如何使用cygwin，hive和hbase在Windows 7 32bit上的c＃中创建和访问hadoop表？ [英] How do I create and access in hadoop tables in c# on windows 7 32bit, using cygwin,hive and hbase?

问题描述

我正在尝试在单集群节点中设置hadoop。我需要在hive和hbase中创建表，以便使用c＃处理表。



我有cygwin，hadoop-1.2.1和hive-1.1.0 on Windows 7 32位。



运行hadoop，它会显示警告：$ HADOOP_HOME已被弃用。仍然可以工作!!



但是当在cygwin上启动hive（bin / hive）时会显示消息Missing Hive CLI Jar并退出。



我的配置设置：



hadoop-env.sh

  export  JAVA_HOME =   C：\\ Java 
 
＃额外的Java CLASSPATH元素。可选的。 
＃ export  HADOOP_CLASSPATH = 
  export  HADOOP_CLASSPATH = $ HIVE_HOME / conf：$ HIVE_HOME / lib 
＃要使用的最大堆数，  MB。默认值为 1000 。 
＃ export  HADOOP_HEAPSIZE =  2000  
 
＃额外的Java运行时选项。由默认清空。 
＃ export  HADOOP_OPTS = -server 
 
＃当指定
时，命令特定选项附加到HADOOP_OPTS  export  HADOOP_NAMENODE_OPTS =    -  Dcom.sun.management.jmxremote $ HADOOP_NAMENODE_OPTS 
  export  HADOOP_SECONDARYNAMENODE_OPTS =    -  Dcom .sun.management.jmxremote $ HADOOP_SECONDARYNAMENODE_OPTS 
  export  HADOOP_DATANODE_OPTS =  < span class =code-string>  -  Dcom.sun.management.jmxremote $ HADOOP_DATANODE_OPTS 
  export  HADOOP_BALANCER_OPTS =    -  Dcom.sun.management.jmxremote $ HADOOP_BALANCER_OPTS 
  export  HADOOP_JOBTRACKER_OPTS =    -  Dcom.sun.management.jmxremote $ HADOOP_JOBTRACKER_OPTS 

/etc/bash.bashrc

 #environment variables 
 
 export JAVA_HOME ='c：\ Java' 
 export CYGWIN_HOME ='c：\ cygwin'
 export HADOOP_HOME ='c：\ cygwin \ home \Administrator \ hadoop-1.2.1'
 export HADOOP_MAPRED_HOME = $ HADOOP_HOME 
 export HADOOP_COMMON_HOME = $ HADOOP_HOME 
 export HADOOP_HDFS_HOME = $ HADOOP_HOME 
 export YARN_HOME = $ HADOOP_HOME 
 export HADOOP_COMMON_LIB = $ HADOOP_HOME / lib 
 export HIVE_HOME ='c：\ cygwin \home \Administrator\hadoop-1.2.1\apache-hive-1.1.0-bin'
 export DERBY_HOME ='c：\ cygwin \home \Administrator \ hadoop-1.2.1 \db-derby-10.4.2.0-bin'
 export SQOOP_HOME ='c：\ cygwin \ home \Administrator \ hadoop-1.2.1 \sqoop-1.4.5.bin__hadoop-1.0.0'
 
 export CLASSPATH = $ CLASSPA TH：C：\cygwin\home\Administrator\hadoop-1.2.1\lib\ * :. 
 export CLASSPATH = $ CLASSPATH：C：\ cygwin \home \Administrator \ hadoop-1.2.1 \ apache-hive-1.1.0-bin \lib \ *：。 
 export PATH = $ JAVA_HOME\\bin：$ CYGWIN_HOME\\bin：$ HADOOP_HOME\\bin：$ HIVE_HOME\\bin：$ SQOOP_HOME\\bin：$ DERBY_HOME \\ \\\bin 

hive-env.sh

＃设置 HADOOP_HOME指向特定的hadoop安装目录
＃ HADOOP_HOME = $ {bin} /../../ hadoop 
  export  HADOOP_HOME = '  C：\ cygwin \home \Administrator \ hadoop-1.2.1' 
 
＃Hive配置目录可以通过以下方式控制： 
  export  HIVE_CONF_DIR = '  c：\ cygwin \home \Administrator\hadoop-1.2.1 \ apache-hive-1.1.0-bin \ conf' 
 
＃包含额外ibraries的文件夹  hive编译/执行可以通过以下方式控制：
＃ export  HIVE_AUX_JARS_PATH = 

mapred-site.xml

 <  配置 >  
 < ;  属性 >  
  <  名称 >  mapred.job.tracker <   / name  >  
 <   value  >  localhost：50001 <   / value  >  
 <   / property  >  
 <   / configuration  >  

core-site.xml

 <   configuration  >  
 <   property  >  
 <  名称 >  fs.default.name <   / name  >  
 <   value  >  hdfs：// localhost：50000 <   / value  >  
 <   / property  >  
 
 <   property  >  
 <   name  < span class =code-keyword>>  hadoop.tmp.dir <   / name  >  
 <   value  >  / cygwin / home / Administrator / hadoop / <   / value  >  
 <   description  > 其他临时目录的基础。 <   / description  >  
 <   / property  >  
 
 <   / configuration  >  

hdfs-site.xml

 <   configuration  >  
 <   property  >  
 <   name  >  dfs.data.dir <   / name  >  
 <   value  >  / home / Administrator / hadoop / datanode <   / value  >  
 <   / property  >  
 <  属性 >  
 <  名称 >  dfs.name.dir <  < span class =code-leadattribute> / name  >  
 <   value  >  / home / Administrator / hadoop / namenode <   / value  >  
 <   / property  >  
 <   property  >  
 <  名称 >  dfs.permissions <   / name  >  
 <   value  >  false <   / value  >  
 <   description  < span class =code-keyword>>  
如果为true，则启用HDFS中的权限检查。 
如果为false，则关闭权限检查，但所有其他行为都保持不变。从一个参数值切换到另一个参数值不会更改模式，所有者或文件或目录组。 
 <   / description  >  
 <   / property  >  
 <   property  > ;  
 <  名称  >  dfs.replication <   / name  >  
 <   value  >  1 <   / value  >  
 <   description  > 默认块复制。 
创建文件时可以指定实际的复制数。如果未在创建时指定复制，则使用默认值。 
 <   / description  >  
 <   / property  >  
 <   / configuration   >  

是否对设置进行了任何更改以及其他更改要做什么？请帮帮我一个出路。



提前致谢

解决方案

HADOOP_HOME已被弃用。它仍然有效!!



但是当在cygwin上启动hive（bin / hive）时会显示消息Missing Hive CLI Jar并退出。



我的配置设置：



hadoop-env.sh

  export  JAVA_HOME =   C：\\ Java 
 
＃额外的Java CLASSPATH元素。可选。
＃ export  HADOOP_CLASSPATH = 
  export  HADOOP_CLASSPATH = 

HIVE_HOME / conf：

< blockquote> HIVE_HOME / lib
＃要使用的最大堆数， MB。默认值为 1000 。
＃ export HADOOP_HE APSIZE = 2000

＃额外的Java运行时选项。由默认清空。
＃ export HADOOP_OPTS = -server

＃当指定
时，命令特定选项附加到HADOOP_OPTS export HADOOP_NAMENODE_OPTS = - Dcom.sun.management.jmxremote

I am trying to set hadoop in single-cluster node. And I need to create tables in hive and hbase inorder to handle the tables using c#.

I have cygwin,hadoop-1.2.1 and hive-1.1.0 on windows 7 32bit.

Running hadoop, it gives "Warning: $HADOOP_HOME is deprecated." still it works!!

But when starting hive (bin/hive) on cygwin gives message "Missing Hive CLI Jar" and exit.

My configuration settings:

hadoop-env.sh

export JAVA_HOME="C:\\Java"

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=
export HADOOP_CLASSPATH=$HIVE_HOME/conf:$HIVE_HOME/lib
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"

/etc/bash.bashrc

#environment variables

export JAVA_HOME='c:\Java'
export CYGWIN_HOME='c:\cygwin'
export HADOOP_HOME='c:\cygwin\home\Administrator\hadoop-1.2.1'
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB=$HADOOP_HOME/lib
export HIVE_HOME='c:\cygwin\home\Administrator\hadoop-1.2.1\apache-hive-1.1.0-bin'
export DERBY_HOME='c:\cygwin\home\Administrator\hadoop-1.2.1\db-derby-10.4.2.0-bin'
export SQOOP_HOME='c:\cygwin\home\Administrator\hadoop-1.2.1\sqoop-1.4.5.bin__hadoop-1.0.0'

export CLASSPATH=$CLASSPATH:C:\cygwin\home\Administrator\hadoop-1.2.1\lib\*:.
export CLASSPATH=$CLASSPATH:C:\cygwin\home\Administrator\hadoop-1.2.1\apache-hive-1.1.0-bin\lib\*:.
export PATH=$JAVA_HOME\\bin:$CYGWIN_HOME\\bin:$HADOOP_HOME\\bin:$HIVE_HOME\\bin:$SQOOP_HOME\\bin:$DERBY_HOME\\bin

hive-env.sh

# Set HADOOP_HOME to point to a specific hadoop install directory
# HADOOP_HOME=${bin}/../../hadoop
export HADOOP_HOME='C:\cygwin\home\Administrator\hadoop-1.2.1'

# Hive Configuration Directory can be controlled by:
 export HIVE_CONF_DIR='c:\cygwin\home\Administrator\hadoop-1.2.1\apache-hive-1.1.0-bin\conf'

# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
# export HIVE_AUX_JARS_PATH=

mapred-site.xml

<configuration>
<property> 
<name>mapred.job.tracker</name> 
<value>localhost:50001</value> 
</property>
</configuration>

core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:50000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/cygwin/home/Administrator/hadoop/</value>
<description>A base for other temporary directories.</description>
</property>

</configuration>

hdfs-site.xml

<configuration>
<property> 
<name>dfs.data.dir</name> 
<value>/home/Administrator/hadoop/datanode</value> 
</property> 
<property> 
<name>dfs.name.dir</name> 
<value>/home/Administrator/hadoop/namenode</value> 
</property> 
<property>
<name>dfs.permissions</name>
<value>false</value>
<description>
If "true", enable permission checking in HDFS.
If "false", permission checking is turned off, but all other behaviour is unchanged. Switching from one parameter value to the other does not change the mode, owner, or group of files or directories.
</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
</description>
</property>
</configuration>

Is there any changes to be made to the settings and what else to be done? Please help me a way out.

Thanks in advance

解决方案

HADOOP_HOME is deprecated." still it works!!

But when starting hive (bin/hive) on cygwin gives message "Missing Hive CLI Jar" and exit.

My configuration settings:

hadoop-env.sh

export JAVA_HOME="C:\\Java"

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=
export HADOOP_CLASSPATH=

HIVE_HOME/conf:

HIVE_HOME/lib # The maximum amount of heap to use, in MB. Default is 1000. # export HADOOP_HEAPSIZE=2000 # Extra Java runtime options. Empty by default. # export HADOOP_OPTS=-server # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote

这篇关于如何使用cygwin，hive和hbase在Windows 7 32bit上的c＃中创建和访问hadoop表？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！