如何连接到远程HDFS [英] How to connect to remote HDFS

问题描述

我在Windows机器上运行eclipse，并且HDFS在Unix机器上运行。

我试图连接到运行在远程机器上的HDFS实例。 。这是我已经尝试过的。

  Configuration conf = new Configuration（）; 
 conf.set（fs.defaultFS，hdfs：// remoteHostName：portNumber）; 
 DFSClient client = null; 
 System.out.println（try）; 
尝试
 {
 System.out.println（trying）; 
 client = new DFSClient（conf）; 
 
 System.out.println（client）; 
 
 catch（IOException e）{
 
 e.printStackTrace（）; 
} 
 
 finally {
 if（client！= null）
 try {
 client.close（）; 
} catch（IOException e）{
 
 e.printStackTrace（）; 
} 
 
 
} 
  

我有以下例外：

 线程main中的异常java.lang.NoSuchMethodError：org.apache.hadoop.ipc.RPC。 getProxy（Ljava /郎/类; JLjava /净/的InetSocketAddress; Lorg /阿帕奇/ hadoop的/安全/ UserGroupInformation; Lorg /阿帕奇/ hadoop的/ CONF /配置; Ljavax /净/的SocketFactory; ILorg /阿帕奇/ hadoop的/ IO /重试/ RetryPolicy; Z）Lorg /阿帕奇/ hadoop的/ IPC / VersionedProtocol; 
 at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode（DFSClient.java:135）
 at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:280）
 at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:245）
 at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:235 ）
 at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:226）
  

顺便说一下，我从远程机器上的hdfs-site.xml获得了portNumber。

这种方法是否正确？

另外，在Python中执行此操作会更容易吗？

编辑

请注意，我的窗口上解压缩了Hadoop二进制文件，并且相应地设置了HADOOP_HOME环境变量。这可能是导致问题的原因吗？

解决方案

请参阅： Hadoop 2.6.0 Browsing filesystem Java for your specific problem。

除此之外，您可能需要考虑使用REST进行远程交互。 Apache Knox可以为您提供对远程集群的访问，并防止您的代码不必知道集群内部组件，例如host：port，kerberos或不等等。这些事情可能会从您的远程客户端下改变。

i am trying to connect to an HDFS instance running on a remote machine.

I am running eclipse on a windows machine and the HDFS is running on a Unix box. Here is what i have tried

         Configuration conf = new Configuration();
         conf.set("fs.defaultFS", "hdfs://remoteHostName:portNumber");
         DFSClient client = null;
         System.out.println("try");
         try 
         {
             System.out.println("trying");   
             client = new DFSClient(conf);

        System.out.println(client);
         } 
         catch (IOException e) {

             e.printStackTrace();
        }

         finally {
             if(client!=null)
                 try {
                    client.close();
                } catch (IOException e) {

                    e.printStackTrace();
                }


         }

but this gives me the following exception

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;ILorg/apache/hadoop/io/retry/RetryPolicy;Z)Lorg/apache/hadoop/ipc/VersionedProtocol;
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:135)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:280)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:235)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:226)

by the way, i got the portNumber from the hdfs-site.xml on the remote machine

Is this approach correct?

Also, would it be easier to do this in Python?

EDIT

Note that i do have the Hadoop binaries unzipped on my windows and i have set the HADOOP_HOME environment variable accordingly. Could this be causing a problem?

解决方案

See: Hadoop 2.6.0 Browsing filesystem Java for your specific problem.

Beyond that, you might want to consider using REST for remote interactions. Apache Knox can provide you with access to the remote cluster and shield your code from having to know cluster internals such as host:port, kerberos or not, etc. These things can change out from under your remote clients.

这篇关于如何连接到远程HDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！