我如何通过java的hdfs协议访问hadoop？ [英] How can I access hadoop via the hdfs protocol from java?

问题描述

  uri = 
我发现了一种通过hftp连接到hadoop的方法，并且它工作正常HFTP：//172.16.xxx.xxx：50070 /; 
 
 System.out.println（uri：+ uri）; 
 Configuration conf = new Configuration（）; 
 
 FileSystem fs = FileSystem.get（URI.create（uri），conf）; 
 fs.printStatistics（）; 
  

然而，我想读/写以及复制文件，也就是我想连接超过hdfs。如何启用hdfs连接，以便我可以编辑实际的远程文件系统？

我尝试将上述协议从hftp - >hdfs更改，但我得到以下异常...

（原谅我对url协议和hadoop的了解不足，我认为这是一个有点奇怪的问题，我会问，但任何帮助真的很感谢！）

< blockquote>

线程main中的异常java.io.IOException：调用
/172.16.112.131:50070失败，出现本地异常：java.io.EOFException在组织中出现
。
中的apache.hadoop.ipc.Client.wrapException（Client.java:1139）org.apache.hadoop.ipc.Client.call（Client.java:1107）at
org.apache.hadoop。 ipc.RPC $ Invoker.invoke（RPC.java:226）at
$ Proxy0.getProtocolVersion（Unknown Source）at
org.apache.hadoop.ipc.RPC.getProxy（RPC.java:398）
org.apache.hadoop.ipc.RPC.getProxy（RPC.java:384）at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode（DFSClient.java:111）
在org.apache.hadoop.hdfs.DFSClient。（DFSClient.java:213）at
org.apache.hadoop.hdfs.DFSClient。（DFSClient.java:180）at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize（DistributedFileSystem.java:89）
at
org.apache.hadoop.fs.FileSystem.access
org.apache.hadoop.fs.FileSystem.createFileSystem（FileSystem.java:1514）
（FileSystem.java:67）at
org.apache.hadoop.fs.FileSystem $ Cache.getInternal（FileSystem.java:1548）
at org.apache.hadoop.fs.FileSystem $ Cache.get（FileSystem.java:1530）
at org.apache.hadoop.fs.FileSystem.get（FileSystem.java:228）at
sb.HadoopRemote.main（HadoopRemote.java:24）

$ b $关于hadoop：您需要确保core-site.xml namenode条目在您的服务器上为0.0.0.0而不是127.0.0.1（本地主机）提供服务hadoop配置。重要的是，出于某种原因，clouderas vm distro默认为localhost。

I found a way to connect to hadoop via hftp, and it works fine, (read only) :

    uri = "hftp://172.16.xxx.xxx:50070/";

    System.out.println( "uri: " + uri );           
    Configuration conf = new Configuration();

    FileSystem fs = FileSystem.get( URI.create( uri ), conf );
    fs.printStatistics();

However, I want to read/write as well as copy files, that is, I want to connect over hdfs . How can I enable hdfs connections so that i can edit the actual , remote filesystem ?

I tried to change the protocol above from "hftp" -> "hdfs", but I got the following exception ...

(forgive my poor knowledge of url protocols and hadoop , I assume this is a somewhat strange question im asking, but any help would really be appreciated !)

Exception in thread "main" java.io.IOException: Call to /172.16.112.131:50070 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1139) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:213) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:180) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228) at sb.HadoopRemote.main(HadoopRemote.java:24)

解决方案

Regarding hadoop : You need to make sure the core-site.xml namenode entry is serving to 0.0.0.0 instead of 127.0.0.1(localhost) in your hadoop configuration. Importantly, clouderas vm distro defaults to localhost for some reason.

这篇关于我如何通过java的hdfs协议访问hadoop？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！