是否有任何现有API可以通过JAVA读写HDFS? [英] Is there any existing API to read and write to HDFS from JAVA
本文介绍了是否有任何现有API可以通过JAVA读写HDFS?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
是否存在任何存在的HDFS读取/写入API以及如何实现它的最佳实践。下面是代码片段
import org.apache.hadoop.conf.Configuration;
导入org.apache.hadoop.fs.FileSystem;
导入org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;
String Dest =/user/pkumar/test.xml;
配置conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(Dest),conf);
路径路径=新路径(Dest);
if(!fs.exists(path)){
OutputStream out = fs.create(path,new Progressable(){
public void progress(){
System.out.print(。);
}
});
System.out.println();
IOUtils.copyBytes(connect,out,4096,true);
$ b $ p $在你的pom.xml中使用下面的依赖关系
<依赖关系>
< dependency>
< groupId> org.apache.hadoop< / groupId>
< artifactId> hadoop-common< / artifactId>
< version> 2.3.0-cdh5.1.3< / version>
< /依赖关系>
< dependency>
< groupId> org.apache.hadoop< / groupId>
< artifactId> hadoop-client< / artifactId>
< version> 2.3.0-cdh5.1.3< / version>
< /依赖关系>
< dependency>
< groupId> org.apache.httpcomponents< / groupId>
< artifactId> httpcore< / artifactId>
< version> 4.3.2< / version>
< /依赖关系>
< dependency>
< groupId> org.apache.httpcomponents< / groupId>
< artifactId> httpclient< / artifactId>
< version> 4.3.2< / version>
< /依赖关系>
< /依赖关系>
Is there any existing API that exists to read/write from HDFS, along with best practices of how to implement it .
解决方案 Below is the code snippet
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;
String Dest = "/user/pkumar/test.xml";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(Dest),conf);
Path path = new Path(Dest);
if(!fs.exists(path)){
OutputStream out = fs.create(path, new Progressable(){
public void progress(){
System.out.print(".");
}
});
System.out.println();
IOUtils.copyBytes(connect, out, 4096, true);
}
use the below dependencies in your pom.xml
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.2</version>
</dependency>
</dependencies>
这篇关于是否有任何现有API可以通过JAVA读写HDFS?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文