用Java读取HDFS和本地文件 [英] Reading HDFS and local files in Java

问题描述

无论是HDFS还是本地文件，我都希望读取文件路径。目前，我使用前缀file：//和带有前缀hdfs：//的HDFS路径传递本地路径，并编写如下代码：

  Configuration configuration = new Configuration（）; 
 FileSystem fileSystem = null; 
 if（filePath.startsWith（hdfs：//））{
 fileSystem = FileSystem.get（configuration）; 
} else if（filePath.startsWith（file：//））{
 fileSystem = FileSystem.getLocal（configuration）.getRawFileSystem（）; 
 
 $ / code> 

从这里我使用FileSystem的API读取文件。

您能否让我知道是否还有其他更好的方法比这个更好？

解决方案

这是否有意义，

  public static void main（String [] args）throws IOException {
 
 Configuration conf = new Configuration（）; 
 conf.addResource（new Path（/ hadoop / projects / hadoop-1.0.4 / conf / core-site.xml））; 
 conf.addResource（new Path（/ hadoop / projects / hadoop-1.0.4 / conf / hdfs-site.xml））; 
 
 BufferedReader br = new BufferedReader（new InputStreamReader（System.in））; 
 System.out.println（输入文件路径...）; 
 String filePath = br.readLine（）; 
 
路径路径=新路径（filePath）; 
 FileSystem fs = path.getFileSystem（conf）; 
 FSDataInputStream inputStream = fs.open（path）; 
 System.out.println（inputStream.available（））; 
 fs.close（）; 
} 
  

如果你这样做的话，你不必做那个检查。直接从Path获得FileSystem，然后做任何你想做的事。

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the following

Configuration configuration = new Configuration();
FileSystem fileSystem = null;
if (filePath.startsWith("hdfs://")) {
  fileSystem = FileSystem.get(configuration);
} else if (filePath.startsWith("file://")) {
  fileSystem = FileSystem.getLocal(configuration).getRawFileSystem();
}

From here I use the API's of the FileSystem to read the file.

Can you please let me know if there is any other better way than this?

解决方案

Does this make sense,

public static void main(String[] args) throws IOException {

    Configuration conf = new Configuration();
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));

    BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
    System.out.println("Enter the file path...");
    String filePath = br.readLine();

    Path path = new Path(filePath);
    FileSystem fs = path.getFileSystem(conf);
    FSDataInputStream inputStream = fs.open(path);
    System.out.println(inputStream.available());
    fs.close();
}

You don't have to put that check if you go this way. Get the FileSystem directly from Path and then do whatever you feel like.

这篇关于用Java读取HDFS和本地文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！