用 Java 读取 HDFS 和本地文件 [英] Reading HDFS and local files in Java

问题描述

我想读取文件路径，而不管它们是 HDFS 还是本地文件.目前，我通过前缀为 file://的本地路径和前缀为 hdfs://的 HDFS 路径并编写如下代码

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the following

Configuration configuration = new Configuration();
FileSystem fileSystem = null;
if (filePath.startsWith("hdfs://")) {
  fileSystem = FileSystem.get(configuration);
} else if (filePath.startsWith("file://")) {
  fileSystem = FileSystem.getLocal(configuration).getRawFileSystem();
}

从这里我使用 FileSystem 的 API 来读取文件.

From here I use the API's of the FileSystem to read the file.

如果有比这更好的方法，你能告诉我吗?

Can you please let me know if there is any other better way than this?

推荐答案

这有意义吗，

public static void main(String[] args) throws IOException {

    Configuration conf = new Configuration();
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));

    BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
    System.out.println("Enter the file path...");
    String filePath = br.readLine();

    Path path = new Path(filePath);
    FileSystem fs = path.getFileSystem(conf);
    FSDataInputStream inputStream = fs.open(path);
    System.out.println(inputStream.available());
    fs.close();
}

如果你走这条路，你就不必进行检查.直接从 Path 获取 FileSystem，然后做任何你想做的事情.

You don't have to put that check if you go this way. Get the FileSystem directly from Path and then do whatever you feel like.

这篇关于用 Java 读取 HDFS 和本地文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！