使用Java在HDFS中访问文件 [英] Accessing files in HDFS using Java

问题描述

我正在尝试使用Java API访问HDFS中的文件，但每次我都收到File Not Found。我正在使用的代码是： -

I am trying to access a file in the HDFS using Java APIs, but everytime I am getting File Not Found. Code which I am using to access is :-

Configuration conf = new Configuration();
    conf.addResource(FileUtilConstants.ENV_HADOOP_HOME + FileUtilConstants.REL_PATH_CORE_SITE);
    conf.addResource(FileUtilConstants.ENV_HADOOP_HOME + FileUtilConstants.REL_PATH_HDFS_SITE);

    try {
        FileSystem fs = FileSystem.get(conf);
        Path hdfsfilePath = new Path(hdfsPath);
        logger.info("Filesystem URI : " + fs.getUri());
        logger.info("Filesystem Home Directory : " + fs.getHomeDirectory());
        logger.info("Filesystem Working Directory : " + fs.getWorkingDirectory());
        logger.info("HDFS File Path : " + hdfsfilePath);
        if (!fs.exists(hdfsfilePath)) {
            logger.error("File does not exists : " + hdfsPath);
        }

这里是代码的命令行输出。

And here is the command line output from the code.

[root@koversevms ~]# java -jar /tmp/thetus-incendiary-koverse-extension-fileutils-1.0-SNAPSHOT.jar 
13/07/10 02:47:18 INFO fileutils.HadoopFileChecksumUtils: Filesystem URI : file:///
13/07/10 02:47:18 INFO fileutils.HadoopFileChecksumUtils: Filesystem Home Directory : file:/root
13/07/10 02:47:18 INFO fileutils.HadoopFileChecksumUtils: Filesystem Working Directory : file:/root
13/07/10 02:47:18 INFO fileutils.HadoopFileChecksumUtils: HDFS File Path : /usr/hadoop/sample/sample.txt
13/07/10 02:47:18 ERROR fileutils.HadoopFileChecksumUtils: File does not exists : /usr/hadoop/sample/sample.txt

我是hadoop的新手，所以我不知道发生了什么。

I am new to hadoop so I don't know what is going wrong.

谢谢，
Nayan

Thanks, Nayan

推荐答案

这里是代码片段，小时ef =https://stackoverflow.com/questions/15941108/hdfs-access-from-remote-host-through-java-api-user-authentication>这个问题。尽管原始问题的意图不同，它也应该解决你的问题。你的代码中的主要问题是你有从方案（ file：// ）开始的问题。请在您的配置中检查 fs.defaultFS 变量。

Here is code fragment originally posted in context of answer to this question. It should solve your question too despite intention of original question was different. Main point in your code is you have issues starting from scheme (file://). Please check fs.defaultFS variable in your configuration.

package org.myorg;

import java.security.PrivilegedExceptionAction;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;

public class HdfsTest {

    public static void main(String args[]) {

        try {
            UserGroupInformation ugi
                = UserGroupInformation.createRemoteUser("hbase");

            ugi.doAs(new PrivilegedExceptionAction<Void>() {

                public Void run() throws Exception {

                    Configuration conf = new Configuration();
                    conf.set("fs.defaultFS", "hdfs://1.2.3.4:8020/user/hbase");
                    conf.set("hadoop.job.ugi", "hbase");

                    FileSystem fs = FileSystem.get(conf);

                    fs.createNewFile(new Path("/user/hbase/test"));

                    FileStatus[] status = fs.listStatus(new Path("/user/hbase"));
                    for(int i=0;i<status.length;i++){
                        System.out.println(status[i].getPath());
                    }
                    return null;
                }
            });
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

这篇关于使用Java在HDFS中访问文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！