从HDFS读取数据 - 我的程序找不到路径 [英] Reading data from HDFS - my program can't find the path

问题描述

我试图从HDFS读取文件的内容。我的代码如下 -

  package gen; 
 import java.io.BufferedReader; 
 import java.io.IOException; 
 import java.io.InputStreamReader; 
 
导入org.apache.hadoop.conf.Configuration; 
 import org.apache.hadoop.fs.FileSystem; 
导入org.apache.hadoop.fs.Path; 
 
 public class ReadFromHDFS {
 
 public static void main（String [] args）throws Exception {
 
 if（args.length <1） {
 System.out.println（Usage：ReadFromHDFS< hdfs-file-path-to-read-from>）; 
 System.out.println（示例：ReadFromHDFS'hdfs：/ localhost：9000 / myFirstSelfWriteFile'）; 
 System.exit（-1）; 
} 
 
 try {
 Path path = new Path（args [0]）; 
 FileSystem fileSystem = FileSystem.get（new Configuration（））; 
 BufferedReader bufferedReader = new BufferedReader（new InputStreamReader（fileSystem.open（path）））; 
 String line = bufferedReader.readLine（）; 
 while（line！= null）{
 System.out.println（line）; 
 line = bufferedReader.readLine（）; 
} 
} catch（IOException e）{
 e.printStackTrace（）; 
 
 
 
 $ / code $ / pre 
 
 
然而，我不能弄清楚如何让这个程序成为我的HDFS目录的路径。我已经试过了 -  
  java -cp< hadoop jar：myjar> gen.ReadFromHDFS< path> 
  
其中有路径，我尝试直接引用目录（当我做hadoop fs -ls时， ，目录中的文件，添加hdfs：/本地主机，hdfs：/和他们都没有工作。任何人都可以帮助我如何将我的文件夹的路径传递给HDFS？例如，当我直接给路径（不带前缀）时，它说文件不存在。
 
 
 
编辑：目前为止，这些解决方案似乎都不适用我。我总是得到异常 -  
  java.io.FileNotFoundExceptoin：File< filename>不存在。 
 at org.apache.hadoop.fs.getFileSystem.getFileStatus（RawLocalFileSystem.java:361）
  
 
 
 <看起来是试图在本地查找文件。
 
解决方案
尝试
  FileSystem fileSystem = FileSystem.get（new Configuration（））; 
 Path path = new Path（fileSystem.getName（）+/+ args [0]）; 
 BufferedReader bufferedReader = new BufferedReader（new InputStreamReader（fileSystem.open（path）））; 
 String line = bufferedReader.readLine（）; 
  
并将HDFS中的文件路径设为（无前缀）
 
 
 / myFirstSelfWriteFile
  
不要包括hdfs：/ localhost
I'm trying to read the contents of a file from HDFS. My code is below - 
package gen;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class ReadFromHDFS {

 public static void main(String[] args) throws Exception {

  if (args.length < 1) {
   System.out.println("Usage: ReadFromHDFS <hdfs-file-path-to-read-from>");
   System.out.println("Example: ReadFromHDFS 'hdfs:/localhost:9000/myFirstSelfWriteFile'");
   System.exit(-1);
  } 

  try {
   Path path = new Path(args[0]);
   FileSystem fileSystem = FileSystem.get(new Configuration());
   BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fileSystem.open(path)));
   String line = bufferedReader.readLine();
   while (line != null) {
    System.out.println(line);
    line = bufferedReader.readLine();
   }
  } catch (IOException e) {
   e.printStackTrace();
  }
 }
}
However, I can't figure out how to give this program the path to my HDFS directory. I have tried - 
java -cp <hadoop jar:myjar> gen.ReadFromHDFS <path>
where with path I tried referencing the directory directly (what I see when I do hadoop fs -ls), the file inside the directory, adding hdfs:/localhost, hdfs:/ and none of them work. Can any one help me with how exactly I should pass the path of my folder to HDFS? For example, when I give the path directly (with no prefix) it says that the file does not exist.

Edit: None of the solutions so far seem to work for me. I always get the exception - 
  java.io.FileNotFoundExceptoin: File <filename> does not exist.
  at org.apache.hadoop.fs.getFileSystem.getFileStatus(RawLocalFileSystem.java:361)
It seems to be trying to find the file locally.
解决方案
try 
FileSystem fileSystem = FileSystem.get(new Configuration());
Path path = new Path(fileSystem.getName() + "/" + args[0]);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fileSystem.open(path)));
String line = bufferedReader.readLine();
and give file path in HDFS as (with no prefix)
"/myFirstSelfWriteFile"
do not include "hdfs:/localhost"

                        
这篇关于从HDFS读取数据 - 我的程序找不到路径的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！