使用JAVA将文件从HDFS复制到本地文件系统 [英] Copying files from HDFS to local file system with JAVA

问题描述

我试图将文件从HDFS复制到本地文件系统进行预处理。下面的代码应该根据文档工作。尽管它不会提供任何错误消息，并且mapreduce作业运行平稳，但在本地硬盘上看不到任何输出。你认为问题是什么？谢谢。

  try {
 Path phdfs_input = new Path（hdfs：// master：54310 / user / hduser / conninput /+ value.toString（））; 
路径plocal_input = new Path（/ home / hduser / Desktop /+ avlue.toString（））; 
 FileSystem fs = FileSystem.get（context.getConfiguration（））; 
 fs.copyToLocalFile（phdfs_input，plocal_input）; 
 $ b / * String localoutput_file =/ home / hduser / Destop / output /+ value.toString（）; 
 String cmd1 [] = {mafia，-mfi，.5，-ascii，〜/ Desktop /+ value.toString（），localoutput_file}; 
文件mafia_dir =新文件（/ home / hduser /）; 
 ShellCommandExecutor s = new ShellCommandExecutor（cmd1，mafia_dir）; * / 
} catch（Exception e）{
 e.printStackTrace（）; 
 
  

解决方案

尝试使用<$ c $在路径构造函数中使用c> /user/hduser/conninput/+ value.toString（），而不是提供 master：54310 部分，它应该从配置中找出 master：54310 / p>

I am trying to copy files from HDFS to local filesystem for preprocessing. The below code should work according to the documentation. Although it doesn't give any error messages and the mapreduce job runs smoothly I can not see any output on my local hard drive. What do you think the problem is? Thanks.

        try {
            Path phdfs_input = new Path("hdfs://master:54310/user/hduser/conninput/"+value.toString());
            Path plocal_input = new Path("/home/hduser/Desktop/"+avlue.toString());
            FileSystem fs = FileSystem.get(context.getConfiguration());
            fs.copyToLocalFile(phdfs_input, plocal_input);

           /* String localoutput_file = "/home/hduser/Destop/output/"+value.toString();
            String cmd1[] = {"mafia", "-mfi", ".5", "-ascii", "~/Desktop/"+value.toString(), localoutput_file };
            File mafia_dir = new File("/home/hduser/");
            ShellCommandExecutor s = new ShellCommandExecutor(cmd1, mafia_dir);*/
        } catch (Exception e) {
            e.printStackTrace();
        }

解决方案

Try using /user/hduser/conninput/"+value.toString() in the Path constructor instead of providing the master:54310 part. It should figure out master:54310 from the Configuration.

这篇关于使用JAVA将文件从HDFS复制到本地文件系统的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！