将文件从Hdfs复制到Hdfs Scala [英] Copy file from Hdfs to Hdfs scala

问题描述

是否存在使用Hadoop API/Spark Scala在Hdfs上将文件从一个目录复制到另一个目录的已知方法?

Is there a known way using Hadoop api / spark scala to copy files from one directory to another on Hdfs ?

我尝试使用copyFromLocalFile，但没有帮助

I have tried using copyFromLocalFile but was not helpful

推荐答案

尝试使用Hadoop的 FileUtil.copy()命令，如下所述:

Try Hadoop's FileUtil.copy() command, as described here: https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/fs/FileUtil.html#copy(org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration)

val conf = new org.apache.hadoop.conf.Configuration()
val srcPath = new org.apache.hadoop.fs.Path("hdfs://my/src/path")
val dstPath = new org.apache.hadoop.fs.Path("hdfs://my/dst/path")

org.apache.hadoop.fs.FileUtil.copy(
srcPath.getFileSystem(conf), 
srcPath, 
dstPath.getFileSystem(conf), 
dstPath, 
true, 
conf
) 

这篇关于将文件从Hdfs复制到Hdfs Scala的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！