Hadoop fs - 只复制特定文件 [英] Hadoop fs -get copy only specific files

问题描述

有没有办法只复制特定的文件，比如基于文件类型使用fs -get或fs -copyToLocal？注意：我希望这是递归的，并遍历整个集群。

Is there a way to copy only specific files, say based on file type using fs -get or fs -copyToLocal? Note: I would like this to be recursive and traverse the entire cluster.

想出来，但我无法回答自己的问题。

Figured it out but I can't answer my own question.

以下是我们如何做到的。刚刚写了一个快速的shell脚本。

Here's how we did it. Just wrote a quick shell script.

mkdir /tmp/txt

    for F in `hadoop fs -fs hdfs://namenode.mycluster -lsr / | grep '/*.txt$' | awk '{print $NF}'; 
    do
       hadoop fs -fs hdfs://namenode.mycluster -copyToLocal $F /tmp/las/ 
    done

推荐答案

我们是这么做的。只写了一个快速的shell脚本。

Here's how we did it. Just wrote a quick shell script.

LOCAL_DIR=/tmp/txt
mkdir $LOCAL_DIR

for F in `hadoop fs -fs hdfs://namenode.mycluster -lsr / | grep '/*.txt$' | awk '{print $NF}'; 
do
   hadoop fs -fs hdfs://namenode.mycluster -copyToLocal $F $LOCAL_DIR 
done`

这篇关于Hadoop fs - 只复制特定文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！