将数据从Azure Blob存储传输到HDFS文件系统 [英] Transfer data from azure blob storage to hdfs file system

问题描述

我在镶木地板格式的天蓝色存储blob中有数据.我需要做的是将所有这些存储文件传输到hdfs.有什么办法可以做到吗?

I have data in azure storage blob which is in parquet format. What I need to do is to transfer all those storage files to a hdfs. Is there any way I can do that?

找不到任何有用的方法来做到这一点，

couldn't find any helpful method to do it,

谢谢.

推荐答案

使用@jay的解决方案，我能够使用以下命令传输数据.

using @jay's solution I was able to transfer data using following command.

命令:

hadoop  distcp -D fs.azure.account.key.<account name>.blob.core.windows.net=<Key> wasb://<container>@<account>.blob.core.windows.net<path to wasb file> hdfs://<hdfs path>

distcp递归复制目录结构以获取更多信息，请阅读此

distcp copies directory structure recursively for more info read this link

这篇关于将数据从Azure Blob存储传输到HDFS文件系统的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！