什么是获得天蓝色斑点存储的最佳方法 [英] what is the best way to get azure blob storage

问题描述

我正在使用scala和spark，需要访问azure blob存储并获取其文件列表.知道Spark版本为2.11的最佳方法是什么.

I'm working with scala and spark and need to access azure blob storage and get its list of files. What is the best way to do that knowing spark version is 2.11.

推荐答案

1. 对于在本地运行的Spark，有一个官方教程了解与wasb和博客有关HDInsight的配置的更多详细信息.
2. 对于在Azure上运行的Spark，不同之处仅在于使用wasb访问HDFS，而其他准备工作则由Azure在 使用Spark创建HDInsight集群.列出文件的方法 是

1. For Spark running on local, there is an official blog which introduces how to access Azure Blob Storage from Spark. The key is that you need to configure Azure Storage account as HDFS-compatible storage in core-site.xml file and add two jars hadoop-azure & azure-storage to your classpath for accessing HDFS via the protocol wasb[s]. You can refer to the official tutorial to know HDFS-compatible storage with wasb, and the blog about configuration for HDInsight more details.
2. For Spark running on Azure, the difference is just only access HDFS with wasb, the other preparations has been done by Azure when creating HDInsight cluster with Spark. The method for listing files is listFiles or wholeTextFiles of SparkContext

这篇关于什么是获得天蓝色斑点存储的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！