pyspark写入wasb blob存储容器 [英] pyspark write to wasb blob storage container
问题描述
我正在运行Ubuntu实例，以使用N系列实例运行azure的计算.计算完成后，我尝试使用wasb之类的wasb来写入Azure blob容器
I am running a Ubuntu instance to run a calculation of azure using a N-series instance. After the calculation I try to write to a Azure blob container using the wasb like URL
wasb://containername/path
我正在尝试使用pyspark命令
I am trying to use the pyspark command
sparkSession.write.save('wasb://containername/path', format='json', mode='append')
但是我从Spark那里收到一个Java io异常，说它不支持wasb文件系统.我想知道是否有人不使用HDInsight实例就知道如何写入wasb地址?
But I receive a Java io exception from spark saying it doesn't support a wasb file system. I was wondering if anyone knows how to write to a wasb address while not using a HDInsight instance?
推荐答案
我还没有使用 pyspark 完成它，但是这是我使用scala和spark的方法.
I haven't done it with the pyspark but here is how I did using scala and spark.
在sbt中添加依赖项
"org.apache.hadoop" % "hadoop-azure" % "2.7.3"
然后定义要在基础 Hadoop 配置中使用的文件系统.
Then define the file system to be used in the underlying Hadoop configurations.
val spark = SparkSession.builder().appName("read azure storage").master("local[*]").getOrCreate()
spark.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
spark.sparkContext.hadoopConfiguration.set("fs.azure.account.key.yourAccount.blob.core.windows.net", "yourKey ")
val baseDir = "wasb[s]://BlobStorageContainer@yourUser.blob.core.windows.net/"
现在将数据框写入Blob容器
Now write the dataframe to blob container
resultDF.write.mode(SaveMode.Append).json(baseDir + outputPath)
希望这很有帮助这里是工作程序
Hope this is helpful here was the working program
这篇关于pyspark写入wasb blob存储容器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
