从AWS S3 pyspark读取文件数据 [英] Read file data from aws s3 pyspark

问题描述

我在s3中放置了一个json文件. s3网址类似于以下网址:

I have a json file placed in s3. The s3 url is similar to the below one:

https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json

但是在pyspark中传递相同的内容时，它没有读取文件.

But in pyspark when pass the same, It is not reading the file.

path = "https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json"
df=spark.read.json(path)

但是我可以通过浏览器下载它.

But I am able to download it through browser.

推荐答案

假定dir-resources是存储桶的名称，您应该能够使用以下URI访问文件:

Assuming that dir-resources is the name of your bucket, you should be able to access to the file with the following URI:

path = "s3://dir-resources/sample.json"

在某些情况下，您可能不得不使用s3n协议:

In some cases, you may have to use the s3n protocol instead:

path = "s3n://dir-resources/sample.json"

这篇关于从AWS S3 pyspark读取文件数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！