如何通过 Python 将数据帧作为 csv 上传到 azure? [英] How can I upload a dataframe as csv to azure by Python?

问题描述

我正在使用 Python 和 Pyspark，我想将 CSV 文件上传到 azure blob 存储.我已经有一个由代码生成的数据框:df.我想做的是下一个:

I am working with Python and Pyspark, and I want to upload a CSV file to an azure blob storage. I have already a dataframe generated by code: df. What I want to do is the next:

# Dataframe generated by code
df

# Create the BlockBlockService that is used to call the Blob service for the storage account
block_blob_service = BlockBlobService(account_name='name', account_key='key') 

container_name ='results-csv'

d = {'one' : pandas.Series([1., 2., 3.], index=['a', 'b', 'c']), 'two' : pandas.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
df = pandas.DataFrame(d)


writer = pandas.ExcelWriter(df, engine='xlsxwriter')

a = df.to_excel(writer, sheet_name='Sheet1', index=False, engine='xlsxwriter')


block_blob_service.create_blob_from_stream(container_name, 'test', a)

我收到错误:

ValueError: stream should not be None.

所以我想将数据帧的内容作为 blob 上传到上面提供的存储位置.有没有办法在不首先在我的本地计算机中生成 CSV 文件的情况下做到这一点?

So I want to upload the content of the dataframe as a blob to the storage location provided above. Is there any way to do that without first generating a CSV file in my local computer?

推荐答案

我们打算做的是使用 dataset.to_csv 函数创建一个文件流，然后将该流发送到 azure blob.对此的替代方法是我们直接将字符串数据集存储到 azure.代码:

What we intent to do is using dataset.to_csv function create a file stream and then send that stream to azure blob. The alternative to this is we directly store the the string dataset to azure. Code :

    blob_client = service.get_blob_client(container=container_name, blob=local_file_name)
    print(str(dataset.to_csv()))
    blob_client.upload_blob(str(dataset.to_csv()))

这会将文件存储到 blob 中.到目前为止，任何其他解决方案都不起作用.问题仍然是现在数据是 blob 不是 csv 格式的那部分我们仍然需要弄清楚.

This will store the file into blob. Any other solution is not working as of now. Still the issue being now the data is blob is not in csv format that part we still need to figure out.

添加了以csv格式发送的代码

Edit : Added the code to send it in csv format

这篇关于如何通过 Python 将数据帧作为 csv 上传到 azure?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！