如何从正在下载csv文件的python请求流加载数据帧? [英] how to load a dataframe from a python requests stream that is downloading a csv file?

问题描述

我想从我将通过流媒体检索的csv文件中创建一个数据框:

I would like to create a dataframe from a csv file that I will retrieve via streaming:

import requests

url = "https://{0}:8443/gateway/default/webhdfs/v1/{1}?op=OPEN".format(host, filepath)

r = requests.get(url, 
                 auth=(username, password), 
                 verify=False, 
                 allow_redirects=True, 
                 stream=True)

chunk_size = 1024
for chunk in r.iter_content(chunk_size):
    # how to load the data

如何将数据从http流加载到spark?

How can the data be loaded into spark from the http stream?

请注意，无法使用HDFS格式检索数据-必须使用WebHDFS.

Note that it isn't possible to use HDFS format for retrieving the data - WebHDFS must be used.

推荐答案

您可以预先生成块边界的RDD，然后使用它来处理worker中的文件.例如:

You can pre-generate the RDD of chunks' boundaries, then use it to process the file inside the worker. For examples:

def process(start, finish):
   // Download file
   // Process downloaded content in range [start, finish)
   // Return a list of item

partition_size = file_size / num_partition
boundaries = [(i, i+paritition_size - 1) for i in range(0, file_size, partition_size)]
rrd = sc.parallelize(boundaries).flatMap(process)
df = sqlContext.createDataFrame(rrd)

这篇关于如何从正在下载csv文件的python请求流加载数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！