在Django中流式传输一个CSV文件 [英] Streaming a CSV file in Django
问题描述
我正在尝试流式传输一个csv文件作为附件下载。 CSV文件的大小要达到4MB以上,我需要一种方式让用户主动下载文件,而无需等待所有的数据创建并首先提交给内存。
I am attempting to stream a csv file as an attachment download. The CSV files are getting to be 4MB in size or more, and I need a way for the user to actively download the files without waiting for all of the data to be created and committed to memory first.
我首先使用我自己的文件包,基于Django的 FileWrapper
类。那失败了然后我看到一个方法来使用生成器来流传输响应:
如何使用Django流式传输HttpResponse
I first used my own file wrapper based on Django's FileWrapper
class. That failed. Then I saw a method here for using a generator to stream the response:
How to stream an HttpResponse with Django
当我在生成器中引发错误时,我可以看到我正在使用 get_row_data()
函数,但是当我尝试返回响应时它返回空。我还禁用了Django GZipMiddleware
。有没有人知道我在做错什么?
When I raise an error within the generator, I can see that I am creating the proper data with the get_row_data()
function, but when I try to return the response it comes back empty. I've also disabled the Django GZipMiddleware
. Does anyone know what I'm doing wrong?
编辑:我遇到的问题是与 ConditionalGetMiddleware
。我不得不更换它,代码在下面的答案。
The issue I was having was with the ConditionalGetMiddleware
. I had to replace it, the code is in an answer below.
这是视图:
from django.views.decorators.http import condition
@condition(etag_func=None)
def csv_view(request, app_label, model_name):
""" Based on the filters in the query, return a csv file for the given model """
#Get the model
model = models.get_model(app_label, model_name)
#if there are filters in the query
if request.method == 'GET':
#if the query is not empty
if request.META['QUERY_STRING'] != None:
keyword_arg_dict = {}
for key, value in request.GET.items():
#get the query filters
keyword_arg_dict[str(key)] = str(value)
#generate a list of row objects, based on the filters
objects_list = model.objects.filter(**keyword_arg_dict)
else:
#get all the model's objects
objects_list = model.objects.all()
else:
#get all the model's objects
objects_list = model.objects.all()
#create the reponse object with a csv mimetype
response = HttpResponse(
stream_response_generator(model, objects_list),
mimetype='text/plain',
)
response['Content-Disposition'] = "attachment; filename=foo.csv"
return response
这里是用于流式传输的生成器响应:
Here is the generator I use to stream the response:
def stream_response_generator(model, objects_list):
"""Streaming function to return data iteratively """
for row_item in objects_list:
yield get_row_data(model, row_item)
time.sleep(1)
这里是如何创建csv行数据:
And here is how I create the csv row data:
def get_row_data(model, row):
"""Get a row of csv data from an object"""
#Create a temporary csv handle
csv_handle = cStringIO.StringIO()
#create the csv output object
csv_output = csv.writer(csv_handle)
value_list = []
for field in model._meta.fields:
#if the field is a related field (ForeignKey, ManyToMany, OneToOne)
if isinstance(field, RelatedField):
#get the related model from the field object
related_model = field.rel.to
for key in row.__dict__.keys():
#find the field in the row that matches the related field
if key.startswith(field.name):
#Get the unicode version of the row in the related model, based on the id
try:
entry = related_model.objects.get(
id__exact=int(row.__dict__[key]),
)
except:
pass
else:
value = entry.__unicode__().encode("utf-8")
break
#if it isn't a related field
else:
#get the value of the field
if isinstance(row.__dict__[field.name], basestring):
value = row.__dict__[field.name].encode("utf-8")
else:
value = row.__dict__[field.name]
value_list.append(value)
#add the row of csv values to the csv file
csv_output.writerow(value_list)
#Return the string value of the csv output
return csv_handle.getvalue()
推荐答案
这里有一些简单的代码,将流式传输CSV;你可以从这个到你需要做的任何事情:
Here's some simple code that'll stream a CSV; you can probably go from this to whatever you need to do:
import cStringIO as StringIO
import csv
def csv(request):
def data():
for i in xrange(10):
csvfile = StringIO.StringIO()
csvwriter = csv.writer(csvfile)
csvwriter.writerow([i,"a","b","c"])
yield csvfile.getvalue()
response = HttpResponse(data(), mimetype="text/csv")
response["Content-Disposition"] = "attachment; filename=test.csv"
return response
这只是将每行写入一个内存中的文件,读取该行并生成它。
This simply writes each row to an in-memory file, reads the row and yields it.
此版本更有效地生成批量数据,但请务必在使用之前了解上述内容:
This version is more efficient for generating bulk data, but be sure to understand the above before using it:
import cStringIO as StringIO
import csv
def csv(request):
csvfile = StringIO.StringIO()
csvwriter = csv.writer(csvfile)
def read_and_flush():
csvfile.seek(0)
data = csvfile.read()
csvfile.seek(0)
csvfile.truncate()
return data
def data():
for i in xrange(10):
csvwriter.writerow([i,"a","b","c"])
data = read_and_flush()
yield data
response = HttpResponse(data(), mimetype="text/csv")
response["Content-Disposition"] = "attachment; filename=test.csv"
return response
这篇关于在Django中流式传输一个CSV文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!