在Django中流式传输CSV文件 [英] Streaming a CSV file in Django

问题描述

我试图将csv文件作为附件下载进行流式传输。 CSV文件的大小为4MB或更大，我需要一种方式让用户主动下载文件，而不必等待所有的数据被创建并提交到内存。

I am attempting to stream a csv file as an attachment download. The CSV files are getting to be 4MB in size or more, and I need a way for the user to actively download the files without waiting for all of the data to be created and committed to memory first.

我首先使用我自己的文件包装器基于Django的 FileWrapper 类。失败了。然后我在这里看到一个方法使用生成器流式响应：
如何使用Django流式传输HttpResponse

I first used my own file wrapper based on Django's FileWrapper class. That failed. Then I saw a method here for using a generator to stream the response: How to stream an HttpResponse with Django

当我在生成器中引发错误时，可以看到我正在创建正确的数据 get_row_data（）函数，但是当我尝试返回响应时，它返回空。我也禁用了Django GZipMiddleware 。有人知道我在做什么吗？

When I raise an error within the generator, I can see that I am creating the proper data with the get_row_data() function, but when I try to return the response it comes back empty. I've also disabled the Django GZipMiddleware. Does anyone know what I'm doing wrong?

编辑：我遇到的问题是与 ConditionalGetMiddleware 。

The issue I was having was with the ConditionalGetMiddleware. I had to replace it, the code is in an answer below.

以下是视图：

from django.views.decorators.http import condition

@condition(etag_func=None)
def csv_view(request, app_label, model_name):
    """ Based on the filters in the query, return a csv file for the given model """

    #Get the model
    model = models.get_model(app_label, model_name)

    #if there are filters in the query
    if request.method == 'GET':
        #if the query is not empty
        if request.META['QUERY_STRING'] != None:
            keyword_arg_dict = {}
            for key, value in request.GET.items():
                #get the query filters
                keyword_arg_dict[str(key)] = str(value)
            #generate a list of row objects, based on the filters
            objects_list = model.objects.filter(**keyword_arg_dict)
        else:
            #get all the model's objects
            objects_list = model.objects.all()
    else:
        #get all the model's objects
        objects_list = model.objects.all()
    #create the reponse object with a csv mimetype
    response = HttpResponse(
        stream_response_generator(model, objects_list),
        mimetype='text/plain',
        )
    response['Content-Disposition'] = "attachment; filename=foo.csv"
    return response

响应：

def stream_response_generator(model, objects_list):
    """Streaming function to return data iteratively """
    for row_item in objects_list:
        yield get_row_data(model, row_item)
        time.sleep(1)

下面是我如何创建csv行数据：

And here is how I create the csv row data:

def get_row_data(model, row):
    """Get a row of csv data from an object"""
    #Create a temporary csv handle
    csv_handle = cStringIO.StringIO()
    #create the csv output object
    csv_output = csv.writer(csv_handle)
    value_list = [] 
    for field in model._meta.fields:
        #if the field is a related field (ForeignKey, ManyToMany, OneToOne)
        if isinstance(field, RelatedField):
            #get the related model from the field object
            related_model = field.rel.to
            for key in row.__dict__.keys():
                #find the field in the row that matches the related field
                if key.startswith(field.name):
                    #Get the unicode version of the row in the related model, based on the id
                    try:
                        entry = related_model.objects.get(
                            id__exact=int(row.__dict__[key]),
                            )
                    except:
                        pass
                    else:
                        value = entry.__unicode__().encode("utf-8")
                        break
        #if it isn't a related field
        else:
            #get the value of the field
            if isinstance(row.__dict__[field.name], basestring):
                value = row.__dict__[field.name].encode("utf-8")
            else:
                value = row.__dict__[field.name]
        value_list.append(value)
    #add the row of csv values to the csv file
    csv_output.writerow(value_list)
    #Return the string value of the csv output
    return csv_handle.getvalue()

推荐答案

这里有一些简单的代码，你可以从这里到任何你需要做的：

Here's some simple code that'll stream a CSV; you can probably go from this to whatever you need to do:

import cStringIO as StringIO
import csv

def csv(request):
    def data():
        for i in xrange(10):
            csvfile = StringIO.StringIO()
            csvwriter = csv.writer(csvfile)
            csvwriter.writerow([i,"a","b","c"])
            yield csvfile.getvalue()

    response = HttpResponse(data(), mimetype="text/csv")
    response["Content-Disposition"] = "attachment; filename=test.csv"
    return response

这只是将每行写入一个内存文件，读取该行并生成它。

This simply writes each row to an in-memory file, reads the row and yields it.

此版本对于生成批量数据更有效，但务必在使用之前了解上述内容：

This version is more efficient for generating bulk data, but be sure to understand the above before using it:

import cStringIO as StringIO
import csv

def csv(request):
    csvfile = StringIO.StringIO()
    csvwriter = csv.writer(csvfile)

    def read_and_flush():
        csvfile.seek(0)
        data = csvfile.read()
        csvfile.seek(0)
        csvfile.truncate()
        return data

    def data():
        for i in xrange(10):
            csvwriter.writerow([i,"a","b","c"])
        data = read_and_flush()
        yield data

    response = HttpResponse(data(), mimetype="text/csv")
    response["Content-Disposition"] = "attachment; filename=test.csv"
    return response

这篇关于在Django中流式传输CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！