如何使客户端下载动态生成的非常大的文件 [英] How to make the client download a very large file that is genereted on the fly

问题描述

我有一个导出功能，可以读取整个数据库并创建一个包含所有记录的.xls文件.然后将文件发送到客户端.

I have an export function that read the entire database and create a .xls file with all the records. Then the file is sent to the client.

当然，导出完整数据库的时间需要很多时间，并且请求很快会因超时错误而结束.

Of course, the time of export the full database requires a lot of time and the request will soon end in a timeout error.

处理这种情况的最佳解决方案是什么?

What is the best solution to handle this case?

例如，我听说过与Redis进行排队的事情，但这需要两个请求:一个请求启动将生成文件的作业，第二个请求下载生成的文件.

I heard something about making a queue with Redis for example but this will require two requests: one for starting the job that will generate the file and the second to download the generated file.

只有来自客户端的单个请求才有可能吗?

Is this possible with a single request from the client?

推荐答案

Excel导出:

使用流.以下是一个大概的想法:

Use Streams. Following is a rough idea of what might be done:

1. 使用 exceljs模块.因为它具有针对此确切问题的流API .

1. Use exceljs module. Because it has a streaming API aimed towards this exact problem.

var Excel = require('exceljs')

由于我们正在尝试启动下载.编写适当的标题以进行响应.

Since we are trying to initiate a download. Write appropriate headers to response.

res.status(200);
res.setHeader('Content-disposition', 'attachment; filename=db_dump.xls');
res.setHeader('Content-type', 'application/vnd.ms-excel');

创建由 Streaming Excel writer 支持的工作簿.提供给writer的流是服务器响应.

Create a workbook backed by Streaming Excel writer. The stream given to writer is server response.

var options = {
    stream: res, // write to server response
    useStyles: false,
    useSharedStrings: false
};

var workbook = new Excel.stream.xlsx.WorkbookWriter(options);

现在，所有输出流均已设置.对于输入流，更喜欢使用将查询结果/游标作为流提供的数据库驱动程序.

Now, the output streaming flow is all set up. for the input streaming, prefer a DB driver that gives query results/cursor as a stream.

定义一个异步函数，该函数将1个表转储到1个工作表中.

Define an async function that dumps 1 table to 1 worksheet.

var tableToSheet = function (name, done) {
    var str = dbDriver.query('SELECT * FROM ' + name).stream();
    var sheet = workbook.addWorksheet(name);

    str.on('data', function (d) {
        sheet.addRow(d).commit(); // format object if required
    });

    str.on('end', function () {
        sheet.commit();
        done();
    });

    str.on('error', function (err) {
        done(err);
    });
}

现在，使用异步模块的 mapSeries :

Now, lets export some db tables, using async module's mapSeries:

async.mapSeries(['cars','planes','trucks'],tableToSheet,function(err){
   if(err){
     // log error
   }
   res.end();
})

---

CSV导出:

对于单个表/收集模块的CSV导出，可以使用 fast-csv :

For CSV export of a single table/collection module fast-csv can be used:

// response headers as usual
res.status(200);
res.setHeader('Content-disposition', 'attachment; filename=mytable_dump.csv');
res.setHeader('Content-type', 'text/csv');

// create csv stream
var csv = require('fast-csv');
var csvStr = csv.createWriteStream({headers: true});

// open database stream
var dbStr = dbDriver.query('SELECT * from mytable').stream();

// connect the streams
dbStr.pipe(csvStr).pipe(res);

---

您现在正在将数据从数据库流式传输到HTTP响应，然后将其即时转换为xls/csv格式.无需将全部数据缓冲或存储在内存或文件中.

---

You are now streaming data from DB to HTTP response, converting it into xls/csv format on the fly. No need to buffer or store the entire data in memory or in a file.

这篇关于如何使客户端下载动态生成的非常大的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！