Node.js流/管道错误处理(错误时更改响应状态) [英] Node.js Streaming/Piping Error Handling (Change Response Status on Error)

问题描述

我在Cassandra数据库中有数百万行，我想通过一个zip文件(不希望在内存中有潜在的巨大zip文件)流传输到客户端.我正在使用Cassandra-Node驱动程序中的stream()函数，将其传递给Transformer，该Transformer从我关心的每一行中提取一个字段并追加换行符，并通过管道将哪些管道归档到Express Response对象.这似乎工作正常，但我不知道如何在流式传输期间正确处理错误.我必须在为客户端进行流传输之前设置适当的标头/状态，但是如果在流传输期间发生错误，例如在dbStream上，我想清理所有管道并将响应状态重置为类似404的状态但是，如果在设置标头并开始流传输后尝试重置状态，则会收到发送标头后无法设置标头.我四处张望，找不到在管道处理/流式传输到Response对象时如何正确处理Node中的错误.如果我无法在错误时发送正确的响应代码，客户端如何判断有效数据是否真的在流传输?有人可以帮忙吗?

I have millions of rows in my Cassandra db that I want to stream to the client in a zip file (don't want a potentially huge zip file in memory). I am using the stream() function from the Cassandra-Node driver, piping to a Transformer which extracts the one field from each row that I care about and appends a newline, and pipes to archive which pipes to the Express Response object. This seems to work fine but I can't figure out how to properly handle errors during streaming. I have to set the appropriate headers/status before streaming for the client, but if there is an error during the streaming, on the dbStream for example, I want to clean up all of the pipes and reset the response status to be something like 404. But If I try to reset the status after the headers are set and the streaming starts, I get Can't set headers after they are sent. I've looked all over and can't find how to properly handle errors in Node when piping/streaming to the Response object. How can the client tell if valid data was actually streamed if I can't send a proper response code on error? Can anyone help?

function streamNamesToWriteStream(query, res, options) {
  return new Promise((resolve, reject) => {

    let success = true;

    const dbStream = db.client.stream(query);
    const rowTransformer = new Transform({
      objectMode: true,
      transform(row, encoding, callback) {
        try {
          const vote = row.name + '\n';
          callback(null, vote);
        } catch (err) {
          callback(null, err.message  + '\n');
        }
      }
    });

    // Handle res events
    res.on('error', (err) => {
      logger.error(`res ${res} error`);
      return reject(err);
    });

    dbStream.on('error', function(err) {
      res.status(404).send() // Can't set headers after they are sent.
      logger.debug(`dbStream error: ${err}`);
      success = false;
      //res.end();
      //return reject(err);
    });

    res.writeHead(200, {
      'Content-Type': 'application/zip',
      'Content-disposition': 'attachment; filename=myFile.zip'
    });

    const archive = archiver.create('zip');
    archive.on('error', function(err) { throw err; });
    archive.on('end', function(err) {
      logger.debug(`Archive done`);
      //res.status(404).end()
    });

    archive.pipe(res, {
      //end:false
    });
    archive.append(dbStream.pipe(rowTransformer), { name: 'file1.txt' });
    archive.append(dbStream.pipe(rowTransformer), { name: 'file1.txt' });
    archive.finalize();
  });
}

推荐答案

显然，更改标头为时已晚，因此必须有应用程序逻辑来检测问题.这是我的一些想法:

Obviously it's too late to change the headers, so there's going to have to be application logic to detect a problem. Here's some ideas I have:

1. 在发生错误时，在流的末尾编写明确的前哨.然后，zip文件的使用者需要寻找该值以检查是否存在问题.

1. Write an unambiguous sentinel of some kind at the end of the stream when an error occurs. The consumer of the zip file will then need to look for that value to check for a problem.

也许更简单些，让使用者对zip归档文件的完整性进行验证.大概如果流失败，则zip将会损坏.

Perhaps more simply, have the consumer execute a verification on the integrity of the zip archive. Presumably if the stream fails the zip will be corrupted.

这篇关于Node.js流/管道错误处理(错误时更改响应状态)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！