节点JS-将数据从Busboy流到AWS S3 [英] Node JS - Stream data from Busboy to AWS S3

问题描述

我正在尝试通过ec2将文件上传到s3。
我的第一个方法是-将文件完全上传到ec2，然后将该文件上传到s3。这种方法不好，因为从ec2到s3的传输时间很浪费时间。

I am trying to upload a file to s3 via ec2. My first approach was - upload file to ec2 completely and then upload that file to s3. This approach is not good because transfer time from ec2 to s3 is waste of time.

当前，我正在尝试使用 busboy上传流来 s3上传流，因此，作为s3 upload 方法支持流作为上传正文，将同时完成ec2和ec2到s3的上传。

Currently I am trying to use busboy upload stream to s3 upload stream so that uploading to ec2 and then ec2 to s3 will be done simultaneously as s3 "upload" method support stream as upload Body.

这是我的代码-

router.post('/s3StreamUpload', function(req, res, next) {
   var busboy = new Busboy({headers: req.headers});
   busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
      console.log('Before Upload: ' + new Date());
      console.log('File [' + fieldname + ']: filename: ' + filename + ', encoding: ' + encoding + ', mimetype: ' + mimetype);

      var s3 = new AWS.S3({
         params: {Bucket: 'sswa', Key: filename, Body: file},
         options: {partSize: 5 * 1024 * 1024, queueSize: 10}   // 5 MB
      });
      s3.upload().on('httpUploadProgress', function (evt) {
         console.log(evt);
      }).send(function (err, data) {
         console.log('After Upload: ' + new Date());
         console.log(err, data);
      });
   });
   busboy.on('field', function(fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) {
      console.log('Field [' + fieldname + ']: value: ' + inspect(val));
   });
   busboy.on('finish', function() {
      console.log('Done parsing form!');
      res.writeHead(303, { Connection: 'close', Location: '/' });
      res.end();
   });
   req.pipe(busboy);
});

我怀疑它是否真的同时作为流上传到s3？这种方法有什么弊端吗？

I have doubt is it really uploading to s3 simultaneously as stream ? Is there any drawbacks of this approach ?

推荐答案

要测试向S3的多部分流上传是否正常工作，我在执行的三个时间点进行了时间记录-

To test whether multi-part streaming upload to S3 is working or not, I took time log at three points of execution -

1. 从客户端开始上传之前（ uploadStartTime ）


2. 上传之后到EC2（ busboyFinishTime ）


3. 转移到S3之后（ s3UploadFinishTime ）

1. Before start upload from client (uploadStartTime)
2. After uploaded to EC2 (busboyFinishTime)
3. After transferred to S3 (s3UploadFinishTime)

然后我从EC2运行。在上传了各种长度的视频文件（36.1 MB，33.3 MB，52.5 MB）之后，我观察到，每上传5个（我定义的）EC2，就会立即将部分传输到S3。将零件上传到S3时，您会看到以下行的日志。它将显示文件零件上传进度以及零件编号。

Then I run from EC2. After uploading various length of video files (36.1 MB, 33.3 MB, 52.5 MB) I observed that parts are transferred to S3 immediately for each 5MB (as I defined) uploaded to EC2. When uploading parts to S3 you will see a log of the following line. It will show file part upload progress with the part number.

console.log(evt);

对于所有三个上载 busboyFinishTime 和 s3UploadFinishTime 相同或几乎没有1秒的差异。

For all three uploads busboyFinishTime and s3UploadFinishTime are same or there is hardly a 1-second difference.

示例：
上传52.5 MB时

Example: When 52.5 MB uploaded

{
  "uploadStartTime": "2016-04-28T14:19:51.365Z",
  "busboyFinishTime": "2016-04-28T14:22:26.292Z",
  "s3UploadFinishTime": "2016-04-28T14:22:26.558Z"
}

完整代码：

router.post('/s3StreamUpload', function(req, res, next) {
   var busboy = new Busboy({headers: req.headers});
   var uploadStartTime = new Date(),
      busboyFinishTime = null,
      s3UploadFinishTime = null;

   busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
      console.log('File [' + fieldname + ']: filename: ' + filename + ', encoding: ' + encoding + ', mimetype: ' + mimetype);

      var s3 = new AWS.S3({
         params: {Bucket: 'sswa', Key: filename, Body: file},
         options: {partSize: 5 * 1024 * 1024, queueSize: 10}   // 5 MB
      });
      s3.upload().on('httpUploadProgress', function (evt) {
         console.log(evt);
      }).send(function (err, data) {
         s3UploadFinishTime = new Date();
         if(busboyFinishTime && s3UploadFinishTime) {
            res.json({
               uploadStartTime: uploadStartTime,
               busboyFinishTime: busboyFinishTime,
               s3UploadFinishTime: s3UploadFinishTime
            });
         }
         console.log(err, data);
      });
   });
   busboy.on('field', function(fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) {
      console.log('Field [' + fieldname + ']: value: ' + inspect(val));
   });
   busboy.on('finish', function() {
      console.log('Done parsing form!');
      busboyFinishTime = new Date();
      if(busboyFinishTime && s3UploadFinishTime) {
         res.json({
            uploadStartTime: uploadStartTime,
            busboyFinishTime: busboyFinishTime,
            s3UploadFinishTime: s3UploadFinishTime
         });
      }
   });
   req.pipe(busboy);
});

根据我的观察，我有信心这是将文件上传至的最佳解决方案之一使用EC2上部署的REST API通过EC2进行S3。

According to my observations, I feel confident that this is one of the best solutions to upload a file to S3 via EC2 using a REST API deployed on EC2.

这篇关于节点JS-将数据从Busboy流到AWS S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！