如何将文件从 url 流式传输到 s3 [英] How to stream file to s3 from url

问题描述

如何从 url 获取文件然后上传到 s3?有很多使用分段上传的示例，但这应该更简单.此代码执行服务器端.S3 上传失败，并显示NetworkingError:无法通过管道".不可读.

How do I grab a file from a url and then upload to s3? There are alot of examples using Multipart Upload out there but this should be simpler.This code executes server side. S3 upload fails with 'NetworkingError: Cannot pipe. Not readable.'

import http from 'http';
import fs from 'fs';
import AWS from 'aws-sdk';
AWS.config.region = 'us-west-1';

const file = fs.createWriteStream('file.jpg');
var request = http.get('http://weknowmemes.com/wp-content/uploads/2012/04/dont-worry-cat-it.jpg', function(response) {
  response.pipe(file);

  file.on('finish', function() {
    // Upload the File
    var s3 = new AWS.S3({params: {Bucket: Config.s3Bucket}});
    var params = {Key: 'mykey', ContentType: 'image/jpg', Body: file};
    s3.putObject(params, function (err, data) {
      console.log('err', err)
      console.log('data', data)
    });
  });
});

推荐答案

好的，我解决了这个问题.这是我的工作解决方案:

Alright I solved this. Here's my working solution:

import request from 'request';
import bl from 'bl';
import AWS from 'aws-sdk';

const regexp = /filename=\"(.*)\"/gi;
let fileName;
let mimeType;

request
  .get(imageUrl)
  .on('response', function(response) {
    fileName = regexp.exec(response.headers['content-disposition'] )[1];
    mimeType = response.headers['content-type'];
  })
  .pipe(bl(function(error, data) {
    const s3 = new AWS.S3({params: {Bucket: Config.s3Bucket}});
    const params = {Key: fileName, ContentType: mimeType, Body: data};
    s3.putObject(params, function(s3Error, data) {
      if (s3Error) {
        console.log('Error uploading data: ', s3Error);
      } else {
        console.log('Successfully uploaded data: ', data);
      }
    });
  }))

我不需要使用 WriteStream 实例，但我确实需要获取图像的原始数据二进制文件，以便分配给 s3 的 putObject 参数的 Body> 方法.这需要使用 BufferList 即.从'bl'导入bl;

I didn't need to use a WriteStream instance but I did need to get the raw data binary for my image in order to assign to the Body parameter of s3's putObject method. This required using a BufferList ie. import bl from 'bl';

实际上，我的主要问题是不了解如何以正确的方式将数据从 request.get 导入并导出到 S3.当您使用 Multipart Form 时，它会为您处理文件，因此我一直在为该用例找到所有简单的示例.

Really my main issue was not understanding how to get the data formatted the right way in from request.get and out to S3. When you are using a Multipart Form it handles the file for you hence all the simplish examples I kept finding for that use case.

这篇关于如何将文件从 url 流式传输到 s3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！