Node.js流直接上传到Google Cloud Storage [英] Node.js stream upload directly to Google Cloud Storage
问题描述
我有一个运行在Google Compute VM实例上的Node.js应用程序，该应用程序直接从POST请求(而不是通过浏览器)接收文件上传，并将传入的数据流传输到Google Cloud Storage(GCS).
I have a Node.js app running on a Google Compute VM instance that receives file uploads directly from POST requests (not via the browser) and streams the incoming data to Google Cloud Storage (GCS).
我正在使用Restify b/c，我不需要Express的额外功能，因为它可以轻松地传输传入的数据.
I'm using Restify b/c I don't need the extra functionality of Express and because it makes it easy to stream the incoming data.
我为该文件创建了一个随机文件名，将传入的req放入一个干净的小GCS节点包装器中(在这里找到: https://developers.google. com/storage/docs/reference-methods#putobject ...它说，如果使用chunked transfer encoding，则不需要Content-Length.
I create a random filename for the file, take the incoming req and toss it to a neat little Node wrapper for GCS (found here: https://github.com/bsphere/node-gcs) which makes a PUT request to GCS. The documentation for GCS using PUT can be found here: https://developers.google.com/storage/docs/reference-methods#putobject ... it says Content-Length is not necessary if using chunked transfer encoding.
好消息:正在相应的GCS存储存储桶"内创建文件 ！
Good news: the file is being created inside the appropriate GCS storage "bucket"!
坏消息:
-
我还没有弄清楚如何从Restify获取传入文件的扩展名(注意，我手动设置了'.jpg'和
content-type).
文件正在发生轻微损坏(几乎可以肯定我对PUT请求做错了什么).如果我从Google下载POSTED文件，则OSX会告诉我它已损坏...但是，如果我使用PhotoShop，它会打开并且看起来还不错.
The file is experiencing slight corruption (almost certainly do to something I'm doing wrong with the PUT request). If I download the POSTed file from Google, OSX tells me its damaged ... BUT, if I use PhotoShop, it opens and looks just fine.
更新/解决方案
正如vkurchatkin所指出的，我需要解析request对象，而不是仅仅将整个对象传送到GCS.在尝试了较轻的busboy模块之后，我认为使用multiparty变得容易得多.为了动态设置Content-Type，我只使用了Mimer( https://github.com/heldr/mimer )，引用传入文件的文件扩展名.重要的是要注意，由于我们正在管道part对象，因此必须清除part.header.否则，会传递意想不到的信息(特别是content-type)，并且可能/将与我们试图明确设置的content-type发生冲突.
As pointed out by vkurchatkin, I needed to parse the request object instead of just piping the whole thing to GCS. After trying out the lighter busboy module, I decided it was just a lot easier to use multiparty. For dynamically setting the Content-Type, I simply used Mimer (https://github.com/heldr/mimer), referencing the file extension of the incoming file. It's important to note that since we're piping the part object, the part.headers must be cleared out. Otherwise, unintended info, specifically content-type, will be passed along and can/will conflict with the content-type we're trying to set explicitly.
以下是适用的修改后的代码:
Here's the applicable, modified code:
var restify = require('restify'),
server = restify.createServer(),
GAPI = require('node-gcs').gapitoken,
GCS = require('node-gcs'),
multiparty = require('multiparty'),
Mimer = require('mimer');
server.post('/upload', function(req, res) {
var form = new multiparty.Form();
form.on('part', function(part){
var fileType = '.' + part.filename.split('.').pop().toLowerCase();
var fileName = Math.random().toString(36).slice(2) + fileType;
// clear out the part's headers to prevent conflicting data being passed to GCS
part.headers = null;
var gapi = new GAPI({
iss: '-- your -- @developer.gserviceaccount.com',
scope: 'https://www.googleapis.com/auth/devstorage.full_control',
keyFile: './key.pem'
},
function(err) {
if (err) { console.log('google cloud authorization error: ' + err); }
var headers = {
'Content-Type': Mimer(fileType),
'Transfer-Encoding': 'Chunked',
'x-goog-acl': 'public-read'
};
var gcs = new GCS(gapi);
gcs.putStream(part, myBucket, '/' + fileName, headers, function(gerr, gres){
console.log('file should be there!');
});
});
});
};
推荐答案
您不能使用原始的req流，因为它会产生整个请求主体，该主体是多部分的.您需要使用 multiparty 之类的内容来解析请求，从而为您提供可读的流和所需的所有元数据
You can't use the raw req stream since it yields whole request body, which is multipart. You need to parse the request with something like multiparty give you a readable steam and all metadata you need.
这篇关于Node.js流直接上传到Google Cloud Storage的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
