Node.js 流直接上传到谷歌云存储 [英] Node.js stream upload directly to Google Cloud Storage

问题描述

我有一个在 Google Compute VM 实例上运行的 Node.js 应用程序，它直接从 POST 请求(而不是通过浏览器)接收文件上传，并将传入的数据流式传输到 Google Cloud Storage (GCS).

I have a Node.js app running on a Google Compute VM instance that receives file uploads directly from POST requests (not via the browser) and streams the incoming data to Google Cloud Storage (GCS).

我正在使用 Restify b/c 我不需要 Express 的额外功能，因为它可以轻松地流式传输传入的数据.

I'm using Restify b/c I don't need the extra functionality of Express and because it makes it easy to stream the incoming data.

我为文件创建了一个随机文件名，将传入的 req 扔到一个简洁的 GCS 小节点包装器中(可在此处找到:https://github.com/bsphere/node-gcs) 向 GCS 发出 PUT 请求.可以在此处找到使用 PUT 的 GCS 文档:https://developers.google.com/storage/docs/reference-methods#putobject ...它说如果使用 chunked transfer encoding，则不需要 Content-Length.

I create a random filename for the file, take the incoming req and toss it to a neat little Node wrapper for GCS (found here: https://github.com/bsphere/node-gcs) which makes a PUT request to GCS. The documentation for GCS using PUT can be found here: https://developers.google.com/storage/docs/reference-methods#putobject ... it says Content-Length is not necessary if using chunked transfer encoding.

好消息:文件正在相应的 GCS 存储桶"内创建！

Good news: the file is being created inside the appropriate GCS storage "bucket"!

坏消息:

1. 我还没有弄清楚如何从 Restify 获取传入文件的扩展名(注意我手动设置 '.jpg' 和 content-type 手动).

文件出现轻微损坏(几乎可以肯定是我在 PUT 请求中做错了什么).如果我从 Google 下载 POSTed 文件，OSX 会告诉我它已损坏......但是，如果我使用 PhotoShop，它会打开并且看起来很好.

The file is experiencing slight corruption (almost certainly do to something I'm doing wrong with the PUT request). If I download the POSTed file from Google, OSX tells me its damaged ... BUT, if I use PhotoShop, it opens and looks just fine.

更新/解决方案

正如 vkurchatkin 所指出的，我需要解析 request 对象，而不是仅仅将整个东西通过管道传送到 GCS.在尝试了较轻的 busboy 模块后，我认为使用 multiparty 会容易得多.为了动态设置 Content-Type，我只使用了 Mimer (https://github.com/heldr/mimer)，引用传入文件的文件扩展名.重要的是要注意，由于我们是在传递 part 对象，因此必须清除 part.headers.否则，意外的信息，特别是 content-type 将被传递，并且可能/将与我们试图显式设置的 content-type 冲突.

As pointed out by vkurchatkin, I needed to parse the request object instead of just piping the whole thing to GCS. After trying out the lighter busboy module, I decided it was just a lot easier to use multiparty. For dynamically setting the Content-Type, I simply used Mimer (https://github.com/heldr/mimer), referencing the file extension of the incoming file. It's important to note that since we're piping the part object, the part.headers must be cleared out. Otherwise, unintended info, specifically content-type, will be passed along and can/will conflict with the content-type we're trying to set explicitly.

这是适用的修改代码:

var restify = require('restify'),
    server = restify.createServer(),
    GAPI = require('node-gcs').gapitoken,
    GCS = require('node-gcs'),
    multiparty = require('multiparty'),
    Mimer = require('mimer');

server.post('/upload', function(req, res) {

    var form = new multiparty.Form();

    form.on('part', function(part){
        var fileType = '.' + part.filename.split('.').pop().toLowerCase();
        var fileName = Math.random().toString(36).slice(2) + fileType;

        // clear out the part's headers to prevent conflicting data being passed to GCS
        part.headers = null;

        var gapi = new GAPI({
            iss: '-- your -- @developer.gserviceaccount.com',
            scope: 'https://www.googleapis.com/auth/devstorage.full_control',
            keyFile: './key.pem'
        }, 
        function(err) {
            if (err) { console.log('google cloud authorization error: ' + err); }

            var headers = {
                'Content-Type': Mimer(fileType),
                'Transfer-Encoding': 'Chunked',
                'x-goog-acl': 'public-read'
            };

            var gcs = new GCS(gapi);

            gcs.putStream(part, myBucket, '/' + fileName, headers, function(gerr, gres){
                console.log('file should be there!');
            });
        });
    });
};

推荐答案

你不能使用原始的 req 流，因为它会产生整个请求体，它是多部分的.您需要使用类似 multiparty 之类的东西来解析请求，从而为您提供可读的蒸汽和您需要的所有元数据.

You can't use the raw req stream since it yields whole request body, which is multipart. You need to parse the request with something like multiparty give you a readable steam and all metadata you need.

这篇关于Node.js 流直接上传到谷歌云存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！