尽管通过磁盘存储位置参数，但我通过流方式上传文件时，内存缓冲区已满 [英] Memory buffers fills up when I stream upload a file, though passing a disk storage location param

问题描述

尝试使用此代码使用船长和skipper-s3流式传输到S3

Trying to stream-upload to S3 using skipper and skipper-s3 using this code

image_upload : function  (req,res) {

    // depends on bodyparser:skipper in confing/http.js

    console.log("Uploading image .. ");

    req.file('image')
        .upload({
            adapter: require('skipper-s3'),
            key: KEY,
            secret: SECRET,
            bucket: 'mybucket',
            tmpdir: 'tmp/s3-multiparts'
        }, function whenDone(err, uploadedFiles) {
            if (err){
                return res.negotiate(err);
            } else {
                dict = {
                    files: uploadedFiles,
                    textParams: req.params.all()
                }
                return res.ok(dict);
            }
        }

    );
};

我有2个问题:

第一:当我尝试上传一些文件时，我看到该文件在内存中缓冲，而不是仅缓冲到磁盘(添加到选项的路径)中.

First: when I try to upload some file, I can see that this file buffers in memory, instead just buffering to the disk (path added to options).

实际上这是一个大问题，因为当有许多用户同时上传时，我的记忆将很快填满

Actually this is a big problem, as my memory will fill up so quickly when having many users concurrently uploading

第二:当上传大小超过5MB时，我的应用程序崩溃，这是我的错误

Second: my application crashes when upload is bigger than 5MBs, giving me this error

events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Request aborted
    at IncomingMessage.onReqAborted (~Desktop/myApp/node_modules/sails/node_modules/skipper/node_modules/multiparty/index.js:175:17)
    at IncomingMessage.emit (events.js:104:17)
    at abortIncoming (_http_server.js:279:11)
    at Socket.serverSocketCloseListener (_http_server.js:292:5)
    at Socket.emit (events.js:129:20)
    at TCP.close (net.js:485:12)

推荐答案

第二个问题的解决方案: 您可以指定要上传的文件的最大大小.默认值为5 MB.

Solution to your second problem: You can specify maximum size of file to be uploaded. Default is 5 MB.

req.file('some_file').upload({
    dirname: 'path to store the file',/* optional. defaults to assets/uploads I guess*/
    saveAs: 'new file name', /* optional. default file name */
    maxBytes: 5 * 1024 * 1024 //5 MB
}, function(err, uploadedFiles) {

});

我不确定第一个问题.您无需在选项中指定临时目录.默认情况下，Skipper-S3将文件写入<your project root>/.tmp/s3-upload-part-queue以进行上传.我不明白为什么要填满内存.您可以删除tempdir并进行检查吗?

I am not sure about the first problem though. You don't need to specify a tempdir in options. Skipper-S3 by default writes the file to <your project root>/.tmp/s3-upload-part-queue for its upload purpose. I don't see why the memory should get filled up. Can you possibly remove the tempdir and check?

这篇关于尽管通过磁盘存储位置参数，但我通过流方式上传文件时，内存缓冲区已满的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！