在Node.js中通过HTTP发送大图像数据 [英] Sending large image data over HTTP in Node.js

问题描述

在我的开发环境中，我有两台服务器。一个通过 POST http请求发送和映像到另一个。

In my development environment I have two servers. One sends and image to the other over a POST http request.

客户端服务器执行此操作：

Client server does this:

    fs.readFile(rawFile.path,'binary',function (err, file){
        restler.post("http://0.0.0.0:5000",{
            data: file,
            headers:{
                "Content-Type": rawFile.type,
            }
        }).on('complete',function(data,response){                               
            console.log(data);
            res.send("file went through")
        })

收到请求的服务器执行此操作：

The server that recieves the request does this:

    server.post('/',function(req,res,next){
        fs.writeFileSync("test.png",req.body,"binary",function(err){
            if(err) throw err;
            res.send("OK")
        })
    })

如果我发送一个小图像它工作正常。但是，如果我发送一个大的im尽管文件保存正确，但只显示图像的第一个上半部分。其余的都是黑色的。图像大小是正确的。

If i send a small image it works fine. However, if i send a large image although the file is saved correctly only the first upper portion of the image is displayed. The rest is black. Image size is correct.

我想这只是图像的第一个块写在文件上。
我试过创建一个 readStream 和一个 writeStream 但它似乎不起作用：

I guess it's just the first chunk of the image that's being written on the file. I've tried creating a readStream and a writeStream but it doesn't seem to work:

req.body.pipe(fs.createWriteStream('test.png'))

我可以直接从二进制数据和管道流入文件吗？对于我所看到的， readStream 通常用于从文件流而非原始二进制数据。

Can i stream directly from the binary data and pipe it into the file? For what i've seen, readStream is often used to stream from files not raw binary data.

我读过一些发布但它似乎对我不起作用。

I read a few posts but it doesn't seem to work for me.

我在客户端服务器中使用 restler 模块，并在 restify 中使用其他。

I'm using restler module in the client server and restify in the other.

谢谢！

推荐答案

很抱歉直言不讳这里有很多错误。

Sorry to be blunt, but there's a lot wrong here.

readFile 在调用回调之前将文件的全部内容读入内存，此时您开始上传文件。

readFile reads the entire contents of a file into memory before invoking the callback, at which point you begin uploading the file.

这很糟糕–特别是在处理像images&ndash这样的大文件时;因为没有理由将文件读入内存。这很浪费;在负载下，您会发现您的服务器内存不足并崩溃。

This is bad–especially when dealing with large files like images–because there's really no reason to read the file into memory. It's wasteful; and under load, you'll find that your server will run out of memory and crash.

相反，您想获得流，它们从磁盘读取时会发出大块数据。您所要做的就是将这些块传递给您的上传流（ pipe ），然后从内存中丢弃数据。这样，你永远不会使用超过少量的缓冲内存。

Instead, you want to get a stream, which emits chunks of data as they're read from disk. All you have to do is pass those chunks along to your upload stream (pipe), and then discard the data from memory. In this way, you never use more than a small amount of buffer memory.

（可读流的默认行为是处理原始二进制数据;只有当你通过时才会这样做它在文本中处理的编码。）

(A readable stream's default behavior is to deal in raw binary data; it's only if you pass an encoding that it deals in text.)

请求模块使这一点变得特别简单：

The request module makes this especially easy:

fs.createReadStream('test.png').pipe(request.post('http://0.0.0.0:5000/'));

在服务器上，您遇到了更大的问题。 从不使用* 同步方法。它阻止您的服务器执行任何（如响应其他请求），直到整个文件被刷新到磁盘，这可能需要几秒钟。

On the server, you have a larger problem. Never use *Sync methods. It blocks your server from doing anything (like responding to other requests) until the entire file is flushed to disk, which can take seconds.

因此，我们希望获取传入的数据流并将其传输到文件系统流。你最初是在正确的轨道上; req.body.pipe（fs.createWriteStream（'test.png'））无效的原因是因为 body 不是流。

So instead, we want to take the incoming data stream and pipe it to a filesystem stream. You were on the right track originally; the reason that req.body.pipe(fs.createWriteStream('test.png')) didn't work is because body is not a stream.

正文由 bodyParser生成中间件。在改进中，该中间件的行为与 readFile 非常相似，因为它将整个传入的请求实体缓存在内存中。在这种情况下，我们不希望这样。禁用正文解析器中间件。

body is generated by the bodyParser middleware. In restify, that middleware acts much like readFile in that it buffers the entire incoming request-entity in memory. In this case, we don't want that. Disable the body parser middleware.

那么传入数据流在哪里？它是 req 对象本身。 restify的请求继承节点的 http.IncomingMessage ，这是一个可读的流。所以：

So where is the incoming data stream? It is the req object itself. restify's Request inherits node's http.IncomingMessage, which is a readable stream. So:

fs.createWriteStream('test.png').pipe(req);

---

我还应该提到这一切都很简单因为没有形式解析开销。请求只发送没有 multipart / form-data 包装器的文件：

POST / HTTP/1.1
host: localhost:5000
content-type: application/octet-stream
Connection: keep-alive
Transfer-Encoding: chunked

<image data>...

这意味着浏览器无法将文件发布到此URL。如果需要，请查看强大的，它会对请求实体进行流解析。

This means that a browser could not post a file to this URL. If that's a need, look in to formidable, which does streaming parsing of request-entities.

这篇关于在Node.js中通过HTTP发送大图像数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！