.pipe()是否在node.js中执行memcpy? [英] Does the .pipe() perform a memcpy in node.js?

问题描述

这是有关系统级优化的概念性查询.通过阅读NodeJS文档，我的理解是管道很容易在流上执行流控制.

This is a conceptual query regarding system level optimisation. My understanding by reading the NodeJS Documentation is that pipes are handy to perform flow control on streams.

背景:我有麦克风流进来，我想避免额外的复制操作以节省整个系统的MIPS.我知道对于音频流，即使引擎盖下有内存复制，也不会花费很多MIPS，但是我还计划扩展以30fps和UHD分辨率在相机帧中进行流传输.以30fps的速度复制UHD分辨率像素数据的多个副本效率极低，因此需要一些建议.

Background: I have microphone stream coming in and I wanted to avoid an extra copy operation to conserve overall system MIPS. I understand that for audio streams this is not a great deal of MIPS being spent even if there was a memcopy under the hood, but I also have an extension planned to stream in camera frames at 30fps and UHD resolution. Making multiple copies of UHD resolution pixel data at 30fps is super inefficient, so needed some advice around this.

示例代码:

var spawn = require('child_process').spawn
var PassThrough = require('stream').PassThrough;

var ps = null;
//var audioStream = new PassThrough;
//var infoStream = new PassThrough;

var start = function() {
    if(ps == null) {
        ps = spawn('rec', ['-b', 16, '--endian', 'little', '-c', 1, '-r', 16000, '-e', 'signed-integer', '-t', 'raw', '-']);
        //ps.stdout.pipe(audioStream);
        //ps.stderr.pipe(infoStream);
        exports.audioStream = ps.stdout;
        exports.infoStream = ps.stderr;
    }
};

var stop = function() {
    if(ps) {
        ps.kill();
        ps = null;
    }
};

//exports.audioStream = audioStream;
//exports.infoStream = infoStream;
exports.startCapture = start;
exports.stopCapture = stop;

以下是问题:

1. 为了能够执行流控制，source.pipe(dest)是在引擎盖下执行从源内存到目标内存的memcpy还是将内存中的引用传递给目标?
2. 注释的代码包含PassThrough类的实例化-我目前假设PassThrough也会引起内存复制，因此我要在整个系统中保存一个memcpy操作，因为我在上面的注释中添加了?
3. 如果我必须在Process和Spawned Child进程之间创建管道(使用child_process.spawn()，如

1. To be able to perform flow control, does the source.pipe(dest) perform a memcpy from the source memory to the destination memory under the hood OR would it pass the reference in memory to the destination?
2. The commented code contains a PassThrough class instantiation - I am currently assuming the PassThrough causes memcopies as well, and so I am saving one memcpy operation in the entire system because I added in the above comments?
3. If I had to create a pipe between a Process and a Spawned Child process (using child_process.spawn() as shown in How to transfer/stream big data from/to child processes in node.js without using the blocking stdio?), I presume that definitely results in memcpy? Is there anyway to make that a reference rather than copy?
4. Does this behaviour differ from OS to OS? I presume it should be OS agnostic, but asking this anyways.

在此先感谢您的帮助.这将极大地帮助我的体系结构.

Thanks in advance for your help. It will help my architecture a great deal.

推荐答案

一些URL供参考: https://github.com/nodejs/node/
https://github.com/nodejs/node/blob/master/src /stream_wrap.cc
https://github.com/nodejs/node/blob/master/src /stream_base.cc
https://github.com/libuv/libuv/blob /v1.x/src/unix/stream.c
https://github.com/libuv/libuv/blob /v1.x/src/win/stream.c

some url's for reference: https://github.com/nodejs/node/
https://github.com/nodejs/node/blob/master/src/stream_wrap.cc
https://github.com/nodejs/node/blob/master/src/stream_base.cc
https://github.com/libuv/libuv/blob/v1.x/src/unix/stream.c
https://github.com/libuv/libuv/blob/v1.x/src/win/stream.c

我尝试根据theese和其他一些文件编写一个复杂的/庞大的解释，但是我得出的结论是，最好总结一下我的经验/阅读内容如何告诉我节点的内部工作:

i tried writing a complicated / huge explaination based on theese and some other files however i came to the conclusion it would be best to give you a summary of how my experience / reading tells me node internally works:

管道简单地连接流，使其看起来好像.write(…)被.write(…)调用一样，中间没有任何膨胀.

pipe simply connects streams making it appear as if .on("data", …) is called by .write(…) without anything bloated in between.

现在我们需要将js世界与c ++/c世界分开.
在js中处理数据时，我们使用缓冲区. https://github.com/nodejs/node/blob/master/src /node_buffer.cc
它们只是代表分配的内存，上面有一些糖果可以对其进行操作.

now we need to separate the js world from the c++ / c world.
when dealing with data in js we use buffers. https://github.com/nodejs/node/blob/master/src/node_buffer.cc
they simply represent allocated memory with some candy on top to operate with it.

如果将进程的标准输出连接到某些.on("data", …)侦听器，它将把传入的块复制到Buffer对象中，以在js世界中进一步使用.
在js世界中，您可以使用.pause()等方法(如您在节点Steam api文档中所见)，以防止进程吃掉内存，以防传入的数据流比其处理的速度快.

if you connect stdout of a process to some .on("data", …) listener it will copy the incoming chunk into a Buffer object for further usage inside the js world.
inside the js world you have methods like .pause() etc. (as you can see in nodes steam api documentation) to prevent the process to eat memory in case incoming data flows faster than its processed.

连接一个进程的标准输出，例如通过管道连接一个传出的tcp端口，将导致类似于nginx操作的连接.通过将传入数据直接复制到传出流中，它将像连接这些流一样，直接连接这些流.

connecting stdout of a process and for example an outgoing tcp port through pipe will result in a connection similar to how nginx operates. it will connect theese streams as if they would directly talk to each other by copying incoming data directly to the outgoing stream.

一旦暂停流，节点将使用内部缓冲以防无法暂停传入的流.

as soon as you pause a stream, node will use internal buffering in case its unable to pause the incoming stream.

因此对于您的情况，您应该进行测试.
尝试通过节点中的传入流接收数据，暂停该流，然后看看会发生什么.
我不确定节点是否将使用内部缓冲，或者您尝试运行的进程是否会暂停，直到它可以继续发送数据.
我希望过程停止，直到您继续播放流媒体为止.

so for your scenario you should just do testing.
try to receive data through an incoming stream in node, pause the stream and see what happens.
i'm not sure if node will use internal buffering or if the process you try to run will just halt untill it can continue to send data.
i expect the process to halt untill you continue the stream.

对于传输大图像，我建议将它们成块传输或直接将其通过管道传输到传出端口.

for transfering huge images i recommend transfering them in chunks or to pipe them directly to an outgoing port.

块方式将使您可以一次将数据发送到多个客户端，并保持较低的内存占用.

the chunk way would allow you to send the data to multiple clients at once and would keep the memory footprint pretty low.

PS，您应该看一下我刚刚发现的要点: https://gist.github. com/joyrexus/10026630
它深入说明了如何与流进行交互

PS you should take a look at this gist that i just found: https://gist.github.com/joyrexus/10026630
it explains in depth how you can interact with streams

这篇关于.pipe()是否在node.js中执行memcpy?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！