Nodejs文件流并发性 [英] Nodejs filestream concurrency

问题描述

它看起来像我在nodejs中的并发问题。我知道这是不可能的。

我正在通过文件处理数据并将其写入另一个文件。在输出文件中，我注意到行被覆盖，意味着我偶尔会看到输出中的行似乎被其他行中途覆盖。
我从读取流中读取数据。它看起来大致是这样的：

$ p $ let iStream = fs.createReadStream（inputFile）;
让oStream = fs.createWriteStream（outputFile）;
let remaining ='';

iStream.on（'data'，（data）=> {
remaining + = data;
let line = remaining.split（/ \ r？\
let line = line.length;
if（lines> 0）{
remaining = line [lines - 1];
line.length = lines - 1;
line.forEach（（curr）=> {
oStream.write（processLine（curr））;
}）;
}）;


有没有这种方案产生写入失败的可能性，还是我不得不去其他地方看？ p>

解决方案

这似乎是一个缓冲区溢出问题。您可能会溢出写入缓冲区，但不会注意流量控制。

您可以将回调传递到 .write（） / code>，只有在调用该回调函数时才继续下一次写入操作，或者您可以关注 .write（）的返回值，并在返回 false ，你必须等待流中的 drain 事件才能写入更多。

另一种方法是编写一个转换流，然后使用 .pipe（），让流媒体基础设施为您管理流量控制。

it looks to me like I have a concurrency problem in nodejs. I am aware of the fact that this is not supposed to be possible.

I am processing data by the line from a file and writing it to another file also in lines. In the output file I notice that lines are being overwritten meaning that every now and then I see lines in the output that seem to be overwritten half way by other lines. I read the data from a read stream. it looks roughly like this:

let iStream = fs.createReadStream(inputFile);
let oStream = fs.createWriteStream(outputFile);
let remaining = '';

iStream.on('data',(data)=>{
  remaining += data;
  let line = remaining.split(/\r?\n/);
  let lines = line.length;
  if(lines > 0) {
    remaining = line[lines - 1];
    line.length = lines - 1;
    line.forEach((curr)=>{
      oStream.write(processLine(curr));
    });
});

Is there any possibility of this scheme producing write failures or do I have to look somewhere else ?

解决方案

This appears to be a buffer overflow issue. You are likely overflowing the write buffer, but not paying any attention to flow control.

You can either pass a callback into .write() and only proceed with the next write when that callback is called or you can pay attention to the return value from .write() and when it returns false, you have to then wait for the drain event on the stream before writing some more.

Another approach would be to write a transform stream and then use .pipe() and let the streaming infrastructure manage the flow control for you.

这篇关于Nodejs文件流并发性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！