使用nodejs流式传输和转换文件 [英] Stream and transform a file in place with nodejs

问题描述

我想做类似的事情：

var fs = require('fs');
var through = require('through');

var file = 'path/to/file.json';

var input = fs.createReadStream(file, 'utf8');
var output = fs.createWriteStream(file, 'utf8');

var buf = '';
input
.pipe(through(function data(chunk) { buf += chunk; }, function end() {
  var data = JSON.parse(buf);
  // Do some transformation on the obj, and then...
  this.queue(JSON.stringify(data, null, ' '));
})
.pipe(output);

但这会失败，因为它试图读取和写入同一目的地。有办法绕过它，就像只从上面的结束回调中输入输出一样。

But this fails because it's trying to read and write to the same destination. There are ways around it, like only piping to output from within the end callback above.

有更好的方法吗？更好的是，我的意思是使用更少的代码或更少的内存。是的，我知道我可以这样做：

Is there a better way? By better, I mean uses less code or less memory. And yes, I'm aware that I could just do:

var fs = require('fs');
var file = 'path/to/file.json';

var str = fs.readFileSync(file, 'utf8');
var data = JSON.parse(str);    
// Do some transformation on the obj, and then...
fs.writeFileSync(file, JSON.stringify(data, null, '  '), 'utf8');

推荐答案

您的代码没有其他方式可以使用 less 内存，因为您需要将整个文件解析为Javascript对象。通过这种方式，您的代码的两个版本都是内存相同的。如果您无需处理完整的JSON对象即可完成某些工作，请查看 JSONStream 。

There is no other way that your code will use less memory, because you need the whole file to parse it into a Javascript object. In this way, both versions of your code are equivalent memory-wise. If you can do some work without having to work on the full JSON object, check out JSONStream.

在您的示例中，应该读取文件，然后解析并转换它，然后将结果写入文件;虽然您不应使用函数的同步版本，但请参阅节点的本段末尾.js文档：

In your example, you should read the file, then parse and transform it, then write the result to a file; although you shouldn't use the synchronous version of the functions, see the end of this paragraph of the Node.js documentation:

在繁忙的进程中，强烈建议程序员使用这些调用的异步版本。同步版本将阻止整个过程，直到它们完成 - 暂停所有连接。

In busy processes, the programmer is strongly encouraged to use the asynchronous versions of these calls. The synchronous versions will block the entire process until they complete--halting all connections.

---

无论如何，我不认为你在覆盖它时可以从文件中读取。请参阅此特定答案以解决同一问题。

这篇关于使用nodejs流式传输和转换文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！