Go io.Pipe的缓冲版本 [英] Buffered version of Go io.Pipe

问题描述

标准库中是否存在 io.Pipe 的缓冲版本.第三方图书馆)，然后再推出自己的图书?

Is there a buffered version of io.Pipe (either in the standard library or a third-party library) before I roll my own?

上下文:我正在尝试使用此解决方案来解析使用bzip2压缩的JSON数据，以便进行解压缩和解析是并行进行的，但是发现加速非常小.解析未压缩的数据每百万条记录大约需要22秒.解压缩那么多数据大约需要相同的时间.如预期的那样，在单线程上完成它们大约需要44秒.使用上述解决方案大约需要41秒钟.

Context: I'm trying to use this solution for parsing of JSON data compressed with bzip2, so that the decompression and parsing happen in parallel, but finding that the speed-up is very small. Parsing uncompressed data takes ~22 sec per million records. Decompressing that much data takes about the same time. Doing them on a single thread takes ~44 seconds, as expected. Using the solution above takes ~41 seconds.

io.Pipe的文档说:

The documentation for io.Pipe says:

管道上的读取和写入一一匹配，除非当需要多次读取才能消耗单个写入.也就是说，每个写入PipeWriter块，直到满足一个或多个条件从PipeReader读取的内容会完全消耗已写入的数据.这数据直接从写入复制到相应的读取(或阅读);没有内部缓冲.

Reads and Writes on the pipe are matched one to one except when multiple Reads are needed to consume a single Write. That is, each Write to the PipeWriter blocks until it has satisfied one or more Reads from the PipeReader that fully consume the written data. The data is copied directly from the Write to the corresponding Read (or Reads); there is no internal buffering.

我怀疑这可能是个问题，具体取决于bzip2解压缩器写入数据的方式以及JSON解析器读取数据的方式，因此我想尝试一个缓冲版本.

I suspect this could be a problem, depending on the way the bzip2 decompressor writes data and the way the JSON parser reads it, so I'd like to try a buffered version.

推荐答案

这就是 bufio 包适用于.它允许您使用 NewReader将任何 io.Reader 转换为缓冲的阅读器. 或任何 io.Writer 放入具有

That's what the bufio package is for. It lets you turn any io.Reader into a buffered reader with NewReader, or any io.Writer into a buffered writer with NewWriter.

(缓冲的IO是否真的可以解决您的特定问题，我不知道...)

(Whether buffered IO will actually help with your specific problem, I have no idea...)

这篇关于Go io.Pipe的缓冲版本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！