并行迭代文件中的行(Scala)? [英] Iterate over lines in a file in parallel (Scala)?

问题描述

我了解 Scala 中的并行集合.他们很方便！但是，我想并行迭代一个对于内存来说太大的文件的行.例如，我可以创建线程并在 Scanner 上设置锁，但如果我可以运行如下代码就太好了:

I know about the parallel collections in Scala. They are handy! However, I would like to iterate over the lines of a file that is too large for memory in parallel. I could create threads and set up a lock over a Scanner, for example, but it would be great if I could run code such as:

Source.fromFile(path).getLines.par foreach { line =>

不幸的是

error: value par is not a member of Iterator[String]

在这里实现一些并行性的最简单方法是什么?现在，我将阅读一些行并并行处理它们.

What is the easiest way to accomplish some parallelism here? For now, I will read in somes lines and handle them in parallel.

推荐答案

您可以使用分组轻松地将迭代器分割成可以加载到内存中然后并行处理的块.

You could use grouping to easily slice the iterator into chunks you can load into memory and then process in parallel.

val chunkSize = 128 * 1024
val iterator = Source.fromFile(path).getLines.grouped(chunkSize)
iterator.foreach { lines => 
    lines.par.foreach { line => process(line) }
}

在我看来，这样的事情是最简单的方法.

In my opinion, something like this is the simplest way to do it.

这篇关于并行迭代文件中的行(Scala)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！