Java：并发读取InputStream [英] Java: Concurrent reads on an InputStream

问题描述

现在看了一会儿，我在这个问题上有点困惑。我想要能够采取一个输入流，并在段中同时读取。段不会彼此交互，它们只是要从上载的文件插入或更新到数据库中的值。是否可以通过设置段大小同时读取输入流，然后在离开新的线程之前跳过来处理转换和插入/更新？

Been looking around for a little while now and I'm a bit confused on this issue. I want to be able to take an input stream and read it concurrently in segments. The segments don't interact with each other they are just values to be inserted or updated in a database from an uploaded file. Is it possible to read an input stream concurrently by setting a segment size and then just skipping forward before spinning off a new thread to handle the conversion and insert/update?

基本上该文件是ID的列表（每行一个ID），虽然如果我可以指定一个分隔符将是更可取的。一些文件可能是巨大的，所以我想处理和转换数据成段，以便插入/更新到数据库后JVM内存可以释放。这可能吗？

Essentially the file is a list of ID's (one ID per line), although it would be preferable if I could specify a separator. Some files can be huge so I would like to process and convert the data to be into segments so that after inserting/updating to the database the JVM memory can be freed up. Is this possible? And if so are there any libraries out there that do this already?

提前和感谢，

阿列克谢蓝。

推荐答案

一个好的方法可能是让一个读取器读取块，然后将每个块移动到工作线程一个线程池。考虑到这些将被插入到数据库中，插入将是读取输入的慢的部分，所以单个线程应该足以读取。

A good approach might instead be to have a single reader that reads chunks and then hands each chunk off to a worker thread from a thread pool. Given that these will be inserted into a database the inserts will be by far the slow parts compared to reading the input so a single thread should suffice for reading.

下面是例如，从 System.in 到工作线程的每个线的处理。如果在单个事务中执行大量插入，那么数据库插入的性能会好得多，因此，传递一组比如说1000行的代码比在示例中传递一行要好。

Below is an example that hands off processing of each line from System.in to a worker thread. Performance of database inserts is much better if you perform a large number inserts within a single transaction so passing in a group of say 1000 lines would be better than passing in a single line as in the example.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
    public static class Worker implements Runnable {
        private final String line;

        public Worker(String line) {
            this.line = line;
        }

        @Override
        public void run() {
            // Process line here.
            System.out.println("Processing line: " + line);
        }
    }

    public static void main(String[] args) throws IOException {
        // Create worker thread pool.
        ExecutorService service = Executors.newFixedThreadPool(4);

        BufferedReader buffer = new BufferedReader(new InputStreamReader(System.in));
        String line;

        // Read each line and hand it off to a worker thread for processing.
        while ((line = buffer.readLine()) != null) {
            service.execute(new Worker(line));
        }
    }
}

这篇关于Java：并发读取InputStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！