Java:对 InputStream 的并发读取 [英] Java: Concurrent reads on an InputStream

问题描述

现在环顾四周，我对这个问题有点困惑.我希望能够获取输入流并在分段中同时读取它.这些段不相互交互，它们只是从上传的文件插入或更新到数据库中的值.是否可以通过设置段大小，然后在分离新线程处理转换和插入/更新之前向前跳过来同时读取输入流?

Been looking around for a little while now and I'm a bit confused on this issue. I want to be able to take an input stream and read it concurrently in segments. The segments don't interact with each other they are just values to be inserted or updated in a database from an uploaded file. Is it possible to read an input stream concurrently by setting a segment size and then just skipping forward before spinning off a new thread to handle the conversion and insert/update?

本质上，该文件是一个 ID 列表(每行一个 ID)，尽管如果我可以指定分隔符会更好.有些文件可能很大，所以我想处理数据并将其转换为段，以便在插入/更新到数据库后可以释放 JVM 内存.这可能吗?如果是这样，是否有任何图书馆已经这样做了?

Essentially the file is a list of ID's (one ID per line), although it would be preferable if I could specify a separator. Some files can be huge so I would like to process and convert the data to be into segments so that after inserting/updating to the database the JVM memory can be freed up. Is this possible? And if so are there any libraries out there that do this already?

提前致谢，

阿列克谢蓝.

推荐答案

一个好的方法可能是让一个读取器读取块，然后将每个块从线程池中交给工作线程.鉴于这些将被插入到数据库中，与读取输入相比，插入将是迄今为止最慢的部分，因此单个线程应该足以读取.

A good approach might instead be to have a single reader that reads chunks and then hands each chunk off to a worker thread from a thread pool. Given that these will be inserted into a database the inserts will be by far the slow parts compared to reading the input so a single thread should suffice for reading.

下面是一个将 System.in 中每一行的处理移交给工作线程的示例.如果您在单个事务中执行大量插入，则数据库插入的性能会好得多，因此传入一组 1000 行比在示例中传入单行要好.

Below is an example that hands off processing of each line from System.in to a worker thread. Performance of database inserts is much better if you perform a large number inserts within a single transaction so passing in a group of say 1000 lines would be better than passing in a single line as in the example.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
    public static class Worker implements Runnable {
        private final String line;

        public Worker(String line) {
            this.line = line;
        }

        @Override
        public void run() {
            // Process line here.
            System.out.println("Processing line: " + line);
        }
    }

    public static void main(String[] args) throws IOException {
        // Create worker thread pool.
        ExecutorService service = Executors.newFixedThreadPool(4);

        BufferedReader buffer = new BufferedReader(new InputStreamReader(System.in));
        String line;

        // Read each line and hand it off to a worker thread for processing.
        while ((line = buffer.readLine()) != null) {
            service.execute(new Worker(line));
        }
    }
}

这篇关于Java:对 InputStream 的并发读取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！