如何使用多个线程处理存储在本地磁盘中的大量文件（使用文件锁） [英] how to use multiple threads to process large number of files stored in the local disk ( using file lock)

问题描述

如何在java中使用多个线程来处理存储在本地磁盘目录中的大量文件（使用文件锁）

how to use multiple threads in java to process large number of files stored in the local disk directory ( using file lock)

推荐答案

您不想以并行方式读取文件（磁盘I / O不能很好地并行化）。最好让单个线程读取文件，将内容发送到工作线程进行并行处理，然后从工作者收集结果。使用优秀的 ExecutorService & c：o来自 java.util.concurrent 为您提供线程的详细信息，使您的解决方案更加灵活。

You don't want to read the files in parallell (disk I/O doesn't parallelize well). Better then to let a single thread read the files, send the contents off to worker threads for parallel processing, and then collect the results from the workers. Using the excellent ExecutorService & c:o from java.util.concurrent spares you the dirty details of threading and makes your solution far more flexible.

这是一个简单的例子。假设 Foo 是处理文件的结果：

Here's a simple example. Assuming Foo is the result of processing a file:

public List<Foo> processFiles(Iterable<File> files){
    List<Future<Foo>> futures = new ArrayList<Future<Foo>>();
    ExecutorService exec = Executors.newFixedThreadPool(
        Runtime.getRuntime().availableProcessors());
    for (File f : files){
        final byte[] bytes = readAllBytes(f); // defined elsewhere
        futures.add(exec.submit(new Callable<Foo>(){
            public Foo call(){
                InputStream in = new ByteArrayInputStream(bytes);
                // Read a Foo object from "in" and return it
            }
        }));
    }
    List<Foo> foos = new List<Foo>(futures.size());
    for (Future<Foo> f : futures) foos.add(f.get());
    exec.shutdown();
    return foos;
}

TODO：添加异常处理等。您可能还想实例化 ExecutorService 在 processFiles 之外，因此您可以在调用之间重复使用它。

TODO: Add exception handling etc. You may also want to instantiate the ExecutorService outside of processFiles so you can reuse it between calls.

这篇关于如何使用多个线程处理存储在本地磁盘中的大量文件（使用文件锁）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！