如何写多线程代码并累积单个文件中所有线程的输出 [英] How write mulithreaded code and accumulate the output from all the threads in a single file

问题描述

这个问题有点类似于使Java中的现有代码并行/ multithread

我没有找到我的问题的答案，所以我在下面发帖。

这里是我的简化版本现有的应用程序代码

  for（MyClass myObject：myObjectSet）{
 String outputString = myLongRunningMethod 
 fileWriter.append（outputString +\\\
）; 
} 
 
 fileWriter.close（）; 
  

这里我试图让它多线程。code> myLongRunningMethod 是最慢的一部分。

  ExecutorService threadExec = Executors.newFixedThreadPool（myObjectSet.size 
  

//信号量实现
ResourcePool resourcePool = new ResourcePool（myObjectSet.size（））;

  for（MyClass myObject：myObjectSet）{
 Object key = resourcePool.getItem 
 
 MyClassMT myClassMT = new MyClassMT（myObject）; 
 threadExec.execute（myClassMT）; 
} 
  

其中MyClassMT是：

  public class MyClassMT实现Runnable {
 
 MyClass myObject; 
 
 public MyClassMT（MyClass myObject）{
 this.myObject = myObject 
} 
 
 @Override 
 public void run（）{
 String outString = myLongRunningMethod（this.myObject）; 
 System.out.println（outString）; 
} 
} 
  

问题/问题 / p>

我尝试的多线程代码似乎运行正常，因为我可以在控制台中看到一个预期的输出，但是我不能安全地写出输出 myLongRunningMethod 到使用 fileWriter 的文件。此外，我可以在分析器中看到所有的线程仍然活着，即使他们已经完成处理 myObject

在处理myObjectSet中的所有元素后将输出写入文件。即以多线程方式恢复我的原始功能。然后停止所有的线程。

有没有更容易/更好的实现？可能是涉及番石榴的一个期货
我真的需要使用MyResource类似于这里的代码 Semaphore

BTW，我尝试将 outString 字段 MyClassMT ，并尝试在 threadExec.execute（myClassMT）; 之后的代码中返回，不工作。

未在代码中显示，将为每个myObjectSet创建一个新的fileWriter。

让我知道如果需要更多的信息。

解决方案

如何在处理myObjectSet中的所有元素后将输出写入文件。

使用 Future< String> 是这样做的正确方法。你需要将 MyClassMT 变成 Callable< String> ，并使用 threadExec.submit ）。这会返回一个 Future< String> ，在提交完所有任务后，可以获取每个线程的工作结果。

  public class MyClassMT implements Callable< String> {
 ... 
 public String call（）{
 ... 
} 
  

您还可以使用 threadExec.invokeAll（...）来调用您的 Callable< String> / code>类。这会返回 List< Future< String>> 。

>

 列表< Future< String>> futures = threadExec.invokeAll（myClassMTCollection）; 
 //完成后一直关闭池提交
 threadExec.shutdown（）; 
 for（Future< String> future：futures）{
 //这可以抛出线程抛出的异常
 String result = future.get（）; 
} 
  

然后停止所有线程。 p>

提交所有任务后，您需要调用 shutdown（）在游泳池。提交的作业继续运行，但一旦作业完成，线程将被关闭。

ExecutorService threadExec = Executors.newFixedThreadPool（myObjectSet.size（））;

如果你这样做，你真的应该使用 Executors.newCachedThreadPool c $ c>它将在需要时分叉一个新的线程。真的，如果你的线程是CPU密集型的，你应该选择固定线程池数量的核心数，并且不为每个任务分配一个新的线程。 p>


This question is somewhat similar to Make an existing code in Java parallel/multithread

I didn't find answer specific to my question, so I am posting below.

I am trying to make an existing application multithreaded to decrease execution time.

Here is my (for brevity) simplified version existing application code

for(MyClass myObject : myObjectSet) {              
    String outputString=myLongRunningMethod(myObject);   
    fileWriter.append(outputString+"\n");
}

fileWriter.close();    

Here I attempted to make it multi-threaded.Becuase myLongRunningMethod is the slowest piece.

ExecutorService threadExec = Executors.newFixedThreadPool(myObjectSet.size());

// Semaphore implementation ResourcePool resourcePool = new ResourcePool(myObjectSet.size());

for(MyClass myObject:myObjectSet) {             
       Object key = resourcePool.getItem();      

       MyClassMT myClassMT = new MyClassMT(myObject);
       threadExec.execute(myClassMT);                      
 }

Where MyClassMT is:

public class MyClassMT implements Runnable{

    MyClass myObject;

    public MyClassMT(MyClass myObject) {
        this.myObject=myObject
    }

    @Override
    public void run() {
       String outString= myLongRunningMethod(this.myObject);
       System.out.println(outString);   
    }
}

Problem/Question

My attempted multi-threaded code seems to be running fine, because I can see an expected output in the console but I no more safely able to write the output of myLongRunningMethod to file using fileWriter. Also, I can see in profiler that all the threads are still alive even after they have done processing myObject

How to write output in a file after all the elements in myObjectSet is processed. i.e. to get back my original functionality in multi-threaded way. And then stop all the threads.

Is there any easier/better implementation? Maybe the one which involves guava Futures Am I really required to use MyResource which is similar to code here Semaphore

BTW, I tried setting outString as field of MyClassMT and tried getting it back in the code after just afterthreadExec.execute(myClassMT); , that didn't work.

Not shown in code, a new fileWriter will be created for every myObjectSet.

Please let me know if need more information on this.

解决方案

How to write output in a file after all the elements in myObjectSet is processed. i.e. to get back my original functionality in multi-threaded way.

Using the Future<String> is the right way of doing this. You need to turn MyClassMT into a Callable<String> and use threadExec.submit(myCallable). This returns a Future<String> which, after you have submitted all of your tasks can be used to get the results of each of the thread's work.

public class MyClassMT implements Callable<String> {
    ...
    public String call() {
       ...
    }

You can also use threadExec.invokeAll(...) to invoke all of your Callable<String> classes. This returns a List<Future<String>>.

Then you can do something like:

List<Future<String>> futures = threadExec.invokeAll(myClassMTCollection);
// always shutdown the pool once you are done submitting
threadExec.shutdown();
for (Future<String> future : futures) {
    // this can throw an exception that the thread threw
    String result = future.get();
}

And then stop all the threads.

After you have submitted all of the tasks, you need to call shutdown() on the pool. The submitted jobs continue to run but once the jobs are done the threads will be shutdown. If you don't do this your application will never finish.

ExecutorService threadExec = Executors.newFixedThreadPool(myObjectSet.size());

If you are doing something like this then you really should be using the Executors.newCachedThreadPool() which will fork a new thread whenever needed. Really, if your threads are CPU intensive, you should pick some number around he number of cores with the fixed thread pool and not allocate a new thread for every task.

这篇关于如何写多线程代码并累积单个文件中所有线程的输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！