Java多线程和迭代器，应该简单，初学者 [英] Java multithreading and iterators, should be simple, beginner

问题描述

首先，我想说我正在从python到更复杂的代码。我现在正在使用Java，而且我非常新。我知道Java非常擅长多线程，这很好，因为我用它来处理太字节数据。

First I'd like to say that I'm working my way up from python to more complicated code. I'm now on to Java and I'm extremely new. I understand that Java is really good at multithreading which is good because I'm using it to process terabytes of data.

数据输入只是输入迭代器而我有一个类封装了一个运行函数，该函数从迭代器获取一行，进行一些分析，然后将分析写入文件。线程必须彼此共享的唯一信息是它们写入的对象的名称。简单吧？我只是想让每个线程同时执行run函数，这样我们就可以快速迭代输入数据。在python中它很简单。

The data input is simply input into an iterator and I have a class that encapsulates a run function that takes one line from the iterator, does some analysis, and then writes the analysis to a file. The only bit of info the threads have to share with each other is the name of the object they are writing to. Simple right? I just want each thread executing the run function simultaneously so we can iterate through the input data quickly. In python it would b e simple.

from multiprocessing import Pool
f = open('someoutput.csv','w');
def run(x):
f.write(analyze(x))

p = Pool(8);
p.map(run,iterator_of_input_data);

所以在Java中，我有10K行分析代码，可以非常轻松地遍历我的输入传递它是我的run函数，它反过来调用我的所有分析代码将它发送到输出对象。

So in Java, I have my 10K lines of analysis code and can very easily iterate through my input passing it my run function which in turn calls on all my analysis code sending it to an output object.

public class cool {
    ...
    public static void run(Input input,output) {
        Analysis an = new Analysis(input,output);    
    }
    public static void main(String args[]) throws Exception {
        Iterator iterator = new Parser(File(input_file)).iterator();
        File output = File(output_object);
        while(iterator.hasNext(){
            cool.run(iterator.next(),output);
        }
    }
}

我想做的就是获取多个线程来获取迭代器对象并执行run语句。一切都是独立的。我一直在看java多线程的东西，但它用于通过网络交谈，共享数据等。这是很简单，因为我认为它是？如果有人可以指出我正确的方向我会很乐意做腿部工作。

All I want to do is get multiple threads taking the iterator objects and executing the run statement. Everything is independent. I keep looking at java multithreading stuff but its for talking over networks, sharing data etc. Is this is simple as I think it is? If someone can just point me in the right direction I would be happy to do the leg work.

谢谢

推荐答案

ExecutorService（ThreadPoolExecutor）将是Java equivelant。

A ExecutorService (ThreadPoolExecutor) would be the Java equivelant.

ExecutorService executorService =
    new ThreadPoolExecutor(
        maxThreads, // core thread pool size
        maxThreads, // maximum thread pool size
        1, // time to wait before resizing pool
        TimeUnit.MINUTES, 
        new ArrayBlockingQueue<Runnable>(maxThreads, true),
        new ThreadPoolExecutor.CallerRunsPolicy());

ConcurrentLinkedQueue<ResultObject> resultQueue;

while (iterator.hasNext()) {
    executorService.execute(new MyJob(iterator.next(), resultQueue))
}

将您的工作作为Runnable实施。

Implement your job as a Runnable.

class MyJob implements Runnable {
    /* collect useful parameters in the constructor */
    public MyJob(...) {
        /* omitted */
    }

    public void run() {
        /* job here, submit result to resultQueue */
    }
}

resultQueue 用于收集工作结果。

参见有关详细信息，请参阅 java api文档 。

See the java api documentation for detailed information.

这篇关于Java多线程和迭代器，应该简单，初学者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！