Java线程在处理结束时变慢 [英] Java threads slow down towards the end of processing
问题描述
我有一个Java程序，它接收一个包含文本文件列表的文本文件，并分别处理每一行。为了加快处理速度，我使用带有24个线程的FixedThreadPool的ExecutorService来使用线程。该机器有24个内核和48GB内存。
我正在处理的文本文件有250万行。我发现，对于前230万行左右，CPU运行率很高，运行良好。然而，超过某些点（大约在2.3行），性能退化，只使用了一个CPU，我的程序几乎停止了。
我调查了很多原因，确保关闭所有文件句柄，并增加提供给JVM的内存量。但是，无论我改变什么，性能总是会降低到最后。我甚至尝试过包含更少行的文本文件，并在处理文件结束时再次降低性能。
除了标准的Java并发库之外，代码还使用Lucene库进行文本处理和分析。
当我不对此代码进行线程化时，性能是恒定的，并且不会在最后退化。我知道这是一个黑暗的镜头，很难描述发生了什么，但我想我会看到是否有人有任何想法可能会导致这种性能退化到底。
编辑
收到评论后，我已粘贴堆栈跟踪此处。如您所见，它似乎没有任何线程阻塞。此外，在分析时，当事情变慢时，GC不是100％。事实上，大多数时候CPU和GC利用率都是0％，CPU偶尔会出现一些文件，然后再次停止。
执行线程的代码
BufferedReader read = new BufferedReader（new FileReader（inputFile））;
ExecutorService executor = Executors.newFixedThreadPool（NTHREADS）;
字符串行;
while（（line = read.readLine（））！= null）{//索引每一行
Runnable worker = new CharikarHashThreader（line，bits，minTokens）;
executor.execute（worker）;
}
read.close（）;
这听起来很像垃圾收集/内存问题。
当垃圾收集运行时，它会暂停所有线程，以便GC线程可以执行是这个可收集的垃圾分析而不会改变它。当GC正在运行时，你会看到100个正好1个线程，其他线程将停留在0％。
我会考虑添加一些Runtime.freeMemory （）调用（或使用分析器）查看GC期间是否停止运行。
我还尝试在第一次运行程序你的文件的10k行，看看是否有效。
我还要看看你的程序在使用StringBuilders时是否构建了太多的中间字符串。 / p>
听起来像你需要描述你的内存使用情况。
I have a Java program that takes in a text file containing a list of text files and processes each line separately. To speed up the processing, I make use of threads using an ExecutorService with a FixedThreadPool with 24 threads. The machine has 24 cores and 48GB of RAM.
The text file that I'm processing has 2.5 million lines. I find that for the first 2.3 million lines or so things run very well with high CPU utilization. However, beyond some point (at around the 2.3 lines), the performance degenerates with only a single CPU being utilized and my program pretty much grinding to a halt.
I've investigated a number of causes, made sure all my file handles are closed, and increased the amount of memory supplied to the JVM. However, regardless of what I change, performance always degrades towards the end. I've even tried on text files containing fewer lines and once again performance decreases towards the end of processing the file.
In addition to the standard Java concurrency libraries, the code also makes use of Lucene libraries for text processing and analysis.
When I don't thread this code, the performance is constant and doesn't degenerate towards the end. I know this is a shot in the dark and it's hard to describe what is going on, but I thought I would just see if anyone has any ideas as to what might be causing this degeneration in performance towards the end.
Edit
After the comments I've received, I've pasted a stack trace here. As you can see, it doesn't appear as if any of the threads are blocking. Also, when profiling, the GC was not at 100% when things slowed down. In fact, both CPU and GC utilization were at 0% most of the time, with the CPU spiking occasionally to process a few files and then stopping again.
Code for executing threads
BufferedReader read = new BufferedReader(new FileReader(inputFile));
ExecutorService executor = Executors.newFixedThreadPool(NTHREADS);
String line;
while ((line = read.readLine()) != null) { //index each line
Runnable worker = new CharikarHashThreader(line, bits, minTokens);
executor.execute(worker);
}
read.close();
This sounds alot like a Garbage Collection / Memory Issue.
When the Garbage Collection runs it pauses all threads so that the GC thread can do its "is this collectable garbage" analysis without things changing on it. While the GC is running you'll see exactly 1 thread at 100%, the other threads will be stuck at 0%.
I would consider adding a few Runtime.freeMemory() calls (or using a profiler) to see if the "grind to a halt" occurs during GC.
I'd also trying running your program on just the first 10k lines of your file to see if that works.
I'd also look to see if your program is building too many intermediate Strings when it should be using StringBuilders.
It sounds to me like you need to profile your memory usage.
这篇关于Java线程在处理结束时变慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
