提高BufferedReader速度 [英] Improve BufferedReader Speed
问题描述
我正在处理许多千兆字节的文本数据,我想知道是否有办法提高性能。例如,当经历10千兆字节的数据并且根本不处理它时,只需逐行迭代,大约需要3分钟。
I am crunching through many gigabytes of text data and I was wondering if there is a way to improve performance. For example when going through 10 gigabytes of data and not processing it at all, just iterating line by line, it takes about 3 minutes.
基本上我有一个dataIterator包装器包含一个BufferedReader。我不断调用这个迭代器,它返回下一行。
Basically I have a dataIterator wrapper that contains a BufferedReader. I continuously call this iterator, which returns the next line.
问题是创建的字符串数量是多少?或者可能是函数调用的数量。我真的不知道如何分析这个应用程序,因为它被编译为一个jar并用作STAF服务。
Is the problem the number of strings being created? Or perhaps the number of function calls. I don't really know how to profile this application because it get compiled as a jar and used as a STAF service.
任何和所有的想法都赞赏?
Any and all ideas appreciated?
推荐答案
让我们从基础开始:你的应用程序是I / O绑定的。由于对象分配,内存或CPU限制,您没有遇到性能不佳的问题。由于磁盘访问,您的应用程序运行缓慢。
Lets start from the basis: your application is I/O-bound. You are not suffering bad performance due to object allocation, or memory, or CPU limits. Your application is running slowly because of disk access.
如果您认为可以改进文件访问,则可能需要使用JNI进行低级编程。如果你自己更有效地处理它,可以改善文件访问,这需要在较低的层次上完成。
If you think you can improve file access, you might need to resort to lower-level programming using the JNI. File access can be improved if you handle it more efficiently by yourself, and that will need to be done on a lower level.
我不确定使用 java.nio
将为您提供更好的性能,尽管它可以让您在I / O运行时更自由地进行CPU /内存密集型操作。
I am not sure that using java.nio
will give you better performance by magnitude which you are looking for, although it might give you some more freedom in doing CPU/memory intensive operations while I/O is running.
原因是,基本上, java.nio
用选择器包装文件读取,让你在收到时收到通知读取缓冲区以供使用,确实为您提供了可能对您的性能有所帮助的异步行为。但读取文件本身是你的瓶颈,而 java.nio
并没有给你任何东西。
The reason being is that basically, java.nio
wraps the file reading with a selector, letting you be notified when a buffer is read for use, indeed giving you the asynchronous behavior which might help your performance a bit. But reading the file itself is your bottleneck, and java.nio
doesn't give you anything in that area.
所以先试试吧,但我不会把它的希望太高了。
So try it out first, but I wouldn't keep my hopes too high for it.
这篇关于提高BufferedReader速度的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!