提高BufferedReader速度 [英] Improve BufferedReader Speed

问题描述

我正在处理许多千兆字节的文本数据，我想知道是否有办法提高性能。例如，当经历10千兆字节的数据并且根本不处理它时，只需逐行迭代，大约需要3分钟。

I am crunching through many gigabytes of text data and I was wondering if there is a way to improve performance. For example when going through 10 gigabytes of data and not processing it at all, just iterating line by line, it takes about 3 minutes.

基本上我有一个dataIterator包装器包含一个BufferedReader。我不断调用这个迭代器，它返回下一行。

Basically I have a dataIterator wrapper that contains a BufferedReader. I continuously call this iterator, which returns the next line.

问题是创建的字符串数量是多少？或者可能是函数调用的数量。我真的不知道如何分析这个应用程序，因为它被编译为一个jar并用作STAF服务。

Is the problem the number of strings being created? Or perhaps the number of function calls. I don't really know how to profile this application because it get compiled as a jar and used as a STAF service.

任何和所有的想法都赞赏？

Any and all ideas appreciated?

推荐答案

让我们从基础开始：你的应用程序是I / O绑定的。由于对象分配，内存或CPU限制，您没有遇到性能不佳的问题。由于磁盘访问，您的应用程序运行缓慢。

Lets start from the basis: your application is I/O-bound. You are not suffering bad performance due to object allocation, or memory, or CPU limits. Your application is running slowly because of disk access.

如果您认为可以改进文件访问，则可能需要使用JNI进行低级编程。如果你自己更有效地处理它，可以改善文件访问，这需要在较低的层次上完成。

If you think you can improve file access, you might need to resort to lower-level programming using the JNI. File access can be improved if you handle it more efficiently by yourself, and that will need to be done on a lower level.

我不确定使用 java.nio 将为您提供更好的性能，尽管它可以让您在I / O运行时更自由地进行CPU /内存密集型操作。

I am not sure that using java.nio will give you better performance by magnitude which you are looking for, although it might give you some more freedom in doing CPU/memory intensive operations while I/O is running.

原因是，基本上， java.nio 用选择器包装文件读取，让你在收到时收到通知读取缓冲区以供使用，确实为您提供了可能对您的性能有所帮助的异步行为。但读取文件本身是你的瓶颈，而 java.nio 并没有给你任何东西。

The reason being is that basically, java.nio wraps the file reading with a selector, letting you be notified when a buffer is read for use, indeed giving you the asynchronous behavior which might help your performance a bit. But reading the file itself is your bottleneck, and java.nio doesn't give you anything in that area.

所以先试试吧，但我不会把它的希望太高了。

So try it out first, but I wouldn't keep my hopes too high for it.

这篇关于提高BufferedReader速度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！