从同一个文件读取的多个线程......是否可能？ [英] Multiple threads reading from the same file... is it possible?

问题描述

你好......我有一个需要处理的巨大文本文件。在

时刻，我将它以小块（x行数）加载到内存中

并以这种方式处理它。我希望这个过程更快，所以我想要b $ b喜欢尝试创建多个线程，然后让它们加载不同的

块的文件。同时和异步处理它。


是否可以做类似的事情，如果是这样的话，那么需要什么？b $ b这样做？ />
WATYF

Hi there... I have a huge text file that needs to be processed. At the
moment, I''m loading it into memory in small chunks (x amount of lines)
and processing it that way. I''d like the process to be faster, so I''d
like to try creating multiple threads, and having them load different
chunks of the file at the same time and process it asynchronously.

Is it possible to do something like that, and if so, what would be
needed to do so?
WATYF

推荐答案

首先，您必须为多个线程设计一种方法来了解哪些行

来处理。如果你打算以某种方式删除处理过的行，你会打开文件并调用读/写锁，这样只有一个线程

可以在时间。如果你这样做，那你显然必须

保护你的文件打开用Try / Catch b / c两个或更多线程可以尝试

同时打开文件并且只有一个人可以一次锁定它。


如果你不打算从文件中删除，那么你仍需要为每个文件设计

a方式线程知道要处理什么。你可以有一个全局的
计数器/指针，在从线程更新时用互斥锁锁定，告诉

你下一个线程的起始位置。但是，如果由于某种原因导致一个或多个线程中止
，则计数器/指针不会更新为

反映未处理的记录。

如果你只想让多个线程同时读取文件，

跟上另一个线程是否已经处理了一条记录，它们可以做b
$ b通过使用读/共享访问选项打开文件。


HTH
Le *@KnowDotNet.com

免费试用我们最新的省时工具Visual Class Organizer 30天。

http://www.knowdotnet.com/articles/V...oductHome.html


" WATYF"写道：

First, you have to devise a way for the multiple threads to know which lines
to process. If you plan to delete the processed lines somehow, you would
have to open the file with read/write lock invoked so that only one thread
can get to it at a time. If you do it this way, then you obviously must
protect your file open with Try/Catch b/c two or more threads can be trying
to open the file concurrently and only one can lock it at one time.

If you are not going to delete from the file, then you still have to devise
a way for each thread to know what to process. You could have a "global"
counter/pointer, locked with a mutex while updating from a thread, that told
you where the next thread was to start. However, if one or more threads
abort for some reason, then the counter/pointer would not be updated to
reflect the unprocessed records.

If you simply want to have multiple threads reading the file concurrently,
keeping up with whether a record has been processed by another thread, they
can do that by opening the file with read/share access options.

HTH
Le*@KnowDotNet.com

Try our latest time saving tool, Visual Class Organizer, free for 30 days.

http://www.knowdotnet.com/articles/V...oductHome.html

"WATYF" wrote:

你好......我有一个需要处理的巨大文本文件。在

时刻，我将它以小块（x行数）加载到内存中

并以这种方式处理它。我希望这个过程更快，所以我想要b $ b喜欢尝试创建多个线程，然后让它们加载不同的

块的文件。同时和异步处理它。


是否可以做类似的事情，如果是这样的话，那么需要什么？b $ b这样做？ />

WATYF

Hi there... I have a huge text file that needs to be processed. At the
moment, I''m loading it into memory in small chunks (x amount of lines)
and processing it that way. I''d like the process to be faster, so I''d
like to try creating multiple threads, and having them load different
chunks of the file at the same time and process it asynchronously.

Is it possible to do something like that, and if so, what would be
needed to do so?
WATYF

为什么不考虑使用单个线程读取文件并让其他人

线程处理块。这对你来说应该更容易处理。


问候，


Trevor Benedict

MCSD

" WATYF" < WA **** @ gmail.com写信息

news：11 ********************** @ k35g2000prh.googlegr oups.com ...

Why not think about reading the file using a single thread and let other
thread(s) process the chunks. This must be easier for you to deal with.

Regards,

Trevor Benedict
MCSD

"WATYF" <WA****@gmail.comwrote in message
news:11**********************@k35g2000prh.googlegr oups.com...

你好......我有一个需要处理的巨大文本文件。在

时刻，我将它以小块（x行数）加载到内存中

并以这种方式处理它。我希望这个过程更快，所以我想要b $ b喜欢尝试创建多个线程，然后让它们加载不同的

块的文件。同时和异步处理它。


是否可以做类似的事情，如果是这样的话，那么需要什么？b $ b这样做？ />

WATYF

Hi there... I have a huge text file that needs to be processed. At the
moment, I''m loading it into memory in small chunks (x amount of lines)
and processing it that way. I''d like the process to be faster, so I''d
like to try creating multiple threads, and having them load different
chunks of the file at the same time and process it asynchronously.

Is it possible to do something like that, and if so, what would be
needed to do so?
WATYF

我需要读数最多的多线程。从

文件中读取的时间最长。实际上处理

文件的块比将它加载到内存所花费的时间要少得多。


我可能已经弄明白了，尽管......虽然...我不确定它是否真的按照我的预期做了。


我使用了TextReaders / TextWriters的组合创建了

使用TextReader.Synchronized和同步的arraylists

是使用ArrayList.Synchronized创建的。 Streams的FileOptions之一是
，它是FileOptions.Asynchronous，所以当打开

文件进行读/写时我用它。


WATYF

9月20日下午1:50，Trevor Benedict < trevorn ... @ yahoo.comwrote：

I need the reading to be multi-threaded the most. Reading from the
file is what takes the longest. Actually processing the chunks of the
file takes much less time than the loading it into memory.

I may have figured it out, though... although I''m not sure yet if it''s
really doing what I expect it to be doing.

I used a combination of TextReaders/TextWriters that were created
using TextReader.Synchronized and also synchronized arraylists which
were created using ArrayList.Synchronized. One of the FileOptions for
the Streams is FileOptions.Asynchronous, so I used that when opening
the files for reading/writing.

WATYF
On Sep 20, 1:50 pm, "Trevor Benedict" <[e-mail]:

为什么不考虑使用单个线程读取文件并让其他

线程（s）处理块。这对你来说应该更容易处理。


问候，


Trevor Benedict

MCSD

这篇关于从同一个文件读取的多个线程......是否可能？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！