递归函数中每个进程的多线程或并行处理 [英] multi-threading or parallel processing for each process within recursive function

问题描述

我的代码确实收集目录列表并将其保存到CSV输出（在我的示例中为DirectoryOutputfile.csv）。使用目录列表CSV文件我使用递归函数并为每个目录执行任务。

I have my code which does collect the list of directories and saves it to an CSV output (DirectoryOutputfile.csv in my example). Using the directory list CSV file I use a recursive function and perform the task for each directory.

任务涉及输入每个目录并启动流程并保存输出到另一个CSV。每个目录都会发生这种情况并保存输出。

The task involves inputting the each directory and initiating the process and saves the output to another CSV. This happens for each directory and saves the output.

此外，我将完成的目录保存到另一个CSV中，以了解在LIVE基础上完成了多少个目录。

Also I am saving the completed directories into another CSV to know how many are being complete to view on LIVE basis.

我问： 我没有在我的情况下使用任何线程，所以想让进程
并行运行使其快速并如上所述保存输出。我是线程新手，请建议我如何在代码中使用并行处理或多线程来加速，因为列表中有大约750K的目录。

My ask: I am not using any threading in my case, so would like to make the process run in parallel to make it fast and have the output saved as mentioned above. I am new to the threading, kindly suggest how can I use parallel processing or multi-threading in my code to speed up as there are around ~750K directories in the list.

我不知道在我的代码中我需要在哪里实现它，因为我在多个地方使用streamwriter和streamreader进行写入和读取操作。

I am not sure where would I need to implement this in my code as I have write and read operations using streamwriter and streamreader at multiple places.

Code下面：

谢谢， Dhilip

Thanks, Dhilip

推荐答案

因此，在（（Directoryline = streamReaderDirectory.ReadLine（））！= null）开始的循环是您阅读每个循环的地方文件中的目录名称。  ？右 很容易将整个循环的内容移动到一个单独的函数中，然后
触发一个线程来进行处理，比如

So the loop that starts while((Directoryline = streamReaderDirectory.ReadLine()) != null) is where you read each directory name from the file.  Right?  It would be easy enough to move the contents of that entire loop into a separate function, then fire off a thread to do the processing, something like

    Thread thread = new Thread（Class.HandleOneDirectory）;

    thread.Start（Directoryline）;

    Thread thread = new Thread( Class.HandleOneDirectory );
    thread.Start( Directoryline );

使用线程池可能更好。 如果你尝试启动750,000个线程，你就会引发骚乱。 另外，请记住，从多个线程写入单个文件时必须小心。 您可能希望同步对DirectoryLineUpdatedSofarInCSV的访问。

It might be better to use a thread pool.  If you try to start 750,000 threads, you'll cause a riot.  Also, remember that you have to be careful when writing to a single file from multiple threads.  You might want to synchronize access to DirectoryLineUpdatedSofarInCSV.

这篇关于递归函数中每个进程的多线程或并行处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！