TPL读取和转换大约50 GB的输入文件到文本格式有多好 [英] How good would TPL be to read and transform around 50 GB of input files to text format

问题描述

您好

我即将开始一个项目，该项目涉及阅读每天到达的大约50 GB的输入文件，转换其内容并将其输出到文本文件。可能有大约200个文件加起来就是那么多GB。

I am about to begin a project which involves reading some 50 GB of input files arriving every day, transforming their content and outputting them to a text file. There could be some 200 files which add up to that many GB.

我只是想知道TPL的技术选择有多好？我正在考虑同时处理多个文件，并希望TPL提供的并行功能可以提高处理效率。我使用的硬件类型是
，如256 GB的RAM，以及4个启用超线程的八核处理器，相当于64个逻辑核心。

I just wish to know how good a choice of technology is TPL for this? I am thinking of processing multiple files concurrently and hoping that the parallel capabilities provided by TPL would make processing more efficient. The kind of hardware I am using is like 256 GB of RAM, and 4 octal core processors with hyperthreading enabled, which amount to 64 logical cores.

TPL如何被利用最佳地执行这种处理？有什么想法吗？

How could TPL be leveraged to perform this kind of processing optimally? Any thoughts?

Ringoo

推荐答案

嗨Ringoo，

Hi Ringoo,

基本上TPL只是Threads上易于管理的包装器，因此即使您考虑使用普通线程，也应考虑使用TPL，因为最终线程将在后台运行。这些是由microsoft开发的非常智能的API
意味着您可以轻松地在具有不同核心数的计算机上切换它们，这些会自动适应这种情况。

Basically TPL is just a easy manageable wrapper on Threads, so even if you are thinking of using normal Threading you should consider TPL over that as at final the threads will be running behind the scenes. These are a very intelligent API developed by microsoft means you can easily switch them on machines with different cores count, these automatically adapts to that kind of scenario.

还有其他的其中内置的功能包括取消，进度，多个任务之间的依赖关系，异常处理等，这些都需要花费很多精力和时间才能通过线程开发。

Also there are others features inbuilt in them like cancellation, progress, dependency between multiple tasks, exception handling, etc which would take lot of effort and time when developed through threading.

另外正如我所说他们也使用线程在低级别，所以它们与使用Threads相同，在核心可管理性方面更好。性能也取决于你所提到的非常好的机器硬件。

Also as i said they too use threads at low level, so they are same as using Threads, even better in core manageability. And performance also depends on the machine hardware which is really very good as you have mentioned.

谢谢

Amit Tonk

Amit Tonk

-------------

-------------

如果这篇文章对你有所帮助，请将此标记为答案。

Please mark this as answer if this post help you.

这篇关于TPL读取和转换大约50 GB的输入文件到文本格式有多好的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！