什么可以解决这个多线程场景比并发集合 [英] What could solves this multi-threaded scenario better than Concurrent collections

问题描述

我有一个持久的B +树，多个线程正在读取树的不同块，并对读取的数据执行一些操作。 有趣的部分：每个线程产生一组结果，作为最终用户，我想在一个地方看到所有的结果。 我做什么：一个ConcurentDictionary和所有线程正在写入它。

一切顺利。但应用程序是时间关键，一个额外的秒表示完全不满意。 ConcurentDictionary因为线程安全开销本身比Dictionary慢。

我可以使用Dictionary，然后每个线程将结果写入不同的字典。但是，我会有合并不同字典的问题。

。

我的问题：

$

1. 如果不是（1），那么我如何合并最佳不同的字典。鉴于（a）复制项目一个一个和（b） LINQ是已知的解决方案，并不是预期的最佳）：


2. 如果不是（2）;-)您会建议什么？

b

快速信息：

• #Thread = processorCount。应用程序可以在标准的笔记本电脑（即4线程）或高端服务器（即<32线程）


• 项目计数上运行。该树通常包含超过1.0E + 12个项目。

解决方案

结果字典的锁定/构建每个线程需要3700ms，而实际处理逻辑只需要300ms。

我建议作为一个实验，让每个线程创建自己的本地结果字典。

如果构建本地字典增加超过300ms，那么它将会创建一个字典。将无法满足您的时间限制。因为没有任何锁定或任何尝试合并结果，它已经花了太长时间。

更新

似乎您可以支付合并价格沿着，锁定导致线程闲置了相当长的时间，或者在后处理合并中支付价格。但是核心问题是锁定意味着你没有充分利用可用的CPU。

从你的核心获得最大性能的唯一真正的解决方案是，阻塞字典实现也是线程安全的。我找不到.NET实现，但发现了一个研究文件，详细说明了一个算法，表明它是可能的。

正确实现这样的算法不是微不足道，但会是有趣！

可扩展和无锁并发词典

I have a persistent B+tree, multiple threads are reading different chunks of the tree and performing some operations on read data. Interesting part: each thread produces a set of results, and as end user I want to see all the results in one place. What I do: one ConcurentDictionary and all threads are writing to it.

Everything works smooth this way. But the application is time critical, one extra second means a total dissatisfaction. ConcurentDictionary because of the thread-safety overhead is intrinsically slow compared to Dictionary.

I can use Dictionary, then each thread will write results to distinct dictionaries. But then I'll have the problem of merging different dictionaries.

.

My Questions:

1. Are concurrent collections a good decision for my scenario ?
2. If Not(1), then how would I merge optimally different dictionaries. Given that, (a) copying items one-by-one and (b) LINQ are known solutions and are not as optimal as expected :)
3. If Not(2) ;-) What would you suggest instead ?

.

A quick info:

• #Thread = processorCount. The application can run on a standard laptop (i.e., 4 threads) or high-end server (i.e., <32 threads)
• Item Count. The tree usually holds more than 1.0E+12 items.

解决方案

From your timings it seems that the locking/building of the result dictionary is taking 3700ms per thread with the actual processing logic taking just 300ms.

I suggest that as an experiment you let each thread create its own local dictionary of results. Then you can see how much time is spent building the dictionary compared to how much is the effect of locking across threads.

If building the local dictionary adds more than 300ms then it will not be possible to meet your time limit. Because without any locking or any attempt to merge the results it has already taken too long.

Update

It seems that you can either pay the merge price as you go along, with the locking causing the threads to sit idle for a significant percentage of time, or pay the price in a post-processing merge. But the core problem is that the locking means you are not fully utilising the available CPU.

The only real solution to getting maximum performance from your cores is it use a non-blocking dictionary implementation that is also thread safe. I could not find a .NET implementation but did find a research paper detailing an algorithm that would indicate it is possible.

Implementing such an algorithm correctly is not trivial but would be fun!

Scalable and Lock-Free Concurrent Dictionaries

这篇关于什么可以解决这个多线程场景比并发集合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！