为C ++中的许多进程处理中央数据缓冲区 [英] handling central data buffer for many processes in C++
问题描述
我遇到以下问题，无法决定如何处理：
我有一个类 Reader ，每1 / T秒获得一块数据（实际上数据来自视频帧，每秒30帧）。这些块将被传递给处理这些块并输出决定的多个对象 Detectors 。然而，在作出决定之前，每个检测器需要读取的块的数量，例如，一些可能只需要一个块，大约51个。
我想有一个数据缓冲区 Reader 读取数据块，实现发布/订阅者注册每个检测器，并在数据缓冲区中有足够数量的帧以供其处理时向其发送信号。这是一个好的方法吗？此外，什么是最好的方式来管理缓冲区和 Detectors 从它读取数据，而无需自己的副本？共享指针？
非常感谢！
C
关于设计问题：
尝试从一个简单的设计开始。例如，仅使用线程，并使用它自己的同步队列*将每个订阅者的shared_ptr传递给作业。由于对数据的访问是只读的，并且AFAICR，boost :: shared_ptr是多线程安全的这样使用，没有同步问题，数据被自动清除。不要担心内存实现（现在），只是确保你使用有限的内存（o（1））（如你所说，大约51 shared_ptrs每个订阅者/线程。
当你有这个工作框架时，你将能够根据遇到的问题开始优化。如果realocations是问题，你可以移动到一个环形缓冲区（如bcat建议）。或者您可以用池分配器替换您的分配器（/ new运算符）。如果你有很多订阅者，它可能是有效的合并队列到一个由所有的线程使用。这需要更多的信息（如果一个线程由于很长的计算非常慢，你有一些方法来表示它停止处理？或者队列增长？如果是这种情况，循环缓冲区可能不工作所以很好...），并可能有其并发症，但记住，我们只是试图保存由shared_ptrs（而不是工作）占用的空间。
底线，尽量避免过早优化。
祝你好运
b
*同步队列 - 线程之间的队列。 push（j）添加作业和pop（）等待队列不为空并返回顶部作业（不同于stl :: queue，当队列被多个线程读取时，这是很重要的）。我通常实现它通过包装一个stl ::队列并使用boost :: mutex保护它。
I ran into the following problem and cannot decide how to proceed:
I have a class, Reader, getting a chunk of data every 1/T seconds (actually the data is from video frames, 30 frames per second). The chunks are to be passed to several objects, Detectors that process the chunks and output a decision. However, the number of chunks that each detector needs to read before making a decision varies, e.g. some may need only one chunk, some 51.
I am thinking of having a data buffer where Reader places the read data chunks, implementing publish/subscriber to register each Detector and sending it a signal when there are enough number of frames in the data buffer for it to process. Is this a good approach? Also, what's the best way to manage the buffer and have Detectors read data from it without making their own copies? Shared pointers?
Thanks a lot!
C
I think (also based on your comment to Maciek) you have to start by understanding the difference between threads and processes and how they can communicate.
Regarding the design problem: Try to start with a simple design. for instance, using only threads and passing each of the subscribers a shared_ptr to the job using it's own synchronized queue*. Since the access to the data is read-only and, AFAICR, boost::shared_ptr is multi-threading safe for such a use, there are no synchronization problems and the data is cleaned automatically. Don't worry about memory realocations (yet), Just make sure you are using a finite amount of memory (o(1)) (as you said, about 51 shared_ptrs at most) per subscriber/thread.
When you'll have this working skeleton, you will be able to start optimizing based on the problems you encounter. If realocations are the problem, you can move to a ring buffer (as suggested by bcat). or you can replace your allocator (/new operator) with a pool allocator. if you have many subscribers, it might be effective to merge the queues into a single one used by all the threads. Doing that requires more information (what if one thread is very slow due to a very long computation? do you have some way to signal it to stop processing? or should the queue grow? if this is the case, a cyclic buffer may not work so well...) and may have its complications, but remember we are only trying to save the room occupied by the shared_ptrs (and not the jobs).
Bottom line, try to avoid premature optimizations. instead, write it with reasonable optimization and extendability in design and go on from there based on what you learn.
Good luck
* synchronized queue - a queue between threads. push(j) adds the job and pop() waits until the queue is not empty and returns the top job (unlike stl::queue. This is important when the queue is read by more than one thread). I usually implement it by wrapping an stl::queue and protecting it using boost::mutex.
这篇关于为C ++中的许多进程处理中央数据缓冲区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
