速度更快的 DelayQueue remove()? [英] DelayQueue with higher speed remove()?

问题描述

我有一个项目跟踪超过 50 万个对象的状态信息，程序每秒接收 10k 次关于这些对象的更新，更新包括新建、更新或删除操作.

I have a project that keeps track of state information in over 500k objects, the program receives 10k updates/second about these objects, the updates consist of new, update or delete operations.

作为程序内务管理的一部分，必须大约每五分钟对这些对象执行一次，为此我将它们放置在实现 Delayed 接口的 DelayQueue 中，允许 DelayQueue 的阻塞功能来控制这些对象的内部管理.

As part of the program house keeping must be performed on these objects roughly every five minutes, for this purpose I've placed them in a DelayQueue implementing the Delayed interface, allowing the blocking functionality of the DelayQueue to control house keeping of these objects.

• 在新建时，一个对象被放置在 DelayQueue 上.

更新后，对象从 DelayQueue 中 remove() 取出，更新后重新插入到更新信息指定的新位置.

Upon update, the object is remove()'d from the DelayQueue, updated and then reinserted at it's new position dictated by the updated information.

删除后，对象从DelayQueue中remove().

我面临的问题是，一旦队列通过了大约 450k 个对象，remove() 方法就会变成一个非常长的操作.

The problem I'm facing is that the remove() method becomes a prohibitively long operation once the queue passes around 450k objects.

该程序是多线程的，一个线程处理更新，另一个线程处理内务.由于 remove() 延迟，我们会遇到严重的锁定性能问题，最终更新线程缓冲区会消耗所有堆空间.

The program is multithreaded, one thread handles updates and another the house keeping. Due to the remove() delay, we get nasty locking performance issues, and eventually the update thread buffer's consumes all of the heap space.

我设法通过创建一个 DelayedWeakReference(扩展 WeakReference 实现 Delayed) 来解决这个问题，它允许我将影子"对象留在队列中，直到它们正常过期.

I've managed to work around this by creating a DelayedWeakReference (extends WeakReference implements Delayed), which allows me to leave "shadow" objects in the queue until they would expire normally.

这消除了性能问题，但会导致内存需求显着增加.这样做会为每个实际需要在队列中的对象产生大约 5 个 DelayedWeakReference.

This takes the performance issue away, but causes an significant increase in memory requirements. Doing this results in around 5 DelayedWeakReference's for every object that actually needs to be in the queue.

是否有人知道带有附加跟踪的 DelayQueue 允许快速 remove() 操作?或者有什么更好的方法可以在不消耗更多内存的情况下处理这个问题?

Is anyone aware of a DelayQueue with additional tracking that permits fast remove() operations? Or has any suggestions of better ways to handle this without consuming significantly more memory?

推荐答案

我花了一些时间考虑这个，
但在阅读了您的有趣问题几分钟后，以下是我的想法:
A.如果你的对象有某种ID，用它来散列，实际上没有一个延迟队列，而是有N个延迟队列.
这会将锁定因子减少 N.
会有一个中央数据结构，
拿着这N个队列.由于 N 已预先配置，
您可以在系统启动时创建所有 N 个队列.

took me some time to think about this,
but after reading your interesting question for some minutes, here are my ideas:
A. if you objects have some sort of ID, use it to hash, and actually don't have one delay queue, but have N delay queues.
This will reduce the locking factor by N.
There will be a central data structure,
holding these N queues. Since N is preconfigured,
you can create all N queues when the system starts.

这篇关于速度更快的 DelayQueue remove()?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！