在Kafka Streams应用程序中关闭或不关闭RocksDB Cache和WriteBufferManager [英] To close or to not close RocksDB Cache and WriteBufferManager in kafka streams app

问题描述

我目前正在通过扩展 RocksDBConfigSetter 界面在流应用程序中使用自定义的 RocksDB 配置.我发现有关关闭 cache &的文档相互冲突 writeBufferManager 实例.

I am currently playing around with a custom RocksDB configuration in my streams app by extending RocksDBConfigSetter interface. I see conflicting documentation around closing cache & writeBufferManager instances.

现在，我看到了javadoc&文档页面之一建议我们需要关闭覆盖了 RocksObject 的所有实例( Cache 和 WriteBufferManager 实例都扩展了该类) RocksDBConfigSetter#close()方法.

Right now, I see that the javadoc & one of the documentation page suggests that we need to close all the instances that extend RocksObject (both Cache & WriteBufferManager instances extend this class) in the overridden RocksDBConfigSetter#close() method.

但是，内存管理文档页面建议我们将这些实例创建为静态实例，而不在覆盖的 RocksDBConfigSetter#close()中关闭 Cache 和 WriteBufferManager 实例.方法.

However, the memory management documentation page suggests that we create these instances as static instances and not close the Cache and WriteBufferManager instances in the overridden RocksDBConfigSetter#close() method.

不确定在此应遵循的内容.如果有人可以帮助我了解哪个文档是正确的，以及如果我们想通过传入自定义的rocksdb配置来限制内存使用的首选方式，将不胜感激.

Not sure what to follow here. Would appreciate if anyone can help me understand which documentation is correct and what is the preferred way if we would want to limit the memory usage by passing in a custom rocksdb configuration.

如果我们将它们声明为静态，可以不关闭这些实例吗?

Is it ok to not close these instances if we declare them as static?

推荐答案

两个文档都是正确的.

在第一文档，您提到缓存是对象的字段.如果您不关闭 close()中的缓存，则在Kafka Streams关闭相应的RocksDB状态存储后，直到JVM退出，它都会泄漏堆外内存.

In the first documentation you mention the cache is a field of the object. If you do not close the cache in close(), it will leak off-heap memory after Kafka Streams closed the corresponding RocksDB state store until the JVM exits.

在第二篇文档中您提到缓存和写缓冲区管理器是静态的.如果在 close()中关闭它们，则被Kafka Streams关闭的第一个RocksDB状态存储将同时关闭它们，而所有其他RocksDB状态存储将很可能崩溃，因为它们的缓存和写入缓冲区管理器已关闭.

In the second documentation you mention the cache and the write buffer manager are static. If you close them in a close() the first RocksDB state store that is closed by Kafka Streams will close both and all other RocksDB state stores will most likely crash because their cache and write buffer manager was closed.

当我们没有回调的类加载器卸载该类时，您将需要关闭静态缓存和静态写缓冲区管理器.我认为卸载是在JVM退出时发生的，因此直到JVM退出之后，堆外内存都不会泄漏，然后无论如何都要释放堆外内存.

You would need to close a static cache and a static write buffer manager when the class is unloaded by the class loader for which we do not have a callback. I think unloading happens when the JVM exits, so no off-heap memory is leaked until the JVM exits and afterwards the off-heap memory is freed anyways.

关于限制RocksDB的内存使用的问题，答案取决于您要限制的范围.您是否要限制一个RocksDB实例的内存使用，还是要限制一个Kafka Streams客户端中使用的所有RocksDB实例的内存?对于前者，您应该使用第一个文档中的示例.对于后者，您应该使用第二个文档中的示例.

Regarding your question about limiting the memory usage of RocksDB the answer depends on what you want to limit. Do you want to limit the memory used by one single instance of RocksDB or do you want to limit the memory of all RocksDB instances that are used within one Kafka Streams client? For the former you should use the example in the first documentation. For the latter, you should use the example in the second documentation.

这篇关于在Kafka Streams应用程序中关闭或不关闭RocksDB Cache和WriteBufferManager的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！