在 kafka 流应用程序中关闭或不关闭 RocksDB Cache 和 WriteBufferManager [英] To close or to not close RocksDB Cache and WriteBufferManager in kafka streams app

问题描述

我目前正在通过扩展 RocksDBConfigSetter 接口在我的流应用程序中使用自定义 RocksDB 配置.我看到有关关闭 cache & 的冲突文档writeBufferManager 实例.

I am currently playing around with a custom RocksDB configuration in my streams app by extending RocksDBConfigSetter interface. I see conflicting documentation around closing cache & writeBufferManager instances.

现在，我看到 javadoc &文档页面之一 建议我们需要关闭所有扩展 RocksObject 的实例(Cache 和 WriteBufferManager 实例都扩展了这个类)RocksDBConfigSetter#close() 方法.

Right now, I see that the javadoc & one of the documentation page suggests that we need to close all the instances that extend RocksObject (both Cache & WriteBufferManager instances extend this class) in the overridden RocksDBConfigSetter#close() method.

但是，内存管理文档页面 建议我们将这些实例创建为静态实例，而不是关闭重写的 RocksDBConfigSetter#close() 中的 Cache 和 WriteBufferManager 实例方法.

However, the memory management documentation page suggests that we create these instances as static instances and not close the Cache and WriteBufferManager instances in the overridden RocksDBConfigSetter#close() method.

不知道在这里要遵循什么.如果有人能帮助我了解哪些文档是正确的，以及如果我们想通过传入自定义的 Rocksdb 配置来限制内存使用量，我们将不胜感激.

Not sure what to follow here. Would appreciate if anyone can help me understand which documentation is correct and what is the preferred way if we would want to limit the memory usage by passing in a custom rocksdb configuration.

如果我们将它们声明为静态实例，可以不关闭它们吗?

Is it ok to not close these instances if we declare them as static?

推荐答案

两个文档都是正确的.

在 首先文档 你提到的缓存是对象的一个​​字段.如果没有在close()中关闭缓存，在Kafka Streams关闭相应的RocksDB状态存储后，直到JVM退出，它才会泄漏堆外内存.

In the first documentation you mention the cache is a field of the object. If you do not close the cache in close(), it will leak off-heap memory after Kafka Streams closed the corresponding RocksDB state store until the JVM exits.

在第二个文档中 你提到缓存和写缓冲区管理器是静态的.如果您在 close() 中关闭它们，Kafka Streams 关闭的第一个 RocksDB 状态存储将同时关闭，所有其他 RocksDB 状态存储很可能会崩溃，因为它们的缓存和写入缓冲区管理器已关闭.

In the second documentation you mention the cache and the write buffer manager are static. If you close them in a close() the first RocksDB state store that is closed by Kafka Streams will close both and all other RocksDB state stores will most likely crash because their cache and write buffer manager was closed.

当类被我们没有回调的类加载器卸载时，您需要关闭静态缓存和静态写入缓冲区管理器.我认为卸载会在 JVM 退出时发生，因此在 JVM 退出之前不会泄漏堆外内存，然后无论如何释放堆外内存.

You would need to close a static cache and a static write buffer manager when the class is unloaded by the class loader for which we do not have a callback. I think unloading happens when the JVM exits, so no off-heap memory is leaked until the JVM exits and afterwards the off-heap memory is freed anyways.

关于限制 RocksDB 内存使用的问题，答案取决于您想要限制的内容.您是要限制单个 RocksDB 实例使用的内存，还是要限制在一个 Kafka Streams 客户端中使用的所有 RocksDB 实例的内存?对于前者，您应该使用第一个文档中的示例.对于后者，您应该使用第二个文档中的示例.

Regarding your question about limiting the memory usage of RocksDB the answer depends on what you want to limit. Do you want to limit the memory used by one single instance of RocksDB or do you want to limit the memory of all RocksDB instances that are used within one Kafka Streams client? For the former you should use the example in the first documentation. For the latter, you should use the example in the second documentation.

这篇关于在 kafka 流应用程序中关闭或不关闭 RocksDB Cache 和 WriteBufferManager的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！