Apache Flink:状态后端在哪里保存状态? [英] Apache Flink: Where do State Backends keep the state?

问题描述

我收到以下声明:

根据您的状态后端，Flink 还可以管理应用程序的状态，这意味着 Flink 处理内存管理(如有必要，可能会溢出到磁盘)以允许应用程序保持非常大的状态."

"Depending on your state backend, Flink can also manage the state for the application, meaning Flink deals with the memory management (possibly spilling to disk if necessary) to allow applications to hold very large state."

https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/state_backends.html

这是否意味着只有当状态后端配置为 RocksDBStateBackend 时，状态才会保留在内存中，并可能在必要时溢出到磁盘?

Does it mean that only when the state backends is configured to RocksDBStateBackend, the state would keep in memory and possibly spilling to disk if necessary?

但是，如果配置为 MemoryStateBackend 或 FsStateBackend，状态只会保留在内存中，永远不会溢出到磁盘.

However if configured to MemoryStateBackend or FsStateBackend, the state only keep in memory and would never be spilled to disk.

推荐答案

总的来说，您是对的.只有使用 RocksDBStateBackend 才会将数据溢出到磁盘.

Yes in general you are right. Only with RocksDBStateBackend there will be spilling data to disk.

在 MemoryStateBackend 和 FsStateBackend 的情况下，状态总是保存在 TaskManagers 内存中，因此必须适合那里.这两个后端之间的区别在于它们检查点数据的方式.

In case of both MemoryStateBackend and FsStateBackend the state is always kept in TaskManagers memory and thus must fit in there. The difference between those two backends is the way they checkpoint data.

• 在 MemoryStateBackend 的情况下，检查点数据被发送到 JobManager 并保存在那里的内存中.

• In case of MemoryStateBackend the checkpoint data is sent to JobManager and kept also in memory there.

FsStateBackend 在文件系统中的检查点存储数据，并且仅将小的元数据发送到 JobManager(或在元数据文件夹中的 HA 场景存储中)

The FsStateBackend stores data upon checkpoint in FileSystem and sends only small metadata to JobManager (or in HA scenario stores in metadata folder)

因此，对于任何生产用例，强烈建议使用 RocksDBStateBackend.您可以在此处找到更深入的信息.

Therefore for any production use-cases the RocksDBStateBackend is highly encouraged. More in-depth information you can find here.

这篇关于Apache Flink:状态后端在哪里保存状态?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！