Apache Flink:状态后端在哪里保存状态? [英] Apache Flink: Where do State Backends keep the state?

问题描述

我在下面收到一条声明:

I got a statement below:

取决于您的状态后端，Flink还可以管理应用程序的状态，这意味着Flink可以处理内存管理(如有必要，可以洒到磁盘上)以允许应用程序保持非常大的状态."

"Depending on your state backend, Flink can also manage the state for the application, meaning Flink deals with the memory management (possibly spilling to disk if necessary) to allow applications to hold very large state."

https://ci .apache.org/projects/flink/flink-docs-master/dev/stream/state/state_backends.html

这是否意味着只有将状态后端配置为RocksDBStateBackend时，状态才会保留在内存中，并在必要时可能溢出到磁盘?

Does it mean that only when the state backends is configured to RocksDBStateBackend, the state would keep in memory and possibly spilling to disk if necessary?

但是，如果将其配置为MemoryStateBackend或FsStateBackend，则状态仅保留在内存中，并且永远不会溢出到磁盘上.

However if configured to MemoryStateBackend or FsStateBackend, the state only keep in memory and would never be spilled to disk.

推荐答案

是的，通常来说您是对的.只有使用RocksDBStateBackend，数据才会泄漏到磁盘上.

Yes in general you are right. Only with RocksDBStateBackend there will be spilling data to disk.

对于MemoryStateBackend和FsStateBackend而言，状态始终保留在TaskManagers内存中，因此必须适合该内存.这两个后端之间的区别在于它们检查点数据的方式.

In case of both MemoryStateBackend and FsStateBackend the state is always kept in TaskManagers memory and thus must fit in there. The difference between those two backends is the way they checkpoint data.

• 如果使用MemoryStateBackend，检查点数据将发送到JobManager，并也保存在内存中.

• In case of MemoryStateBackend the checkpoint data is sent to JobManager and kept also in memory there.

FsStateBackend在检查点上将数据存储在FileSystem中，并且仅将小的元数据发送到JobManager(或在HA场景中存储在元数据文件夹中)

The FsStateBackend stores data upon checkpoint in FileSystem and sends only small metadata to JobManager (or in HA scenario stores in metadata folder)

因此，对于任何生产用例，强烈建议使用RocksDBStateBackend.您可以在此处.

Therefore for any production use-cases the RocksDBStateBackend is highly encouraged. More in-depth information you can find here.

这篇关于Apache Flink:状态后端在哪里保存状态?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！