如何在Apache Flink中将检查点存储到远程RocksDB中 [英] How to store checkpoint into remote RocksDB in Apache Flink

问题描述

我知道Apache Flink中有三种状态后端:MemoryStateBackend，FsStateBackend和RocksDBStateBackend.

I know that there are three kinds of state backends in Apache Flink: MemoryStateBackend, FsStateBackend and RocksDBStateBackend.

MemoryStateBackend将检查点存储在本地RAM中，FsStateBackend将检查点存储在本地FileSystem中，而RocksDBStateBackend将检查点存储在RocksDB中.我对RocksDBStateBackend有一些疑问.

MemoryStateBackend stores the checkpoints into local RAM, FsStateBackend stores the checkpoints into local FileSystem, and RocksDBStateBackend stores the checkpoints into RocksDB. I have some questions about the RocksDBStateBackend.

据我了解，RocksDBStateBackend的机制已嵌入到Apache Flink中. rocksDB是一种键值数据库.因此，如果我是对的，则意味着Flink将所有检查点存储到使用本地磁盘的嵌入式rockDB中.

As my understanding, the mechanism of RocksDBStateBackend has been embedded into Apache Flink. The rocksDB is a kind of key-value DB. So If I'm right, it means that Flink will store all checkpoints into the embedded rocksDB, which uses the local disk.

如果是这样，由于存储在rocksDB中的检查点，我认为在某些情况下磁盘可能会被耗尽.现在，我在考虑是否可以配置一个远程rocksDB来存储这些检查点?如果有可能，我们是否应该担心远程rocksDB崩溃?如果远程rocksDB崩溃，则Flink的作业将无法继续工作，对吧?

If so, I think the disk could be exhausted in some cases because of the checkpoints stored into the rocksDB. Now I'm thinking if it is possible to configure a remote rocksDB to store these checkpoints? If it is possible, should we worry about the remote rocksDB crashing? If the remote rocksDB crashes, the jobs of Flink can not continue working, right?

推荐答案

没有选择将外部或远程RocksDB与Apache Flink一起使用. RocksDB是一个嵌入式键值存储，在每个任务管理器中都有一个本地实例.

There is no option to use an external or remote RocksDB with Apache Flink. RocksDB is an embedded key-value store with a local instance in each task manager.

几点:

• Flink在工作状态(始终为本地(为了获得良好的性能))和状态快照(检查点和保存点)(不是本地)(出于可靠性，应将它们存储在分布式文件中)之间有很大的区别系统).

• Flink makes a strong distinction between the working state, which is always local (for good performance), and state snapshots (checkpoints and savepoints), which are not local (for reliability they should be stored in a distributed file system).

RocksDBStateBackend将本地磁盘用于工作状态.其他两个状态后端将其工作状态保留在Java堆上.

The RocksDBStateBackend uses the local disk for working state. The other two state backends keep their working state on the Java heap.

检查点协调器安排将散布在所有任务管理器中的所有这些数据片收集在一起，以存储在其他位置存储的完整检查点中.如果使用MemoryStateBackend，则这些检查点存储在JobManager堆中；否则，这些检查点将存储在JobManager堆中.对于其他两个，它们都位于分布式文件系统中.

The checkpoint coordinator arranges for all of these slices of data scattered across all of the task managers to be collected together into complete checkpoints that are stored elsewhere. In the case of the MemoryStateBackend those checkpoints are stored on the JobManager heap; for the other two, they are in a distributed file system.

您想将RocksDB配置为使用最快的可用本地文件系统.尝试使用本地连接的SSD，并避免使用网络连接的存储(例如EBS).不要尝试将诸如S3之类的分布式文件系统用作RocksDB的本地存储.

You want to configure RocksDB to use the fastest available local file system. Try to use locally attached SSDs, and avoid network-attached storage (such as EBS). Do not try to use a distributed file system such as S3 as RocksDB's local storage.

state.backend.rocksdb.localdir控制每个本地RocksDB在何处存储其工作状态.

state.backend.rocksdb.localdir controls where each local RocksDB stores its working state.

RocksDBStateBackend构造函数的参数控制检查点的存储位置.例如，按照@ezequiel的建议使用S3是AWS上的明显选择.

The parameter to the RocksDBStateBackend constructor controls where the checkpoints are stored. E.g., using S3 as recommended by @ezequiel is the obvious choice on AWS.

这篇关于如何在Apache Flink中将检查点存储到远程RocksDB中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！