与Ceph链接作为永久存储 [英] Flink with Ceph as the persistent storage

问题描述

Flink文档建议将Ceph用作状态的持久存储. https://ci.apache. org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html

Flink documents suggests that Ceph can be used as a persistent storage for states. https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html

考虑到Ceph是一个事务数据库，它不会对Flink的性能产生不利影响吗?

Considering that Ceph is a transactional database, wouldn't it have adverse effect on Flink's performance?

推荐答案

Ceph 将自己描述为"统一的分布式存储系统" ，并提供了网络文件系统API.因此，它应该与Flink的状态后端(将检查点持久保存到远程文件系统)无缝地协同工作.

Ceph describes itself as a "unified, distributed storage system" and provides a network file system API. As such, it such should be seamlessly working with Flink's state backends that persist checkpoints to a remote file system.

我不知道使用Ceph的人(更常用HDFS和S3)，也没有有关性能的信息.但是，请注意，Flink能够异步写入检查点，因此存储系统的性能不会影响Flink应用程序的处理速度.但是，这可能会限制获取检查点的时间间隔.

I'm not aware of people using Ceph (HDFS and S3 are more commonly used) and have no information about the performance. However, note that Flink is able to write checkpoints asynchronously, such that the performance of the storage system does not affect the processing speed of a Flink application. It might however, constrain the interval in which checkpoints are taken.

更新: (2018年2月)我注意到有多个用户在Flink的用户邮件列表中报告说，他们正在将Ceph与Flink一起使用.

Update: (Feb. 2018) I noticed that multiple users reported on Flink's user mailing list that they are using Ceph with Flink.

更新2: Flink与S3协议以及两者均正常运行(

Update 2: Flink is working fine with S3 protocol and both (Presto & Hadoop) Flink's S3 FileSystem plugins are working fine with it.

这篇关于与Ceph链接作为永久存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！