NFS(Netapp服务器)-> Flink-> s3 [英] NFS (Netapp server)-> Flink ->s3

问题描述

我是flink(java)的新手，它试图将作为文件路径安装的netapp文件服务器上的xml文件移动到已安装flink的服务器上.

I am new to flink (java)and trying to move xml files on a netapp file server mounted as a file path onto server that flink is installed.

如何实时进行批处理或流处理，以获取进入该文件夹的文件并使用s3将其下沉.

How to do batch or stream processing in real time to fetch files coming to the folder and sink it with s3.

我在flink-starter中找不到任何示例来从本地文件系统读取文件，flink是否至少是此用例的正确选择?如果是这样，我在哪里可以找到资源来收听文件夹和管理检查点/保存点?

I couldn't find any examples in flink-starter to read files from local file system, is flink atleast a right choice to this use case? If so where can I find resources to listen to folder and manage checkpoints/ save points?

推荐答案

此问题的完整工作代码在下面的链接中.您需要启用检查点才能将.inprogress文件移动到实际文件

Full working code for this question is in the following link . You need to enable checkpointing to move the .inprogress files to actual files

//每1000毫秒启动一个检查点 env.enableCheckpointing(1000);

// start a checkpoint every 1000 ms env.enableCheckpointing(1000);

StreamingFileSink不会将数据提取到s3

这篇关于NFS(Netapp服务器)-> Flink-> s3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！