如何在NIFI中的PUTFILE处理器上维护唯一的时间戳 [英] How to maintain unique timestamp at PUTFILE processor in NIFI

问题描述

我有一些csv数据，需要将它们放入具有唯一文件名且具有特定文件命名约定的位置.

I have some csv data which needs to put into a location with unique file name with specific file naming convention.

流程顺序

1. 合并记录->在这里，csv行将合并并进入更新属性处理器.

1. Merge record -> At here the csv rows will get merged and gets forward to the update attribute processor.

更新属性->当合并的流文件内容(csv行的集合)流过更新属性处理器时，具有以下语法的当前时间戳将被分配给文件名".流文件属性.语法:Test-$ {now():format("yyyyMMddHHmmssSSS"，"IST")}.csv

Update Attribute -> When the merged flowfile content(collection of csv row) flows through the update attribute processor, the current timestamp with the below syntax will get assigned to the "filename" flowfile-attribute. syntax: Test-${now():format("yyyyMMddHHmmssSSS", "IST")}.csv

PutSftp->现在，在putsftp服务器上，无论接收到的来自流程文件更新属性的流文件如何，都将发布到远程服务器.

PutSftp -> Now at putsftp server whatever the flow file received form update attribute is getting published to an remote server.

问题声明:

我的远程服务器对文件名格式有限制，它应具有特定的定义，例如:语法:Test-$ {now():format("yyyyMMddHHmmssSSS"，"IST")}.csv鸡蛋:Test-202104041836555.csv

My remote server has restriction with the file name format and it should be in specific as defined like: syntax: Test-${now():format("yyyyMMddHHmmssSSS", "IST")}.csv Egg: Test-202104041836555.csv

因此，在更新属性处理器中，由于线程在同一时间戳实例处处理多个流文件，因此多个流文件(合并的内容)被分配了相同的文件名，并且在放入sftp服务器时，csv文件无法处理并放置到远程服务器，因为已经存在同名文件.

So at the update attribute processor the multiple flowfiles (merged content) are getting assigned same filename due to thread processing multiple flowfiles at same instance of timestamp and while placing into sftp server the csv file is getting failed to process and place into the remote server as there is already existing file with same name.

注意:我的sftp服务器客户端与文件命名格式紧密相关，因此与putsftp处理器中的冲突解决策略无关.

Note: I have nothing to do with conflict resolve strategies in putsftp processor as my sftp server client has a tight dependency on the file naming format.

供处理器序列流参考的图像:

Image for reference of the processor sequence flow:

更新属性处理器属性:

推荐答案

我通过介绍ControrRate处理器找到了解决方案.ControlRate处理器的配置:

I found the solution by introduction ControrRate processor. Configuration of ControlRate processor:

通过在一个实例/时间点仅将一个流文件泵送到udpate属性处理器，便能够为每个流文件分配唯一的时间戳.

By pumping only one flowfile at a instance / point of time to the udpate attribute processor, am able to achieve assigning unique timestamp for each flow file.

更新的流程图:

这篇关于如何在NIFI中的PUTFILE处理器上维护唯一的时间戳的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！