流文件绝对路径Nifi [英] Flowfile absolute path Nifi

问题描述

我正在尝试使用批量加载选项将流文件加载到MySQL数据库中。下面是我在 UpdateAttribute 处理器中使用的查询，并在更新参数以进行批量加载后将该查询传递给 PutSQL 。

I'm trying to load the flow files into MySQL database using bulk load option. Below is the query I'm using as part of the UpdateAttribute processor and passing that query to PutSQL after updating the parameters to do bulk load.

LOAD DATA INFILE '${absolute.path}${filename}' INTO TABLE ${dest.database}.${db.table.name} FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'

当我运行流程时，它失败了找不到文件异常。

When I ran the flow it's failing saying file not found exception.

. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry; : java.sql.BatchUpdateException: Unable to open file 'data.csv'for 'LOAD DATA INFILE command.Due to underlying IOException:`

** BEGIN NESTED EXCEPTION ** 

java.io.FileNotFoundException
MESSAGE: data.csv (No such file or directory)
java.io.FileNotFoundException: data.csv (No such file or directory).

这里MySQL Server和Nifi在不同的节点上，所以我不能使用LOAD DATA LOCAL INFILE查询。

Here MySQL Server and Nifi are on different nodes so I can't use LOAD DATA LOCAL INFILE query.

即使我在SQL查询中提到流文件的完整绝对路径，我也不确定为什么我找不到文件异常。

I'm not sure why I'm getting file not found exception even though I mentioned the complete absolute path of the flow file in the SQL Query.

当我使用带有硬编码文件名的查询并在nifi节点中提供文件的绝对路径时，它会按预期工作。

When I use query with hard coded file name and providing the absolute path of the file in nifi node, it's working as expected.

工作：

LOAD DATA LOCAL INFILE '/path/in/nifi/node/to/file/data.csv' INTO TABLE ${dest.database}.${db.table.name} FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'}

问题是如何获取流文件的绝对路径并将相同的流文件加载到mysql。

Question is how to get the absolute path of the flow file and load the same flow file into mysql.

流：

推荐答案

• 停止 PutSQL 处理器，然后让流文件排队。


• 一旦排队，右键单击成功关系
  $ b在 UpdateAttribute 和 PutSQL 之间的$ b并选择 List Queue 。


• 选择任意一个流文件并导航至属性标签，然后查看
  ，如果属性 absolute.path 和 flowfilename 存在，并且如果
  
  确实存在，验证是否已设置期望值。在您的情况下， absolute.path 的值应为 / path / in / nifi / node / to / file 和 flowfilename 的值应为 /data.csv


• Stop the PutSQL processor and let the flowfiles queue up.
• Once they are queued up, right click on the success relationship
  between UpdateAttribute and PutSQL and select List Queue.
• Select any one flowfile and navigate to the Attributes tab and see if the attributes absolute.path and flowfilename exists and if
  they do exist, verify if they have the expected value set. In your case absolute.path should have the value /path/in/nifi/node/to/file and flowfilename should have the value /data.csv

为您提供的问题：您是否使用 UpdateAttribute 自行设置这些属性，原因是NiFi不会生成名为的属性flowfilename ，它会生成一个名称为 filename 的文件。

Question for you: Are you setting these attributes yourself using UpdateAttribute, reason is, NiFi doesn't generate an attribute named flowfilename, it generates one with the name filename.

再做一件事，确保 absolute.path 的值最后以 / 结尾或 flowfilename 以 / 开头。否则，将附加它们，结果将为 /path/in/nifi/node/to/filedata.csv 。您可以尝试@Mahendra建议的追加函数，否则您可以简单地使用 $ {absolute.path} / $ {flowfilename} 。

One more thing, make sure either the value for absolute.path ends with a / in the end or the value of flowfilename begins with a /. If not, they will be appended and the result will be /path/in/nifi/node/to/filedata.csv. You can try the append function that @Mahendra suggested, else you can simply use ${absolute.path}/${flowfilename}.

更新

我只是意识到 absolute.path 是一个核心属性，例如 filename ， filesize ， mime.type 等。某些处理器使用所有核心属性，而某些处理器使用的核心属性很少。 GenerateTableFetch 写入 absolute.path ，但未对此设置任何内容。这就是为什么它具有 ./ 这是默认值的原因。

I just realized that absolute.path is a core attribute like filename, filesize, mime.type, etc. Some processors use all the core attributes while some use very few which are needed. GenerateTableFetch writes absolute.path but doesn't set anything for it. That's why it has ./ which is the default value.

所以我对您的工作方式的建议是，您可以使用 UpdateAttribute 手动设置/覆盖 absolute.path 属性（就像您覆盖了文件名），并设置所需的值，即 / path / in / nifi / node / to / file

So my suggestion for your approach to work is, you can manually set/overwrite absolute.path attribute using UpdateAttribute (just like you have overwritten filename) and set the desired value which is /path/in/nifi/node/to/file

这篇关于流文件绝对路径Nifi的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！