在 nifi usgae 的 Evaluate jsonpath 处理器中是否会因为属性创建而影响性能 [英] In nifi usgae of Evaluate jsonpath processor will it affect performance impact because of attribute creation

问题描述

我正在尝试将 nifi REST API 与我的应用程序集成.因此，通过映射我的应用程序的输入和输出，我试图调用 nifi REST api 来创建流.因此，在我的用例中，大多数时候我会提取 JSON 值并应用表达式语言.

I'm trying to integrate nifi REST API's with my application. So by mapping input and output from my application, I am trying to call nifi REST api for flow creation. So, in my use case most of the times I will extract the JSON values and will apply expression languages.

因此，为了简化所有用例，我使用评估 JSONpath 处理器来使用 jsonpath 获取所有属性，并在提取处理器中应用表达式语言函数.下面是相关的流程图.

So, for simplifying all the use-cases I am using evaluate JSONpath processor for fetching all attributes using jsonpath and apply expression language function on that in extract processor. Below is the flow diagram regarding that.

这是正确的方法，因为对于具有 30 个键的 JSON 到 JSON 操作，这是最简单的方法，并且当我尝试将 nifi REST API 与我的应用程序集成时，我无法根据用户映射动态生成 JOLT 转换逻辑.

Is it the right approach because for JSON to JSON manipulation having 30 keys this is the simplest way, and as I am trying to integrate nifi REST API's with my application I cannot generate JOLT transformation logic dynamically based on the user mapping.

因此，在这种情况下，评估 JSONpath 处理器的使用是否会为大约 50 个具有不同转换逻辑的用例产生任何性能问题，因为正如我在文档中看到的那样，属性使用会产生性能(关于内存)问题.

So, in this case, does the usage of evaluating JSONpath processor creates any performance issues for about 50 use case with different transformation logic because as I saw in documentation attribute usage creates performance(regarding memory) issues.

推荐答案

您对内存中有太多属性的担忧不应该成为问题；每个流文件有 30 个属性比平常高，但如果这些都是 0 - ~100-200 个字符之间的字符串，那么影响应该很小.如果您开始尝试从流文件内容中将 KB 的数据提取到每个流文件上的属性，您将看到堆使用量增加，但框架仍应能够处理此问题，直到您达到非常高的吞吐量(每秒 1000 个流文件)在像现代笔记本电脑这样的商品硬件上).

Your concern about having too many attributes in memory should not be an issue here; having 30 attributes per flowfile is higher than usual, but if these are all strings between 0 - ~100-200 characters, there should be minimal impact. If you start trying to extract KB worth of data from the flowfile content to the attributes on each flowfile, you will see increased heap usage, but the framework should still be able to handle this until you reach very high throughput (1000's of flowfiles per second on commodity hardware like a modern laptop).

您可能想调查 ReplaceTextWithMapping，因为该处理器可以从定义文件加载并处理许多替换使用单个处理器进行操作.

You may want to investigate ReplaceTextWithMapping, as that processor can load from a definition file and handle many replace operations using a single processor.

具有不同配置值的同一流程流程的多个副本通常是流程设计的味道"(数据库交互偶尔例外).相反，看看是否有一种方法可以通用化流程并使用 变量人口(来自传入的流文件属性，变量注册表、环境变量等).

It is usually a flow design "smell" to have multiple copies of the same flow process with different configuration values (with the occasional exception of database interaction). Rather, see if there is a way you can genericize the process and populate the relevant values for each flowfile using variable population (from the incoming flowfile attributes, the variable registry, environment variables, etc.).

这篇关于在 nifi usgae 的 Evaluate jsonpath 处理器中是否会因为属性创建而影响性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！