ExecuteSQL 处理器返回损坏的数据 [英] ExecuteSQL processor returns corrupted data

问题描述

我在 NiFI 中有一个流程，其中我使用 ExecuteSQL 处理器从 hive 表中获取名为 dt 的子分区的整体合并.例如:我的表由 sikid 和 dt 分区.所以我在 sikid=1, dt=1000 和 sikid=2, dt=1000 下.我所做的是select * from my_table where dt=1000.

I have a flow in NiFI in which I use the ExecuteSQL processor to get a whole a merge of sub-partitions named dt from a hive table. For example: My table is partitioned by sikid and dt. So I have under sikid=1, dt=1000, and under sikid=2, dt=1000. What I did is select * from my_table where dt=1000.

不幸的是，我从 ExecuteSQL 处理器返回的是损坏的数据，包括具有 dt=NULL 的行，而原始表中甚至没有一行 dt=NULL.

Unfortunately, what I've got in return from the ExecuteSQL processor is corrupted data, including rows that have dt=NULL while the original table does not have even one row with dt=NULL.

DBCPConnectionPool 配置为使用 HiveJDBC4 jar.后来按照CDH的版本尝试使用兼容的jar，也没有修复.

The DBCPConnectionPool is configured to use HiveJDBC4 jar. Later I tried using the compatible jar according to the CDH release, didn't fix it either.

ExecuteSQL 处理器配置如下:

规范化表/列名:true

使用 Avro 逻辑类型:false

蜂巢版本:1.1.0

CDH:5.7.1

知道发生了什么吗?谢谢！

Any ideas what's happening? Thanks!

显然，我返回的数据包括额外的行......其中几千行......这很奇怪.

Apparently my returned data includes extra rows... a few thousand of them.. which is quite weird.

推荐答案

最终通过使用 hive 属性解决了hive.query.result.fileformat=SequenceFile

Eventually it was solved by using hive property hive.query.result.fileformat=SequenceFile

这篇关于ExecuteSQL 处理器返回损坏的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！