NiFi |流文件随处理器一起移动 [英] NiFi | Flow file movement withing processor

问题描述

我一直在阅读有关NiFi的内容，很少有疑问.考虑一个用例，我想将数据从本地移到HDFS中.我将使用getFile和putHDFS处理器.

I have been reading about NiFi and have few queries . Consider a use case where I want to move data into HDFS from local. I will use getFile and putHDFS processor.

因此，当我将位置传递给getFile时，它将拾取数据并将移入内容存储库，然后将其传递给putHDFS处理器以进行摄取.

So when I pass location to getFile , it will pick up data and will move into content repository and further it will pass to putHDFS processor for ingestion.

问题:

1. 我已经看到流文件的内容是字节表示形式，字节转换是否由Nifi完成?(如果我的源文件是文本文件)?

1. I have seen flow file content is a byte representation , does byte conversion is done by Nifi ?( If my source file is text file)?

如何将数据从内容存储库移至HDFS?

How data is moved to HDFS from content repo ?

推荐答案

1)尚未真正完成转换... GetFile处理器正在从源文件读取字节，并将字节写入内容存储库中的目标.无论源文件的内容是什么，内容存储库中的源文件都是相同的.此操作以流方式执行，因此可以将大文件移动到内容存储库中，而无需将整个文件读到内存中.

1) There is not really a conversion being done... the GetFile processor is reading bytes from the source file and writing bytes to the destination in the content repository. Whatever the content of the source file was, it will be the same in the content repository. This operation is performed in a streaming fashion so that a large file can be moved into the content repository without reading the whole file into memory.

2)PutHDFS处理器使用Apache Hadoop 2.6.2客户端将字节从内容存储库流式传输到HDFS.这类似于从命令行执行"hdfs put".

2) The PutHDFS processor uses the Apache Hadoop 2.6.2 client to stream the bytes from the content repository into HDFS. It is similar to performing an "hdfs put" from the command line.

这篇关于NiFi |流文件随处理器一起移动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！