合并HDFS中的两个实木复合地板文件 [英] Merge two parquet files in HDFS

问题描述

我有一些HDFS格式的HDFS文件.我想将这些文件合并为一个大文件.

I have some files in HDFS in parquet format. I would like to merge these files into one single large file.

我该怎么做?

除了文本文件，我已经做了以下类似的事情.

I have done some thing like below but for text files.

hadoop fs -cat /input_hdfs_dir/* | hadoop fs -put - /output_hdfs_file

但无法以parquet格式获得所需的结果.

But unable to achieve the desired result in parquet format.

我如何达到我的要求?

推荐答案

不可能将parquet文件与hdfs命令合并.

Its not possible to merge parquet files with hdfs commands.

有一个镶木工具库帮助您实现parquet文件的merging.该命令应为

There is a parquet-tools library that can help you achieve the merging of parquet files. The command should be

java jar ./parquet-tools-<VERSION>.jar <command> <input-directory> <output-file>

这篇关于合并HDFS中的两个实木复合地板文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！