如何在类似于pyspark的java分区中编写镶木地板文件? [英] How to write parquet file in partition in java similar to pyspark?

问题描述

我可以像这样将镶木地板文件写入pyspark的分区中:

I can write parquet file into partition in pyspark like this:

rdd.write
 .partitionBy("created_year", "created_month")
 .parquet("hdfs:///my_file")

实木复合地板文件被自动分区为created_year，created_month.如何在Java中做同样的事情?我在ParquetWriter类中看不到任何选项.还有另一种可以做到这一点的课程吗?

The parquet file is auto partition into created_year, created_month. How to do the same in java? I don't see an option in ParquetWriter class. Is there another class that can do that?

谢谢

推荐答案

您必须将RDD转换为DataFrame，然后调用write parquet函数.

You have to convert your RDD into DataFrame and then call write parquet function.

df = sql_context.createDataFrame(rdd)
df.write.parquet("hdfs:///my_file", partitionBy=["created_year", "created_month"])

这篇关于如何在类似于pyspark的java分区中编写镶木地板文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！