Spark DataSet有效地获取整行的长度 [英] Spark DataSet efficiently get length size of entire row

问题描述

我正在处理不同大小的dataSet，每个都有一个动态大小的列 - 对于我的应用程序，我需要知道字符的整行长度，以便以字节或KB为单位估算整行大小。

I'm working with different size of dataSet each one with a dynamic size of columns - for my application, I have a requirement to know the entire row length of characters for estimate the entire row size in Bytes or KBytes.

整行大小的结果（以KB为单位）将写入新列。

The result of entire row size(in KB) will be written to a new column.

private void writeMyData(Dataset<Row> dataSet){

        Column[] columns = Arrays.stream(dfToWrite.columns()).map(col-> functions.col(col)).toArray(Column[]::new);

        dataSet.withColumn("marker", functions.length(functions.concat_ws( dataSet.columns()[3],columns))).write().partitionBy(hivePartitionColumn)
                .option("header", "true")
                .mode(SaveMode.Append).format(storageFormat).save(pathTowrite);

}

因为我没有 org.apache.spark.sql.functions return Column []
所以我必须使用 dataSet.columns（）并收集它。

As I've none of the method of org.apache.spark.sql.functions return Column[] So i had to use dataSet.columns() and Collect it.

但每次使用嵌套操作 function.method 似乎效率不高。

But using nested operation function.method each time don't seem efficient.

我希望函数大小为 Column [] 并返回列的整个长度。
而不是嵌套操作。

I would rather have a function size that's gets Column[] and return the entire length of the columns. instead of having nested operation.

1. 有没有办法可以帮助我使用UDF函数进行这种操作？
   或者是否存在这种操作的现有功能？


2. 使用这种解决方案有多糟糕？

首选Java解决方案。

Java solution is preferred.

推荐答案

使用Spark Dataframe UDF的好解决方案我已经习惯了get Bytes length对我来说更好：

nice solution with spark Dataframe UDF I have used to get Bytes length which is better for my case:

static UDF1 BytesSize = new UDF1<String, Integer>() {
    public Integer call(final String line) throws Exception {
        return line.getBytes().length;
    }
};

private void saveIt(){

sparkSession.udf().register("BytesSize",BytesSize,DataTypes.IntegerType);
    dfToWrite.withColumn("fullLineBytesSize",callUDF("BytesSize",functions.concat_ws( ",",columns)) ).write().partitionBy(hivePartitionColumn)
                    .option("header", "true")
                    .mode(SaveMode.Append).format(storageFormat).save(pathTowrite);
}

这篇关于Spark DataSet有效地获取整行的长度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！