如何配置 Java Spark sparksession 样本大小 [英] How to config Java Spark sparksession samplesize

问题描述

我是 Java Spark 的新手.

I am new to Java Spark.

我目前在使用 Mongodb ETL 到 hive 时遇到问题，这可能导致该字段具有不同的数据类型.所以我想增加样本量，但我在使用 Java 时只看到 Scala 的例子，有谁知道我是否设置正确增加样本量?

I am currently have issue with Mongodb ETL to hive, that could cause the field have different data type. So that I want to increase the sample size but I only see examples of scala while I am using Java, does anyone know if I setup to increase samplesize properly?

SparkSession spark = SparkSession.builder()
                .master("local[2]")
                .appName("SparkReadMgToHive")
                .config("spark.sql.warehouse.dir", warehouseLocation)
                .config("spark.mongodb.input.uri", "mongodb://localhost:27017/test.testcollection")
                .config("sampleSize", 50000)
                .enableHiveSupport()
                .getOrCreate();

非常感谢

推荐答案

spark.mongodb.input.sampleSize

SparkSession spark = SparkSession.builder()
                .master("local[2]")
                .appName("SparkReadMgToHive")
                .config("spark.sql.warehouse.dir", warehouseLocation)
                .config("spark.mongodb.input.uri", "mongodb://localhost:27017/test.testcollection")
                .config("spark.mongodb.input.sampleSize", 50000)
                .enableHiveSupport()
                .getOrCreate();

这篇关于如何配置 Java Spark sparksession 样本大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！