为 Spark 2.x 启用 spark.sql.crossJoin.enabled [英] spark.sql.crossJoin.enabled for Spark 2.x

问题描述

我正在使用带有 Spark 2.0.0 的预览版"Google DataProc Image 1.1.要完成我的一项操作，我必须完成笛卡尔积.从 2.0.0 版本开始，创建了一个 spark 配置参数 (spark.sql.cross Join.enabled) 来禁止笛卡尔积并抛出异常.如何设置 spark.sql.crossJoin.enabled=true，最好使用初始化操作?spark.sql.crossJoin.enabled=true

I am using the 'preview' Google DataProc Image 1.1 with Spark 2.0.0. To complete one of my operations I have to complete a cartesian product. Since version 2.0.0 there has been a spark configuration parameter created (spark.sql.cross Join.enabled) that prohibits cartesian products and an Exception is thrown. How can I set spark.sql.crossJoin.enabled=true, preferably by using an initialization action? spark.sql.crossJoin.enabled=true

推荐答案

要更改 Dataproc 中配置设置的默认值，您甚至不需要 init 操作，您可以使用 --properties 标志:

For changing default values of configuration settings in Dataproc, you don't even need an init action, you can use the --properties flag when creating your cluster from the command-line:

gcloud dataproc clusters create --properties spark:spark.sql.crossJoin.enabled=true my-cluster ...

这篇关于为 Spark 2.x 启用 spark.sql.crossJoin.enabled的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！