全局启用 spark.sql 区分大小写 [英] Enable case sensitivity for spark.sql globally

问题描述

选项 spark.sql.caseSensitive 控制列名等是否应该区分大小写.它可以设置，例如通过

The option spark.sql.caseSensitive controls whether column names etc should be case sensitive or not. It can be set e.g. by

spark_session.sql('set spark.sql.caseSensitive=true')

并且默认为 false.

似乎不可能在 $SPARK_HOME/conf/spark-defaults.conf 中使用

It does not seem to be possible to enable it globally in $SPARK_HOME/conf/spark-defaults.conf with

spark.sql.caseSensitive: True

虽然.这是有意的还是有其他文件可以设置 sql 选项?

though. Is that intended or is there some other file to set sql options?

也在source 据说完全不鼓励启用此功能.该建议背后的基本原理是什么?

Also in the source it is stated that it is highly discouraged to enable this at all. What is the rationale behind that advice?

推荐答案

结果设置

spark.sql.caseSensitive: True

在 $SPARK_HOME/conf/spark-defaults.conf 中确实有效.它也必须在 Spark 驱动程序的配置中完成，而不是在 master 或 worker 中完成.显然我上次尝试时忘记了这一点.

in $SPARK_HOME/conf/spark-defaults.conf DOES work after all. It just has to be done in the configuration of the Spark driver as well, not the master or workers. Apparently I forgot that when I last tried.

这篇关于全局启用 spark.sql 区分大小写的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！