在sparklyr中更改JVM时区 [英] changing the JVM timezone in sparklyr

问题描述

我正拼命尝试在Sparklyr中更改我的JVM的时区(使用spark 2.1.0).我要在任何地方GMT.

I am desperately trying to change the timezone of my JVM in Sparklyr (using spark 2.1.0). I want GMT everywhere.

我正在设置:

config$`driver.extraJavaOptions` <-"Duser.timezone=GMT"

在我的spark_config()文件中，但是不幸的是，在Spark UI中，我仍然(在系统属性下)看到user.timezone设置为America/New_York.

in my spark_config() file but unfortunately, in the Spark UI I still see (under System Properties) that user.timezone is set to America/New_York.

有什么想法吗? 谢谢！

Any ideas? Thanks!

推荐答案

几件事:

• 该属性的名称为spark.driver.extraJavaOptions.
• 该值缺少前导-.应该是-Duser.timezone=GMT.
• 为保持一致性，您同时需要spark.driver.extraJavaOptions和spark.executor.extraJavaOptions.

• 通常情况下，应在应用程序外部设置spark.driver.extraJavaOptions和类似属性.如官方文档所述:

• The name of the property is spark.driver.extraJavaOptions.
• The value is missing leading -. Should be -Duser.timezone=GMT.
• For consistency you need both spark.driver.extraJavaOptions and spark.executor.extraJavaOptions.

• In general case spark.driver.extraJavaOptions and similar properties should be set outside the application. As explained in the official documentation:

在客户端模式下，不得直接在应用程序中通过SparkConf设置此配置，因为此时驱动程序JVM已经启动.相反，请通过--driver-java-options命令行选项或在默认属性文件中进行设置.

In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

在驱动程序上调用相应的Java方法应该起作用

On the driver calling corresponding Java methods should work

# sc is spark_shell_connection / spark_connection
sparklyr::invoke_static(sc, "java.util.TimeZone",  "getTimeZone", "GMT") %>%
  sparklyr::invoke_static(sc, "java.util.TimeZone", "setDefault", .)

，但可能不会反映在用户界面中，您仍然需要spark.executor.extraJavaOptions.

but might not be reflected in the UI, and you'll still need spark.executor.extraJavaOptions.

通常情况下，您应该在配置目录中编辑spark-defualts.conf以包含

In general case you should edit spark-defualts.conf in the configuration directory to include

spark.driver.extraJavaOptions -Duser.timezone=GMT
spark.executor.extraJavaOptions -Duser.timezone=GMT

如果您不能修改主配置，则可以使用SPARK_CONF_DIR环境variabl.e

If you cannot modify main configuration you can create an application specific directory and point to it using SPARK_CONF_DIR environment variabl.e

在最新版本中，您还可以在应用程序本身中设置spark.sql.session.timeZone(请注意，它与相应的JVM选项不同，并且仅影响Spark查询).

In the recent versions you can also set spark.sql.session.timeZone in the application itself (note that it is different than corresponding JVM options and affects only Spark queries).

这篇关于在sparklyr中更改JVM时区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！