星火无法加载来自脂肪-JAR资源的自定义log4j.properties [英] Spark unable to load custom log4j.properties from fat-jar resources

问题描述

我在尝试新的和改进的火花1.4.1，但我遇到什么样子了回归。

I'm trying out new and improved spark 1.4.1 but I experience what looks like a regression.

当火花提交我的罐子，我沿着指令传递给加载在我的脂肪罐子的资源捆绑的定制log4j-light-redis.properties。在火花1.3.1，这个工程就像一个魅力。在1.4.1，资源加载器没有找到它：

When spark-submit my jar, I pass along the instruction to load a custom log4j-light-redis.properties bundled in the resources of my fat-jar. In spark 1.3.1, this works like a charm. In 1.4.1, resource loader does not find it:

log4j: Trying to find [log4j-light-redis.properties] using context classloader sun.misc.Launcher$AppClassLoader@4e25154f.
log4j: Trying to find [log4j-light-redis.properties] using sun.misc.Launcher$AppClassLoader@4e25154f class loader.
log4j: Trying to find [log4j-light-redis.properties] using ClassLoader.getSystemResource().
log4j: Could not find resource: [log4j-light-redis.properties].

在火花1.3.1，它会找到正确的资源：

In spark 1.3.1, it would find the resource correctly:

log4j: Trying to find [log4j-light-redis.properties] using context classloader org.apache.spark.util.MutableURLClassLoader@1a1d6a08.
log4j: Using URL [jar:file:/E:/Projects/.../target/fatjar-1.1-SNAPSHOT-shaded.jar!/log4j-light-redis.properties] for automatic log4j configuration.
log4j: Reading configuration from URL jar:file:/E:/Projects/.../target/fatjar-1.1-SNAPSHOT-shaded.jar!/log4j-light-redis.properties
log4j: Parsing for [root] with value=[WARN,console,redis].

有人也有同样的问题，但火花邮寄名单还没有确认这个问题：

Someone also had the same issue but spark mailing list still haven't acknowledged the question:

<一个href=\"http://apache-spark-user-list.1001560.n3.nabble.com/log4j-xml-bundled-in-jar-vs-log4-properties-in-spark-conf-tt23923.html\" rel=\"nofollow\">http://apache-spark-user-list.1001560.n3.nabble.com/log4j-xml-bundled-in-jar-vs-log4-properties-in-spark-conf-tt23923.html

更新
看起来像它的活动比我以为，我无法从我的脂肪罐子加载自定义Log4j追加了更糟糕！它失败，一个ClassNotFoundException！

Update Looks like it's event worse than what I thought.. I'm not able to load a custom log4j appender from my fat-jar anymore! It fails with a ClassNotFoundException!

推荐答案

在1.4.1，MutableURLClassLoader不是log4j的初始化之前好像是1.3.1设置。

In 1.4.1, MutableURLClassLoader is not set before log4j initialization like it is in 1.3.1.

下面的解释是：

虽然在SparkSubmit.scala解析参数，它​​采用spark.util.Utils。这个对象对log4j的一个新的静态的依赖，通过ShutdownHookManager，触发它是从调用setContextClassLoader（MutableURLClassLoader）之前递交初始化> doRunMain> runMain

While parsing arguments in in SparkSubmit.scala, it uses spark.util.Utils. This object has a new static dependency on log4j, through ShutdownHookManager, that triggers it's initialization before the call to setContextClassLoader(MutableURLClassLoader) from submit > doRunMain > runMain

这篇关于星火无法加载来自脂肪-JAR资源的自定义log4j.properties的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！