JSON路径2.3.0与hadoop 2.7环境冲突JSON-smart1.2.0.jar [英] JSON Path 2.3.0 conflicts with hadoop 2.7 Environment JSON-smart1.2.0.jar
问题描述
- 在HDFS位置中具有JSON输入
- 需要解析JSON并汇总结果
- 要做的是使用使用JSON路径库的PIG UDF
- 在hadoop2.7环境jar上:json-smart1.2，json-path1.2进行了硬绑定
-
每当我执行PIG Mapreduce时，都会将我抛出异常
- Have a Input of JSON inside HDFS location
- It is required to parse the JSON and to aggregate results
- To do am using the PIG UDF which are using JSON-path libraries
- On the hadoop2.7 environment jar: json-smart1.2, json-path1.2 are hardbinded
Whenever I execute the PIG Mapreduce which throws me below Exception
java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)
at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)
为了解决以下选项尝试过的问题
In-order to solve the problem tried below options
选项1: 尝试设置注册json-smart2.3.jar& json-path2.3.0.jar 但是没有可喜的结果(正如Jar所引用的是json-path1.2.jar)
Option 1: Tried setting Registering the json-smart2.3.jar & json-path2.3.0.jar But no promising results (As the Jar it was referencing is json-path1.2.jar)
选项2: 将我的模块依赖项降级为json-path1.2.jar 没有结果
Option 2: Downgrading my module dependencies to json-path1.2.jar No results
选项3: 使用自定义classLoaders尝试将JSON-path2.3.0的jar加载到jar中 加载的类遇到了Org.slf4j绑定的问题
Option 3: Using Custom classLoaders tried to load the jar of JSON-path2.3.0 jar it loaded the class went into issues of Org.slf4j binding
确定了多个绑定路径，但是sun.misc classloader出现了问题
There were multiple binding paths identified, But went problems with sun.misc classloader
Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Caused by: java.lang.ExceptionInInitializerError
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
... 6 more
Caused by: java.lang.IllegalStateException: Unexpected initialization failure
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
... 8 more
Caused by: java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
... 13 more
有人可以建议我解决此问题吗?可以在讲述weblogic的地方找到一个stackoverflow链接，而不是可以在Hadoop2.7上应用的通用解决方案(
Can some one suggest me to solve this problem, Can find one stackoverflow link where it was telling about weblogic and not a generalized solution which can be applied on Hadoop2.7 also (JSON Parser -java.lang.NoSuchFieldError: defaultReader)
推荐答案
提供自定义实现以加载/解析JSON，这样我们就可以避免使用Hadoop/lib的Json-smart-2.x/1.x
Provide the custom implementation to load/parse the JSON so that we can avoid using Json-smart-2.x/1.x of Hadoop/lib
public static void changeJsonPathConfig() {
if (!configChanged) {
Configuration.setDefaults(new Configuration.Defaults() {
private final JsonProvider jsonProvider = new GsonJsonProvider(
new GsonBuilder().serializeNulls().create());
private final MappingProvider mappingProvider = new GsonMappingProvider();
@Override
public JsonProvider jsonProvider() {
return jsonProvider;
}
@Override
public MappingProvider mappingProvider() {
return mappingProvider;
}
@Override
public Set<Option> options() {
return EnumSet.noneOf(Option.class);
}
});
configChanged = true;
}
}
这篇关于JSON路径2.3.0与hadoop 2.7环境冲突JSON-smart1.2.0.jar的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
