JSON Path 2.3.0 与 hadoop 2.7 Environment JSON-smart1.2.0.jar 冲突 [英] JSON Path 2.3.0 conflicts with hadoop 2.7 Environment JSON-smart1.2.0.jar

查看：23
本文介绍了JSON Path 2.3.0 与 hadoop 2.7 Environment JSON-smart1.2.0.jar 冲突的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

  1. 在 HDFS 位置输入 JSON
  2. 需要解析 JSON 并聚合结果
  3. 要做的是使用使用 JSON 路径库的 PIG UDF
  4. 在hadoop2.7环境jar上:json-smart1.2、json-path1.2是硬绑定的

  5. 每当我执行 PIG Mapreduce 时，它​​都会抛出异常

    java.lang.NoSuchFieldError: defaultReader在 com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)在 com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)在 com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)在 com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)在 com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)

为了解决问题尝试了以下选项

选项 1:尝试设置注册 json-smart2.3.jar &json-path2.3.0.jar但没有可喜的结果(因为它引用的 Jar 是 json-path1.2.jar)

选项 2:将我的模块依赖项降级到 json-path1.2.jar没有结果

选项 3:使用自定义类加载器尝试加载 JSON-path2.3.0 jar 的 jar 吧加载类进入 Org.slf4j 绑定问题

确定了多个绑定路径，但是 sun.misc 类加载器出现问题

无法实例化 SLF4J LoggerFactory报告的异常:java.lang.NullPointerException在 sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)在 sun.misc.URLClassPath.getLoader(URLClassPath.java:485)在 sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)在 sun.misc.URLClassPath.access$100(URLClassPath.java:64)在 sun.misc.URLClassPath$1.next(URLClassPath.java:239)在 sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)在 java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)在 java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)在 java.security.AccessController.doPrivileged(Native Method)在 java.net.URLClassLoader$3.next(URLClassLoader.java:598)在 java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)在 sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)在 sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)在 org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)在 org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)在 org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)在 org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)在 org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)在 org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)在 com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)在 com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)在 com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在 java.lang.reflect.Method.invoke(Method.java:498)在 com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)在 com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)线程main"中的异常 java.lang.reflect.InvocationTargetException在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在 java.lang.reflect.Method.invoke(Method.java:498)在 com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)在 com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)引起:java.lang.ExceptionInInitializerError在 com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)在 com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)... 6个引起:java.lang.IllegalStateException:意外的初始化失败在 org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)在 org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)在 org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)在 org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)在 org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)在 com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)... 8个引起:java.lang.NullPointerException在 sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)在 sun.misc.URLClassPath.getLoader(URLClassPath.java:485)在 sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)在 sun.misc.URLClassPath.access$100(URLClassPath.java:64)在 sun.misc.URLClassPath$1.next(URLClassPath.java:239)在 sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)在 java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)在 java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)在 java.security.AccessController.doPrivileged(Native Method)在 java.net.URLClassLoader$3.next(URLClassLoader.java:598)在 java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)在 sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)在 sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)在 org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)在 org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)……还有 13 个

有人可以建议我解决这个问题吗，可以找到一个关于 weblogic 的 stackoverflow 链接，而不是可以应用于 Hadoop2.7 的通用解决方案(JSON 解析器 -java.lang.NoSuchFieldError: defaultReader)

解决方案

提供自定义实现来加载/解析JSON，这样我们就可以避免使用Hadoop/lib的Json-smart-2.x/1.x

 public static void changeJsonPathConfig() {如果 (!configChanged) {Configuration.setDefaults(new Configuration.Defaults() {私有最终 JsonProvider jsonProvider = new GsonJsonProvider(new GsonBuilder().serializeNulls().create());私有最终 MappingProvider mappingProvider = new GsonMappingProvider();@覆盖公共 JsonProvider jsonProvider() {返回 jsonProvider;}@覆盖公共 MappingProvider mappingProvider() {返回映射提供者；}@覆盖公共集<选项>选项() {返回 EnumSet.noneOf(Option.class);}});configChanged = true;}}

  1. Have a Input of JSON inside HDFS location
  2. It is required to parse the JSON and to aggregate results
  3. To do am using the PIG UDF which are using JSON-path libraries
  4. On the hadoop2.7 environment jar: json-smart1.2, json-path1.2 are hardbinded

  5. Whenever I execute the PIG Mapreduce which throws me below Exception

    java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)
at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)

In-order to solve the problem tried below options

Option 1: Tried setting Registering the json-smart2.3.jar & json-path2.3.0.jar But no promising results (As the Jar it was referencing is json-path1.2.jar)

Option 2: Downgrading my module dependencies to json-path1.2.jar No results

Option 3: Using Custom classLoaders tried to load the jar of JSON-path2.3.0 jar it loaded the class went into issues of Org.slf4j binding

There were multiple binding paths identified, But went problems with sun.misc classloader

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NullPointerException
    at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
    at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
    at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
    at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
    at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
    at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
    at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
    at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Caused by: java.lang.ExceptionInInitializerError
    at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
    at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
    ... 6 more
Caused by: java.lang.IllegalStateException: Unexpected initialization failure
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
    at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
    ... 8 more
Caused by: java.lang.NullPointerException
    at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
    at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
    at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
    ... 13 more

Can some one suggest me to solve this problem, Can find one stackoverflow link where it was telling about weblogic and not a generalized solution which can be applied on Hadoop2.7 also (JSON Parser -java.lang.NoSuchFieldError: defaultReader)

解决方案

Provide the custom implementation to load/parse the JSON so that we can avoid using Json-smart-2.x/1.x of Hadoop/lib

  public static void changeJsonPathConfig() {

    if (!configChanged) {

        Configuration.setDefaults(new Configuration.Defaults() {



            private final JsonProvider jsonProvider = new GsonJsonProvider(

                    new GsonBuilder().serializeNulls().create());

            private final MappingProvider mappingProvider = new GsonMappingProvider();



            @Override

            public JsonProvider jsonProvider() {

                return jsonProvider;

            }



            @Override

            public MappingProvider mappingProvider() {

                return mappingProvider;

            }



            @Override

            public Set<Option> options() {

                return EnumSet.noneOf(Option.class);

            }

        });

        configChanged = true;

    }

}

这篇关于JSON Path 2.3.0 与 hadoop 2.7 Environment JSON-smart1.2.0.jar 冲突的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
Java开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆