带Spark的proto2无法运行 [英] proto2 with Spark cannot run

问题描述

我有一个语法为proto2

此外，我需要使用Spark(2.0.2)和HBase.我的项目是使用Gradle构建的.

Also, I need to use Spark (2.0.2) and HBase. My project is built using Gradle.

现在，当我运行Java代码时，出现此错误:

Right now, when I run my Java code, I get this error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, localhost): java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
    at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
    at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
    at com.google.protobuf.Any.getDescriptor(Any.java:129)
    at com.google.protobuf.util.JsonFormat$ParserImpl.buildWellKnownTypeParsers(JsonFormat.java:1100)
    at com.google.protobuf.util.JsonFormat$ParserImpl.<clinit>(JsonFormat.java:1094)
    at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:277)

此堆栈跟踪与此处发布的问题非常相似. >，但在那里发布的决议不适用于我.

This stacktrace is very similar to the issue posted here, but the resolution posted there didn't apply for me.

我尝试过的事情:

我改变了

compile group: 'com.google.protobuf', name: 'protobuf-java', version: '2.5.0'

到

compile group: 'org.spark-project.protobuf', name: 'protobuf-java', version: '2.4.1-shaded'如该链接所示，但仍然存在相同的错误.

compile group: 'org.spark-project.protobuf', name: 'protobuf-java', version: '2.4.1-shaded' as that link suggested, but still, the same error persisted.

任何帮助将不胜感激！

推荐答案

好的，这是解决方法:

relocate是救援方法.

在我的主程序中，我有一个build.gradle像这样:

In my main program, I've a build.gradle like this:

dependencies {
    compile project(path: ':utils:hbasewrapper', configuration: 'shadow')
}
apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'awesome-jar'
    zip64 true
    // This is to avoid the conflict between proto3 we are using and the proto2 hbase is using.
    relocate ('com.google.protobuf', 'importer.com.google.protobuf')
}

然后，我创建了另一个gradle包，其中包含以下build.gradle

Then, I created another gradle package which has following build.gradle

group 'com.abc.hbasewrapper'
version '1.0-SNAPSHOT'

dependencies {
    compile group: 'org.apache.hbase', name: 'hbase-client', version: '1.3.0'
    compile group: 'org.apache.hbase', name: 'hbase-server', version: '1.2.2'
}

apply plugin: 'java'
apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'hbasewrapper'
    version = null
    zip64 true
    relocate ('com.google.protobuf', 'hbasewrapper.com.google.protobuf')
    relocate ('io.netty', 'hbasewrapper.io.netty')
}

根本原因是由于HBase引用了proto2，而我的程序正在使用proto3，因此导致此NoSuchMethodError错误. 解决方案是使用relocate将com.google.protobuf重新定位到另一个gradle项目，其唯一目的是构建具有proto2的hbase jar文件.

The root cause is due the HBase is referencing proto2 and my program is using proto3, thus causing this NoSuchMethodError error. The solution is to use relocate, to relocate the com.google.protobuf to a different gradle project which sole purpose is to build the hbase jar files that has proto2.

希望它可以帮助其他程序员！ ^ ^

Hope it helps other fellow programmers! ^ ^

这篇关于带Spark的proto2无法运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！