阿帕奇Spark和GRPC [英] Apache Spark and gRPC

问题描述

我正在写使用Apache星火的应用程序。对于与客户沟通，我想用GRPC。

I'm writing an application that uses Apache Spark. For communicating with a client, I would like to use gRPC.

在我的摇篮构建文件，我用

In my Gradle build file, I use

dependencies {
  compile('org.apache.spark:spark-core_2.11:1.5.2')
  compile 'org.apache.spark:spark-sql_2.11:1.5.2'
  compile 'io.grpc:grpc-all:0.13.1'
  ...
}

当离开了GRPC，一切工作正常。然而，当使用GRPC，我可以创建构建，但不能执行它，作为网状的各种版本被包使用。星火似乎用网​​状-所有，其中包含了比GRPC使用相同的方法（但具有潜在不同的签名）。

When leaving out gRPC, everything works fine. However, when gRPC is used, I can create the build, but not execute it, as various versions of netty are used by the packages. Spark seems to use netty-all, which contains the same methods (but with potentially different signatures) than what gRPC uses.

我尝试（使用 com.github.johnrengelman.shadow ）的阴影，但不知何故，仍然无法正常工作。我怎样才能解决这个问题？

I tried shadowing (using com.github.johnrengelman.shadow) , but somehow it still does not work. How can I approach this problem?

推荐答案

一般的解决方法，以这样的事情与拆迁阴影。看到答案了类似的问题，protobuf的依赖关系： https：//开头组。 google.com/forum/#​​!topic/grpc-io/ABwMhW9bU34

The general solution to this sort of thing is shading with relocation. See the answer to a similar problem with protobuf dependencies: https://groups.google.com/forum/#!topic/grpc-io/ABwMhW9bU34

这篇关于阿帕奇Spark和GRPC的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！