使用sbt程序集fat JAR的Bigtable错误（Jetty ALPN和OpenSSL均不可用） [英] Bigtable error with sbt assembly fat JAR (Neither Jetty ALPN nor OpenSSL are available)

问题描述

Bigtable客户端API需要 netty-tcnative-boringssl-static 来连接。这在我的Intellij IDE中工作得很好，但是当我用sbt-assembly构建一个胖JAR，然后运行服务器时，出现以下错误：

  2017-01-10 12:03:41错误BigtableSession：129  -  Jetty ALPN和OpenSSL都不可用。 OpenSSL不可用原因：
 java.lang.IllegalArgumentException：无法加载任何给定的库：[netty-tcnative-linux-x86_64，netty-tcnative-linux-x86_64-fedora，netty-tcnative] 
异常在线程mainjava.lang.ExceptionInInitializerError 
 at mycompany.algo.serving.model.algoServingModelLoaderTest $ .loadLastModel（algoServingModelLoaderTest.scala：36）
 at mycompany.algo.serving.algoServingLauncherTest $$ anonfun $ 4.apply（algoServingLauncherTest.scala：38）
 at mycompany.algo.serving.algoServingLauncherTest $$ anonfun $ 4.apply（algoServingLauncherTest.scala：38）
 at mycompany.serving.MultiPredictorEbapServing $$ anonfun $ loadPredictors $ 1.apply（MultiPredictorEbapServing.scala：25）
 at mycompany.serving.MultiPredictorEbapServing $$ anonfun $ loadPredictors $ 1.apply（MultiPredictorEbapServing.scala：25）
 at scala.collection.immutable.List.foreach List.scala：381）mycompany.serving.MultiPredictorEbapSe上的
在mycompany.algo.serving.algoServingLauncherTest $ .delayedEndpoint $ mycompany $ algo $ serving $ algoServingLauncherTest $ 1（algoServingLauncherTest.scala：38）
 at mycompany.algo处显示rving.loadPredictors（MultiPredictorEbapServing.scala：25）
。 serving.algoServingLauncherTest $ delayedInit $ body.apply（algoServingLauncherTest.scala：11）
 at scala.Function0 $ class.apply $ mcV $ sp（Function0.scala：34）
 at scala.runtime.AbstractFunction0。申请$ mcV $ sp（AbstractFunction0.scala：12）
在scala.App $$ anonfun $ main $ 1.apply（app.scala：76）
在scala.App $$ anonfun $ main $ 1。在scala.collection.immutable.List.foreach（List.scala：381）应用（App.scala：76）
在scala.collection.generic.TraversableForwarder $ b.b $ class.foreach（TraversableForwarder.scala： 35）
 at mycompany.algo.serving.algoServingLauncherTest $ .main（algoServingLauncherTest.scala：11）
 at mycompany。scala.App $ class.main（App.scala：76）
。 algo.serving.algo ServingLauncherTest.main（algoServingLauncherTest.scala）
导致：java.lang.IllegalgotateException：在com.google.cloud找不到com.google.cloud.bigtable.hbase1_2.BigtableConnection 
的适当构造函数。 bigtable.hbase.BigtableConfiguration.connect（BigtableConfiguration.java:88）
位于com.google.cloud.bigtable.hbase.BigtableConfiguration.connect（BigtableConfiguration.java:72）
位于mycompany.algo.serving处。 model.algoServingModel $。< init>（algoServingModel.scala：25）
 at mycompany.algo.serving.model.algoServingModel $。< clinit>（algoServingModel.scala）
 ... 18 more 
导致：java.lang.reflect.InvocationTargetException $ b $在sun.reflect.NativeConstructorAccessorImpl.newInstance0（本地方法）
在sun.reflect.NativeConstructorAccessorImpl.newInstance（NativeConstructorAccessorImpl.java:62）
在sun.reflect.DelegatingConstructorAccessorImpl.newInstance（DelegatingConstru ctorAccessorImpl.java:45）
位于java.lang.reflect.Constructor.newInstance（Constructor.java:422）
位于com.google.cloud.bigtable.hbase.BigtableConfiguration.connect（BigtableConfiguration.java： 85）
 ... 21 more 
导致：java.lang.IllegalgotateException：未正确配置Jetty ALPN和通过netty-tcnative的OpenSSL。 
 at com.google.cloud.bigtable.grpc.BigtableSession。< init>（BigtableSession.java:279）
 at org.apache.hadoop.hbase.client.AbstractBigtableConnection。< init>（ AbstractBigtableConnection.java:137）
位于org.apache.hadoop.hbase.client.AbstractBigtableConnection。< init>（AbstractBigtableConnection.java:104）
位于com.google.cloud.bigtable.hbase1_2.BigtableConnection 。< init>（BigtableConnection.java:50）
 ... 26 more 
  

我应该怎么做？

我使用的是Scala 2.11.8，而我的sbt看起来像：

 依赖项++ = Seq（
com.google.cloud％google-cloud％0.7.0，
com.google.cloud.bigtable ％bigtable-hbase-1.2％0.9.4，
io.netty％netty-tcnative-boringssl-static％1.1.33.Fork19，
org。 apache.hbase％hbase-server％1.2.1，
org.apache.hbase％hbase-client％1.2.1，
org.apache。 hbase％hbase-common ％1.2.1，
）
  

感谢您的帮助

编辑自我最初编辑以来，虽然是由akka引起的，但可以在没有Akka的情况下复制它。 编辑我的糟糕之处在于，本地库被我的sbt汇编合并策略驱逐。 编辑这个问题似乎与sbt

解决方案这是由sbt-assembly默认合并策略引起的。
我需要添加下面的设置

$ val $ settings = Seq（
assemblyMergeStrategy in assembly：= {
案例PathList（META-INF，xs @ _ *）=>
xs map {_.toLowerCase} match {
casenative:: xs =>
MergeStrategy.singleOrError
case _ => MergeStrategy.discard
}
casereference.conf=> MergeStrategy.concat
case x => MergeStrategy.first
}
）

解决方案

问题在于SBT合并策略，我使用这个策略来解决问题。

  assembly中的assemblyMergeStrategy：= {
 case PathList（META-INF，MANIFEST.MF）=> MergeStrategy.discard 
 case _ => MergeStrategy.first 
} 
  

I would like to build a Restful API with akka-http able to retrieve data from Bigtable (HBase).

The Bigtable client API requires netty-tcnative-boringssl-static to connect. This works pretty well inside my Intellij IDE, but when I build a fat JAR with sbt-assembly, and then run the server, I get the following error:

2017-01-10 12:03:41 ERROR BigtableSession:129 - Neither Jetty ALPN nor OpenSSL are available. OpenSSL unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
Exception in thread "main" java.lang.ExceptionInInitializerError
        at mycompany.algo.serving.model.algoServingModelLoaderTest$.loadLastModel(algoServingModelLoaderTest.scala:36)
        at mycompany.algo.serving.algoServingLauncherTest$$anonfun$4.apply(algoServingLauncherTest.scala:38)
        at mycompany.algo.serving.algoServingLauncherTest$$anonfun$4.apply(algoServingLauncherTest.scala:38)
        at mycompany.serving.MultiPredictorEbapServing$$anonfun$loadPredictors$1.apply(MultiPredictorEbapServing.scala:25)
        at mycompany.serving.MultiPredictorEbapServing$$anonfun$loadPredictors$1.apply(MultiPredictorEbapServing.scala:25)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at mycompany.serving.MultiPredictorEbapServing.loadPredictors(MultiPredictorEbapServing.scala:25)
        at mycompany.algo.serving.algoServingLauncherTest$.delayedEndpoint$mycompany$algo$serving$algoServingLauncherTest$1(algoServingLauncherTest.scala:38)
        at mycompany.algo.serving.algoServingLauncherTest$delayedInit$body.apply(algoServingLauncherTest.scala:11)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at mycompany.algo.serving.algoServingLauncherTest$.main(algoServingLauncherTest.scala:11)
        at mycompany.algo.serving.algoServingLauncherTest.main(algoServingLauncherTest.scala)
Caused by: java.lang.IllegalgotateException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_2.BigtableConnection
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:88)
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:72)
        at mycompany.algo.serving.model.algoServingModel$.<init>(algoServingModel.scala:25)
        at mycompany.algo.serving.model.algoServingModel$.<clinit>(algoServingModel.scala)
        ... 18 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:85)
        ... 21 more
Caused by: java.lang.IllegalgotateException: Neither Jetty ALPN nor OpenSSL via netty-tcnative were properly configured.
        at com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:279)
        at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:137)
        at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:104)
        at com.google.cloud.bigtable.hbase1_2.BigtableConnection.<init>(BigtableConnection.java:50)
        ... 26 more

What should I do ?

I am using Scala 2.11.8 and my sbt looks like:

dependencies ++= Seq(
  "com.google.cloud" % "google-cloud" % "0.7.0",
  "com.google.cloud.bigtable" % "bigtable-hbase-1.2" % "0.9.4",
  "io.netty" % "netty-tcnative-boringssl-static" % "1.1.33.Fork19",
  "org.apache.hbase" % "hbase-server" % "1.2.1",
  "org.apache.hbase" % "hbase-client" % "1.2.1",
  "org.apache.hbase" % "hbase-common" % "1.2.1",
)

Thanks for your help

EDIT Edited since I originally though it was caused by akka, but could reproduice it without Akka

EDIT My bad, the native libraries were evicted with my sbt assembly merge strategy

EDIT This works with Maven, the issue seems to be with sbt

SOLUTION This was caused by sbt-assembly default merge strategy. I needed to add the below settings

val settings = Seq(
    assemblyMergeStrategy in assembly := {
      case PathList("META-INF", xs @ _*) =>
        xs map {_.toLowerCase} match {
          case "native" :: xs =>
            MergeStrategy.singleOrError
          case _ => MergeStrategy.discard
        }
      case "reference.conf" => MergeStrategy.concat
      case x => MergeStrategy.first
    }
)

解决方案

The problem was with SBT merge strategy, I used this strategy to solve the problem.

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
  case _ => MergeStrategy.first
}

这篇关于使用sbt程序集fat JAR的Bigtable错误（Jetty ALPN和OpenSSL均不可用）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！