与Cloudera Hbase 1.0.0集成时的依赖冲突 [英] Dependency conflict in integrating with Cloudera Hbase 1.0.0
问题描述
我试图将我的play framework(2.4.2)web应用程序连接到cloudera hbase群集。我在我的bulid.sbt文件中包含了hbase依赖项,并使用hbase示例代码将一个单元格插入表中。但是,我得到了这个异常,这似乎是玩框架和Hbase之间的依赖冲突。
我也附加了我的示例代码和build.sbt文件。
[错误] [07/21/2015 12:03:05.919 ] [application-akka.actor.default-dispatcher-5] [ActorSystem(application)]从线程[application-akka.actor.default-dispatcher-5]中取消致命错误,关闭ActorSystem [应用程序]
java。 lang.IllegalAccessError:尝试从org.apache.hadoop.hbase类的org.apache.hadoop.hbase.zookeeper.MetaTableLocator
访问方法com.google.common.base.Stopwatch。< init>()V .zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
at org.apache.hadoop.hbase .client.ConnectionManager $ HConnectionImplementation.locateRegion(ConnectionManager.java:1123)
at org.apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation.locateRegion(ConnectionManager.java:1110)
at org .apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1262)
at org.apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107)
at controllers.Application.index(Application.java:44)
at router.Routes $$ anonfun $ routes $ 1 $$ anonfun $ applyOrElse $ 1 $$ anonfun $ apply $ 1.apply(Routes.scala:95)
在router.Routes $$ anonfun $ routes $ 1 $$ anonfun $ applyOrElse $ 1 $$ anonfun $ apply $ 1.apply(Routes.scala:95)
at play.core.routing.HandlerInvokerFactory $$ anon $ 4.resultCall( HandlerInvoker.scala:136)
at play.core.routing.HandlerInvokerFactory $ JavaActionInvokerFactory $$ anon $ 14 $$ anon $ 3 $$ anon $ 1.invocation(HandlerInvoker.scala:127)
at play.core。 j.JavaAction $$ anon $ 1.call(JavaAction.scala:70)
at play.http.DefaultHttpRequestHandler $ 1.call(DefaultHttpRequestHandler.java:20)
at play.core.j.JavaAction $$ anonfun $ 7.apply(JavaAction.scala:94)
at play.core.j.JavaAction $$ anonfun $ 7.apply(JavaAction.scala:94)
at scala.concurrent.impl.Future $ PromiseCompletingRunnable .liftedTree1 $ 1(Future.scala:24)
at scala.concurrent.impl.Future $ PromiseCompletingRunnable.run(Future.scala:24)
at play.core.j.HttpExecutionContext $$ anon $ 2。运行(HttpExecutionContext.scala:40)
在play.api.libs.iteratee.Exec ution $ trampoline $ .execute(Execution.scala:70)
at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32)
at scala.concurrent.impl.Future $ .apply( Future.scala:31)
at scala.concurrent.Future $ .apply(Future.scala:492)
at play.core.j.JavaAction.apply(JavaAction.scala:94)
at play.api.mvc.Action $$ anonfun $ apply $ 1 $$ anonfun $ apply $ 4 $$ anonfun $ apply $ 5.apply(Action.scala:105)
at play.api.mvc.Action $$ anonfun $ apply $ 1 $$ anonfun $ apply $ 4 $$ anonfun $ apply $ 5.apply(Action.scala:105)
at play.utils.Threads $ .withContextClassLoader(Threads.scala:21)
at play.api.mvc.Action $$ anonfun $ apply $ 1 $$ anonfun $ apply $ 4.apply(Action.scala:104)
at play.api.mvc.Action $$ anonfun $ apply $ 1 $$ anonfun $应用$ 4.apply(Action.scala:103)
at scala.Option.map(Option.scala:146)
at play.api.mvc.Action $$ anonfun $ apply $ 1.apply(Action .scala:103)
在play.api.mvc.Action $$ anonfun $ apply $ 1.apply(Action.scala:96)
在play.api.libs.iteratee.Iteratee $$ anonfun $ mapM $ 1.apply(Iteratee.scala:524)
在play.api.libs。 iteratee.Iteratee $$ anonfun $ mapM $ 1.apply(Iteratee.scala:524)
at play.api.libs.iteratee.Iteratee $$ anonfun $ flatMapM $ 1.apply(Iteratee.scala:560)
at play.api.libs.iteratee.Iteratee $$ anonfun $ flatMapM $ 1.apply(Iteratee.scala:560)
at play.api.libs.iteratee.Iteratee $$ anonfun $ flatMap $ 1 $$ anonfun $申请$ 13.apply(Iteratee.scala:536)
在play.api.libs.iteratee.Iteratee $$ anonfun $ flatMap $ 1 $$ anonfun $ apply $ 13.apply(Iteratee.scala:536)
at scala.concurrent.impl.Future $ PromiseCompletingRunnable.liftedTree1 $ 1(Future.scala:24)
at scala.concurrent.impl.Future $ PromiseCompletingRunnable.run(Future.scala:24)
at akka。 dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator $ AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool $ WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent .forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
这是我的bulid.sbt文件:
name:= HbaseTest
version:=1.0-SNAPSHOT
lazy val root =(项目文件(。))。enablePlugins(PlayJava)
scalaVersion:=2.11.6
libraryDependencies ++ = Seq(
javaJdbc,
cache,
javaWs
)
// hbase
libraryDependencies + =org.apache.hbase%hbase-client%1.0.0-cdh5.4.4
libraryDependencies + =org.apache.hbase %hbase-annotations%1.0.0-cdh5.4.4
libraryDependencies + =org.apache.hbase%hbase-common%1.0.0-cdh5.4.4
libraryDepe ndencies =org.apache.hbase%hbase-protocol%1.0.0-cdh5.4.4
// hadoop
libraryDependencies + =org.apache.hadoop%hadoop -common%2.6.0-cdh5.4.4
libraryDependencies + =org.apache.hadoop%hadoop-annotations%2.6.0-cdh5.4.4
libraryDependencies + = org.apache.hadoop%hadoop-auth%2.6.0-cdh5.4.4
// Play提供了两种类型的路由器,一种预计它的动作被注入,
/ / other,传统风格,静态访问其动作。
routesGenerator:= InjectedRoutesGenerator
这是我的代码:
程序包控制器;
导入播放。*;
import play.mvc。*;
导入views.html。*;
import java.io.IOException;
import java.util.HashMap;
导入org.apache.hadoop.conf.Configuration;
导入org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.compress.Compression.Algorithm;
import org.apache.hadoop.hbase.util.Bytes;
public class Application extends Controller {
public Result index(){
String ZooKeeperIP =10.12.7.43;
String ZooKeeperPort =2181;
String HBaseMaster =10.12.7.43:60000;
配置hBaseConfig;
连接连接=空;
// TableName TABLE_NAME =sample;
hBaseConfig = HBaseConfiguration.create();
hBaseConfig.set(hbase.zookeeper.quorum,ZooKeeperIP);
hBaseConfig.set(hbase.zookeeper.property.clientPort,ZooKeeperPort);
hBaseConfig.set(hbase.master,HBaseMaster);
//连接= ConnectionFactory.createConnection(hBaseConfig);
尝试{
connection = ConnectionFactory.createConnection(hBaseConfig);
HTable表=新的HTable(hBaseConfig,sample);
Put p = new Put(Bytes.toBytes(1));
p.add(Bytes.toBytes(a),Bytes.toBytes(b),Bytes.toBytes(4));
table.put(p);
} catch(Exception e){
e.printStackTrace();
System.out.println(e.getMessage());
}
return ok(index.render(Your new application is ready。));
}
}
正如我所见,麻烦与依赖关系有关。
特别是 guava
库(这是 hadoop
)。
Play 使用较新版本的
guava
我所看到的。它没有 StopWatch
类,其中 hbase
需要。
您可以通过多种方式来解决问题(不幸的是,我知道他们都是'hacky')。
简单的方法是使用像 zipkin
。我们将自己添加 StopWatch
的位置。
另一种方式是以某种方式分离HBase操作。 (这将需要大量的工作和设计变更)
如果 sbt
支持'阴影会更容易',因为我知道它还没有。
您仍然可以使用 sbt
来解决它的一些问题,比如 spark
处理类似问题。
I tried to connect my play framework (2.4.2) web application to a cloudera hbase cluster. I included hbase dependencies in my bulid.sbt file and used hbase sample code to insert a cell into a table. However, I got this exception which seems to be dependency conflict between play framework and Hbase. I also attached my sample code and build.sbt files as well. I would be grateful for your help to resolve this error.
[ERROR] [07/21/2015 12:03:05.919] [application-akka.actor.default-dispatcher-5] [ActorSystem(application)] Uncaught fatal error from thread [application-akka.actor.default-dispatcher-5] shutting down ActorSystem [application]
java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:434)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1123)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1110)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1262)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107)
at controllers.Application.index(Application.java:44)
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$1$$anonfun$apply$1.apply(Routes.scala:95)
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$1$$anonfun$apply$1.apply(Routes.scala:95)
at play.core.routing.HandlerInvokerFactory$$anon$4.resultCall(HandlerInvoker.scala:136)
at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$14$$anon$3$$anon$1.invocation(HandlerInvoker.scala:127)
at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:70)
at play.http.DefaultHttpRequestHandler$1.call(DefaultHttpRequestHandler.java:20)
at play.core.j.JavaAction$$anonfun$7.apply(JavaAction.scala:94)
at play.core.j.JavaAction$$anonfun$7.apply(JavaAction.scala:94)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40)
at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:70)
at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32)
at scala.concurrent.impl.Future$.apply(Future.scala:31)
at scala.concurrent.Future$.apply(Future.scala:492)
at play.core.j.JavaAction.apply(JavaAction.scala:94)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:104)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:103)
at scala.Option.map(Option.scala:146)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:103)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:96)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:536)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:536)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
this is my bulid.sbt file:
name := """HbaseTest"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayJava)
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
javaJdbc,
cache,
javaWs
)
//hbase
libraryDependencies +="org.apache.hbase" % "hbase-client" % "1.0.0-cdh5.4.4"
libraryDependencies +="org.apache.hbase" % "hbase-annotations" % "1.0.0-cdh5.4.4"
libraryDependencies +="org.apache.hbase" % "hbase-common" % "1.0.0-cdh5.4.4"
libraryDependencies +="org.apache.hbase" % "hbase-protocol" % "1.0.0-cdh5.4.4"
//hadoop
libraryDependencies +="org.apache.hadoop" % "hadoop-common"%"2.6.0-cdh5.4.4"
libraryDependencies +="org.apache.hadoop" % "hadoop-annotations"%"2.6.0-cdh5.4.4"
libraryDependencies +="org.apache.hadoop" % "hadoop-auth"%"2.6.0-cdh5.4.4"
// Play provides two styles of routers, one expects its actions to be injected, the
// other, legacy style, accesses its actions statically.
routesGenerator := InjectedRoutesGenerator
this is my code:
package controllers;
import play.*;
import play.mvc.*;
import views.html.*;
import java.io.IOException;
import java.util.HashMap;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.compress.Compression.Algorithm;
import org.apache.hadoop.hbase.util.Bytes;
public class Application extends Controller {
public Result index() {
String ZooKeeperIP = "10.12.7.43";
String ZooKeeperPort = "2181";
String HBaseMaster = "10.12.7.43:60000";
Configuration hBaseConfig;
Connection connection = null;
//TableName TABLE_NAME = "sample";
hBaseConfig = HBaseConfiguration.create();
hBaseConfig.set("hbase.zookeeper.quorum",ZooKeeperIP);
hBaseConfig.set("hbase.zookeeper.property.clientPort", ZooKeeperPort);
hBaseConfig.set("hbase.master", HBaseMaster);
//connection = ConnectionFactory.createConnection(hBaseConfig);
try {
connection = ConnectionFactory.createConnection(hBaseConfig);
HTable table = new HTable(hBaseConfig, "sample");
Put p = new Put(Bytes.toBytes("1"));
p.add(Bytes.toBytes("a"), Bytes.toBytes("b"), Bytes.toBytes("4"));
table.put(p);
}catch (Exception e) {
e.printStackTrace();
System.out.println(e.getMessage());
}
return ok(index.render("Your new application is ready."));
}
}
As I can see, The trouble is with the dependencies.
Specifically guava
library (which is a common problem with hadoop
).
Play
uses newer version of guava
as I can see. It doesn't have the StopWatch
class which hbase
requires.
You could approach the problem this is in multiple ways (All of them I know are 'hacky' unfortunately).
Easy way is to use a hack like zipkin
. Where we would add the StopWatch
ourselves.
Another way would be to somehow separate HBase operations. (Which would require lot of work and design changes)
It would have been much easier if sbt
supported 'shading', As I know it doesn't yet.
You could still workaround it using sbt
with some effort like how spark
deals with a similar problem.
这篇关于与Cloudera Hbase 1.0.0集成时的依赖冲突的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!