Spark Streaming和Phoenix Kerberos问题 [英] Spark Streaming and Phoenix Kerberos issue
问题描述
对于这个问题抱歉，因为这已经引起了无数次的讨论，但是我仍然无法在阅读每篇相关文章后找到解决方案。
我在HDP 2.4.2上的Kerberos env上使用Spark Streaming（1.6.1）和Phoenix（4.4），当尝试从HBase读取或写入时获取低于异常。即使跳过spark-submit中的key-ph.conf文件，我也会遇到同样的问题。
我在下面的帖子中看到了与我的问题相同的问题，但仍然存在我无法为我的问题找到解决方案：
https://community.hortonworks.com/questions/56848/spark-cant-connect-to-secure-phoenix.html
Spark无法连接到安全凤凰 以下是我的Spark-submit命令。 Spark代码： 我可以通过执行以下步骤来解决此问题： 1）将所需的hbase，phoenix jars传递到spark额外classpath选项： kafka-jaas.conf文件为例： 如果您仍然无法连接到安全凤凰，请执行以下步骤。 Sorry for this question as this has been raised numerous time on SO, But I am still not able to find a solution for my issue after going through each relevant post.
I am using Spark Streaming (1.6.1) with Phoenix (4.4) on Kerberos env on HDP 2.4.2 getting below exception when try to read or write from HBase. I get the same issue even after skipping key-ph.conf file from spark-submit. I had a look on below post which has identical problem as mine but still I am not able to find solution for my issue: https://community.hortonworks.com/questions/56848/spark-cant-connect-to-secure-phoenix.html Spark can't connect to secure phoenix Below is my Spark-submit command. Spark Code:
I am able to fix this issue by performing below steps as follows: 1) Pass required hbase,phoenix jars into spark extra classpath option: 2) Valid keytab and jaas conf in spark extra java options: kafka-jaas.conf file as an example: 3) Configuring --files spark options After following the steps if you are still not able to connect to secured phoenix. Please set below SPARK_CLASSPATH and then run/execute spark-submit command.
这篇关于Spark Streaming和Phoenix Kerberos问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
spark-submit \
--verbose \
- 主管簇群\
- 执行程序2 \
- 执行程序 - 内存8g \
--executor-cores 4 \
--conf spark.driver.memory = 1024m \
--files key-ph.conf＃key-ph.conf，user.headless .keytab＃user.headless.keytab，在/ etc / HBase的/ 2.4.2.0 -258 / 0 / hbase-site.xml \
--jars /usr/hdp/2.4.2.0-258/hbase/lib/hbase-common-1.1.2.2.4.2.0-258.jar， /usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server- 1.1.2.2.4.2.0-258.jar，在/ usr / HDP / 2.4.2.0-258 / HBase的/ LIB / HBase的 - 协议 - 1.1.2.2.4.2.0-258.jar，在/ usr / HDP / 2.4。 2.0-258 / HBase的/ LIB / HTRACE核-3.1.0-incubating.jar，在/ usr / HDP / 2.4.2.0-258 / HBase的/ LIB /番石榴12.0.1.jar，在/ usr / HDP / 2.4。 2.0-258 /凤凰/ LIB /凤芯4.4.0.2.4.2.0-258.jar，在/ usr / HDP / 2.4.2.0-258 /凤凰/凤4.4.0.2.4.2.0-258客户端-spark.jar \
--driver-java-options-Djava.security.auth.login.config =。/ key-ph.conf -Dhttp.proxyHost = proxy-host -Dhttp.proxyPort = 8080 -Dhttps.proxyHost = proxy-host -Dhttps.proxyPort = 8080 -Dlog4j.configuration = file：/home/user/spark-log4j/log4j-phoenix-driver.properties\
--confspark。 executor.extraJavaOptions = -Djava.security.auth.login.config = ./key-ph.conf -Dlog4j.configuration = file：/ home / user / spark-log4j / log4j-phoenix-executor。属性\
--class com.spark.demo.SampleInsert /home/user/test-ph.jar tableName ZK_IP：2181：/ hbase-secure：user@CLIENT.LAN：/ home / user / user .headless.keytab
demoArrDataFrame.write
.format（org.apache.phoenix.spark）
.options（Map（table - > tableName.toUpperCase，
zkUrl - > ZK_IP：2181：/ hbase-secure：user@FORSYS.LAN：/home/user/user.headless.keytab））
.mode（SaveMode.Overwrite）
.save
16/12/05 16:11:36 WARR AbstractRpcClient：连接到服务器时遇到异常：javax.security.sasl.SaslException：GSS启动失败[由GSSException引起：没有提供有效的凭据（机制级别：失败找到任何Kerberos tgt）]
16/12/05 16:11:36错误AbstractRpcClient：SASL身份验证失败。最可能的原因是丢失或无效的凭证。考虑'kinit'。
javax.security.sasl.SaslException：GSS启动失败[由GSSException引起：没有提供有效的凭据（机制级别：无法找到任何Kerberos tgt）]
at com.sun.security.sasl.gsskerb .GssKrb5Client.evaluateChallenge（GssKrb5Client.java:211）
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect（HBaseSaslRpcClient.java:179）
at org.apache.hadoop.hbase.ipc .RpcClientImpl $ Connection.setupSaslConnection（RpcClientImpl.java:611）
at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.access $ 600（RpcClientImpl.java:156）
at org.apache。 hadoop.hbase.ipc.RpcClientImpl $ $连接2.run（RpcClientImpl.java:737）在org.apache.hadoop.hbase.ipc.RpcClientImpl $ $连接2.run
（RpcClientImpl.java:734）
在java.security.AccessController.doPrivileged（Native方法）$ b $在javax.security.auth.Subject.doAs（Subject.java:422）
在org.apache.hadoop.security.UserGroupInformation.doAs （UserGroupInformation.java:1709）在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.setupIOstreams
（RpcClientImpl.java:734）
在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.writeRequest（RpcClientImpl。 Java的：887）
在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.tracedWriteRequest（RpcClientImpl.java:856）
在org.apache.hadoop.hbase.ipc.RpcClientImpl.call（ RpcClientImpl.java:1200）
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod（AbstractRpcClient.java:213）
at org.apache.hadoop.hbase.ipc.AbstractRpcClient $ BlockingRpcChannelImplementation。 callBlockingMethod（AbstractRpcClient.java:287）
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos $ MasterService $ BlockingStub.isMasterRunning（MasterProtos.java:58152）
at org.apache.hadoop。 hbase.client.ConnectionManager $ HConnectionImplementation $ MasterServiceStubMaker.isMasterRunning（ConnectionManager.java:1571）
at org.apache.hadoop.hbase.client.ConnectionManager $ HConnec tionImplementation $ StubMaker.makeStubNoRetries（ConnectionManager.java:1509）
at org.apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation $ StubMaker.makeStub（ConnectionManager.java:1531）
at org.apache。 hadoop.hbase.client.ConnectionManager $ HConnectionImplementation $ MasterServiceStubMaker.makeStub（ConnectionManager.java:1560）
在org.apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation.getKeepAliveMasterService（ConnectionManager.java:1711）
。在org.apache.hadoop.hbase.client.MasterCallable.prepare（MasterCallable.java:38）
在org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries（RpcRetryingCaller.java:124）
at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor（HBaseAdmin.java:528）$ b $ org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable（HBaseAdmin.java:4083）
b在org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor（HBaseAdmin.java:550）
在org.apache.p hoenix.query.ConnectionQueryServicesImpl.ensureTableCreated（ConnectionQueryServicesImpl.java:810）
处org.apache.phoenix.query org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable（ConnectionQueryServicesImpl.java:1174）
。 DelegateConnectionQueryServices.createTable（DelegateConnectionQueryServices.java:112）
at org.apache.phoenix.schema.MetaDataClient.createTableInternal（MetaDataClient.java:1974）
at org.apache.phoenix.schema.MetaDataClient.createTable（ MetaDataClient.java:770）
在org.apache.phoenix.compile.CreateTableCompiler $ 2.execute（CreateTableCompiler.java:186）
在org.apache.phoenix.jdbc.PhoenixStatement $ 2.call（PhoenixStatement。 java：305）
at org.apache.phoenix.jdbc.PhoenixStatement $ 2.call（PhoenixStatement.java:297）
at org.apache.phoenix.call.CallRunner.run（CallRunner.java:53 ）
在org.apache.phoenix.jdbc.PhoenixStatement.executeMutation（PhoenixStatement.java:295）
在org.apache.phoenix.jdb c.PhoenixStatement.executeUpdate（PhoenixStatement.java:1244）
在org.apache.phoenix.query.ConnectionQueryServicesImpl $ 12.call（ConnectionQueryServicesImpl.java:1850）
在org.apache.phoenix.query.ConnectionQueryServicesImpl $ 12.call（ConnectionQueryServicesImpl.java:1819）
在org.apache.phoenix.util.PhoenixContextExecutor.call（PhoenixContextExecutor.java:77）
在org.apache.phoenix.query.ConnectionQueryServicesImpl.init（
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices（PhoenixDriver.java:180）
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect（PhoenixEmbeddedDriver.java：
at org.apache.phoenix.jdbc.PhoenixDriver.connect（PhoenixDriver.java:151）
at java.sql.DriverManager.getConnection（DriverManager.java:664）
at java .sql.DriverManager.getConnection（DriverManager.java:208）
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection（Connecti onUtil.java:99）
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection（ConnectionUtil.java:82）
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection（ ConnectionUtil.java:70）
在org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getUpsertColumnMetadataList（PhoenixConfigurationUtil.java:232）
在org.apache.phoenix.spark.DataFrameFunctions $$ anonfun $ 2。应用（DataFrameFunctions.scala：45）
在org.apache.phoenix.spark.DataFrameFunctions $$ anonfun $ 2.apply（DataFrameFunctions.scala：41）
at org.apache.spark.rdd.RDD $ $ anonfun $ mapPartitions $ 1 $$ anonfun $ apply $ 22.apply（RDD.scala：717）
at org.apache.spark.rdd.RDD $$ anonfun $ mapPartitions $ 1 $$ anonfun $ apply $ 22.apply（RDD .scala：717）
在org.apache.spark.rdd.MapPartitionsRDD.compute（MapPartitionsRDD.scala：38）
在org.apache.spark.rdd.RDD.computeOrReadCheckpoint（RDD.scala：313 ）org.apache.spark.rdd.RDD.iterator
（RDD.scala：277）
at org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：66）
在org.apache.spark.scheduler.Task.run（Task.scala：89）
在org.apache。 spark.executor.Executor $ TaskRunner.run（Executor.scala：214）
。在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）
。在java.util.concurrent.ThreadPoolExecutor中$ Worker.run（ThreadPoolExecutor.java:617）在java.lang.Thread.run处的
（Thread.java:745）
引起：GSSException：没有提供有效的凭据（机制级别：无法找到任何Kerberos tgt）
at sun.security.jgss.krb5.Krb5InitCredential.getInstance（Krb5InitCredential.java:147）
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement（Krb5MechFactory.java:122）
。在sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext（Krb5MechFactory.java:187）
处太阳sun.security.jgss.GSSManagerImpl.getMechanismContext（GSSManagerImpl.java:224）
。 security.jgss.GSSContextImpl.initSecContext（GSS ContextImpl.java:212）
at sun.security.jgss.GSSContextImpl.initSecContext（GSSContextImpl.java:179）
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge（GssKrb5Client.java： 192）
... 62 more
$ b
- confspark.executor.extraClassPath = / usr / hdp / 2.4.2.0-258 / hbase / lib / hbase-common-1.1.2.2。 4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/ HBase的/ lib目录/ HBase的服务器 - 1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2.0-258.jar ：/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar：/usr/hdp/2.4.2.0-258/hbase/lib/guava-12.0.1.jar ：/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar：在/ usr / HDP / CURR ent / phoenix-client / phoenix-client.jar\
--confspark.driver.extraClassPath = / usr / hdp / 2.4.2.0-258 / hbase / lib / hbase-common -1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4 .2.0-258 / HBase的/ lib目录/ HBase的服务器 - 1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2。 0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar：/usr/hdp/2.4.2.0-258/hbase/lib/guava- 12.0.1.jar：/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar：在/ usr / HDP /当前/ phoenix-client / phoenix-client.jar\
$ b 2）有效的keytab和jaas conf在Spark额外的java选项中：
- confspark.driver.extraJavaOptions = -XX：+ UseG1GC -Djava.security。 auth.login.config = ./ kafka_jaas.conf -Dhttp.proxyHost = PROXY.IP -Dhttp.proxyPort = 8080 -Dhttps.proxyHost = PROXY.IP2 -Dhttps.proxyPort = 8080\
--confspark.executor.extraJavaOp tions = -XX：+ UseG1GC -Djava.security.auth.login.config =。/ kafka_jaas.conf\
KafkaServer {
com.sun.security .auth.module.Krb5LoginModule需要
useKeyTab = true
keyTab =/ etc / security / keytabs / kafka.service.keytab
storeKey = true
useTicketCache = false
serviceName =kafka
principal =kafka/IP@REALM.LAN;
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache = true
renewTicket = true
serviceName =kafka;
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab = true
keyTab =/ etc / security / keytabs / kafka.service.keytab
storeKey = true
useTicketCache = false
serviceName =zookeeper
principal =kafka/IP@REALM.LAN;
};
<3>配置 - 文件火花选项
- 文件kafka_jaas.conf＃kafka_jaas.conf，user.headless.keytab＃user.headless.keytab，/ etc / hbase / conf / hbase-site.xml＃hbase-site .xml \
export SPARK_CLASSPATH = / usr / hdp / 2.4.2.0- 258 / HBase的/ LIB / HBase的-共1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258的.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server-1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/hbase - 方案 - 1.1.2.2.4.2.0-258.jar：/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar：/usr/hdp/2.4.2.0 -258 / HBase的/ LIB /番石榴12.0.1.jar：/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0- 258.jar：/usr/hdp/current/phoenix-client/phoenix-client.jar
spark-submit \
--verbose \
--master yarn-cluster \
--num-executors 2 \
--executor-memory 8g \
--executor-cores 4 \
--conf spark.driver.memory=1024m \
--files key-ph.conf#key-ph.conf,user.headless.keytab#user.headless.keytab,/etc/hbase/2.4.2.0-258/0/hbase-site.xml \
--jars /usr/hdp/2.4.2.0-258/hbase/lib/hbase-common-1.1.2.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server-1.1.2.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.4.2.0-258/hbase/lib/guava-12.0.1.jar,/usr/hdp/2.4.2.0-258/phoenix/lib/phoenix-core-4.4.0.2.4.2.0-258.jar,/usr/hdp/2.4.2.0-258/phoenix/phoenix-4.4.0.2.4.2.0-258-client-spark.jar \
--driver-java-options "-Djava.security.auth.login.config=./key-ph.conf -Dhttp.proxyHost=proxy-host -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy-host -Dhttps.proxyPort=8080 -Dlog4j.configuration=file:/home/user/spark-log4j/log4j-phoenix-driver.properties" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./key-ph.conf -Dlog4j.configuration=file:/home/user/spark-log4j/log4j-phoenix-executor.properties" \
--class com.spark.demo.SampleInsert /home/user/test-ph.jar tableName ZK_IP:2181:/hbase-secure:user@CLIENT.LAN:/home/user/user.headless.keytab
demoArrDataFrame.write
.format("org.apache.phoenix.spark")
.options(Map("table" -> tableName.toUpperCase,
"zkUrl" -> "ZK_IP:2181:/hbase-secure:user@FORSYS.LAN:/home/user/user.headless.keytab"))
.mode(SaveMode.Overwrite)
.save
16/12/05 16:11:36 WARN AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
16/12/05 16:11:36 ERROR AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58152)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1571)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1509)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1531)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1560)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4083)
at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:528)
at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:550)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:810)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1174)
at org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:112)
at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1974)
at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:770)
at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:186)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:305)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:297)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:295)
at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1244)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1850)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1819)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1819)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:180)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:132)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:151)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:99)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:82)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:70)
at org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getUpsertColumnMetadataList(PhoenixConfigurationUtil.java:232)
at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:45)
at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:41)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:717)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:717)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 62 more
--conf "spark.executor.extraClassPath=/usr/hdp/2.4.2.0-258/hbase/lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hbase/lib/guava-12.0.1.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar" \
--conf "spark.driver.extraClassPath=/usr/hdp/2.4.2.0-258/hbase/lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hbase/lib/guava-12.0.1.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar" \
--conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -Djava.security.auth.login.config=./kafka_jaas.conf -Dhttp.proxyHost=PROXY.IP -Dhttp.proxyPort=8080 -Dhttps.proxyHost=PROXY.IP2 -Dhttps.proxyPort=8080" \
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -Djava.security.auth.login.config=./kafka_jaas.conf" \
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="kafka/IP@REALM.LAN";
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/IP@REALM.LAN";
};
--files kafka_jaas.conf#kafka_jaas.conf,user.headless.keytab#user.headless.keytab,/etc/hbase/conf/hbase-site.xml#hbase-site.xml \
export SPARK_CLASSPATH=/usr/hdp/2.4.2.0-258/hbase/lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hbase/lib/guava-12.0.1.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar
