在纱集群模式下运行时没有发现蜂巢表 [英] Hive tables not found when running in YARN-Cluster mode
本文介绍了在纱集群模式下运行时没有发现蜂巢表的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有一个火花（1.4.1版）的应用上HDP 2.3。纱线客户端模式下运行时，它工作正常。然而，纱线群集模式没有我的蜂巢表运行时，它可以通过应用程序被发现。
我递交申请，像这样：
./斌/火花提交
--class com.myCompany.Main
--master纱线集群
--num-执行人3
--driver-4G内存
--executor内存10克
--executor-1核心
--jars LIB / DataNucleus的-API JDO-3.2.6.jar，LIB / DataNucleus的-RDBMS-3.2.9.jar，LIB / DataNucleus的核心-3.2.10.jar /home/spark/apps/YarnClusterTest.jar
--files /etc/hive/conf/hive-site.xml
下面是从日志的摘录：
02年5月12日11时05分13秒INFO hive.HiveContext：初始化执行蜂巢版本0.13.1
15/12/02 11点05分14秒INFO metastore.HiveMetaStore：0：打开原店与类实行：org.apache.hadoop.hive.metastore.ObjectStore
15/12/02 11点05分14秒INFO metastore.ObjectStore：ObjectStore的，所谓的初始化
15/12/02 11点05分14秒INFO DataNucleus.Persistence：房产hive.metastore.integral.jdo.pushdown未知 - 会被忽略
15/12/02 11点05分14秒INFO DataNucleus.Persistence：房产datanucleus.cache.level2未知 - 会被忽略
15/12/02 11点05分14秒INFO storage.BlockManagerMasterEndpoint：注册块管理器worker2.xxx.com:34697 5.2 GB RAM，BlockManagerId（1，worker2.xxx.com，34697）
15/12/02 11点05分16秒INFO metastore.ObjectStore：设置MetaStore对象脚类与hive.metastore.cache.pinobjtypes=\"Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order\"
15/12/02 11点05分16秒INFO metastore.MetaStoreDirectSql：MySQL的检查失败，假设我们不是在MySQL：词法错误第1行，遇到5列：@（64），后：。
15/12/02 11点05分17秒INFO DataNucleus.Datastore：类org.apache.hadoop.hive.metastore.model.MFieldSchema被标记为嵌入式只有因此不存在其自身的数据存储表。
15/12/02 11点05分17秒INFO DataNucleus.Datastore：类org.apache.hadoop.hive.metastore.model.MOrder被标记为嵌入式只有因此不存在其自身的数据存储表。
15/12/02 11点05分18秒INFO DataNucleus.Datastore：类org.apache.hadoop.hive.metastore.model.MFieldSchema被标记为嵌入式只有因此不存在其自身的数据存储表。
15/12/02 11点05分18秒INFO DataNucleus.Datastore：类org.apache.hadoop.hive.metastore.model.MOrder被标记为嵌入式只有因此不存在其自身的数据存储表。
15/12/02 11点05分18秒INFO metastore.ObjectStore：ObjectStore的初始化
15/12/02 11点05分19秒WARN metastore.ObjectStore：版本信息metastore没有找到。 hive.metastore.schema.verification没有启用，因此记录架构版本0.13.1aa
15/12/02 11点05分19秒INFO metastore.HiveMetaStore：在metastore添加管理员角色
15/12/02 11点05分19秒INFO metastore.HiveMetaStore：在metastore新增公共角色
15/12/02 11点05分19秒INFO metastore.HiveMetaStore：没有用户管理员角色加入，因为配置是空的
15/12/02 11点05分19秒INFO session.SessionState：在这一点无需TEZ会话。 hive.execution.engine = MR。
15/12/02 11点05分19秒INFO parse.ParseDriver：解析命令：SELECT * FROM streamsummary
15/12/02 11点05分20秒INFO parse.ParseDriver：完成解析
15/12/02 11点05分20秒INFO hive.HiveContext：使用星火类初始化HiveMetastoreConnection版本0.13.1。
15/12/02 11点05分20秒INFO metastore.HiveMetaStore 0：get_table：DB =默认TBL = streamsummary
15/12/02 11点05分20秒INFO HiveMetaStore.audit：UGI =火花IP =未知的IP地址CMD = get_table：DB =默认TBL = streamsummary
15/12/02 11点05分20秒DEBUG myCompany.Main $：没有这样的表streamsummary; 1号线POS 14
我基本上碰到同样的没有这样的表'问题，我的应用程序需要读取或写入配置单元表的任何时间。
发布蜂房的site.xml这里，以供参考。
＆LT;结构＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆ambari.hive.db.schema.name LT; /名称＆gt;
＆LT; VALUE＆GT;蜂巢＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆datanucleus.autoCreateSchema LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆datanucleus.cache.level2.type LT; /名称＆gt;
＆LT; VALUE＆GT;没有＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.auto.convert.join LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.auto.convert.join.noconditionaltask LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.auto.convert.join.noconditionaltask.size LT; /名称＆gt;
＆LT; VALUE＆GT; 1968526677＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.auto.convert.sortmerge.join LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.auto.convert.sortmerge.join.to.mapjoin LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.cbo.enable LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.cli.print.header LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.cluster.delegation.token.store.class LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.thrift.ZooKeeperTokenStore＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.cluster.delegation.token.store.zookeeper.connectString LT; /名称＆gt;
＆LT; VALUE＆GT; worker1.xxx.com：2181，worker3.xxx.com：2181，worker2.xxx.com：2181＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.cluster.delegation.token.store.zookeeper.znode LT; /名称＆gt;
＆LT; VALUE＆GT; /蜂巢/组/代表团＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.abortedtxn.threshold LT; /名称＆gt;
＆LT; VALUE＆GT; 1000℃; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.check.interval LT; /名称＆gt;
＆LT; VALUE＆GT; 300L＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.delta.num.threshold LT; /名称＆gt;
＆LT;价值＆GT; 10＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.delta.pct.threshold LT; /名称＆gt;
＆LT; VALUE＆GT;＆0.1F LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.initiator.on LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.worker.threads LT; /名称＆gt;
＆LT; VALUE＆GT; 0℃; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compactor.worker.timeout LT; /名称＆gt;
＆LT; VALUE＆GT; 86400L＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.compute.query.using.stats LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.conf.restricted.list LT; /名称＆gt;
<value>hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role</value>
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.convert.join.bucket.mapjoin.tez LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.default.fileformat LT; /名称＆gt;
＆LT; VALUE＆GT;文本文件＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.default.fileformat.managed LT; /名称＆gt;
＆LT; VALUE＆GT;文本文件＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.enforce.bucketing LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.enforce.sorting LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.enforce.sortmergebucketmapjoin LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.exec.com press.intermediate＆LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.exec.com press.output＆LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.dynamic.partition LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.dynamic.partition.mode LT; /名称＆gt;
＆LT; VALUE＆GT;不严格的＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.failure.hooks LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.hooks.ATSHook＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.max.created.files LT; /名称＆gt;
＆LT;值100000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.max.dynamic.partitions LT; /名称＆gt;
＆LT; VALUE＆GT; 5000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.max.dynamic.partitions.pernode LT; /名称＆gt;
＆LT; VALUE＆GT; 2000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.exec.orc.com pression.strategy＆LT; /名称＆gt;
＆LT; VALUE＆GT;速度＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.exec.orc.default.com preSS＆LT; /名称＆gt;
＆LT; VALUE＆GT;＆ZLIB LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.orc.default.stripe.size LT; /名称＆gt;
＆LT; VALUE＆GT; 67108864＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.orc.encoding.strategy LT; /名称＆gt;
＆LT; VALUE＆GT;速度＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.parallel LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.parallel.thread.number LT; /名称＆gt;
＆LT; VALUE＆GT; 8示/值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.post.hooks LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.hooks.ATSHook＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.exec pre.hooks＆LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.hooks.ATSHook＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.reducers.bytes.per.reducer LT; /名称＆gt;
＆LT; VALUE＆GT; 67108864＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.reducers.max LT; /名称＆gt;
＆LT; VALUE＆GT; 1009＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.scratchdir LT; /名称＆gt;
＆LT; VALUE＆GT;的/ tmp /蜂巢＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.submit.local.task.via.child LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.exec.submitviachild LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.execution.engine LT; /名称＆gt;
＆LT; VALUE＆GT;＆TEZ LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.fetch.task.aggr LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.fetch.task.conversion LT; /名称＆gt;
＆LT; VALUE＆GT;详情＆lt; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.fetch.task.conversion.threshold LT; /名称＆gt;
＆LT; VALUE＆GT; 1073741824＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.limit.optimize.enable LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.limit.pushdown.memory.usage LT; /名称＆gt;
＆LT; VALUE＆GT; 0.04 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.map.aggr LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.map.aggr.hash.force.flush.memory.threshold LT; /名称＆gt;
＆LT; VALUE＆GT; 0.9 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.map.aggr.hash.min.reduction LT; /名称＆gt;
＆LT; VALUE＆GT; 0.5 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.map.aggr.hash.percentmemory LT; /名称＆gt;
＆LT; VALUE＆GT; 0.5 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.mapjoin.bucket.cache.size LT; /名称＆gt;
＆LT; VALUE＆GT; 10000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.mapjoin.optimized.hashtable LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.ma pred.reduce.tasks.speculative.execution＆LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.mapfiles LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.merge.ma predfiles＆LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.orcfile.stripe.level LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.rcfile.block.level LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.size.per.task LT; /名称＆gt;
＆LT; VALUE＆GT;＆2.56 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.smallfiles.avgsize LT; /名称＆gt;
＆LT; VALUE＆GT; 16000000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.merge.tezfiles LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.authorization.storage.checks LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.cache.pinobjtypes LT; /名称＆gt;
＆LT; VALUE＆GT;表，数据库类型，FieldSchema，订单和LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.client.connect.retry.delay LT; /名称＆gt;
＆LT; VALUE＆GT;＆5S LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.client.socket.timeout LT; /名称＆gt;
＆LT; VALUE＆GT; 1800＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.connect.retries LT; /名称＆gt;
＆LT; VALUE＆GT; 24 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.execute.setugi LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.failure.retries LT; /名称＆gt;
＆LT; VALUE＆GT; 24 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.kerberos.keytab.file LT; /名称＆gt;
＆LT; VALUE＆GT; /etc/security/keytabs/hive.service.keytab< /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.kerberos.principal LT; /名称＆gt;
＆LT; VALUE＆GT; hive/_HOST@EXAMPLE.COM< /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt; hive.metastore pre.event.listeners＆LT; /名称＆gt;
<value>org.apache.hadoop.hive.ql.security.authorization.Authorization$p$pEventListener</value>
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.sasl.enabled LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.server.max.threads LT; /名称＆gt;
＆LT;值100000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.uris LT; /名称＆gt;
＆LT; VALUE＆GT;节俭：//host.xxx.com：9083＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.metastore.warehouse.dir LT; /名称＆gt;
＆LT; VALUE＆GT; /应用/蜂巢/仓库和LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.bucketmapjoin LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.bucketmapjoin.sortedmerge LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.constant.propagation LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.index.filter LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.metadataonly LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.null.scan LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.reducededuplication LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.reducededuplication.min.reducer LT; /名称＆gt;
＆LT; VALUE＆GT; 4℃; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.optimize.sort.dynamic.partition LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.orc.compute.splits.num.threads LT; /名称＆gt;
＆LT;价值＆GT; 10＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.orc.splits.include.file.footer LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;蜂巢prewarm.enabled＆LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;蜂巢prewarm.numcontainers＆LT; /名称＆gt;
＆LT; VALUE＆GT;第3版; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.authenticator.manager LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.authorization.enabled LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.authorization.manager LT; /名称＆gt;
<value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory</value>
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.metastore.authenticator.manager LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.metastore.authorization.auth.reads LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.security.metastore.authorization.manager LT; /名称＆gt;
<value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.allow.user.substitution LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.authentication LT; /名称＆gt;
＆LT; VALUE＆GT;无＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.authentication.spnego.keytab LT; /名称＆gt;
＆LT; VALUE＆GT; HTTP/_HOST@EXAMPLE.COM< /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.authentication.spnego.principal LT; /名称＆gt;
＆LT; VALUE＆GT; /etc/security/keytabs/spnego.service.keytab< /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.enable.doAs LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.logging.operation.enabled LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.logging.operation.log.location LT; /名称＆gt;
＆LT; VALUE＆GT; $ {系统：java.io.tmpdir} / $ {系统：user.name} / operation_logs＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.support.dynamic.service.discovery LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.table.type.mapping LT; /名称＆gt;
＆LT; VALUE＆GT;古典和LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.tez.default.queues LT; /名称＆gt;
＆LT; VALUE＆GT;默认＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.tez.initialize.default.sessions LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.tez.sessions.per.default.queue LT; /名称＆gt;
＆LT;价值＆GT; 1 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.thrift.http.path LT; /名称＆gt;
＆LT; VALUE＆GT;的cliservice＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.thrift.http.port LT; /名称＆gt;
＆LT; VALUE＆GT; 10001＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.thrift.max.worker.threads LT; /名称＆gt;
＆LT; VALUE＆GT; 500℃; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.thrift.port LT; /名称＆gt;
＆LT; VALUE＆GT; 10000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.thrift.sasl.qop LT; /名称＆gt;
＆LT; VALUE＆GT; AUTH＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.transport.mode LT; /名称＆gt;
＆LT; VALUE＆GT;二元LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.use.SSL LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.server2.zookeeper.namespace LT; /名称＆gt;
＆LT; VALUE＆GT; hiveserver2＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.smbjoin.cache.rows LT; /名称＆gt;
＆LT; VALUE＆GT; 10000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.stats.autogather LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.stats.dbclass LT; /名称＆gt;
＆LT; VALUE＆GT; FS＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.stats.fetch.column.stats LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.stats.fetch.partition.stats LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.support.concurrency LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.auto.reducer.parallelism LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.container.size LT; /名称＆gt;
＆LT; VALUE＆GT; 5632＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.cpu.vcores LT; /名称＆gt;
＆LT;价值＆GT; -1 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.dynamic.partition.pruning LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.dynamic.partition.pruning.max.data.size LT; /名称＆gt;
＆LT; VALUE＆GT; 104857600＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.dynamic.partition.pruning.max.event.size LT; /名称＆gt;
＆LT; VALUE＆GT; 1048576＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.input.format LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.io.HiveInputFormat＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.java.opts LT; /名称＆gt;
＆LT; VALUE＆GT; -server -Djava.net preferIPv4Stack = TRUE -XX：NewRatio = 8 -XX：+ UseNUMA -XX：+ UseG1GC -XX：+ ResizeTLAB -XX：+ PrintGCDetails -verbose：GC -XX： + PrintGCTimeStamps＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.log.level LT; /名称＆gt;
＆LT; VALUE＆GT;信息与LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.max.partition.factor LT; /名称＆gt;
＆LT; VALUE＆GT; 2.0＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.min.partition.factor LT; /名称＆gt;
＆LT; VALUE＆GT; 0.25 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.tez.smb.number.waves LT; /名称＆gt;
＆LT; VALUE＆GT; 0.5 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.txn.manager LT; /名称＆gt;
＆LT; VALUE＆GT; org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.txn.max.open.batch LT; /名称＆gt;
＆LT; VALUE＆GT; 1000℃; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.txn.timeout LT; /名称＆gt;
＆LT; VALUE＆GT; 300℃/值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.user.install.directory LT; /名称＆gt;
＆LT; VALUE＆GT; /用户/＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.vectorized.execution.enabled LT; /名称＆gt;
＆LT; VALUE＆GT;真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.vectorized.execution.reduce.enabled LT; /名称＆gt;
＆LT; VALUE＆GT;假LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.vectorized.groupby.checkinterval LT; /名称＆gt;
＆LT; VALUE＆GT; 4096＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.vectorized.groupby.flush.percent LT; /名称＆gt;
＆LT; VALUE＆GT; 0.1 LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.vectorized.groupby.maxentries LT; /名称＆gt;
＆LT;值100000＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.zookeeper.client.port LT; /名称＆gt;
＆LT; VALUE＆GT; 2181＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.zookeeper.namespace LT; /名称＆gt;
＆LT; VALUE＆GT; hive_zookeeper_namespace＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆hive.zookeeper.quorum LT; /名称＆gt;
＆LT; VALUE＆GT; worker1.xxx.com：2181年，worker3.xxx.com：2181年，worker2.xxx.com：2181＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆javax.jdo.option.ConnectionDriverName LT; /名称＆gt;
＆LT; VALUE＆GT; com.mysql.jdbc.Driver＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆javax.jdo.option.ConnectionPassword LT; /名称＆gt;
＆LT; VALUE＆GT;蜂巢＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆javax.jdo.option.ConnectionURL LT; /名称＆gt;
＆LT; VALUE＆GT;的jdbc：mysql的：//host.xxx.com/hive createDatabaseIfNotExist =真＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT;性＆gt;
＆LT;名称＆gt;＆javax.jdo.option.ConnectionUserName LT; /名称＆gt;
＆LT; VALUE＆GT;蜂巢＆LT; /值＆GT;
＆LT; /性＆gt; ＆LT; /结构＆gt;
解决方案
我贴在Hortonworks社区同样的问题，我解决了问题用的这个答案。
它的要点是：在递交申请时，在 - 文件参数必须来之前的 - 参数，蜂房的site.xml 来使用的副本是一个在星火 CONF DIR，而不是在 $ HIVE_HOME / conf目录/蜂房的site.xml 。因此：
./bin/spark-submit \\
--class com.myCompany.Main \\
--master纱线集群\\
--num-执行人3 \\
--driver-1G内存\\
--executor内存11克\\
--executor-芯1 \\
--files /usr/hdp/current/spark-client/conf/hive-site.xml \\
--jars LIB / DataNucleus的-API JDO-3.2.6.jar，LIB / DataNucleus的-RDBMS-3.2.9.jar，LIB / DataNucleus的核心-3.2.10.jar \\
/home/spark/apps/YarnClusterTest.jar
I have a Spark (version 1.4.1) application on HDP 2.3. It works fine when running it in YARN-Client mode. However, when running it on YARN-Cluster mode none of my Hive tables can be found by the application.
I submit the application like so:
./bin/spark-submit
--class com.myCompany.Main
--master yarn-cluster
--num-executors 3
--driver-memory 4g
--executor-memory 10g
--executor-cores 1
--jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar /home/spark/apps/YarnClusterTest.jar
--files /etc/hive/conf/hive-site.xml
Here's a excerpt from the logs:
5/12/02 11:05:13 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
15/12/02 11:05:14 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/12/02 11:05:14 INFO metastore.ObjectStore: ObjectStore, initialize called
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/12/02 11:05:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager worker2.xxx.com:34697 with 5.2 GB RAM, BlockManagerId(1, worker2.xxx.com, 34697)
15/12/02 11:05:16 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/12/02 11:05:16 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO metastore.ObjectStore: Initialized ObjectStore
15/12/02 11:05:19 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added admin role in metastore
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added public role in metastore
15/12/02 11:05:19 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
15/12/02 11:05:19 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/12/02 11:05:19 INFO parse.ParseDriver: Parsing command: SELECT * FROM streamsummary
15/12/02 11:05:20 INFO parse.ParseDriver: Parse Completed
15/12/02 11:05:20 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
15/12/02 11:05:20 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=streamsummary
15/12/02 11:05:20 INFO HiveMetaStore.audit: ugi=spark ip=unknown-ip-addr cmd=get_table : db=default tbl=streamsummary
15/12/02 11:05:20 DEBUG myCompany.Main$: no such table streamsummary; line 1 pos 14
I basically run into the same 'no such table' problem for any time my application needs to read from or write to the Hive tables.
Posting hive-site.xml here for reference.
<configuration>
<property>
<name>ambari.hive.db.schema.name</name>
<value>hive</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.cache.level2.type</name>
<value>none</value>
</property>
<property>
<name>hive.auto.convert.join</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.join.noconditionaltask</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.join.noconditionaltask.size</name>
<value>1968526677</value>
</property>
<property>
<name>hive.auto.convert.sortmerge.join</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.sortmerge.join.to.mapjoin</name>
<value>false</value>
</property>
<property>
<name>hive.cbo.enable</name>
<value>true</value>
</property>
<property>
<name>hive.cli.print.header</name>
<value>false</value>
</property>
<property>
<name>hive.cluster.delegation.token.store.class</name>
<value>org.apache.hadoop.hive.thrift.ZooKeeperTokenStore</value>
</property>
<property>
<name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
<value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value>
</property>
<property>
<name>hive.cluster.delegation.token.store.zookeeper.znode</name>
<value>/hive/cluster/delegation</value>
</property>
<property>
<name>hive.compactor.abortedtxn.threshold</name>
<value>1000</value>
</property>
<property>
<name>hive.compactor.check.interval</name>
<value>300L</value>
</property>
<property>
<name>hive.compactor.delta.num.threshold</name>
<value>10</value>
</property>
<property>
<name>hive.compactor.delta.pct.threshold</name>
<value>0.1f</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property>
<property>
<name>hive.compactor.worker.threads</name>
<value>0</value>
</property>
<property>
<name>hive.compactor.worker.timeout</name>
<value>86400L</value>
</property>
<property>
<name>hive.compute.query.using.stats</name>
<value>true</value>
</property>
<property>
<name>hive.conf.restricted.list</name>
<value>hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role</value>
</property>
<property>
<name>hive.convert.join.bucket.mapjoin.tez</name>
<value>false</value>
</property>
<property>
<name>hive.default.fileformat</name>
<value>TextFile</value>
</property>
<property>
<name>hive.default.fileformat.managed</name>
<value>TextFile</value>
</property>
<property>
<name>hive.enforce.bucketing</name>
<value>false</value>
</property>
<property>
<name>hive.enforce.sorting</name>
<value>true</value>
</property>
<property>
<name>hive.enforce.sortmergebucketmapjoin</name>
<value>true</value>
</property>
<property>
<name>hive.exec.compress.intermediate</name>
<value>false</value>
</property>
<property>
<name>hive.exec.compress.output</name>
<value>false</value>
</property>
<property>
<name>hive.exec.dynamic.partition</name>
<value>true</value>
</property>
<property>
<name>hive.exec.dynamic.partition.mode</name>
<value>nonstrict</value>
</property>
<property>
<name>hive.exec.failure.hooks</name>
<value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>
<property>
<name>hive.exec.max.created.files</name>
<value>100000</value>
</property>
<property>
<name>hive.exec.max.dynamic.partitions</name>
<value>5000</value>
</property>
<property>
<name>hive.exec.max.dynamic.partitions.pernode</name>
<value>2000</value>
</property>
<property>
<name>hive.exec.orc.compression.strategy</name>
<value>SPEED</value>
</property>
<property>
<name>hive.exec.orc.default.compress</name>
<value>ZLIB</value>
</property>
<property>
<name>hive.exec.orc.default.stripe.size</name>
<value>67108864</value>
</property>
<property>
<name>hive.exec.orc.encoding.strategy</name>
<value>SPEED</value>
</property>
<property>
<name>hive.exec.parallel</name>
<value>false</value>
</property>
<property>
<name>hive.exec.parallel.thread.number</name>
<value>8</value>
</property>
<property>
<name>hive.exec.post.hooks</name>
<value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>
<property>
<name>hive.exec.pre.hooks</name>
<value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>
<property>
<name>hive.exec.reducers.bytes.per.reducer</name>
<value>67108864</value>
</property>
<property>
<name>hive.exec.reducers.max</name>
<value>1009</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
</property>
<property>
<name>hive.exec.submit.local.task.via.child</name>
<value>true</value>
</property>
<property>
<name>hive.exec.submitviachild</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>
<property>
<name>hive.fetch.task.aggr</name>
<value>false</value>
</property>
<property>
<name>hive.fetch.task.conversion</name>
<value>more</value>
</property>
<property>
<name>hive.fetch.task.conversion.threshold</name>
<value>1073741824</value>
</property>
<property>
<name>hive.limit.optimize.enable</name>
<value>true</value>
</property>
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.04</value>
</property>
<property>
<name>hive.map.aggr</name>
<value>true</value>
</property>
<property>
<name>hive.map.aggr.hash.force.flush.memory.threshold</name>
<value>0.9</value>
</property>
<property>
<name>hive.map.aggr.hash.min.reduction</name>
<value>0.5</value>
</property>
<property>
<name>hive.map.aggr.hash.percentmemory</name>
<value>0.5</value>
</property>
<property>
<name>hive.mapjoin.bucket.cache.size</name>
<value>10000</value>
</property>
<property>
<name>hive.mapjoin.optimized.hashtable</name>
<value>true</value>
</property>
<property>
<name>hive.mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>hive.merge.mapfiles</name>
<value>true</value>
</property>
<property>
<name>hive.merge.mapredfiles</name>
<value>false</value>
</property>
<property>
<name>hive.merge.orcfile.stripe.level</name>
<value>true</value>
</property>
<property>
<name>hive.merge.rcfile.block.level</name>
<value>true</value>
</property>
<property>
<name>hive.merge.size.per.task</name>
<value>256000000</value>
</property>
<property>
<name>hive.merge.smallfiles.avgsize</name>
<value>16000000</value>
</property>
<property>
<name>hive.merge.tezfiles</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.authorization.storage.checks</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.cache.pinobjtypes</name>
<value>Table,Database,Type,FieldSchema,Order</value>
</property>
<property>
<name>hive.metastore.client.connect.retry.delay</name>
<value>5s</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>1800s</value>
</property>
<property>
<name>hive.metastore.connect.retries</name>
<value>24</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.failure.retries</name>
<value>24</value>
</property>
<property>
<name>hive.metastore.kerberos.keytab.file</name>
<value>/etc/security/keytabs/hive.service.keytab</value>
</property>
<property>
<name>hive.metastore.kerberos.principal</name>
<value>hive/_HOST@EXAMPLE.COM</value>
</property>
<property>
<name>hive.metastore.pre.event.listeners</name>
<value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value>
</property>
<property>
<name>hive.metastore.sasl.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.server.max.threads</name>
<value>100000</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://host.xxx.com:9083</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/apps/hive/warehouse</value>
</property>
<property>
<name>hive.optimize.bucketmapjoin</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.bucketmapjoin.sortedmerge</name>
<value>false</value>
</property>
<property>
<name>hive.optimize.constant.propagation</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.index.filter</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.metadataonly</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.null.scan</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.reducededuplication</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.reducededuplication.min.reducer</name>
<value>4</value>
</property>
<property>
<name>hive.optimize.sort.dynamic.partition</name>
<value>false</value>
</property>
<property>
<name>hive.orc.compute.splits.num.threads</name>
<value>10</value>
</property>
<property>
<name>hive.orc.splits.include.file.footer</name>
<value>false</value>
</property>
<property>
<name>hive.prewarm.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.prewarm.numcontainers</name>
<value>3</value>
</property>
<property>
<name>hive.security.authenticator.manager</name>
<value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value>
</property>
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.security.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory</value>
</property>
<property>
<name>hive.security.metastore.authenticator.manager</name>
<value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value>
</property>
<property>
<name>hive.security.metastore.authorization.auth.reads</name>
<value>true</value>
</property>
<property>
<name>hive.security.metastore.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
</property>
<property>
<name>hive.server2.allow.user.substitution</name>
<value>true</value>
</property>
<property>
<name>hive.server2.authentication</name>
<value>NONE</value>
</property>
<property>
<name>hive.server2.authentication.spnego.keytab</name>
<value>HTTP/_HOST@EXAMPLE.COM</value>
</property>
<property>
<name>hive.server2.authentication.spnego.principal</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.server2.logging.operation.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
</property>
<property>
<name>hive.server2.support.dynamic.service.discovery</name>
<value>true</value>
</property>
<property>
<name>hive.server2.table.type.mapping</name>
<value>CLASSIC</value>
</property>
<property>
<name>hive.server2.tez.default.queues</name>
<value>default</value>
</property>
<property>
<name>hive.server2.tez.initialize.default.sessions</name>
<value>false</value>
</property>
<property>
<name>hive.server2.tez.sessions.per.default.queue</name>
<value>1</value>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>500</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.sasl.qop</name>
<value>auth</value>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property>
<property>
<name>hive.server2.use.SSL</name>
<value>false</value>
</property>
<property>
<name>hive.server2.zookeeper.namespace</name>
<value>hiveserver2</value>
</property>
<property>
<name>hive.smbjoin.cache.rows</name>
<value>10000</value>
</property>
<property>
<name>hive.stats.autogather</name>
<value>true</value>
</property>
<property>
<name>hive.stats.dbclass</name>
<value>fs</value>
</property>
<property>
<name>hive.stats.fetch.column.stats</name>
<value>true</value>
</property>
<property>
<name>hive.stats.fetch.partition.stats</name>
<value>true</value>
</property>
<property>
<name>hive.support.concurrency</name>
<value>false</value>
</property>
<property>
<name>hive.tez.auto.reducer.parallelism</name>
<value>true</value>
</property>
<property>
<name>hive.tez.container.size</name>
<value>5632</value>
</property>
<property>
<name>hive.tez.cpu.vcores</name>
<value>-1</value>
</property>
<property>
<name>hive.tez.dynamic.partition.pruning</name>
<value>true</value>
</property>
<property>
<name>hive.tez.dynamic.partition.pruning.max.data.size</name>
<value>104857600</value>
</property>
<property>
<name>hive.tez.dynamic.partition.pruning.max.event.size</name>
<value>1048576</value>
</property>
<property>
<name>hive.tez.input.format</name>
<value>org.apache.hadoop.hive.ql.io.HiveInputFormat</value>
</property>
<property>
<name>hive.tez.java.opts</name>
<value>-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps</value>
</property>
<property>
<name>hive.tez.log.level</name>
<value>INFO</value>
</property>
<property>
<name>hive.tez.max.partition.factor</name>
<value>2.0</value>
</property>
<property>
<name>hive.tez.min.partition.factor</name>
<value>0.25</value>
</property>
<property>
<name>hive.tez.smb.number.waves</name>
<value>0.5</value>
</property>
<property>
<name>hive.txn.manager</name>
<value>org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager</value>
</property>
<property>
<name>hive.txn.max.open.batch</name>
<value>1000</value>
</property>
<property>
<name>hive.txn.timeout</name>
<value>300</value>
</property>
<property>
<name>hive.user.install.directory</name>
<value>/user/</value>
</property>
<property>
<name>hive.vectorized.execution.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.vectorized.execution.reduce.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.vectorized.groupby.checkinterval</name>
<value>4096</value>
</property>
<property>
<name>hive.vectorized.groupby.flush.percent</name>
<value>0.1</value>
</property>
<property>
<name>hive.vectorized.groupby.maxentries</name>
<value>100000</value>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
</property>
<property>
<name>hive.zookeeper.namespace</name>
<value>hive_zookeeper_namespace</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://host.xxx.com/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
</configuration>
解决方案
I posted this same question on the Hortonworks community, and I resolved the issue with the help of this answer.
The gist of it is this: when submitting the application, the --files argument has to come before the --jars argument, and the copy of hive-site.xml to use is the one in the Spark conf dir, not in $HIVE_HOME/conf/hive-site.xml. Hence:
./bin/spark-submit \
--class com.myCompany.Main \
--master yarn-cluster \
--num-executors 3 \
--driver-memory 1g \
--executor-memory 11g \
--executor-cores 1 \
--files /usr/hdp/current/spark-client/conf/hive-site.xml \
--jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar \
/home/spark/apps/YarnClusterTest.jar
这篇关于在纱集群模式下运行时没有发现蜂巢表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
