在使用配置单元配置mysql之后，无法启动配置单元Metastore服务或配置单元 [英] Unable to start hive metastore service or hive-shell after configuring mysql with hive

问题描述

我花了更多时间用配置单元来配置mysql，每当我遇到错误时我不明白它出错的地方.....正如你可以在这篇链接中看到的那样< a>，

hive-site.xml配置可以在这里看到链接

这里是文件结构

所以是造成这个问题的原因....希望这可以帮助我解决问题。

我跟着这些链接

https://dzone.com/articles/how-configure-mysql-metastore

hive0.13.1中的hive-site.xml路径 p>

https：// cwiki .apache.org / confluence / display / Hive / AdminManual + Configuration

https： //cwiki.apache.or g / confluence / display / hive / AdminManual + MetastoreAdmin＃AdminManualMetastoreAdmin-RemoteMetastoreServer

解决方案

.xml文件，这个配置帮助我解决了这个问题。

 <属性> 
< name> javax.jdo.option.ConnectionURL< / name> 
< value> jdbc：mysql：// localhost / metastore_db？createDatabaseIfNotExist = true< / value> 
< description>元数据存储在MySQL服务器< / description> 
< / property> 
 
<属性> 
< name> javax.jdo.option.ConnectionDriverName< / name> 
<值> com.mysql.jdbc.Driver< /值> 
< description> MySQL JDBC驱动程序类< / description> 
< / property> 
 
<属性> 
< name> javax.jdo.option.ConnectionUserName< / name> 
< value> hiveuser< / value> 
< description>用于连接到mysql服务器的用户名< / description> 
< / property> 
 
<属性> 
< name> javax.jdo.option.ConnectionPassword< / name> 
<值> hivepassword< /值> 
< description>连接到mysql服务器的密码< / description> 
< / property> 
<属性> 
< name> javax.jdo.PersistenceManagerFactoryClass< / name> 
< value> org.datanucleus.api.jdo.JDOPersistenceManagerFactory< / value> 
< description>实现jdo持久性的类< / description> 
< / property> 
 
<属性> 
< name> javax.jdo.option.DetachAllOnCommit< / name> 
<值> true< /值> 
< description>从会话中分离所有对象，以便在事务提交后使用它们< / description> 
< / property> 
 
<属性> 
< name> javax.jdo.option.NonTransactionalRead< / name> 
<值> true< /值> 
< description>读取交易外部< / description> 
< / property> 
<属性> 
< name> javax.jdo.option.Multithreaded< / name> 
<值> true< /值> 
< description>如果多个线程同时通过JDO访问元数据，请将其设置为true。< / description> 
< / property> 
 
<属性> 
< name> datanucleus.validateTables< / name> 
<值> false< /值> 
< description>根据代码验证现有模式。如果您想验证现有架构< / description>，请启用此功能
< / property> 
 
<属性> 
< name> datanucleus.validateColumns< / name> 
<值> false< /值> 
< description>根据代码验证现有模式。如果您想验证现有架构< / description>，请启用此功能
< / property> 
 
<属性> 
< name> datanucleus.validateConstraints< / name> 
<值> false< /值> 
< description>根据代码验证现有模式。如果您想验证现有架构< / description>，请启用此功能
< / property> 
 
<属性> 
< name> datanucleus.storeManagerType< / name> 
<值> rdbms< /值> 
< description>元数据存储类型< / description> 
< / property> 
 
<属性> 
<名称> datanucleus.autoCreateSchema< /名称> 
<值> false< /值> 
< / property> 
 
<属性> 
< name> datanucleus.autoStartMechanismMode< / name> 
<值>已检查< /值>如果元数据表不正确，
< description>抛出异常< / description> 
< / property> 
 
<属性> 
< name> datanucleus.autoStartMechanism< / name> 
<值> SchemaTable< /值> 
< / property> 
 
<属性> 
< name> datanucleus.fixedDatastore< / name> 
<值> true< /值> 
< / property> 
 
<属性> 
< name> datanucleus.transactionIsolation< / name> 
<值>读取已提交< /值> 
< description>用于身份生成的默认事务隔离级别。 < /描述> 
< / property> 
 
<属性> 
< name> datanucleus.cache.level2< / name> 
<值> false< /值> 
< description>使用二级缓存。如果元数据独立于Hive Metastore服务器进行更改，请将其关闭< / description> 
< / property> 
 
<属性> 
< name> datanucleus.cache.level2.type< / name> 
<值> SOFT< /值> 
< description> SOFT =基于软参考的高速缓冲存储器，WEAK =基于弱参考的高速缓冲存储器。< / description> 
< / property> 
 
<属性> 
< name> datanucleus.identifierFactory< / name> 
<值> datanucleus1< /值> 
< description>生成表/列名称时使用的标识符工厂的名称。'datanucleus1'用于与DataNucleus v1向后兼容< / description> 
< / property> 
 
 
<属性> 
< name> datanucleus.plugin.pluginRegistryBundleCheck< / name> 
<值> LOG< /值> 
< description>定义当找到并重复插件包时发生的情况[EXCEPTION | LOG | NONE]< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.warehouse.dir< / name> 
< value> / user / hive / warehouse< /值> 
<描述>仓库的默认数据库的位置< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.execute.setugi< / name> 
<值> false< /值> 
< description>在非安全模式下，将此属性设置为true将导致Metastore使用客户端报告的用户和组权限执行DFS操作。请注意，此属性必须在客户端和服务器端都设置。进一步说明它的最大努力。如果客户端将其设置为true，并且服务器将其设置为false，则客户端设置将被忽略。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.event.listeners< / name> 
<值>< /值> 
< description>元数据事件的逗号分隔侦听器列表。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.partition.inherit.table.properties< / name> 
<值>< /值>在表格属性中出现的逗号分隔键的
< description>列表将被继承到新创建的分区。 *表示所有的键都会被继承。< / description> 
< / property> 
 
<属性> 
< name> hive.metadata.export.location< / name> 
<值>< /值> 
< description>与org.apache.hadoop.hive.ql.parse.MetaDataExportListener预事件侦听器一起使用时，元数据将被导出到的位置。默认值为空字符串，这会导致元数据导出到HDFS上当前用户的主目录。< / description> 
< / property> 
 
<属性> 
< name> hive.metadata.move.exported.metadata.to.trash< / name> 
<值>< /值> 
< description>与org.apache.hadoop.hive.ql.parse.MetaDataExportListener预事件侦听器配合使用时，此设置确定导出的元数据是否随后将被移动到用户的垃圾目录旁边丢弃的表数据。这确保了元数据将随着删除的表数据一起被清除。< / description> 
< / property> 
 
<属性> 
<名称> hive.metastore.partition.name.whitelist.pattern< / name> 
<值>< /值> 
< description>将根据此正则表达式模式检查分区名称，如果不匹配则拒绝。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.disallow.incompatible.col.type.change< / name> 
<值>< /值> 
< description>如果为true（默认值为false），则不允许将列类型（例如STRING）更改为不兼容类型（比如MAP& lt; STRING，STRING& gt;）的ALTER TABLE操作。 RCFile默认SerDe（ColumnarSerDe）以这样一种方式序列化值，即数据类型可以从字符串转换为任何类型。该映射也被串行化为一个字符串，它也可以作为字符串读取。但是，对于任何二进制序列化，这是不正确的。当随后尝试访问旧分区时，阻止ALTER TABLE可防止出现ClassCastExceptions。像INT，STRING，BIGINT等原始类型相互兼容，不会被阻塞。 
 
请参阅HIVE-4409了解更多详情。 
< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.end.function.listeners< / name> 
<值>< /值> 
< description>列表中用于元数据转换功能结束的逗号分隔的侦听器。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.event.expiry.duration< / name> 
<值> 0< /值> 
< description>事件过期后的时间表（以秒为单位）< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.event.clean.freq< / name> 
<值> 0< /值> 
< description>定时器任务运行的频率以清除Metastore中的过期事件（以秒为单位）。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.connect.retries< / name> 
<值> 5< /值> 
< description>打开与Metastore的连接时的重试次数< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.failure.retries< / name> 
<值> 3< /值> 
< description> Thrift Metastore呼叫失败时的重试次数< / description> 
< / property> 
 
<属性> 
<名称> hive.metastore.client.connect.retry.delay< / name> 
<值> 1< /值> 
< description>客户端在连续连接尝试之间等待的秒数< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.client.socket.timeout< / name> 
<值> 20< /值> 
< description> MetaStore客户套接字超时秒数< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.rawstore.impl< / name> 
< value> org.apache.hadoop.hive.metastore.ObjectStore< / value> 
< description>实现org.apache.hadoop.hive.metastore.rawstore接口的类的名称。该类用于存储和检索原始元数据对象，如表，数据库< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.batch.retrieve.max< / name> 
<值> 300< /值> 
< description>可以从一个批次的元存储中检索最大对象数（表/分区）。数字越高，Hive Metastore服务器需要的往返次数越少，但也可能导致客户端的内存需求增加。< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.batch.retrieve.table.partition.max< / name> 
<值> 1000< /值> 
< description> Metastore内部在一批中检索的表分区的最大数量。< / description> 
< / property> 
<属性> 
< name> hive.metastore.uris< / name> 
<值> thrift：// localhost：9083< /值> 
< description> Hive metastore Thrift服务器< / description> 
< / property> 
<属性> 
<名称> mapred.reduce.tasks< / name> 
<值> -1< /值> 
< description>每个作业的减少任务的默认数量。通常设置为接近可用主机的数量。 mapred.job.tracker为本地时忽略。 Hadoop默认设置为1，而Hive使用-1作为默认值。通过将此属性设置为-1，Hive将自动计算出应该是减速器数量的数量。 
< / description> 
< / property> 
<属性> 
< name> hive.cli.print.header< / name> 
<值> false< /值> 
< description>是否打印查询输出中列的名称。< / description> 
< / property> 
 
<属性> 
< name> hive.cli.print.current.db< / name> 
<值> false< /值> 
< description>是否在Hive提示符中包含当前数据库。< / description> 
< / property> 
 
<属性> 
<名称> hive.cli.prompt< / name> 
< value>配置单元< /值> 
< description>命令行提示配置值。其他hiveconf可用于此配置值。只有在Hive 
 CLI启动时才会调用变量替换。< / description> 
< / property> 
 
<属性> 
< name> hive.test.mode< / name> 
<值> false< /值> 
< description> Hive是否在测试模式下运行。如果是，则打开采样并为输出表名添加前缀。< / description> 
< / property> 
 
<属性> 
< name> hive.test.mode.prefix< / name> 
<值> test_< /值> 
< description>如果Hive在测试模式下运行，则以该字符串< / description>为输出表的前缀。 
< / property> 
 
 
<属性> 
< name> hive.test.mode.samplefreq< / name> 
<值> 32< /值> 
< description>如果Hive在测试模式下运行并且表没有分段，则采样频率< / description> 
< / property> 
 
<属性> 
< name> hive.test.mode.nosamplelist< / name> 
<值>< /值> 
< description>如果Hive在测试模式下运行，请勿对以上以逗号分隔的表列表进行采样< / description> 
< / property> 
 
<属性> 
< name> hive.metastore.uris< / name> 
<值>< /值> 
< description>远程Metastore的Thrift URI。由Metastore客户端用于连接到远程Metastore。< / description> 
< / property> 
< / configuration> 
  

I know this question asked already but those answers are not helping in any way.

I am spending more time to configure mysql with hive, Every time I encounter errors and I don't understand where it goes wrong ..... as you can see in this link,

hive-site.xml configuration can be seen here in the link

Here is file-structure

So is the reason creating this question.... Hoping this helps me to resolve.

I followed these links

https://dzone.com/articles/how-configure-mysql-metastore

hive-site.xml path in hive0.13.1

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration

How to set configuration in Hive-Site.xml file for hive metastore connection?

https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreServer

解决方案

Problem was with hive-site.xml file and this configuration helped me to resolve the issue.

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>MySQL JDBC driver class</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hiveuser</value>
    <description>user name for connecting to mysql server </description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hivepassword</value>
    <description>password for connecting to mysql server </description>
  </property>
  <property>
    <name>javax.jdo.PersistenceManagerFactoryClass</name>
    <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
    <description>class implementing the jdo persistence</description>
  </property>

  <property>
    <name>javax.jdo.option.DetachAllOnCommit</name>
    <value>true</value>
    <description>detaches all objects from session so that they can be used after transaction is committed</description>
  </property>

  <property>
    <name>javax.jdo.option.NonTransactionalRead</name>
    <value>true</value>
    <description>reads outside of transactions</description>
  </property>
  <property>
    <name>javax.jdo.option.Multithreaded</name>
    <value>true</value>
    <description>Set this to true if multiple threads access metastore through JDO concurrently.</description>
  </property>

  <property>
    <name>datanucleus.validateTables</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.validateColumns</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.validateConstraints</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.storeManagerType</name>
    <value>rdbms</value>
    <description>metadata store type</description>
  </property>

  <property>
    <name>datanucleus.autoCreateSchema</name>
    <value>false</value>
  </property>

  <property>
    <name>datanucleus.autoStartMechanismMode</name>
    <value>checked</value>
    <description>throw exception if metadata tables are incorrect</description>
  </property>

  <property>
    <name>datanucleus.autoStartMechanism</name>
    <value>SchemaTable</value>
  </property>

  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>true</value>
  </property>

  <property>
    <name>datanucleus.transactionIsolation</name>
    <value>read-committed</value>
    <description>Default transaction isolation level for identity generation. </description>
  </property>

  <property>
    <name>datanucleus.cache.level2</name>
    <value>false</value>
    <description>Use a level 2 cache. Turn this off if metadata is changed independently of Hive metastore server</description>
  </property>

  <property>
    <name>datanucleus.cache.level2.type</name>
    <value>SOFT</value>
    <description>SOFT=soft reference based cache, WEAK=weak reference based cache.</description>
  </property>

  <property>
    <name>datanucleus.identifierFactory</name>
    <value>datanucleus1</value>
    <description>Name of the identifier factory to use when generating table/column names etc. 'datanucleus1' is used for backward  compatibility with DataNucleus v1</description>
  </property>


  <property>
    <name>datanucleus.plugin.pluginRegistryBundleCheck</name>
    <value>LOG</value>
    <description>Defines what happens when plugin bundles are found and are duplicated [EXCEPTION|LOG|NONE]</description>
  </property>

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

  <property>
    <name>hive.metastore.execute.setugi</name>
    <value>false</value>
    <description>In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further  note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored.</description>
  </property>

  <property>
    <name>hive.metastore.event.listeners</name>
    <value></value>
    <description>list of comma separated listeners for metastore events.</description>
  </property>

  <property>
    <name>hive.metastore.partition.inherit.table.properties</name>
    <value></value>
    <description>list of comma separated keys occurring in table properties which will get inherited to newly created partitions. *   implies all the keys will get inherited.</description>
  </property>

  <property>
    <name>hive.metadata.export.location</name>
    <value></value>
    <description>When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, it is the location to which the metadata will be exported. The default is an empty string, which results in the metadata being exported   to the current user's home directory on HDFS.</description>
  </property>

  <property>
    <name>hive.metadata.move.exported.metadata.to.trash</name>
    <value></value>
    <description>When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, this setting determines if the metadata that is exported will subsequently be moved to the user's trash directory alongside the   dropped table data. This ensures that the metadata will be cleaned up along with the dropped table data.</description>
  </property>

  <property>
    <name>hive.metastore.partition.name.whitelist.pattern</name>
    <value></value>
    <description>Partition names will be checked against this regex pattern and rejected if not matched.</description>
  </property>

  <property>
    <name>hive.metastore.disallow.incompatible.col.type.change</name>
    <value></value>
    <description>If true (default is false), ALTER TABLE operations which change the type of   a column (say STRING) to an incompatible type (say MAP&lt;STRING, STRING&gt;) are disallowed.    RCFile default SerDe (ColumnarSerDe) serializes the values in such a way that the  datatypes can be converted from string to any type. The map is also serialized as  a string, which can be read as a string as well. However, with any binary   serialization, this is not true. Blocking the ALTER TABLE prevents ClassCastExceptions  when subsequently trying to access old partitions.   Primitive types like INT, STRING, BIGINT, etc are compatible with each other and are   not blocked.  

  See HIVE-4409 for more details.
    </description>
  </property>

  <property>
    <name>hive.metastore.end.function.listeners</name>
    <value></value>
    <description>list of comma separated listeners for the end of metastore functions.</description>
  </property>

  <property>
    <name>hive.metastore.event.expiry.duration</name>
    <value>0</value>
    <description>Duration after which events expire from events table (in seconds)</description>
  </property>

  <property>
    <name>hive.metastore.event.clean.freq</name>
    <value>0</value>
    <description>Frequency at which timer task runs to purge expired events in metastore(in seconds).</description>
  </property>

  <property>
    <name>hive.metastore.connect.retries</name>
    <value>5</value>
    <description>Number of retries while opening a connection to metastore</description>
  </property>

  <property>
    <name>hive.metastore.failure.retries</name>
    <value>3</value>
    <description>Number of retries upon failure of Thrift metastore calls</description>
  </property>

  <property>
    <name>hive.metastore.client.connect.retry.delay</name>
    <value>1</value>
    <description>Number of seconds for the client to wait between consecutive connection attempts</description>
  </property>

  <property>
    <name>hive.metastore.client.socket.timeout</name>
    <value>20</value>
    <description>MetaStore Client socket timeout in seconds</description>
  </property>

  <property>
    <name>hive.metastore.rawstore.impl</name>
    <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
    <description>Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store   and retrieval of raw metadata objects such as table, database</description>
  </property>

  <property>
    <name>hive.metastore.batch.retrieve.max</name>
    <value>300</value>
    <description>Maximum number of objects (tables/partitions) can be retrieved from metastore in one batch. The higher the number, the less the number of round trips is needed to the Hive metastore server, but it may also cause higher memory requirement at the   client side.</description>
  </property>

  <property>
    <name>hive.metastore.batch.retrieve.table.partition.max</name>
    <value>1000</value>
    <description>Maximum number of table partitions that metastore internally retrieves in one batch.</description>
  </property>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9083</value>
    <description>Hive metastore Thrift server</description>
  </property>
  <property>
    <name>mapred.reduce.tasks</name>
    <value>-1</value>
    <description>The default number of reduce tasks per job.  Typically set to a prime close to the number of available hosts.  Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.
    </description>
  </property>
  <property>
    <name>hive.cli.print.header</name>
    <value>false</value>
    <description>Whether to print the names of the columns in query output.</description>
  </property>

  <property>
    <name>hive.cli.print.current.db</name>
    <value>false</value>
    <description>Whether to include the current database in the Hive prompt.</description>
  </property>

  <property>
    <name>hive.cli.prompt</name>
    <value>hive</value>
    <description>Command line prompt configuration value. Other hiveconf can be used in this configuration value. Variable substitution will only be invoked at the Hive
    CLI startup.</description>
  </property>

  <property>
    <name>hive.test.mode</name>
    <value>false</value>
    <description>Whether Hive is running in test mode. If yes, it turns on sampling and prefixes the output tablename.</description>
  </property>

  <property>
    <name>hive.test.mode.prefix</name>
    <value>test_</value>
    <description>if Hive is running in test mode, prefixes the output table by this string</description>
  </property>


  <property>
    <name>hive.test.mode.samplefreq</name>
    <value>32</value>
    <description>if Hive is running in test mode and table is not bucketed, sampling frequency</description>
  </property>

  <property>
    <name>hive.test.mode.nosamplelist</name>
    <value></value>
    <description>if Hive is running in test mode, don't sample the above comma separated list of tables</description>
  </property>

  <property>
    <name>hive.metastore.uris</name>
    <value></value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>
</configuration>

这篇关于在使用配置单元配置mysql之后，无法启动配置单元Metastore服务或配置单元的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！