适用于Spark 2的hbase-spark [英] hbase-spark for Spark 2
问题描述
我想使用 Scala 从 Spark 2 对 hbase 进行全面扫描.
I want to do a full Scan on hbase from Spark 2 using Scala.
我没有固定的目录定义，因此不能选择 SHC 这样的库
I don't have a fixed catalog definition so libraries as SHC are not an option.
我的逻辑选择是使用hbase-spark，在Spark 1.6中可以正常工作
My logical choice was to use hbase-spark, that is working fine in Spark 1.6
除了以前版本中有关此库的不良文档外，令人惊讶的是在检查最新的HBase版本时，例如
In addition to the poor documentation about this library in previous versions, my surprise has been when checking the last HBase releasees, for example tag 2.0, hbase-spark is gone! but still in the master.
所以我的问题是:
- 最新版本的hbase-spark模块在哪里?
- 在哪里可以找到与Spark 2兼容的hbase-spark版本?
thx！
推荐答案
@ bp2010已经回答了部分问题.
@bp2010 already answered part of the question.
有关HBase Spark的信息，请参见下文.它适用于spark 2.
Regarding the HBase Spark see below. It works with spark 2.
有些选项不需要客户端代码提供固定目录:
There are some options that don't demand a fixed catalog from client code:
-
HBase Spark带有示例的源代码在这里: https://github.com/apache/hbase-connectors/tree/master/spark/hbase-spark 在这里，您可以查看有关存储库的说明: https://github.com/apache/hbase-connectors/树/主/火花/hbase-火花
HBase Spark Source code with examples are here: https://github.com/apache/hbase-connectors/tree/master/spark/hbase-spark Here you can see explanations about the repositories: https://github.com/apache/hbase-connectors/tree/master/spark/hbase-spark
Apache Phoenix Spark连接器 https://phoenix.apache.org/phoenix_spark.html
Apache Phoenix Spark connector https://phoenix.apache.org/phoenix_spark.html
我不确定它是否对您有帮助，因为该表必须映射到Phoenix表.如果您有Phoenix，并且问题出在用代码编写目录，但是您可以在HBase Table中标准化类型，对于完整扫描而言，这可能是一种方法.否则，请选择选项1.
I'm not sure if it helps you, since the table must be mapped to a Phoenix table. If you have Phoenix, and you problem is writing the catalog from code, but you can standardize types in HBase Table, for a full scan this can be the way to go. Otherwise, go with option 1.
这篇关于适用于Spark 2的hbase-spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
