无法使用来自 Spark SQL 的现有 Hive 永久 UDF [英] Unable to use an existing Hive permanent UDF from Spark SQL
问题描述
我之前已经在 hive 中注册了一个 UDF.它是永久性的,而不是 TEMPORARY
.它在直线上工作.
CREATE FUNCTION normaliseURL AS 'com.example.hive.udfs.NormaliseURL' USING JAR 'hdfs://udfs/hive-udfs.jar';
我已将 spark 配置为使用 hive Metastore.配置正在工作,因为我可以查询配置单元表.我可以看到UDF;
在[9]中:spark.sql('describe function normaliseURL').show(truncate=False)+--------------------------------------------+|function_desc |+--------------------------------------------+|功能:default.normaliseURL ||类:com.example.hive.udfs.NormaliseURL ||用法:不适用.|+--------------------------------------------+
但是我不能在sql语句中使用UDF;
spark.sql('SELECT normaliseURL("value")')AnalysisException:未定义的函数:'default.normaliseURL'.此函数既不是注册的临时函数,也不是在数据库 'default' 中注册的永久函数.;第 1 行 pos 7"
如果我尝试使用 spark(绕过 Metastore)注册 UDF,它会注册失败,表明它已经存在.
In [12]: spark.sql("create function normaliseURL as 'com.example.hive.udfs.NormaliseURL'")AnalysisException: "函数 'default.normaliseURL' 已经存在于数据库 'default' 中;"
我使用的是 Spark 2.0,hive Metastore 1.1.0.UDF是scala,我的spark驱动代码是python.
我被难住了.
- 我对 Spark 可以使用 Metastore 定义的永久 UDF 的假设是否正确?
- 我是否在 hive 中正确创建了函数?
问题是 Spark 2.0 无法执行 JAR 位于 HDFS 上的函数.
Spark SQL:Thriftserver 无法运行已注册的 Hive UDTF>
一种解决方法是将该函数定义为 Spark 作业中的临时函数,其中 jar 路径指向本地边缘节点路径.然后在同一个 Spark 作业中调用该函数.
CREATE TEMPORARY FUNCTION functionName as 'com.test.HiveUDF' USING JAR '/user/home/dir1/functions.jar'
I have previously registered a UDF with hive. It is permanent not TEMPORARY
. It works in beeline.
CREATE FUNCTION normaliseURL AS 'com.example.hive.udfs.NormaliseURL' USING JAR 'hdfs://udfs/hive-udfs.jar';
I have spark configured to use the hive metastore. The config is working as I can query hive tables. I can see the UDF;
In [9]: spark.sql('describe function normaliseURL').show(truncate=False)
+-------------------------------------------+
|function_desc |
+-------------------------------------------+
|Function: default.normaliseURL |
|Class: com.example.hive.udfs.NormaliseURL |
|Usage: N/A. |
+-------------------------------------------+
However I cannot use the UDF in a sql statement;
spark.sql('SELECT normaliseURL("value")')
AnalysisException: "Undefined function: 'default.normaliseURL'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7"
If I attempt to register the UDF with spark (bypassing the metastore) it fails to register it, suggesting that it does already exist.
In [12]: spark.sql("create function normaliseURL as 'com.example.hive.udfs.NormaliseURL'")
AnalysisException: "Function 'default.normaliseURL' already exists in database 'default';"
I'm using Spark 2.0, hive metastore 1.1.0. The UDF is scala, my spark driver code is python.
I'm stumped.
- Am I correct in my assumption that Spark can utilise metastore-defined permanent UDFs?
- Am I creating the function correctly in hive?
Issue is Spark 2.0 is not able to execute the functions whose JARs are located on HDFS.
Spark SQL: Thriftserver unable to run a registered Hive UDTF
One workaround is to define the function as a temporary function in Spark job with jar path pointing to a local edge-node path. Then call the function in same Spark job.
CREATE TEMPORARY FUNCTION functionName as 'com.test.HiveUDF' USING JAR '/user/home/dir1/functions.jar'
这篇关于无法使用来自 Spark SQL 的现有 Hive 永久 UDF的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!