数据块-创建永久的用户定义函数(UDF) [英] Databricks - Creating permanent User Defined Functions (UDFs)

问题描述

我能够创建一个UDF函数并使用spark.UDF方法注册火花.但是，这仅是每个会话. 群集启动时如何自动注册python UDF功能?这些功能应对所有用户可用.用例示例是将时间从UTC转换为本地时区.

I am able to create a UDF function and register to spark using spark.UDF method. However, this is per session only. How to register python UDF functions automatically when the Cluster starts?. These functions should be available to all users. Example use case is to convert time from UTC to local time zone.

推荐答案

这是不可能的.这不像Hive中的UDF.

This is not possible; this is not like UDFs in Hive.

如果使用spark-submit，则将UDF编码为您提交的包/程序的一部分，或在Spark App随附的jar中.

Code the UDF as part of the package / program you submit or in the jar included in the Spark App, if using spark-submit.

但是

spark.udf.register.udf("...

也必须完成.这适用于Databrick笔记本电脑等.需要根据Spark上下文/会话重新注册UDF.

is required to be done as well. This applies to Databrick notebooks, etc. The UDFs need to be re-registered per Spark Context/Session.

这篇关于数据块-创建永久的用户定义函数(UDF)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！