火花中的哈希函数 [英] Hash function in spark
问题描述
我正在尝试向数据框中添加一列,该列将包含另一列的哈希.
I'm trying to add a column to a dataframe, which will contain hash of another column.
我找到了以下文档:
https://spark.apache.org/docs/2.3 .0/api/sql/index.html#hash
并尝试了此:
I've found this piece of documentation:
https://spark.apache.org/docs/2.3.0/api/sql/index.html#hash
And tried this:
import org.apache.spark.sql.functions._
val df = spark.read.parquet(...)
val withHashedColumn = df.withColumn("hashed", hash($"my_column"))
但是该hash()
使用的哈希函数是什么?是murmur
,sha
,md5
,还有其他内容吗?
But what is the hash function used by that hash()
? Is that murmur
, sha
, md5
, something else?
我在此列中获得的值是整数,因此此处的值范围可能是[-2^(31) ... +2^(31-1)]
.
我可以在这里获得长期价值吗?我可以代替字符串哈希吗?
如何为此指定具体的哈希算法?
我可以使用自定义哈希函数吗?
The value I get in this column is integer, thus range of values here is probably [-2^(31) ... +2^(31-1)]
.
Can I get a long value here? Can I get a string hash instead?
How can I specify a concrete hashing algorithm for that?
Can I use a custom hash function?
推荐答案
这是基于源代码的Murmur.
It is Murmur based on the source code.
/**
* Calculates the hash code of given columns, and returns the result as an int column.
*
* @group misc_funcs
* @since 2.0.0
*/
@scala.annotation.varargs
def hash(cols: Column*): Column = withExpr {
new Murmur3Hash(cols.map(_.expr))
}
这篇关于火花中的哈希函数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!