比较方法违反了Spark中的一般合同 [英] Comparison method violates its general contract in Spark

问题描述

我试图对List [Row]数据集进行排序，这就是我的处理方法.

I tried to sort my List[Row] data set and here is how I made for it.

def getDiffMinute(ts1:Timestamp, ts2:Timestamp) : Long = {
    if(ts1==null || ts2==null) 0 
    else (ts1.getTime - ts2.getTime) / 60000
}
myList.sortWith( (r1: Row, r2: Row) => 
    MYUtils.getDiffMinute( r1.getAs[Timestamp]("time"), r2.getAs[Timestamp]("time")) < 0
)

由于getDiffMinute函数返回Long类型数据和wortWith需要bool类型，因此无法获取异常. 某些数据列表效果很好，但其他数据列表(尤其是大数据，例如大于1GB)则无法解决此错误.

Since getDiffMinute function return Long type data and wortWith need bool type, there is no way to get exception. Some data lists work so well, but others(especially big data like more than 1gb) does not work with this error.

Comparison method violates its general contract

对此有任何想法吗?

推荐答案

我认为这是因为您的比较器getDiffMinute编写不正确. 在您的情况下，假设B为空，则diff(A，B)= 0，diff(B，C)= 0，因此diff(A，C)也应为0，但是如果A和C都不为零，则可以为任意值空值.

I assume its because your comparator getDiffMinute is not properly written. In your case lets say B is null, then diff(A,B) = 0, diff(B,C) = 0 so diff (A,C) should be 0 too, but it can be anything if neither A and C are nulls.

更多信息: http://docs .oracle.com/javase/6/docs/api/java/util/Comparator.html#compare(T，％20T)

这篇关于比较方法违反了Spark中的一般合同的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！