解决"Kryo序列化失败:缓冲区溢出".火花异常 [英] Resolving "Kryo serialization failed: Buffer overflow" Spark exception

问题描述

我正在尝试运行Spark(Java)代码并收到错误消息

I am trying to run Spark (Java) code and getting the error

org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 27".

其他帖子建议将缓冲区设置为最大值.当我尝试最大缓冲区值为512MB时，出现错误

Other posts have suggested setting the buffer to its max value. When I tried this with max buffer value of 512MB I got the error

java.lang.ClassNotFoundException: org.apache.spark.serializer.KryoSerializer.buffer.max', '512'

我该如何解决这个问题?

How can I solve this problem?

推荐答案

尝试使用" spark.kryoserializer.buffer.max.mb "，" 512 "代替 spark.kryoserializer.buffer.max "，" 512MB "

Try using "spark.kryoserializer.buffer.max.mb", "512" instead spark.kryoserializer.buffer.max", "512MB"

这篇关于解决"Kryo序列化失败:缓冲区溢出".火花异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！