“缓冲池被破坏"在 Apache FlinkflapMap Operator 中发现的问题 [英] "Buffer pool is destroyed" issue found in Apache Flink flapMap Operator

问题描述

当我尝试在 flatMap 运算符中写入 OUT 集合时，出现非法状态异常(仅在高负载下):缓冲池被破坏我在这里做错了什么?当 flink 抛出 Buffer pool 错误时?

When I try to write to OUT collection in flatMap operator, I get illegal state exception(only under high load):Buffer pool is destroyed What wrong am I doing here ? When flink throws Buffer pool error ?

java.lang.RuntimeException:  Buffer pool is destroyed.

    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
    at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)

..........
Caused by: java.lang.IllegalStateException:  Buffer pool is destroyed.
    at org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
    at org.apache.flink.runtime.io.network.buffer.BufferBuilder.append(BufferBuilder.java:79)
    at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.copyToBufferBuilder(SpanningRecordSerializer.java:95)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:150)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:128)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:101)
    at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)```

推荐答案

什么是缓冲池?

缓冲池是用于网络堆栈的固定大小的内存段实例池，这意味着您在发送数据时将段放入其中，并在通过网络接收时从中获取段.

Buffer pool is a fixed size pool of memory segments instances for the network stack, which means that you put segments into it when you send data and fetch segments from it when you receive across the network.

为什么会被销毁?

由于网络环境关闭，缓冲池被破坏.

The buffer pool is destroyd because the network environment shuts down.

为什么 NetworkEnvironment 关闭?

Why does NetworkEnvironment shut down?

这可能是我不太确定的特定情况.我建议你阅读更多关于这个缓冲池异常的日志.

It may be a specific circumtances which I'm not very sure. I suggest you read more logs above this buffer pool exception.

这篇关于“缓冲池被破坏"在 Apache FlinkflapMap Operator 中发现的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！