Zookeeper不断收到WARN:“捕获到流末尾异常" [英] Zookeeper keeps getting the WARN: "caught end of stream exception"

问题描述

我现在正在使用CDH-5.3.1集群，其中三个Zookeeper实例位于三个ip中:

I am now using a CDH-5.3.1 cluster with three zookeeper instances located in three ips:

133.0.127.40 n1
133.0.127.42 n2
133.0.127.44 n3

启动时一切正常，但是现在我注意到节点n2一直在收到警告:

Everything works fine when it starts, but these days I notice that the node n2 keeps getting the WARN:

caught end of stream exception

EndOfStreamException: Unable to read additional data from client sessionid **0x0**, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
    at java.lang.Thread.run(Thread.java:722)

它每秒钟发生一次，并且仅在n2上发生，而n1和n3可以. 我仍然可以使用HBase Shell扫描表，并使用Solr WEB UI进行查询.但是我无法启动Flume代理，此过程全部停止在这一步:

it happens every second, and only on n2, while n1 and n3 are fine. I can still use HBase shell to scan my table, and the Solr WEB UI to do querys. But I cannot start Flume agents, the process all stops at this step:

Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

jetty-6.1.26.cloudera.4

Started SelectChannelConnector@0.0.0.0:41414.

几分钟后，我从Cloudera Manager收到警告，Flume代理已超出文件描述符的阈值.

And minutes later I get the warning from Cloudera Manager that Flume agent is exceeding the threshold of File Descriptors.

有人知道出什么问题了吗?预先感谢.

Does anyone know what is going wrong? Thanks in advance.

推荐答案

我记得在ZK中看到了类似的错误(当然不是Flume).我相信当时的问题与存储在节点上和/或传输到客户端的大量数据有关.在zoo.cfg中需要考虑的事项:

I recall seeing similar errors in ZK (admittedly not with Flume). I believe the problem at the time was to do with the large amount of data stored on the node and/or transferred to the client. Things to consider tweaking in zoo.cfg:

• 限制autopurge.snapRetainCount，例如将其设置为10
• 将autopurge.purgeInterval设置为2(小时)

• put a limit on autopurge.snapRetainCount, e.g. set it to 10
• set autopurge.purgeInterval to, say, 2 (hours)

如果ZK客户端(Flume?)正在将大型znode流传输到ZK集群或从ZK集群流传输大型znode，则可能要在客户端JVM上设置Java系统属性 jute.maxbuffer ，并且可能在服务器节点上达到足够大的值.我相信此属性的默认值为1M.恐怕要反复试验才能确定适合您的工作量的价值！

If the ZK client (Flume?) is streaming large znodes to/from the ZK cluster, you may want to set the Java system property jute.maxbuffer on the client JVM(s), and possibly on the server nodes, to a large enough value. I believe the default value for this property is 1M. Determining the appropriate value for your workload is an exercise in trial and error I'm afraid!

这篇关于Zookeeper不断收到WARN:“捕获到流末尾异常"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！