消息的处理过程中发生的节俭传输错误 [英] Thrift transport error occurred during processing of message

问题描述

我正在卡桑德拉1.2.6和Apache 0.8.0星火

下面我从卡桑德拉使用星火的newAPIHadoopRDD创建RDD。

当我运行的火花工作APPLN我发现Cassandra是记录下面的错误消息

  21 INFO：36：28629听力节俭的客户...
DEBUG 21：36：29730传播负载信息...
DEBUG 21：36：57844入门replayAllFailedBatches
DEBUG 21：36：57845 forceFlush要求，但一切都在batchlog干净
DEBUG 21：36：57846成品replayAllFailedBatches
DEBUG 21：37：29731传播负载信息...
DEBUG 21：37：57846入门replayAllFailedBatches
DEBUG 21：37：57847 forceFlush要求，但一切都在batchlog干净
DEBUG 21：37：57847成品replayAllFailedBatches
DEBUG 21：38：29732传播负载信息...
DEBUG 21：38：57847入门replayAllFailedBatches
DEBUG 21：38：57849 forceFlush要求，但一切都在batchlog干净
DEBUG 21：38：57849成品replayAllFailedBatches
DEBUG 21：39：29732传播负载信息...
DEBUG 21：39：57849入门replayAllFailedBatches
DEBUG 21：39：57850 forceFlush要求，但一切都在batchlog干净
DEBUG 21：39：57850成品replayAllFailedBatches
DEBUG 21：39：57956计算范围为-3011659447910895493
DEBUG 21：消息的处理过程中发生00043节俭传输错误：40。
org.apache.thrift.transport.TTransportException
    在org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:132）
    在org.apache.thrift.transport.TTransport.readAll（TTransport.java:84）
    在org.apache.thrift.transport.TFramedTransport.readFrame（TFramedTransport.java:129）
    在org.apache.thrift.transport.TFramedTransport.read（TFramedTransport.java:101）
    在org.apache.thrift.transport.TTransport.readAll（TTransport.java:84）
    在org.apache.thrift.protocol.TBinaryProtocol.readAll（TBi​​naryProtocol.java:378）
    在org.apache.thrift.protocol.TBinaryProtocol.readI32（TBinaryProtocol.java:297）
    在org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin（TBinaryProtocol.java:204）
    在org.apache.thrift.TBaseProcessor.process（TBaseProcessor.java:22）
    在org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
    在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145）
    在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:615）
    在java.lang.Thread.run（Thread.java:744）
DEBUG 21：消息的处理过程中发生00048节俭传输错误：40。
org.apache.thrift.transport.TTransportException
    在org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:132）
    在org.apache.thrift.transport.TTransport.readAll（TTransport.java:84）
    在org.apache.thrift.transport.TFramedTransport.readFrame（TFramedTransport.java:129）
    在org.apache.thrift.transport.TFramedTransport.read（TFramedTransport.java:101）
    在org.apache.thrift.transport.TTransport.readAll（TTransport.java:84）
    在org.apache.thrift.protocol.TBinaryProtocol.readAll（TBi​​naryProtocol.java:378）
    在org.apache.thrift.protocol.TBinaryProtocol.readI32（TBinaryProtocol.java:297）
    在org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin（TBinaryProtocol.java:204）
    在org.apache.thrift.TBaseProcessor.process（TBaseProcessor.java:22）
    在org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
    在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145）
    在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:615）
    在java.lang.Thread.run（Thread.java:744）
调试21：40：03071 execute_cql3_query
DEBUG 21：40：03090请求完成
DEBUG 21：40：03253 prepare_cql3_query
DEBUG 21：40：03267 execute_ prepared_cql3_query
DEBUG 21：40：03275请求完成
DEBUG 21：40：03291 prepare_cql3_query
DEBUG 21：40：03310 exe​​cute_ prepared_cql3_query
DEBUG 21：40：03312请求完成
DEBUG 21：40：03314 prepare_cql3_query
DEBUG 21：40：03326 execute_ prepared_cql3_query
DEBUG 21：40：03327请求完成
 

解决方案

请尝试引发新DataStax卡桑德拉驱动程序（可用的这里）。它能够直接访问卡桑德拉，而不Hadoop的API。

I am running cassandra 1.2.6 and Apache Spark 0.8.0.

Here I am creating a RDD from Cassandra using Spark's newAPIHadoopRDD.

When I run the job from spark appln I find that cassandra is logging the below error messages

INFO 21:36:28,629 Listening for thrift clients...
DEBUG 21:36:29,730 Disseminating load info ...
DEBUG 21:36:57,844 Started replayAllFailedBatches
DEBUG 21:36:57,845 forceFlush requested but everything is clean in batchlog
DEBUG 21:36:57,846 Finished replayAllFailedBatches
DEBUG 21:37:29,731 Disseminating load info ...
DEBUG 21:37:57,846 Started replayAllFailedBatches
DEBUG 21:37:57,847 forceFlush requested but everything is clean in batchlog
DEBUG 21:37:57,847 Finished replayAllFailedBatches
DEBUG 21:38:29,732 Disseminating load info ...
DEBUG 21:38:57,847 Started replayAllFailedBatches
DEBUG 21:38:57,849 forceFlush requested but everything is clean in batchlog
DEBUG 21:38:57,849 Finished replayAllFailedBatches
DEBUG 21:39:29,732 Disseminating load info ...
DEBUG 21:39:57,849 Started replayAllFailedBatches
DEBUG 21:39:57,850 forceFlush requested but everything is clean in batchlog
DEBUG 21:39:57,850 Finished replayAllFailedBatches
DEBUG 21:39:57,956 computing ranges for -3011659447910895493
DEBUG 21:40:00,043 Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
DEBUG 21:40:00,048 Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
DEBUG 21:40:03,071 execute_cql3_query
DEBUG 21:40:03,090 request complete
DEBUG 21:40:03,253 prepare_cql3_query
DEBUG 21:40:03,267 execute_prepared_cql3_query
DEBUG 21:40:03,275 request complete
DEBUG 21:40:03,291 prepare_cql3_query
DEBUG 21:40:03,310 execute_prepared_cql3_query
DEBUG 21:40:03,312 request complete
DEBUG 21:40:03,314 prepare_cql3_query
DEBUG 21:40:03,326 execute_prepared_cql3_query
DEBUG 21:40:03,327 request complete

解决方案

Please try a new DataStax Cassandra driver for Spark (available here). It is able to access Cassandra directly, without Hadoop API.

这篇关于消息的处理过程中发生的节俭传输错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！