java.io.IOException：ensureRemaining：只剩下0个字节，试图读取1 [英] java.io.IOException: ensureRemaining: Only 0 bytes remaining, trying to read 1
本文介绍了java.io.IOException：ensureRemaining：只剩下0个字节，试图读取1的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我在giraph的自定义类中遇到了一些问题。我做了一个VertexInput和Output格式，但我总是得到以下错误：
java.io.IOException：ensureRemaining：Only *剩余字节，试图读取*
，其中放置了*的不同值。
这是在单个节点集群上测试的。
这个问题发生在vertexIterator执行next（）时，并且没有更多的顶点。这个迭代器是从flush方法调用的，但我不明白，基本上，为什么next（）方法失败。这里有一些日志和类...
我的日志如下：
15/09/08 00:52:21信息bsp.BspService：BspService：通过作业连接到ZooKeeper giraph_yarn_application_1441683854213_0001,1在localhost上：22181
15/09/08 00:52:21 INFO zookeeper。 ZooKeeper：客户端环境：zookeeper.version = 3.4.5-1392090，构建于09/30/2012 17:52 GMT
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：主机。 name = localhost
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：java.version = 1.7.0_79
15/09/08 00:52:21信息zookeeper.ZooKeeper ：客户端环境：java.vendor = Oracle Corporation
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：java.home = / usr / lib / jvm / java-7-openjdk-amd64 / jre
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：java.class.path =。：$ {CLASSPATH}：./**/
15/09 / 08 00:52:21 INFO zookeeper.ZooKeeper：客户端环境：java.library.path = / usr / java / packages / lib / amd64：/ us r / lib / x86_64-linux-gnu / jni：/ lib / x86_64-linux-gnu：/ usr / lib / x86_64-linux-gnu：/ usr / lib / jni：/ lib：/ usr / l $
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：java.io.tmpdir = / tmp
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：java .compiler = LT; NA>
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：os.name = Linux
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：os .arch = amd64
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：os.version = 3.13.0-62-generic
15/09/08 00:52： 21 INFO zookeeper.ZooKeeper：客户端环境：user.name = hduser
15/09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：user.home = / home / hduser
15 / 09/08 00:52:21信息zookeeper.ZooKeeper：客户端环境：user.dir = / app / hadoop / tmp / nm-local-dir / usercache / hduser / appcache / application_1441683854213_0001 / container_1441683854213_0001_01_000003
15/09 / 08 0时52分21秒INFO zookeeper.ZooKeeper：启动客户端连接，connectString中=本地主机：22181 sessionTimeout = 60000 watcher=org.apache.giraph.worker.BspServiceWorker@4256d3a0
15/09/08 0时52分21秒INFO zookeeper.ClientCnxn：打开到本地服务器localhost / 127.0.0.1的套接字连接：22181。不会尝试使用SASL进行身份验证（未知错误）
15/09/08 00:52:21 INFO zookeeper.ClientCnxn：建立到localhost / 127.0.0.1的套接字连接：22181，启动会话
15 / 09/08 0时52分21秒INFO zookeeper.ClientCnxn：会话建立在服务器localhost / 127.0.0.1齐全：22181，会话ID = 0x14fab0de0bb0002，协商超时= 40000
15/09/08 0时52分21秒INFO BSP .BspService：进程：异步连接完成。
15/09/08 00:52:21信息netty.NettyServer：NettyServer：使用带有8个线程的执行组for requestFrameDecoder。
15/09/08 00:52:21信息Configuration.deprecation：mapred.map.tasks已弃用。而是使用mapreduce.job.maps
15/09/08 00:52:21 INFO netty.NettyServer：start：已启动服务器通信服务器：localhost / 127.0.0.1：30001绑定尝试时最多有16个线程0 with sendBufferSize = 32768 receiveBufferSize = 524288
15/09/08 00:52:21 INFO netty.NettyClient：NettyClient：在请求编码器后面使用8个线程的执行处理程序。
15/09/08 00:52:21信息graph.GraphTaskManager：setup：注册该工作人员的健康...
15/09/08 00:52:21 INFO yarn.GiraphYarnTask：[STATUS ：task-1] WORKER_ONLY starting ...
15/09/08 00:52:22 INFO bsp.BspService：getJobState：作业状态已经存在（/ _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _masterJobState）
15/09 / 08 00:52:22 INFO bsp.BspService：getApplicationAttempt：Node / _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _applicationAttemptsDir已经存在！
15/09/08 00:52:22 INFO bsp.BspService：getApplicationAttempt：Node / _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _applicationAttemptsDir已经存在！
15/09/08 00:52:22 INFO worker.BspServiceWorker：registerHealth：为attempt = 0创建了我的健康节点，superstep = -1与/ _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _applicationAttemptsDir / 0 / _superstepD $
15/09/08 00:52:22信息netty.NettyServer：start：使用Netty而不进行身份验证。
15/09/08 00:52:22 INFO bsp.BspService：process：partitionAssignmentsReadyChanged（分配已分配）
15/09/08 00:52:22信息worker.BspServiceWorker：startSuperstep：Master（ hostname = localhost，MRtaskID = 0，port = 30000）
15/09/08 00:52:22 INFO worker.BspServiceWorker：startSuperstep：准备好superstep上的计算-1，因为工作者选择和顶点范围分配在/ _hadoopBsp / giraph_yarn_application_1441683854 $
15/09/08 00:52:22 INFO yarn.GiraphYarnTask：[STATUS：task-1] startSuperstep：WORKER_ONLY - Attempt = 0，Superstep = -1
15/09 / 08 00:52:22 INFO netty.NettyClient：使用Netty而不进行身份验证。
15/09/08 00:52:22信息netty.NettyClient：使用Netty而无需身份验证。
15/09/08 00:52:22信息netty.NettyClient：connectAllAddresses：成功添加2个连接，（总共连接2个）0失败，共0次失败。
15/09/08 00:52:22信息netty.NettyServer：start：使用Netty而不进行身份验证。
15/09/08 00:52:22 INFO handler.RequestDecoder：decode：服务器窗口指标MBytes / sec received = 0，MBytesReceived = 0.0001，ave收到req MBytes = 0.0001，secs等待= 1.44168435E9
15/09/08 00:52:22 INFO worker.BspServiceWorker：loadInputSplits：使用1个线程，最初1个线程完成1次分割。
15/09/08 00:52:22 INFO worker.InputSplitsHandler：reserveInputSplit：保留的输入拆分路径/ _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _vertexInputSplitDir / 0，总体上大约0.0％输入拆分rese $
15/09 / 08 00:52:22 INFO worker.InputSplitsCallable：getInputSplit：Reserved / _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _vertexInputSplitDir / 0来自ZooKeeper并获得输入分割'hdfs：// hdnode01：54310 / u $
15/09/08 00： 52:22 INFO worker.InputSplitsCallable：loadFromInputSplit：已完成加载/ _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _vertexInputSplitDir / 0（v = 6，e = 10）
15/09/08 00:52:22 INFO worker.InputSplitsCallable：call：加载1输入拆分0.16241108秒，（v = 6，e = 10）36.94329顶点/秒，61.572155边/秒
15/09/08 00:52:22错误utils.LogStacktraceCallable：执行可调用失败
java.lang.IllegalStateException：next：IOException
at org.apache.giraph.utils.VertexIterator.next（VertexIte （org.apache.giraph.partition.BasicPartition.addPartitionVertices（BasicPartition.java:99）
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest（SendWorkerVerticesRequest。
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest（NettyWorkerClientRequestProcessor.java:466）
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush（NettyWorkerClientRequestProcessor。
at org.apache.giraph.worker.InputSplitsCallable.call（InputSplitsCallable.java:241）
at org.apache.giraph.worker.InputSplitsCallable.call（InputSplitsCallable.java:60）
at org.apache.giraph.utils.LogStacktraceCallable.call（LogStacktraceCallable.java:51）
at java.util.concurrent.FutureTask.run（FutureTask.java:262）
at java .util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145）
at java.util.concurrent.Thr eadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）$ b $在java.lang.Thread.run（Thread.java:745）
导致：java.io.IOException：ensureRemaining：只剩下0个字节，试图在org.apache.giraph.utils.UnsafeArrayReads.readByte（UnsafeArrayReads.java:123）上阅读org.apache.giraph.utils.UnsafeReads.ensureRemaining（UnsafeReads.java:77）
中的1
）
at org.apache.giraph.utils.UnsafeReads.readLine（UnsafeReads.java:100）
at pruebas.TextAndDoubleComplexWritable.readFields（TextAndDoubleComplexWritable.java:37）
at org.apache。 giraph.utils.WritableUtils.reinitializeVertexFromDataInput（WritableUtils.java:540）
at org.apache.giraph.utils.VertexIterator.next（VertexIterator.java:98）
... 11 more
15/09/08 00:52:22错误worker.BspServiceWorker：unregisterHealth：失败，取消注册运行状况/ _hadoopBsp / giraph_yarn_application_1441683854213_0001 / _applicationAttemptsDir / 0 / _superstepDir / -1 / _w orkerHea $
15/09/08 0点52分22秒ERROR yarn.GiraphYarnTask：GiraphYarnTask扔了顶级异常，失败的任务
了java.lang.RuntimeException：运行：捕捉到一个不可恢复的异常WAITFOR：为ExecutionException在等待org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4bbf48f0
时发生org.apache.giraph.yarn.GiraphYarnTask.run（GiraphYarnTask.java:104）
org.apache.giraph .yarn.GiraphYarnTask.main（GiraphYarnTask.java:183）
导致：java.lang.IllegalStateException：waitFor：等待org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4bbf48f0
时发生ExecutionException在org.apache.giraph.utils.ProgressableUtils.waitFor（ProgressableUtils.java:193）
在org.apache.giraph.utils.ProgressableUtils.waitForever（ProgressableUtils.java:151）
在org.apache .giraph.utils.ProgressableUtils.waitForever（ProgressableUtils.java:136）
在org.apache.giraph.utils.ProgressableUtils.getFu tureResult（ProgressableUtils.java:99）
at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables（ProgressableUtils.java:233）
at org.apache.giraph.worker.BspServiceWorker.loadInputSplits（BspServiceWorker。
at org.apache.giraph.worker.BspServiceWorker.loadVertices（BspServiceWorker.java:409）
at org.apache.giraph.worker.BspServiceWorker.setup（BspServiceWorker.java:629）在org.apache.giraph.graph.GraphTaskManager.execute（GraphTaskManager.java:284）
在org.apache.giraph.yarn.GiraphYarnTask.run（GiraphYarnTask.java:92）
... 1 more
引发java.util.concurrent.ExecutionException：java.lang.IllegalStateException：next：IOException $ b $ java.util.concurrent.FutureTask.report（FutureTask.java:122）
at java.util.concurrent.FutureTask.get（FutureTask.java:202）
at org.apache.giraph.utils.ProgressableUtils $ FutureWaitable.waitFor（ProgressableUtils.java:31 2）
在org.apache.giraph.utils.ProgressableUtils.waitFor（ProgressableUtils.java:185）
... 10 more
引起：java.lang.IllegalStateException：next：IOException在org.apache.giraph.utils.VertexIterator.next（VertexIterator.java:101）
在org.apache.giraph.partition.BasicPartition.addPartitionVertices（BasicPartition.java:99）
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest（SendWorkerVerticesRequest.java:115）
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest（NettyWorkerClientRequestProcessor.java:466）
在org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush（NettyWorkerClientRequestProcessor.java:412）
at org.apache.giraph.worker.InputSplitsCallable.call（InputSplitsCallable.java:241）
at org .apache.giraph.worker.InputSplitsCallable.call（InputSplitsCallable.java:60）
at org.apache.giraph.utils.LogStackt $ java.util.concurrent.FutureTask.run（FutureTask.java:262）
在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java： 1145）
在java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）$ b $在java.lang.Thread.run（Thread.java:745）
导致：java.io.IOException：ensureRemaining：只剩下0个字节，试图读取org.apache.giraph.utils.UnsafeReads.ensureRemaining（UnsafeReads.java:77）$ b $ org.apache处的
。 giraph.utils.UnsafeArrayReads.readByte（UnsafeArrayReads.java:123）
at org.apache.giraph.utils.UnsafeReads.readLine（UnsafeReads.java:100）
at pruebas.TextAndDoubleComplexWritable.readFields（TextAndDoubleComplexWritable。 java：37）
at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput（WritableUtils.java:540）
at org.apache.giraph.utils.VertexIterator.next（ VertexIterator.java:98）
... 11更多
我的输入格式： p>
package pruebas;界面。这是至关重要的，因为是让我们在giraph中发送和接收信息的方法。我在
导入org.apache.giraph.edge.Edge;
import org.apache.giraph.edge.EdgeFactory;
import org.apache.giraph.io.formats.AdjacencyListTextVertexInputFormat;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
/ **
*表@author hduser
*的
* /
公共类IdTextWithComplexValueInputFormat
延伸
AdjacencyListTextVertexInputFormat<文本， TextAndDoubleComplexWritable，DoubleWritable> {
@Override
public AdjacencyListTextVertexReader createVertexReader（InputSplit split，
TaskAttemptContext context）{
return new TextComplexValueDoubleAdjacencyListVertexReader（）;
protected class TextComplexValueDoubleAdjacencyListVertexReader extends
AdjacencyListTextVertexReader {
/ **
*带有
*的构造函数* {@link AdjacencyListTextVertexInputFormat .LineSanitizer}。
*
* @param lineSanitizer
*用于读取
* /
的清理程序public TextComplexValueDoubleAdjacencyListVertexReader（）{
super（）;
}
@Override
public Text decodeId（String s）{
return new Text（s）;
$ b @Override
public TextAndDoubleComplexWritable decodeValue（String s）{
TextAndDoubleComplexWritable valorComplejo = new TextAndDoubleComplexWritable（）;
valorComplejo.setVertexData（Double.valueOf（s））;
valorComplejo.setIds_vertices_anteriores（）;
返回valorComplejo;
}
@Override
public Edge< Text，DoubleWritable> decodeEdge（String s1，String s2）{
return EdgeFactory.create（new Text（s1），
new DoubleWritable（Double.valueOf（s2）））;
$ b code
$ b TextAndDoubleComplexWritable：package pruebas;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.Writable;
public class TextAndDoubleComplexWritable implements Writable {
private String idsVerticesAnteriores;
private double vertexData;
public TextAndDoubleComplexWritable（）{
super（）;
this.idsVerticesAnteriores =;
}
public TextAndDoubleComplexWritable（double vertexData）{
super（）;
this.vertexData = vertexData;
public TextAndDoubleComplexWritable（String ids_vertices_anteriores，
double vertexData）{
super（）;
this.idsVerticesAnteriores = ids_vertices_anteriores;
this.vertexData = vertexData;
}
public void write（DataOutput out）throws IOException {
out.writeUTF（idsVerticesAnteriores）;
public void readFields（DataInput in）throws IOException {
idsVerticesAnteriores = in.readLine（）;
}
public String getIds_vertices_anteriores（）{
return idsVerticesAnteriores;
}
public void setIds_vertices_anteriores（String ids_vertices_anteriores）{
this.idsVerticesAnteriores = ids_vertices_anteriores;
}
public double getVertexData（）{
return vertexData;
}
public void setVertexData（double vertexData）{
this.vertexData = vertexData;
我的输入文件：
Portada 0.0 Sugerencias 1.0
Sugerencias 3.0 Portada 1.0
，我用这个命令执行它：
$ HADOOP_HOME / bin / yarn jar $ GIRAPH_HOME / giraph-examples / target / giraph-examples-1.1.0-for-hadoop-2.4.0 -jar -with-dependencies.jar org.apache.giraph.GiraphRunner lectura_de_grafo.BusquedaDeCaminosNavegacionalesWikiquote -vif pruebas.IdTextWithComplexValueInputFormat -vip / user / hduser /input/wiki-graph-chiquito.txt -op / user / hduser / output / caminosNavegacionales -w 2 -yh 250
任何帮助将不胜感激！
更新：
我的输入文件错误。 Giraph（或者我的例子）不能很好地处理未列出的顶点。
但问题仍然存在。我更新了原始问题上的文件数据。
UPDATE 2：
OutputFormat没有被使用，并且计算算法从不被执行。我将两者都删除，以帮助澄清问题。
Update 3，19/11/2015：
问题不在于输入格式，输入格式运行良好并完全读取数据。
问题发生在类TextAndDoubleComplexWritable中，我将它添加到我的原始问题中，以更好地解释这个最终解决方案（我也添加了一个答案） 。解决方案问题出在TextAndDoubleComplexWritable类中。当我们实现
readFields和write时，我并没有意识到方法的重要性。可写入readFields方法中写了一个空字符串，我应该使用该方法写入我的顶点的两个值。我以如下方式更新了这两种方法：
$ b $ pre $public void write（DataOutput out）throws IOException {
out.writeDouble （this.vertexData）;
out.writeUTF（this.idsVerticesAnteriores！=hola
：this.idsVerticesAnteriores）;
}
public void readFields（DataInput in）throws IOException {
this.vertexData = in.readDouble（）;
this.idsVerticesAnteriores = in.readUTF（）;
// idsVerticesAnteriores = in.readLine（）;
}
这是行得通的，最后!!
i'm having some problems with custom classes in giraph. I made a VertexInput and Output format, but i always getting the following error:
java.io.IOException: ensureRemaining: Only * bytes remaining, trying to read *
with different values where the "*" are placed.
This was tested on a Single Node Cluster.
This problem happen when a vertexIterator do next(), and there aren't any more vertex left. This iterator it's invocated from a flush method, but i don't understand, basically, why the "next()" method is failing. Here are some logs and classes...
My log is the following:
15/09/08 00:52:21 INFO bsp.BspService: BspService: Connecting to ZooKeeper with job giraph_yarn_application_1441683854213_0001, 1 on localhost:22181
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:host.name=localhost
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.class.path=.:${CLASSPATH}:./**/
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/l$
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:os.version=3.13.0-62-generic
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:user.name=hduser
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/application_1441683854213_0001/container_1441683854213_0001_01_000003
15/09/08 00:52:21 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:22181 sessionTimeout=60000 watcher=org.apache.giraph.worker.BspServiceWorker@4256d3a0
15/09/08 00:52:21 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:22181. Will not attempt to authenticate using SASL (unknown error)
15/09/08 00:52:21 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:22181, initiating session
15/09/08 00:52:21 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:22181, sessionid = 0x14fab0de0bb0002, negotiated timeout = 40000
15/09/08 00:52:21 INFO bsp.BspService: process: Asynchronous connection complete.
15/09/08 00:52:21 INFO netty.NettyServer: NettyServer: Using execution group with 8 threads for requestFrameDecoder.
15/09/08 00:52:21 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/09/08 00:52:21 INFO netty.NettyServer: start: Started server communication server: localhost/127.0.0.1:30001 with up to 16 threads on bind attempt 0 with sendBufferSize = 32768 receiveBufferSize = 524288
15/09/08 00:52:21 INFO netty.NettyClient: NettyClient: Using execution handler with 8 threads after request-encoder.
15/09/08 00:52:21 INFO graph.GraphTaskManager: setup: Registering health of this worker...
15/09/08 00:52:21 INFO yarn.GiraphYarnTask: [STATUS: task-1] WORKER_ONLY starting...
15/09/08 00:52:22 INFO bsp.BspService: getJobState: Job state already exists (/_hadoopBsp/giraph_yarn_application_1441683854213_0001/_masterJobState)
15/09/08 00:52:22 INFO bsp.BspService: getApplicationAttempt: Node /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_applicationAttemptsDir already exists!
15/09/08 00:52:22 INFO bsp.BspService: getApplicationAttempt: Node /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_applicationAttemptsDir already exists!
15/09/08 00:52:22 INFO worker.BspServiceWorker: registerHealth: Created my health node for attempt=0, superstep=-1 with /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_applicationAttemptsDir/0/_superstepD$
15/09/08 00:52:22 INFO netty.NettyServer: start: Using Netty without authentication.
15/09/08 00:52:22 INFO bsp.BspService: process: partitionAssignmentsReadyChanged (partitions are assigned)
15/09/08 00:52:22 INFO worker.BspServiceWorker: startSuperstep: Master(hostname=localhost, MRtaskID=0, port=30000)
15/09/08 00:52:22 INFO worker.BspServiceWorker: startSuperstep: Ready for computation on superstep -1 since worker selection and vertex range assignments are done in /_hadoopBsp/giraph_yarn_application_1441683854$
15/09/08 00:52:22 INFO yarn.GiraphYarnTask: [STATUS: task-1] startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1
15/09/08 00:52:22 INFO netty.NettyClient: Using Netty without authentication.
15/09/08 00:52:22 INFO netty.NettyClient: Using Netty without authentication.
15/09/08 00:52:22 INFO netty.NettyClient: connectAllAddresses: Successfully added 2 connections, (2 total connected) 0 failed, 0 failures total.
15/09/08 00:52:22 INFO netty.NettyServer: start: Using Netty without authentication.
15/09/08 00:52:22 INFO handler.RequestDecoder: decode: Server window metrics MBytes/sec received = 0, MBytesReceived = 0.0001, ave received req MBytes = 0.0001, secs waited = 1.44168435E9
15/09/08 00:52:22 INFO worker.BspServiceWorker: loadInputSplits: Using 1 thread(s), originally 1 threads(s) for 1 total splits.
15/09/08 00:52:22 INFO worker.InputSplitsHandler: reserveInputSplit: Reserved input split path /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_vertexInputSplitDir/0, overall roughly 0.0% input splits rese$
15/09/08 00:52:22 INFO worker.InputSplitsCallable: getInputSplit: Reserved /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_vertexInputSplitDir/0 from ZooKeeper and got input split 'hdfs://hdnode01:54310/u$
15/09/08 00:52:22 INFO worker.InputSplitsCallable: loadFromInputSplit: Finished loading /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_vertexInputSplitDir/0 (v=6, e=10)
15/09/08 00:52:22 INFO worker.InputSplitsCallable: call: Loaded 1 input splits in 0.16241108 secs, (v=6, e=10) 36.94329 vertices/sec, 61.572155 edges/sec
15/09/08 00:52:22 ERROR utils.LogStacktraceCallable: Execution of callable failed
java.lang.IllegalStateException: next: IOException
at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:101)
at org.apache.giraph.partition.BasicPartition.addPartitionVertices(BasicPartition.java:99)
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:115)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:466)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:412)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:241)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: ensureRemaining: Only 0 bytes remaining, trying to read 1
at org.apache.giraph.utils.UnsafeReads.ensureRemaining(UnsafeReads.java:77)
at org.apache.giraph.utils.UnsafeArrayReads.readByte(UnsafeArrayReads.java:123)
at org.apache.giraph.utils.UnsafeReads.readLine(UnsafeReads.java:100)
at pruebas.TextAndDoubleComplexWritable.readFields(TextAndDoubleComplexWritable.java:37)
at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:540)
at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:98)
... 11 more
15/09/08 00:52:22 ERROR worker.BspServiceWorker: unregisterHealth: Got failure, unregistering health on /_hadoopBsp/giraph_yarn_application_1441683854213_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHea$
15/09/08 00:52:22 ERROR yarn.GiraphYarnTask: GiraphYarnTask threw a top-level exception, failing task
java.lang.RuntimeException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4bbf48f0
at org.apache.giraph.yarn.GiraphYarnTask.run(GiraphYarnTask.java:104)
at org.apache.giraph.yarn.GiraphYarnTask.main(GiraphYarnTask.java:183)
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4bbf48f0
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:193)
at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:151)
at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:136)
at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:99)
at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:233)
at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:316)
at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:409)
at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:629)
at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:284)
at org.apache.giraph.yarn.GiraphYarnTask.run(GiraphYarnTask.java:92)
... 1 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: next: IOException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:312)
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:185)
... 10 more
Caused by: java.lang.IllegalStateException: next: IOException
at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:101)
at org.apache.giraph.partition.BasicPartition.addPartitionVertices(BasicPartition.java:99)
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:115)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:466)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:412)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:241)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: ensureRemaining: Only 0 bytes remaining, trying to read 1
at org.apache.giraph.utils.UnsafeReads.ensureRemaining(UnsafeReads.java:77)
at org.apache.giraph.utils.UnsafeArrayReads.readByte(UnsafeArrayReads.java:123)
at org.apache.giraph.utils.UnsafeReads.readLine(UnsafeReads.java:100)
at pruebas.TextAndDoubleComplexWritable.readFields(TextAndDoubleComplexWritable.java:37)
at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:540)
at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:98)
... 11 more
My Input format:
package pruebas;
import org.apache.giraph.edge.Edge;
import org.apache.giraph.edge.EdgeFactory;
import org.apache.giraph.io.formats.AdjacencyListTextVertexInputFormat;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
/**
* @author hduser
*
*/
public class IdTextWithComplexValueInputFormat
extends
AdjacencyListTextVertexInputFormat<Text, TextAndDoubleComplexWritable, DoubleWritable> {
@Override
public AdjacencyListTextVertexReader createVertexReader(InputSplit split,
TaskAttemptContext context) {
return new TextComplexValueDoubleAdjacencyListVertexReader();
}
protected class TextComplexValueDoubleAdjacencyListVertexReader extends
AdjacencyListTextVertexReader {
/**
* Constructor with
* {@link AdjacencyListTextVertexInputFormat.LineSanitizer}.
*
* @param lineSanitizer
* the sanitizer to use for reading
*/
public TextComplexValueDoubleAdjacencyListVertexReader() {
super();
}
@Override
public Text decodeId(String s) {
return new Text(s);
}
@Override
public TextAndDoubleComplexWritable decodeValue(String s) {
TextAndDoubleComplexWritable valorComplejo = new TextAndDoubleComplexWritable();
valorComplejo.setVertexData(Double.valueOf(s));
valorComplejo.setIds_vertices_anteriores("");
return valorComplejo;
}
@Override
public Edge<Text, DoubleWritable> decodeEdge(String s1, String s2) {
return EdgeFactory.create(new Text(s1),
new DoubleWritable(Double.valueOf(s2)));
}
}
}
TextAndDoubleComplexWritable:
package pruebas;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.Writable;
public class TextAndDoubleComplexWritable implements Writable {
private String idsVerticesAnteriores;
private double vertexData;
public TextAndDoubleComplexWritable() {
super();
this.idsVerticesAnteriores = "";
}
public TextAndDoubleComplexWritable(double vertexData) {
super();
this.vertexData = vertexData;
}
public TextAndDoubleComplexWritable(String ids_vertices_anteriores,
double vertexData) {
super();
this.idsVerticesAnteriores = ids_vertices_anteriores;
this.vertexData = vertexData;
}
public void write(DataOutput out) throws IOException {
out.writeUTF(idsVerticesAnteriores);
}
public void readFields(DataInput in) throws IOException {
idsVerticesAnteriores = in.readLine();
}
public String getIds_vertices_anteriores() {
return idsVerticesAnteriores;
}
public void setIds_vertices_anteriores(String ids_vertices_anteriores) {
this.idsVerticesAnteriores = ids_vertices_anteriores;
}
public double getVertexData() {
return vertexData;
}
public void setVertexData(double vertexData) {
this.vertexData = vertexData;
}
}
My input file:
Portada 0.0 Sugerencias 1.0
Sugerencias 3.0 Portada 1.0
and i execute it with this command:
$HADOOP_HOME/bin/yarn jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner lectura_de_grafo.BusquedaDeCaminosNavegacionalesWikiquote -vif pruebas.IdTextWithComplexValueInputFormat -vip /user/hduser/input/wiki-graph-chiquito.txt -op /user/hduser/output/caminosNavegacionales -w 2 -yh 250
Any help would be appreciated!
UPDATE: My input file was wrong. Giraph (or my example of it) doesn't handle very well outgoing to non listed vertex.
But the problem still happen. I updated the file data on my original question .
UPDATE 2: The OutputFormat it's not used, and the algorithm for computation is never executed either. I remove both for helping to clarify the question.
Update 3, 19/11/2015:
The problem wasn't in the input format, the input format worked well and read the data entirely.
The problem was in the class TextAndDoubleComplexWritable, i add it to my original question, for a better explanation of the final solution for this (i added an answer too).
解决方案
The problem was in the class TextAndDoubleComplexWritable. I wasn't aware of the importance of methods readFields and write when we are implementing the Writable interface. This are crucial because are the methods that let us send and receive information in giraph. I was writing a empty string in the readFields method, and i should use that method for writing both values of my vertex. I updated both methods in the following way:
public void write(DataOutput out) throws IOException {
out.writeDouble(this.vertexData);
out.writeUTF(this.idsVerticesAnteriores != "" ? "hola"
: this.idsVerticesAnteriores);
}
public void readFields(DataInput in) throws IOException {
this.vertexData = in.readDouble();
this.idsVerticesAnteriores = in.readUTF();
// idsVerticesAnteriores = in.readLine();
}
and this is working, finally!!
这篇关于java.io.IOException：ensureRemaining：只剩下0个字节，试图读取1的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
