自定义 hadoop 键和值:如何编写 CompareTo() 方法 [英] Custom hadoop key and value : How to write CompareTo() Method
问题描述
我需要从映射器发出一个二维双精度数组作为键和值.在 Stack Overflow 中发布了一些问题，但没有得到解答.
I need to emit a 2D double array as key and value from mapper. There are questions posted in Stack Overflow, but they are not answered.
我在给定的数据集中做一些矩阵乘法，之后我需要发出 A*Atrns 的值，这将是一个矩阵作为键和 Atrans*D 这也将是一个矩阵作为值.那么如何从映射器中发出这些矩阵.并且 value 应该与 key 本身相对应.
I am doing some of the matrix multiplication in a given dataset, and after that I need to emit the value of A*Atrns which will be a matrix as key and Atrans*D which will also be a matrix as value. So how to emit these matrices from mapper. And the value should be corresponding to the key itself.
ie key -----> A*Atrans--------->after multiplication the result will be a 2D array which is declared as double (matrix) lets say the result be Matrix "Ekey"(double[][] Ekey)
value ------> Atrans*D ---------> after multiplication the result will be Matrix "Eval" (double[][] Eval).
After that I need to emit these matrix to reducer for further calculations.
So in mapper:
context.write(Ekey,Eval);
Reducer:
I need to do further calculations with these Ekey and Eval.
我写了我的课:
更新
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
private double[][] values;
public MatrixWritable() {
// TODO Auto-generated constructor stub
setValue(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void setValue(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return values;
}
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(value.length); // write values
for (int i = 0; i < value.length; i++) {
out.writeInt(value[i].length);
}
for (int i = 0; i < value.length; i++) {
for (int j = 0; j < value[i].length; j++) {
out.writeDouble(value[i][j]);
}
}
}
@Override
public void readFields(DataInput in) throws IOException {
value = new double[in.readInt()][];
for (int i = 0; i < value.length; i++) {
value[i] = new double[in.readInt()];
}
values = new double[value.length][value[0].length];
for(int i=0;i<value.length ; i++){
for(int j= 0 ; j< value[0].length;j++){
values[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + Arrays.hashCode(value);
return result;
}
/* (non-Javadoc)
* @see java.lang.Object#equals(java.lang.Object)
*/
@Override
public boolean equals(Object obj) {
if (this == obj) {
return true;
}
if (obj == null) {
return false;
}
if (!(obj instanceof MatrixWritable)) {
return false;
}
MatrixWritable other = (MatrixWritable) obj;
if (!Arrays.deepEquals(value, other.value)) {
return false;
}
return true;
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
public String toString() { String separator = "|";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < values.length; i++) {
// iterate over the second dimension
for(int j = 0; j < values[i].length; j++){
result.append(values[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append(",");
}
return result.toString();
}
}
我能够从映射器发出一个值作为矩阵
I am able to emit a value as matrix from mapper
context.write(...,new MatrixWritable(AAtrans));
如何从映射器发出矩阵 AtransD 作为键?
How to emit matrix AtransD as key from mapper?
为此，我需要编写 compareto() 方法，对吗?
For that I need to write compareto() method, right?
该方法中应包含哪些内容?
What should be included in that method?
推荐答案
首先，要实现自定义键，您必须实现 WritableComparable.要实现自定义值，您必须实现 Writable.在许多情况下，由于能够方便地交换键和值，因此大多数人将所有自定义类型编写为 WritableComparable.
First, to implement a custom key you must implement WritableComparable. To implement a custom value you must implement Writable. In many cases since it is handy to be able to swap keys and values most people write all custom types as WritableComparable.
这里是 Hadoop: The Definitive Guide 部分的链接，该部分涵盖了编写 WritableComparable.编写自定义可写文件
Here is a link to the section of Hadoop: The Definitive Guide that covers writing a WritableComparable. Writing A Custom Writable
写出数组的诀窍在于，在读取端，您需要知道要读取多少个元素.所以基本模式是……
The trick with writing out an array is that on the read side you need to know how many elements to read. So the basic patter is...
On write:
write the number of elements
write each element
On read:
read the number of elements (n)
create an array of the appropriate size
read 0 - (n-1) elements and populate array
更新
您应该在默认构造函数中将数组实例化为空，以防止以后出现 NullPointerException.
You should instantiate your array as empty in the default constructor to prevent a NullPointerException later.
您的实现的问题在于它假设每个内部数组的长度相同.如果这是真的，您不需要多次计算列长度.如果为false，则需要在写入行的值之前写入每一行的长度.
The problem with your implementation is that it assumes that each inner array is of the same length. If that is true, you don't need to calculate the column length more than once. If false, you need to write the length of each row before writing the values of the row.
我会建议这样的事情:
context.write(row); // as calculated above
for (int i=0; i<row; i++){
double[] rowVals = array[row];
context.write(rowVals.length);
for (int j=0; j<rowVals.length; j++)
context.write(rowVals[j]);
}
这篇关于自定义 hadoop 键和值:如何编写 CompareTo() 方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
