[caffe]:检查失败:检查失败:hdf_blobs_[i]->shape(0) == num (200 vs. 6000) [英] [caffe]: check fails: Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000)
问题描述
我将火车和标签数据作为 data.mat.(我有 200 个训练数据和 6000 个特征,标签是 (-1, +1) 保存在 data.mat 中).
我正在尝试在 hdf5 中转换我的数据并使用以下方法运行 Caffe:
加载data.mathdf5write('my_data.h5', '/new_train_x', single( reshape(new_train_x,[200, 6000, 1, 1]) ) );hdf5write('my_data.h5', '/label_train', single( reshape(label_train,[200, 1, 1, 1]) ), 'WriteMode', 'append' );
而我的 layer.prototxt(只是数据层)是:
层{类型:HDF5Data"名称:数据"top: "new_train_x" # 注意:与 HDF5 中的名称相同顶部:label_train"#hdf5_data_param {来源:/path/to/list/file.txt"批量大小:20}包括{阶段:火车}}
但是,我有一个错误:(检查失败:hdf_blobs_[i]->shape(0) == num (200 vs. 6000))
<块引用>I1222 17:02:48.915861 3941 layer_factory.hpp:76] 创建层数据I1222 17:02:48.915871 3941 net.cpp:110] 创建层数据I1222 17:02:48.915877 3941 net.cpp:433] 数据 ->new_train_xI1222 17:02:48.915890 3941 net.cpp:433] 数据 ->标签火车I1222 17:02:48.915900 3941 hdf5_data_layer.cpp:81] 正在加载 HDF5 文件名列表:file.txtI1222 17:02:48.915923 3941 hdf5_data_layer.cpp:95] HDF5 文件数:1F1222 17:02:48.993865 3941 hdf5_data_layer.cpp:55] 检查失败:hdf_blobs_[i]->shape(0) == num (200 vs. 6000)*** 检查失败堆栈跟踪:***@ 0x7fd2e6608ddd google::LogMessage::Fail()@ 0x7fd2e660ac90 google::LogMessage::SendToLog()@ 0x7fd2e66089a2 google::LogMessage::Flush()@ 0x7fd2e660b6ae google::LogMessageFatal::~LogMessageFatal()@ 0x7fd2e69f9eda caffe::HDF5DataLayer<>::LoadHDF5FileData()@ 0x7fd2e69f901f caffe::HDF5DataLayer<>::LayerSetUp()@ 0x7fd2e6a48030 caffe::Net<>::Init()@ 0x7fd2e6a49278 caffe::Net<>::Net()@ 0x7fd2e6a9157a caffe::Solver<>::InitTrainNet()@ 0x7fd2e6a928b1 caffe::Solver<>::Init()@ 0x7fd2e6a92c19 caffe::Solver<>::Solver()@ 0x41222d caffe::GetSolver<>()@ 0x408ed9 火车()@ 0x406741 主@ 0x7fd2e533ca40(未知)@ 0x406f69 _start中止(核心转储)
非常感谢!!!!任何建议将不胜感激!
问题
看来确实是数组中元素的顺序有冲突:matlab从第一维到最后一维排列元素(类似fortran),而caffe和hdf5是从最后一维到第一维存储数组:
假设我们有形状为 n
xc
xh
xw
的 X
那么X
的第二个元素";在 matlab 中是 X[2,1,1,1]
但在 C 中是 X[0,0,0,1]
(基于 1 和基于 0 的索引不根本不会让生活更轻松).
因此,当您在 Matlab 中保存 size=[200, 6000, 1, 1]
数组时,hdf5 和 caffe 实际看到的是 shape=[6000,200] 数组
.
使用 h5ls
命令行工具可以帮助您发现问题.
在matlab中你保存了
<代码>>>hdf5write('my_data.h5', '/new_train_x',单(重塑(new_train_x,[200, 6000, 1, 1]));>>hdf5write('my_data.h5', '/label_train',单(重塑(label_train,[200, 1, 1, 1])),'WriteMode', '追加' );
现在您可以使用 h5ls
(在 Linux 终端中)检查生成的 my_data.h5
:
user@host:~/$ h5ls ./my_data.h5label_train 数据集{200}new_train_x 数据集 {6000, 200}
如您所见,数组是向后"写的.
解决方案
在从 Matlab 导出数据时考虑到这种冲突,您应该 置换
:
加载data.mathdf5write('my_data.h5', '/new_train_x',单(置换(重塑(new_train_x,[200, 6000, 1, 1]),[4:-1:1]));hdf5write('my_data.h5', '/label_train',单(置换(重塑(标签火车,[200,1,1,1]),[4:-1:1])),'WriteMode', '追加' );
使用 h5ls
检查结果 my_data.h5
现在结果为:
user@host:~/$ h5ls ./my_data.h5label_train 数据集 {200, 1, 1, 1}new_train_x 数据集 {200, 6000, 1, 1}
这是您最初的预期.
I have the train and label data as data.mat. (I have 200 training data with 6000 features and labels are (-1, +1) that have saved in data.mat).
I am trying to convert my data in hdf5 and run Caffe using:
load data.mat
hdf5write('my_data.h5', '/new_train_x', single( reshape(new_train_x,[200, 6000, 1, 1]) ) );
hdf5write('my_data.h5', '/label_train', single( reshape(label_train,[200, 1, 1, 1]) ), 'WriteMode', 'append' );
And my layer.prototxt (just data layer) is:
layer {
type: "HDF5Data"
name: "data"
top: "new_train_x" # note: same name as in HDF5
top: "label_train" #
hdf5_data_param {
source: "/path/to/list/file.txt"
batch_size: 20
}
include { phase: TRAIN }
}
but, i have an error: ( Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000))
I1222 17:02:48.915861 3941 layer_factory.hpp:76] Creating layer data I1222 17:02:48.915871 3941 net.cpp:110] Creating Layer data I1222 17:02:48.915877 3941 net.cpp:433] data -> new_train_x I1222 17:02:48.915890 3941 net.cpp:433] data -> label_train I1222 17:02:48.915900 3941 hdf5_data_layer.cpp:81] Loading list of HDF5 filenames from: file.txt I1222 17:02:48.915923 3941 hdf5_data_layer.cpp:95] Number of HDF5 files: 1 F1222 17:02:48.993865 3941 hdf5_data_layer.cpp:55] Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000) *** Check failure stack trace: *** @ 0x7fd2e6608ddd google::LogMessage::Fail() @ 0x7fd2e660ac90 google::LogMessage::SendToLog() @ 0x7fd2e66089a2 google::LogMessage::Flush() @ 0x7fd2e660b6ae google::LogMessageFatal::~LogMessageFatal() @ 0x7fd2e69f9eda caffe::HDF5DataLayer<>::LoadHDF5FileData() @ 0x7fd2e69f901f caffe::HDF5DataLayer<>::LayerSetUp() @ 0x7fd2e6a48030 caffe::Net<>::Init() @ 0x7fd2e6a49278 caffe::Net<>::Net() @ 0x7fd2e6a9157a caffe::Solver<>::InitTrainNet() @ 0x7fd2e6a928b1 caffe::Solver<>::Init() @ 0x7fd2e6a92c19 caffe::Solver<>::Solver() @ 0x41222d caffe::GetSolver<>() @ 0x408ed9 train() @ 0x406741 main @ 0x7fd2e533ca40 (unknown) @ 0x406f69 _start Aborted (core dumped)
Many thanks!!!! Any advice would be appreciated!
The problem
It seems like there is indeed a conflict with the order of elements in arrays: matlab arranges the elements from the first dimension to the last (like fortran), while caffe and hdf5 stores the arrays from last dimension to first:
Suppose we have X
of shape n
xc
xh
xw
then the "second element of X
" is X[2,1,1,1]
in matlab but X[0,0,0,1]
in C (1-based vs 0-based indexing doesn't make life easier at all).
Therefore, when you save an array of size=[200, 6000, 1, 1]
in Matlab, what hdf5 and caffe are actually seeing is as array of shape=[6000,200]
.
Using the h5ls
command line tool can help you spot the problem.
In matlab you saved
>> hdf5write('my_data.h5', '/new_train_x',
single( reshape(new_train_x,[200, 6000, 1, 1]) );
>> hdf5write('my_data.h5', '/label_train',
single( reshape(label_train,[200, 1, 1, 1]) ),
'WriteMode', 'append' );
Now you can inspect the resulting my_data.h5
using h5ls
(in Linux terminal):
user@host:~/$ h5ls ./my_data.h5
label_train Dataset {200}
new_train_x Dataset {6000, 200}
As you can see, the arrays are written "backwards".
Solution
Taking this conflict into account when exporting data from Matlab, you should permute
:
load data.mat
hdf5write('my_data.h5', '/new_train_x',
single( permute(reshape(new_train_x,[200, 6000, 1, 1]),[4:-1:1] ) );
hdf5write('my_data.h5', '/label_train',
single( permute(reshape(label_train,[200, 1, 1, 1]), [4:-1:1] ) ),
'WriteMode', 'append' );
Inspect the resulting my_data.h5
using h5ls
now results with:
user@host:~/$ h5ls ./my_data.h5
label_train Dataset {200, 1, 1, 1}
new_train_x Dataset {200, 6000, 1, 1}
Which is what you expected in the first place.
这篇关于[caffe]:检查失败:检查失败:hdf_blobs_[i]->shape(0) == num (200 vs. 6000)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!