检查失败:如何在深层使用hdf5数据层? [英] Check fail: how to use hdf5 data layer in deep layer?
问题描述
我的火车和标签数据为data.mat. (我有200个训练数据和6000个功能，并且标签(-1，+1)已保存在data.mat中.)
I have the train and label data as data.mat. (I have 200 training data with 6000 features and labels are (-1, +1) that have saved in data.mat).
我正在尝试在hdf5中转换数据(训练和测试)并使用以下命令运行Caffe:
I am trying to convert my data (train and test) in hdf5 and run Caffe using:
load input.mat
hdf5write('my_data.h5', '/new_train_x', single( permute(reshape(new_train_x,[200, 6000, 1, 1]),[4:-1:1] ) ));
hdf5write('my_data.h5', '/label_train', single( permute(reshape(label_train,[200, 1, 1, 1]), [4:-1:1] ) ) , 'WriteMode', 'append' );
hdf5write('my_data_test.h5', '/test_x', single( permute(reshape(test_x,[77, 6000, 1, 1]),[4:-1:1] ) ));
hdf5write('my_data_test.h5', '/label_test', single( permute(reshape(label_test,[77, 1, 1, 1]), [4:-1:1] ) ) , 'WriteMode', 'append' );
(有关在Matlab中将Mat文件转换为hdf5的信息，请参见此线程).
(See this thread regarding converting mat-files to hdf5 in Matlab).
我的train_val.prototxt是:
layer {
type: "HDF5Data"
name: "data"
top: "new_train_x" # note: same name as in HDF5
top: "label_train" #
hdf5_data_param {
source: "file.txt"
batch_size: 20
}
include { phase: TRAIN }
}
layer {
type: "HDF5Data"
name: "data"
top: "test_x" # note: same name as in HDF5
top: "label_test" #
hdf5_data_param {
source: "file_test.txt"
batch_size: 20
}
include { phase:TEST }
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "new_train_x"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 30
weight_filler {
type: "gaussian" # initialize the filters from a Gaussian
std: 0.01
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "tanh1"
type: "TanH"
bottom: "ip1"
top: "tanh1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "tanh1"
top: "ip2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 1
weight_filler {
type: "gaussian" # initialize the filters from a Gaussian
std: 0.01
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "loss"
type: "TanH"
bottom: "ip2"
bottom: "label_train"
top: "loss"
}
但是我有一个问题.看来，它无法读取我的输入数据.
But I have a problem. It seems, it cannot read my input data.
I1227 10:27:21.880826 7186 layer_factory.hpp:76] Creating layer data
I1227 10:27:21.880851 7186 net.cpp:110] Creating Layer data
I1227 10:27:21.880866 7186 net.cpp:433] data -> new_train_x
I1227 10:27:21.880893 7186 net.cpp:433] data -> label_train
I1227 10:27:21.880915 7186 hdf5_data_layer.cpp:81] Loading list of HDF5 filenames from: file.txt
I1227 10:27:21.880965 7186 hdf5_data_layer.cpp:95] Number of HDF5 files: 1
I1227 10:27:21.962596 7186 net.cpp:155] Setting up data
I1227 10:27:21.962702 7186 net.cpp:163] Top shape: 20 6000 1 1 (120000)
I1227 10:27:21.962738 7186 net.cpp:163] Top shape: 20 1 1 1 (20)
I1227 10:27:21.962772 7186 layer_factory.hpp:76] Creating layer ip1
I1227 10:27:21.962838 7186 net.cpp:110] Creating Layer ip1
I1227 10:27:21.962873 7186 net.cpp:477] ip1 <- new_train_x
I1227 10:27:21.962918 7186 net.cpp:433] ip1 -> ip1
I1227 10:27:21.979375 7186 net.cpp:155] Setting up ip1
I1227 10:27:21.979434 7186 net.cpp:163] Top shape: 20 30 (600)
I1227 10:27:21.979478 7186 layer_factory.hpp:76] Creating layer tanh1
I1227 10:27:21.979529 7186 net.cpp:110] Creating Layer tanh1
I1227 10:27:21.979557 7186 net.cpp:477] tanh1 <- ip1
I1227 10:27:21.979583 7186 net.cpp:433] tanh1 -> tanh1
I1227 10:27:21.979620 7186 net.cpp:155] Setting up tanh1
I1227 10:27:21.979650 7186 net.cpp:163] Top shape: 20 30 (600)
I1227 10:27:21.979670 7186 layer_factory.hpp:76] Creating layer ip2
I1227 10:27:21.979696 7186 net.cpp:110] Creating Layer ip2
I1227 10:27:21.979720 7186 net.cpp:477] ip2 <- tanh1
I1227 10:27:21.979746 7186 net.cpp:433] ip2 -> ip2
I1227 10:27:21.979796 7186 net.cpp:155] Setting up ip2
I1227 10:27:21.979825 7186 net.cpp:163] Top shape: 20 1 (20)
I1227 10:27:21.979854 7186 layer_factory.hpp:76] Creating layer loss
I1227 10:27:21.979881 7186 net.cpp:110] Creating Layer loss
I1227 10:27:21.979909 7186 net.cpp:477] loss <- ip2
I1227 10:27:21.979931 7186 net.cpp:477] loss <- label_train
I1227 10:27:21.979962 7186 net.cpp:433] loss -> loss
F1227 10:27:21.980006 7186 layer.hpp:374] Check failed: ExactNumBottomBlobs() == bottom.size() (1 vs. 2) TanH Layer takes 1 bottom blob(s) as input.
*** Check failure stack trace: ***
@ 0x7f44cbc68ea4 (unknown)
@ 0x7f44cbc68deb (unknown)
@ 0x7f44cbc687bf (unknown)
@ 0x7f44cbc6ba35 (unknown)
@ 0x7f44cbfd0ba8 caffe::Layer<>::CheckBlobCounts()
@ 0x7f44cbfed9da caffe::Net<>::Init()
@ 0x7f44cbfef108 caffe::Net<>::Net()
@ 0x7f44cc03f71a caffe::Solver<>::InitTrainNet()
@ 0x7f44cc040a51 caffe::Solver<>::Init()
@ 0x7f44cc040db9 caffe::Solver<>::Solver()
@ 0x41222d caffe::GetSolver<>()
@ 0x408ed9 train()
@ 0x406741 main
@ 0x7f44ca997a40 (unknown)
@ 0x406f69 _start
@ (nil) (unknown)
Aborted (core dumped)
现在，如果我这样更改损耗层:
Now, if i change loss layer like this:
layer {
name: "loss"
type: "TanH"
bottom: "ip2"
top: "loss"
}
我有这个问题:
F1227 10:53:17.884419 9102 insert_splits.cpp:35] Unknown bottom blob 'new_train_x' (layer 'ip1', bottom index 0)
*** Check failure stack trace: ***
@ 0x7f502ab5dea4 (unknown)
@ 0x7f502ab5ddeb (unknown)
@ 0x7f502ab5d7bf (unknown)
@ 0x7f502ab60a35 (unknown)
@ 0x7f502af1d75b caffe::InsertSplits()
@ 0x7f502aee19e9 caffe::Net<>::Init()
@ 0x7f502aee4108 caffe::Net<>::Net()
@ 0x7f502af35172 caffe::Solver<>::InitTestNets()
@ 0x7f502af35abd caffe::Solver<>::Init()
@ 0x7f502af35db9 caffe::Solver<>::Solver()
@ 0x41222d caffe::GetSolver<>()
@ 0x408ed9 train()
@ 0x406741 main
@ 0x7f502988ca40 (unknown)
@ 0x406f69 _start
@ (nil) (unknown)
Aborted (core dumped)
非常感谢！！！！任何建议将不胜感激！
推荐答案
您的数据层仅针对phase: TRAIN定义.我相信在caffe尝试构建测试时间网络(即phase: TEST网络)时会发生错误).
您应该有一个包含测试数据的附加层:
Your data layer is only defined for phase: TRAIN I believe the error occurs when caffe attempts to construct the test-time net (i.e., the phase: TEST net).
You should have an additional layer with test data:
layer {
type: "HDF5Data"
name: "data"
top: "new_train_x" # note: same name as in HDF5
top: "label_train" #
hdf5_data_param {
source: "test_file.txt"
batch_size: 20
}
include { phase: TEST } # do not forget TEST phase
}
顺便说一句，如果您不想在训练期间测试网络，则可以关闭此选项.有关更多信息，请参见此线程.
BTW, if you do not want to test your net during training, you can switch this option off. See this thread for more information.
更新:
原谅我直言不讳，但你却一团糟.
Update:
Forgive me for being blunt, but you are making quite a mess.
-
"TanH"不是损失层-它是神经元/激活层.它用作线性层(转换/内积)的非线性.这样，它接受单个输入(底部Blob)并输出单个Blob(顶部).
损失层计算标量损失值，通常需要两个输入:要与之比较的预测和地面真实情况. - 您确实更改了网络，并且还在
TEST阶段添加了"HDF5Data"层，但是该层输出top: "test_x"，网络中没有层期望bottom: "test_x"，而只有层期望... "label_text"也是如此.
"TanH"is not a loss layer - it's a neuron/activation layer. It serves as a non-linarity applied to a linear layer (conv/inner-product). As such, it accepts a single input (bottom blob) and outputs a single blob (top).
A loss layer computes a scalar loss value and usually requires two inputs: prediction and ground truth to compare to.- You did change your net and added a
"HDF5Data"layer for theTESTphase as well, but this layer outputs atop: "test_x", no layer in your net expects abottom: "test_x"you only have layers expecting"new_train_x"... same goes for"label_text".
我建议您使用更通用的名称(例如，x和label)重新编写hdf5文件，以进行 训练和测试.只需使用不同的文件名来区分它们.这样，您的网络就可以在两个阶段同时使用"x"和"label"，并且仅根据阶段加载适当的数据集.
I suggest you re-write your hdf5 files with more generic names (e.g., x and label) for both train and test. Just use different file names to distinguish between them. This way your net works with "x" and "label" in both phases and only loads the appropriate dataset according to phase.
这篇关于检查失败:如何在深层使用hdf5数据层?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
