在keras中，如何适应具有不同类型的多个输入数据 [英] In keras, how to fit multiple input data with different type

问题描述

我有3000张(320 * 320)形状的图像，它们的拍摄时间以及标签.现在，我想使用这两种类型的数据(图像和时间)来预测其标签.主要代码如下:

I have 3000 images in (320*320) shape, their shooting time and also their labels. Now I want to use these two type of data(images and time) to predict their label. The main code as follows:

num_classes=10
image_out=GlobalMaxPooling2D()(x)
x=keras.layers.advanced_activations.LeakyReLU(alpha=0.3)(x)

#add time
time_input = Input(shape=(1,), name='time_input')
x = keras.layers.concatenate([image_out, time_input],axis=-1)
x=Dense(num_classes, activation='sigmoid')(x)
predictions = Dense(num_classes,activation='softmax',name='main_output')(x)

我知道我可以使用以下代码来训练模型:

I know I could use following code to train the model:

model.fit([main_input, additional_data], [labels],
      epochs=50, batch_size=32)

但是，由于我使用了数据增强，并且还使用了时间特征来训练模型，因此如何同时使用fit_generator?我可以这样定义一个新的生成器吗?

However, how to use both fit_generator since I use data augumention and also use time features to train the model? May I have to define a new generator like this?

def generator_for2img(X1, X2, Y,batch_size):
    datagen = ImageDataGenerator(featurewise_center=False,  # set input mean to 0 over the dataset
                                 samplewise_center=False,  # set each sample mean to 0
                                 featurewise_std_normalization=False,  # divide inputs by std of the dataset
                                 samplewise_std_normalization=False,  # divide each input by its std
                                 zca_whitening=False,  # apply ZCA whitening
                                 rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
                                 width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
                                 height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
                                 horizontal_flip=True,  # randomly flip images
                                 vertical_flip=False)  # randomly flip image
    time_generator=Generator()
    datagen.fit(X1)
    genX1 = datagen.flow(X1,Y,batch_size=batch_size)
    genX2 = time_generator.flow(X2,batch_size=batch_size)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i ], X1i[1]

model.fit_generator(generator_for2img(X_train_cv, X_train_time, y_train_cv,batch_size),
                                    epochs=epochs,
                                    validation_data=([X_valid,X_valid_time],y_valid),
                                    workers=4,

推荐答案

我最近遇到了一个非常相似的问题.我正在使用CNN对图像进行分类，但是，我还想在分类过程中包括图像的全局特征.我预先计算了这些功能，然后想通过后期融合(即在softmax层合并)将它们包含在网络中.我发现除了使用Keras的ImageDataGenerator之外，我还需要编写3个生成器(实际上，可以减少到2个).以下是包含这些生成器的函数.输入x_train和x_test是图像输入，而x_train_feat和x_test_feat代表全局(正常")特征.

I recently faced a very similar problem. I was having a CNN classify images, however, I also wanted to include global features of the image in the classification process. I precomputed these features and then wanted to include them in the network through late fusing (i.e. merging at the softmax layer). I found that I needed to write 3 generators (in fact, this can be reduced to 2) in addition to using Keras' ImageDataGenerator. The following is the function that includes these generators. The inputs x_train and x_test are the image inputs while x_train_feat and x_test_feat represent the global ("normal") features.

def train_model(model, x_train, x_train_feat, y_train, x_test, x_test_feat, y_test, train_batch_size, test_batch_size, epochs, model_dir, model_name, num_classes, patience=5, monitor='val_acc'):

    '''
    Training function
    '''

    train_datagen = ImageDataGenerator(
        featurewise_center=False, # also use in test gen if activated AND fit test_gen on train data
        samplewise_center=False,
        featurewise_std_normalization=False, # also use in test gen if activated AND fit test_gen on train data
        samplewise_std_normalization=False,
        zca_whitening=False,
        zca_epsilon=0,
        rotation_range=0.05,
        width_shift_range=0.05,
        height_shift_range=0.05,
        channel_shift_range=0,
        fill_mode='nearest',
        cval=0,
        vertical_flip=False,
        rescale=1./255,
        shear_range=0.,
        zoom_range=0.,
        horizontal_flip=False)

    train_datagen.fit(x_train)

    test_datagen = ImageDataGenerator(
        rescale=1./255,
        featurewise_std_normalization=False,
        featurewise_center=False)

    #test_datagen.fit(x_train)

    train_generator = train_datagen.flow(
        x_train,
        y_train,
        batch_size=train_batch_size,
        shuffle=False)

    def train_feat_gen(x_train_feat, train_batch_size):
        while True:
            for batch in range(len(x_train_feat) // train_batch_size + 1):
                if batch > max(range(len(x_train_feat) // train_batch_size)):
                    yield x_train_feat[batch*train_batch_size:]
                else:
                    yield x_train_feat[batch*train_batch_size:(1+batch)*train_batch_size]

    def val_feat_gen(x_val_feat, test_batch_size):
        while True:
            for batch in range(len(x_val_feat) // test_batch_size + 1):
                if batch > max(range(len(x_val_feat) // test_batch_size)):
                    yield x_val_feat[batch*test_batch_size:]
                else:
                    yield x_val_feat[batch*test_batch_size:(1+batch)*test_batch_size]

    def merge_generator(gen1, gen2):
        while True:
            X1 = gen1.__next__()
            X2 = gen2.__next__()
            yield [X1[0], X2], X1[1]

    validation_generator = test_datagen.flow(
        x_test,
        y_test,
        batch_size=test_batch_size)

    final_train_gen = merge_generator(train_generator, train_feat_gen(x_train_feat, train_batch_size))
    final_val_gen = merge_generator(validation_generator, val_feat_gen(x_val_feat, test_batch_size))

    callbacks = [ModelCheckpoint(MODEL_DIR+model_name+'.h5',
                                 monitor=monitor,
                                save_best_only=True),
                EarlyStopping(monitor=monitor, patience=patience),
                TensorBoard(LOG_DIR+model_name+'_'+str(time())),
                ReduceLROnPlateau(monitor='val_loss', factor=0.75, patience=2)]

    model.fit_generator(
        final_train_gen,
        steps_per_epoch=len(x_train) // train_batch_size,
        epochs=epochs,
        validation_data=final_val_gen,
        validation_steps=len(y_test) // test_batch_size,
        callbacks=callbacks,)

请注意生成器train_feat_gen，val_feat_gen和merge_generator.前两个负责训练和验证数据的批处理，而后者则包装两个生成器(train_feat_gen和train_datagen； val_feat_gen和test_datagen)，以生成一个提供两个不同类型输入的生成器.当然，在这里同时拥有train_feat_gen和val_feat_gen是多余的，因为它们本质上是相同的，但是用不同的参数实例化.但是，我认为这种情况更加明显.希望这会有所帮助.

Note the generators train_feat_gen, val_feat_gen and merge_generator. The first two take care of the batching of the training and validation data while the latter wraps two generators (train_feat_gen and train_datagen; val_feat_gen and test_datagen) to produce one generator that provides two inputs of different types. Of course, having both train_feat_gen and val_feat_gen is redundant here, as they essentially do the same but are instantiated with different parameters. However, I think this case makes it more obvious. Hope this helps.

这篇关于在keras中，如何适应具有不同类型的多个输入数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！