将自定义R生成器函数与fit_generator(Keras，R)结合使用 [英] Using a custom R generator function with fit_generator (Keras, R)
问题描述
我想训练一个卷积网络，以解决图像数据上的多类，多标签问题.由于数据的性质，并且出于某种原因，我将为您节省时间，最好是我可以使用自定义的R生成器函数来馈送给fit_generator命令，而不是其内置的image_data_generator和flow_images_from_directory命令(我成功地开始使用它，只是没有解决这个特定问题).
此处( https://www.rdocumentation.org /packages/keras/versions/2.2.0/topics/fit_generator )说，我可以做到这一点，而无需给出任何示例.因此，我尝试了以下方法.这是我要做的事情的精简示例(此代码完全是自包含的):
library(keras)
library(reticulate) #for py_iterator function
play.network = keras_model_sequential() %>%
layer_dense(units = 10, activation = "relu", input_shape = c(10)) %>%
layer_dense(units = 1, activation = "relu")
play.network %>% compile(
optimizer = "rmsprop",
loss = "mse"
)
mikes.custom.generator.function = function() #generates a 2-list of a random 1 x 10 array, and a scalar
{
new.func = function()
{
arr = array(dim = c(1,10))
arr[,] = sample(1:10, 10, replace = TRUE)/10
return(list(arr,runif(1)))
}
}
mikes.custom.iterator = py_iterator(mikes.custom.generator.function()) #creates a python iterator object
generator_next(mikes.custom.iterator) #correctly returns a 2-member list consisting of a 1 x 10 array, and a scalar
generator_next(mikes.custom.iterator)[[1]] #a 1 x 10 array
generator_next(mikes.custom.iterator)[[2]] #a scalar
#try to fit with "fit_generator":
play.network %>% fit_generator( #FREEZES.
mikes.custom.iterator,
steps_per_epoch = 1,
epochs = 1
)
在训练时，东西冻结了，没有给我任何错误消息或其他任何东西.对于原始问题，我还使用了自定义图像数据生成器进行了尝试，结果相同.
请注意，如果我只是使用fit并手动输入训练数据，则此网络的训练就很好:
play.network %>% fit(generator_next(mikes.custom.iterator)[[1]],generator_next(mikes.custom.iterator)[[2]], epochs = 1, batch_size = 1)
#trains just fine
我想我知道问题所在，但是我不知道解决方法.如果您将其作为我的自定义迭代器的类，它将给出
class(mikes.custom.iterator)
[1] "python.builtin.iterator" "rpytools.generator.RGenerator" "python.builtin.object"
而如果我使用内置的image_data_generator和flow_images_from_directory命令构建一个迭代器，它将给出
train_datagen <- image_data_generator(rescale = 1/255)
class(train_datagen)
[1] "keras.preprocessing.image.ImageDataGenerator" "keras_preprocessing.image.ImageDataGenerator" "python.builtin.object"
train_generator <- flow_images_from_directory(
train_dir,
train_datagen,
....
)
class(train_generator)
[1] "python.builtin.iterator" "keras_preprocessing.image.DirectoryIterator" "keras_preprocessing.image.Iterator" "tensorflow.python.keras.utils.data_utils.Sequence" "python.builtin.object"
所以我的猜测是train_datagen和/或train_generator具有mikes.custom.iterator没有的属性，并且fit_generator试图使用除基本generator_next以外的功能调用mikes.custom.iterator从理论上讲，它应该真正需要的).但是，即使在网上搜索了两个小时之后，我也不知道它们可能是什么，或者如何正确构建mikes.custom.iterator.
帮助任何人吗?
sampling_generator <- function(X_data, Y_data, batch_size) {
function() {
rows <- sample(1:nrow(X_data), batch_size, replace = TRUE)
list(X_data[rows,], Y_data[rows,])
}
}
model %>%
fit_generator(sampling_generator(X_train, Y_train, batch_size = 128),
steps_per_epoch = nrow(X_train) / 128, epochs = 10)
我在R keras常见问题解答中找到了这个答案，
I'd like to train a convolutional network to solve a multi-class, multi-label problem on image data. Due to the nature of the data, and for reasons I'll spare you, it would be best if I could use a custom R generator function to feed to the fit_generator command, instead of its built-in image_data_generator and flow_images_from_directory commands (which I was successfully able to get working, just not for this particular problem).
Here (https://www.rdocumentation.org/packages/keras/versions/2.2.0/topics/fit_generator) it says that I can do just that, without giving any examples. So I tried the following. Here is an extremely stripped down example of what I'm trying to do (this code is entirely self contained):
library(keras)
library(reticulate) #for py_iterator function
play.network = keras_model_sequential() %>%
layer_dense(units = 10, activation = "relu", input_shape = c(10)) %>%
layer_dense(units = 1, activation = "relu")
play.network %>% compile(
optimizer = "rmsprop",
loss = "mse"
)
mikes.custom.generator.function = function() #generates a 2-list of a random 1 x 10 array, and a scalar
{
new.func = function()
{
arr = array(dim = c(1,10))
arr[,] = sample(1:10, 10, replace = TRUE)/10
return(list(arr,runif(1)))
}
}
mikes.custom.iterator = py_iterator(mikes.custom.generator.function()) #creates a python iterator object
generator_next(mikes.custom.iterator) #correctly returns a 2-member list consisting of a 1 x 10 array, and a scalar
generator_next(mikes.custom.iterator)[[1]] #a 1 x 10 array
generator_next(mikes.custom.iterator)[[2]] #a scalar
#try to fit with "fit_generator":
play.network %>% fit_generator( #FREEZES.
mikes.custom.iterator,
steps_per_epoch = 1,
epochs = 1
)
The thing freezes at training time, without giving me an error message or anything. I also tried it with a custom image data generator for my original problem, same result.
Note that this network trains just fine if I just use fit and input the training data manually:
play.network %>% fit(generator_next(mikes.custom.iterator)[[1]],generator_next(mikes.custom.iterator)[[2]], epochs = 1, batch_size = 1)
#trains just fine
I think I know the problem, but I don't know the solution. If you ask it for the class of my custom iterator, it gives
class(mikes.custom.iterator)
[1] "python.builtin.iterator" "rpytools.generator.RGenerator" "python.builtin.object"
whereas if I build an iterator using the builtin image_data_generator and flow_images_from_directory commands, it gives
train_datagen <- image_data_generator(rescale = 1/255)
class(train_datagen)
[1] "keras.preprocessing.image.ImageDataGenerator" "keras_preprocessing.image.ImageDataGenerator" "python.builtin.object"
train_generator <- flow_images_from_directory(
train_dir,
train_datagen,
....
)
class(train_generator)
[1] "python.builtin.iterator" "keras_preprocessing.image.DirectoryIterator" "keras_preprocessing.image.Iterator" "tensorflow.python.keras.utils.data_utils.Sequence" "python.builtin.object"
So my guess is that train_datagen and/or train_generator have attributes that mikes.custom.iterator does not, and fit_generator is trying to call upon mikes.custom.iterator using functions other than the basic generator_next (which is in theory all it should really need). But I don't know what they may be, or how to build mikes.custom.iterator correctly, even after searching for two hours online.
Help anyone?
sampling_generator <- function(X_data, Y_data, batch_size) {
function() {
rows <- sample(1:nrow(X_data), batch_size, replace = TRUE)
list(X_data[rows,], Y_data[rows,])
}
}
model %>%
fit_generator(sampling_generator(X_train, Y_train, batch_size = 128),
steps_per_epoch = nrow(X_train) / 128, epochs = 10)
I found this answer in R keras FAQs which seems to work
这篇关于将自定义R生成器函数与fit_generator(Keras，R)结合使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
