tf.transform:向Keras模型添加预处理? [英] tf.transform: add preprocessing to Keras model?
问题描述
我有一个keras
模型,用于使用tensorflow
后端进行文本分类.当前,它假定输入是一个numpy的整数数组.
I have a keras
model for text classification using a tensorflow
backend. It currently assumes the input is a numpy array of integers.
我想对此进行修改,以便可以对原始文本进行训练和预测.根据我的收集,这涉及到使用tf.transform
将字符串的张量转换为整数的张量.
I'd like to modify this so that I can train and predict on raw text. From what I've gathered, this involves using tf.transform
to convert a tensor of strings into a tensor of integers.
我已经使用tf.transform
完成了此操作,但是现在不确定如何将该预处理步骤添加到模型中作为第一层/第一步.需要明确的是,我的输入数据如下:
I've done this using tf.transform
but now am unsure how to add this preprocessing step to my model as the very first layer / step. To be clear, my input data looks like this:
[{"review":"movie is great},
{"review":"awful film"}]
[{"review":"movie is great},
{"review":"awful film"}]
,输出为:
[{"review_out": array([-1, -1, 1, 0, 2])},
{"review_out": array([-1, -1, -1, 3, 4])]
[{"review_out": array([-1, -1, 1, 0, 2])},
{"review_out": array([-1, -1, -1, 3, 4])]
执行此操作的函数称为preprocess
.因此,我只想将运行preprocess
作为DAG的第一步.
The function that does this is called preprocess
. So I just want to include running preprocess
as the first step in my DAG.
我该怎么办?
作为参考,这很重要,因为我想在ML Engine上进行实时预测.
For reference, this is important because I want to do live prediction on ML Engine.
推荐答案
如果使用tf.data.Dataset()
,则存在tf.data.Dataset().map(map_func)
函数,该函数允许您将map_func
应用于数据集的所有元素.这可以用来添加您的预处理步骤.
If you use a tf.data.Dataset()
, there exists a tf.data.Dataset().map(map_func)
function that allows you to apply map_func
to all elements of your dataset. This could be used to add your preprocessing step.
https://www.tensorflow.org/api_docs/python/tf /data/Dataset
例如:
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.map(preprocess)
model.fit(dataset, ....)
通过numpy数组使用tf.data.Dataset()
还有其他优点.
There are other advantages to using tf.data.Dataset()
over a numpy array as well.
这篇关于tf.transform:向Keras模型添加预处理?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!