tf.transform:向Keras模型添加预处理? [英] tf.transform: add preprocessing to Keras model?
问题描述
我有一个keras模型，用于使用tensorflow后端进行文本分类.当前，它假定输入是一个numpy的整数数组.
I have a keras model for text classification using a tensorflow backend. It currently assumes the input is a numpy array of integers.
我想对此进行修改，以便可以对原始文本进行训练和预测.根据我的收集，这涉及到使用tf.transform将字符串的张量转换为整数的张量.
I'd like to modify this so that I can train and predict on raw text. From what I've gathered, this involves using tf.transform to convert a tensor of strings into a tensor of integers.
我已经使用tf.transform完成了此操作，但是现在不确定如何将该预处理步骤添加到模型中作为第一层/第一步.需要明确的是，我的输入数据如下:
I've done this using tf.transform but now am unsure how to add this preprocessing step to my model as the very first layer / step. To be clear, my input data looks like this:
[{"review":"movie is great},
{"review":"awful film"}]
[{"review":"movie is great},
{"review":"awful film"}]
，输出为:
[{"review_out": array([-1, -1, 1, 0, 2])},
{"review_out": array([-1, -1, -1, 3, 4])]
[{"review_out": array([-1, -1, 1, 0, 2])},
{"review_out": array([-1, -1, -1, 3, 4])]
执行此操作的函数称为preprocess.因此，我只想将运行preprocess作为DAG的第一步.
The function that does this is called preprocess. So I just want to include running preprocess as the first step in my DAG.
我该怎么办?
作为参考，这很重要，因为我想在ML Engine上进行实时预测.
For reference, this is important because I want to do live prediction on ML Engine.
推荐答案
如果使用tf.data.Dataset()，则存在tf.data.Dataset().map(map_func)函数，该函数允许您将map_func应用于数据集的所有元素.这可以用来添加您的预处理步骤.
If you use a tf.data.Dataset(), there exists a tf.data.Dataset().map(map_func) function that allows you to apply map_func to all elements of your dataset. This could be used to add your preprocessing step.
https://www.tensorflow.org/api_docs/python/tf /data/Dataset
例如:
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.map(preprocess)
model.fit(dataset, ....)
通过numpy数组使用tf.data.Dataset()还有其他优点.
There are other advantages to using tf.data.Dataset() over a numpy array as well.
这篇关于tf.transform:向Keras模型添加预处理?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
