执行 TFMA 时出现 TFX 管道错误:AttributeError: 'NoneType' 对象没有属性 'ToBatchTensors' [英] TFX Pipeline Error While Executing TFMA: AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'

问题描述

基本上我只重用了 iris utils 和 iris 管道更改服务输入:

Basically I only reused code from iris utils and iris pipeline with minor change on serving input:

def _get_serve_tf_examples_fn(model, tf_transform_output):   
    model.tft_layer = tf_transform_output.transform_features_layer()

    feature_spec = tf_transform_output.raw_feature_spec()
    print(feature_spec)
    feature_spec.pop(_LABEL_KEY)

    @tf.function
    def serve_tf_examples_fn(*args):
        parsed_features = {}
        for arg in args:
            parsed_features[arg.name.split(":")[0]] = arg
        print(parsed_features)

        transformed_features = model.tft_layer(parsed_features)

        return model(transformed_features)


def run_fn(fn_args: TrainerFnArgs):
    ...

    feature_spec = tf_transform_output.raw_feature_spec()
    feature_spec.pop(_LABEL_KEY)

    inputs = [tf.TensorSpec(
                    shape=[None, 1],
                    dtype=feature_spec[f].dtype,
                    name=f) for f in feature_spec]

    signatures = {
        'serving_default':
            _get_serve_tf_examples_fn(model, tf_transform_output).get_concrete_function(*inputs),
    }
    model.save(fn_args.serving_model_dir, save_format='tf', signatures=signatures)

来自 iris 代码的 get_concrete_function() 原始输入只是一个带有 dtype 字符串的 TensorSpec.我已经尝试使用精确输入为模型提供服务，但是当我测试 REST API 时出现解析错误.所以我尝试更改服务输入，以便它可以像这样接收 JSON 输入:

the get_concrete_function() original input from iris codes is only a TensorSpec with dtype string. I already tried serving the model using the exact input but when I test the REST API I got a parsing error. So I tried to change the serving input so it can receive JSON input like this:

{"instances": [{"feat1": 90, "feat2": 23.8, "feat3": 12}]}

当我运行管道时，训练成功，但在运行评估器组件时出现错误.这是最新的日志:

when I run the pipeline, the training was successful but then the error occurred when running the evaluator component. this is the latest logs:

INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/67/serving_model_dir as candidate model.
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/14/serving_model_dir as baseline model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:We decided to produce LargeList and LargeBinary types.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0e44560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77f8a70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
...
Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
    result = self._batch_reducible_process(element)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
    self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in _execute
    response = task()
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 313, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 483, in do_instruction
    getattr(request, request_type), request.instruction_id)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 518, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 983, in process_bundle
    element.data)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 219, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 330, in apache_beam.runners.worker.operations.Operation.output
  ...
  File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
    result = self._batch_reducible_process(element)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
    self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors' [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractBatchPredictions/Predict']
...
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0273050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77fc170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes arg

我认为评估器组件与提供输入功能没有任何关系，因为它只是将新训练的模型与最新发布的模型进行比较，但是我哪里出错了?

I don't think evaluator component has anything to do with serving input function as it just compare to the newly trained model with a latest published model, but then where did I go wrong?

推荐答案

所以最后我误会了评估器组件，或者如果我改用 TFMA 则更合适.它确实使用了服务签名中定义的服务输入函数.根据此链接，TFMA EvalConfig 使用的默认签名是serving_default"；它描述了要序列化示例的服务模型输入.这就是为什么当我更改字符串以外的输入签名时，TFMA 会作为异常引发.

So in the end I was mistaken about the evaluator component, or more appropriately if I address the TFMA instead. it indeed uses the serving input function defined in serving signatures. According to this link, the default signature used by the TFMA EvalConfig is "serving_default" which describes the serving model input to be serialized examples. That's why when I changed the input signature other than string, TFMA would raised as exception.

我认为此签名不打算用于通过 REST API 为模型提供服务，因为serving_default"是仍然需要签名，我不想修改 EvalConfig，我创建了另一个签名来接收我想要的 JSON 输入.为了使 tht 工作，我需要制作另一个由 @tf.function 装饰的函数.就这样.希望我的回答能帮助到遇到类似问题的人.

I think this signature is not meant to be used in serving the model through REST API and because the "serving_default" signature is still needeed and I am not in the mood with tinkering the EvalConfig, I created another signature which would receive the JSON input that I want. For tht to work, I need to make another function decorated by @tf.function. That's all. I hope my answer will help people who struggle with similar problems.

这篇关于执行 TFMA 时出现 TFX 管道错误:AttributeError: 'NoneType' 对象没有属性 'ToBatchTensors'的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！