使用 Java 使用 Tensorflow Serving 进行推理 [英] Inferencing with Tensorflow Serving using Java

问题描述

我们正在将现有的 Java 生产代码转换为使用 Tensorflow Serving (TFS) 进行推理.我们已经重新训练我们的模型并使用新的 SavedModel 格式保存它们(不再有冻结图！！).
从我读过的文档来看，TFS 不直接支持 Java.但是它确实提供了一个 gRPC 接口，并且确实提供了一个 Java 接口.

We are transitioning an existing Java production code to use Tensorflow Serving (TFS) for inferencing. We have already retrained our models and saved them using the new SavedModel format (no more frozen graphs!!).
From the documentation that I have read, TFS does not directly support Java. However it does provide a gRPC interface, and that does provide a Java interface.

我的问题是，启动 Java 应用程序以使用 TFS 所涉及的步骤是什么.

My question, what are the steps involved in bringing up a Java application to use TFS.

推荐答案

由于文档和示例仍然有限，我们花了四天时间将它们拼凑起来.
我确信有更好的方法可以做到这一点，但这是我目前发现的:

It took four days to piece this together, as documentation and examples are still limited.
I'm sure there are better ways to do this, but this is what I found so far:

• 我在 github 上克隆了 tensorflow/tensorflow、tensorflow/serving 和 google/protobuf 存储库.
• 我使用 protoc protobuf 编译了以下 protobuf 文件编译器与 grpc-java 插件.我讨厌有这么多分散的 .proto 文件要编译的事实，但我希望包含最小的集合，并且有这么多不需要的 .proto 文件在将被绘制的各种目录.这是编译我们的 Java 应用程序所需的最小集合:
  • serving_repo/tensorflow_serving/apis/*.proto
  • serving_repo/tensorflow_serving/config/model_server_config.proto
  • serving_repo/tensorflow_serving/core/logging.proto
  • serving_repo/tensorflow_serving/core/logging_config.proto
  • serving_repo/tensorflow_serving/util/status.proto
  • serving_repo/tensorflow_serving/sources/storage_path/file_system_storage_path_source.proto
  • serving_repo/tensorflow_serving/config/log_collector_config.proto
  • tensorflow_repo/tensorflow/core/framework/tensor.proto
  • tensorflow_repo/tensorflow/core/framework/tensor_shape.proto
  • tensorflow_repo/tensorflow/core/framework/types.proto
  • tensorflow_repo/tensorflow/core/framework/resource_handle.proto
  • tensorflow_repo/tensorflow/core/example/example.proto
  • tensorflow_repo/tensorflow/core/protobuf/tensorflow_server.proto
  • tensorflow_repo/tensorflow/core/example/feature.proto
  • tensorflow_repo/tensorflow/core/protobuf/named_tensor.proto
  • tensorflow_repo/tensorflow/core/protobuf/config.proto
  • I cloned the tensorflow/tensorflow, tensorflow/serving and google/protobuf repos on github.
  • I compiled the following protobuf files using the protoc protobuf compiler with the grpc-java plugin. I hate the fact that there are so many scattered .proto files to be compiled, but I wanted the minimal set to include and there are so many unneeded .proto files in the various directories that would have been drawn in. Here is the minimal set I needed to compile our Java app:
    • serving_repo/tensorflow_serving/apis/*.proto
    • serving_repo/tensorflow_serving/config/model_server_config.proto
    • serving_repo/tensorflow_serving/core/logging.proto
    • serving_repo/tensorflow_serving/core/logging_config.proto
    • serving_repo/tensorflow_serving/util/status.proto
    • serving_repo/tensorflow_serving/sources/storage_path/file_system_storage_path_source.proto
    • serving_repo/tensorflow_serving/config/log_collector_config.proto
    • tensorflow_repo/tensorflow/core/framework/tensor.proto
    • tensorflow_repo/tensorflow/core/framework/tensor_shape.proto
    • tensorflow_repo/tensorflow/core/framework/types.proto
    • tensorflow_repo/tensorflow/core/framework/resource_handle.proto
    • tensorflow_repo/tensorflow/core/example/example.proto
    • tensorflow_repo/tensorflow/core/protobuf/tensorflow_server.proto
    • tensorflow_repo/tensorflow/core/example/feature.proto
    • tensorflow_repo/tensorflow/core/protobuf/named_tensor.proto
    • tensorflow_repo/tensorflow/core/protobuf/config.proto

    $ ./protoc -I=/Users/foobar/protobuf_repo/src \
       -I=/Users/foobar/tensorflow_repo \   
       -I=/Users/foobar/tfserving_repo \  
       -plugin=protoc-gen-grpc-java=/Users/foobar/protoc-gen-grpc-java-1.20.0-osx-x86_64.exe \
       --java_out=src \
       --grpc-java_out=src \
       /Users/foobar/tfserving_repo/tensorflow_serving/apis/*.proto
    

    • 按照 gRPC 文档，我创建了一个频道和一个存根:
    • Following the gRPC documentation, I created a Channel and a stub:

    ManagedChannel mChannel;
    PredictionServiceGrpc.PredictionServiceBlockingStub mBlockingstub;
    mChannel = ManagedChannelBuilder.forAddress(host,port).usePlaintext().build();
    mBlockingstub = PredictionServiceGrpc.newBlockingStub(mChannel);
    

    • 我按照几个文档拼凑了以下步骤:
      • gRPC 文档讨论存根(阻塞和异步)
      • 这篇文章 概述过程，但使用 Python
      • 此示例代码 对于 NewBuilder 语法示例至关重要.
      • I followed several documents to piece together the steps that follow:
        • The gRPC documents discuss stubs (Blocking and Asynch)
        • This article overview the process, but with Python
        • This sample code was critical for examples of the NewBuilder syntax.
        • io.grpc:grpc-all
        • org.tensorflow:libtensorflow
        • org.tensorflow:proto
        • com.google.protobuf:protobuf-java

        // Generate features TensorProto
        TensorProto.Builder featuresTensorBuilder = TensorProto.newBuilder();
        
        TensorShapeProto.Dim featuresDim1  = TensorShapeProto.Dim.newBuilder().setSize(1).build();
        TensorShapeProto     featuresShape = TensorShapeProto.newBuilder().addDim(featuresDim1).build();
        featuresTensorBuilder.setDtype(org.tensorflow.framework.DataType).setTensorShape(featuresShape);
        TensorProto featuresTensorProto = featuresTensorBuilder.build();
        
        
        // Now prepare for the inference request over gRPC to the TF Serving server
        com.google.protobuf.Int64Value version = com.google.protobuf.Int64Value.newBuilder().setValue(mGraphVersion).build();
        
        Model.ModelSpec.Builder model = Model.ModelSpec
                                             .newBuilder()
                                             .setName(mGraphName)
                                             .setVersion(version);  // type = Int64Value
        Model.ModelSpec     modelSpec = model.build();
        
        Predict.PredictRequest request;
        request = Predict.PredictRequest.newBuilder()
                                        .setModelSpec(modelSpec)
                                        .putInputs("image", featuresTensorProto)
                                        .build();
        
        Predict.PredictResponse response;
        
        try {
            response = mBlockingstub.predict(request);
            // Refer to https://github.com/thammegowda/tensorflow-grpc-java/blob/master/src/main/java/edu/usc/irds/tensorflow/grpc/TensorflowObjectRecogniser.java
        
            java.util.Map<java.lang.String, org.tensorflow.framework.TensorProto> outputs = response.getOutputsOrDefault();
            for (java.util.Map.Entry<java.lang.String, org.tensorflow.framework.TensorProto> entry : outputs.entrySet()) {
                System.out.println("Response with the key: " + entry.getKey() + ", value: " + entry.getValue());
            }
        } catch (StatusRuntimeException e) {
            logger.log(Level.WARNING, "RPC failed: {0}", e.getStatus());
            success = false;
        }
        
        

        这篇关于使用 Java 使用 Tensorflow Serving 进行推理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！