MLKit对象检测未检测到对象 [英] MLKit Object Detection is not detecting objects
本文介绍了MLKit对象检测未检测到对象的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
MLKit(没有Firebase)是新的，所以我遇到了麻烦.我尝试在此遵循以下示例:
修复后， LocalModel 初始化代码:
LocalModel localModel =新的LocalModel.Builder().setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")//或.setAbsoluteFilePath(tflite模型的绝对文件路径).建造(); 更新:发现了另外一个问题
ImageAnalysis 实例未绑定到 CameraProvider :
...ImageAnalysis imageAnalysis = ...相机camera = cameraProvider.bindToLifecycle((LifecycleOwner)this，cameraSelector，预览);//不使用imageAnalysis 要解决此问题，只需将最后一个参数 imageAnalysis 变量传递给 bindToLifecycle 方法:
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this，cameraSelector，预览，imageAnalysis); 第二次更新:发现了另一个问题
MLKit无法处理图像，因为它在处理过程中或在处理开始之前就已关闭.我说的是 imageProxy.close()在 public void analytics(ImageProxy imageProxy)内部声明的代码行.
close()方法的Java文档:
/***释放此框架以供重复使用.*< p>*调用此方法后，调用此{@code Image}上的任何方法将*导致{@link IllegalStateException}，并尝试从中读取*或写入先前版本返回的{@link ByteBuffer ByteBuffers}* {@link Plane#getBuffer}调用将具有未定义的行为.如果图片*是通过{@link ImageWriter}获得的* {@link ImageWriter#dequeueInputImage()}，在调用此方法后，*应用程序填充的图像数据将丢失，并且图像将被*返回到{@link ImageWriter}以供重用.图片提供给* {@link ImageWriter#queueInputImage queueInputImage()}是自动生成的*关闭.*</p>*/ 要解决将 imageProxy.close()移入失败和成功侦听器的问题，
objectDetector.process(图像).addOnFailureListener(new OnFailureListener(){@Override公共无效onFailure(@NonNull异常e){Toast.makeText(getApplicationContext()，"Fail.Sad！"，Toast.LENGTH_LONG).show();...imageProxy.close();}}).addOnSuccessListener(new OnSuccessListener< List< DetectedObject>>>(){@Overridepublic void onSuccess(List< DetectedObject>结果){Toast.makeText(getBaseContext()，"Success ..."，Toast.LENGTH_LONG).show();...imageProxy.close();}}); 固定解决方案已使用来自以下产品的图像分类模型进行了测试Tensorflow和测试成功.
MLKit by Google (without Firebase) is new, so I'm having trouble. I'm trying to follow this example here: https://developers.google.com/ml-kit/vision/object-detection/custom-models/android
The app opens fine, & the camera works (As in, I can see things). But the actual detection doesn't seem to work.
Am I missing part of the code to actually detect the object? Or is it an issue with the implementation of CameraX or ImageInput?
package com.example.mlkitobjecttest;
import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.camera.core.Camera;
import androidx.camera.core.CameraSelector;
import androidx.camera.core.CameraX;
import androidx.camera.core.ImageAnalysis;
import androidx.camera.core.ImageProxy;
import androidx.camera.core.Preview;
import androidx.camera.core.impl.PreviewConfig;
import androidx.camera.lifecycle.ProcessCameraProvider;
import androidx.camera.view.PreviewView;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;
import androidx.lifecycle.LifecycleOwner;
import android.content.pm.PackageManager;
import android.graphics.Rect;
import android.media.Image;
import android.os.Bundle;
import android.text.Layout;
import android.util.Rational;
import android.util.Size;
import android.view.View;
import android.widget.TextView;
import android.widget.Toast;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.mlkit.common.model.LocalModel;
import com.google.mlkit.vision.common.InputImage;
import com.google.mlkit.vision.objects.DetectedObject;
import com.google.mlkit.vision.objects.ObjectDetection;
import com.google.mlkit.vision.objects.ObjectDetector;
import com.google.mlkit.vision.objects.custom.CustomObjectDetectorOptions;
import org.w3c.dom.Text;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class MainActivity extends AppCompatActivity {
private class YourAnalyzer implements ImageAnalysis.Analyzer {
@Override
@androidx.camera.core.ExperimentalGetImage
public void analyze(ImageProxy imageProxy) {
Image mediaImage = imageProxy.getImage();
if (mediaImage != null) {
InputImage image =
InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
// Pass image to an ML Kit Vision API
// ...
LocalModel localModel =
new LocalModel.Builder()
.setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")
// or .setAbsoluteFilePath(absolute file path to tflite model)
.build();
CustomObjectDetectorOptions customObjectDetectorOptions =
new CustomObjectDetectorOptions.Builder(localModel)
.setDetectorMode(CustomObjectDetectorOptions.SINGLE_IMAGE_MODE)
.enableMultipleObjects()
.enableClassification()
.setClassificationConfidenceThreshold(0.5f)
.setMaxPerObjectLabelCount(3)
.build();
ObjectDetector objectDetector =
ObjectDetection.getClient(customObjectDetectorOptions);
objectDetector
.process(image)
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
//Toast.makeText(getApplicationContext(), "Fail. Sad!", Toast.LENGTH_SHORT).show();
//textView.setText("Fail. Sad!");
imageProxy.close();
}
})
.addOnSuccessListener(new OnSuccessListener<List<DetectedObject>>() {
@Override
public void onSuccess(List<DetectedObject> results) {
for (DetectedObject detectedObject : results) {
Rect box = detectedObject.getBoundingBox();
for (DetectedObject.Label label : detectedObject.getLabels()) {
String text = label.getText();
int index = label.getIndex();
float confidence = label.getConfidence();
textView.setText(text);
}}
imageProxy.close();
}
});
}
//ImageAnalysis.Builder.fromConfig(new ImageAnalysisConfig).setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST);
}
}
PreviewView prevView;
private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
private ExecutorService executor = Executors.newSingleThreadExecutor();
TextView textView;
private int REQUEST_CODE_PERMISSIONS = 101;
private String[] REQUIRED_PERMISSIONS = new String[]{"android.permission.CAMERA"};
/* @NonNull
@Override
public CameraXConfig getCameraXConfig() {
return CameraXConfig.Builder.fromConfig(Camera2Config.defaultConfig())
.setCameraExecutor(ContextCompat.getMainExecutor(this))
.build();
}
*/
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
prevView = findViewById(R.id.viewFinder);
textView = findViewById(R.id.scan_button);
if(allPermissionsGranted()){
startCamera();
}else{
ActivityCompat.requestPermissions(this, REQUIRED_PERMISSIONS, REQUEST_CODE_PERMISSIONS);
}
}
private void startCamera() {
cameraProviderFuture = ProcessCameraProvider.getInstance(this);
cameraProviderFuture.addListener(new Runnable() {
@Override
public void run() {
try {
ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
bindPreview(cameraProvider);
} catch (ExecutionException | InterruptedException e) {
// No errors need to be handled for this Future.
// This should never be reached.
}
}
}, ContextCompat.getMainExecutor(this));
}
void bindPreview(@NonNull ProcessCameraProvider cameraProvider) {
Preview preview = new Preview.Builder()
.build();
CameraSelector cameraSelector = new CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build();
preview.setSurfaceProvider(prevView.createSurfaceProvider());
ImageAnalysis imageAnalysis =
new ImageAnalysis.Builder()
.setTargetResolution(new Size(1280, 720))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();
imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), new YourAnalyzer());
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview, imageAnalysis);
}
private boolean allPermissionsGranted() {
for(String permission: REQUIRED_PERMISSIONS){
if(ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED){
return false;
}
}
return true;
}
@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
if(requestCode == REQUEST_CODE_PERMISSIONS){
if(allPermissionsGranted()){
startCamera();
} else{
Toast.makeText(this, "Permissions not granted by the user.", Toast.LENGTH_SHORT).show();
this.finish();
}
}
}
}
解决方案
Nothing is detected because you defined the wrong path to tflite model file. You emulator or physical device cannot resolve given path as it doesn't exists on mobile device: C:\\Users\\dude\\Documents\\mlkitobjecttest\\app\\src\\main\\assets\\mobilenet_v1_1.0_128_quantized_1_default_1.tflite
Copy your model mobilenet_v1_1.0_128_quantized_1_default_1.tflite into assets directory under you app's project src/main directory.
If you do not have that directory just create a new one named assets.
At the end it should look like this:
After that fix LocalModel initialization code:
LocalModel localModel =
new LocalModel.Builder()
.setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")
// or .setAbsoluteFilePath(absolute file path to tflite model)
.build();
Update: one more issue found
ImageAnalysis instance was not bound to CameraProvider:
...
ImageAnalysis imageAnalysis = ...
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview); // imageAnalysis is not used
To fix it just pass as the last argument imageAnalysis variable into bindToLifecycle method:
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview, imageAnalysis);
Second update: another one issue found
MLKit cannot process an image because it was closed while it was processing or right before processing started. I'm talking about imageProxy.close() line of code declared inside of public void analyze(ImageProxy imageProxy).
Java documentation of close() method:
/**
* Free up this frame for reuse.
* <p>
* After calling this method, calling any methods on this {@code Image} will
* result in an {@link IllegalStateException}, and attempting to read from
* or write to {@link ByteBuffer ByteBuffers} returned by an earlier
* {@link Plane#getBuffer} call will have undefined behavior. If the image
* was obtained from {@link ImageWriter} via
* {@link ImageWriter#dequeueInputImage()}, after calling this method, any
* image data filled by the application will be lost and the image will be
* returned to {@link ImageWriter} for reuse. Images given to
* {@link ImageWriter#queueInputImage queueInputImage()} are automatically
* closed.
* </p>
*/
To fix that move imageProxy.close() into failure and success listeners:
objectDetector
.process(image)
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
Toast.makeText(getApplicationContext(), "Fail. Sad!", Toast.LENGTH_LONG).show();
...
imageProxy.close();
}
})
.addOnSuccessListener(new OnSuccessListener<List<DetectedObject>>() {
@Override
public void onSuccess(List<DetectedObject> results) {
Toast.makeText(getBaseContext(), "Success...", Toast.LENGTH_LONG).show();
...
imageProxy.close();
}
});
The fixed solution was tested with image classification model from Tensorflow and test was successful.
这篇关于MLKit对象检测未检测到对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
