如何在Processing中使用网络摄像头确定上唇和下唇之间的距离？ [英] How to determine the distance between upper lip and lower lip by using webcam in Processing?

查看：267
本文介绍了如何在Processing中使用网络摄像头确定上唇和下唇之间的距离？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我应该从哪里开始？我可以看到使用Python，Java脚本进行大量的人脸识别和分析，但处理怎么样？

Where should I start? I can see plenty of face recognition and analysis using Python, Java script but how about Processing ?

我想通过在上下嘴唇之间使用2点来确定距离他们的最高点和最低点通过网络摄像头在进一步的项目中使用它。

I want to determine the distance by using 2 points between upper and lower lip at their highest and lowest point via webcam to use it in further project.

任何帮助将不胜感激

推荐答案

如果您想单独使用Processing，可以使用 Greg Borenstein的OpenCV for处理库：

If you want to do it in Processing alone you can use Greg Borenstein's OpenCV for Processing library:


  1. 您可以从人脸检测示例

  2. 检测到脸部后，可以检测到嘴巴使用 OpenCV.CASCADE_MOUTH 在面部矩形内。

  3. 一旦你检测到嘴巴，也许你可以使用嘴巴边界框高度逃脱。有关更多详细信息，请使用OpenCV来阈值该矩形。希望张开的嘴巴可以从皮肤的其他部分很好地分割。 查找轮廓应该为您提供点数列表可以使用。

  1. You can start with the Face Detection example
  2. Once you detect a face, you can detect a mouth within the face rectangle using OpenCV.CASCADE_MOUTH.
  3. Once you have mouth detected maybe you can get away with using the mouth bounding box height. For more detail you use OpenCV to threshold that rectangle. Hopefully the open mouth will segment nicely from the rest of the skin. Finding contours should give you lists of points you can work with.

对于更精确的东西，你可以使用Jason Saragih的CLM FaceTracker，可作为OpenFrameworks插件使用。 OpenFrameworks与Processing有相似之处。如果您在Processing中确实需要这种准确性，可以在后台运行 FaceOSC 并使用 oscP5 阅读处理中的口部坐标

For something a lot more exact, you can use Jason Saragih's CLM FaceTracker, which is available as an OpenFrameworks addon. OpenFrameworks has similarities to Processing. If you do need this sort of accuracy in Processing you can run FaceOSC in the background and read the mouth coordinates in Processing using oscP5

更新

对于第一个选项，使用HAAR级联分类器，结果存在以下几个问题：

For the first option, using HAAR cascade classifiers, turns out there are a couple of issues:


  1. OpenCV Processing库可以加载一个级联，第二个实例将覆盖第一个。

  2. OpenCV.CASCADE_MOUTH似乎对闭口更好，但是嘴巴不太好

要通过第一期，你可以使用<一个href =http://docs.opencv.org/java/2.4.9/ =nofollow> OpenCV Java API ，绕过OpenCV Processing进行多级联检测。

To get past the 1st issue, you can use the OpenCV Java API directly, bypassing OpenCV Processing for multiple cascade detection.

有几个参数可以帮助判断例如，将手前的边界框的想法作为提示传递给分类器。
我已经在笔记本电脑上使用网络摄像头进行了基本测试，并测量了不同距离的面部和嘴部边界框。这是一个例子：

There are couple of parameters that can help the detection, such as having idea of the bounding box of the mouth before hand to pass as a hint to the classifier. I've done a basic test using a webcam on my laptop and measure the bounding box for face and mouth at various distances. Here's an example:

import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);
Size minFeatureSizeMouth = new Size(30,10);
Size maxFeatureSizeMouth = new Size(120,60);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150
//laptop webcam mouth rectangle
//far, small scale, ~30,10
//typing distance, ~50,25px
//really close, ~120,60

int mouthHeightHistory = 30;
int[] mouthHeights = new int[mouthHeightHistory]; 

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();
  frameRate(30);

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces >= 1){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.65);
    faceLower.y = face.y + faceLower.height; 
    Mat faceROI = equalizedImg.submat(faceLower);
    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROI,faceImg);
    image(faceImg,width-faceImg.width,0);

    mouthDetections = new MatOfRect();
    mouthDetector.detectMultiScale(faceROI, mouthDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeMouth, maxFeatureSizeMouth);
    Rect[] mouthDetectionResults = mouthDetections.toArray();
    int mouths = mouthDetectionResults.length;
    text("detected mouths: "+mouths,5,25);
    if(mouths >= 1){
      Rect mouth = mouthDetectionResults[0];
      stroke(192,0,0);
      rect(faceLower.x + mouth.x,faceLower.y + mouth.y,mouth.width,mouth.height);
      text("mouth height:"+mouth.height+"~px",5,35);
      updateAndPlotMouthHistory(mouth.height);
    }
  }
}
void updateAndPlotMouthHistory(int newHeight){
  //shift older values by 1
  for(int i = mouthHeightHistory-1; i > 0; i--){
    mouthHeights[i] = mouthHeights[i-1];
  } 
  //add new value at the front
  mouthHeights[0] = newHeight;
  //plot
  float graphWidth = 100.0;
  float elementWidth = graphWidth / mouthHeightHistory;  
  for(int i = 0; i < mouthHeightHistory; i++){
    rect(elementWidth * i,45,elementWidth,mouthHeights[i]);
  }
}
void captureEvent(Capture c) {
  c.read();
}

要做的一个非常重要的注释：我'已将OpenCV Processing库文件夹（〜/ Documents / Processing / libraries / opencv_processing / library / cascade-files ）中的级联xml文件复制到草图的数据文件夹中。我的草图是OpenCVMouthOpen，因此文件夹结构如下所示：

One very imortant note to make: I've copied cascade xml files from the OpenCV Processing library folder (~/Documents/Processing/libraries/opencv_processing/library/cascade-files) to the sketch's data folder. My sketch is OpenCVMouthOpen, so the folder structure looks like this:

OpenCVMouthOpen
├── OpenCVMouthOpen.pde
└── data
    ├── haarcascade_frontalface_alt.xml
    ├── haarcascade_frontalface_alt2.xml
    ├── haarcascade_frontalface_alt_tree.xml
    ├── haarcascade_frontalface_default.xml
    ├── haarcascade_mcs_mouth.xml
    └── lbpcascade_frontalface.xml

如果你不这样做复制级联文件并使用代码，因为它不会出现任何错误，但检测根本不起作用。如果你想检查，你可以做

If you don't copy the cascades files and use the code as it is you won't get any errors, but the detection simply won't work. If you want to check, you can do

println(faceDetector.empty())

在 setup（）函数的末尾，如果你得到 false ，已加载级联，如果你得到 true ，则尚未加载级联。

at the end of the setup() function and if you get false, the cascade has been loaded and if you get true, the cascade hasn't been loaded.

您可能需要使用 minFeatureSize 和 maxFeatureSize 的面部和嘴部值你的设置。第二个问题，级联不能很好地检测到张开的嘴巴很棘手。可能有一个已经训练过的开口级联，但你需要找到它。否则，使用这种方法你可能需要自己训练一个，这可能有点乏味。

You may need to play with the minFeatureSize and maxFeatureSize values for face and mouth for your setup. The second issue, cascade not detecting wide open mouth very well is tricky. There might be an already trained cascade for open mouths, but you'd need to find it. Otherwise, with this method you may need to train one yourself and that can be a bit tedious.

尽管如此，请注意左边画的是一个倒置的情节。检测到嘴巴。在我的测试中，我注意到高度不是非常准确，但图表中有明显的变化。您可能无法获得稳定的嘴高，但通过比较当前与平均的先前高度值，您应该看到一些峰值（值从正到负，反之亦然），这可以让您了解嘴巴张开/近距离变化。

Nevertheless, notice that there is an upside down plot drawn on the left when a mouth is detected. In my tests I noticed that the height isn't super accurate, but there are noticeable changes in the graph. You may not be able to get a steady mouth height, but by comparing current to averaged previous height values you should see some peaks (values going from positive to negative or vice-versa) which give you an idea of a mouth open/close change.

尽管在整个图像中搜索嘴而不是面部只能稍微慢一些，但准确度较低，这是一个更简单的设置。它可以让你的项目更准确，更多误报，这可能更简单：

Although searching through the whole image for a mouth as opposed to a face only can be a bit slower and less accurate, it's a simpler setup. It you can get away with less accuracy and more false positives on your project this could be simpler:

import gab.opencv.*;
import java.awt.Rectangle;
import org.opencv.objdetect.Objdetect;
import processing.video.*;

Capture video;
OpenCV opencv;
Rectangle[] faces,mouths;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
int minFeatureSize = 20;
int maxFeatureSize = 150;
// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 6;

void setup() {
  size(320, 240);
  noFill();
  stroke(0, 192, 0);
  strokeWeight(3);

  video = new Capture(this,width,height);
  video.start();

  opencv  = new OpenCV(this,320,240);
  opencv.loadCascade(OpenCV.CASCADE_MOUTH);
}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  Rectangle[] mouths = opencv.detect(searchScaleFactor,minNeighbors,flags,minFeatureSize, maxFeatureSize);
  for (int i = 0; i < mouths.length; i++) {
    text(mouths[i].x + "," + mouths[i].y + "," + mouths[i].width + "," + mouths[i].height,mouths[i].x, mouths[i].y);
    rect(mouths[i].x, mouths[i].y, mouths[i].width, mouths[i].height);
  }
}
void captureEvent(Capture c) {
  c.read();
}

我也提到了细分/阈值处理。这是一个粗略的例子，使用检测到的面部的下半部分只是一个基本阈值，然后是一些基本的形态滤波器（腐蚀/扩张）来清理阈值图像：

I was mentioning segmenting/thresholding as well. Here's a rough example using the lower part of a detected face just a basic threshold, then some basic morphological filters (erode/dilate) to cleanup the thresholded image a bit:

import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;
import org.opencv.imgproc.Imgproc;
import java.awt.Rectangle;
import java.util.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150

float threshold = 160;
int erodeAmt = 1;
int dilateAmt = 5;

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);

  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces > 0){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.55);
    faceLower.y = face.y + faceLower.height; 
    //submat grabs a portion of the image (submatrix) = our region of interest (ROI)
    Mat faceROI = equalizedImg.submat(faceLower);
    Mat faceROIThresh = faceROI.clone();
    //threshold
    Imgproc.threshold(faceROI, faceROIThresh, threshold, width, Imgproc.THRESH_BINARY_INV);
    Imgproc.erode(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), erodeAmt);
    Imgproc.dilate(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), dilateAmt);
    //find contours
    Mat faceContours = faceROIThresh.clone();
    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    Imgproc.findContours(faceContours, contours, new Mat(), Imgproc.RETR_EXTERNAL , Imgproc.CHAIN_APPROX_SIMPLE);
    //draw contours
    for(int i = 0 ; i < contours.size(); i++){
      MatOfPoint contour = contours.get(i);
      Point[] points = contour.toArray();
      stroke(map(i,0,contours.size()-1,32,255),0,0);
      beginShape();
      for(Point p : points){
        vertex((float)p.x,(float)p.y);
      }
      endShape();
    }

    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROIThresh,faceImg);
    image(faceImg,width-faceImg.width,0);
  }
  text("Drag mouseX to control threshold: " + threshold+
      "\nHold 'e' and drag mouseX to control erodeAmt: " + erodeAmt+
      "\nHold 'd' and drag mouseX to control dilateAmt: " + dilateAmt,5,210);
}
void mouseDragged(){
  if(keyPressed){
    if(key == 'e') erodeAmt = (int)map(mouseX,0,width,1,6);
    if(key == 'd') dilateAmt = (int)map(mouseX,0,width,1,10);
  }else{
    threshold = mouseX;
  }
}
void captureEvent(Capture c) {
  c.read();
}

通过使用YCrCb色彩空间更好地分割皮肤，可以稍微改善一下，但总的来说，你注意到有很多变量可以做到这一点，但这并没有使这个变得非常灵活。

This could be improved a bit by using YCrCb colour space to segment skin better, but overall you notice that there are quite a few variables to get right which doesn't make this a very flexible setup.

你将更好的结果使用 FaceOSC 并通过oscP5读取处理所需的值。以下是 FaceOSCReceiver 的略微简化版本处理示例主要关注嘴巴：

You will be much better results using FaceOSC and reading the values you need in Processing via oscP5. Here is a slightly simplified version of the FaceOSCReceiver Processing example focusing mainly on mouth:

import oscP5.*;
OscP5 oscP5;

// num faces found
int found;

// pose
float poseScale;
PVector posePosition = new PVector();


// gesture
float mouthHeight;
float mouthWidth;

void setup() {
  size(640, 480);
  frameRate(30);

  oscP5 = new OscP5(this, 8338);
  oscP5.plug(this, "found", "/found");
  oscP5.plug(this, "poseScale", "/pose/scale");
  oscP5.plug(this, "posePosition", "/pose/position");
  oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
  oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
}

void draw() {  
  background(255);
  stroke(0);

  if(found > 0) {
    translate(posePosition.x, posePosition.y);
    scale(poseScale);
    noFill();
    ellipse(0, 20, mouthWidth* 3, mouthHeight * 3);
  }
}

// OSC CALLBACK FUNCTIONS

public void found(int i) {
  println("found: " + i);
  found = i;
}

public void poseScale(float s) {
  println("scale: " + s);
  poseScale = s;
}

public void posePosition(float x, float y) {
  println("pose position\tX: " + x + " Y: " + y );
  posePosition.set(x, y, 0);
}

public void mouthWidthReceived(float w) {
  println("mouth Width: " + w);
  mouthWidth = w;
}

public void mouthHeightReceived(float h) {
  println("mouth height: " + h);
  mouthHeight = h;
}


// all other OSC messages end up here
void oscEvent(OscMessage m) {
  if(m.isPlugged() == false) {
    println("UNPLUGGED: " + m);
  }
}

在OSX上你只需下载已编译的FaceOSC应用。
在其他操作系统上，您可能需要设置OpenFrameworks ，下载 of ofFaceTracker 并自行编译FaceOSC。

On OSX you can simply download the compiled FaceOSC app. On other operating systems you may need to setup OpenFrameworks, download ofxFaceTracker and compile FaceOSC yourself.

这篇关于如何在Processing中使用网络摄像头确定上唇和下唇之间的距离？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆