在Android中检测人脸地标点 [英] Detecting face landmarks points in android

问题描述

我正在开发应用程序，需要在像镜子凸轮或化妆凸轮这样的凸轮上获取面部界标点。我也希望它也可用于iOS。请为我提供可靠的解决方案。
我曾经使用过Dlib和Luxand。

I am developing app in which I need to get face landmarks points on a cam like mirror cam or makeup cam. I want it to be available for iOS too. Please guide me for a robust solution. I have used Dlib and Luxand.

DLIB： https://github.com/tzutalin/dlib-android-app

Luxand： http://www.luxand.com/facesdk/download/

Dlib很慢，大约有2秒的延迟（请查看git页面上的演示视频），luxand可以，但已付款。我的首要任务是使用开源解决方案。
我也使用了Google的愿景，但是它们并没有提供太多的面部标志点。
因此，请给我一个解决方案，以使dlib快速运行，或者使用其他任何选项来保持跨平台优先。
预先感谢。

Dlib is slow and having a lag of 2 sec approximately (Please look at the demo video on the git page) and luxand is ok but it's paid. My priority is to use an open source solution. I have also use the Google vision but they are not offering much face landmarks points. So please give me a solution to make the the dlib to work fast or any other option keeping cross-platform in priority. Thanks in advance.

推荐答案

您可以让Dlib在Android上实时检测人脸地标（20-30 fps）（如果您选择一些快捷方式）。这是一个很棒的库。

You can make Dlib detect face landmarks in real-time on Android (20-30 fps) if you take a few shortcuts. It's an awesome library.

初始化

首先，您应该遵循所有建议在Evgeniy的答案中，尤其要确保只初始化一次 frontal_face_detector 和 shape_predictor 对象一次，而不是每帧初始化一次。如果从文件反序列化 frontal_face_detector 而不是使用 get_serialized_frontal_faces（）函数，则初始化速度会更快。 shape_predictor 需要从100Mb文件中初始化，并且需要几秒钟的时间。序列化和反序列化功能被编写为跨平台的，并且可以对数据执行验证，这虽然健壮，但速度却很慢。如果您准备对字节顺序进行假设，则可以编写自己的反序列化函数，该过程将更快。该文件主要由136个浮点值的矩阵组成（其中约有120000个浮点值，这意味着总共有16320000个浮点值）。如果将这些浮点数量化为8或16位，则可以节省大量空间（例如，您可以将最小值和（max-min）/ 255存储为每个矩阵的浮点数并分别量化）。这样可以将文件大小减小到大约18Mb，并且可以在几百毫秒而不是几秒钟的时间内加载。对我而言，使用量化值导致的质量下降似乎可以忽略不计，但是YMMV。

Firstly you should follow all the recommendations in Evgeniy's answer, especially make sure that you only initialize the frontal_face_detector and shape_predictor objects once instead of every frame. The frontal_face_detector will initialize faster if you deserialize it from a file instead of using the get_serialized_frontal_faces() function. The shape_predictor needs to be initialized from a 100Mb file, and takes several seconds. The serialize and deserialize functions are written to be cross-platform and perform validation on the data, which is robust but makes it quite slow. If you are prepared to make assumptions about endianness you can write your own deserialization function that will be much faster. The file is mostly made up of matrices of 136 floating point values (about 120000 of them, meaning 16320000 floats in total). If you quantize these floats down to 8 or 16 bits you can make big space savings (e.g. you can store the min value and (max-min)/255 as floats for each matrix and quantize each separately). This reduces the file size down to about 18Mb and it loads in a few hundred milliseconds instead of several seconds. The decrease in quality from using quantized values seems negligible to me but YMMV.

人脸检测

您可以将相机框架缩小到240x160之类的小尺寸（或其他，保持宽高比正确）以更快地进行人脸检测。这意味着您无法检测到较小的面孔，但这可能不是问题，具体取决于您的应用。另一种更复杂的方法是自适应裁剪并调整用于面部检测的区域的大小：首先检查高分辨率图像（例如480x320）中的所有面部，然后在上一个位置周围裁剪+/-一个面部宽度的区域，按比例缩小如果需要的话。如果您未能检测到一张脸，则恢复为检测下一幅整个区域。

You can scale the camera frames down to something small like 240x160 (or whatever, keeping aspect ratio correct) for faster face detection. It means you can't detect smaller faces but it might not be a problem depending on your app. Another more complex approach is to adaptively crop and resize the region you use for face detections: initially check for all faces in a higher res image (e.g. 480x320) and then crop the area +/- one face width around the previous location, scaling down if need be. If you fail to detect a face one frame then revert to detecting the entire region the next one.

面部跟踪

要更快地进行面部跟踪，可以在一个线程中连续运行面部检测，然后在另一个线程中，跟踪检测到的面部并使用跟踪的矩形执行面部特征检测。在测试中，我发现人脸检测大约需要100到400毫秒，具体取决于我使用的手机（大约240x160），并且在此期间我可以在中间帧上进行7或8次人脸特征检测。如果人脸移动很多，这可能会有点棘手，因为当您获得新的人脸检测（这是从400毫秒前开始）时，您必须决定是从新检测到的位置还是从被跟踪者的位置继续跟踪先前的检测。 Dlib包含一个 correlation_tracker ，但是不幸的是，我无法使其运行速度超过每帧250ms左右，并且降低分辨率（甚至是大幅降低）并没有太大用处有所不同。修补内部参数可以提高速度，但跟踪效果较差。我最终使用了基于预览帧的色度UV平面的CAMShift跟踪器，并基于检测到的脸部矩形生成了颜色直方图。

For faster face tracking, you can run face detections continuously in one thread, and then in another thread, track the detected face(s) and perform face feature detections using the tracked rectangles. In my testing I found that face detection took between 100 - 400ms depending on what phone I used (at about 240x160), and I could do 7 or 8 face feature detections on the intermediate frames in that time. This can get a bit tricky if the face is moving a lot, because when you get a new face detection (which will be from 400ms ago), you have to decide whether to keep tracking from the new detected location or the tracked location of the previous detection. Dlib includes a correlation_tracker however unfortunately I wasn't able to get this to run faster than about 250ms per frame, and scaling down the resolution (even drastically) didn't make much of a difference. Tinkering with internal parameters produced increase speed but poor tracking. I ended up using a CAMShift tracker based on the chroma UV planes of the preview frames, generating the color histogram based on the detected face rectangles. There is an implementation of CAMShift in OpenCV, but it's also pretty simple to roll your own.

希望OpenCV中有CAMShift的实现，但是滚动自己也很简单。希望这会有所帮助，这主要是选择低挂果进行优化首先，继续前进，直到满意为止。在Galaxy Note 5上，Dlib会在大约100毫秒的时间内进行面部+特征检测，即使没有所有这些额外的复杂功能，这也足以满足您的目的。

Hope this helps, it's mostly a matter of picking the low hanging fruit for optimization first and just keep going until you're happy it's fast enough. On a Galaxy Note 5 Dlib does face+feature detections at about 100ms, which might be good enough for your purposes even without all this extra complication.

这篇关于在Android中检测人脸地标点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！