在 Android 上使用 JNI 进行图像解码和处理 [英] image decoding and manipulation using JNI on android
问题描述
在某些应用程序中，重要的是要在没有 OOM 的情况下快速处理大图像.
On some apps, it is important to handle large images without OOM and also quickly.
为此，JNI(或渲染脚本，遗憾的是缺乏文档)可能是一个不错的解决方案.
For this, JNI (or renderscript, which sadly lacks on documentation) can be a nice solution.
过去，我成功地使用 JNI 来旋转巨大的位图，同时避免 OOM(链接 这里，这里和这里).这是一个很好的(但令人讨厌的)体验，但最终它奏效了.
In the past, i've succeeded using JNI for rotating huge bitmaps while avoiding OOM (link here , here and here). it was a nice (yet annoyingly hard) experience, but in the end it worked.
android框架有很多处理位图的函数，但我不知道JNI端的情况如何.
the android framework has plenty of functions to handle bitmaps, but i have no idea what is the situation on the JNI side.
我已经知道如何将位图从 android 的java 世界"传递到JNI 世界"并返回.
I already know how to pass a bitmap from android's "java world" to the "JNI world" and back.
我不知道我可以在 JNI 端使用哪些函数来帮助我处理位图.
What i don't know is which functions I can use on the JNI side to help me with bitmaps.
我希望能够在JNI上做所有的图像操作(包括解码)，这样我在面对大图像时就不用担心OOM，并且在处理结束时，我可以转换数据到 Java 位图(向用户显示)和/或将其写入文件.
I wish to be able to do all image operations (including decoding) on JNI, so that I won't need to worry about OOM when presented with large images, and in the end of the process, I could convert the data to Java-bitmap (to show the user) and/or write it to a file.
再次，我不想将 JNI 端的数据转换为 java 位图，只是为了能够运行这些操作.
again, i don't want to convert the data on the JNI side to a java bitmap just to be able to run those operations.
事实证明，有些库提供了许多功能(例如 JavaCV)，但它们非常大，我不太确定它们的功能以及它们是否真的在 JNI 端进行解码，所以我希望能够通过以下方式知道什么是可能的取而代之的是Android内置的JNI函数.
As it turns out, there are some libraries that offer many functions (like JavaCV), but they are quite large and I'm not quite sure about their features and if they really do the decoding on the JNI-side, so I would prefer to be able to know what is possible via the built-in JNI function of Android instead.
android 的 JNI 端有哪些函数可用于图像处理?
which functions are available for image manipulation on the JNI side on android?
例如，我如何在位图上运行人脸检测、应用矩阵、下采样位图、缩放位图等等...?
for example, how could i run face detection on bitmaps, apply matrices, downsample bitmaps, scale bitmaps, and so on... ?
对于一些操作，我已经想到了实现它们的方法(缩放图像很容易，维基百科可以提供很多帮助)，但有些非常复杂.
for some of the operations, i can already think of a way to implement them (scaling images is quite easy, and wikipedia can help a lot), but some are very complex.
即使我自己实现了这些操作，考虑到 C/C++ 可以进行的许多优化，也许其他人已经提高了它的效率.
even if i do implement the operations by myself, maybe others have made it much more efficiently, thinking of the so many optimizations that C/C++ can have.
当我进入 android 的 JNI 端时，我真的是靠我自己吗，我需要从头开始实现一切?
am i really on my own when going to the JNI side of android, where i need to implement everythign from scratch?
澄清一下，我感兴趣的是:
just to make it clear, what i'm interested in is:
java 上的输入位图 -> 纯粹在 JNI 和 C/C++ 中进行图像处理(不转换为 java 对象) -> java 上的输出位图.
input bitmap on java -> image manipulation purely in JNI and C/C++ (no convertion to java objects whatsoever) ->output bitmap on java.
推荐答案
Android 的内置 JNI 函数"有点矛盾.从技术上讲，许多 Android 框架 Java 类在链的某个位置使用 JNI 来调用本机库是正确的.
"built-in JNI function of Android" is kind of oxymoron. It's technically correct that many Android Framework Java classes use JNI somewhere down the chain to invoke native libraries.
但是对这个声明有三点保留.
But there are three reservations regarding this statement.
这些是实施细节"，可能会在任何下一版本的 Android、任何分支(例如 Kindle)甚至不被视为 "fork"(例如由三星构建，或为 Quallcom SOC 构建).
These are "implementation details", and are subject to change without notice in any next release of Android, or any fork (e.g. Kindle), or even OEM version which is not regarded a "fork" (e.g. built by Samsung, or for Quallcom SOC).
在核心 Java 类中实现本机方法的方式与经典"JNI 不同.这些方法由 JVM 预加载和缓存，因此不会受到 JNI 调用的大部分典型开销的影响.
The way native methods are implemented in core Java classes is different from the "classical" JNI. These methods are preloaded and cached by the JVM and are therefore do not suffer from most of the overhead typical for JNI calls.
您的 Java 或本机代码无法直接与其他类的 JNI 方法进行交互，尤其是构成系统框架的类.
There is nothing your Java or native code can do to interact directly with the JNI methods of other classes, especially classes that constitute the system framework.
说了这么多，你可以自由研究Android的源码，找到原生库支持特定的类和方法(例如面部检测)，并在您的本机代码中使用这些库，或者构建您自己的 JNI 层以从您的 Java 代码中使用这些库.
All this said, you are free to study the source code of Android, to find the native libraries that back specific classes and methods (e.g. face detection), and use these libraries in your native code, or build a JNI layer of your own to use these libraries from your Java code.
举个具体的例子，Android中的人脸检测是通过android.media.FaceDetector 类，它加载 libFFTEm.so.您可以查看本机代码，并根据需要使用它.您不应假设 libFFTEm.so 将存在于设备上，或者设备上的库将具有相同的 API.
To give a specific example, face detection in Android is implemented through the android.media.FaceDetector class, which loads libFFTEm.so. You can look at the native code, and use it as you wish. You should not assume that libFFTEm.so will be present on the device, or that the library on device will have same API.
但在这种特定情况下，这不是问题，因为 neven 的所有工作都是完全基于软件的.因此，您可以完全复制此代码，也可以只复制其中的相关部分，并将其作为您的本地库的一部分.请注意，对于许多设备，您可以简单地加载和使用 /system/lib/libFFTEm.so 并且永远不会感到不适，直到遇到一个行为不端的系统.
But in this specific case, it's not a problem, because all work of neven is entirely software based. Therefore you can copy this code in its entirety, or only relevant parts of it, and make it part of your native library. Note that for many devices you can simply load and use /system/lib/libFFTEm.so and never feel discomfort, until you encounter a system that will misbehave.
您可以通过阅读 本机代码得出一个值得注意的结论，是底层算法忽略了颜色信息.因此，如果要查找人脸坐标的图像来自 YUV 源，调用
One noteworthy conclusion you can make from reading the native code, is that the underlying algorithms ignore the color information. Therefore, if the image for which you want to find face coordinates comes from YUV source, you can avoid a lot of overhead if you call
// run detection
btk_DCR_assignGrayByteImage(hdcr, bwbuffer, width, height);
int numberOfFaces = 0;
if (btk_FaceFinder_putDCR(hfd, hdcr) == btk_STATUS_OK) {
numberOfFaces = btk_FaceFinder_faces(hfd);
} else {
ALOGE("ERROR: Return 0 faces because error exists in btk_FaceFinder_putDCR.
");
}
直接使用您的 YUV(或 Y)字节数组，而不是将其转换为 RGB 并在 android.media.FaceDetector.findFaces().如果您的 YUV 缓冲区来自 Java，您可以构建自己的类 YuvFaceDetector，它将是 android.media.FaceDetector 唯一的区别是 YuvFaceDetector.findFaces()Y(亮度)值只代替Bitmap，避免RGB到Y的转换.
directly with your YUV (or Y) byte array, instead of converting it to RGB and back to YUV in android.media.FaceDetector.findFaces(). If your YUV buffer comes from Java, you can build your own class YuvFaceDetector which will be a copy of android.media.FaceDetector with the only difference that YuvFaceDetector.findFaces() will take Y (luminance) values only instead of a Bitmap, and avoid the RGB to Y conversion.
其他一些情况就没有这么简单了.例如，视频编解码器与硬件平台紧密耦合，您不能简单地从 libstagefright.so 到你的项目.Jpeg 编解码器是一种特殊的野兽.在现代系统(IIRC，自 2.2 起)中，您可以期望 /system/lib/libjpeg.so 存在.但是许多平台也通过 libstagefright.so 或 OpenMAX 有更高效的 Jpeg 编解码器硬件实现，并且这些通常用于 android.graphics.Bitmap.compress() 和android.graphics.BitmapFactory.decode***() 方法.
Some other situations are not as easy as this. For example, the video codecs are tightly coupled to the hardware platform, and you cannot simply copy the code from libstagefright.so to your project. Jpeg codec is a special beast. In modern systems (IIRC, since 2.2), you can expect /system/lib/libjpeg.so to be present. But many platforms also have much more efficient HW implementations of Jpeg codecs through libstagefright.so or OpenMAX, and often these are used in android.graphics.Bitmap.compress() and android.graphics.BitmapFactory.decode***() methods.
还有一个优化的libjpeg-turbo，比<代码>/system/lib/libjpeg.so.
这篇关于在 Android 上使用 JNI 进行图像解码和处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
