将TrueDepth数据转换为灰度图像会产生失真的图像 [英] Converting TrueDepth data to grayscale image produces distorted image
问题描述
我从TrueDepth相机获取深度数据，并将其转换为
实际输出(20％的时间):
实际输出(80％的时间):
TL; DR
您应该始终取消对 CVPixelBufferGetBaseAddress 的调用，以免丢失重要警告.
发现问题是如何访问byteBuffer内部的值.如果不是使用 unsafeBitCast()而是使用Apple在其示例中使用的方法( assumingMemoryBound )，则将获得正确的结果.
虽然看起来像:
//错误代码让byteBuffer = unsafeBitCast(baseAddress，to:UnsafeMutablePointer< Float16> .self)//...让byteBufferIndex = col + row * bytesPerRow让距离= byteBuffer [byteBufferIndex] ...的行为应与以下内容相同:
//良好代码让rowData = baseAddress！+行* bytesPerRow让距离= rowData.assumingMemoryBound(到:Float16.self)[col] ...事实上，两者有很大的不同，前者产生了上述不好的结果，而后者产生了很好的结果.
最终的(固定的)代码应如下所示:
让depthDataMap:CVPixelBuffer = ...let width = CVPixelBufferGetWidth(depthDataMap)//640let height = CVPixelBufferGetHeight(depthDataMap)//480let bytesPerRow = CVPixelBufferGetBytesPerRow(depthDataMap)//1280让baseAddress = CVPixelBufferGetBaseAddress(depthDataMap)！assert(kCVPixelFormatType_DepthFloat16 == CVPixelBufferGetPixelFormatType(depthDataMap))var像素= [Float]()适用于0 ..< height {for col in 0 ..< width {让rowData = baseAddress + row * bytesPerRow让距离= rowData.assumingMemoryBound(到:Float16.self)[col]像素+ = [距离]}}//TODO:将像素渲染为灰度图像 我实际上不确定为什么会这样，因为我们知道:
assert(MemoryLayout< Float16> .size == 2)断言(宽度== 640)断言(bytesPerRow == 1280)断言(宽度* 2 == bytesPerRow) 这似乎意味着在一行的末尾没有多余的字节，我们应该能够将其读取为一个巨型数组.
如果有人知道前者为什么失败，请分享！
更新:
如果您强制取消对 CVPixelBufferGetBaseAddress 的调用:
让baseAddress = CVPixelBufferGetBaseAddress(depthDataMap)！ ...事情开始变得更有意义了.
即，您将在此行看到警告:
let byteBuffer = unsafeBitCast(baseAddress，至:UnsafeMutablePointer< Float16> .self) ⚠️'unsafeBitCast'从'UnsafeMutableRawPointer'到'UnsafeMutablePointer'为原始指针提供了一种类型，并可能导致未定义的行为
⚠️如果已知指针指向内存中现有的值或类型为'Float16'的数组，请使用'assumingMemoryBound'方法
我猜我看到的结果与未定义行为"相关.警告.
因此，本课是，在尝试使用 CVPixelBufferGetBaseAddress 的结果之前(例如，在 unsafeBitCast 中)，应始终将其包装.
I'm getting the depth data from the TrueDepth camera, and converting it to a grayscale image. (I realize I could pass the AVDepthData to a CIImage constructor, however, for testing purposes, I want to make sure my array is populated correctly, therefore manually constructing an image would ensure that is the case.)
I notice that when I try to convert the grayscale image, I get weird results. Namely, the image appears in the top half, and the bottom half is distorted (sometimes showing the image twice, other times showing nonsense).
For example:
Expected output (i.e.
CIImage(depthData: depthData)):
Actual output (20% of the time):
Actual output (80% of the time):
I started with Apple's sample code and tried to extract the pixel in the CVPixelBuffer.
let depthDataMap: CVPixelBuffer = ...
let width = CVPixelBufferGetWidth(depthDataMap) // 640
let height = CVPixelBufferGetHeight(depthDataMap) // 480
let bytesPerRow = CVPixelBufferGetBytesPerRow(depthDataMap) // 1280
let baseAddress = CVPixelBufferGetBaseAddress(depthDataMap)
assert(kCVPixelFormatType_DepthFloat16 == CVPixelBufferGetPixelFormatType(depthDataMap))
let byteBuffer = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Float16>.self)
var pixels = [Float]()
for row in 0..<height {
for col in 0..<width {
let byteBufferIndex = col + row * bytesPerRow
let distance = byteBuffer[byteBufferIndex]
pixels += [distance]
}
}
// TODO: render pixels as a grayscale image
Any idea what is wrong here?
TL;DR
You should always unwrap the call to CVPixelBufferGetBaseAddress so that you don't miss important warnings.
Turns out the problem is how the value inside the byteBuffer is being accessed. If instead of using unsafeBitCast() you use the method Apple uses in their example (assumingMemoryBound), you will get the correct results.
Although it looks like:
// BAD CODE
let byteBuffer = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Float16>.self)
// ...
let byteBufferIndex = col + row * bytesPerRow
let distance = byteBuffer[byteBufferIndex]
... should behave the same as:
// GOOD CODE
let rowData = baseAddress! + row * bytesPerRow
let distance = rowData.assumingMemoryBound(to: Float16.self)[col]
... the two are in fact very different, with the former producing the bad results mentioned above, and the latter producing good results.
The final (fixed) code should look like this:
let depthDataMap: CVPixelBuffer = ...
let width = CVPixelBufferGetWidth(depthDataMap) // 640
let height = CVPixelBufferGetHeight(depthDataMap) // 480
let bytesPerRow = CVPixelBufferGetBytesPerRow(depthDataMap) // 1280
let baseAddress = CVPixelBufferGetBaseAddress(depthDataMap)!
assert(kCVPixelFormatType_DepthFloat16 == CVPixelBufferGetPixelFormatType(depthDataMap))
var pixels = [Float]()
for row in 0..<height {
for col in 0..<width {
let rowData = baseAddress + row * bytesPerRow
let distance = rowData.assumingMemoryBound(to: Float16.self)[col]
pixels += [distance]
}
}
// TODO: render pixels as a grayscale image
I'm actually not sure why this is the case because we know:
assert(MemoryLayout<Float16>.size == 2)
assert(width == 640)
assert(bytesPerRow == 1280)
assert(width * 2 == bytesPerRow)
This seems to imply that there are no extra bytes at the end of a row, and we should be able to read it as one giant array.
If anyone knows why the former fails, please share!
Update:
If you force unwrap the call to CVPixelBufferGetBaseAddress:
let baseAddress = CVPixelBufferGetBaseAddress(depthDataMap)!
... things start to make a bit more sense.
Namely, you will see a warning on this line:
let byteBuffer = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Float16>.self)
⚠️ 'unsafeBitCast' from 'UnsafeMutableRawPointer' to 'UnsafeMutablePointer' gives a type to a raw pointer and may lead to undefined behavior
⚠️ Use the 'assumingMemoryBound' method if the pointer is known to point to an existing value or array of type 'Float16' in memory
I guess the results I seeing were related to the "undefined behavior" warning.
The lesson, therefore, is that you should always unwrap the result of CVPixelBufferGetBaseAddress before attempting to use it (e.g. in unsafeBitCast).
这篇关于将TrueDepth数据转换为灰度图像会产生失真的图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
