如何在Windows 8现代版应用程序的视频流中捕获帧? [英] How do I grab frames from a video stream on Windows 8 modern apps?
问题描述
我正在尝试从mp4视频流中提取图像.在查找内容之后，似乎正确的做法是使用C ++中的Media Foundations，然后从中打开框架/读取内容.
I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
文档和样本的方式很少，但是经过一番挖掘之后，似乎有些人已经成功地做到了这一点，方法是将帧读取到纹理中并将该纹理的内容复制到内存可读的纹理中(我我什至不确定我是否在这里使用正确的术语).尝试我发现的内容会给我带来错误，并且我可能在做很多错误的事情.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
这是我尝试执行的一小段代码(项目本身附加在底部).
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
我得到的错误是:
App1.exe中0x76814B32处的第一次机会异常:Microsoft C ++异常:内存位置0x07AFF60C处的Platform :: InvalidArgumentException ^.HRESULT:0x80070057
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
我不太确定如何继续进行下去，因为就像我说的那样，关于它的文档很少.
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
这是我正在修改的示例.此问题特定于WinRT(Windows 8应用).
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
推荐答案
更新成功！见底部编辑
一些部分成功，但也许足以回答您的问题.请继续阅读.
Some partial success, but maybe enough to answer your question. Please read on.
在我的系统上，调试异常显示尝试调用 TransferVideoFrame()时 OnTimer()函数失败.它给出的错误是 InvalidArgumentException .
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
所以我的第一个更改是在函数 CreateDX11Device()中，如下所示:
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
现在， TransferVideoFrame()仍然失败，但是给出了 E_FAIL (作为HRESULT)，而不是无效的参数.
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
更多谷歌搜索导致我发现-
其中一个示例显示了使用 TransferVideoFrame()而不使用 CreateTexture2D()来预先创建纹理的示例.我看到您在 OnTimer()中已经有一些与此类似的代码，但是并未使用，所以我想您已经找到了相同的链接.
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
无论如何，我现在使用以下代码获取视频帧:
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
这样做之后，我看到 TransferVideoFrame()成功(好！)，但是在复制的纹理上调用 Map()- m_spCopyTexture -失败，因为该纹理不是通过CPU读取访问权限创建的.
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
因此，我只是将您的读/写 m_spRenderTexture 用作副本的目标，因为它具有正确的标志，并且由于之前的更改，我不再使用它.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
现在，在我的系统上， OnTimer()函数不会失败.视频帧将呈现为纹理，并且像素数据已成功复制到内存缓冲区中.
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
在查看是否还有其他问题之前，也许这是个很好的时机，看看您是否可以取得与目前为止相同的进步.如果您用更多信息对此答案发表评论，我将编辑答案以在可能的情况下添加更多帮助.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
编辑
在 FramePlayer :: CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
还请注意，有一段时间需要清除内存泄漏(我确定您已经知道)-在下面的行中分配的内存永远不会释放:
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
成功
我现在已经成功保存了一个单独的帧，但是现在不使用复制纹理.
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
首先，我下载了最新版本的 DirectXTex库，该库提供了DX11纹理助手功能，例如从纹理中提取图像并保存到文件.说明，以将DirectXTex库添加到解决方案中作为现有项目的需要请注意Windows 8 Store Apps所需的更改.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
一旦包含，引用并构建了上述库，请将以下 #include 添加到 FramePlayer.cpp
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
最后， FramePlayer :: OnTimer()中的代码中央部分必须与以下内容相似.您将看到我每次都保存为相同的文件名，因此需要进行修改以添加例如名称的帧号
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
我现在没有时间进行进一步的介绍，但我对到目前为止所取得的成就感到非常满意:-))
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
您可以重新看一下并在评论中更新您的结果吗?
Can you take a fresh look and update your results in comments?
这篇关于如何在Windows 8现代版应用程序的视频流中捕获帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
