设置IMTransform视频处理器以进行色彩空间转换 [英] Setup of IMTransform Video Processor for color space conversion
问题描述
我正在尝试使用视频处理器MFT进行一些基本的色彩空间转换.我的相机本机支持NV12，我需要RGB24，以编写一些可以提供类似卡通效果的着色器.
I am trying to do some basic color space conversion using Video Processor MFT. My camera natively supports NV12 and I need RGB24, to code some shader which will provide cartoon-like effect.
这是用于执行MF的Media类的定义.
Here is definition of class Media which is used to perform MF.
class Media : public IMFSourceReaderCallback //this class inhertis from IMFSourceReaderCallback
{
CRITICAL_SECTION criticalSection;
long referenceCount;
WCHAR *wSymbolicLink;
UINT32 cchSymbolicLink;
IMFSourceReader* sourceReader;
MFT_REGISTER_TYPE_INFO *inputVideoTypes;
MFT_REGISTER_TYPE_INFO *outputVideoTypes;
IMFMediaType* mediaType = NULL;
IMFMediaType* streamType = NULL;
IMFMediaType* streamType2 = NULL;
IMFMediaType* streamType3 = NULL;
IMFTransform **VP;
public:
LONG stride;
float bytesPerPixel;
GUID videoFormat;
UINT height;
UINT width;
WCHAR deviceNameString[2048];
BYTE* rawData;
UINT32 count;
DWORD devices_found = 0;
HRESULT CreateCaptureDevice();
HRESULT SetSourceReader(IMFActivate *device);
HRESULT IsMediaTypeSupported(IMFMediaType* type);
HRESULT GetDefaultStride(IMFMediaType *pType, LONG *plStride);
HRESULT Close();
Media();
~Media();
// the class must implement the methods from IUnknown
STDMETHODIMP QueryInterface(REFIID iid, void** ppv);
STDMETHODIMP_(ULONG) AddRef();
STDMETHODIMP_(ULONG) Release();
// the class must implement the methods from IMFSourceReaderCallback
STDMETHODIMP OnReadSample(HRESULT status, DWORD streamIndex, DWORD streamFlags, LONGLONG timeStamp, IMFSample *sample);
STDMETHODIMP OnEvent(DWORD, IMFMediaEvent *);
STDMETHODIMP OnFlush(DWORD);
};
我通过以下方式进行IMFTransform的设置:
I am doing setup of IMFTransform in the following way:
inputVideoTypes = new MFT_REGISTER_TYPE_INFO;
inputVideoTypes->guidMajorType = MFMediaType_Video;
inputVideoTypes->guidSubtype = MFVideoFormat_NV12;
outputVideoTypes = new MFT_REGISTER_TYPE_INFO;
outputVideoTypes->guidMajorType = MFMediaType_Video;
outputVideoTypes->guidSubtype = MFVideoFormat_RGB24;
hr = sourceReader->GetNativeMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, i, &streamType2);
IMFActivate **transformActivateArray = NULL;
UINT32 MFTcount;
hr = MFTEnumEx(MFT_CATEGORY_VIDEO_PROCESSOR, MFT_ENUM_FLAG_ALL, inputVideoTypes, outputVideoTypes, &transformActivateArray, &MFTcount);
if (FAILED(hr))
{
exit(3);
}
if (MFTcount == 0)
exit(7);
VP = new IMFTransform*[MFTcount];
for (DWORD i = 0; i < MFTcount; i++)
{
hr = transformActivateArray[i]->ActivateObject(__uuidof(IMFTransform), (void**)&VP[i]);
}
DWORD* inputCount = new DWORD[MFTcount];
DWORD* outputCount = new DWORD[MFTcount];
for (DWORD i = 0; i < MFTcount; i++)
{
hr = VP[i]->GetStreamCount(&inputCount[i], &outputCount[i]);
}
DWORD **inputids = new DWORD*[MFTcount];
DWORD **outputids = new DWORD*[MFTcount];
for (DWORD i = 0; i < MFTcount; i++)
{
inputids[i] = new DWORD[inputCount[i]];
outputids[i] = new DWORD[outputCount[i]];
}
for (DWORD i = 0; i < MFTcount; i++)
{
VP[i]->GetStreamIDs(inputCount[i], inputids[i], outputCount[i], outputids[i]);
if (FAILED(hr))
exit(5);
}
DWORD flag1 = -1;
DWORD flag2 = -1;
for (DWORD i = 0; i < MFTcount; i++)
{
for (DWORD j = 0; j < inputCount[i]; j++)
{
hr = VP[i]->GetInputAvailableType(0, 0, &streamType);
if (SUCCEEDED(hr))
{
flag1 = i;
flag2 = j;
break;
}
}
}
if (flag1 == -1 && flag2 == -1)
exit(2);
hr = VP[0]->SetInputType(0, streamType2, 0);
hr = VP[0]->GetOutputAvailableType(0, 0, &streamType3);
hr = VP[0]->SetOutputType(0, streamType3, 0);
问题是SetOutput方法返回:找不到所需的属性，而我却没有真正得到，这是什么问题.谁能指出我在哪里做坏事?谢谢
The problem is that SetOutput method returns: cannot find a demanded attribute, and I dont really get, what is wrong. Can anyone point where I am doing things bad way? Thank you
输入的LogMediaType:
LogMediaType of input:
MF_MT_FRAME_SIZE 1280 x 720
MF_MT_YUV_MATRIX 2
MF_MT_MAJOR_TYPE MFMediaType_Video
MF_MT_VIDEO_LIGHTING 3
MF_MT_VIDEO_CHROMA_SITING 1
MF_MT_AM_FORMAT_TYPE {F72A76A0-EB0A-11D0-ACE4-0000C0CC16BA}
MF_MT_FIXED_SIZE_SAMPLES 1
MF_MT_VIDEO_NOMINAL_RANGE 1
MF_MT_FRAME_RATE 30 x 1
MF_MT_PIXEL_ASPECT_RATIO 1 x 1
MF_MT_ALL_SAMPLES_INDEPENDENT 1
MF_MT_FRAME_RATE_RANGE_MIN 128849018881
MF_MT_VIDEO_PRIMARIES 2
MF_MT_INTERLACE_MODE 2
MF_MT_FRAME_RATE_RANGE_MAX 128849018881
{EA031A62-8BBB-43C5-B5C4-572D2D231C18} 1
MF_MT_SUBTYPE MFVideoFormat_NV12
输出日志无效
Exception thrown: read access violation.
**pType** was nullptr.
EDIT2
我只有一个由EnumEx方法枚举的VP，并且它具有固定数量的输入(1)和输出(1)流，因此以前的日志是唯一一个用于输入的日志
I have only one VP enumerated by EnumEx method, and it has fixed number of input (1) and output (1) streams, so previous log is the only one log for input
编辑3
hr = VP->SetInputType(0, streamType2, 0);
//MediaFoundationSamples::LogMediaType(streamType2);
DWORD dwIndex = 4;
hr = VP->GetOutputAvailableType(0, dwIndex, &streamType3);
hr = MFSetAttributeSize(streamType3, MF_MT_FRAME_SIZE, 1280, 720);
hr = streamType3->SetUINT32(MF_MT_FIXED_SIZE_SAMPLES, 1);
hr = MFSetAttributeRatio(streamType3, MF_MT_FRAME_RATE, 30, 1);
hr = MFSetAttributeRatio(streamType3, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
streamType3->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, 1);
streamType3->SetUINT32(MF_MT_INTERLACE_MODE, 2);
MediaFoundationSamples::LogMediaType(streamType3);
hr = VP->SetOutputType(0, streamType3, 0);
hr = VP->GetInputStreamInfo(0, &InputInfo);
hr = VP->GetOutputStreamInfo(0, &OutputInfo);
，在onReadSample方法中的处理如下:
and processing in onReadSample method looks like:
hr = VP->ProcessMessage(MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, NULL);
hr = VP->ProcessInput(0, sample, 0);
DWORD statusFlags;
hr = VP->GetOutputStatus(&statusFlags);
while (statusFlags == 0)
{
hr = VP->ProcessInput(0, sample, 0);
hr = VP->GetOutputStatus(&statusFlags);
}
DWORD outputStatus = 0;
IMFSample* outputSample;
MFCreateSample(&outputSample);
MFT_OUTPUT_DATA_BUFFER outputBuffer = {};
outputBuffer.pSample = outputSample;
hr = VP->ProcessOutput(0, OutputInfo.cbSize, &outputBuffer, &outputStatus);
推荐答案
从Mediafoundation示例中，您具有LogMediaType函数
From Mediafoundation samples, you have LogMediaType function LogMediaType
您可以显示streamType2/streamType3的日志吗?
Can you show log for streamType2/streamType3.
还可以从此处视频格式属性
它可以帮助您找到缺少的属性.
It could help you to find the missing attribute.
EDIT1
因为我使用Windows 7，所以没有视频处理器MFT.我使用Color Converter DSP，他也可以进行色彩空间转换色彩转换器DSP
Beacause i use Windows Seven, i don't have the Video Processor MFT. I use Color Converter DSP, who can also do the color space conversion Color Converter DSP
IMFTransform* pVideoColorConverter = NULL;
IMFMediaType* pVideoOutputType = NULL;
// RGB24 media type at index 10, but can be different on your system
DWORD dwRGB24Index = 10;
hr = CoCreateInstance(CLSID_CColorConvertDMO, NULL, CLSCTX_INPROC_SERVER, IID_IMFTransform, reinterpret_cast<void**>(&pVideoColorConverter);
hr = pVideoColorConverter->SetInputType(0, pVideoInputType, 0);
hr = pVideoColorConverter->GetOutputAvailableType(0, dwRGB24Index, &pVideoOutputType);
hr = pVideoColorConverter->SetOutputType(0, pVideoOutputType, 0);
LogMediaType(pVideoInputType);
LogMediaType(pVideoOutputType);
这是日志:
pVideoInputType (my camera provide YUY2 not NV12) :
MF_MT_MAJOR_TYPE MFMediaType_Video
MF_MT_SUBTYPE MFVideoFormat_YUY2
MF_MT_FRAME_SIZE 640 x 480
MF_MT_DEFAULT_STRIDE 1280
MF_MT_ALL_SAMPLES_INDEPENDENT 1
MF_MT_FIXED_SIZE_SAMPLES 1
MF_MT_SAMPLE_SIZE 614400
MF_MT_AVG_BITRATE 147456000
MF_MT_FRAME_RATE 30 x 1
MF_MT_PIXEL_ASPECT_RATIO 1 x 1
MF_MT_INTERLACE_MODE 2
MF_MT_AM_FORMAT_TYPE {05589F80-C356-11CE-BF01-00AA0055595A}
MF_MT_FRAME_RATE_RANGE_MAX 128849018881
MF_MT_FRAME_RATE_RANGE_MIN 42949672961333333
pVideoOutputType :
MF_MT_MAJOR_TYPE MFMediaType_Video
MF_MT_SUBTYPE MFVideoFormat_RGB24
MF_MT_FRAME_SIZE 640 x 480
MF_MT_FRAME_RATE 10000000 x 333333
MF_MT_GEOMETRIC_APERTURE <<byte array>>
MF_MT_PIXEL_ASPECT_RATIO 1 x 1
MF_MT_INTERLACE_MODE 2
MF_MT_DEFAULT_STRIDE 1920
MF_MT_ALL_SAMPLES_INDEPENDENT 1
MF_MT_FIXED_SIZE_SAMPLES 1
MF_MT_SAMPLE_SIZE 921600
EDIT2
好的，您的视频输入类型似乎正确.
Ok, your video input type seems to be correct.
您现在可以记录所有VP的所有输入类型(streamType)了吗?
Can you now log all input type (streamType) from all VP :
DWORD* inputCount = new DWORD[MFTcount];
DWORD* outputCount = new DWORD[MFTcount];
for(DWORD i = 0; i < MFTcount; i++){
hr = VP[i]->GetStreamCount(&inputCount[i], &outputCount[i]);
if(FAILED(hr)){
exit(3);
}
}
DWORD** inputids = new DWORD*[MFTcount];
DWORD** outputids = new DWORD*[MFTcount];
for(DWORD i = 0; i < MFTcount; i++){
inputids[i] = new DWORD[inputCount[i]];
outputids[i] = new DWORD[outputCount[i]];
}
for(DWORD i = 0; i < MFTcount; i++){
hr = VP[i]->GetStreamIDs(inputCount[i], inputids[i], outputCount[i], outputids[i]);
// By convention, if an MFT has exactly one fixed input stream and one fixed output stream, it should assign the identifier 0 to both streams
if(hr == E_NOTIMPL && inputCount[i] == 1 && outputCount[i] == 1){
inputids[i][0] = 0;
outputids[i][0] = 0;
}
else if(FAILED(hr)){
exit(4);
}
}
for(DWORD i = 0; i < MFTcount; i++){
// todo : log VP = i
for(DWORD j = 0; j < inputCount[i]; j++){
// todo : log stream id = inputids[i][j]
DWORD dwTypeIndex = 0;
hr = S_OK;
while(hr == S_OK){
// todo :log dwTypeIndex
hr = VP[i]->GetInputAvailableType(inputids[i][j], dwTypeIndex, &streamType);
if(SUCCEEDED(hr)){
LogMediaType(streamType);
SAFE_RELEASE(streamType);
dwTypeIndex++;
}
else{
// todo : check hr, should be MF_E_NO_MORE_TYPES
// if hr == E_NOTIMPL/MF_E_INVALIDSTREAMNUMBER... should be error
}
}
}
}
EDIT3
这里是视频处理器MFT的使用: VideoProcessor MTF示例
Here is a use of the Video Processor MFT : VideoProcessor MTF sample
似乎您需要在使用D3DManager之前先提供它(HRESULT DX11VideoRenderer :: CPresenter :: CreateXVP(void)->第1118行)
It seems that you need to provide a D3DManager before using it (HRESULT DX11VideoRenderer::CPresenter::CreateXVP(void) -> line 1118)
我不确定D3DManager，因为我无法在系统上对其进行测试.
I'm not sure for the D3DManager because i can't test it on my system.
hr = CoCreateInstance(CLSID_VideoProcessorMFT, nullptr, CLSCTX_INPROC_SERVER, IID_IMFTransform, (void**)&m_pXVP);
if (FAILED(hr))
{
break;
}
hr = m_pXVP->ProcessMessage(MFT_MESSAGE_SET_D3D_MANAGER, ULONG_PTR(m_pDXGIManager));
if (FAILED(hr))
{
break;
}
// Tell the XVP that we are the swapchain allocator
hr = m_pXVP->GetAttributes(&pAttributes);
if (FAILED(hr))
{
break;
}
hr = pAttributes->SetUINT32(MF_XVP_PLAYBACK_MODE, TRUE);
if (FAILED(hr))
{
break;
}
从代码中检查所有m_pXVP和m_pXVPControl.
Check all m_pXVP and m_pXVPControl from the code.
EDIT4
根据您的inputType，尝试手动创建outputType，并添加以下属性:
According to your inputType, try to manually create the outputType, adding this attributes :
MF_MT_MAJOR_TYPE MFMediaType_Video
MF_MT_SUBTYPE MFVideoFormat_RGB24
MF_MT_FRAME_SIZE 1280 x 720
MF_MT_FIXED_SIZE_SAMPLES 1
MF_MT_FRAME_RATE 30 x 1
MF_MT_PIXEL_ASPECT_RATIO 1 x 1
MF_MT_ALL_SAMPLES_INDEPENDENT 1
MF_MT_INTERLACE_MODE 2
因此，在SetInputType之后，创建视频媒体输出类型，然后使用此新的mediaType调用SetOutputType.
So just after SetInputType, create the video media output type, and then call SetOutputType with this new mediaType.
您只能先尝试使用MF_MT_MAJOR_TYPE/MF_MT_SUBTYPE/MF_MT_FRAME_SIZE，然后再添加一个.
You can first try only with MF_MT_MAJOR_TYPE/MF_MT_SUBTYPE/MF_MT_FRAME_SIZE, and then add others, one by one.
这篇关于设置IMTransform视频处理器以进行色彩空间转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
