在 Windows 上捕获和显示实时摄像机内容 [英] Capturing and displaying live camera content on Windows

问题描述

我正在开发一个 Windows 应用程序，该应用程序能够显示高质量的视频源、录制视频或从中拍摄照片，并在以后进行编辑(最高可达 4K，在不久的将来可能会达到 8K).我目前有一个工作产品，使用 WPF (C#).为了捕获和显示视频，我使用了 AForge.NET 库.

I am developing a Windows application that is able to display a high-quality video feed, record it or take photos from it, and edit them later (up to 4K, in the near future maybe 8K). I currently have a working product, using WPF (C#). For capturing and displaying video, I used the AForge.NET library.

我的问题是应用程序真的很慢，主要的性能影响来自视频渲染.显然，唯一的方法是从 AForge 库中进行回调，每次可用时提供一个新框架.然后将该框架作为图像放置在 Image 元素中.我相信您可以看到性能下降的来源，尤其是高分辨率图像.

My problem is that the application is really slow, with the main performance hit coming from video rendering. Apparently the only way to do this, is to have a callback from the AForge library, providing a new frame every time one is available. That frame is then placed as an image inside an Image element. I believe you can see where the performance hit comes from, especially for high-res imagery.

我使用 WPF 和这些庞大的库的经验让我重新思考了我想如何编程；我不想制作因为速度慢而占用每个人时间的糟糕软件(我参考手工网络了解更多关于为什么?".

My experience with WPF and these enormous libraries has made me rethink how I want to program in general; I do not want to make bad software which takes up everyone's time by being slow (I refer to the Handmade network for more on "why?".

问题是，在 WPF C# 中，相机捕获和显示是地狱，但我似乎没有比其他任何地方更好(在 Windows 上).我的一个选择是主要使用 C++ 和 DirectShow.这是一个不错的解决方案，但在性能方面感觉已经过时，并且建立在 Microsoft 的 COM 系统上，我更愿意避免使用它.可以选择使用 Direct3D 使用硬件进行渲染，但 DirectShow 和 Direct3D 不能很好地配合使用.

The problem is, camera capture and display was hell in WPF C#, but I do not seem to be better of anywhere else (on Windows, that is). An option would be for me to use mostly C++ and DirectShow. This is an okay-ish solution, but feels outdated in terms of performance, and is built upon Microsoft's COM system, which I prefer to avoid. There are options to render with hardware using Direct3D, but DirectShow and Direct3D do not play nicely together.

我研究了其他应用程序是如何实现这一点的.VLC 使用 DirectShow，但这只能说明 DirectShow 存在较大的延迟.我认为这是因为 VLC 并非用于实时目的.OBS 工作室使用 QT 使用的任何东西，但我无法找到他们是如何做到的.OpenCV 抓取帧并将它们传送到屏幕上，根本没有效率，但这对 OpenCV 观众来说已经足够了.最后，来自 Windows 的集成网络摄像头应用程序.出于某种原因，这个应用程序能够实时记录和回放，而不会对性能造成很大的影响.我无法弄清楚他们是如何做到这一点的，我也没有找到任何其他解决方案可以实现与该工具相当的结果.

I have researched how other applications were able to achieve this. VLC uses DirectShow, but this only shows that DirectShow suffers from large latency. I assume this is because VLC was not intended for real-time purposes. OBS studio uses whatever QT uses, but I was unable to find how they do it. OpenCV grabs frames and blits them to the screen, not efficient at all, but that suffices for the OpenCV audience. Lastly, the integrated webcam app from Windows. For some reason this app is able to record and play back in real time, without a large performance hit. I was not able to figure out how they did this, nor did I find any other solution achieving comparable results to that tool.

TLDR;所以我的问题是:我将如何有效地捕获和渲染相机流，最好是硬件加速；是否可以在不通过 Directshow 的情况下在 Windows 上执行此操作?最后，当我希望它们实时处理 4K 素材时，我是否会要求很多商品设备?

TLDR; So my questions are: How would I go about efficiently capturing and rendering a camera stream, preferably hardware accelerated; Is it possible to do this on Windows without going through Directshow; And lastly, do I ask to much of commodity devices when I want them to process 4K footage in real-time?

我没有发现任何人以足以满足我需求的方式这样做；这让我同时感到绝望和内疚.我宁愿不要因为这个问题打扰 StackOverflow.

I have not found anyone doing this in a way that suffices my needs; this makes me feel both desperate and guilty at the same time. I would have preferred to not bother StackOverflow with this problem.

非常感谢您提供有关此主题的一般性回答或建议.

Many thanks in advance, for an answer, or advice on this topic in general.

推荐答案

您的问题是关于多种技术的组合:视频捕获、视频演示以及将两者连接在一起需要什么.

Your question is about combination of several technologies: video capture, video presentation and what it takes to connect the two together.

在 Windows 上有两个与视频相关的 API(如果我们不考虑古老的 VfW):DirectShow 和 Media Foundation.这两个 API 都有底层，它们大多是共享的，因此 DirectShow 和 Media Foundation 都提供类似的视频捕获功能和性能.这两个 API 都为您提供了良好的视频捕获延迟，而且相当低.就目前情况而言，不建议使用 DirectShow，因为该 API 已接近其生命周期，并且大部分已被废弃.同时，您可能会发现 DirectShow 的文档更完善、功能更丰富，并且提供了数量级更好的补充材料和第三方软件项目.您提到了一些库，它们都建立在上述技术之一(VfW、DirectShow、Media Foundation)之上，其实现质量不如原始操作系统 API.

On Windows there are two video related APIs (if we don't take ancient VfW into consideration): DirectShow and Media Foundation. Both APIs have underlying layers, which are mostly shared and for this reason both DirectShow and Media Foundation offer similar video capture capabilities and performance. Both APIs offer you good video capture latency, reasonably low. As things stand now use of DirectShow is not recommended since the API is at its end of life, and is mostly abandoned technology. Same time, you would probably find DirectShow better documented, more versatile and provided with orders of magnitude better supplementary materials and third party software items. You mentioned a few libraries and they all are built on top of one of the mentioned technologies (VfW, DirectShow, Media Foundation) with implementation quality inferior to original operating system API.

实际上，您可以使用两者之一来捕捉视频，最好使用 Media Foundation 作为当前技术.

Practically, you capture video with either of the two, preferably Media Foundation as the current technology.

在我看来，您问题中最重要的部分是如何组织视频渲染.在性能方面，利用硬件加速至关重要，在这种情况下，您的应用程序所构建的技术以及视频演示/嵌入的可用集成选项很重要.对于 .NET 桌面应用程序，您可能对将 Direct3D 11/12 与 .NET 混合或使用 MediaPlayerElement 控件以及研究如何将视频帧注入其中感兴趣.如上所述，即使第三方库可用，您也不应该期望它们以适当的方式解决问题.您有兴趣至少了解视频管道中的数据流.

In my opinion the most important part of your question is how to organize video rendering. Performance wise it is essential to take advantage of hardware acceleration and in this context it is important what technologies your application is built on and what are the available integration options for video presentation/embedding. For a .NET desktop application you would be interested in either mixing Direct3D 11/12 with .NET or using MediaPlayerElement control and a research how to inject video frames into it. As mentioned above even though third party libraries are available, you should not expect them to solve problems in an appropriate way. You are interested to at least understand the data flow in the video pipeline.

那么你有一个问题，如何连接视频捕获(不是由视频硬件加速)和硬件加速的视频渲染.这里可以有多种解决方案，但重要的是 DirectShow 对硬件加速的支持是有限的，并且在 Direct3D 9 中停止了它的发展，现在听起来已经过时了.这是告别这项 - 毫无疑问 - 优秀技术的另一个原因.您有兴趣研究将捕获的视频内容尽快放入 Direct3D 11/Direct3D 12/Direct2D 的选项，并利用当前的标准技术进行以下处理.实际技术可能取决于:它可以是 Media Foundation、Direct3D 11/12 或提到的 MediaPlayerElement 控件，以及一些其他选项，如 Direct2D，它们也不错.在获得出色或至少合理的性能的过程中，您有兴趣尽量减少使用第三方库(即使是流行的库)，即使它们的标题中有流行语.

Then you have a problem how to connect video capture (not accelerated by video hardware) and hardware accelerated video rendering. There can be multiple solutions here, but important is that DirectShow's support for hardware acceleration is limited and stopped its evolution at Direct3D 9, which sounds as outdated nowadays. This is one another reason to say farewell to this - no doubt - excellent piece of technology. You are interested in investigating your options in placing captured video content into Direct3D 11/Direct3D 12/Direct2D as soon as possible and utilize standard current technologies for the following processing. Actual technologies might depend: it can be Media Foundation, Direct3D 11/12 or mentioned MediaPlayerElement controls, as well as a few other options like Direct2D which are also good. On the way to great or at least reasonable performance you are interested to minimize use of third party - even though popular - libraries, even if they have buzz words in their titles.

可以实时捕获和处理 4K 实时镜头，但是您通常拥有专业的视频捕获硬件或压缩的内容，您应该使用硬件加速进行解压缩.

4K realtime footage can be captured and processed in real time, however you normally either have professional video capture hardware or the content is compressed which you are supposed to decompress with hardware acceleration.

这篇关于在 Windows 上捕获和显示实时摄像机内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！