Python3 在网络摄像头 fps 处处理和显示网络摄像头流 [英] Python3 process and display webcam stream at the webcams fps

问题描述

如何读取相机并以相机帧速率显示图像?

我想从我的网络摄像头连续读取图像(进行一些快速预处理)，然后在窗口中显示图像.这应该以我的网络摄像头提供的帧速率(29 fps)运行.OpenCV GUI 和 Tkinter GUI 似乎太慢了，无法以这样的帧速率显示图像.这些显然是我实验中的瓶颈.即使没有预处理，图像的显示速度也不够快.我使用的是 MacBook Pro 2018.

这是我尝试过的.始终使用 OpenCV 读取网络摄像头:

• 一切都发生在主线程中，图像使用 OpenCV 显示:12 fps
• 在单独的线程中读取相机并进行预处理，在主线程中使用 OpenCV 显示图像:20 fps
• 像上面一样多线程，但不显示图像:29 fps
• 像上面一样多线程，但使用 Tkinter 显示图像:不知道确切的 fps，但感觉就像 <10 fps.

代码如下:

单循环，OpenCV GUI:

导入 cv2导入时间定义主():帽 = cv2.VideoCapture(0)window_name = "FPS 单循环";cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)start_time = time.time()帧数 = 0seconds_to_measure = 10而 start_time + seconds_to_measure >时间.时间():成功，img = cap.read()img = img[:, ::-1] # 镜像time.sleep(0.01) # 模拟一些处理时间cv2.imshow(window_name, img)cv2.waitKey(1)帧数 = 帧数 + 1cv2.destroyAllWindows()打印(f在 {seconds_to_measure} 秒内捕获了 {frames}.FPS:{frames/seconds_to_measure}")如果 __name__ == __main__":主要的()

在 10 秒内捕获了 121.FPS:12.1

多线程，opencv gui:

导入日志导入时间从队列导入完整，队列从线程导入线程，事件导入 cv2logger = logging.getLogger(VideoStream")def setup_webcam_stream(src=0):帽 = cv2.VideoCapture(src)宽度，高度 = (cap.get(cv2.CAP_PROP_FRAME_WIDTH),cap.get(cv2.CAP_PROP_FRAME_HEIGHT),)logger.info(f相机尺寸:{宽度，高度}")logger.info(f"Camera FPS: {cap.get(cv2.CAP_PROP_FPS)}")抓取, frame = cap.read() # 读取一次以初始化如果没有抓住:引发 IOError(无法读取视频流.")返回上限def video_stream_loop(video_stream: cv2.VideoCapture, queue: Queue, stop_event: Event):而不是 stop_event.is_set():尝试:成功，img = video_stream.read()# 我们需要一个超时时间以免在没有从队列中检索到图像时卡住queue.put(img, timeout=1)除了完整:pass # 用更新的帧再试一次def processing_loop(input_queue: Queue, output_queue: Queue, stop_event: Event):而不是 stop_event.is_set():尝试:img = input_queue.get()img = img[:, ::-1] # 镜像time.sleep(0.01) # 模拟一些处理时间# 我们需要一个超时时间以免在没有从队列中检索到图像时卡住output_queue.put(img, timeout=1)除了完整:pass # 用更新的帧再试一次定义主():流 = setup_webcam_stream(0)网络摄像头队列 = 队列()处理队列 = 队列()stop_event = 事件()window_name = "FPS 多线程";cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)start_time = time.time()帧数 = 0seconds_to_measure = 10尝试:线(target=video_stream_loop, args=[stream, webcam_queue, stop_event]).开始()线(目标=processing_loop，args=[网络摄像头队列，已处理队列，停止事件]).开始()而 start_time + seconds_to_measure >时间.时间():img = processing_queue.get()cv2.imshow(window_name, img)cv2.waitKey(1)帧数 = 帧数 + 1最后:stop_event.set()cv2.destroyAllWindows()打印(f在 {seconds_to_measure} 秒内捕获了 {frames} 帧.FPS:{frames/seconds_to_measure}")打印(f网络摄像头队列:{webcam_queue.qsize()}")打印(f处理队列:{processed_queue.qsize()}")如果 __name__ == __main__":logging.basicConfig(level=logging.DEBUG)主要的()

INFO:VideoStream:Camera 尺寸:(1280.0, 720.0)信息:视频流:相机 FPS:29.000049在 10 秒内捕获了 209 帧.每秒帧数:20.9网络摄像头队列:0处理队列:82

在这里您可以看到第二个队列中还有剩余的图像，这些图像被提取用于显示.

当我取消注释这两行时:

cv2.imshow(window_name, img)cv2.waitKey(1)

那么输出是:

INFO:VideoStream:Camera 尺寸:(1280.0, 720.0)信息:视频流:相机 FPS:29.000049在 10 秒内捕获了 291 帧.每秒帧数:29.1网络摄像头队列:0已处理队列:0

因此它能够以网络摄像头的速度处理所有帧，而无需 GUI 显示它们.

多线程，Tkinter gui:

导入日志导入时间导入 tkinter从队列导入完整，队列，空从线程导入线程，事件进口PIL从 PIL 导入 ImageTk导入 cv2logger = logging.getLogger(VideoStream")def setup_webcam_stream(src=0):帽 = cv2.VideoCapture(src)宽度，高度 = cap.get(cv2.CAP_PROP_FRAME_WIDTH), cap.get(cv2.CAP_PROP_FRAME_HEIGHT)logger.info(f相机尺寸:{宽度，高度}")logger.info(f"Camera FPS: {cap.get(cv2.CAP_PROP_FPS)}")抓取, frame = cap.read() # 读取一次以初始化如果没有抓住:引发 IOError(无法读取视频流.")返回上限，宽度，高度def video_stream_loop(video_stream: cv2.VideoCapture, queue: Queue, stop_event: Event):而不是 stop_event.is_set():尝试:成功，img = video_stream.read()# 我们需要一个超时时间以免在没有从队列中检索到图像时卡住queue.put(img, timeout=1)除了完整:pass # 用更新的帧再试一次def processing_loop(input_queue: Queue, output_queue: Queue, stop_event: Event):而不是 stop_event.is_set():尝试:img = input_queue.get()img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = img[:, ::-1] # 镜像time.sleep(0.01) # 模拟一些处理时间# 我们需要一个超时时间以免在没有从队列中检索到图像时卡住output_queue.put(img, timeout=1)除了完整:pass # 用更新的帧再试一次类应用程序:def __init__(self, window, window_title, image_queue: Queue, image_dimensions: tuple):self.window = 窗口self.window.title(window_title)self.image_queue = image_queue# 创建一个可以适合上述视频源大小的画布self.canvas = tkinter.Canvas(window, width=image_dimensions[0], height=image_dimensions[1])self.canvas.pack()# 调用一次后，更新方法会每隔延迟毫秒自动调用一次self.delay = 1自我更新()self.window.mainloop()定义更新(自我):尝试:frame = self.image_queue.get(timeout=0.1) # 超时不会永远阻塞这个方法self.photo = ImageTk.PhotoImage(image=PIL.Image.fromarray(frame))self.canvas.create_image(0, 0, image=self.photo, anchor=tkinter.NW)self.window.after(self.delay, self.update)除了空:pass # 下次再试定义主():流，宽度，高度 = setup_webcam_stream(0)网络摄像头队列 = 队列()处理队列 = 队列()stop_event = 事件()window_name = "FPS 多线程";尝试:线程(目标=video_stream_loop，args=[stream，webcam_queue，stop_event]).开始()线程(目标=processing_loop，args=[网络摄像头队列，已处理队列，停止事件]).开始()App(tkinter.Tk(), window_name,processed_queue, (width, height))最后:stop_event.set()打印(f网络摄像头队列:{webcam_queue.qsize()}")打印(f处理队列:{processed_queue.qsize()}")如果 __name__ == __main__":logging.basicConfig(level=logging.DEBUG)主要的()

INFO:VideoStream:Camera 尺寸:(1280.0, 720.0)信息:视频流:相机 FPS:29.000049网络摄像头队列:0处理队列:968

解决方案

在这个答案中，我分享了关于 camera FPS VS display FPS 的一些注意事项以及一些演示的代码示例:

• FPS 计算的基础知识；
• 如何将显示 FPS 从 29 fps 提高到 300+ fps；
• 如何有效地使用threading和queue以最接近相机支持的最大fps进行捕捉；

对于遇到您的问题的任何人，以下是需要首先回答的几个重要问题:

• 正在捕获的图像的大小是多少?
• 您的网络摄像头支持多少 FPS?(相机 FPS)
• 从网络摄像头抓取帧并将其显示在窗口中的速度有多快?(显示 FPS)

相机 FPS VS 显示器 FPS

相机 fps 是指相机硬件的能力.例如，ffmpeg 告诉我，在 640x480 分辨率下，我的相机可以返回最低 15 fps，最高 30 fps，以及其他格式:

ffmpeg -list_devices true -f dshow -i dummyffmpeg -f dshow -list_options true -i video=HP HD Camera"[dshow @ 00000220181cc600] vcodec=mjpeg min s=640x480 fps=15 max s=640x480 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=320x180 fps=15 max s=320x180 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=320x240 fps=15 max s=320x240 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=424x240 fps=15 max s=424x240 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=640x360 fps=15 max s=640x360 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=848x480 fps=15 max s=848x480 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=960x540 fps=15 max s=960x540 fps=30[dshow @ 00000220181cc600] vcodec=mjpeg min s=1280x720 fps=15 max s=1280x720 fps=30

这里的重要认识是，尽管能够在内部捕获 30 fps，但不能保证应用程序能够在一秒钟内从相机中提取这 30 帧.这背后的原因将在以下部分阐明.

显示 fps 是指每秒可以在一个窗口中绘制多少张图像.这个数字完全不受相机的限制，它通常比相机的 fps 高得多.正如您稍后将看到的，可以创建和应用程序每秒从相机中提取 29 个图像并每秒绘制 300 多次.这意味着在从相机中提取下一帧之前，来自相机的相同图像在一个窗口中多次绘制.

我的网络摄像头可以捕获多少 FPS?

以下应用程序仅演示了如何打印相机使用的默认设置(大小、fps)以及如何从中检索帧、将其显示在窗口中并计算渲染的 FPS 量:

将 numpy 导入为 np导入 cv2导入日期时间定义主():# 创建显示窗口cv2.namedWindow(网络摄像头"，cv2.WINDOW_NORMAL)# 初始化网络摄像头捕获对象帽 = cv2.VideoCapture(0)# 检索捕获对象的属性cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)cap_fps = cap.get(cv2.CAP_PROP_FPS)fps_sleep = int(1000/cap_fps)print('* 捕获宽度:', cap_width)print('* 捕获高度:', cap_height)print('* Capture FPS:', cap_fps, '理想的帧间等待时间:', fps_sleep, 'ms')# 初始化时间和帧计数变量last_time = datetime.datetime.now()帧数 = 0# 主循环:从相机中检索并显示一帧而(真):# 阻塞直到读取整个帧成功，img = cap.read()帧 += 1# 计算 fps: current_time - last_timedelta_time = datetime.datetime.now() - last_timeelapsed_time = delta_time.total_seconds()cur_fps = np.around(frames/elapsed_time, 1)# 绘制 FPS 文本并显示图像cv2.putText(img, 'FPS:' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)cv2.imshow(网络摄像头", img)# 等待 1ms 等待 ESC 被按下键 = cv2.waitKey(1)如果(键== 27):休息#释放资源cv2.destroyAllWindows()cap.release()如果 __name__ == __main__":主要的()

输出:

* 捕获宽度:640.0* 捕获高度:480.0* 捕获 FPS:30.0 帧之间的等待时间:33 毫秒

如前所述，默认情况下，我的相机能够以 30 fps 的速度捕获 640x480 图像，尽管上面的循环非常简单，但我的显示 FPS 较低:我只能检索帧并以 28 或 29 fps 的速度显示它们，并且中间无需执行任何自定义图像处理.怎么回事?

实际情况是，即使循环看起来很简单，但在幕后发生的事情却花费了足够的处理时间，使得循环的一次迭代难以在不到 33 毫秒的时间内发生:

• cap.read() 执行对相机驱动程序的 I/O 调用以提取新数据.此功能会阻止您的应用程序的执行，直到数据完全传输完毕；
• 需要使用新像素设置一个 numpy 数组；
• 需要其他调用来显示窗口并在其中绘制像素，即cv2.imshow()，通常操作较慢；
• 由于 cv2.waitKey(1) 需要保持窗口打开，因此还有 1 毫秒的延迟；

所有这些操作，尽管很小，但使应用程序很难调用 cap.read()、获取新帧并以精确的 30 fps 显示它.

您可以尝试多种方法来加速应用程序，以便能够显示比相机驱动程序允许的更多的帧和 这篇文章 很好地涵盖了它们.请记住这一点:您将无法从相机捕获更多帧，而不是驱动程序支持的帧数.但是，您将能够显示更多帧.

如何将显示 FPS 提高到 300+?线程 示例.

用于增加每秒显示的图像数量的方法之一依赖于 threading 包来创建一个单独的线程来连续从相机中提取帧.发生这种情况是因为应用程序的主循环不再阻塞在 cap.read() 上等待它返回一个新的帧，从而增加了每个可以显示(或绘制)的帧数第二个.

注意:这种方法在一个窗口上多次渲染相同的图像，直到检索到来自相机的下一个图像.请记住，它甚至可能会在其内容仍在使用来自相机的新数据进行更新时绘制图像.

以下应用程序只是一个学术示例，不是我推荐的生产代码，用于增加窗口中每秒显示的帧数:

将 numpy 导入为 np导入 cv2导入日期时间从线程导入线程# 全局变量stop_thread = False # 控制线程执行img = None # 存储相机检索到的图像def start_capture_thread(cap):全局 img，stop_thread# 不断从相机中读取声望为真:_, img = cap.read()如果(停止线程):休息定义主():全局 img，stop_thread# 创建显示窗口cv2.namedWindow(网络摄像头"，cv2.WINDOW_NORMAL)# 初始化网络摄像头捕获对象帽 = cv2.VideoCapture(0)# 检索捕获对象的属性cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)cap_fps = cap.get(cv2.CAP_PROP_FPS)fps_sleep = int(1000/cap_fps)print('* 捕获宽度:', cap_width)print('* 捕获高度:', cap_height)print('* 捕获 FPS:', cap_fps, '帧之间的等待时间:', fps_sleep)# 启动捕获线程:从相机中读取帧(不间断)并将结果存储在 img 中t = Thread(target=start_capture_thread, args=(cap,), daemon=True) # 当应用程序退出时，一个守护线程被杀死t.start()# 初始化时间和帧计数变量last_time = datetime.datetime.now()帧数 = 0cur_fps = 0而(真):# 阻塞直到读取整个帧帧 += 1# 测量运行时间:current_time - last_timedelta_time = datetime.datetime.now() - last_timeelapsed_time = delta_time.total_seconds()# 计算 fps 但避免除以零如果(elapsed_time ！= 0):cur_fps = np.around(frames/elapsed_time, 1)# TODO:复制图像并在需要时在此处处理# 绘制 FPS 文本并显示图像如果(img 不是 None):cv2.putText(img, 'FPS:' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)cv2.imshow(网络摄像头", img)# 等待 1ms 等待 ESC 被按下键 = cv2.waitKey(1)如果(键== 27):stop_thread = 真休息#释放资源cv2.destroyAllWindows()cap.release()如果 __name__ == __main__":主要的()

如何以相机支持的最接近的最大 fps 进行拍摄?threading 和 queue 示例.

使用 queue 的问题在于，在性能方面，您得到的结果取决于应用程序每秒可以从相机提取多少帧.如果相机支持 30 fps，那么只要正在执行的图像处理操作很快，您的应用程序就可以获得.否则，显示的帧数(每秒)会下降，队列的大小将缓慢增加，直到所有 RAM 内存耗尽.为避免该问题，请确保为 queueSize 设置一个数字，以防止队列增长超出您的操作系统可以处理的范围.

下面的代码是一个简单的实现，它创建了一个专用的线程来从相机中抓取帧并将它们放入一个队列，稍后由主循环使用应用程序:

将 numpy 导入为 np导入 cv2导入日期时间导入队列从线程导入线程# 全局变量stop_thread = False # 控制线程执行def start_capture_thread(cap, queue):全局停止线程# 不断从相机中读取声望为真:_, img = cap.read()queue.put(img)如果(停止线程):休息定义主():全局停止线程# 创建显示窗口cv2.namedWindow(网络摄像头"，cv2.WINDOW_NORMAL)# 初始化网络摄像头捕获对象帽 = cv2.VideoCapture(0)#cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)# 检索捕获对象的属性cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)cap_fps = cap.get(cv2.CAP_PROP_FPS)print('* 捕获宽度:', cap_width)print('* 捕获高度:', cap_height)print('* 捕获 FPS:', cap_fps)# 创建队列frame_queue = queue.Queue(maxsize=0)# 启动捕获线程:从相机中读取帧(不间断)并将结果存储在 img 中t = Thread(target=start_capture_thread, args=(cap, frames_queue,), daemon=True) # 应用程序退出时一个守护线程被杀死t.start()# 初始化时间和帧计数变量last_time = datetime.datetime.now()帧数 = 0cur_fps = 0而(真):如果(frames_queue.empty()):继续# 阻塞直到读取整个帧帧 += 1# 测量运行时间:current_time - last_timedelta_time = datetime.datetime.now() - last_timeelapsed_time = delta_time.total_seconds()# 计算 fps 但避免除以零如果(elapsed_time ！= 0):cur_fps = np.around(frames/elapsed_time, 1)# 从队列中检索图像img = frames_queue.get()# TODO:如果需要，在此处处理图像# 绘制 FPS 文本并显示图像如果(img 不是 None):cv2.putText(img, 'FPS:' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)cv2.imshow(网络摄像头", img)# 等待 1ms 等待 ESC 被按下键 = cv2.waitKey(1)如果(键== 27):stop_thread = 真休息#释放资源cv2.destroyAllWindows()cap.release()如果 __name__ == __main__":主要的()

之前我说过可能，这就是我的意思:即使我使用专用的线程从相机和队列中提取帧> 为了存储它们，显示的 fps 仍被限制为 29.3，而本应为 30 fps.在这种情况下，我假设 VideoCapture 使用的相机驱动程序或后端实现可以归咎于这个问题.在 Windows 上，默认使用的后端是 MSMF.

可以通过在构造函数上传递正确的参数来强制 VideoCapture 使用不同的后端:

cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)

我对 DShow 的体验很糟糕:从相机返回的 CAP_PROP_FPS 是 0 并且显示的 FPS 卡在 14 左右.这只是一个示例，用于说明后端捕获驱动程序如何对相机捕获产生负面影响.

但这是你可以探索的东西.也许在您的操作系统上使用不同的后端可以提供更好的结果.这是一个很好的 来自 OpenCV 的视频 I/O 模块的高级概述 列出了支持的后端:

更新

在此答案的其中一条评论中，OP 在 Mac OS 上将 OpenCV 4.1 升级到 4.3，并观察到 ​​FPS 渲染的显着改进.看起来这是一个与 cv2.imshow() 相关的性能问题.

How can I read a camera and display the images at the cameras frame rate?

I want to continuously read images from my webcam, (do some fast preprocessing) and then display the image in a window. This should run at the frame rate, that my webcam provides (29 fps). It seems like the OpenCV GUI and Tkinter GUI is too slow, to display images at such a frame rate. These are clearly the bottlenecks in my experiments. Even without the preprocessing, the images are not displayed fast enough. I am on a MacBook Pro 2018.

Here is what I tried. The webcam is always read with OpenCV:

• Everything happens in the main thread, the images are displayed with OpenCV: 12 fps
• Read camera and do preprocessing in separate threads, show image with OpenCV in the main thread: 20 fps
• multithreaded like above, but do not show the image: 29 fps
• multithreaded like above, but show the images with Tkinter: don't know the exact fps but it feels like <10 fps.

Here is the code:

Single loop, OpenCV GUI:

import cv2
import time


def main():
    cap = cv2.VideoCapture(0)
    window_name = "FPS Single Loop"
    cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)

    start_time = time.time()
    frames = 0

    seconds_to_measure = 10
    while start_time + seconds_to_measure > time.time():
        success, img = cap.read()
        img = img[:, ::-1]  # mirror
        time.sleep(0.01)  # simulate some processing time
        cv2.imshow(window_name, img)
        cv2.waitKey(1)
        frames = frames + 1

    cv2.destroyAllWindows()

    print(
        f"Captured {frames} in {seconds_to_measure} seconds. FPS: {frames/seconds_to_measure}"
    )


if __name__ == "__main__":
    main()

Captured 121 in 10 seconds. FPS: 12.1

Multithreaded, opencv gui:

import logging
import time
from queue import Full, Queue
from threading import Thread, Event

import cv2

logger = logging.getLogger("VideoStream")


def setup_webcam_stream(src=0):
    cap = cv2.VideoCapture(src)
    width, height = (
        cap.get(cv2.CAP_PROP_FRAME_WIDTH),
        cap.get(cv2.CAP_PROP_FRAME_HEIGHT),
    )
    logger.info(f"Camera dimensions: {width, height}")
    logger.info(f"Camera FPS: {cap.get(cv2.CAP_PROP_FPS)}")
    grabbed, frame = cap.read()  # Read once to init
    if not grabbed:
        raise IOError("Cannot read video stream.")
    return cap


def video_stream_loop(video_stream: cv2.VideoCapture, queue: Queue, stop_event: Event):
    while not stop_event.is_set():
        try:
            success, img = video_stream.read()
            # We need a timeout here to not get stuck when no images are retrieved from the queue
            queue.put(img, timeout=1)
        except Full:
            pass  # try again with a newer frame


def processing_loop(input_queue: Queue, output_queue: Queue, stop_event: Event):
    while not stop_event.is_set():
        try:
            img = input_queue.get()
            img = img[:, ::-1]  # mirror
            time.sleep(0.01)  # simulate some processing time
            # We need a timeout here to not get stuck when no images are retrieved from the queue
            output_queue.put(img, timeout=1)
        except Full:
            pass  # try again with a newer frame


def main():
    stream = setup_webcam_stream(0)
    webcam_queue = Queue()
    processed_queue = Queue()
    stop_event = Event()
    window_name = "FPS Multi Threading"
    cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)

    start_time = time.time()
    frames = 0

    seconds_to_measure = 10
    try:
        Thread(
            target=video_stream_loop, args=[stream, webcam_queue, stop_event]
        ).start()
        Thread(
            target=processing_loop, args=[webcam_queue, processed_queue, stop_event]
        ).start()
        while start_time + seconds_to_measure > time.time():
            img = processed_queue.get()
            cv2.imshow(window_name, img)
            cv2.waitKey(1)
            frames = frames + 1
    finally:
        stop_event.set()

    cv2.destroyAllWindows()

    print(
        f"Captured {frames} frames in {seconds_to_measure} seconds. FPS: {frames/seconds_to_measure}"
    )
    print(f"Webcam queue: {webcam_queue.qsize()}")
    print(f"Processed queue: {processed_queue.qsize()}")


if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)
    main()

INFO:VideoStream:Camera dimensions: (1280.0, 720.0)
INFO:VideoStream:Camera FPS: 29.000049
Captured 209 frames in 10 seconds. FPS: 20.9
Webcam queue: 0
Processed queue: 82

Here you can see that there are images remaining in the second queue where the images get fetched for displaying them.

When I uncomment these two lines:

cv2.imshow(window_name, img)
cv2.waitKey(1)

then the output is:

INFO:VideoStream:Camera dimensions: (1280.0, 720.0)
INFO:VideoStream:Camera FPS: 29.000049
Captured 291 frames in 10 seconds. FPS: 29.1
Webcam queue: 0
Processed queue: 0

So it is able to process all frames at the webcams speed without a GUI displaying them.

Multithreaded, Tkinter gui:

import logging
import time
import tkinter
from queue import Full, Queue, Empty
from threading import Thread, Event

import PIL
from PIL import ImageTk
import cv2

logger = logging.getLogger("VideoStream")


def setup_webcam_stream(src=0):
    cap = cv2.VideoCapture(src)
    width, height = cap.get(cv2.CAP_PROP_FRAME_WIDTH), cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
    logger.info(f"Camera dimensions: {width, height}")
    logger.info(f"Camera FPS: {cap.get(cv2.CAP_PROP_FPS)}")
    grabbed, frame = cap.read()  # Read once to init
    if not grabbed:
        raise IOError("Cannot read video stream.")
    return cap, width, height


def video_stream_loop(video_stream: cv2.VideoCapture, queue: Queue, stop_event: Event):
    while not stop_event.is_set():
        try:
            success, img = video_stream.read()
            # We need a timeout here to not get stuck when no images are retrieved from the queue
            queue.put(img, timeout=1)
        except Full:
            pass  # try again with a newer frame


def processing_loop(input_queue: Queue, output_queue: Queue, stop_event: Event):
    while not stop_event.is_set():
        try:
            img = input_queue.get()
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
            img = img[:, ::-1]  # mirror
            time.sleep(0.01)  # simulate some processing time
            # We need a timeout here to not get stuck when no images are retrieved from the queue
            output_queue.put(img, timeout=1)
        except Full:
            pass  # try again with a newer frame


class App:
    def __init__(self, window, window_title, image_queue: Queue, image_dimensions: tuple):
        self.window = window
        self.window.title(window_title)

        self.image_queue = image_queue

        # Create a canvas that can fit the above video source size
        self.canvas = tkinter.Canvas(window, width=image_dimensions[0], height=image_dimensions[1])
        self.canvas.pack()

        # After it is called once, the update method will be automatically called every delay milliseconds
        self.delay = 1
        self.update()

        self.window.mainloop()

    def update(self):
        try:
            frame = self.image_queue.get(timeout=0.1)  # Timeout to not block this method forever
            self.photo = ImageTk.PhotoImage(image=PIL.Image.fromarray(frame))
            self.canvas.create_image(0, 0, image=self.photo, anchor=tkinter.NW)
            self.window.after(self.delay, self.update)
        except Empty:
            pass  # try again next time


def main():
    stream, width, height = setup_webcam_stream(0)
    webcam_queue = Queue()
    processed_queue = Queue()
    stop_event = Event()
    window_name = "FPS Multi Threading"

    try:
        Thread(target=video_stream_loop, args=[stream, webcam_queue, stop_event]).start()
        Thread(target=processing_loop, args=[webcam_queue, processed_queue, stop_event]).start()
        App(tkinter.Tk(), window_name, processed_queue, (width, height))
    finally:
        stop_event.set()

    print(f"Webcam queue: {webcam_queue.qsize()}")
    print(f"Processed queue: {processed_queue.qsize()}")


if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)
    main()

INFO:VideoStream:Camera dimensions: (1280.0, 720.0)
INFO:VideoStream:Camera FPS: 29.000049
Webcam queue: 0
Processed queue: 968

解决方案

On this answer I share some considerations on camera FPS VS display FPS and some code examples that demonstrates:

• The basics on FPS calculation;
• How to increase the display FPS from 29 fps to 300+ fps;
• How to use threading and queue efficiently to capture at the closest maximum fps supported by the camera;

For anyone going through your issue, here is a couple of important questions that need to be answered first:

• What's the size of the images being captured?
• How many FPS does your webcam support? (camera FPS)
• How fast can you grab a frame from the webcam and display it in a window? (display FPS)

Camera FPS VS Display FPS

The camera fps refers to what the hardware of the camera is capable of. For instance, ffmpeg tells that at 640x480 my camera can return 15 fps minimum and 30 at maximum, among other formats:

ffmpeg -list_devices true -f dshow -i dummy
ffmpeg -f dshow -list_options true -i video="HP HD Camera"

[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=640x480 fps=15 max s=640x480 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=320x180 fps=15 max s=320x180 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=320x240 fps=15 max s=320x240 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=424x240 fps=15 max s=424x240 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=640x360 fps=15 max s=640x360 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=848x480 fps=15 max s=848x480 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=960x540 fps=15 max s=960x540 fps=30
[dshow @ 00000220181cc600]   vcodec=mjpeg  min s=1280x720 fps=15 max s=1280x720 fps=30

The important realization here is that despite being able to capture 30 fps internally, there is NO guarantee that an application will be able to pull those 30 frames from the camera in a second. The reasons behind this are clarified on the following sections.

The display fps refers to how many images can be draw in a window per second. This number is not limited by the camera at all and its usually much much higher than the camera fps. As you'll see later, its possible to create and application that pulls 29 images per second from the camera and draws them more than 300 times a second. That means that the same image from the camera is drawn multiple times in a window before the next frame is pulled from the camera.

How many FPS can my webcam capture?

The following application simply demonstrates how to print the default settings used by the camera (size, fps) and how to retrieve frames from it, display it in a window and compute the amount of FPS being rendered:

import numpy as np
import cv2
import datetime
    
def main():
    # create display window
    cv2.namedWindow("webcam", cv2.WINDOW_NORMAL)

    # initialize webcam capture object
    cap = cv2.VideoCapture(0)

    # retrieve properties of the capture object
    cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
    cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
    cap_fps = cap.get(cv2.CAP_PROP_FPS)
    fps_sleep = int(1000 / cap_fps)
    print('* Capture width:', cap_width)
    print('* Capture height:', cap_height)
    print('* Capture FPS:', cap_fps, 'ideal wait time between frames:', fps_sleep, 'ms')

    # initialize time and frame count variables
    last_time = datetime.datetime.now()
    frames = 0

    # main loop: retrieves and displays a frame from the camera
    while (True):
        # blocks until the entire frame is read
        success, img = cap.read()
        frames += 1

        # compute fps: current_time - last_time
        delta_time = datetime.datetime.now() - last_time
        elapsed_time = delta_time.total_seconds()
        cur_fps = np.around(frames / elapsed_time, 1)

        # draw FPS text and display image
        cv2.putText(img, 'FPS: ' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
        cv2.imshow("webcam", img)

        # wait 1ms for ESC to be pressed
        key = cv2.waitKey(1)
        if (key == 27):
            break

    # release resources
    cv2.destroyAllWindows()
    cap.release()


if __name__ == "__main__":
    main()

Output:

* Capture width: 640.0
* Capture height: 480.0
* Capture FPS: 30.0 wait time between frames: 33 ms

As mentioned earlier, my camera is able to capture 640x480 images at 30 fps by default and even though the loop above is pretty simple, my display FPS is lower: I'm only able to retrieve frames and display them at 28 or 29 fps and that's without performing any custom image processing in between. What's going on?

The reality is that even though the loop looks pretty simple, there are things happening under the hood that costs just enough processing time to make it difficult for one iteration of the loop to happen in less than 33ms:

• cap.read() executes I/O calls to the camera driver in order to pull the new data. This function blocks execution of your application until the data has been transferred completely;
• a numpy array needs to be setup with the new pixels;
• other calls are required to display a window and draw the pixels in it, namely cv2.imshow(), which is usually slow operation;
• there's also a 1ms delay thanks to cv2.waitKey(1) which is required to keep the window opened;

All of these operations, as small as they are, make it incredibly difficult for an application to call cap.read(), get a new frame and display it at precisely 30 fps.

There's a number of things you can try to speed up the application to be able to display more frames than the camera driver allows and this post covers them well. Just remember this: you won't be able to capture more frames from the camera than what the driver says it supports. You will, however, be able to display more frames.

How to increase the display FPS to 300+? A threading example.

One of the approaches used to increase the amount of images being displayed per second relies on the threading package to create a separate thread to continuously pull frames from the camera. This happens because the main loop of the application is not blocked on cap.read() anymore waiting for it to return a new frame, thus increasing the number of frames that can be displayed (or draw) per second.

Note: this approach renders the same image multiple times on a window until the next image from the camera is retrieved. Keep in mind that it might even draw an image while it's contents are still being updated with new data from the camera.

The following application is just an academic example, not something I recommend as production code, to increase the amount of frames per second that are display in a window:

import numpy as np
import cv2
import datetime
from threading import Thread

# global variables
stop_thread = False             # controls thread execution
img = None                      # stores the image retrieved by the camera


def start_capture_thread(cap):
    global img, stop_thread

    # continuously read fames from the camera
    while True:
        _, img = cap.read()

        if (stop_thread):
            break


def main():
    global img, stop_thread

    # create display window
    cv2.namedWindow("webcam", cv2.WINDOW_NORMAL)

    # initialize webcam capture object
    cap = cv2.VideoCapture(0)

    # retrieve properties of the capture object
    cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
    cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
    cap_fps = cap.get(cv2.CAP_PROP_FPS)
    fps_sleep = int(1000 / cap_fps)
    print('* Capture width:', cap_width)
    print('* Capture height:', cap_height)
    print('* Capture FPS:', cap_fps, 'wait time between frames:', fps_sleep)

    # start the capture thread: reads frames from the camera (non-stop) and stores the result in img
    t = Thread(target=start_capture_thread, args=(cap,), daemon=True) # a deamon thread is killed when the application exits
    t.start()

    # initialize time and frame count variables
    last_time = datetime.datetime.now()
    frames = 0
    cur_fps = 0

    while (True):
        # blocks until the entire frame is read
        frames += 1

        # measure runtime: current_time - last_time
        delta_time = datetime.datetime.now() - last_time
        elapsed_time = delta_time.total_seconds()

        # compute fps but avoid division by zero
        if (elapsed_time != 0):
            cur_fps = np.around(frames / elapsed_time, 1)

        # TODO: make a copy of the image and process it here if needed

        # draw FPS text and display image
        if (img is not None):
            cv2.putText(img, 'FPS: ' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
            cv2.imshow("webcam", img)

        # wait 1ms for ESC to be pressed
        key = cv2.waitKey(1)
        if (key == 27):
            stop_thread = True
            break

    # release resources
    cv2.destroyAllWindows()
    cap.release()


if __name__ == "__main__":
    main()

How to capture at the closest maximum fps supported by the camera? A threading and queue example.

The problem of using a queue is that, performance-wise, what you get depends on how many frames per second the application can pull from the camera. If the camera supports 30 fps then that's what your application might get as long as the image processing operations being done are fast. Otherwise, there will be a drop in the number of frames being displayed (per second) and the size of the queue will slowly increase until all your RAM memory runs out. To avoid that problem, make sure to set queueSize with a number that prevents the queue from growing beyond what your OS can handle.

The following code is a naive implementation that creates a dedicated thread to grab frames from the camera and puts them in a queue that is later used by the main loop of the application:

import numpy as np
import cv2
import datetime
import queue
from threading import Thread

# global variables
stop_thread = False             # controls thread execution


def start_capture_thread(cap, queue):
    global stop_thread

    # continuously read fames from the camera
    while True:
        _, img = cap.read()
        queue.put(img)

        if (stop_thread):
            break


def main():
    global stop_thread

    # create display window
    cv2.namedWindow("webcam", cv2.WINDOW_NORMAL)

    # initialize webcam capture object
    cap = cv2.VideoCapture(0)
    #cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)

    # retrieve properties of the capture object
    cap_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
    cap_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
    cap_fps = cap.get(cv2.CAP_PROP_FPS)
    print('* Capture width:', cap_width)
    print('* Capture height:', cap_height)
    print('* Capture FPS:', cap_fps)

    # create a queue
    frames_queue = queue.Queue(maxsize=0)

    # start the capture thread: reads frames from the camera (non-stop) and stores the result in img
    t = Thread(target=start_capture_thread, args=(cap, frames_queue,), daemon=True) # a deamon thread is killed when the application exits
    t.start()

    # initialize time and frame count variables
    last_time = datetime.datetime.now()
    frames = 0
    cur_fps = 0

    while (True):
        if (frames_queue.empty()):
            continue

        # blocks until the entire frame is read
        frames += 1

        # measure runtime: current_time - last_time
        delta_time = datetime.datetime.now() - last_time
        elapsed_time = delta_time.total_seconds()

        # compute fps but avoid division by zero
        if (elapsed_time != 0):
            cur_fps = np.around(frames / elapsed_time, 1)

        # retrieve an image from the queue
        img = frames_queue.get()

        # TODO: process the image here if needed

        # draw FPS text and display image
        if (img is not None):
            cv2.putText(img, 'FPS: ' + str(cur_fps), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
            cv2.imshow("webcam", img)

        # wait 1ms for ESC to be pressed
        key = cv2.waitKey(1)
        if (key == 27):
            stop_thread = True
            break

    # release resources
    cv2.destroyAllWindows()
    cap.release()


if __name__ == "__main__":
    main()

Earlier I said might and here is what I meant: even when I use a dedicated thread to pull frames from the camera and a queue to store them, the displayed fps is still capped to 29.3 when it should have been 30 fps. In this case, I assume that the camera driver or the backend implementation used by VideoCapture can be blamed for the issue. On Windows, the backend used by default is MSMF.

It is possible to force VideoCapture to use a different backend by passing the right arguments on the constructor:

cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)

My experience with DShow was terrible: the returned CAP_PROP_FPS from the camera was 0 and the displayed FPS got stuck around 14. This is just an example to illustrate how the backend capture driver can interfere negatively with the camera capture.

But that's something you can explore. Maybe using a different backend on your OS can provide better results. Here's a nice high-level overview of the Video I/O module from OpenCV that lists the supported backends:

Update

In one of the comments of this answer, the OP upgraded OpenCV 4.1 to 4.3 on Mac OS and observed a noticeable improvement on FPS rendering. It looks like it was a performance issue related to cv2.imshow().

这篇关于Python3 在网络摄像头 fps 处处理和显示网络摄像头流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！