Python Opencv 直播桌面截屏流程 [英] Python Opencv live desktop screen process

问题描述

我正在尝试处理实时屏幕.有一个关于捕鱼的游戏.当鱼在圆圈中时，你必须点击它.我想我可以用opencv处理我的屏幕，找到鱼并用pyautogui点击它.

I'm trying to process live screen. There is a game about catching fish. You have to click on fish when it is in circle. I think that I can process my screen with opencv, find fish and click on it with pyautogui.

我做到了，但问题是程序不够快，无法点击.顺便说一下，游戏是 Metin 2 mmorpg 中的迷你游戏.这就像一个黑客或机器人，但我只是想知道我是否可以这样做.

I did it but problem is program not fast enough to click. By the way game is a mini game in Metin 2 mmorpg. It is like a hack or bot but I just wondering if can I do that.

这是我的代码:

import numpy as np
import cv2
from PIL import ImageGrab
import pyautogui
import time

while True:

    img=ImageGrab.grab(bbox=(341,208,430,290))
    img_np=np.array(img)
    #gray=cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)

    #lower=np.array([57,91,120])
    #upper=np.array([65,95,160])

    #mask=cv2.inRange(gray,95,130)

    #sonuc=cv2.bitwise_and(gray,gray,mask=mask)
    #cv2.imshow('frame',mask)


    degsk=np.argwhere(img_np==[123,90,57])
    if len(degsk)!=0:
        #print(degsk)
        yerx=341+degsk[int(len(degsk)/2),1]
        yery=208+degsk[int(len(degsk)/2),0]
        #pyautogui.click(x=yerx, y=yery)
        time.sleep(0.8)

    if cv2.waitKey(1)&0xFF==ord('q'):
        break

cv2.destroyAllWindows()

如您所见，首先我尝试屏蔽屏幕，但我意识到这不是必需的，所以我找到了鱼的 BGR 值并编程以在 numpy 数组中找到它，而不是在数组中取值中间，然后使用鼠标移动功能.正如我所说，这还不够快，无法钓到鱼.

As you can see, first I tried mask the screen than I realise that it is not necessary so I found BGR value of fish and programed to find it in numpy array than I took value middle in the array and than i used mouse move function. As I said this is not fast enough to catch fish.

所以该程序正在运行，但延迟捕获鱼.我怎样才能使这个程序更快?

So the program is working but delayed for catch fish. How can I make faster this program?

这里的游戏画面

推荐答案

我发现 IPython 最适合计时，所以如果你启动 IPython 并粘贴以下代码:

I find IPython best for timing things, so if you start IPython and paste in the following code:

from PIL import ImageGrab

img=ImageGrab.grab(bbox=(341,208,430,290)) 

然后您可以使用以下语句计时:

You can then time a statement with:

%timeit img=ImageGrab.grab(bbox=(341,208,430,290))

我明白了:

552 ms ± 5.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

所以，抓取屏幕需要超过 500 毫秒，所以你只会得到不到 2 帧/秒 - 甚至没有处理它.

So, grabbing the screen takes over 500ms, so you are only going to get under 2 frames/second - without even processing it.

如果你想更快地抓取屏幕，我建议 ffmpeg.我将它安装在运行 macOS 和 homebrew 的 iMac 上，使用:

If you want to grab the screen faster, I would suggest ffmpeg. I installed it on my iMac running macOS with homebrew using:

brew install ffmpeg

然后我可以查看可用视频源的列表并找到我需要用来录制屏幕的内容:

I can then see the list of available video sources and find what I need to record the screen with:

ffmpeg -f avfoundation -list_devices true -i ""

样本输出

[AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation video devices:
[AVFoundation input device @ 0x7fa7dcf05b40] [0] FaceTime HD Camera
[AVFoundation input device @ 0x7fa7dcf05b40] [1] Capture screen 0             <--- THIS ONE IS THE SCREEN
[AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation audio devices:
[AVFoundation input device @ 0x7fa7dcf05b40] [0] MacBook Pro Microphone
[AVFoundation input device @ 0x7fa7dcf05b40] [1] CalDigit Thunderbolt 3 Audio

所以我知道我需要为屏幕输入 1.

So I know I need input 1 for the screen.

因此，如果我想从左上角 (0,0) 以 400px 的宽度和 200px 的高度以 20fps 的速度记录屏幕 10 秒，并将 RGBA8888 数据传递给我的钓鱼程序，我可以这样做:

So, if I want to record the screen from top-left corner (0,0) at a width of 400px and height of 200px at 20fps for 10s and pass RGBA8888 data to my fishing program, I can do this:

ffmpeg -y -pix_fmt bgr0 -f avfoundation -r 20 -t 10 -i 1 -filter:v "crop=400:200:0:0" -f rawvideo - | ./fish.py

我现在可以使用以下作为我的钓鱼程序:

I can now use the following as my fishing program:

#!/usr/bin/env python3

import numpy as np
import pyautogui
import time
import os, sys

# width, height
w, h = 400, 200

# Bytes per frame - assumes bgr0, i.e. 8-bits of blue, 8-bits red. 8-bits green and 8-bits junk
bytesPerFrame = w * h * 4

while True:
    img = sys.stdin.buffer.read(bytesPerFrame)
    if len(img) != bytesPerFrame:
        break
    # Process your video here

关键字:pyautogui、屏幕抓取、屏幕抓取、屏幕抓取、慢速、Mac、macOS、Python、捕获、屏幕捕获、ffmpeg、PIL、OpenCV

Keywords: pyautogui, screen grab, screen-grab, screengrab, slow, Mac, macOS, Python, capture, screen capture, ffmpeg, PIL, OpenCV

这篇关于Python Opencv 直播桌面截屏流程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！