如何将实时视频帧从客户端流式传输到 Flask 服务器并返回到客户端? [英] How to stream live video frames from client to flask server and back to the client?
问题描述
我正在尝试构建一个客户端服务器架构,在该架构中我使用 getUserMedia() 从用户的网络摄像头捕获实时视频.现在不是直接在 标签中显示视频,我想将它发送到我的烧瓶服务器,对帧进行一些处理并将其扔回我的网页.
我使用 socketio 来创建客户端-服务器连接.这是我的 index.html 中的脚本.请原谅我的错误或任何错误的代码.
<video autoplay="true" id="videoElement"></视频>
在我的 app.py -
from flask import Flask,render_template从flask_socketio 导入SocketIO,发出app = Flask(__name__)socketio = SocketIO(app)@app.route('/', methods=['POST', 'GET'])定义索引():返回 render_template('index.html')@socketio.on('catch-frame')def catch_frame(数据):##获取数据框##做一些处理##发送回客户端发射('response_back',数据)## ??如果 __name__ == '__main__':socketio.run(app, host='127.0.0.1')
我也曾想过通过 WebRTC 来做这件事,但我只得到点对点的代码.
那么,有人可以帮我解决这个问题吗?预先感谢您的帮助.
所以,我试图做的是获取客户端网络摄像头捕获的实时视频流并在后端进行处理.
我的后端代码是用 Python 编写的,我使用 SocketIo 将帧从前端发送到后端.您可以查看此设计以更好地了解正在发生的事情-图片
- 我的服务器(app.py)将在后端运行,客户端将访问 index.html
- SocketIo 连接将建立,使用网络摄像头捕获的视频流将逐帧发送到服务器.
- 然后这些帧将在后端处理并发送回客户端.
- 来自服务器的处理过的帧可以显示在 img 标签中.
这是工作代码 -
app.py
@socketio.on('image')定义图像(数据图像):sbuf = StringIO()sbuf.write(data_image)# 解码并转换成图像b = io.BytesIO(base64.b64decode(data_image))pimg = Image.open(b)## 将RGB 转换为BGR,作为opencv 标准框架 = cv2.cvtColor(np.array(pimg), cv2.COLOR_RGB2BGR)# 处理图像帧frame = imutils.resize(frame, width=700)框架 = cv2.flip(frame, 1)imgencode = cv2.imencode('.jpg', frame)[1]# base64 编码stringData = base64.b64encode(imgencode).decode('utf-8')b64_src = '数据:图像/jpg;base64,'stringData = b64_src + stringData# 返回帧发射('response_back',stringData)
index.html
<canvas id="canvasOutput"></canvas><视频自动播放=真"id="videoElement"></video><div 类 = '视频'><img id="图像">
<脚本>var socket = io('http://localhost:5000');socket.on('connect', function(){console.log("已连接...!", socket.connected)});const video = document.querySelector(#videoElement");视频宽度 = 500;视频高度 = 375;;如果(navigator.mediaDevices.getUserMedia){navigator.mediaDevices.getUserMedia({ video: true }).then(函数(流){video.srcObject = 流;视频播放();}).catch(函数(err0r){控制台日志(err0r)console.log(出了点问题!");});}让 src = new cv.Mat(video.height, video.width, cv.CV_8UC4);让 dst = new cv.Mat(video.height, video.width, cv.CV_8UC1);让帽 = 新 cv.VideoCapture(video);const FPS = 22;setInterval(() => {cap.read(src);var type = "image/png";var data = document.getElementById("canvasOutput").toDataURL(type);data = data.replace('data:' + type + ';base64,', '');//分离垃圾一开始socket.emit('图像', 数据);}, 10000/FPS);socket.on('response_back', function(image){const image_id = document.getElementById('image');image_id.src = 图像;});
此外,websockets 在安全源上运行.
I am trying to build a client server architecture where I am capturing the live video from user's webcam using getUserMedia(). Now instead of showing video directly in <video>
tag, I want to send it to my flask server, do some processing on frames and throw it back to my web page.
I have used socketio for creating a client-server connection. This is the script in my index.html. Please pardon my mistakes or any wrong code.
<div id="container">
<video autoplay="true" id="videoElement">
</video>
</div>
<script type="text/javascript" charset="utf-8">
var socket = io('http://127.0.0.1:5000');
// checking for connection
socket.on('connect', function(){
console.log("Connected... ", socket.connected)
});
var video = document.querySelector("#videoElement");
// asking permission to access the system camera of user, capturing live
// video on getting true.
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true })
.then(function (stream) {
// instead of showing it directly in <video>, I want to send these frame to server
//video_t.srcObject = stream
//this code might be wrong, but this is what I want to do.
socket.emit('catch-frame', { image: true, buffer: getFrame() });
})
.catch(function (err0r) {
console.log(err0r)
console.log("Something went wrong!");
});
}
// returns a frame encoded in base64
const getFrame = () => {
const canvas = document.createElement('canvas');
canvas.width = video_t.videoWidth;
canvas.height = video_t.videoHeight;
canvas.getContext('2d').drawImage(video_t, 0, 0);
const data = canvas.toDataURL('image/png');
return data;
}
// receive the frame from the server after processed and now I want display them in either
// <video> or <img>
socket.on('response_back', function(frame){
// this code here is wrong, but again this is what something I want to do.
video.srcObject = frame;
});
</script>
In my app.py -
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
app = Flask(__name__)
socketio = SocketIO(app)
@app.route('/', methods=['POST', 'GET'])
def index():
return render_template('index.html')
@socketio.on('catch-frame')
def catch_frame(data):
## getting the data frames
## do some processing
## send it back to client
emit('response_back', data) ## ??
if __name__ == '__main__':
socketio.run(app, host='127.0.0.1')
I also have thought to do this by WebRTC, but I am only getting code for peer to peer.
So, can anyone help me with this? Thanks in advance for help.
So, what I was trying to do is to take the real time video stream captured by the client's webcam and process them at backend.
My backend code is written in Python and I am using SocketIo to send the frames from frontend to backend. You can have a look at this design to get a better idea about what's happening - image
- My server(app.py) will be running in backend and client will be accessing index.html
- SocketIo connection will get establish and video stream captured using webcam will be send to server frames by frames.
- These frames will be then processed at the backend and emit back to the client.
- Processed frames coming form the server can be shown in img tag.
Here is the working code -
app.py
@socketio.on('image')
def image(data_image):
sbuf = StringIO()
sbuf.write(data_image)
# decode and convert into image
b = io.BytesIO(base64.b64decode(data_image))
pimg = Image.open(b)
## converting RGB to BGR, as opencv standards
frame = cv2.cvtColor(np.array(pimg), cv2.COLOR_RGB2BGR)
# Process the image frame
frame = imutils.resize(frame, width=700)
frame = cv2.flip(frame, 1)
imgencode = cv2.imencode('.jpg', frame)[1]
# base64 encode
stringData = base64.b64encode(imgencode).decode('utf-8')
b64_src = 'data:image/jpg;base64,'
stringData = b64_src + stringData
# emit the frame back
emit('response_back', stringData)
index.html
<div id="container">
<canvas id="canvasOutput"></canvas>
<video autoplay="true" id="videoElement"></video>
</div>
<div class = 'video'>
<img id="image">
</div>
<script>
var socket = io('http://localhost:5000');
socket.on('connect', function(){
console.log("Connected...!", socket.connected)
});
const video = document.querySelector("#videoElement");
video.width = 500;
video.height = 375; ;
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true })
.then(function (stream) {
video.srcObject = stream;
video.play();
})
.catch(function (err0r) {
console.log(err0r)
console.log("Something went wrong!");
});
}
let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);
let dst = new cv.Mat(video.height, video.width, cv.CV_8UC1);
let cap = new cv.VideoCapture(video);
const FPS = 22;
setInterval(() => {
cap.read(src);
var type = "image/png"
var data = document.getElementById("canvasOutput").toDataURL(type);
data = data.replace('data:' + type + ';base64,', ''); //split off junk
at the beginning
socket.emit('image', data);
}, 10000/FPS);
socket.on('response_back', function(image){
const image_id = document.getElementById('image');
image_id.src = image;
});
</script>
Also, websockets runs on secure origin.
这篇关于如何将实时视频帧从客户端流式传输到 Flask 服务器并返回到客户端?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!