将实时镜头从相机流式传输到 Unity3D [英] Streaming live footage from camera to Unity3D

问题描述

假设我有一个无线摄像头，我想实时地将镜头流传输到统一格式.有没有办法做到这一点?

Let's say I have a wireless camera that I want to stream footage from to unity in real-time. Is there a way to achieve this?

奖励问题:

• 角度更广(180 度，甚至 360 度)的相机怎么样
• 如果这是我想与之互动的镜头，延迟会有多大问题
• 除了常规镜头之外，是否可以发送更多数据，例如深度感知(使用深度感知相机)?
• 是我疯了还是这样做了?

提前致谢

推荐答案

我假设这是一个带有以太网端口或 Wi-Fi 的相机，您可以连接到它并从中实时传输图像.

I assume this is a camera with Ethernet port or Wi-Fi that you can connect to and stream images from it live.

如果是，那么是的，可以使用 Unity 来完成.

If so, then yes, it can be done with Unity.

如何在没有外部库的情况下完成:

连接到相机:

1.连接到与相机相同的本地网络，或者如果支持 unpn，您也可以通过互联网连接到它.通常，您需要摄像机的 IP 和端口才能执行此操作.假设摄像机 IP 地址为 192.168.1.5，端口号为 900.要连接的 url 是 http://192.168.1.5:900.

1.Connect to the-same local network with the camera or if unpn is supported, you can also connect to it through internet. Usually, you need the IP and the port of the camera to do this. Let's say that the Camera IP Address is 192.168.1.5 and the port number is 900. The url to connect to is http://192.168.1.5:900.

有时，它只是一个以 .mjpg 或 .bin 结尾的网址，例如 http://192.168.1.5/mjpg/video.mjpg 和 http://192.168.1.5/mjpg/video.bin

Sometimes, it is simply an url that ends with .mjpg or .bin such as http://192.168.1.5/mjpg/video.mjpg and http://192.168.1.5/mjpg/video.bin

每个相机都不同.找到网址的唯一方法是阅读其手册.如果手册不可用，请使用其官方应用程序连接到它，然后使用 Wireshark 发现相机图像的 url.username 和 password(如果需要)也可以在手册中找到.如果不可用，请在谷歌上搜索型号，应该可以找到您需要的一切.

Every camera is different. The only way to find the url is to read its manual. If the manual is not available, connect to it with its official Application then use Wireshark to discover the url of the camera Image. The username and the password(if required) can also be found in the manual. If not available, google the model number and everything you need should be found.

从相机中提取 JPEG:

当连接到相机时，相机会向您发送无穷无尽的数据.您可以扫描这些数据并从中检索图像.

When connected to the camera, the camera will be sending endless data to you. This data you can scan and retrieve the image from it.

2.搜索 0xFF 后跟 0xD8 的 JPEG 标头.如果这两个字节彼此相邻，则开始读取字节并继续将它们保存到数组中.您可以使用 index(int) 变量来记录您收到的字节数.

2.Search for the JPEG header which is 0xFF followed by 0xD8. If these two bytes are next to each other then start reading the bytes and continue to save them to an array. You can use an index(int) variable to keep count of how many bytes you have received.

int counter = 0;
byte[] completeImageByte = new byte[500000];
byte[] receivedBytes = new byte[500000];
receivedBytes[counter] = byteFromCamera;
counter++;

3.从摄像头读取数据时，检查接下来的两个字节是否为0xFF后跟0xD9的JPEG页脚.如果这是真的，那么您已经收到了完整的图像(1 帧).

3.While reading the data from the camera, check if the next two bytes is the JPEG footer which is 0xFF followed by 0xD9. If this is true then you have received the complete image(1 frame).

您的图像字节应该类似于:

Your image bytes should look something like:

0xFF 0xD8 someotherbytes(数千个)..... 然后 0xFF 0xD9

0xFF 0xD8 someotherbytes(thousands of them)..... then 0xFF 0xD9

将 receivedBytes 复制到 completeImageByte 变量，以便以后可以使用它来显示图像.将 counter 变量重置为 0.

Copy receivedBytes to the completeImageByte variable so that it can be used to display the image later on. Reset counter variable to 0.

Buffer.BlockCopy(receivedBytes, 0, completeImageByte, 0, counter);
counter = 0;

在屏幕上显示 JPEG 图像:

4.将图像显示到屏幕

由于您每秒将收到许多图像，因此我发现显示此图像的最有效方法是使用 RawImage 组件.因此，如果您希望在移动设备上运行，请不要为此使用 Image 或 Sprite Renderer.

Since you will be receiving many images per second, the most efficient way I found to display this is to use RawImage Component. So, don't use Image or Sprite Renderer for this if you want this to run on mobile devices.

public RawImage screenDisplay;
if(updateFrame){
Texture2D camTexture = new Texture2D(2, 2);
camTexture.LoadImage(completeImageByte);
screenDisplay.texture = camTexture;
}

你只需要在Start()函数中做一次camTexture = new Texture2D(2, 2);.

You only need to do camTexture = new Texture2D(2, 2); once in the Start() function.

5.跳回步骤 2 并继续这样做，直到您想停下来为止.

5.Jump back to step 2 and continue doing it until you want to quite.

用于连接到相机的 API:.

如果相机需要身份验证(用户名和密码)，请使用 HttpWebRequest.

Use HttpWebRequest if the camera requires authentication (username and password).

对于那些不需要身份验证的，使用UnityWebRequest.使用 UnityWebRequest 时，您必须从 派生自己的类DownloadHandlerScript 否则您的应用程序将崩溃，因为您将不停地接收数据.

For those that does not require authentication, use UnityWebRequest. When using UnityWebRequest, you must derive your own classes from DownloadHandlerScript or your app will crash as you will be receiving data non stop.

从 DownloadHandlerScript 派生自己的类的示例:

Example of deriving your own class from DownloadHandlerScript:

using UnityEngine;
using System.Collections;
using UnityEngine.Networking;

public class CustomWebRequest : DownloadHandlerScript
{    
    // Standard scripted download handler - will allocate memory on each ReceiveData callback
    public CustomWebRequest()
        : base()
    {
    }

    // Pre-allocated scripted download handler
    // Will reuse the supplied byte array to deliver data.
    // Eliminates memory allocation.
    public CustomWebRequest(byte[] buffer)
        : base(buffer)
    {
    }

    // Required by DownloadHandler base class. Called when you address the 'bytes' property.
    protected override byte[] GetData() { return null; }

    // Called once per frame when data has been received from the network.
    protected override bool ReceiveData(byte[] byteFromCamera, int dataLength)
    {
        if (byteFromCamera == null || byteFromCamera.Length < 1)
        {
            //Debug.Log("CustomWebRequest :: ReceiveData - received a null/empty buffer");
            return false;
        }

        //Search of JPEG Image here

        return true;
    }

    // Called when all data has been received from the server and delivered via ReceiveData
    protected override void CompleteContent()
    {
        //Debug.Log("CustomWebRequest :: CompleteContent - DOWNLOAD COMPLETE!");
    }

    // Called when a Content-Length header is received from the server.
    protected override void ReceiveContentLength(int contentLength)
    {
        //Debug.Log(string.Format("CustomWebRequest :: ReceiveContentLength - length {0}", contentLength));
    }
}

用法:

using UnityEngine;
using System.Collections;
using UnityEngine.Networking;

public class Test : MonoBehaviour
{

    CustomWebRequest camImage;
    UnityWebRequest webRequest;
    byte[] bytes = new byte[90000];

    void Start()
    {
        string url = "http://camUrl/mjpg/video.mjpg";
        webRequest = new UnityWebRequest(url);
        webRequest.downloadHandler = new CustomWebRequest(bytes);
        webRequest.Send();
    }
}

您可以在ReceiveData<中执行步骤2、3、4和5CustomWebRequest 脚本中的/code> 函数.

You can them perform step 2,3,4 and 5 in the ReceiveData function from the CustomWebRequest script.

控制相机:

相机具有平移、旋转、翻转、镜像和执行其他功能的命令.这在每个相机中都不同，但很简单，就像向相机的 url 发出 GET/POST 请求并提供查询一样.这些命令可以在相机手册中找到.

Cameras have commands to pan, rotate, flip, mirror and perform other function.This is different in every camera but it is simple as making GET/POST request to a url of the camera and providing the queries. These commands can be found in the camera manual.

例如:http://192.168.1.5?pan=50&rotate=90

其他框架:

AForge - 一个可以同时处理 JPEG/MJPES 和 FFMPEG 来自相机.您必须修改它才能与 Unity 一起使用，如果您不能执行步骤 2、3、4 和 5，则应该修改它.

AForge - A free framework that can handle both JPEG/MJPES and FFMPEG from the camera. You have to modify it to work with Unity and you should if you can't do step 2,3,4 and 5.

这篇关于将实时镜头从相机流式传输到 Unity3D的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！