Xuggler 编码和混合 [英] Xuggler encoding and muxing

问题描述

我正在尝试使用 Xuggler(我相信使用ffmpeg 在幕后)执行以下操作:

I'm trying to use Xuggler (which I believe uses ffmpeg under the hood) to do the following:

• 接受原始 MPJPEG 视频比特流(来自小型 TTL 串行相机)并将其编码/转码为 h.264；和
• 接受原始音频比特流(来自麦克风)并将其编码为 AAC；然后
• 将两个(音频和视频)比特流混合到一个 MPEG-TS 容器中

我看过/阅读过他们的一些优秀教程，到目前为止，我已经掌握了以下内容:

I've watched/read some of their excellent tutorials, and so far here's what I've got:

// I'll worry about implementing this functionality later, but
// involves querying native device drivers.
byte[] nextMjpeg = getNextMjpegFromSerialPort();

// I'll also worry about implementing this functionality as well;
// I'm simply providing these for thoroughness.
BufferedImage mjpeg = MjpegFactory.newMjpeg(nextMjpeg);

// Specify a h.264 video stream (how?)
String h264Stream = "???";

IMediaWriter writer = ToolFactory.makeWriter(h264Stream);
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264);
writer.encodeVideo(0, mjpeg);

一方面，我想我已经接近这里了，但仍然不正确；并且我只是通过阅读视频代码示例(不是音频 - 我找不到任何好的音频示例)来了解这一点.

For one, I think I'm close here, but it's still not correct; and I've only gotten this far by reading the video code examples (not the audio - I can't find any good audio examples).

从字面上看，我将获得对进入我的 Xuggler 实现的原始视频和音频源的字节级访问.但是对于我的生活，我无法弄清楚如何将它们转换为 h.264/AAC/MPEG-TS 格式.在此先感谢您的帮助.

Literally, I'll be getting byte-level access to the raw video and audio feeds coming into my Xuggler implementation. But for the life of me I can't figure out how to get them into an h.264/AAC/MPEG-TS format. Thanks in advance for any help here.

推荐答案

查看 Xuggler 这个示例代码，以下应该可以工作将视频编码为 H.264 并将其复用到 MPEG2TS 容器中:

Looking at Xuggler this sample code, the following should work to encode video as H.264 and mux it into a MPEG2TS container:

IMediaWriter writer = ToolFactory.makeWriter("output.ts");
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, width, height);
for (...)
{

   BufferedImage mjpeg = ...;

   writer.encodeVideo(0, mjpeg);
}

从文件扩展名猜测容器类型，明确指定编解码器.

The container type is guessed from the file extension, the codec is specified explicitly.

要混合音频和视频，您可以这样做:

To mux audio and video, you would do something like this:

writer.addVideoStream(videoStreamIndex, 0, videoCodec, width, height);
writer.addAudioStream(audioStreamIndex, 0, audioCodec, channelCount, sampleRate);

while (... have more data ...)
{
    BufferedImage videoFrame = ...;
    long videoFrameTime = ...; // this is the time to display this frame
    writer.encodeVideo(videoStreamIndex, videoFrame, videoFrameTime, DEFAULT_TIME_UNIT);

    short[] audioSamples = ...; // the size of this array should be number of samples * channelCount
    long audioSamplesTime = ...; // this is the time to play back this bit of audio
    writer.encodeAudio(audioStreamIndex, audioSamples, audioSamplesTime, DEFAULT_TIME_UNIT);
}

在这种情况下，我相信您的代码负责交错音频和视频:您希望在每次循环时调用 encodeAudio() 或 encodeVideo()，基于可用的数据(一段音频样本或一个视频帧)有一个更早的时间戳.

In this case I believe your code is responsible for interleaving the audio and video: you want to call either encodeAudio() or encodeVideo() on each pass through the loop, based on which data available (a chunk of audio samples or a video frame) has an earlier timestamp.

您最终可能会使用另一个较低级别的 API，它基于 IStreamCoder，它可以更好地控制各种参数.我认为您不需要使用它.

There is another, lower-level API you may end up using, based on IStreamCoder, which gives more control over various parameters. I don't think you will need to use that.

回答您提出的具体问题:

To answer the specific questions you asked:

(1) 将 BufferedImage (M/JPEG) 编码为 h.264 流" - 你已经知道了，writer.addVideoStream(..., ICodec.ID.CODEC_ID_H264)确保您获得 H.264 编解码器.要获得传输流 (MPEG2 TS) 容器，只需使用带有 .ts 扩展名的文件名调用 makeWriter().

(1) "Encode a BufferedImage (M/JPEG) into a h.264 stream" - you already figured that out, writer.addVideoStream(..., ICodec.ID.CODEC_ID_H264) makes sure you get the H.264 codec. To get a transport stream (MPEG2 TS) container, simply call makeWriter() with a filename with a .ts extension.

(2)找出原始音频提要的BufferedImage-equivalent"是什么" - 即短[]或IAudioSamples 对象(两者似乎都可以工作，但 IAudioSamples 必须从 IBuffer 构建，这不太直接).

(2) "Figure out what the "BufferedImage-equivalent" for a raw audio feed is" - that is either a short[] or an IAudioSamples object (both seem to work, but IAudioSamples has to be constructed from an IBuffer which is much less straightforward).

(3) 将此音频类编码为 AAC 音频流" - 调用 writer.addAudioStream(..., ICodec.ID.CODEC_ID_AAC, channelCount, sampleRate)

(3) "Encode this audio class into an AAC audio stream" - call writer.addAudioStream(..., ICodec.ID.CODEC_ID_AAC, channelCount, sampleRate)

(4) 将两个流复用到同一个 MPEG-TS 容器中" - 使用 .ts 文件名调用 makeWriter()，它设置容器类型.为了正确的音频/视频同步，您可能需要以正确的顺序调用 encodeVideo()/encodeAudio().

(4) "multiplex both stream into the same MPEG-TS container" - call makeWriter() with a .ts filename, which sets the container type. For correct audio/video sync you probably need to call encodeVideo()/encodeAudio() in the correct order.

附言始终首先传递可用的最早的音频/视频.例如，如果您有 440 个样本长的音频块(以 44000 Hz 采样率，440/44000 = 0.01 秒)和恰好 25fps(1/25 = 0.04 秒)的视频，您可以将它们提供给作者这个顺序:

P.S. Always pass the earliest audio/video available first. For example, if you have audio chunks which are 440 samples long (at 44000 Hz sample rate, 440 / 44000 = 0.01 seconds), and video at exactly 25fps (1 / 25 = 0.04 seconds), you would give them to the writer in this order:

video0 @ 0.00 sec
audio0 @ 0.00 sec
audio1 @ 0.01 sec
audio2 @ 0.02 sec
audio3 @ 0.03 sec
video1 @ 0.04 sec
audio4 @ 0.04 sec
audio5 @ 0.05 sec

...等等

只要连续的音频/视频时间戳相对接近，大多数播放设备可能都可以处理流，但这是您为完美多路复用所做的.

Most playback devices are probably ok with the stream as long as the consecutive audio/video timestamps are relatively close, but this is what you'd do for a perfect mux.

附言您可能需要参考一些文档:Xuggler 类图、工具工厂、IMediaWriter、ICodec.

P.S. There are a few docs you may want to refer to: Xuggler class diagram, ToolFactory, IMediaWriter, ICodec.

这篇关于Xuggler 编码和混合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！