Xuggler编码和复用 [英] Xuggler encoding and muxing

问题描述

我正在尝试使用 Xuggler （我相信使用 ffmpeg 在下面）执行以下操作：

• 接受原始的MPJPEG视频比特流（来自小型TTL串行相机），并将其编码/转码为h.264;和


• 接受来自麦克风的原始音频比特流，并将其编码为AAC;然后


• 将两个（音频和视频）比特流混合在一起成为MPEG-TS容器

我看过/阅读了一些优秀的教程，到目前为止我已经得到了以下内容：

  // I稍后将会担心实现此功能，但
 //涉及查询本地设备驱动程序。 
 byte [] nextMjpeg = getNextMjpegFromSerialPort（）; 
 
 //我也担心实现这个功能; 
 //我只是提供这些彻底的。 
 BufferedImage mjpeg = MjpegFactory.newMjpeg（nextMjpeg）; 
 
 //指定h.264视频流（如何？）
字符串h264Stream =???; 
 
 IMediaWriter writer = ToolFactory.makeWriter（h264Stream）; 
 writer.addVideoStream（0，0，ICodec.ID.CODEC_ID_H264）; 
 writer.encodeVideo（0，mjpeg）; 
  

对于一个，我想我在这里很近，但仍然不正确;通过阅读视频代码示例（而不是音频 - 我找不到任何好的音频示例），我只是得到了这个。

从字面上来说，我会获取字节级访问我的Xuggler实现中的原始视频和音频源。但是对于我来说，我无法弄清楚如何让他们进入h.264 / AAC / MPEG-TS格式。感谢您提前获得任何帮助。

解决方案

查看 Xuggler这个示例代码，以下应该可以将视频编码为H.264，并将其复制到MPEG2TS容器中：

 code> IMediaWriter writer = ToolFactory.makeWriter（output.ts）; 
 writer.addVideoStream（0，0，ICodec.ID.CODEC_ID_H264，width，height）; 
（...）
 {
 
 BufferedImage mjpeg = ...; 
 
 writer.encodeVideo（0，mjpeg）; 
} 
  

从文件扩展名猜测容器类型，显式指定编解码器。

对于多媒体音频和视频，您可以这样做：

  writer.addVideoStream（videoStreamIndex，0，videoCodec，width，height）; 
 writer.addAudioStream（audioStreamIndex，0，audioCodec，channelCount，sampleRate）; 
 
 while（...有更多的数据...）
 {
 BufferedImage videoFrame = ...; 
 long videoFrameTime = ...; //这是显示此框架的时间
 writer.encodeVideo（videoStreamIndex，videoFrame，videoFrameTime，DEFAULT_TIME_UNIT）; 
 
 short [] audioSamples = ...; //这个数组的大小应该是样本数* channelCount 
 long audioSamplesTime = ...; //这是播放这一位音频的时间
 writer.encodeAudio（audioStreamIndex，audioSamples，audioSamplesTime，DEFAULT_TIME_UNIT）; 
} 
  

在这种情况下，我相信你的代码负责交织音频和视频：您可以根据哪些数据可用（音频样本或视频帧块）具有较早的时间戳记，每循环遍历一次encodeAudio（）或 encodeVideo（） / em>的。

根据 IStreamCoder，您可能会使用另一种较低级别的API ，它可以更好地控制各种参数。我不认为你需要使用它。

要回答您提出的具体问题：

（1）将缓冲图像（M / JPEG）编码为h.264流 - 您已经知道， writer.addVideoStream（...，ICodec.ID.CODEC_ID_H264）确保您获得H.264 编解码器。要获取传输流（MPEG2 TS）容器，只需调用带有.ts扩展名的文件名 makeWriter（）。

（2）了解原始音频源的BufferedImage等价物是 - 这是一个简短的[]或 IAudioSamples 对象（两者似乎都有效，但IAudioSamples必须从（3）将这个音频类编码为AAC音频流 - 调用 writer.addAudioStream（。 ..，ICodec.ID.CODEC_ID_AAC，channelCount，sampleRate）

（4）将两个流复用到同一个MPEG-TS容器 - 使用.ts文件名调用 makeWriter（），它设置容器类型。为了正确的音频/视频同步，您可能需要以正确的顺序调用encodeVideo（）/ encodeAudio（）。

始终通过最早的音频/视频。例如，如果您有440个样本长的音频块（44000 Hz采样率，440/44000 = 0.01秒），并且视频完全为25fps（1/25 = 0.04秒），则可以将其提供给作者这个订单：

  video0 @ 0.00 sec 
 audio0 @ 0.00 sec 
 audio1 @ 0.01 sec 
 audio2 @ 0.02 sec 
 audio3 @ 0.03 sec 
 video1 @ 0.04 sec 
 audio4 @ 0.04 sec 
 audio5 @ 0.05 sec 
  pre> 
 
 
 ...等等
 
 
 
大多数回放设备都可以与流一起使用，只要连续的音频/视频时间戳相对较近，但这是您将要做的完美复用。
 
 
 
 PS您可能需要参考几个文档： Xuggler类图， ToolFactory ， IMediaWriter ， ICodec 。 
 
I'm trying to use Xuggler (which I believe uses ffmpeg under the hood) to do the following:


Accept a raw MPJPEG video bitstream (from a small TTL serial camera) and encode/transcode it to h.264; and
Accept a raw audio bitsream (from a microphone) and encode it to AAC; then
Mux the two (audio and video) bitsreams together into a MPEG-TS container


I've watched/read some of their excellent tutorials, and so far here's what I've got:
// I'll worry about implementing this functionality later, but
// involves querying native device drivers.
byte[] nextMjpeg = getNextMjpegFromSerialPort();

// I'll also worry about implementing this functionality as well;
// I'm simply providing these for thoroughness.
BufferedImage mjpeg = MjpegFactory.newMjpeg(nextMjpeg);

// Specify a h.264 video stream (how?)
String h264Stream = "???";

IMediaWriter writer = ToolFactory.makeWriter(h264Stream);
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264);
writer.encodeVideo(0, mjpeg);
For one, I think I'm close here, but it's still not correct; and I've only gotten this far by reading the video code examples (not the audio - I can't find any good audio examples).

Literally, I'll be getting byte-level access to the raw video and audio feeds coming into my Xuggler implementation. But for the life of me I can't figure out how to get them into an h.264/AAC/MPEG-TS format. Thanks in advance for any help here.
解决方案
Looking at Xuggler this sample code, the following should work to encode video as H.264 and mux it into a MPEG2TS container:
IMediaWriter writer = ToolFactory.makeWriter("output.ts");
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, width, height);
for (...)
{

   BufferedImage mjpeg = ...;

   writer.encodeVideo(0, mjpeg);
}
The container type is guessed from the file extension, the codec is specified explicitly.

To mux audio and video, you would do something like this:
writer.addVideoStream(videoStreamIndex, 0, videoCodec, width, height);
writer.addAudioStream(audioStreamIndex, 0, audioCodec, channelCount, sampleRate);

while (... have more data ...)
{
    BufferedImage videoFrame = ...;
    long videoFrameTime = ...; // this is the time to display this frame
    writer.encodeVideo(videoStreamIndex, videoFrame, videoFrameTime, DEFAULT_TIME_UNIT);

    short[] audioSamples = ...; // the size of this array should be number of samples * channelCount
    long audioSamplesTime = ...; // this is the time to play back this bit of audio
    writer.encodeAudio(audioStreamIndex, audioSamples, audioSamplesTime, DEFAULT_TIME_UNIT);
}
In this case I believe your code is responsible for interleaving the audio and video: you want to call either encodeAudio() or encodeVideo() on each pass through the loop, based on which data available (a chunk of audio samples or a video frame) has an earlier timestamp.  

There is another, lower-level API you may end up using, based on IStreamCoder, which gives more control over various parameters.  I don't think you will need to use that.

To answer the specific questions you asked:

(1) "Encode a BufferedImage (M/JPEG) into a h.264 stream" - you already figured that out, writer.addVideoStream(..., ICodec.ID.CODEC_ID_H264) makes sure you get the H.264 codec.  To get a transport stream (MPEG2 TS) container, simply call makeWriter() with a filename with a .ts extension.

(2) "Figure out what the "BufferedImage-equivalent" for a raw audio feed is" - that is either a short[] or an IAudioSamples object (both seem to work, but IAudioSamples has to be constructed from an IBuffer which is much less straightforward).

(3) "Encode this audio class into an AAC audio stream" - call writer.addAudioStream(..., ICodec.ID.CODEC_ID_AAC, channelCount, sampleRate)

(4) "multiplex both stream into the same MPEG-TS container" - call makeWriter() with a .ts filename, which sets the container type.  For correct audio/video sync you probably need to call encodeVideo()/encodeAudio() in the correct order.  

P.S.  Always pass the earliest audio/video available first.  For example, if you have audio chunks which are 440 samples long (at 44000 Hz sample rate, 440 / 44000 = 0.01 seconds), and video at exactly 25fps (1 / 25 = 0.04 seconds), you would give them to the writer in this order:
video0 @ 0.00 sec
audio0 @ 0.00 sec
audio1 @ 0.01 sec
audio2 @ 0.02 sec
audio3 @ 0.03 sec
video1 @ 0.04 sec
audio4 @ 0.04 sec
audio5 @ 0.05 sec
... and so forth

Most playback devices are probably ok with the stream as long as the consecutive audio/video timestamps are relatively close, but this is what you'd do for a perfect mux.

P.S. There are a few docs you may want to refer to: Xuggler class diagram, ToolFactory, IMediaWriter, ICodec.

                        
这篇关于Xuggler编码和复用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！