HTML5音频流：精确测量延迟？ [英] HTML5 audio streaming: precisely measure latency?

问题描述

我正在构建一个跨平台的Web应用程序，在该应用程序中可以在服务器上即时生成音频，并可能通过HTML5音频元素实时流式传输到浏览器客户端。在浏览器上，我将使用Javascript驱动的动画，这些动画必须与播放的音频精确同步。 精确意味着音频和动画必须在一秒钟之内，并且希望在250ms以内（假设唇音同步）。出于各种原因，我无法在服务器上播放音频和动画，并实时传输视频。

理想情况下，服务器上的音频生成和浏览器上的音频播放，但我的理解是延迟将难以控制，可能在3-7秒范围内（浏览器，环境，网络和月相）依赖）。但是，如果我能够精确地测量实时延迟，以便浏览器JavaScript知道何时呈现适当的动画帧，那么我可以处理这个问题。

所以，我需要精确测量传递给流媒体服务器（Icecast？）的传递音频和扬声器主机上扬声器的音频之间的延迟。一些蓝天的可能性：

• 在音频流中添加元数据，并从播放的音频中解析出来（我明白这不是使用标准音频元素是不可能的）




• 为音频添加短时间的纯静音，然后在浏览器上检测它们（音频元素可以产生实际的音频样本？）




• 查询服务器和浏览器的各种缓冲区深度

使用Javascript解码音频流，然后获取元数据 ？

解决方案

利用 timeupdate 事件< ; audio> 元素，每秒触发三到四次，通过检查 .currentTime <$来在媒体流媒体期间执行精确的动画c $ c>< audio> 元素。





如果在浏览器中可用，您可以使用 fetch（） code>请求音频资源，在 .then（） return response.body.getReader（） which返回资源的 ReadableStream ;创建一个新的 MediaSource 对象，设置< audio> 或 new Audio（） code> .src 至 objectURL MediaSource ;将第一个流块添加到 .read（）链接 .then（）到 sourceBuffer MediaSource 与 .mode 设置为sequence;将剩余的块附加到 sourceBuffer 在 sourceBuffer updateend 事件。
$ b $

如果 fetch（） response.body.getReader（）在浏览器中不可用，仍然可以使用 timeupdate 或进度 < ; audio> 元素来检查 .currentTime ，开始或停止播放流媒体时所需的动画或转换。





使用 canplay 事件< audio> 元素可在流累积时播放媒体在 MediaSource >处继续播放。





您可以使用一个对象，其属性设置为 .currentTime < audio> 动画应该发生，值设为 css <
$ b

在 javascript 下面，一个元素应该被动画以执行精确的动画。 imation在每20秒钟发生一次，从 0 开始，并且每60秒钟发生一次，直到媒体播放结束。



 <！DOCTYPE html> 
< html xmlns =http://www.w3.org/1999/xhtml> 
< head> 
< meta charset =utf-8/> 
< title>< / title> 
< style> 
 body {
 width：90vw; 
身高：90vh; 
背景：＃000; 
转换：背景1s; 
} 
 
 span {
 font-family：Georgia; 
 font-size：36px; 
 opacity：0; 
} 
< / style> 
< / head> 
 
< body> 
<音频控制>< /音频> 
< br> 
< span>< / span> 
< script type =text / javascript> 
 window.onload = function（）{
 var url =/ path / to / audio; 
 //给定240秒音频总持续时间
 // 240/12 = 20 
 //属性对应于`<音频>`.currentTime`，
 //值对应于元素
设置的颜色var colors = {
 0：red，
 20：blue，
 40：green，
 60：黄色，
 80：橙色，
 100：紫色，
 120：紫色，
 140：棕色，
 160：棕褐色，
 180：黄金，
 200：sienna，
 220：skyblue
}; 
 var body = document.querySelector（body）; 
 var mediaSource = new MediaSource; 
 var audio = document.querySelector（audio）; 
 var span = document.querySelector（span）; 
 var color = window.getComputedStyle（body）
 .getPropertyValue（background-color）; 
 //console.log(mediaSource.readyState）; //关闭
 var mimecodec =audio / mpeg; 
 
 audio.oncanplay = function（）{
 this.play（）; 
} 
 
 audio.ontimeupdate = function（）{
 // 240/12 = 20 
 var curr = Math.round（this.currentTime）; 
 
 if（colors.hasOwnProperty（curr））{
 //将`color`设为`colors [curr]`
 color = colors [curr] 
} 
 // animate`< span>`每60秒
 if（curr％60 === 0&& span.innerHTML ===）{
 var t = curr / 60; 
 span.innerHTML = t +minute+（t === 1？s）
 +of+ Math.round（this.duration）/ 60 
 +音频分钟; 
 span.animate（[{
 opacity：0 
}，{
 opacity：1 
}，{
 opacity：0 
} ]，{
 duration：2500，
 iterations：1 
}）
 .onfinish = function（）{
 span.innerHTML =
} 
} 
 //每20秒更换`body`的`background-color` 
 body.style.backgroundColor = color; 
 console.log（当前时间：，curr 
，当前背景颜色：，颜色
，duration：，this.duration）; 
} 
 //将`<音频>`.src`设置为`mediaSource` 
 audio.src = URL.createObjectURL（mediaSource）; 
 mediaSource.addEventListener（sourceopen，sourceOpen）; 
 
函数sourceOpen（event）{
 //如果媒体类型被'mediaSource`支持
 //获取资源，开始流读，
 //追加流到`sourceBuffer` 
 if（MediaSource.isTypeSupported（mimecodec））{
 var sourceBuffer = mediaSource.addSourceBuffer（mimecodec）; 
 //将`sourceBuffer`` .mode`设置为``序列``
 sourceBuffer.mode =sequence; 
 
 fetch（url）
 //返回`response`的'readableStream` 
 .then（response => response.body.getReader（））
。然后（阅读器=> {
 
 var processStream =（data）=> {
 if（data.done）{
 return; 
} 
 //将流的块添加到`sourceBuffer` 
 sourceBuffer.appendBuffer（data.value）; 
} 
 //在`sourceBuffer``upndatend``中调用`reader.read（）` ，
 //读取下一个流块，将块附加到
 //`sourceBuffer` 
 sourceBuffer.addEventListener（updateend，function（）{
 reader.read（ ）.then（processStream）; 
}）; 
 //开始处理流
 reader.read（）。then（processStream）; 
 // do stuff`reader` is关闭，
 //读取流完成
 return reader.closed.then（（）=> {
 //流的结束信号到`mediaSource` 
 mediaSource.endOfStream（）; 
返回mediaSource.readyState; 
}）
}）
 //当`reader.closed`，`mediaSource`流结束
 .then（msg => console.log（msg））时做的事情
} 
 //如果'MediaSource`不支持'mimecodec'
 else {
 alert（mimecodec +not supported）; 
} 
}; 
} 
< / script> 
< / body> 
< / html> 
  

plnkr http://plnkr.co/edit/fIm1Qp?p=preview




I'm building a cross-platform web app where audio is generated on-the-fly on the server and live streamed to a browser client, probably via the HTML5 audio element. On the browser, I'll have Javascript-driven animations that must precisely sync with the played audio. "Precise" means that the audio and animation must be within a second of each other, and hopefully within 250ms (think lip-syncing). For various reasons, I can't do the audio and animation on the server and live-stream the resulting video.

Ideally, there would be little or no latency between the audio generation on the server and the audio playback on the browser, but my understanding is that latency will be difficult to control and probably in the 3-7 second range (browser-, environment-, network- and phase-of-the-moon-dependent). I can handle that, though, if I can precisely measure the actual latency on-the-fly so that my browser Javascript knows when to present the proper animated frame.

So, I need to precisely measure the latency between my handing audio to the streaming server (Icecast?), and the audio coming out of the speakers on the computer hosting the speaker. Some blue-sky possibilities:

• Add metadata to the audio stream, and parse it from the playing audio (I understand this isn't possible using the standard audio element)

• Add brief periods of pure silence to the audio, and then detect them on the browser (can audio elements yield the actual audio samples?)

• Query the server and the browser as to the various buffer depths

• Decode the streamed audio in Javascript and then grab the metadata

Any thoughts as to how I could do this?

解决方案

Utilize timeupdate event of <audio> element, which is fired three to four times per second, to perform precise animations during streaming of media by checking .currentTime of <audio> element. Where animations or transitions can be started or stopped up to several times per second.

If available at browser, you can use fetch() to request audio resource, at .then() return response.body.getReader() which returns a ReadableStream of the resource; create a new MediaSource object, set <audio> or new Audio() .src to objectURL of the MediaSource; append first stream chunks at .read() chained .then() to sourceBuffer of MediaSource with .mode set to "sequence"; append remainder of chunks to sourceBuffer at sourceBuffer updateend events.

If fetch() response.body.getReader() is not available at browser, you can still use timeupdate or progress event of <audio> element to check .currentTime, start or stop animations or transitions at required second of streaming media playback.

Use canplay event of <audio> element to play media when stream has accumulated adequate buffers at MediaSource to proceed with playback.

You can use an object with properties set to numbers corresponding to .currentTime of <audio> where animation should occur, and values set to css property of element which should be animated to perform precise animations.

At javascript below, animations occur at every twenty second period, beginning at 0, and at every sixty seconds until the media playback has concluded.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">    
<head>
  <meta charset="utf-8" />
  <title></title>
  <style>
    body {
      width: 90vw;
      height: 90vh;
      background: #000;
      transition: background 1s;
    }

    span {
      font-family: Georgia;
      font-size: 36px;
      opacity: 0;
    }
  </style>
</head>

<body>
  <audio controls></audio>
  <br>
  <span></span>
  <script type="text/javascript">
    window.onload = function() {
      var url = "/path/to/audio";
      // given 240 seconds total duration of audio 
      // 240/12 = 20
      // properties correspond to `<audio>` `.currentTime`,
      // values correspond to color to set at element
      var colors = {
        0: "red",
        20: "blue",
        40: "green",
        60: "yellow",
        80: "orange",
        100: "purple",
        120: "violet",
        140: "brown",
        160: "tan",
        180: "gold",
        200: "sienna",
        220: "skyblue"
      };
      var body = document.querySelector("body");
      var mediaSource = new MediaSource;
      var audio = document.querySelector("audio");
      var span = document.querySelector("span");
      var color = window.getComputedStyle(body)
                  .getPropertyValue("background-color");
      //console.log(mediaSource.readyState); // closed
      var mimecodec = "audio/mpeg";

      audio.oncanplay = function() {
        this.play();
      }

      audio.ontimeupdate = function() {         
        // 240/12 = 20
        var curr = Math.round(this.currentTime);

        if (colors.hasOwnProperty(curr)) {
          // set `color` to `colors[curr]`
          color = colors[curr]
        }
        // animate `<span>` every 60 seconds
        if (curr % 60 === 0 && span.innerHTML === "") {
          var t = curr / 60;
          span.innerHTML = t + " minute" + (t === 1 ? "" : "s") 
                           + " of " + Math.round(this.duration) / 60 
                          + " minutes of audio";
          span.animate([{
              opacity: 0
            }, {
              opacity: 1
            }, {
              opacity: 0
            }], {
              duration: 2500,
              iterations: 1
            })
            .onfinish = function() {
              span.innerHTML = ""
            }
        }
        // change `background-color` of `body` every 20 seconds
        body.style.backgroundColor = color;
        console.log("current time:", curr
                   , "current background color:", color
                  , "duration:", this.duration);
      }
      // set `<audio>` `.src` to `mediaSource`
      audio.src = URL.createObjectURL(mediaSource);
      mediaSource.addEventListener("sourceopen", sourceOpen);

      function sourceOpen(event) {
        // if the media type is supported by `mediaSource`
        // fetch resource, begin stream read, 
        // append stream to `sourceBuffer`
        if (MediaSource.isTypeSupported(mimecodec)) {
          var sourceBuffer = mediaSource.addSourceBuffer(mimecodec);
          // set `sourceBuffer` `.mode` to `"sequence"`
          sourceBuffer.mode = "sequence";

          fetch(url)
          // return `ReadableStream` of `response`
          .then(response => response.body.getReader())
          .then(reader => {

            var processStream = (data) => {
              if (data.done) {
                  return;
              }
              // append chunk of stream to `sourceBuffer`
              sourceBuffer.appendBuffer(data.value);
            }
            // at `sourceBuffer` `updateend` call `reader.read()`,
            // to read next chunk of stream, append chunk to 
            // `sourceBuffer`
            sourceBuffer.addEventListener("updateend", function() {
              reader.read().then(processStream);
            });
            // start processing stream
            reader.read().then(processStream);
            // do stuff `reader` is closed, 
            // read of stream is complete
            return reader.closed.then(() => {
              // signal end of stream to `mediaSource`
              mediaSource.endOfStream();
              return  mediaSource.readyState;
            })
          })
          // do stuff when `reader.closed`, `mediaSource` stream ended
          .then(msg => console.log(msg))
        } 
        // if `mimecodec` is not supported by `MediaSource`  
        else {
          alert(mimecodec + " not supported");
        }
      };
    }
  </script>
</body>
</html>

plnkr http://plnkr.co/edit/fIm1Qp?p=preview

这篇关于HTML5音频流：精确测量延迟？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！