ffmpeg“清单太大"；下载youtube视频时 [英] ffmpeg "Manifest too large" when downloading youtube video

问题描述

我正在尝试以编程方式下载 YouTube视频的一部分.众所周知的程序不适用于某些视频，我想克服这种情况.

I'm trying to programmatically download a part of a YouTube video. The widely known procedure doesn't work for some videos and I'd like to overcome this situation.

我正在尝试以编程方式下载 YouTube视频的一部分.如 如何下载部分中所述使用youtube-dl命令录制视频? ，您可以通过以下命令来实现.

I'm trying to programmatically download a part of a YouTube video. As described in How to download portion of video with youtube-dl command?, you can achieve this by the following commands.

#Converts a human-readable URL to longer URLs for internal use.
~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=POrfo478HRI'
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes

#Stores the result to a variable.
~ $ url_video_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Stores the result to a variable.
~ $ url_audio_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Downloads the video for one minute from 12:45.
~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
frame= 3600 fps= 49 q=-1.0 Lsize=   27047kB time=00:00:59.99 bitrate=3692.9kbits/s speed=0.813x

这确实很好用.

对于某些视频^†1 ，内部URL的格式为 https://manifest.googlevideo.com/... 而不是 https://如上例所示的RANDOM-STRING.googlevideo.com/....

For some videos^†1, the internal URLs are of the form https://manifest.googlevideo.com/... instead of https://RANDOM-STRING.googlevideo.com/... as in the example above.

~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=KRMfnvLODZQ'
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D

下载此类视频始终会失败.

And downloading such videos always fails.

~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
[dash @ 0x55fa6a1a4400] Manifest too large: 3776722
https://manifest.googlevideo.com/...: Invalid data found when processing input

请注意，仅使用 youtube-dl 即可下载全部此类视频.

Please note downloading full of such videos using youtube-dl alone succeeds.

_{†​​1:据我们测试，最近(例如昨天)上传的视频具有此类URL，并且当我们在一天后重试相同的 youtube-dl --get-url 命令时大约，我们得到无辜"的报偿.像第一个示例中那样的URL.}

_{†1: As far as we tested, recently (e.g. yesterday) uploaded videos have such URLs and when we retry the same youtube-dl --get-url command after a day or so, we get "innocent" URLs like in the first example.}

如何下​​载部分视频

通过为 ffmpeg ，

通过某种方式强制检索 https://RANDOM-STRING.googlevideo.com/... 形式的内部URL，

还是我根本不知道的事情?

or by something which I have no idea at all?

我当前的解决方法是等待一两天，直到内部URL更改为止，这比下载完整视频然后进行裁剪要慢得多.

My current workaround is just to wait for a day or two until the internal URLs change, which is much slower than downloading the full video and then cropping it.

~ $ lsb_release -d
Description:    Arch Linux

~ $ ffmpeg -version
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

~ $ youtube-dl --version
2020.07.28

推荐答案

有两种解决方法.

如OP中所述，您只需等待几天，即可从 https://manifest.googlevideo.com/api/manifest更改内部URL †1 /dash/...到 https://r1 --- sn-3pm76n76.googlevideo.com/... .

As described in the OP, you can just wait for a few days for the internal URLs to be changed^†1 from https://manifest.googlevideo.com/api/manifest/dash/... to https://r1---sn-3pm76n76.googlevideo.com/....

据我测试，这至少需要一整天的时间(7-10小时).

As far as I tested, this takes at least a whole day for long (7-10 hours) videos.

通过以下步骤，您可以立即下载视频.

By the following steps, you can download the video right away.

1. url ='https://www.youtube.com/watch?v = pkP-hgGJdvY'

_{(我选择此URL作为示例，因为 youtube-dl --get-url" $ {url}" 返回 https://manifest.googlevideo.com/api/...(写此答案时).}

_{(I chose this URL as an example because youtube-dl --get-url "${url}" returns https://manifest.googlevideo.com/api/... at the time of writing this answer.)}

1. 卷曲"$ {url}"

从 curl 的输出中，提取格式为 https://manifest.googlevideo.com/api/manifest/的URL internal_url hls_variant/....请注意，此URL包含/hls_variant/而不是/dash/.

From the output of curl, extract the URL internal_url of the form https://manifest.googlevideo.com/api/manifest/hls_variant/.... Note that this URL contains /hls_variant/ instead of /dash/.

ffmpeg -i"$ {internal_url}" ，带有或不带有用于下载视频的任意选项.

ffmpeg -i "${internal_url}" with or without arbitrary options to download the video.

第2步和第3步的一个衬里

One liner of the step 2 and 3:

curl --silent "${url}" | grep -o 'https:\\/\\/[^"]\+googlevideo.com\\/[^"]\+' | grep 'hls_variant' | sed 's/\\//g'

2.2已知问题

即使采用上述方法，对于很长时间(2个多小时)的实时存档(最近已完成实时流式传输(例如，几个小时或一天前))，也只能将整个视频的最后2小时准确地保存为下载 ^†2 ，即使您可以在浏览器或官方应用上播放视频也没有问题.此问题并非特定于 ffmpeg ，实际上 youtube-dl 的GitHub存储库有一个相关问题:

2.2 Known Issue

Even by the method above, for a long (2+ hours) live archive which has finished being live-streamed recently (e.g. hours or a day ago), only exactly last 2 hours of the whole video can be downloaded^†2 even if you can play the video on a browser or the official app with no problem. This problem is not specific to ffmpeg and actually youtube-dl's GitHub repository has a related issue: Youtube: Only downloading last 2 hours of 3 hour video. (with hack as a workaround) #26330. (By the way, the OP of the issue includes a workaround of this problem, which I don't write here since I don't tested.)

考虑†1和†2，当目标视频的YouTube服务器端处理尚未完全完成时，似乎会出现此问题.尽管我根本不知道该怎么做，但也许应该有一些优雅的方法来解决最后两个小时"的问题.

Taking †1 and †2 into account, it seems the problem occurs when the YouTube server-side process of the target video has not completely been finished. Although I have no idea at all how to, probably there should be some elegant way to overcome the only-last-two-hour problem.

这篇关于ffmpeg“清单太大"；下载youtube视频时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！