Release 2021.05.20

[redbulltv] fix embed data extraction
https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.11. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.05.11**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.05.20
+ [debug] yt-dlp version 2021.05.11
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.11. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.05.11**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.11. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.05.11**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.11. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.05.11**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.05.20
+ [debug] yt-dlp version 2021.05.11
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.11. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.05.11**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -132,7 +132,7 @@ jobs:
     - name: Upgrade pip and enable wheel support
       run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4 pefile==2019.4.18
+      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py

.gitignore

@@ -1,46 +1,3 @@
-# Config
-*.conf
-*.spec
-cookies
-cookies.txt
-
-# Downloaded
-*.srt
-*.ttml
-*.sbv
-*.vtt
-*.flv
-*.mp4
-*.m4a
-*.m4v
-*.mp3
-*.3gp
-*.webm
-*.wav
-*.ape
-*.mkv
-*.swf
-*.part
-*.part-*
-*.ytdl
-*.dump
-*.frag
-*.frag.urls
-*.aria2
-*.swp
-*.ogg
-*.opus
-*.info.json
-*.live_chat.json
-*.jpg
-*.png
-*.webp
-*.annotations.xml
-*.description
-
-# Allow config/media files in testdata
-!test/testdata/**
-
 # Python
 *.pyc
 *.pyo
@@ -86,6 +43,48 @@ README.txt
 yt-dlp.zip
 *.exe
 
+# Downloaded
+*.srt
+*.ttml
+*.sbv
+*.vtt
+*.flv
+*.mp4
+*.m4a
+*.m4v
+*.mp3
+*.3gp
+*.webm
+*.wav
+*.ape
+*.mkv
+*.swf
+*.part
+*.part-*
+*.ytdl
+*.dump
+*.frag
+*.frag.urls
+*.aria2
+*.swp
+*.ogg
+*.opus
+*.info.json
+*.live_chat.json
+*.jpg
+*.png
+*.webp
+*.annotations.xml
+*.description
+
+# Config
+*.conf
+*.spec
+cookies
+cookies.txt
+
+
+
 # Text Editor / IDE
 .idea
 *.iml

Changelog.md

@@ -10,7 +10,7 @@
 * Commit to master as `Release <version>`
 * Push to origin/release using `git push origin master:release`
     build task will now run
-* Update version.py using `devscripts\update-version.py`
+* Update version.py using devscripts\update-version.py
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
 * Push to origin/master
@@ -19,74 +19,30 @@
 -->
 
 
-### 2021.06.01
-
-* Merge youtube-dl: Upto [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)
-* Pre-check archive and filters during playlist extraction
-* Handle Basic Auth `user:pass` in URLs by [hhirtz](https://github.com/hhirtz) and [pukkandan](https://github.com/pukkandan)
-* [archiveorg] Add YoutubeWebArchiveIE by [colethedj](https://github.com/colethedj) and [alex-gedeon](https://github.com/alex-gedeon)
-* [fancode] Add extractor by [rmsmachine](https://github.com/rmsmachine)
-* [patreon] Support vimeo embeds by [rhsmachine](https://github.com/rhsmachine)
-* [Saitosan] Add new extractor by [llacb47](https://github.com/llacb47)
-* [ShemarooMe] Add extractor by [Ashish0804](https://github.com/Ashish0804) and [pukkandan](https://github.com/pukkandan)
-* [telemundo] Add extractor by [king-millez](https://github.com/king-millez)
-* [SonyLIV] Add SonyLIVSeriesIE and subtitle support by [Ashish0804](https://github.com/Ashish0804)
-* [Hotstar] Add HotStarSeriesIE by [Ashish0804](https://github.com/Ashish0804)
-* [Voot] Add VootSeriesIE by [Ashish0804](https://github.com/Ashish0804)
-* [vidio] Support login and premium videos by [MinePlayersPE](https://github.com/MinePlayersPE)
-* [fragment] When using `-N`, do not keep the fragment content in memory
-* [ffmpeg] Download and merge in a single step if possible
-* [ThumbnailsConvertor] Support conversion to `png` and make it the default by [louie-github](https://github.com/louie-github)
-* [VideoConvertor] Generalize with remuxer and allow conditional recoding
-* [EmbedThumbnail] Embed in `mp4`/`m4a` using mutagen by [tripulse](https://github.com/tripulse) and [pukkandan](https://github.com/pukkandan)
-* [EmbedThumbnail] Embed if any thumbnail was downloaded, not just the best
-* [EmbedThumbnail] Correctly escape filename
-* [update] replace self without launching a subprocess in windows
-* [update] Block further update for unsupported systems
-* Refactor `__process_playlist` by creating `LazyList`
-* Write messages to `stderr` when both `quiet` and `verbose`
-* Sanitize and sort playlist thumbnails
-* Remove `None` values from `info.json`
-* [extractor] Always prefer native hls downloader by default
-* [extractor] Skip subtitles without URI in m3u8 manifests by [hheimbuerger](https://github.com/hheimbuerger)
-* [extractor] Functions to parse `socket.io` response as `json` by [pukkandan](https://github.com/pukkandan) and [llacb47](https://github.com/llacb47)
-* [extractor] Allow `note=False` when extracting manifests
-* [utils] Escape URLs in `sanitized_Request`, not `sanitize_url`
-* [hls] Disable external downloader for `webtt`
-* [youtube] `/live` URLs should raise error if channel is not live
-* [youtube] Bug fixes
-* [zee5] Fix m3u8 formats' extension
-* [ard] Allow URLs without `-` before id by [olifre](https://github.com/olifre)
-* [cleanup] `YoutubeDL._match_entry`
-* [cleanup] Refactor updater
-* [cleanup] Refactor ffmpeg convertors
-* [cleanup] setup.py
-
-
 ### 2021.05.20
 
-* **Youtube improvements**: 
+**Youtube improvements**: 
     * Support youtube music `MP`, `VL` and `browse` pages
-    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+    * Extract more formats for youtube music by @craftingmod, @colethedj, @pukkandan
-    * Extract multiple subtitles in same language by [pukkandan](https://github.com/pukkandan) and [tpikonen](https://github.com/tpikonen)
+    * Extract multiple subtitles in same language by @pukkandan and @tpikonen
     * Redirect channels that doesn't have a `videos` tab to their `UU` playlists
     * Support in-channel search
     * Sort audio-only formats correctly
     * Always extract `maxresdefault` thumbnail
     * Extract audio language
-    * Add subtitle language names by [nixxo](https://github.com/nixxo) and [tpikonen](https://github.com/tpikonen)
+    * Add subtitle language names by @nixxo and @tpikonen
     * Show alerts only from the final webpage
-    * Add `html5=1` param to `get_video_info` page requests by [colethedj](https://github.com/colethedj)
+    * Add `html5=1` param to `get_video_info` page requests by @colethedj
     * Better message when login required
-* **Add option `--print`**: to print any field/template
+**Add option `--print`**: to print any field/template
-    * Deprecates: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
+    Deprecates: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
 * Field `additional_urls` to download additional videos from metadata using [`--parse-metadata`](https://github.com/yt-dlp/yt-dlp#modifying-metadata)
 * Merge youtube-dl: Upto [commit/dfbbe29](https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b)
 * Write thumbnail of playlist and add `pl_thumbnail` outtmpl key
-* [embedthumbnail] Add `flac` support and refactor `mutagen` code by [pukkandan](https://github.com/pukkandan) and [tripulse](https://github.com/tripulse)
+* [embedthumbnail] Add `flac` support and refactor `mutagen` code by @pukkandan and @tripulse
-* [audius:artist] Add extractor by [king-millez](https://github.com/king-millez)
+* [audius:artist] Add extractor by @king-millez
-* [parlview] Add extractor by [king-millez](https://github.com/king-millez)
+* [parlview] Add extractor by @king-millez
-* [tenplay] Fix extractor by [king-millez](https://github.com/king-millez)
+* [tenplay] Fix extractor by @king-millez
 * [rmcdecouverte] Generalize `_VALID_URL`
 * Add compat-option `no-attach-infojson`
 * Add field `name` for subtitles
@@ -97,7 +53,6 @@
 * [options] Refactor callbacks
 * [test:download] Only extract enough videos for `playlist_mincount`
 * [extractor] bugfix for when `compat_opts` is not given
-* [build] Fix x86 build by [shirt](https://github.com/shirt-dev)
 * [cleanup] code formatting, youtube tests and readme
 
 ### 2021.05.11
@@ -654,4 +609,4 @@
 * [generic] Extract embedded youtube and twitter videos by [diegorodriguezv](https://github.com/diegorodriguezv)
 * [ffmpeg] Ensure all streams are copied by [pukkandan](https://github.com/pukkandan)
 * [embedthumbnail] Fix for os.rename error by [pukkandan](https://github.com/pukkandan)
-* make_win.bat: don't use UPX to pack vcruntime140.dll by [jbruchon](https://github.com/jbruchon)
+* make_win.bat: don't use UPX to pack vcruntime140.dll by [jbruchon](https://github.com/jbruchon)

README.md

@@ -66,7 +66,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)**: (v2021.05.16) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl [commit/dfbbe29](https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b)**: (v2021.05.16) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
 
@@ -84,11 +84,11 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au)
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay
 
-* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
+* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [be6202f12b97858b9d716e608394b51065d0419f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
 * **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to using `--paths` (`-P`)
 
@@ -128,9 +128,8 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior
 * The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this
 * Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading
-* Youtube channel URLs are automatically redirected to `/video`. Append a `/featured` to the URL to download only the videos in the home page. If the channel does not have a videos tab, we try to download the equivalent `UU` playlist instead. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections
+* Youtube channel URLs are automatically redirected to `/video`. Either append a `/featured` to the URL or use `--compat-options no-youtube-channel-redirect` to download only the videos in the home page
 * Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
-* If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this
 
 For ease of use, a few more compat options are available:
 * `--compat-options all`: Use all compat options
@@ -671,10 +670,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
 ## Post-Processing Options:
     -x, --extract-audio              Convert video files to audio-only files
                                      (requires ffmpeg and ffprobe)
-    --audio-format FORMAT            Specify audio format to convert the audio
+    --audio-format FORMAT            Specify audio format: "best", "aac",
-                                     to when -x is used. Currently supported
+                                     "flac", "mp3", "m4a", "opus", "vorbis", or
-                                     formats are: best (default) or one of
+                                     "wav"; "best" by default; No effect without
-                                     aac|flac|mp3|m4a|opus|vorbis|wav
+                                     -x
     --audio-quality QUALITY          Specify ffmpeg audio quality, insert a
                                      value between 0 (better) and 9 (worse) for
                                      VBR or a specific bitrate like 128K
@@ -684,12 +683,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      |webm|mov|avi|mp3|mka|m4a|ogg|opus). If
                                      target container does not support the
                                      video/audio codec, remuxing will fail. You
-                                     can specify multiple rules; Eg.
+                                     can specify multiple rules; eg.
                                      "aac>m4a/mov>mp4/mkv" will remux aac to
                                      m4a, mov to mp4 and anything else to mkv.
     --recode-video FORMAT            Re-encode the video into another format if
-                                     re-encoding is necessary. The syntax and
+                                     re-encoding is necessary. The supported
-                                     supported formats are the same as --remux-video
+                                     formats are the same as --remux-video
     --postprocessor-args NAME:ARGS   Give these arguments to the postprocessors.
                                      Specify the postprocessor/executable name
                                      and the arguments separated by a colon ":"
@@ -750,10 +749,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      fields are passed, "%(filepath)s" is
                                      appended to the end of the command
     --convert-subs FORMAT            Convert the subtitles to another format
-                                     (currently supported: srt|vtt|ass|lrc)
+                                     (currently supported: srt|ass|vtt|lrc)
                                      (Alias: --convert-subtitles)
     --convert-thumbnails FORMAT      Convert the thumbnails to another format
-                                     (currently supported: jpg|png)
+                                     (currently supported: jpg)
     --split-chapters                 Split video into multiple files based on
                                      internal chapters. The "chapter:" prefix
                                      can be used with "--paths" and "--output"

pyinst.py

@@ -17,7 +17,7 @@ assert arch in ('32', '64')
 print('Building %sbit version' % arch)
 _x86 = '_x86' if arch == '32' else ''
 
-FILE_DESCRIPTION = 'yt-dlp%s' % (' (32 Bit)' if _x86 else '')
+FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
 
 # root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 # print('Changing working directory to %s' % root_dir)

setup.py

@@ -9,44 +9,45 @@ from distutils.spawn import spawn
 
 
 # Get the version from yt_dlp/version.py without importing the package
-exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
+exec(compile(open('yt_dlp/version.py').read(),
+             'yt_dlp/version.py', 'exec'))
 
 
 DESCRIPTION = 'Command-line program to download videos from YouTube.com and many other other video platforms.'
 
 LONG_DESCRIPTION = '\n\n'.join((
     'Official repository: <https://github.com/yt-dlp/yt-dlp>',
-    '**PS**: Some links in this document will not work since this is a copy of the README.md from Github',
+    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
     open('README.md', 'r', encoding='utf-8').read()))
 
 REQUIREMENTS = ['mutagen', 'pycryptodome']
 
-if sys.argv[1:2] == ['py2exe']:
-    raise NotImplementedError('py2exe is not currently supported; instead, use "pyinst.py" to build with pyinstaller')
 
+if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
+    print('inv')
+else:
+    files_spec = [
+        ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
+        ('share/zsh/site-functions', ['completions/zsh/_yt-dlp']),
+        ('share/fish/vendor_completions.d', ['completions/fish/yt-dlp.fish']),
+        ('share/doc/yt_dlp', ['README.txt']),
+        ('share/man/man1', ['yt-dlp.1'])
+    ]
+    root = os.path.dirname(os.path.abspath(__file__))
+    data_files = []
+    for dirname, files in files_spec:
+        resfiles = []
+        for fn in files:
+            if not os.path.exists(fn):
+                warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first.' % fn)
+            else:
+                resfiles.append(fn)
+        data_files.append((dirname, resfiles))
 
-files_spec = [
+    params = {
-    ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
+        'data_files': data_files,
-    ('share/zsh/site-functions', ['completions/zsh/_yt-dlp']),
+    }
-    ('share/fish/vendor_completions.d', ['completions/fish/yt-dlp.fish']),
+    params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
-    ('share/doc/yt_dlp', ['README.txt']),
-    ('share/man/man1', ['yt-dlp.1'])
-]
-root = os.path.dirname(os.path.abspath(__file__))
-data_files = []
-for dirname, files in files_spec:
-    resfiles = []
-    for fn in files:
-        if not os.path.exists(fn):
-            warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first' % fn)
-        else:
-            resfiles.append(fn)
-    data_files.append((dirname, resfiles))
-
-params = {
-    'data_files': data_files,
-}
-params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
 
 
 class build_lazy_extractors(Command):
@@ -60,8 +61,10 @@ class build_lazy_extractors(Command):
         pass
 
     def run(self):
-        spawn([sys.executable, 'devscripts/make_lazy_extractors.py', 'yt_dlp/extractor/lazy_extractors.py'],
+        spawn(
-              dry_run=self.dry_run)
+            [sys.executable, 'devscripts/make_lazy_extractors.py', 'yt_dlp/extractor/lazy_extractors.py'],
+            dry_run=self.dry_run,
+        )
 
 
 packages = find_packages(exclude=('youtube_dl', 'test', 'ytdlp_plugins'))

supportedsites.md

@@ -307,7 +307,6 @@
  - **EyedoTV**
  - **facebook**
  - **FacebookPluginsVideo**
- - **fancode:vod**
  - **faz.net**
  - **fc2**
  - **fc2:embed**
@@ -393,7 +392,6 @@
  - **HotNewHipHop**
  - **hotstar**
  - **hotstar:playlist**
- - **hotstar:series**
  - **Howcast**
  - **HowStuffWorks**
  - **hrfernsehen**
@@ -861,7 +859,6 @@
  - **safari**: safaribooksonline.com online video
  - **safari:api**
  - **safari:course**: safaribooksonline.com online courses
- - **Saitosan**
  - **SAKTV**
  - **SaltTV**
  - **SampleFocus**
@@ -886,7 +883,6 @@
  - **Shahid**
  - **ShahidShow**
  - **Shared**: shared.sx
- - **ShemarooMe**
  - **ShowRoomLive**
  - **simplecast**
  - **simplecast:episode**
@@ -906,7 +902,6 @@
  - **Snotr**
  - **Sohu**
  - **SonyLIV**
- - **SonyLIVSeries**
  - **soundcloud**
  - **soundcloud:playlist**
  - **soundcloud:search**: Soundcloud search
@@ -985,7 +980,6 @@
  - **Telecinco**: telecinco.es, cuatro.com and mediaset.es
  - **Telegraaf**
  - **TeleMB**
- - **Telemundo**
  - **TeleQuebec**
  - **TeleQuebecEmission**
  - **TeleQuebecLive**
@@ -1169,7 +1163,6 @@
  - **VODPlatform**
  - **VoiceRepublic**
  - **Voot**
- - **VootSeries**
  - **VoxMedia**
  - **VoxMediaVolume**
  - **vpro**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
@@ -1199,7 +1192,6 @@
  - **wdr:mobile**
  - **WDRElefant**
  - **WDRPage**
- - **web.archive:youtube**: web.archive.org saved youtube videos
  - **Webcaster**
  - **WebcasterFeed**
  - **WebOfStories**

test/test_YoutubeDL.py

@@ -756,7 +756,7 @@ class TestYoutubeDL(unittest.TestCase):
             def process_info(self, info_dict):
                 super(YDL, self).process_info(info_dict)
 
-            def _match_entry(self, info_dict, incomplete=False):
+            def _match_entry(self, info_dict, incomplete):
                 res = super(FilterYDL, self)._match_entry(info_dict, incomplete)
                 if res is None:
                     self.downloaded_info_dicts.append(info_dict)

test/test_postprocessors.py

@@ -8,11 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from yt_dlp.postprocessor import (
+from yt_dlp.postprocessor import MetadataFromFieldPP, MetadataFromTitlePP
-    FFmpegThumbnailsConvertorPP,
-    MetadataFromFieldPP,
-    MetadataFromTitlePP,
-)
 
 
 class TestMetadataFromField(unittest.TestCase):
@@ -34,24 +30,3 @@ class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
         self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
-
-
-class TestConvertThumbnail(unittest.TestCase):
-    def test_escaping(self):
-        pp = FFmpegThumbnailsConvertorPP()
-        if not pp.available:
-            print('Skipping: ffmpeg not found')
-            return
-
-        file = 'test/testdata/thumbnails/foo %d bar/foo_%d.{}'
-        tests = (('webp', 'png'), ('png', 'jpg'))
-
-        for inp, out in tests:
-            out_file = file.format(out)
-            if os.path.exists(out_file):
-                os.remove(out_file)
-            pp.convert_thumbnail(file.format(inp), out)
-            assert os.path.exists(out_file)
-
-        for _, out in tests:
-            os.remove(file.format(out))

test/test_utils.py

@@ -66,7 +66,6 @@ from yt_dlp.utils import (
     sanitize_filename,
     sanitize_path,
     sanitize_url,
-    sanitized_Request,
     expand_path,
     prepend_extension,
     replace_extension,
@@ -239,16 +238,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(sanitize_url('httpss://foo.bar'), 'https://foo.bar')
         self.assertEqual(sanitize_url('rmtps://foo.bar'), 'rtmps://foo.bar')
         self.assertEqual(sanitize_url('https://foo.bar'), 'https://foo.bar')
-        self.assertEqual(sanitize_url('foo bar'), 'foo bar')
-
-    def test_extract_basic_auth(self):
-        auth_header = lambda url: sanitized_Request(url).get_header('Authorization')
-        self.assertFalse(auth_header('http://foo.bar'))
-        self.assertFalse(auth_header('http://:foo.bar'))
-        self.assertEqual(auth_header('http://@foo.bar'), 'Basic Og==')
-        self.assertEqual(auth_header('http://:pass@foo.bar'), 'Basic OnBhc3M=')
-        self.assertEqual(auth_header('http://user:@foo.bar'), 'Basic dXNlcjo=')
-        self.assertEqual(auth_header('http://user:pass@foo.bar'), 'Basic dXNlcjpwYXNz')
 
     def test_expand_path(self):
         def env(var):

yt_dlp/YoutubeDL.py

@@ -70,7 +70,6 @@ from .utils import (
     int_or_none,
     iri_to_uri,
     ISO3166Utils,
-    LazyList,
     locked_file,
     make_dir,
     make_HTTPS_handler,
@@ -388,9 +387,8 @@ class YoutubeDL(object):
                        if True, otherwise use ffmpeg/avconv if False, otherwise
                        use downloader suggested by extractor if None.
     compat_opts:       Compatibility options. See "Differences in default behavior".
-                       Note that only format-sort, format-spec, no-live-chat,
+                       Note that only format-sort, format-spec, no-live-chat, no-attach-info-json
-                       no-attach-info-json, playlist-index, list-formats,
+                       playlist-index, list-formats, no-youtube-channel-redirect
-                       no-direct-merge, no-youtube-channel-redirect,
                        and no-youtube-unavailable-videos works when used via the API
 
     The following parameters are not used by YoutubeDL itself, they are used by
@@ -658,10 +656,12 @@ class YoutubeDL(object):
         """Print message to stdout"""
         if self.params.get('logger'):
             self.params['logger'].debug(message)
-        elif not quiet or self.params.get('verbose'):
+        elif not quiet:
-            self._write_string(
+            message = self._bidi_workaround(message)
-                '%s%s' % (self._bidi_workaround(message), ('' if skip_eol else '\n')),
+            terminator = ['\n', ''][skip_eol]
-                self._err_file if quiet else self._screen_file)
+            output = message + terminator
+
+            self._write_string(output, self._screen_file)
 
     def to_stderr(self, message):
         """Print message to stderr"""
@@ -669,7 +669,9 @@ class YoutubeDL(object):
         if self.params.get('logger'):
             self.params['logger'].error(message)
         else:
-            self._write_string('%s\n' % self._bidi_workaround(message), self._err_file)
+            message = self._bidi_workaround(message)
+            output = message + '\n'
+            self._write_string(output, self._err_file)
 
     def to_console_title(self, message):
         if not self.params.get('consoletitle', False):
@@ -731,8 +733,7 @@ class YoutubeDL(object):
                 else:
                     tb_data = traceback.format_list(traceback.extract_stack())
                     tb = ''.join(tb_data)
-            if tb:
+            self.to_stderr(tb)
-                self.to_stderr(tb)
         if not self.params.get('ignoreerrors', False):
             if sys.exc_info()[0] and hasattr(sys.exc_info()[1], 'exc_info') and sys.exc_info()[1].exc_info[0]:
                 exc_info = sys.exc_info()[1].exc_info
@@ -1013,12 +1014,11 @@ class YoutubeDL(object):
             path = encodeFilename(path, True).decode(preferredencoding())
         return sanitize_path(path, force=self.params.get('windowsfilenames'))
 
-    def _match_entry(self, info_dict, incomplete=False, silent=False):
+    def _match_entry(self, info_dict, incomplete):
         """ Returns None if the file should be downloaded """
 
-        video_title = info_dict.get('title', info_dict.get('id', 'video'))
-
         def check_filter():
+            video_title = info_dict.get('title', info_dict.get('id', 'video'))
             if 'title' in info_dict:
                 # This can happen when we're just evaluating the playlist
                 title = info_dict['title']
@@ -1045,6 +1045,8 @@ class YoutubeDL(object):
                     return 'Skipping %s, because it has exceeded the maximum view count (%d/%d)' % (video_title, view_count, max_views)
             if age_restricted(info_dict.get('age_limit'), self.params.get('age_limit')):
                 return 'Skipping "%s" because it is age restricted' % video_title
+            if self.in_download_archive(info_dict):
+                return '%s has already been recorded in archive' % video_title
 
             if not incomplete:
                 match_filter = self.params.get('match_filter')
@@ -1054,17 +1056,13 @@ class YoutubeDL(object):
                         return ret
             return None
 
-        if self.in_download_archive(info_dict):
+        reason = check_filter()
-            reason = '%s has already been recorded in the archive' % video_title
-            break_opt, break_err = 'break_on_existing', ExistingVideoReached
-        else:
-            reason = check_filter()
-            break_opt, break_err = 'break_on_reject', RejectedVideoReached
         if reason is not None:
-            if not silent:
+            self.to_screen('[download] ' + reason)
-                self.to_screen('[download] ' + reason)
+            if reason.endswith('has already been recorded in the archive') and self.params.get('break_on_existing', False):
-            if self.params.get(break_opt, False):
+                raise ExistingVideoReached()
-                raise break_err()
+            elif self.params.get('break_on_reject', False):
+                raise RejectedVideoReached()
         return reason
 
     @staticmethod
@@ -1255,7 +1253,6 @@ class YoutubeDL(object):
 
             self._playlist_level += 1
             self._playlist_urls.add(webpage_url)
-            self._sanitize_thumbnails(ie_result)
             try:
                 return self.__process_playlist(ie_result, download)
             finally:
@@ -1307,7 +1304,7 @@ class YoutubeDL(object):
 
         playlist_results = []
 
-        playliststart = self.params.get('playliststart', 1)
+        playliststart = self.params.get('playliststart', 1) - 1
         playlistend = self.params.get('playlistend')
         # For backwards compatibility, interpret -1 as whole list
         if playlistend == -1:
@@ -1327,43 +1324,50 @@ class YoutubeDL(object):
             playlistitems = orderedSet(iter_playlistitems(playlistitems_str))
 
         ie_entries = ie_result['entries']
-        msg = (
-            'Downloading %d videos' if not isinstance(ie_entries, list)
-            else 'Collected %d videos; downloading %%d of them' % len(ie_entries))
-        if not isinstance(ie_entries, (list, PagedList)):
-            ie_entries = LazyList(ie_entries)
 
-        entries = []
+        def make_playlistitems_entries(list_ie_entries):
-        for i in playlistitems or itertools.count(playliststart):
+            num_entries = len(list_ie_entries)
-            if playlistitems is None and playlistend is not None and playlistend < i:
+            for i in playlistitems:
-                break
+                if -num_entries < i <= num_entries:
-            entry = None
+                    yield list_ie_entries[i - 1]
-            try:
+                elif incomplete_entries:
-                entry = ie_entries[i - 1]
-                if entry is None:
                     raise EntryNotInPlaylist()
-            except (IndexError, EntryNotInPlaylist):
-                if incomplete_entries:
-                    raise EntryNotInPlaylist()
-                elif not playlistitems:
-                    break
-            entries.append(entry)
-            try:
-                if entry is not None:
-                    self._match_entry(entry, incomplete=True, silent=True)
-            except (ExistingVideoReached, RejectedVideoReached):
-                break
-        ie_result['entries'] = entries
 
-        # Save playlist_index before re-ordering
+        if isinstance(ie_entries, list):
-        entries = [
+            n_all_entries = len(ie_entries)
-            ((playlistitems[i - 1] if playlistitems else i), entry)
+            if playlistitems:
-            for i, entry in enumerate(entries, 1)
+                entries = list(make_playlistitems_entries(ie_entries))
-            if entry is not None]
+            else:
-        n_entries = len(entries)
+                entries = ie_entries[playliststart:playlistend]
+            n_entries = len(entries)
+            msg = 'Collected %d videos; downloading %d of them' % (n_all_entries, n_entries)
+        elif isinstance(ie_entries, PagedList):
+            if playlistitems:
+                entries = []
+                for item in playlistitems:
+                    entries.extend(ie_entries.getslice(
+                        item - 1, item
+                    ))
+            else:
+                entries = ie_entries.getslice(
+                    playliststart, playlistend)
+            n_entries = len(entries)
+            msg = 'Downloading %d videos' % n_entries
+        else:  # iterable
+            if playlistitems:
+                entries = list(make_playlistitems_entries(list(itertools.islice(
+                    ie_entries, 0, max(playlistitems)))))
+            else:
+                entries = list(itertools.islice(
+                    ie_entries, playliststart, playlistend))
+            n_entries = len(entries)
+            msg = 'Downloading %d videos' % n_entries
 
+        if any((entry is None for entry in entries)):
+            raise EntryNotInPlaylist()
         if not playlistitems and (playliststart or playlistend):
-            playlistitems = list(range(playliststart, playliststart + n_entries))
+            playlistitems = list(range(1 + playliststart, 1 + playliststart + len(entries)))
+        ie_result['entries'] = entries
         ie_result['requested_entries'] = playlistitems
 
         if self.params.get('allow_playlist_files', True):
@@ -1410,6 +1414,11 @@ class YoutubeDL(object):
                         self.report_error('Cannot write playlist description file ' + descfn)
                         return
 
+        # Save playlist_index before re-ordering
+        entries = [
+            ((playlistitems[i - 1] if playlistitems else i), entry)
+            for i, entry in enumerate(entries, 1)]
+
         if self.params.get('playlistreverse', False):
             entries = entries[::-1]
         if self.params.get('playlistrandom', False):
@@ -1417,7 +1426,7 @@ class YoutubeDL(object):
 
         x_forwarded_for = ie_result.get('__x_forwarded_for_ip')
 
-        self.to_screen('[%s] playlist %s: %s' % (ie_result['extractor'], playlist, msg % n_entries))
+        self.to_screen('[%s] playlist %s: %s' % (ie_result['extractor'], playlist, msg))
         failures = 0
         max_failures = self.params.get('skip_playlist_after_errors') or float('inf')
         for i, entry_tuple in enumerate(entries, 1):
@@ -1904,27 +1913,6 @@ class YoutubeDL(object):
         self.cookiejar.add_cookie_header(pr)
         return pr.get_header('Cookie')
 
-    @staticmethod
-    def _sanitize_thumbnails(info_dict):
-        thumbnails = info_dict.get('thumbnails')
-        if thumbnails is None:
-            thumbnail = info_dict.get('thumbnail')
-            if thumbnail:
-                info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
-        if thumbnails:
-            thumbnails.sort(key=lambda t: (
-                t.get('preference') if t.get('preference') is not None else -1,
-                t.get('width') if t.get('width') is not None else -1,
-                t.get('height') if t.get('height') is not None else -1,
-                t.get('id') if t.get('id') is not None else '',
-                t.get('url')))
-            for i, t in enumerate(thumbnails):
-                t['url'] = sanitize_url(t['url'])
-                if t.get('width') and t.get('height'):
-                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
-                if t.get('id') is None:
-                    t['id'] = '%d' % i
-
     def process_video_result(self, info_dict, download=True):
         assert info_dict.get('_type', 'video') == 'video'
 
@@ -1961,14 +1949,30 @@ class YoutubeDL(object):
             info_dict['playlist'] = None
             info_dict['playlist_index'] = None
 
-        self._sanitize_thumbnails(info_dict)
+        thumbnails = info_dict.get('thumbnails')
+        if thumbnails is None:
+            thumbnail = info_dict.get('thumbnail')
+            if thumbnail:
+                info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
+        if thumbnails:
+            thumbnails.sort(key=lambda t: (
+                t.get('preference') if t.get('preference') is not None else -1,
+                t.get('width') if t.get('width') is not None else -1,
+                t.get('height') if t.get('height') is not None else -1,
+                t.get('id') if t.get('id') is not None else '',
+                t.get('url')))
+            for i, t in enumerate(thumbnails):
+                t['url'] = sanitize_url(t['url'])
+                if t.get('width') and t.get('height'):
+                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
+                if t.get('id') is None:
+                    t['id'] = '%d' % i
 
         if self.params.get('list_thumbnails'):
             self.list_thumbnails(info_dict)
             return
 
         thumbnail = info_dict.get('thumbnail')
-        thumbnails = info_dict.get('thumbnails')
         if thumbnail:
             info_dict['thumbnail'] = sanitize_url(thumbnail)
         elif thumbnails:
@@ -2290,8 +2294,7 @@ class YoutubeDL(object):
         if not test:
             for ph in self._progress_hooks:
                 fd.add_progress_hook(ph)
-            urls = '", "'.join([f['url'] for f in info.get('requested_formats', [])] or [info['url']])
+            self.write_debug('Invoking downloader on %r' % info.get('url'))
-            self.write_debug('Invoking downloader on "%s"' % urls)
         new_info = dict(info)
         if new_info.get('http_headers') is None:
             new_info['http_headers'] = self._calc_headers(new_info)
@@ -2315,7 +2318,7 @@ class YoutubeDL(object):
         if 'format' not in info_dict:
             info_dict['format'] = info_dict['ext']
 
-        if self._match_entry(info_dict) is not None:
+        if self._match_entry(info_dict, incomplete=False) is not None:
             return
 
         self.post_extract(info_dict)
@@ -2530,6 +2533,17 @@ class YoutubeDL(object):
 
                 success = True
                 if info_dict.get('requested_formats') is not None:
+                    downloaded = []
+                    merger = FFmpegMergerPP(self)
+                    if self.params.get('allow_unplayable_formats'):
+                        self.report_warning(
+                            'You have requested merging of multiple formats '
+                            'while also allowing unplayable formats to be downloaded. '
+                            'The formats won\'t be merged to prevent data corruption.')
+                    elif not merger.available:
+                        self.report_warning(
+                            'You have requested merging of multiple formats but ffmpeg is not installed. '
+                            'The formats won\'t be merged.')
 
                     def compatible_formats(formats):
                         # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2577,57 +2591,27 @@ class YoutubeDL(object):
                     temp_filename = correct_ext(temp_filename)
                     dl_filename = existing_file(full_filename, temp_filename)
                     info_dict['__real_download'] = False
-
+                    if dl_filename is None:
-                    _protocols = set(determine_protocol(f) for f in requested_formats)
+                        for f in requested_formats:
-                    if len(_protocols) == 1:
+                            new_info = dict(info_dict)
-                        info_dict['protocol'] = _protocols.pop()
+                            new_info.update(f)
-                    directly_mergable = (
+                            fname = prepend_extension(
-                        'no-direct-merge' not in self.params.get('compat_opts', [])
+                                self.prepare_filename(new_info, 'temp'),
-                        and info_dict.get('protocol') is not None  # All requested formats have same protocol
+                                'f%s' % f['format_id'], new_info['ext'])
-                        and not self.params.get('allow_unplayable_formats')
+                            if not self._ensure_dir_exists(fname):
-                        and get_suitable_downloader(info_dict, self.params).__name__ == 'FFmpegFD')
+                                return
-                    if directly_mergable:
+                            downloaded.append(fname)
-                        info_dict['url'] = requested_formats[0]['url']
+                            partial_success, real_download = self.dl(fname, new_info)
-                        # Treat it as a single download
+                            info_dict['__real_download'] = info_dict['__real_download'] or real_download
-                        dl_filename = existing_file(full_filename, temp_filename)
+                            success = success and partial_success
-                        if dl_filename is None:
+                        if merger.available and not self.params.get('allow_unplayable_formats'):
-                            success, real_download = self.dl(temp_filename, info_dict)
+                            info_dict['__postprocessors'].append(merger)
-                            info_dict['__real_download'] = real_download
+                            info_dict['__files_to_merge'] = downloaded
-                    else:
+                            # Even if there were no downloads, it is being merged only now
-                        downloaded = []
+                            info_dict['__real_download'] = True
-                        merger = FFmpegMergerPP(self)
+                        else:
-                        if self.params.get('allow_unplayable_formats'):
+                            for file in downloaded:
-                            self.report_warning(
+                                files_to_move[file] = None
-                                'You have requested merging of multiple formats '
-                                'while also allowing unplayable formats to be downloaded. '
-                                'The formats won\'t be merged to prevent data corruption.')
-                        elif not merger.available:
-                            self.report_warning(
-                                'You have requested merging of multiple formats but ffmpeg is not installed. '
-                                'The formats won\'t be merged.')
-
-                        if dl_filename is None:
-                            for f in requested_formats:
-                                new_info = dict(info_dict)
-                                del new_info['requested_formats']
-                                new_info.update(f)
-                                fname = prepend_extension(
-                                    self.prepare_filename(new_info, 'temp'),
-                                    'f%s' % f['format_id'], new_info['ext'])
-                                if not self._ensure_dir_exists(fname):
-                                    return
-                                downloaded.append(fname)
-                                partial_success, real_download = self.dl(fname, new_info)
-                                info_dict['__real_download'] = info_dict['__real_download'] or real_download
-                                success = success and partial_success
-                            if merger.available and not self.params.get('allow_unplayable_formats'):
-                                info_dict['__postprocessors'].append(merger)
-                                info_dict['__files_to_merge'] = downloaded
-                                # Even if there were no downloads, it is being merged only now
-                                info_dict['__real_download'] = True
-                            else:
-                                for file in downloaded:
-                                    files_to_move[file] = None
                 else:
                     # Just a single file
                     dl_filename = existing_file(full_filename, temp_filename)
@@ -2777,20 +2761,19 @@ class YoutubeDL(object):
 
     @staticmethod
     def filter_requested_info(info_dict, actually_filter=True):
-        remove_keys = ['__original_infodict']  # Always remove this since this may contain a copy of the entire dict
+        info_dict.pop('__original_infodict', None)  # Always remove this
-        keep_keys = ['_type'],  # Always keep this to facilitate load-info-json
+        if not actually_filter:
-        if actually_filter:
-            remove_keys += ('requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries')
-            empty_values = (None, {}, [], set(), tuple())
-            reject = lambda k, v: k not in keep_keys and (
-                k.startswith('_') or k in remove_keys or v in empty_values)
-        else:
             info_dict['epoch'] = int(time.time())
-            reject = lambda k, v: k in remove_keys
+            return info_dict
+        exceptions = {
+            'remove': ['requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries'],
+            'keep': ['_type'],
+        }
+        keep_key = lambda k: k in exceptions['keep'] or not (k.startswith('_') or k in exceptions['remove'])
         filter_fn = lambda obj: (
-            list(map(filter_fn, obj)) if isinstance(obj, (list, tuple, set))
+            list(map(filter_fn, obj)) if isinstance(obj, (list, tuple))
             else obj if not isinstance(obj, dict)
-            else dict((k, filter_fn(v)) for k, v in obj.items() if not reject(k, v)))
+            else dict((k, filter_fn(v)) for k, v in obj.items() if keep_key(k)))
         return filter_fn(info_dict)
 
     def run_pp(self, pp, infodict):
@@ -3232,7 +3215,7 @@ class YoutubeDL(object):
             thumb_ext = determine_ext(t['url'], 'jpg')
             suffix = '%s.' % t['id'] if multiple else ''
             thumb_display_id = '%s ' % t['id'] if multiple else ''
-            thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
+            t['filepath'] = thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
 
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
                 ret.append(suffix + thumb_ext)
@@ -3248,7 +3231,6 @@ class YoutubeDL(object):
                     ret.append(suffix + thumb_ext)
                     self.to_screen('[%s] %s: Writing thumbnail %sto: %s' %
                                    (info_dict['extractor'], info_dict['id'], thumb_display_id, thumb_filename))
-                    t['filepath'] = thumb_filename
                 except network_exceptions as err:
                     self.report_warning('Unable to download thumbnail "%s": %s' %
                                         (t['url'], error_to_compat_str(err)))

yt_dlp/__init__.py

@@ -31,26 +31,20 @@ from .utils import (
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
+    REMUX_EXTENSIONS,
     render_table,
     SameFileError,
     setproctitle,
     std_headers,
     write_string,
 )
-from .update import run_update
+from .update import update_self
 from .downloader import (
     FileDownloader,
 )
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
-from .postprocessor.ffmpeg import (
-    FFmpegExtractAudioPP,
-    FFmpegSubtitlesConvertorPP,
-    FFmpegThumbnailsConvertorPP,
-    FFmpegVideoConvertorPP,
-    FFmpegVideoRemuxerPP,
-)
 from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
@@ -215,25 +209,25 @@ def _real_main(argv=None):
     if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
         raise ValueError('Playlist end must be greater than playlist start')
     if opts.extractaudio:
-        if opts.audioformat not in ['best'] + list(FFmpegExtractAudioPP.SUPPORTED_EXTS):
+        if opts.audioformat not in ['best', 'aac', 'flac', 'mp3', 'm4a', 'opus', 'vorbis', 'wav']:
             parser.error('invalid audio format specified')
     if opts.audioquality:
         opts.audioquality = opts.audioquality.strip('k').strip('K')
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
     if opts.recodevideo is not None:
-        opts.recodevideo = opts.recodevideo.replace(' ', '')
+        if opts.recodevideo not in REMUX_EXTENSIONS:
-        if not re.match(FFmpegVideoConvertorPP.FORMAT_RE, opts.recodevideo):
+            parser.error('invalid video recode format specified')
-            parser.error('invalid video remux format specified')
     if opts.remuxvideo is not None:
         opts.remuxvideo = opts.remuxvideo.replace(' ', '')
-        if not re.match(FFmpegVideoRemuxerPP.FORMAT_RE, opts.remuxvideo):
+        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
+        if not re.match(remux_regex, opts.remuxvideo):
             parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
-        if opts.convertsubtitles not in FFmpegSubtitlesConvertorPP.SUPPORTED_EXTS:
+        if opts.convertsubtitles not in ('srt', 'vtt', 'ass', 'lrc'):
             parser.error('invalid subtitle format specified')
     if opts.convertthumbnails is not None:
-        if opts.convertthumbnails not in FFmpegThumbnailsConvertorPP.SUPPORTED_EXTS:
+        if opts.convertthumbnails not in ('jpg', ):
             parser.error('invalid thumbnail format specified')
 
     if opts.date is not None:
@@ -264,8 +258,8 @@ def _real_main(argv=None):
         return parsed_compat_opts
 
     all_compat_opts = [
-        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
+        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'multistreams',
-        'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
+        'no-playlist-metafiles', 'no-live-chat', 'playlist-index', 'list-formats',
         'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
     ]
     compat_opts = parse_compat_opts()
@@ -486,10 +480,10 @@ def _real_main(argv=None):
         opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
 
     final_ext = (
-        opts.recodevideo if opts.recodevideo in FFmpegVideoConvertorPP.SUPPORTED_EXTS
+        opts.recodevideo
-        else opts.remuxvideo if opts.remuxvideo in FFmpegVideoRemuxerPP.SUPPORTED_EXTS
+        or (opts.remuxvideo in REMUX_EXTENSIONS) and opts.remuxvideo
-        else opts.audioformat if (opts.extractaudio and opts.audioformat != 'best')
+        or (opts.extractaudio and opts.audioformat != 'best') and opts.audioformat
-        else None)
+        or None)
 
     match_filter = (
         None if opts.match_filter is None
@@ -663,7 +657,7 @@ def _real_main(argv=None):
         # Update version
         if opts.update_self:
             # If updater returns True, exit. Required for windows
-            if run_update(ydl):
+            if update_self(ydl.to_screen, opts.verbose, ydl._opener):
                 if actual_use:
                     sys.exit('ERROR: The program must exit for the update to complete')
                 sys.exit()

yt_dlp/downloader/dash.py

@@ -154,9 +154,8 @@ class DashSegmentsFD(FragmentFD):
             max_workers = self.params.get('concurrent_fragment_downloads', 1)
             if can_threaded_download and max_workers > 1:
                 self.report_warning('The download speed shown is only of one thread. This is a known issue')
-                _download_fragment = lambda f: (f, download_fragment(f)[1])
                 with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments_to_download]
+                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments_to_download]
                     # timeout must be 0 to return instantly
                     done, not_done = concurrent.futures.wait(futures, timeout=0)
                     try:
@@ -170,13 +169,9 @@ class DashSegmentsFD(FragmentFD):
                         # timeout must be none to cancel
                         concurrent.futures.wait(not_done, timeout=None)
                         raise KeyboardInterrupt
+                results = [future.result() for future in futures]
 
-                for fragment, frag_index in map(lambda x: x.result(), futures):
+                for frag_content, frag_index in results:
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                    fragment['fragment_filename_sanitized'] = frag_sanitized
-                    frag_content = down.read()
-                    down.close()
                     result = append_fragment(frag_content, frag_index)
                     if not result:
                         return False

yt_dlp/downloader/external.py

@@ -346,7 +346,7 @@ class FFmpegFD(ExternalFD):
         return FFmpegPostProcessor().available
 
     def _call_downloader(self, tmpfilename, info_dict):
-        urls = [f['url'] for f in info_dict.get('requested_formats', [])] or [info_dict['url']]
+        url = info_dict['url']
         ffpp = FFmpegPostProcessor(downloader=self)
         if not ffpp.available:
             self.report_error('m3u8 download detected but ffmpeg could not be found. Please install')
@@ -378,7 +378,7 @@ class FFmpegFD(ExternalFD):
         # if end_time:
         #     args += ['-t', compat_str(end_time - start_time)]
 
-        if info_dict.get('http_headers') is not None and re.match(r'^https?://', urls[0]):
+        if info_dict.get('http_headers') is not None and re.match(r'^https?://', url):
             # Trailing \r\n after each HTTP header is important to prevent warning from ffmpeg/avconv:
             # [http @ 00000000003d2fa0] No trailing CRLF found in HTTP header.
             headers = handle_youtubedl_headers(info_dict['http_headers'])
@@ -436,15 +436,7 @@ class FFmpegFD(ExternalFD):
             elif isinstance(conn, compat_str):
                 args += ['-rtmp_conn', conn]
 
-        for url in urls:
+        args += ['-i', url, '-c', 'copy']
-            args += ['-i', url]
-        args += ['-c', 'copy']
-        if info_dict.get('requested_formats'):
-            for (i, fmt) in enumerate(info_dict['requested_formats']):
-                if fmt.get('acodec') != 'none':
-                    args.extend(['-map', '%d:a:0' % i])
-                if fmt.get('vcodec') != 'none':
-                    args.extend(['-map', '%d:v:0' % i])
 
         if self.params.get('test', False):
             args += ['-fs', compat_str(self._TEST_FILE_SIZE)]

yt_dlp/downloader/hls.py

@@ -81,6 +81,8 @@ class HlsFD(FragmentFD):
         man_url = info_dict['url']
         self.to_screen('[%s] Downloading m3u8 manifest' % self.FD_NAME)
 
+        is_webvtt = info_dict['ext'] == 'vtt'
+
         urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
         man_url = urlh.geturl()
         s = urlh.read().decode('utf-8', 'ignore')
@@ -99,11 +101,7 @@ class HlsFD(FragmentFD):
             #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
-        is_webvtt = info_dict['ext'] == 'vtt'
+        real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
-        if is_webvtt:
-            real_downloader = None  # Packing the fragments is not currently supported for external downloader
-        else:
-            real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
         if real_downloader and not real_downloader.supports_manifest(s):
             real_downloader = None
         if real_downloader:
@@ -424,9 +422,8 @@ class HlsFD(FragmentFD):
             max_workers = self.params.get('concurrent_fragment_downloads', 1)
             if can_threaded_download and max_workers > 1:
                 self.report_warning('The download speed shown is only of one thread. This is a known issue')
-                _download_fragment = lambda f: (f, download_fragment(f)[1])
                 with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments]
+                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments]
                     # timeout must be 0 to return instantly
                     done, not_done = concurrent.futures.wait(futures, timeout=0)
                     try:
@@ -440,13 +437,9 @@ class HlsFD(FragmentFD):
                         # timeout must be none to cancel
                         concurrent.futures.wait(not_done, timeout=None)
                         raise KeyboardInterrupt
+                results = [future.result() for future in futures]
 
-                for fragment, frag_index in map(lambda x: x.result(), futures):
+                for frag_content, frag_index in results:
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                    fragment['fragment_filename_sanitized'] = frag_sanitized
-                    frag_content = down.read()
-                    down.close()
                     result = append_fragment(frag_content, frag_index)
                     if not result:
                         return False

yt_dlp/extractor/archiveorg.py

@@ -1,36 +1,22 @@
-# coding: utf-8
 from __future__ import unicode_literals
 
 import re
 import json
 
 from .common import InfoExtractor
-from .youtube import YoutubeIE
+from ..compat import compat_urllib_parse_unquote_plus
-from ..compat import (
-    compat_urllib_parse_unquote,
-    compat_urllib_parse_unquote_plus,
-    compat_urlparse,
-    compat_parse_qs,
-    compat_HTTPError
-)
 from ..utils import (
-    clean_html,
-    determine_ext,
-    dict_get,
-    extract_attributes,
-    ExtractorError,
-    HEADRequest,
-    int_or_none,
     KNOWN_EXTENSIONS,
-    merge_dicts,
+
-    mimetype2ext,
+    extract_attributes,
-    parse_duration,
-    RegexNotFoundError,
-    str_to_int,
-    str_or_none,
-    try_get,
     unified_strdate,
     unified_timestamp,
+    clean_html,
+    dict_get,
+    parse_duration,
+    int_or_none,
+    str_or_none,
+    merge_dicts,
 )
 
 
@@ -255,165 +241,3 @@ class ArchiveOrgIE(InfoExtractor):
                     'parent': 'root'})
 
         return info
-
-
-class YoutubeWebArchiveIE(InfoExtractor):
-    IE_NAME = 'web.archive:youtube'
-    IE_DESC = 'web.archive.org saved youtube videos'
-    _VALID_URL = r"""(?x)^
-                (?:https?://)?web\.archive\.org/
-                    (?:web/)?
-                    (?:[0-9A-Za-z_*]+/)?  # /web and the version index is optional
-
-                (?:https?(?::|%3[Aa])//)?
-                (?:
-                    (?:\w+\.)?youtube\.com/watch(?:\?|%3[fF])(?:[^\#]+(?:&|%26))?v(?:=|%3[dD])  # Youtube URL
-                    |(wayback-fakeurl\.archive\.org/yt/)  # Or the internal fake url
-                )
-                (?P<id>[0-9A-Za-z_-]{11})(?:%26|\#|&|$)
-                """
-
-    _TESTS = [
-        {
-            'url': 'https://web.archive.org/web/20150415002341/https://www.youtube.com/watch?v=aYAGB11YrSs',
-            'info_dict': {
-                'id': 'aYAGB11YrSs',
-                'ext': 'webm',
-                'title': 'Team Fortress 2 - Sandviches!'
-            }
-        },
-        {
-            # Internal link
-            'url': 'https://web.archive.org/web/2oe/http://wayback-fakeurl.archive.org/yt/97t7Xj_iBv0',
-            'info_dict': {
-                'id': '97t7Xj_iBv0',
-                'ext': 'mp4',
-                'title': 'How Flexible Machines Could Save The World'
-            }
-        },
-        {
-            # Video from 2012, webm format itag 45.
-            'url': 'https://web.archive.org/web/20120712231619/http://www.youtube.com/watch?v=AkhihxRKcrs&gl=US&hl=en',
-            'info_dict': {
-                'id': 'AkhihxRKcrs',
-                'ext': 'webm',
-                'title': 'Limited Run: Mondo\'s Modern Classic 1 of 3 (SDCC 2012)'
-            }
-        },
-        {
-            # Old flash-only video. Webpage title starts with "YouTube - ".
-            'url': 'https://web.archive.org/web/20081211103536/http://www.youtube.com/watch?v=jNQXAC9IVRw',
-            'info_dict': {
-                'id': 'jNQXAC9IVRw',
-                'ext': 'unknown_video',
-                'title': 'Me at the zoo'
-            }
-        },
-        {
-            # Flash video with .flv extension (itag 34). Title has prefix "YouTube         -"
-            # Title has some weird unicode characters too.
-            'url': 'https://web.archive.org/web/20110712231407/http://www.youtube.com/watch?v=lTx3G6h2xyA',
-            'info_dict': {
-                'id': 'lTx3G6h2xyA',
-                'ext': 'flv',
-                'title': '‪Madeon - Pop Culture (live mashup)‬‏'
-            }
-        },
-        {   # Some versions of Youtube have have "YouTube" as page title in html (and later rewritten by js).
-            'url': 'https://web.archive.org/web/http://www.youtube.com/watch?v=kH-G_aIBlFw',
-            'info_dict': {
-                'id': 'kH-G_aIBlFw',
-                'ext': 'mp4',
-                'title': 'kH-G_aIBlFw'
-            },
-            'expected_warnings': [
-                'unable to extract title',
-            ]
-        },
-        {
-            # First capture is a 302 redirect intermediary page.
-            'url': 'https://web.archive.org/web/20050214000000/http://www.youtube.com/watch?v=0altSZ96U4M',
-            'info_dict': {
-                'id': '0altSZ96U4M',
-                'ext': 'mp4',
-                'title': '0altSZ96U4M'
-            },
-            'expected_warnings': [
-                'unable to extract title',
-            ]
-        },
-        {
-            # Video not archived, only capture is unavailable video page
-            'url': 'https://web.archive.org/web/20210530071008/https://www.youtube.com/watch?v=lHJTf93HL1s&spfreload=10',
-            'only_matching': True,
-        },
-        {   # Encoded url
-            'url': 'https://web.archive.org/web/20120712231619/http%3A//www.youtube.com/watch%3Fgl%3DUS%26v%3DAkhihxRKcrs%26hl%3Den',
-            'only_matching': True,
-        },
-        {
-            'url': 'https://web.archive.org/web/20120712231619/http%3A//www.youtube.com/watch%3Fv%3DAkhihxRKcrs%26gl%3DUS%26hl%3Den',
-            'only_matching': True,
-        }
-    ]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        title = video_id  # if we are not able get a title
-
-        def _extract_title(webpage):
-            page_title = self._html_search_regex(
-                r'<title>([^<]*)</title>', webpage, 'title', fatal=False) or ''
-            # YouTube video pages appear to always have either 'YouTube -' as suffix or '- YouTube' as prefix.
-            try:
-                page_title = self._html_search_regex(
-                    r'(?:YouTube\s*-\s*(.*)$)|(?:(.*)\s*-\s*YouTube$)',
-                    page_title, 'title', default='')
-            except RegexNotFoundError:
-                page_title = None
-
-            if not page_title:
-                self.report_warning('unable to extract title', video_id=video_id)
-                return
-            return page_title
-
-        # If the video is no longer available, the oldest capture may be one before it was removed.
-        # Setting the capture date in url to early date seems to redirect to earliest capture.
-        webpage = self._download_webpage(
-            'https://web.archive.org/web/20050214000000/http://www.youtube.com/watch?v=%s' % video_id,
-            video_id=video_id, fatal=False, errnote='unable to download video webpage (probably not archived).')
-        if webpage:
-            title = _extract_title(webpage) or title
-
-        # Use link translator mentioned in https://github.com/ytdl-org/youtube-dl/issues/13655
-        internal_fake_url = 'https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/%s' % video_id
-        try:
-            video_file_webpage = self._request_webpage(
-                HEADRequest(internal_fake_url), video_id,
-                note='Fetching video file url', expected_status=True)
-        except ExtractorError as e:
-            # HTTP Error 404 is expected if the video is not saved.
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 404:
-                raise ExtractorError(
-                    'HTTP Error %s. Most likely the video is not archived or issue with web.archive.org.' % e.cause.code,
-                    expected=True)
-            raise
-        video_file_url = compat_urllib_parse_unquote(video_file_webpage.url)
-        video_file_url_qs = compat_parse_qs(compat_urlparse.urlparse(video_file_url).query)
-
-        # Attempt to recover any ext & format info from playback url
-        format = {'url': video_file_url}
-        itag = try_get(video_file_url_qs, lambda x: x['itag'][0])
-        if itag and itag in YoutubeIE._formats:  # Naughty access but it works
-            format.update(YoutubeIE._formats[itag])
-            format.update({'format_id': itag})
-        else:
-            mime = try_get(video_file_url_qs, lambda x: x['mime'][0])
-            ext = mimetype2ext(mime) or determine_ext(video_file_url)
-            format.update({'ext': ext})
-        return {
-            'id': video_id,
-            'title': title,
-            'formats': [format],
-            'duration': str_to_int(try_get(video_file_url_qs, lambda x: x['dur'][0]))
-        }

yt_dlp/extractor/ard.py

@@ -290,14 +290,14 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
 
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/(?:[^/?#&]+/)+(?P<id>[^/?#&]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?:video-?)?(?P<id>[0-9]+))\.html'
     _TESTS = [{
         # available till 7.01.2022
         'url': 'https://www.daserste.de/information/talk/maischberger/videos/maischberger-die-woche-video100.html',
         'md5': '867d8aa39eeaf6d76407c5ad1bb0d4c1',
         'info_dict': {
-            'id': 'maischberger-die-woche-video100',
+            'display_id': 'maischberger-die-woche',
-            'display_id': 'maischberger-die-woche-video100',
+            'id': '100',
             'ext': 'mp4',
             'duration': 3687.0,
             'title': 'maischberger. die woche vom 7. Januar 2021',
@@ -305,28 +305,16 @@ class ARDIE(InfoExtractor):
             'thumbnail': r're:^https?://.*\.jpg$',
         },
     }, {
-        'url': 'https://www.daserste.de/information/politik-weltgeschehen/morgenmagazin/videosextern/dominik-kahun-aus-der-nhl-direkt-zur-weltmeisterschaft-100.html',
+        'url': 'https://www.daserste.de/information/reportage-dokumentation/erlebnis-erde/videosextern/woelfe-und-herdenschutzhunde-ungleiche-brueder-102.html',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.daserste.de/information/nachrichten-wetter/tagesthemen/videosextern/tagesthemen-17736.html',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.daserste.de/unterhaltung/serie/in-aller-freundschaft-die-jungen-aerzte/videos/diversity-tag-sanam-afrashteh100.html',
         'only_matching': True,
     }, {
         'url': 'http://www.daserste.de/information/reportage-dokumentation/dokus/videos/die-story-im-ersten-mission-unter-falscher-flagge-100.html',
         'only_matching': True,
-    }, {
-        'url': 'https://www.daserste.de/unterhaltung/serie/in-aller-freundschaft-die-jungen-aerzte/Drehpause-100.html',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.daserste.de/unterhaltung/film/filmmittwoch-im-ersten/videos/making-ofwendezeit-video-100.html',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        display_id = mobj.group('id')
+        display_id = mobj.group('display_id')
 
         player_url = mobj.group('mainurl') + '~playerXml.xml'
         doc = self._download_xml(player_url, display_id)
@@ -377,7 +365,7 @@ class ARDIE(InfoExtractor):
         self._sort_formats(formats)
 
         return {
-            'id': xpath_text(video_node, './videoId', default=display_id),
+            'id': mobj.group('id'),
             'formats': formats,
             'display_id': display_id,
             'title': video_node.find('./title').text,

yt_dlp/extractor/common.py

@@ -952,49 +952,6 @@ class InfoExtractor(object):
             else:
                 self.report_warning(errmsg + str(ve))
 
-    def _parse_socket_response_as_json(self, data, video_id, transform_source=None, fatal=True):
-        return self._parse_json(
-            data[data.find('{'):data.rfind('}') + 1],
-            video_id, transform_source, fatal)
-
-    def _download_socket_json_handle(
-            self, url_or_request, video_id, note='Polling socket',
-            errnote='Unable to poll socket', transform_source=None,
-            fatal=True, encoding=None, data=None, headers={}, query={},
-            expected_status=None):
-        """
-        Return a tuple (JSON object, URL handle).
-
-        See _download_webpage docstring for arguments specification.
-        """
-        res = self._download_webpage_handle(
-            url_or_request, video_id, note, errnote, fatal=fatal,
-            encoding=encoding, data=data, headers=headers, query=query,
-            expected_status=expected_status)
-        if res is False:
-            return res
-        webpage, urlh = res
-        return self._parse_socket_response_as_json(
-            webpage, video_id, transform_source=transform_source,
-            fatal=fatal), urlh
-
-    def _download_socket_json(
-            self, url_or_request, video_id, note='Polling socket',
-            errnote='Unable to poll socket', transform_source=None,
-            fatal=True, encoding=None, data=None, headers={}, query={},
-            expected_status=None):
-        """
-        Return the JSON object as a dict.
-
-        See _download_webpage docstring for arguments specification.
-        """
-        res = self._download_socket_json_handle(
-            url_or_request, video_id, note=note, errnote=errnote,
-            transform_source=transform_source, fatal=fatal, encoding=encoding,
-            data=data, headers=headers, query=query,
-            expected_status=expected_status)
-        return res if res is False else res[0]
-
     def report_warning(self, msg, video_id=None, *args, **kwargs):
         idstr = '' if video_id is None else '%s: ' % video_id
         self._downloader.report_warning(
@@ -1945,15 +1902,15 @@ class InfoExtractor(object):
         return fmts
 
     def _extract_m3u8_formats_and_subtitles(
-            self, m3u8_url, video_id, ext=None, entry_protocol='m3u8_native',
+            self, m3u8_url, video_id, ext=None, entry_protocol='m3u8',
             preference=None, quality=None, m3u8_id=None, note=None,
             errnote=None, fatal=True, live=False, data=None, headers={},
             query={}):
 
         res = self._download_webpage_handle(
             m3u8_url, video_id,
-            note='Downloading m3u8 information' if note is None else note,
+            note=note or 'Downloading m3u8 information',
-            errnote='Failed to download m3u8 information' if errnote is None else errnote,
+            errnote=errnote or 'Failed to download m3u8 information',
             fatal=fatal, data=data, headers=headers, query=query)
 
         if res is False:
@@ -1969,7 +1926,7 @@ class InfoExtractor(object):
             headers=headers, query=query, video_id=video_id)
 
     def _parse_m3u8_formats_and_subtitles(
-            self, m3u8_doc, m3u8_url, ext=None, entry_protocol='m3u8_native',
+            self, m3u8_doc, m3u8_url, ext=None, entry_protocol='m3u8',
             preference=None, quality=None, m3u8_id=None, live=False, note=None,
             errnote=None, fatal=True, data=None, headers={}, query={},
             video_id=None):
@@ -2085,12 +2042,7 @@ class InfoExtractor(object):
             groups.setdefault(group_id, []).append(media)
             # <https://tools.ietf.org/html/rfc8216#section-4.3.4.1>
             if media_type == 'SUBTITLES':
-                # According to RFC 8216 §4.3.4.2.1, URI is REQUIRED in the
+                lang = media['LANGUAGE']  # XXX: normalise?
-                # EXT-X-MEDIA tag if the media type is SUBTITLES.
-                # However, lack of URI has been spotted in the wild.
-                # e.g. NebulaIE; see https://github.com/yt-dlp/yt-dlp/issues/339
-                if not media.get('URI'):
-                    return
                 url = format_url(media['URI'])
                 sub_info = {
                     'url': url,
@@ -2102,7 +2054,6 @@ class InfoExtractor(object):
                     # <https://tools.ietf.org/html/rfc8216#section-3.1>
                     sub_info['ext'] = 'vtt'
                     sub_info['protocol'] = 'm3u8_native'
-                lang = media.get('LANGUAGE') or 'und'
                 subtitles.setdefault(lang, []).append(sub_info)
             if media_type not in ('VIDEO', 'AUDIO'):
                 return
@@ -2508,8 +2459,8 @@ class InfoExtractor(object):
             fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             mpd_url, video_id,
-            note='Downloading MPD manifest' if note is None else note,
+            note=note or 'Downloading MPD manifest',
-            errnote='Failed to download MPD manifest' if errnote is None else errnote,
+            errnote=errnote or 'Failed to download MPD manifest',
             fatal=fatal, data=data, headers=headers, query=query)
         if res is False:
             return [], {}
@@ -2838,8 +2789,8 @@ class InfoExtractor(object):
     def _extract_ism_formats_and_subtitles(self, ism_url, video_id, ism_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             ism_url, video_id,
-            note='Downloading ISM manifest' if note is None else note,
+            note=note or 'Downloading ISM manifest',
-            errnote='Failed to download ISM manifest' if errnote is None else errnote,
+            errnote=errnote or 'Failed to download ISM manifest',
             fatal=fatal, data=data, headers=headers, query=query)
         if res is False:
             return [], {}
@@ -3530,7 +3481,7 @@ class InfoExtractor(object):
         return compat_urllib_parse_unquote(os.path.splitext(url_basename(url))[0])
 
     @staticmethod
-    def _availability(is_private=None, needs_premium=None, needs_subscription=None, needs_auth=None, is_unlisted=None):
+    def _availability(is_private, needs_premium, needs_subscription, needs_auth, is_unlisted):
         all_known = all(map(
             lambda x: x is not None,
             (is_private, needs_premium, needs_subscription, needs_auth, is_unlisted)))

yt_dlp/extractor/egghead.py

@@ -107,7 +107,8 @@ class EggheadLessonIE(EggheadBaseIE):
             ext = determine_ext(format_url)
             if ext == 'm3u8':
                 formats.extend(self._extract_m3u8_formats(
-                    format_url, lesson_id, 'mp4', m3u8_id='hls', fatal=False))
+                    format_url, lesson_id, 'mp4', entry_protocol='m3u8',
+                    m3u8_id='hls', fatal=False))
             elif ext == 'mpd':
                 formats.extend(self._extract_mpd_formats(
                     format_url, lesson_id, mpd_id='dash', fatal=False))

yt_dlp/extractor/extractors.py

@@ -67,10 +67,7 @@ from .appletrailers import (
     AppleTrailersSectionIE,
 )
 from .applepodcasts import ApplePodcastsIE
-from .archiveorg import (
+from .archiveorg import ArchiveOrgIE
-    ArchiveOrgIE,
-    YoutubeWebArchiveIE,
-)
 from .arcpublishing import ArcPublishingIE
 from .arkena import ArkenaIE
 from .ard import (
@@ -399,7 +396,6 @@ from .facebook import (
     FacebookIE,
     FacebookPluginsVideoIE,
 )
-from .fancode import FancodeVodIE
 from .faz import FazIE
 from .fc2 import (
     FC2IE,
@@ -505,7 +501,6 @@ from .hotnewhiphop import HotNewHipHopIE
 from .hotstar import (
     HotStarIE,
     HotStarPlaylistIE,
-    HotStarSeriesIE,
 )
 from .howcast import HowcastIE
 from .howstuffworks import HowStuffWorksIE
@@ -1124,7 +1119,6 @@ from .safari import (
     SafariApiIE,
     SafariCourseIE,
 )
-from .saitosan import SaitosanIE
 from .samplefocus import SampleFocusIE
 from .sapo import SapoIE
 from .savefrom import SaveFromIE
@@ -1157,7 +1151,6 @@ from .shared import (
     SharedIE,
     VivoIE,
 )
-from .shemaroome import ShemarooMeIE
 from .showroomlive import ShowRoomLiveIE
 from .simplecast import (
     SimplecastIE,
@@ -1191,10 +1184,7 @@ from .slideslive import SlidesLiveIE
 from .slutload import SlutloadIE
 from .snotr import SnotrIE
 from .sohu import SohuIE
-from .sonyliv import (
+from .sonyliv import SonyLIVIE
-    SonyLIVIE,
-    SonyLIVSeriesIE,
-)
 from .soundcloud import (
     SoundcloudEmbedIE,
     SoundcloudIE,
@@ -1302,7 +1292,6 @@ from .telebruxelles import TeleBruxellesIE
 from .telecinco import TelecincoIE
 from .telegraaf import TelegraafIE
 from .telemb import TeleMBIE
-from .telemundo import TelemundoIE
 from .telequebec import (
     TeleQuebecIE,
     TeleQuebecSquatIE,
@@ -1564,10 +1553,7 @@ from .vodlocker import VodlockerIE
 from .vodpl import VODPlIE
 from .vodplatform import VODPlatformIE
 from .voicerepublic import VoiceRepublicIE
-from .voot import (
+from .voot import VootIE
-    VootIE,
-    VootSeriesIE,
-)
 from .voxmedia import (
     VoxMediaVolumeIE,
     VoxMediaIE,

yt_dlp/extractor/fancode.py

@@ -1,91 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-
-from ..compat import compat_str
-from ..utils import (
-    parse_iso8601,
-    ExtractorError,
-    try_get
-)
-
-
-class FancodeVodIE(InfoExtractor):
-    IE_NAME = 'fancode:vod'
-
-    _VALID_URL = r'https?://(?:www\.)?fancode\.com/video/(?P<id>[0-9]+)\b'
-
-    _TESTS = [{
-        'url': 'https://fancode.com/video/15043/match-preview-pbks-vs-mi',
-        'params': {
-            'skip_download': True,
-            'format': 'bestvideo'
-        },
-        'info_dict': {
-            'id': '6249806281001',
-            'ext': 'mp4',
-            'title': 'Match Preview: PBKS vs MI',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            "timestamp": 1619081590,
-            'view_count': int,
-            'like_count': int,
-            'upload_date': '20210422',
-            'uploader_id': '6008340455001'
-        }
-    }, {
-        'url': 'https://fancode.com/video/15043',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-
-        BRIGHTCOVE_URL_TEMPLATE = 'https://players.brightcove.net/%s/default_default/index.html?videoId=%s'
-
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        brightcove_user_id = self._html_search_regex(
-            r'(?:https?://)?players\.brightcove\.net/(\d+)/default_default/index(?:\.min)?\.js',
-            webpage, 'user id')
-
-        data = '''{
-            "query":"query Video($id: Int\\u0021, $filter: SegmentFilter) { media(id: $id, filter: $filter) { id contentId title contentId publishedTime totalViews totalUpvotes provider thumbnail { src } mediaSource {brightcove } duration isPremium isUserEntitled tags duration }}",
-            "variables":{
-                "id":%s,
-                "filter":{
-                    "contentDataType":"DEFAULT"
-                }
-            },
-            "operationName":"Video"
-            }''' % video_id
-
-        metadata_json = self._download_json(
-            'https://www.fancode.com/graphql', video_id, data=data.encode(), note='Downloading metadata',
-            headers={
-                'content-type': 'application/json',
-                'origin': 'https://fancode.com',
-                'referer': url,
-            })
-
-        media = try_get(metadata_json, lambda x: x['data']['media'], dict) or {}
-        brightcove_video_id = try_get(media, lambda x: x['mediaSource']['brightcove'], compat_str)
-
-        if brightcove_video_id is None:
-            raise ExtractorError('Unable to extract brightcove Video ID')
-
-        is_premium = media.get('isPremium')
-        if is_premium:
-            self.report_warning('this video requires a premium account', video_id)
-
-        return {
-            '_type': 'url_transparent',
-            'url': BRIGHTCOVE_URL_TEMPLATE % (brightcove_user_id, brightcove_video_id),
-            'ie_key': 'BrightcoveNew',
-            'id': video_id,
-            'title': media['title'],
-            'like_count': media.get('totalUpvotes'),
-            'view_count': media.get('totalViews'),
-            'tags': media.get('tags'),
-            'release_timestamp': parse_iso8601(media.get('publishedTime')),
-            'availability': self._availability(needs_premium=is_premium),
-        }

yt_dlp/extractor/hotstar.py

@@ -87,14 +87,7 @@ class HotStarBaseIE(InfoExtractor):
 
 class HotStarIE(HotStarBaseIE):
     IE_NAME = 'hotstar'
-    _VALID_URL = r'''(?x)
+    _VALID_URL = r'https?://(?:www\.)?hotstar\.com/.*(?P<id>\d{10})'
-                           https?://(?:www\.)?hotstar\.com(?:/in)?/(?!in/)
-                           (?:
-                               tv/(?:[^/?#]+/){3}|
-                               (?!tv/)[^?#]+/
-                           )?
-                           (?P<id>\d{10})
-                   '''
     _TESTS = [{
         # contentData
         'url': 'https://www.hotstar.com/can-you-not-spread-rumours/1000076273',
@@ -242,41 +235,3 @@ class HotStarPlaylistIE(HotStarBaseIE):
             if video.get('contentId')]
 
         return self.playlist_result(entries, playlist_id)
-
-
-class HotStarSeriesIE(HotStarBaseIE):
-    IE_NAME = 'hotstar:series'
-    _VALID_URL = r'(?:https?://)(?:www\.)?hotstar\.com(?:/in)?/tv/[^/]+/(?P<id>\d{10})$'
-    _TESTS = [{
-        'url': 'https://www.hotstar.com/in/tv/radhakrishn/1260000646',
-        'info_dict': {
-            'id': '1260000646',
-        },
-        'playlist_mincount': 690,
-    }, {
-        'url': 'https://www.hotstar.com/tv/dancee-/1260050431',
-        'info_dict': {
-            'id': '1260050431',
-        },
-        'playlist_mincount': 43,
-    }]
-
-    def _real_extract(self, url):
-        series_id = self._match_id(url)
-        headers = {
-            'x-country-code': 'IN',
-            'x-platform-code': 'PCTV',
-        }
-        detail_json = self._download_json('https://api.hotstar.com/o/v1/show/detail?contentId=' + series_id,
-                                          video_id=series_id, headers=headers)
-        id = compat_str(try_get(detail_json, lambda x: x['body']['results']['item']['id'], int))
-        item_json = self._download_json('https://api.hotstar.com/o/v1/tray/g/1/items?etid=0&tao=0&tas=10000&eid=' + id,
-                                        video_id=series_id, headers=headers)
-        entries = [
-            self.url_result(
-                'https://www.hotstar.com/%d' % video['contentId'],
-                ie=HotStarIE.ie_key(), video_id=video['contentId'])
-            for video in item_json['body']['results']['items']
-            if video.get('contentId')]
-
-        return self.playlist_result(entries, series_id)

yt_dlp/extractor/odnoklassniki.py

@@ -247,7 +247,8 @@ class OdnoklassnikiIE(InfoExtractor):
         m3u8_url = metadata.get('hlsMasterPlaylistUrl')
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
-                m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+                m3u8_url, video_id, 'mp4', entry_protocol='m3u8',
+                m3u8_id='hls', fatal=False))
         rtmp_url = metadata.get('rtmpUrl')
         if rtmp_url:
             formats.append({

yt_dlp/extractor/patreon.py

@@ -2,9 +2,6 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from .vimeo import VimeoIE
-
-from ..compat import compat_urllib_parse_unquote
 from ..utils import (
     clean_html,
     determine_ext,
@@ -14,7 +11,6 @@ from ..utils import (
     parse_iso8601,
     str_or_none,
     try_get,
-    url_or_none
 )
 
 
@@ -67,20 +63,6 @@ class PatreonIE(InfoExtractor):
     }, {
         'url': 'https://www.patreon.com/posts/743933',
         'only_matching': True,
-    }, {
-        'url': 'https://www.patreon.com/posts/kitchen-as-seen-51706779',
-        'md5': '96656690071f6d64895866008484251b',
-        'info_dict': {
-            'id': '555089736',
-            'ext': 'mp4',
-            'title': 'KITCHEN AS SEEN ON DEEZ NUTS EXTENDED!',
-            'uploader': 'Cold Ones',
-            'thumbnail': 're:^https?://.*$',
-            'upload_date': '20210526',
-            'description': 'md5:557a409bd79d3898689419094934ba79',
-            'uploader_id': '14936315',
-        },
-        'skip': 'Patron-only content'
     }]
 
     # Currently Patreon exposes download URL via hidden CSS, so login is not
@@ -154,19 +136,6 @@ class PatreonIE(InfoExtractor):
                         'uploader_url': user_attributes.get('url'),
                     })
 
-        if not info.get('url'):
-            # handle Vimeo embeds
-            if try_get(attributes, lambda x: x['embed']['provider']) == 'Vimeo':
-                embed_html = try_get(attributes, lambda x: x['embed']['html'])
-                v_url = url_or_none(compat_urllib_parse_unquote(
-                    self._search_regex(r'src=(https%3A%2F%2Fplayer\.vimeo\.com.+)%3F', embed_html, 'vimeo url', fatal=False)))
-                if v_url:
-                    info.update({
-                        '_type': 'url_transparent',
-                        'url': VimeoIE._smuggle_referrer(v_url, 'https://patreon.com'),
-                        'ie_key': 'Vimeo',
-                    })
-
         if not info.get('url'):
             embed_url = try_get(attributes, lambda x: x['embed']['url'])
             if embed_url:

yt_dlp/extractor/saitosan.py

@@ -1,78 +0,0 @@
-# coding: utf-8
-
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import ExtractorError, try_get
-
-
-class SaitosanIE(InfoExtractor):
-    IE_NAME = 'Saitosan'
-    _VALID_URL = r'https?://(?:www\.)?saitosan\.net/bview.html\?id=(?P<id>[0-9]+)'
-    _TESTS = [{
-        'url': 'http://www.saitosan.net/bview.html?id=10031846',
-        'info_dict': {
-            'id': '10031846',
-            'ext': 'mp4',
-            'title': '井下原 和弥',
-            'uploader': '井下原 和弥',
-            'thumbnail': 'http://111.171.196.85:8088/921f916f-7f55-4c97-b92e-5d9d0fef8f5f/thumb',
-            'is_live': True,
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'skip': 'Broadcasts are ephemeral',
-    },
-        {
-        'url': 'http://www.saitosan.net/bview.html?id=10031795',
-        'info_dict': {
-            'id': '10031795',
-            'ext': 'mp4',
-            'title': '橋本',
-            'uploader': '橋本',
-            'thumbnail': 'http://111.171.196.85:8088/1a3933e1-a01a-483b-8931-af15f37f8082/thumb',
-            'is_live': True,
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'skip': 'Broadcasts are ephemeral',
-    }]
-
-    def _real_extract(self, url):
-        b_id = self._match_id(url)
-
-        base = 'http://hankachi.saitosan-api.net:8002/socket.io/?transport=polling&EIO=3'
-        sid = self._download_socket_json(base, b_id, note='Opening socket').get('sid')
-        base += '&sid=' + sid
-
-        self._download_webpage(base, b_id, note='Polling socket')
-        payload = '420["room_start_join",{"room_id":"%s"}]' % b_id
-        payload = '%s:%s' % (len(payload), payload)
-
-        self._download_webpage(base, b_id, data=payload, note='Polling socket with payload')
-        response = self._download_socket_json(base, b_id, note='Polling socket')
-        if not response.get('ok'):
-            err = response.get('error') or {}
-            raise ExtractorError(
-                '%s said: %s - %s' % (self.IE_NAME, err.get('code', '?'), err.get('msg', 'Unknown')) if err
-                else 'The socket reported that the broadcast could not be joined. Maybe it\'s offline or the URL is incorrect',
-                expected=True, video_id=b_id)
-
-        self._download_webpage(base, b_id, data='26:421["room_finish_join",{}]', note='Polling socket')
-        b_data = self._download_socket_json(base, b_id, note='Getting broadcast metadata from socket')
-        m3u8_url = b_data.get('url')
-
-        self._download_webpage(base, b_id, data='1:1', note='Closing socket', fatal=False)
-
-        return {
-            'id': b_id,
-            'title': b_data.get('name'),
-            'formats': self._extract_m3u8_formats(m3u8_url, b_id, 'mp4', live=True),
-            'thumbnail': m3u8_url.replace('av.m3u8', 'thumb'),
-            'uploader': try_get(b_data, lambda x: x['broadcast_user']['name']),  # same as title
-            'is_live': True
-        }

yt_dlp/extractor/shemaroome.py

@@ -1,76 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..aes import aes_cbc_decrypt
-from ..compat import (
-    compat_b64decode,
-    compat_ord,
-)
-from ..utils import (
-    bytes_to_intlist,
-    intlist_to_bytes,
-    unified_strdate,
-    url_or_none,
-)
-
-
-class ShemarooMeIE(InfoExtractor):
-    _VALID_URL = r'(?:https?://)(?:www\.)?shemaroome\.com/(?:movies|shows)/(?P<id>[^?#]+)'
-    _TESTS = [{
-        'url': 'https://www.shemaroome.com/movies/dil-hai-tumhaara',
-        'info_dict': {
-            'id': 'dil-hai-tumhaara',
-            'ext': 'mp4',
-            'title': 'Dil Hai Tumhaara',
-            'release_date': '20020906',
-            'thumbnail': 'https://daex9l847wg3n.cloudfront.net/shemoutputimages/Dil-Hai-Tumhaara/60599346a609d2faa3000020/large_16_9_1616436538.jpg?1616483693',
-            'description': 'md5:2782c4127807103cf5a6ae2ca33645ce',
-        },
-        'params': {
-            'skip_download': True
-        }
-    }, {
-        'url': 'https://www.shemaroome.com/shows/jurm-aur-jazbaat/laalach',
-        'info_dict': {
-            'id': 'jurm-aur-jazbaat_laalach',
-            'ext': 'mp4',
-            'title': 'Laalach',
-            'description': 'md5:92b79c2dcb539b0ab53f9fa5a048f53c',
-            'release_date': '20210507',
-        },
-        'params': {
-            'skip_download': True
-        }
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url).replace('/', '_')
-        webpage = self._download_webpage(url, video_id)
-        m = re.search(
-            r'params_for_player\s*=\s*"(?P<data>[^|]+)\|key=(?P<key>[^|]+)\|image=(?P<thumbnail>[^|]+)\|title=(?P<title>[^|]+)',
-            webpage)
-        data = bytes_to_intlist(compat_b64decode(m.group('data')))
-        key = bytes_to_intlist(compat_b64decode(m.group('key')))
-        iv = [0] * 16
-        m3u8_url = intlist_to_bytes(aes_cbc_decrypt(data, key, iv))
-        m3u8_url = m3u8_url[:-compat_ord((m3u8_url[-1]))].decode('ascii')
-        formats = self._extract_m3u8_formats(m3u8_url, video_id, fatal=False)
-        self._sort_formats(formats)
-
-        release_date = self._html_search_regex(
-            (r'itemprop="uploadDate">\s*([\d-]+)', r'id="release_date" value="([\d-]+)'),
-            webpage, 'release date', fatal=False)
-
-        description = self._html_search_regex(r'(?s)>Synopsis(</.+?)</', webpage, 'description', fatal=False)
-
-        return {
-            'id': video_id,
-            'formats': formats,
-            'title': m.group('title'),
-            'thumbnail': url_or_none(m.group('thumbnail')),
-            'release_date': unified_strdate(release_date),
-            'description': description,
-        }

yt_dlp/extractor/sonyliv.py

@@ -9,18 +9,11 @@ from ..compat import compat_HTTPError
 from ..utils import (
     ExtractorError,
     int_or_none,
-    try_get,
 )
 
 
 class SonyLIVIE(InfoExtractor):
-    _VALID_URL = r'''(?x)
+    _VALID_URL = r'https?://(?:www\.)?sonyliv\.com/(?:s(?:how|port)s/[^/]+|movies|clip|trailer|music-videos)/[^/?#&]+-(?P<id>\d+)'
-                     (?:
-                        sonyliv:|
-                        https?://(?:www\.)?sonyliv\.com/(?:s(?:how|port)s/[^/]+|movies|clip|trailer|music-videos)/[^/?#&]+-
-                    )
-                    (?P<id>\d+)
-                  '''
     _TESTS = [{
         'url': 'https://www.sonyliv.com/shows/bachelors-delight-1700000113/achaari-cheese-toast-1000022678?watch=true',
         'info_dict': {
@@ -100,14 +93,7 @@ class SonyLIVIE(InfoExtractor):
         metadata = self._call_api(
             '1.6', 'IN/DETAIL/' + video_id, video_id)['containers'][0]['metadata']
         title = metadata['episodeTitle']
-        subtitles = {}
+
-        for sub in content.get('subtitle', []):
-            sub_url = sub.get('subtitleUrl')
-            if not sub_url:
-                continue
-            subtitles.setdefault(sub.get('subtitleLanguageName', 'ENG'), []).append({
-                'url': sub_url,
-            })
         return {
             'id': video_id,
             'title': title,
@@ -120,43 +106,4 @@ class SonyLIVIE(InfoExtractor):
             'series': metadata.get('title'),
             'episode_number': int_or_none(metadata.get('episodeNumber')),
             'release_year': int_or_none(metadata.get('year')),
-            'subtitles': subtitles,
         }
-
-
-class SonyLIVSeriesIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?sonyliv\.com/shows/[^/?#&]+-(?P<id>\d{10})$'
-    _TESTS = [{
-        'url': 'https://www.sonyliv.com/shows/adaalat-1700000091',
-        'playlist_mincount': 456,
-        'info_dict': {
-            'id': '1700000091',
-        },
-    }]
-    _API_SHOW_URL = "https://apiv2.sonyliv.com/AGL/1.9/R/ENG/WEB/IN/DL/DETAIL/{}?kids_safe=false&from=0&to=49"
-    _API_EPISODES_URL = "https://apiv2.sonyliv.com/AGL/1.4/R/ENG/WEB/IN/CONTENT/DETAIL/BUNDLE/{}?from=0&to=1000&orderBy=episodeNumber&sortOrder=asc"
-    _API_SECURITY_URL = 'https://apiv2.sonyliv.com/AGL/1.4/A/ENG/WEB/ALL/GETTOKEN'
-
-    def _entries(self, show_id):
-        headers = {
-            'Accept': 'application/json, text/plain, */*',
-            'Referer': 'https://www.sonyliv.com',
-        }
-        headers['security_token'] = self._download_json(
-            self._API_SECURITY_URL, video_id=show_id, headers=headers,
-            note='Downloading security token')['resultObj']
-        seasons = try_get(
-            self._download_json(self._API_SHOW_URL.format(show_id), video_id=show_id, headers=headers),
-            lambda x: x['resultObj']['containers'][0]['containers'], list)
-        for season in seasons or []:
-            season_id = season['id']
-            episodes = try_get(
-                self._download_json(self._API_EPISODES_URL.format(season_id), video_id=season_id, headers=headers),
-                lambda x: x['resultObj']['containers'][0]['containers'], list)
-            for episode in episodes or []:
-                video_id = episode.get('id')
-                yield self.url_result('sonyliv:%s' % video_id, ie=SonyLIVIE.ie_key(), video_id=video_id)
-
-    def _real_extract(self, url):
-        show_id = self._match_id(url)
-        return self.playlist_result(self._entries(show_id), playlist_id=show_id)

yt_dlp/extractor/ted.py

@@ -123,10 +123,6 @@ class TEDIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
-    }, {
-        # with own formats and private Youtube external
-        'url': 'https://www.ted.com/talks/spencer_wells_a_family_tree_for_humanity',
-        'only_matching': True,
     }]
 
     _NATIVE_FORMATS = {
@@ -214,6 +210,16 @@ class TEDIE(InfoExtractor):
 
         player_talk = talk_info['player_talks'][0]
 
+        external = player_talk.get('external')
+        if isinstance(external, dict):
+            service = external.get('service')
+            if isinstance(service, compat_str):
+                ext_url = None
+                if service.lower() == 'youtube':
+                    ext_url = external.get('code')
+
+                return self.url_result(ext_url or external['uri'])
+
         resources_ = player_talk.get('resources') or talk_info.get('resources')
 
         http_url = None
@@ -288,16 +294,6 @@ class TEDIE(InfoExtractor):
                 'vcodec': 'none',
             })
 
-        if not formats:
-            external = player_talk.get('external')
-            if isinstance(external, dict):
-                service = external.get('service')
-                if isinstance(service, compat_str):
-                    ext_url = None
-                    if service.lower() == 'youtube':
-                        ext_url = external.get('code')
-                    return self.url_result(ext_url or external['uri'])
-
         self._sort_formats(formats)
 
         video_id = compat_str(talk_info['id'])

yt_dlp/extractor/telemundo.py

@@ -1,58 +0,0 @@
-# coding=utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    try_get,
-    unified_timestamp,
-    HEADRequest,
-)
-
-
-class TelemundoIE(InfoExtractor):
-
-    _VALID_URL = r'https?:\/\/(?:www\.)?telemundo\.com\/.+?video\/[^\/]+(?P<id>tmvo\d{7})'
-    _TESTS = [{
-        'url': 'https://www.telemundo.com/noticias/noticias-telemundo-en-la-noche/empleo/video/esta-aplicacion-gratuita-esta-ayudando-los-latinos-encontrar-trabajo-en-estados-unidos-tmvo9829325',
-        'info_dict': {
-            'id': 'tmvo9829325',
-            'timestamp': 1621396800,
-            'title': 'Esta aplicación gratuita está ayudando a los latinos a encontrar trabajo en Estados Unidos',
-            'uploader': 'Telemundo',
-            'uploader_id': 'NBCU_Telemundo',
-            'ext': 'mp4',
-            'upload_date': '20210519',
-        },
-        'params': {
-            'skip_download': True,
-        }
-    }, {
-        'url': 'https://www.telemundo.com/shows/al-rojo-vivo/empleo/video/personajes-de-times-square-piden-que-la-ciudad-de-nueva-york-los-deje-volver-trabajar-tmvo9816272',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        metadata = self._parse_json(
-            self._search_regex(r'<[^>]+id="__NEXT_DATA__"[^>]+>([^<]+)', webpage, 'JSON metadata'), video_id)
-        redirect_url = try_get(
-            metadata,
-            lambda x: x['props']['initialState']['video']['associatedPlaylists'][0]['videos'][0]['videoAssets'][0]['publicUrl'])
-
-        m3u8_url = self._request_webpage(HEADRequest(
-            redirect_url + '?format=redirect&manifest=m3u&format=redirect&Tracking=true&Embedded=true&formats=MPEG4'),
-            video_id, 'Processing m3u8').geturl()
-        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4')
-        self._sort_formats(formats)
-        date = unified_timestamp(try_get(
-            metadata, lambda x: x['props']['initialState']['video']['associatedPlaylists'][0]['videos'][0]['datePublished'].split(' ', 1)[1]))
-        return {
-            'url': url,
-            'id': video_id,
-            'title': self._search_regex(r'<h1[^>]+>([^<]+)', webpage, 'title', fatal=False),
-            'formats': formats,
-            'timestamp': date,
-            'uploader': 'Telemundo',
-            'uploader_id': self._search_regex(r'https?:\/\/(?:[^/]+\/){3}video\/(?P<id>[^\/]+)', m3u8_url, 'Akamai account', fatal=False)
-        }

yt_dlp/extractor/twitch.py

@@ -49,7 +49,6 @@ class TwitchBaseIE(InfoExtractor):
         'ChannelCollectionsContent': '07e3691a1bad77a36aba590c351180439a40baefc1c275356f40fc7082419a84',
         'StreamMetadata': '1c719a40e481453e5c48d9bb585d971b8b372f8ebb105b17076722264dfa5b3e',
         'ComscoreStreamingQuery': 'e1edae8122517d013405f237ffcc124515dc6ded82480a88daef69c83b53ac01',
-        'VideoAccessToken_Clip': '36b89d2507fce29e5ca551df756d27c1cfe079e2609642b4390aa4c35796eb11',
         'VideoPreviewOverlay': '3006e77e51b128d838fa4e835723ca4dc9a05c5efd4466c1085215c6e437e65c',
         'VideoMetadata': '226edb3e692509f727fd56821f5653c05740242c82b0388883e0c0e75dcbf687',
     }
@@ -894,25 +893,7 @@ class TwitchClipsIE(TwitchBaseIE):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        clip = self._download_gql(
+        clip = self._download_base_gql(
-            video_id, [{
-                'operationName': 'VideoAccessToken_Clip',
-                'variables': {
-                    'slug': video_id,
-                },
-            }],
-            'Downloading clip access token GraphQL')[0]['data']['clip']
-
-        if not clip:
-            raise ExtractorError(
-                'This clip is no longer available', expected=True)
-
-        access_query = {
-            'sig': clip['playbackAccessToken']['signature'],
-            'token': clip['playbackAccessToken']['value'],
-        }
-
-        data = self._download_base_gql(
             video_id, {
                 'query': '''{
   clip(slug: "%s") {
@@ -937,10 +918,11 @@ class TwitchClipsIE(TwitchBaseIE):
     }
     viewCount
   }
-}''' % video_id}, 'Downloading clip GraphQL', fatal=False)
+}''' % video_id}, 'Downloading clip GraphQL')['data']['clip']
 
-        if data:
+        if not clip:
-            clip = try_get(data, lambda x: x['data']['clip'], dict) or clip
+            raise ExtractorError(
+                'This clip is no longer available', expected=True)
 
         formats = []
         for option in clip.get('videoQualities', []):
@@ -950,7 +932,7 @@ class TwitchClipsIE(TwitchBaseIE):
             if not source:
                 continue
             formats.append({
-                'url': update_url_query(source, access_query),
+                'url': source,
                 'format_id': option.get('quality'),
                 'height': int_or_none(option.get('quality')),
                 'fps': int_or_none(option.get('frameRate')),

yt_dlp/extractor/ustream.py

@@ -75,7 +75,7 @@ class UstreamIE(InfoExtractor):
     @staticmethod
     def _extract_url(webpage):
         mobj = re.search(
-            r'<iframe[^>]+?src=(["\'])(?P<url>https?://(?:www\.)?(?:ustream\.tv|video\.ibm\.com)/embed/.+?)\1', webpage)
+            r'<iframe[^>]+?src=(["\'])(?P<url>http://(?:www\.)?(?:ustream\.tv|video\.ibm\.com)/embed/.+?)\1', webpage)
         if mobj is not None:
             return mobj.group('url')
 

yt_dlp/extractor/vidio.py

@@ -5,14 +5,11 @@ import re
 
 from .common import InfoExtractor
 from ..utils import (
-    ExtractorError,
-    get_element_by_class,
     int_or_none,
     parse_iso8601,
     str_or_none,
     strip_or_none,
     try_get,
-    urlencode_postdata,
 )
 
 
@@ -44,49 +41,11 @@ class VidioIE(InfoExtractor):
     }, {
         'url': 'https://www.vidio.com/watch/77949-south-korea-test-fires-missile-that-can-strike-all-of-the-north',
         'only_matching': True,
-    }, {
-        # Premier-exclusive video
-        'url': 'https://www.vidio.com/watch/1550718-stand-by-me-doraemon',
-        'only_matching': True
     }]
-    _LOGIN_URL = 'https://www.vidio.com/users/login'
-    _NETRC_MACHINE = 'vidio'
-
-    def _login(self):
-        username, password = self._get_login_info()
-        if username is None:
-            return
-
-        def is_logged_in():
-            res = self._download_json(
-                'https://www.vidio.com/interactions.json', None, 'Checking if logged in', fatal=False) or {}
-            return bool(res.get('current_user'))
-
-        if is_logged_in():
-            return
-
-        login_page = self._download_webpage(
-            self._LOGIN_URL, None, 'Downloading log in page')
-
-        login_form = self._form_hidden_inputs("login-form", login_page)
-        login_form.update({
-            'user[login]': username,
-            'user[password]': password,
-        })
-        login_post, login_post_urlh = self._download_webpage_handle(
-            self._LOGIN_URL, None, 'Logging in', data=urlencode_postdata(login_form), expected_status=[302, 401])
-
-        if login_post_urlh.status == 401:
-            reason = get_element_by_class('onboarding-form__general-error', login_post)
-            if reason:
-                raise ExtractorError(
-                    'Unable to log in: %s' % reason, expected=True)
-            raise ExtractorError('Unable to log in')
 
     def _real_initialize(self):
         self._api_key = self._download_json(
             'https://www.vidio.com/auth', None, data=b'')['api_key']
-        self._login()
 
     def _real_extract(self, url):
         video_id, display_id = re.match(self._VALID_URL, url).groups()
@@ -97,30 +56,9 @@ class VidioIE(InfoExtractor):
             })
         video = data['videos'][0]
         title = video['title'].strip()
-        is_premium = video.get('is_premium')
-        if is_premium:
-            sources = self._download_json(
-                'https://www.vidio.com/interactions_stream.json?video_id=%s&type=videos' % video_id,
-                display_id, note='Downloading premier API JSON')
-            if not (sources.get('source') or sources.get('source_dash')):
-                self.raise_login_required('This video is only available for registered users with a premier subscription.')
-
-            formats, subs = [], {}
-            if sources.get('source'):
-                hls_formats, hls_subs = self._extract_m3u8_formats_and_subtitles(
-                    sources['source'], display_id, 'mp4', 'm3u8_native')
-                formats.extend(hls_formats)
-                subs.update(hls_subs)
-            if sources.get('source_dash'):  # TODO: Find video example with source_dash
-                dash_formats, dash_subs = self._extract_mpd_formats_and_subtitles(
-                    sources['source_dash'], display_id, 'dash')
-                formats.extend(dash_formats)
-                subs.update(dash_subs)
-        else:
-            hls_url = data['clips'][0]['hls_url']
-            formats, subs = self._extract_m3u8_formats_and_subtitles(
-                hls_url, display_id, 'mp4', 'm3u8_native')
 
+        formats = self._extract_m3u8_formats(
+            data['clips'][0]['hls_url'], display_id, 'mp4', 'm3u8_native')
         self._sort_formats(formats)
 
         get_first = lambda x: try_get(data, lambda y: y[x + 's'][0], dict) or {}
@@ -138,7 +76,6 @@ class VidioIE(InfoExtractor):
             'duration': int_or_none(video.get('duration')),
             'like_count': get_count('likes'),
             'formats': formats,
-            'subtitles': subs,
             'uploader': user.get('name'),
             'timestamp': parse_iso8601(video.get('created_at')),
             'uploader_id': username,

yt_dlp/extractor/voot.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..compat import compat_str
 from ..utils import (
     ExtractorError,
     int_or_none,
@@ -12,17 +11,7 @@ from ..utils import (
 
 
 class VootIE(InfoExtractor):
-    _VALID_URL = r'''(?x)
+    _VALID_URL = r'https?://(?:www\.)?voot\.com/(?:[^/]+/)+(?P<id>\d+)'
-                    (?:
-                        voot:|
-                        (?:https?://)(?:www\.)?voot\.com/?
-                        (?:
-                            movies/[^/]+/|
-                            (?:shows|kids)/(?:[^/]+/){4}
-                        )
-                     )
-                    (?P<id>\d{3,})
-                    '''
     _GEO_COUNTRIES = ['IN']
     _TESTS = [{
         'url': 'https://www.voot.com/shows/ishq-ka-rang-safed/1/360558/is-this-the-end-of-kamini-/441353',
@@ -33,6 +22,7 @@ class VootIE(InfoExtractor):
             'description': 'md5:06291fbbbc4dcbe21235c40c262507c1',
             'timestamp': 1472162937,
             'upload_date': '20160825',
+            'duration': 1146,
             'series': 'Ishq Ka Rang Safed',
             'season_number': 1,
             'episode': 'Is this the end of Kamini?',
@@ -54,6 +44,7 @@ class VootIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
+
         media_info = self._download_json(
             'https://wapi.voot.com/ws/ott/getMediaInfo.json', video_id,
             query={
@@ -91,6 +82,7 @@ class VootIE(InfoExtractor):
                 episode = value
             elif key == 'EpisodeNo':
                 episode_number = int_or_none(value)
+
         return {
             'extractor_key': 'Kaltura',
             'id': entry_id,
@@ -106,45 +98,3 @@ class VootIE(InfoExtractor):
             'like_count': int_or_none(media.get('like_counter')),
             'formats': formats,
         }
-
-
-class VootSeriesIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?voot\.com/shows/[^/]+/(?P<id>\d{3,})'
-    _TESTS = [{
-        'url': 'https://www.voot.com/shows/chakravartin-ashoka-samrat/100002',
-        'playlist_mincount': 442,
-        'info_dict': {
-            'id': '100002',
-        },
-    }, {
-        'url': 'https://www.voot.com/shows/ishq-ka-rang-safed/100003',
-        'playlist_mincount': 341,
-        'info_dict': {
-            'id': '100003',
-        },
-    }]
-    _SHOW_API = 'https://psapi.voot.com/media/voot/v1/voot-web/content/generic/season-by-show?sort=season%3Aasc&id={}&responseType=common'
-    _SEASON_API = 'https://psapi.voot.com/media/voot/v1/voot-web/content/generic/series-wise-episode?sort=episode%3Aasc&id={}&responseType=common&page={:d}'
-
-    def _entries(self, show_id):
-        show_json = self._download_json(self._SHOW_API.format(show_id), video_id=show_id)
-        for season in show_json.get('result', []):
-            page_num = 1
-            season_id = try_get(season, lambda x: x['id'], compat_str)
-            season_json = self._download_json(self._SEASON_API.format(season_id, page_num),
-                                              video_id=season_id,
-                                              note='Downloading JSON metadata page %d' % page_num)
-            episodes_json = season_json.get('result', [])
-            while episodes_json:
-                page_num += 1
-                for episode in episodes_json:
-                    video_id = episode.get('id')
-                    yield self.url_result(
-                        'voot:%s' % video_id, ie=VootIE.ie_key(), video_id=video_id)
-                episodes_json = self._download_json(self._SEASON_API.format(season_id, page_num),
-                                                    video_id=season_id,
-                                                    note='Downloading JSON metadata page %d' % page_num)['result']
-
-    def _real_extract(self, url):
-        show_id = self._match_id(url)
-        return self.playlist_result(self._entries(show_id), playlist_id=show_id)

yt_dlp/extractor/whowatch.py

@@ -47,7 +47,8 @@ class WhoWatchIE(InfoExtractor):
 
             if hls_url:
                 hls_fmts = self._extract_m3u8_formats(
-                    hls_url, video_id, ext='mp4', m3u8_id='hls-%s' % name, quality=quality)
+                    hls_url, video_id, ext='mp4', entry_protocol='m3u8',
+                    m3u8_id='hls-%s' % name, quality=quality)
                 formats.extend(hls_fmts)
             else:
                 hls_fmts = []
@@ -70,7 +71,8 @@ class WhoWatchIE(InfoExtractor):
 
         # This contains the same formats as the above manifests and is used only as a fallback
         formats.extend(self._extract_m3u8_formats(
-            hls_url, video_id, ext='mp4', m3u8_id='hls'))
+            hls_url, video_id, ext='mp4', entry_protocol='m3u8',
+            m3u8_id='hls'))
         self._remove_duplicate_formats(formats)
         self._sort_formats(formats)
 

yt_dlp/extractor/youtube.py

@@ -68,7 +68,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
 
     _RESERVED_NAMES = (
         r'channel|c|user|browse|playlist|watch|w|v|embed|e|watch_popup|'
-        r'movies|results|shared|hashtag|trending|feed|feeds|oembed|get_video_info|'
+        r'movies|results|shared|hashtag|trending|feed|feeds|oembed|'
         r'storefront|oops|index|account|reporthistory|t/terms|about|upload|signin|logout')
 
     _NETRC_MACHINE = 'youtube'
@@ -2107,7 +2107,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         r'/itag/(\d+)', f['url'], 'itag', default=None)
                     if itag:
                         f['format_id'] = itag
-                    formats.append(f)
+                formats.append(f)
 
         if self.get_param('youtube_include_dash_manifest', True):
             for sd in (streaming_data, ytm_streaming_data):
@@ -2854,10 +2854,6 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
     }, {
         'url': 'https://www.youtube.com/c/CommanderVideoHq/live',
         'only_matching': True,
-    }, {
-        'note': 'A channel that is not live. Should raise error',
-        'url': 'https://www.youtube.com/user/numberphile/live',
-        'only_matching': True,
     }, {
         'url': 'https://www.youtube.com/feed/trending',
         'only_matching': True,
@@ -3714,26 +3710,23 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
         if tabs:
             selected_tab = self._extract_selected_tab(tabs)
             tab_name = selected_tab.get('title', '')
-            if 'no-youtube-channel-redirect' not in compat_opts:
+            if (mobj['tab'] == '/videos' and tab_name.lower() != mobj['tab'][1:]
-                if mobj['tab'] == '/live':
+                    and 'no-youtube-channel-redirect' not in compat_opts):
-                    # Live tab should have redirected to the video
+                if not mobj['not_channel'] and item_id[:2] == 'UC':
-                    raise ExtractorError('The channel is not currently live', expected=True)
+                    # Topic channels don't have /videos. Use the equivalent playlist instead
-                if mobj['tab'] == '/videos' and tab_name.lower() != mobj['tab'][1:]:
+                    self.report_warning('The URL does not have a %s tab. Trying to redirect to playlist UU%s instead' % (mobj['tab'][1:], item_id[2:]))
-                    if not mobj['not_channel'] and item_id[:2] == 'UC':
+                    pl_id = 'UU%s' % item_id[2:]
-                        # Topic channels don't have /videos. Use the equivalent playlist instead
+                    pl_url = 'https://www.youtube.com/playlist?list=%s%s' % (pl_id, mobj['post'])
-                        self.report_warning('The URL does not have a %s tab. Trying to redirect to playlist UU%s instead' % (mobj['tab'][1:], item_id[2:]))
+                    try:
-                        pl_id = 'UU%s' % item_id[2:]
+                        pl_webpage, pl_data = self._extract_webpage(pl_url, pl_id)
-                        pl_url = 'https://www.youtube.com/playlist?list=%s%s' % (pl_id, mobj['post'])
+                        for alert_type, alert_message in self._extract_alerts(pl_data):
-                        try:
+                            if alert_type == 'error':
-                            pl_webpage, pl_data = self._extract_webpage(pl_url, pl_id)
+                                raise ExtractorError('Youtube said: %s' % alert_message)
-                            for alert_type, alert_message in self._extract_alerts(pl_data):
+                        item_id, url, webpage, data = pl_id, pl_url, pl_webpage, pl_data
-                                if alert_type == 'error':
+                    except ExtractorError:
-                                    raise ExtractorError('Youtube said: %s' % alert_message)
+                        self.report_warning('The playlist gave error. Falling back to channel URL')
-                            item_id, url, webpage, data = pl_id, pl_url, pl_webpage, pl_data
+                else:
-                        except ExtractorError:
+                    self.report_warning('The URL does not have a %s tab. %s is being downloaded instead' % (mobj['tab'][1:], tab_name))
-                            self.report_warning('The playlist gave error. Falling back to channel URL')
-                    else:
-                        self.report_warning('The URL does not have a %s tab. %s is being downloaded instead' % (mobj['tab'][1:], tab_name))
 
         self.write_debug('Final URL: %s' % url)
 
@@ -3756,8 +3749,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             data, lambda x: x['currentVideoEndpoint']['watchEndpoint']['videoId'],
             compat_str) or video_id
         if video_id:
-            if mobj['tab'] != '/live':  # live tab is expected to redirect to video
+            self.report_warning('Unable to recognize playlist. Downloading just video %s' % video_id)
-                self.report_warning('Unable to recognize playlist. Downloading just video %s' % video_id)
             return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id)
 
         raise ExtractorError('Unable to recognize tab page')
@@ -3838,7 +3830,7 @@ class YoutubePlaylistIE(InfoExtractor):
 
     def _real_extract(self, url):
         playlist_id = self._match_id(url)
-        is_music_url = YoutubeBaseInfoExtractor.is_music_url(url)
+        is_music_url = self.is_music_url(url)
         url = update_url_query(
             'https://www.youtube.com/playlist',
             parse_qs(url) or {'list': playlist_id})

yt_dlp/extractor/zee5.py

@@ -98,7 +98,7 @@ class Zee5IE(InfoExtractor):
             compat_str)
         formats = self._extract_m3u8_formats(
             'https://zee5vodnd.akamaized.net' + m3u8_url.replace('/drm', '/hls', 1) + token_request['video_token'],
-            video_id, 'mp4', fatal=False)
+            video_id, fatal=False)
         mpd_url = try_get(
             json_data,
             (lambda x: x['video'][0], lambda x: x['video_details']['url']),

yt_dlp/options.py

@@ -5,6 +5,7 @@ import optparse
 import re
 import sys
 
+from .downloader.external import list_external_downloaders
 from .compat import (
     compat_expanduser,
     compat_get_terminal_size,
@@ -17,18 +18,11 @@ from .utils import (
     get_executable_path,
     OUTTMPL_TYPES,
     preferredencoding,
+    REMUX_EXTENSIONS,
     write_string,
 )
 from .version import __version__
 
-from .downloader.external import list_external_downloaders
-from .postprocessor.ffmpeg import (
-    FFmpegExtractAudioPP,
-    FFmpegSubtitlesConvertorPP,
-    FFmpegThumbnailsConvertorPP,
-    FFmpegVideoRemuxerPP,
-)
-
 
 def _hide_login_info(opts):
     PRIVATE_OPTS = set(['-p', '--password', '-u', '--username', '--video-password', '--ap-password', '--ap-username'])
@@ -1129,9 +1123,7 @@ def parseOpts(overrideArguments=None):
         help='Convert video files to audio-only files (requires ffmpeg and ffprobe)')
     postproc.add_option(
         '--audio-format', metavar='FORMAT', dest='audioformat', default='best',
-        help=(
+        help='Specify audio format: "best", "aac", "flac", "mp3", "m4a", "opus", "vorbis", or "wav"; "%default" by default; No effect without -x')
-            'Specify audio format to convert the audio to when -x is used. Currently supported formats are: '
-            'best (default) or one of %s' % '|'.join(FFmpegExtractAudioPP.SUPPORTED_EXTS)))
     postproc.add_option(
         '--audio-quality', metavar='QUALITY',
         dest='audioquality', default='5',
@@ -1142,14 +1134,14 @@ def parseOpts(overrideArguments=None):
         help=(
             'Remux the video into another container if necessary (currently supported: %s). '
             'If target container does not support the video/audio codec, remuxing will fail. '
-            'You can specify multiple rules; Eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
+            'You can specify multiple rules; eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
-            'and anything else to mkv.' % '|'.join(FFmpegVideoRemuxerPP.SUPPORTED_EXTS)))
+            'and anything else to mkv.' % '|'.join(REMUX_EXTENSIONS)))
     postproc.add_option(
         '--recode-video',
         metavar='FORMAT', dest='recodevideo', default=None,
         help=(
             'Re-encode the video into another format if re-encoding is necessary. '
-            'The syntax and supported formats are the same as --remux-video'))
+            'The supported formats are the same as --remux-video'))
     postproc.add_option(
         '--postprocessor-args', '--ppa',
         metavar='NAME:ARGS', dest='postprocessor_args', default={}, type='str',
@@ -1257,15 +1249,11 @@ def parseOpts(overrideArguments=None):
     postproc.add_option(
         '--convert-subs', '--convert-sub', '--convert-subtitles',
         metavar='FORMAT', dest='convertsubtitles', default=None,
-        help=(
+        help='Convert the subtitles to another format (currently supported: srt|ass|vtt|lrc) (Alias: --convert-subtitles)')
-            'Convert the subtitles to another format (currently supported: %s) '
-            '(Alias: --convert-subtitles)' % '|'.join(FFmpegSubtitlesConvertorPP.SUPPORTED_EXTS)))
     postproc.add_option(
         '--convert-thumbnails',
         metavar='FORMAT', dest='convertthumbnails', default=None,
-        help=(
+        help='Convert the thumbnails to another format (currently supported: jpg)')
-            'Convert the thumbnails to another format '
-            '(currently supported: %s) ' % '|'.join(FFmpegThumbnailsConvertorPP.SUPPORTED_EXTS)))
     postproc.add_option(
         '--split-chapters', '--split-tracks',
         dest='split_chapters', action='store_true', default=False,

yt_dlp/postprocessor/common.py

@@ -100,7 +100,7 @@ class PostProcessor(object):
         else:
             use_compat = False
         return cli_configuration_args(
-            self.get_param('postprocessor_args'),
+            self._downloader.params.get('postprocessor_args'),
             keys, default, use_compat)
 
 

yt_dlp/postprocessor/embedthumbnail.py

@@ -8,10 +8,9 @@ import subprocess
 import re
 
 try:
-    from mutagen.flac import Picture, FLAC
-    from mutagen.mp4 import MP4, MP4Cover
-    from mutagen.oggopus import OggOpus
     from mutagen.oggvorbis import OggVorbis
+    from mutagen.oggopus import OggOpus
+    from mutagen.flac import Picture, FLAC
     has_mutagen = True
 except ImportError:
     has_mutagen = False
@@ -59,9 +58,6 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             return guess()
         return int(mobj.group('w')), int(mobj.group('h'))
 
-    def _report_run(self, exe, filename):
-        self.to_screen('%s: Adding thumbnail to "%s"' % (exe, filename))
-
     def run(self, info):
         filename = info['filepath']
         temp_filename = prepend_extension(filename, 'temp')
@@ -70,29 +66,22 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             self.to_screen('There aren\'t any thumbnails to embed')
             return [], info
 
-        idx = next((-i for i, t in enumerate(info['thumbnails'][::-1], 1) if t.get('filepath')), None)
+        thumbnail_filename = info['thumbnails'][-1]['filepath']
-        if idx is None:
-            self.to_screen('There are no thumbnails on disk')
-            return [], info
-        thumbnail_filename = info['thumbnails'][idx]['filepath']
         if not os.path.exists(encodeFilename(thumbnail_filename)):
             self.report_warning('Skipping embedding the thumbnail because the file is missing.')
             return [], info
 
         # Correct extension for WebP file with wrong extension (see #25687, #25717)
         convertor = FFmpegThumbnailsConvertorPP(self._downloader)
-        convertor.fixup_webp(info, idx)
+        convertor.fixup_webp(info, -1)
 
-        original_thumbnail = thumbnail_filename = info['thumbnails'][idx]['filepath']
+        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filepath']
 
-        # Convert unsupported thumbnail formats to PNG (see #25687, #25717)
+        # Convert unsupported thumbnail formats to JPEG (see #25687, #25717)
-        # Original behavior was to convert to JPG, but since JPG is a lossy
-        # format, there will be some additional data loss.
-        # PNG, on the other hand, is lossless.
         thumbnail_ext = os.path.splitext(thumbnail_filename)[1][1:]
         if thumbnail_ext not in ('jpg', 'png'):
-            thumbnail_filename = convertor.convert_thumbnail(thumbnail_filename, 'png')
+            thumbnail_filename = convertor.convert_thumbnail(thumbnail_filename, 'jpg')
-            thumbnail_ext = 'png'
+            thumbnail_ext = 'jpg'
 
         mtime = os.stat(encodeFilename(filename)).st_mtime
 
@@ -102,7 +91,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                 '-c', 'copy', '-map', '0:0', '-map', '1:0', '-id3v2_version', '3',
                 '-metadata:s:v', 'title="Album cover"', '-metadata:s:v', 'comment="Cover (front)"']
 
-            self._report_run('ffmpeg', filename)
+            self.to_screen('Adding thumbnail to "%s"' % filename)
             self.run_ffmpeg_multiple_files([filename, thumbnail_filename], temp_filename, options)
 
         elif info['ext'] in ['mkv', 'mka']:
@@ -119,51 +108,25 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                 '-metadata:s:%d' % new_stream, 'mimetype=%s' % mimetype,
                 '-metadata:s:%d' % new_stream, 'filename=cover.%s' % thumbnail_ext])
 
-            self._report_run('ffmpeg', filename)
+            self.to_screen('Adding thumbnail to "%s"' % filename)
             self.run_ffmpeg(filename, temp_filename, options)
 
         elif info['ext'] in ['m4a', 'mp4', 'mov']:
-            # Method 1: Use mutagen
+            try:
-            if not has_mutagen:
+                options = ['-c', 'copy', '-map', '0', '-dn', '-map', '1']
-                success = False
-            else:
-                try:
-                    self._report_run('mutagen', filename)
-                    meta = MP4(filename)
-                    # NOTE: the 'covr' atom is a non-standard MPEG-4 atom,
-                    # Apple iTunes 'M4A' files include the 'moov.udta.meta.ilst' atom.
-                    f = {'jpeg': MP4Cover.FORMAT_JPEG, 'png': MP4Cover.FORMAT_PNG}[imghdr.what(thumbnail_filename)]
-                    with open(thumbnail_filename, 'rb') as thumbfile:
-                        thumb_data = thumbfile.read()
-                    meta.tags['covr'] = [MP4Cover(data=thumb_data, imageformat=f)]
-                    meta.save()
-                    temp_filename = filename
-                except Exception as err:
-                    self.report_warning('unable to embed using mutagen; %s' % error_to_compat_str(err))
-                    success = False
 
-            # Method 2: Use ffmpeg+ffprobe
+                old_stream, new_stream = self.get_stream_number(
-            if not success:
+                    filename, ('disposition', 'attached_pic'), 1)
-                success = True
+                if old_stream is not None:
-                try:
+                    options.extend(['-map', '-0:%d' % old_stream])
-                    options = ['-c', 'copy', '-map', '0', '-dn', '-map', '1']
+                    new_stream -= 1
+                options.extend(['-disposition:%s' % new_stream, 'attached_pic'])
 
-                    old_stream, new_stream = self.get_stream_number(
+                self.to_screen('Adding thumbnail to "%s"' % filename)
-                        filename, ('disposition', 'attached_pic'), 1)
+                self.run_ffmpeg_multiple_files([filename, thumbnail_filename], temp_filename, options)
-                    if old_stream is not None:
-                        options.extend(['-map', '-0:%d' % old_stream])
-                        new_stream -= 1
-                    options.extend(['-disposition:%s' % new_stream, 'attached_pic'])
 
-                    self._report_run('ffmpeg', filename)
+            except PostProcessingError as err:
-                    self.run_ffmpeg_multiple_files([filename, thumbnail_filename], temp_filename, options)
+                self.report_warning('unable to embed using ffprobe & ffmpeg; %s' % error_to_compat_str(err))
-                except PostProcessingError as err:
-                    self.report_warning('unable to embed using ffprobe & ffmpeg; %s' % error_to_compat_str(err))
-                    success = False
-
-            # Method 3: Use AtomicParsley
-            if not success:
-                success = True
                 atomicparsley = next((
                     x for x in ['AtomicParsley', 'atomicparsley']
                     if check_executable(x, ['-v'])), None)
@@ -178,7 +141,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                        encodeFilename(temp_filename, True)]
                 cmd += [encodeArgument(o) for o in self._configuration_args('AtomicParsley')]
 
-                self._report_run('atomicparsley', filename)
+                self.to_screen('Adding thumbnail to "%s"' % filename)
                 self.write_debug('AtomicParsley command line: %s' % shell_quote(cmd))
                 p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                 stdout, stderr = process_communicate_or_kill(p)
@@ -195,7 +158,8 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             if not has_mutagen:
                 raise EmbedThumbnailPPError('module mutagen was not found. Please install using `python -m pip install mutagen`')
 
-            self._report_run('mutagen', filename)
+            self.to_screen('Adding thumbnail to "%s"' % filename)
+            temp_filename = filename
             f = {'opus': OggOpus, 'flac': FLAC, 'ogg': OggVorbis}[info['ext']](filename)
 
             pic = Picture()
@@ -203,7 +167,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             with open(thumbnail_filename, 'rb') as thumbfile:
                 pic.data = thumbfile.read()
             pic.type = 3  # front cover
-            res = self._get_thumbnail_resolution(thumbnail_filename, info['thumbnails'][idx])
+            res = self._get_thumbnail_resolution(thumbnail_filename, info['thumbnails'][-1])
             if res is not None:
                 pic.width, pic.height = res
 
@@ -213,7 +177,6 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                 # https://wiki.xiph.org/VorbisComment#METADATA_BLOCK_PICTURE
                 f['METADATA_BLOCK_PICTURE'] = base64.b64encode(pic.write()).decode('ascii')
             f.save()
-            temp_filename = filename
 
         else:
             raise EmbedThumbnailPPError('Supported filetypes for thumbnail embedding are: mp3, mkv/mka, ogg/opus/flac, m4a/mp4/mov')

yt_dlp/postprocessor/ffmpeg.py

@@ -290,12 +290,13 @@ class FFmpegPostProcessor(PostProcessor):
 
 
 class FFmpegExtractAudioPP(FFmpegPostProcessor):
-    COMMON_AUDIO_EXTS = ('wav', 'flac', 'm4a', 'aiff', 'mp3', 'ogg', 'mka', 'opus', 'wma')
+    COMMON_AUDIO_EXTENSIONS = ('wav', 'flac', 'm4a', 'aiff', 'mp3', 'ogg', 'mka', 'opus', 'wma')
-    SUPPORTED_EXTS = ('best', 'aac', 'flac', 'mp3', 'm4a', 'opus', 'vorbis', 'wav')
 
     def __init__(self, downloader=None, preferredcodec=None, preferredquality=None, nopostoverwrites=False):
         FFmpegPostProcessor.__init__(self, downloader)
-        self._preferredcodec = preferredcodec or 'best'
+        if preferredcodec is None:
+            preferredcodec = 'best'
+        self._preferredcodec = preferredcodec
         self._preferredquality = preferredquality
         self._nopostoverwrites = nopostoverwrites
 
@@ -314,7 +315,7 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
         path = information['filepath']
         orig_ext = information['ext']
 
-        if self._preferredcodec == 'best' and orig_ext in self.COMMON_AUDIO_EXTS:
+        if self._preferredcodec == 'best' and orig_ext in self.COMMON_AUDIO_EXTENSIONS:
             self.to_screen('Skipping audio extraction since the file is already in a common audio format')
             return [], information
 
@@ -398,57 +399,62 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
         return [path], information
 
 
-class FFmpegVideoConvertorPP(FFmpegPostProcessor):
+class FFmpegVideoRemuxerPP(FFmpegPostProcessor):
-    SUPPORTED_EXTS = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
-    FORMAT_RE = re.compile(r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(SUPPORTED_EXTS)))
-    _action = 'converting'
-
     def __init__(self, downloader=None, preferedformat=None):
-        super(FFmpegVideoConvertorPP, self).__init__(downloader)
+        super(FFmpegVideoRemuxerPP, self).__init__(downloader)
         self._preferedformats = preferedformat.lower().split('/')
 
-    def _target_ext(self, source_ext):
+    def run(self, information):
+        path = information['filepath']
+        sourceext, targetext = information['ext'].lower(), None
         for pair in self._preferedformats:
             kv = pair.split('>')
-            if len(kv) == 1 or kv[0].strip() == source_ext:
+            if len(kv) == 1 or kv[0].strip() == sourceext:
-                return kv[-1].strip()
+                targetext = kv[-1].strip()
+                break
 
-    @staticmethod
-    def _options(target_ext):
-        if target_ext == 'avi':
-            return ['-c:v', 'libxvid', '-vtag', 'XVID']
-        return []
-
-    def run(self, information):
-        path, source_ext = information['filepath'], information['ext'].lower()
-        target_ext = self._target_ext(source_ext)
         _skip_msg = (
-            'could not find a mapping for %s' if not target_ext
+            'could not find a mapping for %s' if not targetext
-            else 'already is in target format %s' if source_ext == target_ext
+            else 'already is in target format %s' if sourceext == targetext
             else None)
         if _skip_msg:
-            self.to_screen('Not %s media file "%s"; %s' % (self._action, path, _skip_msg % source_ext))
+            self.to_screen('Not remuxing media file %s; %s' % (path, _skip_msg % sourceext))
             return [], information
 
+        options = ['-c', 'copy', '-map', '0', '-dn']
+        if targetext in ['mp4', 'm4a', 'mov']:
+            options.extend(['-movflags', '+faststart'])
         prefix, sep, oldext = path.rpartition('.')
-        outpath = prefix + sep + target_ext
+        outpath = prefix + sep + targetext
-        self.to_screen('%s video from %s to %s; Destination: %s' % (self._action.title(), source_ext, target_ext, outpath))
+        self.to_screen('Remuxing video from %s to %s; Destination: %s' % (sourceext, targetext, outpath))
-        self.run_ffmpeg(path, outpath, self._options(target_ext))
+        self.run_ffmpeg(path, outpath, options)
-
         information['filepath'] = outpath
-        information['format'] = information['ext'] = target_ext
+        information['format'] = targetext
+        information['ext'] = targetext
         return [path], information
 
 
-class FFmpegVideoRemuxerPP(FFmpegVideoConvertorPP):
+class FFmpegVideoConvertorPP(FFmpegPostProcessor):
-    _action = 'remuxing'
+    def __init__(self, downloader=None, preferedformat=None):
+        super(FFmpegVideoConvertorPP, self).__init__(downloader)
+        self._preferedformat = preferedformat
 
-    @staticmethod
+    def run(self, information):
-    def _options(target_ext):
+        path = information['filepath']
-        options = ['-c', 'copy', '-map', '0', '-dn']
+        if information['ext'] == self._preferedformat:
-        if target_ext in ['mp4', 'm4a', 'mov']:
+            self.to_screen('Not converting video file %s - already is in target format %s' % (path, self._preferedformat))
-            options.extend(['-movflags', '+faststart'])
+            return [], information
-        return options
+        options = []
+        if self._preferedformat == 'avi':
+            options.extend(['-c:v', 'libxvid', '-vtag', 'XVID'])
+        prefix, sep, ext = path.rpartition('.')
+        outpath = prefix + sep + self._preferedformat
+        self.to_screen('Converting video from %s to %s, Destination: ' % (information['ext'], self._preferedformat) + outpath)
+        self.run_ffmpeg(path, outpath, options)
+        information['filepath'] = outpath
+        information['format'] = self._preferedformat
+        information['ext'] = self._preferedformat
+        return [path], information
 
 
 class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
@@ -709,8 +715,6 @@ class FFmpegFixupM3u8PP(FFmpegPostProcessor):
 
 
 class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
-    SUPPORTED_EXTS = ('srt', 'vtt', 'ass', 'lrc')
-
     def __init__(self, downloader=None, format=None):
         super(FFmpegSubtitlesConvertorPP, self).__init__(downloader)
         self.format = format
@@ -819,8 +823,6 @@ class FFmpegSplitChaptersPP(FFmpegPostProcessor):
 
 
 class FFmpegThumbnailsConvertorPP(FFmpegPostProcessor):
-    SUPPORTED_EXTS = ('jpg', 'png')
-
     def __init__(self, downloader=None, format=None):
         super(FFmpegThumbnailsConvertorPP, self).__init__(downloader)
         self.format = format
@@ -846,22 +848,25 @@ class FFmpegThumbnailsConvertorPP(FFmpegPostProcessor):
                 info['__files_to_move'][webp_filename] = replace_extension(
                     info['__files_to_move'].pop(thumbnail_filename), 'webp')
 
-    @staticmethod
+    def convert_thumbnail(self, thumbnail_filename, ext):
-    def _options(target_ext):
+        if ext != 'jpg':
-        if target_ext == 'jpg':
+            raise FFmpegPostProcessorError('Only conversion to jpg is currently supported')
-            return ['-bsf:v', 'mjpeg2jpeg']
+        # NB: % is supposed to be escaped with %% but this does not work
-        return []
+        # for input files so working around with standard substitution
-
+        escaped_thumbnail_filename = thumbnail_filename.replace('%', '#')
-    def convert_thumbnail(self, thumbnail_filename, target_ext):
+        os.rename(encodeFilename(thumbnail_filename), encodeFilename(escaped_thumbnail_filename))
-        thumbnail_conv_filename = replace_extension(thumbnail_filename, target_ext)
+        escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
-
+        self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
-        self.to_screen('Converting thumbnail "%s" to %s' % (thumbnail_filename, target_ext))
+        self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
-        self.real_run_ffmpeg(
+        thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
-            [(thumbnail_filename, ['-f', 'image2', '-pattern_type', 'none'])],
+        # Rename back to unescaped
-            [(thumbnail_conv_filename.replace('%', '%%'), self._options(target_ext))])
+        os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
-        return thumbnail_conv_filename
+        os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
+        return thumbnail_jpg_filename
 
     def run(self, info):
+        if self.format != 'jpg':
+            raise FFmpegPostProcessorError('Only conversion to jpg is currently supported')
         files_to_delete = []
         has_thumbnail = False
 

yt_dlp/update.py

@@ -1,12 +1,13 @@
 from __future__ import unicode_literals
 
-import hashlib
+import io
 import json
+import traceback
+import hashlib
 import os
 import platform
 import subprocess
 import sys
-import traceback
 from zipimport import zipimporter
 
 from .compat import compat_realpath
@@ -32,40 +33,6 @@ def rsa_verify(message, signature, key):
 
 
 def update_self(to_screen, verbose, opener):
-    ''' Exists for backward compatibility. Use run_update(ydl) instead '''
-
-    printfn = to_screen
-
-    class FakeYDL():
-        _opener = opener
-        to_screen = printfn
-
-        @staticmethod
-        def report_warning(msg, *args, **kwargs):
-            return printfn('WARNING: %s' % msg, *args, **kwargs)
-
-        @staticmethod
-        def report_error(msg, tb=None):
-            printfn('ERROR: %s' % msg)
-            if not verbose:
-                return
-            if tb is None:
-                # Copied from YoutubeDl.trouble
-                if sys.exc_info()[0]:
-                    tb = ''
-                    if hasattr(sys.exc_info()[1], 'exc_info') and sys.exc_info()[1].exc_info[0]:
-                        tb += ''.join(traceback.format_exception(*sys.exc_info()[1].exc_info))
-                    tb += encode_compat_str(traceback.format_exc())
-                else:
-                    tb_data = traceback.format_list(traceback.extract_stack())
-                    tb = ''.join(tb_data)
-            if tb:
-                printfn(tb)
-
-    return run_update(FakeYDL())
-
-
-def run_update(ydl):
     """
     Update the program file with the latest version from the repository
     Returns whether the program should terminate
@@ -73,11 +40,6 @@ def run_update(ydl):
 
     JSON_URL = 'https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest'
 
-    def report_error(msg, network=False, expected=False, delim=';'):
-        if network:
-            msg += '%s Visit  https://github.com/yt-dlp/yt-dlp/releases/latest' % delim
-        ydl.report_error(msg, tb='' if network or expected else None)
-
     def calc_sha256sum(path):
         h = hashlib.sha256()
         b = bytearray(128 * 1024)
@@ -87,149 +49,170 @@ def run_update(ydl):
                 h.update(mv[:n])
         return h.hexdigest()
 
-    err = None
+    if not isinstance(globals().get('__loader__'), zipimporter) and not hasattr(sys, 'frozen'):
-    if isinstance(globals().get('__loader__'), zipimporter):
+        to_screen('It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update')
-        # We only support python 3.6 or above
+        return
-        if sys.version_info < (3, 6):
-            err = 'This is the last release of yt-dlp for Python version %d.%d! Please update to Python 3.6 or above' % sys.version_info[:2]
-    elif hasattr(sys, 'frozen'):
-        # Python 3.6 supports only vista and above
-        if sys.getwindowsversion()[0] < 6:
-            err = 'This is the last release of yt-dlp for your version of Windows. Please update to Windows Vista or above'
-    else:
-        err = 'It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update'
-    if err:
-        return report_error(err, expected=True)
 
     # sys.executable is set to the full pathname of the exe-file for py2exe
     # though symlinks are not followed so that we need to do this manually
     # with help of realpath
     filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0])
-    ydl.to_screen('Current Build Hash %s' % calc_sha256sum(filename))
+    to_screen('Current Build Hash %s' % calc_sha256sum(filename))
 
     # Download and check versions info
     try:
-        version_info = ydl._opener.open(JSON_URL).read().decode('utf-8')
+        version_info = opener.open(JSON_URL).read().decode('utf-8')
         version_info = json.loads(version_info)
     except Exception:
-        return report_error('can\'t obtain versions info. Please try again later ', True, delim='or')
+        if verbose:
+            to_screen(encode_compat_str(traceback.format_exc()))
+        to_screen('ERROR: can\'t obtain versions info. Please try again later')
+        to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+        return
 
     def version_tuple(version_str):
         return tuple(map(int, version_str.split('.')))
 
     version_id = version_info['tag_name']
     if version_tuple(__version__) >= version_tuple(version_id):
-        ydl.to_screen('yt-dlp is up to date (%s)' % __version__)
+        to_screen('yt-dlp is up to date (%s)' % __version__)
         return
 
-    ydl.to_screen('Updating to version ' + version_id + ' ...')
+    to_screen('Updating to version ' + version_id + ' ...')
 
     version_labels = {
         'zip_3': '',
+        'zip_2': '',
+        # 'zip_2': '_py2',
         'exe_64': '.exe',
         'exe_32': '_x86.exe',
     }
 
     def get_bin_info(bin_or_exe, version):
         label = version_labels['%s_%s' % (bin_or_exe, version)]
-        return next((i for i in version_info['assets'] if i['name'] == 'yt-dlp%s' % label), {})
+        return next(
+            (i for i in version_info['assets'] if i['name'] == 'yt-dlp%s' % label),
+            {})
 
     def get_sha256sum(bin_or_exe, version):
         label = version_labels['%s_%s' % (bin_or_exe, version)]
         urlh = next(
-            (i for i in version_info['assets'] if i['name'] in ('SHA2-256SUMS')),
+            (i for i in version_info['assets']
-            {}).get('browser_download_url')
+                if i['name'] in ('SHA2-256SUMS')), {}).get('browser_download_url')
         if not urlh:
             return None
-        hash_data = ydl._opener.open(urlh).read().decode('utf-8')
+        hash_data = opener.open(urlh).read().decode('utf-8')
         hashes = list(map(lambda x: x.split(':'), hash_data.splitlines()))
-        return next((i[1] for i in hashes if i[0] == 'yt-dlp%s' % label), None)
+        return next(
+            (i[1] for i in hashes if i[0] == 'yt-dlp%s' % label),
+            None)
 
     if not os.access(filename, os.W_OK):
-        return report_error('no write permissions on %s' % filename, expected=True)
+        to_screen('ERROR: no write permissions on %s' % filename)
+        return
 
     # PyInstaller
     if hasattr(sys, 'frozen'):
         exe = filename
         directory = os.path.dirname(exe)
         if not os.access(directory, os.W_OK):
-            return report_error('no write permissions on %s' % directory, expected=True)
+            to_screen('ERROR: no write permissions on %s' % directory)
-        try:
+            return
-            if os.path.exists(filename + '.old'):
-                os.remove(filename + '.old')
-        except (IOError, OSError):
-            return report_error('unable to remove the old version')
 
         try:
             arch = platform.architecture()[0][:2]
             url = get_bin_info('exe', arch).get('browser_download_url')
             if not url:
-                return report_error('unable to fetch updates', True)
+                to_screen('ERROR: unable to fetch updates')
-            urlh = ydl._opener.open(url)
+                to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+                return
+            urlh = opener.open(url)
             newcontent = urlh.read()
             urlh.close()
         except (IOError, OSError, StopIteration):
-            return report_error('unable to download latest version', True)
+            if verbose:
+                to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to download latest version')
+            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+            return
 
         try:
             with open(exe + '.new', 'wb') as outf:
                 outf.write(newcontent)
         except (IOError, OSError):
-            return report_error('unable to write the new version')
+            if verbose:
+                to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to write the new version')
+            return
 
         expected_sum = get_sha256sum('exe', arch)
         if not expected_sum:
-            ydl.report_warning('no hash information found for the release')
+            to_screen('WARNING: no hash information found for the release')
         elif calc_sha256sum(exe + '.new') != expected_sum:
-            report_error('unable to verify the new executable', True)
+            to_screen('ERROR: unable to verify the new executable')
+            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
             try:
                 os.remove(exe + '.new')
             except OSError:
-                return report_error('unable to remove corrupt download')
+                to_screen('ERROR: unable to remove corrupt download')
+            return
 
         try:
-            os.rename(exe, exe + '.old')
+            bat = os.path.join(directory, 'yt-dlp-updater.cmd')
-        except (IOError, OSError):
+            with io.open(bat, 'w') as batfile:
-            return report_error('unable to move current version')
+                batfile.write('''
-        try:
+@(
-            os.rename(exe + '.new', exe)
+    echo.Waiting for file handle to be closed ...
-        except (IOError, OSError):
+    ping 127.0.0.1 -n 5 -w 1000 > NUL
-            report_error('unable to overwrite current version')
+    move /Y "%s.new" "%s" > NUL
-            os.rename(exe + '.old', exe)
+    echo.Updated yt-dlp to version %s
-            return
+)
-        try:
+@start /b "" cmd /c del "%%~f0"&exit /b
-            # Continues to run in the background
+                ''' % (exe, exe, version_id))
-            subprocess.Popen(
+
-                'ping 127.0.0.1 -n 5 -w 1000 & del /F "%s.old"' % exe,
+            subprocess.Popen([bat])  # Continues to run in the background
-                shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-            ydl.to_screen('Updated yt-dlp to version %s' % version_id)
             return True  # Exit app
-        except OSError:
+        except (IOError, OSError):
-            report_error('unable to delete old version')
+            if verbose:
+                to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to overwrite current version')
+            return
 
     # Zip unix package
     elif isinstance(globals().get('__loader__'), zipimporter):
         try:
-            url = get_bin_info('zip', '3').get('browser_download_url')
+            py_ver = platform.python_version()[0]
+            url = get_bin_info('zip', py_ver).get('browser_download_url')
             if not url:
-                return report_error('unable to fetch updates', True)
+                to_screen('ERROR: unable to fetch updates')
-            urlh = ydl._opener.open(url)
+                to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+                return
+            urlh = opener.open(url)
             newcontent = urlh.read()
             urlh.close()
         except (IOError, OSError, StopIteration):
-            return report_error('unable to download latest version', True)
+            if verbose:
+                to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to download latest version')
+            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+            return
 
-        expected_sum = get_sha256sum('zip', '3')
+        expected_sum = get_sha256sum('zip', py_ver)
         if expected_sum and hashlib.sha256(newcontent).hexdigest() != expected_sum:
-            return report_error('unable to verify the new zip', True)
+            to_screen('ERROR: unable to verify the new zip')
+            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+            return
 
         try:
             with open(filename, 'wb') as outf:
                 outf.write(newcontent)
         except (IOError, OSError):
-            return report_error('unable to overwrite current version')
+            if verbose:
+                to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to overwrite current version')
+            return
 
-    ydl.to_screen('Updated yt-dlp to version %s; Restart yt-dlp to use the new version' % version_id)
+    to_screen('Updated yt-dlp. Restart yt-dlp to use the new version')
 
 
 '''  # UNUSED

yt_dlp/utils.py

@@ -1716,6 +1716,8 @@ KNOWN_EXTENSIONS = (
     'wav',
     'f4f', 'f4m', 'm3u8', 'smil')
 
+REMUX_EXTENSIONS = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
+
 # needed for sanitizing filenames in restricted mode
 ACCENT_CHARS = dict(zip('ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ',
                         itertools.chain('AAAAAA', ['AE'], 'CEEEEIIIIDNOOOOOOO', ['OE'], 'UUUUUY', ['TH', 'ss'],
@@ -2165,27 +2167,11 @@ def sanitize_url(url):
     for mistake, fixup in COMMON_TYPOS:
         if re.match(mistake, url):
             return re.sub(mistake, fixup, url)
-    return url
+    return escape_url(url)
-
-
-def extract_basic_auth(url):
-    parts = compat_urlparse.urlsplit(url)
-    if parts.username is None:
-        return url, None
-    url = compat_urlparse.urlunsplit(parts._replace(netloc=(
-        parts.hostname if parts.port is None
-        else '%s:%d' % (parts.hostname, parts.port))))
-    auth_payload = base64.b64encode(
-        ('%s:%s' % (parts.username, parts.password or '')).encode('utf-8'))
-    return url, 'Basic ' + auth_payload.decode('utf-8')
 
 
 def sanitized_Request(url, *args, **kwargs):
-    url, auth_header = extract_basic_auth(escape_url(sanitize_url(url)))
+    return compat_urllib_request.Request(sanitize_url(url), *args, **kwargs)
-    if auth_header is not None:
-        headers = args[1] if len(args) >= 2 else kwargs.setdefault('headers', {})
-        headers['Authorization'] = auth_header
-    return compat_urllib_request.Request(url, *args, **kwargs)
 
 
 def expand_path(s):
@@ -3945,70 +3931,11 @@ def detect_exe_version(output, version_re=None, unrecognized='present'):
         return unrecognized
 
 
-class LazyList(collections.Sequence):
-    ''' Lazy immutable list from an iterable
-    Note that slices of a LazyList are lists and not LazyList'''
-
-    def __init__(self, iterable):
-        self.__iterable = iter(iterable)
-        self.__cache = []
-
-    def __iter__(self):
-        for item in self.__cache:
-            yield item
-        for item in self.__iterable:
-            self.__cache.append(item)
-            yield item
-
-    def exhaust(self):
-        ''' Evaluate the entire iterable '''
-        self.__cache.extend(self.__iterable)
-
-    def __getitem__(self, idx):
-        if isinstance(idx, slice):
-            step = idx.step or 1
-            start = idx.start if idx.start is not None else 1 if step > 0 else -1
-            stop = idx.stop if idx.stop is not None else -1 if step > 0 else 0
-        elif isinstance(idx, int):
-            start = stop = idx
-        else:
-            raise TypeError('indices must be integers or slices')
-        if start < 0 or stop < 0:
-            # We need to consume the entire iterable to be able to slice from the end
-            # Obviously, never use this with infinite iterables
-            self.exhaust()
-        else:
-            n = max(start, stop) - len(self.__cache) + 1
-            if n > 0:
-                self.__cache.extend(itertools.islice(self.__iterable, n))
-        return self.__cache[idx]
-
-    def __bool__(self):
-        try:
-            self[0]
-        except IndexError:
-            return False
-        return True
-
-    def __len__(self):
-        self.exhaust()
-        return len(self.__cache)
-
-
 class PagedList(object):
     def __len__(self):
         # This is only useful for tests
         return len(self.getslice())
 
-    def getslice(self, start, end):
-        raise NotImplementedError('This method must be implemented by subclasses')
-
-    def __getitem__(self, idx):
-        if not isinstance(idx, int) or idx < 0:
-            raise TypeError('indices must be non-negative integers')
-        entries = self.getslice(idx, idx + 1)
-        return entries[0] if entries else None
-
 
 class OnDemandPagedList(PagedList):
     def __init__(self, pagefunc, pagesize, use_cache=True):

yt_dlp/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.05.20'
+__version__ = '2021.05.11'