Release 2021.04.11

[documentation] Improvements
Add option --convert-thumbnails
2026-08-08 13:18:35 +03:00 · 2021-04-12 03:20:07 +05:30 · 2021-04-12 02:36:06 +05:30 · 2021-04-12 02:32:29 +05:30 · 2021-04-12 02:12:46 +05:30 · 2021-04-12 01:53:04 +05:30
96 changed files with 5776 additions and 2118 deletions
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.03. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.03**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.03.2
+ [debug] yt-dlp version 2021.04.03
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.03. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.03**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.03. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.03**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones


@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.03. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.03**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.03.2
+ [debug] yt-dlp version 2021.04.03
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.03. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.03**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones


@@ -60,6 +60,7 @@ yt-dlp.zip
 *.mkv
 *.swf
 *.part
+*.part-*
 *.ytdl
 *.dump
 *.frag
@@ -1,5 +1,6 @@
 pukkandan (owner)
 shirt-dev (collaborator)
+colethedj (collaborator)
 h-h-h-h
 pauldubois98
 nixxo
@@ -23,10 +24,17 @@ tsukumi
 bbepis
 animelover1984
 Pccode66
-Ashish
+Ashish0804
 RobinD42
 hseg
-colethedj
 DennyDai
 codeasashu
 teesid
+kevinoconnor7
+damianoamatruda
+2ShedsJackson
+CXwudi
+xtkoba
+llacb47
+hheimbuerger
+B0pol
@@ -6,8 +6,10 @@
 * Run `make doc`
 * Update Changelog.md and CONTRIBUTORS
 * Change "Merged with ytdl" version in Readme.md if needed
+* Add new/fixed extractors in "new features" section of Readme.md
 * Commit to master as `Release <version>`
-* Push to origin/release - build task will now run
+* Push to origin/release using `git push origin master:release`
+    build task will now run
 * Update version.py using devscripts\update-version.py
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
@@ -17,22 +19,124 @@
 -->


+### 2021.04.11
+* Add option `--convert-thumbnails` (only jpg currently supported)
+* Format selector `mergeall` to download and merge all formats
+* Pass any field to `--exec` using similar syntax to output template
+* Choose downloader for each protocol using `--downloader PROTO:NAME`
+    * Alias `--downloader` for `--external-downloader`
+    * Added `native` as an option for the downloader
+* Merge youtube-dl: Upto [commit/4fb25ff](https://github.com/ytdl-org/youtube-dl/commit/4fb25ff5a3be5206bb72e5c4046715b1529fb2c7) (except vimeo)
+* [DiscoveryPlusIndia] Add DiscoveryPlusIndiaShowIE by [Ashish0804](https://github.com/Ashish0804)
+* [NFHSNetwork] Add extractor by [llacb47](https://github.com/llacb47)
+* [nebula] Add extractor (watchnebula.com) by [hheimbuerger](https://github.com/hheimbuerger)
+* [nitter] Fix extraction of reply tweets and update instance list by [B0pol](https://github.com/B0pol)
+* [nitter] Fix thumbnails by [B0pol](https://github.com/B0pol)
+* [youtube] Fix thumbnail URL
+* [youtube] Parse API parameters from initial webpage by [colethedj](https://github.com/colethedj)
+* [youtube] Extract comments' approximate timestamp by [colethedj](https://github.com/colethedj)
+* [youtube] Fix `\_extract_alerts`
+* [bilibili] Fix uploader
+* [utils] Add `datetime_from_str` and `datetime_add_months` by [colethedj](https://github.com/colethedj)
+* Run some `postprocessors` before actual download
+* Improve argument parsing for `-P`, `-o`, `-S`
+* Fix some `m3u8` not obeying `--allow-unplayable-formats`
+* Fix default of `dynamic_mpd`
+* Deprecate `--all-formats`, `--include-ads`, `--hls-prefer-native`, `--hls-prefer-ffmpeg`
+* [documentation] Improvements
+
+### 2021.04.03
+* Merge youtube-dl: Upto [commit/654b4f4](https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a)
+* Ability to set a specific field in the file's metadata using `--parse-metadata`
+* Ability to select n'th best format like `-f bv*.2`
+* [DiscoveryPlus] Add discoveryplus.in
+* [la7] Add podcasts and podcast playlists by [nixxo](https://github.com/nixxo)
+* [mildom] Update extractor with current proxy by [nao20010128nao](https://github.com/nao20010128nao)
+* [ard:mediathek] Fix video id extraction
+* [generic] Detect Invidious' link element
+* [youtube] Show premium state in `availability` by [colethedj](https://github.com/colethedj)
+* [viewsource] Add extractor to handle `view-source:`
+* [sponskrub] Run before embedding thumbnail
+* [documentation] Improve `--parse-metadata` documentation
+
+
+### 2021.03.24.1
+* Revert [commit/8562218](https://github.com/ytdl-org/youtube-dl/commit/8562218350a79d4709da8593bb0c538aa0824acf)
+
+### 2021.03.24
+* Merge youtube-dl: Upto 2021.03.25 ([commit/8562218](https://github.com/ytdl-org/youtube-dl/commit/8562218350a79d4709da8593bb0c538aa0824acf))
+* Parse metadata from multiple fields using `--parse-metadata`
+* Ability to load playlist infojson using `--load-info-json`
+* Write current epoch to infojson when using `--no-clean-infojson`
+* [youtube_live_chat] fix bug when trying to set cookies
+* [niconico] Fix for when logged in by [CXwudi](https://github.com/CXwudi) and [xtkoba](https://github.com/xtkoba)
+* [linuxacadamy] Fix login
+
+
+### 2021.03.21
+* Merge youtube-dl: Upto [commit/7e79ba7](https://github.com/ytdl-org/youtube-dl/commit/7e79ba7dd6e6649dd2ce3a74004b2044f2182881)
+* Option `--no-clean-infojson` to keep private keys in the infojson
+* [aria2c] Support retry/abort unavailable fragments by [damianoamatruda](https://github.com/damianoamatruda)
+* [aria2c] Better default arguments
+* [movefiles] Fix bugs and make more robust
+* [formatSort] Fix `quality` being ignored
+* [splitchapters] Fix for older ffmpeg
+* [sponskrub] Pass proxy to sponskrub
+* Make sure `post_hook` gets the final filename
+* Recursively remove any private keys from infojson
+* Embed video URL metadata inside `mp4` by [damianoamatruda](https://github.com/damianoamatruda) and [pukkandan](https://github.com/pukkandan)
+* Merge `webm` formats into `mkv` if thumbnails are to be embedded by [damianoamatruda](https://github.com/damianoamatruda)
+* Use headers and cookies when downloading subtitles by [damianoamatruda](https://github.com/damianoamatruda)
+* Parse resolution in info dictionary by [damianoamatruda](https://github.com/damianoamatruda)
+* More consistent warning messages by [damianoamatruda](https://github.com/damianoamatruda) and [pukkandan](https://github.com/pukkandan)
+* [documentation] Add deprecated options and aliases in readme
+* [documentation] Fix some minor mistakes
+
+* [niconico] Partial fix adapted from [animelover1984/youtube-dl@b5eff52](https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade) (login and smile formats still don't work)
+* [niconico] Add user extractor by [animelover1984](https://github.com/animelover1984)
+* [bilibili] Add anthology support by [animelover1984](https://github.com/animelover1984)
+* [amcnetworks] Fix extractor by [2ShedsJackson](https://github.com/2ShedsJackson)
+* [stitcher] Merge from youtube-dl by [nixxo](https://github.com/nixxo)
+* [rcs] Improved extraction by [nixxo](https://github.com/nixxo)
+* [linuxacadamy] Improve regex
+* [youtube] Show if video is `private`, `unlisted` etc in info (`availability`) by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+* [youtube] bugfix for channel playlist extraction
+* [nbc] Improve metadata extraction by [2ShedsJackson](https://github.com/2ShedsJackson)
+
+
+### 2021.03.15
+* **Split video by chapters**: using option `--split-chapters`
+    * The output file of the split files can be set with `-o`/`-P` using the prefix `chapter:`
+    * Additional keys `section_title`, `section_number`, `section_start`, `section_end` are available in the output template
+* **Parallel fragment downloads** by [shirt](https://github.com/shirt-dev)
+    * Use option `--concurrent-fragments` (`-N`) to set the number of threads (default 1)
+* Merge youtube-dl: Upto [commit/3be0980](https://github.com/ytdl-org/youtube-dl/commit/3be098010f667b14075e3dfad1e74e5e2becc8ea)
+* [zee5] Add Show Extractor by [Ashish0804](https://github.com/Ashish0804) and [pukkandan](https://github.com/pukkandan)
+* [rai] fix drm check [nixxo](https://github.com/nixxo)
+* [wimtv] Add extractor by [nixxo](https://github.com/nixxo)
+* [mtv] Add mtv.it and extract series metadata by [nixxo](https://github.com/nixxo)
+* [pluto.tv] Add extractor by [kevinoconnor7](https://github.com/kevinoconnor7)
+* [youtube] Rewrite comment extraction by [colethedj](https://github.com/colethedj)
+* [embedthumbnail] Set mtime correctly
+* Refactor some postprocessor/downloader code by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
+
+
 ### 2021.03.07
 * [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [colethedj](https://github.com/colethedj)
 * [youtube] Fix private feeds/playlists on multi-channel accounts by [colethedj](https://github.com/colethedj)
 * [youtube] Extract alerts from continuation by [colethedj](https://github.com/colethedj)
 * [cbs] Add support for ParamountPlus by [shirt](https://github.com/shirt-dev)
-* [mxplayer] Rewrite extractor with show support by [pukkandan](https://github.com/pukkandan) and [Ashish](https://github.com/Ashish)
+* [mxplayer] Rewrite extractor with show support by [pukkandan](https://github.com/pukkandan) and [Ashish0804](https://github.com/Ashish0804)
 * [gedi] Improvements from youtube-dl by [nixxo](https://github.com/nixxo)
 * [vimeo] Fix videos with password by [teesid](https://github.com/teesid)
-* [lbry] Support lbry:// url by [nixxo](https://github.com/nixxo)
+* [lbry] Support `lbry://` url by [nixxo](https://github.com/nixxo)
 * [bilibili] Change `Accept` header by [pukkandan](https://github.com/pukkandan) and [animelover1984](https://github.com/animelover1984)
 * [trovo] Pass origin header
 * [rai] Check for DRM by [nixxo](https://github.com/nixxo)
-* [downloader] Fix bug for ffmpeg/httpie
+* [downloader] Fix bug for `ffmpeg`/`httpie`
 * [update] Fix updater removing the executable bit on some UNIX distros
 * [update] Fix current build hash for UNIX
-* [documentation] Include wget/curl/aria2c install instructions for Unix by [Ashish](https://github.com/Ashish)
+* [documentation] Include wget/curl/aria2c install instructions for Unix by [Ashish0804](https://github.com/Ashish0804)
 * Fix some videos downloading with `m3u8` extension
 * Remove "fixup is ignored" warning when fixup wasn't passed by user

@@ -40,7 +144,6 @@
 ### 2021.03.03.2
 * [build] Fix bug

-
 ### 2021.03.03
 * [youtube] Use new browse API for continuation page extraction by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
 * Fix HLS playlist downloading by [shirt](https://github.com/shirt-dev)
@@ -82,7 +185,7 @@
 * Merge youtube-dl: Upto [commit/4460329](https://github.com/ytdl-org/youtube-dl/commit/44603290e5002153f3ebad6230cc73aef42cc2cd) (except tmz, gedi)
 * [Readthedocs](https://yt-dlp.readthedocs.io) support by [shirt](https://github.com/shirt-dev)
 * [youtube] Show if video was a live stream in info (`was_live`)
-* [Zee5] Add new extractor by [Ashish](https://github.com/Ashish) and [pukkandan](https://github.com/pukkandan)
+* [Zee5] Add new extractor by [Ashish0804](https://github.com/Ashish0804) and [pukkandan](https://github.com/pukkandan)
 * [jwplatform] Add support for `hyland.com`
 * [tennistv] Fix extractor
 * [hls] Support media initialization by [shirt](https://github.com/shirt-dev)
@@ -278,7 +381,7 @@
 * [archive.org] Fix extractor and add support for audio and playlists by [wporr](https://github.com/wporr)
 * [Animelab] Added by [mariuszskon](https://github.com/mariuszskon)
 * [youtube:search] Fix view_count by [ohnonot](https://github.com/ohnonot)
-* [youtube] Show if video is embeddable in info
+* [youtube] Show if video is embeddable in info (`playable_in_embed`)
 * Update version badge automatically in README
 * Enable `test_youtube_search_matching`
 * Create `to_screen` and similar functions in postprocessor/common
@@ -295,7 +398,7 @@

 ### 2021.01.08
 * Merge youtube-dl: Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.08) except stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f))
-* Moved changelog to seperate file
+* Moved changelog to separate file


 ### 2021.01.07-1
@@ -348,31 +451,33 @@
 * Cleaned up the fork for public use


+**PS**: All uncredited changes above this point are authored by [pukkandan](https://github.com/pukkandan)
+
 ### Unreleased changes in [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc)
-* Updated to youtube-dl release 2020.11.26
-* [youtube]
+* Updated to youtube-dl release 2020.11.26 by [pukkandan](https://github.com/pukkandan)
+* Youtube improvements by [pukkandan](https://github.com/pukkandan)
    * Implemented all Youtube Feeds (ytfav, ytwatchlater, ytsubs, ythistory, ytrec) and SearchURL
-    * Fix ytsearch not returning results sometimes due to promoted content
-    * Temporary fix for automatic captions - disable json3
    * Fix some improper Youtube URLs
    * Redirect channel home to /video
    * Print youtube's warning message
-    * Multiple pages are handled better for feeds
+    * Handle Multiple pages for feeds better
+* [youtube] Fix ytsearch not returning results sometimes due to promoted content by [colethedj](https://github.com/colethedj)
+* [youtube] Temporary fix for automatic captions - disable json3 by [blackjack4494](https://github.com/blackjack4494)
 * Add --break-on-existing by [gergesh](https://github.com/gergesh)
-* Pre-check video IDs in the archive before downloading
-* [bitwave.tv] New extractor
-* [Gedi] Add extractor
-* [Rcs] Add new extractor
-* [skyit] Add support for multiple Sky Italia website and removed old skyitalia extractor
-* [france.tv] Fix thumbnail URL
-* [ina] support mobile links
-* [instagram] Fix extractor
-* [itv] BTCC new pages' URL update (articles instead of races)
-* [SouthparkDe] Support for English URLs
-* [spreaker] fix SpreakerShowIE test URL
-* [Vlive] Fix playlist handling when downloading a channel
-* [generic] Detect embedded bitchute videos
-* [generic] Extract embedded youtube and twitter videos
-* [ffmpeg] Ensure all streams are copied
-* Fix for os.rename error when embedding thumbnail to video in a different drive
-* make_win.bat: don't use UPX to pack vcruntime140.dll
+* Pre-check video IDs in the archive before downloading by [pukkandan](https://github.com/pukkandan)
+* [bitwave.tv] New extractor by [lorpus](https://github.com/lorpus)
+* [Gedi] Add extractor by [nixxo](https://github.com/nixxo)
+* [Rcs] Add new extractor by [nixxo](https://github.com/nixxo)
+* [skyit] New skyitalia extractor by [nixxo](https://github.com/nixxo)
+* [france.tv] Fix thumbnail URL by [renalid](https://github.com/renalid)
+* [ina] support mobile links by [B0pol](https://github.com/B0pol)
+* [instagram] Fix thumbnail extractor by [nao20010128nao](https://github.com/nao20010128nao)
+* [SouthparkDe] Support for English URLs by [xypwn](https://github.com/xypwn)
+* [spreaker] fix SpreakerShowIE test URL by [pukkandan](https://github.com/pukkandan)
+* [Vlive] Fix playlist handling when downloading a channel by [kyuyeunk](https://github.com/kyuyeunk)
+* [tmz] Fix extractor by [diegorodriguezv](https://github.com/diegorodriguezv)
+* [generic] Detect embedded bitchute videos by [pukkandan](https://github.com/pukkandan)
+* [generic] Extract embedded youtube and twitter videos by [diegorodriguezv](https://github.com/diegorodriguezv)
+* [ffmpeg] Ensure all streams are copied by [pukkandan](https://github.com/pukkandan)
+* [embedthumbnail] Fix for os.rename error by [pukkandan](https://github.com/pukkandan)
+* make_win.bat: don't use UPX to pack vcruntime140.dll by [jbruchon](https://github.com/jbruchon)
@@ -1,39 +1,44 @@
+<div align="center">
+
 # YT-DLP
+A command-line program to download videos from YouTube and many other [video platforms](supportedsites.md)

-[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=brightgreen&label=Release)](https://github.com/yt-dlp/yt-dlp/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
+<!-- GHA doesnot have for-the-badge style
 [![CI Status](https://github.com/yt-dlp/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/yt-dlp/yt-dlp/actions)
-[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord)](https://discord.gg/S75JaBna)  
+-->
+[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=brightgreen&label=Release&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/releases/latest)
+[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg?style=for-the-badge)](LICENSE)
+[![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest&style=for-the-badge)](https://yt-dlp.readthedocs.io)
+[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord&style=for-the-badge)](https://discord.gg/H5MNcFW63r)  

-[![Commits](https://img.shields.io/github/commit-activity/m/yt-dlp/yt-dlp?label=commits)](https://github.com/yt-dlp/yt-dlp/commits)
-[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master)](https://github.com/yt-dlp/yt-dlp/commits)
-[![Downloads](https://img.shields.io/github/downloads/yt-dlp/yt-dlp/total)](https://github.com/yt-dlp/yt-dlp/releases/latest)
-[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp)
-[![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest)](https://yt-dlp.readthedocs.io)
+[![Commits](https://img.shields.io/github/commit-activity/m/yt-dlp/yt-dlp?label=commits&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
+[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master?style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
+[![Downloads](https://img.shields.io/github/downloads/yt-dlp/yt-dlp/total?style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/releases/latest)
+[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi&style=for-the-badge)](https://pypi.org/project/yt-dlp)

-A command-line program to download videos from youtube.com and many other [video platforms](supportedsites.md)
+</div>

-This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which is inturn a fork of [youtube-dl](https://github.com/ytdl-org/youtube-dl)
+yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on the now inactive [youtube-dlc](https://github.com/blackjack4494/yt-dlc). The main focus of this project is adding new features and patches while also keeping up to date with the original project

 * [NEW FEATURES](#new-features)
 * [INSTALLATION](#installation)
+    * [Dependancies](#dependancies)
    * [Update](#update)
    * [Compile](#compile)
-* [DESCRIPTION](#description)
-* [OPTIONS](#options)
+* [USAGE AND OPTIONS](#usage-and-options)
+    * [General Options](#general-options)
    * [Network Options](#network-options)
-    * [Geo Restriction](#geo-restriction)
+    * [Geo-restriction](#geo-restriction)
    * [Video Selection](#video-selection)
    * [Download Options](#download-options)
    * [Filesystem Options](#filesystem-options)
-    * [Thumbnail images](#thumbnail-images)
+    * [Thumbnail Options](#thumbnail-options)
    * [Internet Shortcut Options](#internet-shortcut-options)
    * [Verbosity and Simulation Options](#verbosity-and-simulation-options)
    * [Workarounds](#workarounds)
    * [Video Format Options](#video-format-options)
    * [Subtitle Options](#subtitle-options)
    * [Authentication Options](#authentication-options)
-    * [Adobe Pass Options](#adobe-pass-options)
    * [Post-processing Options](#post-processing-options)
    * [SponSkrub (SponsorBlock) Options](#sponskrub-sponsorblock-options)
    * [Extractor Options](#extractor-options)
@@ -46,8 +51,12 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
    * [Filtering Formats](#filtering-formats)
    * [Sorting Formats](#sorting-formats)
    * [Format Selection examples](#format-selection-examples)
+* [MODIFYING METADATA](#modifying-metadata)
+    * [Modifying metadata examples](#modifying-metadata-examples)
 * [PLUGINS](#plugins)
+* [DEPRECATED OPTIONS](#deprecated-options)
 * [MORE](#more)
+</div>


 # NEW FEATURES
@@ -57,7 +66,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/

 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))

-* **Merged with youtube-dl v2021.03.03**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl v2021.04.07**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)

 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.

@@ -66,17 +75,17 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs works correctly
    * Redirect channel's home URL automatically to `/video` to preserve the old behaviour

-* **Aria2c with HLS/DASH**: You can use aria2c as the external downloader for DASH(mpd) and HLS(m3u8) formats. No more slow ffmpeg/native downloads
+* **Split video by chapters**: Videos can be split into multiple files based on chapters using `--split-chapters`

-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5
+* **Multi-threaded fragment downloads**: Download multiple fragments of m3u8/mpd videos in parallel. Use `--concurrent-fragments` (`-N`) option to set the number of threads used

-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina, rumble, tennistv
+* **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats

-* **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula

-* **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [`--paths`](https://github.com/yt-dlp/yt-dlp/#:~:text=-P,%20--paths%20TYPE:PATH) for details
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter

-<!-- Relative link doesn't work for "#:~:text=" -->
+* **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to using `--paths` (`-P`)

 * **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details

@@ -84,6 +93,8 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/

 * **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc

+* **Plugin extractors**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
+
 * **Self-updater**: The releases can be updated using `yt-dlp -U`


@@ -96,6 +107,7 @@ If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the


 # INSTALLATION
+yt-dlp is not platform specific. So it should work on your Unix box, on Windows or on macOS

 You can install yt-dlp using one of the following methods:
 * Download the binary from the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest) (recommended method)
@@ -120,8 +132,14 @@ sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o
 sudo chmod a+rx /usr/local/bin/yt-dlp
 ```

+### DEPENDANCIES
+
+Python versions 2.6, 2.7, or 3.2+ are currently supported. However, 3.2+ is strongly recommended and python2 support will be deprecated in the future.
+
+Although there are no required dependancies, `ffmpeg` and `ffprobe` are highly recommended. Other optional dependancies are `sponskrub`, `AtomicParsley`, `mutagen`, `pycryptodome` and any of the supported external downloaders. Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
+
 ### UPDATE
-Starting from version `2021.02.09`, you can use `yt-dlp -U` to update if you are using the provided release.
+You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.

 ### COMPILE
@@ -143,13 +161,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t

 **Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number

-# DESCRIPTION
-**yt-dlp** is a command-line program to download videos from youtube.com many other [video platforms](supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
+# USAGE AND OPTIONS

    yt-dlp [OPTIONS] [--] URL [URL...]

-
-# OPTIONS
 `Ctrl+F` is your friend :D
 <!-- Autogenerated -->

@@ -194,7 +209,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     only list them
    --no-flat-playlist               Extract the videos of a playlist
    --mark-watched                   Mark videos watched (YouTube only)
-    --no-mark-watched                Do not mark videos watched
+    --no-mark-watched                Do not mark videos watched (default)
    --no-colors                      Do not emit color codes in output

 ## Network Options:
@@ -208,7 +223,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    -4, --force-ipv4                 Make all connections via IPv4
    -6, --force-ipv6                 Make all connections via IPv6

-## Geo Restriction:
+## Geo-restriction:
    --geo-verification-proxy URL     Use this proxy to verify the IP address for
                                     some geo-restricted sites. The default
                                     proxy specified by --proxy (or none, if the
@@ -292,11 +307,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --break-on-reject                Stop the download process when encountering
                                     a file that has been filtered out
    --no-download-archive            Do not use archive file (default)
-    --include-ads                    Download advertisements as well
-                                     (experimental)
-    --no-include-ads                 Do not download advertisements (default)

 ## Download Options:
+    -N, --concurrent-fragments N     Number of fragments of a dash/hlsnative
+                                     video that should be download concurrently
+                                     (default is 1)
    -r, --limit-rate RATE            Maximum download rate in bytes per second
                                     (e.g. 50K or 4.2M)
    -R, --retries RETRIES            Number of retries (default is 10), or
@@ -330,10 +345,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --playlist-random                Download playlist videos in random order
    --xattr-set-filesize             Set file xattribute ytdl.filesize with
                                     expected file size
-    --hls-prefer-native              Use the native HLS downloader instead of
-                                     ffmpeg
-    --hls-prefer-ffmpeg              Use ffmpeg instead of the native HLS
-                                     downloader
    --hls-use-mpegts                 Use the mpegts container for HLS videos;
                                     allowing some players to play the video
                                     while downloading, and reducing the chance
@@ -343,10 +354,19 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --no-hls-use-mpegts              Do not use the mpegts container for HLS
                                     videos. This is default when not
                                     downloading live streams
-    --external-downloader NAME       Name or path of the external downloader to
-                                     use. Currently supports aria2c, avconv,
-                                     axel, curl, ffmpeg, httpie, wget
-                                     (Recommended: aria2c)
+    --downloader [PROTO:]NAME        Name or path of the external downloader to
+                                     use (optionally) prefixed by the protocols
+                                     (http, ftp, m3u8, dash, rstp, rtmp, mms) to
+                                     use it for. Currently supports native,
+                                     aria2c, avconv, axel, curl, ffmpeg, httpie,
+                                     wget (Recommended: aria2c). You can use
+                                     this option multiple times to set different
+                                     downloaders for different protocols. For
+                                     example, --downloader aria2c --downloader
+                                     "dash,m3u8:native" will use aria2c for
+                                     http/ftp downloads, and the native
+                                     downloader for dash/m3u8 downloads
+                                     (Alias: --external-downloader)
    --downloader-args NAME:ARGS      Give these arguments to the external
                                     downloader. Specify the downloader name and
                                     the arguments separated by a colon ":". You
@@ -358,7 +378,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     stdin), one URL per line. Lines starting
                                     with '#', ';' or ']' are considered as
                                     comments and ignored
-    -P, --paths TYPE:PATH            The paths where the files should be
+    -P, --paths TYPES:PATH           The paths where the files should be
                                     downloaded. Specify the type of file and
                                     the path separated by a colon ":". All the
                                     same types as --output are supported.
@@ -369,7 +389,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     home path after download is finished. This
                                     option is ignored if --output is an
                                     absolute path
-    -o, --output [TYPE:]TEMPLATE     Output filename template, see "OUTPUT
+    -o, --output [TYPES:]TEMPLATE    Output filename template; see "OUTPUT
                                     TEMPLATE" for details
    --output-na-placeholder TEXT     Placeholder value for unavailable meta
                                     fields in output filename template
@@ -419,6 +439,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     --write-description etc. (default)
    --no-write-playlist-metafiles    Do not write playlist metadata when using
                                     --write-info-json, --write-description etc.
+    --clean-infojson                 Remove some private fields such as
+                                     filenames from the infojson. Note that it
+                                     could still contain some personal
+                                     information (default)
+    --no-clean-infojson              Write all fields to the infojson
    --get-comments                   Retrieve video comments to be placed in the
                                     .info.json file. The comments are fetched
                                     even without this option if the extraction
@@ -440,7 +465,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --no-cache-dir                   Disable filesystem caching
    --rm-cache-dir                   Delete all filesystem cache files

-## Thumbnail Images:
+## Thumbnail Options:
    --write-thumbnail                Write thumbnail image to disk
    --no-write-thumbnail             Do not write thumbnail image to disk
                                     (default)
@@ -462,7 +487,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --no-warnings                    Ignore warnings
    -s, --simulate                   Do not download the video and do not write
                                     anything to disk
-    --skip-download                  Do not download the video
+    --skip-download                  Do not download the video but write all
+                                     related files (Alias: --no-download)
    -g, --get-url                    Simulate, quiet but print URL
    -e, --get-title                  Simulate, quiet but print title
    --get-id                         Simulate, quiet but print id
@@ -499,7 +525,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --encoding ENCODING              Force the specified encoding (experimental)
    --no-check-certificate           Suppress HTTPS certificate validation
    --prefer-insecure                Use an unencrypted connection to retrieve
-                                     information about the video. (Currently
+                                     information about the video (Currently
                                     supported only for YouTube)
    --user-agent UA                  Specify a custom user agent
    --referer URL                    Specify a custom referer, use if the video
@@ -513,15 +539,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --sleep-requests SECONDS         Number of seconds to sleep between requests
                                     during data extraction
    --sleep-interval SECONDS         Number of seconds to sleep before each
-                                     download when used alone or a lower bound
-                                     of a range for randomized sleep before each
-                                     download (minimum possible number of
-                                     seconds to sleep) when used along with
-                                     --max-sleep-interval
-    --max-sleep-interval SECONDS     Upper bound of a range for randomized sleep
-                                     before each download (maximum possible
-                                     number of seconds to sleep). Must only be
-                                     used along with --min-sleep-interval
+                                     download. This is the minimum time to sleep
+                                     when used along with --max-sleep-interval
+                                     (Alias: --min-sleep-interval)
+    --max-sleep-interval SECONDS     Maximum number of seconds to sleep. Can
+                                     only be used along with --min-sleep-interval
    --sleep-subtitles SECONDS        Number of seconds to sleep before each
                                     subtitle download

@@ -544,7 +566,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     into a single file
    --no-audio-multistreams          Only one audio stream is downloaded for
                                     each output file (default)
-    --all-formats                    Download all available video formats
    --prefer-free-formats            Prefer video formats with free containers
                                     over non-free ones of same quality. Use
                                     with "-S ext" to strictly prefer free
@@ -571,16 +592,16 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --write-subs                     Write subtitle file
    --no-write-subs                  Do not write subtitle file (default)
    --write-auto-subs                Write automatically generated subtitle file
-                                     (YouTube only)
-    --no-write-auto-subs             Do not write automatically generated
-                                     subtitle file (default)
+                                     (Alias: --write-automatic-subs)
+    --no-write-auto-subs             Do not write auto-generated subtitles
+                                     (default) (Alias: --no-write-automatic-subs)
    --all-subs                       Download all the available subtitles of the
                                     video
    --list-subs                      List all available subtitles for the video
    --sub-format FORMAT              Subtitle format, accepts formats
                                     preference, for example: "srt" or
                                     "ass/srt/best"
-    --sub-lang LANGS                 Languages of the subtitles to download
+    --sub-langs LANGS                Languages of the subtitles to download
                                     (optional) separated by commas, use --list-
                                     subs for available language tags

@@ -591,8 +612,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    -2, --twofactor TWOFACTOR        Two-factor authentication code
    -n, --netrc                      Use .netrc authentication data
    --video-password PASSWORD        Video password (vimeo, youku)
-
-## Adobe Pass Options:
    --ap-mso MSO                     Adobe Pass multiple-system operator (TV
                                     provider) identifier, use --ap-list-mso for
                                     a list of available MSOs
@@ -629,23 +648,24 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     Specify the postprocessor/executable name
                                     and the arguments separated by a colon ":"
                                     to give the argument to the specified
-                                     postprocessor/executable. Supported
-                                     postprocessors are: SponSkrub,
-                                     ExtractAudio, VideoRemuxer, VideoConvertor,
-                                     EmbedSubtitle, Metadata, Merger,
-                                     FixupStretched, FixupM4a, FixupM3u8,
-                                     SubtitlesConvertor and EmbedThumbnail. The
-                                     supported executables are: SponSkrub,
-                                     FFmpeg, FFprobe, and AtomicParsley. You can
+                                     postprocessor/executable. Supported PP are:
+                                     Merger, ExtractAudio, SplitChapters,
+                                     Metadata, EmbedSubtitle, EmbedThumbnail,
+                                     SubtitlesConvertor, ThumbnailsConvertor,
+                                     VideoRemuxer, VideoConvertor, SponSkrub,
+                                     FixupStretched, FixupM4a and FixupM3u8. The
+                                     supported executables are: AtomicParsley,
+                                     FFmpeg, FFprobe, and SponSkrub. You can
                                     also specify "PP+EXE:ARGS" to give the
                                     arguments to the specified executable only
                                     when being used by the specified
                                     postprocessor. Additionally, for
-                                     ffmpeg/ffprobe, a number can be appended to
-                                     the exe name seperated by "_i" to pass the
-                                     argument before the specified input file.
-                                     Eg: --ppa "Merger+ffmpeg_i1:-v quiet". You
-                                     can use this option multiple times to give
+                                     ffmpeg/ffprobe, "_i"/"_o" can be appended
+                                     to the prefix optionally followed by a
+                                     number to pass the argument before the
+                                     specified input/output file. Eg: --ppa
+                                     "Merger+ffmpeg_i1:-v quiet". You can use
+                                     this option multiple times to give
                                     different arguments to different
                                     postprocessors. (Alias: --ppa)
    -k, --keep-video                 Keep the intermediate video file on disk
@@ -661,20 +681,9 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --no-embed-thumbnail             Do not embed thumbnail (default)
    --add-metadata                   Write metadata to the video file
    --no-add-metadata                Do not write metadata (default)
-    --parse-metadata FIELD:FORMAT    Parse additional metadata like title/artist
-                                     from other fields. Give field name to
-                                     extract data from, and format of the field
-                                     seperated by a ":". Either regular
-                                     expression with named capture groups or a
-                                     similar syntax to the output template can
-                                     also be used. The parsed parameters replace
-                                     any existing values and can be use in
-                                     output template. This option can be used
-                                     multiple times. Example: --parse-metadata
-                                     "title:%(artist)s - %(title)s" matches a
-                                     title like "Coldplay - Paradise". Example
-                                     (regex): --parse-metadata
-                                     "description:Artist - (?P<artist>.+?)"
+    --parse-metadata FROM:TO         Parse additional metadata like title/artist
+                                     from other fields; see "MODIFYING METADATA"
+                                     for details
    --xattrs                         Write metadata to the video file's xattrs
                                     (using dublin core and xdg standards)
    --fixup POLICY                   Automatically correct known faults of the
@@ -686,11 +695,26 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     path to the binary or its containing
                                     directory
    --exec CMD                       Execute a command on the file after
-                                     downloading and post-processing, similar to
-                                     find's -exec syntax. Example: --exec 'adb
-                                     push {} /sdcard/Music/ && rm {}'
-    --convert-subs FORMAT            Convert the subtitles to other format
+                                     downloading and post-processing. Similar
+                                     syntax to the output template can be used
+                                     to pass any field as arguments to the
+                                     command. An additional field "filepath"
+                                     that contains the final path of the
+                                     downloaded file is also available. If no
+                                     fields are passed, "%(filepath)s" is
+                                     appended to the end of the command
+    --convert-subs FORMAT            Convert the subtitles to another format
                                     (currently supported: srt|ass|vtt|lrc)
+                                     (Alias: --convert-subtitles)
+    --convert-thumbnails FORMAT      Convert the thumbnails to another format
+                                     (currently supported: jpg)
+    --split-chapters                 Split video into multiple files based on
+                                     internal chapters. The "chapter:" prefix
+                                     can be used with "--paths" and "--output"
+                                     to set the output filename for the split
+                                     files. See "OUTPUT TEMPLATE" for details
+    --no-split-chapters              Do not split video based on chapters
+                                     (default)

 ## SponSkrub (SponsorBlock) Options:
 [SponSkrub](https://github.com/yt-dlp/SponSkrub) is a utility to
@@ -806,9 +830,9 @@ The `-o` option is used to indicate a template for the output file names while `

 **tl;dr:** [navigate me to examples](#output-template-examples).

-The basic usage of `-o` is not to set any template arguments when downloading a single file, like in `yt-dlp -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Date/time fields can also be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it inside the parantheses seperated from the field name using a `>`. For example, `%(duration>%H-%M-%S)s`.
+The basic usage of `-o` is not to set any template arguments when downloading a single file, like in `yt-dlp -o funny_video.flv "https://some/video"` (hard-coding file extension like this is not recommended). However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Date/time fields can also be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it inside the parantheses separated from the field name using a `>`. For example, `%(duration>%H-%M-%S)s`.

-Additionally, you can set different output templates for the various metadata files seperately from the general output template by specifying the type of file followed by the template seperated by a colon ":". The different filetypes supported are `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
+Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different filetypes supported are `subtitle`, `thumbnail`, `description`, `annotation`, `infojson`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.

 The available fields are:

@@ -817,6 +841,7 @@ The available fields are:
 - `url` (string): Video URL
 - `ext` (string): Video filename extension
 - `alt_title` (string): A secondary title of the video
+ - `description` (string): The description of the video
 - `display_id` (string): An alternative identifier for the video
 - `uploader` (string): Full name of the video uploader
 - `license` (string): License name the video is licensed under
@@ -840,6 +865,7 @@ The available fields are:
 - `is_live` (boolean): Whether this video is a live stream or a fixed-length video
 - `was_live` (boolean): Whether this video was originally a live stream
 - `playable_in_embed` (string): Whether this video is allowed to play in embedded players on other sites
+ - `availability` (string): Whether the video is 'private', 'premium_only', 'subscriber_only', 'needs_auth', 'unlisted' or 'public'
 - `start_time` (numeric): Time in seconds where the reproduction should start, as specified in the URL
 - `end_time` (numeric): Time in seconds where the reproduction should end, as specified in the URL
 - `format` (string): A human-readable description of the format
@@ -899,6 +925,13 @@ Available for the media that is a track or a part of a music album:
 - `disc_number` (numeric): Number of the disc or other physical medium the track belongs to
 - `release_year` (numeric): Year (YYYY) when the album was released

+Available for `chapter:` prefix when using `--split-chapters` for videos with internal chapters:
+
+ - `section_title` (string): Title of the chapter
+ - `section_number` (numeric): Number of the chapter within the file
+ - `section_start` (numeric): Start time of the chapter in seconds
+ - `section_end` (numeric): End time of the chapter in seconds
+
 Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).

 For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKcj`, this will result in a `yt-dlp test video-BaW_jenozKcj.mp4` file created in the current directory.
@@ -931,7 +964,7 @@ youtube-dl_test_video_.mp4          # A simple file name
 # Download YouTube playlist videos in separate directory indexed by video order in a playlist
 $ yt-dlp -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re

-# Download YouTube playlist videos in seperate directories according to their uploaded year
+# Download YouTube playlist videos in separate directories according to their uploaded year
 $ yt-dlp -o '%(upload_date>%Y)s/%(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re

 # Download all playlists of YouTube channel/user keeping each playlist in separate directory:
@@ -952,7 +985,7 @@ $ yt-dlp -o - BaW_jenozKc
 By default, yt-dlp tries to download the best available quality if you **don't** pass any options.
 This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg is unavailable, or if you use yt-dlp to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.

-The general syntax for format selection is `--f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download.
+The general syntax for format selection is `-f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download.

 **tl;dr:** [navigate me to examples](#format-selection-examples).

@@ -963,8 +996,9 @@ You can also use a file extension (currently `3gp`, `aac`, `flv`, `m4a`, `mp3`,
 You can also use special names to select particular edge case formats:

 - `all`: Select all formats
- - `b*`, `best*`: Select the best quality format irrespective of whether it contains video or audio.
- - `w*`, `worst*`: Select the worst quality format irrespective of whether it contains video or audio.
+ - `mergeall`: Select and merge all formats (Must be used with `--audio-multistreams`, `--video-multistreams` or both)
+ - `b*`, `best*`: Select the best quality format irrespective of whether it contains video or audio
+ - `w*`, `worst*`: Select the worst quality format irrespective of whether it contains video or audio
 - `b`, `best`: Select the best quality format that contains both video and audio. Equivalent to `best*[vcodec!=none][acodec!=none]`
 - `w`, `worst`: Select the worst quality format that contains both video and audio. Equivalent to `worst*[vcodec!=none][acodec!=none]`
 - `bv`, `bestvideo`: Select the best quality video-only format. Equivalent to `best*[acodec=none]`
@@ -976,7 +1010,9 @@ You can also use special names to select particular edge case formats:
 - `ba*`, `bestaudio*`: Select the best quality format that contains audio. It may also contain video. Equivalent to `best*[acodec!=none]`
 - `wa*`, `worstaudio*`: Select the worst quality format that contains audio. It may also contain video. Equivalent to `worst*[acodec!=none]`

-For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recomended to never actually use `worst` and related options. When your format selector is `worst`, the format which is worst in all respects is selected. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps` instead of `-f worst`. See [sorting formats](#sorting-formats) for more details.
+For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recomended not to use `worst` and related options. When your format selector is `worst`, the format which is worst in all respects is selected. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps` instead of `-f worst`. See [sorting formats](#sorting-formats) for more details.
+
+You can select the n'th best format of a type by using `best<type>.<n>`. For example, `best.2` will select the 2nd best combined format. Similarly, `bv*.3` will select the 3rd best format that contains a video stream.

 If you want to download multiple videos and they don't have the same formats available, you can specify the order of preference using slashes. Note that formats on the left hand side are preferred, for example `-f 22/17/18` will download format 22 if it's available, otherwise it will download format 17 if it's available, otherwise it will download format 18 if it's available, otherwise it will complain that no suitable formats are available for download.

@@ -1047,7 +1083,7 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
 - `br`: Equivalent to using `tbr,vbr,abr`
 - `asr`: Audio sample rate in Hz

-Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
+Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.

 The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.

@@ -1076,10 +1112,17 @@ $ yt-dlp
 # by default, bestvideo and bestaudio will have the same file name.
 $ yt-dlp -f 'bv,ba' -o '%(title)s.f%(format_id)s.%(ext)s'

+# Download and merge the best format that has a video stream,
+# and all audio-only formats into one file
+$ yt-dlp -f 'bv*+mergeall[vcodec=none]' --audio-multistreams
+
+# Download and merge the best format that has a video stream,
+# and the best 2 audio-only formats into one file
+$ yt-dlp -f 'bv*+ba+ba.2' --audio-multistreams


 # The following examples show the old method (without -S) of format selection
-# and how to use -S to achieve a similar but better result
+# and how to use -S to achieve a similar but (generally) better result

 # Download the worst video available (old method)
 $ yt-dlp -f 'wv*+wa/w'
@@ -1160,7 +1203,7 @@ $ yt-dlp -S '+codec:h264'
 $ yt-dlp -f '((bv*[fps>30]/bv*)[height<=720]/(wv*[fps>30]/wv*)) + ba / (b[fps>30]/b)[height<=720]/(w[fps>30]/w)'

 # Download the video with the largest resolution no better than 720p,
-# or the video with the smallest resolution available  if there is no such video,
+# or the video with the smallest resolution available if there is no such video,
 # prefering larger framerate for formats with the same resolution
 $ yt-dlp -S 'res:720,fps'

@@ -1172,11 +1215,77 @@ $ yt-dlp -S 'res:720,fps'
 $ yt-dlp -S '+res:480,codec,br'
 ```

+# MODIFYING METADATA
+
+The metadata obtained the the extractors can be modified by using `--parse-metadata FROM:TO`. The general syntax is to give the name of a field or a template (with similar syntax to [output template](#output-template)) to extract data from, and the format to interpret it as, separated by a colon `:`. Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields.
+
+Note that any field created by this can be used in the [output template](#output-template) and will also affect the media file's metadata added when using `--add-metadata`.
+
+You can also use this to change only the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. You can use this to set a different "description" and "synopsis", for example.
+
+## Modifying metadata examples
+
+Note that on Windows you may need to use double quotes instead of single.
+
+```bash
+# Interpret the title as "Artist - Title"
+$ yt-dlp --parse-metadata 'title:%(artist)s - %(title)s'
+
+# Regex example
+$ yt-dlp --parse-metadata 'description:Artist - (?P<artist>.+)'
+
+# Set title as "Series name S01E05"
+$ yt-dlp --parse-metadata '%(series)s S%(season_number)02dE%(episode_number)02d:%(title)s'
+
+# Set "comment" field in video metadata using description instead of webpage_url
+$ yt-dlp --parse-metadata 'description:(?s)(?P<meta_comment>.+)' --add-metadata
+
+```
+
 # PLUGINS

 Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently only `extractor` plugins are supported. Support for `downloader` and `postprocessor` plugins may be added in the future. See [ytdlp_plugins](ytdlp_plugins) for example.

 **Note**: `<root-dir>` is the directory of the binary (`<root-dir>/yt-dlp`), or the root directory of the module if you are running directly from source-code (`<root dir>/yt_dlp/__main__.py`)

+# DEPRECATED OPTIONS
+
+These are all the deprecated options and the current alternative to achieve the same effect
+
+    --cn-verification-proxy URL      --geo-verification-proxy URL
+    --id                             -o "%(id)s.%(ext)s"
+    -A, --auto-number                -o "%(autonumber)s-%(id)s.%(ext)s"
+    -t, --title                      -o "%(title)s-%(id)s.%(ext)s"
+    -l, --literal                    -o accepts literal names
+    --all-formats                    -f all
+    --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
+    --metadata-from-title FORMAT     --parse-metadata "%(title)s:FORMAT"
+    --prefer-avconv                  avconv is no longer officially supported (Alias: --no-prefer-ffmpeg)
+    --prefer-ffmpeg                  Default (Alias: --no-prefer-avconv)
+    --hls-prefer-native              --downloader "m3u8:native"
+    --hls-prefer-ffmpeg              --downloader "m3u8:ffmpeg"
+    --avconv-location                avconv is no longer officially supported
+    -C, --call-home                  Not implemented
+    --no-call-home                   Default
+    --include-ads                    Not implemented
+    --no-include-ads                 Default
+    --write-srt                      --write-subs
+    --no-write-srt                   --no-write-subs
+    --srt-lang LANGS                 --sub-langs LANGS
+    --prefer-unsecure                --prefer-insecure
+    --rate-limit RATE                --limit-rate RATE
+    --force-write-download-archive   --force-write-archive
+    --dump-intermediate-pages        --dump-pages
+    --dump-headers                   --print-traffic
+    --youtube-print-sig-code         No longer supported
+    --trim-file-names LENGTH         --trim-filenames LENGTH
+    --yes-overwrites                 --force-overwrites
+    --load-info                      --load-info-json
+    --split-tracks                   --split-chapters
+    --no-split-tracks                --no-split-chapters
+    --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
+    --test                           Only used for testing extractors
+
+
 # MORE
 For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl#faq)
@@ -3,6 +3,7 @@
 - **20min**
 - **220.ro**
 - **23video**
+ - **247sports**
 - **24video**
 - **3qsdn**: 3Q SDN
 - **3sat**
@@ -97,7 +98,8 @@
 - **bbc**: BBC
 - **bbc.co.uk**: BBC iPlayer
 - **bbc.co.uk:article**: BBC articles
- - **bbc.co.uk:iplayer:playlist**
+ - **bbc.co.uk:iplayer:episodes**
+ - **bbc.co.uk:iplayer:group**
 - **bbc.co.uk:playlist**
 - **BBVTV**
 - **Beatport**
@@ -170,7 +172,8 @@
 - **cbsnews**: CBS News
 - **cbsnews:embed**
 - **cbsnews:livevideo**: CBS News Live Videos
- - **CBSSports**
+ - **cbssports**
+ - **cbssports:embed**
 - **CCMA**
 - **CCTV**: 央视网
 - **CDA**
@@ -248,6 +251,8 @@
 - **DiscoveryGoPlaylist**
 - **DiscoveryNetworksDe**
 - **DiscoveryPlus**
+ - **DiscoveryPlusIndia**
+ - **DiscoveryPlusIndiaShow**
 - **DiscoveryVR**
 - **Disney**
 - **dlive:stream**
@@ -456,6 +461,8 @@
 - **kuwo:singer**: 酷我音乐 - 歌手
 - **kuwo:song**: 酷我音乐
 - **la7.it**
+ - **la7.it:pod:episode**
+ - **la7.it:podcast**
 - **laola1tv**
 - **laola1tv:embed**
 - **lbry**
@@ -479,6 +486,8 @@
 - **limelight**
 - **limelight:channel**
 - **limelight:channel_list**
+ - **LineLive**
+ - **LineLiveChannel**
 - **LineTV**
 - **linkedin:learning**
 - **linkedin:learning:course**
@@ -505,6 +514,7 @@
 - **mangomolo:live**
 - **mangomolo:video**
 - **ManyVids**
+ - **MaoriTV**
 - **Markiza**
 - **MarkizaPage**
 - **massengeschmack.tv**
@@ -543,6 +553,7 @@
 - **mixcloud:playlist**
 - **mixcloud:user**
 - **MLB**
+ - **MLBVideo**
 - **Mnet**
 - **MNetTV**
 - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
@@ -561,6 +572,8 @@
 - **mtg**: MTG services
 - **mtv**
 - **mtv.de**
+ - **mtv.it**
+ - **mtv.it:programma**
 - **mtv:video**
 - **mtvjapan**
 - **mtvservices:embedded**
@@ -602,6 +615,7 @@
 - **ndr:embed**
 - **ndr:embed:base**
 - **NDTV**
+ - **Nebula**
 - **NerdCubedFeed**
 - **netease:album**: 网易云音乐 - 专辑
 - **netease:djradio**: 网易云音乐 - 电台
@@ -620,6 +634,7 @@
 - **NextTV**: 壹電視
 - **Nexx**
 - **NexxEmbed**
+ - **NFHSNetwork**
 - **nfl.com** (Currently broken)
 - **nfl.com:article** (Currently broken)
 - **NhkVod**
@@ -632,6 +647,7 @@
 - **nicknight**
 - **niconico**: ニコニコ動画
 - **NiconicoPlaylist**
+ - **NiconicoUser**
 - **Nintendo**
 - **Nitter**
 - **njoy**: N-JOY
@@ -702,6 +718,9 @@
 - **OutsideTV**
 - **PacktPub**
 - **PacktPubCourse**
+ - **PalcoMP3:artist**
+ - **PalcoMP3:song**
+ - **PalcoMP3:video**
 - **pandora.tv**: 판도라TV
 - **ParamountNetwork**
 - **parliamentlive.tv**: UK parliament videos
@@ -734,6 +753,7 @@
 - **Playwire**
 - **pluralsight**
 - **pluralsight:course**
+ - **PlutoTV**
 - **podomatic**
 - **Pokemon**
 - **PokemonWatch**
@@ -914,6 +934,7 @@
 - **stanfordoc**: Stanford Open ClassRoom
 - **Steam**
 - **Stitcher**
+ - **StitcherShow**
 - **StoryFire**
 - **StoryFireSeries**
 - **StoryFireUser**
@@ -1086,6 +1107,7 @@
 - **Vidbit**
 - **Viddler**
 - **Videa**
+ - **video.arnes.si**: Arnes Video
 - **video.google:search**: Google Video search
 - **video.sky.it**
 - **video.sky.it:live**
@@ -1171,6 +1193,7 @@
 - **Weibo**
 - **WeiboMobile**
 - **WeiqiTV**: WQTV
+ - **WimTV**
 - **Wistia**
 - **WistiaPlaylist**
 - **wnl**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
@@ -1241,7 +1264,9 @@
 - **ZDF**
 - **ZDFChannel**
 - **Zee5**
+ - **zee5:series**
 - **Zhihu**
 - **zingmp3**: mp3.zing.vn
+ - **zingmp3:album**
 - **zoom**
 - **Zype**
@@ -14,10 +14,10 @@ from yt_dlp.postprocessor import MetadataFromFieldPP, MetadataFromTitlePP
 class TestMetadataFromField(unittest.TestCase):
    def test_format_to_regex(self):
        pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
-        self.assertEqual(pp._data[0]['regex'], r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')
+        self.assertEqual(pp._data[0]['regex'], r'(?P<title>.+)\ \-\ (?P<artist>.+)')


 class TestMetadataFromTitle(unittest.TestCase):
    def test_format_to_regex(self):
        pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')
+        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
@@ -23,6 +23,7 @@ from yt_dlp.utils import (
    clean_html,
    clean_podcast_url,
    date_from_str,
+    datetime_from_str,
    DateRange,
    detect_exe_version,
    determine_ext,
@@ -311,8 +312,18 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(date_from_str('yesterday'), date_from_str('now-1day'))
        self.assertEqual(date_from_str('now+7day'), date_from_str('now+1week'))
        self.assertEqual(date_from_str('now+14day'), date_from_str('now+2week'))
-        self.assertEqual(date_from_str('now+365day'), date_from_str('now+1year'))
-        self.assertEqual(date_from_str('now+30day'), date_from_str('now+1month'))
+        self.assertEqual(date_from_str('20200229+365day'), date_from_str('20200229+1year'))
+        self.assertEqual(date_from_str('20210131+28day'), date_from_str('20210131+1month'))
+
+    def test_datetime_from_str(self):
+        self.assertEqual(datetime_from_str('yesterday', precision='day'), datetime_from_str('now-1day', precision='auto'))
+        self.assertEqual(datetime_from_str('now+7day', precision='day'), datetime_from_str('now+1week', precision='auto'))
+        self.assertEqual(datetime_from_str('now+14day', precision='day'), datetime_from_str('now+2week', precision='auto'))
+        self.assertEqual(datetime_from_str('20200229+365day', precision='day'), datetime_from_str('20200229+1year', precision='auto'))
+        self.assertEqual(datetime_from_str('20210131+28day', precision='day'), datetime_from_str('20210131+1month', precision='auto'))
+        self.assertEqual(datetime_from_str('20210131+59day', precision='day'), datetime_from_str('20210131+2month', precision='auto'))
+        self.assertEqual(datetime_from_str('now+1day', precision='hour'), datetime_from_str('now+24hours', precision='auto'))
+        self.assertEqual(datetime_from_str('now+23hours', precision='hour'), datetime_from_str('now+23hours', precision='auto'))

    def test_daterange(self):
        _20century = DateRange("19000101", "20000101")
@@ -180,6 +180,8 @@ def _real_main(argv=None):
    if opts.overwrites:
        # --yes-overwrites implies --no-continue
        opts.continue_dl = False
+    if opts.concurrent_fragment_downloads <= 0:
+        raise ValueError('Concurrent fragments must be positive')

    def parse_retries(retries, name=''):
        if retries in ('inf', 'infinite'):
@@ -226,8 +228,11 @@ def _real_main(argv=None):
        if not re.match(remux_regex, opts.remuxvideo):
            parser.error('invalid video remux format specified')
    if opts.convertsubtitles is not None:
-        if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
+        if opts.convertsubtitles not in ('srt', 'vtt', 'ass', 'lrc'):
            parser.error('invalid subtitle format specified')
+    if opts.convertthumbnails is not None:
+        if opts.convertthumbnails not in ('jpg', ):
+            parser.error('invalid thumbnail format specified')

    if opts.date is not None:
        date = DateRange.day(opts.date)
@@ -277,9 +282,14 @@ def _real_main(argv=None):

    def report_conflict(arg1, arg2):
        write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
+
    if opts.remuxvideo and opts.recodevideo:
        report_conflict('--recode-video', '--remux-video')
        opts.remuxvideo = False
+    if opts.sponskrub_cut and opts.split_chapters and opts.sponskrub is not False:
+        report_conflict('--split-chapter', '--sponskrub-cut')
+        opts.sponskrub_cut = False
+
    if opts.allow_unplayable_formats:
        if opts.extractaudio:
            report_conflict('--allow-unplayable-formats', '--extract-audio')
@@ -315,7 +325,22 @@ def _real_main(argv=None):
        postprocessors.append({
            'key': 'MetadataFromField',
            'formats': opts.metafromfield,
-            'when': 'beforedl'
+            # Run this immediately after extraction is complete
+            'when': 'pre_process'
+        })
+    if opts.convertsubtitles:
+        postprocessors.append({
+            'key': 'FFmpegSubtitlesConvertor',
+            'format': opts.convertsubtitles,
+            # Run this before the actual video download
+            'when': 'before_dl'
+        })
+    if opts.convertthumbnails:
+        postprocessors.append({
+            'key': 'FFmpegThumbnailsConvertor',
+            'format': opts.convertthumbnails,
+            # Run this before the actual video download
+            'when': 'before_dl'
        })
    if opts.extractaudio:
        postprocessors.append({
@@ -344,15 +369,11 @@ def _real_main(argv=None):
    # so metadata can be added here.
    if opts.addmetadata:
        postprocessors.append({'key': 'FFmpegMetadata'})
-    if opts.convertsubtitles:
-        postprocessors.append({
-            'key': 'FFmpegSubtitlesConvertor',
-            'format': opts.convertsubtitles,
-        })
    if opts.embedsubtitles:
        already_have_subtitle = opts.writesubtitles
        postprocessors.append({
            'key': 'FFmpegEmbedSubtitle',
+            # already_have_subtitle = True prevents the file from being deleted after embedding
            'already_have_subtitle': already_have_subtitle
        })
        if not already_have_subtitle:
@@ -361,19 +382,9 @@ def _real_main(argv=None):
    # this was the old behaviour if only --all-sub was given.
    if opts.allsubtitles and not opts.writeautomaticsub:
        opts.writesubtitles = True
-    if opts.embedthumbnail:
-        already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
-        postprocessors.append({
-            'key': 'EmbedThumbnail',
-            'already_have_thumbnail': already_have_thumbnail
-        })
-        if not already_have_thumbnail:
-            opts.writethumbnail = True
-    # XAttrMetadataPP should be run after post-processors that may change file
-    # contents
-    if opts.xattrs:
-        postprocessors.append({'key': 'XAttrMetadata'})
-    # This should be below all ffmpeg PP because it may cut parts out from the video
+    # This should be above EmbedThumbnail since sponskrub removes the thumbnail attachment
+    # but must be below EmbedSubtitle and FFmpegMetadata
+    # See https://github.com/yt-dlp/yt-dlp/issues/204 , https://github.com/faissaloo/SponSkrub/issues/29
    # If opts.sponskrub is None, sponskrub is used, but it silently fails if the executable can't be found
    if opts.sponskrub is not False:
        postprocessors.append({
@@ -384,12 +395,27 @@ def _real_main(argv=None):
            'force': opts.sponskrub_force,
            'ignoreerror': opts.sponskrub is None,
        })
+    if opts.embedthumbnail:
+        already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
+        postprocessors.append({
+            'key': 'EmbedThumbnail',
+            # already_have_thumbnail = True prevents the file from being deleted after embedding
+            'already_have_thumbnail': already_have_thumbnail
+        })
+        if not already_have_thumbnail:
+            opts.writethumbnail = True
+    if opts.split_chapters:
+        postprocessors.append({'key': 'FFmpegSplitChapters'})
+    # XAttrMetadataPP should be run after post-processors that may change file contents
+    if opts.xattrs:
+        postprocessors.append({'key': 'XAttrMetadata'})
    # ExecAfterDownload must be the last PP
    if opts.exec_cmd:
        postprocessors.append({
            'key': 'ExecAfterDownload',
            'exec_cmd': opts.exec_cmd,
-            'when': 'aftermove'
+            # Run this only after the files have been moved to their final locations
+            'when': 'after_move'
        })

    def report_args_compat(arg, name):
@@ -415,7 +441,6 @@ def _real_main(argv=None):
        else match_filter_func(opts.match_filter))

    ydl_opts = {
-        'convertsubtitles': opts.convertsubtitles,
        'usenetrc': opts.usenetrc,
        'username': opts.username,
        'password': opts.password,
@@ -463,6 +488,7 @@ def _real_main(argv=None):
        'extractor_retries': opts.extractor_retries,
        'skip_unavailable_fragments': opts.skip_unavailable_fragments,
        'keep_fragments': opts.keep_fragments,
+        'concurrent_fragment_downloads': opts.concurrent_fragment_downloads,
        'buffersize': opts.buffersize,
        'noresizebuffer': opts.noresizebuffer,
        'http_chunk_size': opts.http_chunk_size,
@@ -482,6 +508,7 @@ def _real_main(argv=None):
        'writeannotations': opts.writeannotations,
        'writeinfojson': opts.writeinfojson,
        'allow_playlist_files': opts.allow_playlist_files,
+        'clean_infojson': opts.clean_infojson,
        'getcomments': opts.getcomments,
        'writethumbnail': opts.writethumbnail,
        'write_all_thumbnails': opts.write_all_thumbnails,
@@ -78,6 +78,15 @@ try:
 except ImportError:  # Python 2
    import Cookie as compat_cookies

+if sys.version_info[0] == 2:
+    class compat_cookies_SimpleCookie(compat_cookies.SimpleCookie):
+        def load(self, rawdata):
+            if isinstance(rawdata, compat_str):
+                rawdata = str(rawdata)
+            return super(compat_cookies_SimpleCookie, self).load(rawdata)
+else:
+    compat_cookies_SimpleCookie = compat_cookies.SimpleCookie
+
 try:
    import html.entities as compat_html_entities
 except ImportError:  # Python 2
@@ -3020,6 +3029,7 @@ __all__ = [
    'compat_cookiejar',
    'compat_cookiejar_Cookie',
    'compat_cookies',
+    'compat_cookies_SimpleCookie',
    'compat_ctypes_WINFUNCTYPE',
    'compat_etree_Element',
    'compat_etree_fromstring',
@@ -1,5 +1,6 @@
 from __future__ import unicode_literals

+from ..compat import compat_str
 from ..utils import (
    determine_protocol,
 )
@@ -42,6 +43,23 @@ PROTOCOL_MAP = {
 }


+def shorten_protocol_name(proto, simplify=False):
+    short_protocol_names = {
+        'm3u8_native': 'm3u8_n',
+        'http_dash_segments': 'dash',
+        'niconico_dmc': 'dmc',
+    }
+    if simplify:
+        short_protocol_names.update({
+            'https': 'http',
+            'ftps': 'ftp',
+            'm3u8_native': 'm3u8',
+            'm3u8_frag_urls': 'm3u8',
+            'dash_frag_urls': 'dash',
+        })
+    return short_protocol_names.get(proto, proto)
+
+
 def get_suitable_downloader(info_dict, params={}, default=HttpFD):
    """Get the downloader class that can handle the info dict."""
    protocol = determine_protocol(info_dict)
@@ -50,8 +68,14 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):
    # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
    #     return FFmpegFD

-    external_downloader = params.get('external_downloader')
-    if external_downloader is not None:
+    downloaders = params.get('external_downloader')
+    external_downloader = (
+        downloaders if isinstance(downloaders, compat_str) or downloaders is None
+        else downloaders.get(shorten_protocol_name(protocol, True), downloaders.get('default')))
+    if external_downloader and external_downloader.lower() == 'native':
+        external_downloader = 'native'
+
+    if external_downloader not in (None, 'native'):
        ed = get_external_downloader(external_downloader)
        if ed.can_download(info_dict, external_downloader):
            return ed
@@ -59,6 +83,8 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):
    if protocol.startswith('m3u8'):
        if info_dict.get('is_live'):
            return FFmpegFD
+        elif external_downloader == 'native':
+            return HlsFD
        elif _get_real_downloader(info_dict, 'frag_urls', params, None):
            return HlsFD
        elif params.get('hls_prefer_native') is True:
@@ -70,6 +96,7 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):


 __all__ = [
-    'get_suitable_downloader',
    'FileDownloader',
+    'get_suitable_downloader',
+    'shorten_protocol_name',
 ]
@@ -326,6 +326,12 @@ class FileDownloader(object):
        """Report it was impossible to resume download."""
        self.to_screen('[download] Unable to resume')

+    @staticmethod
+    def supports_manifest(manifest):
+        """ Whether the downloader can download the fragments from the manifest.
+        Redefine in subclasses if needed. """
+        pass
+
    def download(self, filename, info_dict, subtitle=False):
        """Download to a filename using the info from info_dict
        Return True on success and False otherwise
@@ -1,18 +1,26 @@
 from __future__ import unicode_literals

+try:
+    import concurrent.futures
+    can_threaded_download = True
+except ImportError:
+    can_threaded_download = False
+
 from ..downloader import _get_real_downloader
 from .fragment import FragmentFD

 from ..compat import compat_urllib_error
 from ..utils import (
    DownloadError,
+    sanitize_open,
    urljoin,
 )


 class DashSegmentsFD(FragmentFD):
    """
-    Download segments in a DASH manifest
+    Download segments in a DASH manifest. External downloaders can take over
+    the fragment downloads by supporting the 'dash_frag_urls' protocol
    """

    FD_NAME = 'dashsegments'
@@ -22,7 +30,7 @@ class DashSegmentsFD(FragmentFD):
        fragments = info_dict['fragments'][:1] if self.params.get(
            'test', False) else info_dict['fragments']

-        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+        real_downloader = _get_real_downloader(info_dict, 'dash_frag_urls', self.params, None)

        ctx = {
            'filename': filename,
@@ -37,7 +45,7 @@ class DashSegmentsFD(FragmentFD):
        fragment_retries = self.params.get('fragment_retries', 0)
        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)

-        fragment_urls = []
+        fragments_to_download = []
        frag_index = 0
        for i, fragment in enumerate(fragments):
            frag_index += 1
@@ -48,49 +56,17 @@ class DashSegmentsFD(FragmentFD):
                assert fragment_base_url
                fragment_url = urljoin(fragment_base_url, fragment['path'])

-            if real_downloader:
-                fragment_urls.append(fragment_url)
-                continue
-
-            # In DASH, the first segment contains necessary headers to
-            # generate a valid MP4 file, so always abort for the first segment
-            fatal = i == 0 or not skip_unavailable_fragments
-            count = 0
-            while count <= fragment_retries:
-                try:
-                    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
-                    if not success:
-                        return False
-                    self._append_fragment(ctx, frag_content)
-                    break
-                except compat_urllib_error.HTTPError as err:
-                    # YouTube may often return 404 HTTP error for a fragment causing the
-                    # whole download to fail. However if the same fragment is immediately
-                    # retried with the same request data this usually succeeds (1-2 attempts
-                    # is usually enough) thus allowing to download the whole file successfully.
-                    # To be future-proof we will retry all fragments that fail with any
-                    # HTTP error.
-                    count += 1
-                    if count <= fragment_retries:
-                        self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                except DownloadError:
-                    # Don't retry fragment if error occurred during HTTP downloading
-                    # itself since it has own retry settings
-                    if not fatal:
-                        self.report_skip_fragment(frag_index)
-                        break
-                    raise
-
-            if count > fragment_retries:
-                if not fatal:
-                    self.report_skip_fragment(frag_index)
-                    continue
-                self.report_error('giving up after %s fragment retries' % fragment_retries)
-                return False
+            fragments_to_download.append({
+                'frag_index': frag_index,
+                'index': i,
+                'url': fragment_url,
+            })

        if real_downloader:
+            self.to_screen(
+                '[%s] Fragment downloads will be delegated to %s' % (self.FD_NAME, real_downloader.get_basename()))
            info_copy = info_dict.copy()
-            info_copy['url_list'] = fragment_urls
+            info_copy['fragments'] = fragments_to_download
            fd = real_downloader(self.ydl, self.params)
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
@@ -99,5 +75,104 @@ class DashSegmentsFD(FragmentFD):
            if not success:
                return False
        else:
+            def download_fragment(fragment):
+                i = fragment['index']
+                frag_index = fragment['frag_index']
+                fragment_url = fragment['url']
+
+                ctx['fragment_index'] = frag_index
+
+                # In DASH, the first segment contains necessary headers to
+                # generate a valid MP4 file, so always abort for the first segment
+                fatal = i == 0 or not skip_unavailable_fragments
+                count = 0
+                while count <= fragment_retries:
+                    try:
+                        success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
+                        if not success:
+                            return False, frag_index
+                        break
+                    except compat_urllib_error.HTTPError as err:
+                        # YouTube may often return 404 HTTP error for a fragment causing the
+                        # whole download to fail. However if the same fragment is immediately
+                        # retried with the same request data this usually succeeds (1-2 attempts
+                        # is usually enough) thus allowing to download the whole file successfully.
+                        # To be future-proof we will retry all fragments that fail with any
+                        # HTTP error.
+                        count += 1
+                        if count <= fragment_retries:
+                            self.report_retry_fragment(err, frag_index, count, fragment_retries)
+                    except DownloadError:
+                        # Don't retry fragment if error occurred during HTTP downloading
+                        # itself since it has own retry settings
+                        if not fatal:
+                            break
+                        raise
+
+                if count > fragment_retries:
+                    if not fatal:
+                        return False, frag_index
+                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                    return False, frag_index
+
+                return frag_content, frag_index
+
+            def append_fragment(frag_content, frag_index):
+                if frag_content:
+                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
+                    try:
+                        file, frag_sanitized = sanitize_open(fragment_filename, 'rb')
+                        ctx['fragment_filename_sanitized'] = frag_sanitized
+                        file.close()
+                        self._append_fragment(ctx, frag_content)
+                        return True
+                    except FileNotFoundError:
+                        if skip_unavailable_fragments:
+                            self.report_skip_fragment(frag_index)
+                            return True
+                        else:
+                            self.report_error(
+                                'fragment %s not found, unable to continue' % frag_index)
+                            return False
+                else:
+                    if skip_unavailable_fragments:
+                        self.report_skip_fragment(frag_index)
+                        return True
+                    else:
+                        self.report_error(
+                            'fragment %s not found, unable to continue' % frag_index)
+                        return False
+
+            max_workers = self.params.get('concurrent_fragment_downloads', 1)
+            if can_threaded_download and max_workers > 1:
+                self.report_warning('The download speed shown is only of one thread. This is a known issue')
+                with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
+                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments_to_download]
+                    # timeout must be 0 to return instantly
+                    done, not_done = concurrent.futures.wait(futures, timeout=0)
+                    try:
+                        while not_done:
+                            # Check every 1 second for KeyboardInterrupt
+                            freshly_done, not_done = concurrent.futures.wait(not_done, timeout=1)
+                            done |= freshly_done
+                    except KeyboardInterrupt:
+                        for future in not_done:
+                            future.cancel()
+                        # timeout must be none to cancel
+                        concurrent.futures.wait(not_done, timeout=None)
+                        raise KeyboardInterrupt
+                results = [future.result() for future in futures]
+
+                for frag_content, frag_index in results:
+                    result = append_fragment(frag_content, frag_index)
+                    if not result:
+                        return False
+            else:
+                for fragment in fragments_to_download:
+                    frag_content, frag_index = download_fragment(fragment)
+                    result = append_fragment(frag_content, frag_index)
+                    if not result:
+                        return False
+
            self._finish_frag_download(ctx)
        return True
@@ -24,7 +24,6 @@ from ..utils import (
    cli_bool_option,
    cli_configuration_args,
    encodeFilename,
-    error_to_compat_str,
    encodeArgument,
    handle_youtubedl_headers,
    check_executable,
@@ -82,11 +81,15 @@ class ExternalFD(FileDownloader):

    @property
    def exe(self):
-        return self.params.get('external_downloader')
+        return self.get_basename()

    @classmethod
    def available(cls, path=None):
-        return check_executable(path or cls.get_basename(), [cls.AVAILABLE_OPT])
+        path = check_executable(path or cls.get_basename(), [cls.AVAILABLE_OPT])
+        if path:
+            cls.exe = path
+            return path
+        return False

    @classmethod
    def supports(cls, info_dict):
@@ -108,7 +111,8 @@ class ExternalFD(FileDownloader):
    def _configuration_args(self, *args, **kwargs):
        return cli_configuration_args(
            self.params.get('external_downloader_args'),
-            self.get_basename(), *args, **kwargs)
+            [self.get_basename(), 'default'],
+            *args, **kwargs)

    def _call_downloader(self, tmpfilename, info_dict):
        """ Either overwrite this or implement _make_cmd """
@@ -116,24 +120,43 @@ class ExternalFD(FileDownloader):

        self._debug_cmd(cmd)

-        p = subprocess.Popen(
-            cmd, stderr=subprocess.PIPE)
-        _, stderr = process_communicate_or_kill(p)
-        if p.returncode != 0:
-            self.to_stderr(stderr.decode('utf-8', 'replace'))
+        if 'fragments' in info_dict:
+            fragment_retries = self.params.get('fragment_retries', 0)
+            skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+
+            count = 0
+            while count <= fragment_retries:
+                p = subprocess.Popen(
+                    cmd, stderr=subprocess.PIPE)
+                _, stderr = process_communicate_or_kill(p)
+                if p.returncode == 0:
+                    break
+                # TODO: Decide whether to retry based on error code
+                # https://aria2.github.io/manual/en/html/aria2c.html#exit-status
+                self.to_stderr(stderr.decode('utf-8', 'replace'))
+                count += 1
+                if count <= fragment_retries:
+                    self.to_screen(
+                        '[%s] Got error. Retrying fragments (attempt %d of %s)...'
+                        % (self.get_basename(), count, self.format_retries(fragment_retries)))
+            if count > fragment_retries:
+                if not skip_unavailable_fragments:
+                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                    return -1

-        if 'url_list' in info_dict:
-            file_list = []
-            for [i, url] in enumerate(info_dict['url_list']):
-                tmpsegmentname = '%s_%s.frag' % (tmpfilename, i)
-                file_list.append(tmpsegmentname)
-            key_list = info_dict.get('key_list')
-            decrypt_info = None
            dest, _ = sanitize_open(tmpfilename, 'wb')
-            for i, file in enumerate(file_list):
-                src, _ = sanitize_open(file, 'rb')
-                if key_list:
-                    decrypt_info = next((x for x in key_list if x['INDEX'] == i), decrypt_info)
+            for frag_index, fragment in enumerate(info_dict['fragments']):
+                fragment_filename = '%s-Frag%d' % (tmpfilename, frag_index)
+                try:
+                    src, _ = sanitize_open(fragment_filename, 'rb')
+                except IOError:
+                    if skip_unavailable_fragments and frag_index > 1:
+                        self.to_screen('[%s] Skipping fragment %d ...' % (self.get_basename(), frag_index))
+                        continue
+                    self.report_error('Unable to open fragment %d' % frag_index)
+                    return -1
+                decrypt_info = fragment.get('decrypt_info')
+                if decrypt_info:
                    if decrypt_info['METHOD'] == 'AES-128':
                        iv = decrypt_info.get('IV')
                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
@@ -149,19 +172,16 @@ class ExternalFD(FileDownloader):
                    fragment_data = src.read()
                    dest.write(fragment_data)
                src.close()
+                if not self.params.get('keep_fragments', False):
+                    os.remove(encodeFilename(fragment_filename))
            dest.close()
-            if not self.params.get('keep_fragments', False):
-                for file_path in file_list:
-                    try:
-                        os.remove(file_path)
-                    except OSError as ose:
-                        self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
-                try:
-                    file_path = '%s.frag.urls' % tmpfilename
-                    os.remove(file_path)
-                except OSError as ose:
-                    self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
-
+            os.remove(encodeFilename('%s.frag.urls' % tmpfilename))
+        else:
+            p = subprocess.Popen(
+                cmd, stderr=subprocess.PIPE)
+            _, stderr = process_communicate_or_kill(p)
+            if p.returncode != 0:
+                self.to_stderr(stderr.decode('utf-8', 'replace'))
        return p.returncode

    def _prepare_url(self, info_dict, url):
@@ -243,17 +263,24 @@ class WgetFD(ExternalFD):

 class Aria2cFD(ExternalFD):
    AVAILABLE_OPT = '-v'
-    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'frag_urls')
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'dash_frag_urls', 'm3u8_frag_urls')
+
+    @staticmethod
+    def supports_manifest(manifest):
+        UNSUPPORTED_FEATURES = [
+            r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [1]
+            # 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
+        ]
+        check_results = (not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES)
+        return all(check_results)

    def _make_cmd(self, tmpfilename, info_dict):
-        cmd = [self.exe, '-c']
-        dn = os.path.dirname(tmpfilename)
-        if 'url_list' not in info_dict:
-            cmd += ['--out', os.path.basename(tmpfilename)]
-        verbose_level_args = ['--console-log-level=warn', '--summary-interval=0']
-        cmd += self._configuration_args(['--file-allocation=none', '-x16', '-j16', '-s16'] + verbose_level_args)
-        if dn:
-            cmd += ['--dir', dn]
+        cmd = [self.exe, '-c',
+               '--console-log-level=warn', '--summary-interval=0', '--download-result=hide',
+               '--file-allocation=none', '-x16', '-j16', '-s16']
+        if 'fragments' in info_dict:
+            cmd += ['--allow-overwrite=true', '--allow-piece-length-change=true']
+
        if info_dict.get('http_headers') is not None:
            for key, val in info_dict['http_headers'].items():
                cmd += ['--header', '%s: %s' % (key, val)]
@@ -261,19 +288,25 @@ class Aria2cFD(ExternalFD):
        cmd += self._option('--all-proxy', 'proxy')
        cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
        cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
+        cmd += self._configuration_args()
+
+        dn = os.path.dirname(tmpfilename)
+        if dn:
+            cmd += ['--dir', dn]
+        if 'fragments' not in info_dict:
+            cmd += ['--out', os.path.basename(tmpfilename)]
        cmd += ['--auto-file-renaming=false']
-        if 'url_list' in info_dict:
-            cmd += verbose_level_args
-            cmd += ['--uri-selector', 'inorder', '--download-result=hide']
+
+        if 'fragments' in info_dict:
+            cmd += ['--file-allocation=none', '--uri-selector=inorder']
            url_list_file = '%s.frag.urls' % tmpfilename
            url_list = []
-            for [i, url] in enumerate(info_dict['url_list']):
-                tmpsegmentname = '%s_%s.frag' % (os.path.basename(tmpfilename), i)
-                url_list.append('%s\n\tout=%s' % (url, tmpsegmentname))
+            for frag_index, fragment in enumerate(info_dict['fragments']):
+                fragment_filename = '%s-Frag%d' % (os.path.basename(tmpfilename), frag_index)
+                url_list.append('%s\n\tout=%s' % (fragment['url'], fragment_filename))
            stream, _ = sanitize_open(url_list_file, 'wb')
            stream.write('\n'.join(url_list).encode('utf-8'))
            stream.close()
-
            cmd += ['-i', url_list_file]
        else:
            cmd += ['--', info_dict['url']]
@@ -281,9 +314,11 @@ class Aria2cFD(ExternalFD):


 class HttpieFD(ExternalFD):
+    AVAILABLE_OPT = '--version'
+
    @classmethod
    def available(cls, path=None):
-        return check_executable(path or 'http', ['--version'])
+        return ExternalFD.available(cls, path or 'http')

    def _make_cmd(self, tmpfilename, info_dict):
        cmd = ['http', '--download', '--output', tmpfilename, info_dict['url']]
@@ -298,7 +333,8 @@ class FFmpegFD(ExternalFD):
    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')

    @classmethod
-    def available(cls, path=None):  # path is ignored for ffmpeg
+    def available(cls, path=None):
+        # TODO: Fix path for ffmpeg
        return FFmpegPostProcessor().available

    def _call_downloader(self, tmpfilename, info_dict):
@@ -455,4 +491,4 @@ def get_external_downloader(external_downloader):
        downloader . """
    # Drop .exe extension on Windows
    bn = os.path.splitext(os.path.basename(external_downloader))[0]
-    return _BY_NAME[bn]
+    return _BY_NAME.get(bn)
@@ -7,6 +7,11 @@ try:
    can_decrypt_frag = True
 except ImportError:
    can_decrypt_frag = False
+try:
+    import concurrent.futures
+    can_threaded_download = True
+except ImportError:
+    can_threaded_download = False

 from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
@@ -19,12 +24,17 @@ from ..compat import (
 )
 from ..utils import (
    parse_m3u8_attributes,
+    sanitize_open,
    update_url_query,
 )


 class HlsFD(FragmentFD):
-    """ A limited implementation that does not require ffmpeg """
+    """
+    Download segments in a m3u8 manifest. External downloaders can take over
+    the fragment downloads by supporting the 'm3u8_frag_urls' protocol and
+    re-defining 'supports_manifest' function
+    """

    FD_NAME = 'hlsnative'

@@ -53,12 +63,15 @@ class HlsFD(FragmentFD):
            UNSUPPORTED_FEATURES += [
                r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
            ]
-        check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
-        is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
-        check_results.append(with_crypto or not is_aes128_enc)
-        check_results.append(not (is_aes128_enc and r'#EXT-X-BYTERANGE' in manifest))
-        check_results.append(not info_dict.get('is_live'))
-        return all(check_results)
+
+        def check_results():
+            yield not info_dict.get('is_live')
+            is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
+            yield with_crypto or not is_aes128_enc
+            yield not (is_aes128_enc and r'#EXT-X-BYTERANGE' in manifest)
+            for feature in UNSUPPORTED_FEATURES:
+                yield not re.search(feature, manifest)
+        return all(check_results())

    def real_download(self, filename, info_dict):
        man_url = info_dict['url']
@@ -70,20 +83,24 @@ class HlsFD(FragmentFD):

        if not self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')):
            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
-                self.report_error('pycryptodome not found. Please install it.')
+                self.report_error('pycryptodome not found. Please install')
                return False
            if self.can_download(s, info_dict, with_crypto=True):
-                self.report_warning('pycryptodome is needed to download this file with hlsnative')
-            self.report_warning(
-                'hlsnative has detected features it does not support, '
-                'extraction will be delegated to ffmpeg')
+                self.report_warning('pycryptodome is needed to download this file natively')
            fd = FFmpegFD(self.ydl, self.params)
+            self.report_warning(
+                '%s detected unsupported features; extraction will be delegated to %s' % (self.FD_NAME, fd.get_basename()))
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
            #     fd.add_progress_hook(ph)
            return fd.real_download(filename, info_dict)

-        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+        real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
+        if real_downloader and not real_downloader.supports_manifest(s):
+            real_downloader = None
+        if real_downloader:
+            self.to_screen(
+                '[%s] Fragment downloads will be delegated to %s' % (self.FD_NAME, real_downloader.get_basename()))

        def is_ad_fragment_start(s):
            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
@@ -93,7 +110,7 @@ class HlsFD(FragmentFD):
            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))

-        fragment_urls = []
+        fragments = []

        media_frags = 0
        ad_frags = 0
@@ -136,14 +153,12 @@ class HlsFD(FragmentFD):
        i = 0
        media_sequence = 0
        decrypt_info = {'METHOD': 'NONE'}
-        key_list = []
        byte_range = {}
        discontinuity_count = 0
        frag_index = 0
        ad_frag_next = False
        for line in s.splitlines():
            line = line.strip()
-            download_frag = False
            if line:
                if not line.startswith('#'):
                    if format_index and discontinuity_count != format_index:
@@ -160,17 +175,20 @@ class HlsFD(FragmentFD):
                    if extra_query:
                        frag_url = update_url_query(frag_url, extra_query)

-                    if real_downloader:
-                        fragment_urls.append(frag_url)
-                        continue
-                    download_frag = True
+                    fragments.append({
+                        'frag_index': frag_index,
+                        'url': frag_url,
+                        'decrypt_info': decrypt_info,
+                        'byte_range': byte_range,
+                        'media_sequence': media_sequence,
+                    })

                elif line.startswith('#EXT-X-MAP'):
                    if format_index and discontinuity_count != format_index:
                        continue
                    if frag_index > 0:
                        self.report_error(
-                            'initialization fragment found after media fragments, unable to download')
+                            'Initialization fragment found after media fragments, unable to download')
                        return False
                    frag_index += 1
                    map_info = parse_m3u8_attributes(line[11:])
@@ -180,9 +198,14 @@ class HlsFD(FragmentFD):
                        else compat_urlparse.urljoin(man_url, map_info.get('URI')))
                    if extra_query:
                        frag_url = update_url_query(frag_url, extra_query)
-                    if real_downloader:
-                        fragment_urls.append(frag_url)
-                        continue
+
+                    fragments.append({
+                        'frag_index': frag_index,
+                        'url': frag_url,
+                        'decrypt_info': decrypt_info,
+                        'byte_range': byte_range,
+                        'media_sequence': media_sequence
+                    })

                    if map_info.get('BYTERANGE'):
                        splitted_byte_range = map_info.get('BYTERANGE').split('@')
@@ -191,7 +214,6 @@ class HlsFD(FragmentFD):
                            'start': sub_range_start,
                            'end': sub_range_start + int(splitted_byte_range[0]),
                        }
-                    download_frag = True

                elif line.startswith('#EXT-X-KEY'):
                    decrypt_url = decrypt_info.get('URI')
@@ -206,9 +228,6 @@ class HlsFD(FragmentFD):
                            decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
                        if decrypt_url != decrypt_info['URI']:
                            decrypt_info['KEY'] = None
-                    key_data = decrypt_info.copy()
-                    key_data['INDEX'] = frag_index
-                    key_list.append(key_data)

                elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
                    media_sequence = int(line[22:])
@@ -225,58 +244,16 @@ class HlsFD(FragmentFD):
                    ad_frag_next = False
                elif line.startswith('#EXT-X-DISCONTINUITY'):
                    discontinuity_count += 1
+                i += 1
+                media_sequence += 1

-                if download_frag:
-                    count = 0
-                    headers = info_dict.get('http_headers', {})
-                    if byte_range:
-                        headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
-                    while count <= fragment_retries:
-                        try:
-                            success, frag_content = self._download_fragment(
-                                ctx, frag_url, info_dict, headers)
-                            if not success:
-                                return False
-                            break
-                        except compat_urllib_error.HTTPError as err:
-                            # Unavailable (possibly temporary) fragments may be served.
-                            # First we try to retry then either skip or abort.
-                            # See https://github.com/ytdl-org/youtube-dl/issues/10165,
-                            # https://github.com/ytdl-org/youtube-dl/issues/10448).
-                            count += 1
-                            if count <= fragment_retries:
-                                self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                    if count > fragment_retries:
-                        if skip_unavailable_fragments:
-                            i += 1
-                            media_sequence += 1
-                            self.report_skip_fragment(frag_index)
-                            continue
-                        self.report_error(
-                            'giving up after %s fragment retries' % fragment_retries)
-                        return False
-
-                    if decrypt_info['METHOD'] == 'AES-128':
-                        iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
-                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                        # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                        # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-                        # not what it decrypts to.
-                        if not test:
-                            frag_content = AES.new(
-                                decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-                    self._append_fragment(ctx, frag_content)
-                    # We only download the first fragment during the test
-                    if test:
-                        break
-                    i += 1
-                    media_sequence += 1
+        # We only download the first fragment during the test
+        if test:
+            fragments = [fragments[0] if fragments else None]

        if real_downloader:
            info_copy = info_dict.copy()
-            info_copy['url_list'] = fragment_urls
-            info_copy['key_list'] = key_list
+            info_copy['fragments'] = fragments
            fd = real_downloader(self.ydl, self.params)
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
@@ -285,5 +262,107 @@ class HlsFD(FragmentFD):
            if not success:
                return False
        else:
+            def download_fragment(fragment):
+                frag_index = fragment['frag_index']
+                frag_url = fragment['url']
+                decrypt_info = fragment['decrypt_info']
+                byte_range = fragment['byte_range']
+                media_sequence = fragment['media_sequence']
+
+                ctx['fragment_index'] = frag_index
+
+                count = 0
+                headers = info_dict.get('http_headers', {})
+                if byte_range:
+                    headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
+                while count <= fragment_retries:
+                    try:
+                        success, frag_content = self._download_fragment(
+                            ctx, frag_url, info_dict, headers)
+                        if not success:
+                            return False, frag_index
+                        break
+                    except compat_urllib_error.HTTPError as err:
+                        # Unavailable (possibly temporary) fragments may be served.
+                        # First we try to retry then either skip or abort.
+                        # See https://github.com/ytdl-org/youtube-dl/issues/10165,
+                        # https://github.com/ytdl-org/youtube-dl/issues/10448).
+                        count += 1
+                        if count <= fragment_retries:
+                            self.report_retry_fragment(err, frag_index, count, fragment_retries)
+                if count > fragment_retries:
+                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                    return False, frag_index
+
+                if decrypt_info['METHOD'] == 'AES-128':
+                    iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
+                    decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                        self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                    # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+                    # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+                    # not what it decrypts to.
+                    if not test:
+                        frag_content = AES.new(
+                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+
+                return frag_content, frag_index
+
+            def append_fragment(frag_content, frag_index):
+                if frag_content:
+                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
+                    try:
+                        file, frag_sanitized = sanitize_open(fragment_filename, 'rb')
+                        ctx['fragment_filename_sanitized'] = frag_sanitized
+                        file.close()
+                        self._append_fragment(ctx, frag_content)
+                        return True
+                    except FileNotFoundError:
+                        if skip_unavailable_fragments:
+                            self.report_skip_fragment(frag_index)
+                            return True
+                        else:
+                            self.report_error(
+                                'fragment %s not found, unable to continue' % frag_index)
+                            return False
+                else:
+                    if skip_unavailable_fragments:
+                        self.report_skip_fragment(frag_index)
+                        return True
+                    else:
+                        self.report_error(
+                            'fragment %s not found, unable to continue' % frag_index)
+                        return False
+
+            max_workers = self.params.get('concurrent_fragment_downloads', 1)
+            if can_threaded_download and max_workers > 1:
+                self.report_warning('The download speed shown is only of one thread. This is a known issue')
+                with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
+                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments]
+                    # timeout must be 0 to return instantly
+                    done, not_done = concurrent.futures.wait(futures, timeout=0)
+                    try:
+                        while not_done:
+                            # Check every 1 second for KeyboardInterrupt
+                            freshly_done, not_done = concurrent.futures.wait(not_done, timeout=1)
+                            done |= freshly_done
+                    except KeyboardInterrupt:
+                        for future in not_done:
+                            future.cancel()
+                        # timeout must be none to cancel
+                        concurrent.futures.wait(not_done, timeout=None)
+                        raise KeyboardInterrupt
+                results = [future.result() for future in futures]
+
+                for frag_content, frag_index in results:
+                    result = append_fragment(frag_content, frag_index)
+                    if not result:
+                        return False
+            else:
+                for fragment in fragments:
+                    frag_content, frag_index = download_fragment(fragment)
+                    result = append_fragment(frag_content, frag_index)
+                    if not result:
+                        return False
+
            self._finish_frag_download(ctx)
        return True
@@ -117,7 +117,7 @@ class RtmpFD(FileDownloader):

        # Check for rtmpdump first
        if not check_executable('rtmpdump', ['-h']):
-            self.report_error('RTMP download detected but "rtmpdump" could not be run. Please install it.')
+            self.report_error('RTMP download detected but "rtmpdump" could not be run. Please install')
            return False

        # Download using rtmpdump. rtmpdump returns exit code 2 when
@@ -24,7 +24,7 @@ class RtspFD(FileDownloader):
            args = [
                'mpv', '-really-quiet', '--vo=null', '--stream-dump=' + tmpfilename, url]
        else:
-            self.report_error('MMS or RTSP download detected but neither "mplayer" nor "mpv" could be run. Please install any.')
+            self.report_error('MMS or RTSP download detected but neither "mplayer" nor "mpv" could be run. Please install one')
            return False

        self._debug_cmd(args)
@@ -79,8 +79,7 @@ class YoutubeLiveChatReplayFD(FragmentFD):

        self._prepare_and_start_frag_download(ctx)

-        success, raw_fragment = dl_fragment(
-            'https://www.youtube.com/watch?v={}'.format(video_id))
+        success, raw_fragment = dl_fragment(info_dict['url'])
        if not success:
            return False
        try:
@@ -65,15 +65,35 @@ class AMCNetworksIE(ThePlatformIE):
    def _real_extract(self, url):
        site, display_id = re.match(self._VALID_URL, url).groups()
        requestor_id = self._REQUESTOR_ID_MAP[site]
-        properties = self._download_json(
-            'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/%s/url/%s' % (requestor_id.lower(), display_id),
-            display_id)['data']['properties']
+        page_data = self._download_json(
+            'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/%s/url/%s'
+            % (requestor_id.lower(), display_id), display_id)['data']
+        properties = page_data.get('properties') or {}
        query = {
            'mbr': 'true',
            'manifest': 'm3u',
        }
-        tp_path = 'M_UwQC/media/' + properties['videoPid']
-        media_url = 'https://link.theplatform.com/s/' + tp_path
+
+        video_player_count = 0
+        try:
+            for v in page_data['children']:
+                if v.get('type') == 'video-player':
+                    releasePid = v['properties']['currentVideo']['meta']['releasePid']
+                    tp_path = 'M_UwQC/' + releasePid
+                    media_url = 'https://link.theplatform.com/s/' + tp_path
+                    video_player_count += 1
+        except KeyError:
+            pass
+        if video_player_count > 1:
+            self.report_warning(
+                'The JSON data has %d video players. Only one will be extracted' % video_player_count)
+
+        # Fall back to videoPid if releasePid not found.
+        # TODO: Fall back to videoPid if releasePid manifest uses DRM.
+        if not video_player_count:
+            tp_path = 'M_UwQC/media/' + properties['videoPid']
+            media_url = 'https://link.theplatform.com/s/' + tp_path
+
        theplatform_metadata = self._download_theplatform_metadata(tp_path, display_id)
        info = self._parse_theplatform_metadata(theplatform_metadata)
        video_id = theplatform_metadata['pid']
@@ -90,30 +110,41 @@ class AMCNetworksIE(ThePlatformIE):
        formats, subtitles = self._extract_theplatform_smil(
            media_url, video_id)
        self._sort_formats(formats)
+
+        thumbnails = []
+        thumbnail_urls = [properties.get('imageDesktop')]
+        if 'thumbnail' in info:
+            thumbnail_urls.append(info.pop('thumbnail'))
+        for thumbnail_url in thumbnail_urls:
+            if not thumbnail_url:
+                continue
+            mobj = re.search(r'(\d+)x(\d+)', thumbnail_url)
+            thumbnails.append({
+                'url': thumbnail_url,
+                'width': int(mobj.group(1)) if mobj else None,
+                'height': int(mobj.group(2)) if mobj else None,
+            })
+
        info.update({
+            'age_limit': parse_age_limit(rating),
+            'formats': formats,
            'id': video_id,
            'subtitles': subtitles,
-            'formats': formats,
-            'age_limit': parse_age_limit(parse_age_limit(rating)),
+            'thumbnails': thumbnails,
        })
        ns_keys = theplatform_metadata.get('$xmlns', {}).keys()
        if ns_keys:
            ns = list(ns_keys)[0]
-            series = theplatform_metadata.get(ns + '$show')
-            season_number = int_or_none(
-                theplatform_metadata.get(ns + '$season'))
-            episode = theplatform_metadata.get(ns + '$episodeTitle')
+            episode = theplatform_metadata.get(ns + '$episodeTitle') or None
            episode_number = int_or_none(
                theplatform_metadata.get(ns + '$episode'))
-            if season_number:
-                title = 'Season %d - %s' % (season_number, title)
-            if series:
-                title = '%s - %s' % (series, title)
+            season_number = int_or_none(
+                theplatform_metadata.get(ns + '$season'))
+            series = theplatform_metadata.get(ns + '$show') or None
            info.update({
-                'title': title,
-                'series': series,
-                'season_number': season_number,
                'episode': episode,
                'episode_number': episode_number,
+                'season_number': season_number,
+                'series': series,
            })
        return info
@@ -42,6 +42,7 @@ class ApplePodcastsIE(InfoExtractor):
        ember_data = self._parse_json(self._search_regex(
            r'id="shoebox-ember-data-store"[^>]*>\s*({.+?})\s*<',
            webpage, 'ember data'), episode_id)
+        ember_data = ember_data.get(episode_id) or ember_data
        episode = ember_data['data']['attributes']
        description = episode.get('description') or {}

@@ -272,7 +272,8 @@ class ARDMediathekIE(ARDMediathekBaseIE):
        else:  # request JSON file
            if not document_id:
                video_id = self._search_regex(
-                    r'/play/(?:config|media)/(\d+)', webpage, 'media id')
+                    (r'/play/(?:config|media|sola)/(\d+)', r'contentId["\']\s*:\s*(\d+)'),
+                    webpage, 'media id', default=None)
            info = self._extract_media_info(
                'http://www.ardmediathek.de/play/media/%s' % video_id,
                webpage, video_id)
@@ -0,0 +1,101 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_parse_qs,
+    compat_urllib_parse_urlparse,
+)
+from ..utils import (
+    float_or_none,
+    int_or_none,
+    parse_iso8601,
+    remove_start,
+)
+
+
+class ArnesIE(InfoExtractor):
+    IE_NAME = 'video.arnes.si'
+    IE_DESC = 'Arnes Video'
+    _VALID_URL = r'https?://video\.arnes\.si/(?:[a-z]{2}/)?(?:watch|embed|api/(?:asset|public/video))/(?P<id>[0-9a-zA-Z]{12})'
+    _TESTS = [{
+        'url': 'https://video.arnes.si/watch/a1qrWTOQfVoU?t=10',
+        'md5': '4d0f4d0a03571b33e1efac25fd4a065d',
+        'info_dict': {
+            'id': 'a1qrWTOQfVoU',
+            'ext': 'mp4',
+            'title': 'Linearna neodvisnost, definicija',
+            'description': 'Linearna neodvisnost, definicija',
+            'license': 'PRIVATE',
+            'creator': 'Polona Oblak',
+            'timestamp': 1585063725,
+            'upload_date': '20200324',
+            'channel': 'Polona Oblak',
+            'channel_id': 'q6pc04hw24cj',
+            'channel_url': 'https://video.arnes.si/?channel=q6pc04hw24cj',
+            'duration': 596.75,
+            'view_count': int,
+            'tags': ['linearna_algebra'],
+            'start_time': 10,
+        }
+    }, {
+        'url': 'https://video.arnes.si/api/asset/s1YjnV7hadlC/play.mp4',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.arnes.si/embed/s1YjnV7hadlC',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.arnes.si/en/watch/s1YjnV7hadlC',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.arnes.si/embed/s1YjnV7hadlC?t=123&hideRelated=1',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.arnes.si/api/public/video/s1YjnV7hadlC',
+        'only_matching': True,
+    }]
+    _BASE_URL = 'https://video.arnes.si'
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+
+        video = self._download_json(
+            self._BASE_URL + '/api/public/video/' + video_id, video_id)['data']
+        title = video['title']
+
+        formats = []
+        for media in (video.get('media') or []):
+            media_url = media.get('url')
+            if not media_url:
+                continue
+            formats.append({
+                'url': self._BASE_URL + media_url,
+                'format_id': remove_start(media.get('format'), 'FORMAT_'),
+                'format_note': media.get('formatTranslation'),
+                'width': int_or_none(media.get('width')),
+                'height': int_or_none(media.get('height')),
+            })
+        self._sort_formats(formats)
+
+        channel = video.get('channel') or {}
+        channel_id = channel.get('url')
+        thumbnail = video.get('thumbnailUrl')
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'thumbnail': self._BASE_URL + thumbnail,
+            'description': video.get('description'),
+            'license': video.get('license'),
+            'creator': video.get('author'),
+            'timestamp': parse_iso8601(video.get('creationTime')),
+            'channel': channel.get('name'),
+            'channel_id': channel_id,
+            'channel_url': self._BASE_URL + '/?channel=' + channel_id if channel_id else None,
+            'duration': float_or_none(video.get('duration'), 1000),
+            'view_count': int_or_none(video.get('views')),
+            'tags': video.get('hashtags'),
+            'start_time': int_or_none(compat_parse_qs(
+                compat_urllib_parse_urlparse(url).query).get('t', [None])[0]),
+        }
@@ -49,6 +49,7 @@ class BandcampIE(InfoExtractor):
            'uploader': 'Ben Prunty',
            'timestamp': 1396508491,
            'upload_date': '20140403',
+            'release_timestamp': 1396483200,
            'release_date': '20140403',
            'duration': 260.877,
            'track': 'Lanius (Battle)',
@@ -69,6 +70,7 @@ class BandcampIE(InfoExtractor):
            'uploader': 'Mastodon',
            'timestamp': 1322005399,
            'upload_date': '20111122',
+            'release_timestamp': 1076112000,
            'release_date': '20040207',
            'duration': 120.79,
            'track': 'Hail to Fire',
@@ -197,7 +199,7 @@ class BandcampIE(InfoExtractor):
            'thumbnail': thumbnail,
            'uploader': artist,
            'timestamp': timestamp,
-            'release_date': unified_strdate(tralbum.get('album_release_date')),
+            'release_timestamp': unified_timestamp(tralbum.get('album_release_date')),
            'duration': duration,
            'track': track,
            'track_number': track_number,
@@ -1,17 +1,22 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import functools
 import itertools
+import json
 import re

 from .common import InfoExtractor
 from ..compat import (
    compat_etree_Element,
    compat_HTTPError,
+    compat_parse_qs,
+    compat_urllib_parse_urlparse,
    compat_urlparse,
 )
 from ..utils import (
    ExtractorError,
+    OnDemandPagedList,
    clean_html,
    dict_get,
    float_or_none,
@@ -811,7 +816,7 @@ class BBCIE(BBCCoUkIE):

    @classmethod
    def suitable(cls, url):
-        EXCLUDE_IE = (BBCCoUkIE, BBCCoUkArticleIE, BBCCoUkIPlayerPlaylistIE, BBCCoUkPlaylistIE)
+        EXCLUDE_IE = (BBCCoUkIE, BBCCoUkArticleIE, BBCCoUkIPlayerEpisodesIE, BBCCoUkIPlayerGroupIE, BBCCoUkPlaylistIE)
        return (False if any(ie.suitable(url) for ie in EXCLUDE_IE)
                else super(BBCIE, cls).suitable(url))

@@ -1338,21 +1343,149 @@ class BBCCoUkPlaylistBaseIE(InfoExtractor):
            playlist_id, title, description)


-class BBCCoUkIPlayerPlaylistIE(BBCCoUkPlaylistBaseIE):
-    IE_NAME = 'bbc.co.uk:iplayer:playlist'
-    _VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/iplayer/(?:episodes|group)/(?P<id>%s)' % BBCCoUkIE._ID_REGEX
-    _URL_TEMPLATE = 'http://www.bbc.co.uk/iplayer/episode/%s'
-    _VIDEO_ID_TEMPLATE = r'data-ip-id=["\'](%s)'
+class BBCCoUkIPlayerPlaylistBaseIE(InfoExtractor):
+    _VALID_URL_TMPL = r'https?://(?:www\.)?bbc\.co\.uk/iplayer/%%s/(?P<id>%s)' % BBCCoUkIE._ID_REGEX
+
+    @staticmethod
+    def _get_default(episode, key, default_key='default'):
+        return try_get(episode, lambda x: x[key][default_key])
+
+    def _get_description(self, data):
+        synopsis = data.get(self._DESCRIPTION_KEY) or {}
+        return dict_get(synopsis, ('large', 'medium', 'small'))
+
+    def _fetch_page(self, programme_id, per_page, series_id, page):
+        elements = self._get_elements(self._call_api(
+            programme_id, per_page, page + 1, series_id))
+        for element in elements:
+            episode = self._get_episode(element)
+            episode_id = episode.get('id')
+            if not episode_id:
+                continue
+            thumbnail = None
+            image = self._get_episode_image(episode)
+            if image:
+                thumbnail = image.replace('{recipe}', 'raw')
+            category = self._get_default(episode, 'labels', 'category')
+            yield {
+                '_type': 'url',
+                'id': episode_id,
+                'title': self._get_episode_field(episode, 'subtitle'),
+                'url': 'https://www.bbc.co.uk/iplayer/episode/' + episode_id,
+                'thumbnail': thumbnail,
+                'description': self._get_description(episode),
+                'categories': [category] if category else None,
+                'series': self._get_episode_field(episode, 'title'),
+                'ie_key': BBCCoUkIE.ie_key(),
+            }
+
+    def _real_extract(self, url):
+        pid = self._match_id(url)
+        qs = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
+        series_id = qs.get('seriesId', [None])[0]
+        page = qs.get('page', [None])[0]
+        per_page = 36 if page else self._PAGE_SIZE
+        fetch_page = functools.partial(self._fetch_page, pid, per_page, series_id)
+        entries = fetch_page(int(page) - 1) if page else OnDemandPagedList(fetch_page, self._PAGE_SIZE)
+        playlist_data = self._get_playlist_data(self._call_api(pid, 1))
+        return self.playlist_result(
+            entries, pid, self._get_playlist_title(playlist_data),
+            self._get_description(playlist_data))
+
+
+class BBCCoUkIPlayerEpisodesIE(BBCCoUkIPlayerPlaylistBaseIE):
+    IE_NAME = 'bbc.co.uk:iplayer:episodes'
+    _VALID_URL = BBCCoUkIPlayerPlaylistBaseIE._VALID_URL_TMPL % 'episodes'
    _TESTS = [{
        'url': 'http://www.bbc.co.uk/iplayer/episodes/b05rcz9v',
        'info_dict': {
            'id': 'b05rcz9v',
            'title': 'The Disappearance',
-            'description': 'French thriller serial about a missing teenager.',
+            'description': 'md5:58eb101aee3116bad4da05f91179c0cb',
        },
-        'playlist_mincount': 6,
-        'skip': 'This programme is not currently available on BBC iPlayer',
+        'playlist_mincount': 8,
    }, {
+        # all seasons
+        'url': 'https://www.bbc.co.uk/iplayer/episodes/b094m5t9/doctor-foster',
+        'info_dict': {
+            'id': 'b094m5t9',
+            'title': 'Doctor Foster',
+            'description': 'md5:5aa9195fad900e8e14b52acd765a9fd6',
+        },
+        'playlist_mincount': 10,
+    }, {
+        # explicit season
+        'url': 'https://www.bbc.co.uk/iplayer/episodes/b094m5t9/doctor-foster?seriesId=b094m6nv',
+        'info_dict': {
+            'id': 'b094m5t9',
+            'title': 'Doctor Foster',
+            'description': 'md5:5aa9195fad900e8e14b52acd765a9fd6',
+        },
+        'playlist_mincount': 5,
+    }, {
+        # all pages
+        'url': 'https://www.bbc.co.uk/iplayer/episodes/m0004c4v/beechgrove',
+        'info_dict': {
+            'id': 'm0004c4v',
+            'title': 'Beechgrove',
+            'description': 'Gardening show that celebrates Scottish horticulture and growing conditions.',
+        },
+        'playlist_mincount': 37,
+    }, {
+        # explicit page
+        'url': 'https://www.bbc.co.uk/iplayer/episodes/m0004c4v/beechgrove?page=2',
+        'info_dict': {
+            'id': 'm0004c4v',
+            'title': 'Beechgrove',
+            'description': 'Gardening show that celebrates Scottish horticulture and growing conditions.',
+        },
+        'playlist_mincount': 1,
+    }]
+    _PAGE_SIZE = 100
+    _DESCRIPTION_KEY = 'synopsis'
+
+    def _get_episode_image(self, episode):
+        return self._get_default(episode, 'image')
+
+    def _get_episode_field(self, episode, field):
+        return self._get_default(episode, field)
+
+    @staticmethod
+    def _get_elements(data):
+        return data['entities']['results']
+
+    @staticmethod
+    def _get_episode(element):
+        return element.get('episode') or {}
+
+    def _call_api(self, pid, per_page, page=1, series_id=None):
+        variables = {
+            'id': pid,
+            'page': page,
+            'perPage': per_page,
+        }
+        if series_id:
+            variables['sliceId'] = series_id
+        return self._download_json(
+            'https://graph.ibl.api.bbc.co.uk/', pid, headers={
+                'Content-Type': 'application/json'
+            }, data=json.dumps({
+                'id': '5692d93d5aac8d796a0305e895e61551',
+                'variables': variables,
+            }).encode('utf-8'))['data']['programme']
+
+    @staticmethod
+    def _get_playlist_data(data):
+        return data
+
+    def _get_playlist_title(self, data):
+        return self._get_default(data, 'title')
+
+
+class BBCCoUkIPlayerGroupIE(BBCCoUkIPlayerPlaylistBaseIE):
+    IE_NAME = 'bbc.co.uk:iplayer:group'
+    _VALID_URL = BBCCoUkIPlayerPlaylistBaseIE._VALID_URL_TMPL % 'group'
+    _TESTS = [{
        # Available for over a year unlike 30 days for most other programmes
        'url': 'http://www.bbc.co.uk/iplayer/group/p02tcc32',
        'info_dict': {
@@ -1361,14 +1494,56 @@ class BBCCoUkIPlayerPlaylistIE(BBCCoUkPlaylistBaseIE):
            'description': 'md5:683e901041b2fe9ba596f2ab04c4dbe7',
        },
        'playlist_mincount': 10,
+    }, {
+        # all pages
+        'url': 'https://www.bbc.co.uk/iplayer/group/p081d7j7',
+        'info_dict': {
+            'id': 'p081d7j7',
+            'title': 'Music in Scotland',
+            'description': 'Perfomances in Scotland and programmes featuring Scottish acts.',
+        },
+        'playlist_mincount': 47,
+    }, {
+        # explicit page
+        'url': 'https://www.bbc.co.uk/iplayer/group/p081d7j7?page=2',
+        'info_dict': {
+            'id': 'p081d7j7',
+            'title': 'Music in Scotland',
+            'description': 'Perfomances in Scotland and programmes featuring Scottish acts.',
+        },
+        'playlist_mincount': 11,
    }]
+    _PAGE_SIZE = 200
+    _DESCRIPTION_KEY = 'synopses'

-    def _extract_title_and_description(self, webpage):
-        title = self._search_regex(r'<h1>([^<]+)</h1>', webpage, 'title', fatal=False)
-        description = self._search_regex(
-            r'<p[^>]+class=(["\'])subtitle\1[^>]*>(?P<value>[^<]+)</p>',
-            webpage, 'description', fatal=False, group='value')
-        return title, description
+    def _get_episode_image(self, episode):
+        return self._get_default(episode, 'images', 'standard')
+
+    def _get_episode_field(self, episode, field):
+        return episode.get(field)
+
+    @staticmethod
+    def _get_elements(data):
+        return data['elements']
+
+    @staticmethod
+    def _get_episode(element):
+        return element
+
+    def _call_api(self, pid, per_page, page=1, series_id=None):
+        return self._download_json(
+            'http://ibl.api.bbc.co.uk/ibl/v1/groups/%s/episodes' % pid,
+            pid, query={
+                'page': page,
+                'per_page': per_page,
+            })['group_episodes']
+
+    @staticmethod
+    def _get_playlist_data(data):
+        return data['group']
+
+    def _get_playlist_title(self, data):
+        return data.get('title')


 class BBCCoUkPlaylistIE(BBCCoUkPlaylistBaseIE):
@@ -7,6 +7,7 @@ import re

 from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
+    compat_str,
    compat_parse_qs,
    compat_urlparse,
 )
@@ -15,6 +16,7 @@ from ..utils import (
    int_or_none,
    float_or_none,
    parse_iso8601,
+    try_get,
    smuggle_url,
    str_or_none,
    strip_jsonp,
@@ -113,6 +115,14 @@ class BiliBiliIE(InfoExtractor):
        # new BV video id format
        'url': 'https://www.bilibili.com/video/BV1JE411F741',
        'only_matching': True,
+    }, {
+        # Anthology
+        'url': 'https://www.bilibili.com/video/BV1bK411W797',
+        'info_dict': {
+            'id': 'BV1bK411W797',
+            'title': '物语中的人物是如何吐槽自己的OP的'
+        },
+        'playlist_count': 17,
    }]

    _APP_KEY = 'iVGUTjsxvpLeuDCf'
@@ -138,15 +148,21 @@ class BiliBiliIE(InfoExtractor):
        anime_id = mobj.group('anime_id')
        page_id = mobj.group('page')
        webpage = self._download_webpage(url, video_id)
-        headers = {
-            'Referer': url,
-            'Accept': '*/*'
-        }
-        headers.update(self.geo_verification_headers())
+
+        # Bilibili anthologies are similar to playlists but all videos share the same video ID as the anthology itself.
+        # If the video has no page argument, check to see if it's an anthology
+        if page_id is None:
+            if not self._downloader.params.get('noplaylist'):
+                r = self._extract_anthology_entries(bv_id, video_id, webpage)
+                if r is not None:
+                    self.to_screen('Downloading anthology %s - add --no-playlist to just download video' % video_id)
+                    return r
+            else:
+                self.to_screen('Downloading just video %s because of --no-playlist' % video_id)

        if 'anime/' not in url:
            cid = self._search_regex(
-                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + str(page_id), webpage, 'cid',
+                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + compat_str(page_id), webpage, 'cid',
                default=None
            ) or self._search_regex(
                r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
@@ -160,8 +176,12 @@ class BiliBiliIE(InfoExtractor):
            if 'no_bangumi_tip' not in smuggled_data:
                self.to_screen('Downloading episode %s. To download all videos in anime %s, re-run yt-dlp with %s' % (
                    video_id, anime_id, compat_urlparse.urljoin(url, '//bangumi.bilibili.com/anime/%s' % anime_id)))
+            headers = {
+                'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
+                'Referer': url
+            }
+            headers.update(self.geo_verification_headers())

-            headers['Content-Type'] = 'application/x-www-form-urlencoded; charset=UTF-8'
            js = self._download_json(
                'http://bangumi.bilibili.com/web_api/get_source', video_id,
                data=urlencode_postdata({'episode_id': video_id}),
@@ -170,6 +190,12 @@ class BiliBiliIE(InfoExtractor):
                self._report_error(js)
            cid = js['result']['cid']

+        headers = {
+            'Accept': 'application/json',
+            'Referer': url
+        }
+        headers.update(self.geo_verification_headers())
+
        entries = []

        RENDITIONS = ('qn=80&quality=80&type=', 'quality=2&type=mp4')
@@ -219,7 +245,18 @@ class BiliBiliIE(InfoExtractor):
        title = self._html_search_regex(
            (r'<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
             r'(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
-            group='title') + ('_p' + str(page_id) if page_id is not None else '')
+            group='title')
+
+        # Get part title for anthologies
+        if page_id is not None:
+            # TODO: The json is already downloaded by _extract_anthology_entries. Don't redownload for each video
+            part_title = try_get(
+                self._download_json(
+                    "https://api.bilibili.com/x/player/pagelist?bvid=%s&jsonp=jsonp" % bv_id,
+                    video_id, note='Extracting videos in anthology'),
+                lambda x: x['data'][int(page_id) - 1]['part'])
+            title = part_title or title
+
        description = self._html_search_meta('description', webpage)
        timestamp = unified_timestamp(self._html_search_regex(
            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time',
@@ -229,7 +266,7 @@ class BiliBiliIE(InfoExtractor):

        # TODO 'view_count' requires deobfuscating Javascript
        info = {
-            'id': str(video_id) if page_id is None else '%s_p%s' % (video_id, page_id),
+            'id': compat_str(video_id) if page_id is None else '%s_p%s' % (video_id, page_id),
            'cid': cid,
            'title': title,
            'description': description,
@@ -239,7 +276,7 @@ class BiliBiliIE(InfoExtractor):
        }

        uploader_mobj = re.search(
-            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]*>(?P<name>[^<]+)',
+            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]*>\s*(?P<name>[^<]+?)\s*<',
            webpage)
        if uploader_mobj:
            info.update({
@@ -295,7 +332,7 @@ class BiliBiliIE(InfoExtractor):

            global_info = {
                '_type': 'multi_video',
-                'id': video_id,
+                'id': compat_str(video_id),
                'bv_id': bv_id,
                'title': title,
                'description': description,
@@ -307,6 +344,20 @@ class BiliBiliIE(InfoExtractor):

            return global_info

+    def _extract_anthology_entries(self, bv_id, video_id, webpage):
+        title = self._html_search_regex(
+            (r'<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
+             r'(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
+            group='title')
+        json_data = self._download_json(
+            "https://api.bilibili.com/x/player/pagelist?bvid=%s&jsonp=jsonp" % bv_id,
+            video_id, note='Extracting videos in anthology')
+
+        if len(json_data['data']) > 1:
+            return self.playlist_from_matches(
+                json_data['data'], bv_id, title, ie=BiliBiliIE.ie_key(),
+                getter=lambda entry: 'https://www.bilibili.com/video/%s?p=%d' % (bv_id, entry['page']))
+
    def _get_video_id_set(self, id, is_bv):
        query = {'bvid': id} if is_bv else {'aid': id}
        response = self._download_json(
@@ -501,7 +552,7 @@ class BiliBiliSearchIE(SearchInfoExtractor):

            videos = data['result']
            for video in videos:
-                e = self.url_result(video['arcurl'], 'BiliBili', str(video['aid']))
+                e = self.url_result(video['arcurl'], 'BiliBili', compat_str(video['aid']))
                entries.append(e)

            if(len(entries) >= n or len(videos) >= BiliBiliSearchIE.MAX_NUMBER_OF_RESULTS):
@@ -27,7 +27,7 @@ class CBSBaseIE(ThePlatformFeedIE):


 class CBSIE(CBSBaseIE):
-    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:(?:cbs\.com|paramountplus\.com)/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
+    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:(?:cbs|paramountplus)\.com/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'

    _TESTS = [{
        'url': 'https://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
@@ -53,7 +53,7 @@ class CBSIE(CBSBaseIE):
        'url': 'http://www.colbertlateshow.com/podcasts/dYSwjqPs_X1tvbV_P2FcPWRa_qT6akTC/in-the-bad-room-with-stephen/',
        'only_matching': True,
    }, {
-        'url': 'https://www.paramountplus.com/shows/star-trek-discovery/video/l5ANMH9wM7kxwV1qr4u1xn88XOhYMlZX/star-trek-discovery-the-vulcan-hello/',
+        'url': 'https://www.paramountplus.com/shows/all-rise/video/QmR1WhNkh1a_IrdHZrbcRklm176X_rVc/all-rise-space/',
        'only_matching': True,
    }]

@@ -1,38 +1,113 @@
 from __future__ import unicode_literals

-from .cbs import CBSBaseIE
+import re
+
+# from .cbs import CBSBaseIE
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    try_get,
+)


-class CBSSportsIE(CBSBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/(?:video|news)/(?P<id>[^/?#&]+)'
-
+# class CBSSportsEmbedIE(CBSBaseIE):
+class CBSSportsEmbedIE(InfoExtractor):
+    IE_NAME = 'cbssports:embed'
+    _VALID_URL = r'''(?ix)https?://(?:(?:www\.)?cbs|embed\.247)sports\.com/player/embed.+?
+        (?:
+            ids%3D(?P<id>[\da-f]{8}-(?:[\da-f]{4}-){3}[\da-f]{12})|
+            pcid%3D(?P<pcid>\d+)
+        )'''
    _TESTS = [{
-        'url': 'https://www.cbssports.com/nba/video/donovan-mitchell-flashes-star-potential-in-game-2-victory-over-thunder/',
-        'info_dict': {
-            'id': '1214315075735',
-            'ext': 'mp4',
-            'title': 'Donovan Mitchell flashes star potential in Game 2 victory over Thunder',
-            'description': 'md5:df6f48622612c2d6bd2e295ddef58def',
-            'timestamp': 1524111457,
-            'upload_date': '20180419',
-            'uploader': 'CBSI-NEW',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        }
+        'url': 'https://www.cbssports.com/player/embed/?args=player_id%3Db56c03a6-231a-4bbe-9c55-af3c8a8e9636%26ids%3Db56c03a6-231a-4bbe-9c55-af3c8a8e9636%26resizable%3D1%26autoplay%3Dtrue%26domain%3Dcbssports.com%26comp_ads_enabled%3Dfalse%26watchAndRead%3D0%26startTime%3D0%26env%3Dprod',
+        'only_matching': True,
    }, {
-        'url': 'https://www.cbssports.com/nba/news/nba-playoffs-2018-watch-76ers-vs-heat-game-3-series-schedule-tv-channel-online-stream/',
+        'url': 'https://embed.247sports.com/player/embed/?args=%3fplayer_id%3d1827823171591%26channel%3dcollege-football-recruiting%26pcid%3d1827823171591%26width%3d640%26height%3d360%26autoplay%3dTrue%26comp_ads_enabled%3dFalse%26uvpc%3dhttps%253a%252f%252fwww.cbssports.com%252fapi%252fcontent%252fvideo%252fconfig%252f%253fcfg%253duvp_247sports_v4%2526partner%253d247%26uvpc_m%3dhttps%253a%252f%252fwww.cbssports.com%252fapi%252fcontent%252fvideo%252fconfig%252f%253fcfg%253duvp_247sports_m_v4%2526partner_m%253d247_mobile%26utag%3d247sportssite%26resizable%3dTrue',
        'only_matching': True,
    }]

-    def _extract_video_info(self, filter_query, video_id):
-        return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
+    # def _extract_video_info(self, filter_query, video_id):
+    #     return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)

+    def _real_extract(self, url):
+        uuid, pcid = re.match(self._VALID_URL, url).groups()
+        query = {'id': uuid} if uuid else {'pcid': pcid}
+        video = self._download_json(
+            'https://www.cbssports.com/api/content/video/',
+            uuid or pcid, query=query)[0]
+        video_id = video['id']
+        title = video['title']
+        metadata = video.get('metaData') or {}
+        # return self._extract_video_info('byId=%d' % metadata['mpxOutletId'], video_id)
+        # return self._extract_video_info('byGuid=' + metadata['mpxRefId'], video_id)
+
+        formats = self._extract_m3u8_formats(
+            metadata['files'][0]['url'], video_id, 'mp4',
+            'm3u8_native', m3u8_id='hls', fatal=False)
+        self._sort_formats(formats)
+
+        image = video.get('image')
+        thumbnails = None
+        if image:
+            image_path = image.get('path')
+            if image_path:
+                thumbnails = [{
+                    'url': image_path,
+                    'width': int_or_none(image.get('width')),
+                    'height': int_or_none(image.get('height')),
+                    'filesize': int_or_none(image.get('size')),
+                }]
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'description': video.get('description'),
+            'timestamp': int_or_none(try_get(video, lambda x: x['dateCreated']['epoch'])),
+            'duration': int_or_none(metadata.get('duration')),
+        }
+
+
+class CBSSportsBaseIE(InfoExtractor):
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(
-            [r'(?:=|%26)pcid%3D(\d+)', r'embedVideo(?:Container)?_(\d+)'],
-            webpage, 'video id')
-        return self._extract_video_info('byId=%s' % video_id, video_id)
+        iframe_url = self._search_regex(
+            r'<iframe[^>]+(?:data-)?src="(https?://[^/]+/player/embed[^"]+)"',
+            webpage, 'embed url')
+        return self.url_result(iframe_url, CBSSportsEmbedIE.ie_key())
+
+
+class CBSSportsIE(CBSSportsBaseIE):
+    IE_NAME = 'cbssports'
+    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/video/(?P<id>[^/?#&]+)'
+    _TESTS = [{
+        'url': 'https://www.cbssports.com/college-football/video/cover-3-stanford-spring-gleaning/',
+        'info_dict': {
+            'id': 'b56c03a6-231a-4bbe-9c55-af3c8a8e9636',
+            'ext': 'mp4',
+            'title': 'Cover 3: Stanford Spring Gleaning',
+            'description': 'The Cover 3 crew break down everything you need to know about the Stanford Cardinal this spring.',
+            'timestamp': 1617218398,
+            'upload_date': '20210331',
+            'duration': 502,
+        },
+    }]
+
+
+class TwentyFourSevenSportsIE(CBSSportsBaseIE):
+    IE_NAME = '247sports'
+    _VALID_URL = r'https?://(?:www\.)?247sports\.com/Video/(?:[^/?#&]+-)?(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://247sports.com/Video/2021-QB-Jake-Garcia-senior-highlights-through-five-games-10084854/',
+        'info_dict': {
+            'id': '4f1265cb-c3b5-44a8-bb1d-1914119a0ccc',
+            'ext': 'mp4',
+            'title': '2021 QB Jake Garcia senior highlights through five games',
+            'description': 'md5:8cb67ebed48e2e6adac1701e0ff6e45b',
+            'timestamp': 1607114223,
+            'upload_date': '20201204',
+            'duration': 208,
+        },
+    }]
@@ -17,7 +17,7 @@ import math

 from ..compat import (
    compat_cookiejar_Cookie,
-    compat_cookies,
+    compat_cookies_SimpleCookie,
    compat_etree_Element,
    compat_etree_fromstring,
    compat_getpass,
@@ -231,8 +231,9 @@ class InfoExtractor(object):
    uploader:       Full name of the video uploader.
    license:        License name the video is licensed under.
    creator:        The creator of the video.
+    release_timestamp: UNIX timestamp of the moment the video was released.
    release_date:   The date (YYYYMMDD) when the video was released.
-    timestamp:      UNIX timestamp of the moment the video became available.
+    timestamp:      UNIX timestamp of the moment the video was uploaded
    upload_date:    Video upload date (YYYYMMDD).
                    If not explicitly set, calculated from timestamp.
    uploader_id:    Nickname or id of the video uploader.
@@ -251,8 +252,8 @@ class InfoExtractor(object):
                        * "data": The subtitles file contents
                        * "url": A URL pointing to the subtitles file
                    "ext" will be calculated from URL if missing
-    automatic_captions: Like 'subtitles', used by the YoutubeIE for
-                    automatically generated captions
+    automatic_captions: Like 'subtitles'; contains automatically generated
+                    captions instead of normal subtitles
    duration:       Length of the video in seconds, as an integer or float.
    view_count:     How many users have watched the video on the platform.
    like_count:     Number of positive ratings of the video
@@ -264,6 +265,7 @@ class InfoExtractor(object):
                    properties (all but one of text or html optional):
                        * "author" - human-readable name of the comment author
                        * "author_id" - user ID of the comment author
+                        * "author_thumbnail" - The thumbnail of the comment author
                        * "id" - Comment ID
                        * "html" - Comment as HTML
                        * "text" - Plain text of the comment
@@ -271,6 +273,12 @@ class InfoExtractor(object):
                        * "parent" - ID of the comment this one is replying to.
                                     Set to "root" to indicate that this is a
                                     comment to the original video.
+                        * "like_count" - Number of positive ratings of the comment
+                        * "dislike_count" - Number of negative ratings of the comment
+                        * "is_favorited" - Whether the comment is marked as
+                                           favorite by the video uploader
+                        * "author_is_uploader" - Whether the comment is made by
+                                                 the video uploader
    age_limit:      Age restriction for the video, as an integer (years)
    webpage_url:    The URL to the video webpage, if given to yt-dlp it
                    should allow to get the same result again. (It will be set
@@ -293,7 +301,11 @@ class InfoExtractor(object):
    playable_in_embed: Whether this video is allowed to play in embedded
                    players on other sites. Can be True (=always allowed),
                    False (=never allowed), None (=unknown), or a string
-                    specifying the criteria for embedability (Eg: 'whitelist').
+                    specifying the criteria for embedability (Eg: 'whitelist')
+    availability:   Under what condition the video is available. One of
+                    'private', 'premium_only', 'subscriber_only', 'needs_auth',
+                    'unlisted' or 'public'. Use 'InfoExtractor._availability'
+                    to set it
    __post_extractor: A function to be called just before the metadata is
                    written to either disk, logger or console. The function
                    must return a dict which will be added to the info_dict.
@@ -1296,6 +1308,7 @@ class InfoExtractor(object):

        def extract_video_object(e):
            assert e['@type'] == 'VideoObject'
+            author = e.get('author')
            info.update({
                'url': url_or_none(e.get('contentUrl')),
                'title': unescapeHTML(e.get('name')),
@@ -1303,7 +1316,11 @@ class InfoExtractor(object):
                'thumbnail': url_or_none(e.get('thumbnailUrl') or e.get('thumbnailURL')),
                'duration': parse_duration(e.get('duration')),
                'timestamp': unified_timestamp(e.get('uploadDate')),
-                'uploader': str_or_none(e.get('author')),
+                # author can be an instance of 'Organization' or 'Person' types.
+                # both types can have 'name' property(inherited from 'Thing' type). [1]
+                # however some websites are using 'Text' type instead.
+                # 1. https://schema.org/VideoObject
+                'uploader': author.get('name') if isinstance(author, dict) else author if isinstance(author, compat_str) else None,
                'filesize': float_or_none(e.get('contentSize')),
                'tbr': int_or_none(e.get('bitrate')),
                'width': int_or_none(e.get('width')),
@@ -1386,7 +1403,7 @@ class InfoExtractor(object):
        return self._hidden_inputs(form)

    class FormatSort:
-        regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
+        regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<separator>[~:])(?P<limit>.*?))?)? *$'

        default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
                   'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
@@ -1409,8 +1426,8 @@ class InfoExtractor(object):
            'ie_pref': {'priority': True, 'type': 'extractor'},
            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'lang': {'priority': True, 'convert': 'ignore', 'type': 'extractor', 'field': 'language_preference'},
-            'quality': {'convert': 'float_none', 'type': 'extractor'},
+            'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
+            'quality': {'convert': 'float_none'},
            'filesize': {'convert': 'bytes'},
            'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
            'id': {'convert': 'string', 'field': 'format_id'},
@@ -1421,7 +1438,7 @@ class InfoExtractor(object):
            'vbr': {'convert': 'float_none'},
            'abr': {'convert': 'float_none'},
            'asr': {'convert': 'float_none'},
-            'source': {'convert': 'ignore', 'type': 'extractor', 'field': 'source_preference'},
+            'source': {'convert': 'ignore', 'field': 'source_preference'},

            'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
            'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
@@ -1546,7 +1563,7 @@ class InfoExtractor(object):
                if self._get_field_setting(field, 'type') == 'alias':
                    field = self._get_field_setting(field, 'field')
                reverse = match.group('reverse') is not None
-                closest = match.group('seperator') == '~'
+                closest = match.group('separator') == '~'
                limit_text = match.group('limit')

                has_limit = limit_text is not None
@@ -1563,7 +1580,8 @@ class InfoExtractor(object):
                             else None)

        def print_verbose_info(self, to_screen):
-            to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
+            if self._sort_user:
+                to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
            if self._sort_extractor:
                to_screen('[debug] Sort order given by extractor: %s' % ', '.join(self._sort_extractor))
            to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
@@ -1877,7 +1895,8 @@ class InfoExtractor(object):
        if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
            return []

-        if re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc):  # Apple FairPlay
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc)):  # Apple FairPlay
            return []

        formats = []
@@ -2394,7 +2413,7 @@ class InfoExtractor(object):
            http://standards.iso.org/ittf/PubliclyAvailableStandards/c065274_ISO_IEC_23009-1_2014.zip
         2. https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP
        """
-        if not self._downloader.params.get('dynamic_mpd'):
+        if not self._downloader.params.get('dynamic_mpd', True):
            if mpd_doc.get('type') == 'dynamic':
                return []

@@ -3206,10 +3225,10 @@ class InfoExtractor(object):
        self._downloader.cookiejar.set_cookie(cookie)

    def _get_cookies(self, url):
-        """ Return a compat_cookies.SimpleCookie with the cookies for the url """
+        """ Return a compat_cookies_SimpleCookie with the cookies for the url """
        req = sanitized_Request(url)
        self._downloader.cookiejar.add_cookie_header(req)
-        return compat_cookies.SimpleCookie(req.get_header('Cookie'))
+        return compat_cookies_SimpleCookie(req.get_header('Cookie'))

    def _apply_first_set_cookie_header(self, url_handle, cookie):
        """
@@ -3324,6 +3343,20 @@ class InfoExtractor(object):
    def _generic_title(self, url):
        return compat_urllib_parse_unquote(os.path.splitext(url_basename(url))[0])

+    @staticmethod
+    def _availability(is_private, needs_premium, needs_subscription, needs_auth, is_unlisted):
+        all_known = all(map(
+            lambda x: x is not None,
+            (is_private, needs_premium, needs_subscription, needs_auth, is_unlisted)))
+        return (
+            'private' if is_private
+            else 'premium_only' if needs_premium
+            else 'subscriber_only' if needs_subscription
+            else 'needs_auth' if needs_auth
+            else 'unlisted' if is_unlisted
+            else 'public' if all_known
+            else None)
+

 class SearchInfoExtractor(InfoExtractor):
    """
@@ -1,5 +1,7 @@
 from __future__ import unicode_literals

+import re
+
 from .common import InfoExtractor
 from ..compat import (
    compat_urlparse,
@@ -58,3 +60,16 @@ class MmsIE(InfoExtractor):
            'title': title,
            'url': url,
        }
+
+
+class ViewSourceIE(InfoExtractor):
+    IE_DESC = False
+    _VALID_URL = r'view-source:(?P<url>.+)'
+
+    _TEST = {
+        'url': 'view-source:https://www.youtube.com/watch?v=BaW_jenozKc',
+        'only_matching': True
+    }
+
+    def _real_extract(self, url):
+        return self.url_result(re.match(self._VALID_URL, url).group('url'))
@@ -25,12 +25,12 @@ class CuriosityStreamBaseIE(InfoExtractor):
            raise ExtractorError(
                '%s said: %s' % (self.IE_NAME, error), expected=True)

-    def _call_api(self, path, video_id):
+    def _call_api(self, path, video_id, query=None):
        headers = {}
        if self._auth_token:
            headers['X-Auth-Token'] = self._auth_token
        result = self._download_json(
-            self._API_BASE_URL + path, video_id, headers=headers)
+            self._API_BASE_URL + path, video_id, headers=headers, query=query)
        self._handle_errors(result)
        return result['data']

@@ -52,62 +52,75 @@ class CuriosityStreamIE(CuriosityStreamBaseIE):
    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/video/(?P<id>\d+)'
    _TEST = {
        'url': 'https://app.curiositystream.com/video/2',
-        'md5': '262bb2f257ff301115f1973540de8983',
        'info_dict': {
            'id': '2',
            'ext': 'mp4',
            'title': 'How Did You Develop The Internet?',
            'description': 'Vint Cerf, Google\'s Chief Internet Evangelist, describes how he and Bob Kahn created the internet.',
-        }
+        },
+        'params': {
+            'format': 'bestvideo',
+            # m3u8 download
+            'skip_download': True,
+        },
    }

    def _real_extract(self, url):
        video_id = self._match_id(url)
-        media = self._call_api('media/' + video_id, video_id)
-        title = media['title']

        formats = []
-        for encoding in media.get('encodings', []):
-            m3u8_url = encoding.get('master_playlist_url')
-            if m3u8_url:
-                formats.extend(self._extract_m3u8_formats(
-                    m3u8_url, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
-            encoding_url = encoding.get('url')
-            file_url = encoding.get('file_url')
-            if not encoding_url and not file_url:
-                continue
-            f = {
-                'width': int_or_none(encoding.get('width')),
-                'height': int_or_none(encoding.get('height')),
-                'vbr': int_or_none(encoding.get('video_bitrate')),
-                'abr': int_or_none(encoding.get('audio_bitrate')),
-                'filesize': int_or_none(encoding.get('size_in_bytes')),
-                'vcodec': encoding.get('video_codec'),
-                'acodec': encoding.get('audio_codec'),
-                'container': encoding.get('container_type'),
-            }
-            for f_url in (encoding_url, file_url):
-                if not f_url:
+        for encoding_format in ('m3u8', 'mpd'):
+            media = self._call_api('media/' + video_id, video_id, query={
+                'encodingsNew': 'true',
+                'encodingsFormat': encoding_format,
+            })
+            for encoding in media.get('encodings', []):
+                playlist_url = encoding.get('master_playlist_url')
+                if encoding_format == 'm3u8':
+                    # use `m3u8` entry_protocol until EXT-X-MAP is properly supported by `m3u8_native` entry_protocol
+                    formats.extend(self._extract_m3u8_formats(
+                        playlist_url, video_id, 'mp4',
+                        m3u8_id='hls', fatal=False))
+                elif encoding_format == 'mpd':
+                    formats.extend(self._extract_mpd_formats(
+                        playlist_url, video_id, mpd_id='dash', fatal=False))
+                encoding_url = encoding.get('url')
+                file_url = encoding.get('file_url')
+                if not encoding_url and not file_url:
                    continue
-                fmt = f.copy()
-                rtmp = re.search(r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+))/(?P<playpath>mp[34]:.+)$', f_url)
-                if rtmp:
-                    fmt.update({
-                        'url': rtmp.group('url'),
-                        'play_path': rtmp.group('playpath'),
-                        'app': rtmp.group('app'),
-                        'ext': 'flv',
-                        'format_id': 'rtmp',
-                    })
-                else:
-                    fmt.update({
-                        'url': f_url,
-                        'format_id': 'http',
-                    })
-                formats.append(fmt)
+                f = {
+                    'width': int_or_none(encoding.get('width')),
+                    'height': int_or_none(encoding.get('height')),
+                    'vbr': int_or_none(encoding.get('video_bitrate')),
+                    'abr': int_or_none(encoding.get('audio_bitrate')),
+                    'filesize': int_or_none(encoding.get('size_in_bytes')),
+                    'vcodec': encoding.get('video_codec'),
+                    'acodec': encoding.get('audio_codec'),
+                    'container': encoding.get('container_type'),
+                }
+                for f_url in (encoding_url, file_url):
+                    if not f_url:
+                        continue
+                    fmt = f.copy()
+                    rtmp = re.search(r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+))/(?P<playpath>mp[34]:.+)$', f_url)
+                    if rtmp:
+                        fmt.update({
+                            'url': rtmp.group('url'),
+                            'play_path': rtmp.group('playpath'),
+                            'app': rtmp.group('app'),
+                            'ext': 'flv',
+                            'format_id': 'rtmp',
+                        })
+                    else:
+                        fmt.update({
+                            'url': f_url,
+                            'format_id': 'http',
+                        })
+                    formats.append(fmt)
        self._sort_formats(formats)

+        title = media['title']
+
        subtitles = {}
        for closed_caption in media.get('closed_captions', []):
            sub_url = closed_caption.get('file')
@@ -140,7 +153,7 @@ class CuriosityStreamCollectionIE(CuriosityStreamBaseIE):
            'title': 'Curious Minds: The Internet',
            'description': 'How is the internet shaping our lives in the 21st Century?',
        },
-        'playlist_mincount': 17,
+        'playlist_mincount': 16,
    }, {
        'url': 'https://curiositystream.com/series/2',
        'only_matching': True,
@@ -0,0 +1,100 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from ..compat import compat_str
+from ..utils import try_get
+from .common import InfoExtractor
+from .dplay import DPlayIE
+
+
+class DiscoveryPlusIndiaIE(DPlayIE):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.in/videos?' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.in/videos/how-do-they-do-it/fugu-and-more?seasonId=8&type=EPISODE',
+        'info_dict': {
+            'id': '27104',
+            'ext': 'mp4',
+            'display_id': 'how-do-they-do-it/fugu-and-more',
+            'title': 'Fugu and More',
+            'description': 'The Japanese catch, prepare and eat the deadliest fish on the planet.',
+            'duration': 1319,
+            'timestamp': 1582309800,
+            'upload_date': '20200221',
+            'series': 'How Do They Do It?',
+            'season_number': 8,
+            'episode_number': 2,
+            'creator': 'Discovery Channel',
+        },
+        'params': {
+            'format': 'bestvideo',
+            'skip_download': True,
+        },
+        'skip': 'Cookies (not necessarily logged in) are needed'
+    }]
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['x-disco-params'] = 'realm=%s' % realm
+        headers['x-disco-client'] = 'WEB:UNKNOWN:dplus-india:17.0.0'
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        return self._download_json(
+            disco_base + 'playback/v3/videoPlaybackInfo',
+            video_id, headers=headers, data=json.dumps({
+                'deviceInfo': {
+                    'adBlocker': False,
+                },
+                'videoId': video_id,
+            }).encode('utf-8'))['data']['attributes']['streaming']
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'ap2-prod-direct.discoveryplus.in', 'dplusindia', 'in')
+
+
+class DiscoveryPlusIndiaShowIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.in/show/(?P<show_name>[^/]+)/?(?:[?#]|$)'
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.in/show/how-do-they-do-it',
+        'playlist_mincount': 140,
+        'info_dict': {
+            'id': 'how-do-they-do-it',
+        },
+    }
+    ]
+
+    def _entries(self, show_name):
+        headers = {
+            'x-disco-client': 'WEB:UNKNOWN:dplus-india:prod',
+            'x-disco-params': 'realm=dplusindia',
+            'referer': 'https://www.discoveryplus.in/',
+        }
+        show_url = 'https://ap2-prod-direct.discoveryplus.in/cms/routes/show/{}?include=default'.format(show_name)
+        show_json = self._download_json(show_url,
+                                        video_id=show_name,
+                                        headers=headers)['included'][4]['attributes']['component']
+        show_id = show_json['mandatoryParams'].split('=')[-1]
+        season_url = 'https://ap2-prod-direct.discoveryplus.in/content/videos?sort=episodeNumber&filter[seasonNumber]={}&filter[show.id]={}&page[size]=100&page[number]={}'
+        for season in show_json['filters'][0]['options']:
+            season_id = season['id']
+            total_pages, page_num = 1, 0
+            while page_num < total_pages:
+                season_json = self._download_json(season_url.format(season_id, show_id, compat_str(page_num + 1)),
+                                                  video_id=show_id, headers=headers,
+                                                  note='Downloading JSON metadata%s' % (' page %d' % page_num if page_num else ''))
+                if page_num == 0:
+                    total_pages = try_get(season_json, lambda x: x['meta']['totalPages'], int) or 1
+                episodes_json = season_json['data']
+                for episode in episodes_json:
+                    video_id = episode['attributes']['path']
+                    yield self.url_result(
+                        'https://discoveryplus.in/videos/%s' % video_id,
+                        ie=DiscoveryPlusIndiaIE.ie_key(), video_id=video_id)
+                page_num += 1
+
+    def _real_extract(self, url):
+        show_name = re.match(self._VALID_URL, url).group('show_name')
+        return self.playlist_result(self._entries(show_name), playlist_id=show_name)
@@ -80,6 +80,7 @@ from .arte import (
    ArteTVEmbedIE,
    ArteTVPlaylistIE,
 )
+from .arnes import ArnesIE
 from .asiancrush import (
    AsianCrushIE,
    AsianCrushPlaylistIE,
@@ -108,7 +109,8 @@ from .bandcamp import BandcampIE, BandcampAlbumIE, BandcampWeeklyIE
 from .bbc import (
    BBCCoUkIE,
    BBCCoUkArticleIE,
-    BBCCoUkIPlayerPlaylistIE,
+    BBCCoUkIPlayerEpisodesIE,
+    BBCCoUkIPlayerGroupIE,
    BBCCoUkPlaylistIE,
    BBCIE,
 )
@@ -208,7 +210,11 @@ from .cbsnews import (
    CBSNewsIE,
    CBSNewsLiveVideoIE,
 )
-from .cbssports import CBSSportsIE
+from .cbssports import (
+    CBSSportsEmbedIE,
+    CBSSportsIE,
+    TwentyFourSevenSportsIE,
+)
 from .ccc import (
    CCCIE,
    CCCPlaylistIE,
@@ -263,6 +269,7 @@ from .commonmistakes import CommonMistakesIE, UnicodeBOMIE
 from .commonprotocols import (
    MmsIE,
    RtmpIE,
+    ViewSourceIE,
 )
 from .condenast import CondeNastIE
 from .contv import CONtvIE
@@ -306,6 +313,10 @@ from .democracynow import DemocracynowIE
 from .dfb import DFBIE
 from .dhm import DHMIE
 from .digg import DiggIE
+from .discoveryplusindia import (
+    DiscoveryPlusIndiaIE,
+    DiscoveryPlusIndiaShowIE,
+)
 from .dotsub import DotsubIE
 from .douyutv import (
    DouyuShowIE,
@@ -581,7 +592,11 @@ from .kuwo import (
    KuwoCategoryIE,
    KuwoMvIE,
 )
-from .la7 import LA7IE
+from .la7 import (
+    LA7IE,
+    LA7PodcastEpisodeIE,
+    LA7PodcastIE,
+)
 from .laola1tv import (
    Laola1TvEmbedIE,
    Laola1TvIE,
@@ -622,7 +637,11 @@ from .limelight import (
    LimelightChannelIE,
    LimelightChannelListIE,
 )
-from .line import LineTVIE
+from .line import (
+    LineTVIE,
+    LineLiveIE,
+    LineLiveChannelIE,
+)
 from .linkedin import (
    LinkedInLearningIE,
    LinkedInLearningCourseIE,
@@ -660,6 +679,7 @@ from .mangomolo import (
    MangomoloLiveIE,
 )
 from .manyvids import ManyVidsIE
+from .maoritv import MaoriTVIE
 from .markiza import (
    MarkizaIE,
    MarkizaPageIE,
@@ -708,7 +728,10 @@ from .mixcloud import (
    MixcloudUserIE,
    MixcloudPlaylistIE,
 )
-from .mlb import MLBIE
+from .mlb import (
+    MLBIE,
+    MLBVideoIE,
+)
 from .mnet import MnetIE
 from .moevideo import MoeVideoIE
 from .mofosex import (
@@ -732,6 +755,8 @@ from .mtv import (
    MTVServicesEmbeddedIE,
    MTVDEIE,
    MTVJapanIE,
+    MTVItaliaIE,
+    MTVItaliaProgrammaIE,
 )
 from .muenchentv import MuenchenTVIE
 from .mwave import MwaveIE, MwaveMeetGreetIE
@@ -778,8 +803,9 @@ from .ndr import (
    NJoyEmbedIE,
 )
 from .ndtv import NDTVIE
-from .netzkino import NetzkinoIE
+from .nebula import NebulaIE
 from .nerdcubed import NerdCubedFeedIE
+from .netzkino import NetzkinoIE
 from .neteasemusic import (
    NetEaseMusicIE,
    NetEaseMusicAlbumIE,
@@ -804,6 +830,7 @@ from .nexx import (
    NexxIE,
    NexxEmbedIE,
 )
+from .nfhsnetwork import NFHSNetworkIE
 from .nfl import (
    NFLIE,
    NFLArticleIE,
@@ -820,7 +847,7 @@ from .nick import (
    NickNightIE,
    NickRuIE,
 )
-from .niconico import NiconicoIE, NiconicoPlaylistIE
+from .niconico import NiconicoIE, NiconicoPlaylistIE, NiconicoUserIE
 from .ninecninemedia import NineCNineMediaIE
 from .ninegag import NineGagIE
 from .ninenow import NineNowIE
@@ -915,6 +942,11 @@ from .packtpub import (
    PacktPubIE,
    PacktPubCourseIE,
 )
+from .palcomp3 import (
+    PalcoMP3IE,
+    PalcoMP3ArtistIE,
+    PalcoMP3VideoIE,
+)
 from .pandoratv import PandoraTVIE
 from .parliamentliveuk import ParliamentLiveUKIE
 from .patreon import PatreonIE
@@ -951,6 +983,7 @@ from .plays import PlaysTVIE
 from .playtvak import PlaytvakIE
 from .playvid import PlayvidIE
 from .playwire import PlaywireIE
+from .plutotv import PlutoTVIE
 from .pluralsight import (
    PluralsightIE,
    PluralsightCourseIE,
@@ -1176,7 +1209,10 @@ from .spike import (
    BellatorIE,
    ParamountNetworkIE,
 )
-from .stitcher import StitcherIE
+from .stitcher import (
+    StitcherIE,
+    StitcherShowIE,
+)
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
 from .sportdeutschland import SportDeutschlandIE
@@ -1557,6 +1593,7 @@ from .weibo import (
    WeiboMobileIE
 )
 from .weiqitv import WeiqiTVIE
+from .wimtv import WimTVIE
 from .wistia import (
    WistiaIE,
    WistiaPlaylistIE,
@@ -1663,8 +1700,14 @@ from .zattoo import (
    ZattooLiveIE,
 )
 from .zdf import ZDFIE, ZDFChannelIE
+from .zee5 import (
+    Zee5IE,
+    Zee5SeriesIE,
+)
 from .zhihu import ZhihuIE
-from .zingmp3 import ZingMp3IE
-from .zee5 import Zee5IE
+from .zingmp3 import (
+    ZingMp3IE,
+    ZingMp3AlbumIE,
+)
 from .zoom import ZoomIE
 from .zype import ZypeIE
@@ -401,7 +401,7 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
            (r'player\.load[^;]+src:\s*["\']([^"\']+)',
             r'id-video=([^@]+@[^"]+)',
             r'<a[^>]+href="(?:https?:)?//videos\.francetv\.fr/video/([^@]+@[^"]+)"',
-             r'data-id="([^"]+)"'),
+             r'data-id=["\']([\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'),
            webpage, 'video id')

        return self._make_url_result(video_id)
@@ -17,7 +17,7 @@ class FujiTVFODPlus7IE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
        formats = self._extract_m3u8_formats(
-            self._BASE_URL + 'abr/pc_html5/%s.m3u8' % video_id, video_id)
+            self._BASE_URL + 'abr/pc_html5/%s.m3u8' % video_id, video_id, 'mp4')
        for f in formats:
            wh = self._BITRATE_MAP.get(f.get('tbr'))
            if wh:
@@ -134,6 +134,7 @@ from .rumble import RumbleEmbedIE
 from .arcpublishing import ArcPublishingIE
 from .medialaan import MedialaanIE
 from .simplecast import SimplecastIE
+from .wimtv import WimTVIE


 class GenericIE(InfoExtractor):
@@ -2250,6 +2251,15 @@ class GenericIE(InfoExtractor):
            },
            'playlist_mincount': 52,
        },
+        {
+            # WimTv embed player
+            'url': 'http://www.msmotor.tv/wearefmi-pt-2-2021/',
+            'info_dict': {
+                'id': 'wearefmi-pt-2-2021',
+                'title': '#WEAREFMI – PT.2 – 2021 – MsMotorTV',
+            },
+            'playlist_count': 1,
+        },
    ]

    def report_following_redirect(self, new_url):
@@ -2649,6 +2659,15 @@ class GenericIE(InfoExtractor):
        if vid_me_embed_url is not None:
            return self.url_result(vid_me_embed_url, 'Vidme')

+        # Invidious Instances
+        # https://github.com/yt-dlp/yt-dlp/issues/195
+        # https://github.com/iv-org/invidious/pull/1730
+        youtube_url = self._search_regex(
+            r'<link rel="alternate" href="(https://www\.youtube\.com/watch\?v=[0-9A-Za-z_-]{11})"',
+            webpage, 'youtube link', default=None)
+        if youtube_url:
+            return self.url_result(youtube_url, YoutubeIE.ie_key())
+
        # Look for YouTube embeds
        youtube_urls = YoutubeIE._extract_urls(webpage)
        if youtube_urls:
@@ -2955,7 +2974,7 @@ class GenericIE(InfoExtractor):
            webpage)
        if not mobj:
            mobj = re.search(
-                r'data-video-link=["\'](?P<url>http://m.mlb.com/video/[^"\']+)',
+                r'data-video-link=["\'](?P<url>http://m\.mlb\.com/video/[^"\']+)',
                webpage)
        if mobj is not None:
            return self.url_result(mobj.group('url'), 'MLB')
@@ -3350,6 +3369,11 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                rcs_urls, video_id, video_title, ie=RCSEmbedsIE.ie_key())

+        wimtv_urls = WimTVIE._extract_urls(webpage)
+        if wimtv_urls:
+            return self.playlist_from_matches(
+                wimtv_urls, video_id, video_title, ie=WimTVIE.ie_key())
+
        bitchute_urls = BitChuteIE._extract_urls(webpage)
        if bitchute_urls:
            return self.playlist_from_matches(
@@ -12,6 +12,7 @@ from ..compat import (
 )
 from ..utils import (
    ExtractorError,
+    float_or_none,
    get_element_by_attribute,
    int_or_none,
    lowercase_escape,
@@ -32,6 +33,7 @@ class InstagramIE(InfoExtractor):
            'title': 'Video by naomipq',
            'description': 'md5:1f17f0ab29bd6fe2bfad705f58de3cb8',
            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 0,
            'timestamp': 1371748545,
            'upload_date': '20130620',
            'uploader_id': 'naomipq',
@@ -48,6 +50,7 @@ class InstagramIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Video by britneyspears',
            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 0,
            'timestamp': 1453760977,
            'upload_date': '20160125',
            'uploader_id': 'britneyspears',
@@ -86,6 +89,24 @@ class InstagramIE(InfoExtractor):
            'title': 'Post by instagram',
            'description': 'md5:0f9203fc6a2ce4d228da5754bcf54957',
        },
+    }, {
+        # IGTV
+        'url': 'https://www.instagram.com/tv/BkfuX9UB-eK/',
+        'info_dict': {
+            'id': 'BkfuX9UB-eK',
+            'ext': 'mp4',
+            'title': 'Fingerboarding Tricks with @cass.fb',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 53.83,
+            'timestamp': 1530032919,
+            'upload_date': '20180626',
+            'uploader_id': 'instagram',
+            'uploader': 'Instagram',
+            'like_count': int,
+            'comment_count': int,
+            'comments': list,
+            'description': 'Meet Cass Hirst (@cass.fb), a fingerboarding pro who can perform tiny ollies and kickflips while blindfolded.',
+        }
    }, {
        'url': 'https://instagram.com/p/-Cmh1cukG2/',
        'only_matching': True,
@@ -159,7 +180,9 @@ class InstagramIE(InfoExtractor):
            description = try_get(
                media, lambda x: x['edge_media_to_caption']['edges'][0]['node']['text'],
                compat_str) or media.get('caption')
+            title = media.get('title')
            thumbnail = media.get('display_src') or media.get('display_url')
+            duration = float_or_none(media.get('video_duration'))
            timestamp = int_or_none(media.get('taken_at_timestamp') or media.get('date'))
            uploader = media.get('owner', {}).get('full_name')
            uploader_id = media.get('owner', {}).get('username')
@@ -200,9 +223,10 @@ class InstagramIE(InfoExtractor):
                            continue
                        entries.append({
                            'id': node.get('shortcode') or node['id'],
-                            'title': 'Video %d' % edge_num,
+                            'title': node.get('title') or 'Video %d' % edge_num,
                            'url': node_video_url,
                            'thumbnail': node.get('display_url'),
+                            'duration': float_or_none(node.get('video_duration')),
                            'width': int_or_none(try_get(node, lambda x: x['dimensions']['width'])),
                            'height': int_or_none(try_get(node, lambda x: x['dimensions']['height'])),
                            'view_count': int_or_none(node.get('video_view_count')),
@@ -239,8 +263,9 @@ class InstagramIE(InfoExtractor):
            'id': video_id,
            'formats': formats,
            'ext': 'mp4',
-            'title': 'Video by %s' % uploader_id,
+            'title': title or 'Video by %s' % uploader_id,
            'description': description,
+            'duration': duration,
            'thumbnail': thumbnail,
            'timestamp': timestamp,
            'uploader_id': uploader_id,
@@ -146,7 +146,7 @@ class IviIE(InfoExtractor):
                        expected=True)
                elif not pycryptodomex_found:
                    raise ExtractorError(
-                        'pycryptodomex not found. Please install it.',
+                        'pycryptodomex not found. Please install',
                        expected=True)
                elif message:
                    extractor_msg += ': ' + message
@@ -29,34 +29,51 @@ class JamendoIE(InfoExtractor):
            'id': '196219',
            'display_id': 'stories-from-emona-i',
            'ext': 'flac',
-            'title': 'Maya Filipič - Stories from Emona I',
-            'artist': 'Maya Filipič',
+            # 'title': 'Maya Filipič - Stories from Emona I',
+            'title': 'Stories from Emona I',
+            # 'artist': 'Maya Filipič',
            'track': 'Stories from Emona I',
            'duration': 210,
            'thumbnail': r're:^https?://.*\.jpg',
            'timestamp': 1217438117,
            'upload_date': '20080730',
+            'license': 'by-nc-nd',
+            'view_count': int,
+            'like_count': int,
+            'average_rating': int,
+            'tags': ['piano', 'peaceful', 'newage', 'strings', 'upbeat'],
        }
    }, {
        'url': 'https://licensing.jamendo.com/en/track/1496667/energetic-rock',
        'only_matching': True,
    }]

+    def _call_api(self, resource, resource_id):
+        path = '/api/%ss' % resource
+        rand = compat_str(random.random())
+        return self._download_json(
+            'https://www.jamendo.com' + path, resource_id, query={
+                'id[]': resource_id,
+            }, headers={
+                'X-Jam-Call': '$%s*%s~' % (hashlib.sha1((path + rand).encode()).hexdigest(), rand)
+            })[0]
+
    def _real_extract(self, url):
        track_id, display_id = self._VALID_URL_RE.match(url).groups()
-        webpage = self._download_webpage(
-            'https://www.jamendo.com/track/' + track_id, track_id)
-        models = self._parse_json(self._html_search_regex(
-            r"data-bundled-models='([^']+)",
-            webpage, 'bundled models'), track_id)
-        track = models['track']['models'][0]
+        # webpage = self._download_webpage(
+        #     'https://www.jamendo.com/track/' + track_id, track_id)
+        # models = self._parse_json(self._html_search_regex(
+        #     r"data-bundled-models='([^']+)",
+        #     webpage, 'bundled models'), track_id)
+        # track = models['track']['models'][0]
+        track = self._call_api('track', track_id)
        title = track_name = track['name']
-        get_model = lambda x: try_get(models, lambda y: y[x]['models'][0], dict) or {}
-        artist = get_model('artist')
-        artist_name = artist.get('name')
-        if artist_name:
-            title = '%s - %s' % (artist_name, title)
-        album = get_model('album')
+        # get_model = lambda x: try_get(models, lambda y: y[x]['models'][0], dict) or {}
+        # artist = get_model('artist')
+        # artist_name = artist.get('name')
+        # if artist_name:
+        #     title = '%s - %s' % (artist_name, title)
+        # album = get_model('album')

        formats = [{
            'url': 'https://%s.jamendo.com/?trackid=%s&format=%s&from=app-97dab294'
@@ -74,7 +91,7 @@ class JamendoIE(InfoExtractor):

        urls = []
        thumbnails = []
-        for _, covers in track.get('cover', {}).items():
+        for covers in (track.get('cover') or {}).values():
            for cover_id, cover_url in covers.items():
                if not cover_url or cover_url in urls:
                    continue
@@ -88,13 +105,14 @@ class JamendoIE(InfoExtractor):
                })

        tags = []
-        for tag in track.get('tags', []):
+        for tag in (track.get('tags') or []):
            tag_name = tag.get('name')
            if not tag_name:
                continue
            tags.append(tag_name)

        stats = track.get('stats') or {}
+        license = track.get('licenseCC') or []

        return {
            'id': track_id,
@@ -103,11 +121,11 @@ class JamendoIE(InfoExtractor):
            'title': title,
            'description': track.get('description'),
            'duration': int_or_none(track.get('duration')),
-            'artist': artist_name,
+            # 'artist': artist_name,
            'track': track_name,
-            'album': album.get('name'),
+            # 'album': album.get('name'),
            'formats': formats,
-            'license': '-'.join(track.get('licenseCC', [])) or None,
+            'license': '-'.join(license) if license else None,
            'timestamp': int_or_none(track.get('dateCreated')),
            'view_count': int_or_none(stats.get('listenedAll')),
            'like_count': int_or_none(stats.get('favorited')),
@@ -116,9 +134,9 @@ class JamendoIE(InfoExtractor):
        }


-class JamendoAlbumIE(InfoExtractor):
+class JamendoAlbumIE(JamendoIE):
    _VALID_URL = r'https?://(?:www\.)?jamendo\.com/album/(?P<id>[0-9]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'https://www.jamendo.com/album/121486/duck-on-cover',
        'info_dict': {
            'id': '121486',
@@ -151,17 +169,7 @@ class JamendoAlbumIE(InfoExtractor):
        'params': {
            'playlistend': 2
        }
-    }
-
-    def _call_api(self, resource, resource_id):
-        path = '/api/%ss' % resource
-        rand = compat_str(random.random())
-        return self._download_json(
-            'https://www.jamendo.com' + path, resource_id, query={
-                'id[]': resource_id,
-            }, headers={
-                'X-Jam-Call': '$%s*%s~' % (hashlib.sha1((path + rand).encode()).hexdigest(), rand)
-            })[0]
+    }]

    def _real_extract(self, url):
        album_id = self._match_id(url)
@@ -169,7 +177,7 @@ class JamendoAlbumIE(InfoExtractor):
        album_name = album.get('name')

        entries = []
-        for track in album.get('tracks', []):
+        for track in (album.get('tracks') or []):
            track_id = track.get('id')
            if not track_id:
                continue
@@ -1,9 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import re
+
 from .common import InfoExtractor
 from ..utils import (
+    determine_ext,
+    float_or_none,
+    parse_duration,
    smuggle_url,
+    unified_strdate,
 )


@@ -57,3 +63,141 @@ class LA7IE(InfoExtractor):
            'thumbnail': self._og_search_thumbnail(webpage, default=None),
            'ie_key': 'Kaltura',
        }
+
+
+class LA7PodcastEpisodeIE(InfoExtractor):
+    IE_NAME = 'la7.it:pod:episode'
+    _VALID_URL = r'''(?x)(https?://)?
+        (?:www\.)?la7\.it/[^/]+/podcast/([^/]+-)?(?P<id>\d+)'''
+
+    _TESTS = [{
+        'url': 'https://www.la7.it/voicetown/podcast/la-carezza-delle-memoria-di-carlo-verdone-23-03-2021-371497',
+        'md5': '7737d4d79b3c1a34b3de3e16297119ed',
+        'info_dict': {
+            'id': '371497',
+            'ext': 'mp3',
+            'title': '"La carezza delle memoria" di Carlo Verdone',
+            'description': 'md5:5abf07c3c551a687db80af3f9ceb7d52',
+            'thumbnail': 'https://www.la7.it/sites/default/files/podcast/371497.jpg',
+            'upload_date': '20210323',
+        },
+    }, {
+        # embed url
+        'url': 'https://www.la7.it/embed/podcast/371497',
+        'only_matching': True,
+    }, {
+        # date already in the title
+        'url': 'https://www.la7.it/propagandalive/podcast/lintervista-di-diego-bianchi-ad-annalisa-cuzzocrea-puntata-del-1932021-20-03-2021-371130',
+        'only_matching': True,
+    }, {
+        # title same as show_title
+        'url': 'https://www.la7.it/otto-e-mezzo/podcast/otto-e-mezzo-26-03-2021-372340',
+        'only_matching': True,
+    }]
+
+    def _extract_info(self, webpage, video_id=None, ppn=None):
+        if not video_id:
+            video_id = self._search_regex(
+                r'data-nid=([\'"])(?P<vid>\d+)\1',
+                webpage, 'video_id', group='vid')
+
+        media_url = self._search_regex(
+            (r'src:\s*([\'"])(?P<url>.+?mp3.+?)\1',
+             r'data-podcast=([\'"])(?P<url>.+?mp3.+?)\1'),
+            webpage, 'media_url', group='url')
+        ext = determine_ext(media_url)
+        formats = [{
+            'url': media_url,
+            'format_id': ext,
+            'ext': ext,
+        }]
+        self._sort_formats(formats)
+
+        title = self._html_search_regex(
+            (r'<div class="title">(?P<title>.+?)</',
+             r'<title>(?P<title>[^<]+)</title>',
+             r'title:\s*([\'"])(?P<title>.+?)\1'),
+            webpage, 'title', group='title')
+
+        description = (
+            self._html_search_regex(
+                (r'<div class="description">(.+?)</div>',
+                 r'<div class="description-mobile">(.+?)</div>',
+                 r'<div class="box-txt">([^<]+?)</div>',
+                 r'<div class="field-content"><p>(.+?)</p></div>'),
+                webpage, 'description', default=None)
+            or self._html_search_meta('description', webpage))
+
+        thumb = self._html_search_regex(
+            (r'<div class="podcast-image"><img src="(.+?)"></div>',
+             r'<div class="container-embed"[^<]+url\((.+?)\);">',
+             r'<div class="field-content"><img src="(.+?)"'),
+            webpage, 'thumbnail', fatal=False, default=None)
+
+        duration = parse_duration(self._html_search_regex(
+            r'<span class="(?:durata|duration)">([\d:]+)</span>',
+            webpage, 'duration', fatal=False, default=None))
+
+        date = self._html_search_regex(
+            r'class="data">\s*(?:<span>)?([\d\.]+)\s*</',
+            webpage, 'date', default=None)
+
+        date_alt = self._search_regex(
+            r'(\d+[\./]\d+[\./]\d+)', title, 'date_alt', default=None)
+        ppn = ppn or self._search_regex(
+            r'ppN:\s*([\'"])(?P<ppn>.+?)\1',
+            webpage, 'ppn', group='ppn', default=None)
+        # if the date is not in the title
+        # and title is the same as the show_title
+        # add the date to the title
+        if date and not date_alt and ppn and ppn.lower() == title.lower():
+            title += ' del %s' % date
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'duration': float_or_none(duration),
+            'formats': formats,
+            'thumbnail': thumb,
+            'upload_date': unified_strdate(date),
+        }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+
+        return self._extract_info(webpage, video_id)
+
+
+class LA7PodcastIE(LA7PodcastEpisodeIE):
+    IE_NAME = 'la7.it:podcast'
+    _VALID_URL = r'(https?://)?(www\.)?la7\.it/(?P<id>[^/]+)/podcast/?(?:$|[#?])'
+
+    _TESTS = [{
+        'url': 'https://www.la7.it/propagandalive/podcast',
+        'info_dict': {
+            'id': 'propagandalive',
+            'title': "Propaganda Live",
+        },
+        'playlist_count': 10,
+    }]
+
+    def _real_extract(self, url):
+        playlist_id = self._match_id(url)
+        webpage = self._download_webpage(url, playlist_id)
+
+        title = (
+            self._html_search_regex(
+                r'<h1.*?>(.+?)</h1>', webpage, 'title', fatal=False, default=None)
+            or self._og_search_title(webpage))
+        ppn = self._search_regex(
+            r'window\.ppN\s*=\s*([\'"])(?P<ppn>.+?)\1',
+            webpage, 'ppn', group='ppn', default=None)
+
+        entries = []
+        for episode in re.finditer(
+                r'<div class="container-podcast-property">([\s\S]+?)(?:</div>\s*){3}',
+                webpage):
+            entries.append(self._extract_info(episode.group(1), ppn=ppn))
+
+        return self.playlist_result(entries, playlist_id, title)
@@ -6,8 +6,10 @@ import json

 from .common import InfoExtractor
 from ..compat import (
+    compat_parse_qs,
    compat_str,
    compat_urllib_parse_unquote,
+    compat_urllib_parse_urlparse,
 )
 from ..utils import (
    determine_ext,
@@ -62,6 +64,7 @@ class LBRYBaseIE(InfoExtractor):
            'description': stream_value.get('description'),
            'license': stream_value.get('license'),
            'timestamp': int_or_none(stream.get('timestamp')),
+            'release_timestamp': int_or_none(stream_value.get('release_time')),
            'tags': stream_value.get('tags'),
            'duration': int_or_none(media.get('duration')),
            'channel': try_get(signing_channel, lambda x: x['value']['title']),
@@ -94,6 +97,8 @@ class LBRYIE(LBRYBaseIE):
            'description': 'md5:f6cb5c704b332d37f5119313c2c98f51',
            'timestamp': 1595694354,
            'upload_date': '20200725',
+            'release_timestamp': 1595340697,
+            'release_date': '20200721',
            'width': 1280,
            'height': 720,
        }
@@ -108,6 +113,8 @@ class LBRYIE(LBRYBaseIE):
            'description': 'md5:661ac4f1db09f31728931d7b88807a61',
            'timestamp': 1591312601,
            'upload_date': '20200604',
+            'release_timestamp': 1591312421,
+            'release_date': '20200604',
            'tags': list,
            'duration': 2570,
            'channel': 'The LBRY Foundation',
@@ -189,17 +196,18 @@ class LBRYChannelIE(LBRYBaseIE):
    }]
    _PAGE_SIZE = 50

-    def _fetch_page(self, claim_id, url, page):
+    def _fetch_page(self, claim_id, url, params, page):
        page += 1
+        page_params = {
+            'channel_ids': [claim_id],
+            'claim_type': 'stream',
+            'no_totals': True,
+            'page': page,
+            'page_size': self._PAGE_SIZE,
+        }
+        page_params.update(params)
        result = self._call_api_proxy(
-            'claim_search', claim_id, {
-                'channel_ids': [claim_id],
-                'claim_type': 'stream',
-                'no_totals': True,
-                'page': page,
-                'page_size': self._PAGE_SIZE,
-                'stream_types': self._SUPPORTED_STREAM_TYPES,
-            }, 'page %d' % page)
+            'claim_search', claim_id, page_params, 'page %d' % page)
        for item in (result.get('items') or []):
            stream_claim_name = item.get('name')
            stream_claim_id = item.get('claim_id')
@@ -220,8 +228,31 @@ class LBRYChannelIE(LBRYBaseIE):
        result = self._resolve_url(
            'lbry://' + display_id, display_id, 'channel')
        claim_id = result['claim_id']
+        qs = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
+        content = qs.get('content', [None])[0]
+        params = {
+            'fee_amount': qs.get('fee_amount', ['>=0'])[0],
+            'order_by': {
+                'new': ['release_time'],
+                'top': ['effective_amount'],
+                'trending': ['trending_group', 'trending_mixed'],
+            }[qs.get('order', ['new'])[0]],
+            'stream_types': [content] if content in ['audio', 'video'] else self._SUPPORTED_STREAM_TYPES,
+        }
+        duration = qs.get('duration', [None])[0]
+        if duration:
+            params['duration'] = {
+                'long': '>=1200',
+                'short': '<=240',
+            }[duration]
+        language = qs.get('language', ['all'])[0]
+        if language != 'all':
+            languages = [language]
+            if language == 'en':
+                languages.append('none')
+            params['any_languages'] = languages
        entries = OnDemandPagedList(
-            functools.partial(self._fetch_page, claim_id, url),
+            functools.partial(self._fetch_page, claim_id, url, params),
            self._PAGE_SIZE)
        result_value = result.get('value') or {}
        return self.playlist_result(
@@ -4,7 +4,13 @@ from __future__ import unicode_literals
 import re

 from .common import InfoExtractor
-from ..utils import js_to_json
+from ..compat import compat_str
+from ..utils import (
+    ExtractorError,
+    int_or_none,
+    js_to_json,
+    str_or_none,
+)


 class LineTVIE(InfoExtractor):
@@ -88,3 +94,137 @@ class LineTVIE(InfoExtractor):
                           for thumbnail in video_info.get('thumbnails', {}).get('list', [])],
            'view_count': video_info.get('meta', {}).get('count'),
        }
+
+
+class LineLiveBaseIE(InfoExtractor):
+    _API_BASE_URL = 'https://live-api.line-apps.com/web/v4.0/channel/'
+
+    def _parse_broadcast_item(self, item):
+        broadcast_id = compat_str(item['id'])
+        title = item['title']
+        is_live = item.get('isBroadcastingNow')
+
+        thumbnails = []
+        for thumbnail_id, thumbnail_url in (item.get('thumbnailURLs') or {}).items():
+            if not thumbnail_url:
+                continue
+            thumbnails.append({
+                'id': thumbnail_id,
+                'url': thumbnail_url,
+            })
+
+        channel = item.get('channel') or {}
+        channel_id = str_or_none(channel.get('id'))
+
+        return {
+            'id': broadcast_id,
+            'title': self._live_title(title) if is_live else title,
+            'thumbnails': thumbnails,
+            'timestamp': int_or_none(item.get('createdAt')),
+            'channel': channel.get('name'),
+            'channel_id': channel_id,
+            'channel_url': 'https://live.line.me/channels/' + channel_id if channel_id else None,
+            'duration': int_or_none(item.get('archiveDuration')),
+            'view_count': int_or_none(item.get('viewerCount')),
+            'comment_count': int_or_none(item.get('chatCount')),
+            'is_live': is_live,
+        }
+
+
+class LineLiveIE(LineLiveBaseIE):
+    _VALID_URL = r'https?://live\.line\.me/channels/(?P<channel_id>\d+)/broadcast/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://live.line.me/channels/4867368/broadcast/16331360',
+        'md5': 'bc931f26bf1d4f971e3b0982b3fab4a3',
+        'info_dict': {
+            'id': '16331360',
+            'title': '振りコピ講座😙😙😙',
+            'ext': 'mp4',
+            'timestamp': 1617095132,
+            'upload_date': '20210330',
+            'channel': '白川ゆめか',
+            'channel_id': '4867368',
+            'view_count': int,
+            'comment_count': int,
+            'is_live': False,
+        }
+    }, {
+        # archiveStatus == 'DELETED'
+        'url': 'https://live.line.me/channels/4778159/broadcast/16378488',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        channel_id, broadcast_id = re.match(self._VALID_URL, url).groups()
+        broadcast = self._download_json(
+            self._API_BASE_URL + '%s/broadcast/%s' % (channel_id, broadcast_id),
+            broadcast_id)
+        item = broadcast['item']
+        info = self._parse_broadcast_item(item)
+        protocol = 'm3u8' if info['is_live'] else 'm3u8_native'
+        formats = []
+        for k, v in (broadcast.get(('live' if info['is_live'] else 'archived') + 'HLSURLs') or {}).items():
+            if not v:
+                continue
+            if k == 'abr':
+                formats.extend(self._extract_m3u8_formats(
+                    v, broadcast_id, 'mp4', protocol,
+                    m3u8_id='hls', fatal=False))
+                continue
+            f = {
+                'ext': 'mp4',
+                'format_id': 'hls-' + k,
+                'protocol': protocol,
+                'url': v,
+            }
+            if not k.isdigit():
+                f['vcodec'] = 'none'
+            formats.append(f)
+        if not formats:
+            archive_status = item.get('archiveStatus')
+            if archive_status != 'ARCHIVED':
+                raise ExtractorError('this video has been ' + archive_status.lower(), expected=True)
+        self._sort_formats(formats)
+        info['formats'] = formats
+        return info
+
+
+class LineLiveChannelIE(LineLiveBaseIE):
+    _VALID_URL = r'https?://live\.line\.me/channels/(?P<id>\d+)(?!/broadcast/\d+)(?:[/?&#]|$)'
+    _TEST = {
+        'url': 'https://live.line.me/channels/5893542',
+        'info_dict': {
+            'id': '5893542',
+            'title': 'いくらちゃん',
+            'description': 'md5:c3a4af801f43b2fac0b02294976580be',
+        },
+        'playlist_mincount': 29
+    }
+
+    def _archived_broadcasts_entries(self, archived_broadcasts, channel_id):
+        while True:
+            for row in (archived_broadcasts.get('rows') or []):
+                share_url = str_or_none(row.get('shareURL'))
+                if not share_url:
+                    continue
+                info = self._parse_broadcast_item(row)
+                info.update({
+                    '_type': 'url',
+                    'url': share_url,
+                    'ie_key': LineLiveIE.ie_key(),
+                })
+                yield info
+            if not archived_broadcasts.get('hasNextPage'):
+                return
+            archived_broadcasts = self._download_json(
+                self._API_BASE_URL + channel_id + '/archived_broadcasts',
+                channel_id, query={
+                    'lastId': info['id'],
+                })
+
+    def _real_extract(self, url):
+        channel_id = self._match_id(url)
+        channel = self._download_json(self._API_BASE_URL + channel_id, channel_id)
+        return self.playlist_result(
+            self._archived_broadcasts_entries(channel.get('archivedBroadcasts') or {}, channel_id),
+            channel_id, channel.get('title'), channel.get('information'))
@@ -38,8 +38,8 @@ class LinuxAcademyIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'What Is Data Science',
            'description': 'md5:c574a3c20607144fb36cb65bdde76c99',
-            'timestamp': 1607387907,
-            'upload_date': '20201208',
+            'timestamp': int,  # The timestamp and upload date changes
+            'upload_date': r're:\d+',
            'duration': 304,
        },
        'params': {
@@ -59,6 +59,16 @@ class LinuxAcademyIE(InfoExtractor):
        },
        'playlist_count': 41,
        'skip': 'Requires Linux Academy account credentials',
+    }, {
+        'url': 'https://linuxacademy.com/cp/modules/view/id/39',
+        'info_dict': {
+            'id': '39',
+            'title': 'Red Hat Certified Systems Administrator - RHCSA (EX200) Exam Prep  (legacy)',
+            'description': 'md5:0f1d3369e90c3fb14a79813b863c902f',
+            'duration': 89280,
+        },
+        'playlist_count': 73,
+        'skip': 'Requires Linux Academy account credentials',
    }]

    _AUTHORIZE_URL = 'https://login.linuxacademy.com/authorize'
@@ -102,7 +112,7 @@ class LinuxAcademyIE(InfoExtractor):
            'client_id': self._CLIENT_ID,
            'redirect_uri': self._ORIGIN_URL,
            'tenant': 'lacausers',
-            'connection': 'Username-Password-Authentication',
+            'connection': 'Username-Password-ACG-Proxy',
            'username': username,
            'password': password,
            'sso': 'true',
@@ -162,7 +172,7 @@ class LinuxAcademyIE(InfoExtractor):
        if course_id:
            module = self._parse_json(
                self._search_regex(
-                    r'window\.module\s*=\s*({.+?})\s*;', webpage, 'module'),
+                    r'window\.module\s*=\s*({(?:(?!};)[^"]|"([^"]|\\")*")+})\s*;', webpage, 'module'),
                item_id)
            entries = []
            chapter_number = None
@@ -0,0 +1,31 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class MaoriTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?maoritelevision\.com/shows/(?:[^/]+/)+(?P<id>[^/?&#]+)'
+    _TEST = {
+        'url': 'https://www.maoritelevision.com/shows/korero-mai/S01E054/korero-mai-series-1-episode-54',
+        'md5': '5ade8ef53851b6a132c051b1cd858899',
+        'info_dict': {
+            'id': '4774724855001',
+            'ext': 'mp4',
+            'title': 'Kōrero Mai, Series 1 Episode 54',
+            'upload_date': '20160226',
+            'timestamp': 1456455018,
+            'description': 'md5:59bde32fd066d637a1a55794c56d8dcb',
+            'uploader_id': '1614493167001',
+        },
+    }
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/1614493167001/HJlhIQhQf_default/index.html?videoId=%s'
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+        brightcove_id = self._search_regex(
+            r'data-main-video-id=["\'](\d+)', webpage, 'brightcove id')
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id,
+            'BrightcoveNew', brightcove_id)
@@ -5,6 +5,7 @@ from datetime import datetime
 import itertools
 import json
 import base64
+import re

 from .common import InfoExtractor
 from ..utils import (
@@ -68,7 +69,7 @@ class MildomBaseIE(InfoExtractor):
                self._DISPATCHER_CONFIG = self._parse_json(base64.b64decode(tmp['data']), 'initialization')
            except ExtractorError:
                self._DISPATCHER_CONFIG = self._download_json(
-                    'https://bookish-octo-barnacle.vercel.app/api/dispatcher_config', 'initialization',
+                    'https://bookish-octo-barnacle.vercel.app/api/mildom/dispatcher_config', 'initialization',
                    note='Downloading dispatcher_config fallback')
        return self._DISPATCHER_CONFIG

@@ -110,6 +111,7 @@ class MildomIE(MildomBaseIE):
        enterstudio = self._call_api(
            'https://cloudac.mildom.com/nonolive/gappserv/live/enterstudio', video_id,
            note='Downloading live metadata', query={'user_id': video_id})
+        result_video_id = enterstudio.get('log_id', video_id)

        title = try_get(
            enterstudio, (
@@ -128,7 +130,7 @@ class MildomIE(MildomBaseIE):
            ), compat_str)

        servers = self._call_api(
-            'https://cloudac.mildom.com/nonolive/gappserv/live/liveserver', video_id,
+            'https://cloudac.mildom.com/nonolive/gappserv/live/liveserver', result_video_id,
            note='Downloading live server list', query={
                'user_id': video_id,
                'live_server_type': 'hls',
@@ -139,7 +141,7 @@ class MildomIE(MildomBaseIE):
            'is_lhls': '0',
        })
        m3u8_url = update_url_query(servers['stream_server'] + '/%s_master.m3u8' % video_id, stream_query)
-        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', headers={
+        formats = self._extract_m3u8_formats(m3u8_url, result_video_id, 'mp4', headers={
            'Referer': 'https://www.mildom.com/',
            'Origin': 'https://www.mildom.com',
        }, note='Downloading m3u8 information')
@@ -150,13 +152,13 @@ class MildomIE(MildomBaseIE):
            parsed = parsed._replace(
                netloc='bookish-octo-barnacle.vercel.app',
                query=compat_urllib_parse_urlencode(stream_query, True),
-                path='/api' + parsed.path)
+                path='/api/mildom' + parsed.path)
            fmt['url'] = compat_urlparse.urlunparse(parsed)

        self._sort_formats(formats)

        return {
-            'id': video_id,
+            'id': result_video_id,
            'title': title,
            'description': description,
            'uploader': uploader,
@@ -172,9 +174,8 @@ class MildomVodIE(MildomBaseIE):
    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+)'

    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        m = self._VALID_URL_RE.match(url)
-        user_id = m.group('user_id')
+        m = re.match(self._VALID_URL, url)
+        user_id, video_id = m.group('user_id'), m.group('id')
        url = 'https://www.mildom.com/playback/%s/%s' % (user_id, video_id)

        webpage = self._download_webpage(url, video_id)
@@ -230,7 +231,7 @@ class MildomVodIE(MildomBaseIE):
            parsed = parsed._replace(
                netloc='bookish-octo-barnacle.vercel.app',
                query=compat_urllib_parse_urlencode(stream_query, True),
-                path='/api/vod2/proxy')
+                path='/api/mildom/vod2/proxy')
            fmt['url'] = compat_urlparse.urlunparse(parsed)

        self._sort_formats(formats)
@@ -1,15 +1,91 @@
 from __future__ import unicode_literals

-from .nhl import NHLBaseIE
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    determine_ext,
+    int_or_none,
+    parse_duration,
+    parse_iso8601,
+    try_get,
+)


-class MLBIE(NHLBaseIE):
+class MLBBaseIE(InfoExtractor):
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        video = self._download_video_data(display_id)
+        video_id = video['id']
+        title = video['title']
+        feed = self._get_feed(video)
+
+        formats = []
+        for playback in (feed.get('playbacks') or []):
+            playback_url = playback.get('url')
+            if not playback_url:
+                continue
+            name = playback.get('name')
+            ext = determine_ext(playback_url)
+            if ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    playback_url, video_id, 'mp4',
+                    'm3u8_native', m3u8_id=name, fatal=False))
+            else:
+                f = {
+                    'format_id': name,
+                    'url': playback_url,
+                }
+                mobj = re.search(r'_(\d+)K_(\d+)X(\d+)', name)
+                if mobj:
+                    f.update({
+                        'height': int(mobj.group(3)),
+                        'tbr': int(mobj.group(1)),
+                        'width': int(mobj.group(2)),
+                    })
+                mobj = re.search(r'_(\d+)x(\d+)_(\d+)_(\d+)K\.mp4', playback_url)
+                if mobj:
+                    f.update({
+                        'fps': int(mobj.group(3)),
+                        'height': int(mobj.group(2)),
+                        'tbr': int(mobj.group(4)),
+                        'width': int(mobj.group(1)),
+                    })
+                formats.append(f)
+        self._sort_formats(formats)
+
+        thumbnails = []
+        for cut in (try_get(feed, lambda x: x['image']['cuts'], list) or []):
+            src = cut.get('src')
+            if not src:
+                continue
+            thumbnails.append({
+                'height': int_or_none(cut.get('height')),
+                'url': src,
+                'width': int_or_none(cut.get('width')),
+            })
+
+        language = (video.get('language') or 'EN').lower()
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'description': video.get('description'),
+            'duration': parse_duration(feed.get('duration')),
+            'thumbnails': thumbnails,
+            'timestamp': parse_iso8601(video.get(self._TIMESTAMP_KEY)),
+            'subtitles': self._extract_mlb_subtitles(feed, language),
+        }
+
+
+class MLBIE(MLBBaseIE):
    _VALID_URL = r'''(?x)
                    https?://
-                        (?:[\da-z_-]+\.)*(?P<site>mlb)\.com/
+                        (?:[\da-z_-]+\.)*mlb\.com/
                        (?:
                            (?:
-                                (?:[^/]+/)*c-|
+                                (?:[^/]+/)*video/[^/]+/c-|
                                (?:
                                    shared/video/embed/(?:embed|m-internal-embed)\.html|
                                    (?:[^/]+/)+(?:play|index)\.jsp|
@@ -18,7 +94,6 @@ class MLBIE(NHLBaseIE):
                            (?P<id>\d+)
                        )
                    '''
-    _CONTENT_DOMAIN = 'content.mlb.com'
    _TESTS = [
        {
            'url': 'https://www.mlb.com/mariners/video/ackleys-spectacular-catch/c-34698933',
@@ -76,18 +151,6 @@ class MLBIE(NHLBaseIE):
                'thumbnail': r're:^https?://.*\.jpg$',
            },
        },
-        {
-            'url': 'https://www.mlb.com/news/blue-jays-kevin-pillar-goes-spidey-up-the-wall-to-rob-tim-beckham-of-a-homer/c-118550098',
-            'md5': 'e09e37b552351fddbf4d9e699c924d68',
-            'info_dict': {
-                'id': '75609783',
-                'ext': 'mp4',
-                'title': 'Must C: Pillar climbs for catch',
-                'description': '4/15/15: Blue Jays outfielder Kevin Pillar continues his defensive dominance by climbing the wall in left to rob Tim Beckham of a home run',
-                'timestamp': 1429139220,
-                'upload_date': '20150415',
-            }
-        },
        {
            'url': 'https://www.mlb.com/video/hargrove-homers-off-caldwell/c-1352023483?tid=67793694',
            'only_matching': True,
@@ -113,8 +176,92 @@ class MLBIE(NHLBaseIE):
            'url': 'http://mlb.mlb.com/shared/video/embed/m-internal-embed.html?content_id=75609783&property=mlb&autoplay=true&hashmode=false&siteSection=mlb/multimedia/article_118550098/article_embed&club=mlb',
            'only_matching': True,
        },
-        {
-            'url': 'https://www.mlb.com/cut4/carlos-gomez-borrowed-sunglasses-from-an-as-fan/c-278912842',
-            'only_matching': True,
-        }
    ]
+    _TIMESTAMP_KEY = 'date'
+
+    @staticmethod
+    def _get_feed(video):
+        return video
+
+    @staticmethod
+    def _extract_mlb_subtitles(feed, language):
+        subtitles = {}
+        for keyword in (feed.get('keywordsAll') or []):
+            keyword_type = keyword.get('type')
+            if keyword_type and keyword_type.startswith('closed_captions_location_'):
+                cc_location = keyword.get('value')
+                if cc_location:
+                    subtitles.setdefault(language, []).append({
+                        'url': cc_location,
+                    })
+        return subtitles
+
+    def _download_video_data(self, display_id):
+        return self._download_json(
+            'http://content.mlb.com/mlb/item/id/v1/%s/details/web-v1.json' % display_id,
+            display_id)
+
+
+class MLBVideoIE(MLBBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?mlb\.com/(?:[^/]+/)*video/(?P<id>[^/?&#]+)'
+    _TEST = {
+        'url': 'https://www.mlb.com/mariners/video/ackley-s-spectacular-catch-c34698933',
+        'md5': '632358dacfceec06bad823b83d21df2d',
+        'info_dict': {
+            'id': 'c04a8863-f569-42e6-9f87-992393657614',
+            'ext': 'mp4',
+            'title': "Ackley's spectacular catch",
+            'description': 'md5:7f5a981eb4f3cbc8daf2aeffa2215bf0',
+            'duration': 66,
+            'timestamp': 1405995000,
+            'upload_date': '20140722',
+            'thumbnail': r're:^https?://.+',
+        },
+    }
+    _TIMESTAMP_KEY = 'timestamp'
+
+    @classmethod
+    def suitable(cls, url):
+        return False if MLBIE.suitable(url) else super(MLBVideoIE, cls).suitable(url)
+
+    @staticmethod
+    def _get_feed(video):
+        return video['feeds'][0]
+
+    @staticmethod
+    def _extract_mlb_subtitles(feed, language):
+        subtitles = {}
+        for cc_location in (feed.get('closedCaptions') or []):
+            subtitles.setdefault(language, []).append({
+                'url': cc_location,
+            })
+
+    def _download_video_data(self, display_id):
+        # https://www.mlb.com/data-service/en/videos/[SLUG]
+        return self._download_json(
+            'https://fastball-gateway.mlb.com/graphql',
+            display_id, query={
+                'query': '''{
+  mediaPlayback(ids: "%s") {
+    description
+    feeds(types: CMS) {
+      closedCaptions
+      duration
+      image {
+        cuts {
+          width
+          height
+          src
+        }
+      }
+      playbacks {
+        name
+        url
+      }
+    }
+    id
+    timestamp
+    title
+  }
+}''' % display_id,
+            })['data']['mediaPlayback'][0]
@@ -14,6 +14,7 @@ from ..utils import (
    fix_xml_ampersands,
    float_or_none,
    HEADRequest,
+    int_or_none,
    RegexNotFoundError,
    sanitized_Request,
    strip_or_none,
@@ -176,6 +177,22 @@ class MTVServicesInfoExtractor(InfoExtractor):
            raise ExtractorError('Could not find video title')
        title = title.strip()

+        series = find_xpath_attr(
+            itemdoc, './/{http://search.yahoo.com/mrss/}category',
+            'scheme', 'urn:mtvn:franchise')
+        season = find_xpath_attr(
+            itemdoc, './/{http://search.yahoo.com/mrss/}category',
+            'scheme', 'urn:mtvn:seasonN')
+        episode = find_xpath_attr(
+            itemdoc, './/{http://search.yahoo.com/mrss/}category',
+            'scheme', 'urn:mtvn:episodeN')
+        series = series.text if series is not None else None
+        season = season.text if season is not None else None
+        episode = episode.text if episode is not None else None
+        if season and episode:
+            # episode number includes season, so remove it
+            episode = re.sub(r'^%s' % season, '', episode)
+
        # This a short id that's used in the webpage urls
        mtvn_id = None
        mtvn_id_node = find_xpath_attr(itemdoc, './/{http://search.yahoo.com/mrss/}category',
@@ -201,6 +218,9 @@ class MTVServicesInfoExtractor(InfoExtractor):
            'description': description,
            'duration': float_or_none(content_el.attrib.get('duration')),
            'timestamp': timestamp,
+            'series': series,
+            'season_number': int_or_none(season),
+            'episode_number': int_or_none(episode),
        }

    def _get_feed_query(self, uri):
@@ -255,7 +275,9 @@ class MTVServicesInfoExtractor(InfoExtractor):

    @staticmethod
    def _extract_child_with_type(parent, t):
-        return next(c for c in parent['children'] if c.get('type') == t)
+        for c in parent['children']:
+            if c.get('type') == t:
+                return c

    def _extract_mgid(self, webpage):
        try:
@@ -286,7 +308,8 @@ class MTVServicesInfoExtractor(InfoExtractor):
            data = self._parse_json(self._search_regex(
                r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
            main_container = self._extract_child_with_type(data, 'MainContainer')
-            video_player = self._extract_child_with_type(main_container, 'VideoPlayer')
+            ab_testing = self._extract_child_with_type(main_container, 'ABTesting')
+            video_player = self._extract_child_with_type(ab_testing or main_container, 'VideoPlayer')
            mgid = video_player['props']['media']['video']['config']['uri']

        return mgid
@@ -320,7 +343,7 @@ class MTVServicesEmbeddedIE(MTVServicesInfoExtractor):
    @staticmethod
    def _extract_url(webpage):
        mobj = re.search(
-            r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//media.mtvnservices.com/embed/.+?)\1', webpage)
+            r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//media\.mtvnservices\.com/embed/.+?)\1', webpage)
        if mobj:
            return mobj.group('url')

@@ -483,3 +506,152 @@ class MTVDEIE(MTVServicesInfoExtractor):
            'arcEp': 'mtv.de',
            'mgid': uri,
        }
+
+
+class MTVItaliaIE(MTVServicesInfoExtractor):
+    IE_NAME = 'mtv.it'
+    _VALID_URL = r'https?://(?:www\.)?mtv\.it/(?:episodi|video|musica)/(?P<id>[0-9a-z]+)'
+    _TESTS = [{
+        'url': 'http://www.mtv.it/episodi/24bqab/mario-una-serie-di-maccio-capatonda-cavoli-amario-episodio-completo-S1-E1',
+        'info_dict': {
+            'id': '0f0fc78e-45fc-4cce-8f24-971c25477530',
+            'ext': 'mp4',
+            'title': 'Cavoli amario (episodio completo)',
+            'description': 'md5:4962bccea8fed5b7c03b295ae1340660',
+            'series': 'Mario - Una Serie Di Maccio Capatonda',
+            'season_number': 1,
+            'episode_number': 1,
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }]
+    _GEO_COUNTRIES = ['IT']
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'
+
+    def _get_feed_query(self, uri):
+        return {
+            'arcEp': 'mtv.it',
+            'mgid': uri,
+        }
+
+
+class MTVItaliaProgrammaIE(MTVItaliaIE):
+    IE_NAME = 'mtv.it:programma'
+    _VALID_URL = r'https?://(?:www\.)?mtv\.it/(?:programmi|playlist)/(?P<id>[0-9a-z]+)'
+    _TESTS = [{
+        # program page: general
+        'url': 'http://www.mtv.it/programmi/s2rppv/mario-una-serie-di-maccio-capatonda',
+        'info_dict': {
+            'id': 'a6f155bc-8220-4640-aa43-9b95f64ffa3d',
+            'title': 'Mario - Una Serie Di Maccio Capatonda',
+            'description': 'md5:72fbffe1f77ccf4e90757dd4e3216153',
+        },
+        'playlist_count': 2,
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        # program page: specific season
+        'url': 'http://www.mtv.it/programmi/d9ncjf/mario-una-serie-di-maccio-capatonda-S2',
+        'info_dict': {
+            'id': '4deeb5d8-f272-490c-bde2-ff8d261c6dd1',
+            'title': 'Mario - Una Serie Di Maccio Capatonda - Stagione 2',
+        },
+        'playlist_count': 34,
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        # playlist page + redirect
+        'url': 'http://www.mtv.it/playlist/sexy-videos/ilctal',
+        'info_dict': {
+            'id': 'dee8f9ee-756d-493b-bf37-16d1d2783359',
+            'title': 'Sexy Videos',
+        },
+        'playlist_mincount': 145,
+        'params': {
+            'skip_download': True,
+        },
+    }]
+    _GEO_COUNTRIES = ['IT']
+    _FEED_URL = 'http://www.mtv.it/feeds/triforce/manifest/v8'
+
+    def _get_entries(self, title, url):
+        while True:
+            pg = self._search_regex(r'/(\d+)$', url, 'entries', '1')
+            entries = self._download_json(url, title, 'page %s' % pg)
+            url = try_get(
+                entries, lambda x: x['result']['nextPageURL'], compat_str)
+            entries = try_get(
+                entries, (
+                    lambda x: x['result']['data']['items'],
+                    lambda x: x['result']['data']['seasons']),
+                list)
+            for entry in entries or []:
+                if entry.get('canonicalURL'):
+                    yield self.url_result(entry['canonicalURL'])
+            if not url:
+                break
+
+    def _real_extract(self, url):
+        query = {'url': url}
+        info_url = update_url_query(self._FEED_URL, query)
+        video_id = self._match_id(url)
+        info = self._download_json(info_url, video_id).get('manifest')
+
+        redirect = try_get(
+            info, lambda x: x['newLocation']['url'], compat_str)
+        if redirect:
+            return self.url_result(redirect)
+
+        title = info.get('title')
+        video_id = try_get(
+            info, lambda x: x['reporting']['itemId'], compat_str)
+        parent_id = try_get(
+            info, lambda x: x['reporting']['parentId'], compat_str)
+
+        playlist_url = current_url = None
+        for z in (info.get('zones') or {}).values():
+            if z.get('moduleName') in ('INTL_M304', 'INTL_M209'):
+                info_url = z.get('feed')
+            if z.get('moduleName') in ('INTL_M308', 'INTL_M317'):
+                playlist_url = playlist_url or z.get('feed')
+            if z.get('moduleName') in ('INTL_M300',):
+                current_url = current_url or z.get('feed')
+
+        if not info_url:
+            raise ExtractorError('No info found')
+
+        if video_id == parent_id:
+            video_id = self._search_regex(
+                r'([^\/]+)/[^\/]+$', info_url, 'video_id')
+
+        info = self._download_json(info_url, video_id, 'Show infos')
+        info = try_get(info, lambda x: x['result']['data'], dict)
+        title = title or try_get(
+            info, (
+                lambda x: x['title'],
+                lambda x: x['headline']),
+            compat_str)
+        description = try_get(info, lambda x: x['content'], compat_str)
+
+        if current_url:
+            season = try_get(
+                self._download_json(playlist_url, video_id, 'Seasons info'),
+                lambda x: x['result']['data'], dict)
+            current = try_get(
+                season, lambda x: x['currentSeason'], compat_str)
+            seasons = try_get(
+                season, lambda x: x['seasons'], list) or []
+
+            if current in [s.get('eTitle') for s in seasons]:
+                playlist_url = current_url
+
+        title = re.sub(
+            r'[-|]\s*(?:mtv\s*italia|programma|playlist)',
+            '', title, flags=re.IGNORECASE).strip()
+
+        return self.playlist_result(
+            self._get_entries(title, playlist_url),
+            video_id, title, description)
@@ -10,6 +10,7 @@ from .adobepass import AdobePassIE
 from ..compat import compat_urllib_parse_unquote
 from ..utils import (
    int_or_none,
+    parse_age_limit,
    parse_duration,
    smuggle_url,
    try_get,
@@ -18,7 +19,7 @@ from ..utils import (
 )


-class NBCIE(AdobePassIE):
+class NBCIE(ThePlatformIE):
    _VALID_URL = r'https?(?P<permalink>://(?:www\.)?nbc\.com/(?:classic-tv/)?[^/]+/video/[^/]+/(?P<id>n?\d+))'

    _TESTS = [
@@ -132,7 +133,9 @@ class NBCIE(AdobePassIE):
            'manifest': 'm3u',
        }
        video_id = video_data['mpxGuid']
-        title = video_data['secondaryTitle']
+        tp_path = 'NnzsPC/media/guid/%s/%s' % (video_data.get('mpxAccountId') or '2410887629', video_id)
+        tpm = self._download_theplatform_metadata(tp_path, video_id)
+        title = tpm.get('title') or video_data.get('secondaryTitle')
        if video_data.get('locked'):
            resource = self._get_mvpd_resource(
                video_data.get('resourceId') or 'nbcentertainment',
@@ -142,18 +145,40 @@ class NBCIE(AdobePassIE):
        theplatform_url = smuggle_url(update_url_query(
            'http://link.theplatform.com/s/NnzsPC/media/guid/%s/%s' % (video_data.get('mpxAccountId') or '2410887629', video_id),
            query), {'force_smil_url': True})
+
+        # Empty string or 0 can be valid values for these. So the check must be `is None`
+        description = video_data.get('description')
+        if description is None:
+            description = tpm.get('description')
+        episode_number = int_or_none(video_data.get('episodeNumber'))
+        if episode_number is None:
+            episode_number = int_or_none(tpm.get('nbcu$airOrder'))
+        rating = video_data.get('rating')
+        if rating is None:
+            try_get(tpm, lambda x: x['ratings'][0]['rating'])
+        season_number = int_or_none(video_data.get('seasonNumber'))
+        if season_number is None:
+            season_number = int_or_none(tpm.get('nbcu$seasonNumber'))
+        series = video_data.get('seriesShortTitle')
+        if series is None:
+            series = tpm.get('nbcu$seriesShortTitle')
+        tags = video_data.get('keywords')
+        if tags is None or len(tags) == 0:
+            tags = tpm.get('keywords')
+
        return {
            '_type': 'url_transparent',
+            'age_limit': parse_age_limit(rating),
+            'description': description,
+            'episode': title,
+            'episode_number': episode_number,
            'id': video_id,
+            'ie_key': 'ThePlatform',
+            'season_number': season_number,
+            'series': series,
+            'tags': tags,
            'title': title,
            'url': theplatform_url,
-            'description': video_data.get('description'),
-            'tags': video_data.get('keywords'),
-            'season_number': int_or_none(video_data.get('seasonNumber')),
-            'episode_number': int_or_none(video_data.get('episodeNumber')),
-            'episode': title,
-            'series': video_data.get('seriesShortTitle'),
-            'ie_key': 'ThePlatform',
        }


@@ -0,0 +1,197 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+    ExtractorError,
+    parse_iso8601,
+    try_get,
+    urljoin,
+)
+
+
+class NebulaIE(InfoExtractor):
+
+    _VALID_URL = r'https?://(?:www\.)?watchnebula\.com/videos/(?P<id>[-\w]+)'
+    _TESTS = [
+        {
+            'url': 'https://watchnebula.com/videos/that-time-disney-remade-beauty-and-the-beast',
+            'md5': 'fe79c4df8b3aa2fea98a93d027465c7e',
+            'info_dict': {
+                'id': '5c271b40b13fd613090034fd',
+                'ext': 'mp4',
+                'title': 'That Time Disney Remade Beauty and the Beast',
+                'description': 'Note: this video was originally posted on YouTube with the sponsor read included. We weren’t able to remove it without reducing video quality, so it’s presented here in its original context.',
+                'upload_date': '20180731',
+                'timestamp': 1533009600,
+                'channel': 'Lindsay Ellis',
+                'uploader': 'Lindsay Ellis',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+        {
+            'url': 'https://watchnebula.com/videos/the-logistics-of-d-day-landing-craft-how-the-allies-got-ashore',
+            'md5': '6d4edd14ce65720fa63aba5c583fb328',
+            'info_dict': {
+                'id': '5e7e78171aaf320001fbd6be',
+                'ext': 'mp4',
+                'title': 'Landing Craft - How The Allies Got Ashore',
+                'description': r're:^In this episode we explore the unsung heroes of D-Day, the landing craft.',
+                'upload_date': '20200327',
+                'timestamp': 1585348140,
+                'channel': 'The Logistics of D-Day',
+                'uploader': 'The Logistics of D-Day',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+        {
+            'url': 'https://watchnebula.com/videos/money-episode-1-the-draw',
+            'md5': '8c7d272910eea320f6f8e6d3084eecf5',
+            'info_dict': {
+                'id': '5e779ebdd157bc0001d1c75a',
+                'ext': 'mp4',
+                'title': 'Episode 1: The Draw',
+                'description': r'contains:There’s free money on offer… if the players can all work together.',
+                'upload_date': '20200323',
+                'timestamp': 1584980400,
+                'channel': 'Tom Scott Presents: Money',
+                'uploader': 'Tom Scott Presents: Money',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+    ]
+    _NETRC_MACHINE = 'watchnebula'
+
+    def _retrieve_nebula_auth(self, video_id):
+        """
+        Log in to Nebula, and returns a Nebula API token
+        """
+
+        username, password = self._get_login_info()
+        if not (username and password):
+            self.raise_login_required()
+
+        self.report_login()
+        data = json.dumps({'email': username, 'password': password}).encode('utf8')
+        response = self._download_json(
+            'https://api.watchnebula.com/api/v1/auth/login/',
+            data=data, fatal=False, video_id=video_id,
+            headers={
+                'content-type': 'application/json',
+                # Submitting the 'sessionid' cookie always causes a 403 on auth endpoint
+                'cookie': ''
+            },
+            note='Authenticating to Nebula with supplied credentials',
+            errnote='Authentication failed or rejected')
+        if not response or not response.get('key'):
+            self.raise_login_required()
+        return response['key']
+
+    def _retrieve_zype_api_key(self, page_url, display_id):
+        """
+        Retrieves the Zype API key
+        """
+
+        # Find the js that has the API key from the webpage and download it
+        webpage = self._download_webpage(page_url, video_id=display_id)
+        main_script_relpath = self._search_regex(
+            r'<script[^>]*src="(?P<script_relpath>[^"]*main.[0-9a-f]*.chunk.js)"[^>]*>', webpage,
+            group='script_relpath', name='script relative path', fatal=True)
+        main_script_abspath = urljoin(page_url, main_script_relpath)
+        main_script = self._download_webpage(main_script_abspath, video_id=display_id,
+                                             note='Retrieving Zype API key')
+
+        api_key = self._search_regex(
+            r'REACT_APP_ZYPE_API_KEY\s*:\s*"(?P<api_key>[\w-]*)"', main_script,
+            group='api_key', name='API key', fatal=True)
+
+        return api_key
+
+    def _call_zype_api(self, path, params, video_id, api_key, note):
+        """
+        A helper for making calls to the Zype API.
+        """
+        query = {'api_key': api_key, 'per_page': 1}
+        query.update(params)
+        return self._download_json('https://api.zype.com' + path, video_id, query=query, note=note)
+
+    def _call_nebula_api(self, path, video_id, access_token, note):
+        """
+        A helper for making calls to the Nebula API.
+        """
+        return self._download_json('https://api.watchnebula.com/api/v1' + path, video_id, headers={
+            'Authorization': 'Token {access_token}'.format(access_token=access_token)
+        }, note=note)
+
+    def _fetch_zype_access_token(self, video_id, nebula_token):
+        user_object = self._call_nebula_api('/auth/user/', video_id, nebula_token, note='Retrieving Zype access token')
+        access_token = try_get(user_object, lambda x: x['zype_auth_info']['access_token'], compat_str)
+        if not access_token:
+            if try_get(user_object, lambda x: x['is_subscribed'], bool):
+                # TODO: Reimplement the same Zype token polling the Nebula frontend implements
+                # see https://github.com/ytdl-org/youtube-dl/pull/24805#issuecomment-749231532
+                raise ExtractorError(
+                    'Unable to extract Zype access token from Nebula API authentication endpoint. '
+                    'Open an arbitrary video in a browser with this account to generate a token',
+                    expected=True)
+            raise ExtractorError('Unable to extract Zype access token from Nebula API authentication endpoint')
+        return access_token
+
+    def _extract_channel_title(self, video_meta):
+        # TODO: Implement the API calls giving us the channel list,
+        # so that we can do the title lookup and then figure out the channel URL
+        categories = video_meta.get('categories', []) if video_meta else []
+        # the channel name is the value of the first category
+        for category in categories:
+            if category.get('value'):
+                return category['value'][0]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        nebula_token = self._retrieve_nebula_auth(display_id)
+        api_key = self._retrieve_zype_api_key(url, display_id)
+
+        response = self._call_zype_api('/videos', {'friendly_title': display_id},
+                                       display_id, api_key, note='Retrieving metadata from Zype')
+        if len(response.get('response') or []) != 1:
+            raise ExtractorError('Unable to find video on Zype API')
+        video_meta = response['response'][0]
+
+        video_id = video_meta['_id']
+        zype_access_token = self._fetch_zype_access_token(display_id, nebula_token=nebula_token)
+
+        channel_title = self._extract_channel_title(video_meta)
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            '_type': 'url_transparent',
+            'ie_key': 'Zype',
+            'url': 'https://player.zype.com/embed/%s.html?access_token=%s' % (video_id, zype_access_token),
+            'title': video_meta.get('title'),
+            'description': video_meta.get('description'),
+            'timestamp': parse_iso8601(video_meta.get('published_at')),
+            'thumbnails': [
+                {
+                    'id': tn.get('name'),  # this appears to be null
+                    'url': tn['url'],
+                    'width': tn.get('width'),
+                    'height': tn.get('height'),
+                } for tn in video_meta.get('thumbnails', [])],
+            'duration': video_meta.get('duration'),
+            'channel': channel_title,
+            'uploader': channel_title,  # we chose uploader = channel name
+            # TODO: uploader_url, channel_id, channel_url
+        }
@@ -0,0 +1,144 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+from ..utils import (
+    try_get,
+    unified_strdate,
+    unified_timestamp
+)
+
+
+class NFHSNetworkIE(InfoExtractor):
+    IE_NAME = 'NFHSNetwork'
+    _VALID_URL = r'https?://(?:www\.)?nfhsnetwork\.com/events/[\w-]+/(?P<id>(?:gam|evt|dd|)?[\w\d]{0,10})'
+    _TESTS = [{
+        # Auto-generated two-team sport (pixellot)
+        'url': 'https://www.nfhsnetwork.com/events/rockford-high-school-rockford-mi/gamcf7e54cfbc',
+        'info_dict': {
+            'id': 'gamcf7e54cfbc',
+            'ext': 'mp4',
+            'title': 'Rockford vs Spring Lake - Girls Varsity Lacrosse 03/27/2021',
+            'uploader': 'MHSAA - Michigan: Rockford High School, Rockford, MI',
+            'uploader_id': 'cd2622cf76',
+            'uploader_url': 'https://www.nfhsnetwork.com/schools/rockford-high-school-rockford-mi',
+            'location': 'Rockford, Michigan',
+            'timestamp': 1616859000,
+            'upload_date': '20210327'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Non-sport activity with description
+        'url': 'https://www.nfhsnetwork.com/events/limon-high-school-limon-co/evt4a30e3726c',
+        'info_dict': {
+            'id': 'evt4a30e3726c',
+            'ext': 'mp4',
+            'title': 'Drama Performance Limon High School vs. Limon High School - 12/13/2020',
+            'description': 'Join the broadcast of the Limon High School Musical Performance at 2 PM.',
+            'uploader': 'CHSAA: Limon High School, Limon, CO',
+            'uploader_id': '7d2d121332',
+            'uploader_url': 'https://www.nfhsnetwork.com/schools/limon-high-school-limon-co',
+            'location': 'Limon, Colorado',
+            'timestamp': 1607893200,
+            'upload_date': '20201213'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Postseason game
+        'url': 'https://www.nfhsnetwork.com/events/nfhs-network-special-events/dd8de71d45',
+        'info_dict': {
+            'id': 'dd8de71d45',
+            'ext': 'mp4',
+            'title': '2015 UA Holiday Classic Tournament: National Division  - 12/26/2015',
+            'uploader': 'SoCal Sports Productions',
+            'uploader_id': '063dba0150',
+            'uploader_url': 'https://www.nfhsnetwork.com/affiliates/socal-sports-productions',
+            'location': 'San Diego, California',
+            'timestamp': 1451187000,
+            'upload_date': '20151226'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Video with no broadcasts object
+        'url': 'https://www.nfhsnetwork.com/events/wiaa-wi/9aa2f92f82',
+        'info_dict': {
+            'id': '9aa2f92f82',
+            'ext': 'mp4',
+            'title': 'Competitive Equity  - 01/21/2015',
+            'description': 'Committee members discuss points of their research regarding a competitive equity plan',
+            'uploader': 'WIAA - Wisconsin: Wisconsin Interscholastic Athletic Association',
+            'uploader_id': 'a49f7d1002',
+            'uploader_url': 'https://www.nfhsnetwork.com/associations/wiaa-wi',
+            'location': 'Stevens Point, Wisconsin',
+            'timestamp': 1421856000,
+            'upload_date': '20150121'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }
+    ]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        data = self._download_json(
+            'https://cfunity.nfhsnetwork.com/v2/game_or_event/' + video_id,
+            video_id)
+        publisher = data.get('publishers')[0]  # always exists
+        broadcast = (publisher.get('broadcasts') or publisher.get('vods'))[0]  # some (older) videos don't have a broadcasts object
+        uploader = publisher.get('formatted_name') or publisher.get('name')
+        uploaderID = publisher.get('publisher_key')
+        pubType = publisher.get('type')
+        uploaderPrefix = (
+            "schools" if pubType == "school"
+            else "associations" if "association" in pubType
+            else "affiliates" if (pubType == "publisher" or pubType == "affiliate")
+            else "schools")
+        uploaderPage = 'https://www.nfhsnetwork.com/%s/%s' % (uploaderPrefix, publisher.get('slug'))
+        location = '%s, %s' % (data.get('city'), data.get('state_name'))
+        description = broadcast.get('description')
+        isLive = broadcast.get('on_air') or broadcast.get('status') == 'on_air' or False
+
+        timestamp = unified_timestamp(data.get('local_start_time'))
+        upload_date = unified_strdate(data.get('local_start_time'))
+
+        title = (
+            self._og_search_title(webpage)
+            or self._html_search_regex(r'<h1 class="sr-hidden">(.*?)</h1>', webpage, 'title'))
+        title = title.split('|')[0].strip()
+
+        video_type = 'broadcasts' if isLive else 'vods'
+        key = broadcast.get('key') if isLive else try_get(publisher, lambda x: x['vods'][0]['key'])
+        m3u8_url = self._download_json(
+            'https://cfunity.nfhsnetwork.com/v2/%s/%s/url' % (video_type, key),
+            video_id).get('video_url')
+
+        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', live=isLive)
+        self._sort_formats(formats, ['res', 'tbr'])
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'description': description,
+            'timestamp': timestamp,
+            'uploader': uploader,
+            'uploader_id': uploaderID,
+            'uploader_url': uploaderPage,
+            'location': location,
+            'upload_date': upload_date,
+            'is_live': isLive
+        }
@@ -8,6 +8,7 @@ import datetime
 from .common import InfoExtractor
 from ..postprocessor.ffmpeg import FFmpegPostProcessor
 from ..compat import (
+    compat_str,
    compat_parse_qs,
    compat_urllib_parse_urlparse,
 )
@@ -20,6 +21,7 @@ from ..utils import (
    parse_duration,
    parse_iso8601,
    PostProcessingError,
+    str_or_none,
    remove_start,
    try_get,
    unified_timestamp,
@@ -34,7 +36,7 @@ class NiconicoIE(InfoExtractor):

    _TESTS = [{
        'url': 'http://www.nicovideo.jp/watch/sm22312215',
-        'md5': 'd1a75c0823e2f629128c43e1212760f9',
+        'md5': 'a5bad06f1347452102953f323c69da34s',
        'info_dict': {
            'id': 'sm22312215',
            'ext': 'mp4',
@@ -203,7 +205,7 @@ class NiconicoIE(InfoExtractor):
            'data-api-data="([^"]+)"', webpage,
            'API data', default='{}'), video_id)

-        session_api_data = try_get(api_data, lambda x: x['video']['dmcInfo']['session_api'])
+        session_api_data = try_get(api_data, lambda x: x['media']['delivery']['movie']['session'])
        session_api_endpoint = try_get(session_api_data, lambda x: x['urls'][0])

        # ping
@@ -220,7 +222,7 @@ class NiconicoIE(InfoExtractor):
        yesno = lambda x: 'yes' if x else 'no'

        # m3u8 (encryption)
-        if 'encryption' in try_get(api_data, lambda x: x['video']['dmcInfo']) or {}:
+        if 'encryption' in (try_get(api_data, lambda x: x['media']['delivery']['movie']) or {}):
            protocol = 'm3u8'
            session_api_http_parameters = {
                'parameters': {
@@ -244,8 +246,8 @@ class NiconicoIE(InfoExtractor):
            session_api_http_parameters = {
                'parameters': {
                    'http_output_download_parameters': {
-                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
-                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                        'use_ssl': yesno(session_api_endpoint['isSsl']),
+                        'use_well_known_port': yesno(session_api_endpoint['isWellKnownPort']),
                    }
                }
            }
@@ -258,15 +260,15 @@ class NiconicoIE(InfoExtractor):
            data=json.dumps({
                'session': {
                    'client_info': {
-                        'player_id': session_api_data.get('player_id'),
+                        'player_id': session_api_data.get('playerId'),
                    },
                    'content_auth': {
-                        'auth_type': try_get(session_api_data, lambda x: x['auth_types'][session_api_data['protocols'][0]]),
-                        'content_key_timeout': session_api_data.get('content_key_timeout'),
+                        'auth_type': try_get(session_api_data, lambda x: x['authTypes'][session_api_data['protocols'][0]]),
+                        'content_key_timeout': session_api_data.get('contentKeyTimeout'),
                        'service_id': 'nicovideo',
-                        'service_user_id': session_api_data.get('service_user_id')
+                        'service_user_id': session_api_data.get('serviceUserId')
                    },
-                    'content_id': session_api_data.get('content_id'),
+                    'content_id': session_api_data.get('contentId'),
                    'content_src_id_sets': [{
                        'content_src_ids': [{
                            'src_id_to_mux': {
@@ -279,7 +281,7 @@ class NiconicoIE(InfoExtractor):
                    'content_uri': '',
                    'keep_method': {
                        'heartbeat': {
-                            'lifetime': session_api_data.get('heartbeat_lifetime')
+                            'lifetime': session_api_data.get('heartbeatLifetime')
                        }
                    },
                    'priority': session_api_data.get('priority'),
@@ -289,7 +291,7 @@ class NiconicoIE(InfoExtractor):
                            'http_parameters': session_api_http_parameters
                        }
                    },
-                    'recipe_id': session_api_data.get('recipe_id'),
+                    'recipe_id': session_api_data.get('recipeId'),
                    'session_operation_auth': {
                        'session_operation_auth_by_signature': {
                            'signature': session_api_data.get('signature'),
@@ -308,7 +310,7 @@ class NiconicoIE(InfoExtractor):
            'url': session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT',
            'data': json.dumps(session_response['data']),
            # interval, convert milliseconds to seconds, then halve to make a buffer.
-            'interval': float_or_none(session_api_data.get('heartbeat_lifetime'), scale=2000),
+            'interval': float_or_none(session_api_data.get('heartbeatLifetime'), scale=2000),
        }

        return info_dict, heartbeat_info_dict
@@ -327,15 +329,17 @@ class NiconicoIE(InfoExtractor):
        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
        vdict = parse_format_id(video_quality['id'])
        adict = parse_format_id(audio_quality['id'])
-        resolution = video_quality.get('resolution', {'height': vdict.get('res')})
+        resolution = try_get(video_quality, lambda x: x['metadata']['resolution'], dict) or {'height': vdict.get('res')}
+        vbr = try_get(video_quality, lambda x: x['metadata']['bitrate'], float)

        return {
            'url': '%s:%s/%s/%s' % (protocol, video_id, video_quality['id'], audio_quality['id']),
            'format_id': format_id,
+            'format_note': 'DMC %s' % try_get(video_quality, lambda x: x['metadata']['label'], compat_str),
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'vcodec': vdict.get('codec'),
            'acodec': adict.get('codec'),
-            'vbr': float_or_none(video_quality.get('bitrate'), 1000) or float_or_none(vdict.get('br')),
+            'vbr': float_or_none(vbr, 1000) or float_or_none(vdict.get('br')),
            'abr': float_or_none(audio_quality.get('bitrate'), 1000) or float_or_none(adict.get('br')),
            'height': int_or_none(resolution.get('height', vdict.get('res'))),
            'width': int_or_none(resolution.get('width')),
@@ -394,92 +398,93 @@ class NiconicoIE(InfoExtractor):
        formats = []

        # Get HTML5 videos info
-        try:
-            dmc_info = api_data['video']['dmcInfo']
-        except KeyError:
-            raise ExtractorError('The video can\'t downloaded.',
-                                 expected=True)
+        quality_info = try_get(api_data, lambda x: x['media']['delivery']['movie'])
+        if not quality_info:
+            raise ExtractorError('The video can\'t downloaded.', expected=True)

-        quality_info = dmc_info.get('quality')
        for audio_quality in quality_info.get('audios') or {}:
            for video_quality in quality_info.get('videos') or {}:
-                if not audio_quality.get('available') or not video_quality.get('available'):
+                if not audio_quality.get('isAvailable') or not video_quality.get('isAvailable'):
                    continue
                formats.append(self._extract_format_for_quality(
                    api_data, video_id, audio_quality, video_quality))

        # Get flv/swf info
+        timestamp = None
        video_real_url = try_get(api_data, lambda x: x['video']['smileInfo']['url'])
-        is_economy = video_real_url.endswith('low')
+        if not video_real_url:
+            self.report_warning('Unable to obtain smile video information')
+        else:
+            is_economy = video_real_url.endswith('low')

-        if is_economy:
-            self.report_warning('Site is currently in economy mode! You will only have access to lower quality streams')
+            if is_economy:
+                self.report_warning('Site is currently in economy mode! You will only have access to lower quality streams')

-        # Invoking ffprobe to determine resolution
-        pp = FFmpegPostProcessor(self._downloader)
-        cookies = self._get_cookies('https://nicovideo.jp').output(header='', sep='; path=/; domain=nicovideo.jp;\n')
+            # Invoking ffprobe to determine resolution
+            pp = FFmpegPostProcessor(self._downloader)
+            cookies = self._get_cookies('https://nicovideo.jp').output(header='', sep='; path=/; domain=nicovideo.jp;\n')

-        self.to_screen('%s: %s' % (video_id, 'Checking smile format with ffprobe'))
+            self.to_screen('%s: %s' % (video_id, 'Checking smile format with ffprobe'))

-        try:
-            metadata = pp.get_metadata_object(video_real_url, ['-cookies', cookies])
-        except PostProcessingError as err:
-            raise ExtractorError(err.msg, expected=True)
+            try:
+                metadata = pp.get_metadata_object(video_real_url, ['-cookies', cookies])
+            except PostProcessingError as err:
+                raise ExtractorError(err.msg, expected=True)

-        v_stream = a_stream = {}
+            v_stream = a_stream = {}

-        # Some complex swf files doesn't have video stream (e.g. nm4809023)
-        for stream in metadata['streams']:
-            if stream['codec_type'] == 'video':
-                v_stream = stream
-            elif stream['codec_type'] == 'audio':
-                a_stream = stream
+            # Some complex swf files doesn't have video stream (e.g. nm4809023)
+            for stream in metadata['streams']:
+                if stream['codec_type'] == 'video':
+                    v_stream = stream
+                elif stream['codec_type'] == 'audio':
+                    a_stream = stream

-        # Community restricted videos seem to have issues with the thumb API not returning anything at all
-        filesize = int(
-            (get_video_info_xml('size_high') if not is_economy else get_video_info_xml('size_low'))
-            or metadata['format']['size']
-        )
-        extension = (
-            get_video_info_xml('movie_type')
-            or 'mp4' if 'mp4' in metadata['format']['format_name'] else metadata['format']['format_name']
-        )
+            # Community restricted videos seem to have issues with the thumb API not returning anything at all
+            filesize = int(
+                (get_video_info_xml('size_high') if not is_economy else get_video_info_xml('size_low'))
+                or metadata['format']['size']
+            )
+            extension = (
+                get_video_info_xml('movie_type')
+                or 'mp4' if 'mp4' in metadata['format']['format_name'] else metadata['format']['format_name']
+            )

-        # 'creation_time' tag on video stream of re-encoded SMILEVIDEO mp4 files are '1970-01-01T00:00:00.000000Z'.
-        timestamp = (
-            parse_iso8601(get_video_info_web('first_retrieve'))
-            or unified_timestamp(get_video_info_web('postedDateTime'))
-        )
-        metadata_timestamp = (
-            parse_iso8601(try_get(v_stream, lambda x: x['tags']['creation_time']))
-            or timestamp if extension != 'mp4' else 0
-        )
+            # 'creation_time' tag on video stream of re-encoded SMILEVIDEO mp4 files are '1970-01-01T00:00:00.000000Z'.
+            timestamp = (
+                parse_iso8601(get_video_info_web('first_retrieve'))
+                or unified_timestamp(get_video_info_web('postedDateTime'))
+            )
+            metadata_timestamp = (
+                parse_iso8601(try_get(v_stream, lambda x: x['tags']['creation_time']))
+                or timestamp if extension != 'mp4' else 0
+            )

-        # According to compconf, smile videos from pre-2017 are always better quality than their DMC counterparts
-        smile_threshold_timestamp = parse_iso8601('2016-12-08T00:00:00+09:00')
+            # According to compconf, smile videos from pre-2017 are always better quality than their DMC counterparts
+            smile_threshold_timestamp = parse_iso8601('2016-12-08T00:00:00+09:00')

-        is_source = timestamp < smile_threshold_timestamp or metadata_timestamp > 0
+            is_source = timestamp < smile_threshold_timestamp or metadata_timestamp > 0

-        # If movie file size is unstable, old server movie is not source movie.
-        if filesize > 1:
-            formats.append({
-                'url': video_real_url,
-                'format_id': 'smile' if not is_economy else 'smile_low',
-                'format_note': 'SMILEVIDEO source' if not is_economy else 'SMILEVIDEO low quality',
-                'ext': extension,
-                'container': extension,
-                'vcodec': v_stream.get('codec_name'),
-                'acodec': a_stream.get('codec_name'),
-                # Some complex swf files doesn't have total bit rate metadata (e.g. nm6049209)
-                'tbr': int_or_none(metadata['format'].get('bit_rate'), scale=1000),
-                'vbr': int_or_none(v_stream.get('bit_rate'), scale=1000),
-                'abr': int_or_none(a_stream.get('bit_rate'), scale=1000),
-                'height': int_or_none(v_stream.get('height')),
-                'width': int_or_none(v_stream.get('width')),
-                'source_preference': 5 if not is_economy else -2,
-                'quality': 5 if is_source and not is_economy else None,
-                'filesize': filesize
-            })
+            # If movie file size is unstable, old server movie is not source movie.
+            if filesize > 1:
+                formats.append({
+                    'url': video_real_url,
+                    'format_id': 'smile' if not is_economy else 'smile_low',
+                    'format_note': 'SMILEVIDEO source' if not is_economy else 'SMILEVIDEO low quality',
+                    'ext': extension,
+                    'container': extension,
+                    'vcodec': v_stream.get('codec_name'),
+                    'acodec': a_stream.get('codec_name'),
+                    # Some complex swf files doesn't have total bit rate metadata (e.g. nm6049209)
+                    'tbr': int_or_none(metadata['format'].get('bit_rate'), scale=1000),
+                    'vbr': int_or_none(v_stream.get('bit_rate'), scale=1000),
+                    'abr': int_or_none(a_stream.get('bit_rate'), scale=1000),
+                    'height': int_or_none(v_stream.get('height')),
+                    'width': int_or_none(v_stream.get('width')),
+                    'source_preference': 5 if not is_economy else -2,
+                    'quality': 5 if is_source and not is_economy else None,
+                    'filesize': filesize
+                })

        if len(formats) == 0:
            raise ExtractorError('Unable to find video info.')
@@ -487,13 +492,12 @@ class NiconicoIE(InfoExtractor):
        self._sort_formats(formats)

        # Start extracting information
-        title = get_video_info_web('originalTitle')
-        if not title:
-            title = self._og_search_title(webpage, default=None)
-        if not title:
-            title = self._html_search_regex(
+        title = (
+            get_video_info_web(['originalTitle', 'title'])
+            or self._og_search_title(webpage, default=None)
+            or self._html_search_regex(
                r'<span[^>]+class="videoHeaderTitle"[^>]*>([^<]+)</span>',
-                webpage, 'video title')
+                webpage, 'video title'))

        watch_api_data_string = self._html_search_regex(
            r'<div[^>]+id="watchAPIDataContainer"[^>]+>([^<]+)</div>',
@@ -517,6 +521,7 @@ class NiconicoIE(InfoExtractor):
            timestamp = parse_iso8601(
                video_detail['postedAt'].replace('/', '-'),
                delimiter=' ', timezone=datetime.timedelta(hours=9))
+        timestamp = timestamp or try_get(api_data, lambda x: parse_iso8601(x['video']['registeredAt']))

        view_count = int_or_none(get_video_info_web(['view_counter', 'viewCount']))
        if not view_count:
@@ -525,11 +530,16 @@ class NiconicoIE(InfoExtractor):
                webpage, 'view count', default=None)
            if match:
                view_count = int_or_none(match.replace(',', ''))
-        view_count = view_count or video_detail.get('viewCount')
+        view_count = (
+            view_count
+            or video_detail.get('viewCount')
+            or try_get(api_data, lambda x: x['video']['count']['view']))
+
+        comment_count = (
+            int_or_none(get_video_info_web('comment_num'))
+            or video_detail.get('commentCount')
+            or try_get(api_data, lambda x: x['video']['count']['comment']))

-        comment_count = (int_or_none(get_video_info_web('comment_num'))
-                         or video_detail.get('commentCount')
-                         or try_get(api_data, lambda x: x['thread']['commentCount']))
        if not comment_count:
            match = self._html_search_regex(
                r'>Comments: <strong[^>]*>([^<]+)</strong>',
@@ -559,7 +569,7 @@ class NiconicoIE(InfoExtractor):
        # Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
        # in the JSON, which will cause None to be returned instead of {}.
        owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
-        uploader_id = (
+        uploader_id = str_or_none(
            get_video_info_web(['ch_id', 'user_id'])
            or owner.get('id')
            or channel_id
@@ -589,7 +599,7 @@ class NiconicoIE(InfoExtractor):


 class NiconicoPlaylistIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?nicovideo\.jp/(?:user/\d+/)?mylist/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?nicovideo\.jp/(?:user/\d+/|my/)?mylist/(?P<id>\d+)'

    _TESTS = [{
        'url': 'http://www.nicovideo.jp/mylist/27411728',
@@ -647,3 +657,40 @@ class NiconicoPlaylistIE(InfoExtractor):
            'uploader_id': uploader_id,
            'entries': OnDemandPagedList(pagefunc, 25),
        }
+
+
+class NiconicoUserIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?nicovideo\.jp/user/(?P<id>\d+)/?(?:$|[#?])'
+    _TEST = {
+        'url': 'https://www.nicovideo.jp/user/419948',
+        'info_dict': {
+            'id': '419948',
+        },
+        'playlist_mincount': 101,
+    }
+    _API_URL = "https://nvapi.nicovideo.jp/v1/users/%s/videos?sortKey=registeredAt&sortOrder=desc&pageSize=%s&page=%s"
+    _api_headers = {
+        'X-Frontend-ID': '6',
+        'X-Frontend-Version': '0',
+        'X-Niconico-Language': 'en-us'
+    }
+    _PAGE_SIZE = 100
+
+    def _entries(self, list_id, ):
+        total_count = 1
+        count = page_num = 0
+        while count < total_count:
+            json_parsed = self._download_json(
+                self._API_URL % (list_id, self._PAGE_SIZE, page_num + 1), list_id,
+                headers=self._api_headers,
+                note='Downloading JSON metadata%s' % (' page %d' % page_num if page_num else ''))
+            if not page_num:
+                total_count = int_or_none(json_parsed['data'].get('totalCount'))
+            for entry in json_parsed["data"]["items"]:
+                count += 1
+                yield self.url_result('https://www.nicovideo.jp/watch/%s' % entry['id'])
+            page_num += 1
+
+    def _real_extract(self, url):
+        list_id = self._match_id(url)
+        return self.playlist_result(self._entries(list_id), list_id, ie=NiconicoIE.ie_key())
@@ -11,60 +11,100 @@ from ..utils import (
    determine_ext,
 )
 import re
+import random


 class NitterIE(InfoExtractor):
    # Taken from https://github.com/zedeus/nitter/wiki/Instances
-    INSTANCES = ('nitter.net',
-                 'nitter.snopyta.org',
-                 'nitter.42l.fr',
-                 'nitter.nixnet.services',
-                 'nitter.13ad.de',
-                 'nitter.pussthecat.org',
-                 'nitter.mastodont.cat',
-                 'nitter.dark.fail',
-                 'nitter.tedomum.net',
-                 'nitter.cattube.org',
-                 'nitter.fdn.fr',
-                 'nitter.1d4.us',
-                 'nitter.kavin.rocks',
-                 'tweet.lambda.dance',
-                 'nitter.cc',
-                 'nitter.weaponizedhumiliation.com',
-                 '3nzoldnxplag42gqjs23xvghtzf6t6yzssrtytnntc6ppc7xxuoneoad.onion',
-                 'nitter.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd.onion',
-                 'nitterlgj3n5fgwesu3vxc5h67ruku33nqaoeoocae2mvlzhsu6k7fqd.onion')
+
+    NON_HTTP_INSTANCES = (
+        '3nzoldnxplag42gqjs23xvghtzf6t6yzssrtytnntc6ppc7xxuoneoad.onion',
+        'nitter.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd.onion',
+        'nitter7bryz3jv7e3uekphigvmoyoem4al3fynerxkj22dmoxoq553qd.onion',
+        'npf37k3mtzwxreiw52ccs5ay4e6qt2fkcs2ndieurdyn2cuzzsfyfvid.onion',
+        'nitter.v6vgyqpa7yefkorazmg5d5fimstmvm2vtbirt6676mt7qmllrcnwycqd.onion',
+        'i23nv6w3juvzlw32xzoxcqzktegd4i4fu3nmnc2ewv4ggiu4ledwklad.onion',
+        '26oq3gioiwcmfojub37nz5gzbkdiqp7fue5kvye7d4txv4ny6fb4wwid.onion',
+
+        'nitter.i2p',
+        'u6ikd6zndl3c4dsdq4mmujpntgeevdk5qzkfb57r4tnfeccrn2qa.b32.i2p',
+
+        'nitterlgj3n5fgwesu3vxc5h67ruku33nqaoeoocae2mvlzhsu6k7fqd.onion',
+    )
+
+    HTTP_INSTANCES = (
+        'nitter.42l.fr',
+        'nitter.pussthecat.org',
+        'nitter.nixnet.services',
+        'nitter.mastodont.cat',
+        'nitter.tedomum.net',
+        'nitter.fdn.fr',
+        'nitter.1d4.us',
+        'nitter.kavin.rocks',
+        'tweet.lambda.dance',
+        'nitter.cc',
+        'nitter.vxempire.xyz',
+        'nitter.unixfox.eu',
+        'nitter.domain.glass',
+        'nitter.himiko.cloud',
+        'nitter.eu',
+        'nitter.namazso.eu',
+        'nitter.mailstation.de',
+        'nitter.actionsack.com',
+        'nitter.cattube.org',
+        'nitter.dark.fail',
+        'birdsite.xanny.family',
+        'nitter.40two.app',
+        'nitter.skrep.in',
+
+        # not in the list anymore
+        'nitter.snopyta.org',
+    )
+
+    DEAD_INSTANCES = (
+        # maintenance
+        'nitter.ethibox.fr',
+
+        # official, rate limited
+        'nitter.net',
+        # offline
+        'nitter.13ad.de',
+        'nitter.weaponizedhumiliation.com',
+    )
+
+    INSTANCES = NON_HTTP_INSTANCES + HTTP_INSTANCES + DEAD_INSTANCES

    _INSTANCES_RE = '(?:' + '|'.join([re.escape(instance) for instance in INSTANCES]) + ')'
    _VALID_URL = r'https?://%(instance)s/(?P<uploader_id>.+)/status/(?P<id>[0-9]+)(#.)?' % {'instance': _INSTANCES_RE}
-    current_instance = INSTANCES[0]  # the test and official instance
+    current_instance = random.choice(HTTP_INSTANCES)
+
    _TESTS = [
        {
            # GIF (wrapped in mp4)
-            'url': 'https://' + current_instance + '/firefox/status/1314279897502629888#m',
+            'url': 'https://%s/firefox/status/1314279897502629888#m' % current_instance,
            'info_dict': {
                'id': '1314279897502629888',
                'ext': 'mp4',
-                'title': 'Firefox 🔥 - You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension.   Report harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg  #UnfckTheInternet',
-                'description': 'You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension.   Report harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg  #UnfckTheInternet',
+                'title': 'Firefox 🔥 - You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension. \n\nReport harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg\n\n#UnfckTheInternet',
+                'description': 'You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension. \n\nReport harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg\n\n#UnfckTheInternet',
                'thumbnail': r're:^https?://.*\.jpg$',
                'uploader': 'Firefox 🔥',
                'uploader_id': 'firefox',
-                'uploader_url': 'https://' + current_instance + '/firefox',
+                'uploader_url': 'https://%s/firefox' % current_instance,
                'upload_date': '20201008',
                'timestamp': 1602183720,
            },
        }, {  # normal video
-            'url': 'https://' + current_instance + '/Le___Doc/status/1299715685392756737#m',
+            'url': 'https://%s/Le___Doc/status/1299715685392756737#m' % current_instance,
            'info_dict': {
                'id': '1299715685392756737',
                'ext': 'mp4',
-                'title': 'Le Doc - "Je ne prédis jamais rien" D Raoult, Août 2020...',
-                'description': '"Je ne prédis jamais rien" D Raoult, Août 2020...',
+                'title': 'Le Doc - "Je ne prédis jamais rien"\nD Raoult, Août 2020...',
+                'description': '"Je ne prédis jamais rien"\nD Raoult, Août 2020...',
                'thumbnail': r're:^https?://.*\.jpg$',
                'uploader': 'Le Doc',
                'uploader_id': 'Le___Doc',
-                'uploader_url': 'https://' + current_instance + '/Le___Doc',
+                'uploader_url': 'https://%s/Le___Doc' % current_instance,
                'upload_date': '20200829',
                'timestamp': 1598711341,
                'view_count': int,
@@ -73,31 +113,51 @@ class NitterIE(InfoExtractor):
                'comment_count': int,
            },
        }, {  # video embed in a "Streaming Political Ads" box
-            'url': 'https://' + current_instance + '/mozilla/status/1321147074491092994#m',
+            'url': 'https://%s/mozilla/status/1321147074491092994#m' % current_instance,
            'info_dict': {
                'id': '1321147074491092994',
                'ext': 'mp4',
-                'title': "Mozilla - Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?  This isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few.   Learn more ➡️ https://mzl.la/StreamingAds",
-                'description': "Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?  This isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few.   Learn more ➡️ https://mzl.la/StreamingAds",
+                'title': "Mozilla - Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?\n\nThis isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few. \n\nLearn more ➡️ https://mzl.la/StreamingAds",
+                'description': "Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?\n\nThis isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few. \n\nLearn more ➡️ https://mzl.la/StreamingAds",
                'thumbnail': r're:^https?://.*\.jpg$',
                'uploader': 'Mozilla',
                'uploader_id': 'mozilla',
-                'uploader_url': 'https://' + current_instance + '/mozilla',
+                'uploader_url': 'https://%s/mozilla' % current_instance,
                'upload_date': '20201027',
                'timestamp': 1603820982
            },
-        },
+        }, {  # not the first tweet but main-tweet
+            'url': 'https://%s/TheNaturalNu/status/1379050895539724290#m' % current_instance,
+            'info_dict': {
+                'id': '1379050895539724290',
+                'ext': 'mp4',
+                'title': 'Dorothy Zbornak - This had me hollering!!',
+                'description': 'This had me hollering!!',
+                'thumbnail': r're:^https?://.*\.jpg$',
+                'uploader': 'Dorothy Zbornak',
+                'uploader_id': 'TheNaturalNu',
+                'uploader_url': 'https://%s/TheNaturalNu' % current_instance,
+                'timestamp': 1617626329,
+                'upload_date': '20210405'
+            }
+        }
    ]

    def _real_extract(self, url):
        video_id = self._match_id(url)
        parsed_url = compat_urlparse.urlparse(url)
-        base_url = parsed_url.scheme + '://' + parsed_url.netloc
+        base_url = '%s://%s' % (parsed_url.scheme, parsed_url.netloc)

        self._set_cookie(parsed_url.netloc, 'hlsPlayback', 'on')
-        webpage = self._download_webpage(url, video_id)
+        full_webpage = self._download_webpage(url, video_id)

-        video_url = base_url + self._html_search_regex(r'(?:<video[^>]+data-url|<source[^>]+src)="([^"]+)"', webpage, 'video url')
+        main_tweet_start = full_webpage.find('class="main-tweet"')
+        if main_tweet_start > 0:
+            webpage = full_webpage[main_tweet_start:]
+        if not webpage:
+            webpage = full_webpage
+
+        video_url = '%s%s' % (base_url, self._html_search_regex(r'(?:<video[^>]+data-url|<source[^>]+src)="([^"]+)"', webpage, 'video url'))
        ext = determine_ext(video_url)

        if ext == 'unknown_video':
@@ -108,33 +168,34 @@ class NitterIE(InfoExtractor):
                'ext': ext
            }]

-        title = (
-            self._og_search_description(webpage).replace('\n', ' ')
-            or self._html_search_regex(r'<div class="tweet-content[^>]+>([^<]+)</div>', webpage, 'title'))
+        title = self._og_search_description(full_webpage)
+        if not title:
+            title = self._html_search_regex(r'<div class="tweet-content[^>]+>([^<]+)</div>', webpage, 'title')
        description = title

        mobj = re.match(self._VALID_URL, url)
        uploader_id = (
            mobj.group('uploader_id')
-            or self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False))
+            or self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False)
+        )

        if uploader_id:
-            uploader_url = base_url + '/' + uploader_id
+            uploader_url = '%s/%s' % (base_url, uploader_id)

        uploader = self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False)

        if uploader:
-            title = uploader + ' - ' + title
+            title = '%s - %s' % (uploader, title)

        view_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-play[^>]*></span>\s([^<]+)</div>', webpage, 'view count', fatal=False))
        like_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-heart[^>]*></span>\s([^<]+)</div>', webpage, 'like count', fatal=False))
        repost_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-retweet[^>]*></span>\s([^<]+)</div>', webpage, 'repost count', fatal=False))
        comment_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-comment[^>]*></span>\s([^<]+)</div>', webpage, 'repost count', fatal=False))

-        thumbnail = base_url + (self._html_search_meta('og:image', webpage, 'thumbnail url')
-                                or self._html_search_regex(r'<video[^>]+poster="([^"]+)"', webpage, 'thumbnail url', fatal=False))
-
-        thumbnail = remove_end(thumbnail, '%3Asmall')  # if parsed with regex, it should contain this
+        thumbnail = self._html_search_meta('og:image', full_webpage, 'thumbnail url')
+        if not thumbnail:
+            thumbnail = '%s%s' % (base_url, self._html_search_regex(r'<video[^>]+poster="([^"]+)"', webpage, 'thumbnail url', fatal=False))
+            thumbnail = remove_end(thumbnail, '%3Asmall')

        thumbnails = []
        thumbnail_ids = ('thumb', 'small', 'large', 'medium', 'orig')
@@ -0,0 +1,148 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+    int_or_none,
+    str_or_none,
+    try_get,
+)
+
+
+class PalcoMP3BaseIE(InfoExtractor):
+    _GQL_QUERY_TMPL = '''{
+  artist(slug: "%s") {
+    %s
+  }
+}'''
+    _ARTIST_FIELDS_TMPL = '''music(slug: "%%s") {
+      %s
+    }'''
+    _MUSIC_FIELDS = '''duration
+      hls
+      mp3File
+      musicID
+      plays
+      title'''
+
+    def _call_api(self, artist_slug, artist_fields):
+        return self._download_json(
+            'https://www.palcomp3.com.br/graphql/', artist_slug, query={
+                'query': self._GQL_QUERY_TMPL % (artist_slug, artist_fields),
+            })['data']
+
+    def _parse_music(self, music):
+        music_id = compat_str(music['musicID'])
+        title = music['title']
+
+        formats = []
+        hls_url = music.get('hls')
+        if hls_url:
+            formats.append({
+                'url': hls_url,
+                'protocol': 'm3u8_native',
+                'ext': 'mp4',
+            })
+        mp3_file = music.get('mp3File')
+        if mp3_file:
+            formats.append({
+                'url': mp3_file,
+            })
+
+        return {
+            'id': music_id,
+            'title': title,
+            'formats': formats,
+            'duration': int_or_none(music.get('duration')),
+            'view_count': int_or_none(music.get('plays')),
+        }
+
+    def _real_initialize(self):
+        self._ARTIST_FIELDS_TMPL = self._ARTIST_FIELDS_TMPL % self._MUSIC_FIELDS
+
+    def _real_extract(self, url):
+        artist_slug, music_slug = re.match(self._VALID_URL, url).groups()
+        artist_fields = self._ARTIST_FIELDS_TMPL % music_slug
+        music = self._call_api(artist_slug, artist_fields)['artist']['music']
+        return self._parse_music(music)
+
+
+class PalcoMP3IE(PalcoMP3BaseIE):
+    IE_NAME = 'PalcoMP3:song'
+    _VALID_URL = r'https?://(?:www\.)?palcomp3\.com(?:\.br)?/(?P<artist>[^/]+)/(?P<id>[^/?&#]+)'
+    _TESTS = [{
+        'url': 'https://www.palcomp3.com/maiaraemaraisaoficial/nossas-composicoes-cuida-bem-dela/',
+        'md5': '99fd6405b2d8fd589670f6db1ba3b358',
+        'info_dict': {
+            'id': '3162927',
+            'ext': 'mp3',
+            'title': 'Nossas Composições - CUIDA BEM DELA',
+            'duration': 210,
+            'view_count': int,
+        }
+    }]
+
+    @classmethod
+    def suitable(cls, url):
+        return False if PalcoMP3VideoIE.suitable(url) else super(PalcoMP3IE, cls).suitable(url)
+
+
+class PalcoMP3ArtistIE(PalcoMP3BaseIE):
+    IE_NAME = 'PalcoMP3:artist'
+    _VALID_URL = r'https?://(?:www\.)?palcomp3\.com(?:\.br)?/(?P<id>[^/?&#]+)'
+    _TESTS = [{
+        'url': 'https://www.palcomp3.com.br/condedoforro/',
+        'info_dict': {
+            'id': '358396',
+            'title': 'Conde do Forró',
+        },
+        'playlist_mincount': 188,
+    }]
+    _ARTIST_FIELDS_TMPL = '''artistID
+    musics {
+      nodes {
+        %s
+      }
+    }
+    name'''
+
+    @ classmethod
+    def suitable(cls, url):
+        return False if re.match(PalcoMP3IE._VALID_URL, url) else super(PalcoMP3ArtistIE, cls).suitable(url)
+
+    def _real_extract(self, url):
+        artist_slug = self._match_id(url)
+        artist = self._call_api(artist_slug, self._ARTIST_FIELDS_TMPL)['artist']
+
+        def entries():
+            for music in (try_get(artist, lambda x: x['musics']['nodes'], list) or []):
+                yield self._parse_music(music)
+
+        return self.playlist_result(
+            entries(), str_or_none(artist.get('artistID')), artist.get('name'))
+
+
+class PalcoMP3VideoIE(PalcoMP3BaseIE):
+    IE_NAME = 'PalcoMP3:video'
+    _VALID_URL = r'https?://(?:www\.)?palcomp3\.com(?:\.br)?/(?P<artist>[^/]+)/(?P<id>[^/?&#]+)/?#clipe'
+    _TESTS = [{
+        'url': 'https://www.palcomp3.com/maiaraemaraisaoficial/maiara-e-maraisa-voce-faz-falta-aqui-ao-vivo-em-vicosa-mg/#clipe',
+        'add_ie': ['Youtube'],
+        'info_dict': {
+            'id': '_pD1nR2qqPg',
+            'ext': 'mp4',
+            'title': 'Maiara e Maraisa - Você Faz Falta Aqui - DVD Ao Vivo Em Campo Grande',
+            'description': 'md5:7043342c09a224598e93546e98e49282',
+            'upload_date': '20161107',
+            'uploader_id': 'maiaramaraisaoficial',
+            'uploader': 'Maiara e Maraisa',
+        }
+    }]
+    _MUSIC_FIELDS = 'youtubeID'
+
+    def _parse_music(self, music):
+        youtube_id = music['youtubeID']
+        return self.url_result(youtube_id, 'Youtube', youtube_id)
@@ -599,11 +599,13 @@ class PeerTubeIE(InfoExtractor):
        else:
            age_limit = None

+        webpage_url = 'https://%s/videos/watch/%s' % (host, video_id)
+
        return {
            'id': video_id,
            'title': title,
            'description': description,
-            'thumbnail': urljoin(url, video.get('thumbnailPath')),
+            'thumbnail': urljoin(webpage_url, video.get('thumbnailPath')),
            'timestamp': unified_timestamp(video.get('publishedAt')),
            'uploader': account_data('displayName', compat_str),
            'uploader_id': str_or_none(account_data('id', int)),
@@ -621,5 +623,6 @@ class PeerTubeIE(InfoExtractor):
            'tags': try_get(video, lambda x: x['tags'], list),
            'categories': categories,
            'formats': formats,
-            'subtitles': subtitles
+            'subtitles': subtitles,
+            'webpage_url': webpage_url,
        }
@@ -1,22 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals

-import re
-import time
-
 from .common import InfoExtractor
-from ..compat import compat_str
 from ..utils import (
    ExtractorError,
    js_to_json,
-    try_get,
-    update_url_query,
-    urlencode_postdata,
 )


 class PicartoIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www.)?picarto\.tv/(?P<id>[a-zA-Z0-9]+)(?:/(?P<token>[a-zA-Z0-9]+))?'
+    _VALID_URL = r'https?://(?:www.)?picarto\.tv/(?P<id>[a-zA-Z0-9]+)'
    _TEST = {
        'url': 'https://picarto.tv/Setz',
        'info_dict': {
@@ -34,65 +27,46 @@ class PicartoIE(InfoExtractor):
        return False if PicartoVodIE.suitable(url) else super(PicartoIE, cls).suitable(url)

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        channel_id = mobj.group('id')
+        channel_id = self._match_id(url)

-        metadata = self._download_json(
-            'https://api.picarto.tv/v1/channel/name/' + channel_id,
-            channel_id)
+        data = self._download_json(
+            'https://ptvintern.picarto.tv/ptvapi', channel_id, query={
+                'query': '''{
+  channel(name: "%s") {
+    adult
+    id
+    online
+    stream_name
+    title
+  }
+  getLoadBalancerUrl(channel_name: "%s") {
+    url
+  }
+}''' % (channel_id, channel_id),
+            })['data']
+        metadata = data['channel']

-        if metadata.get('online') is False:
+        if metadata.get('online') == 0:
            raise ExtractorError('Stream is offline', expected=True)
+        title = metadata['title']

        cdn_data = self._download_json(
-            'https://picarto.tv/process/channel', channel_id,
-            data=urlencode_postdata({'loadbalancinginfo': channel_id}),
-            note='Downloading load balancing info')
+            data['getLoadBalancerUrl']['url'] + '/stream/json_' + metadata['stream_name'] + '.js',
+            channel_id, 'Downloading load balancing info')

-        token = mobj.group('token') or 'public'
-        params = {
-            'con': int(time.time() * 1000),
-            'token': token,
-        }
-
-        prefered_edge = cdn_data.get('preferedEdge')
        formats = []
-
-        for edge in cdn_data['edges']:
-            edge_ep = edge.get('ep')
-            if not edge_ep or not isinstance(edge_ep, compat_str):
+        for source in (cdn_data.get('source') or []):
+            source_url = source.get('url')
+            if not source_url:
                continue
-            edge_id = edge.get('id')
-            for tech in cdn_data['techs']:
-                tech_label = tech.get('label')
-                tech_type = tech.get('type')
-                preference = 0
-                if edge_id == prefered_edge:
-                    preference += 1
-                format_id = []
-                if edge_id:
-                    format_id.append(edge_id)
-                if tech_type == 'application/x-mpegurl' or tech_label == 'HLS':
-                    format_id.append('hls')
-                    formats.extend(self._extract_m3u8_formats(
-                        update_url_query(
-                            'https://%s/hls/%s/index.m3u8'
-                            % (edge_ep, channel_id), params),
-                        channel_id, 'mp4', quality=preference,
-                        m3u8_id='-'.join(format_id), fatal=False))
-                    continue
-                elif tech_type == 'video/mp4' or tech_label == 'MP4':
-                    format_id.append('mp4')
-                    formats.append({
-                        'url': update_url_query(
-                            'https://%s/mp4/%s.mp4' % (edge_ep, channel_id),
-                            params),
-                        'format_id': '-'.join(format_id),
-                        'quality': preference,
-                    })
-                else:
-                    # rtmp format does not seem to work
-                    continue
+            source_type = source.get('type')
+            if source_type == 'html5/application/vnd.apple.mpegurl':
+                formats.extend(self._extract_m3u8_formats(
+                    source_url, channel_id, 'mp4', m3u8_id='hls', fatal=False))
+            elif source_type == 'html5/video/mp4':
+                formats.append({
+                    'url': source_url,
+                })
        self._sort_formats(formats)

        mature = metadata.get('adult')
@@ -103,10 +77,10 @@ class PicartoIE(InfoExtractor):

        return {
            'id': channel_id,
-            'title': self._live_title(metadata.get('title') or channel_id),
+            'title': self._live_title(title.strip()),
            'is_live': True,
-            'thumbnail': try_get(metadata, lambda x: x['thumbnails']['web']),
            'channel': channel_id,
+            'channel_id': metadata.get('id'),
            'channel_url': 'https://picarto.tv/%s' % channel_id,
            'age_limit': age_limit,
            'formats': formats,
@@ -31,6 +31,7 @@ class PinterestBaseIE(InfoExtractor):

        title = (data.get('title') or data.get('grid_title') or video_id).strip()

+        urls = []
        formats = []
        duration = None
        if extract_formats:
@@ -38,8 +39,9 @@ class PinterestBaseIE(InfoExtractor):
                if not isinstance(format_dict, dict):
                    continue
                format_url = url_or_none(format_dict.get('url'))
-                if not format_url:
+                if not format_url or format_url in urls:
                    continue
+                urls.append(format_url)
                duration = float_or_none(format_dict.get('duration'), scale=1000)
                ext = determine_ext(format_url)
                if 'hls' in format_id.lower() or ext == 'm3u8':
@@ -0,0 +1,164 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+import uuid
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import (
+    ExtractorError,
+    float_or_none,
+    int_or_none,
+    try_get,
+    url_or_none,
+)
+
+
+class PlutoTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?pluto\.tv/on-demand/(?P<video_type>movies|series)/(?P<slug>.*)/?$'
+    _INFO_URL = 'https://service-vod.clusters.pluto.tv/v3/vod/slugs/'
+    _INFO_QUERY_PARAMS = {
+        'appName': 'web',
+        'appVersion': 'na',
+        'clientID': compat_str(uuid.uuid1()),
+        'clientModelNumber': 'na',
+        'serverSideAds': 'false',
+        'deviceMake': 'unknown',
+        'deviceModel': 'web',
+        'deviceType': 'web',
+        'deviceVersion': 'unknown',
+        'sid': compat_str(uuid.uuid1()),
+    }
+    _TESTS = [
+        {
+            'url': 'https://pluto.tv/on-demand/series/i-love-money/season/2/episode/its-in-the-cards-2009-2-3',
+            'md5': 'ebcdd8ed89aaace9df37924f722fd9bd',
+            'info_dict': {
+                'id': '5de6c598e9379ae4912df0a8',
+                'ext': 'mp4',
+                'title': 'It\'s In The Cards',
+                'episode': 'It\'s In The Cards',
+                'description': 'The teams face off against each other in a 3-on-2 soccer showdown.  Strategy comes into play, though, as each team gets to select their opposing teams’ two defenders.',
+                'series': 'I Love Money',
+                'season_number': 2,
+                'episode_number': 3,
+                'duration': 3600,
+            }
+        },
+        {
+            'url': 'https://pluto.tv/on-demand/series/i-love-money/season/1/',
+            'playlist_count': 11,
+            'info_dict': {
+                'id': '5de6c582e9379ae4912dedbd',
+                'title': 'I Love Money - Season 1',
+            }
+        },
+        {
+            'url': 'https://pluto.tv/on-demand/series/i-love-money/',
+            'playlist_count': 26,
+            'info_dict': {
+                'id': '5de6c582e9379ae4912dedbd',
+                'title': 'I Love Money',
+            }
+        },
+        {
+            'url': 'https://pluto.tv/on-demand/movies/arrival-2015-1-1',
+            'md5': '3cead001d317a018bf856a896dee1762',
+            'info_dict': {
+                'id': '5e83ac701fa6a9001bb9df24',
+                'ext': 'mp4',
+                'title': 'Arrival',
+                'description': 'When mysterious spacecraft touch down across the globe, an elite team - led by expert translator Louise Banks (Academy Award® nominee Amy Adams) – races against time to decipher their intent.',
+                'duration': 9000,
+            }
+        },
+    ]
+
+    def _to_ad_free_formats(self, video_id, formats):
+        ad_free_formats = []
+        m3u8_urls = set()
+        for format in formats:
+            res = self._download_webpage(
+                format.get('url'), video_id, note='Downloading m3u8 playlist',
+                fatal=False)
+            if not res:
+                continue
+            first_segment_url = re.search(
+                r'^(https?://.*/)0\-(end|[0-9]+)/[^/]+\.ts$', res,
+                re.MULTILINE)
+            if not first_segment_url:
+                continue
+            m3u8_urls.add(
+                compat_urlparse.urljoin(first_segment_url.group(1), '0-end/master.m3u8'))
+
+        for m3u8_url in m3u8_urls:
+            ad_free_formats.extend(
+                self._extract_m3u8_formats(
+                    m3u8_url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id='hls', fatal=False))
+        self._sort_formats(ad_free_formats)
+        return ad_free_formats
+
+    def _get_video_info(self, video_json, slug, series_name=None):
+        video_id = video_json.get('_id', slug)
+        formats = []
+        for video_url in try_get(video_json, lambda x: x['stitched']['urls'], list) or []:
+            if video_url.get('type') != 'hls':
+                continue
+            url = url_or_none(video_url.get('url'))
+            formats.extend(
+                self._extract_m3u8_formats(
+                    url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id='hls', fatal=False))
+        info = {
+            'id': video_id,
+            'formats': self._to_ad_free_formats(video_id, formats),
+            'title': video_json.get('name'),
+            'description': video_json.get('description'),
+            'duration': float_or_none(video_json.get('duration'), scale=1000),
+        }
+        if series_name:
+            info.update({
+                'series': series_name,
+                'episode': video_json.get('name'),
+                'season_number': int_or_none(video_json.get('season')),
+                'episode_number': int_or_none(video_json.get('number')),
+            })
+        return info
+
+    def _real_extract(self, url):
+        path = compat_urlparse.urlparse(url).path
+        path_components = path.split('/')
+        video_type = path_components[2]
+        info_slug = path_components[3]
+        video_json = self._download_json(self._INFO_URL + info_slug, info_slug,
+                                         query=self._INFO_QUERY_PARAMS)
+
+        if video_type == 'series':
+            series_name = video_json.get('name', info_slug)
+            season_number = int_or_none(try_get(path_components, lambda x: x[5]))
+            episode_slug = try_get(path_components, lambda x: x[7])
+
+            videos = []
+            for season in video_json['seasons']:
+                if season_number is not None and season_number != int_or_none(season.get('number')):
+                    continue
+                for episode in season['episodes']:
+                    if episode_slug is not None and episode_slug != episode.get('slug'):
+                        continue
+                    videos.append(self._get_video_info(episode, episode_slug, series_name))
+            if not videos:
+                raise ExtractorError('Failed to find any videos to extract')
+            if episode_slug is not None and len(videos) == 1:
+                return videos[0]
+            playlist_title = series_name
+            if season_number is not None:
+                playlist_title += ' - Season %d' % season_number
+            return self.playlist_result(videos,
+                                        playlist_id=video_json.get('_id', info_slug),
+                                        playlist_title=playlist_title)
+        return self._get_video_info(video_json, info_slug)
@@ -167,6 +167,7 @@ class PornHubIE(PornHubBaseIE):
        'params': {
            'skip_download': True,
        },
+        'skip': 'Video has been flagged for verification in accordance with our trust and safety policy',
    }, {
        # subtitles
        'url': 'https://www.pornhub.com/view_video.php?viewkey=ph5af5fef7c2aa7',
@@ -265,7 +266,8 @@ class PornHubIE(PornHubBaseIE):
        webpage = dl_webpage('pc')

        error_msg = self._html_search_regex(
-            r'(?s)<div[^>]+class=(["\'])(?:(?!\1).)*\b(?:removed|userMessageSection)\b(?:(?!\1).)*\1[^>]*>(?P<error>.+?)</div>',
+            (r'(?s)<div[^>]+class=(["\'])(?:(?!\1).)*\b(?:removed|userMessageSection)\b(?:(?!\1).)*\1[^>]*>(?P<error>.+?)</div>',
+             r'(?s)<section[^>]+class=["\']noVideo["\'][^>]*>(?P<error>.+?)</section>'),
            webpage, 'error message', default=None, group='error')
        if error_msg:
            error_msg = re.sub(r'\s+', ' ', error_msg)
@@ -394,34 +396,50 @@ class PornHubIE(PornHubBaseIE):

        upload_date = None
        formats = []
+
+        def add_format(format_url, height=None):
+            ext = determine_ext(format_url)
+            if ext == 'mpd':
+                formats.extend(self._extract_mpd_formats(
+                    format_url, video_id, mpd_id='dash', fatal=False))
+                return
+            if ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    format_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                    m3u8_id='hls', fatal=False))
+                return
+            tbr = None
+            mobj = re.search(r'(?P<height>\d+)[pP]?_(?P<tbr>\d+)[kK]', format_url)
+            if mobj:
+                if not height:
+                    height = int(mobj.group('height'))
+                tbr = int(mobj.group('tbr'))
+            formats.append({
+                'url': format_url,
+                'format_id': '%dp' % height if height else None,
+                'height': height,
+                'tbr': tbr,
+            })
+
        for video_url, height in video_urls:
            if not upload_date:
                upload_date = self._search_regex(
                    r'/(\d{6}/\d{2})/', video_url, 'upload data', default=None)
                if upload_date:
                    upload_date = upload_date.replace('/', '')
-            ext = determine_ext(video_url)
-            if ext == 'mpd':
-                formats.extend(self._extract_mpd_formats(
-                    video_url, video_id, mpd_id='dash', fatal=False))
+            if '/video/get_media' in video_url:
+                medias = self._download_json(video_url, video_id, fatal=False)
+                if isinstance(medias, list):
+                    for media in medias:
+                        if not isinstance(media, dict):
+                            continue
+                        video_url = url_or_none(media.get('videoUrl'))
+                        if not video_url:
+                            continue
+                        height = int_or_none(media.get('quality'))
+                        add_format(video_url, height)
                continue
-            elif ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
-                    video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
-                continue
-            tbr = None
-            mobj = re.search(r'(?P<height>\d+)[pP]?_(?P<tbr>\d+)[kK]', video_url)
-            if mobj:
-                if not height:
-                    height = int(mobj.group('height'))
-                tbr = int(mobj.group('tbr'))
-            formats.append({
-                'url': video_url,
-                'format_id': '%dp' % height if height else None,
-                'height': height,
-                'tbr': tbr,
-            })
+            add_format(video_url)
        self._sort_formats(formats)

        video_uploader = self._html_search_regex(
@@ -170,7 +170,7 @@ class RaiPlayIE(RaiBaseIE):
        media = self._download_json(
            base + '.json', video_id, 'Downloading video JSON')

-        if not self.params.get('allow_unplayable_formats'):
+        if not self._downloader.params.get('allow_unplayable_formats'):
            if try_get(
                    media,
                    (lambda x: x['rights_management']['rights']['drm'],
@@ -15,6 +15,9 @@ from ..utils import (


 class RCSBaseIE(InfoExtractor):
+    # based on VideoPlayerLoader.prototype.getVideoSrc
+    # and VideoPlayerLoader.prototype.transformSrc from
+    # https://js2.corriereobjects.it/includes2013/LIBS/js/corriere_video.sjs
    _ALL_REPLACE = {
        'media2vam.corriere.it.edgesuite.net':
            'media2vam-corriere-it.akamaized.net',
@@ -191,10 +194,10 @@ class RCSBaseIE(InfoExtractor):
            urls.get('m3u8'), video_id, 'mp4', entry_protocol='m3u8_native',
            m3u8_id='hls', fatal=False)

-        if not formats:
+        if urls.get('mp4'):
            formats.append({
                'format_id': 'http-mp4',
-                'url': urls.get('mp4')
+                'url': urls['mp4']
            })
        self._sort_formats(formats)
        return formats
@@ -216,10 +219,12 @@ class RCSBaseIE(InfoExtractor):
        video_data = None
        # look for json video data url
        json = self._search_regex(
-            r'''(?x)var url\s*=\s*["']((?:https?:)?
-                //video\.rcs\.it
-                /fragment-includes/video-includes/.+?\.json)["'];''',
-            page, video_id, default=None)
+            r'''(?x)url\s*=\s*(["'])
+            (?P<url>
+                (?:https?:)?//video\.rcs\.it
+                /fragment-includes/video-includes/.+?\.json
+            )\1;''',
+            page, video_id, group='url', default=None)
        if json:
            if json.startswith('//'):
                json = 'https:%s' % json
@@ -227,13 +232,16 @@ class RCSBaseIE(InfoExtractor):

        # if json url not found, look for json video data directly in the page
        else:
+            # RCS normal pages and most of the embeds
            json = self._search_regex(
                r'[\s;]video\s*=\s*({[\s\S]+?})(?:;|,playlist=)',
                page, video_id, default=None)
-            if json:
-                video_data = self._parse_json(
-                    json, video_id, transform_source=js_to_json)
-            else:
+            if not json and 'video-embed' in url:
+                page = self._download_webpage(url.replace('video-embed', 'video-json'), video_id)
+                json = self._search_regex(
+                    r'##start-video##({[\s\S]+?})##end-video##',
+                    page, video_id, default=None)
+            if not json:
                # if no video data found try search for iframes
                emb = RCSEmbedsIE._extract_url(page)
                if emb:
@@ -242,6 +250,9 @@ class RCSBaseIE(InfoExtractor):
                        'url': emb,
                        'ie_key': RCSEmbedsIE.ie_key()
                    }
+            if json:
+                video_data = self._parse_json(
+                    json, video_id, transform_source=js_to_json)

        if not video_data:
            raise ExtractorError('Video data not found in the page')
@@ -250,7 +261,8 @@ class RCSBaseIE(InfoExtractor):
            self._get_video_src(video_data), video_id)

        description = (video_data.get('description')
-                       or clean_html(video_data.get('htmlDescription')))
+                       or clean_html(video_data.get('htmlDescription'))
+                       or self._html_search_meta('description', page))
        uploader = video_data.get('provider') or mobj.group('cdn')

        return {
@@ -283,6 +295,7 @@ class RCSEmbedsIE(RCSBaseIE):
            'uploader': 'rcs.it',
        }
    }, {
+        # redownload the page changing 'video-embed' in 'video-json'
        'url': 'https://video.gazzanet.gazzetta.it/video-embed/gazzanet-mo05-0000260789',
        'md5': 'a043e3fecbe4d9ed7fc5d888652a5440',
        'info_dict': {
@@ -359,6 +372,7 @@ class RCSIE(RCSBaseIE):
            'uploader': 'Corriere Tv',
        }
    }, {
+        # video data inside iframe
        'url': 'https://viaggi.corriere.it/video/norvegia-il-nuovo-ponte-spettacolare-sopra-la-cascata-di-voringsfossen/',
        'md5': 'da378e4918d2afbf7d61c35abb948d4c',
        'info_dict': {
@@ -389,15 +403,15 @@ class RCSVariousIE(RCSBaseIE):
                    (?P<cdn>
                        leitv\.it|
                        youreporter\.it
-                    )/(?:video/)?(?P<id>[^/]+?)(?:$|\?|/)'''
+                    )/(?:[^/]+/)?(?P<id>[^/]+?)(?:$|\?|/)'''
    _TESTS = [{
-        'url': 'https://www.leitv.it/video/marmellata-di-ciliegie-fatta-in-casa/',
-        'md5': '618aaabac32152199c1af86784d4d554',
+        'url': 'https://www.leitv.it/benessere/mal-di-testa-come-combatterlo-ed-evitarne-la-comparsa/',
+        'md5': '92b4e63667b8f95acb0a04da25ae28a1',
        'info_dict': {
-            'id': 'marmellata-di-ciliegie-fatta-in-casa',
+            'id': 'mal-di-testa-come-combatterlo-ed-evitarne-la-comparsa',
            'ext': 'mp4',
-            'title': 'Marmellata di ciliegie fatta in casa',
-            'description': 'md5:89133864d6aad456dbcf6e7a29f86263',
+            'title': 'Cervicalgia e mal di testa, il video con i suggerimenti dell\'esperto',
+            'description': 'md5:ae21418f34cee0b8d02a487f55bcabb5',
            'uploader': 'leitv.it',
        }
    }, {
@@ -2,8 +2,9 @@
 from __future__ import unicode_literals

 import base64
+import io
 import re
-import time
+import sys

 from .common import InfoExtractor
 from ..compat import (
@@ -14,56 +15,13 @@ from ..utils import (
    determine_ext,
    ExtractorError,
    float_or_none,
+    qualities,
    remove_end,
    remove_start,
-    sanitized_Request,
    std_headers,
 )

-
-def _decrypt_url(png):
-    encrypted_data = compat_b64decode(png)
-    text_index = encrypted_data.find(b'tEXt')
-    text_chunk = encrypted_data[text_index - 4:]
-    length = compat_struct_unpack('!I', text_chunk[:4])[0]
-    # Use bytearray to get integers when iterating in both python 2.x and 3.x
-    data = bytearray(text_chunk[8:8 + length])
-    data = [chr(b) for b in data if b != 0]
-    hash_index = data.index('#')
-    alphabet_data = data[:hash_index]
-    url_data = data[hash_index + 1:]
-    if url_data[0] == 'H' and url_data[3] == '%':
-        # remove useless HQ%% at the start
-        url_data = url_data[4:]
-
-    alphabet = []
-    e = 0
-    d = 0
-    for l in alphabet_data:
-        if d == 0:
-            alphabet.append(l)
-            d = e = (e + 1) % 4
-        else:
-            d -= 1
-    url = ''
-    f = 0
-    e = 3
-    b = 1
-    for letter in url_data:
-        if f == 0:
-            l = int(letter) * 10
-            f = 1
-        else:
-            if e == 0:
-                l += int(letter)
-                url += alphabet[l]
-                e = (b + 3) % 4
-                f = 0
-                b += 1
-            else:
-                e -= 1
-
-    return url
+_bytes_to_chr = (lambda x: x) if sys.version_info[0] == 2 else (lambda x: map(chr, x))


 class RTVEALaCartaIE(InfoExtractor):
@@ -79,28 +37,31 @@ class RTVEALaCartaIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Balonmano - Swiss Cup masculina. Final: España-Suecia',
            'duration': 5024.566,
+            'series': 'Balonmano',
        },
+        'expected_warnings': ['Failed to download MPD manifest', 'Failed to download m3u8 information'],
    }, {
        'note': 'Live stream',
        'url': 'http://www.rtve.es/alacarta/videos/television/24h-live/1694255/',
        'info_dict': {
            'id': '1694255',
-            'ext': 'flv',
-            'title': 'TODO',
+            'ext': 'mp4',
+            'title': 're:^24H LIVE [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
+            'is_live': True,
+        },
+        'params': {
+            'skip_download': 'live stream',
        },
-        'skip': 'The f4m manifest can\'t be used yet',
    }, {
        'url': 'http://www.rtve.es/alacarta/videos/servir-y-proteger/servir-proteger-capitulo-104/4236788/',
-        'md5': 'e55e162379ad587e9640eda4f7353c0f',
+        'md5': 'd850f3c8731ea53952ebab489cf81cbf',
        'info_dict': {
            'id': '4236788',
            'ext': 'mp4',
-            'title': 'Servir y proteger - Capítulo 104 ',
+            'title': 'Servir y proteger - Capítulo 104',
            'duration': 3222.0,
        },
-        'params': {
-            'skip_download': True,  # requires ffmpeg
-        },
+        'expected_warnings': ['Failed to download MPD manifest', 'Failed to download m3u8 information'],
    }, {
        'url': 'http://www.rtve.es/m/alacarta/videos/cuentame-como-paso/cuentame-como-paso-t16-ultimo-minuto-nuestra-vida-capitulo-276/2969138/?media=tve',
        'only_matching': True,
@@ -111,58 +72,102 @@ class RTVEALaCartaIE(InfoExtractor):

    def _real_initialize(self):
        user_agent_b64 = base64.b64encode(std_headers['User-Agent'].encode('utf-8')).decode('utf-8')
-        manager_info = self._download_json(
+        self._manager = self._download_json(
            'http://www.rtve.es/odin/loki/' + user_agent_b64,
-            None, 'Fetching manager info')
-        self._manager = manager_info['manager']
+            None, 'Fetching manager info')['manager']
+
+    @staticmethod
+    def _decrypt_url(png):
+        encrypted_data = io.BytesIO(compat_b64decode(png)[8:])
+        while True:
+            length = compat_struct_unpack('!I', encrypted_data.read(4))[0]
+            chunk_type = encrypted_data.read(4)
+            if chunk_type == b'IEND':
+                break
+            data = encrypted_data.read(length)
+            if chunk_type == b'tEXt':
+                alphabet_data, text = data.split(b'\0')
+                quality, url_data = text.split(b'%%')
+                alphabet = []
+                e = 0
+                d = 0
+                for l in _bytes_to_chr(alphabet_data):
+                    if d == 0:
+                        alphabet.append(l)
+                        d = e = (e + 1) % 4
+                    else:
+                        d -= 1
+                url = ''
+                f = 0
+                e = 3
+                b = 1
+                for letter in _bytes_to_chr(url_data):
+                    if f == 0:
+                        l = int(letter) * 10
+                        f = 1
+                    else:
+                        if e == 0:
+                            l += int(letter)
+                            url += alphabet[l]
+                            e = (b + 3) % 4
+                            f = 0
+                            b += 1
+                        else:
+                            e -= 1
+
+                yield quality.decode(), url
+            encrypted_data.read(4)  # CRC
+
+    def _extract_png_formats(self, video_id):
+        png = self._download_webpage(
+            'http://www.rtve.es/ztnr/movil/thumbnail/%s/videos/%s.png' % (self._manager, video_id),
+            video_id, 'Downloading url information', query={'q': 'v2'})
+        q = qualities(['Media', 'Alta', 'HQ', 'HD_READY', 'HD_FULL'])
+        formats = []
+        for quality, video_url in self._decrypt_url(png):
+            ext = determine_ext(video_url)
+            if ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    video_url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id='hls', fatal=False))
+            elif ext == 'mpd':
+                formats.extend(self._extract_mpd_formats(
+                    video_url, video_id, 'dash', fatal=False))
+            else:
+                formats.append({
+                    'format_id': quality,
+                    'quality': q(quality),
+                    'url': video_url,
+                })
+        self._sort_formats(formats)
+        return formats

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
+        video_id = self._match_id(url)
        info = self._download_json(
            'http://www.rtve.es/api/videos/%s/config/alacarta_videos.json' % video_id,
            video_id)['page']['items'][0]
        if info['state'] == 'DESPU':
            raise ExtractorError('The video is no longer available', expected=True)
-        title = info['title']
-        png_url = 'http://www.rtve.es/ztnr/movil/thumbnail/%s/videos/%s.png' % (self._manager, video_id)
-        png_request = sanitized_Request(png_url)
-        png_request.add_header('Referer', url)
-        png = self._download_webpage(png_request, video_id, 'Downloading url information')
-        video_url = _decrypt_url(png)
-        ext = determine_ext(video_url)
-
-        formats = []
-        if not video_url.endswith('.f4m') and ext != 'm3u8':
-            if '?' not in video_url:
-                video_url = video_url.replace('resources/', 'auth/resources/')
-            video_url = video_url.replace('.net.rtve', '.multimedia.cdn.rtve')
-
-        if ext == 'm3u8':
-            formats.extend(self._extract_m3u8_formats(
-                video_url, video_id, ext='mp4', entry_protocol='m3u8_native',
-                m3u8_id='hls', fatal=False))
-        elif ext == 'f4m':
-            formats.extend(self._extract_f4m_formats(
-                video_url, video_id, f4m_id='hds', fatal=False))
-        else:
-            formats.append({
-                'url': video_url,
-            })
-        self._sort_formats(formats)
+        title = info['title'].strip()
+        formats = self._extract_png_formats(video_id)

        subtitles = None
-        if info.get('sbtFile') is not None:
-            subtitles = self.extract_subtitles(video_id, info['sbtFile'])
+        sbt_file = info.get('sbtFile')
+        if sbt_file:
+            subtitles = self.extract_subtitles(video_id, sbt_file)
+
+        is_live = info.get('live') is True

        return {
            'id': video_id,
-            'title': title,
+            'title': self._live_title(title) if is_live else title,
            'formats': formats,
            'thumbnail': info.get('image'),
-            'page_url': url,
            'subtitles': subtitles,
-            'duration': float_or_none(info.get('duration'), scale=1000),
+            'duration': float_or_none(info.get('duration'), 1000),
+            'is_live': is_live,
+            'series': info.get('programTitle'),
        }

    def _get_subtitles(self, video_id, sub_file):
@@ -174,48 +179,26 @@ class RTVEALaCartaIE(InfoExtractor):
            for s in subs)


-class RTVEInfantilIE(InfoExtractor):
+class RTVEInfantilIE(RTVEALaCartaIE):
    IE_NAME = 'rtve.es:infantil'
    IE_DESC = 'RTVE infantil'
-    _VALID_URL = r'https?://(?:www\.)?rtve\.es/infantil/serie/(?P<show>[^/]*)/video/(?P<short_title>[^/]*)/(?P<id>[0-9]+)/'
+    _VALID_URL = r'https?://(?:www\.)?rtve\.es/infantil/serie/[^/]+/video/[^/]+/(?P<id>[0-9]+)/'

    _TESTS = [{
        'url': 'http://www.rtve.es/infantil/serie/cleo/video/maneras-vivir/3040283/',
-        'md5': '915319587b33720b8e0357caaa6617e6',
+        'md5': '5747454717aedf9f9fdf212d1bcfc48d',
        'info_dict': {
            'id': '3040283',
            'ext': 'mp4',
            'title': 'Maneras de vivir',
-            'thumbnail': 'http://www.rtve.es/resources/jpg/6/5/1426182947956.JPG',
+            'thumbnail': r're:https?://.+/1426182947956\.JPG',
            'duration': 357.958,
        },
+        'expected_warnings': ['Failed to download MPD manifest', 'Failed to download m3u8 information'],
    }]

-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        info = self._download_json(
-            'http://www.rtve.es/api/videos/%s/config/alacarta_videos.json' % video_id,
-            video_id)['page']['items'][0]

-        webpage = self._download_webpage(url, video_id)
-        vidplayer_id = self._search_regex(
-            r' id="vidplayer([0-9]+)"', webpage, 'internal video ID')
-
-        png_url = 'http://www.rtve.es/ztnr/movil/thumbnail/default/videos/%s.png' % vidplayer_id
-        png = self._download_webpage(png_url, video_id, 'Downloading url information')
-        video_url = _decrypt_url(png)
-
-        return {
-            'id': video_id,
-            'ext': 'mp4',
-            'title': info['title'],
-            'url': video_url,
-            'thumbnail': info.get('image'),
-            'duration': float_or_none(info.get('duration'), scale=1000),
-        }
-
-
-class RTVELiveIE(InfoExtractor):
+class RTVELiveIE(RTVEALaCartaIE):
    IE_NAME = 'rtve.es:live'
    IE_DESC = 'RTVE.es live streams'
    _VALID_URL = r'https?://(?:www\.)?rtve\.es/directo/(?P<id>[a-zA-Z0-9-]+)'
@@ -225,7 +208,7 @@ class RTVELiveIE(InfoExtractor):
        'info_dict': {
            'id': 'la-1',
            'ext': 'mp4',
-            'title': 're:^La 1 [0-9]{4}-[0-9]{2}-[0-9]{2}Z[0-9]{6}$',
+            'title': 're:^La 1 [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
        },
        'params': {
            'skip_download': 'live stream',
@@ -234,29 +217,22 @@ class RTVELiveIE(InfoExtractor):

    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
-        start_time = time.gmtime()
        video_id = mobj.group('id')

        webpage = self._download_webpage(url, video_id)
        title = remove_end(self._og_search_title(webpage), ' en directo en RTVE.es')
        title = remove_start(title, 'Estoy viendo ')
-        title += ' ' + time.strftime('%Y-%m-%dZ%H%M%S', start_time)

        vidplayer_id = self._search_regex(
            (r'playerId=player([0-9]+)',
             r'class=["\'].*?\blive_mod\b.*?["\'][^>]+data-assetid=["\'](\d+)',
             r'data-id=["\'](\d+)'),
            webpage, 'internal video ID')
-        png_url = 'http://www.rtve.es/ztnr/movil/thumbnail/amonet/videos/%s.png' % vidplayer_id
-        png = self._download_webpage(png_url, video_id, 'Downloading url information')
-        m3u8_url = _decrypt_url(png)
-        formats = self._extract_m3u8_formats(m3u8_url, video_id, ext='mp4')
-        self._sort_formats(formats)

        return {
            'id': video_id,
-            'title': title,
-            'formats': formats,
+            'title': self._live_title(title),
+            'formats': self._extract_png_formats(vidplayer_id),
            'is_live': True,
        }

@@ -10,7 +10,7 @@ from ..utils import (

 class SBSIE(InfoExtractor):
    IE_DESC = 'sbs.com.au'
-    _VALID_URL = r'https?://(?:www\.)?sbs\.com\.au/(?:ondemand(?:/video/(?:single/)?|.*?\bplay=)|news/(?:embeds/)?video/)(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://(?:www\.)?sbs\.com\.au/(?:ondemand(?:/video/(?:single/)?|.*?\bplay=|/watch/)|news/(?:embeds/)?video/)(?P<id>[0-9]+)'

    _TESTS = [{
        # Original URL is handled by the generic IE which finds the iframe:
@@ -43,6 +43,9 @@ class SBSIE(InfoExtractor):
    }, {
        'url': 'https://www.sbs.com.au/news/embeds/video/1840778819866',
        'only_matching': True,
+    }, {
+        'url': 'https://www.sbs.com.au/ondemand/watch/1698704451971',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
@@ -2,12 +2,18 @@
 from __future__ import unicode_literals

 from .common import InfoExtractor
-from ..utils import js_to_json
+from ..utils import (
+    get_element_by_class,
+    int_or_none,
+    remove_start,
+    strip_or_none,
+    unified_strdate,
+)


 class ScreencastOMaticIE(InfoExtractor):
-    _VALID_URL = r'https?://screencast-o-matic\.com/watch/(?P<id>[0-9a-zA-Z]+)'
-    _TEST = {
+    _VALID_URL = r'https?://screencast-o-matic\.com/(?:(?:watch|player)/|embed\?.*?\bsc=)(?P<id>[0-9a-zA-Z]+)'
+    _TESTS = [{
        'url': 'http://screencast-o-matic.com/watch/c2lD3BeOPl',
        'md5': '483583cb80d92588f15ccbedd90f0c18',
        'info_dict': {
@@ -16,22 +22,30 @@ class ScreencastOMaticIE(InfoExtractor):
            'title': 'Welcome to 3-4 Philosophy @ DECV!',
            'thumbnail': r're:^https?://.*\.jpg$',
            'description': 'as the title says! also: some general info re 1) VCE philosophy and 2) distance learning.',
-            'duration': 369.163,
+            'duration': 369,
+            'upload_date': '20141216',
        }
-    }
+    }, {
+        'url': 'http://screencast-o-matic.com/player/c2lD3BeOPl',
+        'only_matching': True,
+    }, {
+        'url': 'http://screencast-o-matic.com/embed?ff=true&sc=cbV2r4Q5TL&fromPH=true&a=1',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-
-        jwplayer_data = self._parse_json(
-            self._search_regex(
-                r"(?s)jwplayer\('mp4Player'\).setup\((\{.*?\})\);", webpage, 'setup code'),
-            video_id, transform_source=js_to_json)
-
-        info_dict = self._parse_jwplayer_data(jwplayer_data, video_id, require_title=False)
-        info_dict.update({
-            'title': self._og_search_title(webpage),
-            'description': self._og_search_description(webpage),
+        webpage = self._download_webpage(
+            'https://screencast-o-matic.com/player/' + video_id, video_id)
+        info = self._parse_html5_media_entries(url, webpage, video_id)[0]
+        info.update({
+            'id': video_id,
+            'title': get_element_by_class('overlayTitle', webpage),
+            'description': strip_or_none(get_element_by_class('overlayDescription', webpage)) or None,
+            'duration': int_or_none(self._search_regex(
+                r'player\.duration\s*=\s*function\(\)\s*{\s*return\s+(\d+);\s*};',
+                webpage, 'duration', default=None)),
+            'upload_date': unified_strdate(remove_start(
+                get_element_by_class('overlayPublished', webpage), 'Published: ')),
        })
-        return info_dict
+        return info
@@ -51,13 +51,16 @@ class ShahidIE(ShahidBaseIE):
    _NETRC_MACHINE = 'shahid'
    _VALID_URL = r'https?://shahid\.mbc\.net/ar/(?:serie|show|movie)s/[^/]+/(?P<type>episode|clip|movie)-(?P<id>\d+)'
    _TESTS = [{
-        'url': 'https://shahid.mbc.net/ar/shows/%D9%85%D8%AC%D9%84%D8%B3-%D8%A7%D9%84%D8%B4%D8%A8%D8%A7%D8%A8-%D8%A7%D9%84%D9%85%D9%88%D8%B3%D9%85-1-%D9%83%D9%84%D9%8A%D8%A8-1/clip-275286',
+        'url': 'https://shahid.mbc.net/ar/shows/%D9%85%D8%AA%D8%AD%D9%81-%D8%A7%D9%84%D8%AF%D8%AD%D9%8A%D8%AD-%D8%A7%D9%84%D9%85%D9%88%D8%B3%D9%85-1-%D9%83%D9%84%D9%8A%D8%A8-1/clip-816924',
        'info_dict': {
-            'id': '275286',
+            'id': '816924',
            'ext': 'mp4',
-            'title': 'مجلس الشباب الموسم 1 كليب 1',
-            'timestamp': 1506988800,
-            'upload_date': '20171003',
+            'title': 'متحف الدحيح الموسم 1 كليب 1',
+            'timestamp': 1602806400,
+            'upload_date': '20201016',
+            'description': 'برومو',
+            'duration': 22,
+            'categories': ['كوميديا'],
        },
        'params': {
            # m3u8 download
@@ -109,12 +112,15 @@ class ShahidIE(ShahidBaseIE):
            page_type = 'episode'

        playout = self._call_api(
-            'playout/url/' + video_id, video_id)['playout']
+            'playout/new/url/' + video_id, video_id)['playout']

        if not self._downloader.params.get('allow_unplayable_formats') and playout.get('drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)

-        formats = self._extract_m3u8_formats(playout['url'], video_id, 'mp4')
+        formats = self._extract_m3u8_formats(re.sub(
+            # https://docs.aws.amazon.com/mediapackage/latest/ug/manifest-filtering.html
+            r'aws\.manifestfilter=[\w:;,-]+&?',
+            '', playout['url']), video_id, 'mp4')
        self._sort_formats(formats)

        # video = self._call_api(
@@ -6,9 +6,9 @@ from .mtv import MTVServicesInfoExtractor

 class SouthParkIE(MTVServicesInfoExtractor):
    IE_NAME = 'southpark.cc.com'
-    _VALID_URL = r'https?://(?:www\.)?(?P<url>southpark\.cc\.com/(?:clips|(?:full-)?episodes|collections)/(?P<id>.+?)(\?|#|$))'
+    _VALID_URL = r'https?://(?:www\.)?(?P<url>southpark(?:\.cc|studios)\.com/(?:clips|(?:full-)?episodes|collections)/(?P<id>.+?)(\?|#|$))'

-    _FEED_URL = 'http://www.southparkstudios.com/feeds/video-player/mrss'
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'

    _TESTS = [{
        'url': 'http://southpark.cc.com/clips/104437/bat-daded#tab=featured',
@@ -23,8 +23,20 @@ class SouthParkIE(MTVServicesInfoExtractor):
    }, {
        'url': 'http://southpark.cc.com/collections/7758/fan-favorites/1',
        'only_matching': True,
+    }, {
+        'url': 'https://www.southparkstudios.com/episodes/h4o269/south-park-stunning-and-brave-season-19-ep-1',
+        'only_matching': True,
    }]

+    def _get_feed_query(self, uri):
+        return {
+            'accountOverride': 'intl.mtvi.com',
+            'arcEp': 'shared.southpark.global',
+            'ep': '90877963',
+            'imageEp': 'shared.southpark.global',
+            'mgid': uri,
+        }
+

 class SouthParkEsIE(SouthParkIE):
    IE_NAME = 'southpark.cc.com:español'
@@ -1,82 +1,105 @@
 # coding: utf-8
 from __future__ import unicode_literals

-import re
-
 from .common import InfoExtractor
+from ..compat import (
+    compat_parse_qs,
+    compat_urllib_parse_urlparse,
+)
 from ..utils import (
+    clean_html,
+    float_or_none,
+    int_or_none,
    parse_iso8601,
-    sanitized_Request,
+    strip_or_none,
+    try_get,
 )


 class SportDeutschlandIE(InfoExtractor):
-    _VALID_URL = r'https?://sportdeutschland\.tv/(?P<sport>[^/?#]+)/(?P<id>[^?#/]+)(?:$|[?#])'
+    _VALID_URL = r'https?://sportdeutschland\.tv/(?P<id>(?:[^/]+/)?[^?#/&]+)'
    _TESTS = [{
        'url': 'https://sportdeutschland.tv/badminton/re-live-deutsche-meisterschaften-2020-halbfinals?playlistId=0',
        'info_dict': {
-            'id': 're-live-deutsche-meisterschaften-2020-halbfinals',
+            'id': '5318cac0275701382770543d7edaf0a0',
            'ext': 'mp4',
-            'title': 're:Re-live: Deutsche Meisterschaften 2020.*Halbfinals',
-            'categories': ['Badminton-Deutschland'],
-            'view_count': int,
-            'thumbnail': r're:^https?://.*\.(?:jpg|png)$',
-            'timestamp': int,
-            'upload_date': '20200201',
-            'description': 're:.*',  # meaningless description for THIS video
+            'title': 'Re-live: Deutsche Meisterschaften 2020 - Halbfinals - Teil 1',
+            'duration': 16106.36,
        },
+        'params': {
+            'noplaylist': True,
+            # m3u8 download
+            'skip_download': True,
+        },
+    }, {
+        'url': 'https://sportdeutschland.tv/badminton/re-live-deutsche-meisterschaften-2020-halbfinals?playlistId=0',
+        'info_dict': {
+            'id': 'c6e2fdd01f63013854c47054d2ab776f',
+            'title': 'Re-live: Deutsche Meisterschaften 2020 - Halbfinals',
+            'description': 'md5:5263ff4c31c04bb780c9f91130b48530',
+            'duration': 31397,
+        },
+        'playlist_count': 2,
+    }, {
+        'url': 'https://sportdeutschland.tv/freeride-world-tour-2021-fieberbrunn-oesterreich',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
-        sport_id = mobj.group('sport')
-
-        api_url = 'https://proxy.vidibusdynamic.net/ssl/backend.sportdeutschland.tv/api/permalinks/%s/%s?access_token=true' % (
-            sport_id, video_id)
-        req = sanitized_Request(api_url, headers={
-            'Accept': 'application/vnd.vidibus.v2.html+json',
-            'Referer': url,
-        })
-        data = self._download_json(req, video_id)
-
+        display_id = self._match_id(url)
+        data = self._download_json(
+            'https://backend.sportdeutschland.tv/api/permalinks/' + display_id,
+            display_id, query={'access_token': 'true'})
        asset = data['asset']
-        categories = [data['section']['title']]
-
-        formats = []
-        smil_url = asset['video']
-        if '.smil' in smil_url:
-            m3u8_url = smil_url.replace('.smil', '.m3u8')
-            formats.extend(
-                self._extract_m3u8_formats(m3u8_url, video_id, ext='mp4'))
-
-            smil_doc = self._download_xml(
-                smil_url, video_id, note='Downloading SMIL metadata')
-            base_url_el = smil_doc.find('./head/meta')
-            if base_url_el:
-                base_url = base_url_el.attrib['base']
-            formats.extend([{
-                'format_id': 'rmtp',
-                'url': base_url if base_url_el else n.attrib['src'],
-                'play_path': n.attrib['src'],
-                'ext': 'flv',
-                'preference': -100,
-                'format_note': 'Seems to fail at example stream',
-            } for n in smil_doc.findall('./body/video')])
-        else:
-            formats.append({'url': smil_url})
-
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'formats': formats,
-            'title': asset['title'],
-            'thumbnail': asset.get('image'),
-            'description': asset.get('teaser'),
-            'duration': asset.get('duration'),
-            'categories': categories,
-            'view_count': asset.get('views'),
-            'rtmp_live': asset.get('live'),
-            'timestamp': parse_iso8601(asset.get('date')),
+        title = (asset.get('title') or asset['label']).strip()
+        asset_id = asset.get('id') or asset.get('uuid')
+        info = {
+            'id': asset_id,
+            'title': title,
+            'description': clean_html(asset.get('body') or asset.get('description')) or asset.get('teaser'),
+            'duration': int_or_none(asset.get('seconds')),
        }
+        videos = asset.get('videos') or []
+        if len(videos) > 1:
+            playlist_id = compat_parse_qs(compat_urllib_parse_urlparse(url).query).get('playlistId', [None])[0]
+            if playlist_id:
+                if self._downloader.params.get('noplaylist'):
+                    videos = [videos[int(playlist_id)]]
+                    self.to_screen('Downloading just a single video because of --no-playlist')
+                else:
+                    self.to_screen('Downloading playlist %s - add --no-playlist to just download video' % asset_id)
+
+            def entries():
+                for i, video in enumerate(videos, 1):
+                    video_id = video.get('uuid')
+                    video_url = video.get('url')
+                    if not (video_id and video_url):
+                        continue
+                    formats = self._extract_m3u8_formats(
+                        video_url.replace('.smil', '.m3u8'), video_id, 'mp4', fatal=False)
+                    if not formats:
+                        continue
+                    yield {
+                        'id': video_id,
+                        'formats': formats,
+                        'title': title + ' - ' + (video.get('label') or 'Teil %d' % i),
+                        'duration': float_or_none(video.get('duration')),
+                    }
+            info.update({
+                '_type': 'multi_video',
+                'entries': entries(),
+            })
+        else:
+            formats = self._extract_m3u8_formats(
+                videos[0]['url'].replace('.smil', '.m3u8'), asset_id, 'mp4')
+            section_title = strip_or_none(try_get(data, lambda x: x['section']['title']))
+            info.update({
+                'formats': formats,
+                'display_id': asset.get('permalink'),
+                'thumbnail': try_get(asset, lambda x: x['images'][0]),
+                'categories': [section_title] if section_title else None,
+                'view_count': int_or_none(asset.get('views')),
+                'is_live': asset.get('is_live') is True,
+                'timestamp': parse_iso8601(asset.get('date') or asset.get('published_at')),
+            })
+        return info
@@ -1,19 +1,61 @@
 from __future__ import unicode_literals

-import re
-
 from .common import InfoExtractor
+from ..compat import compat_str
 from ..utils import (
    clean_html,
+    clean_podcast_url,
    ExtractorError,
    int_or_none,
    str_or_none,
    try_get,
+    url_or_none,
 )


-class StitcherIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?stitcher\.com/(?:podcast|show)/(?:[^/]+/)+e(?:pisode)?/(?:(?P<display_id>[^/#?&]+?)-)?(?P<id>\d+)(?:[/#?&]|$)'
+class StitcherBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?stitcher\.com/(?:podcast|show)/'
+
+    def _call_api(self, path, video_id, query):
+        resp = self._download_json(
+            'https://api.prod.stitcher.com/' + path,
+            video_id, query=query)
+        error_massage = try_get(resp, lambda x: x['errors'][0]['message'])
+        if error_massage:
+            raise ExtractorError(error_massage, expected=True)
+        return resp['data']
+
+    def _extract_description(self, data):
+        return clean_html(data.get('html_description') or data.get('description'))
+
+    def _extract_audio_url(self, episode):
+        return url_or_none(episode.get('audio_url') or episode.get('guid'))
+
+    def _extract_show_info(self, show):
+        return {
+            'thumbnail': show.get('image_base_url'),
+            'series': show.get('title'),
+        }
+
+    def _extract_episode(self, episode, audio_url, show_info):
+        info = {
+            'id': compat_str(episode['id']),
+            'display_id': episode.get('slug'),
+            'title': episode['title'].strip(),
+            'description': self._extract_description(episode),
+            'duration': int_or_none(episode.get('duration')),
+            'url': clean_podcast_url(audio_url),
+            'vcodec': 'none',
+            'timestamp': int_or_none(episode.get('date_published')),
+            'season_number': int_or_none(episode.get('season')),
+            'season_id': str_or_none(episode.get('season_id')),
+        }
+        info.update(show_info)
+        return info
+
+
+class StitcherIE(StitcherBaseIE):
+    _VALID_URL = StitcherBaseIE._VALID_URL_BASE + r'(?:[^/]+/)+e(?:pisode)?/(?:[^/#?&]+-)?(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.stitcher.com/podcast/the-talking-machines/e/40789481?autoplay=true',
        'md5': 'e9635098e0da10b21a0e2b85585530f6',
@@ -24,8 +66,9 @@ class StitcherIE(InfoExtractor):
            'description': 'md5:547adb4081864be114ae3831b4c2b42f',
            'duration': 1604,
            'thumbnail': r're:^https?://.*\.jpg',
-            'upload_date': '20180126',
-            'timestamp': 1516989316,
+            'upload_date': '20151008',
+            'timestamp': 1444285800,
+            'series': 'Talking Machines',
        },
    }, {
        'url': 'http://www.stitcher.com/podcast/panoply/vulture-tv/e/the-rare-hourlong-comedy-plus-40846275?autoplay=true',
@@ -55,33 +98,47 @@ class StitcherIE(InfoExtractor):
    }]

    def _real_extract(self, url):
-        display_id, audio_id = re.match(self._VALID_URL, url).groups()
+        audio_id = self._match_id(url)
+        data = self._call_api(
+            'shows/episodes', audio_id, {'episode_ids': audio_id})
+        episode = data['episodes'][0]
+        audio_url = self._extract_audio_url(episode)
+        if not audio_url:
+            self.raise_login_required()
+        show = try_get(data, lambda x: x['shows'][0], dict) or {}
+        return self._extract_episode(
+            episode, audio_url, self._extract_show_info(show))

-        resp = self._download_json(
-            'https://api.prod.stitcher.com/episode/' + audio_id,
-            display_id or audio_id)
-        episode = try_get(resp, lambda x: x['data']['episodes'][0], dict)
-        if not episode:
-            raise ExtractorError(resp['errors'][0]['message'], expected=True)

-        title = episode['title'].strip()
-        audio_url = episode['audio_url']
+class StitcherShowIE(StitcherBaseIE):
+    _VALID_URL = StitcherBaseIE._VALID_URL_BASE + r'(?P<id>[^/#?&]+)/?(?:[?#&]|$)'
+    _TESTS = [{
+        'url': 'http://www.stitcher.com/podcast/the-talking-machines',
+        'info_dict': {
+            'id': 'the-talking-machines',
+            'title': 'Talking Machines',
+            'description': 'md5:831f0995e40f26c10231af39cf1ebf0b',
+        },
+        'playlist_mincount': 106,
+    }, {
+        'url': 'https://www.stitcher.com/show/the-talking-machines',
+        'only_matching': True,
+    }]

-        thumbnail = None
-        show_id = episode.get('show_id')
-        if show_id and episode.get('classic_id') != -1:
-            thumbnail = 'https://stitcher-classic.imgix.net/feedimages/%s.jpg' % show_id
+    def _real_extract(self, url):
+        show_slug = self._match_id(url)
+        data = self._call_api(
+            'search/show/%s/allEpisodes' % show_slug, show_slug, {'count': 10000})
+        show = try_get(data, lambda x: x['shows'][0], dict) or {}
+        show_info = self._extract_show_info(show)

-        return {
-            'id': audio_id,
-            'display_id': display_id,
-            'title': title,
-            'description': clean_html(episode.get('html_description') or episode.get('description')),
-            'duration': int_or_none(episode.get('duration')),
-            'thumbnail': thumbnail,
-            'url': audio_url,
-            'vcodec': 'none',
-            'timestamp': int_or_none(episode.get('date_created')),
-            'season_number': int_or_none(episode.get('season')),
-            'season_id': str_or_none(episode.get('season_id')),
-        }
+        entries = []
+        for episode in (data.get('episodes') or []):
+            audio_url = self._extract_audio_url(episode)
+            if not audio_url:
+                continue
+            entries.append(self._extract_episode(episode, audio_url, show_info))
+
+        return self.playlist_result(
+            entries, show_slug, show.get('title'),
+            self._extract_description(show))
@@ -146,18 +146,19 @@ class SVTPlayIE(SVTPlayBaseIE):
                        )
                        (?P<svt_id>[^/?#&]+)|
                        https?://(?:www\.)?(?:svtplay|oppetarkiv)\.se/(?:video|klipp|kanaler)/(?P<id>[^/?#&]+)
+                        (?:.*?modalId=(?P<modal_id>[\da-zA-Z-]+))?
                    )
                    '''
    _TESTS = [{
-        'url': 'https://www.svtplay.se/video/26194546/det-har-ar-himlen',
+        'url': 'https://www.svtplay.se/video/30479064',
        'md5': '2382036fd6f8c994856c323fe51c426e',
        'info_dict': {
-            'id': 'jNwpV9P',
+            'id': '8zVbDPA',
            'ext': 'mp4',
-            'title': 'Det här är himlen',
-            'timestamp': 1586044800,
-            'upload_date': '20200405',
-            'duration': 3515,
+            'title': 'Designdrömmar i Stenungsund',
+            'timestamp': 1615770000,
+            'upload_date': '20210315',
+            'duration': 3519,
            'thumbnail': r're:^https?://(?:.*[\.-]jpg|www.svtstatic.se/image/.*)$',
            'age_limit': 0,
            'subtitles': {
@@ -173,6 +174,9 @@ class SVTPlayIE(SVTPlayBaseIE):
            # AssertionError: Expected test_SVTPlay_jNwpV9P.mp4 to be at least 9.77KiB, but it's only 864.00B
            'skip_download': True,
        },
+    }, {
+        'url': 'https://www.svtplay.se/video/30479064/husdrommar/husdrommar-sasong-8-designdrommar-i-stenungsund?modalId=8zVbDPA',
+        'only_matching': True,
    }, {
        # geo restricted to Sweden
        'url': 'http://www.oppetarkiv.se/video/5219710/trollflojten',
@@ -219,7 +223,8 @@ class SVTPlayIE(SVTPlayBaseIE):

    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
-        video_id, svt_id = mobj.group('id', 'svt_id')
+        video_id = mobj.group('id')
+        svt_id = mobj.group('svt_id') or mobj.group('modal_id')

        if svt_id:
            return self._extract_by_video_id(svt_id)
@@ -254,6 +259,7 @@ class SVTPlayIE(SVTPlayBaseIE):
        if not svt_id:
            svt_id = self._search_regex(
                (r'<video[^>]+data-video-id=["\']([\da-zA-Z-]+)',
+                 r'<[^>]+\bdata-rt=["\']top-area-play-button["\'][^>]+\bhref=["\'][^"\']*video/%s/[^"\']*\bmodalId=([\da-zA-Z-]+)' % re.escape(video_id),
                 r'["\']videoSvtId["\']\s*:\s*["\']([\da-zA-Z-]+)',
                 r'["\']videoSvtId\\?["\']\s*:\s*\\?["\']([\da-zA-Z-]+)',
                 r'"content"\s*:\s*{.*?"id"\s*:\s*"([\da-zA-Z-]+)"',
@@ -143,7 +143,10 @@ class TikTokIE(TikTokBaseIE):
        props_data = try_get(json_data, lambda x: x['props'], expected_type=dict)

        # Chech statusCode for success
-        if props_data.get('pageProps').get('statusCode') == 0:
+        status = props_data.get('pageProps').get('statusCode')
+        if status == 0:
            return self._extract_aweme(props_data, webpage, url)
+        elif status == 10216:
+            raise ExtractorError('This video is private', expected=True)

        raise ExtractorError('Video not available', video_id=video_id)
@@ -9,6 +9,7 @@ from ..utils import (
    int_or_none,
    remove_start,
    smuggle_url,
+    strip_or_none,
    try_get,
 )

@@ -25,6 +26,10 @@ class TVerIE(InfoExtractor):
    }, {
        'url': 'https://tver.jp/episode/79622438',
        'only_matching': True,
+    }, {
+        # subtitle = ' '
+        'url': 'https://tver.jp/corner/f0068870',
+        'only_matching': True,
    }]
    _TOKEN = None
    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/default_default/index.html?videoId=%s'
@@ -47,8 +52,12 @@ class TVerIE(InfoExtractor):
        }

        if service == 'cx':
+            title = main['title']
+            subtitle = strip_or_none(main.get('subtitle'))
+            if subtitle:
+                title += ' - ' + subtitle
            info.update({
-                'title': main.get('subtitle') or main['title'],
+                'title': title,
                'url': 'https://i.fod.fujitv.co.jp/plus7/web/%s/%s.html' % (p_id[:4], p_id),
                'ie_key': 'FujiTVFODPlus7',
            })
@@ -23,6 +23,8 @@ class VGTVIE(XstreamIE):
        'fvn.no/fvntv': 'fvntv',
        'aftenposten.no/webtv': 'aptv',
        'ap.vgtv.no/webtv': 'aptv',
+        'tv.aftonbladet.se': 'abtv',
+        # obsolete URL schemas, kept in order to save one HTTP redirect
        'tv.aftonbladet.se/abtv': 'abtv',
        'www.aftonbladet.se/tv': 'abtv',
    }
@@ -140,6 +142,10 @@ class VGTVIE(XstreamIE):
            'url': 'http://www.vgtv.no/#!/video/127205/inside-the-mind-of-favela-funk',
            'only_matching': True,
        },
+        {
+            'url': 'https://tv.aftonbladet.se/video/36015/vulkanutbrott-i-rymden-nu-slapper-nasa-bilderna',
+            'only_matching': True,
+        },
        {
            'url': 'http://tv.aftonbladet.se/abtv/articles/36015',
            'only_matching': True,
@@ -24,6 +24,7 @@ from ..utils import (
    merge_dicts,
    OnDemandPagedList,
    parse_filesize,
+    parse_iso8601,
    RegexNotFoundError,
    sanitized_Request,
    smuggle_url,
@@ -74,25 +75,28 @@ class VimeoBaseInfoExtractor(InfoExtractor):
                    expected=True)
            raise ExtractorError('Unable to log in')

-    def _verify_video_password(self, url, video_id, webpage):
+    def _get_video_password(self):
        password = self._downloader.params.get('videopassword')
        if password is None:
-            raise ExtractorError('This video is protected by a password, use the --video-password option', expected=True)
-        token, vuid = self._extract_xsrft_and_vuid(webpage)
-        data = urlencode_postdata({
-            'password': password,
-            'token': token,
-        })
+            raise ExtractorError(
+                'This video is protected by a password, use the --video-password option',
+                expected=True)
+        return password
+
+    def _verify_video_password(self, url, video_id, password, token, vuid):
        if url.startswith('http://'):
            # vimeo only supports https now, but the user can give an http url
            url = url.replace('http://', 'https://')
-        password_request = sanitized_Request(url + '/password', data)
-        password_request.add_header('Content-Type', 'application/x-www-form-urlencoded')
-        password_request.add_header('Referer', url)
        self._set_vimeo_cookie('vuid', vuid)
        return self._download_webpage(
-            password_request, video_id,
-            'Verifying the password', 'Wrong password')
+            url + '/password', video_id, 'Verifying the password',
+            'Wrong password', data=urlencode_postdata({
+                'password': password,
+                'token': token,
+            }), headers={
+                'Content-Type': 'application/x-www-form-urlencoded',
+                'Referer': url,
+            })

    def _extract_xsrft_and_vuid(self, webpage):
        xsrft = self._search_regex(
@@ -273,7 +277,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
                            )?
                        (?:videos?/)?
                        (?P<id>[0-9]+)
-                        (?:/[\da-f]+)?
+                        (?:/(?P<unlisted_hash>[\da-f]{10}))?
                        /?(?:[?&].*)?(?:[#].*)?$
                    '''
    IE_NAME = 'vimeo'
@@ -326,9 +330,9 @@ class VimeoIE(VimeoBaseInfoExtractor):
                'id': '54469442',
                'ext': 'mp4',
                'title': 'Kathy Sierra: Building the minimum Badass User, Business of Software 2012',
-                'uploader': 'The BLN & Business of Software',
-                'uploader_url': r're:https?://(?:www\.)?vimeo\.com/theblnbusinessofsoftware',
-                'uploader_id': 'theblnbusinessofsoftware',
+                'uploader': 'Business of Software',
+                'uploader_url': r're:https?://(?:www\.)?vimeo\.com/businessofsoftware',
+                'uploader_id': 'businessofsoftware',
                'duration': 3610,
                'description': None,
            },
@@ -463,6 +467,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
                'skip_download': True,
            },
            'expected_warnings': ['Unable to download JSON metadata'],
+            'skip': 'this page is no longer available.',
        },
        {
            'url': 'http://player.vimeo.com/video/68375962',
@@ -563,9 +568,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
        return urls[0] if urls else None

    def _verify_player_video_password(self, url, video_id, headers):
-        password = self._downloader.params.get('videopassword')
-        if password is None:
-            raise ExtractorError('This video is protected by a password, use the --video-password option', expected=True)
+        password = self._get_video_password()
        data = urlencode_postdata({
            'password': base64.b64encode(password.encode()),
        })
@@ -628,11 +631,37 @@ class VimeoIE(VimeoBaseInfoExtractor):
        if 'Referer' not in headers:
            headers['Referer'] = url

-        channel_id = self._search_regex(
-            r'vimeo\.com/channels/([^/]+)', url, 'channel id', default=None)
-
        # Extract ID from URL
-        video_id = self._match_id(url)
+        video_id, unlisted_hash = re.match(self._VALID_URL, url).groups()
+        if unlisted_hash:
+            token = self._download_json(
+                'https://vimeo.com/_rv/jwt', video_id, headers={
+                    'X-Requested-With': 'XMLHttpRequest'
+                })['token']
+            video = self._download_json(
+                'https://api.vimeo.com/videos/%s:%s' % (video_id, unlisted_hash),
+                video_id, headers={
+                    'Authorization': 'jwt ' + token,
+                }, query={
+                    'fields': 'config_url,created_time,description,license,metadata.connections.comments.total,metadata.connections.likes.total,release_time,stats.plays',
+                })
+            info = self._parse_config(self._download_json(
+                video['config_url'], video_id), video_id)
+            self._vimeo_sort_formats(info['formats'])
+            get_timestamp = lambda x: parse_iso8601(video.get(x + '_time'))
+            info.update({
+                'description': video.get('description'),
+                'license': video.get('license'),
+                'release_timestamp': get_timestamp('release'),
+                'timestamp': get_timestamp('created'),
+                'view_count': int_or_none(try_get(video, lambda x: x['stats']['plays'])),
+            })
+            connections = try_get(
+                video, lambda x: x['metadata']['connections'], dict) or {}
+            for k in ('comment', 'like'):
+                info[k + '_count'] = int_or_none(try_get(connections, lambda x: x[k + 's']['total']))
+            return info
+
        orig_url = url
        is_pro = 'vimeopro.com/' in url
        is_player = '://player.vimeo.com/video/' in url
@@ -722,7 +751,10 @@ class VimeoIE(VimeoBaseInfoExtractor):
            if re.search(r'<form[^>]+?id="pw_form"', webpage) is not None:
                if '_video_password_verified' in data:
                    raise ExtractorError('video password verification failed!')
-                self._verify_video_password(redirect_url, video_id, webpage)
+                video_password = self._get_video_password()
+                token, vuid = self._extract_xsrft_and_vuid(webpage)
+                self._verify_video_password(
+                    redirect_url, video_id, video_password, token, vuid)
                return self._real_extract(
                    smuggle_url(redirect_url, {'_video_password_verified': 'verified'}))
            else:
@@ -808,6 +840,8 @@ class VimeoIE(VimeoBaseInfoExtractor):
                r'<link[^>]+rel=["\']license["\'][^>]+href=(["\'])(?P<license>(?:(?!\1).)+)\1',
                webpage, 'license', default=None, group='license')

+        channel_id = self._search_regex(
+            r'vimeo\.com/channels/([^/]+)', url, 'channel id', default=None)
        channel_url = 'https://vimeo.com/channels/%s' % channel_id if channel_id else None

        info_dict = {
@@ -991,11 +1025,15 @@ class VimeoAlbumIE(VimeoBaseInfoExtractor):
        }
        if hashed_pass:
            query['_hashed_pass'] = hashed_pass
-        videos = self._download_json(
-            'https://api.vimeo.com/albums/%s/videos' % album_id,
-            album_id, 'Downloading page %d' % api_page, query=query, headers={
-                'Authorization': 'jwt ' + authorization,
-            })['data']
+        try:
+            videos = self._download_json(
+                'https://api.vimeo.com/albums/%s/videos' % album_id,
+                album_id, 'Downloading page %d' % api_page, query=query, headers={
+                    'Authorization': 'jwt ' + authorization,
+                })['data']
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
+                return
        for video in videos:
            link = video.get('link')
            if not link:
@@ -1110,10 +1148,23 @@ class VimeoReviewIE(VimeoBaseInfoExtractor):

    def _real_extract(self, url):
        page_url, video_id = re.match(self._VALID_URL, url).groups()
-        clip_data = self._download_json(
-            page_url.replace('/review/', '/review/data/'),
-            video_id)['clipData']
-        config_url = clip_data['configUrl']
+        data = self._download_json(
+            page_url.replace('/review/', '/review/data/'), video_id)
+        if data.get('isLocked') is True:
+            video_password = self._get_video_password()
+            viewer = self._download_json(
+                'https://vimeo.com/_rv/viewer', video_id)
+            webpage = self._verify_video_password(
+                'https://vimeo.com/' + video_id, video_id,
+                video_password, viewer['xsrft'], viewer['vuid'])
+            clip_page_config = self._parse_json(self._search_regex(
+                r'window\.vimeo\.clip_page_config\s*=\s*({.+?});',
+                webpage, 'clip page config'), video_id)
+            config_url = clip_page_config['player']['config_url']
+            clip_data = clip_page_config.get('clip') or {}
+        else:
+            clip_data = data['clipData']
+            config_url = clip_data['configUrl']
        config = self._download_json(config_url, video_id)
        info_dict = self._parse_config(config, video_id)
        source_format = self._extract_original_format(
@@ -113,7 +113,7 @@ class VLiveIE(VLiveBaseIE):
            raise ExtractorError('Unable to log in', expected=True)

    def _call_api(self, path_template, video_id, fields=None, limit=None):
-        query = {'appId': self._APP_ID, 'gcc': 'KR'}
+        query = {'appId': self._APP_ID, 'gcc': 'KR', 'platformType': 'PC'}
        if fields:
            query['fields'] = fields
        if limit:
@@ -7,6 +7,8 @@ from ..compat import compat_urllib_parse_unquote
 from ..utils import (
    ExtractorError,
    int_or_none,
+    try_get,
+    unified_timestamp,
 )


@@ -19,14 +21,17 @@ class VoxMediaVolumeIE(OnceIE):

        setup = self._parse_json(self._search_regex(
            r'setup\s*=\s*({.+});', webpage, 'setup'), video_id)
-        video_data = setup.get('video') or {}
+        player_setup = setup.get('player_setup') or setup
+        video_data = player_setup.get('video') or {}
+        formatted_metadata = video_data.get('formatted_metadata') or {}
        info = {
            'id': video_id,
-            'title': video_data.get('title_short'),
+            'title': player_setup.get('title') or video_data.get('title_short'),
            'description': video_data.get('description_long') or video_data.get('description_short'),
-            'thumbnail': video_data.get('brightcove_thumbnail')
+            'thumbnail': formatted_metadata.get('thumbnail') or video_data.get('brightcove_thumbnail'),
+            'timestamp': unified_timestamp(formatted_metadata.get('video_publish_date')),
        }
-        asset = setup.get('asset') or setup.get('params') or {}
+        asset = try_get(setup, lambda x: x['embed_assets']['chorus'], dict) or {}

        formats = []
        hls_url = asset.get('hls_url')
@@ -47,6 +52,7 @@ class VoxMediaVolumeIE(OnceIE):
        if formats:
            self._sort_formats(formats)
            info['formats'] = formats
+            info['duration'] = int_or_none(asset.get('duration'))
            return info

        for provider_video_type in ('ooyala', 'youtube', 'brightcove'):
@@ -84,7 +90,7 @@ class VoxMediaIE(InfoExtractor):
    }, {
        # Volume embed, Youtube
        'url': 'http://www.theverge.com/2014/10/21/7025853/google-nexus-6-hands-on-photos-video-android-phablet',
-        'md5': '4c8f4a0937752b437c3ebc0ed24802b5',
+        'md5': 'fd19aa0cf3a0eea515d4fd5c8c0e9d68',
        'info_dict': {
            'id': 'Gy8Md3Eky38',
            'ext': 'mp4',
@@ -93,6 +99,7 @@ class VoxMediaIE(InfoExtractor):
            'uploader_id': 'TheVerge',
            'upload_date': '20141021',
            'uploader': 'The Verge',
+            'timestamp': 1413907200,
        },
        'add_ie': ['Youtube'],
        'skip': 'similar to the previous test',
@@ -100,13 +107,13 @@ class VoxMediaIE(InfoExtractor):
        # Volume embed, Youtube
        'url': 'http://www.vox.com/2016/3/31/11336640/mississippi-lgbt-religious-freedom-bill',
        'info_dict': {
-            'id': 'YCjDnX-Xzhg',
+            'id': '22986359b',
            'ext': 'mp4',
            'title': "Mississippi's laws are so bad that its anti-LGBTQ law isn't needed to allow discrimination",
            'description': 'md5:fc1317922057de31cd74bce91eb1c66c',
-            'uploader_id': 'voxdotcom',
            'upload_date': '20150915',
-            'uploader': 'Vox',
+            'timestamp': 1442332800,
+            'duration': 285,
        },
        'add_ie': ['Youtube'],
        'skip': 'similar to the previous test',
@@ -160,6 +167,9 @@ class VoxMediaIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Post-Post-PC CEO: The Full Code Conference Video of Microsoft\'s Satya Nadella',
            'description': 'The longtime veteran was chosen earlier this year as the software giant\'s third leader in its history.',
+            'timestamp': 1402938000,
+            'upload_date': '20140616',
+            'duration': 4114,
        },
        'add_ie': ['VoxMediaVolume'],
    }]
@@ -182,17 +182,20 @@ class VVVVIDIE(InfoExtractor):
            if not embed_code:
                continue
            embed_code = ds(embed_code)
-            if video_type in ('video/rcs', 'video/kenc'):
-                if video_type == 'video/kenc':
-                    kenc = self._download_json(
-                        'https://www.vvvvid.it/kenc', video_id, query={
-                            'action': 'kt',
-                            'conn_id': self._conn_id,
-                            'url': embed_code,
-                        }, fatal=False) or {}
-                    kenc_message = kenc.get('message')
-                    if kenc_message:
-                        embed_code += '?' + ds(kenc_message)
+            if video_type == 'video/kenc':
+                embed_code = re.sub(r'https?(://[^/]+)/z/', r'https\1/i/', embed_code).replace('/manifest.f4m', '/master.m3u8')
+                kenc = self._download_json(
+                    'https://www.vvvvid.it/kenc', video_id, query={
+                        'action': 'kt',
+                        'conn_id': self._conn_id,
+                        'url': embed_code,
+                    }, fatal=False) or {}
+                kenc_message = kenc.get('message')
+                if kenc_message:
+                    embed_code += '?' + ds(kenc_message)
+                formats.extend(self._extract_m3u8_formats(
+                    embed_code, video_id, 'mp4', m3u8_id='hls', fatal=False))
+            elif video_type == 'video/rcs':
                formats.extend(self._extract_akamai_formats(embed_code, video_id))
            elif video_type == 'video/youtube':
                info.update({
@@ -0,0 +1,163 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    determine_ext,
+    parse_duration,
+    urlencode_postdata,
+    ExtractorError,
+)
+
+
+class WimTVIE(InfoExtractor):
+    _player = None
+    _UUID_RE = r'[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12}'
+    _VALID_URL = r'''(?x)
+        https?://platform.wim.tv/
+        (?:
+            (?:embed/)?\?
+            |\#/webtv/.+?/
+        )
+        (?P<type>vod|live|cast)[=/]
+        (?P<id>%s).*?''' % _UUID_RE
+    _TESTS = [{
+        # vod stream
+        'url': 'https://platform.wim.tv/embed/?vod=db29fb32-bade-47b6-a3a6-cb69fe80267a',
+        'md5': 'db29fb32-bade-47b6-a3a6-cb69fe80267a',
+        'info_dict': {
+            'id': 'db29fb32-bade-47b6-a3a6-cb69fe80267a',
+            'ext': 'mp4',
+            'title': 'AMA SUPERCROSS 2020 - R2 ST. LOUIS',
+            'duration': 6481,
+            'thumbnail': r're:https?://.+?/thumbnail/.+?/720$'
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        # live stream
+        'url': 'https://platform.wim.tv/embed/?live=28e22c22-49db-40f3-8c37-8cbb0ff44556&autostart=true',
+        'info_dict': {
+            'id': '28e22c22-49db-40f3-8c37-8cbb0ff44556',
+            'ext': 'mp4',
+            'title': 'Streaming MSmotorTV',
+            'is_live': True,
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        'url': 'https://platform.wim.tv/#/webtv/automotornews/vod/422492b6-539e-474d-9c6b-68c9d5893365',
+        'only_matching': True,
+    }, {
+        'url': 'https://platform.wim.tv/#/webtv/renzoarborechannel/cast/f47e0d15-5b45-455e-bf0d-dba8ffa96365',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _extract_urls(webpage):
+        return [
+            mobj.group('url')
+            for mobj in re.finditer(
+                r'<iframe[^>]+src=["\'](?P<url>%s)' % WimTVIE._VALID_URL,
+                webpage)]
+
+    def _real_initialize(self):
+        if not self._player:
+            self._get_player_data()
+
+    def _get_player_data(self):
+        msg_id = 'Player data'
+        self._player = {}
+
+        datas = [{
+            'url': 'https://platform.wim.tv/common/libs/player/wimtv/wim-rest.js',
+            'vars': [{
+                'regex': r'appAuth = "(.+?)"',
+                'variable': 'app_auth',
+            }]
+        }, {
+            'url': 'https://platform.wim.tv/common/config/endpointconfig.js',
+            'vars': [{
+                'regex': r'PRODUCTION_HOSTNAME_THUMB = "(.+?)"',
+                'variable': 'thumb_server',
+            }, {
+                'regex': r'PRODUCTION_HOSTNAME_THUMB\s*\+\s*"(.+?)"',
+                'variable': 'thumb_server_path',
+            }]
+        }]
+
+        for data in datas:
+            temp = self._download_webpage(data['url'], msg_id)
+            for var in data['vars']:
+                val = self._search_regex(var['regex'], temp, msg_id)
+                if not val:
+                    raise ExtractorError('%s not found' % var['variable'])
+                self._player[var['variable']] = val
+
+    def _generate_token(self):
+        json = self._download_json(
+            'https://platform.wim.tv/wimtv-server/oauth/token', 'Token generation',
+            headers={'Authorization': 'Basic %s' % self._player['app_auth']},
+            data=urlencode_postdata({'grant_type': 'client_credentials'}))
+        token = json.get('access_token')
+        if not token:
+            raise ExtractorError('access token not generated')
+        return token
+
+    def _generate_thumbnail(self, thumb_id, width='720'):
+        if not thumb_id or not self._player.get('thumb_server'):
+            return None
+        if not self._player.get('thumb_server_path'):
+            self._player['thumb_server_path'] = ''
+        return '%s%s/asset/thumbnail/%s/%s' % (
+            self._player['thumb_server'],
+            self._player['thumb_server_path'],
+            thumb_id, width)
+
+    def _real_extract(self, url):
+        urlc = re.match(self._VALID_URL, url).groupdict()
+        video_id = urlc['id']
+        stream_type = is_live = None
+        if urlc['type'] in {'live', 'cast'}:
+            stream_type = urlc['type'] + '/channel'
+            is_live = True
+        else:
+            stream_type = 'vod'
+            is_live = False
+        token = self._generate_token()
+        json = self._download_json(
+            'https://platform.wim.tv/wimtv-server/api/public/%s/%s/play' % (
+                stream_type, video_id), video_id,
+            headers={'Authorization': 'Bearer %s' % token,
+                     'Content-Type': 'application/json'},
+            data=bytes('{}', 'utf-8'))
+
+        formats = []
+        for src in json.get('srcs') or []:
+            if src.get('mimeType') == 'application/x-mpegurl':
+                formats.extend(
+                    self._extract_m3u8_formats(
+                        src.get('uniqueStreamer'), video_id, 'mp4'))
+            if src.get('mimeType') == 'video/flash':
+                formats.append({
+                    'format_id': 'rtmp',
+                    'url': src.get('uniqueStreamer'),
+                    'ext': determine_ext(src.get('uniqueStreamer'), 'flv'),
+                    'rtmp_live': is_live,
+                })
+        json = json.get('resource')
+        thumb = self._generate_thumbnail(json.get('thumbnailId'))
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': json.get('title') or json.get('name'),
+            'duration': parse_duration(json.get('duration')),
+            'formats': formats,
+            'thumbnail': thumb,
+            'is_live': is_live,
+        }
@@ -154,7 +154,7 @@ class YoukuIE(InfoExtractor):
        # request basic data
        basic_data_params = {
            'vid': video_id,
-            'ccode': '0590',
+            'ccode': '0532',
            'client_ip': '192.168.1.1',
            'utid': cna,
            'client_ts': time.time() / 1000,
@@ -2,6 +2,7 @@

 from __future__ import unicode_literals

+import calendar
 import hashlib
 import itertools
 import json
@@ -15,7 +16,6 @@ from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
    compat_chr,
    compat_HTTPError,
-    compat_kwargs,
    compat_parse_qs,
    compat_str,
    compat_urllib_parse_unquote_plus,
@@ -25,8 +25,10 @@ from ..compat import (
 )
 from ..jsinterp import JSInterpreter
 from ..utils import (
+    bool_or_none,
    clean_html,
    dict_get,
+    datetime_from_str,
    ExtractorError,
    format_field,
    float_or_none,
@@ -46,7 +48,7 @@ from ..utils import (
    update_url_query,
    url_or_none,
    urlencode_postdata,
-    urljoin,
+    urljoin
 )


@@ -260,27 +262,30 @@ class YoutubeBaseInfoExtractor(InfoExtractor):

        return True

-    def _download_webpage_handle(self, *args, **kwargs):
-        query = kwargs.get('query', {}).copy()
-        kwargs['query'] = query
-        return super(YoutubeBaseInfoExtractor, self)._download_webpage_handle(
-            *args, **compat_kwargs(kwargs))
+    def _initialize_consent(self):
+        cookies = self._get_cookies('https://www.youtube.com/')
+        if cookies.get('__Secure-3PSID'):
+            return
+        consent_id = None
+        consent = cookies.get('CONSENT')
+        if consent:
+            if 'YES' in consent.value:
+                return
+            consent_id = self._search_regex(
+                r'PENDING\+(\d+)', consent.value, 'consent', default=None)
+        if not consent_id:
+            consent_id = random.randint(100, 999)
+        self._set_cookie('.youtube.com', 'CONSENT', 'YES+cb.20210328-17-p0.en+FX+%s' % consent_id)

    def _real_initialize(self):
+        self._initialize_consent()
        if self._downloader is None:
            return
        if not self._login():
            return

-    _DEFAULT_API_DATA = {
-        'context': {
-            'client': {
-                'clientName': 'WEB',
-                'clientVersion': '2.20210301.08.00',
-            }
-        },
-    }
-
+    _YT_WEB_CLIENT_VERSION = '2.20210407.08.00'
+    _YT_INNERTUBE_API_KEY = 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8'
    _YT_INITIAL_DATA_RE = r'(?:window\s*\[\s*["\']ytInitialData["\']\s*\]|ytInitialData)\s*=\s*({.+?})\s*;'
    _YT_INITIAL_PLAYER_RESPONSE_RE = r'ytInitialPlayerResponse\s*=\s*({.+?})\s*;'
    _YT_INITIAL_BOUNDARY_RE = r'(?:var\s+meta|</script|\n)'
@@ -294,19 +299,23 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
        return "SAPISIDHASH %s_%s" % (time_now, sapisidhash)

    def _call_api(self, ep, query, video_id, fatal=True, headers=None,
-                  note='Downloading API JSON', errnote='Unable to download API page'):
-        data = self._DEFAULT_API_DATA.copy()
+                  note='Downloading API JSON', errnote='Unable to download API page',
+                  context=None, api_key=None):
+
+        data = {'context': context} if context else {'context': self._extract_context()}
        data.update(query)
-        headers = headers or {}
-        headers.update({'content-type': 'application/json'})
-        auth = self._generate_sapisidhash_header()
-        if auth is not None:
-            headers.update({'Authorization': auth, 'X-Origin': 'https://www.youtube.com'})
+        real_headers = self._generate_api_headers()
+        real_headers.update({'content-type': 'application/json'})
+        if headers:
+            real_headers.update(headers)
        return self._download_json(
            'https://www.youtube.com/youtubei/v1/%s' % ep,
            video_id=video_id, fatal=fatal, note=note, errnote=errnote,
-            data=json.dumps(data).encode('utf8'), headers=headers,
-            query={'key': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8'})
+            data=json.dumps(data).encode('utf8'), headers=real_headers,
+            query={'key': api_key or self._extract_api_key()})
+
+    def _extract_api_key(self, ytcfg=None):
+        return try_get(ytcfg, lambda x: x['INNERTUBE_API_KEY'], compat_str) or self._YT_INNERTUBE_API_KEY

    def _extract_yt_initial_data(self, video_id, webpage):
        return self._parse_json(
@@ -315,11 +324,72 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 self._YT_INITIAL_DATA_RE), webpage, 'yt initial data'),
            video_id)

+    def _extract_identity_token(self, webpage, item_id):
+        ytcfg = self._extract_ytcfg(item_id, webpage)
+        if ytcfg:
+            token = try_get(ytcfg, lambda x: x['ID_TOKEN'], compat_str)
+            if token:
+                return token
+        return self._search_regex(
+            r'\bID_TOKEN["\']\s*:\s*["\'](.+?)["\']', webpage,
+            'identity token', default=None)
+
+    @staticmethod
+    def _extract_account_syncid(data):
+        """Extract syncId required to download private playlists of secondary channels"""
+        sync_ids = (
+            try_get(data, lambda x: x['responseContext']['mainAppWebResponseContext']['datasyncId'], compat_str)
+            or '').split("||")
+        if len(sync_ids) >= 2 and sync_ids[1]:
+            # datasyncid is of the form "channel_syncid||user_syncid" for secondary channel
+            # and just "user_syncid||" for primary channel. We only want the channel_syncid
+            return sync_ids[0]
+
    def _extract_ytcfg(self, video_id, webpage):
        return self._parse_json(
            self._search_regex(
                r'ytcfg\.set\s*\(\s*({.+?})\s*\)\s*;', webpage, 'ytcfg',
-                default='{}'), video_id, fatal=False)
+                default='{}'), video_id, fatal=False) or {}
+
+    def __extract_client_version(self, ytcfg):
+        return try_get(ytcfg, lambda x: x['INNERTUBE_CLIENT_VERSION'], compat_str) or self._YT_WEB_CLIENT_VERSION
+
+    def _extract_context(self, ytcfg=None):
+        context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'], dict)
+        if context:
+            return context
+
+        # Recreate the client context (required)
+        client_version = self.__extract_client_version(ytcfg)
+        client_name = try_get(ytcfg, lambda x: x['INNERTUBE_CLIENT_NAME'], compat_str) or 'WEB'
+        context = {
+            'client': {
+                'clientName': client_name,
+                'clientVersion': client_version,
+            }
+        }
+        visitor_data = try_get(ytcfg, lambda x: x['VISITOR_DATA'], compat_str)
+        if visitor_data:
+            context['client']['visitorData'] = visitor_data
+        return context
+
+    def _generate_api_headers(self, ytcfg=None, identity_token=None, account_syncid=None, visitor_data=None):
+        headers = {
+            'X-YouTube-Client-Name': '1',
+            'X-YouTube-Client-Version': self.__extract_client_version(ytcfg),
+        }
+        if identity_token:
+            headers['x-youtube-identity-token'] = identity_token
+        if account_syncid:
+            headers['X-Goog-PageId'] = account_syncid
+            headers['X-Goog-AuthUser'] = 0
+        if visitor_data:
+            headers['x-goog-visitor-id'] = visitor_data
+        auth = self._generate_sapisidhash_header()
+        if auth is not None:
+            headers['Authorization'] = auth
+            headers['X-Origin'] = 'https://www.youtube.com'
+        return headers

    def _extract_video(self, renderer):
        video_id = renderer.get('videoId')
@@ -342,7 +412,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
            (lambda x: x['ownerText']['runs'][0]['text'],
             lambda x: x['shortBylineText']['runs'][0]['text']), compat_str)
        return {
-            '_type': 'url_transparent',
+            '_type': 'url',
            'ie_key': YoutubeIE.ie_key(),
            'id': video_id,
            'url': video_id,
@@ -1209,6 +1279,23 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            'url': 'https://www.youtube.com/watch?v=nGC3D_FkCmg',
            'only_matching': True,
        },
+        {
+            # restricted location, https://github.com/ytdl-org/youtube-dl/issues/28685
+            'url': 'cBvYw8_A0vQ',
+            'info_dict': {
+                'id': 'cBvYw8_A0vQ',
+                'ext': 'mp4',
+                'title': '4K Ueno Okachimachi  Street  Scenes  上野御徒町歩き',
+                'description': 'md5:ea770e474b7cd6722b4c95b833c03630',
+                'upload_date': '20201120',
+                'uploader': 'Walk around Japan',
+                'uploader_id': 'UC3o_t8PzBmXf5S9b7GLx1Mw',
+                'uploader_url': r're:https?://(?:www\.)?youtube\.com/channel/UC3o_t8PzBmXf5S9b7GLx1Mw',
+            },
+            'params': {
+                'skip_download': True,
+            },
+        },
    ]

    def __init__(self, *args, **kwargs):
@@ -1462,23 +1549,297 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            (r'%s\s*%s' % (regex, self._YT_INITIAL_BOUNDARY_RE),
             regex), webpage, name, default='{}'), video_id, fatal=False)

+    @staticmethod
+    def parse_time_text(time_text):
+        """
+        Parse the comment time text
+        time_text is in the format 'X units ago (edited)'
+        """
+        time_text_split = time_text.split(' ')
+        if len(time_text_split) >= 3:
+            return datetime_from_str('now-%s%s' % (time_text_split[0], time_text_split[1]), precision='auto')
+
+    @staticmethod
+    def _join_text_entries(runs):
+        text = None
+        for run in runs:
+            if not isinstance(run, dict):
+                continue
+            sub_text = try_get(run, lambda x: x['text'], compat_str)
+            if sub_text:
+                if not text:
+                    text = sub_text
+                    continue
+                text += sub_text
+        return text
+
+    def _extract_comment(self, comment_renderer, parent=None):
+        comment_id = comment_renderer.get('commentId')
+        if not comment_id:
+            return
+        comment_text_runs = try_get(comment_renderer, lambda x: x['contentText']['runs']) or []
+        text = self._join_text_entries(comment_text_runs) or ''
+        comment_time_text = try_get(comment_renderer, lambda x: x['publishedTimeText']['runs']) or []
+        time_text = self._join_text_entries(comment_time_text)
+        timestamp = calendar.timegm(self.parse_time_text(time_text).timetuple())
+        author = try_get(comment_renderer, lambda x: x['authorText']['simpleText'], compat_str)
+        author_id = try_get(comment_renderer,
+                            lambda x: x['authorEndpoint']['browseEndpoint']['browseId'], compat_str)
+        votes = str_to_int(try_get(comment_renderer, (lambda x: x['voteCount']['simpleText'],
+                                                      lambda x: x['likeCount']), compat_str)) or 0
+        author_thumbnail = try_get(comment_renderer,
+                                   lambda x: x['authorThumbnail']['thumbnails'][-1]['url'], compat_str)
+
+        author_is_uploader = try_get(comment_renderer, lambda x: x['authorIsChannelOwner'], bool)
+        is_liked = try_get(comment_renderer, lambda x: x['isLiked'], bool)
+        return {
+            'id': comment_id,
+            'text': text,
+            'timestamp': timestamp,
+            'time_text': time_text,
+            'like_count': votes,
+            'is_favorited': is_liked,
+            'author': author,
+            'author_id': author_id,
+            'author_thumbnail': author_thumbnail,
+            'author_is_uploader': author_is_uploader,
+            'parent': parent or 'root'
+        }
+
+    def _comment_entries(self, root_continuation_data, identity_token, account_syncid,
+                         ytcfg, session_token_list, parent=None, comment_counts=None):
+
+        def extract_thread(parent_renderer):
+            contents = try_get(parent_renderer, lambda x: x['contents'], list) or []
+            if not parent:
+                comment_counts[2] = 0
+            for content in contents:
+                comment_thread_renderer = try_get(content, lambda x: x['commentThreadRenderer'])
+                comment_renderer = try_get(
+                    comment_thread_renderer, (lambda x: x['comment']['commentRenderer'], dict)) or try_get(
+                    content, (lambda x: x['commentRenderer'], dict))
+
+                if not comment_renderer:
+                    continue
+                comment = self._extract_comment(comment_renderer, parent)
+                if not comment:
+                    continue
+                comment_counts[0] += 1
+                yield comment
+                # Attempt to get the replies
+                comment_replies_renderer = try_get(
+                    comment_thread_renderer, lambda x: x['replies']['commentRepliesRenderer'], dict)
+
+                if comment_replies_renderer:
+                    comment_counts[2] += 1
+                    comment_entries_iter = self._comment_entries(
+                        comment_replies_renderer, identity_token, account_syncid, ytcfg,
+                        parent=comment.get('id'), session_token_list=session_token_list,
+                        comment_counts=comment_counts)
+
+                    for reply_comment in comment_entries_iter:
+                        yield reply_comment
+
+        if not comment_counts:
+            # comment so far, est. total comments, current comment thread #
+            comment_counts = [0, 0, 0]
+
+        # TODO: Generalize the download code with TabIE
+        context = self._extract_context(ytcfg)
+        visitor_data = try_get(context, lambda x: x['client']['visitorData'], compat_str)
+        continuation = YoutubeTabIE._extract_continuation(root_continuation_data)  # TODO
+        first_continuation = False
+        if parent is None:
+            first_continuation = True
+
+        for page_num in itertools.count(0):
+            if not continuation:
+                break
+            headers = self._generate_api_headers(ytcfg, identity_token, account_syncid, visitor_data)
+            retries = self._downloader.params.get('extractor_retries', 3)
+            count = -1
+            last_error = None
+
+            while count < retries:
+                count += 1
+                if last_error:
+                    self.report_warning('%s. Retrying ...' % last_error)
+                try:
+                    query = {
+                        'ctoken': continuation['ctoken'],
+                        'pbj': 1,
+                        'type': 'next',
+                    }
+                    if parent:
+                        query['action_get_comment_replies'] = 1
+                    else:
+                        query['action_get_comments'] = 1
+
+                    comment_prog_str = '(%d/%d)' % (comment_counts[0], comment_counts[1])
+                    if page_num == 0:
+                        if first_continuation:
+                            note_prefix = 'Downloading initial comment continuation page'
+                        else:
+                            note_prefix = '    Downloading comment reply thread %d %s' % (comment_counts[2], comment_prog_str)
+                    else:
+                        note_prefix = '%sDownloading comment%s page %d %s' % (
+                            '       ' if parent else '',
+                            ' replies' if parent else '',
+                            page_num,
+                            comment_prog_str)
+
+                    browse = self._download_json(
+                        'https://www.youtube.com/comment_service_ajax', None,
+                        '%s %s' % (note_prefix, '(retry #%d)' % count if count else ''),
+                        headers=headers, query=query,
+                        data=urlencode_postdata({
+                            'session_token': session_token_list[0]
+                        }))
+                except ExtractorError as e:
+                    if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503, 404, 413):
+                        if e.cause.code == 413:
+                            self.report_warning('Assumed end of comments (received HTTP Error 413)')
+                            return
+                        # Downloading page may result in intermittent 5xx HTTP error
+                        # Sometimes a 404 is also recieved. See: https://github.com/ytdl-org/youtube-dl/issues/28289
+                        last_error = 'HTTP Error %s' % e.cause.code
+                        if e.cause.code == 404:
+                            last_error = last_error + ' (this API is probably deprecated)'
+                        if count < retries:
+                            continue
+                    raise
+                else:
+                    session_token = try_get(browse, lambda x: x['xsrf_token'], compat_str)
+                    if session_token:
+                        session_token_list[0] = session_token
+
+                    response = try_get(browse,
+                                       (lambda x: x['response'],
+                                        lambda x: x[1]['response'])) or {}
+
+                    if response.get('continuationContents'):
+                        break
+
+                    # YouTube sometimes gives reload: now json if something went wrong (e.g. bad auth)
+                    if browse.get('reload'):
+                        raise ExtractorError('Invalid or missing params in continuation request', expected=False)
+
+                    # TODO: not tested, merged from old extractor
+                    err_msg = browse.get('externalErrorMessage')
+                    if err_msg:
+                        raise ExtractorError('YouTube said: %s' % err_msg, expected=False)
+
+                    # Youtube sometimes sends incomplete data
+                    # See: https://github.com/ytdl-org/youtube-dl/issues/28194
+                    last_error = 'Incomplete data received'
+                    if count >= retries:
+                        self._downloader.report_error(last_error)
+
+            if not response:
+                break
+            visitor_data = try_get(
+                response,
+                lambda x: x['responseContext']['webResponseContextExtensionData']['ytConfigData']['visitorData'],
+                compat_str) or visitor_data
+
+            known_continuation_renderers = {
+                'itemSectionContinuation': extract_thread,
+                'commentRepliesContinuation': extract_thread
+            }
+
+            # extract next root continuation from the results
+            continuation_contents = try_get(
+                response, lambda x: x['continuationContents'], dict) or {}
+
+            for key, value in continuation_contents.items():
+                if key not in known_continuation_renderers:
+                    continue
+                continuation_renderer = value
+
+                if first_continuation:
+                    first_continuation = False
+                    expected_comment_count = try_get(
+                        continuation_renderer,
+                        (lambda x: x['header']['commentsHeaderRenderer']['countText']['runs'][0]['text'],
+                         lambda x: x['header']['commentsHeaderRenderer']['commentsCount']['runs'][0]['text']),
+                        compat_str)
+
+                    if expected_comment_count:
+                        comment_counts[1] = str_to_int(expected_comment_count)
+                        self.to_screen('Downloading ~%d comments' % str_to_int(expected_comment_count))
+                        yield comment_counts[1]
+
+                    # TODO: cli arg.
+                    # 1/True for newest, 0/False for popular (default)
+                    comment_sort_index = int(True)
+                    sort_continuation_renderer = try_get(
+                        continuation_renderer,
+                        lambda x: x['header']['commentsHeaderRenderer']['sortMenu']['sortFilterSubMenuRenderer']['subMenuItems']
+                        [comment_sort_index]['continuation']['reloadContinuationData'], dict)
+                    # If this fails, the initial continuation page
+                    # starts off with popular anyways.
+                    if sort_continuation_renderer:
+                        continuation = YoutubeTabIE._build_continuation_query(
+                            continuation=sort_continuation_renderer.get('continuation'),
+                            ctp=sort_continuation_renderer.get('clickTrackingParams'))
+                        self.to_screen('Sorting comments by %s' % ('popular' if comment_sort_index == 0 else 'newest'))
+                        break
+
+                for entry in known_continuation_renderers[key](continuation_renderer):
+                    yield entry
+
+                continuation = YoutubeTabIE._extract_continuation(continuation_renderer)  # TODO
+                break
+
+    def _extract_comments(self, ytcfg, video_id, contents, webpage, xsrf_token):
+        """Entry for comment extraction"""
+        comments = []
+        known_entry_comment_renderers = (
+            'itemSectionRenderer',
+        )
+        estimated_total = 0
+        for entry in contents:
+            for key, renderer in entry.items():
+                if key not in known_entry_comment_renderers:
+                    continue
+
+                comment_iter = self._comment_entries(
+                    renderer,
+                    identity_token=self._extract_identity_token(webpage, item_id=video_id),
+                    account_syncid=self._extract_account_syncid(ytcfg),
+                    ytcfg=ytcfg,
+                    session_token_list=[xsrf_token])
+
+                for comment in comment_iter:
+                    if isinstance(comment, int):
+                        estimated_total = comment
+                        continue
+                    comments.append(comment)
+                break
+        self.to_screen('Downloaded %d/%d comments' % (len(comments), estimated_total))
+        return {
+            'comments': comments,
+            'comment_count': len(comments),
+        }
+
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
        video_id = self._match_id(url)
        base_url = self.http_scheme() + '//www.youtube.com/'
        webpage_url = base_url + 'watch?v=' + video_id
        webpage = self._download_webpage(
-            webpage_url + '&has_verified=1&bpctr=9999999999',
-            video_id, fatal=False)
+            webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)

        player_response = None
        if webpage:
            player_response = self._extract_yt_initial_variable(
                webpage, self._YT_INITIAL_PLAYER_RESPONSE_RE,
                video_id, 'initial player response')
+
+        ytcfg = self._extract_ytcfg(video_id, webpage)
        if not player_response:
            player_response = self._call_api(
-                'player', {'videoId': video_id}, video_id)
+                'player', {'videoId': video_id}, video_id, api_key=self._extract_api_key(ytcfg))

        playability_status = player_response.get('playabilityStatus') or {}
        if playability_status.get('reason') == 'Sign in to confirm your age':
@@ -1506,7 +1867,13 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        def get_text(x):
            if not x:
                return
-            return x.get('simpleText') or ''.join([r['text'] for r in x['runs']])
+            text = x.get('simpleText')
+            if text and isinstance(text, compat_str):
+                return text
+            runs = x.get('runs')
+            if not isinstance(runs, list):
+                return
+            return ''.join([r['text'] for r in runs if isinstance(r.get('text'), compat_str)])

        search_meta = (
            lambda x: self._html_search_meta(x, webpage, default=None)) \
@@ -1655,7 +2022,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    f['format_id'] = itag
                formats.append(f)

-        if self._downloader.params.get('youtube_include_dash_manifest'):
+        if self._downloader.params.get('youtube_include_dash_manifest', True):
            dash_manifest_url = streaming_data.get('dashManifestUrl')
            if dash_manifest_url:
                for f in self._extract_mpd_formats(
@@ -1723,6 +2090,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                thumbnail_url = thumbnail.get('url')
                if not thumbnail_url:
                    continue
+                # Sometimes youtube gives a wrong thumbnail URL. See:
+                # https://github.com/yt-dlp/yt-dlp/issues/233
+                # https://github.com/ytdl-org/youtube-dl/issues/28023
+                if 'maxresdefault' in thumbnail_url:
+                    thumbnail_url = thumbnail_url.split('?')[0]
                thumbnails.append({
                    'height': int_or_none(thumbnail.get('height')),
                    'url': thumbnail_url,
@@ -1775,7 +2147,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            'tags': keywords,
            'is_live': is_live,
            'playable_in_embed': playability_status.get('playableInEmbed'),
-            'was_live': video_details.get('isLiveContent')
+            'was_live': video_details.get('isLiveContent'),
        }

        pctr = try_get(
@@ -1851,13 +2223,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                'yt initial data')
        if not initial_data:
            initial_data = self._call_api(
-                'next', {'videoId': video_id}, video_id, fatal=False)
+                'next', {'videoId': video_id}, video_id, fatal=False, api_key=self._extract_api_key(ytcfg))

        if not is_live:
            try:
                # This will error if there is no livechat
                initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
                info['subtitles']['live_chat'] = [{
+                    'url': 'https://www.youtube.com/watch?v=%s' % video_id,  # url is needed to set cookies
                    'video_id': video_id,
                    'ext': 'json',
                    'protocol': 'youtube_live_chat_replay',
@@ -1951,7 +2324,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    info['channel'] = get_text(try_get(
                        vsir,
                        lambda x: x['owner']['videoOwnerRenderer']['title'],
-                        compat_str))
+                        dict))
                    rows = try_get(
                        vsir,
                        lambda x: x['metadataRowContainer']['metadataRowContainerRenderer']['rows'],
@@ -1992,6 +2365,35 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            if v:
                info[d_k] = v

+        is_private = bool_or_none(video_details.get('isPrivate'))
+        is_unlisted = bool_or_none(microformat.get('isUnlisted'))
+        is_membersonly = None
+        is_premium = None
+        if initial_data and is_private is not None:
+            is_membersonly = False
+            is_premium = False
+            contents = try_get(initial_data, lambda x: x['contents']['twoColumnWatchNextResults']['results']['results']['contents'], list)
+            for content in contents or []:
+                badges = try_get(content, lambda x: x['videoPrimaryInfoRenderer']['badges'], list)
+                for badge in badges or []:
+                    label = try_get(badge, lambda x: x['metadataBadgeRenderer']['label']) or ''
+                    if label.lower() == 'members only':
+                        is_membersonly = True
+                        break
+                    elif label.lower() == 'premium':
+                        is_premium = True
+                        break
+                if is_membersonly or is_premium:
+                    break
+
+        # TODO: Add this for playlists
+        info['availability'] = self._availability(
+            is_private=is_private,
+            needs_premium=is_premium,
+            needs_subscription=is_membersonly,
+            needs_auth=info['age_limit'] >= 18,
+            is_unlisted=None if is_private is None else is_unlisted)
+
        # get xsrf for annotations or comments
        get_annotations = self._downloader.params.get('writeannotations', False)
        get_comments = self._downloader.params.get('getcomments', False)
@@ -2024,156 +2426,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    errnote='Unable to download video annotations', fatal=False,
                    data=urlencode_postdata({xsrf_field_name: xsrf_token}))

-        # Get comments
-        # TODO: Refactor and move to seperate function
-        def extract_comments():
-            expected_video_comment_count = 0
-            video_comments = []
-            comment_xsrf = xsrf_token
-
-            def find_value(html, key, num_chars=2, separator='"'):
-                pos_begin = html.find(key) + len(key) + num_chars
-                pos_end = html.find(separator, pos_begin)
-                return html[pos_begin: pos_end]
-
-            def search_dict(partial, key):
-                if isinstance(partial, dict):
-                    for k, v in partial.items():
-                        if k == key:
-                            yield v
-                        else:
-                            for o in search_dict(v, key):
-                                yield o
-                elif isinstance(partial, list):
-                    for i in partial:
-                        for o in search_dict(i, key):
-                            yield o
-
-            continuations = []
-            if initial_data:
-                try:
-                    ncd = next(search_dict(initial_data, 'nextContinuationData'))
-                    continuations = [ncd['continuation']]
-                # Handle videos where comments have been disabled entirely
-                except StopIteration:
-                    pass
-
-            def get_continuation(continuation, session_token, replies=False):
-                query = {
-                    'pbj': 1,
-                    'ctoken': continuation,
-                }
-                if replies:
-                    query['action_get_comment_replies'] = 1
-                else:
-                    query['action_get_comments'] = 1
-
-                while True:
-                    content, handle = self._download_webpage_handle(
-                        'https://www.youtube.com/comment_service_ajax',
-                        video_id,
-                        note=False,
-                        expected_status=[413],
-                        data=urlencode_postdata({
-                            'session_token': session_token
-                        }),
-                        query=query,
-                        headers={
-                            'Accept': '*/*',
-                            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0',
-                            'X-YouTube-Client-Name': '1',
-                            'X-YouTube-Client-Version': '2.20201202.06.01'
-                        }
-                    )
-
-                    response_code = handle.getcode()
-                    if (response_code == 200):
-                        return self._parse_json(content, video_id)
-                    if (response_code == 413):
-                        return None
-                    raise ExtractorError('Unexpected HTTP error code: %s' % response_code)
-
-            first_continuation = True
-            chain_msg = ''
-            self.to_screen('Downloading comments')
-            while continuations:
-                continuation = continuations.pop()
-                comment_response = get_continuation(continuation, comment_xsrf)
-                if not comment_response:
-                    continue
-                if list(search_dict(comment_response, 'externalErrorMessage')):
-                    raise ExtractorError('Error returned from server: ' + next(search_dict(comment_response, 'externalErrorMessage')))
-
-                if 'continuationContents' not in comment_response['response']:
-                    # Something is wrong here. Youtube won't accept this continuation token for some reason and responds with a user satisfaction dialog (error?)
-                    continue
-                # not sure if this actually helps
-                if 'xsrf_token' in comment_response:
-                    comment_xsrf = comment_response['xsrf_token']
-
-                item_section = comment_response['response']['continuationContents']['itemSectionContinuation']
-                if first_continuation:
-                    expected_video_comment_count = int(item_section['header']['commentsHeaderRenderer']['countText']['runs'][0]['text'].replace(' Comments', '').replace('1 Comment', '1').replace(',', ''))
-                    first_continuation = False
-                if 'contents' not in item_section:
-                    # continuation returned no comments?
-                    # set an empty array as to not break the for loop
-                    item_section['contents'] = []
-
-                for meta_comment in item_section['contents']:
-                    comment = meta_comment['commentThreadRenderer']['comment']['commentRenderer']
-                    video_comments.append({
-                        'id': comment['commentId'],
-                        'text': ''.join([c['text'] for c in try_get(comment, lambda x: x['contentText']['runs'], list) or []]),
-                        'time_text': ''.join([c['text'] for c in comment['publishedTimeText']['runs']]),
-                        'author': comment.get('authorText', {}).get('simpleText', ''),
-                        'votes': comment.get('voteCount', {}).get('simpleText', '0'),
-                        'author_thumbnail': comment['authorThumbnail']['thumbnails'][-1]['url'],
-                        'parent': 'root'
-                    })
-                    if 'replies' not in meta_comment['commentThreadRenderer']:
-                        continue
-
-                    reply_continuations = [rcn['nextContinuationData']['continuation'] for rcn in meta_comment['commentThreadRenderer']['replies']['commentRepliesRenderer']['continuations']]
-                    while reply_continuations:
-                        time.sleep(1)
-                        continuation = reply_continuations.pop()
-                        replies_data = get_continuation(continuation, comment_xsrf, True)
-                        if not replies_data or 'continuationContents' not in replies_data[1]['response']:
-                            continue
-
-                        if self._downloader.params.get('verbose', False):
-                            chain_msg = ' (chain %s)' % comment['commentId']
-                        self.to_screen('Comments downloaded: %d of ~%d%s' % (len(video_comments), expected_video_comment_count, chain_msg))
-                        reply_comment_meta = replies_data[1]['response']['continuationContents']['commentRepliesContinuation']
-                        for reply_meta in reply_comment_meta.get('contents', {}):
-                            reply_comment = reply_meta['commentRenderer']
-                            video_comments.append({
-                                'id': reply_comment['commentId'],
-                                'text': ''.join([c['text'] for c in reply_comment['contentText']['runs']]),
-                                'time_text': ''.join([c['text'] for c in reply_comment['publishedTimeText']['runs']]),
-                                'author': reply_comment.get('authorText', {}).get('simpleText', ''),
-                                'votes': reply_comment.get('voteCount', {}).get('simpleText', '0'),
-                                'author_thumbnail': reply_comment['authorThumbnail']['thumbnails'][-1]['url'],
-                                'parent': comment['commentId']
-                            })
-                        if 'continuations' not in reply_comment_meta or len(reply_comment_meta['continuations']) == 0:
-                            continue
-                        reply_continuations += [rcn['nextContinuationData']['continuation'] for rcn in reply_comment_meta['continuations']]
-
-                self.to_screen('Comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
-                if 'continuations' in item_section:
-                    continuations += [ncd['nextContinuationData']['continuation'] for ncd in item_section['continuations']]
-                time.sleep(1)
-
-            self.to_screen('Total comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
-            return {
-                'comments': video_comments,
-                'comment_count': expected_video_comment_count
-            }
-
        if get_comments:
-            info['__post_extractor'] = extract_comments
+            info['__post_extractor'] = lambda: self._extract_comments(ytcfg, video_id, contents, webpage, xsrf_token)

        self.mark_watched(video_id, player_response)

@@ -2499,6 +2753,13 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
    }, {
        'url': 'https://www.youtube.com/TheYoungTurks/live',
        'only_matching': True,
+    }, {
+        'url': 'https://www.youtube.com/hashtag/cctv9',
+        'info_dict': {
+            'id': 'cctv9',
+            'title': '#cctv9',
+        },
+        'playlist_mincount': 350,
    }]

    @classmethod
@@ -2523,7 +2784,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
    def _extract_basic_item_renderer(item):
        # Modified from _extract_grid_item_renderer
        known_renderers = (
-            'playlistRenderer', 'videoRenderer', 'channelRenderer'
+            'playlistRenderer', 'videoRenderer', 'channelRenderer',
            'gridPlaylistRenderer', 'gridVideoRenderer', 'gridChannelRenderer'
        )
        for key, renderer in item.items():
@@ -2659,6 +2920,16 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            for entry in self._post_thread_entries(renderer):
                yield entry

+    r''' # unused
+    def _rich_grid_entries(self, contents):
+        for content in contents:
+            video_renderer = try_get(content, lambda x: x['richItemRenderer']['content']['videoRenderer'], dict)
+            if video_renderer:
+                entry = self._video_entry(video_renderer)
+                if entry:
+                    yield entry
+    '''
+
    @staticmethod
    def _build_continuation_query(continuation, ctp=None):
        query = {
@@ -2704,7 +2975,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            ctp = continuation_ep.get('clickTrackingParams')
            return YoutubeTabIE._build_continuation_query(continuation, ctp)

-    def _entries(self, tab, item_id, identity_token, account_syncid):
+    def _entries(self, tab, item_id, identity_token, account_syncid, ytcfg):

        def extract_entries(parent_renderer):  # this needs to called again for continuation to work with feeds
            contents = try_get(parent_renderer, lambda x: x['contents'], list) or []
@@ -2756,21 +3027,13 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
        for entry in extract_entries(parent_renderer):
            yield entry
        continuation = continuation_list[0]
-
-        headers = {
-            'x-youtube-client-name': '1',
-            'x-youtube-client-version': '2.20201112.04.01',
-        }
-        if identity_token:
-            headers['x-youtube-identity-token'] = identity_token
-
-        if account_syncid:
-            headers['X-Goog-PageId'] = account_syncid
-            headers['X-Goog-AuthUser'] = 0
+        context = self._extract_context(ytcfg)
+        visitor_data = try_get(context, lambda x: x['client']['visitorData'], compat_str)

        for page_num in itertools.count(1):
            if not continuation:
                break
+            headers = self._generate_api_headers(ytcfg, identity_token, account_syncid, visitor_data)
            retries = self._downloader.params.get('extractor_retries', 3)
            count = -1
            last_error = None
@@ -2780,12 +3043,14 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                    self.report_warning('%s. Retrying ...' % last_error)
                try:
                    response = self._call_api(
-                        ep="browse", fatal=True, headers=headers,
+                        ep='browse', fatal=True, headers=headers,
                        video_id='%s page %s' % (item_id, page_num),
                        query={
                            'continuation': continuation['continuation'],
                            'clickTracking': {'clickTrackingParams': continuation['itct']},
                        },
+                        context=context,
+                        api_key=self._extract_api_key(ytcfg),
                        note='Downloading API JSON%s' % (' (retry #%d)' % count if count else ''))
                except ExtractorError as e:
                    if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503, 404):
@@ -2811,6 +3076,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):

            if not response:
                break
+            visitor_data = try_get(
+                response, lambda x: x['responseContext']['visitorData'], compat_str) or visitor_data

            known_continuation_renderers = {
                'playlistVideoListContinuation': self._playlist_entries,
@@ -2841,9 +3108,9 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                'richItemRenderer': (extract_entries, 'contents'),  # for hashtag
                'backstagePostThreadRenderer': (self._post_thread_continuation_entries, 'contents')
            }
+            on_response_received = dict_get(response, ('onResponseReceivedActions', 'onResponseReceivedEndpoints'))
            continuation_items = try_get(
-                response,
-                lambda x: dict_get(x, ('onResponseReceivedActions', 'onResponseReceivedEndpoints'))[0]['appendContinuationItemsAction']['continuationItems'], list)
+                on_response_received, lambda x: x[0]['appendContinuationItemsAction']['continuationItems'], list)
            continuation_item = try_get(continuation_items, lambda x: x[0], dict) or {}
            video_items_renderer = None
            for key, value in continuation_item.items():
@@ -2901,10 +3168,10 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            channel_name = renderer.get('title')
            channel_url = renderer.get('channelUrl')
            channel_id = renderer.get('externalId')
-
-        if not renderer:
+        else:
            renderer = try_get(
                data, lambda x: x['metadata']['playlistMetadataRenderer'], dict)
+
        if renderer:
            title = renderer.get('title')
            description = renderer.get('description', '')
@@ -2930,11 +3197,12 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                'width': int_or_none(t.get('width')),
                'height': int_or_none(t.get('height')),
            })
-
        if playlist_id is None:
            playlist_id = item_id
        if title is None:
-            title = playlist_id
+            title = (
+                try_get(data, lambda x: x['header']['hashtagHeaderRenderer']['hashtag']['simpleText'])
+                or playlist_id)
        title += format_field(selected_tab, 'title', ' - %s')

        metadata = {
@@ -2957,7 +3225,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            self._entries(
                selected_tab, playlist_id,
                self._extract_identity_token(webpage, item_id),
-                self._extract_account_syncid(data)),
+                self._extract_account_syncid(data),
+                self._extract_ytcfg(item_id, webpage)),
            **metadata)

    def _extract_mix_playlist(self, playlist, playlist_id):
@@ -3011,46 +3280,26 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                    alert_type = alert.get('type')
                    if not alert_type:
                        continue
-                    message = try_get(alert, lambda x: x['text']['simpleText'], compat_str)
+                    message = try_get(alert, lambda x: x['text']['simpleText'], compat_str) or ''
                    if message:
                        yield alert_type, message
                    for run in try_get(alert, lambda x: x['text']['runs'], list) or []:
-                        message = try_get(run, lambda x: x['text'], compat_str)
-                        if message:
-                            yield alert_type, message
+                        message += try_get(run, lambda x: x['text'], compat_str)
+                    if message:
+                        yield alert_type, message

-        err_msg = None
+        errors = []
+        warnings = []
        for alert_type, alert_message in _real_extract_alerts():
            if alert_type.lower() == 'error':
-                if err_msg:
-                    self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg))
-                err_msg = alert_message
+                errors.append([alert_type, alert_message])
            else:
-                self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+                warnings.append([alert_type, alert_message])

-        if err_msg:
-            raise ExtractorError('YouTube said: %s' % err_msg, expected=expected)
-
-    def _extract_identity_token(self, webpage, item_id):
-        ytcfg = self._extract_ytcfg(item_id, webpage)
-        if ytcfg:
-            token = try_get(ytcfg, lambda x: x['ID_TOKEN'], compat_str)
-            if token:
-                return token
-        return self._search_regex(
-            r'\bID_TOKEN["\']\s*:\s*["\'](.+?)["\']', webpage,
-            'identity token', default=None)
-
-    @staticmethod
-    def _extract_account_syncid(data):
-        """Extract syncId required to download private playlists of secondary channels"""
-        sync_ids = (
-            try_get(data, lambda x: x['responseContext']['mainAppWebResponseContext']['datasyncId'], compat_str)
-            or '').split("||")
-        if len(sync_ids) >= 2 and sync_ids[1]:
-            # datasyncid is of the form "channel_syncid||user_syncid" for secondary channel
-            # and just "user_syncid||" for primary channel. We only want the channel_syncid
-            return sync_ids[0]
+        for alert_type, alert_message in (warnings + errors[:-1]):
+            self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+        if errors:
+            raise ExtractorError('YouTube said: %s' % errors[-1][1], expected=expected)

    def _extract_webpage(self, url, item_id):
        retries = self._downloader.params.get('extractor_retries', 3)
@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import re

 from .common import InfoExtractor
+from ..compat import compat_str
 from ..utils import (
    int_or_none,
    parse_age_limit,
@@ -16,24 +17,34 @@ from ..utils import (


 class Zee5IE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?zee5\.com/[^#?]*/(?P<display_id>[-\w]+)/(?P<id>[-\d]+)'
+    _VALID_URL = r'''(?x)
+                     (?:
+                        zee5:|
+                        (?:https?://)(?:www\.)?zee5\.com/(?:[^#?]+/)?
+                        (?:
+                            (?:tvshows|kids|zee5originals)(?:/[^#/?]+){3}
+                            |movies/[^#/?]+
+                        )/(?P<display_id>[^#/?]+)/
+                     )
+                     (?P<id>[^#/?]+)/?(?:$|[?#])
+                     '''
    _TESTS = [{
        'url': 'https://www.zee5.com/movies/details/krishna-the-birth/0-0-63098',
        'info_dict': {
-            "id": "0-0-63098",
-            "ext": "m3u8",
-            "display_id": "krishna-the-birth",
-            "title": "Krishna - The Birth",
-            "duration": 4368,
-            "average_rating": 4,
-            "description": str,
-            "alt_title": "Krishna - The Birth",
-            "uploader": "Zee Entertainment Enterprises Ltd",
-            "release_date": "20060101",
-            "upload_date": "20060101",
-            "timestamp": 1136073600,
-            "thumbnail": "https://akamaividz.zee5.com/resources/0-0-63098/list/270x152/0063098_list_80888170.jpg",
-            "tags": list
+            'id': '0-0-63098',
+            'ext': 'mp4',
+            'display_id': 'krishna-the-birth',
+            'title': 'Krishna - The Birth',
+            'duration': 4368,
+            'average_rating': 4,
+            'description': str,
+            'alt_title': 'Krishna - The Birth',
+            'uploader': 'Zee Entertainment Enterprises Ltd',
+            'release_date': '20060101',
+            'upload_date': '20060101',
+            'timestamp': 1136073600,
+            'thumbnail': 'https://akamaividz.zee5.com/resources/0-0-63098/list/270x152/0063098_list_80888170.jpg',
+            'tags': list
        },
        'params': {
            'format': 'bv',
@@ -41,37 +52,43 @@ class Zee5IE(InfoExtractor):
    }, {
        'url': 'https://zee5.com/tvshows/details/krishna-balram/0-6-1871/episode-1-the-test-of-bramha/0-1-233402',
        'info_dict': {
-            "id": "0-1-233402",
-            'ext': 'm3u8',
-            "display_id": "episode-1-the-test-of-bramha",
-            "title": "Episode 1 - The Test Of Bramha",
-            "duration": 1336,
-            "average_rating": 4,
-            "description": str,
-            "alt_title": "Episode 1 - The Test Of Bramha",
-            "uploader": "Green Gold",
-            "release_date": "20090101",
-            "upload_date": "20090101",
-            "timestamp": 1230768000,
-            "thumbnail": "https://akamaividz.zee5.com/resources/0-1-233402/list/270x152/01233402_list.jpg",
-            "series": "Krishna Balram",
-            "season_number": 1,
-            "episode_number": 1,
-            "tags": list,
+            'id': '0-1-233402',
+            'ext': 'mp4',
+            'display_id': 'episode-1-the-test-of-bramha',
+            'title': 'Episode 1 - The Test Of Bramha',
+            'duration': 1336,
+            'average_rating': 4,
+            'description': str,
+            'alt_title': 'Episode 1 - The Test Of Bramha',
+            'uploader': 'Green Gold',
+            'release_date': '20090101',
+            'upload_date': '20090101',
+            'timestamp': 1230768000,
+            'thumbnail': 'https://akamaividz.zee5.com/resources/0-1-233402/list/270x152/01233402_list.jpg',
+            'series': 'Krishna Balram',
+            'season_number': 1,
+            'episode_number': 1,
+            'tags': list,
        },
        'params': {
            'format': 'bv',
        },
+    }, {
+        'url': 'https://www.zee5.com/hi/tvshows/details/kundali-bhagya/0-6-366/kundali-bhagya-march-08-2021/0-1-manual_7g9jv1os7730?country=IN',
+        'only_matching': True
+    }, {
+        'url': 'https://www.zee5.com/global/hi/tvshows/details/kundali-bhagya/0-6-366/kundali-bhagya-march-08-2021/0-1-manual_7g9jv1os7730',
+        'only_matching': True
    }]

    def _real_extract(self, url):
        video_id, display_id = re.match(self._VALID_URL, url).group('id', 'display_id')
        access_token_request = self._download_json(
            'https://useraction.zee5.com/token/platform_tokens.php?platform_name=web_app',
-            video_id, note="Downloading access token")
+            video_id, note='Downloading access token')
        token_request = self._download_json(
            'https://useraction.zee5.com/tokennd',
-            video_id, note="Downloading video token")
+            video_id, note='Downloading video token')
        json_data = self._download_json(
            'https://gwapi.zee5.com/content/details/{}?translation=en&country=IN'.format(video_id),
            video_id, headers={'X-Access-Token': access_token_request['token']})
@@ -111,3 +128,78 @@ class Zee5IE(InfoExtractor):
            'episode_number': int_or_none(try_get(json_data, lambda x: x['index'])),
            'tags': try_get(json_data, lambda x: x['tags'], list)
        }
+
+
+class Zee5SeriesIE(InfoExtractor):
+    IE_NAME = 'zee5:series'
+    _VALID_URL = r'''(?x)
+                     (?:
+                        zee5:series:|
+                        (?:https?://)(?:www\.)?zee5\.com/(?:[^#?]+/)?
+                        (?:tvshows|kids|zee5originals)(?:/[^#/?]+){2}/
+                     )
+                     (?P<id>[^#/?]+)/?(?:$|[?#])
+                     '''
+    _TESTS = [{
+        'url': 'https://www.zee5.com/kids/kids-shows/krishna-balram/0-6-1871',
+        'playlist_mincount': 43,
+        'info_dict': {
+            'id': '0-6-1871',
+        },
+    }, {
+        'url': 'https://www.zee5.com/tvshows/details/bhabi-ji-ghar-par-hai/0-6-199',
+        'playlist_mincount': 1500,
+        'info_dict': {
+            'id': '0-6-199',
+        },
+    }, {
+        'url': 'https://www.zee5.com/tvshows/details/agent-raghav-crime-branch/0-6-965',
+        'playlist_mincount': 25,
+        'info_dict': {
+            'id': '0-6-965',
+        },
+    }, {
+        'url': 'https://www.zee5.com/ta/tvshows/details/nagabhairavi/0-6-3201',
+        'playlist_mincount': 3,
+        'info_dict': {
+            'id': '0-6-3201',
+        },
+    }, {
+        'url': 'https://www.zee5.com/global/hi/tvshows/details/khwaabon-ki-zamin-par/0-6-270',
+        'playlist_mincount': 150,
+        'info_dict': {
+            'id': '0-6-270',
+        },
+    }
+    ]
+
+    def _entries(self, show_id):
+        access_token_request = self._download_json(
+            'https://useraction.zee5.com/token/platform_tokens.php?platform_name=web_app',
+            show_id, note='Downloading access token')
+        headers = {
+            'X-Access-Token': access_token_request['token'],
+            'Referer': 'https://www.zee5.com/',
+        }
+        show_url = 'https://gwapi.zee5.com/content/tvshow/{}?translation=en&country=IN'.format(show_id)
+
+        page_num = 0
+        show_json = self._download_json(show_url, video_id=show_id, headers=headers)
+        for season in show_json.get('seasons') or []:
+            season_id = try_get(season, lambda x: x['id'], compat_str)
+            next_url = 'https://gwapi.zee5.com/content/tvshow/?season_id={}&type=episode&translation=en&country=IN&on_air=false&asset_subtype=tvshow&page=1&limit=100'.format(season_id)
+            while next_url:
+                page_num += 1
+                episodes_json = self._download_json(
+                    next_url, video_id=show_id, headers=headers,
+                    note='Downloading JSON metadata page %d' % page_num)
+                for episode in try_get(episodes_json, lambda x: x['episode'], list) or []:
+                    video_id = episode.get('id')
+                    yield self.url_result(
+                        'zee5:%s' % video_id,
+                        ie=Zee5IE.ie_key(), video_id=video_id)
+                next_url = url_or_none(episodes_json.get('next_episode_api'))
+
+    def _real_extract(self, url):
+        show_id = self._match_id(url)
+        return self.playlist_result(self._entries(show_id), playlist_id=show_id)
@@ -1,93 +1,94 @@
 # coding: utf-8
 from __future__ import unicode_literals

-import re
-
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
-    update_url_query,
 )


-class ZingMp3BaseInfoExtractor(InfoExtractor):
+class ZingMp3BaseIE(InfoExtractor):
+    _VALID_URL_TMPL = r'https?://(?:mp3\.zing|zingmp3)\.vn/(?:%s)/[^/]+/(?P<id>\w+)\.html'
+    _GEO_COUNTRIES = ['VN']

-    def _extract_item(self, item, page_type, fatal=True):
-        error_message = item.get('msg')
-        if error_message:
-            if not fatal:
-                return
-            raise ExtractorError(
-                '%s returned error: %s' % (self.IE_NAME, error_message),
-                expected=True)
+    def _extract_item(self, item, fatal):
+        item_id = item['id']
+        title = item.get('name') or item['title']

        formats = []
-        for quality, source_url in zip(item.get('qualities') or item.get('quality', []), item.get('source_list') or item.get('source', [])):
-            if not source_url or source_url == 'require vip':
+        for k, v in (item.get('source') or {}).items():
+            if not v:
                continue
-            if not re.match(r'https?://', source_url):
-                source_url = '//' + source_url
-            source_url = self._proto_relative_url(source_url, 'http:')
-            quality_num = int_or_none(quality)
-            f = {
-                'format_id': quality,
-                'url': source_url,
-            }
-            if page_type == 'video':
-                f.update({
-                    'height': quality_num,
-                    'ext': 'mp4',
-                })
+            if k in ('mp4', 'hls'):
+                for res, video_url in v.items():
+                    if not video_url:
+                        continue
+                    if k == 'hls':
+                        formats.extend(self._extract_m3u8_formats(
+                            video_url, item_id, 'mp4',
+                            'm3u8_native', m3u8_id=k, fatal=False))
+                    elif k == 'mp4':
+                        formats.append({
+                            'format_id': 'mp4-' + res,
+                            'url': video_url,
+                            'height': int_or_none(self._search_regex(
+                                r'^(\d+)p', res, 'resolution', default=None)),
+                        })
            else:
-                f.update({
-                    'abr': quality_num,
+                formats.append({
                    'ext': 'mp3',
+                    'format_id': k,
+                    'tbr': int_or_none(k),
+                    'url': self._proto_relative_url(v),
+                    'vcodec': 'none',
                })
-            formats.append(f)
+        if not formats:
+            if not fatal:
+                return
+            msg = item['msg']
+            if msg == 'Sorry, this content is not available in your country.':
+                self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
+            raise ExtractorError(msg, expected=True)
+        self._sort_formats(formats)

-        cover = item.get('cover')
+        subtitles = None
+        lyric = item.get('lyric')
+        if lyric:
+            subtitles = {
+                'origin': [{
+                    'url': lyric,
+                }],
+            }
+
+        album = item.get('album') or {}

        return {
-            'title': (item.get('name') or item.get('title')).strip(),
+            'id': item_id,
+            'title': title,
            'formats': formats,
-            'thumbnail': 'http:/' + cover if cover else None,
-            'artist': item.get('artist'),
+            'thumbnail': item.get('thumbnail'),
+            'subtitles': subtitles,
+            'duration': int_or_none(item.get('duration')),
+            'track': title,
+            'artist': item.get('artists_names'),
+            'album': album.get('name') or album.get('title'),
+            'album_artist': album.get('artists_names'),
        }

-    def _extract_player_json(self, player_json_url, id, page_type, playlist_title=None):
-        player_json = self._download_json(player_json_url, id, 'Downloading Player JSON')
-        items = player_json['data']
-        if 'item' in items:
-            items = items['item']
-
-        if len(items) == 1:
-            # one single song
-            data = self._extract_item(items[0], page_type)
-            data['id'] = id
-
-            return data
-        else:
-            # playlist of songs
-            entries = []
-
-            for i, item in enumerate(items, 1):
-                entry = self._extract_item(item, page_type, fatal=False)
-                if not entry:
-                    continue
-                entry['id'] = '%s-%d' % (id, i)
-                entries.append(entry)
-
-            return {
-                '_type': 'playlist',
-                'id': id,
-                'title': playlist_title,
-                'entries': entries,
-            }
+    def _real_extract(self, url):
+        page_id = self._match_id(url)
+        webpage = self._download_webpage(
+            url.replace('://zingmp3.vn/', '://mp3.zing.vn/'),
+            page_id, query={'play_song': 1})
+        data_path = self._search_regex(
+            r'data-xml="([^"]+)', webpage, 'data path')
+        return self._process_data(self._download_json(
+            'https://mp3.zing.vn/xhr' + data_path, page_id)['data'])


-class ZingMp3IE(ZingMp3BaseInfoExtractor):
-    _VALID_URL = r'https?://mp3\.zing\.vn/(?:bai-hat|album|playlist|video-clip)/[^/]+/(?P<id>\w+)\.html'
+class ZingMp3IE(ZingMp3BaseIE):
+    _VALID_URL = ZingMp3BaseIE._VALID_URL_TMPL % 'bai-hat|video-clip'
    _TESTS = [{
        'url': 'http://mp3.zing.vn/bai-hat/Xa-Mai-Xa-Bao-Thy/ZWZB9WAB.html',
        'md5': 'ead7ae13693b3205cbc89536a077daed',
@@ -95,49 +96,66 @@ class ZingMp3IE(ZingMp3BaseInfoExtractor):
            'id': 'ZWZB9WAB',
            'title': 'Xa Mãi Xa',
            'ext': 'mp3',
-            'thumbnail': r're:^https?://.*\.jpg$',
+            'thumbnail': r're:^https?://.+\.jpg',
+            'subtitles': {
+                'origin': [{
+                    'ext': 'lrc',
+                }]
+            },
+            'duration': 255,
+            'track': 'Xa Mãi Xa',
+            'artist': 'Bảo Thy',
+            'album': 'Special Album',
+            'album_artist': 'Bảo Thy',
        },
    }, {
-        'url': 'http://mp3.zing.vn/video-clip/Let-It-Go-Frozen-OST-Sungha-Jung/ZW6BAEA0.html',
-        'md5': '870295a9cd8045c0e15663565902618d',
+        'url': 'https://mp3.zing.vn/video-clip/Suong-Hoa-Dua-Loi-K-ICM-RYO/ZO8ZF7C7.html',
+        'md5': 'e9c972b693aa88301ef981c8151c4343',
        'info_dict': {
-            'id': 'ZW6BAEA0',
-            'title': 'Let It Go (Frozen OST)',
+            'id': 'ZO8ZF7C7',
+            'title': 'Sương Hoa Đưa Lối',
            'ext': 'mp4',
+            'thumbnail': r're:^https?://.+\.jpg',
+            'duration': 207,
+            'track': 'Sương Hoa Đưa Lối',
+            'artist': 'K-ICM, RYO',
        },
    }, {
-        'url': 'http://mp3.zing.vn/album/Lau-Dai-Tinh-Ai-Bang-Kieu-Minh-Tuyet/ZWZBWDAF.html',
-        'info_dict': {
-            '_type': 'playlist',
-            'id': 'ZWZBWDAF',
-            'title': 'Lâu Đài Tình Ái - Bằng Kiều,Minh Tuyết | Album 320 lossless',
-        },
-        'playlist_count': 10,
-        'skip': 'removed at the request of the owner',
-    }, {
-        'url': 'http://mp3.zing.vn/playlist/Duong-Hong-Loan-apollobee/IWCAACCB.html',
+        'url': 'https://zingmp3.vn/bai-hat/Xa-Mai-Xa-Bao-Thy/ZWZB9WAB.html',
        'only_matching': True,
    }]
    IE_NAME = 'zingmp3'
    IE_DESC = 'mp3.zing.vn'

-    def _real_extract(self, url):
-        page_id = self._match_id(url)
+    def _process_data(self, data):
+        return self._extract_item(data, True)

-        webpage = self._download_webpage(url, page_id)

-        player_json_url = self._search_regex([
-            r'data-xml="([^"]+)',
-            r'&amp;xmlURL=([^&]+)&'
-        ], webpage, 'player xml url')
+class ZingMp3AlbumIE(ZingMp3BaseIE):
+    _VALID_URL = ZingMp3BaseIE._VALID_URL_TMPL % 'album|playlist'
+    _TESTS = [{
+        'url': 'http://mp3.zing.vn/album/Lau-Dai-Tinh-Ai-Bang-Kieu-Minh-Tuyet/ZWZBWDAF.html',
+        'info_dict': {
+            '_type': 'playlist',
+            'id': 'ZWZBWDAF',
+            'title': 'Lâu Đài Tình Ái',
+        },
+        'playlist_count': 10,
+    }, {
+        'url': 'http://mp3.zing.vn/playlist/Duong-Hong-Loan-apollobee/IWCAACCB.html',
+        'only_matching': True,
+    }, {
+        'url': 'https://zingmp3.vn/album/Lau-Dai-Tinh-Ai-Bang-Kieu-Minh-Tuyet/ZWZBWDAF.html',
+        'only_matching': True,
+    }]
+    IE_NAME = 'zingmp3:album'

-        playlist_title = None
-        page_type = self._search_regex(r'/(?:html5)?xml/([^/-]+)', player_json_url, 'page type')
-        if page_type == 'video':
-            player_json_url = update_url_query(player_json_url, {'format': 'json'})
-        else:
-            player_json_url = player_json_url.replace('/xml/', '/html5xml/')
-            if page_type == 'album':
-                playlist_title = self._og_search_title(webpage)
-
-        return self._extract_player_json(player_json_url, page_id, page_type, playlist_title)
+    def _process_data(self, data):
+        def entries():
+            for item in (data.get('items') or []):
+                entry = self._extract_item(item, False)
+                if entry:
+                    yield entry
+        info = data.get('info') or {}
+        return self.playlist_result(
+            entries(), info.get('id'), info.get('name') or info.get('title'))
@@ -1,82 +1,68 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import re
+
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
-    url_or_none,
+    js_to_json,
    parse_filesize,
-    urlencode_postdata
+    urlencode_postdata,
 )


 class ZoomIE(InfoExtractor):
    IE_NAME = 'zoom'
-    _VALID_URL = r'https://(?:.*).?zoom.us/rec(?:ording)?/(play|share)/(?P<id>[A-Za-z0-9\-_.]+)'
-
+    _VALID_URL = r'(?P<base_url>https?://(?:[^.]+\.)?zoom.us/)rec(?:ording)?/(?:play|share)/(?P<id>[A-Za-z0-9_.-]+)'
    _TEST = {
-        'url': 'https://zoom.us/recording/play/SILVuCL4bFtRwWTtOCFQQxAsBQsJljFtm9e4Z_bvo-A8B-nzUSYZRNuPl3qW5IGK',
+        'url': 'https://economist.zoom.us/rec/play/dUk_CNBETmZ5VA2BwEl-jjakPpJ3M1pcfVYAPRsoIbEByGsLjUZtaa4yCATQuOL3der8BlTwxQePl_j0.EImBkXzTIaPvdZO5',
+        'md5': 'ab445e8c911fddc4f9adc842c2c5d434',
        'info_dict': {
-            'md5': '031a5b379f1547a8b29c5c4c837dccf2',
-            'title': "GAZ Transformational Tuesdays W/ Landon & Stapes",
-            'id': "SILVuCL4bFtRwWTtOCFQQxAsBQsJljFtm9e4Z_bvo-A8B-nzUSYZRNuPl3qW5IGK",
-            'ext': "mp4"
+            'id': 'dUk_CNBETmZ5VA2BwEl-jjakPpJ3M1pcfVYAPRsoIbEByGsLjUZtaa4yCATQuOL3der8BlTwxQePl_j0.EImBkXzTIaPvdZO5',
+            'ext': 'mp4',
+            'title': 'China\'s "two sessions" and the new five-year plan',
        }
    }

    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
+        base_url, play_id = re.match(self._VALID_URL, url).groups()
+        webpage = self._download_webpage(url, play_id)

-        password_protected = self._search_regex(r'<form[^>]+?id="(password_form)"', webpage, 'password field', fatal=False, default=None)
-        if password_protected is not None:
-            self._verify_video_password(url, display_id, webpage)
-            webpage = self._download_webpage(url, display_id)
+        try:
+            form = self._form_hidden_inputs('password_form', webpage)
+        except ExtractorError:
+            form = None
+        if form:
+            password = self._downloader.params.get('videopassword')
+            if not password:
+                raise ExtractorError(
+                    'This video is protected by a passcode, use the --video-password option', expected=True)
+            is_meeting = form.get('useWhichPasswd') == 'meeting'
+            validation = self._download_json(
+                base_url + 'rec/validate%s_passwd' % ('_meet' if is_meeting else ''),
+                play_id, 'Validating passcode', 'Wrong passcode', data=urlencode_postdata({
+                    'id': form[('meet' if is_meeting else 'file') + 'Id'],
+                    'passwd': password,
+                    'action': form.get('action'),
+                }))
+            if not validation.get('status'):
+                raise ExtractorError(validation['errorMessage'], expected=True)
+            webpage = self._download_webpage(url, play_id)

-        video_url = self._search_regex(r"viewMp4Url: \'(.*)\'", webpage, 'video url')
-        title = self._html_search_regex([r"topic: \"(.*)\",", r"<title>(.*) - Zoom</title>"], webpage, 'title')
-        viewResolvtionsWidth = self._search_regex(r"viewResolvtionsWidth: (\d*)", webpage, 'res width', fatal=False)
-        viewResolvtionsHeight = self._search_regex(r"viewResolvtionsHeight: (\d*)", webpage, 'res height', fatal=False)
-        fileSize = parse_filesize(self._search_regex(r"fileSize: \'(.+)\'", webpage, 'fileSize', fatal=False))
-
-        urlprefix = url.split("zoom.us")[0] + "zoom.us/"
-
-        formats = []
-        formats.append({
-            'url': url_or_none(video_url),
-            'width': int_or_none(viewResolvtionsWidth),
-            'height': int_or_none(viewResolvtionsHeight),
-            'http_headers': {'Accept': 'video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5',
-                             'Referer': urlprefix},
-            'ext': "mp4",
-            'filesize_approx': int_or_none(fileSize)
-        })
-        self._sort_formats(formats)
+        data = self._parse_json(self._search_regex(
+            r'(?s)window\.__data__\s*=\s*({.+?});',
+            webpage, 'data'), play_id, js_to_json)

        return {
-            'id': display_id,
-            'title': title,
-            'formats': formats
+            'id': play_id,
+            'title': data['topic'],
+            'url': data['viewMp4Url'],
+            'width': int_or_none(data.get('viewResolvtionsWidth')),
+            'height': int_or_none(data.get('viewResolvtionsHeight')),
+            'http_headers': {
+                'Referer': base_url,
+            },
+            'filesize_approx': parse_filesize(data.get('fileSize')),
        }
-
-    def _verify_video_password(self, url, video_id, webpage):
-        password = self._downloader.params.get('videopassword')
-        if password is None:
-            raise ExtractorError('This video is protected by a password, use the --video-password option', expected=True)
-        meetId = self._search_regex(r'<input[^>]+?id="meetId" value="([^\"]+)"', webpage, 'meetId')
-        data = urlencode_postdata({
-            'id': meetId,
-            'passwd': password,
-            'action': "viewdetailedpage",
-            'recaptcha': ""
-        })
-        validation_url = url.split("zoom.us")[0] + "zoom.us/rec/validate_meet_passwd"
-        validation_response = self._download_json(
-            validation_url, video_id,
-            note='Validating Password...',
-            errnote='Wrong password?',
-            data=data)
-
-        if validation_response['errorCode'] != 0:
-            raise ExtractorError('Login failed, %s said: %r' % (self.IE_NAME, validation_response['errorMessage']))
@@ -107,22 +107,31 @@ def parseOpts(overrideArguments=None):

        return ''.join(opts)

-    def _comma_separated_values_options_callback(option, opt_str, value, parser):
-        setattr(parser.values, option.dest, value.split(','))
+    def _comma_separated_values_options_callback(option, opt_str, value, parser, prepend=True):
+        setattr(
+            parser.values, option.dest,
+            value.split(',') if not prepend
+            else value.split(',') + getattr(parser.values, option.dest))

    def _dict_from_multiple_values_options_callback(
-            option, opt_str, value, parser, allowed_keys=r'[\w-]+', delimiter=':', default_key=None, process=None):
+            option, opt_str, value, parser,
+            allowed_keys=r'[\w-]+', delimiter=':', default_key=None, process=None, multiple_keys=True):

        out_dict = getattr(parser.values, option.dest)
-        mobj = re.match(r'(?i)(?P<key>%s)%s(?P<val>.*)$' % (allowed_keys, delimiter), value)
+        if multiple_keys:
+            allowed_keys = r'(%s)(,(%s))*' % (allowed_keys, allowed_keys)
+        mobj = re.match(r'(?i)(?P<keys>%s)%s(?P<val>.*)$' % (allowed_keys, delimiter), value)
        if mobj is not None:
-            key, val = mobj.group('key').lower(), mobj.group('val')
+            keys = [k.strip() for k in mobj.group('keys').lower().split(',')]
+            val = mobj.group('val')
        elif default_key is not None:
-            key, val = default_key, value
+            keys, val = [default_key], value
        else:
            raise optparse.OptionValueError(
                'wrong %s formatting; it should be %s, not "%s"' % (opt_str, option.metavar, value))
-        out_dict[key] = process(val) if callable(process) else val
+        val = process(val) if callable(process) else val
+        for key in keys:
+            out_dict[key] = val

    # No need to wrap help messages if we're on a wide console
    columns = compat_get_terminal_size().columns
@@ -214,12 +223,11 @@ def parseOpts(overrideArguments=None):
        help='Mark videos watched (YouTube only)')
    general.add_option(
        '--no-mark-watched',
-        action='store_false', dest='mark_watched', default=False,
-        help='Do not mark videos watched')
+        action='store_false', dest='mark_watched',
+        help='Do not mark videos watched (default)')
    general.add_option(
        '--no-colors',
-        action='store_true', dest='no_color',
-        default=False,
+        action='store_true', dest='no_color', default=False,
        help='Do not emit color codes in output')

    network = optparse.OptionGroup(parser, 'Network Options')
@@ -251,7 +259,7 @@ def parseOpts(overrideArguments=None):
        help='Make all connections via IPv6',
    )

-    geo = optparse.OptionGroup(parser, 'Geo Restriction')
+    geo = optparse.OptionGroup(parser, 'Geo-restriction')
    geo.add_option(
        '--geo-verification-proxy',
        dest='geo_verification_proxy', default=None, metavar='URL',
@@ -394,11 +402,11 @@ def parseOpts(overrideArguments=None):
    selection.add_option(
        '--include-ads',
        dest='include_ads', action='store_true',
-        help='Download advertisements as well (experimental)')
+        help=optparse.SUPPRESS_HELP)
    selection.add_option(
        '--no-include-ads',
        dest='include_ads', action='store_false',
-        help='Do not download advertisements (default)')
+        help=optparse.SUPPRESS_HELP)

    authentication = optparse.OptionGroup(parser, 'Authentication Options')
    authentication.add_option(
@@ -421,21 +429,19 @@ def parseOpts(overrideArguments=None):
        '--video-password',
        dest='videopassword', metavar='PASSWORD',
        help='Video password (vimeo, youku)')
-
-    adobe_pass = optparse.OptionGroup(parser, 'Adobe Pass Options')
-    adobe_pass.add_option(
+    authentication.add_option(
        '--ap-mso',
        dest='ap_mso', metavar='MSO',
        help='Adobe Pass multiple-system operator (TV provider) identifier, use --ap-list-mso for a list of available MSOs')
-    adobe_pass.add_option(
+    authentication.add_option(
        '--ap-username',
        dest='ap_username', metavar='USERNAME',
        help='Multiple-system operator account login')
-    adobe_pass.add_option(
+    authentication.add_option(
        '--ap-password',
        dest='ap_password', metavar='PASSWORD',
        help='Multiple-system operator account password. If this option is left out, yt-dlp will ask interactively')
-    adobe_pass.add_option(
+    authentication.add_option(
        '--ap-list-mso',
        action='store_true', dest='ap_list_mso', default=False,
        help='List all supported multiple-system operators')
@@ -481,7 +487,7 @@ def parseOpts(overrideArguments=None):
    video_format.add_option(
        '--all-formats',
        action='store_const', dest='format', const='all',
-        help='Download all available video formats')
+        help=optparse.SUPPRESS_HELP)
    video_format.add_option(
        '--prefer-free-formats',
        action='store_true', dest='prefer_free_formats', default=False,
@@ -534,11 +540,11 @@ def parseOpts(overrideArguments=None):
    subtitles.add_option(
        '--write-auto-subs', '--write-automatic-subs',
        action='store_true', dest='writeautomaticsub', default=False,
-        help='Write automatically generated subtitle file (YouTube only)')
+        help='Write automatically generated subtitle file (Alias: --write-automatic-subs)')
    subtitles.add_option(
        '--no-write-auto-subs', '--no-write-automatic-subs',
        action='store_false', dest='writeautomaticsub', default=False,
-        help='Do not write automatically generated subtitle file (default)')
+        help='Do not write auto-generated subtitles (default) (Alias: --no-write-automatic-subs)')
    subtitles.add_option(
        '--all-subs',
        action='store_true', dest='allsubtitles', default=False,
@@ -552,12 +558,16 @@ def parseOpts(overrideArguments=None):
        action='store', dest='subtitlesformat', metavar='FORMAT', default='best',
        help='Subtitle format, accepts formats preference, for example: "srt" or "ass/srt/best"')
    subtitles.add_option(
-        '--sub-lang', '--sub-langs', '--srt-lang',
+        '--sub-langs', '--srt-langs',
        action='callback', dest='subtitleslangs', metavar='LANGS', type='str',
        default=[], callback=_comma_separated_values_options_callback,
        help='Languages of the subtitles to download (optional) separated by commas, use --list-subs for available language tags')

    downloader = optparse.OptionGroup(parser, 'Download Options')
+    downloader.add_option(
+        '-N', '--concurrent-fragments',
+        dest='concurrent_fragment_downloads', metavar='N', default=1, type=int,
+        help='Number of fragments of a dash/hlsnative video that should be download concurrently (default is %default)')
    downloader.add_option(
        '-r', '--limit-rate', '--rate-limit',
        dest='ratelimit', metavar='RATE',
@@ -627,11 +637,11 @@ def parseOpts(overrideArguments=None):
    downloader.add_option(
        '--hls-prefer-native',
        dest='hls_prefer_native', action='store_true', default=None,
-        help='Use the native HLS downloader instead of ffmpeg')
+        help=optparse.SUPPRESS_HELP)
    downloader.add_option(
        '--hls-prefer-ffmpeg',
        dest='hls_prefer_native', action='store_false', default=None,
-        help='Use ffmpeg instead of the native HLS downloader')
+        help=optparse.SUPPRESS_HELP)
    downloader.add_option(
        '--hls-use-mpegts',
        dest='hls_use_mpegts', action='store_true', default=None,
@@ -647,11 +657,20 @@ def parseOpts(overrideArguments=None):
            'Do not use the mpegts container for HLS videos. '
            'This is default when not downloading live streams'))
    downloader.add_option(
-        '--external-downloader',
-        dest='external_downloader', metavar='NAME',
+        '--downloader', '--external-downloader',
+        dest='external_downloader', metavar='[PROTO:]NAME', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': 'http|ftp|m3u8|dash|rtsp|rtmp|mms',
+            'default_key': 'default', 'process': lambda x: x.strip()},
        help=(
-            'Name or path of the external downloader to use. '
-            'Currently supports %s (Recommended: aria2c)' % ', '.join(list_external_downloaders())))
+            'Name or path of the external downloader to use (optionally) prefixed by '
+            'the protocols (http, ftp, m3u8, dash, rstp, rtmp, mms) to use it for. '
+            'Currently supports native, %s (Recommended: aria2c). '
+            'You can use this option multiple times to set different downloaders for different protocols. '
+            'For example, --downloader aria2c --downloader "dash,m3u8:native" will use '
+            'aria2c for http/ftp downloads, and the native downloader for dash/m3u8 downloads '
+            '(Alias: --external-downloader)' % ', '.join(list_external_downloaders())))
    downloader.add_option(
        '--downloader-args', '--external-downloader-args',
        metavar='NAME:ARGS', dest='external_downloader_args', default={}, type='str',
@@ -676,7 +695,7 @@ def parseOpts(overrideArguments=None):
    workarounds.add_option(
        '--prefer-insecure', '--prefer-unsecure',
        action='store_true', dest='prefer_insecure',
-        help='Use an unencrypted connection to retrieve information about the video. (Currently supported only for YouTube)')
+        help='Use an unencrypted connection to retrieve information about the video (Currently supported only for YouTube)')
    workarounds.add_option(
        '--user-agent',
        metavar='UA', dest='user_agent',
@@ -690,6 +709,7 @@ def parseOpts(overrideArguments=None):
        '--add-header',
        metavar='FIELD:VALUE', dest='headers', default={}, type='str',
        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={'multiple_keys': False},
        help='Specify a custom HTTP header and its value, separated by a colon ":". You can use this option multiple times',
    )
    workarounds.add_option(
@@ -704,17 +724,13 @@ def parseOpts(overrideArguments=None):
        '--sleep-interval', '--min-sleep-interval', metavar='SECONDS',
        dest='sleep_interval', type=float,
        help=(
-            'Number of seconds to sleep before each download when used alone '
-            'or a lower bound of a range for randomized sleep before each download '
-            '(minimum possible number of seconds to sleep) when used along with '
-            '--max-sleep-interval'))
+            'Number of seconds to sleep before each download. '
+            'This is the minimum time to sleep when used along with --max-sleep-interval '
+            '(Alias: --min-sleep-interval)'))
    workarounds.add_option(
        '--max-sleep-interval', metavar='SECONDS',
        dest='max_sleep_interval', type=float,
-        help=(
-            'Upper bound of a range for randomized sleep before each download '
-            '(maximum possible number of seconds to sleep). Must only be used '
-            'along with --min-sleep-interval'))
+        help='Maximum number of seconds to sleep. Can only be used along with --min-sleep-interval')
    workarounds.add_option(
        '--sleep-subtitles', metavar='SECONDS',
        dest='sleep_interval_subtitles', default=0, type=int,
@@ -736,7 +752,7 @@ def parseOpts(overrideArguments=None):
    verbosity.add_option(
        '--skip-download', '--no-download',
        action='store_true', dest='skip_download', default=False,
-        help='Do not download the video')
+        help='Do not download the video but write all related files (Alias: --no-download)')
    verbosity.add_option(
        '-g', '--get-url',
        action='store_true', dest='geturl', default=False,
@@ -843,7 +859,7 @@ def parseOpts(overrideArguments=None):
        action='store_true', dest='useid', help=optparse.SUPPRESS_HELP)
    filesystem.add_option(
        '-P', '--paths',
-        metavar='TYPE:PATH', dest='paths', default={}, type='str',
+        metavar='TYPES:PATH', dest='paths', default={}, type='str',
        action='callback', callback=_dict_from_multiple_values_options_callback,
        callback_kwargs={
            'allowed_keys': 'home|temp|%s' % '|'.join(OUTTMPL_TYPES.keys()),
@@ -858,12 +874,12 @@ def parseOpts(overrideArguments=None):
            'This option is ignored if --output is an absolute path'))
    filesystem.add_option(
        '-o', '--output',
-        metavar='[TYPE:]TEMPLATE', dest='outtmpl', default={}, type='str',
+        metavar='[TYPES:]TEMPLATE', dest='outtmpl', default={}, type='str',
        action='callback', callback=_dict_from_multiple_values_options_callback,
        callback_kwargs={
            'allowed_keys': '|'.join(OUTTMPL_TYPES.keys()),
            'default_key': 'default', 'process': lambda x: x.strip()},
-        help='Output filename template, see "OUTPUT TEMPLATE" for details')
+        help='Output filename template; see "OUTPUT TEMPLATE" for details')
    filesystem.add_option(
        '--output-na-placeholder',
        dest='outtmpl_na_placeholder', metavar='TEXT', default='NA',
@@ -979,9 +995,17 @@ def parseOpts(overrideArguments=None):
    filesystem.add_option(
        '--no-write-playlist-metafiles',
        action='store_false', dest='allow_playlist_files',
+        help='Do not write playlist metadata when using --write-info-json, --write-description etc.')
+    filesystem.add_option(
+        '--clean-infojson',
+        action='store_true', dest='clean_infojson', default=True,
        help=(
-            'Do not write playlist metadata when using '
-            '--write-info-json, --write-description etc.'))
+            'Remove some private fields such as filenames from the infojson. '
+            'Note that it could still contain some personal information (default)'))
+    filesystem.add_option(
+        '--no-clean-infojson',
+        action='store_false', dest='clean_infojson',
+        help='Write all fields to the infojson')
    filesystem.add_option(
        '--get-comments',
        action='store_true', dest='getcomments', default=False,
@@ -1011,7 +1035,7 @@ def parseOpts(overrideArguments=None):
        action='store_true', dest='rm_cachedir',
        help='Delete all filesystem cache files')

-    thumbnail = optparse.OptionGroup(parser, 'Thumbnail Images')
+    thumbnail = optparse.OptionGroup(parser, 'Thumbnail Options')
    thumbnail.add_option(
        '--write-thumbnail',
        action='store_true', dest='writethumbnail', default=False,
@@ -1077,18 +1101,21 @@ def parseOpts(overrideArguments=None):
        '--postprocessor-args', '--ppa',
        metavar='NAME:ARGS', dest='postprocessor_args', default={}, type='str',
        action='callback', callback=_dict_from_multiple_values_options_callback,
-        callback_kwargs={'default_key': 'default-compat', 'allowed_keys': r'\w+(?:\+\w+)?', 'process': compat_shlex_split},
+        callback_kwargs={
+            'allowed_keys': r'\w+(?:\+\w+)?', 'default_key': 'default-compat',
+            'process': compat_shlex_split, 'multiple_keys': False},
        help=(
            'Give these arguments to the postprocessors. '
            'Specify the postprocessor/executable name and the arguments separated by a colon ":" '
-            'to give the argument to the specified postprocessor/executable. Supported postprocessors are: '
-            'SponSkrub, ExtractAudio, VideoRemuxer, VideoConvertor, EmbedSubtitle, Metadata, Merger, '
-            'FixupStretched, FixupM4a, FixupM3u8, SubtitlesConvertor and EmbedThumbnail. '
-            'The supported executables are: SponSkrub, FFmpeg, FFprobe, and AtomicParsley. '
+            'to give the argument to the specified postprocessor/executable. Supported PP are: '
+            'Merger, ExtractAudio, SplitChapters, Metadata, EmbedSubtitle, EmbedThumbnail, '
+            'SubtitlesConvertor, ThumbnailsConvertor, VideoRemuxer, VideoConvertor, '
+            'SponSkrub, FixupStretched, FixupM4a and FixupM3u8. '
+            'The supported executables are: AtomicParsley, FFmpeg, FFprobe, and SponSkrub. '
            'You can also specify "PP+EXE:ARGS" to give the arguments to the specified executable '
            'only when being used by the specified postprocessor. Additionally, for ffmpeg/ffprobe, '
-            'a number can be appended to the exe name seperated by "_i" to pass the argument '
-            'before the specified input file. Eg: --ppa "Merger+ffmpeg_i1:-v quiet". '
+            '"_i"/"_o" can be appended to the prefix optionally followed by a number to pass the argument '
+            'before the specified input/output file. Eg: --ppa "Merger+ffmpeg_i1:-v quiet". '
            'You can use this option multiple times to give different arguments to different '
            'postprocessors. (Alias: --ppa)'))
    postproc.add_option(
@@ -1137,17 +1164,10 @@ def parseOpts(overrideArguments=None):
        help=optparse.SUPPRESS_HELP)
    postproc.add_option(
        '--parse-metadata',
-        metavar='FIELD:FORMAT', dest='metafromfield', action='append',
+        metavar='FROM:TO', dest='metafromfield', action='append',
        help=(
-            'Parse additional metadata like title/artist from other fields. '
-            'Give field name to extract data from, and format of the field seperated by a ":". '
-            'Either regular expression with named capture groups or a '
-            'similar syntax to the output template can also be used. '
-            'The parsed parameters replace any existing values and can be use in output template. '
-            'This option can be used multiple times. '
-            'Example: --parse-metadata "title:%(artist)s - %(title)s" matches a title like '
-            '"Coldplay - Paradise". '
-            'Example (regex): --parse-metadata "description:Artist - (?P<artist>.+?)"'))
+            'Parse additional metadata like title/artist from other fields; '
+            'see "MODIFYING METADATA" for details'))
    postproc.add_option(
        '--xattrs',
        action='store_true', dest='xattrs', default=False,
@@ -1174,11 +1194,30 @@ def parseOpts(overrideArguments=None):
    postproc.add_option(
        '--exec',
        metavar='CMD', dest='exec_cmd',
-        help='Execute a command on the file after downloading and post-processing, similar to find\'s -exec syntax. Example: --exec \'adb push {} /sdcard/Music/ && rm {}\'')
+        help=(
+            'Execute a command on the file after downloading and post-processing. '
+            'Similar syntax to the output template can be used to pass any field as arguments to the command. '
+            'An additional field "filepath" that contains the final path of the downloaded file is also available. '
+            'If no fields are passed, "%(filepath)s" is appended to the end of the command'))
    postproc.add_option(
-        '--convert-subs', '--convert-subtitles',
+        '--convert-subs', '--convert-sub', '--convert-subtitles',
        metavar='FORMAT', dest='convertsubtitles', default=None,
-        help='Convert the subtitles to other format (currently supported: srt|ass|vtt|lrc)')
+        help='Convert the subtitles to another format (currently supported: srt|ass|vtt|lrc) (Alias: --convert-subtitles)')
+    postproc.add_option(
+        '--convert-thumbnails',
+        metavar='FORMAT', dest='convertthumbnails', default=None,
+        help='Convert the thumbnails to another format (currently supported: jpg)')
+    postproc.add_option(
+        '--split-chapters', '--split-tracks',
+        dest='split_chapters', action='store_true', default=False,
+        help=(
+            'Split video into multiple files based on internal chapters. '
+            'The "chapter:" prefix can be used with "--paths" and "--output" to '
+            'set the output filename for the split files. See "OUTPUT TEMPLATE" for details'))
+    postproc.add_option(
+        '--no-split-chapters', '--no-split-tracks',
+        dest='split_chapters', action='store_false',
+        help='Do not split video based on chapters (default)')

    sponskrub = optparse.OptionGroup(parser, 'SponSkrub (SponsorBlock) Options', description=(
        'SponSkrub (https://github.com/yt-dlp/SponSkrub) is a utility to mark/remove sponsor segments '
@@ -1269,7 +1308,6 @@ def parseOpts(overrideArguments=None):
    parser.add_option_group(video_format)
    parser.add_option_group(subtitles)
    parser.add_option_group(authentication)
-    parser.add_option_group(adobe_pass)
    parser.add_option_group(postproc)
    parser.add_option_group(sponskrub)
    parser.add_option_group(extractor)
@@ -10,9 +10,11 @@ from .ffmpeg import (
    FFmpegFixupM4aPP,
    FFmpegMergerPP,
    FFmpegMetadataPP,
+    FFmpegSubtitlesConvertorPP,
+    FFmpegThumbnailsConvertorPP,
+    FFmpegSplitChaptersPP,
    FFmpegVideoConvertorPP,
    FFmpegVideoRemuxerPP,
-    FFmpegSubtitlesConvertorPP,
 )
 from .xattrpp import XAttrMetadataPP
 from .execafterdownload import ExecAfterDownloadPP
@@ -27,17 +29,19 @@ def get_postprocessor(key):


 __all__ = [
+    'FFmpegPostProcessor',
    'EmbedThumbnailPP',
    'ExecAfterDownloadPP',
    'FFmpegEmbedSubtitlePP',
    'FFmpegExtractAudioPP',
+    'FFmpegSplitChaptersPP',
    'FFmpegFixupM3u8PP',
    'FFmpegFixupM4aPP',
    'FFmpegFixupStretchedPP',
    'FFmpegMergerPP',
    'FFmpegMetadataPP',
-    'FFmpegPostProcessor',
    'FFmpegSubtitlesConvertorPP',
+    'FFmpegThumbnailsConvertorPP',
    'FFmpegVideoConvertorPP',
    'FFmpegVideoRemuxerPP',
    'MetadataFromFieldPP',
@@ -91,10 +91,18 @@ class PostProcessor(object):
        except Exception:
            self.report_warning(errnote)

-    def _configuration_args(self, *args, **kwargs):
+    def _configuration_args(self, exe, keys=None, default=[], use_compat=True):
+        pp_key = self.pp_key().lower()
+        exe = exe.lower()
+        root_key = exe if pp_key == exe else '%s+%s' % (pp_key, exe)
+        keys = ['%s%s' % (root_key, k) for k in (keys or [''])]
+        if root_key in keys:
+            keys += [root_key] + ([] if pp_key == exe else [(self.pp_key(), exe)]) + ['default']
+        else:
+            use_compat = False
        return cli_configuration_args(
            self._downloader.params.get('postprocessor_args'),
-            self.pp_key().lower(), *args, **kwargs)
+            keys, default, use_compat)


 class AudioConversionError(PostProcessingError):
@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals

-
 import os
 import subprocess
 import struct
@@ -14,8 +13,10 @@ try:
 except ImportError:
    has_mutagen = False

-from .ffmpeg import FFmpegPostProcessor
-
+from .ffmpeg import (
+    FFmpegPostProcessor,
+    FFmpegThumbnailsConvertorPP,
+)
 from ..utils import (
    check_executable,
    encodeArgument,
@@ -24,7 +25,6 @@ from ..utils import (
    PostProcessingError,
    prepend_extension,
    process_communicate_or_kill,
-    replace_extension,
    shell_quote,
 )

@@ -36,7 +36,7 @@ class EmbedThumbnailPPError(PostProcessingError):
 class EmbedThumbnailPP(FFmpegPostProcessor):

    def __init__(self, downloader=None, already_have_thumbnail=False):
-        super(EmbedThumbnailPP, self).__init__(downloader)
+        FFmpegPostProcessor.__init__(self, downloader)
        self._already_have_thumbnail = already_have_thumbnail

    def run(self, info):
@@ -47,44 +47,25 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
            self.to_screen('There aren\'t any thumbnails to embed')
            return [], info

-        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filename']
-
+        thumbnail_filename = info['thumbnails'][-1]['filepath']
        if not os.path.exists(encodeFilename(thumbnail_filename)):
            self.report_warning('Skipping embedding the thumbnail because the file is missing.')
            return [], info

-        def is_webp(path):
-            with open(encodeFilename(path), 'rb') as f:
-                b = f.read(12)
-            return b[0:4] == b'RIFF' and b[8:] == b'WEBP'
-
        # Correct extension for WebP file with wrong extension (see #25687, #25717)
-        _, thumbnail_ext = os.path.splitext(thumbnail_filename)
-        if thumbnail_ext:
-            thumbnail_ext = thumbnail_ext[1:].lower()
-            if thumbnail_ext != 'webp' and is_webp(thumbnail_filename):
-                self.to_screen('Correcting extension to webp and escaping path for thumbnail "%s"' % thumbnail_filename)
-                thumbnail_webp_filename = replace_extension(thumbnail_filename, 'webp')
-                os.rename(encodeFilename(thumbnail_filename), encodeFilename(thumbnail_webp_filename))
-                original_thumbnail = thumbnail_filename = thumbnail_webp_filename
-                thumbnail_ext = 'webp'
+        convertor = FFmpegThumbnailsConvertorPP(self._downloader)
+        convertor.fixup_webp(info, -1)
+
+        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filepath']

        # Convert unsupported thumbnail formats to JPEG (see #25687, #25717)
-        if thumbnail_ext not in ['jpg', 'png']:
-            # NB: % is supposed to be escaped with %% but this does not work
-            # for input files so working around with standard substitution
-            escaped_thumbnail_filename = thumbnail_filename.replace('%', '#')
-            os.rename(encodeFilename(thumbnail_filename), encodeFilename(escaped_thumbnail_filename))
-            escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
-            self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
-            self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
-            thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
-            # Rename back to unescaped for further processing
-            os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
-            os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
-            thumbnail_filename = thumbnail_jpg_filename
+        _, thumbnail_ext = os.path.splitext(thumbnail_filename)
+        if thumbnail_ext not in ('jpg', 'png'):
+            thumbnail_filename = convertor.convert_thumbnail(thumbnail_filename, 'jpg')
            thumbnail_ext = 'jpg'

+        mtime = os.stat(encodeFilename(filename)).st_mtime
+
        success = True
        if info['ext'] == 'mp3':
            options = [
@@ -131,7 +112,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                    x for x in ['AtomicParsley', 'atomicparsley']
                    if check_executable(x, ['-v'])), None)
                if atomicparsley is None:
-                    raise EmbedThumbnailPPError('AtomicParsley was not found. Please install.')
+                    raise EmbedThumbnailPPError('AtomicParsley was not found. Please install')

                cmd = [encodeFilename(atomicparsley, True),
                       encodeFilename(filename, True),
@@ -139,7 +120,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                       encodeFilename(thumbnail_filename, True),
                       encodeArgument('-o'),
                       encodeFilename(temp_filename, True)]
-                cmd += [encodeArgument(o) for o in self._configuration_args(exe='AtomicParsley')]
+                cmd += [encodeArgument(o) for o in self._configuration_args('AtomicParsley')]

                self.to_screen('Adding thumbnail to "%s"' % filename)
                self.write_debug('AtomicParsley command line: %s' % shell_quote(cmd))
@@ -187,10 +168,10 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
            os.remove(encodeFilename(filename))
            os.rename(encodeFilename(temp_filename), encodeFilename(filename))

+        self.try_utime(filename, mtime, mtime)
+
        files_to_delete = [thumbnail_filename]
        if self._already_have_thumbnail:
-            info['__files_to_move'][original_thumbnail] = replace_extension(
-                info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:])
            if original_thumbnail == thumbnail_filename:
                files_to_delete = []
        elif original_thumbnail != thumbnail_filename:
@@ -20,12 +20,13 @@ class ExecAfterDownloadPP(PostProcessor):
    def pp_key(cls):
        return 'Exec'

-    def run(self, information):
-        cmd = self.exec_cmd
-        if '{}' not in cmd:
-            cmd += ' {}'
-
-        cmd = cmd.replace('{}', compat_shlex_quote(information['filepath']))
+    def run(self, info):
+        tmpl, info_copy = self._downloader.prepare_outtmpl(self.exec_cmd, info)
+        cmd = tmpl % info_copy
+        if cmd == self.exec_cmd:  # No replacements were made
+            if '{}' not in self.exec_cmd:
+                self.exec_cmd += ' {}'
+            cmd = self.exec_cmd.replace('{}', compat_shlex_quote(info['filepath']))

        self.to_screen('Executing command: %s' % cmd)
        retCode = subprocess.call(encodeArgument(cmd), shell=True)
@@ -33,4 +34,4 @@ class ExecAfterDownloadPP(PostProcessor):
            raise PostProcessingError(
                'Command returned error code %d' % retCode)

-        return [], information
+        return [], info
@@ -10,6 +10,7 @@ import json

 from .common import AudioConversionError, PostProcessor

+from ..compat import compat_str, compat_numeric_types
 from ..utils import (
    encodeArgument,
    encodeFilename,
@@ -18,7 +19,6 @@ from ..utils import (
    PostProcessingError,
    prepend_extension,
    shell_quote,
-    subtitles_filename,
    dfxp2srt,
    ISO639Utils,
    process_communicate_or_kill,
@@ -61,7 +61,7 @@ class FFmpegPostProcessor(PostProcessor):

    def check_version(self):
        if not self.available:
-            raise FFmpegPostProcessorError('ffmpeg not found. Please install')
+            raise FFmpegPostProcessorError('ffmpeg not found. Please install or provide the path using --ffmpeg-location')

        required_version = '10-0' if self.basename == 'avconv' else '1.0'
        if is_outdated_version(
@@ -165,7 +165,7 @@ class FFmpegPostProcessor(PostProcessor):

    def get_audio_codec(self, path):
        if not self.probe_available and not self.available:
-            raise PostProcessingError('ffprobe and ffmpeg not found. Please install')
+            raise PostProcessingError('ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location')
        try:
            if self.probe_available:
                cmd = [
@@ -207,7 +207,7 @@ class FFmpegPostProcessor(PostProcessor):
        if self.probe_basename != 'ffprobe':
            if self.probe_available:
                self.report_warning('Only ffprobe is supported for metadata extraction')
-            raise PostProcessingError('ffprobe not found. Please install.')
+            raise PostProcessingError('ffprobe not found. Please install or provide the path using --ffmpeg-location')
        self.check_version()

        cmd = [
@@ -234,25 +234,35 @@ class FFmpegPostProcessor(PostProcessor):
        return num, len(streams)

    def run_ffmpeg_multiple_files(self, input_paths, out_path, opts):
+        return self.real_run_ffmpeg(
+            [(path, []) for path in input_paths],
+            [(out_path, opts)])
+
+    def real_run_ffmpeg(self, input_path_opts, output_path_opts):
        self.check_version()

        oldest_mtime = min(
-            os.stat(encodeFilename(path)).st_mtime for path in input_paths)
+            os.stat(encodeFilename(path)).st_mtime for path, _ in input_path_opts)

        cmd = [encodeFilename(self.executable, True), encodeArgument('-y')]
        # avconv does not have repeat option
        if self.basename == 'ffmpeg':
            cmd += [encodeArgument('-loglevel'), encodeArgument('repeat+info')]

-        def make_args(file, pre=[], post=[], *args, **kwargs):
-            args = pre + self._configuration_args(*args, **kwargs) + post
+        def make_args(file, args, name, number):
+            keys = ['_%s%d' % (name, number), '_%s' % name]
+            if name == 'o' and number == 1:
+                keys.append('')
+            args += self._configuration_args(self.basename, keys)
+            if name == 'i':
+                args.append('-i')
            return (
-                [encodeArgument(o) for o in args]
+                [encodeArgument(arg) for arg in args]
                + [encodeFilename(self._ffmpeg_filename_argument(file), True)])

-        for i, path in enumerate(input_paths):
-            cmd += make_args(path, post=['-i'], exe='%s_i%d' % (self.basename, i + 1), use_default_arg=False)
-        cmd += make_args(out_path, pre=opts, exe=self.basename)
+        for arg_type, path_opts in (('i', input_path_opts), ('o', output_path_opts)):
+            cmd += [arg for i, o in enumerate(path_opts)
+                    for arg in make_args(o[0], o[1], arg_type, i + 1)]

        self.write_debug('ffmpeg command line: %s' % shell_quote(cmd))
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
@@ -262,7 +272,8 @@ class FFmpegPostProcessor(PostProcessor):
            if self.get_param('verbose', False):
                self.report_error(stderr)
            raise FFmpegPostProcessorError(stderr.split('\n')[-1])
-        self.try_utime(out_path, oldest_mtime, oldest_mtime)
+        for out_path, _ in output_path_opts:
+            self.try_utime(out_path, oldest_mtime, oldest_mtime)
        return stderr.decode('utf-8', 'replace')

    def run_ffmpeg(self, path, out_path, opts):
@@ -474,7 +485,7 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
                self.report_warning('JSON subtitles cannot be embedded')
            elif ext != 'webm' or ext == 'webm' and sub_ext == 'vtt':
                sub_langs.append(lang)
-                sub_filenames.append(subtitles_filename(filename, lang, sub_ext, ext))
+                sub_filenames.append(sub_info['filepath'])
            else:
                if not webm_vtt_warn and ext == 'webm' and sub_ext != 'vtt':
                    webm_vtt_warn = True
@@ -519,6 +530,8 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
        metadata = {}

        def add(meta_list, info_list=None):
+            if not meta_list:
+                return
            if not info_list:
                info_list = meta_list
            if not isinstance(meta_list, (list, tuple)):
@@ -526,7 +539,7 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
            if not isinstance(info_list, (list, tuple)):
                info_list = (info_list,)
            for info_f in info_list:
-                if info.get(info_f) is not None:
+                if isinstance(info.get(info_f), (compat_str, compat_numeric_types)):
                    for meta_f in meta_list:
                        metadata[meta_f] = info[info_f]
                    break
@@ -539,8 +552,8 @@ class FFmpegMetadataPP(FFmpegPostProcessor):

        add('title', ('track', 'title'))
        add('date', 'upload_date')
-        add(('description', 'comment'), 'description')
-        add('purl', 'webpage_url')
+        add(('description', 'synopsis'), 'description')
+        add(('purl', 'comment'), 'webpage_url')
        add('track', 'track_number')
        add('artist', ('artist', 'creator', 'uploader', 'uploader_id'))
        add('genre')
@@ -552,6 +565,10 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
        add('episode_id', ('episode', 'episode_id'))
        add('episode_sort', 'episode_number')

+        prefix = 'meta_'
+        for key in filter(lambda k: k.startswith(prefix), info.keys()):
+            add(key[len(prefix):], key)
+
        if not metadata:
            self.to_screen('There isn\'t any metadata to add')
            return [], info
@@ -566,7 +583,7 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
        else:
            options.extend(['-c', 'copy'])

-        for (name, value) in metadata.items():
+        for name, value in metadata.items():
            options.extend(['-metadata', '%s=%s' % (name, value)])

        chapters = info.get('chapters', [])
@@ -700,7 +717,6 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):

    def run(self, info):
        subs = info.get('requested_subtitles')
-        filename = info['filepath']
        new_ext = self.format
        new_format = new_ext
        if new_format == 'vtt':
@@ -720,9 +736,9 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
                    'You have requested to convert json subtitles into another format, '
                    'which is currently not possible')
                continue
-            old_file = subtitles_filename(filename, lang, ext, info.get('ext'))
+            old_file = sub['filepath']
            sub_filenames.append(old_file)
-            new_file = subtitles_filename(filename, lang, new_ext, info.get('ext'))
+            new_file = replace_extension(old_file, new_ext)

            if ext in ('dfxp', 'ttml', 'tt'):
                self.report_warning(
@@ -730,7 +746,7 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
                    'which results in style information loss')

                dfxp_file = old_file
-                srt_file = subtitles_filename(filename, lang, 'srt', info.get('ext'))
+                srt_file = replace_extension(old_file, 'srt')

                with open(dfxp_file, 'rb') as f:
                    srt_data = dfxp2srt(f.read())
@@ -741,7 +757,8 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):

                subs[lang] = {
                    'ext': 'srt',
-                    'data': srt_data
+                    'data': srt_data,
+                    'filepath': srt_file,
                }

                if new_ext == 'srt':
@@ -755,6 +772,117 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
                subs[lang] = {
                    'ext': new_ext,
                    'data': f.read(),
+                    'filepath': new_file,
                }

+            info['__files_to_move'][new_file] = replace_extension(
+                info['__files_to_move'][old_file], new_ext)
+
        return sub_filenames, info
+
+
+class FFmpegSplitChaptersPP(FFmpegPostProcessor):
+
+    def _prepare_filename(self, number, chapter, info):
+        info = info.copy()
+        info.update({
+            'section_number': number,
+            'section_title': chapter.get('title'),
+            'section_start': chapter.get('start_time'),
+            'section_end': chapter.get('end_time'),
+        })
+        return self._downloader.prepare_filename(info, 'chapter')
+
+    def _ffmpeg_args_for_chapter(self, number, chapter, info):
+        destination = self._prepare_filename(number, chapter, info)
+        if not self._downloader._ensure_dir_exists(encodeFilename(destination)):
+            return
+
+        chapter['filepath'] = destination
+        self.to_screen('Chapter %03d; Destination: %s' % (number, destination))
+        return (
+            destination,
+            ['-ss', compat_str(chapter['start_time']),
+             '-t', compat_str(chapter['end_time'] - chapter['start_time'])])
+
+    def run(self, info):
+        chapters = info.get('chapters') or []
+        if not chapters:
+            self.report_warning('Chapter information is unavailable')
+            return [], info
+
+        self.to_screen('Splitting video by chapters; %d chapters found' % len(chapters))
+        for idx, chapter in enumerate(chapters):
+            destination, opts = self._ffmpeg_args_for_chapter(idx + 1, chapter, info)
+            self.real_run_ffmpeg([(info['filepath'], opts)], [(destination, ['-c', 'copy'])])
+        return [], info
+
+
+class FFmpegThumbnailsConvertorPP(FFmpegPostProcessor):
+    def __init__(self, downloader=None, format=None):
+        super(FFmpegThumbnailsConvertorPP, self).__init__(downloader)
+        self.format = format
+
+    @staticmethod
+    def is_webp(path):
+        with open(encodeFilename(path), 'rb') as f:
+            b = f.read(12)
+        return b[0:4] == b'RIFF' and b[8:] == b'WEBP'
+
+    def fixup_webp(self, info, idx=-1):
+        thumbnail_filename = info['thumbnails'][idx]['filepath']
+        _, thumbnail_ext = os.path.splitext(thumbnail_filename)
+        if thumbnail_ext:
+            thumbnail_ext = thumbnail_ext[1:].lower()
+            if thumbnail_ext != 'webp' and self.is_webp(thumbnail_filename):
+                self.to_screen('Correcting thumbnail "%s" extension to webp' % thumbnail_filename)
+                webp_filename = replace_extension(thumbnail_filename, 'webp')
+                if os.path.exists(webp_filename):
+                    os.remove(webp_filename)
+                os.rename(encodeFilename(thumbnail_filename), encodeFilename(webp_filename))
+                info['thumbnails'][idx]['filepath'] = webp_filename
+                info['__files_to_move'][webp_filename] = replace_extension(
+                    info['__files_to_move'].pop(thumbnail_filename), 'webp')
+
+    def convert_thumbnail(self, thumbnail_filename, ext):
+        if ext != 'jpg':
+            raise FFmpegPostProcessorError('Only conversion to jpg is currently supported')
+        # NB: % is supposed to be escaped with %% but this does not work
+        # for input files so working around with standard substitution
+        escaped_thumbnail_filename = thumbnail_filename.replace('%', '#')
+        os.rename(encodeFilename(thumbnail_filename), encodeFilename(escaped_thumbnail_filename))
+        escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
+        self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
+        self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
+        thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
+        # Rename back to unescaped
+        os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
+        os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
+        return thumbnail_jpg_filename
+
+    def run(self, info):
+        if self.format != 'jpg':
+            raise FFmpegPostProcessorError('Only conversion to jpg is currently supported')
+        files_to_delete = []
+        has_thumbnail = False
+
+        for idx, thumbnail_dict in enumerate(info['thumbnails']):
+            if 'filepath' not in thumbnail_dict:
+                continue
+            has_thumbnail = True
+            self.fixup_webp(info, idx)
+            original_thumbnail = thumbnail_dict['filepath']
+            _, thumbnail_ext = os.path.splitext(original_thumbnail)
+            if thumbnail_ext:
+                thumbnail_ext = thumbnail_ext[1:].lower()
+            if thumbnail_ext == self.format:
+                self.to_screen('Thumbnail "%s" is already in the requested format' % original_thumbnail)
+                continue
+            thumbnail_dict['filepath'] = self.convert_thumbnail(original_thumbnail, self.format)
+            files_to_delete.append(original_thumbnail)
+            info['__files_to_move'][thumbnail_dict['filepath']] = replace_extension(
+                info['__files_to_move'][original_thumbnail], self.format)
+
+        if not has_thumbnail:
+            self.to_screen('There aren\'t any thumbnails to convert')
+        return files_to_delete, info
@@ -4,11 +4,10 @@ import re

 from .common import PostProcessor
 from ..compat import compat_str
-from ..utils import str_or_none


 class MetadataFromFieldPP(PostProcessor):
-    regex = r'(?P<field>\w+):(?P<format>.+)$'
+    regex = r'(?P<in>.+):(?P<out>.+)$'

    def __init__(self, downloader, formats):
        PostProcessor.__init__(self, downloader)
@@ -19,11 +18,20 @@ class MetadataFromFieldPP(PostProcessor):
            match = re.match(self.regex, f)
            assert match is not None
            self._data.append({
-                'field': match.group('field'),
-                'format': match.group('format'),
-                'regex': self.format_to_regex(match.group('format'))})
+                'in': match.group('in'),
+                'out': match.group('out'),
+                'tmpl': self.field_to_template(match.group('in')),
+                'regex': self.format_to_regex(match.group('out')),
+            })

-    def format_to_regex(self, fmt):
+    @staticmethod
+    def field_to_template(tmpl):
+        if re.match(r'\w+$', tmpl):
+            return '%%(%s)s' % tmpl
+        return tmpl
+
+    @staticmethod
+    def format_to_regex(fmt):
        r"""
        Converts a string like
           '%(title)s - %(artist)s'
@@ -37,7 +45,7 @@ class MetadataFromFieldPP(PostProcessor):
        # replace %(..)s with regex group and escape other string parts
        for match in re.finditer(r'%\((\w+)\)s', fmt):
            regex += re.escape(fmt[lastpos:match.start()])
-            regex += r'(?P<' + match.group(1) + r'>[^\r\n]+)'
+            regex += r'(?P<%s>.+)' % match.group(1)
            lastpos = match.end()
        if lastpos < len(fmt):
            regex += re.escape(fmt[lastpos:])
@@ -45,22 +53,16 @@ class MetadataFromFieldPP(PostProcessor):

    def run(self, info):
        for dictn in self._data:
-            field, regex = dictn['field'], dictn['regex']
-            if field not in info:
-                self.report_warning('Video doesnot have a %s' % field)
-                continue
-            data_to_parse = str_or_none(info[field])
-            if data_to_parse is None:
-                self.report_warning('Field %s cannot be parsed' % field)
-                continue
-            self.write_debug('Searching for r"%s" in %s' % (regex, field))
-            match = re.search(regex, data_to_parse)
+            tmpl, info_copy = self._downloader.prepare_outtmpl(dictn['tmpl'], info)
+            data_to_parse = tmpl % info_copy
+            self.write_debug('Searching for r"%s" in %s' % (dictn['regex'], tmpl))
+            match = re.search(dictn['regex'], data_to_parse)
            if match is None:
-                self.report_warning('Could not interpret video %s as "%s"' % (field, dictn['format']))
+                self.report_warning('Could not interpret video %s as "%s"' % (dictn['in'], dictn['out']))
                continue
            for attribute, value in match.groupdict().items():
                info[attribute] = value
-                self.to_screen('parsed %s from %s: %s' % (attribute, field, value if value is not None else 'NA'))
+                self.to_screen('parsed %s from "%s": %s' % (attribute, dictn['in'], value if value is not None else 'NA'))
        return [], info


@@ -13,9 +13,9 @@ from ..utils import (

 class MoveFilesAfterDownloadPP(PostProcessor):

-    def __init__(self, downloader, files_to_move):
+    def __init__(self, downloader=None, downloaded=True):
        PostProcessor.__init__(self, downloader)
-        self.files_to_move = files_to_move
+        self._downloaded = downloaded

    @classmethod
    def pp_key(cls):
@@ -25,11 +25,11 @@ class MoveFilesAfterDownloadPP(PostProcessor):
        dl_path, dl_name = os.path.split(encodeFilename(info['filepath']))
        finaldir = info.get('__finaldir', dl_path)
        finalpath = os.path.join(finaldir, dl_name)
-        self.files_to_move.update(info['__files_to_move'])
-        self.files_to_move[info['filepath']] = decodeFilename(finalpath)
+        if self._downloaded:
+            info['__files_to_move'][info['filepath']] = decodeFilename(finalpath)

        make_newfilename = lambda old: decodeFilename(os.path.join(finaldir, os.path.basename(encodeFilename(old))))
-        for oldfile, newfile in self.files_to_move.items():
+        for oldfile, newfile in info['__files_to_move'].items():
            if not newfile:
                newfile = make_newfilename(oldfile)
            if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
@@ -6,6 +6,7 @@ from .common import PostProcessor
 from ..compat import compat_shlex_split
 from ..utils import (
    check_executable,
+    cli_option,
    encodeArgument,
    encodeFilename,
    shell_quote,
@@ -31,7 +32,7 @@ class SponSkrubPP(PostProcessor):
            if path:
                raise PostProcessingError('sponskrub not found in "%s"' % path)
            else:
-                raise PostProcessingError('sponskrub not found. Please install or provide the path using --sponskrub-path.')
+                raise PostProcessingError('sponskrub not found. Please install or provide the path using --sponskrub-path')

    def get_exe(self, path=''):
        if not path or not check_executable(path, ['-h']):
@@ -70,8 +71,9 @@ class SponSkrubPP(PostProcessor):
        cmd = [self.path]
        if not self.cutout:
            cmd += ['-chapter']
+        cmd += cli_option(self._downloader.params, '-proxy', 'proxy')
        cmd += compat_shlex_split(self.args)  # For backward compatibility
-        cmd += self._configuration_args(exe=self._exe_name, use_default_arg='no_compat')
+        cmd += self._configuration_args(self._exe_name, use_compat=False)
        cmd += ['--', information['id'], filename, temp_filename]
        cmd = [encodeArgument(i) for i in cmd]

@@ -1836,7 +1836,7 @@ def write_json_file(obj, fn):

    try:
        with tf:
-            json.dump(obj, tf)
+            json.dump(obj, tf, default=repr)
        if sys.platform == 'win32':
            # Need to remove existing file on Windows, else os.rename raises
            # WindowsError or FileExistsError.
@@ -2423,6 +2423,15 @@ class DownloadError(YoutubeDLError):
        self.exc_info = exc_info


+class EntryNotInPlaylist(YoutubeDLError):
+    """Entry not in playlist exception.
+
+    This exception will be thrown by YoutubeDL when a requested entry
+    is not found in the playlist info_dict
+    """
+    pass
+
+
 class SameFileError(YoutubeDLError):
    """Same File exception.

@@ -3043,33 +3052,83 @@ def subtitles_filename(filename, sub_lang, sub_format, expected_real_ext=None):
    return replace_extension(filename, sub_lang + '.' + sub_format, expected_real_ext)


-def date_from_str(date_str):
+def datetime_from_str(date_str, precision='auto', format='%Y%m%d'):
    """
    Return a datetime object from a string in the format YYYYMMDD or
-    (now|today)[+-][0-9](day|week|month|year)(s)?"""
-    today = datetime.date.today()
+    (now|today|date)[+-][0-9](microsecond|second|minute|hour|day|week|month|year)(s)?
+
+    format: string date format used to return datetime object from
+    precision: round the time portion of a datetime object.
+                auto|microsecond|second|minute|hour|day.
+                auto: round to the unit provided in date_str (if applicable).
+    """
+    auto_precision = False
+    if precision == 'auto':
+        auto_precision = True
+        precision = 'microsecond'
+    today = datetime_round(datetime.datetime.now(), precision)
    if date_str in ('now', 'today'):
        return today
    if date_str == 'yesterday':
        return today - datetime.timedelta(days=1)
-    match = re.match(r'(now|today)(?P<sign>[+-])(?P<time>\d+)(?P<unit>day|week|month|year)(s)?', date_str)
+    match = re.match(
+        r'(?P<start>.+)(?P<sign>[+-])(?P<time>\d+)(?P<unit>microsecond|second|minute|hour|day|week|month|year)(s)?',
+        date_str)
    if match is not None:
-        sign = match.group('sign')
-        time = int(match.group('time'))
-        if sign == '-':
-            time = -time
+        start_time = datetime_from_str(match.group('start'), precision, format)
+        time = int(match.group('time')) * (-1 if match.group('sign') == '-' else 1)
        unit = match.group('unit')
-        # A bad approximation?
-        if unit == 'month':
+        if unit == 'month' or unit == 'year':
+            new_date = datetime_add_months(start_time, time * 12 if unit == 'year' else time)
            unit = 'day'
-            time *= 30
-        elif unit == 'year':
-            unit = 'day'
-            time *= 365
-        unit += 's'
-        delta = datetime.timedelta(**{unit: time})
-        return today + delta
-    return datetime.datetime.strptime(date_str, '%Y%m%d').date()
+        else:
+            if unit == 'week':
+                unit = 'day'
+                time *= 7
+            delta = datetime.timedelta(**{unit + 's': time})
+            new_date = start_time + delta
+        if auto_precision:
+            return datetime_round(new_date, unit)
+        return new_date
+
+    return datetime_round(datetime.datetime.strptime(date_str, format), precision)
+
+
+def date_from_str(date_str, format='%Y%m%d'):
+    """
+    Return a datetime object from a string in the format YYYYMMDD or
+    (now|today|date)[+-][0-9](microsecond|second|minute|hour|day|week|month|year)(s)?
+
+    format: string date format used to return datetime object from
+    """
+    return datetime_from_str(date_str, precision='microsecond', format=format).date()
+
+
+def datetime_add_months(dt, months):
+    """Increment/Decrement a datetime object by months."""
+    month = dt.month + months - 1
+    year = dt.year + month // 12
+    month = month % 12 + 1
+    day = min(dt.day, calendar.monthrange(year, month)[1])
+    return dt.replace(year, month, day)
+
+
+def datetime_round(dt, precision='day'):
+    """
+    Round a datetime object's time to a specific precision
+    """
+    if precision == 'microsecond':
+        return dt
+
+    unit_seconds = {
+        'day': 86400,
+        'hour': 3600,
+        'minute': 60,
+        'second': 1,
+    }
+    roundto = lambda x, n: ((x + n / 2) // n) * n
+    timestamp = calendar.timegm(dt.timetuple())
+    return datetime.datetime.utcfromtimestamp(roundto(timestamp, unit_seconds[precision]))


 def hyphenate_date(date_str):
@@ -4106,6 +4165,7 @@ def parse_age_limit(s):
    m = re.match(r'^(?P<age>\d{1,2})\+?$', s)
    if m:
        return int(m.group('age'))
+    s = s.upper()
    if s in US_RATINGS:
        return US_RATINGS[s]
    m = re.match(r'^TV[_-]?(%s)$' % '|'.join(k[3:] for k in TV_PARENTAL_GUIDELINES), s)
@@ -4182,8 +4242,10 @@ def qualities(quality_ids):

 DEFAULT_OUTTMPL = {
    'default': '%(title)s [%(id)s].%(ext)s',
+    'chapter': '%(title)s - %(section_number)03d %(section_title)s [%(id)s].%(ext)s',
 }
 OUTTMPL_TYPES = {
+    'chapter': None,
    'subtitle': None,
    'thumbnail': None,
    'description': 'description',
@@ -4193,6 +4255,20 @@ OUTTMPL_TYPES = {
    'pl_infojson': 'info.json',
 }

+# As of [1] format syntax is:
+#  %[mapping_key][conversion_flags][minimum_width][.precision][length_modifier]type
+# 1. https://docs.python.org/2/library/stdtypes.html#string-formatting
+FORMAT_RE = r'''(?x)
+    (?<!%)
+    %
+    \({0}\)  # mapping key
+    (?:[#0\-+ ]+)?  # conversion flags (optional)
+    (?:\d+)?  # minimum field width (optional)
+    (?:\.\d+)?  # precision (optional)
+    [hlL]?  # length modifier (optional)
+    (?P<type>[diouxXeEfFgGcrs%])  # conversion type
+'''
+

 def limit_length(s, length):
    """ Add ellipses to overly long strings """
@@ -4692,36 +4768,26 @@ def cli_valueless_option(params, command_option, param, expected_value=True):
    return [command_option] if param == expected_value else []


-def cli_configuration_args(argdict, key, default=[], exe=None, use_default_arg=True):
-    # use_default_arg can be True, False, or 'no_compat'
+def cli_configuration_args(argdict, keys, default=[], use_compat=True):
    if isinstance(argdict, (list, tuple)):  # for backward compatibility
-        if use_default_arg is True:
+        if use_compat:
            return argdict
        else:
            argdict = None
-
    if argdict is None:
        return default
    assert isinstance(argdict, dict)

-    key = key.lower()
-    args = exe_args = None
-    if exe is not None:
-        assert isinstance(exe, compat_str)
-        exe = exe.lower()
-        args = argdict.get('%s+%s' % (key, exe))
-        if args is None:
-            exe_args = argdict.get(exe)
-
-    if args is None:
-        args = argdict.get(key) if key != exe else None
-    if args is None and exe_args is None:
-        args = argdict.get('default', default) if use_default_arg else default
-
-    args, exe_args = args or [], exe_args or []
-    assert isinstance(args, (list, tuple))
-    assert isinstance(exe_args, (list, tuple))
-    return args + exe_args
+    assert isinstance(keys, (list, tuple))
+    for key_list in keys:
+        if isinstance(key_list, compat_str):
+            key_list = (key_list,)
+        arg_list = list(filter(
+            lambda x: x is not None,
+            [argdict.get(key.lower()) for key in key_list]))
+        if arg_list:
+            return [arg for args in arg_list for arg in args]
+    return default


 class ISO639Utils(object):
@@ -1,3 +1,3 @@
 from __future__ import unicode_literals

-__version__ = '2021.03.03.2'
+__version__ = '2021.04.03'
Author	SHA1	Message	Date
pukkandan	a0f30f194a	Release 2021.04.11	2021-04-12 03:20:07 +05:30
pukkandan	b31fdeedfd	[documentation] Improvements	2021-04-12 02:36:06 +05:30
pukkandan	8fa43c73d8	Add option `--convert-thumbnails` Closes: https://github.com/yt-dlp/yt-dlp/issues/99 https://github.com/yt-dlp/yt-dlp/issues/102	2021-04-12 02:32:29 +05:30
pukkandan	56d868dbb7	Allow running some `postprocessors` before actual download	2021-04-12 02:12:46 +05:30
coletdjnzandGitHub	f4f751af40	[youtube] Parse API parameters from initial webpage (#230 ) * Obtain innertube_context, api_key and x-goog-visitor-id from webpage * Generalize the header & Innertube_context extraction across YouTube extractors Related: https://github.com/ytdl-org/youtube-dl/commit/1b0a13f33cfb3644cc718d35951ea85bb1905459 Authored by: colethedj	2021-04-12 01:53:04 +05:30
pukkandan	1988fab7e3	[youtube] Fix thumbnail URL Closes: https://github.com/yt-dlp/yt-dlp/issues/233 https://github.com/ytdl-org/youtube-dl/issues/28023	2021-04-11 06:06:13 +05:30
pukkandan	9de3ea3126	Pass any field to `--exec` using similar syntax to output template Related: https://github.com/ytdl-org/youtube-dl/issues/28642	2021-04-11 06:06:12 +05:30
pukkandan	e01d6aa435	Fix `mergeall` when requested formats are unavailable	2021-04-10 22:29:58 +05:30
pukkandan	f7ad71607d	Update to ytdl-commit-4fb25ff [maoritv] Add new extractor https://github.com/ytdl-org/youtube-dl/commit/4fb25ff5a3be5206bb72e5c4046715b1529fb2c7 Except: [vimeo] improve extraction https://github.com/ytdl-org/youtube-dl/commit/3ae9c0f410b1d4f63e8bada67dd62a8d2852be32 [youtube:tab] Pass innertube context... https://github.com/ytdl-org/youtube-dl/commit/1b0a13f33cfb3644cc718d35951ea85bb1905459	2021-04-10 22:19:54 +05:30
pukkandan	68379de561	[downloaders] Fix API access Bug introduced by: `52a8a1e1b9`	2021-04-10 21:15:32 +05:30
pukkandan	d9aa233295	Deprecate more options * `--all-formats` = `-f all` * `--include-ads` is not implemented	2021-04-10 20:58:36 +05:30
pukkandan	f37468c41f	Fix default of `dynamic_mpd`	2021-04-10 20:57:53 +05:30
pukkandan	52a8a1e1b9	Option to choose different downloader for different protocols * Renamed `--external-downloader-args` to `--downloader-args` * Added `native` as an option for the downloader * Use similar syntax to `--downloader-args` etc. Eg: `--downloader dash:native --downloader aria2c` * Deprecated `--hls-prefer-native` and `--hls-prefer-ffmpeg` since the same can now be done with `--downloader "m3u8:native"` and `m3u8:ffmpeg` respectively * Split `frag_urls` protocol into `m3u8_frag_urls` and `dash_frag_urls` * Standardize shortening of protocol names with `downloader.shorten_protocol_name`	2021-04-10 20:57:52 +05:30
pukkandan	d818eb7473	Improve argument parsing for `-P`, `-o`, `-S` * `-P "subtitle,thumbnail:PATH"` is now possible. Similarly for `-o` * `-S "fps,br" -S "res,codec"` is now interpreted as `-S res,codec,fps,br`. Previously, `-S fps,br` was ignored in this case.	2021-04-10 20:57:27 +05:30
pukkandan	f8d4ad9ab0	Format selector `mergeall` to download and merge all formats	2021-04-10 20:57:27 +05:30
pukkandan	3ffc7c89b0	[youtube] Fix _extract_alerts * Sometimes one warning is split between multiple runs * Also simplified code	2021-04-10 20:57:27 +05:30
Henrik Heimbuergerandpukkandan	f1823403b0	[nebula] Add extractor (watchnebula.com) (#122 ) Authored by: hheimbuerger	2021-04-09 16:57:38 +05:30
LEandGitHub	384fb069ec	[NFHSNetwork] Add extractor (#217 ) Authored by: llacb47	2021-04-09 15:33:10 +05:30
bopolandGitHub	a4ddaf231e	[nitter] Fix extraction of reply tweets (#218 ) And update instance list Closes #215 Authored by: B0pol	2021-04-08 22:33:36 +05:30
pukkandan	7e60c06925	[bilibili] Fix uploader Related: https://github.com/yt-dlp/yt-dlp/issues/222	2021-04-08 13:22:04 +05:30
coletdjnzandpukkandan	d92f5d5a90	[youtube] Extract comments' approximate timestamp (#221 ) Authored by: colethedj	2021-04-07 17:09:06 +05:30
colethedjandpukkandan	9e62f283ff	[utils] Add `datetime_from_str` to parse relative time (#221 ) and `datetime_add_months` to accurately add/subtract months Authored by: colethedj	2021-04-07 17:09:06 +05:30
bopolandGitHub	c24ce07a84	[nitter] Fix thumbnails (#216 ) Authored by: B0pol	2021-04-05 16:53:33 +05:30
Ashish GitHub Ashish pukkandan	de6758128e	[DiscoveryPlusIndia] Add DiscoveryPlusIndiaShowIE (#213 ) Co-authored-by: Ashish <ashish@pop-os.localdomain> Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>	2021-04-05 16:20:08 +05:30
pukkandan	73d4343e39	Fix some `m3u8` not obeying `--allow-unplayable-formats`	2021-04-04 17:53:26 +05:30
pukkandan	57d104424f	[version] update :ci skip all	2021-04-04 03:57:29 +05:30
pukkandan	02aabd45d0	Release 2021.04.03	2021-04-04 03:55:41 +05:30
pukkandan	39ed931e53	Update to ytdl-commit-654b4f4 [youtube] prioritize information from YoutubeIE for playlist entries https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a	2021-04-04 03:49:04 +05:30
MatthewandGitHub	b28f8d244a	[YouTube] Show premium state in `availability` (#209 ) Authored by colethedj	2021-04-03 14:20:58 +05:30
pukkandan	73cd218f5a	[documentation] Improve `--parse-metadata` documentation	2021-04-03 14:07:29 +05:30
pukkandan	84601bb72b	Ability to set a specific field in the file's metadata Eg: `--parse-metadata "description:(?s)(?P<meta_comment>.+)"` sets the "comment" field using `description`	2021-04-03 14:07:28 +05:30
pukkandan	54df8fc5b2	[ViewSource] Add extractor to handle `view-source:`	2021-04-03 14:07:27 +05:30
The Hatsune DaishiandGitHub	5d39972ed0	[mildom] Update extractor to comply with current proxy (#212 ) Authored by nao20010128nao	2021-04-03 13:43:49 +05:30
pukkandanandGitHub	0481374e1d	[DiscoveryPlus] Add discoveryplus.in (#203 ) Closes #201	2021-04-03 13:18:40 +05:30
pukkandan	eff635394a	Ability to select nth best format Eg: -f `bv.2` Also cleaned up the related format selection code	2021-04-02 22:16:52 +05:30
pukkandan	df0c81513e	[generic] Detect Invidious' link element (Closes #195 )	2021-04-01 17:13:35 +05:30
pukkandan	3f6a90eb63	[ard:mediathek] Fix video id extraction (Closes #202 )	2021-04-01 16:17:15 +05:30
pukkandan	b050d210df	Fix some typos and linter	2021-04-01 14:16:10 +05:30
pukkandan	f4e4be19f0	[sponskrub] Run before embedding thumbnail Workaround for https://github.com/faissaloo/SponSkrub/issues/29 Closes https://github.com/yt-dlp/yt-dlp/issues/204	2021-04-01 14:13:24 +05:30
pukkandan	cce889b900	Update to ytdl v2021-04-01 https://github.com/ytdl-org/youtube-dl/commit/14f29f087e6097feb46bdb84878924bc410a57eb Closes #205	2021-04-01 13:59:22 +05:30
nixxoandGitHub	a6ae61a4c2	[la7] Add podcasts and podcast playlists (#198 ) Authored by: nixxo	2021-03-28 17:06:59 +05:30
pukkandan	b23b9eefd9	[version] update	2021-03-25 04:44:22 +05:30
pukkandan	a2f0b0c672	Release 2021.03.24.1	2021-03-25 04:33:34 +05:30
pukkandan	b704fc1a68	Revert commit 8562218 [ard] improve clip id extraction https://github.com/ytdl-org/youtube-dl/commit/8562218350a79d4709da8593bb0c538aa0824acf This was merged incorrectly	2021-03-25 04:31:32 +05:30
pukkandan	a3affbe6a0	Release 2021.03.24	2021-03-25 04:18:37 +05:30
pukkandan	1418a0437f	Update to ytdl-commit-8562218 [ard] improve clip id extraction https://github.com/ytdl-org/youtube-dl/commit/8562218350a79d4709da8593bb0c538aa0824acf	2021-03-25 04:09:21 +05:30
pukkandan	143db31d48	Parse metadata from multiple fields Closes #196	2021-03-25 03:32:15 +05:30
pukkandan	3700c7ef10	[linuxacadamy] Fix login	2021-03-24 04:06:26 +05:30
pukkandan	498f560638	Ability to load playlist infojson * If `--no-clean-infojson` is given, the video ids are saved/loaded from in the infojson along with their playlist index * If a video entry that was not saved is requested, we fallback to using `webpage_url` to re-extract the entries Related: https://github.com/yt-dlp/yt-dlp/issues/190#issuecomment-804921024	2021-03-24 01:57:50 +05:30
pukkandan	394dcd4486	Write current epoch to infojson when using `--no-clean-infojson` Closes #191	2021-03-24 00:34:33 +05:30
pukkandan	83b20a970d	[youtube_live_chat] fix bug when trying to set cookies Closes #194	2021-03-23 17:53:36 +05:30
pukkandan	e1feb88fdf	[niconico] Fix for when logged in Related: https://github.com/yt-dlp/yt-dlp/issues/171#issuecomment-803692716 Co-authored by: CXwudi, xtkoba	2021-03-23 14:03:07 +05:30
pukkandan	389b9dbbcc	[version] update	2021-03-22 03:03:06 +05:30
pukkandan	a7f347d9c9	Release 2021.03.21	2021-03-22 02:58:41 +05:30
pukkandan	421a459573	Update to ytdl-commit-7e79ba7 https://github.com/ytdl-org/youtube-dl/commit/7e79ba7dd6e6649dd2ce3a74004b2044f2182881 [vimeo:album] Fix extraction for albums with number of videos multiple to page size	2021-03-22 02:58:41 +05:30
Matthewandpukkandan	c224251aad	[youtube] Show if video is `private`, `unlisted` etc in new field `availability` (#188 ) Closes: #185, https://github.com/ytdl-org/youtube-dl/issues/25631 Authored by: colethedj, pukkandan	2021-03-22 02:58:41 +05:30
pukkandan	037cc66ec8	[linuxacadamy] Improve regex TODO: We need to make a more robust standard regex for fetching js objects from html	2021-03-21 21:27:33 +05:30
Two Sheds" JacksonandGitHub	9160a0c6a2	[nbc] Improve metadata extraction (#187 ) Authored by: 2ShedsJackson	2021-03-21 21:26:31 +05:30
2ShedsJacksonandGitHub	5c5fae6d2f	[amcnetworks] Fix extractor (#179 ) * Prefer use of manifest based on `releasePid` since the one based on `videoPid` may have Fairplay * Additional thumbnail images were added * Don't add `season_number` and `series` to `title` * `series` is now set to `None` rather than "_" when empty * fix bug with age limit Authored by: 2ShedsJackson	2021-03-20 16:11:11 +05:30
pukkandan	c1d3a4a8f0	[niconico] Add user extractor Co-authored by: animelover1984	2021-03-20 14:59:39 +05:30
pukkandan	adc74b3c6d	[bilibili] Add anthology support Closes: #118 Co-authored by: animelover1984	2021-03-20 14:56:42 +05:30
pukkandan	beb4b92a66	More consistent warning messages (#173 ) Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>	2021-03-20 09:34:52 +05:30
pukkandan	cd9b384cc3	Embed video URL metadata inside MP4 (#173 ) `mp4` has `comment` and `synopsis`; the synopsis is expected to have the long description So we save the `webpage_url` to `comment` and `description` to `synopsis` Related: https://github.com/ytdl-org/youtube-dl/issues/28478 Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>	2021-03-20 09:34:52 +05:30
pukkandan	4d971a16b8	Merge webm formats into mkv if thumbnails are to be embedded (#173 ) Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>	2021-03-20 09:34:52 +05:30
Damiano Amatrudaandpukkandan	3561530776	Parse resolution in info dictionary (#173 )	2021-03-20 09:34:52 +05:30
Damiano Amatrudaandpukkandan	4690688658	Use headers and cookies when downloading subtitles (#173 )	2021-03-20 09:34:52 +05:30
pukkandan	fe845284c4	[aria2c] Support retry/abort unavailable fragments (#173 ) Co-authored by: Damiano Amatruda <damiano.amatruda@outlook.com>	2021-03-20 09:33:33 +05:30
pukkandan	2b3bf01c90	[aria2c] Better default arguments	2021-03-20 09:31:53 +05:30
pukkandan	23c1a66730	Make sure `post_hook` gets the final filename	2021-03-20 09:31:53 +05:30
pukkandan	dd18a58cb1	[documentation] Add `SplitChapters` to `--ppa`	2021-03-20 09:31:53 +05:30
pukkandan	a94bfd6cfe	[splitchapters] Fix for older ffmpeg Older versions of ffmpeg doesn't allow `-to` as an input option Closes #180	2021-03-20 09:31:53 +05:30
pukkandan	a515a78dd3	fix some typos and linter	2021-03-18 22:19:31 +05:30
pukkandan	e167860ce7	[documentation] Add deprecated options and aliases in readme	2021-03-18 22:07:50 +05:30
pukkandan	75d43ca080	Option to keep private keys in the infojson Options: --clean-infojson, --no-clean-infojson Related: https://github.com/yt-dlp/yt-dlp/issues/42#issuecomment-800778391	2021-03-18 21:00:45 +05:30
pukkandan	5226731e2d	Recursively remove private keys from infojson Related: https://github.com/yt-dlp/yt-dlp/issues/42	2021-03-18 21:00:45 +05:30
pukkandan	dcf64d43e0	[movefiles] Fix bugs and make more robust	2021-03-18 20:54:53 +05:30
pukkandan	e3c076970e	[youtube] bugfix for channel playlist extraction Bug introduced in `cd7c66cf01`	2021-03-17 17:35:02 +05:30
pukkandan	7978e172f3	[niconico] Partial fix TODO: Smile formats don't work Extraction code that no longer works needs to be removed Adapted from: https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade by animelover1984 Related: https://github.com/yt-dlp/yt-dlp/issues/171 https://github.com/animelover1984/youtube-dl/issues/29 https://github.com/ytdl-org/youtube-dl/issues/28470	2021-03-17 15:38:24 +05:30
nixxoandGitHub	605d299f83	[stitcher] Merge from youtube-dl (#175 ) This cherry-picks the commits: https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f#diff-cc37718b18689c1c0588c4c74518853ef43ab020bbd674f3c04d4eae037798d2 https://github.com/ytdl-org/youtube-dl/commit/8522bcd97c4173407261a3fa0283dd7800c39601#diff-cc37718b18689c1c0588c4c74518853ef43ab020bbd674f3c04d4eae037798d2 Authored by: nixxo	2021-03-16 15:32:10 +05:30
nixxoandGitHub	18c1f04362	[rcs] Improved extraction (See desc) (#170 ) * improved `video_data` extraction * added an extra fallback value for `description` * improved regex in `RCSVariousIE` Authored by: nixxo	2021-03-16 00:26:29 +05:30
pukkandan	e4beae703d	[formatSort] Fix `quality` being ignored Closes #172	2021-03-15 23:55:53 +05:30
pukkandan	d034ab669c	[sponskrub] Pass proxy to sponskrub Needs atleast sponskrub version: faissaloo/sponskrub 3.7.0 OR yt-dlp/sponskrub 2021.03.15	2021-03-15 06:27:21 +05:30
pukkandan	5aeefbd633	[version] update :ci skip all	2021-03-15 06:03:59 +05:30
pukkandan	597c18665e	Release 2021.03.15	2021-03-15 05:54:39 +05:30
pukkandan	10db0d2f57	Update to ytdl-commit-3be0980 https://github.com/ytdl-org/youtube-dl/commit/3be098010f667b14075e3dfad1e74e5e2becc8ea	2021-03-15 04:52:06 +05:30
pukkandanandGitHub	7275535116	Split video by chapters (#158 ) * New options `--split-chapters` and `--no-split-chapters` * The output/path of the split files can be given using the key `chapter` * Additional keys `section_title`, `section_number`, `section_start`, `section_end` are available in the output template * Alias `--split-tracks` for parity with animelover/youtube-dl * `--sponskrub-cut` and `--split-chapter` cannot work together Closes: https://github.com/blackjack4494/yt-dlc/issues/277 https://github.com/ytdl-org/youtube-dl/issues/28438 https://github.com/ytdl-org/youtube-dl/issues/12907 https://github.com/ytdl-org/youtube-dl/issues/6480 https://github.com/ytdl-org/youtube-dl/pull/25005 Rewritten from the implementation by: femaref and Wattux https://github.com/Wattux/youtube-dl/tree/split-at-timestamps https://github.com/ytdl-org/youtube-dl/pull/25005 https://github.com/femaref/youtube-dl/tree/split-track	2021-03-15 04:32:13 +05:30
MatthewandGitHub	a1c5d2ca64	[Youtube] Rewrite comment extraction (#167 ) Closes #121 TODO: * Add an option for the user to specify newest/popular and max number of comments * Refactor the download code and generalize with TabIE * Parse time_text to timestamp	2021-03-15 04:11:11 +05:30
pukkandan	ca87974543	[embedthumbnail] Set mtime correctly Related: https://github.com/yt-dlp/yt-dlp/issues/67	2021-03-14 21:56:04 +05:30
pukkandan	e92caff5d5	Refactor (See desc) * Create `FFmpegPostProcessor.real_run_ffmpeg` that can accept multiple input/output files along with switches for each * Rewrite `cli_configuration_args` and related functions * Create `YoutubeDL._ensure_dir_exists` - this was previously defined in multiple places	2021-03-14 20:02:55 +05:30
CHJ85andpukkandan	ea3a012d2a	[pluto.tv] Add extractor (#163 ) https://github.com/ytdl-org/youtube-dl/pull/27621 Authored by: kevinoconnor7	2021-03-14 16:02:16 +05:30
pukkandan	5b8917fb52	[zee5] Support zee5originals	2021-03-14 15:22:29 +05:30
nixxoandGitHub	8eec0120a2	[rai] fix drm check (#168 ) Bug introduced by #150 Authored by: nixxo	2021-03-13 21:08:50 +05:30
shirtandGitHub	4cf1e5d2f9	Native concurrent downloading of fragments (#166 ) * Option `--concurrent-fragments` (`-N`) to set the number of threads Related: #165 Known issues: * When receiving Ctrl+C, the process will exit only after finishing the currently downloading fragments * The download progress shows the speed of only one thread Authored by shirt-dev	2021-03-13 10:16:58 +05:30
pukkandan	0a473f2f0f	More improvements to HLS/DASH external downloader code * Fix error when there is no `protocol` in `info_dict` * Move HLS byte range detection to `Aria2cFD` so that the download will fall back to the native downloader instead of ffmpeg * Fix bug with getting no fragments in DASH * Convert `check_results` in `can_download` to a generator	2021-03-11 22:07:42 +05:30
nixxoandGitHub	e4edeb6226	[wimtv] Add extractor (#161 ) Added support for VODs, live and embeds Authored by: nixxo	2021-03-11 13:28:51 +05:30
Ashish GitHub Ashish pukkandan	d488e254d9	[Zee5] Add Show Extractor (#160 ) Co-authored-by: Ashish <ashish@pop-os.localdomain> Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>	2021-03-11 13:18:09 +05:30
shirt-devandGitHub	d7009caa03	Improve HLS/DASH external downloader code (#162 ) Authored by: shirt	2021-03-10 20:09:40 +05:30
pukkandan	54759df586	[zee5] Improve regex	2021-03-09 15:17:16 +05:30
nixxoandGitHub	605b684c2d	[mtv] Add mtv.it and extract series metadata (#156 ) * New extractors: MTVItalia, MTVItaliaProgramma * Extract fields: series, season_number, episode_number Authored-by: nixxo	2021-03-08 19:10:27 +05:30
pukkandan	994443d24d	[version] update :ci skip all	2021-03-08 00:16:25 +05:30