Release 2021.03.07

https://github.com/ytdl-org/youtube-dl/pull/27992

Fixes: https://github.com/ytdl-org/youtube-dl/issues/28354

Authored by teesid

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.03.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.01
+ [debug] yt-dlp version 2021.03.03.2
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.03.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.01
+ [debug] yt-dlp version 2021.03.03.2
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.03.2. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.03.03.2**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

CONTRIBUTORS

@@ -21,6 +21,7 @@ nao20010128nao
 kurumigi
 tsukumi
 bbepis
+animelover1984
 Pccode66
 Ashish
 RobinD42
@@ -28,3 +29,4 @@ hseg
 colethedj
 DennyDai
 codeasashu
+teesid

Changelog.md

@@ -17,6 +17,26 @@
 -->
 
 
+### 2021.03.07
+* [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [colethedj](https://github.com/colethedj)
+* [youtube] Fix private feeds/playlists on multi-channel accounts by [colethedj](https://github.com/colethedj)
+* [youtube] Extract alerts from continuation by [colethedj](https://github.com/colethedj)
+* [cbs] Add support for ParamountPlus by [shirt](https://github.com/shirt-dev)
+* [mxplayer] Rewrite extractor with show support by [pukkandan](https://github.com/pukkandan) and [Ashish](https://github.com/Ashish)
+* [gedi] Improvements from youtube-dl by [nixxo](https://github.com/nixxo)
+* [vimeo] Fix videos with password by [teesid](https://github.com/teesid)
+* [lbry] Support lbry:// url by [nixxo](https://github.com/nixxo)
+* [bilibili] Change `Accept` header by [pukkandan](https://github.com/pukkandan) and [animelover1984](https://github.com/animelover1984)
+* [trovo] Pass origin header
+* [rai] Check for DRM by [nixxo](https://github.com/nixxo)
+* [downloader] Fix bug for ffmpeg/httpie
+* [update] Fix updater removing the executable bit on some UNIX distros
+* [update] Fix current build hash for UNIX
+* [documentation] Include wget/curl/aria2c install instructions for Unix by [Ashish](https://github.com/Ashish)
+* Fix some videos downloading with `m3u8` extension
+* Remove "fixup is ignored" warning when fixup wasn't passed by user
+
+
 ### 2021.03.03.2
 * [build] Fix bug
 
@@ -24,10 +44,10 @@
 ### 2021.03.03
 * [youtube] Use new browse API for continuation page extraction by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
 * Fix HLS playlist downloading by [shirt](https://github.com/shirt-dev)
-* **Merge youtube-dl:** Upto [2021.03.03](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.03.03)
+* Merge youtube-dl: Upto [2021.03.03](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.03.03)
 * [mtv] Fix extractor
 * [nick] Fix extractor by [DennyDai](https://github.com/DennyDai)
-* [mxplayer] Add new extractor by[codeasashu](https://github.com/codeasashu)
+* [mxplayer] Add new extractor by [codeasashu](https://github.com/codeasashu)
 * [youtube] Throw error when `--extractor-retries` are exhausted
 * Reduce default of `--extractor-retries` to 3
 * Fix packaging bugs by [hseg](https://github.com/hseg)
@@ -59,7 +79,7 @@
 * Moved project to an organization [yt-dlp](https://github.com/yt-dlp)
 * **Completely changed project name to yt-dlp** by [Pccode66](https://github.com/Pccode66) and [pukkandan](https://github.com/pukkandan)
     * Also, `youtube-dlc` config files are no longer loaded
-* **Merge youtube-dl:** Upto [commit/4460329](https://github.com/ytdl-org/youtube-dl/commit/44603290e5002153f3ebad6230cc73aef42cc2cd) (except tmz, gedi)
+* Merge youtube-dl: Upto [commit/4460329](https://github.com/ytdl-org/youtube-dl/commit/44603290e5002153f3ebad6230cc73aef42cc2cd) (except tmz, gedi)
 * [Readthedocs](https://yt-dlp.readthedocs.io) support by [shirt](https://github.com/shirt-dev)
 * [youtube] Show if video was a live stream in info (`was_live`)
 * [Zee5] Add new extractor by [Ashish](https://github.com/Ashish) and [pukkandan](https://github.com/pukkandan)
@@ -77,7 +97,7 @@
 
 
 ### 2021.02.19
-* **Merge youtube-dl:** Upto [commit/cf2dbec](https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d) (except kakao)
+* Merge youtube-dl: Upto [commit/cf2dbec](https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d) (except kakao)
 * [viki] Fix extractor
 * [niconico] Extract `channel` and `channel_id` by [kurumigi](https://github.com/kurumigi)
 * [youtube] Multiple page support for hashtag URLs
@@ -102,7 +122,7 @@
 
 
 ### 2021.02.15
-* **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
+* Merge youtube-dl: Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
 * [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
 * Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
 * [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
@@ -145,7 +165,7 @@
 
 
 ### 2021.02.04
-* **Merge youtube-dl:** Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
+* Merge youtube-dl: Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
 * **Date/time formatting in output template:**
     * You can use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
 * **Multiple output templates:**
@@ -199,7 +219,7 @@
 
 
 ### 2021.01.24
-* **Merge youtube-dl:** Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* Merge youtube-dl: Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
 * Plugin support ([documentation](https://github.com/yt-dlp/yt-dlp#plugins))
 * **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
     * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
@@ -228,7 +248,7 @@
 
 
 ### 2021.01.16
-* **Merge youtube-dl:** Upto [2021.01.16](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* Merge youtube-dl: Upto [2021.01.16](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
 * **Configuration files:**
     * Portable configuration file: `./yt-dlp.conf`
     * Allow the configuration files to be named `yt-dlp` instead of `youtube-dlc`. See [this](https://github.com/yt-dlp/yt-dlp#configuration) for details
@@ -274,8 +294,7 @@
 
 
 ### 2021.01.08
-* **Merge youtube-dl:** Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.08)
+* Merge youtube-dl: Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.08) except stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f))
-    * Extractor stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f)) have not been merged
 * Moved changelog to seperate file
 
 
@@ -314,7 +333,7 @@
     * Changed video format sorting to show video only files and video+audio files together.
     * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
     * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
+* Shortcut Options: Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-sponsorblock-options) for details
 * Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
 * Added `--list-formats-as-table`,  `--list-formats-old`
@@ -324,7 +343,7 @@
 * Relaxed validation for format filters so that any arbitrary field can be used
 * Fix for embedding thumbnail in mp3 by [pauldubois98](https://github.com/pauldubois98) ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
 * Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
-* **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
+* Merge youtube-dl: Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
     * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
 * Cleaned up the fork for public use
 

README.md

@@ -103,6 +103,23 @@ You can install yt-dlp using one of the following methods:
 * Use pip+git: `python -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp.git@release`
 * Install master branch: `python -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp`
 
+UNIX users (Linux, macOS, BSD) can also install the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest) one of the following ways:
+
+```
+sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
+sudo chmod a+rx /usr/local/bin/yt-dlp
+```
+
+```
+sudo wget https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -O /usr/local/bin/yt-dlp
+sudo chmod a+rx /usr/local/bin/yt-dlp
+```
+
+```
+sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
+sudo chmod a+rx /usr/local/bin/yt-dlp
+```
+
 ### UPDATE
 Starting from version `2021.02.09`, you can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.

supportedsites.md

@@ -347,8 +347,7 @@
  - **Gaskrank**
  - **Gazeta**
  - **GDCVault**
- - **Gedi**
+ - **GediDigital**
- - **GediEmbeds**
  - **generic**: Generic downloader that works on some sites
  - **Gfycat**
  - **GiantBomb**

test/test_all_urls.py

@@ -37,7 +37,6 @@ class TestAllURLsMatching(unittest.TestCase):
         assertPlaylist('PL63F0C78739B09958')
         assertTab('https://www.youtube.com/AsapSCIENCE')
         assertTab('https://www.youtube.com/embedded')
-        assertTab('https://www.youtube.com/feed')  # Own channel's home page
         assertTab('https://www.youtube.com/playlist?list=UUBABnxM4Ar9ten8Mdjj1j0Q')
         assertTab('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
         assertTab('https://www.youtube.com/playlist?list=PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC')

yt_dlp/downloader/external.py

@@ -282,8 +282,8 @@ class Aria2cFD(ExternalFD):
 
 class HttpieFD(ExternalFD):
     @classmethod
-    def available(cls):
+    def available(cls, path=None):
-        return check_executable('http', ['--version'])
+        return check_executable(path or 'http', ['--version'])
 
     def _make_cmd(self, tmpfilename, info_dict):
         cmd = ['http', '--download', '--output', tmpfilename, info_dict['url']]
@@ -298,7 +298,7 @@ class FFmpegFD(ExternalFD):
     SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
 
     @classmethod
-    def available(cls):
+    def available(cls, path=None):  # path is ignored for ffmpeg
         return FFmpegPostProcessor().available
 
     def _call_downloader(self, tmpfilename, info_dict):

yt_dlp/extractor/bilibili.py

@@ -138,6 +138,11 @@ class BiliBiliIE(InfoExtractor):
         anime_id = mobj.group('anime_id')
         page_id = mobj.group('page')
         webpage = self._download_webpage(url, video_id)
+        headers = {
+            'Referer': url,
+            'Accept': '*/*'
+        }
+        headers.update(self.geo_verification_headers())
 
         if 'anime/' not in url:
             cid = self._search_regex(
@@ -155,12 +160,8 @@ class BiliBiliIE(InfoExtractor):
             if 'no_bangumi_tip' not in smuggled_data:
                 self.to_screen('Downloading episode %s. To download all videos in anime %s, re-run yt-dlp with %s' % (
                     video_id, anime_id, compat_urlparse.urljoin(url, '//bangumi.bilibili.com/anime/%s' % anime_id)))
-            headers = {
-                'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
-                'Referer': url
-            }
-            headers.update(self.geo_verification_headers())
 
+            headers['Content-Type'] = 'application/x-www-form-urlencoded; charset=UTF-8'
             js = self._download_json(
                 'http://bangumi.bilibili.com/web_api/get_source', video_id,
                 data=urlencode_postdata({'episode_id': video_id}),
@@ -169,11 +170,6 @@ class BiliBiliIE(InfoExtractor):
                 self._report_error(js)
             cid = js['result']['cid']
 
-        headers = {
-            'Referer': url
-        }
-        headers.update(self.geo_verification_headers())
-
         entries = []
 
         RENDITIONS = ('qn=80&quality=80&type=', 'quality=2&type=mp4')

yt_dlp/extractor/cbs.py

@@ -27,10 +27,10 @@ class CBSBaseIE(ThePlatformFeedIE):
 
 
 class CBSIE(CBSBaseIE):
-    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
+    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:(?:cbs\.com|paramountplus\.com)/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
 
     _TESTS = [{
-        'url': 'http://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
+        'url': 'https://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
         'info_dict': {
             'id': '_u7W953k6la293J7EPTd9oHkSPs6Xn6_',
             'ext': 'mp4',
@@ -52,16 +52,19 @@ class CBSIE(CBSBaseIE):
     }, {
         'url': 'http://www.colbertlateshow.com/podcasts/dYSwjqPs_X1tvbV_P2FcPWRa_qT6akTC/in-the-bad-room-with-stephen/',
         'only_matching': True,
+    }, {
+        'url': 'https://www.paramountplus.com/shows/star-trek-discovery/video/l5ANMH9wM7kxwV1qr4u1xn88XOhYMlZX/star-trek-discovery-the-vulcan-hello/',
+        'only_matching': True,
     }]
 
     def _extract_video_info(self, content_id, site='cbs', mpx_acc=2198311517):
         items_data = self._download_xml(
-            'http://can.cbs.com/thunder/player/videoPlayerService.php',
+            'https://can.cbs.com/thunder/player/videoPlayerService.php',
             content_id, query={'partner': site, 'contentId': content_id})
         video_data = xpath_element(items_data, './/item')
         title = xpath_text(video_data, 'videoTitle', 'title') or xpath_text(video_data, 'videotitle', 'title')
         tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
-        tp_release_url = 'http://link.theplatform.com/s/' + tp_path
+        tp_release_url = 'https://link.theplatform.com/s/' + tp_path
 
         asset_types = []
         subtitles = {}

yt_dlp/extractor/common.py

@@ -1849,8 +1849,9 @@ class InfoExtractor(object):
 
     def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
                               entry_protocol='m3u8', preference=None, quality=None,
-                              m3u8_id=None, live=False, note=None, errnote=None,
+                              m3u8_id=None, note=None, errnote=None,
-                              fatal=True, data=None, headers={}, query={}):
+                              fatal=True, live=False, data=None, headers={},
+                              query={}):
         res = self._download_webpage_handle(
             m3u8_url, video_id,
             note=note or 'Downloading m3u8 information',
@@ -2050,11 +2051,11 @@ class InfoExtractor(object):
                 playlist_formats = _extract_m3u8_playlist_formats(manifest_url, video_id=video_id,
                                                                   fatal=fatal, data=data, headers=headers)
 
-                for format in playlist_formats:
+                for frmt in playlist_formats:
                     format_id = []
                     if m3u8_id:
                         format_id.append(m3u8_id)
-                    format_index = format.get('index')
+                    format_index = frmt.get('index')
                     stream_name = build_stream_name()
                     # Bandwidth of live streams may differ over time thus making
                     # format_id unpredictable. So it's better to keep provided
@@ -2109,6 +2110,8 @@ class InfoExtractor(object):
                             # TODO: update acodec for audio only formats with
                             # the same GROUP-ID
                             f['acodec'] = 'none'
+                    if not f.get('ext'):
+                        f['ext'] = 'm4a' if f.get('vcodec') == 'none' else 'mp4'
                     formats.append(f)
 
                     # for DailyMotion

yt_dlp/extractor/extractors.py

@@ -450,10 +450,7 @@ from .gamestar import GameStarIE
 from .gaskrank import GaskrankIE
 from .gazeta import GazetaIE
 from .gdcvault import GDCVaultIE
-from .gedi import (
+from .gedidigital import GediDigitalIE
-    GediIE,
-    GediEmbedsIE,
-)
 from .generic import GenericIE
 from .gfycat import GfycatIE
 from .giantbomb import GiantBombIE

yt_dlp/extractor/gedi.py

@@ -1,266 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..compat import compat_str
-from ..utils import (
-    base_url,
-    url_basename,
-    urljoin,
-)
-
-
-class GediBaseIE(InfoExtractor):
-    @staticmethod
-    def _clean_audio_fmts(formats):
-        unique_formats = []
-        for f in formats:
-            if 'acodec' in f:
-                unique_formats.append(f)
-        formats[:] = unique_formats
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, video_id)
-        player_data = re.findall(
-            r'PlayerFactory\.setParam\(\'(?P<type>.+?)\',\s*\'(?P<name>.+?)\',\s*\'(?P<val>.+?)\'\);',
-            webpage)
-
-        formats = []
-        audio_fmts = []
-        hls_fmts = []
-        http_fmts = []
-        title = ''
-        thumb = ''
-
-        fmt_reg = r'(?P<t>video|audio)-(?P<p>rrtv|hls)-(?P<h>[\w\d]+)(?:-(?P<br>[\w\d]+))?$'
-        br_reg = r'video-rrtv-(?P<br>\d+)-'
-
-        for t, n, v in player_data:
-            if t == 'format':
-                m = re.match(fmt_reg, n)
-                if m:
-                    # audio formats
-                    if m.group('t') == 'audio':
-                        if m.group('p') == 'hls':
-                            audio_fmts.extend(self._extract_m3u8_formats(
-                                v, video_id, 'm4a', m3u8_id='hls', fatal=False))
-                        elif m.group('p') == 'rrtv':
-                            audio_fmts.append({
-                                'format_id': 'mp3',
-                                'url': v,
-                                'tbr': 128,
-                                'ext': 'mp3',
-                                'vcodec': 'none',
-                                'acodec': 'mp3',
-                            })
-
-                    # video formats
-                    elif m.group('t') == 'video':
-                        # hls manifest video
-                        if m.group('p') == 'hls':
-                            hls_fmts.extend(self._extract_m3u8_formats(
-                                v, video_id, 'mp4', m3u8_id='hls', fatal=False))
-                        # direct mp4 video
-                        elif m.group('p') == 'rrtv':
-                            if not m.group('br'):
-                                mm = re.search(br_reg, v)
-                            http_fmts.append({
-                                'format_id': 'https-' + m.group('h'),
-                                'protocol': 'https',
-                                'url': v,
-                                'tbr': int(m.group('br')) if m.group('br') else
-                                (int(mm.group('br')) if mm.group('br') else 0),
-                                'height': int(m.group('h'))
-                            })
-
-            elif t == 'param':
-                if n == 'videotitle':
-                    title = v
-                if n == 'image_full_play':
-                    thumb = v
-
-        title = self._og_search_title(webpage) if title == '' else title
-
-        # clean weird char
-        title = compat_str(title).encode('utf8', 'replace').replace(b'\xc3\x82', b'').decode('utf8', 'replace')
-
-        if audio_fmts:
-            self._clean_audio_fmts(audio_fmts)
-            self._sort_formats(audio_fmts)
-        if hls_fmts:
-            self._sort_formats(hls_fmts)
-        if http_fmts:
-            self._sort_formats(http_fmts)
-
-        formats.extend(audio_fmts)
-        formats.extend(hls_fmts)
-        formats.extend(http_fmts)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': self._html_search_meta('twitter:description', webpage),
-            'thumbnail': thumb,
-            'formats': formats,
-        }
-
-
-class GediIE(GediBaseIE):
-    _VALID_URL = r'''(?x)https?://video\.
-                    (?:
-                        (?:espresso\.)?repubblica
-                        |lastampa
-                        |huffingtonpost
-                        |ilsecoloxix
-                        |iltirreno
-                        |messaggeroveneto
-                        |ilpiccolo
-                        |gazzettadimantova
-                        |mattinopadova
-                        |laprovinciapavese
-                        |tribunatreviso
-                        |nuovavenezia
-                        |gazzettadimodena
-                        |lanuovaferrara
-                        |corrierealpi
-                        |lasentinella
-                    )
-                    (?:\.gelocal)?\.it/(?!embed/).+?/(?P<id>[\d/]+)(?:\?|\&|$)'''
-    _TESTS = [{
-        'url': 'https://video.lastampa.it/politica/il-paradosso-delle-regionali-la-lega-vince-ma-sembra-aver-perso/121559/121683',
-        'md5': '84658d7fb9e55a6e57ecc77b73137494',
-        'info_dict': {
-            'id': '121559/121683',
-            'ext': 'mp4',
-            'title': 'Il paradosso delle Regionali: ecco perché la Lega vince ma sembra aver perso',
-            'description': 'md5:de7f4d6eaaaf36c153b599b10f8ce7ca',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }, {
-        'url': 'https://video.repubblica.it/motori/record-della-pista-a-spa-francorchamps-la-pagani-huayra-roadster-bc-stupisce/367415/367963',
-        'md5': 'e763b94b7920799a0e0e23ffefa2d157',
-        'info_dict': {
-            'id': '367415/367963',
-            'ext': 'mp4',
-            'title': 'Record della pista a Spa Francorchamps, la Pagani Huayra Roadster BC stupisce',
-            'description': 'md5:5deb503cefe734a3eb3f07ed74303920',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }, {
-        'url': 'https://video.ilsecoloxix.it/sport/cassani-e-i-brividi-azzurri-ai-mondiali-di-imola-qui-mi-sono-innamorato-del-ciclismo-da-ragazzino-incredibile-tornarci-da-ct/66184/66267',
-        'md5': 'e48108e97b1af137d22a8469f2019057',
-        'info_dict': {
-            'id': '66184/66267',
-            'ext': 'mp4',
-            'title': 'Cassani e i brividi azzurri ai Mondiali di Imola: \\"Qui mi sono innamorato del ciclismo da ragazzino, incredibile tornarci da ct\\"',
-            'description': 'md5:fc9c50894f70a2469bb9b54d3d0a3d3b',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }, {
-        'url': 'https://video.iltirreno.gelocal.it/sport/dentro-la-notizia-ferrari-cosa-succede-a-maranello/141059/142723',
-        'md5': 'a6e39f3bdc1842bbd92abbbbef230817',
-        'info_dict': {
-            'id': '141059/142723',
-            'ext': 'mp4',
-            'title': 'Dentro la notizia - Ferrari, cosa succede a Maranello',
-            'description': 'md5:9907d65b53765681fa3a0b3122617c1f',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }]
-
-
-class GediEmbedsIE(GediBaseIE):
-    _VALID_URL = r'''(?x)https?://video\.
-                    (?:
-                        (?:espresso\.)?repubblica
-                        |lastampa
-                        |huffingtonpost
-                        |ilsecoloxix
-                        |iltirreno
-                        |messaggeroveneto
-                        |ilpiccolo
-                        |gazzettadimantova
-                        |mattinopadova
-                        |laprovinciapavese
-                        |tribunatreviso
-                        |nuovavenezia
-                        |gazzettadimodena
-                        |lanuovaferrara
-                        |corrierealpi
-                        |lasentinella
-                    )
-                    (?:\.gelocal)?\.it/embed/.+?/(?P<id>[\d/]+)(?:\?|\&|$)'''
-    _TESTS = [{
-        'url': 'https://video.huffingtonpost.it/embed/politica/cotticelli-non-so-cosa-mi-sia-successo-sto-cercando-di-capire-se-ho-avuto-un-malore/29312/29276?responsive=true&el=video971040871621586700',
-        'md5': 'f4ac23cadfea7fef89bea536583fa7ed',
-        'info_dict': {
-            'id': '29312/29276',
-            'ext': 'mp4',
-            'title': 'Cotticelli: \\"Non so cosa mi sia successo. Sto cercando di capire se ho avuto un malore\\"',
-            'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }, {
-        'url': 'https://video.espresso.repubblica.it/embed/tutti-i-video/01-ted-villa/14772/14870&width=640&height=360',
-        'md5': '0391c2c83c6506581003aaf0255889c0',
-        'info_dict': {
-            'id': '14772/14870',
-            'ext': 'mp4',
-            'title': 'Festival EMERGENCY, Villa: «La buona informazione aiuta la salute» (14772-14870)',
-            'description': 'md5:2bce954d278248f3c950be355b7c2226',
-            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-social-play\.jpg$',
-        },
-    }]
-
-    @staticmethod
-    def _sanitize_urls(urls):
-        # add protocol if missing
-        for i, e in enumerate(urls):
-            if e.startswith('//'):
-                urls[i] = 'https:%s' % e
-        # clean iframes urls
-        for i, e in enumerate(urls):
-            urls[i] = urljoin(base_url(e), url_basename(e))
-        return urls
-
-    @staticmethod
-    def _extract_urls(webpage):
-        entries = [
-            mobj.group('url')
-            for mobj in re.finditer(r'''(?x)
-            (?:
-                data-frame-src=|
-                <iframe[^\n]+src=
-            )
-            (["'])
-                (?P<url>https?://video\.
-                    (?:
-                        (?:espresso\.)?repubblica
-                        |lastampa
-                        |huffingtonpost
-                        |ilsecoloxix
-                        |iltirreno
-                        |messaggeroveneto
-                        |ilpiccolo
-                        |gazzettadimantova
-                        |mattinopadova
-                        |laprovinciapavese
-                        |tribunatreviso
-                        |nuovavenezia
-                        |gazzettadimodena
-                        |lanuovaferrara
-                        |corrierealpi
-                        |lasentinella
-                    )
-                    (?:\.gelocal)?\.it/embed/.+?)
-            \1''', webpage)]
-        return GediEmbedsIE._sanitize_urls(entries)
-
-    @staticmethod
-    def _extract_url(webpage):
-        urls = GediEmbedsIE._extract_urls(webpage)
-        return urls[0] if urls else None

yt_dlp/extractor/gedidigital.py

@@ -0,0 +1,210 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    base_url,
+    determine_ext,
+    int_or_none,
+    url_basename,
+    urljoin,
+)
+
+
+class GediDigitalIE(InfoExtractor):
+    _VALID_URL = r'''(?x)(?P<url>(?:https?:)//video\.
+        (?:
+            (?:
+                (?:espresso\.)?repubblica
+                |lastampa
+                |ilsecoloxix
+                |huffingtonpost
+            )|
+            (?:
+                iltirreno
+                |messaggeroveneto
+                |ilpiccolo
+                |gazzettadimantova
+                |mattinopadova
+                |laprovinciapavese
+                |tribunatreviso
+                |nuovavenezia
+                |gazzettadimodena
+                |lanuovaferrara
+                |corrierealpi
+                |lasentinella
+            )\.gelocal
+        )\.it(?:/[^/]+){2,4}/(?P<id>\d+))(?:$|[?&].*)'''
+    _TESTS = [{
+        'url': 'https://video.lastampa.it/politica/il-paradosso-delle-regionali-la-lega-vince-ma-sembra-aver-perso/121559/121683',
+        'md5': '84658d7fb9e55a6e57ecc77b73137494',
+        'info_dict': {
+            'id': '121683',
+            'ext': 'mp4',
+            'title': 'Il paradosso delle Regionali: ecco perché la Lega vince ma sembra aver perso',
+            'description': 'md5:de7f4d6eaaaf36c153b599b10f8ce7ca',
+            'thumbnail': r're:^https://www\.repstatic\.it/video/photo/.+?-thumb-full-.+?\.jpg$',
+            'duration': 125,
+        },
+    }, {
+        'url': 'https://video.huffingtonpost.it/embed/politica/cotticelli-non-so-cosa-mi-sia-successo-sto-cercando-di-capire-se-ho-avuto-un-malore/29312/29276?responsive=true&el=video971040871621586700',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.espresso.repubblica.it/embed/tutti-i-video/01-ted-villa/14772/14870&width=640&height=360',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.repubblica.it/motori/record-della-pista-a-spa-francorchamps-la-pagani-huayra-roadster-bc-stupisce/367415/367963',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.ilsecoloxix.it/sport/cassani-e-i-brividi-azzurri-ai-mondiali-di-imola-qui-mi-sono-innamorato-del-ciclismo-da-ragazzino-incredibile-tornarci-da-ct/66184/66267',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.iltirreno.gelocal.it/sport/dentro-la-notizia-ferrari-cosa-succede-a-maranello/141059/142723',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.messaggeroveneto.gelocal.it/locale/maria-giovanna-elmi-covid-vaccino/138155/139268',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.ilpiccolo.gelocal.it/dossier/big-john/dinosauro-big-john-al-via-le-visite-guidate-a-trieste/135226/135751',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.gazzettadimantova.gelocal.it/locale/dal-ponte-visconteo-di-valeggio-l-and-8217sos-dei-ristoratori-aprire-anche-a-cena/137310/137818',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.mattinopadova.gelocal.it/dossier/coronavirus-in-veneto/covid-a-vo-un-anno-dopo-un-cuore-tricolore-per-non-dimenticare/138402/138964',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.laprovinciapavese.gelocal.it/locale/mede-zona-rossa-via-alle-vaccinazioni-per-gli-over-80/137545/138120',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.tribunatreviso.gelocal.it/dossier/coronavirus-in-veneto/ecco-le-prima-vaccinazioni-di-massa-nella-marca/134485/135024',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.nuovavenezia.gelocal.it/locale/camion-troppo-alto-per-il-ponte-ferroviario-perde-il-carico/135734/136266',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.gazzettadimodena.gelocal.it/locale/modena-scoperta-la-proteina-che-predice-il-livello-di-gravita-del-covid/139109/139796',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.lanuovaferrara.gelocal.it/locale/due-bombole-di-gpl-aperte-e-abbandonate-i-vigili-bruciano-il-gas/134391/134957',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.corrierealpi.gelocal.it/dossier/cortina-2021-i-mondiali-di-sci-alpino/mondiali-di-sci-il-timelapse-sulla-splendida-olympia/133760/134331',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.lasentinella.gelocal.it/locale/vestigne-centra-un-auto-e-si-ribalta/138931/139466',
+        'only_matching': True,
+    }, {
+        'url': 'https://video.espresso.repubblica.it/tutti-i-video/01-ted-villa/14772',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _sanitize_urls(urls):
+        # add protocol if missing
+        for i, e in enumerate(urls):
+            if e.startswith('//'):
+                urls[i] = 'https:%s' % e
+        # clean iframes urls
+        for i, e in enumerate(urls):
+            urls[i] = urljoin(base_url(e), url_basename(e))
+        return urls
+
+    @staticmethod
+    def _extract_urls(webpage):
+        entries = [
+            mobj.group('eurl')
+            for mobj in re.finditer(r'''(?x)
+            (?:
+                data-frame-src=|
+                <iframe[^\n]+src=
+            )
+            (["'])(?P<eurl>%s)\1''' % GediDigitalIE._VALID_URL, webpage)]
+        return GediDigitalIE._sanitize_urls(entries)
+
+    @staticmethod
+    def _extract_url(webpage):
+        urls = GediDigitalIE._extract_urls(webpage)
+        return urls[0] if urls else None
+
+    @staticmethod
+    def _clean_formats(formats):
+        format_urls = set()
+        clean_formats = []
+        for f in formats:
+            if f['url'] not in format_urls:
+                if f.get('audio_ext') != 'none' and not f.get('acodec'):
+                    continue
+                format_urls.add(f['url'])
+                clean_formats.append(f)
+        formats[:] = clean_formats
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        url = re.match(self._VALID_URL, url).group('url')
+        webpage = self._download_webpage(url, video_id)
+        title = self._html_search_meta(
+            ['twitter:title', 'og:title'], webpage, fatal=True)
+        player_data = re.findall(
+            r"PlayerFactory\.setParam\('(?P<type>format|param)',\s*'(?P<name>[^']+)',\s*'(?P<val>[^']+)'\);",
+            webpage)
+
+        formats = []
+        duration = thumb = None
+        for t, n, v in player_data:
+            if t == 'format':
+                if n in ('video-hds-vod-ec', 'video-hls-vod-ec', 'video-viralize', 'video-youtube-pfp'):
+                    continue
+                elif n.endswith('-vod-ak'):
+                    formats.extend(self._extract_akamai_formats(
+                        v, video_id, {'http': 'media.gedidigital.it'}))
+                else:
+                    ext = determine_ext(v)
+                    if ext == 'm3u8':
+                        formats.extend(self._extract_m3u8_formats(
+                            v, video_id, 'mp4', 'm3u8_native', m3u8_id=n, fatal=False))
+                        continue
+                    f = {
+                        'format_id': n,
+                        'url': v,
+                    }
+                    if ext == 'mp3':
+                        abr = int_or_none(self._search_regex(
+                            r'-mp3-audio-(\d+)', v, 'abr', default=None))
+                        f.update({
+                            'abr': abr,
+                            'tbr': abr,
+                            'acodec': ext,
+                            'vcodec': 'none'
+                        })
+                    else:
+                        mobj = re.match(r'^video-rrtv-(\d+)(?:-(\d+))?$', n)
+                        if mobj:
+                            f.update({
+                                'height': int(mobj.group(1)),
+                                'vbr': int_or_none(mobj.group(2)),
+                            })
+                        if not f.get('vbr'):
+                            f['vbr'] = int_or_none(self._search_regex(
+                                r'-video-rrtv-(\d+)', v, 'abr', default=None))
+                    formats.append(f)
+            elif t == 'param':
+                if n in ['image_full', 'image']:
+                    thumb = v
+                elif n == 'videoDuration':
+                    duration = int_or_none(v)
+
+        self._clean_formats(formats)
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': self._html_search_meta(
+                ['twitter:description', 'og:description', 'description'], webpage),
+            'thumbnail': thumb or self._og_search_thumbnail(webpage),
+            'formats': formats,
+            'duration': duration,
+        }

yt_dlp/extractor/generic.py

@@ -127,7 +127,7 @@ from .expressen import ExpressenIE
 from .zype import ZypeIE
 from .odnoklassniki import OdnoklassnikiIE
 from .kinja import KinjaEmbedIE
-from .gedi import GediEmbedsIE
+from .gedidigital import GediDigitalIE
 from .rcs import RCSEmbedsIE
 from .bitchute import BitChuteIE
 from .rumble import RumbleEmbedIE
@@ -3339,12 +3339,12 @@ class GenericIE(InfoExtractor):
             return self.playlist_from_matches(
                 zype_urls, video_id, video_title, ie=ZypeIE.ie_key())
 
-        # Look for RCS media group embeds
+        gedi_urls = GediDigitalIE._extract_urls(webpage)
-        gedi_urls = GediEmbedsIE._extract_urls(webpage)
         if gedi_urls:
             return self.playlist_from_matches(
-                gedi_urls, video_id, video_title, ie=GediEmbedsIE.ie_key())
+                gedi_urls, video_id, video_title, ie=GediDigitalIE.ie_key())
 
+        # Look for RCS media group embeds
         rcs_urls = RCSEmbedsIE._extract_urls(webpage)
         if rcs_urls:
             return self.playlist_from_matches(

yt_dlp/extractor/lbry.py

@@ -21,9 +21,9 @@ from ..utils import (
 
 
 class LBRYBaseIE(InfoExtractor):
-    _BASE_URL_REGEX = r'https?://(?:www\.)?(?:lbry\.tv|odysee\.com)/'
+    _BASE_URL_REGEX = r'(?:https?://(?:www\.)?(?:lbry\.tv|odysee\.com)/|lbry://)'
     _CLAIM_ID_REGEX = r'[0-9a-f]{1,40}'
-    _OPT_CLAIM_ID = '[^:/?#&]+(?::%s)?' % _CLAIM_ID_REGEX
+    _OPT_CLAIM_ID = '[^:/?#&]+(?:[:#]%s)?' % _CLAIM_ID_REGEX
     _SUPPORTED_STREAM_TYPES = ['video', 'audio']
 
     def _call_api_proxy(self, method, display_id, params, resource):
@@ -41,7 +41,9 @@ class LBRYBaseIE(InfoExtractor):
             'resolve', display_id, {'urls': url}, resource)[url]
 
     def _permanent_url(self, url, claim_name, claim_id):
-        return urljoin(url, '/%s:%s' % (claim_name, claim_id))
+        return urljoin(
+            url.replace('lbry://', 'https://lbry.tv/'),
+            '/%s:%s' % (claim_name, claim_id))
 
     def _parse_stream(self, stream, url):
         stream_value = stream.get('value') or {}
@@ -137,6 +139,9 @@ class LBRYIE(LBRYBaseIE):
     }, {
         'url': 'https://lbry.tv/@lacajadepandora:a/TRUMP-EST%C3%81-BIEN-PUESTO-con-Pilar-Baselga,-Carlos-Senra,-Luis-Palacios-(720p_30fps_H264-192kbit_AAC):1',
         'only_matching': True,
+    }, {
+        'url': 'lbry://@lbry#3f/odysee#7',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -166,7 +171,7 @@ class LBRYIE(LBRYBaseIE):
 
 class LBRYChannelIE(LBRYBaseIE):
     IE_NAME = 'lbry:channel'
-    _VALID_URL = LBRYBaseIE._BASE_URL_REGEX + r'(?P<id>@%s)/?(?:[?#&]|$)' % LBRYBaseIE._OPT_CLAIM_ID
+    _VALID_URL = LBRYBaseIE._BASE_URL_REGEX + r'(?P<id>@%s)/?(?:[?&]|$)' % LBRYBaseIE._OPT_CLAIM_ID
     _TESTS = [{
         'url': 'https://lbry.tv/@LBRYFoundation:0',
         'info_dict': {
@@ -178,6 +183,9 @@ class LBRYChannelIE(LBRYBaseIE):
     }, {
         'url': 'https://lbry.tv/@LBRYFoundation',
         'only_matching': True,
+    }, {
+        'url': 'lbry://@lbry#3f',
+        'only_matching': True,
     }]
     _PAGE_SIZE = 50
 

yt_dlp/extractor/mxplayer.py

@@ -6,98 +6,122 @@ from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
     js_to_json,
+    qualities,
+    try_get,
     url_or_none,
     urljoin,
 )
 
 
-VALID_STREAMS = ('dash', )
-
-
 class MxplayerIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?mxplayer\.in/movie/(?P<slug>[a-z0-9]+(?:-[a-z0-9]+)*)'
+    _VALID_URL = r'https?://(?:www\.)?mxplayer\.in/(?:show|movie)/(?:(?P<display_id>[-/a-z0-9]+)-)?(?P<id>[a-z0-9]+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'https://www.mxplayer.in/movie/watch-knock-knock-hindi-dubbed-movie-online-b9fa28df3bfb8758874735bbd7d2655a?watch=true',
         'info_dict': {
             'id': 'b9fa28df3bfb8758874735bbd7d2655a',
             'ext': 'mp4',
-            'title': 'Knock Knock Movie | Watch 2015 Knock Knock Full Movie Online- MX Player',
+            'title': 'Knock Knock (Hindi Dubbed)',
             'description': 'md5:b195ba93ff1987309cfa58e2839d2a5b'
         },
         'params': {
             'skip_download': True,
             'format': 'bestvideo'
         }
-    }
+    }, {
-
+        'url': 'https://www.mxplayer.in/show/watch-shaitaan/season-1/the-infamous-taxi-gang-of-meerut-online-45055d5bcff169ad48f2ad7552a83d6c',
-    def _get_best_stream_url(self, stream):
+        'info_dict': {
-        best_stream = list(filter(None, [v for k, v in stream.items()]))
+            'id': '45055d5bcff169ad48f2ad7552a83d6c',
-        return best_stream.pop(0) if len(best_stream) else None
+            'ext': 'm3u8',
+            'title': 'The infamous taxi gang of Meerut',
+            'description': 'md5:033a0a7e3fd147be4fb7e07a01a3dc28',
+            'season': 'Season 1',
+            'series': 'Shaitaan'
+        },
+        'params': {
+            'skip_download': True,
+        }
+    }, {
+        'url': 'https://www.mxplayer.in/show/watch-aashram/chapter-1/duh-swapna-online-d445579792b0135598ba1bc9088a84cb',
+        'info_dict': {
+            'id': 'd445579792b0135598ba1bc9088a84cb',
+            'ext': 'mp4',
+            'title': 'Duh Swapna',
+            'description': 'md5:35ff39c4bdac403c53be1e16a04192d8',
+            'season': 'Chapter 1',
+            'series': 'Aashram'
+        },
+        'expected_warnings': ['Unknown MIME type application/mp4 in DASH manifest'],
+        'params': {
+            'skip_download': True,
+            'format': 'bestvideo'
+        }
+    }]
 
     def _get_stream_urls(self, video_dict):
-        stream_dict = video_dict.get('stream', {'provider': {}})
+        stream_provider_dict = try_get(
-        stream_provider = stream_dict.get('provider')
+            video_dict,
+            lambda x: x['stream'][x['stream']['provider']])
+        if not stream_provider_dict:
+            raise ExtractorError('No stream provider found', expected=True)
 
-        if not stream_dict[stream_provider]:
+        for stream_name, stream in stream_provider_dict.items():
-            message = 'No stream provider found'
+            if stream_name in ('hls', 'dash', 'hlsUrl', 'dashUrl'):
-            raise ExtractorError('%s said: %s' % (self.IE_NAME, message), expected=True)
+                stream_type = stream_name.replace('Url', '')
-
+                if isinstance(stream, dict):
-        streams = []
+                    for quality, stream_url in stream.items():
-        for stream_name, v in stream_dict[stream_provider].items():
+                        if stream_url:
-            if stream_name in VALID_STREAMS:
+                            yield stream_type, quality, stream_url
-                stream_url = self._get_best_stream_url(v)
+                else:
-                if stream_url is None:
+                    yield stream_type, 'base', stream
-                    continue
-                streams.append((stream_name, stream_url))
-        return streams
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        display_id, video_id = re.match(self._VALID_URL, url).groups()
-        video_slug = mobj.group('slug')
-
-        video_id = video_slug.split('-')[-1]
-
         webpage = self._download_webpage(url, video_id)
 
-        window_state_json = self._html_search_regex(
+        source = self._parse_json(
-            r'(?s)<script>window\.state\s*[:=]\s(\{.+\})\n(\w+).*(</script>).*',
+            js_to_json(self._html_search_regex(
-            webpage, 'WindowState')
+                r'(?s)<script>window\.state\s*[:=]\s(\{.+\})\n(\w+).*(</script>).*',
-
+                webpage, 'WindowState')),
-        source = self._parse_json(js_to_json(window_state_json), video_id)
+            video_id)
         if not source:
             raise ExtractorError('Cannot find source', expected=True)
 
         config_dict = source['config']
         video_dict = source['entities'][video_id]
-        stream_urls = self._get_stream_urls(video_dict)
 
-        title = self._og_search_title(webpage, fatal=True, default=video_dict['title'])
+        thumbnails = []
+        for i in video_dict.get('imageInfo') or []:
+            thumbnails.append({
+                'url': urljoin(config_dict['imageBaseUrl'], i['url']),
+                'width': i['width'],
+                'height': i['height'],
+            })
 
         formats = []
-        headers = {'Referer': url}
+        get_quality = qualities(['main', 'base', 'high'])
-        for stream_name, stream_url in stream_urls:
+        for stream_type, quality, stream_url in self._get_stream_urls(video_dict):
-            if stream_name == 'dash':
+            format_url = url_or_none(urljoin(config_dict['videoCdnBaseUrl'], stream_url))
-                format_url = url_or_none(urljoin(config_dict['videoCdnBaseUrl'], stream_url))
+            if not format_url:
-                if not format_url:
+                continue
-                    continue
+            if stream_type == 'dash':
-                formats.extend(self._extract_mpd_formats(
+                dash_formats = self._extract_mpd_formats(
-                    format_url, video_id, mpd_id='dash', headers=headers))
+                    format_url, video_id, mpd_id='dash-%s' % quality, headers={'Referer': url})
+                for frmt in dash_formats:
+                    frmt['quality'] = get_quality(quality)
+                formats.extend(dash_formats)
+            elif stream_type == 'hls':
+                formats.extend(self._extract_m3u8_formats(
+                    format_url, video_id, fatal=False,
+                    m3u8_id='hls-%s' % quality, quality=get_quality(quality)))
 
         self._sort_formats(formats)
-        info = {
+        return {
             'id': video_id,
-            'title': title,
+            'display_id': display_id.replace('/', '-'),
+            'title': video_dict['title'] or self._og_search_title(webpage),
+            'formats': formats,
             'description': video_dict.get('description'),
-            'formats': formats
+            'season': try_get(video_dict, lambda x: x['container']['title']),
+            'series': try_get(video_dict, lambda x: x['container']['container']['title']),
+            'thumbnails': thumbnails,
         }
-
-        if video_dict.get('imageInfo'):
-            info['thumbnails'] = list(map(lambda i: dict(i, **{
-                'url': urljoin(config_dict['imageBaseUrl'], i['url'])
-            }), video_dict['imageInfo']))
-
-        if video_dict.get('webUrl'):
-            last_part = video_dict['webUrl'].split("/")[-1]
-            info['display_id'] = last_part.replace(video_id, "").rstrip("-")
-
-        return info

yt_dlp/extractor/rai.py

@@ -158,6 +158,10 @@ class RaiPlayIE(RaiBaseIE):
         # subtitles at 'subtitlesArray' key (see #27698)
         'url': 'https://www.raiplay.it/video/2020/12/Report---04-01-2021-2e90f1de-8eee-4de4-ac0e-78d21db5b600.html',
         'only_matching': True,
+    }, {
+        # DRM protected
+        'url': 'https://www.raiplay.it/video/2020/09/Lo-straordinario-mondo-di-Zoey-S1E1-Lo-straordinario-potere-di-Zoey-ed493918-1d32-44b7-8454-862e473d00ff.html',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -166,6 +170,14 @@ class RaiPlayIE(RaiBaseIE):
         media = self._download_json(
             base + '.json', video_id, 'Downloading video JSON')
 
+        if not self.params.get('allow_unplayable_formats'):
+            if try_get(
+                    media,
+                    (lambda x: x['rights_management']['rights']['drm'],
+                     lambda x: x['program_info']['rights_management']['rights']['drm']),
+                    dict):
+                raise ExtractorError('This video is DRM protected.', expected=True)
+
         title = media['name']
         video = media['video']
 

yt_dlp/extractor/trovo.py

@@ -14,6 +14,7 @@ from ..utils import (
 
 class TrovoBaseIE(InfoExtractor):
     _VALID_URL_BASE = r'https?://(?:www\.)?trovo\.live/'
+    _HEADERS = {'Origin': 'https://trovo.live'}
 
     def _extract_streamer_info(self, data):
         streamer_info = data.get('streamerInfo') or {}
@@ -68,6 +69,7 @@ class TrovoIE(TrovoBaseIE):
                 'format_id': format_id,
                 'height': int_or_none(format_id[:-1]) if format_id else None,
                 'url': play_url,
+                'http_headers': self._HEADERS,
             })
         self._sort_formats(formats)
 
@@ -153,6 +155,7 @@ class TrovoVodIE(TrovoBaseIE):
                 'protocol': 'm3u8_native',
                 'tbr': int_or_none(play_info.get('bitrate')),
                 'url': play_url,
+                'http_headers': self._HEADERS,
             })
         self._sort_formats(formats)
 

yt_dlp/extractor/vimeo.py

@@ -498,6 +498,24 @@ class VimeoIE(VimeoBaseInfoExtractor):
             'url': 'https://vimeo.com/album/2632481/video/79010983',
             'only_matching': True,
         },
+        {
+            'url': 'https://vimeo.com/showcase/3253534/video/119195465',
+            'note': 'A video in a password protected album (showcase)',
+            'info_dict': {
+                'id': '119195465',
+                'ext': 'mp4',
+                'title': 'youtube-dl test video \'ä"BaW_jenozKc',
+                'uploader': 'Philipp Hagemeister',
+                'uploader_id': 'user20132939',
+                'description': 'md5:fa7b6c6d8db0bdc353893df2f111855b',
+                'upload_date': '20150209',
+                'timestamp': 1423518307,
+            },
+            'params': {
+                'format': 'best[protocol=https]',
+                'videopassword': 'youtube-dl',
+            },
+        },
         {
             # source file returns 403: Forbidden
             'url': 'https://vimeo.com/7809605',
@@ -564,6 +582,44 @@ class VimeoIE(VimeoBaseInfoExtractor):
     def _real_initialize(self):
         self._login()
 
+    def _try_album_password(self, url):
+        album_id = self._search_regex(
+            r'vimeo\.com/(?:album|showcase)/([^/]+)', url, 'album id', default=None)
+        if not album_id:
+            return
+        viewer = self._download_json(
+            'https://vimeo.com/_rv/viewer', album_id, fatal=False)
+        if not viewer:
+            webpage = self._download_webpage(url, album_id)
+            viewer = self._parse_json(self._search_regex(
+                r'bootstrap_data\s*=\s*({.+?})</script>',
+                webpage, 'bootstrap data'), album_id)['viewer']
+        jwt = viewer['jwt']
+        album = self._download_json(
+            'https://api.vimeo.com/albums/' + album_id,
+            album_id, headers={'Authorization': 'jwt ' + jwt},
+            query={'fields': 'description,name,privacy'})
+        if try_get(album, lambda x: x['privacy']['view']) == 'password':
+            password = self._downloader.params.get('videopassword')
+            if not password:
+                raise ExtractorError(
+                    'This album is protected by a password, use the --video-password option',
+                    expected=True)
+            self._set_vimeo_cookie('vuid', viewer['vuid'])
+            try:
+                self._download_json(
+                    'https://vimeo.com/showcase/%s/auth' % album_id,
+                    album_id, 'Verifying the password', data=urlencode_postdata({
+                        'password': password,
+                        'token': viewer['xsrft'],
+                    }), headers={
+                        'X-Requested-With': 'XMLHttpRequest',
+                    })
+            except ExtractorError as e:
+                if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
+                    raise ExtractorError('Wrong password', expected=True)
+                raise
+
     def _real_extract(self, url):
         url, data = unsmuggle_url(url, {})
         headers = std_headers.copy()
@@ -591,6 +647,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
         elif any(p in url for p in ('play_redirect_hls', 'moogaloop.swf')):
             url = 'https://vimeo.com/' + video_id
 
+        self._try_album_password(url)
         try:
             # Retrieve video webpage to extract further information
             webpage, urlh = self._download_webpage_handle(

yt_dlp/extractor/youtube.py

@@ -26,6 +26,7 @@ from ..compat import (
 from ..jsinterp import JSInterpreter
 from ..utils import (
     clean_html,
+    dict_get,
     ExtractorError,
     format_field,
     float_or_none,
@@ -59,9 +60,9 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
     _TFA_URL = 'https://accounts.google.com/_/signin/challenge?hl=en&TL={0}'
 
     _RESERVED_NAMES = (
-        r'embed|e|watch_popup|channel|c|user|playlist|watch|w|v|movies|results|shared|hashtag|'
+        r'channel|c|user|playlist|watch|w|v|embed|e|watch_popup|'
-        r'storefront|oops|index|account|reporthistory|t/terms|about|upload|signin|logout|'
+        r'movies|results|shared|hashtag|trending|feed|feeds|'
-        r'feed/(?:watch_later|history|subscriptions|library|trending|recommended)')
+        r'storefront|oops|index|account|reporthistory|t/terms|about|upload|signin|logout')
 
     _NETRC_MACHINE = 'youtube'
     # If True it will raise an error if no login info is provided
@@ -301,7 +302,6 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
         auth = self._generate_sapisidhash_header()
         if auth is not None:
             headers.update({'Authorization': auth, 'X-Origin': 'https://www.youtube.com'})
-
         return self._download_json(
             'https://www.youtube.com/youtubei/v1/%s' % ep,
             video_id=video_id, fatal=fatal, note=note, errnote=errnote,
@@ -2520,17 +2520,22 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             channel_url, 'channel id')
 
     @staticmethod
-    def _extract_grid_item_renderer(item):
+    def _extract_basic_item_renderer(item):
-        for item_kind in ('Playlist', 'Video', 'Channel'):
+        # Modified from _extract_grid_item_renderer
-            renderer = item.get('grid%sRenderer' % item_kind)
+        known_renderers = (
-            if renderer:
+            'playlistRenderer', 'videoRenderer', 'channelRenderer'
-                return renderer
+            'gridPlaylistRenderer', 'gridVideoRenderer', 'gridChannelRenderer'
+        )
+        for key, renderer in item.items():
+            if key not in known_renderers:
+                continue
+            return renderer
 
     def _grid_entries(self, grid_renderer):
         for item in grid_renderer['items']:
             if not isinstance(item, dict):
                 continue
-            renderer = self._extract_grid_item_renderer(item)
+            renderer = self._extract_basic_item_renderer(item)
             if not isinstance(renderer, dict):
                 continue
             title = try_get(
@@ -2559,7 +2564,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
         content = shelf_renderer.get('content')
         if not isinstance(content, dict):
             return
-        renderer = content.get('gridRenderer')
+        renderer = content.get('gridRenderer') or content.get('expandedShelfContentsRenderer')
         if renderer:
             # TODO: add support for nested playlists so each shelf is processed
             # as separate playlist
@@ -2601,20 +2606,6 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 continue
             yield self._extract_video(renderer)
 
-    r""" # Not needed in the new implementation
-    def _itemSection_entries(self, item_sect_renderer):
-        for content in item_sect_renderer['contents']:
-            if not isinstance(content, dict):
-                continue
-            renderer = content.get('videoRenderer', {})
-            if not isinstance(renderer, dict):
-                continue
-            video_id = renderer.get('videoId')
-            if not video_id:
-                continue
-            yield self._extract_video(renderer)
-    """
-
     def _rich_entries(self, rich_grid_renderer):
         renderer = try_get(
             rich_grid_renderer, lambda x: x['content']['videoRenderer'], dict) or {}
@@ -2713,7 +2704,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             ctp = continuation_ep.get('clickTrackingParams')
             return YoutubeTabIE._build_continuation_query(continuation, ctp)
 
-    def _entries(self, tab, identity_token, item_id):
+    def _entries(self, tab, item_id, identity_token, account_syncid):
 
         def extract_entries(parent_renderer):  # this needs to called again for continuation to work with feeds
             contents = try_get(parent_renderer, lambda x: x['contents'], list) or []
@@ -2773,6 +2764,10 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
         if identity_token:
             headers['x-youtube-identity-token'] = identity_token
 
+        if account_syncid:
+            headers['X-Goog-PageId'] = account_syncid
+            headers['X-Goog-AuthUser'] = 0
+
         for page_num in itertools.count(1):
             if not continuation:
                 break
@@ -2803,9 +2798,14 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 else:
                     # Youtube sometimes sends incomplete data
                     # See: https://github.com/ytdl-org/youtube-dl/issues/28194
-                    if response.get('continuationContents') or response.get('onResponseReceivedActions'):
+                    if dict_get(response,
+                                ('continuationContents', 'onResponseReceivedActions', 'onResponseReceivedEndpoints')):
                         break
-                    last_error = 'Incomplete data recieved'
+
+                    # Youtube may send alerts if there was an issue with the continuation page
+                    self._extract_alerts(response, expected=False)
+
+                    last_error = 'Incomplete data received'
                     if count >= retries:
                         self._downloader.report_error(last_error)
 
@@ -2837,11 +2837,13 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 'gridPlaylistRenderer': (self._grid_entries, 'items'),
                 'gridVideoRenderer': (self._grid_entries, 'items'),
                 'playlistVideoRenderer': (self._playlist_entries, 'contents'),
-                'itemSectionRenderer': (self._playlist_entries, 'contents'),
+                'itemSectionRenderer': (extract_entries, 'contents'),  # for feeds
                 'richItemRenderer': (extract_entries, 'contents'),  # for hashtag
+                'backstagePostThreadRenderer': (self._post_thread_continuation_entries, 'contents')
             }
             continuation_items = try_get(
-                response, lambda x: x['onResponseReceivedActions'][0]['appendContinuationItemsAction']['continuationItems'], list)
+                response,
+                lambda x: dict_get(x, ('onResponseReceivedActions', 'onResponseReceivedEndpoints'))[0]['appendContinuationItemsAction']['continuationItems'], list)
             continuation_item = try_get(continuation_items, lambda x: x[0], dict) or {}
             video_items_renderer = None
             for key, value in continuation_item.items():
@@ -2888,7 +2890,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                         try_get(owner, lambda x: x['navigationEndpoint']['browseEndpoint']['canonicalBaseUrl'], compat_str))
         return {k: v for k, v in uploader.items() if v is not None}
 
-    def _extract_from_tabs(self, item_id, webpage, data, tabs, identity_token):
+    def _extract_from_tabs(self, item_id, webpage, data, tabs):
         playlist_id = title = description = channel_url = channel_name = channel_id = None
         thumbnails_list = tags = []
 
@@ -2952,16 +2954,41 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'channel_id': metadata['uploader_id'],
             'channel_url': metadata['uploader_url']})
         return self.playlist_result(
-            self._entries(selected_tab, identity_token, playlist_id),
+            self._entries(
+                selected_tab, playlist_id,
+                self._extract_identity_token(webpage, item_id),
+                self._extract_account_syncid(data)),
             **metadata)
 
+    def _extract_mix_playlist(self, playlist, playlist_id):
+        first_id = last_id = None
+        for page_num in itertools.count(1):
+            videos = list(self._playlist_entries(playlist))
+            if not videos:
+                return
+            start = next((i for i, v in enumerate(videos) if v['id'] == last_id), -1) + 1
+            if start >= len(videos):
+                return
+            for video in videos[start:]:
+                if video['id'] == first_id:
+                    self.to_screen('First video %s found again; Assuming end of Mix' % first_id)
+                    return
+                yield video
+            first_id = first_id or videos[0]['id']
+            last_id = videos[-1]['id']
+
+            _, data = self._extract_webpage(
+                'https://www.youtube.com/watch?list=%s&v=%s' % (playlist_id, last_id),
+                '%s page %d' % (playlist_id, page_num))
+            playlist = try_get(
+                data, lambda x: x['contents']['twoColumnWatchNextResults']['playlist']['playlist'], dict)
+
     def _extract_from_playlist(self, item_id, url, data, playlist):
         title = playlist.get('title') or try_get(
             data, lambda x: x['titleText']['simpleText'], compat_str)
         playlist_id = playlist.get('playlistId') or item_id
-        # Inline playlist rendition continuation does not always work
+
-        # at Youtube side, so delegating regular tab-based playlist URL
+        # Delegating everything except mix playlists to regular tab-based playlist URL
-        # processing whenever possible.
         playlist_url = urljoin(url, try_get(
             playlist, lambda x: x['endpoint']['commandMetadata']['webCommandMetadata']['url'],
             compat_str))
@@ -2969,27 +2996,40 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             return self.url_result(
                 playlist_url, ie=YoutubeTabIE.ie_key(), video_id=playlist_id,
                 video_title=title)
-        return self.playlist_result(
-            self._playlist_entries(playlist), playlist_id=playlist_id,
-            playlist_title=title)
 
-    @staticmethod
+        return self.playlist_result(
-    def _extract_alerts(data):
+            self._extract_mix_playlist(playlist, playlist_id),
-        for alert_dict in try_get(data, lambda x: x['alerts'], list) or []:
+            playlist_id=playlist_id, playlist_title=title)
-            if not isinstance(alert_dict, dict):
+
-                continue
+    def _extract_alerts(self, data, expected=False):
-            for renderer in alert_dict:
+
-                alert = alert_dict[renderer]
+        def _real_extract_alerts():
-                alert_type = alert.get('type')
+            for alert_dict in try_get(data, lambda x: x['alerts'], list) or []:
-                if not alert_type:
+                if not isinstance(alert_dict, dict):
                     continue
-                message = try_get(alert, lambda x: x['text']['simpleText'], compat_str)
+                for alert in alert_dict.values():
-                if message:
+                    alert_type = alert.get('type')
-                    yield alert_type, message
+                    if not alert_type:
-                for run in try_get(alert, lambda x: x['text']['runs'], list) or []:
+                        continue
-                    message = try_get(run, lambda x: x['text'], compat_str)
+                    message = try_get(alert, lambda x: x['text']['simpleText'], compat_str)
                     if message:
                         yield alert_type, message
+                    for run in try_get(alert, lambda x: x['text']['runs'], list) or []:
+                        message = try_get(run, lambda x: x['text'], compat_str)
+                        if message:
+                            yield alert_type, message
+
+        err_msg = None
+        for alert_type, alert_message in _real_extract_alerts():
+            if alert_type.lower() == 'error':
+                if err_msg:
+                    self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg))
+                err_msg = alert_message
+            else:
+                self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+
+        if err_msg:
+            raise ExtractorError('YouTube said: %s' % err_msg, expected=expected)
 
     def _extract_identity_token(self, webpage, item_id):
         ytcfg = self._extract_ytcfg(item_id, webpage)
@@ -3001,35 +3041,18 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             r'\bID_TOKEN["\']\s*:\s*["\'](.+?)["\']', webpage,
             'identity token', default=None)
 
-    def _real_extract(self, url):
+    @staticmethod
-        item_id = self._match_id(url)
+    def _extract_account_syncid(data):
-        url = compat_urlparse.urlunparse(
+        """Extract syncId required to download private playlists of secondary channels"""
-            compat_urlparse.urlparse(url)._replace(netloc='www.youtube.com'))
+        sync_ids = (
-        is_home = re.match(r'(?P<pre>%s)(?P<post>/?(?![^#?]).*$)' % self._VALID_URL, url)
+            try_get(data, lambda x: x['responseContext']['mainAppWebResponseContext']['datasyncId'], compat_str)
-        if is_home is not None and is_home.group('not_channel') is None and item_id != 'feed':
+            or '').split("||")
-            self._downloader.report_warning(
+        if len(sync_ids) >= 2 and sync_ids[1]:
-                'A channel/user page was given. All the channel\'s videos will be downloaded. '
+            # datasyncid is of the form "channel_syncid||user_syncid" for secondary channel
-                'To download only the videos in the home page, add a "/featured" to the URL')
+            # and just "user_syncid||" for primary channel. We only want the channel_syncid
-            url = '%s/videos%s' % (is_home.group('pre'), is_home.group('post') or '')
+            return sync_ids[0]
-
-        # Handle both video/playlist URLs
-        qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
-        video_id = qs.get('v', [None])[0]
-        playlist_id = qs.get('list', [None])[0]
-
-        if is_home is not None and is_home.group('not_channel') is not None and is_home.group('not_channel').startswith('watch') and not video_id:
-            if playlist_id:
-                self._downloader.report_warning('%s is not a valid Youtube URL. Trying to download playlist %s' % (url, playlist_id))
-                url = 'https://www.youtube.com/playlist?list=%s' % playlist_id
-                # return self.url_result(playlist_id, ie=YoutubePlaylistIE.ie_key())
-            else:
-                raise ExtractorError('Unable to recognize tab page')
-        if video_id and playlist_id:
-            if self._downloader.params.get('noplaylist'):
-                self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
-                return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id)
-            self.to_screen('Downloading playlist %s - add --no-playlist to just download video %s' % (playlist_id, video_id))
 
+    def _extract_webpage(self, url, item_id):
         retries = self._downloader.params.get('extractor_retries', 3)
         count = -1
         last_error = 'Incomplete yt initial data recieved'
@@ -3041,40 +3064,67 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 self.report_warning('%s. Retrying ...' % last_error)
             webpage = self._download_webpage(
                 url, item_id,
-                'Downloading webpage%s' % ' (retry #%d)' % count if count else '')
+                'Downloading webpage%s' % (' (retry #%d)' % count if count else ''))
-            identity_token = self._extract_identity_token(webpage, item_id)
             data = self._extract_yt_initial_data(item_id, webpage)
-            err_msg = None
+            self._extract_alerts(data, expected=True)
-            for alert_type, alert_message in self._extract_alerts(data):
-                if alert_type.lower() == 'error':
-                    if err_msg:
-                        self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg))
-                    err_msg = alert_message
-                else:
-                    self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
-            if err_msg:
-                raise ExtractorError('YouTube said: %s' % err_msg, expected=True)
             if data.get('contents') or data.get('currentVideoEndpoint'):
                 break
             if count >= retries:
                 self._downloader.report_error(last_error)
+        return webpage, data
+
+    def _real_extract(self, url):
+        item_id = self._match_id(url)
+        url = compat_urlparse.urlunparse(
+            compat_urlparse.urlparse(url)._replace(netloc='www.youtube.com'))
+
+        # This is not matched in a channel page with a tab selected
+        mobj = re.match(r'(?P<pre>%s)(?P<post>/?(?![^#?]).*$)' % self._VALID_URL, url)
+        mobj = mobj.groupdict() if mobj else {}
+        if mobj and not mobj.get('not_channel'):
+            self._downloader.report_warning(
+                'A channel/user page was given. All the channel\'s videos will be downloaded. '
+                'To download only the videos in the home page, add a "/featured" to the URL')
+            url = '%s/videos%s' % (mobj.get('pre'), mobj.get('post') or '')
+
+        # Handle both video/playlist URLs
+        qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
+        video_id = qs.get('v', [None])[0]
+        playlist_id = qs.get('list', [None])[0]
+
+        if not video_id and (mobj.get('not_channel') or '').startswith('watch'):
+            if not playlist_id:
+                # If there is neither video or playlist ids,
+                # youtube redirects to home page, which is undesirable
+                raise ExtractorError('Unable to recognize tab page')
+            self._downloader.report_warning('A video URL was given without video ID. Trying to download playlist %s' % playlist_id)
+            url = 'https://www.youtube.com/playlist?list=%s' % playlist_id
+
+        if video_id and playlist_id:
+            if self._downloader.params.get('noplaylist'):
+                self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
+                return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id)
+            self.to_screen('Downloading playlist %s; add --no-playlist to just download video %s' % (playlist_id, video_id))
+
+        webpage, data = self._extract_webpage(url, item_id)
 
         tabs = try_get(
             data, lambda x: x['contents']['twoColumnBrowseResultsRenderer']['tabs'], list)
         if tabs:
-            return self._extract_from_tabs(item_id, webpage, data, tabs, identity_token)
+            return self._extract_from_tabs(item_id, webpage, data, tabs)
+
         playlist = try_get(
             data, lambda x: x['contents']['twoColumnWatchNextResults']['playlist']['playlist'], dict)
         if playlist:
             return self._extract_from_playlist(item_id, url, data, playlist)
-        # Fallback to video extraction if no playlist alike page is recognized.
+
-        # First check for the current video then try the v attribute of URL query.
         video_id = try_get(
             data, lambda x: x['currentVideoEndpoint']['watchEndpoint']['videoId'],
             compat_str) or video_id
         if video_id:
+            self._downloader.report_warning('Unable to recognize playlist. Downloading just video %s' % video_id)
             return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id)
-        # Failed to recognize
+
         raise ExtractorError('Unable to recognize tab page')
 
 
@@ -3338,7 +3388,6 @@ class YoutubeFeedsInfoExtractor(YoutubeTabIE):
     Subclasses must define the _FEED_NAME property.
     """
     _LOGIN_REQUIRED = True
-    # _MAX_PAGES = 5
     _TESTS = []
 
     @property

yt_dlp/options.py

@@ -1154,7 +1154,7 @@ def parseOpts(overrideArguments=None):
         help='Write metadata to the video file\'s xattrs (using dublin core and xdg standards)')
     postproc.add_option(
         '--fixup',
-        metavar='POLICY', dest='fixup', default='detect_or_warn',
+        metavar='POLICY', dest='fixup', default=None,
         help=(
             'Automatically correct known faults of the file. '
             'One of never (do nothing), warn (only emit a warning), '

yt_dlp/update.py

@@ -49,12 +49,16 @@ def update_self(to_screen, verbose, opener):
                 h.update(mv[:n])
         return h.hexdigest()
 
-    to_screen('Current Build Hash %s' % calc_sha256sum(sys.executable))
-
     if not isinstance(globals().get('__loader__'), zipimporter) and not hasattr(sys, 'frozen'):
         to_screen('It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update.')
         return
 
+    # sys.executable is set to the full pathname of the exe-file for py2exe
+    # though symlinks are not followed so that we need to do this manually
+    # with help of realpath
+    filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0])
+    to_screen('Current Build Hash %s' % calc_sha256sum(filename))
+
     # Download and check versions info
     try:
         version_info = opener.open(JSON_URL).read().decode('utf-8')
@@ -103,11 +107,6 @@ def update_self(to_screen, verbose, opener):
             (i[1] for i in hashes if i[0] == 'yt-dlp%s' % label),
             None)
 
-    # sys.executable is set to the full pathname of the exe-file for py2exe
-    # though symlinks are not followed so that we need to do this manually
-    # with help of realpath
-    filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0])
-
     if not os.access(filename, os.W_OK):
         to_screen('ERROR: no write permissions on %s' % filename)
         return
@@ -198,28 +197,18 @@ def update_self(to_screen, verbose, opener):
             to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
             return
 
+        expected_sum = get_sha256sum('zip', py_ver)
+        if expected_sum and hashlib.sha256(newcontent).hexdigest() != expected_sum:
+            to_screen('ERROR: unable to verify the new zip')
+            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
+            return
+
         try:
-            with open(filename + '.new', 'wb') as outf:
+            with open(filename, 'wb') as outf:
                 outf.write(newcontent)
         except (IOError, OSError):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
-            to_screen('ERROR: unable to write the new version')
-            return
-
-        expected_sum = get_sha256sum('zip', py_ver)
-        if expected_sum and calc_sha256sum(filename + '.new') != expected_sum:
-            to_screen('ERROR: unable to verify the new zip')
-            to_screen('Visit https://github.com/yt-dlp/yt-dlp/releases/latest')
-            try:
-                os.remove(filename + '.new')
-            except OSError:
-                to_screen('ERROR: unable to remove corrupt zip')
-            return
-
-        try:
-            os.rename(filename + '.new', filename)
-        except OSError:
             to_screen('ERROR: unable to overwrite current version')
             return
 

yt_dlp/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.03.03.1'
+__version__ = '2021.03.03.2'