minor logging improvements

Authored by: doe1080

CONTRIBUTORS

@@ -758,3 +758,5 @@ somini
 thedenv
 vallovic
 arabcoders
+mireq
+mlabeeb03

Changelog.md

@@ -4,6 +4,42 @@
 # To create a release, dispatch the https://github.com/yt-dlp/yt-dlp/actions/workflows/release.yml workflow on master
 -->
 
+### 2025.03.31
+
+#### Core changes
+- [Add `--compat-options 2024`](https://github.com/yt-dlp/yt-dlp/commit/22e34adbd741e1c7072015debd615dc3fb71c401) ([#12789](https://github.com/yt-dlp/yt-dlp/issues/12789)) by [seproDev](https://github.com/seproDev)
+
+#### Extractor changes
+- **francaisfacile**: [Add extractor](https://github.com/yt-dlp/yt-dlp/commit/bb321cfdc3fd4400598ddb12a15862bc2ac8fc10) ([#12787](https://github.com/yt-dlp/yt-dlp/issues/12787)) by [mlabeeb03](https://github.com/mlabeeb03)
+- **generic**: [Validate response before checking m3u8 live status](https://github.com/yt-dlp/yt-dlp/commit/9a1ec1d36e172d252714cef712a6d091e0a0c4f2) ([#12784](https://github.com/yt-dlp/yt-dlp/issues/12784)) by [bashonly](https://github.com/bashonly)
+- **microsoftlearnepisode**: [Extract more formats](https://github.com/yt-dlp/yt-dlp/commit/d63696f23a341ee36a3237ccb5d5e14b34c2c579) ([#12799](https://github.com/yt-dlp/yt-dlp/issues/12799)) by [bashonly](https://github.com/bashonly)
+- **mlbtv**: [Fix radio-only extraction](https://github.com/yt-dlp/yt-dlp/commit/f033d86b96b36f8c5289dd7c3304f42d4d9f6ff4) ([#12792](https://github.com/yt-dlp/yt-dlp/issues/12792)) by [bashonly](https://github.com/bashonly)
+- **on24**: [Support `mainEvent` URLs](https://github.com/yt-dlp/yt-dlp/commit/e465b078ead75472fcb7b86f6ccaf2b5d3bc4c21) ([#12800](https://github.com/yt-dlp/yt-dlp/issues/12800)) by [bashonly](https://github.com/bashonly)
+- **sbs**: [Fix subtitles extraction](https://github.com/yt-dlp/yt-dlp/commit/29560359120f28adaaac67c86fa8442eb72daa0d) ([#12785](https://github.com/yt-dlp/yt-dlp/issues/12785)) by [bashonly](https://github.com/bashonly)
+- **stvr**: [Rename extractor from RTVS to STVR](https://github.com/yt-dlp/yt-dlp/commit/5fc521cbd0ce7b2410d0935369558838728e205d) ([#12788](https://github.com/yt-dlp/yt-dlp/issues/12788)) by [mireq](https://github.com/mireq)
+- **twitch**: clips: [Extract portrait formats](https://github.com/yt-dlp/yt-dlp/commit/61046c31612b30c749cbdae934b7fe26abe659d7) ([#12763](https://github.com/yt-dlp/yt-dlp/issues/12763)) by [DmitryScaletta](https://github.com/DmitryScaletta)
+- **youtube**
+    - [Add `player_js_variant` extractor-arg](https://github.com/yt-dlp/yt-dlp/commit/07f04005e40ebdb368920c511e36e98af0077ed3) ([#12767](https://github.com/yt-dlp/yt-dlp/issues/12767)) by [bashonly](https://github.com/bashonly)
+    - tab: [Fix playlist continuation extraction](https://github.com/yt-dlp/yt-dlp/commit/6a6d97b2cbc78f818de05cc96edcdcfd52caa259) ([#12777](https://github.com/yt-dlp/yt-dlp/issues/12777)) by [coletdjnz](https://github.com/coletdjnz)
+
+#### Misc. changes
+- **cleanup**: Miscellaneous: [5e457af](https://github.com/yt-dlp/yt-dlp/commit/5e457af57fae9645b1b8fa0ed689229c8fb9656b) by [bashonly](https://github.com/bashonly)
+
+### 2025.03.27
+
+#### Core changes
+- **jsinterp**: [Fix nested attributes and object extraction](https://github.com/yt-dlp/yt-dlp/commit/a8b9ff3c2a0ae25735e580173becc78545b92572) ([#12760](https://github.com/yt-dlp/yt-dlp/issues/12760)) by [bashonly](https://github.com/bashonly), [seproDev](https://github.com/seproDev)
+
+#### Extractor changes
+- **youtube**: [Make signature and nsig extraction more robust](https://github.com/yt-dlp/yt-dlp/commit/48be862b32648bff5b3e553e40fca4dcc6e88b28) ([#12761](https://github.com/yt-dlp/yt-dlp/issues/12761)) by [bashonly](https://github.com/bashonly), [seproDev](https://github.com/seproDev)
+
+### 2025.03.26
+
+#### Extractor changes
+- **youtube**
+    - [Fix signature and nsig extraction for player `4fcd6e4a`](https://github.com/yt-dlp/yt-dlp/commit/a550dfc904a02843a26369ae50dbb7c0febfb30e) ([#12748](https://github.com/yt-dlp/yt-dlp/issues/12748)) by [seproDev](https://github.com/seproDev)
+    - [Only cache nsig code on successful decoding](https://github.com/yt-dlp/yt-dlp/commit/ecee97b4fa90d51c48f9154c3a6d5a8ffe46cd5c) ([#12750](https://github.com/yt-dlp/yt-dlp/issues/12750)) by [bashonly](https://github.com/bashonly), [seproDev](https://github.com/seproDev)
+
 ### 2025.03.25
 
 #### Core changes

README.md

@@ -1770,7 +1770,7 @@ The following extractors use this feature:
 * `lang`: Prefer translated metadata (`title`, `description` etc) of this language code (case-sensitive). By default, the video primary language metadata is preferred, with a fallback to `en` translated. See [youtube.py](https://github.com/yt-dlp/yt-dlp/blob/c26f9b991a0681fd3ea548d535919cec1fbbd430/yt_dlp/extractor/youtube.py#L381-L390) for list of supported content language codes
 * `skip`: One or more of `hls`, `dash` or `translated_subs` to skip extraction of the m3u8 manifests, dash manifests and [auto-translated subtitles](https://github.com/yt-dlp/yt-dlp/issues/4090#issuecomment-1158102032) respectively
 * `player_client`: Clients to extract video data from. The currently available clients are `web`, `web_safari`, `web_embedded`, `web_music`, `web_creator`, `mweb`, `ios`, `android`, `android_vr`, `tv` and `tv_embedded`. By default, `tv,ios,web` is used, or `tv,web` is used when authenticating with cookies. The `web_music` client is added for `music.youtube.com` URLs when logged-in cookies are used. The `tv_embedded` and `web_creator` clients are added for age-restricted videos if account age-verification is required. Some clients, such as `web` and `web_music`, require a `po_token` for their formats to be downloadable. Some clients, such as `web_creator`, will only work with authentication. Not all clients support authentication via cookies. You can use `default` for the default clients, or you can use `all` for all clients (not recommended). You can prefix a client with `-` to exclude it, e.g. `youtube:player_client=default,-ios`
-* `player_skip`: Skip some network requests that are generally needed for robust extraction. One or more of `configs` (skip client configs), `webpage` (skip initial webpage), `js` (skip js player). While these options can help reduce the number of requests needed or avoid some rate-limiting, they could cause some issues. See [#860](https://github.com/yt-dlp/yt-dlp/pull/860) for more details
+* `player_skip`: Skip some network requests that are generally needed for robust extraction. One or more of `configs` (skip client configs), `webpage` (skip initial webpage), `js` (skip js player), `initial_data` (skip initial data/next ep request). While these options can help reduce the number of requests needed or avoid some rate-limiting, they could cause issues such as missing formats or metadata.  See [#860](https://github.com/yt-dlp/yt-dlp/pull/860) and [#12826](https://github.com/yt-dlp/yt-dlp/issues/12826) for more details
 * `player_params`: YouTube player parameters to use for player requests. Will overwrite any default ones set by yt-dlp.
 * `comment_sort`: `top` or `new` (default) - choose comment sorting mode (on YouTube's side)
 * `max_comments`: Limit the amount of comments to gather. Comma-separated list of integers representing `max-comments,max-parents,max-replies,max-replies-per-thread`. Default is `all,all,all,all`
@@ -1782,6 +1782,15 @@ The following extractors use this feature:
 * `data_sync_id`: Overrides the account Data Sync ID used in Innertube API requests. This may be needed if you are using an account with `youtube:player_skip=webpage,configs` or `youtubetab:skip=webpage`
 * `visitor_data`: Overrides the Visitor Data used in Innertube API requests. This should be used with `player_skip=webpage,configs` and without cookies. Note: this may have adverse effects if used improperly. If a session from a browser is wanted, you should pass cookies instead (which contain the Visitor ID)
 * `po_token`:  Proof of Origin (PO) Token(s) to use. Comma seperated list of PO Tokens in the format `CLIENT.CONTEXT+PO_TOKEN`, e.g. `youtube:po_token=web.gvs+XXX,web.player=XXX,web_safari.gvs+YYY`. Context can be either `gvs` (Google Video Server URLs) or `player` (Innertube player request)
+* `player_js_variant`: The player javascript variant to use for signature and nsig deciphering. The known variants are: `main`, `tce`, `tv`, `tv_es6`, `phone`, `tablet`. Only `main` is recommended as a possible workaround; the others are for debugging purposes. The default is to use what is prescribed by the site, and can be selected with `actual`
+
+##### PO Token settings
+* `po_token`:  Proof of Origin (PO) Token(s) to use. Comma seperated list of PO Tokens in the format `CLIENT.CONTEXT+PO_TOKEN`, e.g. `youtube:po_token=web.gvs+XXX,web.player=XXX,web_safari.gvs+YYY`. Context can be either `gvs` (Google Video Server URLs) or `player` (Innertube player request)
+* `pot_log_level`: PO Token provider log level. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`. Default is `DEBUG` if `-v` is used, otherwise `INFO`
+* `fetch_pot`: Policy to use for fetching a PO Token from providers. `always` to always try fetch a PO Token regardless if the client requires one for the given context. `when_required` to only fetch a PO Token if the client requires one for the given context (default)
+
+###### youtubepot-webpo
+* `bind_to_visitor_id`: Whether to use the Visitor ID instead of Visitor Data for caching WebPO tokens. Either `true` or `false` (default `true`)
 
 #### youtubetab (YouTube playlists, channels, feeds, etc.)
 * `skip`: One or more of `webpage` (skip initial webpage download), `authcheck` (allow the download of playlists requiring authentication when no initial webpage is downloaded. This may cause unwanted behavior, see [#1122](https://github.com/yt-dlp/yt-dlp/pull/1122) for more details)
@@ -2218,7 +2227,7 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * Live chats (if available) are considered as subtitles. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent any live chat/danmaku from downloading
 * YouTube channel URLs download all uploads of the channel. To download only the videos in a specific tab, pass the tab's URL. If the channel does not show the requested tab, an error will be raised. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections
 * Unavailable videos are also listed for YouTube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
-* The upload dates extracted from YouTube are in UTC [when available](https://github.com/yt-dlp/yt-dlp/blob/89e4d86171c7b7c997c77d4714542e0383bf0db0/yt_dlp/extractor/youtube.py#L3898-L3900). Use `--compat-options no-youtube-prefer-utc-upload-date` to prefer the non-UTC upload date.
+* The upload dates extracted from YouTube are in UTC.
 * If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this
 * Thumbnail embedding in `mp4` is done with mutagen if possible. Use `--compat-options embed-thumbnail-atomicparsley` to force the use of AtomicParsley instead
 * Some internal metadata such as filenames are removed by default from the infojson. Use `--no-clean-infojson` or `--compat-options no-clean-infojson` to revert this
@@ -2237,9 +2246,10 @@ For ease of use, a few more compat options are available:
 * `--compat-options all`: Use all compat options (**Do NOT use this!**)
 * `--compat-options youtube-dl`: Same as `--compat-options all,-multistreams,-playlist-match-filter,-manifest-filesize-approx,-allow-unsafe-ext,-prefer-vp9-sort`
 * `--compat-options youtube-dlc`: Same as `--compat-options all,-no-live-chat,-no-youtube-channel-redirect,-playlist-match-filter,-manifest-filesize-approx,-allow-unsafe-ext,-prefer-vp9-sort`
-* `--compat-options 2021`: Same as `--compat-options 2022,no-certifi,filename-sanitization,no-youtube-prefer-utc-upload-date`
+* `--compat-options 2021`: Same as `--compat-options 2022,no-certifi,filename-sanitization`
 * `--compat-options 2022`: Same as `--compat-options 2023,playlist-match-filter,no-external-downloader-progress,prefer-legacy-http-handler,manifest-filesize-approx`
-* `--compat-options 2023`: Same as `--compat-options prefer-vp9-sort`. Use this to enable all future compat options
+* `--compat-options 2023`: Same as `--compat-options 2024,prefer-vp9-sort`
+* `--compat-options 2024`: Currently does nothing. Use this to enable all future compat options
 
 The following compat options restore vulnerable behavior from before security patches:
 

supportedsites.md

@@ -472,6 +472,7 @@ The only reliable way to check if a site is supported is to try it.
  - **FoxNewsVideo**
  - **FoxSports**
  - **fptplay**: fptplay.vn
+ - **FrancaisFacile**
  - **FranceCulture**
  - **FranceInter**
  - **francetv**
@@ -1251,7 +1252,6 @@ The only reliable way to check if a site is supported is to try it.
  - **rtve.es:infantil**: RTVE infantil
  - **rtve.es:live**: RTVE.es live streams
  - **rtve.es:television**
- - **RTVS**
  - **rtvslo.si**
  - **rtvslo.si:show**
  - **RudoVideo**
@@ -1407,6 +1407,7 @@ The only reliable way to check if a site is supported is to try it.
  - **StretchInternet**
  - **Stripchat**
  - **stv:player**
+ - **stvr**: Slovak Television and Radio (formerly RTVS)
  - **Subsplash**
  - **subsplash:playlist**
  - **Substack**

test/helper.py

@@ -136,7 +136,7 @@ def _iter_differences(got, expected, field):
             return
 
         if op == 'startswith':
-            if not val.startswith(got):
+            if not got.startswith(val):
                 yield field, f'should start with {val!r}, got {got!r}'
             return
 

test/test_YoutubeDL.py

@@ -1435,6 +1435,27 @@ class TestYoutubeDL(unittest.TestCase):
         FakeYDL().close()
         assert all_plugins_loaded.value
 
+    def test_close_hooks(self):
+        # Should call all registered close hooks on close
+        close_hook_called = False
+        close_hook_two_called = False
+
+        def close_hook():
+            nonlocal close_hook_called
+            close_hook_called = True
+
+        def close_hook_two():
+            nonlocal close_hook_two_called
+            close_hook_two_called = True
+
+        ydl = FakeYDL()
+        ydl.add_close_hook(close_hook)
+        ydl.add_close_hook(close_hook_two)
+
+        ydl.close()
+        self.assertTrue(close_hook_called, 'Close hook was not called')
+        self.assertTrue(close_hook_two_called, 'Close hook two was not called')
+
 
 if __name__ == '__main__':
     unittest.main()

test/test_jsinterp.py

@@ -118,6 +118,7 @@ class TestJSInterpreter(unittest.TestCase):
         self._test('function f(){var x = 20; x = 30 + 1; return x;}', 31)
         self._test('function f(){var x = 20; x += 30 + 1; return x;}', 51)
         self._test('function f(){var x = 20; x -= 30 + 1; return x;}', -11)
+        self._test('function f(){var x = 2; var y = ["a", "b"]; y[x%y["length"]]="z"; return y}', ['z', 'b'])
 
     @unittest.skip('Not implemented')
     def test_comments(self):
@@ -403,6 +404,8 @@ class TestJSInterpreter(unittest.TestCase):
         test_result = list('test')
         tests = [
             'function f(a, b){return a.split(b)}',
+            'function f(a, b){return a["split"](b)}',
+            'function f(a, b){let x = ["split"]; return a[x[0]](b)}',
             'function f(a, b){return String.prototype.split.call(a, b)}',
             'function f(a, b){return String.prototype.split.apply(a, [b])}',
         ]
@@ -441,6 +444,9 @@ class TestJSInterpreter(unittest.TestCase):
         self._test('function f(){return "012345678".slice(-1, 1)}', '')
         self._test('function f(){return "012345678".slice(-3, -1)}', '67')
 
+    def test_splice(self):
+        self._test('function f(){var T = ["0", "1", "2"]; T["splice"](2, 1, "0")[0]; return T }', ['0', '1', '0'])
+
     def test_js_number_to_string(self):
         for test, radix, expected in [
             (0, None, '0'),

test/test_networking.py

@@ -39,6 +39,7 @@ from yt_dlp.cookies import YoutubeDLCookieJar
 from yt_dlp.dependencies import brotli, curl_cffi, requests, urllib3
 from yt_dlp.networking import (
     HEADRequest,
+    PATCHRequest,
     PUTRequest,
     Request,
     RequestDirector,
@@ -1856,6 +1857,7 @@ class TestRequest:
 
     def test_request_helpers(self):
         assert HEADRequest('http://example.com').method == 'HEAD'
+        assert PATCHRequest('http://example.com').method == 'PATCH'
         assert PUTRequest('http://example.com').method == 'PUT'
 
     def test_headers(self):

test/test_networking_utils.py

@@ -20,7 +20,6 @@ from yt_dlp.networking._helper import (
     add_accept_encoding_header,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
     ssl_load_certs,
 )
 from yt_dlp.networking.exceptions import (
@@ -28,7 +27,7 @@ from yt_dlp.networking.exceptions import (
     IncompleteRead,
 )
 from yt_dlp.socks import ProxyType
-from yt_dlp.utils.networking import HTTPHeaderDict
+from yt_dlp.utils.networking import HTTPHeaderDict, select_proxy
 
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 

test/test_pot/conftest.py

@@ -0,0 +1,74 @@
+import pytest
+
+from yt_dlp import YoutubeDL
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.extractor.common import InfoExtractor
+from yt_dlp.extractor.youtube.pot._provider import IEContentProviderLogger
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest, PoTokenContext
+from yt_dlp.utils.networking import HTTPHeaderDict
+
+
+class MockLogger(IEContentProviderLogger):
+
+    log_level = IEContentProviderLogger.LogLevel.TRACE
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.messages = {}
+
+    def trace(self, message: str):
+        self.messages.setdefault('trace', []).append(message)
+        pass
+
+    def debug(self, message: str):
+        self.messages.setdefault('debug', []).append(message)
+        pass
+
+    def info(self, message: str):
+        self.messages.setdefault('info', []).append(message)
+        pass
+
+    def warning(self, message: str, *, once=False):
+        self.messages.setdefault('warning', []).append(message)
+        pass
+
+    def error(self, message: str):
+        self.messages.setdefault('error', []).append(message)
+        pass
+
+
+@pytest.fixture
+def ie() -> InfoExtractor:
+    ydl = YoutubeDL()
+    return ydl.get_info_extractor('Youtube')
+
+
+@pytest.fixture
+def logger() -> MockLogger:
+    return MockLogger()
+
+
+@pytest.fixture()
+def pot_request() -> PoTokenRequest:
+    return PoTokenRequest(
+        context=PoTokenContext.GVS,
+        innertube_context={'client': {'clientName': 'WEB'}},
+        innertube_host='youtube.com',
+        session_index=None,
+        player_url=None,
+        is_authenticated=False,
+        video_webpage=None,
+
+        visitor_data='example-visitor-data',
+        data_sync_id='example-data-sync-id',
+        video_id='example-video-id',
+
+        request_cookiejar=YoutubeDLCookieJar(),
+        request_proxy=None,
+        request_headers=HTTPHeaderDict(),
+        request_timeout=None,
+        request_source_address=None,
+        request_verify_tls=True,
+
+        bypass_cache=False,
+    )

test/test_pot/test_pot_builtin_memorycache.py

@@ -0,0 +1,117 @@
+import threading
+import time
+from collections import OrderedDict
+import pytest
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider, BuiltInIEContentProvider
+from yt_dlp.utils import bug_reports_message
+from yt_dlp.extractor.youtube.pot._builtin.memory_cache import MemoryLRUPCP, memorylru_preference, initialize_global_cache
+from yt_dlp.version import __version__
+from yt_dlp.extractor.youtube.pot._registry import _pot_cache_providers, _pot_memory_cache
+
+
+class TestMemoryLRUPCS:
+
+    def test_base_type(self):
+        assert issubclass(MemoryLRUPCP, IEContentProvider)
+        assert issubclass(MemoryLRUPCP, BuiltInIEContentProvider)
+
+    @pytest.fixture
+    def pcp(self, ie, logger) -> MemoryLRUPCP:
+        return MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), max_size))
+
+    def test_is_registered(self):
+        assert _pot_cache_providers.value.get('MemoryLRU') == MemoryLRUPCP
+
+    def test_initialization(self, pcp):
+        assert pcp.PROVIDER_NAME == 'memory'
+        assert pcp.PROVIDER_VERSION == __version__
+        assert pcp.BUG_REPORT_MESSAGE == bug_reports_message(before='')
+        assert pcp.is_available()
+
+    def test_store_and_get(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value1'
+        assert len(pcp.cache) == 1
+
+    def test_store_ignore_expired(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) - 1)
+        assert len(pcp.cache) == 0
+        assert pcp.get('key1') is None
+        assert len(pcp.cache) == 0
+
+    def test_store_override_existing_key(self, ie, logger):
+        MAX_SIZE = 2
+        pcp = MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), MAX_SIZE))
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        pcp.store('key2', 'value2', int(time.time()) + 60)
+        assert len(pcp.cache) == 2
+        pcp.store('key1', 'value2', int(time.time()) + 60)
+        # Ensure that the override key gets added to the end of the cache instead of in the same position
+        pcp.store('key3', 'value3', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value2'
+
+    def test_store_ignore_expired_existing_key(self, pcp):
+        pcp.store('key1', 'value2', int(time.time()) + 60)
+        pcp.store('key1', 'value1', int(time.time()) - 1)
+        assert len(pcp.cache) == 1
+        assert pcp.get('key1') == 'value2'
+        assert len(pcp.cache) == 1
+
+    def test_get_key_expired(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value1'
+        assert len(pcp.cache) == 1
+        pcp.cache['key1'] = ('value1', int(time.time()) - 1)
+        assert pcp.get('key1') is None
+        assert len(pcp.cache) == 0
+
+    def test_lru_eviction(self, ie, logger):
+        MAX_SIZE = 2
+        provider = MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), MAX_SIZE))
+        provider.store('key1', 'value1', int(time.time()) + 5)
+        provider.store('key2', 'value2', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key1') == 'value1'
+
+        provider.store('key3', 'value3', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key2') is None
+
+        provider.store('key4', 'value4', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key1') is None
+        assert provider.get('key3') == 'value3'
+        assert provider.get('key4') == 'value4'
+
+    def test_delete(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 5)
+        assert len(pcp.cache) == 1
+        assert pcp.get('key1') == 'value1'
+        pcp.delete('key1')
+        assert len(pcp.cache) == 0
+        assert pcp.get('key1') is None
+
+    def test_use_global_cache_default(self, ie, logger):
+        pcp = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+        assert pcp.cache is _pot_memory_cache.value['cache']
+        assert pcp.lock is _pot_memory_cache.value['lock']
+
+        pcp2 = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == pcp2.max_size == _pot_memory_cache.value['max_size'] == 25
+        assert pcp.cache is pcp2.cache is _pot_memory_cache.value['cache']
+        assert pcp.lock is pcp2.lock is _pot_memory_cache.value['lock']
+
+    def test_fail_max_size_change_global(self, ie, logger):
+        pcp = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+        with pytest.raises(ValueError, match='Cannot change max_size of initialized global memory cache'):
+            initialize_global_cache(50)
+
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+
+    def test_memory_lru_preference(self, pcp, ie, pot_request):
+        assert memorylru_preference(pcp, pot_request) == 10000

test/test_pot/test_pot_builtin_utils.py

@@ -0,0 +1,46 @@
+import pytest
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenContext,
+
+)
+
+from yt_dlp.extractor.youtube.pot.utils import get_webpo_content_binding, ContentBindingType
+
+
+class TestGetWebPoContentBinding:
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated, expected', [
+        *[(client, context, is_authenticated, expected) for client in [
+            'WEB', 'MWEB', 'TVHTML5', 'WEB_EMBEDDED_PLAYER', 'WEB_CREATOR', 'TVHTML5_SIMPLY_EMBEDDED_PLAYER']
+          for context, is_authenticated, expected in [
+            (PoTokenContext.GVS, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+            (PoTokenContext.PLAYER, False, ('example-video-id', ContentBindingType.VIDEO_ID)),
+            (PoTokenContext.GVS, True, ('example-data-sync-id', ContentBindingType.DATASYNC_ID)),
+        ]],
+        ('WEB_REMIX', PoTokenContext.GVS, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+        ('WEB_REMIX', PoTokenContext.PLAYER, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+        ('ANDROID', PoTokenContext.GVS, False, (None, None)),
+        ('IOS', PoTokenContext.GVS, False, (None, None)),
+    ])
+    def test_get_webpo_content_binding(self, pot_request, client_name, context, is_authenticated, expected):
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        assert get_webpo_content_binding(pot_request) == expected
+
+    def test_extract_visitor_id(self, pot_request):
+        pot_request.visitor_data = 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == ('123abcXYZ_-', ContentBindingType.VISITOR_ID)
+
+    def test_invalid_visitor_id(self, pot_request):
+        # visitor id not alphanumeric (i.e. protobuf extraction failed)
+        pot_request.visitor_data = 'CggxMjM0NTY3OCiA4s-qBg%3D%3D'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)
+
+    def test_no_visitor_id(self, pot_request):
+        pot_request.visitor_data = 'KIDiz6oG'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)
+
+    def test_invalid_base64(self, pot_request):
+        pot_request.visitor_data = 'invalid-base64'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)

test/test_pot/test_pot_builtin_webpospec.py

@@ -0,0 +1,92 @@
+import pytest
+
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider, BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot.cache import CacheProviderWritePolicy
+from yt_dlp.utils import bug_reports_message
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+
+)
+from yt_dlp.version import __version__
+
+from yt_dlp.extractor.youtube.pot._builtin.webpo_cachespec import WebPoPCSP
+from yt_dlp.extractor.youtube.pot._registry import _pot_pcs_providers
+
+
+@pytest.fixture()
+def pot_request(pot_request) -> PoTokenRequest:
+    pot_request.visitor_data = 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D'  # visitor_id=123abcXYZ_-
+    return pot_request
+
+
+class TestWebPoPCSP:
+    def test_base_type(self):
+        assert issubclass(WebPoPCSP, IEContentProvider)
+        assert issubclass(WebPoPCSP, BuiltInIEContentProvider)
+
+    def test_init(self, ie, logger):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        assert pcs.PROVIDER_NAME == 'webpo'
+        assert pcs.PROVIDER_VERSION == __version__
+        assert pcs.BUG_REPORT_MESSAGE == bug_reports_message(before='')
+        assert pcs.is_available()
+
+    def test_is_registered(self):
+        assert _pot_pcs_providers.value.get('WebPo') == WebPoPCSP
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated', [
+        ('ANDROID', PoTokenContext.GVS, False),
+        ('IOS', PoTokenContext.GVS, False),
+        ('IOS', PoTokenContext.PLAYER, False),
+    ])
+    def test_not_supports(self, ie, logger, pot_request, client_name, context, is_authenticated):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        assert pcs.generate_cache_spec(pot_request) is None
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated, remote_host, source_address, request_proxy, expected', [
+        *[(client, context, is_authenticated, remote_host, source_address, request_proxy, expected) for client in [
+            'WEB', 'MWEB', 'TVHTML5', 'WEB_EMBEDDED_PLAYER', 'WEB_CREATOR', 'TVHTML5_SIMPLY_EMBEDDED_PLAYER']
+          for context, is_authenticated, remote_host, source_address, request_proxy, expected in [
+            (PoTokenContext.GVS, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id'}),
+            (PoTokenContext.PLAYER, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'video_id'}),
+            (PoTokenContext.GVS, True, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': 'example-data-sync-id', 'cbt': 'datasync_id'}),
+        ]],
+        ('WEB_REMIX', PoTokenContext.PLAYER, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id'}),
+        ('WEB', PoTokenContext.GVS, False, None, None, None, {'t': 'webpo', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id', 'ip': None, 'sa': None, 'px': None}),
+        ('TVHTML5', PoTokenContext.PLAYER, False, None, None, 'http://example.com', {'t': 'webpo', 'cb': '123abcXYZ_-', 'cbt': 'video_id', 'ip': None, 'sa': None, 'px': 'http://example.com'}),
+
+    ])
+    def test_generate_key_bindings(self, ie, logger, pot_request, client_name, context, is_authenticated, remote_host, source_address, request_proxy, expected):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        pot_request.innertube_context['client']['remoteHost'] = remote_host
+        pot_request.request_source_address = source_address
+        pot_request.request_proxy = request_proxy
+        pot_request.video_id = '123abcXYZ_-'  # same as visitor id to test type
+
+        assert pcs.generate_cache_spec(pot_request).key_bindings == expected
+
+    def test_no_bind_visitor_id(self, ie, logger, pot_request):
+        # Should not bind to visitor id if setting is set to False
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={'bind_to_visitor_id': ['false']})
+        pot_request.innertube_context['client']['clientName'] = 'WEB'
+        pot_request.context = PoTokenContext.GVS
+        pot_request.is_authenticated = False
+        assert pcs.generate_cache_spec(pot_request).key_bindings == {'t': 'webpo', 'ip': None, 'sa': None, 'px': None, 'cb': 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D', 'cbt': 'visitor_data'}
+
+    def test_default_ttl(self, ie, logger, pot_request):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        assert pcs.generate_cache_spec(pot_request).default_ttl == 6 * 60 * 60  # should default to 6 hours
+
+    def test_write_policy(self, ie, logger, pot_request):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.context = PoTokenContext.GVS
+        assert pcs.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_ALL
+        pot_request.context = PoTokenContext.PLAYER
+        assert pcs.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_FIRST

test/test_pot/test_pot_framework.py

@@ -0,0 +1,629 @@
+import pytest
+
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.utils.networking import HTTPHeaderDict
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+    ExternalRequestFeature,
+
+)
+
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    CacheProviderWritePolicy,
+)
+
+import yt_dlp.extractor.youtube.pot.cache as cache
+
+from yt_dlp.networking import Request
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenResponse,
+    PoTokenProvider,
+    PoTokenProviderRejectedRequest,
+    provider_bug_report_message,
+    register_provider,
+    register_preference,
+)
+
+from yt_dlp.extractor.youtube.pot._registry import _pot_providers, _ptp_preferences, _pot_pcs_providers, _pot_cache_providers, _pot_cache_provider_preferences
+
+
+class ExamplePTP(PoTokenProvider):
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    _SUPPORTED_CLIENTS = ('WEB',)
+    _SUPPORTED_CONTEXTS = (PoTokenContext.GVS, )
+
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+        ExternalRequestFeature.PROXY_SCHEME_HTTP,
+        ExternalRequestFeature.PROXY_SCHEME_SOCKS5H,
+    )
+
+    def is_available(self) -> bool:
+        return True
+
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        return PoTokenResponse('example-token', expires_at=123)
+
+
+class ExampleCacheProviderPCP(PoTokenCacheProvider):
+
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    def is_available(self) -> bool:
+        return True
+
+    def get(self, key: str):
+        return 'example-cache'
+
+    def store(self, key: str, value: str, expires_at: int):
+        pass
+
+    def delete(self, key: str):
+        pass
+
+
+class ExampleCacheSpecProviderPCSP(PoTokenCacheSpecProvider):
+
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    def generate_cache_spec(self, request: PoTokenRequest):
+        return PoTokenCacheSpec(
+            key_bindings={'field': 'example-key'},
+            default_ttl=60,
+            write_policy=CacheProviderWritePolicy.WRITE_FIRST,
+        )
+
+
+class TestPoTokenProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenProvider, IEContentProvider)
+
+    def test_create_provider_missing_fetch_method(self, ie, logger):
+        class MissingMethodsPTP(PoTokenProvider):
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPTP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_available_method(self, ie, logger):
+        class MissingMethodsPTP(PoTokenProvider):
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+        with pytest.raises(TypeError):
+            MissingMethodsPTP(ie=ie, logger=logger, settings={})
+
+    def test_barebones_provider(self, ie, logger):
+        class BarebonesProviderPTP(PoTokenProvider):
+            def is_available(self) -> bool:
+                return True
+
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+        provider = BarebonesProviderPTP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+
+    def test_example_provider_success(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'Example'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+
+        response = provider.request_pot(pot_request)
+
+        assert response.po_token == 'example-token'
+        assert response.expires_at == 123
+
+    def test_provider_unsupported_context(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.context = PoTokenContext.PLAYER
+
+        with pytest.raises(PoTokenProviderRejectedRequest):
+            provider.request_pot(pot_request)
+
+    def test_provider_unsupported_client(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = 'ANDROID'
+
+        with pytest.raises(PoTokenProviderRejectedRequest):
+            provider.request_pot(pot_request)
+
+    def test_provider_unsupported_proxy_scheme(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.request_proxy = 'socks4://example.com'
+
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support proxy scheme "socks4". Supported proxy '
+            'schemes: http, socks5h',
+        ):
+            provider.request_pot(pot_request)
+
+        pot_request.request_proxy = 'http://example.com'
+
+        assert provider.request_pot(pot_request)
+
+    def test_provider_ignore_external_request_features(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = None
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_proxy = 'socks5://example.com'
+        assert provider.request_pot(pot_request)
+        pot_request.request_source_address = '0.0.0.0'
+        assert provider.request_pot(pot_request)
+
+    def test_provider_unsupported_external_request_source_address(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = tuple()
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_source_address = None
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_source_address = '0.0.0.0'
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support setting source address',
+        ):
+            provider.request_pot(pot_request)
+
+    def test_provider_supported_external_request_source_address(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+                ExternalRequestFeature.SOURCE_ADDRESS,
+            )
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_source_address = None
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_source_address = '0.0.0.0'
+        assert provider.request_pot(pot_request)
+
+    def test_provider_unsupported_external_request_tls_verification(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = tuple()
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_verify_tls = True
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_verify_tls = False
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support ignoring TLS certificate failures',
+        ):
+            provider.request_pot(pot_request)
+
+    def test_provider_supported_external_request_tls_verification(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+                ExternalRequestFeature.DISABLE_TLS_VERIFICATION,
+            )
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_verify_tls = True
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_verify_tls = False
+        assert provider.request_pot(pot_request)
+
+    def test_provider_request_webpage(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        cookiejar = YoutubeDLCookieJar()
+        pot_request.request_headers = HTTPHeaderDict({'User-Agent': 'example-user-agent'})
+        pot_request.request_proxy = 'socks5://example-proxy.com'
+        pot_request.request_cookiejar = cookiejar
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), pot_request=pot_request)
+
+        assert sent_request.url == 'https://example.com'
+        assert sent_request.headers['User-Agent'] == 'example-user-agent'
+        assert sent_request.proxies == {'all': 'socks5://example-proxy.com'}
+        assert sent_request.extensions['cookiejar'] is cookiejar
+        assert 'Requesting webpage' in logger.messages['info']
+
+    def test_provider_request_webpage_override(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        cookiejar_request = YoutubeDLCookieJar()
+        pot_request.request_headers = HTTPHeaderDict({'User-Agent': 'example-user-agent'})
+        pot_request.request_proxy = 'socks5://example-proxy.com'
+        pot_request.request_cookiejar = cookiejar_request
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+            headers={'User-Agent': 'override-user-agent-override'},
+            proxies={'http': 'http://example-proxy-override.com'},
+            extensions={'cookiejar': YoutubeDLCookieJar()},
+        ), pot_request=pot_request, note='Custom requesting webpage')
+
+        assert sent_request.url == 'https://example.com'
+        assert sent_request.headers['User-Agent'] == 'override-user-agent-override'
+        assert sent_request.proxies == {'http': 'http://example-proxy-override.com'}
+        assert sent_request.extensions['cookiejar'] is not cookiejar_request
+        assert 'Custom requesting webpage' in logger.messages['info']
+
+    def test_provider_request_webpage_no_log(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), note=False)
+
+        assert sent_request.url == 'https://example.com'
+        assert 'info' not in logger.messages
+
+    def test_provider_request_webpage_no_pot_request(self, ie, logger):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), pot_request=None)
+
+        assert sent_request.url == 'https://example.com'
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenProvider):
+            PROVIDER_NAME = 'invalid-suffix'
+
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+            def is_available(self) -> bool:
+                return True
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenCacheProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenCacheProvider, IEContentProvider)
+
+    def test_create_provider_missing_get_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+                def is_available(self) -> bool:
+                    return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_store_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_delete_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_is_available_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_barebones_provider(self, ie, logger):
+        class BarebonesProviderPCP(PoTokenCacheProvider):
+
+            def is_available(self) -> bool:
+                return True
+
+            def get(self, key: str):
+                return 'example-cache'
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+        provider = BarebonesProviderPCP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+
+    def test_create_provider_example(self, ie, logger):
+        provider = ExampleCacheProviderPCP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'ExampleCacheProvider'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExampleCacheProviderPCP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenCacheProvider):
+            def get(self, key: str):
+                return 'example-cache'
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenCacheSpecProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenCacheSpecProvider, IEContentProvider)
+
+    def test_create_provider_missing_supports_method(self, ie, logger):
+        class MissingMethodsPCS(PoTokenCacheSpecProvider):
+            pass
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCS(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_barebones(self, ie, pot_request, logger):
+        class BarebonesProviderPCSP(PoTokenCacheSpecProvider):
+            def generate_cache_spec(self, request: PoTokenRequest):
+                return PoTokenCacheSpec(
+                    default_ttl=100,
+                    key_bindings={},
+                )
+
+        provider = BarebonesProviderPCSP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+        assert provider.is_available()
+        assert provider.generate_cache_spec(request=pot_request).default_ttl == 100
+        assert provider.generate_cache_spec(request=pot_request).key_bindings == {}
+        assert provider.generate_cache_spec(request=pot_request).write_policy == CacheProviderWritePolicy.WRITE_ALL
+
+    def test_create_provider_example(self, ie, pot_request, logger):
+        provider = ExampleCacheSpecProviderPCSP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'ExampleCacheSpecProvider'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+        assert provider.generate_cache_spec(pot_request)
+        assert provider.generate_cache_spec(pot_request).key_bindings == {'field': 'example-key'}
+        assert provider.generate_cache_spec(pot_request).default_ttl == 60
+        assert provider.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_FIRST
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExampleCacheSpecProviderPCSP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenCacheSpecProvider):
+            def generate_cache_spec(self, request: PoTokenRequest):
+                return None
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenRequest:
+    def test_copy_request(self, pot_request):
+        copied_request = pot_request.copy()
+
+        assert copied_request is not pot_request
+        assert copied_request.context == pot_request.context
+        assert copied_request.innertube_context == pot_request.innertube_context
+        assert copied_request.innertube_context is not pot_request.innertube_context
+        copied_request.innertube_context['client']['clientName'] = 'ANDROID'
+        assert pot_request.innertube_context['client']['clientName'] != 'ANDROID'
+        assert copied_request.innertube_host == pot_request.innertube_host
+        assert copied_request.session_index == pot_request.session_index
+        assert copied_request.player_url == pot_request.player_url
+        assert copied_request.is_authenticated == pot_request.is_authenticated
+        assert copied_request.visitor_data == pot_request.visitor_data
+        assert copied_request.data_sync_id == pot_request.data_sync_id
+        assert copied_request.video_id == pot_request.video_id
+        assert copied_request.request_cookiejar is pot_request.request_cookiejar
+        assert copied_request.request_proxy == pot_request.request_proxy
+        assert copied_request.request_headers == pot_request.request_headers
+        assert copied_request.request_headers is not pot_request.request_headers
+        assert copied_request.request_timeout == pot_request.request_timeout
+        assert copied_request.request_source_address == pot_request.request_source_address
+        assert copied_request.request_verify_tls == pot_request.request_verify_tls
+        assert copied_request.bypass_cache == pot_request.bypass_cache
+
+
+def test_provider_bug_report_message(ie, logger):
+    provider = ExamplePTP(ie=ie, logger=logger, settings={})
+    assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+
+    message = provider_bug_report_message(provider)
+    assert message == '; please report this issue to the provider developer at  https://example.com/issues  .'
+
+    message_before = provider_bug_report_message(provider, before='custom message!')
+    assert message_before == 'custom message! Please report this issue to the provider developer at  https://example.com/issues  .'
+
+
+def test_register_provider(ie):
+
+    @register_provider
+    class UnavailableProviderPTP(PoTokenProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+            raise PoTokenProviderRejectedRequest('Not implemented')
+
+    assert _pot_providers.value.get('UnavailableProvider') == UnavailableProviderPTP
+    _pot_providers.value.pop('UnavailableProvider')
+
+
+def test_register_pot_preference(ie):
+    before = len(_ptp_preferences.value)
+
+    @register_preference(ExamplePTP)
+    def unavailable_preference(provider: PoTokenProvider, request: PoTokenRequest):
+        return 1
+
+    assert len(_ptp_preferences.value) == before + 1
+
+
+def test_register_cache_provider(ie):
+
+    @cache.register_provider
+    class UnavailableCacheProviderPCP(PoTokenCacheProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def get(self, key: str):
+            return 'example-cache'
+
+        def store(self, key: str, value: str, expires_at: int):
+            pass
+
+        def delete(self, key: str):
+            pass
+
+    assert _pot_cache_providers.value.get('UnavailableCacheProvider') == UnavailableCacheProviderPCP
+    _pot_cache_providers.value.pop('UnavailableCacheProvider')
+
+
+def test_register_cache_provider_spec(ie):
+
+    @cache.register_spec
+    class UnavailableCacheProviderPCSP(PoTokenCacheSpecProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def generate_cache_spec(self, request: PoTokenRequest):
+            return None
+
+    assert _pot_pcs_providers.value.get('UnavailableCacheProvider') == UnavailableCacheProviderPCSP
+    _pot_pcs_providers.value.pop('UnavailableCacheProvider')
+
+
+def test_register_cache_provider_preference(ie):
+    before = len(_pot_cache_provider_preferences.value)
+
+    @cache.register_preference(ExampleCacheProviderPCP)
+    def unavailable_preference(provider: PoTokenCacheProvider, request: PoTokenRequest):
+        return 1
+
+    assert len(_pot_cache_provider_preferences.value) == before + 1
+
+
+def test_logger_log_level(logger):
+    assert logger.LogLevel('INFO') == logger.LogLevel.INFO
+    assert logger.LogLevel('debuG') == logger.LogLevel.DEBUG
+    assert logger.LogLevel(10) == logger.LogLevel.DEBUG
+    assert logger.LogLevel('UNKNOWN') == logger.LogLevel.INFO

test/test_utils.py

@@ -659,6 +659,8 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(url_or_none('mms://foo.de'), 'mms://foo.de')
         self.assertEqual(url_or_none('rtspu://foo.de'), 'rtspu://foo.de')
         self.assertEqual(url_or_none('ftps://foo.de'), 'ftps://foo.de')
+        self.assertEqual(url_or_none('ws://foo.de'), 'ws://foo.de')
+        self.assertEqual(url_or_none('wss://foo.de'), 'wss://foo.de')
 
     def test_parse_age_limit(self):
         self.assertEqual(parse_age_limit(None), None)

test/test_youtube_signature.py

@@ -88,6 +88,51 @@ _SIG_TESTS = [
         '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
         '0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpz2ICs6EVdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
     ),
+    (
+        'https://www.youtube.com/s/player/363db69b/player_ias_tce.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpz2ICs6EVdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+    ),
+    (
+        'https://www.youtube.com/s/player/4fcd6e4a/player_ias.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        'wAOAOq0QJ8ARAIgXmPlOPSBkkUs1bYFYlJCfe29xx8q7v1pDL0QwbdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0',
+    ),
+    (
+        'https://www.youtube.com/s/player/4fcd6e4a/player_ias_tce.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        'wAOAOq0QJ8ARAIgXmPlOPSBkkUs1bYFYlJCfe29xx8q7v1pDL0QwbdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player_ias.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '7AOq0QJ8wRAIgXmPlOPSBkkAs1bYFYlJCfe29xx8jOv1pDL0Q2bdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0qaw',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player_ias_tce.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '7AOq0QJ8wRAIgXmPlOPSBkkAs1bYFYlJCfe29xx8jOv1pDL0Q2bdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0qaw',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player-plasma-ias-phone-en_US.vflset/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '7AOq0QJ8wRAIgXmPlOPSBkkAs1bYFYlJCfe29xx8jOv1pDL0Q2bdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0qaw',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player-plasma-ias-tablet-en_US.vflset/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '7AOq0QJ8wRAIgXmPlOPSBkkAs1bYFYlJCfe29xx8jOv1pDL0Q2bdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_EMu-m37KtXJoOySqa0qaw',
+    ),
+    (
+        'https://www.youtube.com/s/player/8a8ac953/player_ias_tce.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        'IAOAOq0QJ8wRAAgXmPlOPSBkkUs1bYFYlJCfe29xx8j7v1pDL0QwbdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_E2u-m37KtXJoOySqa0',
+    ),
+    (
+        'https://www.youtube.com/s/player/8a8ac953/tv-player-es6.vflset/tv-player-es6.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        'IAOAOq0QJ8wRAAgXmPlOPSBkkUs1bYFYlJCfe29xx8j7v1pDL0QwbdV96sCIEzpWqMGkFR20CFOg51Tp-7vj_E2u-m37KtXJoOySqa0',
+    ),
 ]
 
 _NSIG_TESTS = [
@@ -243,6 +288,34 @@ _NSIG_TESTS = [
         'https://www.youtube.com/s/player/363db69b/player_ias.vflset/en_US/base.js',
         'eWYu5d5YeY_4LyEDc', 'XJQqf-N7Xra3gg',
     ),
+    (
+        'https://www.youtube.com/s/player/4fcd6e4a/player_ias.vflset/en_US/base.js',
+        'o_L251jm8yhZkWtBW', 'lXoxI3XvToqn6A',
+    ),
+    (
+        'https://www.youtube.com/s/player/4fcd6e4a/player_ias_tce.vflset/en_US/base.js',
+        'o_L251jm8yhZkWtBW', 'lXoxI3XvToqn6A',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/tv-player-ias.vflset/tv-player-ias.js',
+        'ir9-V6cdbCiyKxhr', '9YE85kNjZiS4',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player-plasma-ias-phone-en_US.vflset/base.js',
+        'ir9-V6cdbCiyKxhr', '9YE85kNjZiS4',
+    ),
+    (
+        'https://www.youtube.com/s/player/20830619/player-plasma-ias-tablet-en_US.vflset/base.js',
+        'ir9-V6cdbCiyKxhr', '9YE85kNjZiS4',
+    ),
+    (
+        'https://www.youtube.com/s/player/8a8ac953/player_ias_tce.vflset/en_US/base.js',
+        'MiBYeXx_vRREbiCCmh', 'RtZYMVvmkE0JE',
+    ),
+    (
+        'https://www.youtube.com/s/player/8a8ac953/tv-player-es6.vflset/tv-player-es6.js',
+        'MiBYeXx_vRREbiCCmh', 'RtZYMVvmkE0JE',
+    ),
 ]
 
 
@@ -293,33 +366,33 @@ def t_factory(name, sig_func, url_pattern):
         test_id = re.sub(r'[/.-]', '_', m.group('id') or m.group('compat_id'))
 
         def test_func(self):
-            basename = f'player-{name}-{test_id}.js'
+            basename = f'player-{test_id}.js'
             fn = os.path.join(self.TESTDATA_DIR, basename)
 
             if not os.path.exists(fn):
                 urllib.request.urlretrieve(url, fn)
             with open(fn, encoding='utf-8') as testf:
                 jscode = testf.read()
-            self.assertEqual(sig_func(jscode, sig_input), expected_sig)
+            self.assertEqual(sig_func(jscode, sig_input, url), expected_sig)
 
         test_func.__name__ = f'test_{name}_js_{test_id}'
         setattr(TestSignature, test_func.__name__, test_func)
     return make_tfunc
 
 
-def signature(jscode, sig_input):
-    func = YoutubeIE(FakeYDL())._parse_sig_js(jscode)
+def signature(jscode, sig_input, player_url):
+    func = YoutubeIE(FakeYDL())._parse_sig_js(jscode, player_url)
     src_sig = (
         str(string.printable[:sig_input])
         if isinstance(sig_input, int) else sig_input)
     return func(src_sig)
 
 
-def n_sig(jscode, sig_input):
+def n_sig(jscode, sig_input, player_url):
     ie = YoutubeIE(FakeYDL())
-    funcname = ie._extract_n_function_name(jscode)
+    funcname = ie._extract_n_function_name(jscode, player_url=player_url)
     jsi = JSInterpreter(jscode)
-    func = jsi.extract_function_from_code(*ie._fixup_n_function_code(*jsi.extract_function_code(funcname), jscode))
+    func = jsi.extract_function_from_code(*ie._fixup_n_function_code(*jsi.extract_function_code(funcname), jscode, player_url))
     return func([sig_input])
 
 

yt_dlp/YoutubeDL.py

@@ -640,6 +640,7 @@ class YoutubeDL:
         self._printed_messages = set()
         self._first_webpage_request = True
         self._post_hooks = []
+        self._close_hooks = []
         self._progress_hooks = []
         self._postprocessor_hooks = []
         self._download_retcode = 0
@@ -908,6 +909,10 @@ class YoutubeDL:
         """Add the post hook"""
         self._post_hooks.append(ph)
 
+    def add_close_hook(self, ch):
+        """Add a close hook, called when YoutubeDL.close() is called"""
+        self._close_hooks.append(ch)
+
     def add_progress_hook(self, ph):
         """Add the download progress hook"""
         self._progress_hooks.append(ph)
@@ -1016,6 +1021,9 @@ class YoutubeDL:
             self._request_director.close()
             del self._request_director
 
+        for close_hook in self._close_hooks:
+            close_hook()
+
     def trouble(self, message=None, tb=None, is_error=True):
         """Determine action to take when a download problem appears.
 

yt_dlp/downloader/niconico.py

@@ -85,6 +85,7 @@ class NiconicoLiveFD(FileDownloader):
                             'quality': live_quality,
                             'protocol': 'hls+fmp4',
                             'latency': live_latency,
+                            'accessRightMethod': 'single_cookie',
                             'chasePlay': False,
                         },
                         'room': {

yt_dlp/extractor/_extractors.py

@@ -683,6 +683,7 @@ from .foxnews import (
 )
 from .foxsports import FoxSportsIE
 from .fptplay import FptplayIE
+from .francaisfacile import FrancaisFacileIE
 from .franceinter import FranceInterIE
 from .francetv import (
     FranceTVIE,
@@ -902,6 +903,7 @@ from .ivi import (
     IviIE,
 )
 from .ivideon import IvideonIE
+from .ivoox import IvooxIE
 from .iwara import (
     IwaraIE,
     IwaraPlaylistIE,
@@ -959,7 +961,10 @@ from .kick import (
 )
 from .kicker import KickerIE
 from .kickstarter import KickStarterIE
-from .kika import KikaIE
+from .kika import (
+    KikaIE,
+    KikaPlaylistIE,
+)
 from .kinja import KinjaEmbedIE
 from .kinopoisk import KinoPoiskIE
 from .kommunetv import KommunetvIE
@@ -1060,6 +1065,7 @@ from .loom import (
 from .lovehomeporn import LoveHomePornIE
 from .lrt import (
     LRTVODIE,
+    LRTRadioIE,
     LRTStreamIE,
 )
 from .lsm import (
@@ -1492,6 +1498,10 @@ from .paramountplus import (
 )
 from .parler import ParlerIE
 from .parlview import ParlviewIE
+from .parti import (
+    PartiLivestreamIE,
+    PartiVideoIE,
+)
 from .patreon import (
     PatreonCampaignIE,
     PatreonIE,
@@ -1738,6 +1748,7 @@ from .roosterteeth import (
     RoosterTeethSeriesIE,
 )
 from .rottentomatoes import RottenTomatoesIE
+from .roya import RoyaLiveIE
 from .rozhlas import (
     MujRozhlasIE,
     RozhlasIE,

yt_dlp/extractor/agora.py

@@ -146,7 +146,7 @@ class TokFMPodcastIE(InfoExtractor):
         'url': 'https://audycje.tokfm.pl/podcast/91275,-Systemowy-rasizm-Czy-zamieszki-w-USA-po-morderstwie-w-Minneapolis-doprowadza-do-zmian-w-sluzbach-panstwowych',
         'info_dict': {
             'id': '91275',
-            'ext': 'aac',
+            'ext': 'mp3',
             'title': 'md5:a9b15488009065556900169fb8061cce',
             'episode': 'md5:a9b15488009065556900169fb8061cce',
             'series': 'Analizy',
@@ -164,23 +164,20 @@ class TokFMPodcastIE(InfoExtractor):
             raise ExtractorError('No such podcast', expected=True)
         metadata = metadata[0]
 
-        formats = []
-        for ext in ('aac', 'mp3'):
-            url_data = self._download_json(
-                f'https://api.podcast.radioagora.pl/api4/getSongUrl?podcast_id={media_id}&device_id={uuid.uuid4()}&ppre=false&audio={ext}',
-                media_id, f'Downloading podcast {ext} URL')
-            # prevents inserting the mp3 (default) multiple times
-            if 'link_ssl' in url_data and f'.{ext}' in url_data['link_ssl']:
-                formats.append({
-                    'url': url_data['link_ssl'],
-                    'ext': ext,
-                    'vcodec': 'none',
-                    'acodec': ext,
-                })
+        mp3_url = self._download_json(
+            'https://api.podcast.radioagora.pl/api4/getSongUrl',
+            media_id, 'Downloading podcast mp3 URL', query={
+                'podcast_id': media_id,
+                'device_id': str(uuid.uuid4()),
+                'ppre': 'false',
+                'audio': 'mp3',
+            })['link_ssl']
 
         return {
             'id': media_id,
-            'formats': formats,
+            'url': mp3_url,
+            'vcodec': 'none',
+            'ext': 'mp3',
             'title': metadata.get('podcast_name'),
             'series': metadata.get('series_name'),
             'episode': metadata.get('podcast_name'),

yt_dlp/extractor/common.py

@@ -1570,6 +1570,8 @@ class InfoExtractor:
         """Yield all json ld objects in the html"""
         if default is not NO_DEFAULT:
             fatal = False
+        if not fatal and not isinstance(html, str):
+            return
         for mobj in re.finditer(JSON_LD_RE, html):
             json_ld_item = self._parse_json(
                 mobj.group('json_ld'), video_id, fatal=fatal,

yt_dlp/extractor/crowdbunker.py

@@ -5,7 +5,9 @@ from ..utils import (
     int_or_none,
     try_get,
     unified_strdate,
+    url_or_none,
 )
+from ..utils.traversal import traverse_obj
 
 
 class CrowdBunkerIE(InfoExtractor):
@@ -44,16 +46,15 @@ class CrowdBunkerIE(InfoExtractor):
                 'url': sub_url,
             })
 
-        mpd_url = try_get(video_json, lambda x: x['dashManifest']['url'])
-        if mpd_url:
-            fmts, subs = self._extract_mpd_formats_and_subtitles(mpd_url, video_id)
+        if mpd_url := traverse_obj(video_json, ('dashManifest', 'url', {url_or_none})):
+            fmts, subs = self._extract_mpd_formats_and_subtitles(mpd_url, video_id, mpd_id='dash', fatal=False)
             formats.extend(fmts)
-            subtitles = self._merge_subtitles(subtitles, subs)
-        m3u8_url = try_get(video_json, lambda x: x['hlsManifest']['url'])
-        if m3u8_url:
-            fmts, subs = self._extract_m3u8_formats_and_subtitles(mpd_url, video_id)
+            self._merge_subtitles(subs, target=subtitles)
+
+        if m3u8_url := traverse_obj(video_json, ('hlsManifest', 'url', {url_or_none})):
+            fmts, subs = self._extract_m3u8_formats_and_subtitles(m3u8_url, video_id, m3u8_id='hls', fatal=False)
             formats.extend(fmts)
-            subtitles = self._merge_subtitles(subtitles, subs)
+            self._merge_subtitles(subs, target=subtitles)
 
         thumbnails = [{
             'url': image['url'],

yt_dlp/extractor/francaisfacile.py

@@ -0,0 +1,87 @@
+import urllib.parse
+
+from .common import InfoExtractor
+from ..networking.exceptions import HTTPError
+from ..utils import (
+    ExtractorError,
+    float_or_none,
+    url_or_none,
+)
+from ..utils.traversal import traverse_obj
+
+
+class FrancaisFacileIE(InfoExtractor):
+    _VALID_URL = r'https?://francaisfacile\.rfi\.fr/[a-z]{2}/(?:actualit%C3%A9|podcasts/[^/#?]+)/(?P<id>[^/#?]+)'
+    _TESTS = [{
+        'url': 'https://francaisfacile.rfi.fr/fr/actualit%C3%A9/20250305-r%C3%A9concilier-les-jeunes-avec-la-lecture-gr%C3%A2ce-aux-r%C3%A9seaux-sociaux',
+        'md5': '4f33674cb205744345cc835991100afa',
+        'info_dict': {
+            'id': 'WBMZ58952-FLE-FR-20250305',
+            'display_id': '20250305-réconcilier-les-jeunes-avec-la-lecture-grâce-aux-réseaux-sociaux',
+            'title': 'Réconcilier les jeunes avec la lecture grâce aux réseaux sociaux',
+            'url': 'https://aod-fle.akamaized.net/fle/sounds/fr/2025/03/05/6b6af52a-f9ba-11ef-a1f8-005056a97652.mp3',
+            'ext': 'mp3',
+            'description': 'md5:b903c63d8585bd59e8cc4d5f80c4272d',
+            'duration': 103.15,
+            'timestamp': 1741177984,
+            'upload_date': '20250305',
+        },
+    }, {
+        'url': 'https://francaisfacile.rfi.fr/fr/actualit%C3%A9/20250307-argentine-le-sac-d-un-alpiniste-retrouv%C3%A9-40-ans-apr%C3%A8s-sa-mort',
+        'md5': 'b8c3a63652d4ae8e8092dda5700c1cd9',
+        'info_dict': {
+            'id': 'WBMZ59102-FLE-FR-20250307',
+            'display_id': '20250307-argentine-le-sac-d-un-alpiniste-retrouvé-40-ans-après-sa-mort',
+            'title': 'Argentine: le sac d\'un alpiniste retrouvé 40 ans après sa mort',
+            'url': 'https://aod-fle.akamaized.net/fle/sounds/fr/2025/03/07/8edf4082-fb46-11ef-8a37-005056bf762b.mp3',
+            'ext': 'mp3',
+            'description': 'md5:7fd088fbdf4a943bb68cf82462160dca',
+            'duration': 117.74,
+            'timestamp': 1741352789,
+            'upload_date': '20250307',
+        },
+    }, {
+        'url': 'https://francaisfacile.rfi.fr/fr/podcasts/un-mot-une-histoire/20250317-le-mot-de-david-foenkinos-peut-%C3%AAtre',
+        'md5': 'db83c2cc2589b4c24571c6b6cf14f5f1',
+        'info_dict': {
+            'id': 'WBMZ59441-FLE-FR-20250317',
+            'display_id': '20250317-le-mot-de-david-foenkinos-peut-être',
+            'title': 'Le mot de David Foenkinos: «peut-être» - Un mot, une histoire',
+            'url': 'https://aod-fle.akamaized.net/fle/sounds/fr/2025/03/17/4ca6cbbe-0315-11f0-a85b-005056a97652.mp3',
+            'ext': 'mp3',
+            'description': 'md5:3fe35fae035803df696bfa7af2496e49',
+            'duration': 198.96,
+            'timestamp': 1742210897,
+            'upload_date': '20250317',
+        },
+    }]
+
+    def _real_extract(self, url):
+        display_id = urllib.parse.unquote(self._match_id(url))
+
+        try:  # yt-dlp's default user-agents are too old and blocked by the site
+            webpage = self._download_webpage(url, display_id, headers={
+                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:136.0) Gecko/20100101 Firefox/136.0',
+            })
+        except ExtractorError as e:
+            if not isinstance(e.cause, HTTPError) or e.cause.status != 403:
+                raise
+            # Retry with impersonation if hardcoded UA is insufficient
+            webpage = self._download_webpage(url, display_id, impersonate=True)
+
+        data = self._search_json(
+            r'<script[^>]+\bdata-media-id=[^>]+\btype="application/json"[^>]*>',
+            webpage, 'audio data', display_id)
+
+        return {
+            'id': data['mediaId'],
+            'display_id': display_id,
+            'vcodec': 'none',
+            'title': self._html_extract_title(webpage),
+            **self._search_json_ld(webpage, display_id, fatal=False),
+            **traverse_obj(data, {
+                'title': ('title', {str}),
+                'url': ('sources', ..., 'url', {url_or_none}, any),
+                'duration': ('sources', ..., 'duration', {float_or_none}, any),
+            }),
+        }

yt_dlp/extractor/generic.py

@@ -2214,10 +2214,21 @@ class GenericIE(InfoExtractor):
             if is_live is not None:
                 info['live_status'] = 'not_live' if is_live == 'false' else 'is_live'
                 return
-            headers = m3u8_format.get('http_headers') or info.get('http_headers')
-            duration = self._extract_m3u8_vod_duration(
-                m3u8_format['url'], info.get('id'), note='Checking m3u8 live status',
-                errnote='Failed to download m3u8 media playlist', headers=headers)
+            headers = m3u8_format.get('http_headers') or info.get('http_headers') or {}
+            display_id = info.get('id')
+            urlh = self._request_webpage(
+                m3u8_format['url'], display_id, 'Checking m3u8 live status', errnote=False,
+                headers={**headers, 'Accept-Encoding': 'identity'}, fatal=False)
+            if urlh is False:
+                return
+            first_bytes = urlh.read(512)
+            if not first_bytes.startswith(b'#EXTM3U'):
+                return
+            m3u8_doc = self._webpage_read_content(
+                urlh, urlh.url, display_id, prefix=first_bytes, fatal=False, errnote=False)
+            if not m3u8_doc:
+                return
+            duration = self._parse_m3u8_vod_duration(m3u8_doc, display_id)
             if not duration:
                 info['live_status'] = 'is_live'
             info['duration'] = info.get('duration') or duration

yt_dlp/extractor/ivoox.py

@@ -0,0 +1,78 @@
+from .common import InfoExtractor
+from ..utils import int_or_none, parse_iso8601, url_or_none, urljoin
+from ..utils.traversal import traverse_obj
+
+
+class IvooxIE(InfoExtractor):
+    _VALID_URL = (
+        r'https?://(?:www\.)?ivoox\.com/(?:\w{2}/)?[^/?#]+_rf_(?P<id>[0-9]+)_1\.html',
+        r'https?://go\.ivoox\.com/rf/(?P<id>[0-9]+)',
+    )
+    _TESTS = [{
+        'url': 'https://www.ivoox.com/dex-08x30-rostros-del-mal-los-asesinos-en-audios-mp3_rf_143594959_1.html',
+        'md5': '993f712de5b7d552459fc66aa3726885',
+        'info_dict': {
+            'id': '143594959',
+            'ext': 'mp3',
+            'timestamp': 1742731200,
+            'channel': 'DIAS EXTRAÑOS con Santiago Camacho',
+            'title': 'DEx 08x30 Rostros del mal: Los asesinos en serie que aterrorizaron España',
+            'description': 'md5:eae8b4b9740d0216d3871390b056bb08',
+            'uploader': 'Santiago Camacho',
+            'thumbnail': 'https://static-1.ivoox.com/audios/c/d/5/2/cd52f46783fe735000c33a803dce2554_XXL.jpg',
+            'upload_date': '20250323',
+            'episode': 'DEx 08x30 Rostros del mal: Los asesinos en serie que aterrorizaron España',
+            'duration': 11837,
+            'tags': ['españa', 'asesinos en serie', 'arropiero', 'historia criminal', 'mataviejas'],
+        },
+    }, {
+        'url': 'https://go.ivoox.com/rf/143594959',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.ivoox.com/en/campodelgas-28-03-2025-audios-mp3_rf_144036942_1.html',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        media_id = self._match_id(url)
+        webpage = self._download_webpage(url, media_id, fatal=False)
+
+        data = self._search_nuxt_data(
+            webpage, media_id, fatal=False, traverse=('data', 0, 'data', 'audio'))
+
+        direct_download = self._download_json(
+            f'https://vcore-web.ivoox.com/v1/public/audios/{media_id}/download-url', media_id, fatal=False,
+            note='Fetching direct download link', headers={'Referer': url})
+
+        download_paths = {
+            *traverse_obj(direct_download, ('data', 'downloadUrl', {str}, filter, all)),
+            *traverse_obj(data, (('downloadUrl', 'mediaUrl'), {str}, filter)),
+        }
+
+        formats = []
+        for path in download_paths:
+            formats.append({
+                'url': urljoin('https://ivoox.com', path),
+                'http_headers': {'Referer': url},
+            })
+
+        return {
+            'id': media_id,
+            'formats': formats,
+            'uploader': self._html_search_regex(r'data-prm-author="([^"]+)"', webpage, 'author', default=None),
+            'timestamp': parse_iso8601(
+                self._html_search_regex(r'data-prm-pubdate="([^"]+)"', webpage, 'timestamp', default=None)),
+            'channel': self._html_search_regex(r'data-prm-podname="([^"]+)"', webpage, 'channel', default=None),
+            'title': self._html_search_regex(r'data-prm-title="([^"]+)"', webpage, 'title', default=None),
+            'thumbnail': self._og_search_thumbnail(webpage, default=None),
+            'description': self._og_search_description(webpage, default=None),
+            **self._search_json_ld(webpage, media_id, default={}),
+            **traverse_obj(data, {
+                'title': ('title', {str}),
+                'description': ('description', {str}),
+                'thumbnail': ('image', {url_or_none}),
+                'timestamp': ('uploadDate', {parse_iso8601(delimiter=' ')}),
+                'duration': ('duration', {int_or_none}),
+                'tags': ('tags', ..., 'name', {str}),
+            }),
+        }

yt_dlp/extractor/kika.py

@@ -1,3 +1,5 @@
+import itertools
+
 from .common import InfoExtractor
 from ..utils import (
     determine_ext,
@@ -124,3 +126,43 @@ class KikaIE(InfoExtractor):
                         'vbr': ('bitrateVideo', {int_or_none}, {lambda x: None if x == -1 else x}),
                     }),
                 }
+
+
+class KikaPlaylistIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?kika\.de/[\w-]+/(?P<id>[a-z-]+\d+)'
+
+    _TESTS = [{
+        'url': 'https://www.kika.de/logo/logo-die-welt-und-ich-562',
+        'info_dict': {
+            'id': 'logo-die-welt-und-ich-562',
+            'title': 'logo!',
+            'description': 'md5:7b9d7f65561b82fa512f2cfb553c397d',
+        },
+        'playlist_count': 100,
+    }]
+
+    def _entries(self, playlist_url, playlist_id):
+        for page in itertools.count(1):
+            data = self._download_json(playlist_url, playlist_id, note=f'Downloading page {page}')
+            for item in traverse_obj(data, ('content', lambda _, v: url_or_none(v['api']['url']))):
+                yield self.url_result(
+                    item['api']['url'], ie=KikaIE,
+                    **traverse_obj(item, {
+                        'id': ('id', {str}),
+                        'title': ('title', {str}),
+                        'duration': ('duration', {int_or_none}),
+                        'timestamp': ('date', {parse_iso8601}),
+                    }))
+
+            playlist_url = traverse_obj(data, ('links', 'next', {url_or_none}))
+            if not playlist_url:
+                break
+
+    def _real_extract(self, url):
+        playlist_id = self._match_id(url)
+        brand_data = self._download_json(
+            f'https://www.kika.de/_next-api/proxy/v1/brands/{playlist_id}', playlist_id)
+
+        return self.playlist_result(
+            self._entries(brand_data['videoSubchannel']['videosPageUrl'], playlist_id),
+            playlist_id, title=brand_data.get('title'), description=brand_data.get('description'))

yt_dlp/extractor/lrt.py

@@ -2,8 +2,11 @@ from .common import InfoExtractor
 from ..utils import (
     clean_html,
     merge_dicts,
+    str_or_none,
     traverse_obj,
+    unified_timestamp,
     url_or_none,
+    urljoin,
 )
 
 
@@ -80,7 +83,7 @@ class LRTVODIE(LRTBaseIE):
     }]
 
     def _real_extract(self, url):
-        path, video_id = self._match_valid_url(url).groups()
+        path, video_id = self._match_valid_url(url).group('path', 'id')
         webpage = self._download_webpage(url, video_id)
 
         media_url = self._extract_js_var(webpage, 'main_url', path)
@@ -106,3 +109,42 @@ class LRTVODIE(LRTBaseIE):
         }
 
         return merge_dicts(clean_info, jw_data, json_ld_data)
+
+
+class LRTRadioIE(LRTBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?lrt\.lt/radioteka/irasas/(?P<id>\d+)/(?P<path>[^?#/]+)'
+    _TESTS = [{
+        # m3u8 download
+        'url': 'https://www.lrt.lt/radioteka/irasas/2000359728/nemarios-eiles-apie-pragarus-ir-skaistyklas-su-aiste-kiltinaviciute',
+        'info_dict': {
+            'id': '2000359728',
+            'ext': 'm4a',
+            'title': 'Nemarios eilės: apie pragarus ir skaistyklas su Aiste Kiltinavičiūte',
+            'description': 'md5:5eee9a0e86a55bf547bd67596204625d',
+            'timestamp': 1726143120,
+            'upload_date': '20240912',
+            'tags': 'count:5',
+            'thumbnail': r're:https?://.+/.+\.jpe?g',
+            'categories': ['Daiktiniai įrodymai'],
+        },
+    }, {
+        'url': 'https://www.lrt.lt/radioteka/irasas/2000304654/vakaras-su-knyga-svetlana-aleksijevic-cernobylio-malda-v-dalis?season=%2Fmediateka%2Faudio%2Fvakaras-su-knyga%2F2023',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        video_id, path = self._match_valid_url(url).group('id', 'path')
+        media = self._download_json(
+            'https://www.lrt.lt/radioteka/api/media', video_id,
+            query={'url': f'/mediateka/irasas/{video_id}/{path}'})
+
+        return traverse_obj(media, {
+            'id': ('id', {int}, {str_or_none}),
+            'title': ('title', {str}),
+            'tags': ('tags', ..., 'name', {str}),
+            'categories': ('playlist_item', 'category', {str}, filter, all, filter),
+            'description': ('content', {clean_html}, {str}),
+            'timestamp': ('date', {lambda x: x.replace('.', '/')}, {unified_timestamp}),
+            'thumbnail': ('playlist_item', 'image', {urljoin('https://www.lrt.lt')}),
+            'formats': ('playlist_item', 'file', {lambda x: self._extract_m3u8_formats(x, video_id)}),
+        })

yt_dlp/extractor/microsoftembed.py

@@ -4,6 +4,7 @@ from .common import InfoExtractor
 from ..utils import (
     int_or_none,
     parse_iso8601,
+    parse_resolution,
     traverse_obj,
     unified_timestamp,
     url_basename,
@@ -83,8 +84,8 @@ class MicrosoftMediusBaseIE(InfoExtractor):
             subtitles.setdefault(sub.pop('tag', 'und'), []).append(sub)
         return subtitles
 
-    def _extract_ism(self, ism_url, video_id):
-        formats = self._extract_ism_formats(ism_url, video_id)
+    def _extract_ism(self, ism_url, video_id, fatal=True):
+        formats = self._extract_ism_formats(ism_url, video_id, fatal=fatal)
         for fmt in formats:
             if fmt['language'] != 'eng' and 'English' not in fmt['format_id']:
                 fmt['language_preference'] = -10
@@ -218,9 +219,21 @@ class MicrosoftLearnEpisodeIE(MicrosoftMediusBaseIE):
             'description': 'md5:7bbbfb593d21c2cf2babc3715ade6b88',
             'timestamp': 1676339547,
             'upload_date': '20230214',
-            'thumbnail': r're:https://learn\.microsoft\.com/video/media/.*\.png',
+            'thumbnail': r're:https://learn\.microsoft\.com/video/media/.+\.png',
             'subtitles': 'count:14',
         },
+    }, {
+        'url': 'https://learn.microsoft.com/en-gb/shows/on-demand-instructor-led-training-series/az-900-module-1',
+        'info_dict': {
+            'id': '4fe10f7c-d83c-463b-ac0e-c30a8195e01b',
+            'ext': 'mp4',
+            'title': 'AZ-900 Cloud fundamentals (1 of 6)',
+            'description': 'md5:3c2212ce865e9142f402c766441bd5c9',
+            'thumbnail': r're:https://.+/.+\.jpg',
+            'timestamp': 1706605184,
+            'upload_date': '20240130',
+        },
+        'params': {'format': 'bv[protocol=https]'},
     }]
 
     def _real_extract(self, url):
@@ -230,9 +243,32 @@ class MicrosoftLearnEpisodeIE(MicrosoftMediusBaseIE):
         entry_id = self._html_search_meta('entryId', webpage, 'entryId', fatal=True)
         video_info = self._download_json(
             f'https://learn.microsoft.com/api/video/public/v1/entries/{entry_id}', video_id)
+
+        formats = []
+        if ism_url := traverse_obj(video_info, ('publicVideo', 'adaptiveVideoUrl', {url_or_none})):
+            formats.extend(self._extract_ism(ism_url, video_id, fatal=False))
+        if hls_url := traverse_obj(video_info, ('publicVideo', 'adaptiveVideoHLSUrl', {url_or_none})):
+            formats.extend(self._extract_m3u8_formats(hls_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+        if mpd_url := traverse_obj(video_info, ('publicVideo', 'adaptiveVideoDashUrl', {url_or_none})):
+            formats.extend(self._extract_mpd_formats(mpd_url, video_id, mpd_id='dash', fatal=False))
+        for key in ('low', 'medium', 'high'):
+            if video_url := traverse_obj(video_info, ('publicVideo', f'{key}QualityVideoUrl', {url_or_none})):
+                formats.append({
+                    'url': video_url,
+                    'format_id': f'video-http-{key}',
+                    'acodec': 'none',
+                    **parse_resolution(video_url),
+                })
+        if audio_url := traverse_obj(video_info, ('publicVideo', 'audioUrl', {url_or_none})):
+            formats.append({
+                'url': audio_url,
+                'format_id': 'audio-http',
+                'vcodec': 'none',
+            })
+
         return {
             'id': entry_id,
-            'formats': self._extract_ism(video_info['publicVideo']['adaptiveVideoUrl'], video_id),
+            'formats': formats,
             'subtitles': self._sub_to_dict(traverse_obj(video_info, (
                 'publicVideo', 'captions', lambda _, v: url_or_none(v['url']), {
                     'tag': ('language', {str}),

yt_dlp/extractor/mixcloud.py

@@ -10,7 +10,9 @@ from ..utils import (
     parse_iso8601,
     strip_or_none,
     try_get,
+    url_or_none,
 )
+from ..utils.traversal import traverse_obj
 
 
 class MixcloudBaseIE(InfoExtractor):
@@ -37,7 +39,7 @@ class MixcloudIE(MixcloudBaseIE):
             'ext': 'm4a',
             'title': 'Cryptkeeper',
             'description': 'After quite a long silence from myself, finally another Drum\'n\'Bass mix with my favourite current dance floor bangers.',
-            'uploader': 'Daniel Holbach',
+            'uploader': 'dholbach',
             'uploader_id': 'dholbach',
             'thumbnail': r're:https?://.*\.jpg',
             'view_count': int,
@@ -46,10 +48,11 @@ class MixcloudIE(MixcloudBaseIE):
             'uploader_url': 'https://www.mixcloud.com/dholbach/',
             'artist': 'Submorphics & Chino , Telekinesis, Porter Robinson, Enei, Breakage ft Jess Mills',
             'duration': 3723,
-            'tags': [],
+            'tags': ['liquid drum and bass', 'drum and bass'],
             'comment_count': int,
             'repost_count': int,
             'like_count': int,
+            'artists': list,
         },
         'params': {'skip_download': 'm3u8'},
     }, {
@@ -67,7 +70,7 @@ class MixcloudIE(MixcloudBaseIE):
             'upload_date': '20150203',
             'uploader_url': 'https://www.mixcloud.com/gillespeterson/',
             'duration': 2992,
-            'tags': [],
+            'tags': ['jazz', 'soul', 'world music', 'funk'],
             'comment_count': int,
             'repost_count': int,
             'like_count': int,
@@ -149,8 +152,6 @@ class MixcloudIE(MixcloudBaseIE):
         elif reason:
             raise ExtractorError('Track is restricted', expected=True)
 
-        title = cloudcast['name']
-
         stream_info = cloudcast['streamInfo']
         formats = []
 
@@ -182,47 +183,39 @@ class MixcloudIE(MixcloudBaseIE):
             self.raise_login_required(metadata_available=True)
 
         comments = []
-        for edge in (try_get(cloudcast, lambda x: x['comments']['edges']) or []):
-            node = edge.get('node') or {}
+        for node in traverse_obj(cloudcast, ('comments', 'edges', ..., 'node', {dict})):
             text = strip_or_none(node.get('comment'))
             if not text:
                 continue
-            user = node.get('user') or {}
             comments.append({
-                'author': user.get('displayName'),
-                'author_id': user.get('username'),
                 'text': text,
-                'timestamp': parse_iso8601(node.get('created')),
+                **traverse_obj(node, {
+                    'author': ('user', 'displayName', {str}),
+                    'author_id': ('user', 'username', {str}),
+                    'timestamp': ('created', {parse_iso8601}),
+                }),
             })
 
-        tags = []
-        for t in cloudcast.get('tags'):
-            tag = try_get(t, lambda x: x['tag']['name'], str)
-            if not tag:
-                tags.append(tag)
-
-        get_count = lambda x: int_or_none(try_get(cloudcast, lambda y: y[x]['totalCount']))
-
-        owner = cloudcast.get('owner') or {}
-
         return {
             'id': track_id,
-            'title': title,
             'formats': formats,
-            'description': cloudcast.get('description'),
-            'thumbnail': try_get(cloudcast, lambda x: x['picture']['url'], str),
-            'uploader': owner.get('displayName'),
-            'timestamp': parse_iso8601(cloudcast.get('publishDate')),
-            'uploader_id': owner.get('username'),
-            'uploader_url': owner.get('url'),
-            'duration': int_or_none(cloudcast.get('audioLength')),
-            'view_count': int_or_none(cloudcast.get('plays')),
-            'like_count': get_count('favorites'),
-            'repost_count': get_count('reposts'),
-            'comment_count': get_count('comments'),
             'comments': comments,
-            'tags': tags,
-            'artist': ', '.join(cloudcast.get('featuringArtistList') or []) or None,
+            **traverse_obj(cloudcast, {
+                'title': ('name', {str}),
+                'description': ('description', {str}),
+                'thumbnail': ('picture', 'url', {url_or_none}),
+                'timestamp': ('publishDate', {parse_iso8601}),
+                'duration': ('audioLength', {int_or_none}),
+                'uploader': ('owner', 'displayName', {str}),
+                'uploader_id': ('owner', 'username', {str}),
+                'uploader_url': ('owner', 'url', {url_or_none}),
+                'view_count': ('plays', {int_or_none}),
+                'like_count': ('favorites', 'totalCount', {int_or_none}),
+                'repost_count': ('reposts', 'totalCount', {int_or_none}),
+                'comment_count': ('comments', 'totalCount', {int_or_none}),
+                'tags': ('tags', ..., 'tag', 'name', {str}, filter, all, filter),
+                'artists': ('featuringArtistList', ..., {str}, filter, all, filter),
+            }),
         }
 
 
@@ -295,7 +288,7 @@ class MixcloudUserIE(MixcloudPlaylistBaseIE):
         'url': 'http://www.mixcloud.com/dholbach/',
         'info_dict': {
             'id': 'dholbach_uploads',
-            'title': 'Daniel Holbach (uploads)',
+            'title': 'dholbach (uploads)',
             'description': 'md5:a3f468a60ac8c3e1f8616380fc469b2b',
         },
         'playlist_mincount': 36,
@@ -303,7 +296,7 @@ class MixcloudUserIE(MixcloudPlaylistBaseIE):
         'url': 'http://www.mixcloud.com/dholbach/uploads/',
         'info_dict': {
             'id': 'dholbach_uploads',
-            'title': 'Daniel Holbach (uploads)',
+            'title': 'dholbach (uploads)',
             'description': 'md5:a3f468a60ac8c3e1f8616380fc469b2b',
         },
         'playlist_mincount': 36,
@@ -311,7 +304,7 @@ class MixcloudUserIE(MixcloudPlaylistBaseIE):
         'url': 'http://www.mixcloud.com/dholbach/favorites/',
         'info_dict': {
             'id': 'dholbach_favorites',
-            'title': 'Daniel Holbach (favorites)',
+            'title': 'dholbach (favorites)',
             'description': 'md5:a3f468a60ac8c3e1f8616380fc469b2b',
         },
         # 'params': {
@@ -337,7 +330,7 @@ class MixcloudUserIE(MixcloudPlaylistBaseIE):
             'title': 'First Ear (stream)',
             'description': 'we maraud for ears',
         },
-        'playlist_mincount': 269,
+        'playlist_mincount': 267,
     }]
 
     _TITLE_KEY = 'displayName'
@@ -361,7 +354,7 @@ class MixcloudPlaylistIE(MixcloudPlaylistBaseIE):
             'id': 'maxvibes_jazzcat-on-ness-radio',
             'title': 'Ness Radio sessions',
         },
-        'playlist_mincount': 59,
+        'playlist_mincount': 58,
     }]
     _TITLE_KEY = 'name'
     _DESCRIPTION_KEY = 'description'

yt_dlp/extractor/mlb.py

@@ -449,9 +449,7 @@ mutation initPlaybackSession(
 
         if not (m3u8_url and token):
             errors = '; '.join(traverse_obj(response, ('errors', ..., 'message', {str})))
-            if 'not entitled' in errors:
-                raise ExtractorError(errors, expected=True)
-            elif errors:  # Only warn when 'blacked out' since radio formats are available
+            if errors:  # Only warn when 'blacked out' or 'not entitled'; radio formats may be available
                 self.report_warning(f'API returned errors for {format_id}: {errors}')
             else:
                 self.report_warning(f'No formats available for {format_id} broadcast; skipping')

yt_dlp/extractor/niconico.py

@@ -27,6 +27,7 @@ from ..utils import (
     traverse_obj,
     try_get,
     unescapeHTML,
+    unified_timestamp,
     update_url_query,
     url_basename,
     url_or_none,
@@ -985,6 +986,7 @@ class NiconicoLiveIE(InfoExtractor):
                     'quality': 'abr',
                     'protocol': 'hls+fmp4',
                     'latency': latency,
+                    'accessRightMethod': 'single_cookie',
                     'chasePlay': False,
                 },
                 'room': {
@@ -1005,6 +1007,7 @@ class NiconicoLiveIE(InfoExtractor):
             if data.get('type') == 'stream':
                 m3u8_url = data['data']['uri']
                 qualities = data['data']['availableQualities']
+                cookies = data['data']['cookies']
                 break
             elif data.get('type') == 'disconnect':
                 self.write_debug(recv)
@@ -1043,6 +1046,11 @@ class NiconicoLiveIE(InfoExtractor):
                     **res,
                 })
 
+        for cookie in cookies:
+            self._set_cookie(
+                cookie['domain'], cookie['name'], cookie['value'],
+                expire_time=unified_timestamp(cookie['expires']), path=cookie['path'], secure=cookie['secure'])
+
         formats = self._extract_m3u8_formats(m3u8_url, video_id, ext='mp4', live=True)
         for fmt, q in zip(formats, reversed(qualities[1:])):
             fmt.update({

yt_dlp/extractor/on24.py

@@ -11,12 +11,15 @@ class On24IE(InfoExtractor):
     IE_NAME = 'on24'
     IE_DESC = 'ON24'
 
-    _VALID_URL = r'''(?x)
-                    https?://event\.on24\.com/(?:
-                        wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
-                        eventRegistration/(?:console/EventConsoleApollo|EventLobbyServlet\?target=lobby30)
-                            \.jsp\?(?:[^/#?]*&)?eventid=(?P<id_2>\d{7})[^/#?]*&key=(?P<key_2>[0-9A-F]{32})
-                    )'''
+    _ID_RE = r'(?P<id>\d{7})'
+    _KEY_RE = r'(?P<key>[0-9A-F]{32})'
+    _URL_BASE_RE = r'https?://event\.on24\.com'
+    _URL_QUERY_RE = rf'(?:[^#]*&)?eventid={_ID_RE}&(?:[^#]+&)?key={_KEY_RE}'
+    _VALID_URL = [
+        rf'{_URL_BASE_RE}/wcc/r/{_ID_RE}/{_KEY_RE}',
+        rf'{_URL_BASE_RE}/eventRegistration/console/(?:EventConsoleApollo\.jsp|apollox/mainEvent/?)\?{_URL_QUERY_RE}',
+        rf'{_URL_BASE_RE}/eventRegistration/EventLobbyServlet/?\?{_URL_QUERY_RE}',
+    ]
 
     _TESTS = [{
         'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
@@ -34,12 +37,16 @@ class On24IE(InfoExtractor):
     }, {
         'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
         'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/EventLobbyServlet?target=reg20.jsp&eventid=3543176&key=BC0F6B968B67C34B50D461D40FDB3E18&groupId=3143628',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/apollox/mainEvent?&eventid=4843671&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=4EAC9B5C564CC98FF29E619B06A2F743&newConsole=true&nxChe=true&newTabCon=true&consoleEarEventConsole=false&consoleEarCloudApi=false&text_language_id=en&playerwidth=748&playerheight=526&referrer=https%3A%2F%2Fevent.on24.com%2Finterface%2Fregistration%2Fautoreg%2Findex.html%3Fsessionid%3D1%26eventid%3D4843671%26key%3D4EAC9B5C564CC98FF29E619B06A2F743%26email%3D000a3e42-7952-4dd6-8f8a-34c38ea3cf02%2540platform%26firstname%3Ds%26lastname%3Ds%26deletecookie%3Dtrue%26event_email%3DN%26marketing_email%3DN%26std1%3D0642572014177%26std2%3D0642572014179%26std3%3D550165f7-a44e-4725-9fe6-716f89908c2b%26std4%3D0&eventuserid=745776448&contenttype=A&mediametricsessionid=640613707&mediametricid=6810717&usercd=745776448&mode=launch',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        mobj = self._match_valid_url(url)
-        event_id = mobj.group('id_1') or mobj.group('id_2')
-        event_key = mobj.group('key_1') or mobj.group('key_2')
+        event_id, event_key = self._match_valid_url(url).group('id', 'key')
 
         event_data = self._download_json(
             'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',

yt_dlp/extractor/parti.py

@@ -0,0 +1,101 @@
+from .common import InfoExtractor
+from ..utils import UserNotLive, int_or_none, parse_iso8601, url_or_none, urljoin
+from ..utils.traversal import traverse_obj
+
+
+class PartiBaseIE(InfoExtractor):
+    def _call_api(self, path, video_id, note=None):
+        return self._download_json(
+            f'https://api-backend.parti.com/parti_v2/profile/{path}', video_id, note)
+
+
+class PartiVideoIE(PartiBaseIE):
+    IE_NAME = 'parti:video'
+    _VALID_URL = r'https?://(?:www\.)?parti\.com/video/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://parti.com/video/66284',
+        'info_dict': {
+            'id': '66284',
+            'ext': 'mp4',
+            'title': 'NOW LIVE ',
+            'upload_date': '20250327',
+            'categories': ['Gaming'],
+            'thumbnail': 'https://assets.parti.com/351424_eb9e5250-2821-484a-9c5f-ca99aa666c87.png',
+            'channel': 'ItZTMGG',
+            'timestamp': 1743044379,
+        },
+        'params': {'skip_download': 'm3u8'},
+    }]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        data = self._call_api(f'get_livestream_channel_info/recent/{video_id}', video_id)
+
+        return {
+            'id': video_id,
+            'formats': self._extract_m3u8_formats(
+                urljoin('https://watch.parti.com', data['livestream_recording']), video_id, 'mp4'),
+            **traverse_obj(data, {
+                'title': ('event_title', {str}),
+                'channel': ('user_name', {str}),
+                'thumbnail': ('event_file', {url_or_none}),
+                'categories': ('category_name', {str}, filter, all),
+                'timestamp': ('event_start_ts', {int_or_none}),
+            }),
+        }
+
+
+class PartiLivestreamIE(PartiBaseIE):
+    IE_NAME = 'parti:livestream'
+    _VALID_URL = r'https?://(?:www\.)?parti\.com/creator/(?P<service>[\w]+)/(?P<id>[\w/-]+)'
+    _TESTS = [{
+        'url': 'https://parti.com/creator/parti/Capt_Robs_Adventures',
+        'info_dict': {
+            'id': 'Capt_Robs_Adventures',
+            'ext': 'mp4',
+            'title': r"re:I'm Live on Parti \d{4}-\d{2}-\d{2} \d{2}:\d{2}",
+            'view_count': int,
+            'thumbnail': r're:https://assets\.parti\.com/.+\.png',
+            'timestamp': 1743879776,
+            'upload_date': '20250405',
+            'live_status': 'is_live',
+        },
+        'params': {'skip_download': 'm3u8'},
+    }, {
+        'url': 'https://parti.com/creator/discord/sazboxgaming/0',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        service, creator_slug = self._match_valid_url(url).group('service', 'id')
+
+        encoded_creator_slug = creator_slug.replace('/', '%23')
+        creator_id = self._call_api(
+            f'get_user_by_social_media/{service}/{encoded_creator_slug}',
+            creator_slug, note='Fetching user ID')
+
+        data = self._call_api(
+            f'get_livestream_channel_info/{creator_id}', creator_id,
+            note='Fetching user profile feed')['channel_info']
+
+        if not traverse_obj(data, ('channel', 'is_live', {bool})):
+            raise UserNotLive(video_id=creator_id)
+
+        channel_info = data['channel']
+
+        return {
+            'id': creator_slug,
+            'formats': self._extract_m3u8_formats(
+                channel_info['playback_url'], creator_slug, live=True, query={
+                    'token': channel_info['playback_auth_token'],
+                    'player_version': '1.17.0',
+                }),
+            'is_live': True,
+            **traverse_obj(data, {
+                'title': ('livestream_event_info', 'event_name', {str}),
+                'description': ('livestream_event_info', 'event_description', {str}),
+                'thumbnail': ('livestream_event_info', 'livestream_preview_file', {url_or_none}),
+                'timestamp': ('stream', 'start_time', {parse_iso8601}),
+                'view_count': ('stream', 'viewer_count', {int_or_none}),
+            }),
+        }

yt_dlp/extractor/roya.py

@@ -0,0 +1,43 @@
+from .common import InfoExtractor
+from ..utils.traversal import traverse_obj
+
+
+class RoyaLiveIE(InfoExtractor):
+    _VALID_URL = r'https?://roya\.tv/live-stream/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://roya.tv/live-stream/1',
+        'info_dict': {
+            'id': '1',
+            'title': r're:Roya TV \d{4}-\d{2}-\d{2} \d{2}:\d{2}',
+            'ext': 'mp4',
+            'live_status': 'is_live',
+        },
+    }, {
+        'url': 'https://roya.tv/live-stream/21',
+        'info_dict': {
+            'id': '21',
+            'title': r're:Roya News \d{4}-\d{2}-\d{2} \d{2}:\d{2}',
+            'ext': 'mp4',
+            'live_status': 'is_live',
+        },
+    }, {
+        'url': 'https://roya.tv/live-stream/10000',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        media_id = self._match_id(url)
+
+        stream_url = self._download_json(
+            f'https://ticket.roya-tv.com/api/v5/fastchannel/{media_id}', media_id)['data']['secured_url']
+
+        title = traverse_obj(
+            self._download_json('https://backend.roya.tv/api/v01/channels/schedule-pagination', media_id, fatal=False),
+            ('data', 0, 'channel', lambda _, v: str(v['id']) == media_id, 'title', {str}, any))
+
+        return {
+            'id': media_id,
+            'formats': self._extract_m3u8_formats(stream_url, media_id, 'mp4', m3u8_id='hls', live=True),
+            'title': title,
+            'is_live': True,
+        }

yt_dlp/extractor/rtvs.py

@@ -9,7 +9,9 @@ from ..utils import (
 
 
 class RTVSIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?rtvs\.sk/(?:radio|televizia)/archiv(?:/\d+)?/(?P<id>\d+)/?(?:[#?]|$)'
+    IE_NAME = 'stvr'
+    IE_DESC = 'Slovak Television and Radio (formerly RTVS)'
+    _VALID_URL = r'https?://(?:www\.)?(?:rtvs|stvr)\.sk/(?:radio|televizia)/archiv(?:/\d+)?/(?P<id>\d+)/?(?:[#?]|$)'
     _TESTS = [{
         # radio archive
         'url': 'http://www.rtvs.sk/radio/archiv/11224/414872',
@@ -19,7 +21,7 @@ class RTVSIE(InfoExtractor):
             'ext': 'mp3',
             'title': 'Ostrov pokladov 1 časť.mp3',
             'duration': 2854,
-            'thumbnail': 'https://www.rtvs.sk/media/a501/image/file/2/0000/b1R8.rtvs.jpg',
+            'thumbnail': 'https://www.stvr.sk/media/a501/image/file/2/0000/rtvs-00009383.png',
             'display_id': '135331',
         },
     }, {
@@ -30,7 +32,7 @@ class RTVSIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Amaro Džives - Náš deň',
             'description': 'Galavečer pri príležitosti Medzinárodného dňa Rómov.',
-            'thumbnail': 'https://www.rtvs.sk/media/a501/image/file/2/0031/L7Qm.amaro_dzives_png.jpg',
+            'thumbnail': 'https://www.stvr.sk/media/a501/image/file/2/0031/L7Qm.amaro_dzives_png.jpg',
             'timestamp': 1428555900,
             'upload_date': '20150409',
             'duration': 4986,
@@ -47,8 +49,11 @@ class RTVSIE(InfoExtractor):
             'display_id': '307655',
             'duration': 831,
             'upload_date': '20211111',
-            'thumbnail': 'https://www.rtvs.sk/media/a501/image/file/2/0916/robin.jpg',
+            'thumbnail': 'https://www.stvr.sk/media/a501/image/file/2/0916/robin.jpg',
         },
+    }, {
+        'url': 'https://www.stvr.sk/radio/archiv/11224/414872',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):

yt_dlp/extractor/rumble.py

@@ -7,7 +7,6 @@ from ..utils import (
     ExtractorError,
     UnsupportedError,
     clean_html,
-    determine_ext,
     extract_attributes,
     format_field,
     get_element_by_class,
@@ -36,7 +35,7 @@ class RumbleEmbedIE(InfoExtractor):
             'upload_date': '20191020',
             'channel_url': 'https://rumble.com/c/WMAR',
             'channel': 'WMAR',
-            'thumbnail': 'https://sp.rmbl.ws/s8/1/5/M/z/1/5Mz1a.qR4e-small-WMAR-2-News-Latest-Headline.jpg',
+            'thumbnail': r're:https://.+\.jpg',
             'duration': 234,
             'uploader': 'WMAR',
             'live_status': 'not_live',
@@ -52,7 +51,7 @@ class RumbleEmbedIE(InfoExtractor):
             'upload_date': '20220217',
             'channel_url': 'https://rumble.com/c/CyberTechNews',
             'channel': 'CTNews',
-            'thumbnail': 'https://sp.rmbl.ws/s8/6/7/i/9/h/7i9hd.OvCc.jpg',
+            'thumbnail': r're:https://.+\.jpg',
             'duration': 901,
             'uploader': 'CTNews',
             'live_status': 'not_live',
@@ -114,6 +113,22 @@ class RumbleEmbedIE(InfoExtractor):
             'live_status': 'was_live',
         },
         'params': {'skip_download': True},
+    }, {
+        'url': 'https://rumble.com/embed/v6pezdb',
+        'info_dict': {
+            'id': 'v6pezdb',
+            'ext': 'mp4',
+            'title': '"Es war einmal ein Mädchen" – Ein filmisches Zeitzeugnis aus Leningrad 1944',
+            'uploader': 'RT DE',
+            'channel': 'RT DE',
+            'channel_url': 'https://rumble.com/c/RTDE',
+            'duration': 309,
+            'thumbnail': 'https://1a-1791.com/video/fww1/dc/s8/1/n/z/2/y/nz2yy.qR4e-small-Es-war-einmal-ein-Mdchen-Ei.jpg',
+            'timestamp': 1743703500,
+            'upload_date': '20250403',
+            'live_status': 'not_live',
+        },
+        'params': {'skip_download': True},
     }, {
         'url': 'https://rumble.com/embed/ufe9n.v5pv5f',
         'only_matching': True,
@@ -168,40 +183,42 @@ class RumbleEmbedIE(InfoExtractor):
             live_status = None
 
         formats = []
-        for ext, ext_info in (video.get('ua') or {}).items():
-            if isinstance(ext_info, dict):
-                for height, video_info in ext_info.items():
+        for format_type, format_info in (video.get('ua') or {}).items():
+            if isinstance(format_info, dict):
+                for height, video_info in format_info.items():
                     if not traverse_obj(video_info, ('meta', 'h', {int_or_none})):
                         video_info.setdefault('meta', {})['h'] = height
-                ext_info = ext_info.values()
+                format_info = format_info.values()
 
-            for video_info in ext_info:
+            for video_info in format_info:
                 meta = video_info.get('meta') or {}
                 if not video_info.get('url'):
                     continue
-                if ext == 'hls':
+                # With default query params returns m3u8 variants which are duplicates, without returns tar files
+                if format_type == 'tar':
+                    continue
+                if format_type == 'hls':
                     if meta.get('live') is True and video.get('live') == 1:
                         live_status = 'post_live'
                     formats.extend(self._extract_m3u8_formats(
                         video_info['url'], video_id,
                         ext='mp4', m3u8_id='hls', fatal=False, live=live_status == 'is_live'))
                     continue
-                timeline = ext == 'timeline'
-                if timeline:
-                    ext = determine_ext(video_info['url'])
+                is_timeline = format_type == 'timeline'
+                is_audio = format_type == 'audio'
                 formats.append({
-                    'ext': ext,
-                    'acodec': 'none' if timeline else None,
+                    'acodec': 'none' if is_timeline else None,
+                    'vcodec': 'none' if is_audio else None,
                     'url': video_info['url'],
-                    'format_id': join_nonempty(ext, format_field(meta, 'h', '%sp')),
-                    'format_note': 'Timeline' if timeline else None,
-                    'fps': None if timeline else video.get('fps'),
+                    'format_id': join_nonempty(format_type, format_field(meta, 'h', '%sp')),
+                    'format_note': 'Timeline' if is_timeline else None,
+                    'fps': None if is_timeline or is_audio else video.get('fps'),
                     **traverse_obj(meta, {
-                        'tbr': 'bitrate',
-                        'filesize': 'size',
-                        'width': 'w',
-                        'height': 'h',
-                    }, expected_type=lambda x: int(x) or None),
+                        'tbr': ('bitrate', {int_or_none}),
+                        'filesize': ('size', {int_or_none}),
+                        'width': ('w', {int_or_none}),
+                        'height': ('h', {int_or_none}),
+                    }),
                 })
 
         subtitles = {

yt_dlp/extractor/sbs.py

@@ -122,6 +122,15 @@ class SBSIE(InfoExtractor):
         if traverse_obj(media, ('partOfSeries', {dict})):
             media['epName'] = traverse_obj(media, ('title', {str}))
 
+        # Need to set different language for forced subs or else they have priority over full subs
+        fixed_subtitles = {}
+        for lang, subs in subtitles.items():
+            for sub in subs:
+                fixed_lang = lang
+                if sub['url'].lower().endswith('_fe.vtt'):
+                    fixed_lang += '-forced'
+                fixed_subtitles.setdefault(fixed_lang, []).append(sub)
+
         return {
             'id': video_id,
             **traverse_obj(media, {
@@ -151,6 +160,6 @@ class SBSIE(InfoExtractor):
                 }),
             }),
             'formats': formats,
-            'subtitles': subtitles,
+            'subtitles': fixed_subtitles,
             'uploader': 'SBSC',
         }

yt_dlp/extractor/twitch.py

@@ -14,19 +14,20 @@ from ..utils import (
     dict_get,
     float_or_none,
     int_or_none,
+    join_nonempty,
     make_archive_id,
     parse_duration,
     parse_iso8601,
     parse_qs,
     qualities,
     str_or_none,
-    traverse_obj,
     try_get,
     unified_timestamp,
     update_url_query,
     url_or_none,
     urljoin,
 )
+from ..utils.traversal import traverse_obj, value
 
 
 class TwitchBaseIE(InfoExtractor):
@@ -42,10 +43,10 @@ class TwitchBaseIE(InfoExtractor):
         'CollectionSideBar': '27111f1b382effad0b6def325caef1909c733fe6a4fbabf54f8d491ef2cf2f14',
         'FilterableVideoTower_Videos': 'a937f1d22e269e39a03b509f65a7490f9fc247d7f83d6ac1421523e3b68042cb',
         'ClipsCards__User': 'b73ad2bfaecfd30a9e6c28fada15bd97032c83ec77a0440766a56fe0bd632777',
+        'ShareClipRenderStatus': 'f130048a462a0ac86bb54d653c968c514e9ab9ca94db52368c1179e97b0f16eb',
         'ChannelCollectionsContent': '447aec6a0cc1e8d0a8d7732d47eb0762c336a2294fdb009e9c9d854e49d484b9',
         'StreamMetadata': 'a647c2a13599e5991e175155f798ca7f1ecddde73f7f341f39009c14dbf59962',
         'ComscoreStreamingQuery': 'e1edae8122517d013405f237ffcc124515dc6ded82480a88daef69c83b53ac01',
-        'VideoAccessToken_Clip': '36b89d2507fce29e5ca551df756d27c1cfe079e2609642b4390aa4c35796eb11',
         'VideoPreviewOverlay': '3006e77e51b128d838fa4e835723ca4dc9a05c5efd4466c1085215c6e437e65c',
         'VideoMetadata': '49b5b8f268cdeb259d75b58dcb0c1a748e3b575003448a2333dc5cdafd49adad',
         'VideoPlayer_ChapterSelectButtonVideo': '8d2793384aac3773beab5e59bd5d6f585aedb923d292800119e03d40cd0f9b41',
@@ -1083,16 +1084,44 @@ class TwitchClipsIE(TwitchBaseIE):
         'url': 'https://clips.twitch.tv/FaintLightGullWholeWheat',
         'md5': '761769e1eafce0ffebfb4089cb3847cd',
         'info_dict': {
-            'id': '42850523',
+            'id': '396245304',
             'display_id': 'FaintLightGullWholeWheat',
             'ext': 'mp4',
             'title': 'EA Play 2016 Live from the Novo Theatre',
+            'duration': 32,
+            'view_count': int,
             'thumbnail': r're:^https?://.*\.jpg',
             'timestamp': 1465767393,
             'upload_date': '20160612',
-            'creator': 'EA',
-            'uploader': 'stereotype_',
-            'uploader_id': '43566419',
+            'creators': ['EA'],
+            'channel': 'EA',
+            'channel_id': '25163635',
+            'channel_is_verified': False,
+            'channel_follower_count': int,
+            'uploader': 'EA',
+            'uploader_id': '25163635',
+        },
+    }, {
+        'url': 'https://www.twitch.tv/xqc/clip/CulturedAmazingKuduDatSheffy-TiZ_-ixAGYR3y2Uy',
+        'md5': 'e90fe616b36e722a8cfa562547c543f0',
+        'info_dict': {
+            'id': '3207364882',
+            'display_id': 'CulturedAmazingKuduDatSheffy-TiZ_-ixAGYR3y2Uy',
+            'ext': 'mp4',
+            'title': 'A day in the life of xQc',
+            'duration': 60,
+            'view_count': int,
+            'thumbnail': r're:^https?://.*\.jpg',
+            'timestamp': 1742869615,
+            'upload_date': '20250325',
+            'creators': ['xQc'],
+            'channel': 'xQc',
+            'channel_id': '71092938',
+            'channel_is_verified': True,
+            'channel_follower_count': int,
+            'uploader': 'xQc',
+            'uploader_id': '71092938',
+            'categories': ['Just Chatting'],
         },
     }, {
         # multiple formats
@@ -1116,16 +1145,14 @@ class TwitchClipsIE(TwitchBaseIE):
     }]
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
+        slug = self._match_id(url)
 
         clip = self._download_gql(
-            video_id, [{
-                'operationName': 'VideoAccessToken_Clip',
-                'variables': {
-                    'slug': video_id,
-                },
+            slug, [{
+                'operationName': 'ShareClipRenderStatus',
+                'variables': {'slug': slug},
             }],
-            'Downloading clip access token GraphQL')[0]['data']['clip']
+            'Downloading clip GraphQL')[0]['data']['clip']
 
         if not clip:
             raise ExtractorError(
@@ -1135,81 +1162,71 @@ class TwitchClipsIE(TwitchBaseIE):
             'sig': clip['playbackAccessToken']['signature'],
             'token': clip['playbackAccessToken']['value'],
         }
-
-        data = self._download_base_gql(
-            video_id, {
-                'query': '''{
-  clip(slug: "%s") {
-    broadcaster {
-      displayName
-    }
-    createdAt
-    curator {
-      displayName
-      id
-    }
-    durationSeconds
-    id
-    tiny: thumbnailURL(width: 86, height: 45)
-    small: thumbnailURL(width: 260, height: 147)
-    medium: thumbnailURL(width: 480, height: 272)
-    title
-    videoQualities {
-      frameRate
-      quality
-      sourceURL
-    }
-    viewCount
-  }
-}''' % video_id}, 'Downloading clip GraphQL', fatal=False)  # noqa: UP031
-
-        if data:
-            clip = try_get(data, lambda x: x['data']['clip'], dict) or clip
+        asset_default = traverse_obj(clip, ('assets', 0, {dict})) or {}
+        asset_portrait = traverse_obj(clip, ('assets', 1, {dict})) or {}
 
         formats = []
-        for option in clip.get('videoQualities', []):
-            if not isinstance(option, dict):
-                continue
-            source = url_or_none(option.get('sourceURL'))
-            if not source:
-                continue
+        default_aspect_ratio = float_or_none(asset_default.get('aspectRatio'))
+        formats.extend(traverse_obj(asset_default, ('videoQualities', lambda _, v: url_or_none(v['sourceURL']), {
+            'url': ('sourceURL', {update_url_query(query=access_query)}),
+            'format_id': ('quality', {str}),
+            'height': ('quality', {int_or_none}),
+            'fps': ('frameRate', {float_or_none}),
+            'aspect_ratio': {value(default_aspect_ratio)},
+        })))
+        portrait_aspect_ratio = float_or_none(asset_portrait.get('aspectRatio'))
+        for source in traverse_obj(asset_portrait, ('videoQualities', lambda _, v: url_or_none(v['sourceURL']))):
             formats.append({
-                'url': update_url_query(source, access_query),
-                'format_id': option.get('quality'),
-                'height': int_or_none(option.get('quality')),
-                'fps': int_or_none(option.get('frameRate')),
+                'url': update_url_query(source['sourceURL'], access_query),
+                'format_id': join_nonempty('portrait', source.get('quality')),
+                'height': int_or_none(source.get('quality')),
+                'fps': float_or_none(source.get('frameRate')),
+                'aspect_ratio': portrait_aspect_ratio,
+                'quality': -2,
             })
 
         thumbnails = []
-        for thumbnail_id in ('tiny', 'small', 'medium'):
-            thumbnail_url = clip.get(thumbnail_id)
-            if not thumbnail_url:
-                continue
-            thumb = {
-                'id': thumbnail_id,
-                'url': thumbnail_url,
-            }
-            mobj = re.search(r'-(\d+)x(\d+)\.', thumbnail_url)
-            if mobj:
-                thumb.update({
-                    'height': int(mobj.group(2)),
-                    'width': int(mobj.group(1)),
-                })
-            thumbnails.append(thumb)
+        thumb_asset_default_url = url_or_none(asset_default.get('thumbnailURL'))
+        if thumb_asset_default_url:
+            thumbnails.append({
+                'id': 'default',
+                'url': thumb_asset_default_url,
+                'preference': 0,
+            })
+        if thumb_asset_portrait_url := url_or_none(asset_portrait.get('thumbnailURL')):
+            thumbnails.append({
+                'id': 'portrait',
+                'url': thumb_asset_portrait_url,
+                'preference': -1,
+            })
+        thumb_default_url = url_or_none(clip.get('thumbnailURL'))
+        if thumb_default_url and thumb_default_url != thumb_asset_default_url:
+            thumbnails.append({
+                'id': 'small',
+                'url': thumb_default_url,
+                'preference': -2,
+            })
 
         old_id = self._search_regex(r'%7C(\d+)(?:-\d+)?.mp4', formats[-1]['url'], 'old id', default=None)
 
         return {
-            'id': clip.get('id') or video_id,
+            'id': clip.get('id') or slug,
             '_old_archive_ids': [make_archive_id(self, old_id)] if old_id else None,
-            'display_id': video_id,
-            'title': clip.get('title'),
+            'display_id': slug,
             'formats': formats,
-            'duration': int_or_none(clip.get('durationSeconds')),
-            'view_count': int_or_none(clip.get('viewCount')),
-            'timestamp': unified_timestamp(clip.get('createdAt')),
             'thumbnails': thumbnails,
-            'creator': try_get(clip, lambda x: x['broadcaster']['displayName'], str),
-            'uploader': try_get(clip, lambda x: x['curator']['displayName'], str),
-            'uploader_id': try_get(clip, lambda x: x['curator']['id'], str),
+            **traverse_obj(clip, {
+                'title': ('title', {str}),
+                'duration': ('durationSeconds', {int_or_none}),
+                'view_count': ('viewCount', {int_or_none}),
+                'timestamp': ('createdAt', {parse_iso8601}),
+                'creators': ('broadcaster', 'displayName', {str}, filter, all),
+                'channel': ('broadcaster', 'displayName', {str}),
+                'channel_id': ('broadcaster', 'id', {str}),
+                'channel_follower_count': ('broadcaster', 'followers', 'totalCount', {int_or_none}),
+                'channel_is_verified': ('broadcaster', 'isPartner', {bool}),
+                'uploader': ('broadcaster', 'displayName', {str}),
+                'uploader_id': ('broadcaster', 'id', {str}),
+                'categories': ('game', 'displayName', {str}, filter, all, filter),
+            }),
         }

yt_dlp/extractor/vk.py

@@ -544,7 +544,7 @@ class VKIE(VKBaseIE):
                 'uploader_id': (('author_id', 'authorId'), {str_or_none}, any),
                 'duration': ('duration', {int_or_none}),
                 'chapters': ('time_codes', lambda _, v: isinstance(v['time'], int), {
-                    'title': ('text', {str}),
+                    'title': ('text', {unescapeHTML}),
                     'start_time': 'time',
                 }),
             }),

yt_dlp/extractor/yandexvideo.py

@@ -2,15 +2,17 @@ import itertools
 
 from .common import InfoExtractor
 from ..utils import (
+    bug_reports_message,
     determine_ext,
-    extract_attributes,
     int_or_none,
     lowercase_escape,
     parse_qs,
-    traverse_obj,
+    qualities,
     try_get,
+    update_url_query,
     url_or_none,
 )
+from ..utils.traversal import traverse_obj
 
 
 class YandexVideoIE(InfoExtractor):
@@ -186,7 +188,22 @@ class YandexVideoPreviewIE(InfoExtractor):
         return self.url_result(data_json['video']['url'])
 
 
-class ZenYandexIE(InfoExtractor):
+class ZenYandexBaseIE(InfoExtractor):
+    def _fetch_ssr_data(self, url, video_id):
+        webpage = self._download_webpage(url, video_id)
+        redirect = self._search_json(
+            r'(?:var|let|const)\s+it\s*=', webpage, 'redirect', video_id, default={}).get('retpath')
+        if redirect:
+            video_id = self._match_id(redirect)
+            webpage = self._download_webpage(redirect, video_id, note='Redirecting')
+        return video_id, self._search_json(
+            r'(?:var|let|const)\s+_params\s*=\s*\(', webpage, 'metadata', video_id,
+            contains_pattern=r'{["\']ssrData.+}')['ssrData']
+
+
+class ZenYandexIE(ZenYandexBaseIE):
+    IE_NAME = 'dzen.ru'
+    IE_DESC = 'Дзен (dzen) formerly Яндекс.Дзен (Yandex Zen)'
     _VALID_URL = r'https?://(zen\.yandex|dzen)\.ru(?:/video)?/(media|watch)/(?:(?:id/[^/]+/|[^/]+/)(?:[a-z0-9-]+)-)?(?P<id>[a-z0-9-]+)'
     _TESTS = [{
         'url': 'https://zen.yandex.ru/media/id/606fd806cc13cb3c58c05cf5/vot-eto-focus-dedy-morozy-na-gidrociklah-60c7c443da18892ebfe85ed7',
@@ -216,6 +233,7 @@ class ZenYandexIE(InfoExtractor):
             'timestamp': 1573465585,
         },
         'params': {'skip_download': 'm3u8'},
+        'skip': 'The page does not exist',
     }, {
         'url': 'https://zen.yandex.ru/video/watch/6002240ff8b1af50bb2da5e3',
         'info_dict': {
@@ -227,6 +245,9 @@ class ZenYandexIE(InfoExtractor):
             'uploader': 'TechInsider',
             'timestamp': 1611378221,
             'upload_date': '20210123',
+            'view_count': int,
+            'duration': 243,
+            'tags': ['опыт', 'эксперимент', 'огонь'],
         },
         'params': {'skip_download': 'm3u8'},
     }, {
@@ -240,6 +261,9 @@ class ZenYandexIE(InfoExtractor):
             'uploader': 'TechInsider',
             'upload_date': '20210123',
             'timestamp': 1611378221,
+            'view_count': int,
+            'duration': 243,
+            'tags': ['опыт', 'эксперимент', 'огонь'],
         },
         'params': {'skip_download': 'm3u8'},
     }, {
@@ -252,44 +276,56 @@ class ZenYandexIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        redirect = self._search_json(r'var it\s*=', webpage, 'redirect', id, default={}).get('retpath')
-        if redirect:
-            video_id = self._match_id(redirect)
-            webpage = self._download_webpage(redirect, video_id, note='Redirecting')
-        data_json = self._search_json(
-            r'("data"\s*:|data\s*=)', webpage, 'metadata', video_id, contains_pattern=r'{["\']_*serverState_*video.+}')
-        serverstate = self._search_regex(r'(_+serverState_+video-site_[^_]+_+)', webpage, 'server state')
-        uploader = self._search_regex(r'(<a\s*class=["\']card-channel-link[^"\']+["\'][^>]+>)',
-                                      webpage, 'uploader', default='<a>')
-        uploader_name = extract_attributes(uploader).get('aria-label')
-        item_id = traverse_obj(data_json, (serverstate, 'videoViewer', 'openedItemId', {str}))
-        video_json = traverse_obj(data_json, (serverstate, 'videoViewer', 'items', item_id, {dict})) or {}
+        video_id, ssr_data = self._fetch_ssr_data(url, video_id)
+        video_data = ssr_data['videoMetaResponse']
 
         formats, subtitles = [], {}
-        for s_url in traverse_obj(video_json, ('video', 'streams', ..., {url_or_none})):
+        quality = qualities(('4', '0', '1', '2', '3', '5', '6', '7'))
+        # Deduplicate stream URLs. The "dzen_dash" query parameter is present in some URLs but can be omitted
+        stream_urls = set(traverse_obj(video_data, (
+            'video', ('id', ('streams', ...), ('mp4Streams', ..., 'url'), ('oneVideoStreams', ..., 'url')),
+            {url_or_none}, {update_url_query(query={'dzen_dash': []})})))
+        for s_url in stream_urls:
             ext = determine_ext(s_url)
-            if ext == 'mpd':
-                fmts, subs = self._extract_mpd_formats_and_subtitles(s_url, video_id, mpd_id='dash')
-            elif ext == 'm3u8':
-                fmts, subs = self._extract_m3u8_formats_and_subtitles(s_url, video_id, 'mp4')
+            content_type = traverse_obj(parse_qs(s_url), ('ct', 0))
+            if ext == 'mpd' or content_type == '6':
+                fmts, subs = self._extract_mpd_formats_and_subtitles(s_url, video_id, mpd_id='dash', fatal=False)
+            elif ext == 'm3u8' or content_type == '8':
+                fmts, subs = self._extract_m3u8_formats_and_subtitles(s_url, video_id, 'mp4', m3u8_id='hls', fatal=False)
+            elif content_type == '0':
+                format_type = traverse_obj(parse_qs(s_url), ('type', 0))
+                formats.append({
+                    'url': s_url,
+                    'format_id': format_type,
+                    'ext': 'mp4',
+                    'quality': quality(format_type),
+                })
+                continue
+            else:
+                self.report_warning(f'Unsupported stream URL: {s_url}{bug_reports_message()}')
+                continue
             formats.extend(fmts)
-            subtitles = self._merge_subtitles(subtitles, subs)
+            self._merge_subtitles(subs, target=subtitles)
+
         return {
             'id': video_id,
-            'title': video_json.get('title') or self._og_search_title(webpage),
             'formats': formats,
             'subtitles': subtitles,
-            'duration': int_or_none(video_json.get('duration')),
-            'view_count': int_or_none(video_json.get('views')),
-            'timestamp': int_or_none(video_json.get('publicationDate')),
-            'uploader': uploader_name or data_json.get('authorName') or try_get(data_json, lambda x: x['publisher']['name']),
-            'description': video_json.get('description') or self._og_search_description(webpage),
-            'thumbnail': self._og_search_thumbnail(webpage) or try_get(data_json, lambda x: x['og']['imageUrl']),
+            **traverse_obj(video_data, {
+                'title': ('title', {str}),
+                'description': ('description', {str}),
+                'thumbnail': ('image', {url_or_none}),
+                'duration': ('video', 'duration', {int_or_none}),
+                'view_count': ('video', 'views', {int_or_none}),
+                'timestamp': ('publicationDate', {int_or_none}),
+                'tags': ('tags', ..., {str}),
+                'uploader': ('source', 'title', {str}),
+            }),
         }
 
 
-class ZenYandexChannelIE(InfoExtractor):
+class ZenYandexChannelIE(ZenYandexBaseIE):
+    IE_NAME = 'dzen.ru:channel'
     _VALID_URL = r'https?://(zen\.yandex|dzen)\.ru/(?!media|video)(?:id/)?(?P<id>[a-z0-9-_]+)'
     _TESTS = [{
         'url': 'https://zen.yandex.ru/tok_media',
@@ -323,8 +359,8 @@ class ZenYandexChannelIE(InfoExtractor):
         'url': 'https://zen.yandex.ru/jony_me',
         'info_dict': {
             'id': 'jony_me',
-            'description': 'md5:ce0a5cad2752ab58701b5497835b2cc5',
-            'title': 'JONY ',
+            'description': 'md5:7c30d11dc005faba8826feae99da3113',
+            'title': 'JONY',
         },
         'playlist_count': 18,
     }, {
@@ -333,9 +369,8 @@ class ZenYandexChannelIE(InfoExtractor):
         'url': 'https://zen.yandex.ru/tatyanareva',
         'info_dict': {
             'id': 'tatyanareva',
-            'description': 'md5:40a1e51f174369ec3ba9d657734ac31f',
+            'description': 'md5:92e56fa730a932ca2483ba5c2186ad96',
             'title': 'Татьяна Рева',
-            'entries': 'maxcount:200',
         },
         'playlist_mincount': 46,
     }, {
@@ -348,43 +383,31 @@ class ZenYandexChannelIE(InfoExtractor):
         'playlist_mincount': 657,
     }]
 
-    def _entries(self, item_id, server_state_json, server_settings_json):
-        items = (traverse_obj(server_state_json, ('feed', 'items', ...))
-                 or traverse_obj(server_settings_json, ('exportData', 'items', ...)))
-
-        more = (traverse_obj(server_state_json, ('links', 'more'))
-                or traverse_obj(server_settings_json, ('exportData', 'more', 'link')))
-
+    def _entries(self, feed_data, channel_id):
         next_page_id = None
         for page in itertools.count(1):
-            for item in items or []:
-                if item.get('type') != 'gif':
-                    continue
-                video_id = traverse_obj(item, 'publication_id', 'publicationId') or ''
-                yield self.url_result(item['link'], ZenYandexIE, video_id.split(':')[-1])
+            for item in traverse_obj(feed_data, (
+                (None, ('items', lambda _, v: v['tab'] in ('shorts', 'longs'))),
+                'items', lambda _, v: url_or_none(v['link']),
+            )):
+                yield self.url_result(item['link'], ZenYandexIE, item.get('id'), title=item.get('title'))
 
+            more = traverse_obj(feed_data, ('more', 'link', {url_or_none}))
             current_page_id = next_page_id
             next_page_id = traverse_obj(parse_qs(more), ('next_page_id', -1))
-            if not all((more, items, next_page_id, next_page_id != current_page_id)):
+            if not all((more, next_page_id, next_page_id != current_page_id)):
                 break
 
-            data = self._download_json(more, item_id, note=f'Downloading Page {page}')
-            items, more = data.get('items'), traverse_obj(data, ('more', 'link'))
+            feed_data = self._download_json(more, channel_id, note=f'Downloading Page {page}')
 
     def _real_extract(self, url):
-        item_id = self._match_id(url)
-        webpage = self._download_webpage(url, item_id)
-        redirect = self._search_json(
-            r'var it\s*=', webpage, 'redirect', item_id, default={}).get('retpath')
-        if redirect:
-            item_id = self._match_id(redirect)
-            webpage = self._download_webpage(redirect, item_id, note='Redirecting')
-        data = self._search_json(
-            r'("data"\s*:|data\s*=)', webpage, 'channel data', item_id, contains_pattern=r'{\"__serverState__.+}')
-        server_state_json = traverse_obj(data, lambda k, _: k.startswith('__serverState__'), get_all=False)
-        server_settings_json = traverse_obj(data, lambda k, _: k.startswith('__serverSettings__'), get_all=False)
+        channel_id = self._match_id(url)
+        channel_id, ssr_data = self._fetch_ssr_data(url, channel_id)
+        channel_data = ssr_data['exportResponse']
 
         return self.playlist_result(
-            self._entries(item_id, server_state_json, server_settings_json),
-            item_id, traverse_obj(server_state_json, ('channel', 'source', 'title')),
-            traverse_obj(server_state_json, ('channel', 'source', 'description')))
+            self._entries(channel_data['feedData'], channel_id),
+            channel_id, **traverse_obj(channel_data, ('channel', 'source', {
+                'title': ('title', {str}),
+                'description': ('description', {str}),
+            })))

yt_dlp/extractor/youtube/_base.py

@@ -803,12 +803,14 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
 
     @classmethod
     def _extract_continuation_ep_data(cls, continuation_ep: dict):
-        if isinstance(continuation_ep, dict):
-            continuation = try_get(
-                continuation_ep, lambda x: x['continuationCommand']['token'], str)
+        continuation_commands = traverse_obj(
+            continuation_ep, ('commandExecutorCommand', 'commands', ..., {dict}))
+        continuation_commands.append(continuation_ep)
+        for command in continuation_commands:
+            continuation = traverse_obj(command, ('continuationCommand', 'token', {str}))
             if not continuation:
-                return
-            ctp = continuation_ep.get('clickTrackingParams')
+                continue
+            ctp = command.get('clickTrackingParams')
             return cls._build_api_continuation_query(continuation, ctp)
 
     @classmethod

yt_dlp/extractor/youtube/_tab.py

@@ -524,10 +524,16 @@ class YoutubeTabBaseInfoExtractor(YoutubeBaseInfoExtractor):
             response = self._extract_response(
                 item_id=f'{item_id} page {page_num}',
                 query=continuation, headers=headers, ytcfg=ytcfg,
-                check_get_keys=('continuationContents', 'onResponseReceivedActions', 'onResponseReceivedEndpoints'))
+                check_get_keys=(
+                    'continuationContents', 'onResponseReceivedActions', 'onResponseReceivedEndpoints',
+                    # Playlist recommendations may return with no data - ignore
+                    ('responseContext', 'serviceTrackingParams', ..., 'params', ..., lambda k, v: k == 'key' and v == 'GetRecommendedMusicPlaylists_rid'),
+                ))
 
             if not response:
                 break
+
+            continuation = None
             # Extracting updated visitor data is required to prevent an infinite extraction loop in some cases
             # See: https://github.com/ytdl-org/youtube-dl/issues/28702
             visitor_data = self._extract_visitor_data(response) or visitor_data
@@ -564,7 +570,13 @@ class YoutubeTabBaseInfoExtractor(YoutubeBaseInfoExtractor):
                 yield from func(video_items_renderer)
                 continuation = continuation_list[0] or self._extract_continuation(video_items_renderer)
 
-            if not video_items_renderer:
+            # In the case only a continuation is returned, try to follow it.
+            # We extract this after trying to extract non-continuation items as otherwise this
+            # may be prioritized over other continuations.
+            # see: https://github.com/yt-dlp/yt-dlp/issues/12933
+            continuation = continuation or self._extract_continuation({'contents': [continuation_item]})
+
+            if not continuation and not video_items_renderer:
                 break
 
     @staticmethod
@@ -999,14 +1011,14 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 94,
         'info_dict': {
             'id': 'UCqj7Cz7revf5maW9g5pgNcg',
-            'title': 'Igor Kleiner Ph.D. - Playlists',
+            'title': 'Igor Kleiner  - Playlists',
             'description': 'md5:15d7dd9e333cb987907fcb0d604b233a',
-            'uploader': 'Igor Kleiner Ph.D.',
+            'uploader': 'Igor Kleiner ',
             'uploader_id': '@IgorDataScience',
             'uploader_url': 'https://www.youtube.com/@IgorDataScience',
-            'channel': 'Igor Kleiner Ph.D.',
+            'channel': 'Igor Kleiner ',
             'channel_id': 'UCqj7Cz7revf5maW9g5pgNcg',
-            'tags': ['критическое мышление', 'наука просто', 'математика', 'анализ данных'],
+            'tags': 'count:23',
             'channel_url': 'https://www.youtube.com/channel/UCqj7Cz7revf5maW9g5pgNcg',
             'channel_follower_count': int,
         },
@@ -1016,18 +1028,19 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 94,
         'info_dict': {
             'id': 'UCqj7Cz7revf5maW9g5pgNcg',
-            'title': 'Igor Kleiner Ph.D. - Playlists',
+            'title': 'Igor Kleiner  - Playlists',
             'description': 'md5:15d7dd9e333cb987907fcb0d604b233a',
-            'uploader': 'Igor Kleiner Ph.D.',
+            'uploader': 'Igor Kleiner ',
             'uploader_id': '@IgorDataScience',
             'uploader_url': 'https://www.youtube.com/@IgorDataScience',
-            'tags': ['критическое мышление', 'наука просто', 'математика', 'анализ данных'],
+            'tags': 'count:23',
             'channel_id': 'UCqj7Cz7revf5maW9g5pgNcg',
-            'channel': 'Igor Kleiner Ph.D.',
+            'channel': 'Igor Kleiner ',
             'channel_url': 'https://www.youtube.com/channel/UCqj7Cz7revf5maW9g5pgNcg',
             'channel_follower_count': int,
         },
     }, {
+        # TODO: fix channel_is_verified extraction
         'note': 'playlists, series',
         'url': 'https://www.youtube.com/c/3blue1brown/playlists?view=50&sort=dd&shelf_id=3',
         'playlist_mincount': 5,
@@ -1066,22 +1079,23 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'url': 'https://www.youtube.com/c/ChristophLaimer/playlists',
         'only_matching': True,
     }, {
+        # TODO: fix availability extraction
         'note': 'basic, single video playlist',
-        'url': 'https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc',
+        'url': 'https://www.youtube.com/playlist?list=PLt5yu3-wZAlSLRHmI1qNm0wjyVNWw1pCU',
         'info_dict': {
-            'id': 'PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc',
-            'title': 'youtube-dl public playlist',
+            'id': 'PLt5yu3-wZAlSLRHmI1qNm0wjyVNWw1pCU',
+            'title': 'single video playlist',
             'description': '',
             'tags': [],
             'view_count': int,
-            'modified_date': '20201130',
-            'channel': 'Sergey M.',
-            'channel_id': 'UCmlqkdCBesrv2Lak1mF_MxA',
-            'channel_url': 'https://www.youtube.com/channel/UCmlqkdCBesrv2Lak1mF_MxA',
+            'modified_date': '20250417',
+            'channel': 'cole-dlp-test-acc',
+            'channel_id': 'UCiu-3thuViMebBjw_5nWYrA',
+            'channel_url': 'https://www.youtube.com/channel/UCiu-3thuViMebBjw_5nWYrA',
             'availability': 'public',
-            'uploader': 'Sergey M.',
-            'uploader_url': 'https://www.youtube.com/@sergeym.6173',
-            'uploader_id': '@sergeym.6173',
+            'uploader': 'cole-dlp-test-acc',
+            'uploader_url': 'https://www.youtube.com/@coletdjnz',
+            'uploader_id': '@coletdjnz',
         },
         'playlist_count': 1,
     }, {
@@ -1171,11 +1185,11 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         },
         'playlist_mincount': 17,
     }, {
-        'note': 'Community tab',
+        'note': 'Posts tab',
         'url': 'https://www.youtube.com/channel/UCKfVa3S1e4PHvxWcwyMMg8w/community',
         'info_dict': {
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
-            'title': 'lex will - Community',
+            'title': 'lex will - Posts',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
             'channel': 'lex will',
             'channel_url': 'https://www.youtube.com/channel/UCKfVa3S1e4PHvxWcwyMMg8w',
@@ -1188,30 +1202,14 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         },
         'playlist_mincount': 18,
     }, {
-        'note': 'Channels tab',
-        'url': 'https://www.youtube.com/channel/UCKfVa3S1e4PHvxWcwyMMg8w/channels',
-        'info_dict': {
-            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
-            'title': 'lex will - Channels',
-            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
-            'channel': 'lex will',
-            'channel_url': 'https://www.youtube.com/channel/UCKfVa3S1e4PHvxWcwyMMg8w',
-            'channel_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
-            'tags': ['bible', 'history', 'prophesy'],
-            'channel_follower_count': int,
-            'uploader_url': 'https://www.youtube.com/@lexwill718',
-            'uploader_id': '@lexwill718',
-            'uploader': 'lex will',
-        },
-        'playlist_mincount': 12,
-    }, {
+        # TODO: fix channel_is_verified extraction
         'note': 'Search tab',
         'url': 'https://www.youtube.com/c/3blue1brown/search?query=linear%20algebra',
         'playlist_mincount': 40,
         'info_dict': {
             'id': 'UCYO_jab_esuFRV4b17AJtAw',
             'title': '3Blue1Brown - Search - linear algebra',
-            'description': 'md5:4d1da95432004b7ba840ebc895b6b4c9',
+            'description': 'md5:602e3789e6a0cb7d9d352186b720e395',
             'channel_url': 'https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw',
             'tags': ['Mathematics'],
             'channel': '3Blue1Brown',
@@ -1232,6 +1230,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'url': 'https://music.youtube.com/channel/UCmlqkdCBesrv2Lak1mF_MxA',
         'only_matching': True,
     }, {
+        # TODO: fix availability extraction
         'note': 'Playlist with deleted videos (#651). As a bonus, the video #51 is also twice in this list.',
         'url': 'https://www.youtube.com/playlist?list=PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC',
         'info_dict': {
@@ -1294,24 +1293,25 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         },
         'playlist_mincount': 21,
     }, {
+        # TODO: fix availability extraction
         'note': 'Playlist with "show unavailable videos" button',
-        'url': 'https://www.youtube.com/playlist?list=UUTYLiWFZy8xtPwxFwX9rV7Q',
+        'url': 'https://www.youtube.com/playlist?list=PLYwq8WOe86_xGmR7FrcJq8Sb7VW8K3Tt2',
         'info_dict': {
-            'title': 'Uploads from Phim Siêu Nhân Nhật Bản',
-            'id': 'UUTYLiWFZy8xtPwxFwX9rV7Q',
+            'title': 'The Memes Of 2010s.....',
+            'id': 'PLYwq8WOe86_xGmR7FrcJq8Sb7VW8K3Tt2',
             'view_count': int,
-            'channel': 'Phim Siêu Nhân Nhật Bản',
+            'channel': "I'm Not JiNxEd",
             'tags': [],
-            'description': '',
-            'channel_url': 'https://www.youtube.com/channel/UCTYLiWFZy8xtPwxFwX9rV7Q',
-            'channel_id': 'UCTYLiWFZy8xtPwxFwX9rV7Q',
+            'description': 'md5:44dc3b315ba69394feaafa2f40e7b2a1',
+            'channel_url': 'https://www.youtube.com/channel/UC5H5H85D1QE5-fuWWQ1hdNg',
+            'channel_id': 'UC5H5H85D1QE5-fuWWQ1hdNg',
             'modified_date': r're:\d{8}',
             'availability': 'public',
-            'uploader_url': 'https://www.youtube.com/@phimsieunhannhatban',
-            'uploader_id': '@phimsieunhannhatban',
-            'uploader': 'Phim Siêu Nhân Nhật Bản',
+            'uploader_url': 'https://www.youtube.com/@imnotjinxed1998',
+            'uploader_id': '@imnotjinxed1998',
+            'uploader': "I'm Not JiNxEd",
         },
-        'playlist_mincount': 200,
+        'playlist_mincount': 150,
         'expected_warnings': [r'[Uu]navailable videos (are|will be) hidden'],
     }, {
         'note': 'Playlist with unavailable videos in page 7',
@@ -1334,6 +1334,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 1000,
         'expected_warnings': [r'[Uu]navailable videos (are|will be) hidden'],
     }, {
+        # TODO: fix availability extraction
         'note': 'https://github.com/ytdl-org/youtube-dl/issues/21844',
         'url': 'https://www.youtube.com/playlist?list=PLzH6n4zXuckpfMu_4Ff8E7Z1behQks5ba',
         'info_dict': {
@@ -1384,7 +1385,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
     }, {
         'url': 'https://www.youtube.com/channel/UCoMdktPbSTixAyNGwb-UYkQ/live',
         'info_dict': {
-            'id': 'hGkQjiJLjWQ',  # This will keep changing
+            'id': 'YDvsBbKfLPA',  # This will keep changing
             'ext': 'mp4',
             'title': str,
             'upload_date': r're:\d{8}',
@@ -1409,6 +1410,8 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             'uploader_id': '@SkyNews',
             'uploader': 'Sky News',
             'channel_is_verified': True,
+            'media_type': 'livestream',
+            'timestamp': int,
         },
         'params': {
             'skip_download': True,
@@ -1496,6 +1499,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'url': 'https://music.youtube.com/browse/UC1a8OFewdjuLq6KlF8M_8Ng',
         'only_matching': True,
     }, {
+        # TODO: fix availability extraction
         'note': 'VLPL, should redirect to playlist?list=PL...',
         'url': 'https://music.youtube.com/browse/VLPLRBp0Fe2GpgmgoscNFLxNyBVSFVdYmFkq',
         'info_dict': {
@@ -1537,6 +1541,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
     }, {
         # Destination channel with only a hidden self tab (tab id is UCtFRv9O2AHqOZjjynzrv-xg)
         # Treat as a general feed
+        # TODO: fix extraction
         'url': 'https://www.youtube.com/channel/UCtFRv9O2AHqOZjjynzrv-xg',
         'info_dict': {
             'id': 'UCtFRv9O2AHqOZjjynzrv-xg',
@@ -1560,21 +1565,21 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'expected_warnings': ['YouTube Music is not directly supported'],
     }, {
         'note': 'unlisted single video playlist',
-        'url': 'https://www.youtube.com/playlist?list=PLwL24UFy54GrB3s2KMMfjZscDi1x5Dajf',
+        'url': 'https://www.youtube.com/playlist?list=PLt5yu3-wZAlQLfIN0MMgp0wVV6MP3bM4_',
         'info_dict': {
-            'id': 'PLwL24UFy54GrB3s2KMMfjZscDi1x5Dajf',
-            'title': 'yt-dlp unlisted playlist test',
+            'id': 'PLt5yu3-wZAlQLfIN0MMgp0wVV6MP3bM4_',
+            'title': 'unlisted playlist',
             'availability': 'unlisted',
             'tags': [],
-            'modified_date': '20220418',
-            'channel': 'colethedj',
+            'modified_date': '20250417',
+            'channel': 'cole-dlp-test-acc',
             'view_count': int,
             'description': '',
-            'channel_id': 'UC9zHu_mHU96r19o-wV5Qs1Q',
-            'channel_url': 'https://www.youtube.com/channel/UC9zHu_mHU96r19o-wV5Qs1Q',
-            'uploader_url': 'https://www.youtube.com/@colethedj1894',
-            'uploader_id': '@colethedj1894',
-            'uploader': 'colethedj',
+            'channel_id': 'UCiu-3thuViMebBjw_5nWYrA',
+            'channel_url': 'https://www.youtube.com/channel/UCiu-3thuViMebBjw_5nWYrA',
+            'uploader_url': 'https://www.youtube.com/@coletdjnz',
+            'uploader_id': '@coletdjnz',
+            'uploader': 'cole-dlp-test-acc',
         },
         'playlist': [{
             'info_dict': {
@@ -1596,6 +1601,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_count': 1,
         'params': {'extract_flat': True},
     }, {
+        # By default, recommended is always empty.
         'note': 'API Fallback: Recommended - redirects to home page. Requires visitorData',
         'url': 'https://www.youtube.com/feed/recommended',
         'info_dict': {
@@ -1603,7 +1609,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             'title': 'recommended',
             'tags': [],
         },
-        'playlist_mincount': 50,
+        'playlist_count': 0,
         'params': {
             'skip_download': True,
             'extractor_args': {'youtubetab': {'skip': ['webpage']}},
@@ -1628,6 +1634,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         },
         'skip': 'Query for sorting no longer works',
     }, {
+        # TODO: fix 'unviewable' issue with this playlist when reloading with unavailable videos
         'note': 'API Fallback: Topic, should redirect to playlist?list=UU...',
         'url': 'https://music.youtube.com/browse/UC9ALqqC4aIeG5iDs7i90Bfw',
         'info_dict': {
@@ -1654,11 +1661,12 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'url': 'https://www.youtube.com/channel/UCwVVpHQ2Cs9iGJfpdFngePQ',
         'only_matching': True,
     }, {
+        # TODO: fix metadata extraction
         'note': 'collaborative playlist (uploader name in the form "by <uploader> and x other(s)")',
         'url': 'https://www.youtube.com/playlist?list=PLx-_-Kk4c89oOHEDQAojOXzEzemXxoqx6',
         'info_dict': {
             'id': 'PLx-_-Kk4c89oOHEDQAojOXzEzemXxoqx6',
-            'modified_date': '20220407',
+            'modified_date': '20250115',
             'channel_url': 'https://www.youtube.com/channel/UCKcqXmCcyqnhgpA5P0oHH_Q',
             'tags': [],
             'availability': 'unlisted',
@@ -1692,6 +1700,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'expected_warnings': ['Preferring "ja"'],
     }, {
         # XXX: this should really check flat playlist entries, but the test suite doesn't support that
+        # TODO: fix availability extraction
         'note': 'preferred lang set with playlist with translated video titles',
         'url': 'https://www.youtube.com/playlist?list=PLt5yu3-wZAlQAaPZ5Z-rJoTdbT-45Q7c0',
         'info_dict': {
@@ -1714,6 +1723,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
     }, {
         # shorts audio pivot for 2GtVksBMYFM.
         'url': 'https://www.youtube.com/feed/sfv_audio_pivot?bp=8gUrCikSJwoLMkd0VmtzQk1ZRk0SCzJHdFZrc0JNWUZNGgsyR3RWa3NCTVlGTQ==',
+        # TODO: fix extraction
         'info_dict': {
             'id': 'sfv_audio_pivot',
             'title': 'sfv_audio_pivot',
@@ -1751,6 +1761,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 8,
     }, {
         # Should get three playlists for videos, shorts and streams tabs
+        # TODO: fix channel_is_verified extraction
         'url': 'https://www.youtube.com/channel/UCK9V2B22uJYu3N7eR_BT9QA',
         'info_dict': {
             'id': 'UCK9V2B22uJYu3N7eR_BT9QA',
@@ -1758,7 +1769,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             'channel_follower_count': int,
             'channel_id': 'UCK9V2B22uJYu3N7eR_BT9QA',
             'channel_url': 'https://www.youtube.com/channel/UCK9V2B22uJYu3N7eR_BT9QA',
-            'description': 'md5:49809d8bf9da539bc48ed5d1f83c33f2',
+            'description': 'md5:01e53f350ab8ad6fcf7c4fedb3c1b99f',
             'channel': 'Polka Ch. 尾丸ポルカ',
             'tags': 'count:35',
             'uploader_url': 'https://www.youtube.com/@OmaruPolka',
@@ -1769,14 +1780,14 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_count': 3,
     }, {
         # Shorts tab with channel with handle
-        # TODO: fix channel description
+        # TODO: fix channel_is_verified extraction
         'url': 'https://www.youtube.com/@NotJustBikes/shorts',
         'info_dict': {
             'id': 'UC0intLFzLaudFG-xAvUEO-A',
             'title': 'Not Just Bikes - Shorts',
             'tags': 'count:10',
             'channel_url': 'https://www.youtube.com/channel/UC0intLFzLaudFG-xAvUEO-A',
-            'description': 'md5:5e82545b3a041345927a92d0585df247',
+            'description': 'md5:1d9fc1bad7f13a487299d1fe1712e031',
             'channel_follower_count': int,
             'channel_id': 'UC0intLFzLaudFG-xAvUEO-A',
             'channel': 'Not Just Bikes',
@@ -1797,7 +1808,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             'channel_url': 'https://www.youtube.com/channel/UC3eYAvjCVwNHgkaGbXX3sig',
             'channel': '中村悠一',
             'channel_follower_count': int,
-            'description': 'md5:e744f6c93dafa7a03c0c6deecb157300',
+            'description': 'md5:e8fd705073a594f27d6d6d020da560dc',
             'uploader_url': 'https://www.youtube.com/@Yuichi-Nakamura',
             'uploader_id': '@Yuichi-Nakamura',
             'uploader': '中村悠一',
@@ -1815,6 +1826,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'only_matching': True,
     }, {
         # No videos tab but has a shorts tab
+        # TODO: fix metadata extraction
         'url': 'https://www.youtube.com/c/TKFShorts',
         'info_dict': {
             'id': 'UCgJ5_1F6yJhYLnyMszUdmUg',
@@ -1851,6 +1863,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
     }, {
         # Shorts url result in shorts tab
         # TODO: Fix channel id extraction
+        # TODO: fix test suite, 208163447408c78673b08c172beafe5c310fb167 broke this test
         'url': 'https://www.youtube.com/channel/UCiu-3thuViMebBjw_5nWYrA/shorts',
         'info_dict': {
             'id': 'UCiu-3thuViMebBjw_5nWYrA',
@@ -1879,6 +1892,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'params': {'extract_flat': True},
     }, {
         # Live video status should be extracted
+        # TODO: fix test suite, 208163447408c78673b08c172beafe5c310fb167 broke this test
         'url': 'https://www.youtube.com/channel/UCQvWX73GQygcwXOTSf_VDVg/live',
         'info_dict': {
             'id': 'UCQvWX73GQygcwXOTSf_VDVg',
@@ -1907,6 +1921,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 1,
     }, {
         # Channel renderer metadata. Contains number of videos on the channel
+        # TODO: channels tab removed, change this test to use another page with channel renderer
         'url': 'https://www.youtube.com/channel/UCiu-3thuViMebBjw_5nWYrA/channels',
         'info_dict': {
             'id': 'UCiu-3thuViMebBjw_5nWYrA',
@@ -1940,7 +1955,9 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             },
         }],
         'params': {'extract_flat': True},
+        'skip': 'channels tab removed',
     }, {
+        # TODO: fix channel_is_verified extraction
         'url': 'https://www.youtube.com/@3blue1brown/about',
         'info_dict': {
             'id': '@3blue1brown',
@@ -1950,7 +1967,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
             'channel_id': 'UCYO_jab_esuFRV4b17AJtAw',
             'channel': '3Blue1Brown',
             'channel_url': 'https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw',
-            'description': 'md5:4d1da95432004b7ba840ebc895b6b4c9',
+            'description': 'md5:602e3789e6a0cb7d9d352186b720e395',
             'uploader_url': 'https://www.youtube.com/@3blue1brown',
             'uploader_id': '@3blue1brown',
             'uploader': '3Blue1Brown',
@@ -1976,6 +1993,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_count': 5,
     }, {
         # Releases tab, with rich entry playlistRenderers (same as Podcasts tab)
+        # TODO: fix channel_is_verified extraction
         'url': 'https://www.youtube.com/@AHimitsu/releases',
         'info_dict': {
             'id': 'UCgFwu-j5-xNJml2FtTrrB3A',
@@ -2015,6 +2033,7 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
         'playlist_mincount': 100,
         'expected_warnings': [r'[Uu]navailable videos (are|will be) hidden'],
     }, {
+        # TODO: fix channel_is_verified extraction
         'note': 'Tags containing spaces',
         'url': 'https://www.youtube.com/channel/UC7_YxT-KID8kRbqZo7MyscQ',
         'playlist_count': 3,
@@ -2035,6 +2054,24 @@ class YoutubeTabIE(YoutubeTabBaseInfoExtractor):
                      'challenges', 'sketches', 'scary games', 'funny games', 'rage games',
                      'mark fischbach'],
         },
+    }, {
+        # https://github.com/yt-dlp/yt-dlp/issues/12933
+        'note': 'streams tab, some scheduled streams. Empty intermediate response with only continuation - must follow',
+        'url': 'https://www.youtube.com/@sbcitygov/streams',
+        'playlist_mincount': 150,
+        'info_dict': {
+            'id': 'UCH6-qfQwlUgz9SAf05jvc_w',
+            'channel': 'sbcitygov',
+            'channel_id': 'UCH6-qfQwlUgz9SAf05jvc_w',
+            'title': 'sbcitygov - Live',
+            'channel_follower_count': int,
+            'description': 'md5:ca1a92059835c071e33b3db52f4a6d67',
+            'uploader_id': '@sbcitygov',
+            'uploader_url': 'https://www.youtube.com/@sbcitygov',
+            'uploader': 'sbcitygov',
+            'channel_url': 'https://www.youtube.com/channel/UCH6-qfQwlUgz9SAf05jvc_w',
+            'tags': [],
+        },
     }]
 
     @classmethod

yt_dlp/extractor/youtube/_video.py

@@ -23,6 +23,8 @@ from ._base import (
     _split_innertube_client,
     short_client_name,
 )
+from .pot._director import initialize_pot_director
+from .pot.provider import PoTokenContext, PoTokenRequest
 from ..openload import PhantomJSwrapper
 from ...jsinterp import JSInterpreter
 from ...networking.exceptions import HTTPError
@@ -34,6 +36,7 @@ from ...utils import (
     clean_html,
     datetime_from_str,
     filesize_from_tbr,
+    filter_dict,
     float_or_none,
     format_field,
     get_first,
@@ -65,6 +68,7 @@ from ...utils import (
     urljoin,
     variadic,
 )
+from ...utils.networking import clean_headers, clean_proxies, select_proxy
 
 STREAMING_DATA_CLIENT_NAME = '__yt_dlp_client'
 STREAMING_DATA_INITIAL_PO_TOKEN = '__yt_dlp_po_token'
@@ -1760,6 +1764,16 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         },
     ]
 
+    _PLAYER_JS_VARIANT_MAP = {
+        'main': 'player_ias.vflset/en_US/base.js',
+        'tce': 'player_ias_tce.vflset/en_US/base.js',
+        'tv': 'tv-player-ias.vflset/tv-player-ias.js',
+        'tv_es6': 'tv-player-es6.vflset/tv-player-es6.js',
+        'phone': 'player-plasma-ias-phone-en_US.vflset/base.js',
+        'tablet': 'player-plasma-ias-tablet-en_US.vflset/base.js',
+    }
+    _INVERSE_PLAYER_JS_VARIANT_MAP = {v: k for k, v in _PLAYER_JS_VARIANT_MAP.items()}
+
     @classmethod
     def suitable(cls, url):
         from yt_dlp.utils import parse_qs
@@ -1773,6 +1787,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         super().__init__(*args, **kwargs)
         self._code_cache = {}
         self._player_cache = {}
+        self._pot_director = None
+
+    def _real_initialize(self):
+        super()._real_initialize()
+        self._pot_director = initialize_pot_director(self)
 
     def _prepare_live_from_start_formats(self, formats, video_id, live_start_time, url, webpage_url, smuggled_data, is_live):
         lock = threading.Lock()
@@ -1939,6 +1958,21 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             get_all=False, expected_type=str)
         if not player_url:
             return
+
+        requested_js_variant = self._configuration_arg('player_js_variant', [''])[0] or 'actual'
+        if requested_js_variant in self._PLAYER_JS_VARIANT_MAP:
+            player_id = self._extract_player_info(player_url)
+            original_url = player_url
+            player_url = f'/s/player/{player_id}/{self._PLAYER_JS_VARIANT_MAP[requested_js_variant]}'
+            if original_url != player_url:
+                self.write_debug(
+                    f'Forcing "{requested_js_variant}" player JS variant for player {player_id}\n'
+                    f'        original url = {original_url}', only_once=True)
+        elif requested_js_variant != 'actual':
+            self.report_warning(
+                f'Invalid player JS variant name "{requested_js_variant}" requested. '
+                f'Valid choices are: {", ".join(self._PLAYER_JS_VARIANT_MAP)}', only_once=True)
+
         return urljoin('https://www.youtube.com', player_url)
 
     def _download_player_url(self, video_id, fatal=False):
@@ -1953,6 +1987,17 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             if player_version:
                 return f'https://www.youtube.com/s/player/{player_version}/player_ias.vflset/en_US/base.js'
 
+    def _player_js_cache_key(self, player_url):
+        player_id = self._extract_player_info(player_url)
+        player_path = remove_start(urllib.parse.urlparse(player_url).path, f'/s/player/{player_id}/')
+        variant = self._INVERSE_PLAYER_JS_VARIANT_MAP.get(player_path)
+        if not variant:
+            self.write_debug(
+                f'Unable to determine player JS variant\n'
+                f'        player = {player_url}', only_once=True)
+            variant = re.sub(r'[^a-zA-Z0-9]', '_', remove_end(player_path, '.js'))
+        return join_nonempty(player_id, variant)
+
     def _signature_cache_id(self, example_sig):
         """ Return a string representation of a signature """
         return '.'.join(str(len(part)) for part in example_sig.split('.'))
@@ -1968,30 +2013,29 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         return id_m.group('id')
 
     def _load_player(self, video_id, player_url, fatal=True):
-        player_id = self._extract_player_info(player_url)
-        if player_id not in self._code_cache:
+        player_js_key = self._player_js_cache_key(player_url)
+        if player_js_key not in self._code_cache:
             code = self._download_webpage(
                 player_url, video_id, fatal=fatal,
-                note='Downloading player ' + player_id,
-                errnote=f'Download of {player_url} failed')
+                note=f'Downloading player {player_js_key}',
+                errnote=f'Download of {player_js_key} failed')
             if code:
-                self._code_cache[player_id] = code
-        return self._code_cache.get(player_id)
+                self._code_cache[player_js_key] = code
+        return self._code_cache.get(player_js_key)
 
     def _extract_signature_function(self, video_id, player_url, example_sig):
-        player_id = self._extract_player_info(player_url)
-
         # Read from filesystem cache
-        func_id = f'js_{player_id}_{self._signature_cache_id(example_sig)}'
+        func_id = join_nonempty(
+            self._player_js_cache_key(player_url), self._signature_cache_id(example_sig))
         assert os.path.basename(func_id) == func_id
 
         self.write_debug(f'Extracting signature function {func_id}')
-        cache_spec, code = self.cache.load('youtube-sigfuncs', func_id), None
+        cache_spec, code = self.cache.load('youtube-sigfuncs', func_id, min_ver='2025.03.31'), None
 
         if not cache_spec:
             code = self._load_player(video_id, player_url)
         if code:
-            res = self._parse_sig_js(code)
+            res = self._parse_sig_js(code, player_url)
             test_string = ''.join(map(chr, range(len(example_sig))))
             cache_spec = [ord(c) for c in res(test_string)]
             self.cache.store('youtube-sigfuncs', func_id, cache_spec)
@@ -2039,7 +2083,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 f'    return {expr_code}\n')
         self.to_screen('Extracted signature function:\n' + code)
 
-    def _parse_sig_js(self, jscode):
+    def _parse_sig_js(self, jscode, player_url):
         # Examples where `sig` is funcname:
         # sig=function(a){a=a.split(""); ... ;return a.join("")};
         # ;c&&(c=sig(decodeURIComponent(c)),a.set(b,encodeURIComponent(c)));return a};
@@ -2063,12 +2107,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
              r'\bc\s*&&\s*[a-zA-Z0-9]+\.set\([^,]+\s*,\s*\([^)]*\)\s*\(\s*(?P<sig>[a-zA-Z0-9$]+)\('),
             jscode, 'Initial JS player signature function name', group='sig')
 
+        varname, global_list = self._interpret_player_js_global_var(jscode, player_url)
         jsi = JSInterpreter(jscode)
-        global_var_map = {}
-        _, varname, value = self._extract_player_js_global_var(jscode)
-        if varname:
-            global_var_map[varname] = jsi.interpret_expression(value, {}, allow_recursion=100)
-        initial_function = jsi.extract_function(funcname, global_var_map)
+        initial_function = jsi.extract_function(funcname, filter_dict({varname: global_list}))
         return lambda s: initial_function([s])
 
     def _cached(self, func, *cache_id):
@@ -2087,6 +2128,24 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             return ret
         return inner
 
+    def _load_nsig_code_from_cache(self, player_url):
+        cache_id = ('youtube-nsig', self._player_js_cache_key(player_url))
+
+        if func_code := self._player_cache.get(cache_id):
+            return func_code
+
+        func_code = self.cache.load(*cache_id, min_ver='2025.03.31')
+        if func_code:
+            self._player_cache[cache_id] = func_code
+
+        return func_code
+
+    def _store_nsig_code_to_cache(self, player_url, func_code):
+        cache_id = ('youtube-nsig', self._player_js_cache_key(player_url))
+        if cache_id not in self._player_cache:
+            self.cache.store(*cache_id, func_code)
+            self._player_cache[cache_id] = func_code
+
     def _decrypt_signature(self, s, video_id, player_url):
         """Turn the encrypted s field into a working signature"""
         extract_sig = self._cached(
@@ -2127,9 +2186,31 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 video_id=video_id, note='Executing signature code').strip()
 
         self.write_debug(f'Decrypted nsig {s} => {ret}')
+        # Only cache nsig func JS code to disk if successful, and only once
+        self._store_nsig_code_to_cache(player_url, func_code)
         return ret
 
     def _extract_n_function_name(self, jscode, player_url=None):
+        varname, global_list = self._interpret_player_js_global_var(jscode, player_url)
+        if debug_str := traverse_obj(global_list, (lambda _, v: v.endswith('_w8_'), any)):
+            funcname = self._search_regex(
+                r'''(?xs)
+                    [;\n](?:
+                        (?P<f>function\s+)|
+                        (?:var\s+)?
+                    )(?P<funcname>[a-zA-Z0-9_$]+)\s*(?(f)|=\s*function\s*)
+                    \((?P<argname>[a-zA-Z0-9_$]+)\)\s*\{
+                    (?:(?!\}[;\n]).)+
+                    \}\s*catch\(\s*[a-zA-Z0-9_$]+\s*\)\s*
+                    \{\s*return\s+%s\[%d\]\s*\+\s*(?P=argname)\s*\}\s*return\s+[^}]+\}[;\n]
+                ''' % (re.escape(varname), global_list.index(debug_str)),
+                jscode, 'nsig function name', group='funcname', default=None)
+            if funcname:
+                return funcname
+            self.write_debug(join_nonempty(
+                'Initial search was unable to find nsig function name',
+                player_url and f'        player = {player_url}', delim='\n'), only_once=True)
+
         # Examples (with placeholders nfunc, narray, idx):
         # *  .get("n"))&&(b=nfunc(b)
         # *  .get("n"))&&(b=narray[idx](b)
@@ -2159,7 +2240,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         if not funcname:
             self.report_warning(join_nonempty(
                 'Falling back to generic n function search',
-                player_url and f'         player = {player_url}', delim='\n'))
+                player_url and f'         player = {player_url}', delim='\n'), only_once=True)
             return self._search_regex(
                 r'''(?xs)
                 ;\s*(?P<name>[a-zA-Z0-9_$]+)\s*=\s*function\([a-zA-Z0-9_$]+\)
@@ -2172,9 +2253,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             rf'var {re.escape(funcname)}\s*=\s*(\[.+?\])\s*[,;]', jscode,
             f'Initial JS player n function list ({funcname}.{idx})')))[int(idx)]
 
-    def _extract_player_js_global_var(self, jscode):
+    def _extract_player_js_global_var(self, jscode, player_url):
         """Returns tuple of strings: variable assignment code, variable name, variable value code"""
-        return self._search_regex(
+        extract_global_var = self._cached(self._search_regex, 'js global array', player_url)
+        varcode, varname, varvalue = extract_global_var(
             r'''(?x)
                 (?P<q1>["\'])use\s+strict(?P=q1);\s*
                 (?P<code>
@@ -2182,24 +2264,49 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     (?P<value>
                         (?P<q2>["\'])(?:(?!(?P=q2)).|\\.)+(?P=q2)
                         \.split\((?P<q3>["\'])(?:(?!(?P=q3)).)+(?P=q3)\)
+                        |\[\s*(?:(?P<q4>["\'])(?:(?!(?P=q4)).|\\.)*(?P=q4)\s*,?\s*)+\]
                     )
                 )[;,]
             ''', jscode, 'global variable', group=('code', 'name', 'value'), default=(None, None, None))
+        if not varcode:
+            self.write_debug(join_nonempty(
+                'No global array variable found in player JS',
+                player_url and f'        player = {player_url}', delim='\n'), only_once=True)
+        return varcode, varname, varvalue
 
-    def _fixup_n_function_code(self, argnames, code, full_code):
-        global_var, varname, _ = self._extract_player_js_global_var(full_code)
-        if global_var:
-            self.write_debug(f'Prepending n function code with global array variable "{varname}"')
-            code = global_var + '; ' + code
+    def _interpret_player_js_global_var(self, jscode, player_url):
+        """Returns tuple of: variable name string, variable value list"""
+        _, varname, array_code = self._extract_player_js_global_var(jscode, player_url)
+        jsi = JSInterpreter(array_code)
+        interpret_global_var = self._cached(jsi.interpret_expression, 'js global list', player_url)
+        return varname, interpret_global_var(array_code, {}, allow_recursion=10)
+
+    def _fixup_n_function_code(self, argnames, nsig_code, jscode, player_url):
+        varcode, varname, _ = self._extract_player_js_global_var(jscode, player_url)
+        if varcode and varname:
+            nsig_code = varcode + '; ' + nsig_code
+            _, global_list = self._interpret_player_js_global_var(jscode, player_url)
         else:
-            self.write_debug('No global array variable found in player JS')
-        return argnames, re.sub(
-            rf';\s*if\s*\(\s*typeof\s+[a-zA-Z0-9_$]+\s*===?\s*(?:(["\'])undefined\1|{varname}\[\d+\])\s*\)\s*return\s+{argnames[0]};',
-            ';', code)
+            varname = 'dlp_wins'
+            global_list = []
+
+        undefined_idx = global_list.index('undefined') if 'undefined' in global_list else r'\d+'
+        fixed_code = re.sub(
+            rf'''(?x)
+                ;\s*if\s*\(\s*typeof\s+[a-zA-Z0-9_$]+\s*===?\s*(?:
+                    (["\'])undefined\1|
+                    {re.escape(varname)}\[{undefined_idx}\]
+                )\s*\)\s*return\s+{re.escape(argnames[0])};
+            ''', ';', nsig_code)
+        if fixed_code == nsig_code:
+            self.write_debug(join_nonempty(
+                'No typeof statement found in nsig function code',
+                player_url and f'        player = {player_url}', delim='\n'), only_once=True)
+        return argnames, fixed_code
 
     def _extract_n_function_code(self, video_id, player_url):
         player_id = self._extract_player_info(player_url)
-        func_code = self.cache.load('youtube-nsig', player_id, min_ver='2025.03.25')
+        func_code = self._load_nsig_code_from_cache(player_url)
         jscode = func_code or self._load_player(video_id, player_url)
         jsi = JSInterpreter(jscode)
 
@@ -2209,9 +2316,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         func_name = self._extract_n_function_name(jscode, player_url=player_url)
 
         # XXX: Workaround for the global array variable and lack of `typeof` implementation
-        func_code = self._fixup_n_function_code(*jsi.extract_function_code(func_name), jscode)
+        func_code = self._fixup_n_function_code(*jsi.extract_function_code(func_name), jscode, player_url)
 
-        self.cache.store('youtube-nsig', player_id, func_code)
         return jsi, player_id, func_code
 
     def _extract_n_function_from_code(self, jsi, func_code):
@@ -2720,7 +2826,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 continue
 
     def fetch_po_token(self, client='web', context=_PoTokenContext.GVS, ytcfg=None, visitor_data=None,
-                       data_sync_id=None, session_index=None, player_url=None, video_id=None, **kwargs):
+                       data_sync_id=None, session_index=None, player_url=None, video_id=None, webpage=None, **kwargs):
         """
         Fetch a PO Token for a given client and context. This function will validate required parameters for a given context and client.
 
@@ -2734,10 +2840,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         @param session_index: session index.
         @param player_url: player URL.
         @param video_id: video ID.
+        @param webpage: video webpage.
         @param kwargs: Additional arguments to pass down. May be more added in the future.
         @return: The fetched PO Token. None if it could not be fetched.
         """
 
+        # TODO(future): This validation should be moved into pot framework.
+        #  Some sort of middleware or validation provider perhaps?
+
         # GVS WebPO Token is bound to visitor_data / Visitor ID when logged out.
         # Must have visitor_data for it to function.
         if player_url and context == _PoTokenContext.GVS and not visitor_data and not self.is_authenticated:
@@ -2759,6 +2869,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     f'Got a GVS PO Token for {client} client, but missing Data Sync ID for account. Formats may not work.'
                     f'You may need to pass a Data Sync ID with --extractor-args "youtube:data_sync_id=XXX"')
 
+            self.write_debug(f'{video_id}: Retrieved a {context.value} PO Token for {client} client from config')
             return config_po_token
 
         # Require GVS WebPO Token if logged in for external fetching
@@ -2768,7 +2879,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 f'You may need to pass a Data Sync ID with --extractor-args "youtube:data_sync_id=XXX"')
             return
 
-        return self._fetch_po_token(
+        po_token = self._fetch_po_token(
             client=client,
             context=context.value,
             ytcfg=ytcfg,
@@ -2777,11 +2888,61 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             session_index=session_index,
             player_url=player_url,
             video_id=video_id,
+            video_webpage=webpage,
             **kwargs,
         )
 
+        if po_token:
+            self.write_debug(f'{video_id}: Retrieved a {context.value} PO Token for {client} client')
+            return po_token
+
     def _fetch_po_token(self, client, **kwargs):
-        """(Unstable) External PO Token fetch stub"""
+
+        context = kwargs.get('context')
+
+        # Avoid fetching PO Tokens when not required
+        if not (
+            _PoTokenContext(context) in self._get_default_ytcfg(client)['PO_TOKEN_REQUIRED_CONTEXTS']
+            or self._configuration_arg('fetch_pot', ['when_required'], ie_key=YoutubeIE)[0] == 'always'
+        ):
+            return None
+
+        headers = self.get_param('http_headers').copy()
+        proxies = self._downloader.proxies.copy()
+        clean_headers(headers)
+        clean_proxies(proxies, headers)
+
+        innertube_host = self._select_api_hostname(None, default_client=client)
+
+        pot_request = PoTokenRequest(
+            context=PoTokenContext(context),
+            innertube_context=traverse_obj(kwargs, ('ytcfg', 'INNERTUBE_CONTEXT')),
+            innertube_host=innertube_host,
+            internal_client_name=client,
+            session_index=kwargs.get('session_index'),
+            player_url=kwargs.get('player_url'),
+            video_webpage=kwargs.get('video_webpage'),
+            is_authenticated=self.is_authenticated,
+            visitor_data=kwargs.get('visitor_data'),
+            data_sync_id=kwargs.get('data_sync_id'),
+            video_id=kwargs.get('video_id'),
+            request_cookiejar=self._downloader.cookiejar,
+
+            # All requests that would need to be proxied should be in the
+            # context of www.youtube.com or the innertube host
+            request_proxy=(
+                select_proxy('https://www.youtube.com', proxies)
+                or select_proxy(f'https://{innertube_host}', proxies)
+            ),
+            request_headers=headers,
+            request_timeout=self.get_param('socket_timeout'),
+            request_verify_tls=not self.get_param('nocheckcertificate'),
+            request_source_address=self.get_param('source_address'),
+
+            bypass_cache=False,
+        )
+
+        return self._pot_director.get_po_token(pot_request)
 
     @staticmethod
     def _is_agegated(player_response):
@@ -2939,8 +3100,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'video_id': video_id,
                 'data_sync_id': data_sync_id if self.is_authenticated else None,
                 'player_url': player_url if require_js_player else None,
+                'webpage': webpage,
                 'session_index': self._extract_session_index(master_ytcfg, player_ytcfg),
-                'ytcfg': player_ytcfg,
+                'ytcfg': player_ytcfg or self._get_default_ytcfg(client),
             }
 
             player_po_token = self.fetch_po_token(
@@ -3163,7 +3325,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     if player_url:
                         self.report_warning(
                             f'nsig extraction failed: Some formats may be missing\n'
-                            f'         n = {query["n"][0]} ; player = {player_url}',
+                            f'         n = {query["n"][0]} ; player = {player_url}\n'
+                            f'         {bug_reports_message(before="")}',
                             video_id=video_id, only_once=True)
                         self.write_debug(e, only_once=True)
                     else:
@@ -3181,7 +3344,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             is_damaged = try_call(lambda: format_duration < duration // 2)
             if is_damaged:
                 self.report_warning(
-                    f'{video_id}: Some formats are possibly damaged. They will be deprioritized', only_once=True)
+                    'Some formats are possibly damaged. They will be deprioritized', video_id, only_once=True)
 
             po_token = fmt.get(STREAMING_DATA_INITIAL_PO_TOKEN)
 
@@ -3547,6 +3710,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 if 'sign in' in reason.lower():
                     reason = remove_end(reason, 'This helps protect our community. Learn more')
                     reason = f'{remove_end(reason.strip(), ".")}. {self._youtube_login_hint}'
+                elif get_first(playability_statuses, ('errorScreen', 'playerCaptchaViewModel', {dict})):
+                    reason += '. YouTube is requiring a captcha challenge before playback'
                 self.raise_no_formats(reason, expected=True)
 
         keywords = get_first(video_details, 'keywords', expected_type=list) or []
@@ -3775,7 +3940,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             if not traverse_obj(initial_data, 'contents'):
                 self.report_warning('Incomplete data received in embedded initial data; re-fetching using API.')
                 initial_data = None
-        if not initial_data:
+        if not initial_data and 'initial_data' not in self._configuration_arg('player_skip'):
             query = {'videoId': video_id}
             query.update(self._get_checkok_params())
             initial_data = self._extract_response(

yt_dlp/extractor/youtube/pot/README.md

@@ -0,0 +1,285 @@
+# YoutubeIE PO Token Provider Framework
+
+As part of the YouTube extractor, we have a framework for providing PO Tokens programmatically. This can be used in plugins.
+
+Refer to the [PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide) for more information on PO Tokens.
+
+## Public APIs
+
+- `yt_dlp.extractor.youtube.pot.cache`
+- `yt_dlp.extractor.youtube.pot.provider`
+- `yt_dlp.extractor.youtube.pot.utils`
+
+Everything else is internal-only and no guarantees are made about the API stability.
+
+> [!WARNING]
+> We will try our best to maintain stability with the public APIs.
+> However, due to the nature of extractors and YouTube, we may need to remove or change APIs in the future.
+> If you are using these APIs outside yt-dlp plugins, please account for this by importing them safely.
+
+## PO Token Provider
+
+`yt_dlp.extractor.youtube.pot.provider`
+
+```python
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+    PoTokenProvider,
+    PoTokenResponse,
+    PoTokenProviderError,
+    PoTokenProviderRejectedRequest,
+    register_provider,
+    register_preference,
+    ExternalRequestFeature,
+)
+from yt_dlp.networking.common import Request
+from yt_dlp.extractor.youtube.pot.utils import get_webpo_content_binding
+from yt_dlp.utils import traverse_obj
+from yt_dlp.networking.exceptions import RequestError
+import json
+
+
+@register_provider
+class MyPoTokenProviderPTP(PoTokenProvider):  # Provider name must end with "PTP"
+    PROVIDER_VERSION = '0.2.1'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'my-provider'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    # -- Validation shortcuts. Set these to None to disable. --
+
+    # Innertube Client Name.
+    # For example, "WEB", "ANDROID", "TVHTML5".
+    # For a list of WebPO client names, see yt_dlp.extractor.youtube.pot.utils.WEBPO_CLIENTS.
+    # Also see yt_dlp.extractor.youtube._base.INNERTUBE_CLIENTS for a list of client names currently supported by the YouTube extractor.
+    _SUPPORTED_CLIENTS = ('WEB', 'TVHTML5')
+
+    _SUPPORTED_CONTEXTS = (
+        PoTokenContext.GVS,
+    )
+
+    # If your provider makes external requests to websites (i.e. to youtube.com) using another library or service (i.e., not _request_webpage),
+    # set the request features that are supported here.
+    # If only using _request_webpage to make external requests, set this to None.
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+        ExternalRequestFeature.PROXY_SCHEME_HTTP, 
+        ExternalRequestFeature.SOURCE_ADDRESS, 
+        ExternalRequestFeature.DISABLE_TLS_VERIFICATION
+    )
+
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method SHOULD NOT make any network requests or perform any expensive operations.
+
+        Since this is called multiple times, we recommend caching the result.
+        """
+        return True
+
+    def close(self):
+        # Optional close hook, called when YoutubeDL is closed.
+        pass
+
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        # ℹ️ If you need to validate the request before making the request to the external source
+        # Raise yt_dlp.extractor.youtube.pot.provider.PoTokenProviderRejectedRequest if the request is not supported
+        if request.is_authenticated:
+            raise PoTokenProviderRejectedRequest(
+                'This provider does not support authenticated requests'
+            )
+
+        # ℹ️ Settings are pulled from extractor args passed to yt-dlp with the key `youtubepot-<PROVIDER_KEY>`.
+        # For this example, the extractor arg would be `--extractor-args "youtubepot-mypotokenprovider:url=https://custom.example.com/get_pot"`
+        external_provider_url = self._configuration_arg('url', default=['https://provider.example.com/get_pot'])[0]
+        
+        # See below for logging guidelines
+        self.logger.trace(f'Using external provider URL: {external_provider_url}')
+        
+        # You should use the internal HTTP client to make requests where possible,
+        # as it will handle cookies and other networking settings passed to yt-dlp.
+        try:
+            # See docstring in _request_webpage method for request tips
+            response = self._request_webpage(
+                Request(external_provider_url, data=json.dumps({
+                    'content_binding': get_webpo_content_binding(request),
+                    'proxy': request.request_proxy,
+                    'headers': request.request_headers,
+                    'source_address': request.request_source_address,
+                    'verify_tls': request.request_verify_tls,
+                    # Important: If your provider has its own caching, please respect `bypass_cache`.
+                    # This may be used in the future to request a fresh PO Token if required.
+                    'do_not_cache': request.bypass_cache,
+                }).encode(), proxies={'all': None}), 
+                pot_request=request, 
+                note=f'Requesting {request.context.value} PO Token for {request.internal_client_name} client from external provider',
+            )
+
+        except RequestError as e:
+            # ℹ️ If there is an error, raise PoTokenProviderError.
+            # You can specify whether it is expected or not. If it is unexpected, the log will include a link to the bug report location (BUG_REPORT_LOCATION).
+            raise PoTokenProviderError(
+                'Networking error while fetching to get PO Token from external provider',
+                expected=True
+            ) from e
+
+        # Note: PO Token is expected to be base64url encoded
+        po_token = traverse_obj(response, 'po_token')
+        if not po_token:
+            raise PoTokenProviderError(
+                'Bad PO Token Response from external provider',
+                expected=False
+            )
+
+        return PoTokenResponse(
+            po_token=po_token,
+            # Optional, add a custom expiration time for the token. Use for caching.
+            # By default, yt-dlp will use the default ttl from a registered cache spec (see below)
+            # Set to 0 or -1 to not cache this response.
+            expires_at=None,
+        )
+
+
+# If there are multiple PO Token Providers that can handle the same PoTokenRequest,
+# you can define a preference function to increase/decrease the priority of providers.
+
+@register_preference(MyPoTokenProviderPTP)
+def my_provider_preference(provider: PoTokenProvider, request: PoTokenRequest, *_, **__) -> int:
+    return 50
+```
+
+## Logging Guidelines
+
+- Use the `self.logger` object to log messages.
+- When making HTTP requests, use `self.logger.info` to log a message to standard non-verbose output. This lets users know what is happening when a time-expensive operation is taking place.
+  - For example, `self.logger.info(f'Requesting {request.context.value} PO Token for {request.internal_client_name} client from external provider')`
+- Use `self.logger.debug` to log a message to the verbose output (`--verbose`). Try to keep this to a minimum.
+- Use `self.logger.trace` to log a message to the PO Token debug output (`--extractor-args "youtube:pot_debug=true"`). Log as much as you like here as needed for debugging your provider.
+- Avoid logging PO Tokens or any sensitive information to debug or info output.
+
+## Debugging
+
+- Use `-v --extractor-args "youtube:pot_debug=true"` to enable PO Token debug output.
+
+## Caching
+
+> [!IMPORTANT]
+> yt-dlp currently has a built-in LRU Memory Cache Provider and a cache spec provider for WebPO Tokens. 
+> You should only need to implement cache providers if you want an external cache, or a cache spec if you are handling non-WebPO Tokens.
+
+### Cache Providers
+
+`yt_dlp.extractor.youtube.pot.cache`
+
+```python
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    register_preference,
+    register_provider
+)
+
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+@register_provider
+class MyCacheProviderPCP(PoTokenCacheProvider):  # Provider name must end with "PCP"
+    PROVIDER_VERSION = '0.1.0'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'my-cache-provider'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method SHOULD NOT make any network requests or perform any expensive operations.
+
+        Since this is called multiple times, we recommend caching the result.
+        """
+        return True
+
+    """
+    Implement the below cache operations.
+
+    - expires_at is a timestamp in UTC. It MUST be respected - cache entries should not be returned if they have expired.
+    """
+
+    def get(self, key: str):
+        # ℹ️ Similar to PO Token Providers, Cache Providers and Cache Spec Providers are passed down extractor args matching key youtubepot-<PROVIDER_KEY>.
+        some_setting = self._configuration_arg('some_setting', default=['default_value'])[0]
+        return self.my_cache.get(key)
+
+    def store(self, key: str, value: str, expires_at: int):
+        self.my_cache.store(key, value, expires_at)
+
+    def delete(self, key: str):
+        self.my_cache.delete(key)
+
+    def close(self):
+        # Optional close hook, called when YoutubeDL is closed.
+        pass
+
+# If there are multiple PO Token Cache Providers available, you can define a preference function to increase/decrease the priority of providers. 
+# IMPORTANT: Providers should be in preference of cache lookup time. For example, a memory cache should have a higher preference than a disk cache. 
+# VERY IMPORTANT: yt-dlp has a built-in memory cache with a priority of 10000. Your cache provider should be lower than this.
+
+
+@register_preference(MyCacheProviderPCP)
+def my_cache_preference(provider: PoTokenCacheProvider, request: PoTokenRequest, *_, **__) -> int:
+    return 50
+```
+
+### Cache Specs
+
+`yt_dlp.extractor.youtube.pot.cache`
+
+These are used to provide information on how to cache a particular PO Token Request. 
+You might have a different cache spec for different kinds of PO Tokens (e.g. WebPO vs iOSGuard)
+
+```python
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    CacheProviderWritePolicy,
+    register_spec,
+)
+from yt_dlp.utils import traverse_obj
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+@register_spec
+class MyCacheSpecProviderPCSP(PoTokenCacheSpecProvider):  # Provider name must end with "PCSP"
+    PROVIDER_VERSION = '0.1.0'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'mycachespec'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    def generate_cache_spec(self, request: PoTokenRequest):
+
+        client_name = traverse_obj(request.innertube_context, ('client', 'clientName'))
+        if client_name != 'ANDROID':
+            # ℹ️ If the request is not supported by the cache spec, return None
+            return None
+
+        # Generate a cache spec for the request
+        return PoTokenCacheSpec(
+            # Key bindings to uniquely identify the request. These are used to generate a cache key.
+            key_bindings={
+                'client_name': client_name,
+                'content_binding': 'unique_content_binding',
+                'ip': traverse_obj(request.innertube_context, ('client', 'remoteHost')),
+                'source_address': request.request_source_address,
+                'proxy': request.request_proxy,
+            },
+            # Default Cache TTL in seconds
+            default_ttl=21600,
+
+            # Optional: Specify a write policy.
+            # WRITE_FIRST will write to the highest priority provider only, whereas WRITE_ALL will write to all providers.
+            # WRITE_FIRST may be useful if the PO Token is short-lived and there is no use writing to all providers.
+            write_policy=CacheProviderWritePolicy.WRITE_ALL,
+        )
+```

yt_dlp/extractor/youtube/pot/__init__.py

@@ -0,0 +1,2 @@
+# Trigger import of built-in providers
+from ._builtin import _trigger_import  # noqa: F401

yt_dlp/extractor/youtube/pot/_builtin/__init__.py

@@ -0,0 +1,4 @@
+from .memory_cache import MemoryLRUPCP as _MemoryLRUPCP  # noqa: F401
+from .webpo_cachespec import WebPoPCSP as _WebPoPCSP  # noqa: F401
+
+_trigger_import = None  # noqa: F401

yt_dlp/extractor/youtube/pot/_builtin/memory_cache.py

@@ -0,0 +1,80 @@
+from __future__ import annotations
+
+import datetime as dt
+import typing
+from collections import OrderedDict
+from threading import Lock
+
+from yt_dlp.extractor.youtube.pot._provider import BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot._registry import _pot_memory_cache
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    register_preference,
+    register_provider,
+)
+
+
+def initialize_global_cache(max_size: int):
+    if _pot_memory_cache.value.get('cache') is None:
+        _pot_memory_cache.value['cache'] = OrderedDict()
+        _pot_memory_cache.value['lock'] = Lock()
+        _pot_memory_cache.value['max_size'] = max_size
+
+    if _pot_memory_cache.value['max_size'] != max_size:
+        raise ValueError('Cannot change max_size of initialized global memory cache')
+
+    return _pot_memory_cache.value['cache'], _pot_memory_cache.value['lock'], _pot_memory_cache.value['max_size']
+
+
+@register_provider
+class MemoryLRUPCP(PoTokenCacheProvider, BuiltInIEContentProvider):
+    PROVIDER_NAME = 'memory'
+    DEFAULT_CACHE_SIZE = 25
+
+    def __init__(
+        self,
+        *args,
+        initialize_cache: typing.Callable[[int], tuple[OrderedDict, Lock, int]] = initialize_global_cache,
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        self.cache, self.lock, self.max_size = initialize_cache(self.DEFAULT_CACHE_SIZE)
+
+    def is_available(self) -> bool:
+        return True
+
+    def get(self, key: str) -> str | None:
+        with self.lock:
+            if key not in self.cache:
+                self.logger.trace('cache miss')
+                return None
+            value, expires_at = self.cache.pop(key)
+            if expires_at < int(dt.datetime.now(dt.timezone.utc).timestamp()):
+                self.logger.trace(f'cache expired key={key}')
+                return None
+            self.cache[key] = (value, expires_at)
+            self.logger.trace(f'cache hit key={key}')
+            return value
+
+    def store(self, key: str, value: str, expires_at: int):
+        with self.lock:
+            if expires_at < int(dt.datetime.now(dt.timezone.utc).timestamp()):
+                self.logger.trace(f'ignoring expired key={key}')
+                return
+            if key in self.cache:
+                self.cache.pop(key)
+            self.cache[key] = (value, expires_at)
+            if len(self.cache) > self.max_size:
+                self.cache.popitem(last=False)
+            self.logger.trace(f'storing key={key}')
+
+    def delete(self, key: str):
+        with self.lock:
+            self.logger.trace(f'deleting key={key}')
+            self.cache.pop(key, None)
+
+
+@register_preference(MemoryLRUPCP)
+def memorylru_preference(*_, **__):
+    # Memory LRU Cache SHOULD be the highest priority
+    return 10000

yt_dlp/extractor/youtube/pot/_builtin/webpo_cachespec.py

@@ -0,0 +1,46 @@
+from __future__ import annotations
+
+from yt_dlp.extractor.youtube.pot._provider import BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot.cache import (
+    CacheProviderWritePolicy,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    register_spec,
+)
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+)
+from yt_dlp.extractor.youtube.pot.utils import ContentBindingType, get_webpo_content_binding
+from yt_dlp.utils import traverse_obj
+
+
+@register_spec
+class WebPoPCSP(PoTokenCacheSpecProvider, BuiltInIEContentProvider):
+    PROVIDER_NAME = 'webpo'
+
+    def generate_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        bind_to_visitor_id = self._configuration_arg('bind_to_visitor_id', default=['true'])[0] == 'true'
+        content_binding, content_binding_type = get_webpo_content_binding(
+            request, bind_to_visitor_id=bind_to_visitor_id)
+
+        if not content_binding:
+            return None
+
+        write_policy = CacheProviderWritePolicy.WRITE_ALL
+        if content_binding_type == ContentBindingType.VIDEO_ID:
+            write_policy = CacheProviderWritePolicy.WRITE_FIRST
+
+        return PoTokenCacheSpec(
+            key_bindings={
+                't': 'webpo',
+                'cb': content_binding,
+                'cbt': content_binding_type.value,
+                'ip': traverse_obj(request.innertube_context, ('client', 'remoteHost')),
+                'sa': request.request_source_address,
+                'px': request.request_proxy,
+            },
+            # Integrity token response usually states it has a ttl of 12 hours (43200 seconds).
+            # We will default to 6 hours to be safe.
+            default_ttl=21600,
+            write_policy=write_policy,
+        )

yt_dlp/extractor/youtube/pot/_director.py

@@ -0,0 +1,433 @@
+from __future__ import annotations
+
+import base64
+import binascii
+import dataclasses
+import datetime as dt
+import hashlib
+import json
+import typing
+import urllib.parse
+from collections.abc import Iterable
+
+from yt_dlp.extractor.youtube.pot._provider import (
+    BuiltInIEContentProvider,
+    IEContentProvider,
+    IEContentProviderLogger,
+)
+from yt_dlp.extractor.youtube.pot._registry import (
+    _pot_cache_provider_preferences,
+    _pot_cache_providers,
+    _pot_pcs_providers,
+    _pot_providers,
+    _ptp_preferences,
+)
+from yt_dlp.extractor.youtube.pot.cache import (
+    CacheProviderWritePolicy,
+    PoTokenCacheProvider,
+    PoTokenCacheProviderError,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+)
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenProvider,
+    PoTokenProviderError,
+    PoTokenProviderRejectedRequest,
+    PoTokenRequest,
+    PoTokenResponse,
+    provider_bug_report_message,
+)
+from yt_dlp.utils import ExtractorError, bug_reports_message, format_field, join_nonempty, traverse_obj
+
+if typing.TYPE_CHECKING:
+    from yt_dlp.extractor.youtube.pot.cache import PCPPreference
+
+
+class YoutubeIEContentProviderLogger(IEContentProviderLogger):
+    def __init__(self, ie, prefix, log_level=IEContentProviderLogger.LogLevel.INFO):
+        self.__ie = ie
+        self.prefix = prefix
+        self.log_level = log_level
+
+    def _format_msg(self, message: str):
+        prefixstr = format_field(self.prefix, None, '[%s] ')
+        return f'{prefixstr}{message}'
+
+    def trace(self, message: str):
+        if self.log_level <= self.LogLevel.TRACE:
+            self.__ie.write_debug(self._format_msg('TRACE: ' + message))
+
+    def debug(self, message: str):
+        if self.log_level <= self.LogLevel.DEBUG:
+            self.__ie.write_debug(self._format_msg(message))
+
+    def info(self, message: str):
+        if self.log_level <= self.LogLevel.INFO:
+            self.__ie.to_screen(self._format_msg(message))
+
+    def warning(self, message: str, *, once=False):
+        if self.log_level <= self.LogLevel.WARNING:
+            self.__ie.report_warning(self._format_msg(message), only_once=once)
+
+    def error(self, message: str):
+        if self.log_level <= self.LogLevel.ERROR:
+            self.__ie._downloader.report_error(self._format_msg(message), is_error=False)
+
+
+class PoTokenCache:
+
+    def __init__(
+        self,
+        logger: IEContentProviderLogger,
+        cache_providers: list[PoTokenCacheProvider],
+        cache_spec_providers: list[PoTokenCacheSpecProvider],
+        cache_provider_preferences: list[PCPPreference] | None = None,
+    ):
+        self.cache_providers: dict[str, PoTokenCacheProvider] = {
+            provider.PROVIDER_KEY: provider for provider in (cache_providers or [])
+        }
+        self.cache_provider_preferences: list[PCPPreference] = cache_provider_preferences or []
+
+        self.cache_spec_providers: dict[str, PoTokenCacheSpecProvider] = {
+            provider.PROVIDER_KEY: provider for provider in (cache_spec_providers or [])
+        }
+
+        self.logger = logger
+
+    def _get_cache_providers(self, request: PoTokenRequest) -> Iterable[PoTokenCacheProvider]:
+        """Sorts available cache providers by preference, given a request"""
+        preferences = {
+            provider: sum(pref(provider, request) for pref in self.cache_provider_preferences)
+            for provider in self.cache_providers.values()
+        }
+        if self.logger.log_level <= self.logger.LogLevel.TRACE:
+            # calling is_available() for every PO Token provider upfront may have some overhead
+            self.logger.trace(f'PO Token Cache Providers: {provider_display_list(self.cache_providers.values())}')
+            self.logger.trace('Cache Provider preferences for this request: {}'.format(', '.join(
+                f'{provider.PROVIDER_KEY}={pref}' for provider, pref in preferences.items())))
+
+        return (
+            provider for provider in sorted(self.cache_providers.values(), key=preferences.get, reverse=True) if provider.is_available()
+        )
+
+    def _get_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        for provider in self.cache_spec_providers.values():
+            if not provider.is_available():
+                continue
+            try:
+                spec = provider.generate_cache_spec(request)
+                if not spec:
+                    continue
+                if not validate_cache_spec(spec):
+                    self.logger.error(f'PoTokenCacheSpecProvider "{provider.PROVIDER_KEY}" generate_cache_spec() returned invalid spec {spec}{provider_bug_report_message(provider)}')
+                    continue
+                spec = dataclasses.replace(spec, _provider=provider)
+                self.logger.trace(f'Retrieved cache spec {spec} from cache spec provider "{provider.PROVIDER_NAME}"')
+                return spec
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache spec provider: {e!r}{provider_bug_report_message(provider)}',
+                )
+                continue
+
+    def _generate_key_bindings(self, spec: PoTokenCacheSpec) -> dict[str, str]:
+        bindings_cleaned = {
+            **{k: v for k, v in spec.key_bindings.items() if v is not None},
+            # Allow us to invalidate caches if such need arises
+            '_yt': 'v1',
+            '_p': spec._provider.PROVIDER_KEY,
+        }
+        self.logger.trace('Generate cache key bindings: {}'.format(', '.join(f'{k}={v}' for k, v in bindings_cleaned.items())))
+        return bindings_cleaned
+
+    def _generate_key(self, bindings: dict) -> str:
+        binding_string = ''.join(f'{k}{v}' for k, v in sorted(bindings.items()))
+        return hashlib.sha256(binding_string.encode()).hexdigest()
+
+    def get(self, request: PoTokenRequest) -> PoTokenResponse | None:
+        spec = self._get_cache_spec(request)
+        if not spec:
+            self.logger.trace('No cache spec available for this request, unable to fetch from cache')
+            return None
+
+        cache_key = self._generate_key(self._generate_key_bindings(spec))
+        self.logger.trace(f'Attempting to access PO Token cache using key: {cache_key}')
+
+        for idx, provider in enumerate(self._get_cache_providers(request)):
+            try:
+                self.logger.trace(f'Attempting to fetch PO Token response from "{provider.PROVIDER_NAME}" cache provider')
+                cache_response = provider.get(cache_key)
+                if not cache_response:
+                    continue
+                try:
+                    po_token_response = PoTokenResponse(**json.loads(cache_response))
+                except (TypeError, ValueError, json.JSONDecodeError):
+                    po_token_response = None
+                if not validate_response(po_token_response):
+                    self.logger.error(f'Invalid PO Token response retrieved from cache provider "{provider.PROVIDER_NAME}": {cache_response}{provider_bug_report_message(provider)}')
+                    provider.delete(cache_key)
+                    continue
+                self.logger.trace(f'PO Token response retrieved from cache using "{provider.PROVIDER_NAME}" provider: {po_token_response}')
+                if idx > 0:
+                    # Write back to the highest priority cache provider,
+                    # so we stop trying to fetch from lower priority providers
+                    self.logger.trace('Writing PO Token response to highest priority cache provider')
+                    self.store(request, po_token_response, write_policy=CacheProviderWritePolicy.WRITE_FIRST)
+
+                return po_token_response
+            except PoTokenCacheProviderError as e:
+                self.logger.warning(
+                    f'Error from "{provider.PROVIDER_NAME}" PO Token cache provider: {e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+                continue
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache provider: {e!r}{provider_bug_report_message(provider)}',
+                )
+                continue
+
+    def store(self, request: PoTokenRequest, response: PoTokenResponse, write_policy: CacheProviderWritePolicy | None = None):
+        spec = self._get_cache_spec(request)
+        if not spec:
+            self.logger.trace('No cache spec available for this request. Not caching.')
+            return
+
+        if not validate_response(response):
+            self.logger.error(f'Invalid PO Token response provided to PoTokenCache.store(): {response}{bug_reports_message()}')
+            return
+
+        cache_key = self._generate_key(self._generate_key_bindings(spec))
+        self.logger.trace(f'Attempting to access PO Token cache using key: {cache_key}')
+
+        default_expires_at = int(dt.datetime.now(dt.timezone.utc).timestamp()) + spec.default_ttl
+        cache_response = dataclasses.replace(response, expires_at=response.expires_at or default_expires_at)
+
+        write_policy = write_policy or spec.write_policy
+        self.logger.trace(f'Using write policy: {write_policy}')
+
+        for idx, provider in enumerate(self._get_cache_providers(request)):
+            try:
+                self.logger.trace(
+                    f'Caching PO Token response in "{provider.PROVIDER_NAME}" cache provider (key={cache_key}, expires_at={cache_response.expires_at})',
+                )
+                provider.store(key=cache_key, value=json.dumps(dataclasses.asdict(cache_response)), expires_at=cache_response.expires_at)
+            except PoTokenCacheProviderError as e:
+                self.logger.warning(
+                    f'Error from "{provider.PROVIDER_NAME}" PO Token cache provider: {e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache provider: {e!r}{provider_bug_report_message(provider)}',
+                )
+
+            # WRITE_FIRST should not write to lower priority providers in the case the highest priority provider fails
+            if idx == 0 and write_policy == CacheProviderWritePolicy.WRITE_FIRST:
+                return
+
+    def close(self):
+        for provider in self.cache_providers.values():
+            provider.close()
+        for spec_provider in self.cache_spec_providers.values():
+            spec_provider.close()
+
+
+class PoTokenRequestDirector:
+
+    def __init__(self, logger: IEContentProviderLogger, cache: PoTokenCache):
+        self.providers = {}
+        self.preferences = []
+        self.cache = cache
+        self.logger = logger
+
+    def register_provider(self, provider: PoTokenProvider):
+        self.providers[provider.PROVIDER_KEY] = provider
+
+    def register_preference(self, preference):
+        self.preferences.append(preference)
+
+    def _get_providers(self, request: PoTokenRequest) -> Iterable[PoTokenProvider]:
+        """Sorts available providers by preference, given a request"""
+        preferences = {
+            provider: sum(pref(provider, request) for pref in self.preferences)
+            for provider in self.providers.values()
+        }
+        if self.logger.log_level <= self.logger.LogLevel.TRACE:
+            # calling is_available() for every PO Token provider upfront may have some overhead
+            self.logger.trace(f'PO Token Providers: {provider_display_list(self.providers.values())}')
+            self.logger.trace('Provider preferences for this request: {}'.format(', '.join(
+                f'{provider.PROVIDER_NAME}={pref}' for provider, pref in preferences.items())))
+
+        return (
+            provider for provider in sorted(self.providers.values(), key=preferences.get, reverse=True) if provider.is_available()
+        )
+
+    def _get_po_token(self, request) -> PoTokenResponse | None:
+        for provider in self._get_providers(request):
+            try:
+                self.logger.trace(f'Attempting to fetch a PO Token from "{provider.PROVIDER_NAME}" provider')
+                response = provider.request_pot(request.copy())
+            except PoTokenProviderRejectedRequest as e:
+                self.logger.trace(
+                    f'PO Token Provider "{provider.PROVIDER_NAME}" rejected this request, trying next available provider. Reason: {e}')
+                continue
+            except PoTokenProviderError as e:
+                self.logger.warning(
+                    f'Error fetching PO Token from "{provider.PROVIDER_NAME}" provider: {e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+                continue
+            except Exception as e:
+                self.logger.error(
+                    f'Unexpected error when fetching PO Token from "{provider.PROVIDER_NAME}" provider: {e!r}{provider_bug_report_message(provider)}')
+                continue
+
+            self.logger.trace(f'PO Token response from "{provider.PROVIDER_NAME}" provider: {response}')
+
+            if not validate_response(response):
+                self.logger.error(
+                    f'Invalid PO Token response received from "{provider.PROVIDER_NAME}" provider: {response}{provider_bug_report_message(provider)}')
+                continue
+
+            return response
+
+        self.logger.trace('No PO Token providers were able to provide a valid PO Token')
+        return None
+
+    def get_po_token(self, request: PoTokenRequest) -> str | None:
+        if not request.bypass_cache:
+            if pot_response := self.cache.get(request):
+                return clean_pot(pot_response.po_token)
+
+        if not self.providers:
+            self.logger.trace('No PO Token providers registered')
+            return None
+
+        pot_response = self._get_po_token(request)
+        if not pot_response:
+            return None
+
+        pot_response.po_token = clean_pot(pot_response.po_token)
+
+        if pot_response.expires_at is None or pot_response.expires_at > 0:
+            self.cache.store(request, pot_response)
+        else:
+            self.logger.trace(f'PO Token response will not be cached (expires_at={pot_response.expires_at})')
+
+        return pot_response.po_token
+
+    def close(self):
+        for provider in self.providers.values():
+            provider.close()
+
+        self.cache.close()
+
+
+EXTRACTOR_ARG_PREFIX = 'youtubepot'
+
+
+def initialize_pot_director(ie):
+    if not ie._downloader:
+        raise ExtractorError('Downloader not set', expected=False)
+
+    log_level = min(
+        IEContentProviderLogger.LogLevel(ie._configuration_arg('pot_log_level', ['INFO'], ie_key='youtube', casesense=False)[0].upper()),
+        IEContentProviderLogger.LogLevel.DEBUG if ie._downloader.params.get('verbose', False) else IEContentProviderLogger.LogLevel.INFO,
+    )
+
+    cache_providers = []
+    for cache_provider in _pot_cache_providers.value.values():
+        settings = traverse_obj(ie._downloader.params, ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{cache_provider.PROVIDER_KEY.lower()}'))
+        cache_provider_logger = YoutubeIEContentProviderLogger(ie, f'pot:cache:{cache_provider.PROVIDER_NAME}', log_level=log_level)
+        cache_providers.append(cache_provider(ie, cache_provider_logger, settings or {}))
+
+    cache_spec_providers = []
+    for cache_spec_provider in _pot_pcs_providers.value.values():
+        settings = traverse_obj(ie._downloader.params, ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{cache_spec_provider.PROVIDER_KEY.lower()}'))
+        cache_spec_provider_logger = YoutubeIEContentProviderLogger(ie, f'pot:cache:spec:{cache_spec_provider.PROVIDER_NAME}', log_level=log_level)
+        cache_spec_providers.append(cache_spec_provider(ie, cache_spec_provider_logger, settings or {}))
+
+    cache = PoTokenCache(
+        logger=YoutubeIEContentProviderLogger(ie, 'pot:cache', log_level=log_level),
+        cache_providers=cache_providers,
+        cache_spec_providers=cache_spec_providers,
+        cache_provider_preferences=list(_pot_cache_provider_preferences.value),
+    )
+
+    director = PoTokenRequestDirector(
+        logger=YoutubeIEContentProviderLogger(ie, 'pot', log_level=log_level),
+        cache=cache,
+    )
+
+    ie._downloader.add_close_hook(director.close)
+
+    for provider in _pot_providers.value.values():
+        settings = traverse_obj(ie._downloader.params, ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{provider.PROVIDER_KEY.lower()}'))
+        logger = YoutubeIEContentProviderLogger(ie, f'pot:{provider.PROVIDER_NAME}', log_level=log_level)
+        director.register_provider(provider(ie, logger, settings or {}))
+
+    for preference in _ptp_preferences.value:
+        director.register_preference(preference)
+
+    if director.logger.log_level <= director.logger.LogLevel.DEBUG:
+        director.logger.debug(f'PO Token Providers: {provider_display_list(director.providers.values())}')
+        director.logger.debug(f'PO Token Cache Providers: {provider_display_list(cache.cache_providers.values())}')
+        director.logger.debug(f'PO Token Cache Spec Providers: {provider_display_list(cache.cache_spec_providers.values())}')
+        director.logger.trace(f'Registered {len(director.preferences)} provider preferences')
+        director.logger.trace(f'Registered {len(cache.cache_provider_preferences)} cache provider preferences')
+
+    return director
+
+
+def provider_display_list(providers: Iterable[IEContentProvider]):
+    def provider_display_name(provider):
+        display_str = join_nonempty(provider.PROVIDER_NAME, provider.PROVIDER_VERSION if not isinstance(provider, BuiltInIEContentProvider) else None)
+        statuses = []
+        if not isinstance(provider, BuiltInIEContentProvider):
+            statuses.append('external')
+        if not provider.is_available():
+            statuses.append('unavailable')
+        if statuses:
+            display_str += f' ({", ".join(statuses)})'
+        return display_str
+
+    return ', '.join(provider_display_name(provider) for provider in providers) or 'none'
+
+
+def clean_pot(po_token: str):
+    # Clean and validate the PO Token. This will strip invalid characters off
+    # (e.g. additional url params the user may accidentally include)
+    try:
+        return base64.urlsafe_b64encode(base64.urlsafe_b64decode(urllib.parse.unquote(po_token))).decode()
+    except (binascii.Error, ValueError):
+        raise ValueError('Invalid PO Token')
+
+
+def validate_response(response: PoTokenResponse):
+    if (
+        not isinstance(response, PoTokenResponse)
+        or not response.po_token
+        or not isinstance(response.po_token, str)
+    ):  # noqa: SIM103
+        return False
+
+    try:
+        clean_pot(response.po_token)
+    except ValueError:
+        return False
+
+    return (
+        response.expires_at is None
+        or (
+            isinstance(response.expires_at, int)
+            and (response.expires_at <= 0 or response.expires_at > int(dt.datetime.now(dt.timezone.utc).timestamp()))
+        )
+    )
+
+
+def validate_cache_spec(spec: PoTokenCacheSpec):
+
+    return (
+        isinstance(spec, PoTokenCacheSpec)
+        and isinstance(spec.write_policy, CacheProviderWritePolicy)
+        and isinstance(spec.default_ttl, int)
+        and isinstance(spec.key_bindings, dict)
+        and all(isinstance(k, str) for k in spec.key_bindings)
+        and all(v is None or isinstance(v, str) for v in spec.key_bindings.values())
+        and len({k for k in spec.key_bindings.values() if k is not None}) > 0
+    )

yt_dlp/extractor/youtube/pot/_provider.py

@@ -0,0 +1,151 @@
+from __future__ import annotations
+
+import abc
+import enum
+import functools
+
+from yt_dlp.extractor.common import InfoExtractor
+from yt_dlp.utils import NO_DEFAULT, bug_reports_message, classproperty, traverse_obj
+from yt_dlp.version import __version__
+
+# xxx: these could be generalized outside YoutubeIE eventually
+
+
+class IEContentProviderLogger(abc.ABC):
+
+    class LogLevel(enum.IntEnum):
+        TRACE = 0
+        DEBUG = 10
+        INFO = 20
+        WARNING = 30
+        ERROR = 40
+
+        @classmethod
+        def _missing_(cls, value):
+            if isinstance(value, str):
+                value = value.upper()
+                if value in dir(cls):
+                    return cls[value]
+
+            return cls.INFO
+
+    log_level = LogLevel.INFO
+
+    @abc.abstractmethod
+    def trace(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def debug(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def info(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def warning(self, message: str, *, once=False):
+        pass
+
+    @abc.abstractmethod
+    def error(self, message: str):
+        pass
+
+
+class IEContentProviderError(Exception):
+    def __init__(self, msg=None, expected=False):
+        super().__init__(msg)
+        self.expected = expected
+
+
+class IEContentProvider(abc.ABC):
+    PROVIDER_VERSION: str = '0.0.0'
+    BUG_REPORT_LOCATION: str = '(developer has not provided a bug report location)'
+
+    def __init__(self, ie: InfoExtractor, logger: IEContentProviderLogger, settings: dict[str, list[str]], *_, **__):
+        self.ie = ie
+        self.settings = settings or {}
+        self.logger = logger
+        super().__init__()
+
+    @classmethod
+    def __init_subclass__(cls, *, suffix=None, **kwargs):
+        if suffix:
+            cls._PROVIDER_KEY_SUFFIX = suffix
+        return super().__init_subclass__(**kwargs)
+
+    @classproperty
+    def PROVIDER_NAME(cls) -> str:
+        return cls.__name__[:-len(cls._PROVIDER_KEY_SUFFIX)]
+
+    @classproperty
+    def BUG_REPORT_MESSAGE(cls):
+        return f'please report this issue to the provider developer at  {cls.BUG_REPORT_LOCATION}  .'
+
+    @classproperty
+    def PROVIDER_KEY(cls) -> str:
+        assert hasattr(cls, '_PROVIDER_KEY_SUFFIX'), 'Content Provider implementation must define a suffix for the provider key'
+        assert cls.__name__.endswith(cls._PROVIDER_KEY_SUFFIX), f'PoTokenProvider class names must end with "{cls._PROVIDER_KEY_SUFFIX}"'
+        return cls.__name__[:-len(cls._PROVIDER_KEY_SUFFIX)]
+
+    @abc.abstractmethod
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method should not make any network requests or perform any expensive operations.
+         It is called multiple times.
+        """
+        raise NotImplementedError
+
+    def close(self):  # noqa: B027
+        pass
+
+    def _configuration_arg(self, key, default=NO_DEFAULT, *, casesense=False):
+        '''
+        @returns            A list of values for the setting given by "key"
+                            or "default" if no such key is present
+        @param default      The default value to return when the key is not present (default: [])
+        @param casesense    When false, the values are converted to lower case
+        '''
+        val = traverse_obj(self.settings, key)
+        if val is None:
+            return [] if default is NO_DEFAULT else default
+        return list(val) if casesense else [x.lower() for x in val]
+
+
+class BuiltInIEContentProvider(IEContentProvider, abc.ABC):
+    PROVIDER_VERSION = __version__
+    BUG_REPORT_MESSAGE = bug_reports_message(before='')
+
+
+def register_provider_generic(
+    provider,
+    base_class,
+    registry,
+):
+    """Generic function to register a provider class"""
+    assert issubclass(provider, base_class), f'{provider} must be a subclass of {base_class.__name__}'
+    assert provider.PROVIDER_KEY not in registry, f'{base_class.__name__} {provider.PROVIDER_KEY} already registered'
+    registry[provider.PROVIDER_KEY] = provider
+    return provider
+
+
+def register_preference_generic(
+    base_class,
+    registry,
+    *providers,
+):
+    """Generic function to register a preference for a provider"""
+    assert all(issubclass(provider, base_class) for provider in providers)
+
+    def outer(preference):
+        @functools.wraps(preference)
+        def inner(provider, *args, **kwargs):
+            if not providers or isinstance(provider, providers):
+                return preference(provider, *args, **kwargs)
+            return 0
+        registry.add(inner)
+        return preference
+    return outer

yt_dlp/extractor/youtube/pot/_registry.py

@@ -0,0 +1,8 @@
+from yt_dlp.globals import Indirect
+
+_pot_providers = Indirect({})
+_ptp_preferences = Indirect(set())
+_pot_pcs_providers = Indirect({})
+_pot_cache_providers = Indirect({})
+_pot_cache_provider_preferences = Indirect(set())
+_pot_memory_cache = Indirect({})

yt_dlp/extractor/youtube/pot/cache.py

@@ -0,0 +1,97 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import abc
+import dataclasses
+import enum
+import typing
+
+from yt_dlp.extractor.youtube.pot._provider import (
+    IEContentProvider,
+    IEContentProviderError,
+    register_preference_generic,
+    register_provider_generic,
+)
+from yt_dlp.extractor.youtube.pot._registry import (
+    _pot_cache_provider_preferences,
+    _pot_cache_providers,
+    _pot_pcs_providers,
+)
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+class PoTokenCacheProviderError(IEContentProviderError):
+    """An error occurred while fetching a PO Token"""
+
+
+class PoTokenCacheProvider(IEContentProvider, abc.ABC, suffix='PCP'):
+    @abc.abstractmethod
+    def get(self, key: str) -> str | None:
+        pass
+
+    @abc.abstractmethod
+    def store(self, key: str, value: str, expires_at: int):
+        pass
+
+    @abc.abstractmethod
+    def delete(self, key: str):
+        pass
+
+
+class CacheProviderWritePolicy(enum.Enum):
+    WRITE_ALL = enum.auto()    # Write to all cache providers
+    WRITE_FIRST = enum.auto()  # Write to only the first cache provider
+
+
+@dataclasses.dataclass
+class PoTokenCacheSpec:
+    key_bindings: dict[str, str | None]
+    default_ttl: int
+    write_policy: CacheProviderWritePolicy = CacheProviderWritePolicy.WRITE_ALL
+
+    # Internal
+    _provider: PoTokenCacheSpecProvider | None = None
+
+
+class PoTokenCacheSpecProvider(IEContentProvider, abc.ABC, suffix='PCSP'):
+
+    def is_available(self) -> bool:
+        return True
+
+    @abc.abstractmethod
+    def generate_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        """Generate a cache spec for the given request"""
+        pass
+
+
+def register_provider(provider: type[PoTokenCacheProvider]):
+    """Register a PoTokenCacheProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenCacheProvider,
+        registry=_pot_cache_providers.value,
+    )
+
+
+def register_spec(provider: type[PoTokenCacheSpecProvider]):
+    """Register a PoTokenCacheSpecProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenCacheSpecProvider,
+        registry=_pot_pcs_providers.value,
+    )
+
+
+# XXX: I don't think the typing is correct, and that we need py3.10 to properly type this
+def register_preference(*providers: type[PoTokenCacheProvider]) -> typing.Callable[[PCPPreference], PCPPreference]:
+    """Register a preference for a PoTokenCacheProvider"""
+    return register_preference_generic(
+        PoTokenCacheProvider,
+        _pot_cache_provider_preferences.value,
+        *providers,
+    )
+
+
+if typing.TYPE_CHECKING:
+    PCPPreference = typing.Callable[[PoTokenCacheProvider, PoTokenRequest, ...], int]

yt_dlp/extractor/youtube/pot/provider.py

@@ -0,0 +1,262 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import abc
+import copy
+import dataclasses
+import enum
+import functools
+import typing
+import urllib.parse
+
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.extractor.youtube.pot._provider import (
+    IEContentProvider,
+    IEContentProviderError,
+    register_preference_generic,
+    register_provider_generic,
+)
+from yt_dlp.extractor.youtube.pot._registry import _pot_providers, _ptp_preferences
+from yt_dlp.networking import Request, Response
+from yt_dlp.utils import traverse_obj
+from yt_dlp.utils.networking import HTTPHeaderDict
+
+__all__ = [
+    'ExternalRequestFeature',
+    'PoTokenContext',
+    'PoTokenProvider',
+    'PoTokenProviderError',
+    'PoTokenProviderRejectedRequest',
+    'PoTokenRequest',
+    'PoTokenResponse',
+    'provider_bug_report_message',
+    'register_preference',
+    'register_provider',
+]
+
+
+class PoTokenContext(enum.Enum):
+    GVS = 'gvs'
+    PLAYER = 'player'
+
+
+@dataclasses.dataclass
+class PoTokenRequest:
+    # YouTube parameters
+    context: PoTokenContext
+    innertube_context: InnertubeContext
+    innertube_host: str | None = None
+    session_index: str | None = None
+    player_url: str | None = None
+    is_authenticated: bool = False
+    video_webpage: str | None = None
+    internal_client_name: str | None = None
+
+    # Content binding parameters
+    visitor_data: str | None = None
+    data_sync_id: str | None = None
+    video_id: str | None = None
+
+    # Networking parameters
+    request_cookiejar: YoutubeDLCookieJar = dataclasses.field(default_factory=YoutubeDLCookieJar)
+    request_proxy: str | None = None
+    request_headers: HTTPHeaderDict = dataclasses.field(default_factory=HTTPHeaderDict)
+    request_timeout: float | None = None
+    request_source_address: str | None = None
+    request_verify_tls: bool = True
+
+    # Generate a new token, do not used a cached token
+    # The token should still be cached for future requests
+    bypass_cache: bool = False
+
+    def copy(self):
+        return dataclasses.replace(
+            self,
+            request_headers=HTTPHeaderDict(self.request_headers),
+            innertube_context=copy.deepcopy(self.innertube_context),
+        )
+
+
+@dataclasses.dataclass
+class PoTokenResponse:
+    po_token: str
+    expires_at: int | None = None
+
+
+class PoTokenProviderRejectedRequest(IEContentProviderError):
+    """Reject the PoTokenRequest (cannot handle the request)"""
+
+
+class PoTokenProviderError(IEContentProviderError):
+    """An error occurred while fetching a PO Token"""
+
+
+class ExternalRequestFeature(enum.Enum):
+    PROXY_SCHEME_HTTP = enum.auto()
+    PROXY_SCHEME_HTTPS = enum.auto()
+    PROXY_SCHEME_SOCKS4 = enum.auto()
+    PROXY_SCHEME_SOCKS4A = enum.auto()
+    PROXY_SCHEME_SOCKS5 = enum.auto()
+    PROXY_SCHEME_SOCKS5H = enum.auto()
+    SOURCE_ADDRESS = enum.auto()
+    DISABLE_TLS_VERIFICATION = enum.auto()
+
+
+class PoTokenProvider(IEContentProvider, abc.ABC, suffix='PTP'):
+
+    # Set to None to disable the check
+    _SUPPORTED_CONTEXTS: tuple[PoTokenContext] | None = ()
+
+    # Innertube Client Name.
+    # For example, "WEB", "ANDROID", "TVHTML5".
+    # For a list of WebPO client names, see yt_dlp.extractor.youtube.pot._builtin.utils.WEBPO_CLIENTS.
+    # Also see yt_dlp.extractor.youtube._base.INNERTUBE_CLIENTS for a list of client names currently supported by the YouTube extractor.
+    _SUPPORTED_CLIENTS: tuple[str] | None = ()
+
+    # If making external requests to websites (i.e. to youtube.com) using another library or service (i.e., not _request_webpage),
+    # add the request features that are supported.
+    # If only using _request_webpage to make external requests, set this to None.
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES: tuple[ExternalRequestFeature] | None = ()
+
+    def __validate_request(self, request: PoTokenRequest):
+        if not self.is_available():
+            raise PoTokenProviderRejectedRequest(f'{self.PROVIDER_NAME} is not available')
+
+        # Validate request using built-in settings
+        if self._SUPPORTED_CONTEXTS is not None and request.context not in self._SUPPORTED_CONTEXTS:
+            raise PoTokenProviderRejectedRequest(f'PO Token Context "{request.context}" is not supported by {self.PROVIDER_NAME}')
+
+        if self._SUPPORTED_CLIENTS is not None:
+            client_name = traverse_obj(request.innertube_context, ('client', 'clientName'))
+            if client_name not in self._SUPPORTED_CLIENTS:
+                raise PoTokenProviderRejectedRequest(
+                    f'Client "{client_name}" is not supported by {self.PROVIDER_NAME}. Supported clients: {", ".join(self._SUPPORTED_CLIENTS) or "none"}')
+
+        self.__validate_external_request_features(request)
+
+    @functools.cached_property
+    def _supported_proxy_schemes(self):
+        return {
+            scheme: feature
+            for scheme, feature in {
+                'http': ExternalRequestFeature.PROXY_SCHEME_HTTP,
+                'https': ExternalRequestFeature.PROXY_SCHEME_HTTPS,
+                'socks4': ExternalRequestFeature.PROXY_SCHEME_SOCKS4,
+                'socks4a': ExternalRequestFeature.PROXY_SCHEME_SOCKS4A,
+                'socks5': ExternalRequestFeature.PROXY_SCHEME_SOCKS5,
+                'socks5h': ExternalRequestFeature.PROXY_SCHEME_SOCKS5H,
+            }.items()
+            if feature in (self._SUPPORTED_EXTERNAL_REQUEST_FEATURES or [])
+        }
+
+    def __validate_external_request_features(self, request: PoTokenRequest):
+        if self._SUPPORTED_EXTERNAL_REQUEST_FEATURES is None:
+            return
+
+        if request.request_proxy:
+            scheme = urllib.parse.urlparse(request.request_proxy).scheme
+            if scheme.lower() not in self._supported_proxy_schemes:
+                raise PoTokenProviderRejectedRequest(
+                    f'External requests by "{self.PROVIDER_NAME}" provider do not support proxy scheme "{scheme}". Supported proxy schemes: {", ".join(self._supported_proxy_schemes) or "none"}')
+
+        if request.request_source_address and ExternalRequestFeature.SOURCE_ADDRESS not in self._SUPPORTED_EXTERNAL_REQUEST_FEATURES:
+            raise PoTokenProviderRejectedRequest(
+                f'External requests by "{self.PROVIDER_NAME}" provider do not support setting source address')
+
+        if not request.request_verify_tls and ExternalRequestFeature.DISABLE_TLS_VERIFICATION not in self._SUPPORTED_EXTERNAL_REQUEST_FEATURES:
+            raise PoTokenProviderRejectedRequest(
+                f'External requests by "{self.PROVIDER_NAME}" provider do not support ignoring TLS certificate failures')
+
+    def request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        self.__validate_request(request)
+        return self._real_request_pot(request)
+
+    @abc.abstractmethod
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        """To be implemented by subclasses"""
+        pass
+
+    # Helper functions
+
+    def _request_webpage(self, request: Request, pot_request: PoTokenRequest | None = None, note=None, **kwargs) -> Response:
+        """Make a request using the internal HTTP Client.
+        Use this instead of calling requests, urllib3 or other HTTP client libraries directly!
+
+        YouTube cookies will be automatically applied if this request is made to YouTube.
+
+        @param request: The request to make
+        @param pot_request: The PoTokenRequest to use. Request parameters will be merged from it.
+        @param note: Custom log message to display when making the request. Set to `False` to disable logging.
+
+        Tips:
+        - Disable proxy (e.g. if calling local service): Request(..., proxies={'all': None})
+        - Set request timeout:  Request(..., extensions={'timeout': 5.0})
+        """
+        req = request.copy()
+
+        # Merge some ctx request settings into the request
+        # Most of these will already be used by the configured ydl instance,
+        # however, the YouTube extractor may override some.
+        if pot_request is not None:
+            req.headers = HTTPHeaderDict(pot_request.request_headers, req.headers)
+            req.proxies = req.proxies or ({'all': pot_request.request_proxy} if pot_request.request_proxy else {})
+
+            if pot_request.request_cookiejar is not None:
+                req.extensions['cookiejar'] = req.extensions.get('cookiejar', pot_request.request_cookiejar)
+
+        if note is not False:
+            self.logger.info(str(note) if note else 'Requesting webpage')
+        return self.ie._downloader.urlopen(req)
+
+
+def register_provider(provider: type[PoTokenProvider]):
+    """Register a PoTokenProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenProvider,
+        registry=_pot_providers.value,
+    )
+
+
+def provider_bug_report_message(provider: IEContentProvider, before=';'):
+    msg = provider.BUG_REPORT_MESSAGE
+
+    before = before.rstrip()
+    if not before or before.endswith(('.', '!', '?')):
+        msg = msg[0].title() + msg[1:]
+
+    return (before + ' ' if before else '') + msg
+
+
+# XXX: I don't think the typing is correct, and that we need py3.10 to properly type this
+def register_preference(*providers: type[PoTokenProvider]) -> typing.Callable[[Preference], Preference]:
+    """Register a preference for a PoTokenProvider"""
+    return register_preference_generic(
+        PoTokenProvider,
+        _ptp_preferences.value,
+        *providers,
+    )
+
+
+if typing.TYPE_CHECKING:
+    Preference = typing.Callable[[PoTokenProvider, PoTokenRequest, ...], int]
+
+    # Barebones innertube context. There may be more fields.
+    class ClientInfo(typing.TypedDict, total=False):
+        hl: str | None
+        gl: str | None
+        remoteHost: str | None
+        deviceMake: str | None
+        deviceModel: str | None
+        visitorData: str | None
+        userAgent: str | None
+        clientName: str
+        clientVersion: str
+        osName: str | None
+        osVersion: str | None
+
+    class InnertubeContext(typing.TypedDict, total=False):
+        client: ClientInfo
+        request: dict
+        user: dict

yt_dlp/extractor/youtube/pot/utils.py

@@ -0,0 +1,65 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import base64
+import contextlib
+import enum
+import re
+import urllib.parse
+
+from yt_dlp.extractor.youtube.pot.provider import PoTokenContext, PoTokenRequest
+from yt_dlp.utils import traverse_obj
+
+__all__ = ['WEBPO_CLIENTS', 'ContentBindingType', 'get_webpo_content_binding']
+
+WEBPO_CLIENTS = (
+    'WEB',
+    'MWEB',
+    'TVHTML5',
+    'WEB_EMBEDDED_PLAYER',
+    'WEB_CREATOR',
+    'WEB_REMIX',
+    'TVHTML5_SIMPLY_EMBEDDED_PLAYER',
+)
+
+
+class ContentBindingType(enum.Enum):
+    VISITOR_DATA = 'visitor_data'
+    DATASYNC_ID = 'datasync_id'
+    VIDEO_ID = 'video_id'
+    VISITOR_ID = 'visitor_id'
+
+
+def get_webpo_content_binding(request: PoTokenRequest, webpo_clients=WEBPO_CLIENTS, bind_to_visitor_id=False) -> tuple[str | None, ContentBindingType | None]:
+    client_name = traverse_obj(request.innertube_context, ('client', 'clientName'))
+    if not client_name or client_name not in webpo_clients:
+        return None, None
+
+    if request.context == PoTokenContext.GVS or client_name in ('WEB_REMIX', ):
+        if request.is_authenticated:
+            return request.data_sync_id, ContentBindingType.DATASYNC_ID
+        else:
+            if bind_to_visitor_id:
+                visitor_id = _extract_visitor_id(request.visitor_data)
+                if visitor_id:
+                    return visitor_id, ContentBindingType.VISITOR_ID
+            return request.visitor_data, ContentBindingType.VISITOR_DATA
+
+    elif request.context == PoTokenContext.PLAYER or client_name != 'WEB_REMIX':
+        return request.video_id, ContentBindingType.VIDEO_ID
+
+
+def _extract_visitor_id(visitor_data):
+    if not visitor_data:
+        return
+
+    # Attempt to extract the visitor ID from the visitor_data protobuf
+    # xxx: ideally should use a protobuf parser
+    with contextlib.suppress(Exception):
+        visitor_id = base64.urlsafe_b64decode(urllib.parse.unquote_plus(visitor_data))[2:13].decode()
+        # check that visitor id is all letters and numbers
+        if re.fullmatch(r'[A-Za-z0-9_-]{11}', visitor_id):
+            return visitor_id
+
+    return None

yt_dlp/jsinterp.py

@@ -188,6 +188,7 @@ _COMP_OPERATORS = {'===', '!==', '==', '!=', '<=', '>=', '<', '>'}
 _NAME_RE = r'[a-zA-Z_$][\w$]*'
 _MATCHING_PARENS = dict(zip(*zip('()', '{}', '[]')))
 _QUOTES = '\'"/'
+_NESTED_BRACKETS = r'[^[\]]+(?:\[[^[\]]+(?:\[[^\]]+\])?\])?'
 
 
 class JS_Undefined:
@@ -606,15 +607,18 @@ class JSInterpreter:
 
         m = re.match(fr'''(?x)
             (?P<assign>
-                (?P<out>{_NAME_RE})(?:\[(?P<index>[^\]]+?)\])?\s*
+                (?P<out>{_NAME_RE})(?:\[(?P<index>{_NESTED_BRACKETS})\])?\s*
                 (?P<op>{"|".join(map(re.escape, set(_OPERATORS) - _COMP_OPERATORS))})?
                 =(?!=)(?P<expr>.*)$
             )|(?P<return>
                 (?!if|return|true|false|null|undefined|NaN)(?P<name>{_NAME_RE})$
+            )|(?P<attribute>
+                (?P<var>{_NAME_RE})(?:
+                    (?P<nullish>\?)?\.(?P<member>[^(]+)|
+                    \[(?P<member2>{_NESTED_BRACKETS})\]
+                )\s*
             )|(?P<indexing>
                 (?P<in>{_NAME_RE})\[(?P<idx>.+)\]$
-            )|(?P<attribute>
-                (?P<var>{_NAME_RE})(?:(?P<nullish>\?)?\.(?P<member>[^(]+)|\[(?P<member2>[^\]]+)\])\s*
             )|(?P<function>
                 (?P<fname>{_NAME_RE})\((?P<args>.*)\)$
             )''', expr)
@@ -707,7 +711,7 @@ class JSInterpreter:
                 if obj is NO_DEFAULT:
                     if variable not in self._objects:
                         try:
-                            self._objects[variable] = self.extract_object(variable)
+                            self._objects[variable] = self.extract_object(variable, local_vars)
                         except self.Exception:
                             if not nullish:
                                 raise
@@ -847,7 +851,7 @@ class JSInterpreter:
             raise self.Exception('Cannot return from an expression', expr)
         return ret
 
-    def extract_object(self, objname):
+    def extract_object(self, objname, *global_stack):
         _FUNC_NAME_RE = r'''(?:[a-zA-Z$0-9]+|"[a-zA-Z$0-9]+"|'[a-zA-Z$0-9]+')'''
         obj = {}
         obj_m = re.search(
@@ -869,7 +873,8 @@ class JSInterpreter:
         for f in fields_m:
             argnames = f.group('args').split(',')
             name = remove_quotes(f.group('key'))
-            obj[name] = function_with_repr(self.build_function(argnames, f.group('code')), f'F<{name}>')
+            obj[name] = function_with_repr(
+                self.build_function(argnames, f.group('code'), *global_stack), f'F<{name}>')
 
         return obj
 

yt_dlp/networking/__init__.py

@@ -3,6 +3,7 @@ import warnings
 
 from .common import (
     HEADRequest,
+    PATCHRequest,
     PUTRequest,
     Request,
     RequestDirector,

yt_dlp/networking/_curlcffi.py

@@ -6,7 +6,8 @@ import math
 import re
 import urllib.parse
 
-from ._helper import InstanceStoreMixin, select_proxy
+from ._helper import InstanceStoreMixin
+from ..utils.networking import select_proxy
 from .common import (
     Features,
     Request,

yt_dlp/networking/_helper.py

@@ -13,7 +13,6 @@ import urllib.request
 from .exceptions import RequestError
 from ..dependencies import certifi
 from ..socks import ProxyType, sockssocket
-from ..utils import format_field, traverse_obj
 
 if typing.TYPE_CHECKING:
     from collections.abc import Iterable
@@ -82,19 +81,6 @@ def make_socks_proxy_opts(socks_proxy):
     }
 
 
-def select_proxy(url, proxies):
-    """Unified proxy selector for all backends"""
-    url_components = urllib.parse.urlparse(url)
-    if 'no' in proxies:
-        hostport = url_components.hostname + format_field(url_components.port, None, ':%s')
-        if urllib.request.proxy_bypass_environment(hostport, {'no': proxies['no']}):
-            return
-        elif urllib.request.proxy_bypass(hostport):  # check system settings
-            return
-
-    return traverse_obj(proxies, url_components.scheme or 'http', 'all')
-
-
 def get_redirect_method(method, status):
     """Unified redirect method handling"""
 

yt_dlp/networking/_requests.py

@@ -10,7 +10,7 @@ import warnings
 
 from ..dependencies import brotli, requests, urllib3
 from ..utils import bug_reports_message, int_or_none, variadic
-from ..utils.networking import normalize_url
+from ..utils.networking import normalize_url, select_proxy
 
 if requests is None:
     raise ImportError('requests module is not installed')
@@ -41,7 +41,6 @@ from ._helper import (
     create_socks_proxy_socket,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
 )
 from .common import (
     Features,

yt_dlp/networking/_urllib.py

@@ -26,7 +26,6 @@ from ._helper import (
     create_socks_proxy_socket,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
 )
 from .common import Features, RequestHandler, Response, register_rh
 from .exceptions import (
@@ -41,7 +40,7 @@ from .exceptions import (
 from ..dependencies import brotli
 from ..socks import ProxyError as SocksProxyError
 from ..utils import update_url_query
-from ..utils.networking import normalize_url
+from ..utils.networking import normalize_url, select_proxy
 
 SUPPORTED_ENCODINGS = ['gzip', 'deflate']
 CONTENT_DECODE_ERRORS = [zlib.error, OSError]

yt_dlp/networking/_websockets.py

@@ -11,8 +11,8 @@ from ._helper import (
     create_connection,
     create_socks_proxy_socket,
     make_socks_proxy_opts,
-    select_proxy,
 )
+from ..utils.networking import select_proxy
 from .common import Features, Response, register_rh
 from .exceptions import (
     CertificateVerifyError,

yt_dlp/networking/common.py

@@ -505,6 +505,7 @@ class Request:
 
 
 HEADRequest = functools.partial(Request, method='HEAD')
+PATCHRequest = functools.partial(Request, method='PATCH')
 PUTRequest = functools.partial(Request, method='PUT')
 
 

yt_dlp/options.py

@@ -500,7 +500,8 @@ def create_parser():
                 'youtube-dlc': ['all', '-no-youtube-channel-redirect', '-no-live-chat', '-playlist-match-filter', '-manifest-filesize-approx', '-allow-unsafe-ext', '-prefer-vp9-sort'],
                 '2021': ['2022', 'no-certifi', 'filename-sanitization'],
                 '2022': ['2023', 'no-external-downloader-progress', 'playlist-match-filter', 'prefer-legacy-http-handler', 'manifest-filesize-approx'],
-                '2023': ['prefer-vp9-sort'],
+                '2023': ['2024', 'prefer-vp9-sort'],
+                '2024': [],
             },
         }, help=(
             'Options that can help keep compatibility with youtube-dl or youtube-dlc '

yt_dlp/utils/_utils.py

@@ -2044,7 +2044,7 @@ def url_or_none(url):
     if not url or not isinstance(url, str):
         return None
     url = url.strip()
-    return url if re.match(r'(?:(?:https?|rt(?:m(?:pt?[es]?|fp)|sp[su]?)|mms|ftps?):)?//', url) else None
+    return url if re.match(r'(?:(?:https?|rt(?:m(?:pt?[es]?|fp)|sp[su]?)|mms|ftps?|wss?):)?//', url) else None
 
 
 def strftime_or_none(timestamp, date_format='%Y%m%d', default=None):

yt_dlp/utils/networking.py

@@ -10,7 +10,8 @@ import urllib.request
 if typing.TYPE_CHECKING:
     T = typing.TypeVar('T')
 
-from ._utils import NO_DEFAULT, remove_start
+from ._utils import NO_DEFAULT, remove_start, format_field
+from .traversal import traverse_obj
 
 
 def random_user_agent():
@@ -278,3 +279,16 @@ def normalize_url(url):
         query=escape_rfc3986(url_parsed.query),
         fragment=escape_rfc3986(url_parsed.fragment),
     ).geturl()
+
+
+def select_proxy(url, proxies):
+    """Unified proxy selector for all backends"""
+    url_components = urllib.parse.urlparse(url)
+    if 'no' in proxies:
+        hostport = url_components.hostname + format_field(url_components.port, None, ':%s')
+        if urllib.request.proxy_bypass_environment(hostport, {'no': proxies['no']}):
+            return
+        elif urllib.request.proxy_bypass(hostport):  # check system settings
+            return
+
+    return traverse_obj(proxies, url_components.scheme or 'http', 'all')

yt_dlp/version.py

@@ -1,8 +1,8 @@
 # Autogenerated by devscripts/update-version.py
 
-__version__ = '2025.03.25'
+__version__ = '2025.03.31'
 
-RELEASE_GIT_HEAD = '9dde546e7ee3e1515d88ee3af08b099351455dc0'
+RELEASE_GIT_HEAD = '5e457af57fae9645b1b8fa0ed689229c8fb9656b'
 
 VARIANT = None
 
@@ -12,4 +12,4 @@ CHANNEL = 'stable'
 
 ORIGIN = 'yt-dlp/yt-dlp'
 
-_pkg_version = '2025.03.25'
+_pkg_version = '2025.03.31'