clarify class name requirements

Closes #10823

Authored by: subrat-lima

.github/workflows/build.yml

@@ -192,7 +192,7 @@ jobs:
         with:
           path: ./repo
       - name: Virtualized Install, Prepare & Build
-        uses: yt-dlp/run-on-arch-action@v2
+        uses: yt-dlp/run-on-arch-action@v3
         with:
           # Ref: https://github.com/uraimo/run-on-arch-action/issues/55
           env: |
@@ -411,7 +411,7 @@ jobs:
         run: | # Custom pyinstaller built with https://github.com/yt-dlp/pyinstaller-builds
           python devscripts/install_deps.py -o --include build
           python devscripts/install_deps.py --include curl-cffi
-          python -m pip install -U "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-6.11.1-py3-none-any.whl"
+          python -m pip install -U "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-6.13.0-py3-none-any.whl"
 
       - name: Prepare
         run: |
@@ -460,7 +460,7 @@ jobs:
         run: |
           python devscripts/install_deps.py -o --include build
           python devscripts/install_deps.py
-          python -m pip install -U "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-6.11.1-py3-none-any.whl"
+          python -m pip install -U "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-6.13.0-py3-none-any.whl"
 
       - name: Prepare
         run: |

README.md

@@ -1792,7 +1792,7 @@ The following extractors use this feature:
 #### youtube
 * `lang`: Prefer translated metadata (`title`, `description` etc) of this language code (case-sensitive). By default, the video primary language metadata is preferred, with a fallback to `en` translated. See [youtube.py](https://github.com/yt-dlp/yt-dlp/blob/c26f9b991a0681fd3ea548d535919cec1fbbd430/yt_dlp/extractor/youtube.py#L381-L390) for list of supported content language codes
 * `skip`: One or more of `hls`, `dash` or `translated_subs` to skip extraction of the m3u8 manifests, dash manifests and [auto-translated subtitles](https://github.com/yt-dlp/yt-dlp/issues/4090#issuecomment-1158102032) respectively
-* `player_client`: Clients to extract video data from. The currently available clients are `web`, `web_safari`, `web_embedded`, `web_music`, `web_creator`, `mweb`, `ios`, `android`, `android_vr`, `tv` and `tv_embedded`. By default, `tv,ios,web` is used, or `tv,web` is used when authenticating with cookies. The `web_music` client is added for `music.youtube.com` URLs when logged-in cookies are used. The `tv_embedded` and `web_creator` clients are added for age-restricted videos if account age-verification is required. Some clients, such as `web` and `web_music`, require a `po_token` for their formats to be downloadable. Some clients, such as `web_creator`, will only work with authentication. Not all clients support authentication via cookies. You can use `default` for the default clients, or you can use `all` for all clients (not recommended). You can prefix a client with `-` to exclude it, e.g. `youtube:player_client=default,-ios`
+* `player_client`: Clients to extract video data from. The currently available clients are `web`, `web_safari`, `web_embedded`, `web_music`, `web_creator`, `mweb`, `ios`, `android`, `android_vr`, `tv` and `tv_embedded`. By default, `tv,ios,web` is used, or `tv,web` is used when authenticating with cookies. The `web_music` client is added for `music.youtube.com` URLs when logged-in cookies are used. The `web_embedded` client is added for age-restricted videos but only works if the video is embeddable. The `tv_embedded` and `web_creator` clients are added for age-restricted videos if account age-verification is required. Some clients, such as `web` and `web_music`, require a `po_token` for their formats to be downloadable. Some clients, such as `web_creator`, will only work with authentication. Not all clients support authentication via cookies. You can use `default` for the default clients, or you can use `all` for all clients (not recommended). You can prefix a client with `-` to exclude it, e.g. `youtube:player_client=default,-ios`
 * `player_skip`: Skip some network requests that are generally needed for robust extraction. One or more of `configs` (skip client configs), `webpage` (skip initial webpage), `js` (skip js player), `initial_data` (skip initial data/next ep request). While these options can help reduce the number of requests needed or avoid some rate-limiting, they could cause issues such as missing formats or metadata.  See [#860](https://github.com/yt-dlp/yt-dlp/pull/860) and [#12826](https://github.com/yt-dlp/yt-dlp/issues/12826) for more details
 * `player_params`: YouTube player parameters to use for player requests. Will overwrite any default ones set by yt-dlp.
 * `comment_sort`: `top` or `new` (default) - choose comment sorting mode (on YouTube's side)
@@ -1804,8 +1804,13 @@ The following extractors use this feature:
 * `raise_incomplete_data`: `Incomplete Data Received` raises an error instead of reporting a warning
 * `data_sync_id`: Overrides the account Data Sync ID used in Innertube API requests. This may be needed if you are using an account with `youtube:player_skip=webpage,configs` or `youtubetab:skip=webpage`
 * `visitor_data`: Overrides the Visitor Data used in Innertube API requests. This should be used with `player_skip=webpage,configs` and without cookies. Note: this may have adverse effects if used improperly. If a session from a browser is wanted, you should pass cookies instead (which contain the Visitor ID)
-* `po_token`:  Proof of Origin (PO) Token(s) to use. Comma seperated list of PO Tokens in the format `CLIENT.CONTEXT+PO_TOKEN`, e.g. `youtube:po_token=web.gvs+XXX,web.player=XXX,web_safari.gvs+YYY`. Context can be either `gvs` (Google Video Server URLs) or `player` (Innertube player request)
 * `player_js_variant`: The player javascript variant to use for signature and nsig deciphering. The known variants are: `main`, `tce`, `tv`, `tv_es6`, `phone`, `tablet`. Only `main` is recommended as a possible workaround; the others are for debugging purposes. The default is to use what is prescribed by the site, and can be selected with `actual`
+* `po_token`:  Proof of Origin (PO) Token(s) to use. Comma seperated list of PO Tokens in the format `CLIENT.CONTEXT+PO_TOKEN`, e.g. `youtube:po_token=web.gvs+XXX,web.player=XXX,web_safari.gvs+YYY`. Context can be either `gvs` (Google Video Server URLs) or `player` (Innertube player request)
+* `pot_trace`: Enable debug logging for PO Token fetching. Either `true` or `false` (default)
+* `fetch_pot`: Policy to use for fetching a PO Token from providers. `always` to always try fetch a PO Token regardless if the client requires one for the given context, `never` to never fetch a PO Token, or `auto` to only fetch a PO Token if the client requires one for the given context (default)
+
+#### youtubepot-webpo
+* `bind_to_visitor_id`: Whether to use the Visitor ID instead of Visitor Data for caching WebPO tokens. Either `true` or `false` (default `true`)
 
 #### youtubetab (YouTube playlists, channels, feeds, etc.)
 * `skip`: One or more of `webpage` (skip initial webpage download), `authcheck` (allow the download of playlists requiring authentication when no initial webpage is downloaded. This may cause unwanted behavior, see [#1122](https://github.com/yt-dlp/yt-dlp/pull/1122) for more details)

pyproject.toml

@@ -82,7 +82,7 @@ test = [
     "pytest-rerunfailures~=14.0",
 ]
 pyinstaller = [
-    "pyinstaller>=6.11.1",  # Windows temp cleanup fixed in 6.11.1
+    "pyinstaller>=6.13.0",  # Windows temp cleanup fixed in 6.13.0
 ]
 
 [project.urls]

test/test_YoutubeDL.py

@@ -1435,6 +1435,27 @@ class TestYoutubeDL(unittest.TestCase):
         FakeYDL().close()
         assert all_plugins_loaded.value
 
+    def test_close_hooks(self):
+        # Should call all registered close hooks on close
+        close_hook_called = False
+        close_hook_two_called = False
+
+        def close_hook():
+            nonlocal close_hook_called
+            close_hook_called = True
+
+        def close_hook_two():
+            nonlocal close_hook_two_called
+            close_hook_two_called = True
+
+        ydl = FakeYDL()
+        ydl.add_close_hook(close_hook)
+        ydl.add_close_hook(close_hook_two)
+
+        ydl.close()
+        self.assertTrue(close_hook_called, 'Close hook was not called')
+        self.assertTrue(close_hook_two_called, 'Close hook two was not called')
+
 
 if __name__ == '__main__':
     unittest.main()

test/test_networking_utils.py

@@ -20,7 +20,6 @@ from yt_dlp.networking._helper import (
     add_accept_encoding_header,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
     ssl_load_certs,
 )
 from yt_dlp.networking.exceptions import (
@@ -28,7 +27,7 @@ from yt_dlp.networking.exceptions import (
     IncompleteRead,
 )
 from yt_dlp.socks import ProxyType
-from yt_dlp.utils.networking import HTTPHeaderDict
+from yt_dlp.utils.networking import HTTPHeaderDict, select_proxy
 
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 

test/test_pot/conftest.py

@@ -0,0 +1,74 @@
+import pytest
+
+from yt_dlp import YoutubeDL
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.extractor.common import InfoExtractor
+from yt_dlp.extractor.youtube.pot._provider import IEContentProviderLogger
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest, PoTokenContext
+from yt_dlp.utils.networking import HTTPHeaderDict
+
+
+class MockLogger(IEContentProviderLogger):
+
+    log_level = IEContentProviderLogger.LogLevel.TRACE
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.messages = {}
+
+    def trace(self, message: str):
+        self.messages.setdefault('trace', []).append(message)
+        pass
+
+    def debug(self, message: str):
+        self.messages.setdefault('debug', []).append(message)
+        pass
+
+    def info(self, message: str):
+        self.messages.setdefault('info', []).append(message)
+        pass
+
+    def warning(self, message: str, *, once=False):
+        self.messages.setdefault('warning', []).append(message)
+        pass
+
+    def error(self, message: str):
+        self.messages.setdefault('error', []).append(message)
+        pass
+
+
+@pytest.fixture
+def ie() -> InfoExtractor:
+    ydl = YoutubeDL()
+    return ydl.get_info_extractor('Youtube')
+
+
+@pytest.fixture
+def logger() -> MockLogger:
+    return MockLogger()
+
+
+@pytest.fixture()
+def pot_request() -> PoTokenRequest:
+    return PoTokenRequest(
+        context=PoTokenContext.GVS,
+        innertube_context={'client': {'clientName': 'WEB'}},
+        innertube_host='youtube.com',
+        session_index=None,
+        player_url=None,
+        is_authenticated=False,
+        video_webpage=None,
+
+        visitor_data='example-visitor-data',
+        data_sync_id='example-data-sync-id',
+        video_id='example-video-id',
+
+        request_cookiejar=YoutubeDLCookieJar(),
+        request_proxy=None,
+        request_headers=HTTPHeaderDict(),
+        request_timeout=None,
+        request_source_address=None,
+        request_verify_tls=True,
+
+        bypass_cache=False,
+    )

test/test_pot/test_pot_builtin_memorycache.py

@@ -0,0 +1,117 @@
+import threading
+import time
+from collections import OrderedDict
+import pytest
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider, BuiltInIEContentProvider
+from yt_dlp.utils import bug_reports_message
+from yt_dlp.extractor.youtube.pot._builtin.memory_cache import MemoryLRUPCP, memorylru_preference, initialize_global_cache
+from yt_dlp.version import __version__
+from yt_dlp.extractor.youtube.pot._registry import _pot_cache_providers, _pot_memory_cache
+
+
+class TestMemoryLRUPCS:
+
+    def test_base_type(self):
+        assert issubclass(MemoryLRUPCP, IEContentProvider)
+        assert issubclass(MemoryLRUPCP, BuiltInIEContentProvider)
+
+    @pytest.fixture
+    def pcp(self, ie, logger) -> MemoryLRUPCP:
+        return MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), max_size))
+
+    def test_is_registered(self):
+        assert _pot_cache_providers.value.get('MemoryLRU') == MemoryLRUPCP
+
+    def test_initialization(self, pcp):
+        assert pcp.PROVIDER_NAME == 'memory'
+        assert pcp.PROVIDER_VERSION == __version__
+        assert pcp.BUG_REPORT_MESSAGE == bug_reports_message(before='')
+        assert pcp.is_available()
+
+    def test_store_and_get(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value1'
+        assert len(pcp.cache) == 1
+
+    def test_store_ignore_expired(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) - 1)
+        assert len(pcp.cache) == 0
+        assert pcp.get('key1') is None
+        assert len(pcp.cache) == 0
+
+    def test_store_override_existing_key(self, ie, logger):
+        MAX_SIZE = 2
+        pcp = MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), MAX_SIZE))
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        pcp.store('key2', 'value2', int(time.time()) + 60)
+        assert len(pcp.cache) == 2
+        pcp.store('key1', 'value2', int(time.time()) + 60)
+        # Ensure that the override key gets added to the end of the cache instead of in the same position
+        pcp.store('key3', 'value3', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value2'
+
+    def test_store_ignore_expired_existing_key(self, pcp):
+        pcp.store('key1', 'value2', int(time.time()) + 60)
+        pcp.store('key1', 'value1', int(time.time()) - 1)
+        assert len(pcp.cache) == 1
+        assert pcp.get('key1') == 'value2'
+        assert len(pcp.cache) == 1
+
+    def test_get_key_expired(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 60)
+        assert pcp.get('key1') == 'value1'
+        assert len(pcp.cache) == 1
+        pcp.cache['key1'] = ('value1', int(time.time()) - 1)
+        assert pcp.get('key1') is None
+        assert len(pcp.cache) == 0
+
+    def test_lru_eviction(self, ie, logger):
+        MAX_SIZE = 2
+        provider = MemoryLRUPCP(ie, logger, {}, initialize_cache=lambda max_size: (OrderedDict(), threading.Lock(), MAX_SIZE))
+        provider.store('key1', 'value1', int(time.time()) + 5)
+        provider.store('key2', 'value2', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key1') == 'value1'
+
+        provider.store('key3', 'value3', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key2') is None
+
+        provider.store('key4', 'value4', int(time.time()) + 5)
+        assert len(provider.cache) == 2
+
+        assert provider.get('key1') is None
+        assert provider.get('key3') == 'value3'
+        assert provider.get('key4') == 'value4'
+
+    def test_delete(self, pcp):
+        pcp.store('key1', 'value1', int(time.time()) + 5)
+        assert len(pcp.cache) == 1
+        assert pcp.get('key1') == 'value1'
+        pcp.delete('key1')
+        assert len(pcp.cache) == 0
+        assert pcp.get('key1') is None
+
+    def test_use_global_cache_default(self, ie, logger):
+        pcp = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+        assert pcp.cache is _pot_memory_cache.value['cache']
+        assert pcp.lock is _pot_memory_cache.value['lock']
+
+        pcp2 = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == pcp2.max_size == _pot_memory_cache.value['max_size'] == 25
+        assert pcp.cache is pcp2.cache is _pot_memory_cache.value['cache']
+        assert pcp.lock is pcp2.lock is _pot_memory_cache.value['lock']
+
+    def test_fail_max_size_change_global(self, ie, logger):
+        pcp = MemoryLRUPCP(ie, logger, {})
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+        with pytest.raises(ValueError, match='Cannot change max_size of initialized global memory cache'):
+            initialize_global_cache(50)
+
+        assert pcp.max_size == _pot_memory_cache.value['max_size'] == 25
+
+    def test_memory_lru_preference(self, pcp, ie, pot_request):
+        assert memorylru_preference(pcp, pot_request) == 10000

test/test_pot/test_pot_builtin_utils.py

@@ -0,0 +1,46 @@
+import pytest
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenContext,
+
+)
+
+from yt_dlp.extractor.youtube.pot.utils import get_webpo_content_binding, ContentBindingType
+
+
+class TestGetWebPoContentBinding:
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated, expected', [
+        *[(client, context, is_authenticated, expected) for client in [
+            'WEB', 'MWEB', 'TVHTML5', 'WEB_EMBEDDED_PLAYER', 'WEB_CREATOR', 'TVHTML5_SIMPLY_EMBEDDED_PLAYER']
+          for context, is_authenticated, expected in [
+            (PoTokenContext.GVS, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+            (PoTokenContext.PLAYER, False, ('example-video-id', ContentBindingType.VIDEO_ID)),
+            (PoTokenContext.GVS, True, ('example-data-sync-id', ContentBindingType.DATASYNC_ID)),
+        ]],
+        ('WEB_REMIX', PoTokenContext.GVS, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+        ('WEB_REMIX', PoTokenContext.PLAYER, False, ('example-visitor-data', ContentBindingType.VISITOR_DATA)),
+        ('ANDROID', PoTokenContext.GVS, False, (None, None)),
+        ('IOS', PoTokenContext.GVS, False, (None, None)),
+    ])
+    def test_get_webpo_content_binding(self, pot_request, client_name, context, is_authenticated, expected):
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        assert get_webpo_content_binding(pot_request) == expected
+
+    def test_extract_visitor_id(self, pot_request):
+        pot_request.visitor_data = 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == ('123abcXYZ_-', ContentBindingType.VISITOR_ID)
+
+    def test_invalid_visitor_id(self, pot_request):
+        # visitor id not alphanumeric (i.e. protobuf extraction failed)
+        pot_request.visitor_data = 'CggxMjM0NTY3OCiA4s-qBg%3D%3D'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)
+
+    def test_no_visitor_id(self, pot_request):
+        pot_request.visitor_data = 'KIDiz6oG'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)
+
+    def test_invalid_base64(self, pot_request):
+        pot_request.visitor_data = 'invalid-base64'
+        assert get_webpo_content_binding(pot_request, bind_to_visitor_id=True) == (pot_request.visitor_data, ContentBindingType.VISITOR_DATA)

test/test_pot/test_pot_builtin_webpospec.py

@@ -0,0 +1,92 @@
+import pytest
+
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider, BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot.cache import CacheProviderWritePolicy
+from yt_dlp.utils import bug_reports_message
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+
+)
+from yt_dlp.version import __version__
+
+from yt_dlp.extractor.youtube.pot._builtin.webpo_cachespec import WebPoPCSP
+from yt_dlp.extractor.youtube.pot._registry import _pot_pcs_providers
+
+
+@pytest.fixture()
+def pot_request(pot_request) -> PoTokenRequest:
+    pot_request.visitor_data = 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D'  # visitor_id=123abcXYZ_-
+    return pot_request
+
+
+class TestWebPoPCSP:
+    def test_base_type(self):
+        assert issubclass(WebPoPCSP, IEContentProvider)
+        assert issubclass(WebPoPCSP, BuiltInIEContentProvider)
+
+    def test_init(self, ie, logger):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        assert pcs.PROVIDER_NAME == 'webpo'
+        assert pcs.PROVIDER_VERSION == __version__
+        assert pcs.BUG_REPORT_MESSAGE == bug_reports_message(before='')
+        assert pcs.is_available()
+
+    def test_is_registered(self):
+        assert _pot_pcs_providers.value.get('WebPo') == WebPoPCSP
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated', [
+        ('ANDROID', PoTokenContext.GVS, False),
+        ('IOS', PoTokenContext.GVS, False),
+        ('IOS', PoTokenContext.PLAYER, False),
+    ])
+    def test_not_supports(self, ie, logger, pot_request, client_name, context, is_authenticated):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        assert pcs.generate_cache_spec(pot_request) is None
+
+    @pytest.mark.parametrize('client_name, context, is_authenticated, remote_host, source_address, request_proxy, expected', [
+        *[(client, context, is_authenticated, remote_host, source_address, request_proxy, expected) for client in [
+            'WEB', 'MWEB', 'TVHTML5', 'WEB_EMBEDDED_PLAYER', 'WEB_CREATOR', 'TVHTML5_SIMPLY_EMBEDDED_PLAYER']
+          for context, is_authenticated, remote_host, source_address, request_proxy, expected in [
+            (PoTokenContext.GVS, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id'}),
+            (PoTokenContext.PLAYER, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'video_id'}),
+            (PoTokenContext.GVS, True, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': 'example-data-sync-id', 'cbt': 'datasync_id'}),
+        ]],
+        ('WEB_REMIX', PoTokenContext.PLAYER, False, 'example-remote-host', 'example-source-address', 'example-request-proxy', {'t': 'webpo', 'ip': 'example-remote-host', 'sa': 'example-source-address', 'px': 'example-request-proxy', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id'}),
+        ('WEB', PoTokenContext.GVS, False, None, None, None, {'t': 'webpo', 'cb': '123abcXYZ_-', 'cbt': 'visitor_id', 'ip': None, 'sa': None, 'px': None}),
+        ('TVHTML5', PoTokenContext.PLAYER, False, None, None, 'http://example.com', {'t': 'webpo', 'cb': '123abcXYZ_-', 'cbt': 'video_id', 'ip': None, 'sa': None, 'px': 'http://example.com'}),
+
+    ])
+    def test_generate_key_bindings(self, ie, logger, pot_request, client_name, context, is_authenticated, remote_host, source_address, request_proxy, expected):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = client_name
+        pot_request.context = context
+        pot_request.is_authenticated = is_authenticated
+        pot_request.innertube_context['client']['remoteHost'] = remote_host
+        pot_request.request_source_address = source_address
+        pot_request.request_proxy = request_proxy
+        pot_request.video_id = '123abcXYZ_-'  # same as visitor id to test type
+
+        assert pcs.generate_cache_spec(pot_request).key_bindings == expected
+
+    def test_no_bind_visitor_id(self, ie, logger, pot_request):
+        # Should not bind to visitor id if setting is set to False
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={'bind_to_visitor_id': ['false']})
+        pot_request.innertube_context['client']['clientName'] = 'WEB'
+        pot_request.context = PoTokenContext.GVS
+        pot_request.is_authenticated = False
+        assert pcs.generate_cache_spec(pot_request).key_bindings == {'t': 'webpo', 'ip': None, 'sa': None, 'px': None, 'cb': 'CgsxMjNhYmNYWVpfLSiA4s%2DqBg%3D%3D', 'cbt': 'visitor_data'}
+
+    def test_default_ttl(self, ie, logger, pot_request):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        assert pcs.generate_cache_spec(pot_request).default_ttl == 6 * 60 * 60  # should default to 6 hours
+
+    def test_write_policy(self, ie, logger, pot_request):
+        pcs = WebPoPCSP(ie=ie, logger=logger, settings={})
+        pot_request.context = PoTokenContext.GVS
+        assert pcs.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_ALL
+        pot_request.context = PoTokenContext.PLAYER
+        assert pcs.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_FIRST

test/test_pot/test_pot_framework.py

@@ -0,0 +1,629 @@
+import pytest
+
+from yt_dlp.extractor.youtube.pot._provider import IEContentProvider
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.utils.networking import HTTPHeaderDict
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+    ExternalRequestFeature,
+
+)
+
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    CacheProviderWritePolicy,
+)
+
+import yt_dlp.extractor.youtube.pot.cache as cache
+
+from yt_dlp.networking import Request
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenResponse,
+    PoTokenProvider,
+    PoTokenProviderRejectedRequest,
+    provider_bug_report_message,
+    register_provider,
+    register_preference,
+)
+
+from yt_dlp.extractor.youtube.pot._registry import _pot_providers, _ptp_preferences, _pot_pcs_providers, _pot_cache_providers, _pot_cache_provider_preferences
+
+
+class ExamplePTP(PoTokenProvider):
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    _SUPPORTED_CLIENTS = ('WEB',)
+    _SUPPORTED_CONTEXTS = (PoTokenContext.GVS, )
+
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+        ExternalRequestFeature.PROXY_SCHEME_HTTP,
+        ExternalRequestFeature.PROXY_SCHEME_SOCKS5H,
+    )
+
+    def is_available(self) -> bool:
+        return True
+
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        return PoTokenResponse('example-token', expires_at=123)
+
+
+class ExampleCacheProviderPCP(PoTokenCacheProvider):
+
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    def is_available(self) -> bool:
+        return True
+
+    def get(self, key: str):
+        return 'example-cache'
+
+    def store(self, key: str, value: str, expires_at: int):
+        pass
+
+    def delete(self, key: str):
+        pass
+
+
+class ExampleCacheSpecProviderPCSP(PoTokenCacheSpecProvider):
+
+    PROVIDER_NAME = 'example'
+    PROVIDER_VERSION = '0.0.1'
+    BUG_REPORT_LOCATION = 'https://example.com/issues'
+
+    def generate_cache_spec(self, request: PoTokenRequest):
+        return PoTokenCacheSpec(
+            key_bindings={'field': 'example-key'},
+            default_ttl=60,
+            write_policy=CacheProviderWritePolicy.WRITE_FIRST,
+        )
+
+
+class TestPoTokenProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenProvider, IEContentProvider)
+
+    def test_create_provider_missing_fetch_method(self, ie, logger):
+        class MissingMethodsPTP(PoTokenProvider):
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPTP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_available_method(self, ie, logger):
+        class MissingMethodsPTP(PoTokenProvider):
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+        with pytest.raises(TypeError):
+            MissingMethodsPTP(ie=ie, logger=logger, settings={})
+
+    def test_barebones_provider(self, ie, logger):
+        class BarebonesProviderPTP(PoTokenProvider):
+            def is_available(self) -> bool:
+                return True
+
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+        provider = BarebonesProviderPTP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+
+    def test_example_provider_success(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'Example'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+
+        response = provider.request_pot(pot_request)
+
+        assert response.po_token == 'example-token'
+        assert response.expires_at == 123
+
+    def test_provider_unsupported_context(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.context = PoTokenContext.PLAYER
+
+        with pytest.raises(PoTokenProviderRejectedRequest):
+            provider.request_pot(pot_request)
+
+    def test_provider_unsupported_client(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.innertube_context['client']['clientName'] = 'ANDROID'
+
+        with pytest.raises(PoTokenProviderRejectedRequest):
+            provider.request_pot(pot_request)
+
+    def test_provider_unsupported_proxy_scheme(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+        pot_request.request_proxy = 'socks4://example.com'
+
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support proxy scheme "socks4". Supported proxy '
+            'schemes: http, socks5h',
+        ):
+            provider.request_pot(pot_request)
+
+        pot_request.request_proxy = 'http://example.com'
+
+        assert provider.request_pot(pot_request)
+
+    def test_provider_ignore_external_request_features(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = None
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_proxy = 'socks5://example.com'
+        assert provider.request_pot(pot_request)
+        pot_request.request_source_address = '0.0.0.0'
+        assert provider.request_pot(pot_request)
+
+    def test_provider_unsupported_external_request_source_address(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = tuple()
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_source_address = None
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_source_address = '0.0.0.0'
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support setting source address',
+        ):
+            provider.request_pot(pot_request)
+
+    def test_provider_supported_external_request_source_address(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+                ExternalRequestFeature.SOURCE_ADDRESS,
+            )
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_source_address = None
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_source_address = '0.0.0.0'
+        assert provider.request_pot(pot_request)
+
+    def test_provider_unsupported_external_request_tls_verification(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = tuple()
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_verify_tls = True
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_verify_tls = False
+        with pytest.raises(
+            PoTokenProviderRejectedRequest,
+            match='External requests by "example" provider do not support ignoring TLS certificate failures',
+        ):
+            provider.request_pot(pot_request)
+
+    def test_provider_supported_external_request_tls_verification(self, ie, logger, pot_request):
+        class InternalPTP(ExamplePTP):
+            _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+                ExternalRequestFeature.DISABLE_TLS_VERIFICATION,
+            )
+
+        provider = InternalPTP(ie=ie, logger=logger, settings={})
+
+        pot_request.request_verify_tls = True
+        assert provider.request_pot(pot_request)
+
+        pot_request.request_verify_tls = False
+        assert provider.request_pot(pot_request)
+
+    def test_provider_request_webpage(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        cookiejar = YoutubeDLCookieJar()
+        pot_request.request_headers = HTTPHeaderDict({'User-Agent': 'example-user-agent'})
+        pot_request.request_proxy = 'socks5://example-proxy.com'
+        pot_request.request_cookiejar = cookiejar
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), pot_request=pot_request)
+
+        assert sent_request.url == 'https://example.com'
+        assert sent_request.headers['User-Agent'] == 'example-user-agent'
+        assert sent_request.proxies == {'all': 'socks5://example-proxy.com'}
+        assert sent_request.extensions['cookiejar'] is cookiejar
+        assert 'Requesting webpage' in logger.messages['info']
+
+    def test_provider_request_webpage_override(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        cookiejar_request = YoutubeDLCookieJar()
+        pot_request.request_headers = HTTPHeaderDict({'User-Agent': 'example-user-agent'})
+        pot_request.request_proxy = 'socks5://example-proxy.com'
+        pot_request.request_cookiejar = cookiejar_request
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+            headers={'User-Agent': 'override-user-agent-override'},
+            proxies={'http': 'http://example-proxy-override.com'},
+            extensions={'cookiejar': YoutubeDLCookieJar()},
+        ), pot_request=pot_request, note='Custom requesting webpage')
+
+        assert sent_request.url == 'https://example.com'
+        assert sent_request.headers['User-Agent'] == 'override-user-agent-override'
+        assert sent_request.proxies == {'http': 'http://example-proxy-override.com'}
+        assert sent_request.extensions['cookiejar'] is not cookiejar_request
+        assert 'Custom requesting webpage' in logger.messages['info']
+
+    def test_provider_request_webpage_no_log(self, ie, logger, pot_request):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), note=False)
+
+        assert sent_request.url == 'https://example.com'
+        assert 'info' not in logger.messages
+
+    def test_provider_request_webpage_no_pot_request(self, ie, logger):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={})
+
+        def mock_urlopen(request):
+            return request
+
+        ie._downloader.urlopen = mock_urlopen
+
+        sent_request = provider._request_webpage(Request(
+            'https://example.com',
+        ), pot_request=None)
+
+        assert sent_request.url == 'https://example.com'
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExamplePTP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenProvider):
+            PROVIDER_NAME = 'invalid-suffix'
+
+            def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+                raise PoTokenProviderRejectedRequest('Not implemented')
+
+            def is_available(self) -> bool:
+                return True
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenCacheProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenCacheProvider, IEContentProvider)
+
+    def test_create_provider_missing_get_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+                def is_available(self) -> bool:
+                    return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_store_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_delete_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_missing_is_available_method(self, ie, logger):
+        class MissingMethodsPCP(PoTokenCacheProvider):
+            def get(self, key: str):
+                pass
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCP(ie=ie, logger=logger, settings={})
+
+    def test_barebones_provider(self, ie, logger):
+        class BarebonesProviderPCP(PoTokenCacheProvider):
+
+            def is_available(self) -> bool:
+                return True
+
+            def get(self, key: str):
+                return 'example-cache'
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+        provider = BarebonesProviderPCP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+
+    def test_create_provider_example(self, ie, logger):
+        provider = ExampleCacheProviderPCP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'ExampleCacheProvider'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExampleCacheProviderPCP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenCacheProvider):
+            def get(self, key: str):
+                return 'example-cache'
+
+            def store(self, key: str, value: str, expires_at: int):
+                pass
+
+            def delete(self, key: str):
+                pass
+
+            def is_available(self) -> bool:
+                return True
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenCacheSpecProvider:
+
+    def test_base_type(self):
+        assert issubclass(PoTokenCacheSpecProvider, IEContentProvider)
+
+    def test_create_provider_missing_supports_method(self, ie, logger):
+        class MissingMethodsPCS(PoTokenCacheSpecProvider):
+            pass
+
+        with pytest.raises(TypeError):
+            MissingMethodsPCS(ie=ie, logger=logger, settings={})
+
+    def test_create_provider_barebones(self, ie, pot_request, logger):
+        class BarebonesProviderPCSP(PoTokenCacheSpecProvider):
+            def generate_cache_spec(self, request: PoTokenRequest):
+                return PoTokenCacheSpec(
+                    default_ttl=100,
+                    key_bindings={},
+                )
+
+        provider = BarebonesProviderPCSP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'BarebonesProvider'
+        assert provider.PROVIDER_KEY == 'BarebonesProvider'
+        assert provider.PROVIDER_VERSION == '0.0.0'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  (developer has not provided a bug report location)  .'
+        assert provider.is_available()
+        assert provider.generate_cache_spec(request=pot_request).default_ttl == 100
+        assert provider.generate_cache_spec(request=pot_request).key_bindings == {}
+        assert provider.generate_cache_spec(request=pot_request).write_policy == CacheProviderWritePolicy.WRITE_ALL
+
+    def test_create_provider_example(self, ie, pot_request, logger):
+        provider = ExampleCacheSpecProviderPCSP(ie=ie, logger=logger, settings={})
+        assert provider.PROVIDER_NAME == 'example'
+        assert provider.PROVIDER_KEY == 'ExampleCacheSpecProvider'
+        assert provider.PROVIDER_VERSION == '0.0.1'
+        assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+        assert provider.is_available()
+        assert provider.generate_cache_spec(pot_request)
+        assert provider.generate_cache_spec(pot_request).key_bindings == {'field': 'example-key'}
+        assert provider.generate_cache_spec(pot_request).default_ttl == 60
+        assert provider.generate_cache_spec(pot_request).write_policy == CacheProviderWritePolicy.WRITE_FIRST
+
+    def test_get_config_arg(self, ie, logger):
+        provider = ExampleCacheSpecProviderPCSP(ie=ie, logger=logger, settings={'abc': ['123D'], 'xyz': ['456a', '789B']})
+
+        assert provider._configuration_arg('abc') == ['123d']
+        assert provider._configuration_arg('abc', default=['default']) == ['123d']
+        assert provider._configuration_arg('ABC', default=['default']) == ['default']
+        assert provider._configuration_arg('abc', casesense=True) == ['123D']
+        assert provider._configuration_arg('xyz', casesense=False) == ['456a', '789b']
+
+    def test_require_class_end_with_suffix(self, ie, logger):
+        class InvalidSuffix(PoTokenCacheSpecProvider):
+            def generate_cache_spec(self, request: PoTokenRequest):
+                return None
+
+        provider = InvalidSuffix(ie=ie, logger=logger, settings={})
+
+        with pytest.raises(AssertionError):
+            provider.PROVIDER_KEY  # noqa: B018
+
+
+class TestPoTokenRequest:
+    def test_copy_request(self, pot_request):
+        copied_request = pot_request.copy()
+
+        assert copied_request is not pot_request
+        assert copied_request.context == pot_request.context
+        assert copied_request.innertube_context == pot_request.innertube_context
+        assert copied_request.innertube_context is not pot_request.innertube_context
+        copied_request.innertube_context['client']['clientName'] = 'ANDROID'
+        assert pot_request.innertube_context['client']['clientName'] != 'ANDROID'
+        assert copied_request.innertube_host == pot_request.innertube_host
+        assert copied_request.session_index == pot_request.session_index
+        assert copied_request.player_url == pot_request.player_url
+        assert copied_request.is_authenticated == pot_request.is_authenticated
+        assert copied_request.visitor_data == pot_request.visitor_data
+        assert copied_request.data_sync_id == pot_request.data_sync_id
+        assert copied_request.video_id == pot_request.video_id
+        assert copied_request.request_cookiejar is pot_request.request_cookiejar
+        assert copied_request.request_proxy == pot_request.request_proxy
+        assert copied_request.request_headers == pot_request.request_headers
+        assert copied_request.request_headers is not pot_request.request_headers
+        assert copied_request.request_timeout == pot_request.request_timeout
+        assert copied_request.request_source_address == pot_request.request_source_address
+        assert copied_request.request_verify_tls == pot_request.request_verify_tls
+        assert copied_request.bypass_cache == pot_request.bypass_cache
+
+
+def test_provider_bug_report_message(ie, logger):
+    provider = ExamplePTP(ie=ie, logger=logger, settings={})
+    assert provider.BUG_REPORT_MESSAGE == 'please report this issue to the provider developer at  https://example.com/issues  .'
+
+    message = provider_bug_report_message(provider)
+    assert message == '; please report this issue to the provider developer at  https://example.com/issues  .'
+
+    message_before = provider_bug_report_message(provider, before='custom message!')
+    assert message_before == 'custom message! Please report this issue to the provider developer at  https://example.com/issues  .'
+
+
+def test_register_provider(ie):
+
+    @register_provider
+    class UnavailableProviderPTP(PoTokenProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+            raise PoTokenProviderRejectedRequest('Not implemented')
+
+    assert _pot_providers.value.get('UnavailableProvider') == UnavailableProviderPTP
+    _pot_providers.value.pop('UnavailableProvider')
+
+
+def test_register_pot_preference(ie):
+    before = len(_ptp_preferences.value)
+
+    @register_preference(ExamplePTP)
+    def unavailable_preference(provider: PoTokenProvider, request: PoTokenRequest):
+        return 1
+
+    assert len(_ptp_preferences.value) == before + 1
+
+
+def test_register_cache_provider(ie):
+
+    @cache.register_provider
+    class UnavailableCacheProviderPCP(PoTokenCacheProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def get(self, key: str):
+            return 'example-cache'
+
+        def store(self, key: str, value: str, expires_at: int):
+            pass
+
+        def delete(self, key: str):
+            pass
+
+    assert _pot_cache_providers.value.get('UnavailableCacheProvider') == UnavailableCacheProviderPCP
+    _pot_cache_providers.value.pop('UnavailableCacheProvider')
+
+
+def test_register_cache_provider_spec(ie):
+
+    @cache.register_spec
+    class UnavailableCacheProviderPCSP(PoTokenCacheSpecProvider):
+        def is_available(self) -> bool:
+            return False
+
+        def generate_cache_spec(self, request: PoTokenRequest):
+            return None
+
+    assert _pot_pcs_providers.value.get('UnavailableCacheProvider') == UnavailableCacheProviderPCSP
+    _pot_pcs_providers.value.pop('UnavailableCacheProvider')
+
+
+def test_register_cache_provider_preference(ie):
+    before = len(_pot_cache_provider_preferences.value)
+
+    @cache.register_preference(ExampleCacheProviderPCP)
+    def unavailable_preference(provider: PoTokenCacheProvider, request: PoTokenRequest):
+        return 1
+
+    assert len(_pot_cache_provider_preferences.value) == before + 1
+
+
+def test_logger_log_level(logger):
+    assert logger.LogLevel('INFO') == logger.LogLevel.INFO
+    assert logger.LogLevel('debuG') == logger.LogLevel.DEBUG
+    assert logger.LogLevel(10) == logger.LogLevel.DEBUG
+    assert logger.LogLevel('UNKNOWN') == logger.LogLevel.INFO

yt_dlp/YoutubeDL.py

@@ -640,6 +640,7 @@ class YoutubeDL:
         self._printed_messages = set()
         self._first_webpage_request = True
         self._post_hooks = []
+        self._close_hooks = []
         self._progress_hooks = []
         self._postprocessor_hooks = []
         self._download_retcode = 0
@@ -908,6 +909,10 @@ class YoutubeDL:
         """Add the post hook"""
         self._post_hooks.append(ph)
 
+    def add_close_hook(self, ch):
+        """Add a close hook, called when YoutubeDL.close() is called"""
+        self._close_hooks.append(ch)
+
     def add_progress_hook(self, ph):
         """Add the download progress hook"""
         self._progress_hooks.append(ph)
@@ -1016,6 +1021,9 @@ class YoutubeDL:
             self._request_director.close()
             del self._request_director
 
+        for close_hook in self._close_hooks:
+            close_hook()
+
     def trouble(self, message=None, tb=None, is_error=True):
         """Determine action to take when a download problem appears.
 

yt_dlp/extractor/_extractors.py

@@ -338,7 +338,6 @@ from .canalc2 import Canalc2IE
 from .canalplus import CanalplusIE
 from .canalsurmas import CanalsurmasIE
 from .caracoltv import CaracolTvPlayIE
-from .cartoonnetwork import CartoonNetworkIE
 from .cbc import (
     CBCIE,
     CBCGemIE,
@@ -929,7 +928,10 @@ from .jiocinema import (
 )
 from .jiosaavn import (
     JioSaavnAlbumIE,
+    JioSaavnArtistIE,
     JioSaavnPlaylistIE,
+    JioSaavnShowIE,
+    JioSaavnShowPlaylistIE,
     JioSaavnSongIE,
 )
 from .joj import JojIE
@@ -1964,7 +1966,6 @@ from .spreaker import (
     SpreakerShowIE,
 )
 from .springboardplatform import SpringboardPlatformIE
-from .sprout import SproutIE
 from .sproutvideo import (
     SproutVideoIE,
     VidsIoIE,

yt_dlp/extractor/amcnetworks.py

@@ -1,32 +1,24 @@
-import re
+from .brightcove import BrightcoveNewIE
-
+from .common import InfoExtractor
-from .theplatform import ThePlatformIE
+from ..utils.traversal import traverse_obj
-from ..utils import (
-    int_or_none,
-    parse_age_limit,
-    try_get,
-    update_url_query,
-)
 
 
-class AMCNetworksIE(ThePlatformIE):  # XXX: Do not subclass from concrete IE
+class AMCNetworksIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>amc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?P<id>(?:movies|shows(?:/[^/]+)+)/[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?P<id>(?:movies|shows(?:/[^/?#]+)+)/[^/?#&]+)'
     _TESTS = [{
-        'url': 'https://www.bbcamerica.com/shows/the-graham-norton-show/videos/tina-feys-adorable-airline-themed-family-dinner--51631',
+        'url': 'https://www.amc.com/shows/dark-winds/videos/dark-winds-a-look-at-season-3--1072027',
         'info_dict': {
-            'id': '4Lq1dzOnZGt0',
+            'id': '6369261343112',
             'ext': 'mp4',
-            'title': "The Graham Norton Show - Season 28 - Tina Fey's Adorable Airline-Themed Family Dinner",
+            'title': 'Dark Winds: A Look at Season 3',
-            'description': "It turns out child stewardesses are very generous with the wine! All-new episodes of 'The Graham Norton Show' premiere Fridays at 11/10c on BBC America.",
+            'uploader_id': '6240731308001',
-            'upload_date': '20201120',
+            'duration': 176.427,
-            'timestamp': 1605904350,
+            'thumbnail': r're:https://[^/]+\.boltdns\.net/.+/image\.jpg',
-            'uploader': 'AMCN',
+            'tags': [],
+            'timestamp': 1740414792,
+            'upload_date': '20250224',
         },
-        'params': {
+        'params': {'skip_download': 'm3u8'},
-            # m3u8 download
-            'skip_download': True,
-        },
-        'skip': '404 Not Found',
     }, {
         'url': 'http://www.bbcamerica.com/shows/the-hunt/full-episodes/season-1/episode-01-the-hardest-challenge',
         'only_matching': True,
@@ -52,96 +44,18 @@ class AMCNetworksIE(ThePlatformIE):  # XXX: Do not subclass from concrete IE
         'url': 'https://www.sundancetv.com/shows/riviera/full-episodes/season-1/episode-01-episode-1',
         'only_matching': True,
     }]
-    _REQUESTOR_ID_MAP = {
-        'amc': 'AMC',
-        'bbcamerica': 'BBCA',
-        'ifc': 'IFC',
-        'sundancetv': 'SUNDANCE',
-        'wetv': 'WETV',
-    }
 
     def _real_extract(self, url):
-        site, display_id = self._match_valid_url(url).groups()
+        display_id = self._match_id(url)
-        requestor_id = self._REQUESTOR_ID_MAP[site]
+        webpage = self._download_webpage(url, display_id)
-        page_data = self._download_json(
+        initial_data = self._search_json(
-            f'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/{requestor_id.lower()}/url/{display_id}',
+            r'window\.initialData\s*=\s*JSON\.parse\(String\.raw`', webpage, 'initial data', display_id)
-            display_id)['data']
+        video_id = traverse_obj(initial_data, ('initialData', 'properties', 'videoId', {str}))
-        properties = page_data.get('properties') or {}
+        if not video_id:  # All locked videos are now DRM-protected
-        query = {
+            self.report_drm(display_id)
-            'mbr': 'true',
+        account_id = initial_data['config']['brightcove']['accountId']
-            'manifest': 'm3u',
+        player_id = initial_data['config']['brightcove']['playerId']
-        }
 
-        video_player_count = 0
+        return self.url_result(
-        try:
+            f'https://players.brightcove.net/{account_id}/{player_id}_default/index.html?videoId={video_id}',
-            for v in page_data['children']:
+            BrightcoveNewIE, video_id)
-                if v.get('type') == 'video-player':
-                    release_pid = v['properties']['currentVideo']['meta']['releasePid']
-                    tp_path = 'M_UwQC/' + release_pid
-                    media_url = 'https://link.theplatform.com/s/' + tp_path
-                    video_player_count += 1
-        except KeyError:
-            pass
-        if video_player_count > 1:
-            self.report_warning(
-                f'The JSON data has {video_player_count} video players. Only one will be extracted')
-
-        # Fall back to videoPid if releasePid not found.
-        # TODO: Fall back to videoPid if releasePid manifest uses DRM.
-        if not video_player_count:
-            tp_path = 'M_UwQC/media/' + properties['videoPid']
-            media_url = 'https://link.theplatform.com/s/' + tp_path
-
-        theplatform_metadata = self._download_theplatform_metadata(tp_path, display_id)
-        info = self._parse_theplatform_metadata(theplatform_metadata)
-        video_id = theplatform_metadata['pid']
-        title = theplatform_metadata['title']
-        rating = try_get(
-            theplatform_metadata, lambda x: x['ratings'][0]['rating'])
-        video_category = properties.get('videoCategory')
-        if video_category and video_category.endswith('-Auth'):
-            resource = self._get_mvpd_resource(
-                requestor_id, title, video_id, rating)
-            query['auth'] = self._extract_mvpd_auth(
-                url, video_id, requestor_id, resource)
-        media_url = update_url_query(media_url, query)
-        formats, subtitles = self._extract_theplatform_smil(
-            media_url, video_id)
-
-        thumbnails = []
-        thumbnail_urls = [properties.get('imageDesktop')]
-        if 'thumbnail' in info:
-            thumbnail_urls.append(info.pop('thumbnail'))
-        for thumbnail_url in thumbnail_urls:
-            if not thumbnail_url:
-                continue
-            mobj = re.search(r'(\d+)x(\d+)', thumbnail_url)
-            thumbnails.append({
-                'url': thumbnail_url,
-                'width': int(mobj.group(1)) if mobj else None,
-                'height': int(mobj.group(2)) if mobj else None,
-            })
-
-        info.update({
-            'age_limit': parse_age_limit(rating),
-            'formats': formats,
-            'id': video_id,
-            'subtitles': subtitles,
-            'thumbnails': thumbnails,
-        })
-        ns_keys = theplatform_metadata.get('$xmlns', {}).keys()
-        if ns_keys:
-            ns = next(iter(ns_keys))
-            episode = theplatform_metadata.get(ns + '$episodeTitle') or None
-            episode_number = int_or_none(
-                theplatform_metadata.get(ns + '$episode'))
-            season_number = int_or_none(
-                theplatform_metadata.get(ns + '$season'))
-            series = theplatform_metadata.get(ns + '$show') or None
-            info.update({
-                'episode': episode,
-                'episode_number': episode_number,
-                'season_number': season_number,
-                'series': series,
-            })
-        return info

yt_dlp/extractor/bitchute.py

@@ -1,30 +1,32 @@
 import functools
+import json
 import re
 
 from .common import InfoExtractor
 from ..networking import HEADRequest
+from ..networking.exceptions import HTTPError
 from ..utils import (
     ExtractorError,
     OnDemandPagedList,
     clean_html,
-    extract_attributes,
+    determine_ext,
+    format_field,
     get_element_by_class,
-    get_element_by_id,
-    get_element_html_by_class,
     get_elements_html_by_class,
     int_or_none,
     orderedSet,
     parse_count,
     parse_duration,
-    traverse_obj,
+    parse_iso8601,
-    unified_strdate,
+    url_or_none,
     urlencode_postdata,
     urljoin,
 )
+from ..utils.traversal import traverse_obj
 
 
 class BitChuteIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:www|old)\.)?bitchute\.com/(?:video|embed|torrent/[^/]+)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:(?:www|old)\.)?bitchute\.com/(?:video|embed|torrent/[^/?#]+)/(?P<id>[^/?#&]+)'
     _EMBED_REGEX = [rf'<(?:script|iframe)[^>]+\bsrc=(["\'])(?P<url>{_VALID_URL})']
     _TESTS = [{
         'url': 'https://www.bitchute.com/video/UGlrF9o9b-Q/',
@@ -34,12 +36,17 @@ class BitChuteIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'This is the first video on #BitChute !',
             'description': 'md5:a0337e7b1fe39e32336974af8173a034',
-            'thumbnail': r're:^https?://.*\.jpg$',
+            'thumbnail': r're:https?://.+/.+\.jpg$',
             'uploader': 'BitChute',
             'upload_date': '20170103',
             'uploader_url': 'https://www.bitchute.com/profile/I5NgtHZn9vPj/',
             'channel': 'BitChute',
             'channel_url': 'https://www.bitchute.com/channel/bitchute/',
+            'uploader_id': 'I5NgtHZn9vPj',
+            'channel_id': '1VBwRfyNcKdX',
+            'view_count': int,
+            'duration': 16.0,
+            'timestamp': 1483425443,
         },
     }, {
         # test case: video with different channel and uploader
@@ -49,13 +56,18 @@ class BitChuteIE(InfoExtractor):
             'id': 'Yti_j9A-UZ4',
             'ext': 'mp4',
             'title': 'Israel at War | Full Measure',
-            'description': 'md5:38cf7bc6f42da1a877835539111c69ef',
+            'description': 'md5:e60198b89971966d6030d22b3268f08f',
-            'thumbnail': r're:^https?://.*\.jpg$',
+            'thumbnail': r're:https?://.+/.+\.jpg$',
             'uploader': 'sharylattkisson',
             'upload_date': '20231106',
             'uploader_url': 'https://www.bitchute.com/profile/9K0kUWA9zmd9/',
             'channel': 'Full Measure with Sharyl Attkisson',
             'channel_url': 'https://www.bitchute.com/channel/sharylattkisson/',
+            'uploader_id': '9K0kUWA9zmd9',
+            'channel_id': 'NpdxoCRv3ZLb',
+            'view_count': int,
+            'duration': 554.0,
+            'timestamp': 1699296106,
         },
     }, {
         # video not downloadable in browser, but we can recover it
@@ -66,25 +78,21 @@ class BitChuteIE(InfoExtractor):
             'ext': 'mp4',
             'filesize': 71537926,
             'title': 'STYXHEXENHAMMER666 - Election Fraud, Clinton 2020, EU Armies, and Gun Control',
-            'description': 'md5:228ee93bd840a24938f536aeac9cf749',
+            'description': 'md5:2029c7c212ccd4b040f52bb2d036ef4e',
-            'thumbnail': r're:^https?://.*\.jpg$',
+            'thumbnail': r're:https?://.+/.+\.jpg$',
             'uploader': 'BitChute',
             'upload_date': '20181113',
             'uploader_url': 'https://www.bitchute.com/profile/I5NgtHZn9vPj/',
             'channel': 'BitChute',
             'channel_url': 'https://www.bitchute.com/channel/bitchute/',
+            'uploader_id': 'I5NgtHZn9vPj',
+            'channel_id': '1VBwRfyNcKdX',
+            'view_count': int,
+            'duration': 1701.0,
+            'tags': ['bitchute'],
+            'timestamp': 1542130287,
         },
         'params': {'check_formats': None},
-    }, {
-        # restricted video
-        'url': 'https://www.bitchute.com/video/WEnQU7XGcTdl/',
-        'info_dict': {
-            'id': 'WEnQU7XGcTdl',
-            'ext': 'mp4',
-            'title': 'Impartial Truth - Ein Letzter Appell an die Vernunft',
-        },
-        'params': {'skip_download': True},
-        'skip': 'Georestricted in DE',
     }, {
         'url': 'https://www.bitchute.com/embed/lbb5G1hjPhw/',
         'only_matching': True,
@@ -96,11 +104,8 @@ class BitChuteIE(InfoExtractor):
         'only_matching': True,
     }]
     _GEO_BYPASS = False
-
+    _UPLOADER_URL_TMPL = 'https://www.bitchute.com/profile/%s/'
-    _HEADERS = {
+    _CHANNEL_URL_TMPL = 'https://www.bitchute.com/channel/%s/'
-        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.57 Safari/537.36',
-        'Referer': 'https://www.bitchute.com/',
-    }
 
     def _check_format(self, video_url, video_id):
         urls = orderedSet(
@@ -112,7 +117,7 @@ class BitChuteIE(InfoExtractor):
         for url in urls:
             try:
                 response = self._request_webpage(
-                    HEADRequest(url), video_id=video_id, note=f'Checking {url}', headers=self._HEADERS)
+                    HEADRequest(url), video_id=video_id, note=f'Checking {url}')
             except ExtractorError as e:
                 self.to_screen(f'{video_id}: URL is invalid, skipping: {e.cause}')
                 continue
@@ -121,54 +126,79 @@ class BitChuteIE(InfoExtractor):
                 'filesize': int_or_none(response.headers.get('Content-Length')),
             }
 
-    def _raise_if_restricted(self, webpage):
+    def _call_api(self, endpoint, data, display_id, fatal=True):
-        page_title = clean_html(get_element_by_class('page-title', webpage)) or ''
+        note = endpoint.rpartition('/')[2]
-        if re.fullmatch(r'(?:Channel|Video) Restricted', page_title):
+        try:
-            reason = clean_html(get_element_by_id('page-detail', webpage)) or page_title
+            return self._download_json(
-            self.raise_geo_restricted(reason)
+                f'https://api.bitchute.com/api/beta/{endpoint}', display_id,
-
+                f'Downloading {note} API JSON', f'Unable to download {note} API JSON',
-    @staticmethod
+                data=json.dumps(data).encode(),
-    def _make_url(html):
+                headers={
-        path = extract_attributes(get_element_html_by_class('spa', html) or '').get('href')
+                    'Accept': 'application/json',
-        return urljoin('https://www.bitchute.com', path)
+                    'Content-Type': 'application/json',
+                })
+        except ExtractorError as e:
+            if isinstance(e.cause, HTTPError) and e.cause.status == 403:
+                errors = '. '.join(traverse_obj(e.cause.response.read().decode(), (
+                    {json.loads}, 'errors', lambda _, v: v['context'] == 'reason', 'message', {str})))
+                if errors and 'location' in errors:
+                    # Can always be fatal since the video/media call will reach this code first
+                    self.raise_geo_restricted(errors)
+            if fatal:
+                raise
+            self.report_warning(e.msg)
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(
+        data = {'video_id': video_id}
-            f'https://old.bitchute.com/video/{video_id}', video_id, headers=self._HEADERS)
+        media_url = self._call_api('video/media', data, video_id)['media_url']
-
-        self._raise_if_restricted(webpage)
-        publish_date = clean_html(get_element_by_class('video-publish-date', webpage))
-        entries = self._parse_html5_media_entries(url, webpage, video_id)
 
         formats = []
-        for format_ in traverse_obj(entries, (0, 'formats', ...)):
+        if determine_ext(media_url) == 'm3u8':
+            formats.extend(
+                self._extract_m3u8_formats(media_url, video_id, 'mp4', m3u8_id='hls', live=True))
+        else:
             if self.get_param('check_formats') is not False:
-                format_.update(self._check_format(format_.pop('url'), video_id) or {})
+                if fmt := self._check_format(media_url, video_id):
-                if 'url' not in format_:
+                    formats.append(fmt)
-                    continue
+            else:
-            formats.append(format_)
+                formats.append({'url': media_url})
 
         if not formats:
             self.raise_no_formats(
                 'Video is unavailable. Please make sure this video is playable in the browser '
                 'before reporting this issue.', expected=True, video_id=video_id)
 
-        details = get_element_by_class('details', webpage) or ''
+        video = self._call_api('video', data, video_id, fatal=False)
-        uploader_html = get_element_html_by_class('creator', details) or ''
+        channel = None
-        channel_html = get_element_html_by_class('name', details) or ''
+        if channel_id := traverse_obj(video, ('channel', 'channel_id', {str})):
+            channel = self._call_api('channel', {'channel_id': channel_id}, video_id, fatal=False)
 
         return {
+            **traverse_obj(video, {
+                'title': ('video_name', {str}),
+                'description': ('description', {str}),
+                'thumbnail': ('thumbnail_url', {url_or_none}),
+                'channel': ('channel', 'channel_name', {str}),
+                'channel_id': ('channel', 'channel_id', {str}),
+                'channel_url': ('channel', 'channel_url', {urljoin('https://www.bitchute.com/')}),
+                'uploader_id': ('profile_id', {str}),
+                'uploader_url': ('profile_id', {format_field(template=self._UPLOADER_URL_TMPL)}, filter),
+                'timestamp': ('date_published', {parse_iso8601}),
+                'duration': ('duration', {parse_duration}),
+                'tags': ('hashtags', ..., {str}, filter, all, filter),
+                'view_count': ('view_count', {int_or_none}),
+                'is_live': ('state_id', {lambda x: x == 'live'}),
+            }),
+            **traverse_obj(channel, {
+                'channel': ('channel_name', {str}),
+                'channel_id': ('channel_id', {str}),
+                'channel_url': ('url_slug', {format_field(template=self._CHANNEL_URL_TMPL)}, filter),
+                'uploader': ('profile_name', {str}),
+                'uploader_id': ('profile_id', {str}),
+                'uploader_url': ('profile_id', {format_field(template=self._UPLOADER_URL_TMPL)}, filter),
+            }),
             'id': video_id,
-            'title': self._html_extract_title(webpage) or self._og_search_title(webpage),
-            'description': self._og_search_description(webpage, default=None),
-            'thumbnail': self._og_search_thumbnail(webpage),
-            'uploader': clean_html(uploader_html),
-            'uploader_url': self._make_url(uploader_html),
-            'channel': clean_html(channel_html),
-            'channel_url': self._make_url(channel_html),
-            'upload_date': unified_strdate(self._search_regex(
-                r'at \d+:\d+ UTC on (.+?)\.', publish_date, 'upload date', fatal=False)),
             'formats': formats,
         }
 
@@ -190,7 +220,7 @@ class BitChuteChannelIE(InfoExtractor):
                     'ext': 'mp4',
                     'title': 'This is the first video on #BitChute !',
                     'description': 'md5:a0337e7b1fe39e32336974af8173a034',
-                    'thumbnail': r're:^https?://.*\.jpg$',
+                    'thumbnail': r're:https?://.+/.+\.jpg$',
                     'uploader': 'BitChute',
                     'upload_date': '20170103',
                     'uploader_url': 'https://www.bitchute.com/profile/I5NgtHZn9vPj/',
@@ -198,6 +228,9 @@ class BitChuteChannelIE(InfoExtractor):
                     'channel_url': 'https://www.bitchute.com/channel/bitchute/',
                     'duration': 16,
                     'view_count': int,
+                    'uploader_id': 'I5NgtHZn9vPj',
+                    'channel_id': '1VBwRfyNcKdX',
+                    'timestamp': 1483425443,
                 },
             },
         ],
@@ -213,6 +246,7 @@ class BitChuteChannelIE(InfoExtractor):
             'title': 'Bruce MacDonald and "The Light of Darkness"',
             'description': 'md5:747724ef404eebdfc04277714f81863e',
         },
+        'skip': '404 Not Found',
     }, {
         'url': 'https://old.bitchute.com/playlist/wV9Imujxasw9/',
         'only_matching': True,

yt_dlp/extractor/cartoonnetwork.py

@@ -1,59 +0,0 @@
-from .turner import TurnerBaseIE
-from ..utils import int_or_none
-
-
-class CartoonNetworkIE(TurnerBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cartoonnetwork\.com/video/(?:[^/]+/)+(?P<id>[^/?#]+)-(?:clip|episode)\.html'
-    _TEST = {
-        'url': 'https://www.cartoonnetwork.com/video/ben-10/how-to-draw-upgrade-episode.html',
-        'info_dict': {
-            'id': '6e3375097f63874ebccec7ef677c1c3845fa850e',
-            'ext': 'mp4',
-            'title': 'How to Draw Upgrade',
-            'description': 'md5:2061d83776db7e8be4879684eefe8c0f',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-    }
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-
-        def find_field(global_re, name, content_re=None, value_re='[^"]+', fatal=False):
-            metadata_re = ''
-            if content_re:
-                metadata_re = r'|video_metadata\.content_' + content_re
-            return self._search_regex(
-                rf'(?:_cnglobal\.currentVideo\.{global_re}{metadata_re})\s*=\s*"({value_re})";',
-                webpage, name, fatal=fatal)
-
-        media_id = find_field('mediaId', 'media id', 'id', '[0-9a-f]{40}', True)
-        title = find_field('episodeTitle', 'title', '(?:episodeName|name)', fatal=True)
-
-        info = self._extract_ngtv_info(
-            media_id, {'networkId': 'cartoonnetwork'}, {
-                'url': url,
-                'site_name': 'CartoonNetwork',
-                'auth_required': find_field('authType', 'auth type') != 'unauth',
-            })
-
-        series = find_field(
-            'propertyName', 'series', 'showName') or self._html_search_meta('partOfSeries', webpage)
-        info.update({
-            'id': media_id,
-            'display_id': display_id,
-            'title': title,
-            'description': self._html_search_meta('description', webpage),
-            'series': series,
-            'episode': title,
-        })
-
-        for field in ('season', 'episode'):
-            field_name = field + 'Number'
-            info[field + '_number'] = int_or_none(find_field(
-                field_name, field + ' number', value_re=r'\d+') or self._html_search_meta(field_name, webpage))
-
-        return info

yt_dlp/extractor/jiosaavn.py

@@ -1,23 +1,33 @@
 import functools
+import itertools
 import math
 import re
 
 from .common import InfoExtractor
 from ..utils import (
     InAdvancePagedList,
+    ISO639Utils,
+    OnDemandPagedList,
     clean_html,
     int_or_none,
+    js_to_json,
     make_archive_id,
+    orderedSet,
     smuggle_url,
+    unified_strdate,
+    unified_timestamp,
     unsmuggle_url,
     url_basename,
     url_or_none,
     urlencode_postdata,
+    urljoin,
+    variadic,
 )
 from ..utils.traversal import traverse_obj
 
 
 class JioSaavnBaseIE(InfoExtractor):
+    _URL_BASE_RE = r'https?://(?:www\.)?(?:jio)?saavn\.com'
     _API_URL = 'https://www.jiosaavn.com/api.php'
     _VALID_BITRATES = {'16', '32', '64', '128', '320'}
 
@@ -30,16 +40,20 @@ class JioSaavnBaseIE(InfoExtractor):
                 f'Valid bitrates are: {", ".join(sorted(self._VALID_BITRATES, key=int))}')
         return requested_bitrates
 
-    def _extract_formats(self, song_data):
+    def _extract_formats(self, item_data):
+        # Show/episode JSON data has a slightly different structure than song JSON data
+        if media_url := traverse_obj(item_data, ('more_info', 'encrypted_media_url', {str})):
+            item_data.setdefault('encrypted_media_url', media_url)
+
         for bitrate in self.requested_bitrates:
             media_data = self._download_json(
-                self._API_URL, song_data['id'],
+                self._API_URL, item_data['id'],
                 f'Downloading format info for {bitrate}',
                 fatal=False, data=urlencode_postdata({
                     '__call': 'song.generateAuthToken',
                     '_format': 'json',
                     'bitrate': bitrate,
-                    'url': song_data['encrypted_media_url'],
+                    'url': item_data['encrypted_media_url'],
                 }))
             if not traverse_obj(media_data, ('auth_url', {url_or_none})):
                 self.report_warning(f'Unable to extract format info for {bitrate}')
@@ -53,24 +67,6 @@ class JioSaavnBaseIE(InfoExtractor):
                 'vcodec': 'none',
             }
 
-    def _extract_song(self, song_data, url=None):
-        info = traverse_obj(song_data, {
-            'id': ('id', {str}),
-            'title': ('song', {clean_html}),
-            'album': ('album', {clean_html}),
-            'thumbnail': ('image', {url_or_none}, {lambda x: re.sub(r'-\d+x\d+\.', '-500x500.', x)}),
-            'duration': ('duration', {int_or_none}),
-            'view_count': ('play_count', {int_or_none}),
-            'release_year': ('year', {int_or_none}),
-            'artists': ('primary_artists', {lambda x: x.split(', ') if x else None}),
-            'webpage_url': ('perma_url', {url_or_none}),
-        })
-        if webpage_url := info.get('webpage_url') or url:
-            info['display_id'] = url_basename(webpage_url)
-            info['_old_archive_ids'] = [make_archive_id(JioSaavnSongIE, info['display_id'])]
-
-        return info
-
     def _call_api(self, type_, token, note='API', params={}):
         return self._download_json(
             self._API_URL, token, f'Downloading {note} JSON', f'Unable to download {note} JSON',
@@ -84,19 +80,89 @@ class JioSaavnBaseIE(InfoExtractor):
                 **params,
             })
 
-    def _yield_songs(self, playlist_data):
+    @staticmethod
-        for song_data in traverse_obj(playlist_data, ('songs', lambda _, v: v['id'] and v['perma_url'])):
+    def _extract_song(song_data, url=None):
-            song_info = self._extract_song(song_data)
+        info = traverse_obj(song_data, {
-            url = smuggle_url(song_info['webpage_url'], {
+            'id': ('id', {str}),
-                'id': song_data['id'],
+            'title': (('song', 'title'), {clean_html}, any),
-                'encrypted_media_url': song_data['encrypted_media_url'],
+            'album': ((None, 'more_info'), 'album', {clean_html}, any),
-            })
+            'duration': ((None, 'more_info'), 'duration', {int_or_none}, any),
-            yield self.url_result(url, JioSaavnSongIE, url_transparent=True, **song_info)
+            'channel': ((None, 'more_info'), 'label', {str}, any),
+            'channel_id': ((None, 'more_info'), 'label_id', {str}, any),
+            'channel_url': ((None, 'more_info'), 'label_url', {urljoin('https://www.jiosaavn.com/')}, any),
+            'release_date': ((None, 'more_info'), 'release_date', {unified_strdate}, any),
+            'release_year': ('year', {int_or_none}),
+            'thumbnail': ('image', {url_or_none}, {lambda x: re.sub(r'-\d+x\d+\.', '-500x500.', x)}),
+            'view_count': ('play_count', {int_or_none}),
+            'language': ('language', {lambda x: ISO639Utils.short2long(x.casefold()) or 'und'}),
+            'webpage_url': ('perma_url', {url_or_none}),
+            'artists': ('more_info', 'artistMap', 'primary_artists', ..., 'name', {str}, filter, all),
+        })
+        if webpage_url := info.get('webpage_url') or url:
+            info['display_id'] = url_basename(webpage_url)
+            info['_old_archive_ids'] = [make_archive_id(JioSaavnSongIE, info['display_id'])]
+
+        if primary_artists := traverse_obj(song_data, ('primary_artists', {lambda x: x.split(', ') if x else None})):
+            info['artists'].extend(primary_artists)
+        if featured_artists := traverse_obj(song_data, ('featured_artists', {str}, filter)):
+            info['artists'].extend(featured_artists.split(', '))
+        info['artists'] = orderedSet(info['artists']) or None
+
+        return info
+
+    @staticmethod
+    def _extract_episode(episode_data, url=None):
+        info = JioSaavnBaseIE._extract_song(episode_data, url)
+        info.pop('_old_archive_ids', None)
+        info.update(traverse_obj(episode_data, {
+            'description': ('more_info', 'description', {str}),
+            'timestamp': ('more_info', 'release_time', {unified_timestamp}),
+            'series': ('more_info', 'show_title', {str}),
+            'series_id': ('more_info', 'show_id', {str}),
+            'season': ('more_info', 'season_title', {str}),
+            'season_number': ('more_info', 'season_no', {int_or_none}),
+            'season_id': ('more_info', 'season_id', {str}),
+            'episode_number': ('more_info', 'episode_number', {int_or_none}),
+            'cast': ('starring', {lambda x: x.split(', ') if x else None}),
+        }))
+        return info
+
+    def _extract_jiosaavn_result(self, url, endpoint, response_key, parse_func):
+        url, smuggled_data = unsmuggle_url(url)
+        data = traverse_obj(smuggled_data, ({
+            'id': ('id', {str}),
+            'encrypted_media_url': ('encrypted_media_url', {str}),
+        }))
+
+        if 'id' in data and 'encrypted_media_url' in data:
+            result = {'id': data['id']}
+        else:
+            # only extract metadata if this is not a url_transparent result
+            data = self._call_api(endpoint, self._match_id(url))[response_key][0]
+            result = parse_func(data, url)
+
+        result['formats'] = list(self._extract_formats(data))
+        return result
+
+    def _yield_items(self, playlist_data, keys=None, parse_func=None):
+        """Subclasses using this method must set _ENTRY_IE"""
+        if parse_func is None:
+            parse_func = self._extract_song
+
+        for item_data in traverse_obj(playlist_data, (
+            *variadic(keys, (str, bytes, dict, set)), lambda _, v: v['id'] and v['perma_url'],
+        )):
+            info = parse_func(item_data)
+            url = smuggle_url(info['webpage_url'], traverse_obj(item_data, {
+                'id': ('id', {str}),
+                'encrypted_media_url': ((None, 'more_info'), 'encrypted_media_url', {str}, any),
+            }))
+            yield self.url_result(url, self._ENTRY_IE, url_transparent=True, **info)
 
 
 class JioSaavnSongIE(JioSaavnBaseIE):
     IE_NAME = 'jiosaavn:song'
-    _VALID_URL = r'https?://(?:www\.)?(?:jiosaavn\.com/song/[^/?#]+/|saavn\.com/s/song/(?:[^/?#]+/){3})(?P<id>[^/?#]+)'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'(?:/song/[^/?#]+/|/s/song/(?:[^/?#]+/){3})(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'https://www.jiosaavn.com/song/leja-re/OQsEfQFVUXk',
         'md5': '3b84396d15ed9e083c3106f1fa589c04',
@@ -106,12 +172,38 @@ class JioSaavnSongIE(JioSaavnBaseIE):
             'ext': 'm4a',
             'title': 'Leja Re',
             'album': 'Leja Re',
-            'thumbnail': r're:https?://c.saavncdn.com/258/Leja-Re-Hindi-2018-20181124024539-500x500.jpg',
+            'thumbnail': r're:https?://.+/.+\.jpg',
             'duration': 205,
             'view_count': int,
             'release_year': 2018,
             'artists': ['Sandesh Shandilya', 'Dhvani Bhanushali', 'Tanishk Bagchi'],
             '_old_archive_ids': ['jiosaavnsong OQsEfQFVUXk'],
+            'channel': 'T-Series',
+            'language': 'hin',
+            'channel_id': '34297',
+            'channel_url': 'https://www.jiosaavn.com/label/t-series-albums/6DLuXO3VoTo_',
+            'release_date': '20181124',
+        },
+    }, {
+        'url': 'https://www.jiosaavn.com/song/chuttamalle/P1FfWjZkQ0Q',
+        'md5': '96296c58d6ce488a417ef0728fd2d680',
+        'info_dict': {
+            'id': 'O94kBTtw',
+            'display_id': 'P1FfWjZkQ0Q',
+            'ext': 'm4a',
+            'title': 'Chuttamalle',
+            'album': 'Devara Part 1 - Telugu',
+            'thumbnail': r're:https?://.+/.+\.jpg',
+            'duration': 222,
+            'view_count': int,
+            'release_year': 2024,
+            'artists': 'count:3',
+            '_old_archive_ids': ['jiosaavnsong P1FfWjZkQ0Q'],
+            'channel': 'T-Series',
+            'language': 'tel',
+            'channel_id': '34297',
+            'channel_url': 'https://www.jiosaavn.com/label/t-series-albums/6DLuXO3VoTo_',
+            'release_date': '20240926',
         },
     }, {
         'url': 'https://www.saavn.com/s/song/hindi/Saathiya/O-Humdum-Suniyo-Re/KAMiazoCblU',
@@ -119,26 +211,51 @@ class JioSaavnSongIE(JioSaavnBaseIE):
     }]
 
     def _real_extract(self, url):
-        url, smuggled_data = unsmuggle_url(url)
+        return self._extract_jiosaavn_result(url, 'song', 'songs', self._extract_song)
-        song_data = traverse_obj(smuggled_data, ({
-            'id': ('id', {str}),
-            'encrypted_media_url': ('encrypted_media_url', {str}),
-        }))
 
-        if 'id' in song_data and 'encrypted_media_url' in song_data:
-            result = {'id': song_data['id']}
-        else:
-            # only extract metadata if this is not a url_transparent result
-            song_data = self._call_api('song', self._match_id(url))['songs'][0]
-            result = self._extract_song(song_data, url)
 
-        result['formats'] = list(self._extract_formats(song_data))
+class JioSaavnShowIE(JioSaavnBaseIE):
-        return result
+    IE_NAME = 'jiosaavn:show'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'/shows/[^/?#]+/(?P<id>[^/?#]{11,})/?(?:$|[?#])'
+    _TESTS = [{
+        'url': 'https://www.jiosaavn.com/shows/non-food-ways-to-boost-your-energy/XFMcKICOCgc_',
+        'md5': '0733cd254cfe74ef88bea1eaedcf1f4f',
+        'info_dict': {
+            'id': 'qqzh3RKZ',
+            'display_id': 'XFMcKICOCgc_',
+            'ext': 'mp3',
+            'title': 'Non-Food Ways To Boost Your Energy',
+            'description': 'md5:26e7129644b5c6aada32b8851c3997c8',
+            'episode': 'Episode 1',
+            'timestamp': 1640563200,
+            'series': 'Holistic Lifestyle With Neha Ranglani',
+            'series_id': '52397',
+            'season': 'Holistic Lifestyle With Neha Ranglani',
+            'season_number': 1,
+            'season_id': '61273',
+            'thumbnail': r're:https?://.+/.+\.jpg',
+            'duration': 311,
+            'view_count': int,
+            'release_year': 2021,
+            'language': 'eng',
+            'channel': 'Saavn OG',
+            'channel_id': '1953876',
+            'episode_number': 1,
+            'upload_date': '20211227',
+            'release_date': '20211227',
+        },
+    }, {
+        'url': 'https://www.jiosaavn.com/shows/himesh-reshammiya/Kr8fmfSN4vo_',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        return self._extract_jiosaavn_result(url, 'episode', 'episodes', self._extract_episode)
 
 
 class JioSaavnAlbumIE(JioSaavnBaseIE):
     IE_NAME = 'jiosaavn:album'
-    _VALID_URL = r'https?://(?:www\.)?(?:jio)?saavn\.com/album/[^/?#]+/(?P<id>[^/?#]+)'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'/album/[^/?#]+/(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'https://www.jiosaavn.com/album/96/buIOjYZDrNA_',
         'info_dict': {
@@ -147,18 +264,19 @@ class JioSaavnAlbumIE(JioSaavnBaseIE):
         },
         'playlist_count': 10,
     }]
+    _ENTRY_IE = JioSaavnSongIE
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
         album_data = self._call_api('album', display_id)
 
         return self.playlist_result(
-            self._yield_songs(album_data), display_id, traverse_obj(album_data, ('title', {str})))
+            self._yield_items(album_data, 'songs'), display_id, traverse_obj(album_data, ('title', {str})))
 
 
 class JioSaavnPlaylistIE(JioSaavnBaseIE):
     IE_NAME = 'jiosaavn:playlist'
-    _VALID_URL = r'https?://(?:www\.)?(?:jio)?saavn\.com/(?:s/playlist/(?:[^/?#]+/){2}|featured/[^/?#]+/)(?P<id>[^/?#]+)'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'/(?:s/playlist/(?:[^/?#]+/){2}|featured/[^/?#]+/)(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'https://www.jiosaavn.com/s/playlist/2279fbe391defa793ad7076929a2f5c9/mood-english/LlJ8ZWT1ibN5084vKHRj2Q__',
         'info_dict': {
@@ -172,15 +290,16 @@ class JioSaavnPlaylistIE(JioSaavnBaseIE):
             'id': 'DVR,pFUOwyXqIp77B1JF,A__',
             'title': 'Mood Hindi',
         },
-        'playlist_mincount': 801,
+        'playlist_mincount': 750,
     }, {
         'url': 'https://www.jiosaavn.com/featured/taaza-tunes/Me5RridRfDk_',
         'info_dict': {
             'id': 'Me5RridRfDk_',
             'title': 'Taaza Tunes',
         },
-        'playlist_mincount': 301,
+        'playlist_mincount': 50,
     }]
+    _ENTRY_IE = JioSaavnSongIE
     _PAGE_SIZE = 50
 
     def _fetch_page(self, token, page):
@@ -189,7 +308,7 @@ class JioSaavnPlaylistIE(JioSaavnBaseIE):
 
     def _entries(self, token, first_page_data, page):
         page_data = first_page_data if not page else self._fetch_page(token, page + 1)
-        yield from self._yield_songs(page_data)
+        yield from self._yield_items(page_data, 'songs')
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
@@ -199,3 +318,95 @@ class JioSaavnPlaylistIE(JioSaavnBaseIE):
         return self.playlist_result(InAdvancePagedList(
             functools.partial(self._entries, display_id, playlist_data),
             total_pages, self._PAGE_SIZE), display_id, traverse_obj(playlist_data, ('listname', {str})))
+
+
+class JioSaavnShowPlaylistIE(JioSaavnBaseIE):
+    IE_NAME = 'jiosaavn:show:playlist'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'/shows/(?P<show>[^#/?]+)/(?P<season>\d+)/[^/?#]+'
+    _TESTS = [{
+        'url': 'https://www.jiosaavn.com/shows/talking-music/1/PjReFP-Sguk_',
+        'info_dict': {
+            'id': 'talking-music-1',
+            'title': 'Talking Music',
+        },
+        'playlist_mincount': 11,
+    }]
+    _ENTRY_IE = JioSaavnShowIE
+    _PAGE_SIZE = 10
+
+    def _fetch_page(self, show_id, season_id, page):
+        return self._call_api('show', show_id, f'show page {page}', {
+            'p': page,
+            '__call': 'show.getAllEpisodes',
+            'show_id': show_id,
+            'season_number': season_id,
+            'api_version': '4',
+            'sort_order': 'desc',
+        })
+
+    def _entries(self, show_id, season_id, page):
+        page_data = self._fetch_page(show_id, season_id, page + 1)
+        yield from self._yield_items(page_data, keys=None, parse_func=self._extract_episode)
+
+    def _real_extract(self, url):
+        show_slug, season_id = self._match_valid_url(url).group('show', 'season')
+        playlist_id = f'{show_slug}-{season_id}'
+        webpage = self._download_webpage(url, playlist_id)
+
+        show_info = self._search_json(
+            r'window\.__INITIAL_DATA__\s*=', webpage, 'initial data',
+            playlist_id, transform_source=js_to_json)['showView']
+        show_id = show_info['current_id']
+
+        entries = OnDemandPagedList(functools.partial(self._entries, show_id, season_id), self._PAGE_SIZE)
+        return self.playlist_result(
+            entries, playlist_id, traverse_obj(show_info, ('show', 'title', 'text', {str})))
+
+
+class JioSaavnArtistIE(JioSaavnBaseIE):
+    IE_NAME = 'jiosaavn:artist'
+    _VALID_URL = JioSaavnBaseIE._URL_BASE_RE + r'/artist/[^/?#]+/(?P<id>[^/?#]+)'
+    _TESTS = [{
+        'url': 'https://www.jiosaavn.com/artist/krsna-songs/rYLBEve2z3U_',
+        'info_dict': {
+            'id': 'rYLBEve2z3U_',
+            'title': 'KR$NA',
+        },
+        'playlist_mincount': 38,
+    }, {
+        'url': 'https://www.jiosaavn.com/artist/sanam-puri-songs/SkNEv3qRhDE_',
+        'info_dict': {
+            'id': 'SkNEv3qRhDE_',
+            'title': 'Sanam Puri',
+        },
+        'playlist_mincount': 51,
+    }]
+    _ENTRY_IE = JioSaavnSongIE
+    _PAGE_SIZE = 50
+
+    def _fetch_page(self, artist_id, page):
+        return self._call_api('artist', artist_id, f'artist page {page + 1}', {
+            'p': page,
+            'n_song': self._PAGE_SIZE,
+            'n_album': self._PAGE_SIZE,
+            'sub_type': '',
+            'includeMetaTags': '',
+            'api_version': '4',
+            'category': 'alphabetical',
+            'sort_order': 'asc',
+        })
+
+    def _entries(self, artist_id, first_page):
+        for page in itertools.count():
+            playlist_data = first_page if not page else self._fetch_page(artist_id, page)
+            if not traverse_obj(playlist_data, ('topSongs', ..., {dict})):
+                break
+            yield from self._yield_items(playlist_data, 'topSongs')
+
+    def _real_extract(self, url):
+        artist_id = self._match_id(url)
+        first_page = self._fetch_page(artist_id, 0)
+
+        return self.playlist_result(
+            self._entries(artist_id, first_page), artist_id,
+            traverse_obj(first_page, ('name', {str})))

yt_dlp/extractor/niconico.py

@@ -16,6 +16,7 @@ from ..utils import (
     determine_ext,
     float_or_none,
     int_or_none,
+    parse_bitrate,
     parse_duration,
     parse_iso8601,
     parse_qs,
@@ -23,7 +24,6 @@ from ..utils import (
     qualities,
     remove_start,
     str_or_none,
-    try_get,
     unescapeHTML,
     unified_timestamp,
     update_url_query,
@@ -785,8 +785,6 @@ class NiconicoLiveIE(NiconicoBaseIE):
         'only_matching': True,
     }]
 
-    _KNOWN_LATENCY = ('high', 'low')
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
         webpage, urlh = self._download_webpage_handle(f'https://live.nicovideo.jp/watch/{video_id}', video_id)
@@ -802,22 +800,19 @@ class NiconicoLiveIE(NiconicoBaseIE):
         })
 
         hostname = remove_start(urllib.parse.urlparse(urlh.url).hostname, 'sp.')
-        latency = try_get(self._configuration_arg('latency'), lambda x: x[0])
-        if latency not in self._KNOWN_LATENCY:
-            latency = 'high'
 
         ws = self._request_webpage(
             Request(ws_url, headers={'Origin': f'https://{hostname}'}),
             video_id=video_id, note='Connecting to WebSocket server')
 
-        self.write_debug('[debug] Sending HLS server request')
+        self.write_debug('Sending HLS server request')
         ws.send(json.dumps({
             'type': 'startWatching',
             'data': {
                 'stream': {
                     'quality': 'abr',
-                    'protocol': 'hls+fmp4',
+                    'protocol': 'hls',
-                    'latency': latency,
+                    'latency': 'high',
                     'accessRightMethod': 'single_cookie',
                     'chasePlay': False,
                 },
@@ -881,18 +876,29 @@ class NiconicoLiveIE(NiconicoBaseIE):
         for cookie in cookies:
             self._set_cookie(
                 cookie['domain'], cookie['name'], cookie['value'],
-                expire_time=unified_timestamp(cookie['expires']), path=cookie['path'], secure=cookie['secure'])
+                expire_time=unified_timestamp(cookie.get('expires')), path=cookie['path'], secure=cookie['secure'])
+
+        fmt_common = {
+            'live_latency': 'high',
+            'origin': hostname,
+            'protocol': 'niconico_live',
+            'video_id': video_id,
+            'ws': ws,
+        }
+        q_iter = (q for q in qualities[1:] if not q.startswith('audio_'))  # ignore initial 'abr'
+        a_map = {96: 'audio_low', 192: 'audio_high'}
 
         formats = self._extract_m3u8_formats(m3u8_url, video_id, ext='mp4', live=True)
-        for fmt, q in zip(formats, reversed(qualities[1:])):
+        for fmt in formats:
-            fmt.update({
+            if fmt.get('acodec') == 'none':
-                'format_id': q,
+                fmt['format_id'] = next(q_iter, fmt['format_id'])
-                'protocol': 'niconico_live',
+            elif fmt.get('vcodec') == 'none':
-                'ws': ws,
+                abr = parse_bitrate(fmt['url'].lower())
-                'video_id': video_id,
+                fmt.update({
-                'live_latency': latency,
+                    'abr': abr,
-                'origin': hostname,
+                    'format_id': a_map.get(abr, fmt['format_id']),
-            })
+                })
+            fmt.update(fmt_common)
 
         return {
             'id': video_id,

yt_dlp/extractor/nytimes.py

@@ -181,6 +181,7 @@ class NYTimesArticleIE(NYTimesBaseIE):
             'thumbnail': r're:https?://\w+\.nyt.com/images/.*\.jpg',
             'duration': 119.0,
         },
+        'skip': 'HTTP Error 500: Internal Server Error',
     }, {
         # article with audio and no video
         'url': 'https://www.nytimes.com/2023/09/29/health/mosquitoes-genetic-engineering.html',
@@ -190,13 +191,14 @@ class NYTimesArticleIE(NYTimesBaseIE):
             'ext': 'mp3',
             'title': 'The Gamble: Can Genetically Modified Mosquitoes End Disease?',
             'description': 'md5:9ff8b47acbaf7f3ca8c732f5c815be2e',
-            'timestamp': 1695960700,
+            'timestamp': 1696008129,
             'upload_date': '20230929',
-            'creator': 'Stephanie Nolen, Natalija Gormalova',
+            'creators': ['Stephanie Nolen', 'Natalija Gormalova'],
             'thumbnail': r're:https?://\w+\.nyt.com/images/.*\.jpg',
             'duration': 1322,
         },
     }, {
+        # lede_media_block already has sourceId
         'url': 'https://www.nytimes.com/2023/11/29/business/dealbook/kamala-harris-biden-voters.html',
         'md5': '3eb5ddb1d6f86254fe4f233826778737',
         'info_dict': {
@@ -207,7 +209,7 @@ class NYTimesArticleIE(NYTimesBaseIE):
             'timestamp': 1701290997,
             'upload_date': '20231129',
             'uploader': 'By The New York Times',
-            'creator': 'Katie Rogers',
+            'creators': ['Katie Rogers'],
             'thumbnail': r're:https?://\w+\.nyt.com/images/.*\.jpg',
             'duration': 97.631,
         },
@@ -222,10 +224,22 @@ class NYTimesArticleIE(NYTimesBaseIE):
             'title': 'Drunk and Asleep on the Job: Air Traffic Controllers Pushed to the Brink',
             'description': 'md5:549e5a5e935bf7d048be53ba3d2c863d',
             'upload_date': '20231202',
-            'creator': 'Emily Steel, Sydney Ember',
+            'creators': ['Emily Steel', 'Sydney Ember'],
             'timestamp': 1701511264,
         },
         'playlist_count': 3,
+    }, {
+        # lede_media_block does not have sourceId
+        'url': 'https://www.nytimes.com/2025/04/30/well/move/hip-mobility-routine.html',
+        'info_dict': {
+            'id': 'hip-mobility-routine',
+            'title': 'Tight Hips? These Moves Can Help.',
+            'description': 'Sitting all day is hard on your hips. Try this simple routine for better mobility.',
+            'creators': ['Alyssa Ages', 'Theodore Tae'],
+            'timestamp': 1746003629,
+            'upload_date': '20250430',
+        },
+        'playlist_count': 7,
     }, {
         'url': 'https://www.nytimes.com/2023/12/02/business/media/netflix-squid-game-challenge.html',
         'only_matching': True,
@@ -256,14 +270,18 @@ class NYTimesArticleIE(NYTimesBaseIE):
 
     def _real_extract(self, url):
         page_id = self._match_id(url)
-        webpage = self._download_webpage(url, page_id)
+        webpage = self._download_webpage(url, page_id, impersonate=True)
         art_json = self._search_json(
             r'window\.__preloadedData\s*=', webpage, 'media details', page_id,
             transform_source=lambda x: x.replace('undefined', 'null'))['initialData']['data']['article']
+        content = art_json['sprinkledBody']['content']
 
-        blocks = traverse_obj(art_json, (
+        blocks = []
-            'sprinkledBody', 'content', ..., ('ledeMedia', None),
+        block_filter = lambda k, v: k == 'media' and v['__typename'] in ('Video', 'Audio')
-            lambda _, v: v['__typename'] in ('Video', 'Audio')))
+        if lede_media_block := traverse_obj(content, (..., 'ledeMedia', block_filter, any)):
+            lede_media_block.setdefault('sourceId', art_json.get('sourceId'))
+            blocks.append(lede_media_block)
+        blocks.extend(traverse_obj(content, (..., block_filter)))
         if not blocks:
             raise ExtractorError('Unable to extract any media blocks from webpage')
 
@@ -273,8 +291,7 @@ class NYTimesArticleIE(NYTimesBaseIE):
                 'sprinkledBody', 'content', ..., 'summary', 'content', ..., 'text', {str}),
                 get_all=False) or self._html_search_meta(['og:description', 'twitter:description'], webpage),
             'timestamp': traverse_obj(art_json, ('firstPublished', {parse_iso8601})),
-            'creator': ', '.join(
+            'creators': traverse_obj(art_json, ('bylines', ..., 'creators', ..., 'displayName', {str})),
-                traverse_obj(art_json, ('bylines', ..., 'creators', ..., 'displayName'))),  # TODO: change to 'creators' (list)
             'thumbnails': self._extract_thumbnails(traverse_obj(
                 art_json, ('promotionalMedia', 'assetCrops', ..., 'renditions', ...))),
         }

yt_dlp/extractor/playsuisse.py

@@ -7,11 +7,13 @@ from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
     int_or_none,
+    join_nonempty,
     parse_qs,
     traverse_obj,
     update_url_query,
     urlencode_postdata,
 )
+from ..utils.traversal import unpack
 
 
 class PlaySuisseIE(InfoExtractor):
@@ -26,12 +28,12 @@ class PlaySuisseIE(InfoExtractor):
         {
             # episode in a series
             'url': 'https://www.playsuisse.ch/watch/763182?episodeId=763211',
-            'md5': '82df2a470b2dfa60c2d33772a8a60cf8',
+            'md5': 'e20d1ede6872a03b41905ca1060a1ef2',
             'info_dict': {
                 'id': '763211',
                 'ext': 'mp4',
                 'title': 'Knochen',
-                'description': 'md5:8ea7a8076ba000cd9e8bc132fd0afdd8',
+                'description': 'md5:3bdd80e2ce20227c47aab1df2a79a519',
                 'duration': 3344,
                 'series': 'Wilder',
                 'season': 'Season 1',
@@ -42,24 +44,33 @@ class PlaySuisseIE(InfoExtractor):
             },
         }, {
             # film
-            'url': 'https://www.playsuisse.ch/watch/808675',
+            'url': 'https://www.playsuisse.ch/detail/2573198',
-            'md5': '818b94c1d2d7c4beef953f12cb8f3e75',
+            'md5': '1f115bb0a5191477b1a5771643a4283d',
             'info_dict': {
-                'id': '808675',
+                'id': '2573198',
                 'ext': 'mp4',
-                'title': 'Der Läufer',
+                'title': 'Azor',
-                'description': 'md5:9f61265c7e6dcc3e046137a792b275fd',
+                'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
-                'duration': 5280,
+                'genres': ['Fiction'],
+                'creators': ['Andreas Fontana'],
+                'cast': ['Fabrizio Rongione', 'Stéphanie Cléau', 'Gilles Privat', 'Alexandre Trocki'],
+                'location': 'France; Argentine',
+                'release_year': 2021,
+                'duration': 5981,
                 'thumbnail': 're:https://playsuisse-img.akamaized.net/',
             },
         }, {
             # series (treated as a playlist)
             'url': 'https://www.playsuisse.ch/detail/1115687',
             'info_dict': {
-                'description': 'md5:e4a2ae29a8895823045b5c3145a02aa3',
                 'id': '1115687',
                 'series': 'They all came out to Montreux',
                 'title': 'They all came out to Montreux',
+                'description': 'md5:0fefd8c5b4468a0bb35e916887681520',
+                'genres': ['Documentary'],
+                'creators': ['Oliver Murray'],
+                'location': 'Switzerland',
+                'release_year': 2021,
             },
             'playlist': [{
                 'info_dict': {
@@ -120,6 +131,12 @@ class PlaySuisseIE(InfoExtractor):
             id
             name
             description
+            descriptionLong
+            year
+            contentTypes
+            directors
+            mainCast
+            productionCountries
             duration
             episodeNumber
             seasonNumber
@@ -215,9 +232,7 @@ class PlaySuisseIE(InfoExtractor):
         if not self._ID_TOKEN:
             raise ExtractorError('Login failed')
 
-    def _get_media_data(self, media_id):
+    def _get_media_data(self, media_id, locale=None):
-        # NOTE In the web app, the "locale" header is used to switch between languages,
-        # However this doesn't seem to take effect when passing the header here.
         response = self._download_json(
             'https://www.playsuisse.ch/api/graphql',
             media_id, data=json.dumps({
@@ -225,7 +240,7 @@ class PlaySuisseIE(InfoExtractor):
                 'query': self._GRAPHQL_QUERY,
                 'variables': {'assetId': media_id},
             }).encode(),
-            headers={'Content-Type': 'application/json', 'locale': 'de'})
+            headers={'Content-Type': 'application/json', 'locale': locale or 'de'})
 
         return response['data']['assetV2']
 
@@ -234,7 +249,7 @@ class PlaySuisseIE(InfoExtractor):
             self.raise_login_required(method='password')
 
         media_id = self._match_id(url)
-        media_data = self._get_media_data(media_id)
+        media_data = self._get_media_data(media_id, traverse_obj(parse_qs(url), ('locale', 0)))
         info = self._extract_single(media_data)
         if media_data.get('episodes'):
             info.update({
@@ -257,15 +272,22 @@ class PlaySuisseIE(InfoExtractor):
             self._merge_subtitles(subs, target=subtitles)
 
         return {
-            'id': media_data['id'],
-            'title': media_data.get('name'),
-            'description': media_data.get('description'),
             'thumbnails': thumbnails,
-            'duration': int_or_none(media_data.get('duration')),
             'formats': formats,
             'subtitles': subtitles,
-            'series': media_data.get('seriesName'),
+            **traverse_obj(media_data, {
-            'season_number': int_or_none(media_data.get('seasonNumber')),
+                'id': ('id', {str}),
-            'episode': media_data.get('name') if media_data.get('episodeNumber') else None,
+                'title': ('name', {str}),
-            'episode_number': int_or_none(media_data.get('episodeNumber')),
+                'description': (('descriptionLong', 'description'), {str}, any),
+                'genres': ('contentTypes', ..., {str}),
+                'creators': ('directors', ..., {str}),
+                'cast': ('mainCast', ..., {str}),
+                'location': ('productionCountries', ..., {str}, all, {unpack(join_nonempty, delim='; ')}, filter),
+                'release_year': ('year', {str}, {lambda x: x[:4]}, {int_or_none}),
+                'duration': ('duration', {int_or_none}),
+                'series': ('seriesName', {str}),
+                'season_number': ('seasonNumber', {int_or_none}),
+                'episode': ('name', {str}, {lambda x: x if media_data['episodeNumber'] is not None else None}),
+                'episode_number': ('episodeNumber', {int_or_none}),
+            }),
         }

yt_dlp/extractor/sprout.py

@@ -1,61 +0,0 @@
-from .adobepass import AdobePassIE
-from ..utils import (
-    int_or_none,
-    smuggle_url,
-    update_url_query,
-)
-
-
-class SproutIE(AdobePassIE):
-    _VALID_URL = r'https?://(?:www\.)?(?:sproutonline|universalkids)\.com/(?:watch|(?:[^/]+/)*videos)/(?P<id>[^/?#]+)'
-    _TESTS = [{
-        'url': 'https://www.universalkids.com/shows/remy-and-boo/season/1/videos/robot-bike-race',
-        'info_dict': {
-            'id': 'bm0foJFaTKqb',
-            'ext': 'mp4',
-            'title': 'Robot Bike Race',
-            'description': 'md5:436b1d97117cc437f54c383f4debc66d',
-            'timestamp': 1606148940,
-            'upload_date': '20201123',
-            'uploader': 'NBCU-MPAT',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://www.sproutonline.com/watch/cowboy-adventure',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.universalkids.com/watch/robot-bike-race',
-        'only_matching': True,
-    }]
-    _GEO_COUNTRIES = ['US']
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        mpx_metadata = self._download_json(
-            # http://nbcuunikidsprod.apps.nbcuni.com/networks/universalkids/content/videos/
-            'https://www.universalkids.com/_api/videos/' + display_id,
-            display_id)['mpxMetadata']
-        media_pid = mpx_metadata['mediaPid']
-        theplatform_url = 'https://link.theplatform.com/s/HNK2IC/' + media_pid
-        query = {
-            'mbr': 'true',
-            'manifest': 'm3u',
-        }
-        if mpx_metadata.get('entitlement') == 'auth':
-            query['auth'] = self._extract_mvpd_auth(url, media_pid, 'sprout', 'sprout')
-        theplatform_url = smuggle_url(
-            update_url_query(theplatform_url, query), {
-                'force_smil_url': True,
-                'geo_countries': self._GEO_COUNTRIES,
-            })
-        return {
-            '_type': 'url_transparent',
-            'id': media_pid,
-            'url': theplatform_url,
-            'series': mpx_metadata.get('seriesName'),
-            'season_number': int_or_none(mpx_metadata.get('seasonNumber')),
-            'episode_number': int_or_none(mpx_metadata.get('episodeNumber')),
-            'ie_key': 'ThePlatform',
-        }

yt_dlp/extractor/svt.py

@@ -471,8 +471,7 @@ class SVTPageIE(SVTBaseIE):
         webpage = self._download_webpage(url, display_id)
         title = self._og_search_title(webpage)
 
-        urql_state = self._search_json(
+        urql_state = self._search_json(r'urqlState\s*[=:]', webpage, 'json data', display_id)
-            r'window\.svt\.(?:nyh\.)?urqlState\s*=', webpage, 'json data', display_id)
 
         data = traverse_obj(urql_state, (..., 'data', {str}, {json.loads}), get_all=False) or {}
 

yt_dlp/extractor/wat.py

@@ -2,9 +2,11 @@ from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
     int_or_none,
+    join_nonempty,
     try_get,
     unified_strdate,
 )
+from ..utils.traversal import traverse_obj
 
 
 class WatIE(InfoExtractor):
@@ -70,8 +72,14 @@ class WatIE(InfoExtractor):
 
         error_desc = video_info.get('error_desc')
         if error_desc:
-            if video_info.get('error_code') == 'GEOBLOCKED':
+            error_code = video_info.get('error_code')
+            if error_code == 'GEOBLOCKED':
                 self.raise_geo_restricted(error_desc, video_info.get('geoList'))
+            elif error_code == 'DELIVERY_ERROR':
+                if traverse_obj(video_data, ('delivery', 'code')) == 500:
+                    self.report_drm(video_id)
+                error_desc = join_nonempty(
+                    error_desc, traverse_obj(video_data, ('delivery', 'error', {str})), delim=': ')
             raise ExtractorError(error_desc, expected=True)
 
         title = video_info['title']

yt_dlp/extractor/weverse.py

@@ -290,12 +290,14 @@ class WeverseIE(WeverseBaseIE):
 
         elif live_status == 'is_live':
             video_info = self._call_api(
-                f'/video/v1.2/lives/{api_video_id}/playInfo?preview.format=json&preview.version=v2',
+                f'/video/v1.3/lives/{api_video_id}/playInfo?preview.format=json&preview.version=v2',
                 video_id, note='Downloading live JSON')
             playback = self._parse_json(video_info['lipPlayback'], video_id)
             m3u8_url = traverse_obj(playback, (
                 'media', lambda _, v: v['protocol'] == 'HLS', 'path', {url_or_none}), get_all=False)
-            formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', m3u8_id='hls', live=True)
+            # Live subtitles are not downloadable, but extract to silence "ignoring subs" warning
+            formats, _ = self._extract_m3u8_formats_and_subtitles(
+                m3u8_url, video_id, 'mp4', m3u8_id='hls', live=True)
 
         elif live_status == 'post_live':
             if availability in ('premium_only', 'subscriber_only'):

yt_dlp/extractor/youtube/_clip.py

@@ -37,6 +37,7 @@ class YoutubeClipIE(YoutubeTabBaseInfoExtractor):
             'chapters': 'count:20',
             'comment_count': int,
             'heatmap': 'count:100',
+            'media_type': 'clip',
         },
     }]
 
@@ -59,6 +60,7 @@ class YoutubeClipIE(YoutubeTabBaseInfoExtractor):
             'url': f'https://www.youtube.com/watch?v={video_id}',
             'ie_key': YoutubeIE.ie_key(),
             'id': clip_id,
+            'media_type': 'clip',
             'section_start': int(clip_data['startTimeMs']) / 1000,
             'section_end': int(clip_data['endTimeMs']) / 1000,
             '_format_sort_fields': (  # https protocol is prioritized for ffmpeg compatibility

yt_dlp/extractor/youtube/_redirect.py

@@ -35,6 +35,7 @@ class YoutubeYtBeIE(YoutubeBaseInfoExtractor):
             'duration': 59,
             'comment_count': int,
             'channel_follower_count': int,
+            'media_type': 'short',
         },
         'params': {
             'noplaylist': True,

yt_dlp/extractor/youtube/_video.py

@@ -23,6 +23,8 @@ from ._base import (
     _split_innertube_client,
     short_client_name,
 )
+from .pot._director import initialize_pot_director
+from .pot.provider import PoTokenContext, PoTokenRequest
 from ..openload import PhantomJSwrapper
 from ...jsinterp import JSInterpreter
 from ...networking.exceptions import HTTPError
@@ -66,6 +68,7 @@ from ...utils import (
     urljoin,
     variadic,
 )
+from ...utils.networking import clean_headers, clean_proxies, select_proxy
 
 STREAMING_DATA_CLIENT_NAME = '__yt_dlp_client'
 STREAMING_DATA_INITIAL_PO_TOKEN = '__yt_dlp_po_token'
@@ -376,6 +379,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader': 'Afrojack',
                 'uploader_url': 'https://www.youtube.com/@Afrojack',
                 'uploader_id': '@Afrojack',
+                'media_type': 'video',
             },
             'params': {
                 'youtube_include_dash_manifest': True,
@@ -413,10 +417,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
                 'timestamp': 1401991663,
+                'media_type': 'video',
             },
         },
         {
-            'note': 'Age-gate video with embed allowed in public site',
+            'note': 'Formerly an age-gate video with embed allowed in public site',
             'url': 'https://youtube.com/watch?v=HsUATh_Nc2U',
             'info_dict': {
                 'id': 'HsUATh_Nc2U',
@@ -424,8 +429,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'title': 'Godzilla 2 (Official Video)',
                 'description': 'md5:bf77e03fcae5529475e500129b05668a',
                 'upload_date': '20200408',
-                'age_limit': 18,
+                'age_limit': 0,
-                'availability': 'needs_auth',
+                'availability': 'public',
                 'channel_id': 'UCYQT13AtrJC0gsM1far_zJg',
                 'channel': 'FlyingKitty',
                 'channel_url': 'https://www.youtube.com/channel/UCYQT13AtrJC0gsM1far_zJg',
@@ -443,8 +448,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@FlyingKitty900',
                 'comment_count': int,
                 'channel_is_verified': True,
+                'media_type': 'video',
             },
-            'skip': 'Age-restricted; requires authentication',
         },
         {
             'note': 'Age-gate video embedable only with clientScreen=EMBED',
@@ -507,6 +512,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader': 'Herr Lurik',
                 'uploader_url': 'https://www.youtube.com/@HerrLurik',
                 'uploader_id': '@HerrLurik',
+                'media_type': 'video',
             },
         },
         {
@@ -546,6 +552,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader': 'deadmau5',
                 'uploader_url': 'https://www.youtube.com/@deadmau5',
                 'uploader_id': '@deadmau5',
+                'media_type': 'video',
             },
             'expected_warnings': [
                 'DASH manifest missing',
@@ -581,6 +588,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@Olympics',
                 'channel_is_verified': True,
                 'timestamp': 1440707674,
+                'media_type': 'livestream',
             },
             'params': {
                 'skip_download': 'requires avconv',
@@ -615,6 +623,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_url': 'https://www.youtube.com/@AllenMeow',
                 'uploader_id': '@AllenMeow',
                 'timestamp': 1299776999,
+                'media_type': 'video',
             },
         },
         # url_encoded_fmt_stream_map is empty string
@@ -809,6 +818,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'like_count': int,
                 'age_limit': 0,
                 'channel_follower_count': int,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -868,6 +878,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@BKCHarvard',
                 'uploader_url': 'https://www.youtube.com/@BKCHarvard',
                 'timestamp': 1422422076,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -904,6 +915,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
                 'timestamp': 1447987198,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -968,6 +980,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'comment_count': int,
                 'channel_is_verified': True,
                 'timestamp': 1484761047,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1070,6 +1083,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'tags': 'count:11',
                 'live_status': 'not_live',
                 'channel_follower_count': int,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1124,6 +1138,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_url': 'https://www.youtube.com/@ElevageOrVert',
                 'uploader_id': '@ElevageOrVert',
                 'timestamp': 1497343210,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1163,6 +1178,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
                 'timestamp': 1377976349,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1207,6 +1223,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_follower_count': int,
                 'uploader': 'The Cinematic Orchestra',
                 'comment_count': int,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1275,6 +1292,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_url': 'https://www.youtube.com/@walkaroundjapan7124',
                 'uploader_id': '@walkaroundjapan7124',
                 'timestamp': 1605884416,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1371,6 +1389,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
                 'timestamp': 1395685455,
+                'media_type': 'video',
             }, 'params': {'format': 'mhtml', 'skip_download': True},
         }, {
             # Ensure video upload_date is in UTC timezone (video was uploaded 1641170939)
@@ -1401,6 +1420,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@LeonNguyen',
                 'heatmap': 'count:100',
                 'timestamp': 1641170939,
+                'media_type': 'video',
             },
         }, {
             # date text is premiered video, ensure upload date in UTC (published 1641172509)
@@ -1434,6 +1454,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
                 'timestamp': 1641172509,
+                'media_type': 'video',
             },
         },
         {   # continuous livestream.
@@ -1495,6 +1516,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader': 'Lesmiscore',
                 'uploader_url': 'https://www.youtube.com/@lesmiscore',
                 'timestamp': 1648005313,
+                'media_type': 'short',
             },
         }, {
             # Prefer primary title+description language metadata by default
@@ -1523,6 +1545,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@coletdjnz',
                 'uploader': 'cole-dlp-test-acc',
                 'timestamp': 1662677394,
+                'media_type': 'video',
             },
             'params': {'skip_download': True},
         }, {
@@ -1551,6 +1574,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader': 'cole-dlp-test-acc',
                 'timestamp': 1659073275,
                 'like_count': int,
+                'media_type': 'video',
             },
             'params': {'skip_download': True, 'extractor_args': {'youtube': {'lang': ['fr']}}},
             'expected_warnings': [r'Preferring "fr" translated fields'],
@@ -1587,6 +1611,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'comment_count': int,
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
+                'media_type': 'video',
             },
             'params': {'extractor_args': {'youtube': {'player_client': ['ios']}}, 'format': '233-1'},
         }, {
@@ -1687,6 +1712,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'comment_count': int,
                 'channel_is_verified': True,
                 'heatmap': 'count:100',
+                'media_type': 'video',
             },
             'params': {
                 'extractor_args': {'youtube': {'player_client': ['ios'], 'player_skip': ['webpage']}},
@@ -1719,6 +1745,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'channel_follower_count': int,
                 'categories': ['People & Blogs'],
                 'tags': [],
+                'media_type': 'short',
             },
         },
     ]
@@ -1754,6 +1781,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'uploader_id': '@ChristopherSykesDocumentaries',
                 'heatmap': 'count:100',
                 'timestamp': 1211825920,
+                'media_type': 'video',
             },
             'params': {
                 'skip_download': True,
@@ -1784,6 +1812,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         super().__init__(*args, **kwargs)
         self._code_cache = {}
         self._player_cache = {}
+        self._pot_director = None
+
+    def _real_initialize(self):
+        super()._real_initialize()
+        self._pot_director = initialize_pot_director(self)
 
     def _prepare_live_from_start_formats(self, formats, video_id, live_start_time, url, webpage_url, smuggled_data, is_live):
         lock = threading.Lock()
@@ -1819,6 +1852,12 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     else:
                         retry.error = f'Cannot find refreshed manifest for format {format_id}{bug_reports_message()}'
                     continue
+
+                # Formats from ended premieres will be missing a manifest_url
+                # See https://github.com/yt-dlp/yt-dlp/issues/8543
+                if not f.get('manifest_url'):
+                    break
+
                 return f['manifest_url'], f['manifest_stream_number'], is_live
             return None
 
@@ -2824,7 +2863,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 continue
 
     def fetch_po_token(self, client='web', context=_PoTokenContext.GVS, ytcfg=None, visitor_data=None,
-                       data_sync_id=None, session_index=None, player_url=None, video_id=None, **kwargs):
+                       data_sync_id=None, session_index=None, player_url=None, video_id=None, webpage=None, **kwargs):
         """
         Fetch a PO Token for a given client and context. This function will validate required parameters for a given context and client.
 
@@ -2838,10 +2877,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         @param session_index: session index.
         @param player_url: player URL.
         @param video_id: video ID.
+        @param webpage: video webpage.
         @param kwargs: Additional arguments to pass down. May be more added in the future.
         @return: The fetched PO Token. None if it could not be fetched.
         """
 
+        # TODO(future): This validation should be moved into pot framework.
+        #  Some sort of middleware or validation provider perhaps?
+
         # GVS WebPO Token is bound to visitor_data / Visitor ID when logged out.
         # Must have visitor_data for it to function.
         if player_url and context == _PoTokenContext.GVS and not visitor_data and not self.is_authenticated:
@@ -2863,6 +2906,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     f'Got a GVS PO Token for {client} client, but missing Data Sync ID for account. Formats may not work.'
                     f'You may need to pass a Data Sync ID with --extractor-args "youtube:data_sync_id=XXX"')
 
+            self.write_debug(f'{video_id}: Retrieved a {context.value} PO Token for {client} client from config')
             return config_po_token
 
         # Require GVS WebPO Token if logged in for external fetching
@@ -2872,7 +2916,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 f'You may need to pass a Data Sync ID with --extractor-args "youtube:data_sync_id=XXX"')
             return
 
-        return self._fetch_po_token(
+        po_token = self._fetch_po_token(
             client=client,
             context=context.value,
             ytcfg=ytcfg,
@@ -2881,11 +2925,66 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             session_index=session_index,
             player_url=player_url,
             video_id=video_id,
+            video_webpage=webpage,
             **kwargs,
         )
 
+        if po_token:
+            self.write_debug(f'{video_id}: Retrieved a {context.value} PO Token for {client} client')
+            return po_token
+
     def _fetch_po_token(self, client, **kwargs):
-        """(Unstable) External PO Token fetch stub"""
+        context = kwargs.get('context')
+
+        # Avoid fetching PO Tokens when not required
+        fetch_pot_policy = self._configuration_arg('fetch_pot', [''], ie_key=YoutubeIE)[0]
+        if fetch_pot_policy not in ('never', 'auto', 'always'):
+            fetch_pot_policy = 'auto'
+        if (
+            fetch_pot_policy == 'never'
+            or (
+                fetch_pot_policy == 'auto'
+                and _PoTokenContext(context) not in self._get_default_ytcfg(client)['PO_TOKEN_REQUIRED_CONTEXTS']
+            )
+        ):
+            return None
+
+        headers = self.get_param('http_headers').copy()
+        proxies = self._downloader.proxies.copy()
+        clean_headers(headers)
+        clean_proxies(proxies, headers)
+
+        innertube_host = self._select_api_hostname(None, default_client=client)
+
+        pot_request = PoTokenRequest(
+            context=PoTokenContext(context),
+            innertube_context=traverse_obj(kwargs, ('ytcfg', 'INNERTUBE_CONTEXT')),
+            innertube_host=innertube_host,
+            internal_client_name=client,
+            session_index=kwargs.get('session_index'),
+            player_url=kwargs.get('player_url'),
+            video_webpage=kwargs.get('video_webpage'),
+            is_authenticated=self.is_authenticated,
+            visitor_data=kwargs.get('visitor_data'),
+            data_sync_id=kwargs.get('data_sync_id'),
+            video_id=kwargs.get('video_id'),
+            request_cookiejar=self._downloader.cookiejar,
+
+            # All requests that would need to be proxied should be in the
+            # context of www.youtube.com or the innertube host
+            request_proxy=(
+                select_proxy('https://www.youtube.com', proxies)
+                or select_proxy(f'https://{innertube_host}', proxies)
+            ),
+            request_headers=headers,
+            request_timeout=self.get_param('socket_timeout'),
+            request_verify_tls=not self.get_param('nocheckcertificate'),
+            request_source_address=self.get_param('source_address'),
+
+            bypass_cache=False,
+        )
+
+        return self._pot_director.get_po_token(pot_request)
 
     @staticmethod
     def _is_agegated(player_response):
@@ -3043,8 +3142,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'video_id': video_id,
                 'data_sync_id': data_sync_id if self.is_authenticated else None,
                 'player_url': player_url if require_js_player else None,
+                'webpage': webpage,
                 'session_index': self._extract_session_index(master_ytcfg, player_ytcfg),
-                'ytcfg': player_ytcfg,
+                'ytcfg': player_ytcfg or self._get_default_ytcfg(client),
             }
 
             player_po_token = self.fetch_po_token(
@@ -3109,9 +3209,19 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 else:
                     prs.append(pr)
 
+            # web_embedded can work around age-gate and age-verification for some embeddable videos
+            if self._is_agegated(pr) and variant != 'web_embedded':
+                append_client(f'web_embedded.{base_client}')
+            # Unauthenticated users will only get web_embedded client formats if age-gated
+            if self._is_agegated(pr) and not self.is_authenticated:
+                self.to_screen(
+                    f'{video_id}: This video is age-restricted; some formats may be missing '
+                    f'without authentication. {self._youtube_login_hint}', only_once=True)
+
             # EU countries require age-verification for accounts to access age-restricted videos
             # If account is not age-verified, _is_agegated() will be truthy for non-embedded clients
-            if self.is_authenticated and self._is_agegated(pr):
+            embedding_is_disabled = variant == 'web_embedded' and self._is_unplayable(pr)
+            if self.is_authenticated and (self._is_agegated(pr) or embedding_is_disabled):
                 self.to_screen(
                     f'{video_id}: This video is age-restricted and YouTube is requiring '
                     'account age-verification; some formats may be missing', only_once=True)
@@ -3771,7 +3881,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'tags': keywords,
             'playable_in_embed': get_first(playability_statuses, 'playableInEmbed'),
             'live_status': live_status,
-            'media_type': 'livestream' if get_first(video_details, 'isLiveContent') else None,
+            'media_type': (
+                'livestream' if get_first(video_details, 'isLiveContent')
+                else 'short' if get_first(microformats, 'isShortsEligible')
+                else 'video'),
             'release_timestamp': live_start_time,
             '_format_sort_fields': (  # source_preference is lower for potentially damaged formats
                 'quality', 'res', 'fps', 'hdr:12', 'source', 'vcodec', 'channels', 'acodec', 'lang', 'proto'),

yt_dlp/extractor/youtube/pot/README.md

@@ -0,0 +1,305 @@
+# YoutubeIE PO Token Provider Framework
+
+As part of the YouTube extractor, we have a framework for providing PO Tokens programmatically. This can be used in plugins.
+
+Refer to the [PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide) for more information on PO Tokens.
+
+## Public APIs
+
+- `yt_dlp.extractor.youtube.pot.cache`
+- `yt_dlp.extractor.youtube.pot.provider`
+- `yt_dlp.extractor.youtube.pot.utils`
+
+Everything else is internal-only and no guarantees are made about the API stability.
+
+> [!WARNING]
+> We will try our best to maintain stability with the public APIs.
+> However, due to the nature of extractors and YouTube, we may need to remove or change APIs in the future.
+> If you are using these APIs outside yt-dlp plugins, please account for this by importing them safely.
+
+## PO Token Provider
+
+`yt_dlp.extractor.youtube.pot.provider`
+
+```python
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+    PoTokenContext,
+    PoTokenProvider,
+    PoTokenResponse,
+    PoTokenProviderError,
+    PoTokenProviderRejectedRequest,
+    register_provider,
+    register_preference,
+    ExternalRequestFeature,
+)
+from yt_dlp.networking.common import Request
+from yt_dlp.extractor.youtube.pot.utils import get_webpo_content_binding
+from yt_dlp.utils import traverse_obj
+from yt_dlp.networking.exceptions import RequestError
+import json
+
+
+@register_provider
+class MyPoTokenProviderPTP(PoTokenProvider):  # Provider class name must end with "PTP"
+    PROVIDER_VERSION = '0.2.1'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'my-provider'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    # -- Validation shortcuts. Set these to None to disable. --
+
+    # Innertube Client Name.
+    # For example, "WEB", "ANDROID", "TVHTML5".
+    # For a list of WebPO client names, 
+    #  see yt_dlp.extractor.youtube.pot.utils.WEBPO_CLIENTS.
+    # Also see yt_dlp.extractor.youtube._base.INNERTUBE_CLIENTS 
+    #  for a list of client names currently supported by the YouTube extractor.
+    _SUPPORTED_CLIENTS = ('WEB', 'TVHTML5')
+
+    _SUPPORTED_CONTEXTS = (
+        PoTokenContext.GVS,
+    )
+
+    # If your provider makes external requests to websites (i.e. to youtube.com) 
+    #  using another library or service (i.e., not _request_webpage),
+    # set the request features that are supported here.
+    # If only using _request_webpage to make external requests, set this to None.
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES = (
+        ExternalRequestFeature.PROXY_SCHEME_HTTP, 
+        ExternalRequestFeature.SOURCE_ADDRESS, 
+        ExternalRequestFeature.DISABLE_TLS_VERIFICATION
+    )
+
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method SHOULD NOT make any network requests or perform any expensive operations.
+
+        Since this is called multiple times, we recommend caching the result.
+        """
+        return True
+
+    def close(self):
+        # Optional close hook, called when YoutubeDL is closed.
+        pass
+
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        # ℹ️ If you need to validate the request before making the request to the external source.
+        # Raise yt_dlp.extractor.youtube.pot.provider.PoTokenProviderRejectedRequest if the request is not supported.
+        if request.is_authenticated:
+            raise PoTokenProviderRejectedRequest(
+                'This provider does not support authenticated requests'
+            )
+
+        # ℹ️ Settings are pulled from extractor args passed to yt-dlp with the key `youtubepot-<PROVIDER_KEY>`.
+        # For this example, the extractor arg would be:
+        # `--extractor-args "youtubepot-mypotokenprovider:url=https://custom.example.com/get_pot"`
+        external_provider_url = self._configuration_arg(
+            'url', default=['https://provider.example.com/get_pot'])[0]
+        
+        # See below for logging guidelines
+        self.logger.trace(f'Using external provider URL: {external_provider_url}')
+        
+        # You should use the internal HTTP client to make requests where possible,
+        # as it will handle cookies and other networking settings passed to yt-dlp.
+        try:
+            # See docstring in _request_webpage method for request tips
+            response = self._request_webpage(
+                Request(external_provider_url, data=json.dumps({
+                    'content_binding': get_webpo_content_binding(request),
+                    'proxy': request.request_proxy,
+                    'headers': request.request_headers,
+                    'source_address': request.request_source_address,
+                    'verify_tls': request.request_verify_tls,
+                    # Important: If your provider has its own caching, please respect `bypass_cache`.
+                    # This may be used in the future to request a fresh PO Token if required.
+                    'do_not_cache': request.bypass_cache,
+                }).encode(), proxies={'all': None}), 
+                pot_request=request, 
+                note=(
+                  f'Requesting {request.context.value} PO Token '
+                  f'for {request.internal_client_name} client from external provider'),
+            )
+
+        except RequestError as e:
+            # ℹ️ If there is an error, raise PoTokenProviderError.
+            # You can specify whether it is expected or not. If it is unexpected, 
+            #  the log will include a link to the bug report location (BUG_REPORT_LOCATION).
+            raise PoTokenProviderError(
+                'Networking error while fetching to get PO Token from external provider',
+                expected=True
+            ) from e
+
+        # Note: PO Token is expected to be base64url encoded
+        po_token = traverse_obj(response, 'po_token')
+        if not po_token:
+            raise PoTokenProviderError(
+                'Bad PO Token Response from external provider',
+                expected=False
+            )
+
+        return PoTokenResponse(
+            po_token=po_token,
+            # Optional, add a custom expiration time for the token. Use for caching.
+            # By default, yt-dlp will use the default ttl from a registered cache spec (see below)
+            # Set to 0 or -1 to not cache this response.
+            expires_at=None,
+        )
+
+
+# If there are multiple PO Token Providers that can handle the same PoTokenRequest,
+# you can define a preference function to increase/decrease the priority of providers.
+
+@register_preference(MyPoTokenProviderPTP)
+def my_provider_preference(provider: PoTokenProvider, request: PoTokenRequest) -> int:
+    return 50
+```
+
+## Logging Guidelines
+
+- Use the `self.logger` object to log messages.
+- When making HTTP requests or any other expensive operation, use `self.logger.info` to log a message to standard non-verbose output.
+  - This lets users know what is happening when a time-expensive operation is taking place.
+  - It is recommended to include the PO Token context and internal client name in the message if possible.
+  - For example, `self.logger.info(f'Requesting {request.context.value} PO Token for {request.internal_client_name} client from external provider')`.
+- Use `self.logger.debug` to log a message to the verbose output (`--verbose`).
+  - For debugging information visible to users posting verbose logs.
+  - Try to not log too much, prefer using trace logging for detailed debug messages.
+- Use `self.logger.trace` to log a message to the PO Token debug output (`--extractor-args "youtube:pot_debug=true"`). 
+  - Log as much as you like here as needed for debugging your provider.
+- Avoid logging PO Tokens or any sensitive information to debug or info output.
+
+## Debugging
+
+- Use `-v --extractor-args "youtube:pot_trace=true"` to enable PO Token debug output.
+
+## Caching
+
+> [!WARNING]
+> The following describes more advance features that most users/developers will not need to use.
+
+> [!IMPORTANT]
+> yt-dlp currently has a built-in LRU Memory Cache Provider and a cache spec provider for WebPO Tokens. 
+> You should only need to implement cache providers if you want an external cache, or a cache spec if you are handling non-WebPO Tokens.
+
+### Cache Providers
+
+`yt_dlp.extractor.youtube.pot.cache`
+
+```python
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    register_preference,
+    register_provider
+)
+
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+@register_provider
+class MyCacheProviderPCP(PoTokenCacheProvider):  # Provider class name must end with "PCP"
+    PROVIDER_VERSION = '0.1.0'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'my-cache-provider'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method SHOULD NOT make any network requests or perform any expensive operations.
+
+        Since this is called multiple times, we recommend caching the result.
+        """
+        return True
+
+    def get(self, key: str):
+        # ℹ️ Similar to PO Token Providers, Cache Providers and Cache Spec Providers 
+        # are passed down extractor args matching key youtubepot-<PROVIDER_KEY>.
+        some_setting = self._configuration_arg('some_setting', default=['default_value'])[0]
+        return self.my_cache.get(key)
+
+    def store(self, key: str, value: str, expires_at: int):
+        # ⚠ expires_at MUST be respected. 
+        # Cache entries should not be returned if they have expired.
+        self.my_cache.store(key, value, expires_at)
+
+    def delete(self, key: str):
+        self.my_cache.delete(key)
+
+    def close(self):
+        # Optional close hook, called when the YoutubeDL instance is closed.
+        pass
+
+# If there are multiple PO Token Cache Providers available, you can 
+# define a preference function to increase/decrease the priority of providers. 
+
+# IMPORTANT: Providers should be in preference of cache lookup time. 
+# For example, a memory cache should have a higher preference than a disk cache. 
+
+# VERY IMPORTANT: yt-dlp has a built-in memory cache with a priority of 10000. 
+# Your cache provider should be lower than this.
+
+
+@register_preference(MyCacheProviderPCP)
+def my_cache_preference(provider: PoTokenCacheProvider, request: PoTokenRequest) -> int:
+    return 50
+```
+
+### Cache Specs
+
+`yt_dlp.extractor.youtube.pot.cache`
+
+These are used to provide information on how to cache a particular PO Token Request. 
+You might have a different cache spec for different kinds of PO Tokens.
+
+```python
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    CacheProviderWritePolicy,
+    register_spec,
+)
+from yt_dlp.utils import traverse_obj
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+@register_spec
+class MyCacheSpecProviderPCSP(PoTokenCacheSpecProvider):  # Provider class name must end with "PCSP"
+    PROVIDER_VERSION = '0.1.0'
+    # Define a unique display name for the provider
+    PROVIDER_NAME = 'mycachespec'
+    BUG_REPORT_LOCATION = 'https://issues.example.com/report'
+
+    def generate_cache_spec(self, request: PoTokenRequest):
+
+        client_name = traverse_obj(request.innertube_context, ('client', 'clientName'))
+        if client_name != 'ANDROID':
+            # ℹ️ If the request is not supported by the cache spec, return None
+            return None
+
+        # Generate a cache spec for the request
+        return PoTokenCacheSpec(
+            # Key bindings to uniquely identify the request. These are used to generate a cache key.
+            key_bindings={
+                'client_name': client_name,
+                'content_binding': 'unique_content_binding',
+                'ip': traverse_obj(request.innertube_context, ('client', 'remoteHost')),
+                'source_address': request.request_source_address,
+                'proxy': request.request_proxy,
+            },
+            # Default Cache TTL in seconds
+            default_ttl=21600,
+
+            # Optional: Specify a write policy.
+            # WRITE_FIRST will write to the highest priority provider only, 
+            #  whereas WRITE_ALL will write to all providers.
+            # WRITE_FIRST may be useful if the PO Token is short-lived 
+            #  and there is no use writing to all providers.
+            write_policy=CacheProviderWritePolicy.WRITE_ALL,
+        )
+```

yt_dlp/extractor/youtube/pot/__init__.py

@@ -0,0 +1,2 @@
+# Trigger import of built-in providers
+from ._builtin import _trigger_import  # noqa: F401

yt_dlp/extractor/youtube/pot/_builtin/__init__.py

@@ -0,0 +1,4 @@
+from .memory_cache import MemoryLRUPCP as _MemoryLRUPCP  # noqa: F401
+from .webpo_cachespec import WebPoPCSP as _WebPoPCSP  # noqa: F401
+
+_trigger_import = None  # noqa: F401

yt_dlp/extractor/youtube/pot/_builtin/memory_cache.py

@@ -0,0 +1,78 @@
+from __future__ import annotations
+
+import datetime as dt
+import typing
+from collections import OrderedDict
+from threading import Lock
+
+from yt_dlp.extractor.youtube.pot._provider import BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot._registry import _pot_memory_cache
+from yt_dlp.extractor.youtube.pot.cache import (
+    PoTokenCacheProvider,
+    register_preference,
+    register_provider,
+)
+
+
+def initialize_global_cache(max_size: int):
+    if _pot_memory_cache.value.get('cache') is None:
+        _pot_memory_cache.value['cache'] = OrderedDict()
+        _pot_memory_cache.value['lock'] = Lock()
+        _pot_memory_cache.value['max_size'] = max_size
+
+    if _pot_memory_cache.value['max_size'] != max_size:
+        raise ValueError('Cannot change max_size of initialized global memory cache')
+
+    return (
+        _pot_memory_cache.value['cache'],
+        _pot_memory_cache.value['lock'],
+        _pot_memory_cache.value['max_size'],
+    )
+
+
+@register_provider
+class MemoryLRUPCP(PoTokenCacheProvider, BuiltInIEContentProvider):
+    PROVIDER_NAME = 'memory'
+    DEFAULT_CACHE_SIZE = 25
+
+    def __init__(
+        self,
+        *args,
+        initialize_cache: typing.Callable[[int], tuple[OrderedDict, Lock, int]] = initialize_global_cache,
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        self.cache, self.lock, self.max_size = initialize_cache(self.DEFAULT_CACHE_SIZE)
+
+    def is_available(self) -> bool:
+        return True
+
+    def get(self, key: str) -> str | None:
+        with self.lock:
+            if key not in self.cache:
+                return None
+            value, expires_at = self.cache.pop(key)
+            if expires_at < int(dt.datetime.now(dt.timezone.utc).timestamp()):
+                return None
+            self.cache[key] = (value, expires_at)
+            return value
+
+    def store(self, key: str, value: str, expires_at: int):
+        with self.lock:
+            if expires_at < int(dt.datetime.now(dt.timezone.utc).timestamp()):
+                return
+            if key in self.cache:
+                self.cache.pop(key)
+            self.cache[key] = (value, expires_at)
+            if len(self.cache) > self.max_size:
+                self.cache.popitem(last=False)
+
+    def delete(self, key: str):
+        with self.lock:
+            self.cache.pop(key, None)
+
+
+@register_preference(MemoryLRUPCP)
+def memorylru_preference(*_, **__):
+    # Memory LRU Cache SHOULD be the highest priority
+    return 10000

yt_dlp/extractor/youtube/pot/_builtin/webpo_cachespec.py

@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+from yt_dlp.extractor.youtube.pot._provider import BuiltInIEContentProvider
+from yt_dlp.extractor.youtube.pot.cache import (
+    CacheProviderWritePolicy,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+    register_spec,
+)
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenRequest,
+)
+from yt_dlp.extractor.youtube.pot.utils import ContentBindingType, get_webpo_content_binding
+from yt_dlp.utils import traverse_obj
+
+
+@register_spec
+class WebPoPCSP(PoTokenCacheSpecProvider, BuiltInIEContentProvider):
+    PROVIDER_NAME = 'webpo'
+
+    def generate_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        bind_to_visitor_id = self._configuration_arg(
+            'bind_to_visitor_id', default=['true'])[0] == 'true'
+
+        content_binding, content_binding_type = get_webpo_content_binding(
+            request, bind_to_visitor_id=bind_to_visitor_id)
+
+        if not content_binding or not content_binding_type:
+            return None
+
+        write_policy = CacheProviderWritePolicy.WRITE_ALL
+        if content_binding_type == ContentBindingType.VIDEO_ID:
+            write_policy = CacheProviderWritePolicy.WRITE_FIRST
+
+        return PoTokenCacheSpec(
+            key_bindings={
+                't': 'webpo',
+                'cb': content_binding,
+                'cbt': content_binding_type.value,
+                'ip': traverse_obj(request.innertube_context, ('client', 'remoteHost')),
+                'sa': request.request_source_address,
+                'px': request.request_proxy,
+            },
+            # Integrity token response usually states it has a ttl of 12 hours (43200 seconds).
+            # We will default to 6 hours to be safe.
+            default_ttl=21600,
+            write_policy=write_policy,
+        )

yt_dlp/extractor/youtube/pot/_director.py

@@ -0,0 +1,482 @@
+from __future__ import annotations
+
+import base64
+import binascii
+import dataclasses
+import datetime as dt
+import hashlib
+import json
+import typing
+import urllib.parse
+from collections.abc import Iterable
+
+from yt_dlp.extractor.youtube.pot._provider import (
+    BuiltInIEContentProvider,
+    IEContentProvider,
+    IEContentProviderLogger,
+)
+from yt_dlp.extractor.youtube.pot._registry import (
+    _pot_cache_provider_preferences,
+    _pot_cache_providers,
+    _pot_pcs_providers,
+    _pot_providers,
+    _ptp_preferences,
+)
+from yt_dlp.extractor.youtube.pot.cache import (
+    CacheProviderWritePolicy,
+    PoTokenCacheProvider,
+    PoTokenCacheProviderError,
+    PoTokenCacheSpec,
+    PoTokenCacheSpecProvider,
+)
+from yt_dlp.extractor.youtube.pot.provider import (
+    PoTokenProvider,
+    PoTokenProviderError,
+    PoTokenProviderRejectedRequest,
+    PoTokenRequest,
+    PoTokenResponse,
+    provider_bug_report_message,
+)
+from yt_dlp.utils import ExtractorError, bug_reports_message, format_field, join_nonempty, traverse_obj
+
+if typing.TYPE_CHECKING:
+    from yt_dlp.extractor.youtube.pot.cache import CacheProviderPreference
+    from yt_dlp.extractor.youtube.pot.provider import Preference
+
+
+class YoutubeIEContentProviderLogger(IEContentProviderLogger):
+    def __init__(self, ie, prefix, log_level: IEContentProviderLogger.LogLevel | None = None):
+        self.__ie = ie
+        self.prefix = prefix
+        self.log_level = log_level if log_level is not None else self.LogLevel.INFO
+
+    def _format_msg(self, message: str):
+        prefixstr = format_field(self.prefix, None, '[%s] ')
+        return f'{prefixstr}{message}'
+
+    def trace(self, message: str):
+        if self.log_level <= self.LogLevel.TRACE:
+            self.__ie.write_debug(self._format_msg('TRACE: ' + message))
+
+    def debug(self, message: str):
+        if self.log_level <= self.LogLevel.DEBUG:
+            self.__ie.write_debug(self._format_msg(message))
+
+    def info(self, message: str):
+        if self.log_level <= self.LogLevel.INFO:
+            self.__ie.to_screen(self._format_msg(message))
+
+    def warning(self, message: str, *, once=False):
+        if self.log_level <= self.LogLevel.WARNING:
+            self.__ie.report_warning(self._format_msg(message), only_once=once)
+
+    def error(self, message: str):
+        if self.log_level <= self.LogLevel.ERROR:
+            self.__ie._downloader.report_error(self._format_msg(message), is_error=False)
+
+
+class PoTokenCache:
+
+    def __init__(
+        self,
+        logger: IEContentProviderLogger,
+        cache_providers: list[PoTokenCacheProvider],
+        cache_spec_providers: list[PoTokenCacheSpecProvider],
+        cache_provider_preferences: list[CacheProviderPreference] | None = None,
+    ):
+        self.cache_providers: dict[str, PoTokenCacheProvider] = {
+            provider.PROVIDER_KEY: provider for provider in (cache_providers or [])}
+        self.cache_provider_preferences: list[CacheProviderPreference] = cache_provider_preferences or []
+        self.cache_spec_providers: dict[str, PoTokenCacheSpecProvider] = {
+            provider.PROVIDER_KEY: provider for provider in (cache_spec_providers or [])}
+        self.logger = logger
+
+    def _get_cache_providers(self, request: PoTokenRequest) -> Iterable[PoTokenCacheProvider]:
+        """Sorts available cache providers by preference, given a request"""
+        preferences = {
+            provider: sum(pref(provider, request) for pref in self.cache_provider_preferences)
+            for provider in self.cache_providers.values()
+        }
+        if self.logger.log_level <= self.logger.LogLevel.TRACE:
+            # calling is_available() for every PO Token provider upfront may have some overhead
+            self.logger.trace(f'PO Token Cache Providers: {provider_display_list(self.cache_providers.values())}')
+            self.logger.trace('Cache Provider preferences for this request: {}'.format(', '.join(
+                f'{provider.PROVIDER_KEY}={pref}' for provider, pref in preferences.items())))
+
+        return (
+            provider for provider in sorted(
+                self.cache_providers.values(), key=preferences.get, reverse=True) if provider.is_available())
+
+    def _get_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        for provider in self.cache_spec_providers.values():
+            if not provider.is_available():
+                continue
+            try:
+                spec = provider.generate_cache_spec(request)
+                if not spec:
+                    continue
+                if not validate_cache_spec(spec):
+                    self.logger.error(
+                        f'PoTokenCacheSpecProvider "{provider.PROVIDER_KEY}" generate_cache_spec() '
+                        f'returned invalid spec {spec}{provider_bug_report_message(provider)}')
+                    continue
+                spec = dataclasses.replace(spec, _provider=provider)
+                self.logger.trace(
+                    f'Retrieved cache spec {spec} from cache spec provider "{provider.PROVIDER_NAME}"')
+                return spec
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache spec provider: '
+                    f'{e!r}{provider_bug_report_message(provider)}')
+                continue
+        return None
+
+    def _generate_key_bindings(self, spec: PoTokenCacheSpec) -> dict[str, str]:
+        bindings_cleaned = {
+            **{k: v for k, v in spec.key_bindings.items() if v is not None},
+            # Allow us to invalidate caches if such need arises
+            '_yt': 'v1',
+        }
+        if spec._provider:
+            bindings_cleaned['_p'] = spec._provider.PROVIDER_KEY
+        self.logger.trace(
+            'Generate cache key bindings: {}'.format(', '.join(f'{k}={v}' for k, v in bindings_cleaned.items())))
+        return bindings_cleaned
+
+    def _generate_key(self, bindings: dict) -> str:
+        binding_string = ''.join(f'{k}{v}' for k, v in sorted(bindings.items()))
+        return hashlib.sha256(binding_string.encode()).hexdigest()
+
+    def get(self, request: PoTokenRequest) -> PoTokenResponse | None:
+        spec = self._get_cache_spec(request)
+        if not spec:
+            self.logger.trace('No cache spec available for this request, unable to fetch from cache')
+            return None
+
+        cache_key = self._generate_key(self._generate_key_bindings(spec))
+        self.logger.trace(f'Attempting to access PO Token cache using key: {cache_key}')
+
+        for idx, provider in enumerate(self._get_cache_providers(request)):
+            try:
+                self.logger.trace(
+                    f'Attempting to fetch PO Token response from "{provider.PROVIDER_NAME}" cache provider')
+                cache_response = provider.get(cache_key)
+                if not cache_response:
+                    continue
+                try:
+                    po_token_response = PoTokenResponse(**json.loads(cache_response))
+                except (TypeError, ValueError, json.JSONDecodeError):
+                    po_token_response = None
+                if not validate_response(po_token_response):
+                    self.logger.error(
+                        f'Invalid PO Token response retrieved from cache provider "{provider.PROVIDER_NAME}": '
+                        f'{cache_response}{provider_bug_report_message(provider)}')
+                    provider.delete(cache_key)
+                    continue
+                self.logger.trace(
+                    f'PO Token response retrieved from cache using "{provider.PROVIDER_NAME}" provider: '
+                    f'{po_token_response}')
+                if idx > 0:
+                    # Write back to the highest priority cache provider,
+                    # so we stop trying to fetch from lower priority providers
+                    self.logger.trace('Writing PO Token response to highest priority cache provider')
+                    self.store(request, po_token_response, write_policy=CacheProviderWritePolicy.WRITE_FIRST)
+
+                return po_token_response
+            except PoTokenCacheProviderError as e:
+                self.logger.warning(
+                    f'Error from "{provider.PROVIDER_NAME}" PO Token cache provider: '
+                    f'{e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+                continue
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache provider: '
+                    f'{e!r}{provider_bug_report_message(provider)}',
+                )
+                continue
+        return None
+
+    def store(
+        self,
+        request: PoTokenRequest,
+        response: PoTokenResponse,
+        write_policy: CacheProviderWritePolicy | None = None,
+    ):
+        spec = self._get_cache_spec(request)
+        if not spec:
+            self.logger.trace('No cache spec available for this request. Not caching.')
+            return
+
+        if not validate_response(response):
+            self.logger.error(
+                f'Invalid PO Token response provided to PoTokenCache.store(): '
+                f'{response}{bug_reports_message()}')
+            return
+
+        cache_key = self._generate_key(self._generate_key_bindings(spec))
+        self.logger.trace(f'Attempting to access PO Token cache using key: {cache_key}')
+
+        default_expires_at = int(dt.datetime.now(dt.timezone.utc).timestamp()) + spec.default_ttl
+        cache_response = dataclasses.replace(response, expires_at=response.expires_at or default_expires_at)
+
+        write_policy = write_policy or spec.write_policy
+        self.logger.trace(f'Using write policy: {write_policy}')
+
+        for idx, provider in enumerate(self._get_cache_providers(request)):
+            try:
+                self.logger.trace(
+                    f'Caching PO Token response in "{provider.PROVIDER_NAME}" cache provider '
+                    f'(key={cache_key}, expires_at={cache_response.expires_at})')
+                provider.store(
+                    key=cache_key,
+                    value=json.dumps(dataclasses.asdict(cache_response)),
+                    expires_at=cache_response.expires_at)
+            except PoTokenCacheProviderError as e:
+                self.logger.warning(
+                    f'Error from "{provider.PROVIDER_NAME}" PO Token cache provider: '
+                    f'{e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+            except Exception as e:
+                self.logger.error(
+                    f'Error occurred with "{provider.PROVIDER_NAME}" PO Token cache provider: '
+                    f'{e!r}{provider_bug_report_message(provider)}')
+
+            # WRITE_FIRST should not write to lower priority providers in the case the highest priority provider fails
+            if idx == 0 and write_policy == CacheProviderWritePolicy.WRITE_FIRST:
+                return
+
+    def close(self):
+        for provider in self.cache_providers.values():
+            provider.close()
+        for spec_provider in self.cache_spec_providers.values():
+            spec_provider.close()
+
+
+class PoTokenRequestDirector:
+
+    def __init__(self, logger: IEContentProviderLogger, cache: PoTokenCache):
+        self.providers: dict[str, PoTokenProvider] = {}
+        self.preferences: list[Preference] = []
+        self.cache = cache
+        self.logger = logger
+
+    def register_provider(self, provider: PoTokenProvider):
+        self.providers[provider.PROVIDER_KEY] = provider
+
+    def register_preference(self, preference: Preference):
+        self.preferences.append(preference)
+
+    def _get_providers(self, request: PoTokenRequest) -> Iterable[PoTokenProvider]:
+        """Sorts available providers by preference, given a request"""
+        preferences = {
+            provider: sum(pref(provider, request) for pref in self.preferences)
+            for provider in self.providers.values()
+        }
+        if self.logger.log_level <= self.logger.LogLevel.TRACE:
+            # calling is_available() for every PO Token provider upfront may have some overhead
+            self.logger.trace(f'PO Token Providers: {provider_display_list(self.providers.values())}')
+            self.logger.trace('Provider preferences for this request: {}'.format(', '.join(
+                f'{provider.PROVIDER_NAME}={pref}' for provider, pref in preferences.items())))
+
+        return (
+            provider for provider in sorted(
+                self.providers.values(), key=preferences.get, reverse=True)
+            if provider.is_available()
+        )
+
+    def _get_po_token(self, request) -> PoTokenResponse | None:
+        for provider in self._get_providers(request):
+            try:
+                self.logger.trace(
+                    f'Attempting to fetch a PO Token from "{provider.PROVIDER_NAME}" provider')
+                response = provider.request_pot(request.copy())
+            except PoTokenProviderRejectedRequest as e:
+                self.logger.trace(
+                    f'PO Token Provider "{provider.PROVIDER_NAME}" rejected this request, '
+                    f'trying next available provider. Reason: {e}')
+                continue
+            except PoTokenProviderError as e:
+                self.logger.warning(
+                    f'Error fetching PO Token from "{provider.PROVIDER_NAME}" provider: '
+                    f'{e!r}{provider_bug_report_message(provider) if not e.expected else ""}')
+                continue
+            except Exception as e:
+                self.logger.error(
+                    f'Unexpected error when fetching PO Token from "{provider.PROVIDER_NAME}" provider: '
+                    f'{e!r}{provider_bug_report_message(provider)}')
+                continue
+
+            self.logger.trace(f'PO Token response from "{provider.PROVIDER_NAME}" provider: {response}')
+
+            if not validate_response(response):
+                self.logger.error(
+                    f'Invalid PO Token response received from "{provider.PROVIDER_NAME}" provider: '
+                    f'{response}{provider_bug_report_message(provider)}')
+                continue
+
+            return response
+
+        self.logger.trace('No PO Token providers were able to provide a valid PO Token')
+        return None
+
+    def get_po_token(self, request: PoTokenRequest) -> str | None:
+        if not request.bypass_cache:
+            if pot_response := self.cache.get(request):
+                return clean_pot(pot_response.po_token)
+
+        if not self.providers:
+            self.logger.trace('No PO Token providers registered')
+            return None
+
+        pot_response = self._get_po_token(request)
+        if not pot_response:
+            return None
+
+        pot_response.po_token = clean_pot(pot_response.po_token)
+
+        if pot_response.expires_at is None or pot_response.expires_at > 0:
+            self.cache.store(request, pot_response)
+        else:
+            self.logger.trace(
+                f'PO Token response will not be cached (expires_at={pot_response.expires_at})')
+
+        return pot_response.po_token
+
+    def close(self):
+        for provider in self.providers.values():
+            provider.close()
+        self.cache.close()
+
+
+EXTRACTOR_ARG_PREFIX = 'youtubepot'
+
+
+def initialize_pot_director(ie):
+    if not ie._downloader:
+        raise ExtractorError('Downloader not set', expected=False)
+
+    enable_trace = ie._configuration_arg(
+        'pot_trace', ['false'], ie_key='youtube', casesense=False)[0] == 'true'
+
+    if enable_trace:
+        log_level = IEContentProviderLogger.LogLevel.TRACE
+    elif ie._downloader.params.get('verbose', False):
+        log_level = IEContentProviderLogger.LogLevel.DEBUG
+    else:
+        log_level = IEContentProviderLogger.LogLevel.INFO
+
+    cache_providers = []
+    for cache_provider in _pot_cache_providers.value.values():
+        settings = traverse_obj(
+            ie._downloader.params,
+            ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{cache_provider.PROVIDER_KEY.lower()}'))
+        cache_provider_logger = YoutubeIEContentProviderLogger(
+            ie, f'pot:cache:{cache_provider.PROVIDER_NAME}', log_level=log_level)
+        cache_providers.append(cache_provider(ie, cache_provider_logger, settings or {}))
+
+    cache_spec_providers = []
+    for cache_spec_provider in _pot_pcs_providers.value.values():
+        settings = traverse_obj(
+            ie._downloader.params,
+            ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{cache_spec_provider.PROVIDER_KEY.lower()}'))
+        cache_spec_provider_logger = YoutubeIEContentProviderLogger(
+            ie, f'pot:cache:spec:{cache_spec_provider.PROVIDER_NAME}', log_level=log_level)
+        cache_spec_providers.append(cache_spec_provider(ie, cache_spec_provider_logger, settings or {}))
+
+    cache = PoTokenCache(
+        logger=YoutubeIEContentProviderLogger(ie, 'pot:cache', log_level=log_level),
+        cache_providers=cache_providers,
+        cache_spec_providers=cache_spec_providers,
+        cache_provider_preferences=list(_pot_cache_provider_preferences.value),
+    )
+
+    director = PoTokenRequestDirector(
+        logger=YoutubeIEContentProviderLogger(ie, 'pot', log_level=log_level),
+        cache=cache,
+    )
+
+    ie._downloader.add_close_hook(director.close)
+
+    for provider in _pot_providers.value.values():
+        settings = traverse_obj(
+            ie._downloader.params,
+            ('extractor_args', f'{EXTRACTOR_ARG_PREFIX}-{provider.PROVIDER_KEY.lower()}'))
+        logger = YoutubeIEContentProviderLogger(
+            ie, f'pot:{provider.PROVIDER_NAME}', log_level=log_level)
+        director.register_provider(provider(ie, logger, settings or {}))
+
+    for preference in _ptp_preferences.value:
+        director.register_preference(preference)
+
+    if director.logger.log_level <= director.logger.LogLevel.DEBUG:
+        # calling is_available() for every PO Token provider upfront may have some overhead
+        director.logger.debug(f'PO Token Providers: {provider_display_list(director.providers.values())}')
+        director.logger.debug(f'PO Token Cache Providers: {provider_display_list(cache.cache_providers.values())}')
+        director.logger.debug(f'PO Token Cache Spec Providers: {provider_display_list(cache.cache_spec_providers.values())}')
+        director.logger.trace(f'Registered {len(director.preferences)} provider preferences')
+        director.logger.trace(f'Registered {len(cache.cache_provider_preferences)} cache provider preferences')
+
+    return director
+
+
+def provider_display_list(providers: Iterable[IEContentProvider]):
+    def provider_display_name(provider):
+        display_str = join_nonempty(
+            provider.PROVIDER_NAME,
+            provider.PROVIDER_VERSION if not isinstance(provider, BuiltInIEContentProvider) else None)
+        statuses = []
+        if not isinstance(provider, BuiltInIEContentProvider):
+            statuses.append('external')
+        if not provider.is_available():
+            statuses.append('unavailable')
+        if statuses:
+            display_str += f' ({", ".join(statuses)})'
+        return display_str
+
+    return ', '.join(provider_display_name(provider) for provider in providers) or 'none'
+
+
+def clean_pot(po_token: str):
+    # Clean and validate the PO Token. This will strip invalid characters off
+    # (e.g. additional url params the user may accidentally include)
+    try:
+        return base64.urlsafe_b64encode(
+            base64.urlsafe_b64decode(urllib.parse.unquote(po_token))).decode()
+    except (binascii.Error, ValueError):
+        raise ValueError('Invalid PO Token')
+
+
+def validate_response(response: PoTokenResponse | None):
+    if (
+        not isinstance(response, PoTokenResponse)
+        or not response.po_token
+        or not isinstance(response.po_token, str)
+    ):  # noqa: SIM103
+        return False
+
+    try:
+        clean_pot(response.po_token)
+    except ValueError:
+        return False
+
+    return (
+        response.expires_at is None
+        or (
+            isinstance(response.expires_at, int)
+            and (
+                response.expires_at <= 0
+                or response.expires_at > int(dt.datetime.now(dt.timezone.utc).timestamp())
+            )
+        )
+    )
+
+
+def validate_cache_spec(spec: PoTokenCacheSpec):
+    return (
+        isinstance(spec, PoTokenCacheSpec)
+        and isinstance(spec.write_policy, CacheProviderWritePolicy)
+        and isinstance(spec.default_ttl, int)
+        and isinstance(spec.key_bindings, dict)
+        and all(isinstance(k, str) for k in spec.key_bindings)
+        and all(v is None or isinstance(v, str) for v in spec.key_bindings.values())
+        and len({k for k in spec.key_bindings.values() if k is not None}) > 0
+    )

yt_dlp/extractor/youtube/pot/_provider.py

@@ -0,0 +1,156 @@
+from __future__ import annotations
+
+import abc
+import enum
+import functools
+
+from yt_dlp.extractor.common import InfoExtractor
+from yt_dlp.utils import NO_DEFAULT, bug_reports_message, classproperty, traverse_obj
+from yt_dlp.version import __version__
+
+# xxx: these could be generalized outside YoutubeIE eventually
+
+
+class IEContentProviderLogger(abc.ABC):
+
+    class LogLevel(enum.IntEnum):
+        TRACE = 0
+        DEBUG = 10
+        INFO = 20
+        WARNING = 30
+        ERROR = 40
+
+        @classmethod
+        def _missing_(cls, value):
+            if isinstance(value, str):
+                value = value.upper()
+                if value in dir(cls):
+                    return cls[value]
+
+            return cls.INFO
+
+    log_level = LogLevel.INFO
+
+    @abc.abstractmethod
+    def trace(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def debug(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def info(self, message: str):
+        pass
+
+    @abc.abstractmethod
+    def warning(self, message: str, *, once=False):
+        pass
+
+    @abc.abstractmethod
+    def error(self, message: str):
+        pass
+
+
+class IEContentProviderError(Exception):
+    def __init__(self, msg=None, expected=False):
+        super().__init__(msg)
+        self.expected = expected
+
+
+class IEContentProvider(abc.ABC):
+    PROVIDER_VERSION: str = '0.0.0'
+    BUG_REPORT_LOCATION: str = '(developer has not provided a bug report location)'
+
+    def __init__(
+        self,
+        ie: InfoExtractor,
+        logger: IEContentProviderLogger,
+        settings: dict[str, list[str]], *_, **__,
+    ):
+        self.ie = ie
+        self.settings = settings or {}
+        self.logger = logger
+        super().__init__()
+
+    @classmethod
+    def __init_subclass__(cls, *, suffix=None, **kwargs):
+        if suffix:
+            cls._PROVIDER_KEY_SUFFIX = suffix
+        return super().__init_subclass__(**kwargs)
+
+    @classproperty
+    def PROVIDER_NAME(cls) -> str:
+        return cls.__name__[:-len(cls._PROVIDER_KEY_SUFFIX)]
+
+    @classproperty
+    def BUG_REPORT_MESSAGE(cls):
+        return f'please report this issue to the provider developer at  {cls.BUG_REPORT_LOCATION}  .'
+
+    @classproperty
+    def PROVIDER_KEY(cls) -> str:
+        assert hasattr(cls, '_PROVIDER_KEY_SUFFIX'), 'Content Provider implementation must define a suffix for the provider key'
+        assert cls.__name__.endswith(cls._PROVIDER_KEY_SUFFIX), f'PoTokenProvider class names must end with "{cls._PROVIDER_KEY_SUFFIX}"'
+        return cls.__name__[:-len(cls._PROVIDER_KEY_SUFFIX)]
+
+    @abc.abstractmethod
+    def is_available(self) -> bool:
+        """
+        Check if the provider is available (e.g. all required dependencies are available)
+        This is used to determine if the provider should be used and to provide debug information.
+
+        IMPORTANT: This method should not make any network requests or perform any expensive operations.
+         It is called multiple times.
+        """
+        raise NotImplementedError
+
+    def close(self):  # noqa: B027
+        pass
+
+    def _configuration_arg(self, key, default=NO_DEFAULT, *, casesense=False):
+        """
+        @returns            A list of values for the setting given by "key"
+                            or "default" if no such key is present
+        @param default      The default value to return when the key is not present (default: [])
+        @param casesense    When false, the values are converted to lower case
+        """
+        val = traverse_obj(self.settings, key)
+        if val is None:
+            return [] if default is NO_DEFAULT else default
+        return list(val) if casesense else [x.lower() for x in val]
+
+
+class BuiltInIEContentProvider(IEContentProvider, abc.ABC):
+    PROVIDER_VERSION = __version__
+    BUG_REPORT_MESSAGE = bug_reports_message(before='')
+
+
+def register_provider_generic(
+    provider,
+    base_class,
+    registry,
+):
+    """Generic function to register a provider class"""
+    assert issubclass(provider, base_class), f'{provider} must be a subclass of {base_class.__name__}'
+    assert provider.PROVIDER_KEY not in registry, f'{base_class.__name__} {provider.PROVIDER_KEY} already registered'
+    registry[provider.PROVIDER_KEY] = provider
+    return provider
+
+
+def register_preference_generic(
+    base_class,
+    registry,
+    *providers,
+):
+    """Generic function to register a preference for a provider"""
+    assert all(issubclass(provider, base_class) for provider in providers)
+
+    def outer(preference):
+        @functools.wraps(preference)
+        def inner(provider, *args, **kwargs):
+            if not providers or isinstance(provider, providers):
+                return preference(provider, *args, **kwargs)
+            return 0
+        registry.add(inner)
+        return preference
+    return outer

yt_dlp/extractor/youtube/pot/_registry.py

@@ -0,0 +1,8 @@
+from yt_dlp.globals import Indirect
+
+_pot_providers = Indirect({})
+_ptp_preferences = Indirect(set())
+_pot_pcs_providers = Indirect({})
+_pot_cache_providers = Indirect({})
+_pot_cache_provider_preferences = Indirect(set())
+_pot_memory_cache = Indirect({})

yt_dlp/extractor/youtube/pot/cache.py

@@ -0,0 +1,97 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import abc
+import dataclasses
+import enum
+import typing
+
+from yt_dlp.extractor.youtube.pot._provider import (
+    IEContentProvider,
+    IEContentProviderError,
+    register_preference_generic,
+    register_provider_generic,
+)
+from yt_dlp.extractor.youtube.pot._registry import (
+    _pot_cache_provider_preferences,
+    _pot_cache_providers,
+    _pot_pcs_providers,
+)
+from yt_dlp.extractor.youtube.pot.provider import PoTokenRequest
+
+
+class PoTokenCacheProviderError(IEContentProviderError):
+    """An error occurred while fetching a PO Token"""
+
+
+class PoTokenCacheProvider(IEContentProvider, abc.ABC, suffix='PCP'):
+    @abc.abstractmethod
+    def get(self, key: str) -> str | None:
+        pass
+
+    @abc.abstractmethod
+    def store(self, key: str, value: str, expires_at: int):
+        pass
+
+    @abc.abstractmethod
+    def delete(self, key: str):
+        pass
+
+
+class CacheProviderWritePolicy(enum.Enum):
+    WRITE_ALL = enum.auto()    # Write to all cache providers
+    WRITE_FIRST = enum.auto()  # Write to only the first cache provider
+
+
+@dataclasses.dataclass
+class PoTokenCacheSpec:
+    key_bindings: dict[str, str | None]
+    default_ttl: int
+    write_policy: CacheProviderWritePolicy = CacheProviderWritePolicy.WRITE_ALL
+
+    # Internal
+    _provider: PoTokenCacheSpecProvider | None = None
+
+
+class PoTokenCacheSpecProvider(IEContentProvider, abc.ABC, suffix='PCSP'):
+
+    def is_available(self) -> bool:
+        return True
+
+    @abc.abstractmethod
+    def generate_cache_spec(self, request: PoTokenRequest) -> PoTokenCacheSpec | None:
+        """Generate a cache spec for the given request"""
+        pass
+
+
+def register_provider(provider: type[PoTokenCacheProvider]):
+    """Register a PoTokenCacheProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenCacheProvider,
+        registry=_pot_cache_providers.value,
+    )
+
+
+def register_spec(provider: type[PoTokenCacheSpecProvider]):
+    """Register a PoTokenCacheSpecProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenCacheSpecProvider,
+        registry=_pot_pcs_providers.value,
+    )
+
+
+def register_preference(
+        *providers: type[PoTokenCacheProvider]) -> typing.Callable[[CacheProviderPreference], CacheProviderPreference]:
+    """Register a preference for a PoTokenCacheProvider"""
+    return register_preference_generic(
+        PoTokenCacheProvider,
+        _pot_cache_provider_preferences.value,
+        *providers,
+    )
+
+
+if typing.TYPE_CHECKING:
+    CacheProviderPreference = typing.Callable[[PoTokenCacheProvider, PoTokenRequest], int]

yt_dlp/extractor/youtube/pot/provider.py

@@ -0,0 +1,280 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import abc
+import copy
+import dataclasses
+import enum
+import functools
+import typing
+import urllib.parse
+
+from yt_dlp.cookies import YoutubeDLCookieJar
+from yt_dlp.extractor.youtube.pot._provider import (
+    IEContentProvider,
+    IEContentProviderError,
+    register_preference_generic,
+    register_provider_generic,
+)
+from yt_dlp.extractor.youtube.pot._registry import _pot_providers, _ptp_preferences
+from yt_dlp.networking import Request, Response
+from yt_dlp.utils import traverse_obj
+from yt_dlp.utils.networking import HTTPHeaderDict
+
+__all__ = [
+    'ExternalRequestFeature',
+    'PoTokenContext',
+    'PoTokenProvider',
+    'PoTokenProviderError',
+    'PoTokenProviderRejectedRequest',
+    'PoTokenRequest',
+    'PoTokenResponse',
+    'provider_bug_report_message',
+    'register_preference',
+    'register_provider',
+]
+
+
+class PoTokenContext(enum.Enum):
+    GVS = 'gvs'
+    PLAYER = 'player'
+
+
+@dataclasses.dataclass
+class PoTokenRequest:
+    # YouTube parameters
+    context: PoTokenContext
+    innertube_context: InnertubeContext
+    innertube_host: str | None = None
+    session_index: str | None = None
+    player_url: str | None = None
+    is_authenticated: bool = False
+    video_webpage: str | None = None
+    internal_client_name: str | None = None
+
+    # Content binding parameters
+    visitor_data: str | None = None
+    data_sync_id: str | None = None
+    video_id: str | None = None
+
+    # Networking parameters
+    request_cookiejar: YoutubeDLCookieJar = dataclasses.field(default_factory=YoutubeDLCookieJar)
+    request_proxy: str | None = None
+    request_headers: HTTPHeaderDict = dataclasses.field(default_factory=HTTPHeaderDict)
+    request_timeout: float | None = None
+    request_source_address: str | None = None
+    request_verify_tls: bool = True
+
+    # Generate a new token, do not used a cached token
+    # The token should still be cached for future requests
+    bypass_cache: bool = False
+
+    def copy(self):
+        return dataclasses.replace(
+            self,
+            request_headers=HTTPHeaderDict(self.request_headers),
+            innertube_context=copy.deepcopy(self.innertube_context),
+        )
+
+
+@dataclasses.dataclass
+class PoTokenResponse:
+    po_token: str
+    expires_at: int | None = None
+
+
+class PoTokenProviderRejectedRequest(IEContentProviderError):
+    """Reject the PoTokenRequest (cannot handle the request)"""
+
+
+class PoTokenProviderError(IEContentProviderError):
+    """An error occurred while fetching a PO Token"""
+
+
+class ExternalRequestFeature(enum.Enum):
+    PROXY_SCHEME_HTTP = enum.auto()
+    PROXY_SCHEME_HTTPS = enum.auto()
+    PROXY_SCHEME_SOCKS4 = enum.auto()
+    PROXY_SCHEME_SOCKS4A = enum.auto()
+    PROXY_SCHEME_SOCKS5 = enum.auto()
+    PROXY_SCHEME_SOCKS5H = enum.auto()
+    SOURCE_ADDRESS = enum.auto()
+    DISABLE_TLS_VERIFICATION = enum.auto()
+
+
+class PoTokenProvider(IEContentProvider, abc.ABC, suffix='PTP'):
+
+    # Set to None to disable the check
+    _SUPPORTED_CONTEXTS: tuple[PoTokenContext] | None = ()
+
+    # Innertube Client Name.
+    # For example, "WEB", "ANDROID", "TVHTML5".
+    # For a list of WebPO client names, see yt_dlp.extractor.youtube.pot.utils.WEBPO_CLIENTS.
+    # Also see yt_dlp.extractor.youtube._base.INNERTUBE_CLIENTS
+    #  for a list of client names currently supported by the YouTube extractor.
+    _SUPPORTED_CLIENTS: tuple[str] | None = ()
+
+    # If making external requests to websites (i.e. to youtube.com)
+    #  using another library or service (i.e., not _request_webpage),
+    #  add the request features that are supported.
+    # If only using _request_webpage to make external requests, set this to None.
+    _SUPPORTED_EXTERNAL_REQUEST_FEATURES: tuple[ExternalRequestFeature] | None = ()
+
+    def __validate_request(self, request: PoTokenRequest):
+        if not self.is_available():
+            raise PoTokenProviderRejectedRequest(f'{self.PROVIDER_NAME} is not available')
+
+        # Validate request using built-in settings
+        if (
+            self._SUPPORTED_CONTEXTS is not None
+            and request.context not in self._SUPPORTED_CONTEXTS
+        ):
+            raise PoTokenProviderRejectedRequest(
+                f'PO Token Context "{request.context}" is not supported by {self.PROVIDER_NAME}')
+
+        if self._SUPPORTED_CLIENTS is not None:
+            client_name = traverse_obj(
+                request.innertube_context, ('client', 'clientName'))
+            if client_name not in self._SUPPORTED_CLIENTS:
+                raise PoTokenProviderRejectedRequest(
+                    f'Client "{client_name}" is not supported by {self.PROVIDER_NAME}. '
+                    f'Supported clients: {", ".join(self._SUPPORTED_CLIENTS) or "none"}')
+
+        self.__validate_external_request_features(request)
+
+    @functools.cached_property
+    def _supported_proxy_schemes(self):
+        return {
+            scheme: feature
+            for scheme, feature in {
+                'http': ExternalRequestFeature.PROXY_SCHEME_HTTP,
+                'https': ExternalRequestFeature.PROXY_SCHEME_HTTPS,
+                'socks4': ExternalRequestFeature.PROXY_SCHEME_SOCKS4,
+                'socks4a': ExternalRequestFeature.PROXY_SCHEME_SOCKS4A,
+                'socks5': ExternalRequestFeature.PROXY_SCHEME_SOCKS5,
+                'socks5h': ExternalRequestFeature.PROXY_SCHEME_SOCKS5H,
+            }.items()
+            if feature in (self._SUPPORTED_EXTERNAL_REQUEST_FEATURES or [])
+        }
+
+    def __validate_external_request_features(self, request: PoTokenRequest):
+        if self._SUPPORTED_EXTERNAL_REQUEST_FEATURES is None:
+            return
+
+        if request.request_proxy:
+            scheme = urllib.parse.urlparse(request.request_proxy).scheme
+            if scheme.lower() not in self._supported_proxy_schemes:
+                raise PoTokenProviderRejectedRequest(
+                    f'External requests by "{self.PROVIDER_NAME}" provider do not '
+                    f'support proxy scheme "{scheme}". Supported proxy schemes: '
+                    f'{", ".join(self._supported_proxy_schemes) or "none"}')
+
+        if (
+            request.request_source_address
+            and ExternalRequestFeature.SOURCE_ADDRESS not in self._SUPPORTED_EXTERNAL_REQUEST_FEATURES
+        ):
+            raise PoTokenProviderRejectedRequest(
+                f'External requests by "{self.PROVIDER_NAME}" provider '
+                f'do not support setting source address')
+
+        if (
+            not request.request_verify_tls
+            and ExternalRequestFeature.DISABLE_TLS_VERIFICATION not in self._SUPPORTED_EXTERNAL_REQUEST_FEATURES
+        ):
+            raise PoTokenProviderRejectedRequest(
+                f'External requests by "{self.PROVIDER_NAME}" provider '
+                f'do not support ignoring TLS certificate failures')
+
+    def request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        self.__validate_request(request)
+        return self._real_request_pot(request)
+
+    @abc.abstractmethod
+    def _real_request_pot(self, request: PoTokenRequest) -> PoTokenResponse:
+        """To be implemented by subclasses"""
+        pass
+
+    # Helper functions
+
+    def _request_webpage(self, request: Request, pot_request: PoTokenRequest | None = None, note=None, **kwargs) -> Response:
+        """Make a request using the internal HTTP Client.
+        Use this instead of calling requests, urllib3 or other HTTP client libraries directly!
+
+        YouTube cookies will be automatically applied if this request is made to YouTube.
+
+        @param request: The request to make
+        @param pot_request: The PoTokenRequest to use. Request parameters will be merged from it.
+        @param note: Custom log message to display when making the request. Set to `False` to disable logging.
+
+        Tips:
+        - Disable proxy (e.g. if calling local service): Request(..., proxies={'all': None})
+        - Set request timeout:  Request(..., extensions={'timeout': 5.0})
+        """
+        req = request.copy()
+
+        # Merge some ctx request settings into the request
+        # Most of these will already be used by the configured ydl instance,
+        # however, the YouTube extractor may override some.
+        if pot_request is not None:
+            req.headers = HTTPHeaderDict(pot_request.request_headers, req.headers)
+            req.proxies = req.proxies or ({'all': pot_request.request_proxy} if pot_request.request_proxy else {})
+
+            if pot_request.request_cookiejar is not None:
+                req.extensions['cookiejar'] = req.extensions.get('cookiejar', pot_request.request_cookiejar)
+
+        if note is not False:
+            self.logger.info(str(note) if note else 'Requesting webpage')
+        return self.ie._downloader.urlopen(req)
+
+
+def register_provider(provider: type[PoTokenProvider]):
+    """Register a PoTokenProvider class"""
+    return register_provider_generic(
+        provider=provider,
+        base_class=PoTokenProvider,
+        registry=_pot_providers.value,
+    )
+
+
+def provider_bug_report_message(provider: IEContentProvider, before=';'):
+    msg = provider.BUG_REPORT_MESSAGE
+
+    before = before.rstrip()
+    if not before or before.endswith(('.', '!', '?')):
+        msg = msg[0].title() + msg[1:]
+
+    return (before + ' ' if before else '') + msg
+
+
+def register_preference(*providers: type[PoTokenProvider]) -> typing.Callable[[Preference], Preference]:
+    """Register a preference for a PoTokenProvider"""
+    return register_preference_generic(
+        PoTokenProvider,
+        _ptp_preferences.value,
+        *providers,
+    )
+
+
+if typing.TYPE_CHECKING:
+    Preference = typing.Callable[[PoTokenProvider, PoTokenRequest], int]
+    __all__.append('Preference')
+
+    # Barebones innertube context. There may be more fields.
+    class ClientInfo(typing.TypedDict, total=False):
+        hl: str | None
+        gl: str | None
+        remoteHost: str | None
+        deviceMake: str | None
+        deviceModel: str | None
+        visitorData: str | None
+        userAgent: str | None
+        clientName: str
+        clientVersion: str
+        osName: str | None
+        osVersion: str | None
+
+    class InnertubeContext(typing.TypedDict, total=False):
+        client: ClientInfo
+        request: dict
+        user: dict

yt_dlp/extractor/youtube/pot/utils.py

@@ -0,0 +1,73 @@
+"""PUBLIC API"""
+
+from __future__ import annotations
+
+import base64
+import contextlib
+import enum
+import re
+import urllib.parse
+
+from yt_dlp.extractor.youtube.pot.provider import PoTokenContext, PoTokenRequest
+from yt_dlp.utils import traverse_obj
+
+__all__ = ['WEBPO_CLIENTS', 'ContentBindingType', 'get_webpo_content_binding']
+
+WEBPO_CLIENTS = (
+    'WEB',
+    'MWEB',
+    'TVHTML5',
+    'WEB_EMBEDDED_PLAYER',
+    'WEB_CREATOR',
+    'WEB_REMIX',
+    'TVHTML5_SIMPLY_EMBEDDED_PLAYER',
+)
+
+
+class ContentBindingType(enum.Enum):
+    VISITOR_DATA = 'visitor_data'
+    DATASYNC_ID = 'datasync_id'
+    VIDEO_ID = 'video_id'
+    VISITOR_ID = 'visitor_id'
+
+
+def get_webpo_content_binding(
+    request: PoTokenRequest,
+    webpo_clients=WEBPO_CLIENTS,
+    bind_to_visitor_id=False,
+) -> tuple[str | None, ContentBindingType | None]:
+
+    client_name = traverse_obj(request.innertube_context, ('client', 'clientName'))
+    if not client_name or client_name not in webpo_clients:
+        return None, None
+
+    if request.context == PoTokenContext.GVS or client_name in ('WEB_REMIX', ):
+        if request.is_authenticated:
+            return request.data_sync_id, ContentBindingType.DATASYNC_ID
+        else:
+            if bind_to_visitor_id:
+                visitor_id = _extract_visitor_id(request.visitor_data)
+                if visitor_id:
+                    return visitor_id, ContentBindingType.VISITOR_ID
+            return request.visitor_data, ContentBindingType.VISITOR_DATA
+
+    elif request.context == PoTokenContext.PLAYER or client_name != 'WEB_REMIX':
+        return request.video_id, ContentBindingType.VIDEO_ID
+
+    return None, None
+
+
+def _extract_visitor_id(visitor_data):
+    if not visitor_data:
+        return None
+
+    # Attempt to extract the visitor ID from the visitor_data protobuf
+    # xxx: ideally should use a protobuf parser
+    with contextlib.suppress(Exception):
+        visitor_id = base64.urlsafe_b64decode(
+            urllib.parse.unquote_plus(visitor_data))[2:13].decode()
+        # check that visitor id is all letters and numbers
+        if re.fullmatch(r'[A-Za-z0-9_-]{11}', visitor_id):
+            return visitor_id
+
+    return None

yt_dlp/networking/_curlcffi.py

@@ -6,7 +6,8 @@ import math
 import re
 import urllib.parse
 
-from ._helper import InstanceStoreMixin, select_proxy
+from ._helper import InstanceStoreMixin
+from ..utils.networking import select_proxy
 from .common import (
     Features,
     Request,

yt_dlp/networking/_helper.py

@@ -13,7 +13,6 @@ import urllib.request
 from .exceptions import RequestError
 from ..dependencies import certifi
 from ..socks import ProxyType, sockssocket
-from ..utils import format_field, traverse_obj
 
 if typing.TYPE_CHECKING:
     from collections.abc import Iterable
@@ -82,19 +81,6 @@ def make_socks_proxy_opts(socks_proxy):
     }
 
 
-def select_proxy(url, proxies):
-    """Unified proxy selector for all backends"""
-    url_components = urllib.parse.urlparse(url)
-    if 'no' in proxies:
-        hostport = url_components.hostname + format_field(url_components.port, None, ':%s')
-        if urllib.request.proxy_bypass_environment(hostport, {'no': proxies['no']}):
-            return
-        elif urllib.request.proxy_bypass(hostport):  # check system settings
-            return
-
-    return traverse_obj(proxies, url_components.scheme or 'http', 'all')
-
-
 def get_redirect_method(method, status):
     """Unified redirect method handling"""
 

yt_dlp/networking/_requests.py

@@ -10,7 +10,7 @@ import warnings
 
 from ..dependencies import brotli, requests, urllib3
 from ..utils import bug_reports_message, int_or_none, variadic
-from ..utils.networking import normalize_url
+from ..utils.networking import normalize_url, select_proxy
 
 if requests is None:
     raise ImportError('requests module is not installed')
@@ -41,7 +41,6 @@ from ._helper import (
     create_socks_proxy_socket,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
 )
 from .common import (
     Features,

yt_dlp/networking/_urllib.py

@@ -26,7 +26,6 @@ from ._helper import (
     create_socks_proxy_socket,
     get_redirect_method,
     make_socks_proxy_opts,
-    select_proxy,
 )
 from .common import Features, RequestHandler, Response, register_rh
 from .exceptions import (
@@ -41,7 +40,7 @@ from .exceptions import (
 from ..dependencies import brotli
 from ..socks import ProxyError as SocksProxyError
 from ..utils import update_url_query
-from ..utils.networking import normalize_url
+from ..utils.networking import normalize_url, select_proxy
 
 SUPPORTED_ENCODINGS = ['gzip', 'deflate']
 CONTENT_DECODE_ERRORS = [zlib.error, OSError]

yt_dlp/networking/_websockets.py

@@ -11,8 +11,8 @@ from ._helper import (
     create_connection,
     create_socks_proxy_socket,
     make_socks_proxy_opts,
-    select_proxy,
 )
+from ..utils.networking import select_proxy
 from .common import Features, Response, register_rh
 from .exceptions import (
     CertificateVerifyError,

yt_dlp/utils/networking.py

@@ -10,7 +10,8 @@ import urllib.request
 if typing.TYPE_CHECKING:
     T = typing.TypeVar('T')
 
-from ._utils import NO_DEFAULT, remove_start
+from ._utils import NO_DEFAULT, remove_start, format_field
+from .traversal import traverse_obj
 
 
 def random_user_agent():
@@ -278,3 +279,16 @@ def normalize_url(url):
         query=escape_rfc3986(url_parsed.query),
         fragment=escape_rfc3986(url_parsed.fragment),
     ).geturl()
+
+
+def select_proxy(url, proxies):
+    """Unified proxy selector for all backends"""
+    url_components = urllib.parse.urlparse(url)
+    if 'no' in proxies:
+        hostport = url_components.hostname + format_field(url_components.port, None, ':%s')
+        if urllib.request.proxy_bypass_environment(hostport, {'no': proxies['no']}):
+            return
+        elif urllib.request.proxy_bypass(hostport):  # check system settings
+            return
+
+    return traverse_obj(proxies, url_components.scheme or 'http', 'all')