Release 2021.02.04

Syntax: -o common_template -o type:type_template
Types supported: subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.05-2. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.05-2**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version 2021.01.05-2
+ [debug] yt-dlp version 2021.01.29
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.05-2. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. youtube-dlc does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.05-2**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.05-2. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.05-2**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,16 +21,16 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.05-2. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlc
+- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlp
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.05-2**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version 2021.01.05-2
+ [debug] yt-dlp version 2021.01.29
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.05-2. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.05-2**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/6_question.md

@@ -20,8 +20,8 @@ assignees: ''
 ## Checklist
 
 <!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
+Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- Look through the README (https://github.com/blackjack4494/yt-dlc) and FAQ (https://github.com/blackjack4494/yt-dlc) for similar questions
+- Look through the README (https://github.com/pukkandan/yt-dlp) and FAQ (https://github.com/pukkandan/yt-dlp) for similar questions
 - Search the bugtracker for similar questions: https://github.com/blackjack4494/yt-dlc
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version %(version)s
+ [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. youtube-dlc does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -21,16 +21,16 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlc
+- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlp
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version %(version)s
+ [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,7 +8,7 @@
 
 ### Before submitting a *pull request* make sure you have:
 - [ ] At least skimmed through [adding new extractor tutorial](https://github.com/ytdl-org/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/ytdl-org/youtube-dl#youtube-dl-coding-conventions) sections
-- [ ] [Searched](https://github.com/pukkandan/yt-dlc/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
+- [ ] [Searched](https://github.com/pukkandan/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
 - [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
 
 ### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:

.github/workflows/build.yml

@@ -25,8 +25,8 @@ jobs:
       run: sudo apt-get -y install zip pandoc man
     - name: Bump version
       id: bump_version
-      run: python scripts/update-version-workflow.py
+      run: python devscripts/update-version.py
-    - name: Check the output from My action
+    - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run Make
       run: make
@@ -37,7 +37,7 @@ jobs:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       with:
         tag_name: ${{ steps.bump_version.outputs.ytdlc_version }}
-        release_name: youtube-dlc ${{ steps.bump_version.outputs.ytdlc_version }}
+        release_name: yt-dlp ${{ steps.bump_version.outputs.ytdlc_version }}
         body: |
           Changelog:
           PLACEHOLDER
@@ -58,18 +58,18 @@ jobs:
       env:
         SHA2: ${{ hashFiles('youtube-dlc') }}
       run: echo "::set-output name=sha2_unix::$SHA2"
-    # - name: Install dependencies for pypi
+    - name: Install dependencies for pypi
-    #   run: |
+      run: |
-    #     python -m pip install --upgrade pip
+        python -m pip install --upgrade pip
-    #     pip install setuptools wheel twine
+        pip install setuptools wheel twine
-    # - name: Build and publish
+    - name: Build and publish on pypi
-    #   env:
+      env:
-    #     TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+        TWINE_USERNAME: __token__
-    #     TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
-    #   run: |
+      run: |
-    #     rm -rf dist/*
+        rm -rf dist/*
-    #     python setup.py sdist bdist_wheel
+        python setup.py sdist bdist_wheel
-    #     twine upload dist/*
+        twine upload dist/*
 
   build_windows:
 
@@ -84,11 +84,14 @@ jobs:
       with:
           python-version: '3.8'
     - name: Install Requirements
-      run: pip install pyinstaller
+      run: pip install pyinstaller mutagen
     - name: Bump version
-      run: python scripts/update-version-workflow.py
+      id: bump_version
+      run: python devscripts/update-version.py
+    - name: Print version
+      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script
-      run: python pyinst.py
+      run: python devscripts/pyinst.py 64
     - name: Upload youtube-dlc.exe Windows binary
       id: upload-release-windows
       uses: actions/upload-release-asset@v1
@@ -119,11 +122,14 @@ jobs:
           python-version: '3.4.4'
           architecture: 'x86'
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5
+      run: pip install pyinstaller==3.5 mutagen
     - name: Bump version
-      run: python scripts/update-version-workflow.py
+      id: bump_version
+      run: python devscripts/update-version.py
+    - name: Print version
+      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script for 32 Bit
-      run: python pyinst32.py
+      run: python devscripts/pyinst.py 32
     - name: Upload Executable youtube-dlc_x86.exe
       id: upload-release-windows32
       uses: actions/upload-release-asset@v1
@@ -161,3 +167,16 @@ jobs:
         asset_path: ./SHA2-256SUMS
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
+
+#  update_version_badge:
+#    runs-on: ubuntu-latest
+#    needs: build_unix
+#    steps:
+#    - name: Create Version Badge
+#      uses: schneegans/dynamic-badges-action@v1.0.0
+#      with:
+#        auth: ${{ secrets.GIST_TOKEN }}
+#        gistID: c69cb23c3c5b3316248e52022790aa57
+#        filename: version.json
+#        label: Version
+#        message: ${{ needs.build_unix.outputs.ytdlc_version }}

.github/workflows/core.yml

@@ -1,18 +1,18 @@
-name: CI
+name: Core Tests
-on: [push]
+on: [push, pull_request]
 jobs:
   tests:
-    name: Tests
+    name: Core Tests
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: true
       matrix:
-        os: [ubuntu-latest]
+        os: [ubuntu-18.04]
         # TODO: python 2.6
-        # 3.3, 3.4 are not running
+        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
         python-impl: [cpython]
-        ytdl-test-set: [core, download]
+        ytdl-test-set: [core]
         run-tests-ext: [sh]
         include:
         # python 3.2 is only available on windows via setup-python
@@ -21,20 +21,11 @@ jobs:
           python-impl: cpython
           ytdl-test-set: core
           run-tests-ext: bat
-        - os: windows-latest
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: download
-          run-tests-ext: bat
         # jython
         - os: ubuntu-latest
           python-impl: jython
           ytdl-test-set: core
           run-tests-ext: sh
-        - os: ubuntu-latest
-          python-impl: jython
-          ytdl-test-set: download
-          run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
@@ -60,16 +51,4 @@ jobs:
       env:
         YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
-  flake8:
+  # Linter is in quick-test
-    name: Linter
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: 3.9
-    - name: Install flake8
-      run: pip install flake8
-    - name: Run flake8
-      run: flake8 .

.github/workflows/download.yml

@@ -0,0 +1,53 @@
+name: Download Tests
+on: [push, pull_request]
+jobs:
+  tests:
+    name: Download Tests
+    if: "!contains(github.event.head_commit.message, 'ci skip dl') && !contains(github.event.head_commit.message, 'ci skip all')"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: true
+      matrix:
+        os: [ubuntu-18.04]
+        # TODO: python 2.6
+        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
+        python-impl: [cpython]
+        ytdl-test-set: [download]
+        run-tests-ext: [sh]
+        include:
+        # python 3.2 is only available on windows via setup-python
+        - os: windows-latest
+          python-version: 3.2
+          python-impl: cpython
+          ytdl-test-set: download
+          run-tests-ext: bat
+        # jython - disable for now since it takes too long to complete
+        # - os: ubuntu-latest
+        #   python-impl: jython
+        #   ytdl-test-set: download
+        #   run-tests-ext: sh
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      if: ${{ matrix.python-impl == 'cpython' }}
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Set up Java 8
+      if: ${{ matrix.python-impl == 'jython' }}
+      uses: actions/setup-java@v1
+      with:
+        java-version: 8
+    - name: Install Jython
+      if: ${{ matrix.python-impl == 'jython' }}
+      run: |
+        wget http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
+        java -jar jython-installer.jar -s -d "$HOME/jython"
+        echo "$HOME/jython/bin" >> $GITHUB_PATH
+    - name: Install nose
+      run: pip install nose
+    - name: Run tests
+      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      env:
+        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

.github/workflows/python-publish.yml.disable

@@ -1,33 +0,0 @@
-# This workflows will upload a Python Package using Twine when a release is created
-# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
-
-name: Upload Python Package
-
-on:
-  push:
-    branches:
-      - release
-
-jobs:
-  deploy:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: '3.x'
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install setuptools wheel twine
-    - name: Build and publish
-      env:
-        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-      run: |
-        rm -rf dist/*
-        python setup.py sdist bdist_wheel
-        twine upload dist/*

.github/workflows/quick-test.yml

@@ -0,0 +1,33 @@
+name: Quick Test
+on: [push, pull_request]
+jobs:
+  tests:
+    name: Core Test
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.9
+    - name: Install nose
+      run: pip install nose
+    - name: Run tests
+      env:
+        YTDL_TEST_SET: core
+      run: ./devscripts/run_tests.sh
+  flake8:
+    name: Linter
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.9
+    - name: Install flake8
+      run: pip install flake8
+    - name: Run flake8
+      run: flake8 .

.gitignore

@@ -1,35 +1,43 @@
+# Python
 *.pyc
 *.pyo
-*.class
-*~
-*.DS_Store
 wine-py2exe/
 py2exe.log
-*.kate-swp
 build/
 dist/
 zip/
+tmp/
+venv/
+
+# Misc
+*~
+*.DS_Store
+*.kate-swp
 MANIFEST
-README.txt
+test/local_parameters.json
-youtube-dl.1
-youtube-dlc.1
-youtube-dl.bash-completion
-youtube-dlc.bash-completion
-youtube-dl.fish
-youtube-dlc.fish
-youtube_dl/extractor/lazy_extractors.py
-youtube_dlc/extractor/lazy_extractors.py
-youtube-dl
-youtube-dlc
-youtube-dl.exe
-youtube-dlc.exe
-youtube-dl.tar.gz
-youtube-dlc.tar.gz
-youtube-dlc.spec
 .coverage
 cover/
 updates_key.pem
 *.egg-info
+.tox
+*.class
+
+# Generated
+README.txt
+*.1
+*.bash-completion
+*.fish
+*.exe
+*.tar.gz
+*.zsh
+*.spec
+
+# Binary
+youtube-dl
+youtube-dlc
+*.exe
+
+# Downloaded
 *.srt
 *.ttml
 *.sbv
@@ -47,22 +55,32 @@ updates_key.pem
 *.part
 *.ytdl
 *.swp
+*.ogg
+*.opus
+*.info.json
+*.annotations.xml
+*.description
+
+# Config
+*.conf
 *.spec
-test/local_parameters.json
+cookies
-.tox
-youtube-dl.zsh
-youtube-dlc.zsh
-
-# IntelliJ related files
-.idea
-*.iml
-
-tmp/
-venv/
-
-# VS Code related files
-.vscode
-
 cookies.txt
 
-*.sublime-workspace
+
+
+# Text Editor / IDE
+.idea
+*.iml
+.vscode
+*.sublime-workspace
+*.sublime-project
+!yt-dlp.sublime-project
+
+# Lazy extractors
+*/extractor/lazy_extractors.py
+
+# Plugins
+ytdlp_plugins/extractor/*
+!ytdlp_plugins/extractor/__init__.py
+!ytdlp_plugins/extractor/sample.py

AUTHORS

@@ -1,248 +0,0 @@
-Ricardo Garcia Gonzalez
-Danny Colligan
-Benjamin Johnson
-Vasyl' Vavrychuk
-Witold Baryluk
-Paweł Paprota
-Gergely Imreh
-Rogério Brito
-Philipp Hagemeister
-Sören Schulze
-Kevin Ngo
-Ori Avtalion
-shizeeg
-Filippo Valsorda
-Christian Albrecht
-Dave Vasilevsky
-Jaime Marquínez Ferrándiz
-Jeff Crouse
-Osama Khalid
-Michael Walter
-M. Yasoob Ullah Khalid
-Julien Fraichard
-Johny Mo Swag
-Axel Noack
-Albert Kim
-Pierre Rudloff
-Huarong Huo
-Ismael Mejía
-Steffan Donal
-Andras Elso
-Jelle van der Waa
-Marcin Cieślak
-Anton Larionov
-Takuya Tsuchida
-Sergey M.
-Michael Orlitzky
-Chris Gahan
-Saimadhav Heblikar
-Mike Col
-Oleg Prutz
-pulpe
-Andreas Schmitz
-Michael Kaiser
-Niklas Laxström
-David Triendl
-Anthony Weems
-David Wagner
-Juan C. Olivares
-Mattias Harrysson
-phaer
-Sainyam Kapoor
-Nicolas Évrard
-Jason Normore
-Hoje Lee
-Adam Thalhammer
-Georg Jähnig
-Ralf Haring
-Koki Takahashi
-Ariset Llerena
-Adam Malcontenti-Wilson
-Tobias Bell
-Naglis Jonaitis
-Charles Chen
-Hassaan Ali
-Dobrosław Żybort
-David Fabijan
-Sebastian Haas
-Alexander Kirk
-Erik Johnson
-Keith Beckman
-Ole Ernst
-Aaron McDaniel (mcd1992)
-Magnus Kolstad
-Hari Padmanaban
-Carlos Ramos
-5moufl
-lenaten
-Dennis Scheiba
-Damon Timm
-winwon
-Xavier Beynon
-Gabriel Schubiner
-xantares
-Jan Matějka
-Mauroy Sébastien
-William Sewell
-Dao Hoang Son
-Oskar Jauch
-Matthew Rayfield
-t0mm0
-Tithen-Firion
-Zack Fernandes
-cryptonaut
-Adrian Kretz
-Mathias Rav
-Petr Kutalek
-Will Glynn
-Max Reimann
-Cédric Luthi
-Thijs Vermeir
-Joel Leclerc
-Christopher Krooss
-Ondřej Caletka
-Dinesh S
-Johan K. Jensen
-Yen Chi Hsuan
-Enam Mijbah Noor
-David Luhmer
-Shaya Goldberg
-Paul Hartmann
-Frans de Jonge
-Robin de Rooij
-Ryan Schmidt
-Leslie P. Polzer
-Duncan Keall
-Alexander Mamay
-Devin J. Pohly
-Eduardo Ferro Aldama
-Jeff Buchbinder
-Amish Bhadeshia
-Joram Schrijver
-Will W.
-Mohammad Teimori Pabandi
-Roman Le Négrate
-Matthias Küch
-Julian Richen
-Ping O.
-Mister Hat
-Peter Ding
-jackyzy823
-George Brighton
-Remita Amine
-Aurélio A. Heckert
-Bernhard Minks
-sceext
-Zach Bruggeman
-Tjark Saul
-slangangular
-Behrouz Abbasi
-ngld
-nyuszika7h
-Shaun Walbridge
-Lee Jenkins
-Anssi Hannula
-Lukáš Lalinský
-Qijiang Fan
-Rémy Léone
-Marco Ferragina
-reiv
-Muratcan Simsek
-Evan Lu
-flatgreen
-Brian Foley
-Vignesh Venkat
-Tom Gijselinck
-Founder Fang
-Andrew Alexeyew
-Saso Bezlaj
-Erwin de Haan
-Jens Wille
-Robin Houtevelts
-Patrick Griffis
-Aidan Rowe
-mutantmonkey
-Ben Congdon
-Kacper Michajłow
-José Joaquín Atria
-Viťas Strádal
-Kagami Hiiragi
-Philip Huppert
-blahgeek
-Kevin Deldycke
-inondle
-Tomáš Čech
-Déstin Reed
-Roman Tsiupa
-Artur Krysiak
-Jakub Adam Wieczorek
-Aleksandar Topuzović
-Nehal Patel
-Rob van Bekkum
-Petr Zvoníček
-Pratyush Singh
-Aleksander Nitecki
-Sebastian Blunt
-Matěj Cepl
-Xie Yanbo
-Philip Xu
-John Hawkinson
-Rich Leeper
-Zhong Jianxin
-Thor77
-Mattias Wadman
-Arjan Verwer
-Costy Petrisor
-Logan B
-Alex Seiler
-Vijay Singh
-Paul Hartmann
-Stephen Chen
-Fabian Stahl
-Bagira
-Odd Stråbø
-Philip Herzog
-Thomas Christlieb
-Marek Rusinowski
-Tobias Gruetzmacher
-Olivier Bilodeau
-Lars Vierbergen
-Juanjo Benages
-Xiao Di Guan
-Thomas Winant
-Daniel Twardowski
-Jeremie Jarosh
-Gerard Rovira
-Marvin Ewald
-Frédéric Bournival
-Timendum
-gritstub
-Adam Voss
-Mike Fährmann
-Jan Kundrát
-Giuseppe Fabiano
-Örn Guðjónsson
-Parmjit Virk
-Genki Sky
-Ľuboš Katrinec
-Corey Nicholson
-Ashutosh Chaudhary
-John Dong
-Tatsuyuki Ishi
-Daniel Weber
-Kay Bouché
-Yang Hongbo
-Lei Wang
-Petr Novák
-Leonardo Taccari
-Martin Weinelt
-Surya Oktafendri
-TingPing
-Alexandre Macabies
-Bastian de Groot
-Niklas Haas
-András Veres-Szentkirályi
-Enes Solak
-Nathan Rossi
-Thomas van der Berg
-Luca Cherubin

AUTHORS-Fork

@@ -1,3 +0,0 @@
-pukkandan
-h-h-h-h
-pauldubois98

CONTRIBUTORS

@@ -0,0 +1,19 @@
+pukkandan (owner)
+h-h-h-h
+pauldubois98
+nixxo
+GreyAlien502
+kyuyeunk
+siikamiika
+jbruchon
+alexmerkel
+glenn-slayden
+Unrud
+wporr
+mariuszskon
+ohnonot
+samiksome
+alxnull
+FelixFrog
+Zocker1999NET
+nao20010128nao

Changelog.md

@@ -0,0 +1,231 @@
+# Changelog
+
+<!--
+# Instuctions for creating release
+
+* Run `make doc`
+* Update Changelog.md and CONTRIBUTORS
+* Change "Merged with youtube-dl" version in Readme.md if needed
+* Commit to master as `Release <version>`
+* Push to origin/release - build task will now run
+* Update version.py using devscripts\update-version.py (be wary of timezones)
+* Run `make issuetemplates`
+* Commit to master as `[version] update :ci skip all`
+* Push to origin/master
+* Update changelog in /releases
+
+-->
+
+
+### 2021.02.04
+* **Merge youtube-dl:** Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
+* **Date/time formatting in output template:** You can now use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
+* **Multiple output templates:**
+    * Seperate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
+    * The alowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
+* [youtube] More metadata extraction for channel/playlist URLs (channel, uploader, thumbnail, tags)
+* New option `--no-write-playlist-metafiles` to prevent writing playlist metadata files
+* [audius] Fix extractor
+* [youtube_live_chat] Fix `parse_yt_initial_data` and add `fragment_retries`
+* [postprocessor] Raise errors correctly
+* [metadatafromtitle] Fix bug when extracting data from numeric fields
+* Fix issue with overwriting files
+* Fix "Default format spec" appearing in quiet mode
+* [FormatSort] Allow user to prefer av01 over vp9 (The default is still vp9)
+* [FormatSort] fix bug where `quality` had more priority than `hasvid`
+* [pyinst] Automatically detect python architecture and working directory
+* Strip out internal fields such as `_filename` from infojson
+
+
+### 2021.01.29
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by @animelover1984 and @bbepis
+    * Add `--get-comments`
+    * [youtube] Extract comments
+    * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
+    * [billibilli] Extract comments
+    * [billibilli] Better video extraction
+    * Write playlist data to infojson
+    * [FFmpegMetadata] Embed infojson inside the video
+    * [EmbedThumbnail] Try embedding in mp4 using ffprobe and `-disposition`
+    * [EmbedThumbnail] Treat mka like mkv and mov like mp4
+    * [EmbedThumbnail] Embed in ogg/opus
+    * [VideoRemuxer] Conditionally remux video
+    * [VideoRemuxer] Add `-movflags +faststart` when remuxing to mp4
+    * [ffmpeg] Print entire stderr in verbose when there is error
+    * [EmbedSubtitle] Warn when embedding ass in mp4
+    * [anvato] Use NFLTokenGenerator if possible
+* **Parse additional metadata**: New option `--parse-metadata` to extract additional metadata from existing fields
+    * The extracted fields can be used in `--output`
+    * Deprecated `--metadata-from-title`
+* [Audius] Add extractor
+* [youtube] Extract playlist description and write it to `.description` file
+* Detect existing files even when using `recode`/`remux` (`extract-audio` is partially fixed)
+* Fix wrong user config from v2021.01.24
+* [youtube] Report error message from youtube as error instead of warning
+* [FormatSort] Fix some fields not sorting from v2021.01.24
+* [postprocessor] Deprecate `avconv`/`avprobe`.  All current functionality is left untouched. But don't expect any new features to work with avconv
+* [postprocessor] fix `write_debug` to not throw error when there is no `_downloader`
+* [movefiles] Don't give "cant find" warning when move is unnecessary
+* Refactor `update-version`, `pyinst.py` and related files
+* [ffmpeg] Document more formats that are supported for remux/recode
+
+
+### 2021.01.24
+* **Merge youtube-dl:** Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* Plugin support ([documentation](https://github.com/pukkandan/yt-dlp#plugins))
+* **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
+    * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
+    * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
+    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
+* Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
+* [mildom] Add extractor by @nao20010128nao
+* Warn when using old style `--external-downloader-args` and `--post-processor-args`
+* Fix `--no-overwrite` when using `--write-link`
+* [sponskrub] Output `unrecognized argument` error message correctly
+* [cbs] Make failure to extract title non-fatal
+* Fix typecasting when pre-checking archive
+* Fix issue with setting title on UNIX
+* Deprecate redundant aliases in `formatSort`. The aliases remain functional for backward compatibility, but will be left undocumented
+* [tests] Fix test_post_hooks
+* [tests] Split core and download tests
+
+
+### 2021.01.20
+* [TrovoLive] Add extractor (only VODs)
+* [pokemon] Add `/#/player` URLs
+* Improved parsing of multiple postprocessor-args, add `--ppa` as alias
+* [EmbedThumbnail] Simplify embedding in mkv
+* [sponskrub] Encode filenames correctly, better debug output and error message
+* [readme] Cleanup options
+
+
+### 2021.01.16
+* **Merge youtube-dl:** Upto [2021.01.16](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* **Configuration files:**
+    * Portable configuration file: `./yt-dlp.conf`
+    * Allow the configuration files to be named `yt-dlp` instead of `youtube-dlc`. See [this](https://github.com/pukkandan/yt-dlp#configuration) for details
+* Add PyPI release
+
+
+### 2021.01.14
+* Added option `--break-on-reject`
+* [roosterteeth.com] Fix for bonus episodes by @Zocker1999NET
+* [tiktok] Fix for when share_info is empty
+* [EmbedThumbnail] Fix bug due to incorrect function name
+* [documentation] Changed sponskrub links to point to [pukkandan/sponskrub](https://github.com/pukkandan/SponSkrub) since I am now providing both linux and windows releases
+* [documentation] Change all links to correctly point to new fork URL
+* [documentation] Fixes typos
+
+
+### 2021.01.12
+* [roosterteeth.com] Add subtitle support by @samiksome
+* Added `--force-overwrites`, `--no-force-overwrites` by @alxnull
+* Changed fork name to `yt-dlp`
+* Fix typos by @FelixFrog
+* [ci] Option to skip
+* [changelog] Added unreleased changes in blackjack4494/yt-dlc
+
+
+### 2021.01.10
+* [archive.org] Fix extractor and add support for audio and playlists by @wporr
+* [Animelab] Added by @mariuszskon
+* [youtube:search] Fix view_count by @ohnonot
+* [youtube] Show if video is embeddable in info
+* Update version badge automatically in README
+* Enable `test_youtube_search_matching`
+* Create `to_screen` and similar functions in postprocessor/common
+
+
+### 2021.01.09
+* [youtube] Fix bug in automatic caption extraction
+* Add `post_hooks` to YoutubeDL by @alexmerkel
+* Batch file enumeration improvements by @glenn-slayden
+* Stop immediately when reaching `--max-downloads` by @glenn-slayden
+* Fix incorrect ANSI sequence for restoring console-window title by @glenn-slayden
+* Kill child processes when yt-dlc is killed by @Unrud
+
+
+### 2021.01.08
+* **Merge youtube-dl:** Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.08)
+    * Extractor stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f)) have not been merged
+* Moved changelog to seperate file
+
+
+### 2021.01.07-1
+* [Akamai] fix by @nixxo
+* [Tiktok] merge youtube-dl tiktok extractor by @GreyAlien502
+* [vlive] add support for playlists by @kyuyeunk
+* [youtube_live_chat] make sure playerOffsetMs is positive by @siikamiika
+* Ignore extra data streams in ffmpeg by @jbruchon
+* Allow passing different arguments to different postprocessors using `--postprocessor-args`
+* Deprecated `--sponskrub-args`. The same can now be done using `--postprocessor-args "sponskrub:<args>"`
+* [CI] Split tests into core-test and full-test
+
+
+### 2021.01.07
+* Removed priority of `av01` codec in `-S` since most devices don't support it yet
+* Added `duration_string` to be used in `--output`
+* Created First Release
+
+
+### 2021.01.05-1
+* **Changed defaults:**
+    * Enabled `--ignore`
+    * Disabled `--video-multistreams` and `--audio-multistreams`
+    * Changed default format selection to `bv*+ba/b` when `--audio-multistreams` is disabled
+    * Changed default format sort order to `res,fps,codec,size,br,asr,proto,ext,has_audio,source,format_id`
+    * Changed `webm` to be more preferable than `flv` in format sorting
+    * Changed default output template to `%(title)s [%(id)s].%(ext)s`
+    * Enabled `--list-formats-as-table`
+
+
+### 2021.01.05
+* **Format Sort:** Added `--format-sort` (`-S`), `--format-sort-force` (`--S-force`) - See [Sorting Formats](README.md#sorting-formats) for details
+* **Format Selection:** See [Format Selection](README.md#format-selection) for details
+    * New format selectors: `best*`, `worst*`, `bestvideo*`, `bestaudio*`, `worstvideo*`, `worstaudio*`
+    * Changed video format sorting to show video only files and video+audio files together.
+    * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
+    * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
+* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
+* **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
+* Added `--force-download-archive` (`--force-write-archive`) by @h-h-h-h
+* Added `--list-formats-as-table`,  `--list-formats-old`
+* **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
+    * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
+    * Renamed: `--write-subs`, `--no-write-subs`, `--no-write-auto-subs`, `--write-auto-subs`. Note that these can still be used without the ending "s"
+* Relaxed validation for format filters so that any arbitrary field can be used
+* Fix for embedding thumbnail in mp3 by @pauldubois98 ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
+* Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
+* **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
+    * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
+* Cleaned up the fork for public use
+
+
+### Unreleased changes in [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc)
+* Updated to youtube-dl release 2020.11.26
+* [youtube]
+    * Implemented all Youtube Feeds (ytfav, ytwatchlater, ytsubs, ythistory, ytrec) and SearchURL
+    * Fix ytsearch not returning results sometimes due to promoted content
+    * Temporary fix for automatic captions - disable json3
+    * Fix some improper Youtube URLs
+    * Redirect channel home to /video
+    * Print youtube's warning message
+    * Multiple pages are handled better for feeds
+* Add --break-on-existing by @gergesh
+* Pre-check video IDs in the archive before downloading
+* [bitwave.tv] New extractor
+* [Gedi] Add extractor
+* [Rcs] Add new extractor
+* [skyit] Add support for multiple Sky Italia website and removed old skyitalia extractor
+* [france.tv] Fix thumbnail URL
+* [ina] support mobile links
+* [instagram] Fix extractor
+* [itv] BTCC new pages' URL update (articles instead of races)
+* [SouthparkDe] Support for English URLs
+* [spreaker] fix SpreakerShowIE test URL
+* [Vlive] Fix playlist handling when downloading a channel
+* [generic] Detect embedded bitchute videos
+* [generic] Extract embedded youtube and twitter videos
+* [ffmpeg] Ensure all streams are copied
+* Fix for os.rename error when embedding thumbnail to video in a different drive
+* make_win.bat: don't use UPX to pack vcruntime140.dll

Makefile

@@ -1,8 +1,10 @@
-all: youtube-dlc README.md CONTRIBUTING.md README.txt issuetemplates youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtube-dlc.fish supportedsites
+all: youtube-dlc doc man
-doc: README.md CONTRIBUTING.md issuetemplates supportedsites clean
+doc: README.md CONTRIBUTING.md issuetemplates supportedsites
+man: README.txt youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtube-dlc.fish
+
 
 clean:
-	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
+	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
 
@@ -10,7 +12,8 @@ PREFIX ?= /usr/local
 BINDIR ?= $(PREFIX)/bin
 MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share
-PYTHON ?= /usr/bin/env python
+# make_supportedsites.py doesnot work correctly in python2
+PYTHON ?= /usr/bin/env python3
 
 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
 SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
@@ -46,11 +49,13 @@ offlinetest: codetest
 		--exclude test_age_restriction.py \
 		--exclude test_download.py \
 		--exclude test_iqiyi_sdk_interpreter.py \
+		--exclude test_overwrites.py \
 		--exclude test_socks.py \
 		--exclude test_subtitles.py \
 		--exclude test_write_annotations.py \
 		--exclude test_youtube_lists.py \
-		--exclude test_youtube_signature.py
+		--exclude test_youtube_signature.py \
+		--exclude test_post_hooks.py
 
 tar: youtube-dlc.tar.gz
 

README.md

@@ -1,16 +1,21 @@
-[![Build Status](https://github.com/pukkandan/yt-dlc/workflows/CI/badge.svg)](https://github.com/pukkandan/yt-dlc/actions?query=workflow%3ACI)
+# YT-DLP
-[![Release Version](https://img.shields.io/badge/Release-2021.01.07-brightgreen)](https://github.com/pukkandan/yt-dlc/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](https://github.com/pukkandan/yt-dlc/blob/master/LICENSE)
 
-youtube-dlc - download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
+[![Release version](https://img.shields.io/github/v/release/pukkandan/yt-dlp?color=brightgreen&label=Release)](https://github.com/pukkandan/yt-dlp/releases/latest)
+[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
+[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)  
+[![Commits](https://img.shields.io/github/commit-activity/m/pukkandan/yt-dlp?label=commits)](https://github.com/pukkandan/yt-dlp/commits)
+[![Last Commit](https://img.shields.io/github/last-commit/pukkandan/yt-dlp/master)](https://github.com/pukkandan/yt-dlp/commits)
+[![Downloads](https://img.shields.io/github/downloads/pukkandan/yt-dlp/total)](https://github.com/pukkandan/yt-dlp/releases/latest)
+[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp)
+
+A command-line program to download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
 
 This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which is inturn a fork of [youtube-dl](https://github.com/ytdl-org/youtube-dl)
 
-* [CHANGES FROM YOUTUBE-DLC](#changes)
+* [NEW FEATURES](#new-features)
 * [INSTALLATION](#installation)
-    * [UPDATE](#update)
+    * [Update](#update)
-    * [COMPILE](#compile)
+    * [Compile](#compile)
-* [YOUTUBE-DLC](#youtube-dlc)
 * [DESCRIPTION](#description)
 * [OPTIONS](#options)
     * [Network Options](#network-options)
@@ -27,7 +32,7 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
     * [Authentication Options](#authentication-options)
     * [Adobe Pass Options](#adobe-pass-options)
     * [Post-processing Options](#post-processing-options)
-    * [SponSkrub Options (SponsorBlock)](#sponskrub-options-sponsorblock)
+    * [SponSkrub (SponsorBlock) Options](#sponskrub-sponsorblock-options)
     * [Extractor Options](#extractor-options)
 * [CONFIGURATION](#configuration)
     * [Authentication with .netrc file](#authentication-with-netrc-file)
@@ -38,73 +43,74 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
     * [Filtering Formats](#filtering-formats)
     * [Sorting Formats](#sorting-formats)
     * [Format Selection examples](#format-selection-examples)
-* [VIDEO SELECTION](#video-selection-1)
+* [PLUGINS](#plugins)
 * [MORE](#more)
 
 
-# CHANGES
+# NEW FEATURES
-See [commits](https://github.com/pukkandan/yt-dlc/commits) for more details
+The major new features from the latest release of [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc) are:
 
-### 2021.01.05
+* **[SponSkrub Integration](#sponskrub-sponsorblock-options)**: You can use [SponSkrub](https://github.com/pukkandan/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
-* **Format Sort:** Added `--format-sort` (`-S`), `--format-sort-force` (`--S-force`) - See [Sorting Formats](#sorting-formats) for details
-* **Format Selection:** See [Format Selection](#format-selection) for details
-    * New format selectors: `best*`, `worst*`, `bestvideo*`, `bestaudio*`, `worstvideo*`, `worstaudio*`
-    * Changed video format sorting to show video only files and video+audio files together.
-    * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
-    * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options](#internet-shortcut-options) for details
-* **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](#sponskrub-options-sponsorblock) for details
-* Added `--force-download-archive` (`--force-write-archive`) by by h-h-h-h
-* Added `--list-formats-as-table`,  `--list-formats-old`
-* **Negative Options:** Makes it possible to negate boolean options by adding a `no-` to the switch
-    * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
-    * Renamed: `--write-subs`, --no-write-subs`, `--no-write-auto-subs, `--write-auto-subs`. Note that these can still be used without the ending "s"
-* Relaxed validation for format filters so that any arbitrary field can be used
-* Fix for embedding thumbnail in mp3 by @pauldubois98
-* Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
-* **Merge youtube-dl:** Upto [2020.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
-* Cleaned up the fork for public use
 
-### 2021.01.05-2
+* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
-* **Changed defaults:**
+
-    * Enabled `--ignore`
+* **Merged with youtube-dl v2021.02.04.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
-    * Disabled `--video-multistreams` and `--audio-multistreams`
+
-    * Changed default format selection to `bv*+ba/b` when `--audio-multistreams` is disabled
+* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/pukkandan/yt-dlp/pull/31) for details.
-    * Changed default format sort order to `res,fps,codec,size,br,asr,proto,ext,has_audio,source,format_id`
+
-    * Changed `webm` to be more preferable than `flv` in format sorting
+* **Youtube improvements**:
-    * Changed default output template to `%(title)s [%(id)s].%(ext)s`
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
-    * Enabled `--list-formats-as-table`
+    * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
+    * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
+
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius
+
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
+
+* **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
+
+* **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [`--paths`](https://github.com/pukkandan/yt-dlp/#:~:text=-P,%20--paths%20TYPE:PATH) for details
+
+<!-- Relative link doesn't work for "#:~:text=" -->
+
+* **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details
+
+* **Other new options**: `--parse-metadata`, `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+
+* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc
+
+
+See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes
+
+
+**PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](Changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details
+
+If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](docs/supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated.
 
-### 2021.01.07
-* Removed priority of `av01` codec in `-S` since most devices don't support it yet
-* Added `duration_string` to be used in `--output`
-* Created First Release
 
 # INSTALLATION
 
-To use the latest version, simply download and run the [latest release](https://github.com/pukkandan/yt-dlc/releases/latest).
+You can install yt-dlp using one of the following methods:
-Currently, there is no support for any package managers.
+* Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
-
+* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest)
-If you want to install the current master branch
+* Use pip+git: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp.git@release`
-
+* Install master branch: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp`
-    python -m pip install git+https://github.com/pukkandan/yt-dlc
 
 ### UPDATE
-**DO NOT UPDATE using `-U` !** instead download binaries again
+`-U` does not work. Simply repeat the install process to update.
 
 ### COMPILE
 
 **For Windows**:
-To build the Windows executable yourself (without version info!)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen for embedding thumbnail in opus/ogg files)
+
+    python -m pip install --upgrade pyinstaller mutagen
+
+Once you have all the necessary dependancies installed, just run `py devscripts\pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
+
+You can also build the executable without any version info or metadata by using:
 
-    python -m pip install --upgrade pyinstaller
     pyinstaller.exe youtube_dlc\__main__.py --onefile --name youtube-dlc
-    
-Or simply execute the `make_win.bat` if pyinstaller is installed.
-There will be a `youtube-dlc.exe` in `/dist`  
-
-New way to build Windows is to use `python pyinst.py` (please use python3 64Bit)  
-For 32Bit Version use a 32Bit Version of python (3 preferred here as well) and run `python pyinst32.py`  
 
 **For Unix**:
 You will need the required build tools  
@@ -113,11 +119,12 @@ Then simply type this
 
     make
 
+**Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number
 
 # DESCRIPTION
-**youtube-dlc** is a command-line program to download videos from YouTube.com and a few more sites. It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
+**youtube-dlc** is a command-line program to download videos from youtube.com many other [video platforms](docs/supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
 
-    youtube-dlc [OPTIONS] URL [URL...]
+    youtube-dlc [OPTIONS] [--] URL [URL...]
 
 
 # OPTIONS
@@ -132,9 +139,9 @@ Then simply type this
                                      permissions (run with sudo if needed)
     -i, --ignore-errors              Continue on download errors, for example to
                                      skip unavailable videos in a playlist
-                                     (default) (Same as --no-abort-on-error)
+                                     (default) (Alias: --no-abort-on-error)
-    --abort-on-error                 Abort downloading of further videos if an 
+    --abort-on-error                 Abort downloading of further videos if an
-                                     error occurs (Same as --no-ignore-errors)
+                                     error occurs (Alias: --no-ignore-errors)
     --dump-user-agent                Display the current browser identification
     --list-extractors                List all supported extractors
     --extractor-descriptions         Output descriptions of all supported
@@ -149,23 +156,25 @@ Then simply type this
                                      warning when guessing). "error" just throws
                                      an error. The default value "fixup_error"
                                      repairs broken URLs, but emits an error if
-                                     this is not possible instead of searching.
+                                     this is not possible instead of searching
-    --ignore-config, --no-config     Do not read configuration files. When given
+    --ignore-config, --no-config     Disable loading any configuration files
-                                     in the global configuration file
+                                     except the one provided by --config-location.
-                                     /etc/youtube-dl.conf: Do not read the user
+                                     When given inside a configuration
-                                     configuration in ~/.config/youtube-
+                                     file, no further configuration files are
-                                     dl/config (%APPDATA%/youtube-dl/config.txt
+                                     loaded. Additionally, (for backward
-                                     on Windows)
+                                     compatibility) if this option is found
-    --config-location PATH           Location of the configuration file; either
+                                     inside the system configuration file, the
-                                     the path to the config or its containing
+                                     user configuration is not loaded
-                                     directory.
+    --config-location PATH           Location of the main configuration file;
+                                     either the path to the config or its
+                                     containing directory
     --flat-playlist                  Do not extract the videos of a playlist,
-                                     only list them.
+                                     only list them
     --flat-videos                    Do not resolve the video urls
     --no-flat-playlist               Extract the videos of a playlist
     --mark-watched                   Mark videos watched (YouTube only)
     --no-mark-watched                Do not mark videos watched
-    --no-color                       Do not emit color codes in output
+    --no-colors                      Do not emit color codes in output
 
 ## Network Options:
     --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
@@ -183,7 +192,7 @@ Then simply type this
                                      some geo-restricted sites. The default
                                      proxy specified by --proxy (or none, if the
                                      option is not present) is used for the
-                                     actual downloading.
+                                     actual downloading
     --geo-bypass                     Bypass geographic restriction via faking
                                      X-Forwarded-For HTTP header
     --no-geo-bypass                  Do not bypass geographic restriction via
@@ -205,7 +214,7 @@ Then simply type this
                                      indexed 1, 2, 5, 8 in the playlist. You can
                                      specify range: "--playlist-items
                                      1-3,7,10-13", it will download the videos
-                                     at index 1, 2, 3, 7, 10, 11, 12 and 13.
+                                     at index 1, 2, 3, 7, 10, 11, 12 and 13
     --match-title REGEX              Download only matching titles (regex or
                                      caseless sub-string)
     --reject-title REGEX             Skip download for matching titles (regex or
@@ -229,36 +238,38 @@ Then simply type this
     --max-views COUNT                Do not download any videos with more than
                                      COUNT views
     --match-filter FILTER            Generic video filter. Specify any key (see
-                                     the "OUTPUT TEMPLATE" for a list of
+                                     "OUTPUT TEMPLATE" for a list of available
-                                     available keys) to match if the key is
+                                     keys) to match if the key is present, !key
-                                     present, !key to check if the key is not
+                                     to check if the key is not present,
-                                     present, key > NUMBER (like "comment_count
+                                     key>NUMBER (like "comment_count > 12", also
-                                     > 12", also works with >=, <, <=, !=, =) to
+                                     works with >=, <, <=, !=, =) to compare
-                                     compare against a number, key = 'LITERAL'
+                                     against a number, key = 'LITERAL' (like
-                                     (like "uploader = 'Mike Smith'", also works
+                                     "uploader = 'Mike Smith'", also works with
-                                     with !=) to match against a string literal
+                                     !=) to match against a string literal and &
-                                     and & to require multiple matches. Values
+                                     to require multiple matches. Values which
-                                     which are not known are excluded unless you
+                                     are not known are excluded unless you put a
-                                     put a question mark (?) after the operator.
+                                     question mark (?) after the operator. For
-                                     For example, to only match videos that have
+                                     example, to only match videos that have
                                      been liked more than 100 times and disliked
                                      less than 50 times (or the dislike
                                      functionality is not available at the given
                                      service), but who also have a description,
                                      use --match-filter "like_count > 100 &
-                                     dislike_count <? 50 & description" .
+                                     dislike_count <? 50 & description"
     --no-match-filter                Do not use generic video filter (default)
     --no-playlist                    Download only the video, if the URL refers
-                                     to a video and a playlist.
+                                     to a video and a playlist
     --yes-playlist                   Download the playlist, if the URL refers to
-                                     a video and a playlist.
+                                     a video and a playlist
     --age-limit YEARS                Download only videos suitable for the given
                                      age
     --download-archive FILE          Download only videos not listed in the
                                      archive file. Record the IDs of all
-                                     downloaded videos in it.
+                                     downloaded videos in it
-    --break-on-existing              Stop the download process after attempting
+    --break-on-existing              Stop the download process when encountering
-                                     to download a file that's in the archive.
+                                     a file that is in the archive
+    --break-on-reject                Stop the download process when encountering
+                                     a file that has been filtered out
     --no-download-archive            Do not use archive file (default)
     --include-ads                    Download advertisements as well
                                      (experimental)
@@ -268,15 +279,15 @@ Then simply type this
     -r, --limit-rate RATE            Maximum download rate in bytes per second
                                      (e.g. 50K or 4.2M)
     -R, --retries RETRIES            Number of retries (default is 10), or
-                                     "infinite".
+                                     "infinite"
     --fragment-retries RETRIES       Number of retries for a fragment (default
                                      is 10), or "infinite" (DASH, hlsnative and
                                      ISM)
     --skip-unavailable-fragments     Skip unavailable fragments for DASH,
                                      hlsnative and ISM (default)
-                                     (Same as --no-abort-on-unavailable-fragment)
+                                     (Alias: --no-abort-on-unavailable-fragment)
     --abort-on-unavailable-fragment  Abort downloading if a fragment is unavailable
-                                     (Same as --no-skip-unavailable-fragments)
+                                     (Alias: --no-skip-unavailable-fragments)
     --keep-fragments                 Keep downloaded fragments on disk after
                                      downloading is finished
     --no-keep-fragments              Delete downloaded fragments after
@@ -306,19 +317,36 @@ Then simply type this
                                      allowing to play the video while
                                      downloading (some players may not be able
                                      to play it)
-    --external-downloader COMMAND    Use the specified external downloader.
+    --external-downloader NAME       Use the specified external downloader.
-                                     Currently supports
+                                     Currently supports aria2c, avconv, axel,
-                                     aria2c,avconv,axel,curl,ffmpeg,httpie,wget
+                                     curl, ffmpeg, httpie, wget
-    --external-downloader-args ARGS  Give these arguments to the external
+    --downloader-args NAME:ARGS      Give these arguments to the external
-                                     downloader
+                                     downloader. Specify the downloader name and
+                                     the arguments separated by a colon ":". You
+                                     can use this option multiple times
+                                     (Alias: --external-downloader-args)
 
 ## Filesystem Options:
     -a, --batch-file FILE            File containing URLs to download ('-' for
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
-                                     comments and ignored.
+                                     comments and ignored
-    -o, --output TEMPLATE            Output filename template, see the "OUTPUT
+    -P, --paths TYPE:PATH            The paths where the files should be
+                                     downloaded. Specify the type of file and
+                                     the path separated by a colon ":". All the
+                                     same types as --output are supported.
+                                     Additionally, you can also provide "home"
+                                     and "temp" paths. All intermediary files
+                                     are first downloaded to the temp path and
+                                     then the final files are moved over to the
+                                     home path after download is finished. This
+                                     option is ignored if --output is an
+                                     absolute path
+    -o, --output [TYPE:]TEMPLATE     Output filename template, see "OUTPUT
                                      TEMPLATE" for details
+    --output-na-placeholder TEXT     Placeholder value for unavailable meta
+                                     fields in output filename template
+                                     (default: "NA")
     --autonumber-start NUMBER        Specify the start value for %(autonumber)s
                                      (default is 1)
     --restrict-filenames             Restrict filenames to only ASCII
@@ -326,10 +354,16 @@ Then simply type this
                                      filenames
     --no-restrict-filenames          Allow Unicode characters, "&" and spaces in
                                      filenames (default)
-    -w, --no-overwrites              Do not overwrite files
+    -w, --no-overwrites              Do not overwrite any files
-    -c, --continue                   Resume partially downloaded files (default)
+    --force-overwrites               Overwrite all video and metadata files.
-    --no-continue                    Restart download of partially downloaded
+                                     This option includes --no-continue
-                                     files from beginning
+    --no-force-overwrites            Do not overwrite the video, but overwrite
+                                     related files (default)
+    -c, --continue                   Resume partially downloaded files/fragments
+                                     (default)
+    --no-continue                    Do not resume partially downloaded
+                                     fragments. If the file is unfragmented,
+                                     restart download of the entire file
     --part                           Use .part files instead of writing directly
                                      into output file (default)
     --no-part                        Do not use .part files - write directly
@@ -342,10 +376,18 @@ Then simply type this
                                      file
     --no-write-description           Do not write video description (default)
     --write-info-json                Write video metadata to a .info.json file
+                                     (this may contain personal information)
     --no-write-info-json             Do not write video metadata (default)
     --write-annotations              Write video annotations to a
                                      .annotations.xml file
     --no-write-annotations           Do not write video annotations (default)
+    --write-playlist-metafiles       Write playlist metadata in addition to the
+                                     video metadata when using --write-info-json,
+                                     --write-description etc. (default)
+    --no-write-playlist-metafiles    Do not write playlist metadata when using
+                                     --write-info-json, --write-description etc.
+    --get-comments                   Retrieve video comments to be placed in the
+                                     .info.json file
     --load-info-json FILE            JSON file containing the video information
                                      (created with the "--write-info-json"
                                      option)
@@ -359,7 +401,7 @@ Then simply type this
                                      ~/.cache/youtube-dl . At the moment, only
                                      YouTube player files (for videos with
                                      obfuscated signatures) are cached, but that
-                                     may change.
+                                     may change
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
     --trim-file-name LENGTH          Limit the filename length (extension
@@ -374,13 +416,13 @@ Then simply type this
                                      formats
 
 ## Internet Shortcut Options:
-    --write-link                     Write an internet shortcut file, depending on 
+    --write-link                     Write an internet shortcut file, depending
-                                     the current platform (.url/.webloc/.desktop). 
+                                     on the current platform (.url, .webloc or
-                                     The URL may be cached by the OS.
+                                     .desktop). The URL may be cached by the OS
-    --write-url-link                 Write a Windows .url internet shortcut file. 
+    --write-url-link                 Write a .url Windows internet shortcut. The
-                                     (The OS caches the URL based on the file path)
+                                     OS caches the URL based on the file path
-    --write-webloc-link              Write a .webloc macOS internet shortcut file
+    --write-webloc-link              Write a .webloc macOS internet shortcut
-    --write-desktop-link             Write a .desktop Linux internet shortcut file
+    --write-desktop-link             Write a .desktop Linux internet shortcut
 
 ## Verbosity / Simulation Options:
     -q, --quiet                      Activate quiet mode
@@ -397,18 +439,18 @@ Then simply type this
     --get-filename                   Simulate, quiet but print output filename
     --get-format                     Simulate, quiet but print output format
     -j, --dump-json                  Simulate, quiet but print JSON information.
-                                     See the "OUTPUT TEMPLATE" for a description
+                                     See "OUTPUT TEMPLATE" for a description of
-                                     of available keys.
+                                     available keys
     -J, --dump-single-json           Simulate, quiet but print JSON information
                                      for each command-line argument. If the URL
                                      refers to a playlist, dump the whole
-                                     playlist information in a single line.
+                                     playlist information in a single line
     --print-json                     Be quiet and print the video information as
-                                     JSON (video is still being downloaded).
+                                     JSON (video is still being downloaded)
-    --force-write-archive            Force download archive entries to be written 
+    --force-write-archive            Force download archive entries to be
-                                     as far as no errors occur, even if -s or 
+                                     written as far as no errors occur,even if
-                                     another simulation switch is used.
+                                     -s or another simulation switch is used
-                                     (Same as --force-download-archive)
+                                     (Alias: --force-download-archive)
     --newline                        Output progress bar as new lines
     --no-progress                    Do not print progress bar
     --console-title                  Display progress in console titlebar
@@ -419,7 +461,8 @@ Then simply type this
                                      files in the current directory to debug
                                      problems
     --print-traffic                  Display sent and read HTTP traffic
-    -C, --call-home                  Contact the youtube-dlc server for debugging
+    -C, --call-home                  [Broken] Contact the youtube-dlc server for
+                                     debugging
     --no-call-home                   Do not contact the youtube-dlc server for
                                      debugging (default)
 
@@ -433,7 +476,7 @@ Then simply type this
     --referer URL                    Specify a custom referer, use if the video
                                      access is restricted to one domain
     --add-header FIELD:VALUE         Specify a custom HTTP header and its value,
-                                     separated by a colon ':'. You can use this
+                                     separated by a colon ":". You can use this
                                      option multiple times
     --bidi-workaround                Work around terminals that lack
                                      bidirectional text support. Requires bidiv
@@ -443,11 +486,11 @@ Then simply type this
                                      of a range for randomized sleep before each
                                      download (minimum possible number of
                                      seconds to sleep) when used along with
-                                     --max-sleep-interval.
+                                     --max-sleep-interval
     --max-sleep-interval SECONDS     Upper bound of a range for randomized sleep
                                      before each download (maximum possible
                                      number of seconds to sleep). Must only be
-                                     used along with --min-sleep-interval.
+                                     used along with --min-sleep-interval
     --sleep-subtitles SECONDS        Enforce sleep interval on subtitles as well
 
 ## Video Format Options:
@@ -455,8 +498,8 @@ Then simply type this
                                      for more details
     -S, --format-sort SORTORDER      Sort the formats by the fields given, see
                                      "Sorting Formats" for more details
-    --S-force, --format-sort-force   Force user specified sort order to have 
+    --S-force, --format-sort-force   Force user specified sort order to have
-                                     precedence over all fields, see "Sorting 
+                                     precedence over all fields, see "Sorting
                                      Formats" for more details
     --no-format-sort-force           Some fields have precedence over the user
                                      specified sort order (default), see
@@ -470,26 +513,26 @@ Then simply type this
     --no-audio-multistreams          Only one audio stream is downloaded for
                                      each output file (default)
     --all-formats                    Download all available video formats
-    --prefer-free-formats            Prefer free video formats unless a specific
+    --prefer-free-formats            Prefer free video formats over non-free
-                                     one is requested
+                                     formats of same quality
     -F, --list-formats               List all available formats of requested
                                      videos
-    --list-formats-as-table          Present the output of -F in a more tabular
+    --list-formats-as-table          Present the output of -F in tabular form
-                                     form (default)
+                                     (default)
-                                     (Same as --no-list-formats-as-table)
     --list-formats-old               Present the output of -F in the old form
-    --youtube-include-dash-manifest  Download the DASH manifests and related data 
+                                     (Alias: --no-list-formats-as-table)
-                                     on YouTube videos (default)
+    --youtube-include-dash-manifest  Download the DASH manifests and related
-                                     (Same as --no-youtube-skip-dash-manifest)
+                                     data on YouTube videos (default)
+                                     (Alias: --no-youtube-skip-dash-manifest)
     --youtube-skip-dash-manifest     Do not download the DASH manifests and
                                      related data on YouTube videos
-                                     (Same as --no-youtube-include-dash-manifest)
+                                     (Alias: --no-youtube-include-dash-manifest)
-    --youtube-include-hls-manifest   Download the HLS manifests and related data 
+    --youtube-include-hls-manifest   Download the HLS manifests and related data
                                      on YouTube videos (default)
-                                     (Same as --no-youtube-skip-hls-manifest)
+                                     (Alias: --no-youtube-skip-hls-manifest)
     --youtube-skip-hls-manifest      Do not download the HLS manifests and
                                      related data on YouTube videos
-                                     (Same as --no-youtube-include-hls-manifest)
+                                     (Alias: --no-youtube-include-hls-manifest)
     --merge-output-format FORMAT     If a merge is required (e.g.
                                      bestvideo+bestaudio), output to given
                                      container format. One of mkv, mp4, ogg,
@@ -515,7 +558,7 @@ Then simply type this
 ## Authentication Options:
     -u, --username USERNAME          Login with this account ID
     -p, --password PASSWORD          Account password. If this option is left
-                                     out, youtube-dlc will ask interactively.
+                                     out, youtube-dlc will ask interactively
     -2, --twofactor TWOFACTOR        Two-factor authentication code
     -n, --netrc                      Use .netrc authentication data
     --video-password PASSWORD        Video password (vimeo, youku)
@@ -527,30 +570,51 @@ Then simply type this
     --ap-username USERNAME           Multiple-system operator account login
     --ap-password PASSWORD           Multiple-system operator account password.
                                      If this option is left out, youtube-dlc
-                                     will ask interactively.
+                                     will ask interactively
     --ap-list-mso                    List all supported multiple-system
                                      operators
 
 ## Post-Processing Options:
     -x, --extract-audio              Convert video files to audio-only files
-                                     (requires ffmpeg or avconv and ffprobe or
+                                     (requires ffmpeg and ffprobe)
-                                     avprobe)
     --audio-format FORMAT            Specify audio format: "best", "aac",
                                      "flac", "mp3", "m4a", "opus", "vorbis", or
                                      "wav"; "best" by default; No effect without
                                      -x
-    --audio-quality QUALITY          Specify ffmpeg/avconv audio quality, insert
+    --audio-quality QUALITY          Specify ffmpeg audio quality, insert a
-                                     a value between 0 (better) and 9 (worse)
+                                     value between 0 (better) and 9 (worse) for
-                                     for VBR or a specific bitrate like 128K
+                                     VBR or a specific bitrate like 128K
                                      (default 5)
     --remux-video FORMAT             Remux the video into another container if
-                                     necessary (currently supported: mp4|mkv).
+                                     necessary (currently supported: mp4|mkv|flv
-                                     If target container does not support the
+                                     |webm|mov|avi|mp3|mka|m4a|ogg|opus). If
-                                     video/audio codec, remuxing will fail
+                                     target container does not support the
+                                     video/audio codec, remuxing will fail. You
+                                     can specify multiple rules; eg.
+                                     "aac>m4a/mov>mp4/mkv" will remux aac to
+                                     m4a, mov to mp4 and anything else to mkv.
     --recode-video FORMAT            Re-encode the video into another format if
-                                     re-encoding is necessary (currently
+                                     re-encoding is necessary. The supported
-                                     supported: mp4|flv|ogg|webm|mkv|avi)
+                                     formats are the same as --remux-video
-    --postprocessor-args ARGS        Give these arguments to the postprocessor
+    --postprocessor-args NAME:ARGS   Give these arguments to the postprocessors.
+                                     Specify the postprocessor/executable name
+                                     and the arguments separated by a colon ":"
+                                     to give the argument to the specified
+                                     postprocessor/executable. Supported
+                                     postprocessors are: SponSkrub,
+                                     ExtractAudio, VideoRemuxer, VideoConvertor,
+                                     EmbedSubtitle, Metadata, Merger,
+                                     FixupStretched, FixupM4a, FixupM3u8,
+                                     SubtitlesConvertor and EmbedThumbnail. The
+                                     supported executables are: SponSkrub,
+                                     FFmpeg, FFprobe, and AtomicParsley. You can
+                                     use this option multiple times to give
+                                     different arguments to different
+                                     postprocessors. You can also specify
+                                     "PP+EXE:ARGS" to give the arguments to the
+                                     specified executable only when being used
+                                     by the specified postprocessor. You can use
+                                     this option multiple times (Alias: --ppa)
     -k, --keep-video                 Keep the intermediate video file on disk
                                      after post-processing
     --no-keep-video                  Delete the intermediate video file after
@@ -564,16 +628,20 @@ Then simply type this
     --no-embed-thumbnail             Do not embed thumbnail (default)
     --add-metadata                   Write metadata to the video file
     --no-add-metadata                Do not write metadata (default)
-    --metadata-from-title FORMAT     Parse additional metadata like song title /
+    --parse-metadata FIELD:FORMAT    Parse additional metadata like title/artist
-                                     artist from the video title. The format
+                                     from other fields. Give field name to
-                                     syntax is the same as --output. Regular
+                                     extract data from, and format of the field
-                                     expression with named capture groups may
+                                     seperated by a ":". Either regular
+                                     expression with named capture groups or a
+                                     similar syntax to the output template can
                                      also be used. The parsed parameters replace
-                                     existing values. Example: --metadata-from-
+                                     any existing values and can be use in
-                                     title "%(artist)s - %(title)s" matches a
+                                     output templateThis option can be used
+                                     multiple times. Example: --parse-metadata
+                                     "title:%(artist)s - %(title)s" matches a
                                      title like "Coldplay - Paradise". Example
-                                     (regex): --metadata-from-title
+                                     (regex): --parse-metadata
-                                     "(?P<artist>.+?) - (?P<title>.+)"
+                                     "description:Artist - (?P<artist>.+?)"
     --xattrs                         Write metadata to the video file's xattrs
                                      (using dublin core and xdg standards)
     --fixup POLICY                   Automatically correct known faults of the
@@ -581,14 +649,9 @@ Then simply type this
                                      emit a warning), detect_or_warn (the
                                      default; fix file if we can, warn
                                      otherwise)
-    --prefer-avconv                  Prefer avconv over ffmpeg for running the
+    --ffmpeg-location PATH           Location of the ffmpeg binary; either the
-                                     postprocessors (Same as --no-prefer-ffmpeg)
+                                     path to the binary or its containing
-    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
+                                     directory
-                                     postprocessors (default)
-                                     (Same as --no-prefer-avconv)
-    --ffmpeg-location PATH           Location of the ffmpeg/avconv binary;
-                                     either the path to the binary or its
-                                     containing directory.
     --exec CMD                       Execute a command on the file after
                                      downloading and post-processing, similar to
                                      find's -exec syntax. Example: --exec 'adb
@@ -596,11 +659,14 @@ Then simply type this
     --convert-subs FORMAT            Convert the subtitles to other format
                                      (currently supported: srt|ass|vtt|lrc)
 
-## [SponSkrub](https://github.com/faissaloo/SponSkrub) Options ([SponsorBlock](https://sponsor.ajay.app)):
+## SponSkrub (SponsorBlock) Options:
-    --sponskrub                      Use sponskrub to mark sponsored sections
+[SponSkrub](https://github.com/pukkandan/SponSkrub) is a utility to
-                                     with the data available in SponsorBlock
+    mark/remove sponsor segments from downloaded YouTube videos using
-                                     API. This is enabled by default if the
+    [SponsorBlock API](https://sponsor.ajay.app)
-                                     sponskrub binary exists (Youtube only)
+
+    --sponskrub                      Use sponskrub to mark sponsored sections.
+                                     This is enabled by default if the sponskrub
+                                     binary exists (Youtube only)
     --no-sponskrub                   Do not use sponskrub
     --sponskrub-cut                  Cut out the sponsor sections instead of
                                      simply marking them
@@ -612,20 +678,33 @@ Then simply type this
                                      video was already downloaded (default)
     --sponskrub-location PATH        Location of the sponskrub binary; either
                                      the path to the binary or its containing
-                                     directory.
+                                     directory
-    --sponskrub-args None            Give these arguments to sponskrub
 
 ## Extractor Options:
-    --ignore-dynamic-mpd             Do not process dynamic DASH manifests
-                                     (Same as --no-allow-dynamic-mpd)
     --allow-dynamic-mpd              Process dynamic DASH manifests (default)
-                                     (Same as --no-ignore-dynamic-mpd)
+                                     (Alias: --no-ignore-dynamic-mpd)
+    --ignore-dynamic-mpd             Do not process dynamic DASH manifests
+                                     (Alias: --no-allow-dynamic-mpd)
 
 # CONFIGURATION
 
-You can configure youtube-dlc by placing any supported command line option to a configuration file. On Linux and macOS, the system wide configuration file is located at `/etc/youtube-dlc.conf` and the user wide configuration file at `~/.config/youtube-dlc/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dlc\config.txt` or `C:\Users\<user name>\youtube-dlc.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
+You can configure youtube-dlc by placing any supported command line option to a configuration file. The configuration is loaded from the following locations:
 
-For example, with the following configuration file youtube-dlc will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
+1. **Main Configuration**: The file given by `--config-location`
+1. **Portable Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the same directory as the bundled binary. If you are running from source-code (`<root dir>/youtube_dlc/__main__.py`), the root directory is used instead.
+1. **Home Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the home path given by `-P "home:<path>"`, or in the current directory if no such path is given
+1. **User Configuration**:
+    * `%XDG_CONFIG_HOME%/yt-dlp/config` (recommended on Linux/macOS)
+    * `%XDG_CONFIG_HOME%/yt-dlp.conf`
+    * `%APPDATA%/yt-dlp/config` (recommended on Windows)
+    * `%APPDATA%/yt-dlp/config.txt`
+    * `~/yt-dlp.conf`
+    * `~/yt-dlp.conf.txt`
+
+    If none of these files are found, the search is performed again by replacing `yt-dlp` with `youtube-dlc`. Note that `~` points to `C:\Users\<user name>` on windows. Also, `%XDG_CONFIG_HOME%` defaults to `~/.config` if undefined
+1. **System Configuration**: `/etc/yt-dlp.conf` or `/etc/youtube-dlc.conf`
+
+For example, with the following configuration file youtube-dlc will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
 ```
 # Lines starting with # are comments
 
@@ -638,15 +717,13 @@ For example, with the following configuration file youtube-dlc will always extra
 # Use this proxy
 --proxy 127.0.0.1:3128
 
-# Save all videos under Movies directory in your home directory
+# Save all videos under YouTube directory in your home directory
--o ~/Movies/%(title)s.%(ext)s
+-o ~/YouTube/%(title)s.%(ext)s
 ```
 
-Note that options in configuration file are just the same options aka switches used in regular command line calls thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`.
+Note that options in configuration file are just the same options aka switches used in regular command line calls; thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`.
 
-You can use `--ignore-config` if you want to disable the configuration file for a particular youtube-dlc run.
+You can use `--ignore-config` if you want to disable all configuration files for a particular youtube-dlc run. If `--ignore-config` is found inside any configuration file, no further configuration will be loaded. For example, having the option in the portable configuration file prevents loading of user and system configurations. Additionally, (for backward compatibility) if `--ignore-config` is found inside the system configuration file, the user configuration is not loaded.
-
-You can also use `--config-location` if you want to use custom configuration file for a particular youtube-dlc run.
 
 ### Authentication with `.netrc` file
 
@@ -673,11 +750,15 @@ set HOME=%USERPROFILE%
 
 # OUTPUT TEMPLATE
 
-The `-o` option allows users to indicate a template for the output file names.
+The `-o` option is used to indicate a template for the output file names while `-P` option is used to specify the path each type of file should be saved to.
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
-The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dlc -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Allowed names along with sequence type are:
+The basic usage of `-o` is not to set any template arguments when downloading a single file, like in `youtube-dlc -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Date/time fields can also be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it inside the parantheses seperated from the field name using a `>`. For example, `%(duration>%H-%M-%S)s`.
+
+Additionally, you can set different output templates for the various metadata files seperately from the general output template by specifying the type of file followed by the template seperated by a colon ":". The different filetypes supported are `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
+
+The available fields are:
 
  - `id` (string): Video identifier
  - `title` (string): Video title
@@ -707,7 +788,7 @@ The basic usage is not to set any template arguments when downloading a single f
  - `is_live` (boolean): Whether this video is a live stream or a fixed-length video
  - `start_time` (numeric): Time in seconds where the reproduction should start, as specified in the URL
  - `end_time` (numeric): Time in seconds where the reproduction should end, as specified in the URL
- - `format` (string): A human-readable description of the format 
+ - `format` (string): A human-readable description of the format
  - `format_id` (string): Format code specified by `--format`
  - `format_note` (string): Additional info about the format
  - `width` (numeric): Width of the video
@@ -764,7 +845,7 @@ Available for the media that is a track or a part of a music album:
  - `disc_number` (numeric): Number of the disc or other physical medium the track belongs to
  - `release_year` (numeric): Year (YYYY) when the album was released
 
-Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with `NA`.
+Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
 For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `youtube-dlc test video` and id `BaW_jenozKcj`, this will result in a `youtube-dlc test video-BaW_jenozKcj.mp4` file created in the current directory.
 
@@ -784,7 +865,7 @@ If you are using an output template inside a Windows batch file then you must es
 
 #### Output template examples
 
-Note that on Windows you may need to use double quotes instead of single.
+Note that on Windows you need to use double quotes instead of single.
 
 ```bash
 $ youtube-dlc --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc
@@ -796,14 +877,17 @@ youtube-dlc_test_video_.mp4          # A simple file name
 # Download YouTube playlist videos in separate directory indexed by video order in a playlist
 $ youtube-dlc -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re
 
+# Download YouTube playlist videos in seperate directories according to their uploaded year
+$ youtube-dlc -o '%(upload_date>%Y)s/%(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re
+
 # Download all playlists of YouTube channel/user keeping each playlist in separate directory:
 $ youtube-dlc -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists
 
 # Download Udemy course keeping each chapter in separate directory under MyVideos directory in your home
-$ youtube-dlc -u user -p password -o '~/MyVideos/%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/
+$ youtube-dlc -u user -p password -P '~/MyVideos' -o '%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/
 
 # Download entire series season keeping each series and each season in separate directory under C:/MyVideos
-$ youtube-dlc -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617
+$ youtube-dlc -P "C:/MyVideos" -o "%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617
 
 # Stream the video being downloaded to stdout
 $ youtube-dlc -o - BaW_jenozKc
@@ -812,7 +896,7 @@ $ youtube-dlc -o - BaW_jenozKc
 # FORMAT SELECTION
 
 By default, youtube-dlc tries to download the best available quality if you **don't** pass any options.
-This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg and avconv are unavailable, or if you use youtube-dlc to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.
+This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg is unavailable, or if you use youtube-dlc to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.
 
 The general syntax for format selection is `--f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download.
 
@@ -843,7 +927,7 @@ If you want to download multiple videos and they don't have the same formats ava
 
 If you want to download several formats of the same video use a comma as a separator, e.g. `-f 22,17,18` will download all these three formats, of course if they are available. Or a more sophisticated example combined with the precedence feature: `-f 136/137/mp4/bestvideo,140/m4a/bestaudio`.
 
-You can merge the video and audio of multiple formats into a single file using `-f <format1>+<format2>+...` (requires ffmpeg or avconv installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg/avconv. If `--no-video-multistreams` is used, all formats with a video stream except the first one are ignored. Similarly, if `--no-audio-multistreams` is used, all formats with an audio stream except the first one are ignored. For example, `-f bestvideo+best+bestaudio` will download and merge all 3 given formats. The resulting file will have 2 video streams and 2 audio streams. But `-f bestvideo+best+bestaudio --no-video-multistreams` will download and merge only `bestvideo` and `bestaudio`. `best` is ignored since another format containing a video stream (`bestvideo`) has already been selected. The order of the formats is therefore important. `-f best+bestaudio --no-audio-multistreams` will download and merge both formats while `-f bestaudio+best --no-audio-multistreams` will ignore `best` and download only `bestaudio`.
+You can merge the video and audio of multiple formats into a single file using `-f <format1>+<format2>+...` (requires ffmpeg installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg. If `--no-video-multistreams` is used, all formats with a video stream except the first one are ignored. Similarly, if `--no-audio-multistreams` is used, all formats with an audio stream except the first one are ignored. For example, `-f bestvideo+best+bestaudio` will download and merge all 3 given formats. The resulting file will have 2 video streams and 2 audio streams. But `-f bestvideo+best+bestaudio --no-video-multistreams` will download and merge only `bestvideo` and `bestaudio`. `best` is ignored since another format containing a video stream (`bestvideo`) has already been selected. The order of the formats is therefore important. `-f best+bestaudio --no-audio-multistreams` will download and merge both formats while `-f bestaudio+best --no-audio-multistreams` will ignore `best` and download only `bestaudio`.
 
 ## Filtering Formats
 
@@ -882,35 +966,35 @@ Format selectors can also be grouped using parentheses, for example if you want
 
 You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. The available fields are:
 
- - `video`, `has_video`: Gives priority to formats that has a video stream
+ - `hasvid`: Gives priority to formats that has a video stream
- - `audio`, `has_audio`: Gives priority to formats that has a audio stream
+ - `hasaud`: Gives priority to formats that has a audio stream
- - `extractor`, `preference`, `extractor_preference`: The format preference as given by the extractor
+ - `ie_pref`: The format preference as given by the extractor
- - `lang`, `language_preference`: Language preference as given by the extractor
+ - `lang`: Language preference as given by the extractor
  - `quality`: The quality of the format. This is a metadata field available in some websites
- - `source`, `source_preference`: Preference of the source as given by the extractor
+ - `source`: Preference of the source as given by the extractor
- - `proto`, `protocol`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
+ - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`, `video_codec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `vcodec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- - `acodec`, `audio_codec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
+ - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
- - `vext`, `video_ext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
+ - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
- - `aext`, `audio_ext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
+ - `aext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
- - `ext`, `extension`: Equivalent to `vext,aext`
+ - `ext`: Equivalent to `vext,aext`
  - `filesize`: Exact filesize, if know in advance. This will be unavailable for mu38 and DASH formats.
- - `filesize_approx`: Approximate filesize calculated from the manifests
+ - `fs_approx`: Approximate filesize calculated from the manifests
- - `size`, `filesize_estimate`: Exact filesize if available, otherwise approximate filesize
+ - `size`: Exact filesize if available, otherwise approximate filesize
  - `height`: Height of video
  - `width`: Width of video
- - `res`, `dimension`: Video resolution, calculated as the smallest dimension.
+ - `res`: Video resolution, calculated as the smallest dimension.
- - `fps`, `framerate`: Framerate of video
+ - `fps`: Framerate of video
- - `tbr`, `total_bitrate`: Total average bitrate in KBit/s
+ - `tbr`: Total average bitrate in KBit/s
- - `vbr`, `video_bitrate`: Average video bitrate in KBit/s
+ - `vbr`: Average video bitrate in KBit/s
- - `abr`, `audio_bitrate`: Average audio bitrate in KBit/s
+ - `abr`: Average audio bitrate in KBit/s
- - `br`, `bitrate`: Equivalent to using `tbr,vbr,abr`
+ - `br`: Equivalent to using `tbr,vbr,abr`
- - `samplerate`, `asr`: Audio sample rate in Hz
+ - `asr`: Audio sample rate in Hz
 
 Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `has_video`, `extractor`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,has_audio,source,format_id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec:vp9,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
@@ -932,16 +1016,24 @@ $ youtube-dlc -f 'bv*+ba/b'
 # Same as above
 $ youtube-dlc
 
+# Download the best video-only format and the best audio-only format without merging them
+# For this case, an output template should be used since
+# by default, bestvideo and bestaudio will have the same file name.
+$ youtube-dlc -f 'bv,ba' -o '%(title)s.f%(format_id)s.%(ext)s'
 
 
-# Download the worst video available
+
+# The following examples show the old method (without -S) of format selection
+# and how to use -S to achieve a similar but better result
+
+# Download the worst video available (old method)
 $ youtube-dlc -f 'wv*+wa/w'
 
 # Download the best video available but with the smallest resolution
 $ youtube-dlc -S '+res'
 
 # Download the smallest video available
-$ youtube-dlc -S '+size,+bitrate'
+$ youtube-dlc -S '+size,+br'
 
 
 
@@ -989,14 +1081,7 @@ $ youtube-dlc -f '(bv*+ba/b)[protocol^=http][protocol!*=dash] / (bv*+ba/b)'
 
 # Download best video available via the best protocol
 # (https/ftps > http/ftp > m3u8_native > m3u8 > http_dash_segments ...)
-$ youtube-dlc -S 'protocol'
+$ youtube-dlc -S 'proto'
-
-
-
-# Download the best video-only format and the best audio-only format without merging them
-# For this case, an output template should be used since
-# by default, bestvideo and bestaudio will have the same file name.
-$ youtube-dlc -f 'bv,ba' -o '%(title)s.f%(format_id)s.%(ext)s'
 
 
 
@@ -1032,9 +1117,11 @@ $ youtube-dlc -S 'res:720,fps'
 $ youtube-dlc -S '+res:480,codec,br'
 ```
 
+# PLUGINS
 
+Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently only `extractor` plugins are supported. Support for `downloader` and `postprocessor` plugins may be added in the future. See [ytdlp_plugins](ytdlp_plugins) for example.
 
-
+**Note**: `<root-dir>` is the directory of the binary (`<root-dir>/youtube-dlc`), or the root directory of the module if you are running directly from source-code (`<root dir>/youtube_dlc/__main__.py`)
 
 # MORE
-For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl)
+For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl#faq)

devscripts/pyinst.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+from __future__ import unicode_literals
+import sys
+import os
+import platform
+
+from PyInstaller.utils.win32.versioninfo import (
+    VarStruct, VarFileInfo, StringStruct, StringTable,
+    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
+)
+import PyInstaller.__main__
+
+arch = sys.argv[1] if len(sys.argv) > 1 else platform.architecture()[0][:2]
+assert arch in ('32', '64')
+print('Building %sbit version' % arch)
+_x86 = '_x86' if arch == '32' else ''
+
+FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
+SHORT_URLS = {'32': 'git.io/JUGsM', '64': 'git.io/JLh7K'}
+
+root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+print('Changing working directory to %s' % root_dir)
+os.chdir(root_dir)
+
+exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
+VERSION = locals()['__version__']
+
+VERSION_LIST = VERSION.replace('-', '.').split('.')
+VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
+
+print('Version: %s%s' % (VERSION, _x86))
+print('Remember to update the version using devscipts\\update-version.py')
+
+VERSION_FILE = VSVersionInfo(
+    ffi=FixedFileInfo(
+        filevers=VERSION_LIST,
+        prodvers=VERSION_LIST,
+        mask=0x3F,
+        flags=0x0,
+        OS=0x4,
+        fileType=0x1,
+        subtype=0x0,
+        date=(0, 0),
+    ),
+    kids=[
+        StringFileInfo([
+            StringTable(
+                '040904B0', [
+                    StringStruct('Comments', 'Youtube-dlc%s Command Line Interface.' % _x86),
+                    StringStruct('CompanyName', 'pukkandan@gmail.com'),
+                    StringStruct('FileDescription', FILE_DESCRIPTION),
+                    StringStruct('FileVersion', VERSION),
+                    StringStruct('InternalName', 'youtube-dlc%s' % _x86),
+                    StringStruct(
+                        'LegalCopyright',
+                        'pukkandan@gmail.com | UNLICENSE',
+                    ),
+                    StringStruct('OriginalFilename', 'youtube-dlc%s.exe' % _x86),
+                    StringStruct('ProductName', 'Youtube-dlc%s' % _x86),
+                    StringStruct('ProductVersion', '%s%s | %s' % (VERSION, _x86, SHORT_URLS[arch])),
+                ])]),
+        VarFileInfo([VarStruct('Translation', [0, 1200])])
+    ]
+)
+
+PyInstaller.__main__.run([
+    '--name=youtube-dlc%s' % _x86,
+    '--onefile',
+    '--icon=devscripts/cloud.ico',
+    '--exclude-module=youtube_dl',
+    '--exclude-module=test',
+    '--exclude-module=ytdlp_plugins',
+    '--hidden-import=mutagen',
+    'youtube_dlc/__main__.py',
+])
+SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

devscripts/run_tests.bat

@@ -1,7 +1,7 @@
 @echo off
 
 rem Keep this list in sync with the `offlinetest` target in Makefile
-set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature"
+set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature^|post_hooks"
 
 if "%YTDL_TEST_SET%" == "core" (
     set test_set="-I test_("%DOWNLOAD_TESTS%")\.py"

devscripts/run_tests.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 # Keep this list in sync with the `offlinetest` target in Makefile
-DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|socks|subtitles|write_annotations|youtube_lists|youtube_signature"
+DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|overwrites|socks|subtitles|write_annotations|youtube_lists|youtube_signature|post_hooks"
 
 test_set=""
 multiprocess_args=""

devscripts/update-version.py

@@ -0,0 +1,31 @@
+from __future__ import unicode_literals
+from datetime import datetime
+# import urllib.request
+
+# response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
+# old_version = response.read().decode('utf-8')
+
+exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
+old_version = locals()['__version__']
+
+old_version_list = old_version.replace('-', '.').split(".", 4)
+
+old_ver = '.'.join(old_version_list[:3])
+old_rev = old_version_list[3] if len(old_version_list) > 3 else ''
+
+ver = datetime.now().strftime("%Y.%m.%d")
+rev = str(int(old_rev or 0) + 1) if old_ver == ver else ''
+
+VERSION = '.'.join((ver, rev)) if rev else ver
+# VERSION_LIST = [(int(v) for v in ver.split(".") + [rev or 0])]
+
+print('::set-output name=ytdlc_version::' + VERSION)
+
+file_version_py = open('youtube_dlc/version.py', 'rt')
+data = file_version_py.read()
+data = data.replace(old_version, VERSION)
+file_version_py.close()
+
+file_version_py = open('youtube_dlc/version.py', 'wt')
+file_version_py.write(data)
+file_version_py.close()

docs/supportedsites.md

@@ -47,17 +47,21 @@
  - **Amara**
  - **AMCNetworks**
  - **AmericasTestKitchen**
+ - **AmericasTestKitchenSeason**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
+ - **AnimeLab**
+ - **AnimeLabShows**
  - **AnimeOnDemand**
  - **Anvato**
- - **aol.com**
+ - **aol.com**: Yahoo screen and movies
  - **APA**
  - **Aparat**
  - **AppleConnect**
  - **AppleDaily**: 臺灣蘋果日報
+ - **ApplePodcasts**
  - **appletrailers**
  - **appletrailers:section**
- - **archive.org**: archive.org videos
+ - **archive.org**: archive.org video and audio
  - **ArcPublishing**
  - **ARD**
  - **ARD:mediathek**
@@ -76,6 +80,9 @@
  - **AudioBoom**
  - **audiomack**
  - **audiomack:album**
+ - **Audius**: Audius.co
+ - **audius:playlist**: Audius.co playlists
+ - **audius:track**: Audius track ID or API link. Prepend with "audius:"
  - **AWAAN**
  - **awaan:live**
  - **awaan:season**
@@ -99,12 +106,18 @@
  - **BellMedia**
  - **Bet**
  - **bfi:player**
+ - **bfmtv**
+ - **bfmtv:article**
+ - **bfmtv:live**
+ - **BibelTV**
  - **Bigflix**
  - **Bild**: Bild.de
  - **BiliBili**
  - **BilibiliAudio**
  - **BilibiliAudioAlbum**
+ - **BilibiliChannel**
  - **BiliBiliPlayer**
+ - **BiliBiliSearch**: Bilibili video search, "bilisearch" keyword
  - **BioBioChileTV**
  - **Biography**
  - **BIQLE**
@@ -190,8 +203,6 @@
  - **CNNArticle**
  - **CNNBlogs**
  - **ComedyCentral**
- - **ComedyCentralFullEpisodes**
- - **ComedyCentralShortname**
  - **ComedyCentralTV**
  - **CondeNast**: Condé Nast media group: Allure, Architectural Digest, Ars Technica, Bon Appétit, Brides, Condé Nast, Condé Nast Traveler, Details, Epicurious, GQ, Glamour, Golf Digest, SELF, Teen Vogue, The New Yorker, Vanity Fair, Vogue, W Magazine, WIRED
  - **CONtv**
@@ -346,6 +357,8 @@
  - **Go**
  - **GodTube**
  - **Golem**
+ - **google:podcasts**
+ - **google:podcasts:feed**
  - **GoogleDrive**
  - **Goshgay**
  - **GPUTechConf**
@@ -381,6 +394,8 @@
  - **HungamaSong**
  - **Hypem**
  - **ign.com**
+ - **IHeartRadio**
+ - **iheartradio:podcast**
  - **imdb**: Internet Movie Database trailers
  - **imdb:list**: Internet Movie Database lists
  - **Imgur**
@@ -420,7 +435,8 @@
  - **Katsomo**
  - **KeezMovies**
  - **Ketnet**
- - **KhanAcademy**
+ - **khanacademy**
+ - **khanacademy:unit**
  - **KickStarter**
  - **KinjaEmbed**
  - **KinoPoisk**
@@ -508,6 +524,12 @@
  - **Mgoon**
  - **MGTV**: 芒果TV
  - **MiaoPai**
+ - **mildom**: Record ongoing live by specific user in Mildom
+ - **mildom:user:vod**: Download all VODs from specific user in Mildom
+ - **mildom:vod**: Download a VOD in Mildom
+ - **minds**
+ - **minds:channel**
+ - **minds:group**
  - **MinistryGrid**
  - **Minoto**
  - **miomio.tv**
@@ -537,6 +559,7 @@
  - **mtv:video**
  - **mtvjapan**
  - **mtvservices:embedded**
+ - **MTVUutisetArticle**
  - **MuenchenTV**: münchen.tv
  - **mva**: Microsoft Virtual Academy videos
  - **mva:course**: Microsoft Virtual Academy courses
@@ -706,7 +729,6 @@
  - **Playwire**
  - **pluralsight**
  - **pluralsight:course**
- - **plus.google**: Google Plus
  - **podomatic**
  - **Pokemon**
  - **PokemonWatch**
@@ -869,6 +891,8 @@
  - **Sport5**
  - **SportBox**
  - **SportDeutschland**
+ - **spotify**
+ - **spotify:show**
  - **Spreaker**
  - **SpreakerPage**
  - **SpreakerShow**
@@ -951,12 +975,13 @@
  - **TNAFlixNetworkEmbed**
  - **toggle**
  - **ToonGoggles**
- - **Tosh**: Tosh.0
  - **tou.tv**
  - **Toypics**: Toypics video
  - **ToypicsUser**: Toypics user profile
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
+ - **Trovo**
+ - **TrovoVod**
  - **TruNews**
  - **TruTV**
  - **Tube8**
@@ -1066,7 +1091,6 @@
  - **vidme**
  - **vidme:user**
  - **vidme:user:likes**
- - **Vidzi**
  - **vier**: vier.be and vijf.be
  - **vier:videos**
  - **viewlift**
@@ -1111,6 +1135,7 @@
  - **vrv**
  - **vrv:series**
  - **VShare**
+ - **VTM**
  - **VTXTV**
  - **vube**: Vube.com
  - **VuClip**
@@ -1146,7 +1171,7 @@
  - **WWE**
  - **XBef**
  - **XboxClips**
- - **XFileShare**: XFileShare based sites: ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, XVideoSharing
+ - **XFileShare**: XFileShare based sites: Aparat, ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, XVideoSharing
  - **XHamster**
  - **XHamsterEmbed**
  - **XHamsterUser**

make_win.bat

@@ -1 +0,0 @@
-py -m PyInstaller youtube_dlc\__main__.py --onefile --name youtube-dlc --version-file win\ver.txt --icon win\icon\cloud.ico --upx-exclude=vcruntime140.dll

pyinst.py

@@ -1,92 +0,0 @@
-from __future__ import unicode_literals
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
-
-from datetime import datetime
-
-FILE_DESCRIPTION = 'Media Downloader'
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print(version)
-
-version_list = ver.split(".")
-_year, _month, _day = [int(value) for value in version_list]
-_rev = 0
-if rev:
-    _rev = rev
-_ver_tuple = _year, _month, _day, _rev
-
-version_file = VSVersionInfo(
-    ffi=FixedFileInfo(
-        filevers=_ver_tuple,
-        prodvers=_ver_tuple,
-        mask=0x3F,
-        flags=0x0,
-        OS=0x4,
-        fileType=0x1,
-        subtype=0x0,
-        date=(0, 0),
-    ),
-    kids=[
-        StringFileInfo(
-            [
-                StringTable(
-                    "040904B0",
-                    [
-                        StringStruct("Comments", "Youtube-dlc Command Line Interface."),
-                        StringStruct("CompanyName", "theidel@uni-bremen.de"),
-                        StringStruct("FileDescription", FILE_DESCRIPTION),
-                        StringStruct("FileVersion", version),
-                        StringStruct("InternalName", "youtube-dlc"),
-                        StringStruct(
-                            "LegalCopyright",
-                            "theidel@uni-bremen.de | UNLICENSE",
-                        ),
-                        StringStruct("OriginalFilename", "youtube-dlc.exe"),
-                        StringStruct("ProductName", "Youtube-dlc"),
-                        StringStruct("ProductVersion", version + " | git.io/JUGsM"),
-                    ],
-                )
-            ]
-        ),
-        VarFileInfo([VarStruct("Translation", [0, 1200])])
-    ]
-)
-
-PyInstaller.__main__.run([
-    '--name=youtube-dlc',
-    '--onefile',
-    '--icon=win/icon/cloud.ico',
-    'youtube_dlc/__main__.py',
-])
-SetVersion('dist/youtube-dlc.exe', version_file)

pyinst32.py

@@ -1,92 +0,0 @@
-from __future__ import unicode_literals
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
-
-from datetime import datetime
-
-FILE_DESCRIPTION = 'Media Downloader 32 Bit Version'
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print(version)
-
-version_list = ver.split(".")
-_year, _month, _day = [int(value) for value in version_list]
-_rev = 0
-if rev:
-    _rev = rev
-_ver_tuple = _year, _month, _day, _rev
-
-version_file = VSVersionInfo(
-    ffi=FixedFileInfo(
-        filevers=_ver_tuple,
-        prodvers=_ver_tuple,
-        mask=0x3F,
-        flags=0x0,
-        OS=0x4,
-        fileType=0x1,
-        subtype=0x0,
-        date=(0, 0),
-    ),
-    kids=[
-        StringFileInfo(
-            [
-                StringTable(
-                    "040904B0",
-                    [
-                        StringStruct("Comments", "Youtube-dlc_x86 Command Line Interface."),
-                        StringStruct("CompanyName", "theidel@uni-bremen.de"),
-                        StringStruct("FileDescription", FILE_DESCRIPTION),
-                        StringStruct("FileVersion", version),
-                        StringStruct("InternalName", "youtube-dlc_x86"),
-                        StringStruct(
-                            "LegalCopyright",
-                            "theidel@uni-bremen.de | UNLICENSE",
-                        ),
-                        StringStruct("OriginalFilename", "youtube-dlc_x86.exe"),
-                        StringStruct("ProductName", "Youtube-dlc_x86"),
-                        StringStruct("ProductVersion", version + "_x86 | git.io/JUGsM"),
-                    ],
-                )
-            ]
-        ),
-        VarFileInfo([VarStruct("Translation", [0, 1200])])
-    ]
-)
-
-PyInstaller.__main__.run([
-    '--name=youtube-dlc_x86',
-    '--onefile',
-    '--icon=win/icon/cloud.ico',
-    'youtube_dlc/__main__.py',
-])
-SetVersion('dist/youtube-dlc_x86.exe', version_file)

requirements.txt

@@ -0,0 +1 @@
+mutagen

scripts/update-version-workflow.py

@@ -1,44 +0,0 @@
-from __future__ import unicode_literals
-from datetime import datetime
-# import urllib.request
-
-# response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
-# _LATEST_VERSION = response.read().decode('utf-8')
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print('::set-output name=ytdlc_version::' + version)
-
-file_version_py = open('youtube_dlc/version.py', 'rt')
-data = file_version_py.read()
-data = data.replace(locals()['__version__'], version)
-file_version_py.close()
-file_version_py = open('youtube_dlc/version.py', 'wt')
-file_version_py.write(data)
-file_version_py.close()

scripts/update-version.py

@@ -1,33 +0,0 @@
-# Unused
-
-from __future__ import unicode_literals
-from datetime import datetime
-import urllib.request
-
-response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
-
-_LATEST_VERSION = response.read().decode('utf-8')
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))

setup.cfg

@@ -2,5 +2,5 @@
 universal = True
 
 [flake8]
-exclude = youtube_dlc/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv,devscripts/create-github-release.py,devscripts/release.sh,devscripts/show-downloads-statistics.py,scripts/update-version.py
+exclude = youtube_dlc/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv,devscripts/create-github-release.py,devscripts/release.sh,devscripts/show-downloads-statistics.py
 ignore = E402,E501,E731,E741,W503

setup.py

@@ -7,12 +7,21 @@ import warnings
 import sys
 from distutils.spawn import spawn
 
+
 # Get the version from youtube_dlc/version.py without importing the package
 exec(compile(open('youtube_dlc/version.py').read(),
              'youtube_dlc/version.py', 'exec'))
 
-DESCRIPTION = 'Media downloader supporting various sites such as youtube'
+
-LONG_DESCRIPTION = 'Command-line program to download videos from YouTube.com and other video sites. Based on a more active community fork.'
+DESCRIPTION = 'Command-line program to download videos from YouTube.com and many other other video platforms.'
+
+LONG_DESCRIPTION = '\n\n'.join((
+    'Official repository: <https://github.com/pukkandan/yt-dlp>',
+    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
+    open("README.md", "r", encoding="utf-8").read()))
+
+REQUIREMENTS = ['mutagen']
+
 
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
     print("inv")
@@ -37,10 +46,8 @@ else:
     params = {
         'data_files': data_files,
     }
-    #if setuptools_available:
     params['entry_points'] = {'console_scripts': ['youtube-dlc = youtube_dlc:main']}
-    #else:
+
-    #    params['scripts'] = ['bin/youtube-dlc']
 
 class build_lazy_extractors(Command):
     description = 'Build the extractor lazy loading module'
@@ -58,20 +65,26 @@ class build_lazy_extractors(Command):
             dry_run=self.dry_run,
         )
 
+
+packages = find_packages(exclude=("youtube_dl", "test", "ytdlp_plugins"))
+
 setup(
-    name="youtube_dlc",
+    name="yt-dlp",
     version=__version__,
-    maintainer="Tom-Oliver Heidel",
+    maintainer="pukkandan",
-    maintainer_email="theidel@uni-bremen.de",
+    maintainer_email="pukkandan@gmail.com",
     description=DESCRIPTION,
     long_description=LONG_DESCRIPTION,
-    # long_description_content_type="text/markdown",
+    long_description_content_type="text/markdown",
-    url="https://github.com/pukkandan/yt-dlc",
+    url="https://github.com/pukkandan/yt-dlp",
-    packages=find_packages(exclude=("youtube_dl","test",)),
+    packages=packages,
-	#packages=[
+    install_requires=REQUIREMENTS,
-    #    'youtube_dlc',
+    project_urls={
-    #    'youtube_dlc.extractor', 'youtube_dlc.downloader',
+        'Documentation': 'https://github.com/pukkandan/yt-dlp#yt-dlp',
-    #    'youtube_dlc.postprocessor'],
+        'Source': 'https://github.com/pukkandan/yt-dlp',
+        'Tracker': 'https://github.com/pukkandan/yt-dlp/issues',
+        #'Funding': 'https://donate.pypi.org',
+    },
     classifiers=[
 	    "Topic :: Multimedia :: Video",
         "Development Status :: 5 - Production/Stable",

test/parameters.json

@@ -14,7 +14,7 @@
     "logtostderr": false, 
     "matchtitle": null, 
     "max_downloads": null, 
-    "nooverwrites": false, 
+    "overwrites": null, 
     "nopart": false, 
     "noprogress": false, 
     "outtmpl": "%(id)s.%(ext)s", 

test/test_YoutubeDL.py

@@ -637,13 +637,20 @@ class TestYoutubeDL(unittest.TestCase):
             'title2': '%PATH%',
         }
 
-        def fname(templ):
+        def fname(templ, na_placeholder='NA'):
-            ydl = YoutubeDL({'outtmpl': templ})
+            params = {'outtmpl': templ}
+            if na_placeholder != 'NA':
+                params['outtmpl_na_placeholder'] = na_placeholder
+            ydl = YoutubeDL(params)
             return ydl.prepare_filename(info)
         self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
         self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
-        # Replace missing fields with 'NA'
+        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
-        self.assertEqual(fname('%(uploader_date)s-%(id)s.%(ext)s'), 'NA-1234.mp4')
+        # Replace missing fields with 'NA' by default
+        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
+        # Or by provided placeholder
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
         self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
         self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')

test/test_all_urls.py

@@ -69,9 +69,9 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('https://www.youtube.com/feed/watch_later', ['youtube:tab'])
         self.assertMatch('https://www.youtube.com/feed/subscriptions', ['youtube:tab'])
 
-    # def test_youtube_search_matching(self):
+    def test_youtube_search_matching(self):
-    #     self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
+        self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
-    #     self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
+        self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
 
     def test_youtube_extract(self):
         assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)

test/test_overwrites.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python
+from __future__ import unicode_literals
+
+import os
+from os.path import join
+import subprocess
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from test.helper import try_rm
+
+
+root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+download_file = join(root_dir, 'test.webm')
+
+
+class TestOverwrites(unittest.TestCase):
+    def setUp(self):
+        # create an empty file
+        open(download_file, 'a').close()
+
+    def test_default_overwrites(self):
+        outp = subprocess.Popen(
+            [
+                sys.executable, 'youtube_dlc/__main__.py',
+                '-o', 'test.webm',
+                'https://www.youtube.com/watch?v=jNQXAC9IVRw'
+            ], cwd=root_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        sout, serr = outp.communicate()
+        self.assertTrue(b'has already been downloaded' in sout)
+        # if the file has no content, it has not been redownloaded
+        self.assertTrue(os.path.getsize(download_file) < 1)
+
+    def test_yes_overwrites(self):
+        outp = subprocess.Popen(
+            [
+                sys.executable, 'youtube_dlc/__main__.py', '--yes-overwrites',
+                '-o', 'test.webm',
+                'https://www.youtube.com/watch?v=jNQXAC9IVRw'
+            ], cwd=root_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        sout, serr = outp.communicate()
+        self.assertTrue(b'has already been downloaded' not in sout)
+        # if the file has no content, it has not been redownloaded
+        self.assertTrue(os.path.getsize(download_file) > 1)
+
+    def tearDown(self):
+        try_rm(join(root_dir, 'test.webm'))
+
+
+if __name__ == '__main__':
+    unittest.main()

test/test_post_hooks.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+
+from __future__ import unicode_literals
+
+import os
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from test.helper import get_params, try_rm
+import youtube_dlc.YoutubeDL
+from youtube_dlc.utils import DownloadError
+
+
+class YoutubeDL(youtube_dlc.YoutubeDL):
+    def __init__(self, *args, **kwargs):
+        super(YoutubeDL, self).__init__(*args, **kwargs)
+        self.to_stderr = self.to_screen
+
+
+TEST_ID = 'gr51aVj-mLg'
+EXPECTED_NAME = 'gr51aVj-mLg'
+
+
+class TestPostHooks(unittest.TestCase):
+    def setUp(self):
+        self.stored_name_1 = None
+        self.stored_name_2 = None
+        self.params = get_params({
+            'skip_download': False,
+            'writeinfojson': False,
+            'quiet': True,
+            'verbose': False,
+            'cachedir': False,
+        })
+        self.files = []
+
+    def test_post_hooks(self):
+        self.params['post_hooks'] = [self.hook_one, self.hook_two]
+        ydl = YoutubeDL(self.params)
+        ydl.download([TEST_ID])
+        self.assertEqual(self.stored_name_1, EXPECTED_NAME, 'Not the expected name from hook 1')
+        self.assertEqual(self.stored_name_2, EXPECTED_NAME, 'Not the expected name from hook 2')
+
+    def test_post_hook_exception(self):
+        self.params['post_hooks'] = [self.hook_three]
+        ydl = YoutubeDL(self.params)
+        self.assertRaises(DownloadError, ydl.download, [TEST_ID])
+
+    def hook_one(self, filename):
+        self.stored_name_1, _ = os.path.splitext(os.path.basename(filename))
+        self.files.append(filename)
+
+    def hook_two(self, filename):
+        self.stored_name_2, _ = os.path.splitext(os.path.basename(filename))
+        self.files.append(filename)
+
+    def hook_three(self, filename):
+        self.files.append(filename)
+        raise Exception('Test exception for \'%s\'' % filename)
+
+    def tearDown(self):
+        for f in self.files:
+            try_rm(f)
+
+
+if __name__ == '__main__':
+    unittest.main()

test/test_postprocessors.py

@@ -8,10 +8,16 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from youtube_dlc.postprocessor import MetadataFromTitlePP
+from youtube_dlc.postprocessor import MetadataFromFieldPP, MetadataFromTitlePP
+
+
+class TestMetadataFromField(unittest.TestCase):
+    def test_format_to_regex(self):
+        pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
+        self.assertEqual(pp._data[0]['regex'], r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')
 
 
 class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(pp._titleregex, r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')

test/test_subtitles.py

@@ -264,16 +264,24 @@ class TestNRKSubtitles(BaseTestSubtitles):
 
 
 class TestRaiPlaySubtitles(BaseTestSubtitles):
-    url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
     IE = RaiPlayIE
 
-    def test_allsubtitles(self):
+    def test_subtitles_key(self):
+        self.url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
         self.DL.params['writesubtitles'] = True
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['it']))
         self.assertEqual(md5(subtitles['it']), 'b1d90a98755126b61e667567a1f6680a')
 
+    def test_subtitles_array_key(self):
+        self.url = 'https://www.raiplay.it/video/2020/12/Report---04-01-2021-2e90f1de-8eee-4de4-ac0e-78d21db5b600.html'
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['allsubtitles'] = True
+        subtitles = self.getSubtitles()
+        self.assertEqual(set(subtitles.keys()), set(['it']))
+        self.assertEqual(md5(subtitles['it']), '4b3264186fbb103508abe5311cfcb9cd')
+
 
 class TestVikiSubtitles(BaseTestSubtitles):
     url = 'http://www.viki.com/videos/1060846v-punch-episode-18'

test/test_unicode_literals.py

@@ -15,8 +15,6 @@ IGNORED_FILES = [
     'setup.py',  # http://bugs.python.org/issue13943
     'conf.py',
     'buildserver.py',
-    'pyinst.py',
-    'pyinst32.py',
 ]
 
 IGNORED_DIRS = [

test/test_utils.py

@@ -21,6 +21,7 @@ from youtube_dlc.utils import (
     encode_base_n,
     caesar,
     clean_html,
+    clean_podcast_url,
     date_from_str,
     DateRange,
     detect_exe_version,
@@ -1497,6 +1498,10 @@ Line 1
             iri_to_uri('http://导航.中国/'),
             'http://xn--fet810g.xn--fiqs8s/')
 
+    def test_clean_podcast_url(self):
+        self.assertEqual(clean_podcast_url('https://www.podtrac.com/pts/redirect.mp3/chtbl.com/track/5899E/traffic.megaphone.fm/HSW7835899191.mp3'), 'https://traffic.megaphone.fm/HSW7835899191.mp3')
+        self.assertEqual(clean_podcast_url('https://play.podtrac.com/npr-344098539/edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3'), 'https://edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3')
+
 
 if __name__ == '__main__':
     unittest.main()

test/test_youtube_chapters.py

@@ -1,275 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-from __future__ import unicode_literals
-
-# Allow direct execution
-import os
-import sys
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from test.helper import expect_value
-from youtube_dlc.extractor import YoutubeIE
-
-
-class TestYoutubeChapters(unittest.TestCase):
-
-    _TEST_CASES = [
-        (
-            # https://www.youtube.com/watch?v=A22oy8dFjqc
-            # pattern: 00:00 - <title>
-            '''This is the absolute ULTIMATE experience of Queen's set at LIVE AID, this is the best video mixed to the absolutely superior stereo radio broadcast. This vastly superior audio mix takes a huge dump on all of the official mixes. Best viewed in 1080p. ENJOY! ***MAKE SURE TO READ THE DESCRIPTION***<br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+36);return false;">00:36</a> - Bohemian Rhapsody<br /><a href="#" onclick="yt.www.watch.player.seekTo(02*60+42);return false;">02:42</a> - Radio Ga Ga<br /><a href="#" onclick="yt.www.watch.player.seekTo(06*60+53);return false;">06:53</a> - Ay Oh!<br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+34);return false;">07:34</a> - Hammer To Fall<br /><a href="#" onclick="yt.www.watch.player.seekTo(12*60+08);return false;">12:08</a> - Crazy Little Thing Called Love<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+03);return false;">16:03</a> - We Will Rock You<br /><a href="#" onclick="yt.www.watch.player.seekTo(17*60+18);return false;">17:18</a> - We Are The Champions<br /><a href="#" onclick="yt.www.watch.player.seekTo(21*60+12);return false;">21:12</a> - Is This The World We Created...?<br /><br />Short song analysis:<br /><br />- "Bohemian Rhapsody": Although it's a short medley version, it's one of the best performances of the ballad section, with Freddie nailing the Bb4s with the correct studio phrasing (for the first time ever!).<br /><br />- "Radio Ga Ga": Although it's missing one chorus, this is one of - if not the best - the best versions ever, Freddie nails all the Bb4s and sounds very clean! Spike Edney's Roland Jupiter 8 also really shines through on this mix, compared to the DVD releases!<br /><br />- "Audience Improv": A great improv, Freddie sounds strong and confident. You gotta love when he sustains that A4 for 4 seconds!<br /><br />- "Hammer To Fall": Despite missing a verse and a chorus, it's a strong version (possibly the best ever). Freddie sings the song amazingly, and even ad-libs a C#5 and a C5! Also notice how heavy Brian's guitar sounds compared to the thin DVD mixes - it roars!<br /><br />- "Crazy Little Thing Called Love": A great version, the crowd loves the song, the jam is great as well! Only downside to this is the slight feedback issues.<br /><br />- "We Will Rock You": Although cut down to the 1st verse and chorus, Freddie sounds strong. He nails the A4, and the solo from Dr. May is brilliant!<br /><br />- "We Are the Champions": Perhaps the high-light of the performance - Freddie is very daring on this version, he sustains the pre-chorus Bb4s, nails the 1st C5, belts great A4s, but most importantly: He nails the chorus Bb4s, in all 3 choruses! This is the only time he has ever done so! It has to be said though, the last one sounds a bit rough, but that's a side effect of belting high notes for the past 18 minutes, with nodules AND laryngitis!<br /><br />- "Is This The World We Created... ?": Freddie and Brian perform a beautiful version of this, and it is one of the best versions ever. It's both sad and hilarious that a couple of BBC engineers are talking over the song, one of them being completely oblivious of the fact that he is interrupting the performance, on live television... Which was being televised to almost 2 billion homes.<br /><br /><br />All rights go to their respective owners!<br />-----Copyright Disclaimer Under Section 107 of the Copyright Act 1976, allowance is made for fair use for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing. Non-profit, educational or personal use tips the balance in favor of fair use''',
-            1477,
-            [{
-                'start_time': 36,
-                'end_time': 162,
-                'title': 'Bohemian Rhapsody',
-            }, {
-                'start_time': 162,
-                'end_time': 413,
-                'title': 'Radio Ga Ga',
-            }, {
-                'start_time': 413,
-                'end_time': 454,
-                'title': 'Ay Oh!',
-            }, {
-                'start_time': 454,
-                'end_time': 728,
-                'title': 'Hammer To Fall',
-            }, {
-                'start_time': 728,
-                'end_time': 963,
-                'title': 'Crazy Little Thing Called Love',
-            }, {
-                'start_time': 963,
-                'end_time': 1038,
-                'title': 'We Will Rock You',
-            }, {
-                'start_time': 1038,
-                'end_time': 1272,
-                'title': 'We Are The Champions',
-            }, {
-                'start_time': 1272,
-                'end_time': 1477,
-                'title': 'Is This The World We Created...?',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=ekYlRhALiRQ
-            # pattern: <num>. <title> 0:00
-            '1.  Those Beaten Paths of Confusion <a href="#" onclick="yt.www.watch.player.seekTo(0*60+00);return false;">0:00</a><br />2.  Beyond the Shadows of Emptiness & Nothingness <a href="#" onclick="yt.www.watch.player.seekTo(11*60+47);return false;">11:47</a><br />3.  Poison Yourself...With Thought <a href="#" onclick="yt.www.watch.player.seekTo(26*60+30);return false;">26:30</a><br />4.  The Agents of Transformation <a href="#" onclick="yt.www.watch.player.seekTo(35*60+57);return false;">35:57</a><br />5.  Drowning in the Pain of Consciousness <a href="#" onclick="yt.www.watch.player.seekTo(44*60+32);return false;">44:32</a><br />6.  Deny the Disease of Life <a href="#" onclick="yt.www.watch.player.seekTo(53*60+07);return false;">53:07</a><br /><br />More info/Buy: http://crepusculonegro.storenvy.com/products/257645-cn-03-arizmenda-within-the-vacuum-of-infinity<br /><br />No copyright is intended. The rights to this video are assumed by the owner and its affiliates.',
-            4009,
-            [{
-                'start_time': 0,
-                'end_time': 707,
-                'title': '1. Those Beaten Paths of Confusion',
-            }, {
-                'start_time': 707,
-                'end_time': 1590,
-                'title': '2. Beyond the Shadows of Emptiness & Nothingness',
-            }, {
-                'start_time': 1590,
-                'end_time': 2157,
-                'title': '3. Poison Yourself...With Thought',
-            }, {
-                'start_time': 2157,
-                'end_time': 2672,
-                'title': '4. The Agents of Transformation',
-            }, {
-                'start_time': 2672,
-                'end_time': 3187,
-                'title': '5. Drowning in the Pain of Consciousness',
-            }, {
-                'start_time': 3187,
-                'end_time': 4009,
-                'title': '6. Deny the Disease of Life',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=WjL4pSzog9w
-            # pattern: 00:00 <title>
-            '<a href="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" class="yt-uix-servicelink  " data-target-new-window="True" data-servicelink="CDAQ6TgYACITCNf1raqT2dMCFdRjGAod_o0CBSj4HQ" data-url="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" rel="nofollow noopener" target="_blank">https://arizmenda.bandcamp.com/merch/...</a><br /><br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+00);return false;">00:00</a> Christening Unborn Deformities <br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+08);return false;">07:08</a> Taste of Purity<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+16);return false;">16:16</a> Sculpting Sins of a Universal Tongue<br /><a href="#" onclick="yt.www.watch.player.seekTo(24*60+45);return false;">24:45</a> Birth<br /><a href="#" onclick="yt.www.watch.player.seekTo(31*60+24);return false;">31:24</a> Neves<br /><a href="#" onclick="yt.www.watch.player.seekTo(37*60+55);return false;">37:55</a> Libations in Limbo',
-            2705,
-            [{
-                'start_time': 0,
-                'end_time': 428,
-                'title': 'Christening Unborn Deformities',
-            }, {
-                'start_time': 428,
-                'end_time': 976,
-                'title': 'Taste of Purity',
-            }, {
-                'start_time': 976,
-                'end_time': 1485,
-                'title': 'Sculpting Sins of a Universal Tongue',
-            }, {
-                'start_time': 1485,
-                'end_time': 1884,
-                'title': 'Birth',
-            }, {
-                'start_time': 1884,
-                'end_time': 2275,
-                'title': 'Neves',
-            }, {
-                'start_time': 2275,
-                'end_time': 2705,
-                'title': 'Libations in Limbo',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=o3r1sn-t3is
-            # pattern: <title> 00:00 <note>
-            'Download this show in MP3: <a href="http://sh.st/njZKK" class="yt-uix-servicelink  " data-url="http://sh.st/njZKK" data-target-new-window="True" data-servicelink="CDAQ6TgYACITCK3j8_6o2dMCFVDCGAoduVAKKij4HQ" rel="nofollow noopener" target="_blank">http://sh.st/njZKK</a><br /><br />Setlist:<br />I-E-A-I-A-I-O <a href="#" onclick="yt.www.watch.player.seekTo(00*60+45);return false;">00:45</a><br />Suite-Pee <a href="#" onclick="yt.www.watch.player.seekTo(4*60+26);return false;">4:26</a>  (Incomplete)<br />Attack <a href="#" onclick="yt.www.watch.player.seekTo(5*60+31);return false;">5:31</a> (First live performance since 2011)<br />Prison Song <a href="#" onclick="yt.www.watch.player.seekTo(8*60+42);return false;">8:42</a><br />Know <a href="#" onclick="yt.www.watch.player.seekTo(12*60+32);return false;">12:32</a> (First live performance since 2011)<br />Aerials <a href="#" onclick="yt.www.watch.player.seekTo(15*60+32);return false;">15:32</a><br />Soldier Side - Intro <a href="#" onclick="yt.www.watch.player.seekTo(19*60+13);return false;">19:13</a><br />B.Y.O.B. <a href="#" onclick="yt.www.watch.player.seekTo(20*60+09);return false;">20:09</a><br />Soil <a href="#" onclick="yt.www.watch.player.seekTo(24*60+32);return false;">24:32</a><br />Darts <a href="#" onclick="yt.www.watch.player.seekTo(27*60+48);return false;">27:48</a><br />Radio/Video <a href="#" onclick="yt.www.watch.player.seekTo(30*60+38);return false;">30:38</a><br />Hypnotize <a href="#" onclick="yt.www.watch.player.seekTo(35*60+05);return false;">35:05</a><br />Temper <a href="#" onclick="yt.www.watch.player.seekTo(38*60+08);return false;">38:08</a> (First live performance since 1999)<br />CUBErt <a href="#" onclick="yt.www.watch.player.seekTo(41*60+00);return false;">41:00</a><br />Needles <a href="#" onclick="yt.www.watch.player.seekTo(42*60+57);return false;">42:57</a><br />Deer Dance <a href="#" onclick="yt.www.watch.player.seekTo(46*60+27);return false;">46:27</a><br />Bounce <a href="#" onclick="yt.www.watch.player.seekTo(49*60+38);return false;">49:38</a><br />Suggestions <a href="#" onclick="yt.www.watch.player.seekTo(51*60+25);return false;">51:25</a><br />Psycho <a href="#" onclick="yt.www.watch.player.seekTo(53*60+52);return false;">53:52</a><br />Chop Suey! <a href="#" onclick="yt.www.watch.player.seekTo(58*60+13);return false;">58:13</a><br />Lonely Day <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+01*60+15);return false;">1:01:15</a><br />Question! <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+04*60+14);return false;">1:04:14</a><br />Lost in Hollywood <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+08*60+10);return false;">1:08:10</a><br />Vicinity of Obscenity  <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+13*60+40);return false;">1:13:40</a>(First live performance since 2012)<br />Forest <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+16*60+17);return false;">1:16:17</a><br />Cigaro <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+20*60+02);return false;">1:20:02</a><br />Toxicity <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+23*60+57);return false;">1:23:57</a>(with Chino Moreno)<br />Sugar <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+27*60+53);return false;">1:27:53</a>',
-            5640,
-            [{
-                'start_time': 45,
-                'end_time': 266,
-                'title': 'I-E-A-I-A-I-O',
-            }, {
-                'start_time': 266,
-                'end_time': 331,
-                'title': 'Suite-Pee (Incomplete)',
-            }, {
-                'start_time': 331,
-                'end_time': 522,
-                'title': 'Attack (First live performance since 2011)',
-            }, {
-                'start_time': 522,
-                'end_time': 752,
-                'title': 'Prison Song',
-            }, {
-                'start_time': 752,
-                'end_time': 932,
-                'title': 'Know (First live performance since 2011)',
-            }, {
-                'start_time': 932,
-                'end_time': 1153,
-                'title': 'Aerials',
-            }, {
-                'start_time': 1153,
-                'end_time': 1209,
-                'title': 'Soldier Side - Intro',
-            }, {
-                'start_time': 1209,
-                'end_time': 1472,
-                'title': 'B.Y.O.B.',
-            }, {
-                'start_time': 1472,
-                'end_time': 1668,
-                'title': 'Soil',
-            }, {
-                'start_time': 1668,
-                'end_time': 1838,
-                'title': 'Darts',
-            }, {
-                'start_time': 1838,
-                'end_time': 2105,
-                'title': 'Radio/Video',
-            }, {
-                'start_time': 2105,
-                'end_time': 2288,
-                'title': 'Hypnotize',
-            }, {
-                'start_time': 2288,
-                'end_time': 2460,
-                'title': 'Temper (First live performance since 1999)',
-            }, {
-                'start_time': 2460,
-                'end_time': 2577,
-                'title': 'CUBErt',
-            }, {
-                'start_time': 2577,
-                'end_time': 2787,
-                'title': 'Needles',
-            }, {
-                'start_time': 2787,
-                'end_time': 2978,
-                'title': 'Deer Dance',
-            }, {
-                'start_time': 2978,
-                'end_time': 3085,
-                'title': 'Bounce',
-            }, {
-                'start_time': 3085,
-                'end_time': 3232,
-                'title': 'Suggestions',
-            }, {
-                'start_time': 3232,
-                'end_time': 3493,
-                'title': 'Psycho',
-            }, {
-                'start_time': 3493,
-                'end_time': 3675,
-                'title': 'Chop Suey!',
-            }, {
-                'start_time': 3675,
-                'end_time': 3854,
-                'title': 'Lonely Day',
-            }, {
-                'start_time': 3854,
-                'end_time': 4090,
-                'title': 'Question!',
-            }, {
-                'start_time': 4090,
-                'end_time': 4420,
-                'title': 'Lost in Hollywood',
-            }, {
-                'start_time': 4420,
-                'end_time': 4577,
-                'title': 'Vicinity of Obscenity (First live performance since 2012)',
-            }, {
-                'start_time': 4577,
-                'end_time': 4802,
-                'title': 'Forest',
-            }, {
-                'start_time': 4802,
-                'end_time': 5037,
-                'title': 'Cigaro',
-            }, {
-                'start_time': 5037,
-                'end_time': 5273,
-                'title': 'Toxicity (with Chino Moreno)',
-            }, {
-                'start_time': 5273,
-                'end_time': 5640,
-                'title': 'Sugar',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=PkYLQbsqCE8
-            # pattern: <num> - <title> [<latinized title>] 0:00:00
-            '''Затемно (Zatemno) is an Obscure Black Metal Band from Russia.<br /><br />"Во прах (Vo prakh)'' Into The Ashes", Debut mini-album released may 6, 2016, by Death Knell Productions<br />Released on 6 panel digipak CD, limited to 100 copies only<br />And digital format on Bandcamp<br /><br />Tracklist<br /><br />1 - Во прах [Vo prakh] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+00*60+00);return false;">0:00:00</a><br />2 - Искупление [Iskupleniye] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+08*60+10);return false;">0:08:10</a><br />3 - Из серпов луны...[Iz serpov luny] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+14*60+30);return false;">0:14:30</a><br /><br />Links:<br /><a href="https://deathknellprod.bandcamp.com/album/--2" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://deathknellprod.bandcamp.com/album/--2" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://deathknellprod.bandcamp.com/a...</a><br /><a href="https://www.facebook.com/DeathKnellProd/" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://www.facebook.com/DeathKnellProd/" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://www.facebook.com/DeathKnellProd/</a><br /><br /><br />I don't have any right about this artifact, my only intention is to spread the music of the band, all rights are reserved to the Затемно (Zatemno) and his producers, Death Knell Productions.<br /><br />------------------------------------------------------------------<br /><br />Subscribe for more videos like this.<br />My link: <a href="https://web.facebook.com/AttackOfTheDragons" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://web.facebook.com/AttackOfTheDragons" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://web.facebook.com/AttackOfTheD...</a>''',
-            1138,
-            [{
-                'start_time': 0,
-                'end_time': 490,
-                'title': '1 - Во прах [Vo prakh]',
-            }, {
-                'start_time': 490,
-                'end_time': 870,
-                'title': '2 - Искупление [Iskupleniye]',
-            }, {
-                'start_time': 870,
-                'end_time': 1138,
-                'title': '3 - Из серпов луны...[Iz serpov luny]',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=xZW70zEasOk
-            # time point more than duration
-            '''● LCS Spring finals: Saturday and Sunday from <a href="#" onclick="yt.www.watch.player.seekTo(13*60+30);return false;">13:30</a> outside the venue! <br />● PAX East: Fri, Sat & Sun - more info in tomorrows video on the main channel!''',
-            283,
-            []
-        ),
-    ]
-
-    def test_youtube_chapters(self):
-        for description, duration, expected_chapters in self._TEST_CASES:
-            ie = YoutubeIE()
-            expect_value(
-                self, ie._extract_chapters_from_description(description, duration),
-                expected_chapters, None)
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_youtube_signature.py

@@ -86,13 +86,9 @@ class TestPlayerInfo(unittest.TestCase):
             ('https://www.youtube.com/yts/jsbin/player-en_US-vflaxXRn1/base.js', 'vflaxXRn1'),
             ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js', 'vflXGBaUN'),
             ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js', 'vflKjOTVq'),
-            ('http://s.ytimg.com/yt/swfbin/watch_as3-vflrEm9Nq.swf', 'vflrEm9Nq'),
-            ('https://s.ytimg.com/yts/swfbin/player-vflenCdZL/watch_as3.swf', 'vflenCdZL'),
         )
         for player_url, expected_player_id in PLAYER_URLS:
-            expected_player_type = player_url.split('.')[-1]
+            player_id = YoutubeIE._extract_player_info(player_url)
-            player_type, player_id = YoutubeIE._extract_player_info(player_url)
-            self.assertEqual(player_type, expected_player_type)
             self.assertEqual(player_id, expected_player_id)
 
 

win/ver.txt

@@ -1,45 +0,0 @@
-# UTF-8
-#
-# For more details about fixed file info 'ffi' see:
-# http://msdn.microsoft.com/en-us/library/ms646997.aspx
-VSVersionInfo(
-  ffi=FixedFileInfo(
-    # filevers and prodvers should be always a tuple with four items: (1, 2, 3, 4)
-    # Set not needed items to zero 0.
-    filevers=(16, 9, 2020, 0),
-    prodvers=(16, 9, 2020, 0),
-    # Contains a bitmask that specifies the valid bits 'flags'r
-    mask=0x3f,
-    # Contains a bitmask that specifies the Boolean attributes of the file.
-    flags=0x0,
-    # The operating system for which this file was designed.
-    # 0x4 - NT and there is no need to change it.
-    # OS=0x40004,
-	OS=0x4,
-    # The general type of file.
-    # 0x1 - the file is an application.
-    fileType=0x1,
-    # The function of the file.
-    # 0x0 - the function is not defined for this fileType
-    subtype=0x0,
-    # Creation date and time stamp.
-    date=(0, 0)
-    ),
-  kids=[
-    StringFileInfo(
-      [
-      StringTable(
-        u'040904B0',
-        [StringStruct(u'Comments', u'Youtube-dlc Command Line Interface.'),
-	StringStruct(u'CompanyName', u'theidel@uni-bremen.de'),
-        StringStruct(u'FileDescription', u'Media Downloader'),
-        StringStruct(u'FileVersion', u'16.9.2020.0'),
-        StringStruct(u'InternalName', u'youtube-dlc'),
-        StringStruct(u'LegalCopyright', u'theidel@uni-bremen.de | UNLICENSE'),
-        StringStruct(u'OriginalFilename', u'youtube-dlc.exe'),
-        StringStruct(u'ProductName', u'Youtube-dlc'),
-        StringStruct(u'ProductVersion', u'16.9.2020.0 | git.io/JUGsM')])
-      ]), 
-    VarFileInfo([VarStruct(u'Translation', [0, 1200])])
-  ]
-)

youtube-dlc.cmd

@@ -1 +1 @@
-py "%~dp0\youtube_dl\__main__.py"
+@py "%~dp0youtube_dlc\__main__.py" %*

youtube_dlc/__init__.py

@@ -8,17 +8,17 @@ __license__ = 'Public Domain'
 import codecs
 import io
 import os
-import re
 import random
+import re
 import sys
 
 
 from .options import (
     parseOpts,
+    _remux_formats,
 )
 from .compat import (
     compat_getpass,
-    compat_shlex_split,
     workaround_optparse_bug9161,
 )
 from .utils import (
@@ -26,11 +26,13 @@ from .utils import (
     decodeOption,
     DEFAULT_OUTTMPL,
     DownloadError,
+    ExistingVideoReached,
     expand_path,
     match_filter_func,
     MaxDownloadsReached,
     preferredencoding,
     read_batch_urls,
+    RejectedVideoReached,
     SameFileError,
     setproctitle,
     std_headers,
@@ -44,6 +46,7 @@ from .downloader import (
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
+from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
 
@@ -68,14 +71,7 @@ def _real_main(argv=None):
         std_headers['Referer'] = opts.referer
 
     # Custom HTTP headers
-    if opts.headers is not None:
+    std_headers.update(opts.headers)
-        for h in opts.headers:
-            if ':' not in h:
-                parser.error('wrong header formatting, it should be key:value, not "%s"' % h)
-            key, value = h.split(':', 1)
-            if opts.verbose:
-                write_string('[debug] Adding header from command line option %s:%s\n' % (key, value))
-            std_headers[key] = value
 
     # Dump user agent
     if opts.dump_user_agent:
@@ -176,6 +172,9 @@ def _real_main(argv=None):
         opts.max_sleep_interval = opts.sleep_interval
     if opts.ap_mso and opts.ap_mso not in MSO_INFO:
         parser.error('Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers')
+    if opts.overwrites:
+        # --yes-overwrites implies --no-continue
+        opts.continue_dl = False
 
     def parse_retries(retries):
         if retries in ('inf', 'infinite'):
@@ -211,12 +210,15 @@ def _real_main(argv=None):
         opts.audioquality = opts.audioquality.strip('k').strip('K')
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
-    if opts.remuxvideo is not None:
-        if opts.remuxvideo not in ['mp4', 'mkv']:
-            parser.error('invalid video container format specified')
     if opts.recodevideo is not None:
-        if opts.recodevideo not in ['mp4', 'flv', 'webm', 'ogg', 'mkv', 'avi']:
+        if opts.recodevideo not in _remux_formats:
             parser.error('invalid video recode format specified')
+    if opts.remuxvideo and opts.recodevideo:
+        opts.remuxvideo = None
+        write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
+    if opts.remuxvideo is not None:
+        if opts.remuxvideo not in _remux_formats:
+            parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
         if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
             parser.error('invalid subtitle format specified')
@@ -235,32 +237,45 @@ def _real_main(argv=None):
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
 
-    outtmpl = ((opts.outtmpl is not None and opts.outtmpl)
+    outtmpl = opts.outtmpl
-               or (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s')
+    if not outtmpl:
-               or (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s')
+        outtmpl = {'default': (
-               or (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s')
+            '%(title)s-%(id)s-%(format)s.%(ext)s' if opts.format == '-1' and opts.usetitle
-               or (opts.usetitle and '%(title)s-%(id)s.%(ext)s')
+            else '%(id)s-%(format)s.%(ext)s' if opts.format == '-1'
-               or (opts.useid and '%(id)s.%(ext)s')
+            else '%(autonumber)s-%(title)s-%(id)s.%(ext)s' if opts.usetitle and opts.autonumber
-               or (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s')
+            else '%(title)s-%(id)s.%(ext)s' if opts.usetitle
-               or DEFAULT_OUTTMPL)
+            else '%(id)s.%(ext)s' if opts.useid
-    if not os.path.splitext(outtmpl)[1] and opts.extractaudio:
+            else '%(autonumber)s-%(id)s.%(ext)s' if opts.autonumber
+            else None)}
+    outtmpl_default = outtmpl.get('default')
+    if outtmpl_default is not None and not os.path.splitext(outtmpl_default)[1] and opts.extractaudio:
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
-                     ' template'.format(outtmpl))
+                     ' template'.format(outtmpl_default))
+
     for f in opts.format_sort:
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
             parser.error('invalid format sort string "%s" specified' % f)
 
+    if opts.metafromfield is None:
+        opts.metafromfield = []
+    if opts.metafromtitle is not None:
+        opts.metafromfield.append('title:%s' % opts.metafromtitle)
+    for f in opts.metafromfield:
+        if re.match(MetadataFromFieldPP.regex, f) is None:
+            parser.error('invalid format string "%s" specified for --parse-metadata' % f)
+
     any_getting = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
     any_printing = opts.print_json
     download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
 
     # PostProcessors
     postprocessors = []
-    if opts.metafromtitle:
+    if opts.metafromfield:
         postprocessors.append({
-            'key': 'MetadataFromTitle',
+            'key': 'MetadataFromField',
-            'titleformat': opts.metafromtitle
+            'formats': opts.metafromfield,
+            'when': 'beforedl'
         })
     if opts.extractaudio:
         postprocessors.append({
@@ -321,19 +336,25 @@ def _real_main(argv=None):
             'force': opts.sponskrub_force,
             'ignoreerror': opts.sponskrub is None,
         })
-    # Please keep ExecAfterDownload towards the bottom as it allows the user to modify the final file in any way.
+    # ExecAfterDownload must be the last PP
-    # So if the user is able to remove the file before your postprocessor runs it might cause a few problems.
     if opts.exec_cmd:
         postprocessors.append({
             'key': 'ExecAfterDownload',
             'exec_cmd': opts.exec_cmd,
+            'when': 'aftermove'
         })
-    external_downloader_args = None
+
-    if opts.external_downloader_args:
+    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
-        external_downloader_args = compat_shlex_split(opts.external_downloader_args)
+    if 'default' in opts.external_downloader_args:
-    postprocessor_args = None
+        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
-    if opts.postprocessor_args:
+
-        postprocessor_args = compat_shlex_split(opts.postprocessor_args)
+    if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
+        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
+        opts.postprocessor_args.setdefault('sponskrub', [])
+        opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
+
+    audio_ext = opts.audioformat if (opts.extractaudio and opts.audioformat != 'best') else None
+
     match_filter = (
         None if opts.match_filter is None
         else match_filter_func(opts.match_filter))
@@ -371,13 +392,15 @@ def _real_main(argv=None):
         'listformats': opts.listformats,
         'listformats_table': opts.listformats_table,
         'outtmpl': outtmpl,
+        'outtmpl_na_placeholder': opts.outtmpl_na_placeholder,
+        'paths': opts.paths,
         'autonumber_size': opts.autonumber_size,
         'autonumber_start': opts.autonumber_start,
         'restrictfilenames': opts.restrictfilenames,
         'ignoreerrors': opts.ignoreerrors,
         'force_generic_extractor': opts.force_generic_extractor,
         'ratelimit': opts.ratelimit,
-        'nooverwrites': opts.nooverwrites,
+        'overwrites': opts.overwrites,
         'retries': opts.retries,
         'fragment_retries': opts.fragment_retries,
         'skip_unavailable_fragments': opts.skip_unavailable_fragments,
@@ -393,13 +416,14 @@ def _real_main(argv=None):
         'playlistreverse': opts.playlist_reverse,
         'playlistrandom': opts.playlist_random,
         'noplaylist': opts.noplaylist,
-        'logtostderr': opts.outtmpl == '-',
+        'logtostderr': outtmpl_default == '-',
         'consoletitle': opts.consoletitle,
         'nopart': opts.nopart,
         'updatetime': opts.updatetime,
         'writedescription': opts.writedescription,
         'writeannotations': opts.writeannotations,
-        'writeinfojson': opts.writeinfojson,
+        'writeinfojson': opts.writeinfojson or opts.getcomments,
+        'getcomments': opts.getcomments,
         'writethumbnail': opts.writethumbnail,
         'write_all_thumbnails': opts.write_all_thumbnails,
         'writelink': opts.writelink,
@@ -432,6 +456,7 @@ def _real_main(argv=None):
         'age_limit': opts.age_limit,
         'download_archive': download_archive_fn,
         'break_on_existing': opts.break_on_existing,
+        'break_on_reject': opts.break_on_reject,
         'cookiefile': opts.cookiefile,
         'nocheckcertificate': opts.no_check_certificate,
         'prefer_insecure': opts.prefer_insecure,
@@ -449,6 +474,7 @@ def _real_main(argv=None):
         'extract_flat': opts.extract_flat,
         'mark_watched': opts.mark_watched,
         'merge_output_format': opts.merge_output_format,
+        'final_ext': opts.recodevideo or opts.remuxvideo or audio_ext,
         'postprocessors': postprocessors,
         'fixup': opts.fixup,
         'source_address': opts.source_address,
@@ -465,8 +491,8 @@ def _real_main(argv=None):
         'ffmpeg_location': opts.ffmpeg_location,
         'hls_prefer_native': opts.hls_prefer_native,
         'hls_use_mpegts': opts.hls_use_mpegts,
-        'external_downloader_args': external_downloader_args,
+        'external_downloader_args': opts.external_downloader_args,
-        'postprocessor_args': postprocessor_args,
+        'postprocessor_args': opts.postprocessor_args,
         'cn_verification_proxy': opts.cn_verification_proxy,
         'geo_verification_proxy': opts.geo_verification_proxy,
         'config_location': opts.config_location,
@@ -502,8 +528,8 @@ def _real_main(argv=None):
                 retcode = ydl.download_with_info_file(expand_path(opts.load_info_filename))
             else:
                 retcode = ydl.download(all_urls)
-        except MaxDownloadsReached:
+        except (MaxDownloadsReached, ExistingVideoReached, RejectedVideoReached):
-            ydl.to_screen('--max-download limit reached, aborting.')
+            ydl.to_screen('Aborting remaining downloads')
             retcode = 101
 
     sys.exit(retcode)

youtube_dlc/compat.py

@@ -2896,6 +2896,7 @@ else:
     _terminal_size = collections.namedtuple('terminal_size', ['columns', 'lines'])
 
     def compat_get_terminal_size(fallback=(80, 24)):
+        from .utils import process_communicate_or_kill
         columns = compat_getenv('COLUMNS')
         if columns:
             columns = int(columns)
@@ -2912,7 +2913,7 @@ else:
                 sp = subprocess.Popen(
                     ['stty', 'size'],
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                out, err = sp.communicate()
+                out, err = process_communicate_or_kill(sp)
                 _lines, _columns = map(int, out.split())
             except Exception:
                 _columns, _lines = _terminal_size(*fallback)

youtube_dlc/downloader/common.py

@@ -332,7 +332,7 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            self.params.get('nooverwrites', False)
+            not self.params.get('overwrites', subtitle)
             and os.path.exists(encodeFilename(filename))
         )
 

youtube_dlc/downloader/external.py

@@ -22,6 +22,7 @@ from ..utils import (
     handle_youtubedl_headers,
     check_executable,
     is_outdated_version,
+    process_communicate_or_kill,
 )
 
 
@@ -94,7 +95,8 @@ class ExternalFD(FileDownloader):
         return cli_valueless_option(self.params, command_option, param, expected_value)
 
     def _configuration_args(self, default=[]):
-        return cli_configuration_args(self.params, 'external_downloader_args', default)
+        return cli_configuration_args(
+            self.params, 'external_downloader_args', self.get_basename(), default)[0]
 
     def _call_downloader(self, tmpfilename, info_dict):
         """ Either overwrite this or implement _make_cmd """
@@ -104,7 +106,7 @@ class ExternalFD(FileDownloader):
 
         p = subprocess.Popen(
             cmd, stderr=subprocess.PIPE)
-        _, stderr = p.communicate()
+        _, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
             self.to_stderr(stderr.decode('utf-8', 'replace'))
         return p.returncode
@@ -143,7 +145,7 @@ class CurlFD(ExternalFD):
 
         # curl writes the progress to stderr so don't capture it.
         p = subprocess.Popen(cmd)
-        p.communicate()
+        process_communicate_or_kill(p)
         return p.returncode
 
 
@@ -231,7 +233,7 @@ class FFmpegFD(ExternalFD):
         url = info_dict['url']
         ffpp = FFmpegPostProcessor(downloader=self)
         if not ffpp.available:
-            self.report_error('m3u8 download detected but ffmpeg or avconv could not be found. Please install one.')
+            self.report_error('m3u8 download detected but ffmpeg could not be found. Please install')
             return False
         ffpp.check_version()
 
@@ -343,14 +345,17 @@ class FFmpegFD(ExternalFD):
         proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
         try:
             retval = proc.wait()
-        except KeyboardInterrupt:
+        except BaseException as e:
             # subprocces.run would send the SIGKILL signal to ffmpeg and the
             # mp4 file couldn't be played, but if we ask ffmpeg to quit it
             # produces a file that is playable (this is mostly useful for live
             # streams). Note that Windows is not affected and produces playable
             # files (see https://github.com/ytdl-org/youtube-dl/issues/8300).
-            if sys.platform != 'win32':
+            if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32':
-                proc.communicate(b'q')
+                process_communicate_or_kill(proc, b'q')
+            else:
+                proc.kill()
+                proc.wait()
             raise
         return retval
 

youtube_dlc/downloader/hls.py

@@ -172,8 +172,12 @@ class HlsFD(FragmentFD):
                         iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                         decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
                             self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                        frag_content = AES.new(
+                        # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+                        # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+                        # not what it decrypts to.
+                        if not test:
+                            frag_content = AES.new(
+                                decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
                     self._append_fragment(ctx, frag_content)
                     # We only download the first fragment during the test
                     if test:

youtube_dlc/downloader/rtmp.py

@@ -89,11 +89,13 @@ class RtmpFD(FileDownloader):
                                 self.to_screen('')
                             cursor_in_new_line = True
                             self.to_screen('[rtmpdump] ' + line)
-            finally:
+                if not cursor_in_new_line:
+                    self.to_screen('')
+                return proc.wait()
+            except BaseException:  # Including KeyboardInterrupt
+                proc.kill()
                 proc.wait()
-            if not cursor_in_new_line:
+                raise
-                self.to_screen('')
-            return proc.returncode
 
         url = info_dict['url']
         player_url = info_dict.get('player_url')

youtube_dlc/downloader/youtube_live_chat.py

@@ -4,6 +4,9 @@ import re
 import json
 
 from .fragment import FragmentFD
+from ..compat import compat_urllib_error
+from ..utils import try_get
+from ..extractor.youtube import YoutubeBaseInfoExtractor as YT_BaseIE
 
 
 class YoutubeLiveChatReplayFD(FragmentFD):
@@ -15,6 +18,7 @@ class YoutubeLiveChatReplayFD(FragmentFD):
         video_id = info_dict['video_id']
         self.to_screen('[%s] Downloading live chat' % self.FD_NAME)
 
+        fragment_retries = self.params.get('fragment_retries', 0)
         test = self.params.get('test', False)
 
         ctx = {
@@ -28,15 +32,52 @@ class YoutubeLiveChatReplayFD(FragmentFD):
             return self._download_fragment(ctx, url, info_dict, headers)
 
         def parse_yt_initial_data(data):
-            window_patt = b'window\\["ytInitialData"\\]\\s*=\\s*(.*?)(?<=});'
+            patterns = (
-            var_patt = b'var\\s+ytInitialData\\s*=\\s*(.*?)(?<=});'
+                r'%s\\s*%s' % (YT_BaseIE._YT_INITIAL_DATA_RE, YT_BaseIE._YT_INITIAL_BOUNDARY_RE),
-            for patt in window_patt, var_patt:
+                r'%s' % YT_BaseIE._YT_INITIAL_DATA_RE)
+            data = data.decode('utf-8', 'replace')
+            for patt in patterns:
                 try:
                     raw_json = re.search(patt, data).group(1)
                     return json.loads(raw_json)
                 except AttributeError:
                     continue
 
+        def download_and_parse_fragment(url, frag_index):
+            count = 0
+            while count <= fragment_retries:
+                try:
+                    success, raw_fragment = dl_fragment(url)
+                    if not success:
+                        return False, None, None
+                    data = parse_yt_initial_data(raw_fragment) or json.loads(raw_fragment)['response']
+
+                    live_chat_continuation = try_get(
+                        data,
+                        lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
+                    offset = continuation_id = None
+                    processed_fragment = bytearray()
+                    for action in live_chat_continuation.get('actions', []):
+                        if 'replayChatItemAction' in action:
+                            replay_chat_item_action = action['replayChatItemAction']
+                            offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
+                        processed_fragment.extend(
+                            json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
+                    if offset is not None:
+                        continuation_id = try_get(
+                            live_chat_continuation,
+                            lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
+                    self._append_fragment(ctx, processed_fragment)
+
+                    return True, continuation_id, offset
+                except compat_urllib_error.HTTPError as err:
+                    count += 1
+                    if count <= fragment_retries:
+                        self.report_retry_fragment(err, frag_index, count, fragment_retries)
+            if count > fragment_retries:
+                self.report_error('giving up after %s fragment retries' % fragment_retries)
+                return False, None, None
+
         self._prepare_and_start_frag_download(ctx)
 
         success, raw_fragment = dl_fragment(
@@ -44,54 +85,25 @@ class YoutubeLiveChatReplayFD(FragmentFD):
         if not success:
             return False
         data = parse_yt_initial_data(raw_fragment)
-        continuation_id = data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
+        continuation_id = try_get(
+            data,
+            lambda x: x['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation'])
         # no data yet but required to call _append_fragment
         self._append_fragment(ctx, b'')
 
-        first = True
+        frag_index = offset = 0
-        offset = None
         while continuation_id is not None:
-            data = None
+            frag_index += 1
-            if first:
+            url = ''.join((
-                url = 'https://www.youtube.com/live_chat_replay?continuation={}'.format(continuation_id)
+                'https://www.youtube.com/live_chat_replay',
-                success, raw_fragment = dl_fragment(url)
+                '/get_live_chat_replay' if frag_index > 1 else '',
-                if not success:
+                '?continuation=%s' % continuation_id,
-                    return False
+                '&playerOffsetMs=%d&hidden=false&pbj=1' % max(offset - 5000, 0) if frag_index > 1 else ''))
-                data = parse_yt_initial_data(raw_fragment)
+            success, continuation_id, offset = download_and_parse_fragment(url, frag_index)
-            else:
+            if not success:
-                url = ('https://www.youtube.com/live_chat_replay/get_live_chat_replay'
+                return False
-                       + '?continuation={}'.format(continuation_id)
+            if test:
-                       + '&playerOffsetMs={}'.format(offset - 5000)
-                       + '&hidden=false'
-                       + '&pbj=1')
-                success, raw_fragment = dl_fragment(url)
-                if not success:
-                    return False
-                data = json.loads(raw_fragment)['response']
-
-            first = False
-            continuation_id = None
-
-            live_chat_continuation = data['continuationContents']['liveChatContinuation']
-            offset = None
-            processed_fragment = bytearray()
-            if 'actions' in live_chat_continuation:
-                for action in live_chat_continuation['actions']:
-                    if 'replayChatItemAction' in action:
-                        replay_chat_item_action = action['replayChatItemAction']
-                        offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
-                    processed_fragment.extend(
-                        json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
-                try:
-                    continuation_id = live_chat_continuation['continuations'][0]['liveChatReplayContinuationData']['continuation']
-                except KeyError:
-                    continuation_id = None
-
-            self._append_fragment(ctx, processed_fragment)
-
-            if test or offset is None:
                 break
 
         self._finish_frag_download(ctx)
-
         return True

youtube_dlc/extractor/__init__.py

@@ -1,13 +1,20 @@
 from __future__ import unicode_literals
 
+from ..utils import load_plugins
+
 try:
     from .lazy_extractors import *
     from .lazy_extractors import _ALL_CLASSES
     _LAZY_LOADER = True
+    _PLUGIN_CLASSES = []
 except ImportError:
     _LAZY_LOADER = False
+
+if not _LAZY_LOADER:
     from .extractors import *
 
+    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+
     _ALL_CLASSES = [
         klass
         for name, klass in globals().items()

youtube_dlc/extractor/abcnews.py

@@ -1,14 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import calendar
 import re
-import time
 
 from .amp import AMPIE
 from .common import InfoExtractor
-from .youtube import YoutubeIE
+from ..utils import (
-from ..compat import compat_urlparse
+    parse_duration,
+    parse_iso8601,
+    try_get,
+)
 
 
 class AbcNewsVideoIE(AMPIE):
@@ -18,8 +19,8 @@ class AbcNewsVideoIE(AMPIE):
                         (?:
                             abcnews\.go\.com/
                             (?:
-                                [^/]+/video/(?P<display_id>[0-9a-z-]+)-|
+                                (?:[^/]+/)*video/(?P<display_id>[0-9a-z-]+)-|
-                                video/embed\?.*?\bid=
+                                video/(?:embed|itemfeed)\?.*?\bid=
                             )|
                             fivethirtyeight\.abcnews\.go\.com/video/embed/\d+/
                         )
@@ -36,6 +37,8 @@ class AbcNewsVideoIE(AMPIE):
             'description': 'George Stephanopoulos goes one-on-one with Iranian Foreign Minister Dr. Javad Zarif.',
             'duration': 180,
             'thumbnail': r're:^https?://.*\.jpg$',
+            'timestamp': 1380454200,
+            'upload_date': '20130929',
         },
         'params': {
             # m3u8 download
@@ -47,6 +50,12 @@ class AbcNewsVideoIE(AMPIE):
     }, {
         'url': 'http://abcnews.go.com/2020/video/2020-husband-stands-teacher-jail-student-affairs-26119478',
         'only_matching': True,
+    }, {
+        'url': 'http://abcnews.go.com/video/itemfeed?id=46979033',
+        'only_matching': True,
+    }, {
+        'url': 'https://abcnews.go.com/GMA/News/video/history-christmas-story-67894761',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -67,28 +76,23 @@ class AbcNewsIE(InfoExtractor):
     _VALID_URL = r'https?://abcnews\.go\.com/(?:[^/]+/)+(?P<display_id>[0-9a-z-]+)/story\?id=(?P<id>\d+)'
 
     _TESTS = [{
-        'url': 'http://abcnews.go.com/Blotter/News/dramatic-video-rare-death-job-america/story?id=10498713#.UIhwosWHLjY',
+        # Youtube Embeds
+        'url': 'https://abcnews.go.com/Entertainment/peter-billingsley-child-actor-christmas-story-hollywood-power/story?id=51286501',
         'info_dict': {
-            'id': '10505354',
+            'id': '51286501',
-            'ext': 'flv',
+            'title': "Peter Billingsley: From child actor in 'A Christmas Story' to Hollywood power player",
-            'display_id': 'dramatic-video-rare-death-job-america',
+            'description': 'Billingsley went from a child actor to Hollywood power player.',
-            'title': 'Occupational Hazards',
-            'description': 'Nightline investigates the dangers that lurk at various jobs.',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'upload_date': '20100428',
-            'timestamp': 1272412800,
         },
-        'add_ie': ['AbcNewsVideo'],
+        'playlist_count': 5,
     }, {
         'url': 'http://abcnews.go.com/Entertainment/justin-timberlake-performs-stop-feeling-eurovision-2016/story?id=39125818',
         'info_dict': {
             'id': '38897857',
             'ext': 'mp4',
-            'display_id': 'justin-timberlake-performs-stop-feeling-eurovision-2016',
             'title': 'Justin Timberlake Drops Hints For Secret Single',
             'description': 'Lara Spencer reports the buzziest stories of the day in "GMA" Pop News.',
-            'upload_date': '20160515',
+            'upload_date': '20160505',
-            'timestamp': 1463329500,
+            'timestamp': 1462442280,
         },
         'params': {
             # m3u8 download
@@ -100,49 +104,55 @@ class AbcNewsIE(InfoExtractor):
     }, {
         'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
         'only_matching': True,
+    }, {
+        # inline.type == 'video'
+        'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        story_id = self._match_id(url)
-        display_id = mobj.group('display_id')
+        webpage = self._download_webpage(url, story_id)
-        video_id = mobj.group('id')
+        story = self._parse_json(self._search_regex(
+            r"window\['__abcnews__'\]\s*=\s*({.+?});",
+            webpage, 'data'), story_id)['page']['content']['story']['everscroll'][0]
+        article_contents = story.get('articleContents') or {}
 
-        webpage = self._download_webpage(url, video_id)
+        def entries():
-        video_url = self._search_regex(
+            featured_video = story.get('featuredVideo') or {}
-            r'window\.abcnvideo\.url\s*=\s*"([^"]+)"', webpage, 'video URL')
+            feed = try_get(featured_video, lambda x: x['video']['feed'])
-        full_video_url = compat_urlparse.urljoin(url, video_url)
+            if feed:
+                yield {
+                    '_type': 'url',
+                    'id': featured_video.get('id'),
+                    'title': featured_video.get('name'),
+                    'url': feed,
+                    'thumbnail': featured_video.get('images'),
+                    'description': featured_video.get('description'),
+                    'timestamp': parse_iso8601(featured_video.get('uploadDate')),
+                    'duration': parse_duration(featured_video.get('duration')),
+                    'ie_key': AbcNewsVideoIE.ie_key(),
+                }
 
-        youtube_url = YoutubeIE._extract_url(webpage)
+            for inline in (article_contents.get('inlines') or []):
+                inline_type = inline.get('type')
+                if inline_type == 'iframe':
+                    iframe_url = try_get(inline, lambda x: x['attrs']['src'])
+                    if iframe_url:
+                        yield self.url_result(iframe_url)
+                elif inline_type == 'video':
+                    video_id = inline.get('id')
+                    if video_id:
+                        yield {
+                            '_type': 'url',
+                            'id': video_id,
+                            'url': 'http://abcnews.go.com/video/embed?id=' + video_id,
+                            'thumbnail': inline.get('imgSrc') or inline.get('imgDefault'),
+                            'description': inline.get('description'),
+                            'duration': parse_duration(inline.get('duration')),
+                            'ie_key': AbcNewsVideoIE.ie_key(),
+                        }
 
-        timestamp = None
+        return self.playlist_result(
-        date_str = self._html_search_regex(
+            entries(), story_id, article_contents.get('headline'),
-            r'<span[^>]+class="timestamp">([^<]+)</span>',
+            article_contents.get('subHead'))
-            webpage, 'timestamp', fatal=False)
-        if date_str:
-            tz_offset = 0
-            if date_str.endswith(' ET'):  # Eastern Time
-                tz_offset = -5
-                date_str = date_str[:-3]
-            date_formats = ['%b. %d, %Y', '%b %d, %Y, %I:%M %p']
-            for date_format in date_formats:
-                try:
-                    timestamp = calendar.timegm(time.strptime(date_str.strip(), date_format))
-                except ValueError:
-                    continue
-            if timestamp is not None:
-                timestamp -= tz_offset * 3600
-
-        entry = {
-            '_type': 'url_transparent',
-            'ie_key': AbcNewsVideoIE.ie_key(),
-            'url': full_video_url,
-            'id': video_id,
-            'display_id': display_id,
-            'timestamp': timestamp,
-        }
-
-        if youtube_url:
-            entries = [entry, self.url_result(youtube_url, ie=YoutubeIE.ie_key())]
-            return self.playlist_result(entries)
-
-        return entry

youtube_dlc/extractor/acast.py

@@ -6,6 +6,7 @@ import re
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
+    clean_podcast_url,
     int_or_none,
     parse_iso8601,
 )
@@ -17,7 +18,7 @@ class ACastBaseIE(InfoExtractor):
         info = {
             'id': episode['id'],
             'display_id': episode.get('episodeUrl'),
-            'url': episode['url'],
+            'url': clean_podcast_url(episode['url']),
             'title': title,
             'description': clean_html(episode.get('description') or episode.get('summary')),
             'thumbnail': episode.get('image'),

youtube_dlc/extractor/adn.py

@@ -10,6 +10,7 @@ import random
 from .common import InfoExtractor
 from ..aes import aes_cbc_decrypt
 from ..compat import (
+    compat_HTTPError,
     compat_b64decode,
     compat_ord,
 )
@@ -18,11 +19,14 @@ from ..utils import (
     bytes_to_long,
     ExtractorError,
     float_or_none,
+    int_or_none,
     intlist_to_bytes,
     long_to_bytes,
     pkcs1pad,
     strip_or_none,
-    urljoin,
+    try_get,
+    unified_strdate,
+    urlencode_postdata,
 )
 
 
@@ -31,16 +35,30 @@ class ADNIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?animedigitalnetwork\.fr/video/[^/]+/(?P<id>\d+)'
     _TEST = {
         'url': 'http://animedigitalnetwork.fr/video/blue-exorcist-kyoto-saga/7778-episode-1-debut-des-hostilites',
-        'md5': 'e497370d847fd79d9d4c74be55575c7a',
+        'md5': '0319c99885ff5547565cacb4f3f9348d',
         'info_dict': {
             'id': '7778',
             'ext': 'mp4',
-            'title': 'Blue Exorcist - Kyôto Saga - Épisode 1',
+            'title': 'Blue Exorcist - Kyôto Saga - Episode 1',
             'description': 'md5:2f7b5aa76edbc1a7a92cedcda8a528d5',
+            'series': 'Blue Exorcist - Kyôto Saga',
+            'duration': 1467,
+            'release_date': '20170106',
+            'comment_count': int,
+            'average_rating': float,
+            'season_number': 2,
+            'episode': 'Début des hostilités',
+            'episode_number': 1,
         }
     }
+
+    _NETRC_MACHINE = 'animedigitalnetwork'
     _BASE_URL = 'http://animedigitalnetwork.fr'
-    _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
+    _API_BASE_URL = 'https://gw.api.animedigitalnetwork.fr/'
+    _PLAYER_BASE_URL = _API_BASE_URL + 'player/'
+    _HEADERS = {}
+    _LOGIN_ERR_MESSAGE = 'Unable to log in'
+    _RSA_KEY = (0x9B42B08905199A5CCE2026274399CA560ECB209EE9878A708B1C0812E1BB8CB5D1FB7441861147C1A1F2F3A0476DD63A9CAC20D3E983613346850AA6CB38F16DC7D720FD7D86FC6E5B3D5BBC72E14CD0BF9E869F2CEA2CCAD648F1DCE38F1FF916CEFB2D339B64AA0264372344BC775E265E8A852F88144AB0BD9AA06C1A4ABB, 65537)
     _POS_ALIGN_MAP = {
         'start': 1,
         'end': 3,
@@ -54,26 +72,24 @@ class ADNIE(InfoExtractor):
     def _ass_subtitles_timecode(seconds):
         return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
 
-    def _get_subtitles(self, sub_path, video_id):
+    def _get_subtitles(self, sub_url, video_id):
-        if not sub_path:
+        if not sub_url:
             return None
 
         enc_subtitles = self._download_webpage(
-            urljoin(self._BASE_URL, sub_path),
+            sub_url, video_id, 'Downloading subtitles location', fatal=False) or '{}'
-            video_id, 'Downloading subtitles location', fatal=False) or '{}'
         subtitle_location = (self._parse_json(enc_subtitles, video_id, fatal=False) or {}).get('location')
         if subtitle_location:
             enc_subtitles = self._download_webpage(
-                urljoin(self._BASE_URL, subtitle_location),
+                subtitle_location, video_id, 'Downloading subtitles data',
-                video_id, 'Downloading subtitles data', fatal=False,
+                fatal=False, headers={'Origin': 'https://animedigitalnetwork.fr'})
-                headers={'Origin': 'https://animedigitalnetwork.fr'})
         if not enc_subtitles:
             return None
 
         # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
         dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
             bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(binascii.unhexlify(self._K + '4b8ef13ec1872730')),
+            bytes_to_intlist(binascii.unhexlify(self._K + 'ab9f52f5baae7c72')),
             bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
         ))
         subtitles_json = self._parse_json(
@@ -117,61 +133,100 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
             }])
         return subtitles
 
+    def _real_initialize(self):
+        username, password = self._get_login_info()
+        if not username:
+            return
+        try:
+            access_token = (self._download_json(
+                self._API_BASE_URL + 'authentication/login', None,
+                'Logging in', self._LOGIN_ERR_MESSAGE, fatal=False,
+                data=urlencode_postdata({
+                    'password': password,
+                    'rememberMe': False,
+                    'source': 'Web',
+                    'username': username,
+                })) or {}).get('accessToken')
+            if access_token:
+                self._HEADERS = {'authorization': 'Bearer ' + access_token}
+        except ExtractorError as e:
+            message = None
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
+                resp = self._parse_json(
+                    e.cause.read().decode(), None, fatal=False) or {}
+                message = resp.get('message') or resp.get('code')
+            self.report_warning(message or self._LOGIN_ERR_MESSAGE)
+
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        video_base_url = self._PLAYER_BASE_URL + 'video/%s/' % video_id
-        player_config = self._parse_json(self._search_regex(
+        player = self._download_json(
-            r'playerConfig\s*=\s*({.+});', webpage,
+            video_base_url + 'configuration', video_id,
-            'player config', default='{}'), video_id, fatal=False)
+            'Downloading player config JSON metadata',
-        if not player_config:
+            headers=self._HEADERS)['player']
-            config_url = urljoin(self._BASE_URL, self._search_regex(
+        options = player['options']
-                r'(?:id="player"|class="[^"]*adn-player-container[^"]*")[^>]+data-url="([^"]+)"',
-                webpage, 'config url'))
-            player_config = self._download_json(
-                config_url, video_id,
-                'Downloading player config JSON metadata')['player']
 
-        video_info = {}
+        user = options['user']
-        video_info_str = self._search_regex(
+        if not user.get('hasAccess'):
-            r'videoInfo\s*=\s*({.+});', webpage,
+            self.raise_login_required()
-            'video info', fatal=False)
-        if video_info_str:
-            video_info = self._parse_json(
-                video_info_str, video_id, fatal=False) or {}
 
-        options = player_config.get('options') or {}
+        token = self._download_json(
-        metas = options.get('metas') or {}
+            user.get('refreshTokenUrl') or (self._PLAYER_BASE_URL + 'refresh/token'),
-        links = player_config.get('links') or {}
+            video_id, 'Downloading access token', headers={
-        sub_path = player_config.get('subtitles')
+                'x-player-refresh-token': user['refreshToken']
-        error = None
+            }, data=b'')['token']
-        if not links:
+
-            links_url = player_config.get('linksurl') or options['videoUrl']
+        links_url = try_get(options, lambda x: x['video']['url']) or (video_base_url + 'link')
-            token = options['token']
+        self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
-            self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
+        message = bytes_to_intlist(json.dumps({
-            message = bytes_to_intlist(json.dumps({
+            'k': self._K,
-                'k': self._K,
+            't': token,
-                'e': 60,
+        }))
-                't': token,
+
-            }))
+        # Sometimes authentication fails for no good reason, retry with
+        # a different random padding
+        links_data = None
+        for _ in range(3):
             padded_message = intlist_to_bytes(pkcs1pad(message, 128))
             n, e = self._RSA_KEY
             encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
             authorization = base64.b64encode(encrypted_message).decode()
-            links_data = self._download_json(
+
-                urljoin(self._BASE_URL, links_url), video_id,
+            try:
-                'Downloading links JSON metadata', headers={
+                links_data = self._download_json(
-                    'Authorization': 'Bearer ' + authorization,
+                    links_url, video_id, 'Downloading links JSON metadata', headers={
-                })
+                        'X-Player-Token': authorization
-            links = links_data.get('links') or {}
+                    }, query={
-            metas = metas or links_data.get('meta') or {}
+                        'freeWithAds': 'true',
-            sub_path = sub_path or links_data.get('subtitles') or \
+                        'adaptive': 'false',
-                'index.php?option=com_vodapi&task=subtitles.getJSON&format=json&id=' + video_id
+                        'withMetadata': 'true',
-            sub_path += '&token=' + token
+                        'source': 'Web'
-            error = links_data.get('error')
+                    })
-        title = metas.get('title') or video_info['title']
+                break
+            except ExtractorError as e:
+                if not isinstance(e.cause, compat_HTTPError):
+                    raise e
+
+                if e.cause.code == 401:
+                    # This usually goes away with a different random pkcs1pad, so retry
+                    continue
+
+                error = self._parse_json(e.cause.read(), video_id)
+                message = error.get('message')
+                if e.cause.code == 403 and error.get('code') == 'player-bad-geolocation-country':
+                    self.raise_geo_restricted(msg=message)
+                raise ExtractorError(message)
+        else:
+            raise ExtractorError('Giving up retrying')
+
+        links = links_data.get('links') or {}
+        metas = links_data.get('metadata') or {}
+        sub_url = (links.get('subtitles') or {}).get('all')
+        video_info = links_data.get('video') or {}
+        title = metas['title']
 
         formats = []
-        for format_id, qualities in links.items():
+        for format_id, qualities in (links.get('streaming') or {}).items():
             if not isinstance(qualities, dict):
                 continue
             for quality, load_balancer_url in qualities.items():
@@ -189,19 +244,26 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                     for f in m3u8_formats:
                         f['language'] = 'fr'
                 formats.extend(m3u8_formats)
-        if not error:
-            error = options.get('error')
-        if not formats and error:
-            raise ExtractorError('%s said: %s' % (self.IE_NAME, error), expected=True)
         self._sort_formats(formats)
 
+        video = (self._download_json(
+            self._API_BASE_URL + 'video/%s' % video_id, video_id,
+            'Downloading additional video metadata', fatal=False) or {}).get('video') or {}
+        show = video.get('show') or {}
+
         return {
             'id': video_id,
             'title': title,
-            'description': strip_or_none(metas.get('summary') or video_info.get('resume')),
+            'description': strip_or_none(metas.get('summary') or video.get('summary')),
-            'thumbnail': video_info.get('image'),
+            'thumbnail': video_info.get('image') or player.get('image'),
             'formats': formats,
-            'subtitles': self.extract_subtitles(sub_path, video_id),
+            'subtitles': self.extract_subtitles(sub_url, video_id),
-            'episode': metas.get('subtitle') or video_info.get('videoTitle'),
+            'episode': metas.get('subtitle') or video.get('name'),
-            'series': video_info.get('playlistTitle'),
+            'episode_number': int_or_none(video.get('shortNumber')),
+            'series': show.get('title'),
+            'season_number': int_or_none(video.get('season')),
+            'duration': int_or_none(video_info.get('duration') or video.get('duration')),
+            'release_date': unified_strdate(video.get('releaseDate')),
+            'average_rating': float_or_none(video.get('rating') or metas.get('rating')),
+            'comment_count': int_or_none(video.get('commentsCount')),
         }

youtube_dlc/extractor/adobepass.py

@@ -9,6 +9,7 @@ from .common import InfoExtractor
 from ..compat import (
     compat_kwargs,
     compat_urlparse,
+    compat_getpass
 )
 from ..utils import (
     unescapeHTML,
@@ -60,6 +61,10 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Philo': {
+        'name': 'Philo',
+        'username_field': 'ident'
+    },
     'Verizon': {
         'name': 'Verizon FiOS',
         'username_field': 'IDToken1',
@@ -1467,6 +1472,22 @@ class AdobePassIE(InfoExtractor):
                         mvpd_confirm_page, urlh = mvpd_confirm_page_res
                         if '<button class="submit" value="Resume">Resume</button>' in mvpd_confirm_page:
                             post_form(mvpd_confirm_page_res, 'Confirming Login')
+                elif mso_id == 'Philo':
+                    # Philo has very unique authentication method
+                    self._download_webpage(
+                        'https://idp.philo.com/auth/init/login_code', video_id, 'Requesting auth code', data=urlencode_postdata({
+                            'ident': username,
+                            'device': 'web',
+                            'send_confirm_link': False,
+                            'send_token': True
+                        }))
+                    philo_code = compat_getpass('Type auth code you have received [Return]: ')
+                    self._download_webpage(
+                        'https://idp.philo.com/auth/update/login_code', video_id, 'Submitting token', data=urlencode_postdata({
+                            'token': philo_code
+                        }))
+                    mvpd_confirm_page_res = self._download_webpage_handle('https://idp.philo.com/idp/submit', video_id, 'Confirming Philo Login')
+                    post_form(mvpd_confirm_page_res, 'Confirming Login')
                 elif mso_id == 'Verizon':
                     # In general, if you're connecting from a Verizon-assigned IP,
                     # you will not actually pass your credentials.

youtube_dlc/extractor/aenetworks.py

@@ -252,11 +252,11 @@ class AENetworksShowIE(AENetworksListBaseIE):
     _TESTS = [{
         'url': 'http://www.history.com/shows/ancient-aliens',
         'info_dict': {
-            'id': 'SH012427480000',
+            'id': 'SERIES1574',
             'title': 'Ancient Aliens',
             'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
         },
-        'playlist_mincount': 168,
+        'playlist_mincount': 150,
     }]
     _RESOURCE = 'series'
     _ITEMS_KEY = 'episodes'

youtube_dlc/extractor/aljazeera.py

@@ -1,13 +1,16 @@
 from __future__ import unicode_literals
 
+import json
+import re
+
 from .common import InfoExtractor
 
 
 class AlJazeeraIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?:programmes|video)/.*?/(?P<id>[^/]+)\.html'
+    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?P<type>program/[^/]+|(?:feature|video)s)/\d{4}/\d{1,2}/\d{1,2}/(?P<id>[^/?&#]+)'
 
     _TESTS = [{
-        'url': 'http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html',
+        'url': 'https://www.aljazeera.com/program/episode/2014/9/19/deliverance',
         'info_dict': {
             'id': '3792260579001',
             'ext': 'mp4',
@@ -20,14 +23,34 @@ class AlJazeeraIE(InfoExtractor):
         'add_ie': ['BrightcoveNew'],
         'skip': 'Not accessible from Travis CI server',
     }, {
-        'url': 'http://www.aljazeera.com/video/news/2017/05/sierra-leone-709-carat-diamond-auctioned-170511100111930.html',
+        'url': 'https://www.aljazeera.com/videos/2017/5/11/sierra-leone-709-carat-diamond-to-be-auctioned-off',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.aljazeera.com/features/2017/8/21/transforming-pakistans-buses-into-art',
         'only_matching': True,
     }]
-    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/665003303001/default_default/index.html?videoId=%s'
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        program_name = self._match_id(url)
+        post_type, name = re.match(self._VALID_URL, url).groups()
-        webpage = self._download_webpage(url, program_name)
+        post_type = {
-        brightcove_id = self._search_regex(
+            'features': 'post',
-            r'RenderPagesVideo\(\'(.+?)\'', webpage, 'brightcove id')
+            'program': 'episode',
-        return self.url_result(self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id, 'BrightcoveNew', brightcove_id)
+            'videos': 'video',
+        }[post_type.split('/')[0]]
+        video = self._download_json(
+            'https://www.aljazeera.com/graphql', name, query={
+                'operationName': 'SingleArticleQuery',
+                'variables': json.dumps({
+                    'name': name,
+                    'postType': post_type,
+                }),
+            }, headers={
+                'wp-site': 'aje',
+            })['data']['article']['video']
+        video_id = video['id']
+        account_id = video.get('accountId') or '665003303001'
+        player_id = video.get('playerId') or 'BkeSH5BDb'
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
+            'BrightcoveNew', video_id)

youtube_dlc/extractor/americastestkitchen.py

@@ -1,13 +1,16 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
+    int_or_none,
     try_get,
     unified_strdate,
+    unified_timestamp,
 )
 
 
@@ -22,8 +25,8 @@ class AmericasTestKitchenIE(InfoExtractor):
             'ext': 'mp4',
             'description': 'md5:64e606bfee910627efc4b5f050de92b3',
             'thumbnail': r're:^https?://',
-            'timestamp': 1523664000,
+            'timestamp': 1523318400,
-            'upload_date': '20180414',
+            'upload_date': '20180410',
             'release_date': '20180410',
             'series': "America's Test Kitchen",
             'season_number': 18,
@@ -33,6 +36,27 @@ class AmericasTestKitchenIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
+    }, {
+        # Metadata parsing behaves differently for newer episodes (705) as opposed to older episodes (582 above)
+        'url': 'https://www.americastestkitchen.com/episode/705-simple-chicken-dinner',
+        'md5': '06451608c57651e985a498e69cec17e5',
+        'info_dict': {
+            'id': '5fbe8c61bda2010001c6763b',
+            'title': 'Simple Chicken Dinner',
+            'ext': 'mp4',
+            'description': 'md5:eb68737cc2fd4c26ca7db30139d109e7',
+            'thumbnail': r're:^https?://',
+            'timestamp': 1610755200,
+            'upload_date': '20210116',
+            'release_date': '20210116',
+            'series': "America's Test Kitchen",
+            'season_number': 21,
+            'episode': 'Simple Chicken Dinner',
+            'episode_number': 3,
+        },
+        'params': {
+            'skip_download': True,
+        },
     }, {
         'url': 'https://www.americastestkitchen.com/videos/3420-pan-seared-salmon',
         'only_matching': True,
@@ -60,7 +84,76 @@ class AmericasTestKitchenIE(InfoExtractor):
             'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % video['zypeId'],
             'ie_key': 'Zype',
             'description': clean_html(video.get('description')),
+            'timestamp': unified_timestamp(video.get('publishDate')),
             'release_date': unified_strdate(video.get('publishDate')),
+            'episode_number': int_or_none(episode.get('number')),
+            'season_number': int_or_none(episode.get('season')),
             'series': try_get(episode, lambda x: x['show']['title']),
             'episode': episode.get('title'),
         }
+
+
+class AmericasTestKitchenSeasonIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?(?P<show>americastestkitchen|cookscountry)\.com/episodes/browse/season_(?P<id>\d+)'
+    _TESTS = [{
+        # ATK Season
+        'url': 'https://www.americastestkitchen.com/episodes/browse/season_1',
+        'info_dict': {
+            'id': 'season_1',
+            'title': 'Season 1',
+        },
+        'playlist_count': 13,
+    }, {
+        # Cooks Country Season
+        'url': 'https://www.cookscountry.com/episodes/browse/season_12',
+        'info_dict': {
+            'id': 'season_12',
+            'title': 'Season 12',
+        },
+        'playlist_count': 13,
+    }]
+
+    def _real_extract(self, url):
+        show_name, season_number = re.match(self._VALID_URL, url).groups()
+        season_number = int(season_number)
+
+        slug = 'atk' if show_name == 'americastestkitchen' else 'cco'
+
+        season = 'Season %d' % season_number
+
+        season_search = self._download_json(
+            'https://y1fnzxui30-dsn.algolia.net/1/indexes/everest_search_%s_season_desc_production' % slug,
+            season, headers={
+                'Origin': 'https://www.%s.com' % show_name,
+                'X-Algolia-API-Key': '8d504d0099ed27c1b73708d22871d805',
+                'X-Algolia-Application-Id': 'Y1FNZXUI30',
+            }, query={
+                'facetFilters': json.dumps([
+                    'search_season_list:' + season,
+                    'search_document_klass:episode',
+                    'search_show_slug:' + slug,
+                ]),
+                'attributesToRetrieve': 'description,search_%s_episode_number,search_document_date,search_url,title' % slug,
+                'attributesToHighlight': '',
+                'hitsPerPage': 1000,
+            })
+
+        def entries():
+            for episode in (season_search.get('hits') or []):
+                search_url = episode.get('search_url')
+                if not search_url:
+                    continue
+                yield {
+                    '_type': 'url',
+                    'url': 'https://www.%s.com%s' % (show_name, search_url),
+                    'id': try_get(episode, lambda e: e['objectID'].split('_')[-1]),
+                    'title': episode.get('title'),
+                    'description': episode.get('description'),
+                    'timestamp': unified_timestamp(episode.get('search_document_date')),
+                    'season_number': season_number,
+                    'episode_number': int_or_none(episode.get('search_%s_episode_number' % slug)),
+                    'ie_key': AmericasTestKitchenIE.ie_key(),
+                }
+
+        return self.playlist_result(
+            entries(), 'season_%d' % season_number, season)

youtube_dlc/extractor/amp.py

@@ -8,6 +8,7 @@ from ..utils import (
     int_or_none,
     mimetype2ext,
     parse_iso8601,
+    unified_timestamp,
     url_or_none,
 )
 
@@ -88,7 +89,7 @@ class AMPIE(InfoExtractor):
 
         self._sort_formats(formats)
 
-        timestamp = parse_iso8601(item.get('pubDate'), ' ') or parse_iso8601(item.get('dc-date'))
+        timestamp = unified_timestamp(item.get('pubDate'), ' ') or parse_iso8601(item.get('dc-date'))
 
         return {
             'id': video_id,

youtube_dlc/extractor/animelab.py

@@ -0,0 +1,285 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+from ..utils import (
+    ExtractorError,
+    urlencode_postdata,
+    int_or_none,
+    str_or_none,
+    determine_ext,
+)
+
+from ..compat import compat_HTTPError
+
+
+class AnimeLabBaseIE(InfoExtractor):
+    _LOGIN_REQUIRED = True
+    _LOGIN_URL = 'https://www.animelab.com/login'
+    _NETRC_MACHINE = 'animelab'
+
+    def _login(self):
+        def is_logged_in(login_webpage):
+            return 'Sign In' not in login_webpage
+
+        login_page = self._download_webpage(
+            self._LOGIN_URL, None, 'Downloading login page')
+
+        # Check if already logged in
+        if is_logged_in(login_page):
+            return
+
+        (username, password) = self._get_login_info()
+        if username is None and self._LOGIN_REQUIRED:
+            self.raise_login_required('Login is required to access any AnimeLab content')
+
+        login_form = {
+            'email': username,
+            'password': password,
+        }
+
+        try:
+            response = self._download_webpage(
+                self._LOGIN_URL, None, 'Logging in', 'Wrong login info',
+                data=urlencode_postdata(login_form),
+                headers={'Content-Type': 'application/x-www-form-urlencoded'})
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
+                raise ExtractorError('Unable to log in (wrong credentials?)', expected=True)
+            else:
+                raise
+
+        # if login was successful
+        if is_logged_in(response):
+            return
+
+        raise ExtractorError('Unable to login (cannot verify if logged in)')
+
+    def _real_initialize(self):
+        self._login()
+
+
+class AnimeLabIE(AnimeLabBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?animelab\.com/player/(?P<id>[^/]+)'
+
+    # the following tests require authentication, but a free account will suffice
+    # just set 'usenetrc' to true in test/local_parameters.json if you use a .netrc file
+    # or you can set 'username' and 'password' there
+    # the tests also select a specific format so that the same video is downloaded
+    # regardless of whether the user is premium or not (needs testing on a premium account)
+    _TEST = {
+        'url': 'https://www.animelab.com/player/fullmetal-alchemist-brotherhood-episode-42',
+        'md5': '05bde4b91a5d1ff46ef5b94df05b0f7f',
+        'info_dict': {
+            'id': '383',
+            'ext': 'mp4',
+            'display_id': 'fullmetal-alchemist-brotherhood-episode-42',
+            'title': 'Fullmetal Alchemist: Brotherhood - Episode 42 - Signs of a Counteroffensive',
+            'description': 'md5:103eb61dd0a56d3dfc5dbf748e5e83f4',
+            'series': 'Fullmetal Alchemist: Brotherhood',
+            'episode': 'Signs of a Counteroffensive',
+            'episode_number': 42,
+            'duration': 1469,
+            'season': 'Season 1',
+            'season_number': 1,
+            'season_id': '38',
+        },
+        'params': {
+            'format': '[format_id=21711_yeshardsubbed_ja-JP][height=480]',
+        },
+        'skip': 'All AnimeLab content requires authentication',
+    }
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+
+        # unfortunately we can get different URLs for the same formats
+        # e.g. if we are using a "free" account so no dubs available
+        # (so _remove_duplicate_formats is not effective)
+        # so we use a dictionary as a workaround
+        formats = {}
+        for language_option_url in ('https://www.animelab.com/player/%s/subtitles',
+                                    'https://www.animelab.com/player/%s/dubbed'):
+            actual_url = language_option_url % display_id
+            webpage = self._download_webpage(actual_url, display_id, 'Downloading URL ' + actual_url)
+
+            video_collection = self._parse_json(self._search_regex(r'new\s+?AnimeLabApp\.VideoCollection\s*?\((.*?)\);', webpage, 'AnimeLab VideoCollection'), display_id)
+            position = int_or_none(self._search_regex(r'playlistPosition\s*?=\s*?(\d+)', webpage, 'Playlist Position'))
+
+            raw_data = video_collection[position]['videoEntry']
+
+            video_id = str_or_none(raw_data['id'])
+
+            # create a title from many sources (while grabbing other info)
+            # TODO use more fallback sources to get some of these
+            series = raw_data.get('showTitle')
+            video_type = raw_data.get('videoEntryType', {}).get('name')
+            episode_number = raw_data.get('episodeNumber')
+            episode_name = raw_data.get('name')
+
+            title_parts = (series, video_type, episode_number, episode_name)
+            if None not in title_parts:
+                title = '%s - %s %s - %s' % title_parts
+            else:
+                title = episode_name
+
+            description = raw_data.get('synopsis') or self._og_search_description(webpage, default=None)
+
+            duration = int_or_none(raw_data.get('duration'))
+
+            thumbnail_data = raw_data.get('images', [])
+            thumbnails = []
+            for thumbnail in thumbnail_data:
+                for instance in thumbnail['imageInstances']:
+                    image_data = instance.get('imageInfo', {})
+                    thumbnails.append({
+                        'id': str_or_none(image_data.get('id')),
+                        'url': image_data.get('fullPath'),
+                        'width': image_data.get('width'),
+                        'height': image_data.get('height'),
+                    })
+
+            season_data = raw_data.get('season', {}) or {}
+            season = str_or_none(season_data.get('name'))
+            season_number = int_or_none(season_data.get('seasonNumber'))
+            season_id = str_or_none(season_data.get('id'))
+
+            for video_data in raw_data['videoList']:
+                current_video_list = {}
+                current_video_list['language'] = video_data.get('language', {}).get('languageCode')
+
+                is_hardsubbed = video_data.get('hardSubbed')
+
+                for video_instance in video_data['videoInstances']:
+                    httpurl = video_instance.get('httpUrl')
+                    url = httpurl if httpurl else video_instance.get('rtmpUrl')
+                    if url is None:
+                        # this video format is unavailable to the user (not premium etc.)
+                        continue
+
+                    current_format = current_video_list.copy()
+
+                    format_id_parts = []
+
+                    format_id_parts.append(str_or_none(video_instance.get('id')))
+
+                    if is_hardsubbed is not None:
+                        if is_hardsubbed:
+                            format_id_parts.append('yeshardsubbed')
+                        else:
+                            format_id_parts.append('nothardsubbed')
+
+                    format_id_parts.append(current_format['language'])
+
+                    format_id = '_'.join([x for x in format_id_parts if x is not None])
+
+                    ext = determine_ext(url)
+                    if ext == 'm3u8':
+                        for format_ in self._extract_m3u8_formats(
+                                url, video_id, m3u8_id=format_id, fatal=False):
+                            formats[format_['format_id']] = format_
+                        continue
+                    elif ext == 'mpd':
+                        for format_ in self._extract_mpd_formats(
+                                url, video_id, mpd_id=format_id, fatal=False):
+                            formats[format_['format_id']] = format_
+                        continue
+
+                    current_format['url'] = url
+                    quality_data = video_instance.get('videoQuality')
+                    if quality_data:
+                        quality = quality_data.get('name') or quality_data.get('description')
+                    else:
+                        quality = None
+
+                    height = None
+                    if quality:
+                        height = int_or_none(self._search_regex(r'(\d+)p?$', quality, 'Video format height', default=None))
+
+                    if height is None:
+                        self.report_warning('Could not get height of video')
+                    else:
+                        current_format['height'] = height
+                    current_format['format_id'] = format_id
+
+                    formats[current_format['format_id']] = current_format
+
+        formats = list(formats.values())
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+            'series': series,
+            'episode': episode_name,
+            'episode_number': int_or_none(episode_number),
+            'thumbnails': thumbnails,
+            'duration': duration,
+            'formats': formats,
+            'season': season,
+            'season_number': season_number,
+            'season_id': season_id,
+        }
+
+
+class AnimeLabShowsIE(AnimeLabBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?animelab\.com/shows/(?P<id>[^/]+)'
+
+    _TEST = {
+        'url': 'https://www.animelab.com/shows/attack-on-titan',
+        'info_dict': {
+            'id': '45',
+            'title': 'Attack on Titan',
+            'description': 'md5:989d95a2677e9309368d5cf39ba91469',
+        },
+        'playlist_count': 59,
+        'skip': 'All AnimeLab content requires authentication',
+    }
+
+    def _real_extract(self, url):
+        _BASE_URL = 'http://www.animelab.com'
+        _SHOWS_API_URL = '/api/videoentries/show/videos/'
+        display_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, display_id, 'Downloading requested URL')
+
+        show_data_str = self._search_regex(r'({"id":.*}),\svideoEntry', webpage, 'AnimeLab show data')
+        show_data = self._parse_json(show_data_str, display_id)
+
+        show_id = str_or_none(show_data.get('id'))
+        title = show_data.get('name')
+        description = show_data.get('shortSynopsis') or show_data.get('longSynopsis')
+
+        entries = []
+        for season in show_data['seasons']:
+            season_id = season['id']
+            get_data = urlencode_postdata({
+                'seasonId': season_id,
+                'limit': 1000,
+            })
+            # despite using urlencode_postdata, we are sending a GET request
+            target_url = _BASE_URL + _SHOWS_API_URL + show_id + "?" + get_data.decode('utf-8')
+            response = self._download_webpage(
+                target_url,
+                None, 'Season id %s' % season_id)
+
+            season_data = self._parse_json(response, display_id)
+
+            for video_data in season_data['list']:
+                entries.append(self.url_result(
+                    _BASE_URL + '/player/' + video_data['slug'], 'AnimeLab',
+                    str_or_none(video_data.get('id')), video_data.get('name')
+                ))
+
+        return {
+            '_type': 'playlist',
+            'id': show_id,
+            'title': title,
+            'description': description,
+            'entries': entries,
+        }
+
+# TODO implement myqueue

youtube_dlc/extractor/animeondemand.py

@@ -116,8 +116,6 @@ class AnimeOnDemandIE(InfoExtractor):
             r'(?s)<div[^>]+itemprop="description"[^>]*>(.+?)</div>',
             webpage, 'anime description', default=None)
 
-        entries = []
-
         def extract_info(html, video_id, num=None):
             title, description = [None] * 2
             formats = []
@@ -233,7 +231,7 @@ class AnimeOnDemandIE(InfoExtractor):
                 self._sort_formats(info['formats'])
                 f = common_info.copy()
                 f.update(info)
-                entries.append(f)
+                yield f
 
             # Extract teaser/trailer only when full episode is not available
             if not info['formats']:
@@ -247,7 +245,7 @@ class AnimeOnDemandIE(InfoExtractor):
                         'title': m.group('title'),
                         'url': urljoin(url, m.group('href')),
                     })
-                    entries.append(f)
+                    yield f
 
         def extract_episodes(html):
             for num, episode_html in enumerate(re.findall(
@@ -275,7 +273,8 @@ class AnimeOnDemandIE(InfoExtractor):
                     'episode_number': episode_number,
                 }
 
-                extract_entries(episode_html, video_id, common_info)
+                for e in extract_entries(episode_html, video_id, common_info):
+                    yield e
 
         def extract_film(html, video_id):
             common_info = {
@@ -283,11 +282,18 @@ class AnimeOnDemandIE(InfoExtractor):
                 'title': anime_title,
                 'description': anime_description,
             }
-            extract_entries(html, video_id, common_info)
+            for e in extract_entries(html, video_id, common_info):
+                yield e
 
-        extract_episodes(webpage)
+        def entries():
+            has_episodes = False
+            for e in extract_episodes(webpage):
+                has_episodes = True
+                yield e
 
-        if not entries:
+            if not has_episodes:
-            extract_film(webpage, anime_id)
+                for e in extract_film(webpage, anime_id):
+                    yield e
 
-        return self.playlist_result(entries, anime_id, anime_title, anime_description)
+        return self.playlist_result(
+            entries(), anime_id, anime_title, anime_description)

youtube_dlc/extractor/anvato.py

@@ -21,6 +21,16 @@ from ..utils import (
     unsmuggle_url,
 )
 
+# This import causes a ModuleNotFoundError on some systems for unknown reason.
+# See issues:
+# https://github.com/pukkandan/yt-dlp/issues/35
+# https://github.com/ytdl-org/youtube-dl/issues/27449
+# https://github.com/animelover1984/youtube-dl/issues/17
+try:
+    from .anvato_token_generator import NFLTokenGenerator
+except ImportError:
+    NFLTokenGenerator = None
+
 
 def md5_text(s):
     if not isinstance(s, compat_str):
@@ -203,6 +213,10 @@ class AnvatoIE(InfoExtractor):
         'telemundo': 'anvato_mcp_telemundo_web_prod_c5278d51ad46fda4b6ca3d0ea44a7846a054f582'
     }
 
+    _TOKEN_GENERATORS = {
+        'GXvEgwyJeWem8KCYXfeoHWknwP48Mboj': NFLTokenGenerator,
+    }
+
     _API_KEY = '3hwbSuqqT690uxjNYBktSQpa5ZrpYYR0Iofx7NcJHyA'
 
     _ANVP_RE = r'<script[^>]+\bdata-anvp\s*=\s*(["\'])(?P<anvp>(?:(?!\1).)+)\1'
@@ -262,9 +276,12 @@ class AnvatoIE(InfoExtractor):
             'anvrid': anvrid,
             'anvts': server_time,
         }
-        api['anvstk'] = md5_text('%s|%s|%d|%s' % (
+        if self._TOKEN_GENERATORS.get(access_key) is not None:
-            access_key, anvrid, server_time,
+            api['anvstk2'] = self._TOKEN_GENERATORS[access_key].generate(self, access_key, video_id)
-            self._ANVACK_TABLE.get(access_key, self._API_KEY)))
+        else:
+            api['anvstk'] = md5_text('%s|%s|%d|%s' % (
+                access_key, anvrid, server_time,
+                self._ANVACK_TABLE.get(access_key, self._API_KEY)))
 
         return self._download_json(
             video_data_url, video_id, transform_source=strip_jsonp,

youtube_dlc/extractor/aol.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 
 import re
 
-from .common import InfoExtractor
+from .yahoo import YahooIE
 from ..compat import (
     compat_parse_qs,
     compat_urllib_parse_urlparse,
@@ -15,9 +15,9 @@ from ..utils import (
 )
 
 
-class AolIE(InfoExtractor):
+class AolIE(YahooIE):
     IE_NAME = 'aol.com'
-    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>[0-9a-f]+)'
+    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>\d{9}|[0-9a-f]{24}|[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})'
 
     _TESTS = [{
         # video with 5min ID
@@ -76,10 +76,16 @@ class AolIE(InfoExtractor):
     }, {
         'url': 'https://www.aol.jp/video/playlist/5a28e936a1334d000137da0c/5a28f3151e642219fde19831/',
         'only_matching': True,
+    }, {
+        # Yahoo video
+        'url': 'https://www.aol.com/video/play/991e6700-ac02-11ea-99ff-357400036f61/24bbc846-3e30-3c46-915e-fe8ccd7fcc46/',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
+        if '-' in video_id:
+            return self._extract_yahoo_video(video_id, 'us')
 
         response = self._download_json(
             'https://feedapi.b2c.on.aol.com/v1.0/app/videos/aolon/%s/details' % video_id,

youtube_dlc/extractor/applepodcasts.py

@@ -0,0 +1,61 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_podcast_url,
+    int_or_none,
+    parse_iso8601,
+    try_get,
+)
+
+
+class ApplePodcastsIE(InfoExtractor):
+    _VALID_URL = r'https?://podcasts\.apple\.com/(?:[^/]+/)?podcast(?:/[^/]+){1,2}.*?\bi=(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://podcasts.apple.com/us/podcast/207-whitney-webb-returns/id1135137367?i=1000482637777',
+        'md5': 'df02e6acb11c10e844946a39e7222b08',
+        'info_dict': {
+            'id': '1000482637777',
+            'ext': 'mp3',
+            'title': '207 - Whitney Webb Returns',
+            'description': 'md5:13a73bade02d2e43737751e3987e1399',
+            'upload_date': '20200705',
+            'timestamp': 1593921600,
+            'duration': 6425,
+            'series': 'The Tim Dillon Show',
+        }
+    }, {
+        'url': 'https://podcasts.apple.com/podcast/207-whitney-webb-returns/id1135137367?i=1000482637777',
+        'only_matching': True,
+    }, {
+        'url': 'https://podcasts.apple.com/podcast/207-whitney-webb-returns?i=1000482637777',
+        'only_matching': True,
+    }, {
+        'url': 'https://podcasts.apple.com/podcast/id1135137367?i=1000482637777',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        episode_id = self._match_id(url)
+        webpage = self._download_webpage(url, episode_id)
+        ember_data = self._parse_json(self._search_regex(
+            r'id="shoebox-ember-data-store"[^>]*>\s*({.+?})\s*<',
+            webpage, 'ember data'), episode_id)
+        episode = ember_data['data']['attributes']
+        description = episode.get('description') or {}
+
+        series = None
+        for inc in (ember_data.get('included') or []):
+            if inc.get('type') == 'media/podcast':
+                series = try_get(inc, lambda x: x['attributes']['name'])
+
+        return {
+            'id': episode_id,
+            'title': episode['name'],
+            'url': clean_podcast_url(episode['assetUrl']),
+            'description': description.get('standard') or description.get('short'),
+            'timestamp': parse_iso8601(episode.get('releaseDateTime')),
+            'duration': int_or_none(episode.get('durationInMilliseconds'), 1000),
+            'series': series,
+        }

youtube_dlc/extractor/archiveorg.py

@@ -1,27 +1,43 @@
 from __future__ import unicode_literals
 
+import re
+import json
+
 from .common import InfoExtractor
+from ..compat import compat_urllib_parse_unquote_plus
 from ..utils import (
+    KNOWN_EXTENSIONS,
+
+    extract_attributes,
     unified_strdate,
+    unified_timestamp,
     clean_html,
+    dict_get,
+    parse_duration,
+    int_or_none,
+    str_or_none,
+    merge_dicts,
 )
 
 
 class ArchiveOrgIE(InfoExtractor):
     IE_NAME = 'archive.org'
-    IE_DESC = 'archive.org videos'
+    IE_DESC = 'archive.org video and audio'
-    _VALID_URL = r'https?://(?:www\.)?archive\.org/(?:details|embed)/(?P<id>[^/?#]+)(?:[?].*)?$'
+    _VALID_URL = r'https?://(?:www\.)?archive\.org/(?:details|embed)/(?P<id>[^?#]+)(?:[?].*)?$'
     _TESTS = [{
         'url': 'http://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
         'md5': '8af1d4cf447933ed3c7f4871162602db',
         'info_dict': {
             'id': 'XD300-23_68HighlightsAResearchCntAugHumanIntellect',
-            'ext': 'ogg',
+            'ext': 'ogv',
             'title': '1968 Demo - FJCC Conference Presentation Reel #1',
             'description': 'md5:da45c349df039f1cc8075268eb1b5c25',
-            'upload_date': '19681210',
+            'release_date': '19681210',
-            'uploader': 'SRI International'
+            'timestamp': 1268695290,
-        }
+            'upload_date': '20100315',
+            'creator': 'SRI International',
+            'uploader': 'laura@archive.org',
+        },
     }, {
         'url': 'https://archive.org/details/Cops1922',
         'md5': '0869000b4ce265e8ca62738b336b268a',
@@ -29,37 +45,199 @@ class ArchiveOrgIE(InfoExtractor):
             'id': 'Cops1922',
             'ext': 'mp4',
             'title': 'Buster Keaton\'s "Cops" (1922)',
-            'description': 'md5:89e7c77bf5d965dd5c0372cfb49470f6',
+            'description': 'md5:43a603fd6c5b4b90d12a96b921212b9c',
-        }
+            'uploader': 'yorkmba99@hotmail.com',
+            'timestamp': 1387699629,
+            'upload_date': "20131222",
+        },
     }, {
         'url': 'http://archive.org/embed/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
         'only_matching': True,
+    }, {
+        'url': 'https://archive.org/details/Election_Ads',
+        'md5': '284180e857160cf866358700bab668a3',
+        'info_dict': {
+            'id': 'Election_Ads/Commercial-JFK1960ElectionAdCampaignJingle.mpg',
+            'title': 'Commercial-JFK1960ElectionAdCampaignJingle.mpg',
+            'ext': 'mp4',
+        },
+    }, {
+        'url': 'https://archive.org/details/Election_Ads/Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+        'md5': '7915213ef02559b5501fe630e1a53f59',
+        'info_dict': {
+            'id': 'Election_Ads/Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+            'title': 'Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+            'ext': 'mp4',
+            'timestamp': 1205588045,
+            'uploader': 'mikedavisstripmaster@yahoo.com',
+            'description': '1960 Presidential Campaign Election Commercials John F Kennedy, Richard M Nixon',
+            'upload_date': '20080315',
+        },
+    }, {
+        'url': 'https://archive.org/details/gd1977-05-08.shure57.stevenson.29303.flac16',
+        'md5': '7d07ffb42aba6537c28e053efa4b54c9',
+        'info_dict': {
+            'id': 'gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t01.flac',
+            'title': 'Turning',
+            'ext': 'flac',
+        },
+    }, {
+        'url': 'https://archive.org/details/gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t07.flac',
+        'md5': 'a07cd8c6ab4ee1560f8a0021717130f3',
+        'info_dict': {
+            'id': 'gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t07.flac',
+            'title': 'Deal',
+            'ext': 'flac',
+            'timestamp': 1205895624,
+            'uploader': 'mvernon54@yahoo.com',
+            'description': 'md5:6a31f1996db0aa0fc9da6d6e708a1bb0',
+            'upload_date': '20080319',
+            'location': 'Barton Hall - Cornell University',
+        },
+    }, {
+        'url': 'https://archive.org/details/lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik',
+        'md5': '7cb019baa9b332e82ea7c10403acd180',
+        'info_dict': {
+            'id': 'lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/01.01. Bells Of Rostov.mp3',
+            'title': 'Bells Of Rostov',
+            'ext': 'mp3',
+        },
+    }, {
+        'url': 'https://archive.org/details/lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/02.02.+Song+And+Chorus+In+The+Polovetsian+Camp+From+%22Prince+Igor%22+(Act+2%2C+Scene+1).mp3',
+        'md5': '1d0aabe03edca83ca58d9ed3b493a3c3',
+        'info_dict': {
+            'id': 'lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/02.02. Song And Chorus In The Polovetsian Camp From "Prince Igor" (Act 2, Scene 1).mp3',
+            'title': 'Song And Chorus In The Polovetsian Camp From "Prince Igor" (Act 2, Scene 1)',
+            'ext': 'mp3',
+            'timestamp': 1569662587,
+            'uploader': 'associate-joygen-odiongan@archive.org',
+            'description': 'md5:012b2d668ae753be36896f343d12a236',
+            'upload_date': '20190928',
+        },
     }]
 
-    def _real_extract(self, url):
+    @staticmethod
-        video_id = self._match_id(url)
+    def _playlist_data(webpage):
-        webpage = self._download_webpage(
+        element = re.findall(r'''(?xs)
-            'http://archive.org/embed/' + video_id, video_id)
+            <input
-        jwplayer_playlist = self._parse_json(self._search_regex(
+            (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'|))*?
-            r"(?s)Play\('[^']+'\s*,\s*(\[.+\])\s*,\s*{.*?}\)",
+            \s+class=['"]?js-play8-playlist['"]?
-            webpage, 'jwplayer playlist'), video_id)
+            (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'|))*?
-        info = self._parse_jwplayer_data(
+            \s*/>
-            {'playlist': jwplayer_playlist}, video_id, base_url=url)
+        ''', webpage)[0]
 
-        def get_optional(metadata, field):
+        return json.loads(extract_attributes(element)['value'])
-            return metadata.get(field, [None])[0]
+
+    def _real_extract(self, url):
+        video_id = compat_urllib_parse_unquote_plus(self._match_id(url))
+        identifier, entry_id = (video_id.split('/', 1) + [None])[:2]
+
+        # Archive.org metadata API doesn't clearly demarcate playlist entries
+        # or subtitle tracks, so we get them from the embeddable player.
+        embed_page = self._download_webpage(
+            'https://archive.org/embed/' + identifier, identifier)
+        playlist = self._playlist_data(embed_page)
+
+        entries = {}
+        for p in playlist:
+            # If the user specified a playlist entry in the URL, ignore the
+            # rest of the playlist.
+            if entry_id and p['orig'] != entry_id:
+                continue
+
+            entries[p['orig']] = {
+                'formats': [],
+                'thumbnails': [],
+                'artist': p.get('artist'),
+                'track': p.get('title'),
+                'subtitles': {}}
+
+            for track in p.get('tracks', []):
+                if track['kind'] != 'subtitles':
+                    continue
+
+                entries[p['orig']][track['label']] = {
+                    'url': 'https://archive.org/' + track['file'].lstrip('/')}
 
         metadata = self._download_json(
-            'http://archive.org/details/' + video_id, video_id, query={
+            'http://archive.org/metadata/' + identifier, identifier)
-                'output': 'json',
+        m = metadata['metadata']
-            })['metadata']
+        identifier = m['identifier']
-        info.update({
+
-            'title': get_optional(metadata, 'title') or info.get('title'),
+        info = {
-            'description': clean_html(get_optional(metadata, 'description')),
+            'id': identifier,
-        })
+            'title': m['title'],
-        if info.get('_type') != 'playlist':
+            'description': clean_html(m.get('description')),
-            info.update({
+            'uploader': dict_get(m, ['uploader', 'adder']),
-                'uploader': get_optional(metadata, 'creator'),
+            'creator': m.get('creator'),
-                'upload_date': unified_strdate(get_optional(metadata, 'date')),
+            'license': m.get('licenseurl'),
-            })
+            'release_date': unified_strdate(m.get('date')),
+            'timestamp': unified_timestamp(dict_get(m, ['publicdate', 'addeddate'])),
+            'webpage_url': 'https://archive.org/details/' + identifier,
+            'location': m.get('venue'),
+            'release_year': int_or_none(m.get('year'))}
+
+        for f in metadata['files']:
+            if f['name'] in entries:
+                entries[f['name']] = merge_dicts(entries[f['name']], {
+                    'id': identifier + '/' + f['name'],
+                    'title': f.get('title') or f['name'],
+                    'display_id': f['name'],
+                    'description': clean_html(f.get('description')),
+                    'creator': f.get('creator'),
+                    'duration': parse_duration(f.get('length')),
+                    'track_number': int_or_none(f.get('track')),
+                    'album': f.get('album'),
+                    'discnumber': int_or_none(f.get('disc')),
+                    'release_year': int_or_none(f.get('year'))})
+                entry = entries[f['name']]
+            elif f.get('original') in entries:
+                entry = entries[f['original']]
+            else:
+                continue
+
+            if f.get('format') == 'Thumbnail':
+                entry['thumbnails'].append({
+                    'id': f['name'],
+                    'url': 'https://archive.org/download/' + identifier + '/' + f['name'],
+                    'width': int_or_none(f.get('width')),
+                    'height': int_or_none(f.get('width')),
+                    'filesize': int_or_none(f.get('size'))})
+
+            extension = (f['name'].rsplit('.', 1) + [None])[1]
+            if extension in KNOWN_EXTENSIONS:
+                entry['formats'].append({
+                    'url': 'https://archive.org/download/' + identifier + '/' + f['name'],
+                    'format': f.get('format'),
+                    'width': int_or_none(f.get('width')),
+                    'height': int_or_none(f.get('height')),
+                    'filesize': int_or_none(f.get('size')),
+                    'protocol': 'https'})
+
+        # Sort available formats by filesize
+        for entry in entries.values():
+            entry['formats'] = list(sorted(entry['formats'], key=lambda x: x.get('filesize', -1)))
+
+        if len(entries) == 1:
+            # If there's only one item, use it as the main info dict
+            only_video = entries[list(entries.keys())[0]]
+            if entry_id:
+                info = merge_dicts(only_video, info)
+            else:
+                info = merge_dicts(info, only_video)
+        else:
+            # Otherwise, we have a playlist.
+            info['_type'] = 'playlist'
+            info['entries'] = list(entries.values())
+
+        if metadata.get('reviews'):
+            info['comments'] = []
+            for review in metadata['reviews']:
+                info['comments'].append({
+                    'id': review.get('review_id'),
+                    'author': review.get('reviewer'),
+                    'text': str_or_none(review.get('reviewtitle'), '') + '\n\n' + review.get('reviewbody'),
+                    'timestamp': unified_timestamp(review.get('createdate')),
+                    'parent': 'root'})
+
         return info

youtube_dlc/extractor/ard.py

@@ -226,13 +226,13 @@ class ARDMediathekIE(ARDMediathekBaseIE):
             if doc.tag == 'rss':
                 return GenericIE()._extract_rss(url, video_id, doc)
 
-        title = self._html_search_regex(
+        title = self._og_search_title(webpage, default=None) or self._html_search_regex(
             [r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
              r'<meta name="dcterms\.title" content="(.*?)"/>',
              r'<h4 class="headline">(.*?)</h4>',
              r'<title[^>]*>(.*?)</title>'],
             webpage, 'title')
-        description = self._html_search_meta(
+        description = self._og_search_description(webpage, default=None) or self._html_search_meta(
             'dcterms.abstract', webpage, 'description', default=None)
         if description is None:
             description = self._html_search_meta(
@@ -289,18 +289,18 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
 
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?:video-?)?(?P<id>[0-9]+))\.html'
     _TESTS = [{
-        # available till 14.02.2019
+        # available till 7.01.2022
-        'url': 'http://www.daserste.de/information/talk/maischberger/videos/das-groko-drama-zerlegen-sich-die-volksparteien-video-102.html',
+        'url': 'https://www.daserste.de/information/talk/maischberger/videos/maischberger-die-woche-video100.html',
-        'md5': '8e4ec85f31be7c7fc08a26cdbc5a1f49',
+        'md5': '867d8aa39eeaf6d76407c5ad1bb0d4c1',
         'info_dict': {
-            'display_id': 'das-groko-drama-zerlegen-sich-die-volksparteien-video',
+            'display_id': 'maischberger-die-woche',
-            'id': '102',
+            'id': '100',
             'ext': 'mp4',
-            'duration': 4435.0,
+            'duration': 3687.0,
-            'title': 'Das GroKo-Drama: Zerlegen sich die Volksparteien?',
+            'title': 'maischberger. die woche vom 7. Januar 2021',
-            'upload_date': '20180214',
+            'upload_date': '20210107',
             'thumbnail': r're:^https?://.*\.jpg$',
         },
     }, {
@@ -355,17 +355,17 @@ class ARDIE(InfoExtractor):
 class ARDBetaMediathekIE(ARDMediathekBaseIE):
     _VALID_URL = r'https://(?:(?:beta|www)\.)?ardmediathek\.de/(?P<client>[^/]+)/(?P<mode>player|live|video|sendung|sammlung)/(?P<display_id>(?:[^/]+/)*)(?P<video_id>[a-zA-Z0-9]+)'
     _TESTS = [{
-        'url': 'https://ardmediathek.de/ard/video/die-robuste-roswita/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
+        'url': 'https://www.ardmediathek.de/mdr/video/die-robuste-roswita/Y3JpZDovL21kci5kZS9iZWl0cmFnL2Ntcy84MWMxN2MzZC0wMjkxLTRmMzUtODk4ZS0wYzhlOWQxODE2NGI/',
-        'md5': 'dfdc87d2e7e09d073d5a80770a9ce88f',
+        'md5': 'a1dc75a39c61601b980648f7c9f9f71d',
         'info_dict': {
             'display_id': 'die-robuste-roswita',
-            'id': '70153354',
+            'id': '78566716',
             'title': 'Die robuste Roswita',
-            'description': r're:^Der Mord.*trüber ist als die Ilm.',
+            'description': r're:^Der Mord.*totgeglaubte Ehefrau Roswita',
             'duration': 5316,
-            'thumbnail': 'https://img.ardmediathek.de/standard/00/70/15/33/90/-1852531467/16x9/960?mandant=ard',
+            'thumbnail': 'https://img.ardmediathek.de/standard/00/78/56/67/84/575672121/16x9/960?mandant=ard',
-            'timestamp': 1577047500,
+            'timestamp': 1596658200,
-            'upload_date': '20191222',
+            'upload_date': '20200805',
             'ext': 'mp4',
         },
     }, {

youtube_dlc/extractor/audius.py

@@ -0,0 +1,247 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import random
+import re
+
+from .common import InfoExtractor
+from ..utils import ExtractorError, try_get, compat_str, str_or_none
+from ..compat import compat_urllib_parse_unquote
+
+
+class AudiusBaseIE(InfoExtractor):
+    _API_BASE = None
+    _API_V = '/v1'
+
+    def _get_response_data(self, response):
+        if isinstance(response, dict):
+            response_data = response.get('data')
+            if response_data is not None:
+                return response_data
+            if len(response) == 1 and 'message' in response:
+                raise ExtractorError('API error: %s' % response['message'],
+                                     expected=True)
+        raise ExtractorError('Unexpected API response')
+
+    def _select_api_base(self):
+        """Selecting one of the currently available API hosts"""
+        response = super(AudiusBaseIE, self)._download_json(
+            'https://api.audius.co/', None,
+            note='Requesting available API hosts',
+            errnote='Unable to request available API hosts')
+        hosts = self._get_response_data(response)
+        if isinstance(hosts, list):
+            self._API_BASE = random.choice(hosts)
+            return
+        raise ExtractorError('Unable to get available API hosts')
+
+    @staticmethod
+    def _prepare_url(url, title):
+        """
+        Audius removes forward slashes from the uri, but leaves backslashes.
+        The problem is that the current version of Chrome replaces backslashes
+        in the address bar with a forward slashes, so if you copy the link from
+        there and paste it into youtube-dl, you won't be able to download
+        anything from this link, since the Audius API won't be able to resolve
+        this url
+        """
+        url = compat_urllib_parse_unquote(url)
+        title = compat_urllib_parse_unquote(title)
+        if '/' in title or '%2F' in title:
+            fixed_title = title.replace('/', '%5C').replace('%2F', '%5C')
+            return url.replace(title, fixed_title)
+        return url
+
+    def _api_request(self, path, item_id=None, note='Downloading JSON metadata',
+                     errnote='Unable to download JSON metadata',
+                     expected_status=None):
+        if self._API_BASE is None:
+            self._select_api_base()
+        try:
+            response = super(AudiusBaseIE, self)._download_json(
+                '%s%s%s' % (self._API_BASE, self._API_V, path), item_id, note=note,
+                errnote=errnote, expected_status=expected_status)
+        except ExtractorError as exc:
+            # some of Audius API hosts may not work as expected and return HTML
+            if 'Failed to parse JSON' in compat_str(exc):
+                raise ExtractorError('An error occurred while receiving data. Try again',
+                                     expected=True)
+            raise exc
+        return self._get_response_data(response)
+
+    def _resolve_url(self, url, item_id):
+        return self._api_request('/resolve?url=%s' % url, item_id,
+                                 expected_status=404)
+
+
+class AudiusIE(AudiusBaseIE):
+    _VALID_URL = r'''(?x)https?://(?:www\.)?(?:audius\.co/(?P<uploader>[\w\d-]+)(?!/album|/playlist)/(?P<title>\S+))'''
+    IE_DESC = 'Audius.co'
+    _TESTS = [
+        {
+            # URL from Chrome address bar which replace backslash to forward slash
+            'url': 'https://audius.co/test_acc/t%D0%B5%D0%B5%D0%B5est-1.%5E_%7B%7D/%22%3C%3E.%E2%84%96~%60-198631',
+            'md5': '92c35d3e754d5a0f17eef396b0d33582',
+            'info_dict': {
+                'id': 'xd8gY',
+                'title': '''Tеееest/ 1.!@#$%^&*()_+=[]{};'\\\":<>,.?/№~`''',
+                'ext': 'mp3',
+                'description': 'Description',
+                'duration': 30,
+                'track': '''Tеееest/ 1.!@#$%^&*()_+=[]{};'\\\":<>,.?/№~`''',
+                'artist': 'test',
+                'genre': 'Electronic',
+                'thumbnail': r're:https?://.*\.jpg',
+                'view_count': int,
+                'like_count': int,
+                'repost_count': int,
+            }
+        },
+        {
+            # Regular track
+            'url': 'https://audius.co/voltra/radar-103692',
+            'md5': '491898a0a8de39f20c5d6a8a80ab5132',
+            'info_dict': {
+                'id': 'KKdy2',
+                'title': 'RADAR',
+                'ext': 'mp3',
+                'duration': 318,
+                'track': 'RADAR',
+                'artist': 'voltra',
+                'genre': 'Trance',
+                'thumbnail': r're:https?://.*\.jpg',
+                'view_count': int,
+                'like_count': int,
+                'repost_count': int,
+            }
+        },
+    ]
+
+    _ARTWORK_MAP = {
+        "150x150": 150,
+        "480x480": 480,
+        "1000x1000": 1000
+    }
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        track_id = try_get(mobj, lambda x: x.group('track_id'))
+        if track_id is None:
+            title = mobj.group('title')
+            # uploader = mobj.group('uploader')
+            url = self._prepare_url(url, title)
+            track_data = self._resolve_url(url, title)
+        else:  # API link
+            title = None
+            # uploader = None
+            track_data = self._api_request('/tracks/%s' % track_id, track_id)
+
+        if not isinstance(track_data, dict):
+            raise ExtractorError('Unexpected API response')
+
+        track_id = track_data.get('id')
+        if track_id is None:
+            raise ExtractorError('Unable to get ID of the track')
+
+        artworks_data = track_data.get('artwork')
+        thumbnails = []
+        if isinstance(artworks_data, dict):
+            for quality_key, thumbnail_url in artworks_data.items():
+                thumbnail = {
+                    "url": thumbnail_url
+                }
+                quality_code = self._ARTWORK_MAP.get(quality_key)
+                if quality_code is not None:
+                    thumbnail['preference'] = quality_code
+                thumbnails.append(thumbnail)
+
+        return {
+            'id': track_id,
+            'title': track_data.get('title', title),
+            'url': '%s/v1/tracks/%s/stream' % (self._API_BASE, track_id),
+            'ext': 'mp3',
+            'description': track_data.get('description'),
+            'duration': track_data.get('duration'),
+            'track': track_data.get('title'),
+            'artist': try_get(track_data, lambda x: x['user']['name'], compat_str),
+            'genre': track_data.get('genre'),
+            'thumbnails': thumbnails,
+            'view_count': track_data.get('play_count'),
+            'like_count': track_data.get('favorite_count'),
+            'repost_count': track_data.get('repost_count'),
+        }
+
+
+class AudiusTrackIE(AudiusIE):
+    _VALID_URL = r'''(?x)(?:audius:)(?:https?://(?:www\.)?.+/v1/tracks/)?(?P<track_id>\w+)'''
+    IE_NAME = 'audius:track'
+    IE_DESC = 'Audius track ID or API link. Prepend with "audius:"'
+    _TESTS = [
+        {
+            'url': 'audius:9RWlo',
+            'only_matching': True
+        },
+        {
+            'url': 'audius:http://discoveryprovider.audius.prod-us-west-2.staked.cloud/v1/tracks/9RWlo',
+            'only_matching': True
+        },
+    ]
+
+
+class AudiusPlaylistIE(AudiusBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?audius\.co/(?P<uploader>[\w\d-]+)/(?:album|playlist)/(?P<title>\S+)'
+    IE_NAME = 'audius:playlist'
+    IE_DESC = 'Audius.co playlists'
+    _TEST = {
+        'url': 'https://audius.co/test_acc/playlist/test-playlist-22910',
+        'info_dict': {
+            'id': 'DNvjN',
+            'title': 'test playlist',
+            'description': 'Test description\n\nlol',
+        },
+        'playlist_count': 175,
+    }
+
+    def _build_playlist(self, tracks):
+        entries = []
+        for track in tracks:
+            if not isinstance(track, dict):
+                raise ExtractorError('Unexpected API response')
+            track_id = str_or_none(track.get('id'))
+            if not track_id:
+                raise ExtractorError('Unable to get track ID from playlist')
+            entries.append(self.url_result(
+                'audius:%s' % track_id,
+                ie=AudiusTrackIE.ie_key(), video_id=track_id))
+        return entries
+
+    def _real_extract(self, url):
+        self._select_api_base()
+        mobj = re.match(self._VALID_URL, url)
+        title = mobj.group('title')
+        # uploader = mobj.group('uploader')
+        url = self._prepare_url(url, title)
+        playlist_response = self._resolve_url(url, title)
+
+        if not isinstance(playlist_response, list) or len(playlist_response) != 1:
+            raise ExtractorError('Unexpected API response')
+
+        playlist_data = playlist_response[0]
+        if not isinstance(playlist_data, dict):
+            raise ExtractorError('Unexpected API response')
+
+        playlist_id = playlist_data.get('id')
+        if playlist_id is None:
+            raise ExtractorError('Unable to get playlist ID')
+
+        playlist_tracks = self._api_request(
+            '/playlists/%s/tracks' % playlist_id,
+            title, note='Downloading playlist tracks metadata',
+            errnote='Unable to download playlist tracks metadata')
+        if not isinstance(playlist_tracks, list):
+            raise ExtractorError('Unexpected API response')
+
+        entries = self._build_playlist(playlist_tracks)
+        return self.playlist_result(entries, playlist_id,
+                                    playlist_data.get('playlist_name', title),
+                                    playlist_data.get('description'))

youtube_dlc/extractor/awaan.py

@@ -48,6 +48,7 @@ class AWAANBaseIE(InfoExtractor):
             'duration': int_or_none(video_data.get('duration')),
             'timestamp': parse_iso8601(video_data.get('create_time'), ' '),
             'is_live': is_live,
+            'uploader_id': video_data.get('user_id'),
         }
 
 
@@ -107,6 +108,7 @@ class AWAANLiveIE(AWAANBaseIE):
             'title': 're:Dubai Al Oula [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
             'upload_date': '20150107',
             'timestamp': 1420588800,
+            'uploader_id': '71',
         },
         'params': {
             # m3u8 download

youtube_dlc/extractor/azmedien.py

@@ -47,7 +47,7 @@ class AZMedienIE(InfoExtractor):
         'url': 'https://www.telebaern.tv/telebaern-news/montag-1-oktober-2018-ganze-sendung-133531189#video=0_7xjo9lf1',
         'only_matching': True
     }]
-    _API_TEMPL = 'https://www.%s/api/pub/gql/%s/NewsArticleTeaser/cb9f2f81ed22e9b47f4ca64ea3cc5a5d13e88d1d'
+    _API_TEMPL = 'https://www.%s/api/pub/gql/%s/NewsArticleTeaser/a4016f65fe62b81dc6664dd9f4910e4ab40383be'
     _PARTNER_ID = '1719221'
 
     def _real_extract(self, url):

youtube_dlc/extractor/bfmtv.py

@@ -0,0 +1,103 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import extract_attributes
+
+
+class BFMTVBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?bfmtv\.com/'
+    _VALID_URL_TMPL = _VALID_URL_BASE + r'(?:[^/]+/)*[^/?&#]+_%s[A-Z]-(?P<id>\d{12})\.html'
+    _VIDEO_BLOCK_REGEX = r'(<div[^>]+class="video_block"[^>]*>)'
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
+
+    def _brightcove_url_result(self, video_id, video_block):
+        account_id = video_block.get('accountid') or '876450612001'
+        player_id = video_block.get('playerid') or 'I2qBTln4u'
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
+            'BrightcoveNew', video_id)
+
+
+class BFMTVIE(BFMTVBaseIE):
+    IE_NAME = 'bfmtv'
+    _VALID_URL = BFMTVBaseIE._VALID_URL_TMPL % 'V'
+    _TESTS = [{
+        'url': 'https://www.bfmtv.com/politique/emmanuel-macron-l-islam-est-une-religion-qui-vit-une-crise-aujourd-hui-partout-dans-le-monde_VN-202010020146.html',
+        'info_dict': {
+            'id': '6196747868001',
+            'ext': 'mp4',
+            'title': 'Emmanuel Macron: "L\'Islam est une religion qui vit une crise aujourd’hui, partout dans le monde"',
+            'description': 'Le Président s\'exprime sur la question du séparatisme depuis les Mureaux, dans les Yvelines.',
+            'uploader_id': '876450610001',
+            'upload_date': '20201002',
+            'timestamp': 1601629620,
+        },
+    }]
+
+    def _real_extract(self, url):
+        bfmtv_id = self._match_id(url)
+        webpage = self._download_webpage(url, bfmtv_id)
+        video_block = extract_attributes(self._search_regex(
+            self._VIDEO_BLOCK_REGEX, webpage, 'video block'))
+        return self._brightcove_url_result(video_block['videoid'], video_block)
+
+
+class BFMTVLiveIE(BFMTVIE):
+    IE_NAME = 'bfmtv:live'
+    _VALID_URL = BFMTVBaseIE._VALID_URL_BASE + '(?P<id>(?:[^/]+/)?en-direct)'
+    _TESTS = [{
+        'url': 'https://www.bfmtv.com/en-direct/',
+        'info_dict': {
+            'id': '5615950982001',
+            'ext': 'mp4',
+            'title': r're:^le direct BFMTV WEB \d{4}-\d{2}-\d{2} \d{2}:\d{2}$',
+            'uploader_id': '876450610001',
+            'upload_date': '20171018',
+            'timestamp': 1508329950,
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        'url': 'https://www.bfmtv.com/economie/en-direct/',
+        'only_matching': True,
+    }]
+
+
+class BFMTVArticleIE(BFMTVBaseIE):
+    IE_NAME = 'bfmtv:article'
+    _VALID_URL = BFMTVBaseIE._VALID_URL_TMPL % 'A'
+    _TESTS = [{
+        'url': 'https://www.bfmtv.com/sante/covid-19-un-responsable-de-l-institut-pasteur-se-demande-quand-la-france-va-se-reconfiner_AV-202101060198.html',
+        'info_dict': {
+            'id': '202101060198',
+            'title': 'Covid-19: un responsable de l\'Institut Pasteur se demande "quand la France va se reconfiner"',
+            'description': 'md5:947974089c303d3ac6196670ae262843',
+        },
+        'playlist_count': 2,
+    }, {
+        'url': 'https://www.bfmtv.com/international/pour-bolsonaro-le-bresil-est-en-faillite-mais-il-ne-peut-rien-faire_AD-202101060232.html',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.bfmtv.com/sante/covid-19-oui-le-vaccin-de-pfizer-distribue-en-france-a-bien-ete-teste-sur-des-personnes-agees_AN-202101060275.html',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        bfmtv_id = self._match_id(url)
+        webpage = self._download_webpage(url, bfmtv_id)
+
+        entries = []
+        for video_block_el in re.findall(self._VIDEO_BLOCK_REGEX, webpage):
+            video_block = extract_attributes(video_block_el)
+            video_id = video_block.get('videoid')
+            if not video_id:
+                continue
+            entries.append(self._brightcove_url_result(video_id, video_block))
+
+        return self.playlist_result(
+            entries, bfmtv_id, self._og_search_title(webpage, fatal=False),
+            self._html_search_meta(['og:description', 'description'], webpage))

youtube_dlc/extractor/bibeltv.py

@@ -0,0 +1,30 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class BibelTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?bibeltv\.de/mediathek/videos/(?:crn/)?(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://www.bibeltv.de/mediathek/videos/329703-sprachkurs-in-malaiisch',
+        'md5': '252f908192d611de038b8504b08bf97f',
+        'info_dict': {
+            'id': 'ref:329703',
+            'ext': 'mp4',
+            'title': 'Sprachkurs in Malaiisch',
+            'description': 'md5:3e9f197d29ee164714e67351cf737dfe',
+            'timestamp': 1608316701,
+            'uploader_id': '5840105145001',
+            'upload_date': '20201218',
+        }
+    }, {
+        'url': 'https://www.bibeltv.de/mediathek/videos/crn/326374',
+        'only_matching': True,
+    }]
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/5840105145001/default_default/index.html?videoId=ref:%s'
+
+    def _real_extract(self, url):
+        crn_id = self._match_id(url)
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % crn_id, 'BrightcoveNew')

youtube_dlc/extractor/bilibili.py

@@ -2,9 +2,10 @@
 from __future__ import unicode_literals
 
 import hashlib
+import json
 import re
 
-from .common import InfoExtractor
+from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
     compat_parse_qs,
     compat_urlparse,
@@ -32,13 +33,14 @@ class BiliBiliIE(InfoExtractor):
                             (?:
                                 video/[aA][vV]|
                                 anime/(?P<anime_id>\d+)/play\#
-                            )(?P<id_bv>\d+)|
+                            )(?P<id>\d+)|
-                            video/[bB][vV](?P<id>[^/?#&]+)
+                            video/[bB][vV](?P<id_bv>[^/?#&]+)
                         )
+                        (?:/?\?p=(?P<page>\d+))?
                     '''
 
     _TESTS = [{
-        'url': 'http://www.bilibili.tv/video/av1074402/',
+        'url': 'http://www.bilibili.com/video/av1074402/',
         'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
         'info_dict': {
             'id': '1074402',
@@ -56,6 +58,10 @@ class BiliBiliIE(InfoExtractor):
         # Tested in BiliBiliBangumiIE
         'url': 'http://bangumi.bilibili.com/anime/1869/play#40062',
         'only_matching': True,
+    }, {
+        # bilibili.tv
+        'url': 'http://www.bilibili.tv/video/av1074402/',
+        'only_matching': True,
     }, {
         'url': 'http://bangumi.bilibili.com/anime/5802/play#100643',
         'md5': '3f721ad1e75030cc06faf73587cfec57',
@@ -124,12 +130,20 @@ class BiliBiliIE(InfoExtractor):
         url, smuggled_data = unsmuggle_url(url, {})
 
         mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id') or mobj.group('id_bv')
+        video_id = mobj.group('id_bv') or mobj.group('id')
+
+        av_id, bv_id = self._get_video_id_set(video_id, mobj.group('id_bv') is not None)
+        video_id = av_id
+
         anime_id = mobj.group('anime_id')
+        page_id = mobj.group('page')
         webpage = self._download_webpage(url, video_id)
 
         if 'anime/' not in url:
             cid = self._search_regex(
+                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + str(page_id), webpage, 'cid',
+                default=None
+            ) or self._search_regex(
                 r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
                 default=None
             ) or compat_parse_qs(self._search_regex(
@@ -207,9 +221,9 @@ class BiliBiliIE(InfoExtractor):
             break
 
         title = self._html_search_regex(
-            ('<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
+            (r'<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
-             '(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
+             r'(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
-            group='title')
+            group='title') + ('_p' + str(page_id) if page_id is not None else '')
         description = self._html_search_meta('description', webpage)
         timestamp = unified_timestamp(self._html_search_regex(
             r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time',
@@ -219,7 +233,8 @@ class BiliBiliIE(InfoExtractor):
 
         # TODO 'view_count' requires deobfuscating Javascript
         info = {
-            'id': video_id,
+            'id': str(video_id) if page_id is None else '%s_p%s' % (video_id, page_id),
+            'cid': cid,
             'title': title,
             'description': description,
             'timestamp': timestamp,
@@ -235,27 +250,134 @@ class BiliBiliIE(InfoExtractor):
                 'uploader': uploader_mobj.group('name'),
                 'uploader_id': uploader_mobj.group('id'),
             })
+
         if not info.get('uploader'):
             info['uploader'] = self._html_search_meta(
                 'author', webpage, 'uploader', default=None)
 
+        comments = None
+        if self._downloader.params.get('getcomments', False):
+            comments = self._get_all_comment_pages(video_id)
+
+        raw_danmaku = self._get_raw_danmaku(video_id, cid)
+
+        raw_tags = self._get_tags(video_id)
+        tags = list(map(lambda x: x['tag_name'], raw_tags))
+
+        top_level_info = {
+            'raw_danmaku': raw_danmaku,
+            'comments': comments,
+            'comment_count': len(comments) if comments is not None else None,
+            'tags': tags,
+            'raw_tags': raw_tags,
+        }
+
+        '''
+        # Requires https://github.com/m13253/danmaku2ass which is licenced under GPL3
+        # See https://github.com/animelover1984/youtube-dl
+        danmaku = NiconicoIE.CreateDanmaku(raw_danmaku, commentType='Bilibili', x=1024, y=576)
+        entries[0]['subtitles'] = {
+            'danmaku': [{
+                'ext': 'ass',
+                'data': danmaku
+            }]
+        }
+        '''
+
         for entry in entries:
             entry.update(info)
 
         if len(entries) == 1:
+            entries[0].update(top_level_info)
             return entries[0]
         else:
             for idx, entry in enumerate(entries):
                 entry['id'] = '%s_part%d' % (video_id, (idx + 1))
 
-            return {
+            global_info = {
                 '_type': 'multi_video',
                 'id': video_id,
+                'bv_id': bv_id,
                 'title': title,
                 'description': description,
                 'entries': entries,
             }
 
+            global_info.update(info)
+            global_info.update(top_level_info)
+
+            return global_info
+
+    def _get_video_id_set(self, id, is_bv):
+        query = {'bvid': id} if is_bv else {'aid': id}
+        response = self._download_json(
+            "http://api.bilibili.cn/x/web-interface/view",
+            id, query=query,
+            note='Grabbing original ID via API')
+
+        if response['code'] == -400:
+            raise ExtractorError('Video ID does not exist', expected=True, video_id=id)
+        elif response['code'] != 0:
+            raise ExtractorError('Unknown error occurred during API check (code %s)' % response['code'], expected=True, video_id=id)
+        return (response['data']['aid'], response['data']['bvid'])
+
+    # recursive solution to getting every page of comments for the video
+    # we can stop when we reach a page without any comments
+    def _get_all_comment_pages(self, video_id, commentPageNumber=0):
+        comment_url = "https://api.bilibili.com/x/v2/reply?jsonp=jsonp&pn=%s&type=1&oid=%s&sort=2&_=1567227301685" % (commentPageNumber, video_id)
+        json_str = self._download_webpage(
+            comment_url, video_id,
+            note='Extracting comments from page %s' % (commentPageNumber))
+        replies = json.loads(json_str)['data']['replies']
+        if replies is None:
+            return []
+        return self._get_all_children(replies) + self._get_all_comment_pages(video_id, commentPageNumber + 1)
+
+    # extracts all comments in the tree
+    def _get_all_children(self, replies):
+        if replies is None:
+            return []
+
+        ret = []
+        for reply in replies:
+            author = reply['member']['uname']
+            author_id = reply['member']['mid']
+            id = reply['rpid']
+            text = reply['content']['message']
+            timestamp = reply['ctime']
+            parent = reply['parent'] if reply['parent'] != 0 else 'root'
+
+            comment = {
+                "author": author,
+                "author_id": author_id,
+                "id": id,
+                "text": text,
+                "timestamp": timestamp,
+                "parent": parent,
+            }
+            ret.append(comment)
+
+            # from the JSON, the comment structure seems arbitrarily deep, but I could be wrong.
+            # Regardless, this should work.
+            ret += self._get_all_children(reply['replies'])
+
+        return ret
+
+    def _get_raw_danmaku(self, video_id, cid):
+        # This will be useful if I decide to scrape all pages instead of doing them individually
+        # cid_url = "https://www.bilibili.com/widget/getPageList?aid=%s" % (video_id)
+        # cid_str = self._download_webpage(cid_url, video_id, note=False)
+        # cid = json.loads(cid_str)[0]['cid']
+
+        danmaku_url = "https://comment.bilibili.com/%s.xml" % (cid)
+        danmaku = self._download_webpage(danmaku_url, video_id, note='Downloading danmaku comments')
+        return danmaku
+
+    def _get_tags(self, video_id):
+        tags_url = "https://api.bilibili.com/x/tag/archive/tags?aid=%s" % (video_id)
+        tags_json = self._download_json(tags_url, video_id, note='Downloading tags')
+        return tags_json['data']
+
 
 class BiliBiliBangumiIE(InfoExtractor):
     _VALID_URL = r'https?://bangumi\.bilibili\.com/anime/(?P<id>\d+)'
@@ -324,6 +446,73 @@ class BiliBiliBangumiIE(InfoExtractor):
             season_info.get('bangumi_title'), season_info.get('evaluate'))
 
 
+class BilibiliChannelIE(InfoExtractor):
+    _VALID_URL = r'https?://space.bilibili\.com/(?P<id>\d+)'
+    # May need to add support for pagination? Need to find a user with many video uploads to test
+    _API_URL = "https://api.bilibili.com/x/space/arc/search?mid=%s&pn=1&ps=25&jsonp=jsonp"
+    _TEST = {}  # TODO: Add tests
+
+    def _real_extract(self, url):
+        list_id = self._match_id(url)
+        json_str = self._download_webpage(self._API_URL % list_id, "None")
+
+        json_parsed = json.loads(json_str)
+        entries = [{
+            '_type': 'url',
+            'ie_key': BiliBiliIE.ie_key(),
+            'url': ('https://www.bilibili.com/video/%s' %
+                    entry['bvid']),
+            'id': entry['bvid'],
+        } for entry in json_parsed['data']['list']['vlist']]
+
+        return {
+            '_type': 'playlist',
+            'id': list_id,
+            'entries': entries
+        }
+
+
+class BiliBiliSearchIE(SearchInfoExtractor):
+    IE_DESC = 'Bilibili video search, "bilisearch" keyword'
+    _MAX_RESULTS = 100000
+    _SEARCH_KEY = 'bilisearch'
+    MAX_NUMBER_OF_RESULTS = 1000
+
+    def _get_n_results(self, query, n):
+        """Get a specified number of results for a query"""
+
+        entries = []
+        pageNumber = 0
+        while True:
+            pageNumber += 1
+            # FIXME
+            api_url = "https://api.bilibili.com/x/web-interface/search/type?context=&page=%s&order=pubdate&keyword=%s&duration=0&tids_2=&__refresh__=true&search_type=video&tids=0&highlight=1" % (pageNumber, query)
+            json_str = self._download_webpage(
+                api_url, "None", query={"Search_key": query},
+                note='Extracting results from page %s' % pageNumber)
+            data = json.loads(json_str)['data']
+
+            # FIXME: this is hideous
+            if "result" not in data:
+                return {
+                    '_type': 'playlist',
+                    'id': query,
+                    'entries': entries[:n]
+                }
+
+            videos = data['result']
+            for video in videos:
+                e = self.url_result(video['arcurl'], 'BiliBili', str(video['aid']))
+                entries.append(e)
+
+            if(len(entries) >= n or len(videos) >= BiliBiliSearchIE.MAX_NUMBER_OF_RESULTS):
+                return {
+                    '_type': 'playlist',
+                    'id': query,
+                    'entries': entries[:n]
+                }
+
+
 class BilibiliAudioBaseIE(InfoExtractor):
     def _call_api(self, path, sid, query=None):
         if not query:

youtube_dlc/extractor/bleacherreport.py

@@ -90,13 +90,19 @@ class BleacherReportCMSIE(AMPIE):
     _VALID_URL = r'https?://(?:www\.)?bleacherreport\.com/video_embed\?id=(?P<id>[0-9a-f-]{36}|\d{5})'
     _TESTS = [{
         'url': 'http://bleacherreport.com/video_embed?id=8fd44c2f-3dc5-4821-9118-2c825a98c0e1&library=video-cms',
-        'md5': '2e4b0a997f9228ffa31fada5c53d1ed1',
+        'md5': '670b2d73f48549da032861130488c681',
         'info_dict': {
             'id': '8fd44c2f-3dc5-4821-9118-2c825a98c0e1',
-            'ext': 'flv',
+            'ext': 'mp4',
             'title': 'Cena vs. Rollins Would Expose the Heavyweight Division',
             'description': 'md5:984afb4ade2f9c0db35f3267ed88b36e',
+            'upload_date': '20150723',
+            'timestamp': 1437679032,
+
         },
+        'expected_warnings': [
+            'Unable to download f4m manifest'
+        ]
     }]
 
     def _real_extract(self, url):

youtube_dlc/extractor/bravotv.py

@@ -12,7 +12,7 @@ from ..utils import (
 
 
 class BravoTVIE(AdobePassIE):
-    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?P<req_id>bravotv|oxygen)\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'https://www.bravotv.com/top-chef/season-16/episode-15/videos/the-top-chef-season-16-winner-is',
         'md5': 'e34684cfea2a96cd2ee1ef3a60909de9',
@@ -28,10 +28,13 @@ class BravoTVIE(AdobePassIE):
     }, {
         'url': 'http://www.bravotv.com/below-deck/season-3/ep-14-reunion-part-1',
         'only_matching': True,
+    }, {
+        'url': 'https://www.oxygen.com/in-ice-cold-blood/season-2/episode-16/videos/handling-the-horwitz-house-after-the-murder-season-2',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
+        site, display_id = re.match(self._VALID_URL, url).groups()
         webpage = self._download_webpage(url, display_id)
         settings = self._parse_json(self._search_regex(
             r'<script[^>]+data-drupal-selector="drupal-settings-json"[^>]*>({.+?})</script>', webpage, 'drupal settings'),
@@ -53,11 +56,14 @@ class BravoTVIE(AdobePassIE):
                 tp_path = release_pid = tve['release_pid']
             if tve.get('entitlement') == 'auth':
                 adobe_pass = settings.get('tve_adobe_auth', {})
+                if site == 'bravotv':
+                    site = 'bravo'
                 resource = self._get_mvpd_resource(
-                    adobe_pass.get('adobePassResourceId', 'bravo'),
+                    adobe_pass.get('adobePassResourceId') or site,
                     tve['title'], release_pid, tve.get('rating'))
                 query['auth'] = self._extract_mvpd_auth(
-                    url, release_pid, adobe_pass.get('adobePassRequestorId', 'bravo'), resource)
+                    url, release_pid,
+                    adobe_pass.get('adobePassRequestorId') or site, resource)
         else:
             shared_playlist = settings['ls_playlist']
             account_pid = shared_playlist['account_pid']

youtube_dlc/extractor/canvas.py

@@ -7,12 +7,12 @@ from .common import InfoExtractor
 from .gigya import GigyaBaseIE
 from ..compat import compat_HTTPError
 from ..utils import (
+    extract_attributes,
     ExtractorError,
     strip_or_none,
     float_or_none,
     int_or_none,
     merge_dicts,
-    parse_iso8601,
     str_or_none,
     url_or_none,
 )
@@ -37,6 +37,7 @@ class CanvasIE(InfoExtractor):
         'url': 'https://mediazone.vrt.be/api/v1/canvas/assets/mz-ast-5e5f90b6-2d72-4c40-82c2-e134f884e93e',
         'only_matching': True,
     }]
+    _GEO_BYPASS = False
     _HLS_ENTRY_PROTOCOLS_MAP = {
         'HLS': 'm3u8_native',
         'HLS_AES': 'm3u8',
@@ -47,29 +48,34 @@ class CanvasIE(InfoExtractor):
         mobj = re.match(self._VALID_URL, url)
         site_id, video_id = mobj.group('site_id'), mobj.group('id')
 
-        # Old API endpoint, serves more formats but may fail for some videos
+        data = None
-        data = self._download_json(
+        if site_id != 'vrtvideo':
-            'https://mediazone.vrt.be/api/v1/%s/assets/%s'
+            # Old API endpoint, serves more formats but may fail for some videos
-            % (site_id, video_id), video_id, 'Downloading asset JSON',
+            data = self._download_json(
-            'Unable to download asset JSON', fatal=False)
+                'https://mediazone.vrt.be/api/v1/%s/assets/%s'
+                % (site_id, video_id), video_id, 'Downloading asset JSON',
+                'Unable to download asset JSON', fatal=False)
 
         # New API endpoint
         if not data:
+            headers = self.geo_verification_headers()
+            headers.update({'Content-Type': 'application/json'})
             token = self._download_json(
                 '%s/tokens' % self._REST_API_BASE, video_id,
-                'Downloading token', data=b'',
+                'Downloading token', data=b'', headers=headers)['vrtPlayerToken']
-                headers={'Content-Type': 'application/json'})['vrtPlayerToken']
             data = self._download_json(
                 '%s/videos/%s' % (self._REST_API_BASE, video_id),
-                video_id, 'Downloading video JSON', fatal=False, query={
+                video_id, 'Downloading video JSON', query={
                     'vrtPlayerToken': token,
                     'client': '%s@PROD' % site_id,
                 }, expected_status=400)
-            message = data.get('message')
+            if not data.get('title'):
-            if message and not data.get('title'):
+                code = data.get('code')
-                if data.get('code') == 'AUTHENTICATION_REQUIRED':
+                if code == 'AUTHENTICATION_REQUIRED':
-                    self.raise_login_required(message)
+                    self.raise_login_required()
-                raise ExtractorError(message, expected=True)
+                elif code == 'INVALID_LOCATION':
+                    self.raise_geo_restricted(countries=['BE'])
+                raise ExtractorError(data.get('message') or code, expected=True)
 
         title = data['title']
         description = data.get('description')
@@ -205,20 +211,24 @@ class CanvasEenIE(InfoExtractor):
 
 class VrtNUIE(GigyaBaseIE):
     IE_DESC = 'VrtNU.be'
-    _VALID_URL = r'https?://(?:www\.)?vrt\.be/(?P<site_id>vrtnu)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?vrt\.be/vrtnu/a-z/(?:[^/]+/){2}(?P<id>[^/?#&]+)'
     _TESTS = [{
         # Available via old API endpoint
-        'url': 'https://www.vrt.be/vrtnu/a-z/postbus-x/1/postbus-x-s1a1/',
+        'url': 'https://www.vrt.be/vrtnu/a-z/postbus-x/1989/postbus-x-s1989a1/',
         'info_dict': {
-            'id': 'pbs-pub-2e2d8c27-df26-45c9-9dc6-90c78153044d$vid-90c932b1-e21d-4fb8-99b1-db7b49cf74de',
+            'id': 'pbs-pub-e8713dac-899e-41de-9313-81269f4c04ac$vid-90c932b1-e21d-4fb8-99b1-db7b49cf74de',
             'ext': 'mp4',
-            'title': 'De zwarte weduwe',
+            'title': 'Postbus X - Aflevering 1 (Seizoen 1989)',
-            'description': 'md5:db1227b0f318c849ba5eab1fef895ee4',
+            'description': 'md5:b704f669eb9262da4c55b33d7c6ed4b7',
             'duration': 1457.04,
             'thumbnail': r're:^https?://.*\.jpg$',
-            'season': 'Season 1',
+            'series': 'Postbus X',
-            'season_number': 1,
+            'season': 'Seizoen 1989',
+            'season_number': 1989,
+            'episode': 'De zwarte weduwe',
             'episode_number': 1,
+            'timestamp': 1595822400,
+            'upload_date': '20200727',
         },
         'skip': 'This video is only available for registered users',
         'params': {
@@ -300,69 +310,25 @@ class VrtNUIE(GigyaBaseIE):
     def _real_extract(self, url):
         display_id = self._match_id(url)
 
-        webpage, urlh = self._download_webpage_handle(url, display_id)
+        webpage = self._download_webpage(url, display_id)
+
+        attrs = extract_attributes(self._search_regex(
+            r'(<nui-media[^>]+>)', webpage, 'media element'))
+        video_id = attrs['videoid']
+        publication_id = attrs.get('publicationid')
+        if publication_id:
+            video_id = publication_id + '$' + video_id
+
+        page = (self._parse_json(self._search_regex(
+            r'digitalData\s*=\s*({.+?});', webpage, 'digial data',
+            default='{}'), video_id, fatal=False) or {}).get('page') or {}
 
         info = self._search_json_ld(webpage, display_id, default={})
-
-        # title is optional here since it may be extracted by extractor
-        # that is delegated from here
-        title = strip_or_none(self._html_search_regex(
-            r'(?ms)<h1 class="content__heading">(.+?)</h1>',
-            webpage, 'title', default=None))
-
-        description = self._html_search_regex(
-            r'(?ms)<div class="content__description">(.+?)</div>',
-            webpage, 'description', default=None)
-
-        season = self._html_search_regex(
-            [r'''(?xms)<div\ class="tabs__tab\ tabs__tab--active">\s*
-                    <span>seizoen\ (.+?)</span>\s*
-                </div>''',
-             r'<option value="seizoen (\d{1,3})" data-href="[^"]+?" selected>'],
-            webpage, 'season', default=None)
-
-        season_number = int_or_none(season)
-
-        episode_number = int_or_none(self._html_search_regex(
-            r'''(?xms)<div\ class="content__episode">\s*
-                    <abbr\ title="aflevering">afl</abbr>\s*<span>(\d+)</span>
-                </div>''',
-            webpage, 'episode_number', default=None))
-
-        release_date = parse_iso8601(self._html_search_regex(
-            r'(?ms)<div class="content__broadcastdate">\s*<time\ datetime="(.+?)"',
-            webpage, 'release_date', default=None))
-
-        # If there's a ? or a # in the URL, remove them and everything after
-        clean_url = urlh.geturl().split('?')[0].split('#')[0].strip('/')
-        securevideo_url = clean_url + '.mssecurevideo.json'
-
-        try:
-            video = self._download_json(securevideo_url, display_id)
-        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                self.raise_login_required()
-            raise
-
-        # We are dealing with a '../<show>.relevant' URL
-        redirect_url = video.get('url')
-        if redirect_url:
-            return self.url_result(self._proto_relative_url(redirect_url, 'https:'))
-
-        # There is only one entry, but with an unknown key, so just get
-        # the first one
-        video_id = list(video.values())[0].get('videoid')
-
         return merge_dicts(info, {
             '_type': 'url_transparent',
             'url': 'https://mediazone.vrt.be/api/v1/vrtvideo/assets/%s' % video_id,
             'ie_key': CanvasIE.ie_key(),
             'id': video_id,
             'display_id': display_id,
-            'title': title,
+            'season_number': int_or_none(page.get('episode_season')),
-            'description': description,
-            'season': season,
-            'season_number': season_number,
-            'episode_number': episode_number,
-            'release_date': release_date,
         })

youtube_dlc/extractor/cbs.py

@@ -59,7 +59,7 @@ class CBSIE(CBSBaseIE):
             'http://can.cbs.com/thunder/player/videoPlayerService.php',
             content_id, query={'partner': site, 'contentId': content_id})
         video_data = xpath_element(items_data, './/item')
-        title = xpath_text(video_data, 'videoTitle', 'title', True)
+        title = xpath_text(video_data, 'videoTitle', 'title') or xpath_text(video_data, 'videotitle', 'title')
         tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
         tp_release_url = 'http://link.theplatform.com/s/' + tp_path
 

youtube_dlc/extractor/ccma.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import datetime
 import re
 
 from .common import InfoExtractor
@@ -8,8 +9,8 @@ from ..utils import (
     clean_html,
     int_or_none,
     parse_duration,
-    parse_iso8601,
     parse_resolution,
+    try_get,
     url_or_none,
 )
 
@@ -24,8 +25,9 @@ class CCMAIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'L\'espot de La Marató de TV3',
             'description': 'md5:f12987f320e2f6e988e9908e4fe97765',
-            'timestamp': 1470918540,
+            'timestamp': 1478608140,
-            'upload_date': '20160811',
+            'upload_date': '20161108',
+            'age_limit': 0,
         }
     }, {
         'url': 'http://www.ccma.cat/catradio/alacarta/programa/el-consell-de-savis-analitza-el-derbi/audio/943685/',
@@ -35,8 +37,24 @@ class CCMAIE(InfoExtractor):
             'ext': 'mp3',
             'title': 'El Consell de Savis analitza el derbi',
             'description': 'md5:e2a3648145f3241cb9c6b4b624033e53',
-            'upload_date': '20171205',
+            'upload_date': '20170512',
-            'timestamp': 1512507300,
+            'timestamp': 1494622500,
+            'vcodec': 'none',
+            'categories': ['Esports'],
+        }
+    }, {
+        'url': 'http://www.ccma.cat/tv3/alacarta/crims/crims-josep-tallada-lespereu-me-capitol-1/video/6031387/',
+        'md5': 'b43c3d3486f430f3032b5b160d80cbc3',
+        'info_dict': {
+            'id': '6031387',
+            'ext': 'mp4',
+            'title': 'Crims - Josep Talleda, l\'"Espereu-me" (capítol 1)',
+            'description': 'md5:7cbdafb640da9d0d2c0f62bad1e74e60',
+            'timestamp': 1582577700,
+            'upload_date': '20200224',
+            'subtitles': 'mincount:4',
+            'age_limit': 16,
+            'series': 'Crims',
         }
     }]
 
@@ -72,17 +90,27 @@ class CCMAIE(InfoExtractor):
 
         informacio = media['informacio']
         title = informacio['titol']
-        durada = informacio.get('durada', {})
+        durada = informacio.get('durada') or {}
         duration = int_or_none(durada.get('milisegons'), 1000) or parse_duration(durada.get('text'))
-        timestamp = parse_iso8601(informacio.get('data_emissio', {}).get('utc'))
+        tematica = try_get(informacio, lambda x: x['tematica']['text'])
+
+        timestamp = None
+        data_utc = try_get(informacio, lambda x: x['data_emissio']['utc'])
+        try:
+            timestamp = datetime.datetime.strptime(
+                data_utc, '%Y-%d-%mT%H:%M:%S%z').timestamp()
+        except TypeError:
+            pass
 
         subtitles = {}
-        subtitols = media.get('subtitols', {})
+        subtitols = media.get('subtitols') or []
-        if subtitols:
+        if isinstance(subtitols, dict):
-            sub_url = subtitols.get('url')
+            subtitols = [subtitols]
+        for st in subtitols:
+            sub_url = st.get('url')
             if sub_url:
                 subtitles.setdefault(
-                    subtitols.get('iso') or subtitols.get('text') or 'ca', []).append({
+                    st.get('iso') or st.get('text') or 'ca', []).append({
                         'url': sub_url,
                     })
 
@@ -97,6 +125,16 @@ class CCMAIE(InfoExtractor):
                     'height': int_or_none(imatges.get('alcada')),
                 }]
 
+        age_limit = None
+        codi_etic = try_get(informacio, lambda x: x['codi_etic']['id'])
+        if codi_etic:
+            codi_etic_s = codi_etic.split('_')
+            if len(codi_etic_s) == 2:
+                if codi_etic_s[1] == 'TP':
+                    age_limit = 0
+                else:
+                    age_limit = int_or_none(codi_etic_s[1])
+
         return {
             'id': media_id,
             'title': title,
@@ -106,4 +144,9 @@ class CCMAIE(InfoExtractor):
             'thumbnails': thumbnails,
             'subtitles': subtitles,
             'formats': formats,
+            'age_limit': age_limit,
+            'alt_title': informacio.get('titol_complet'),
+            'episode_number': int_or_none(informacio.get('capitol')),
+            'categories': [tematica] if tematica else None,
+            'series': informacio.get('programa'),
         }

youtube_dlc/extractor/cda.py

@@ -96,7 +96,7 @@ class CDAIE(InfoExtractor):
             raise ExtractorError('This video is only available for premium users.', expected=True)
 
         need_confirm_age = False
-        if self._html_search_regex(r'(<form[^>]+action="/a/validatebirth")',
+        if self._html_search_regex(r'(<form[^>]+action="[^"]*/a/validatebirth[^"]*")',
                                    webpage, 'birthday validate form', default=None):
             webpage = self._download_age_confirm_page(
                 url, video_id, note='Confirming age')

youtube_dlc/extractor/comedycentral.py

@@ -1,142 +1,51 @@
 from __future__ import unicode_literals
 
 from .mtv import MTVServicesInfoExtractor
-from .common import InfoExtractor
 
 
 class ComedyCentralIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
+    _VALID_URL = r'https?://(?:www\.)?cc\.com/(?:episodes|video(?:-clips)?)/(?P<id>[0-9a-z]{6})'
-        (video-clips|episodes|cc-studios|video-collections|shows(?=/[^/]+/(?!full-episodes)))
-        /(?P<title>.*)'''
     _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
 
     _TESTS = [{
-        'url': 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother',
+        'url': 'http://www.cc.com/video-clips/5ke9v2/the-daily-show-with-trevor-noah-doc-rivers-and-steve-ballmer---the-nba-player-strike',
-        'md5': 'c4f48e9eda1b16dd10add0744344b6d8',
+        'md5': 'b8acb347177c680ff18a292aa2166f80',
         'info_dict': {
-            'id': 'cef0cbb3-e776-4bc9-b62e-8016deccb354',
+            'id': '89ccc86e-1b02-4f83-b0c9-1d9592ecd025',
             'ext': 'mp4',
-            'title': 'CC:Stand-Up|August 18, 2013|1|0101|Uncensored - Too Good of a Mother',
+            'title': 'The Daily Show with Trevor Noah|August 28, 2020|25|25149|Doc Rivers and Steve Ballmer - The NBA Player Strike',
-            'description': 'After a certain point, breastfeeding becomes c**kblocking.',
+            'description': 'md5:5334307c433892b85f4f5e5ac9ef7498',
-            'timestamp': 1376798400,
+            'timestamp': 1598670000,
-            'upload_date': '20130818',
+            'upload_date': '20200829',
         },
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/interviews/6yx39d/exclusive-rand-paul-extended-interview',
+        'url': 'http://www.cc.com/episodes/pnzzci/drawn-together--american-idol--parody-clip-show-season-3-ep-314',
         'only_matching': True,
-    }]
-
-
-class ComedyCentralFullEpisodesIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
-        (?:full-episodes|shows(?=/[^/]+/full-episodes))
-        /(?P<id>[^?]+)'''
-    _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
-
-    _TESTS = [{
-        'url': 'http://www.cc.com/full-episodes/pv391a/the-daily-show-with-trevor-noah-november-28--2016---ryan-speedo-green-season-22-ep-22028',
-        'info_dict': {
-            'description': 'Donald Trump is accused of exploiting his president-elect status for personal gain, Cuban leader Fidel Castro dies, and Ryan Speedo Green discusses "Sing for Your Life."',
-            'title': 'November 28, 2016 - Ryan Speedo Green',
-        },
-        'playlist_count': 4,
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
+        'url': 'https://www.cc.com/video/k3sdvm/the-daily-show-with-jon-stewart-exclusive-the-fourth-estate',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        playlist_id = self._match_id(url)
-        webpage = self._download_webpage(url, playlist_id)
-        mgid = self._extract_mgid(webpage, url, data_zone='t2_lc_promo1')
-        videos_info = self._get_videos_info(mgid)
-        return videos_info
-
-
-class ToshIE(MTVServicesInfoExtractor):
-    IE_DESC = 'Tosh.0'
-    _VALID_URL = r'^https?://tosh\.cc\.com/video-(?:clips|collections)/[^/]+/(?P<videotitle>[^/?#]+)'
-    _FEED_URL = 'http://tosh.cc.com/feeds/mrss'
-
-    _TESTS = [{
-        'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
-        'info_dict': {
-            'description': 'Tosh asked fans to share their summer plans.',
-            'title': 'Twitter Users Share Summer Plans',
-        },
-        'playlist': [{
-            'md5': 'f269e88114c1805bb6d7653fecea9e06',
-            'info_dict': {
-                'id': '90498ec2-ed00-11e0-aca6-0026b9414f30',
-                'ext': 'mp4',
-                'title': 'Tosh.0|June 9, 2077|2|211|Twitter Users Share Summer Plans',
-                'description': 'Tosh asked fans to share their summer plans.',
-                'thumbnail': r're:^https?://.*\.jpg',
-                # It's really reported to be published on year 2077
-                'upload_date': '20770610',
-                'timestamp': 3390510600,
-                'subtitles': {
-                    'en': 'mincount:3',
-                },
-            },
-        }]
-    }, {
-        'url': 'http://tosh.cc.com/video-collections/x2iz7k/just-plain-foul/m5q4fp',
         'only_matching': True,
     }]
 
 
 class ComedyCentralTVIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/(?:staffeln|shows)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/folgen/(?P<id>[0-9a-z]{6})'
     _TESTS = [{
-        'url': 'http://www.comedycentral.tv/staffeln/7436-the-mindy-project-staffel-4',
+        'url': 'https://www.comedycentral.tv/folgen/pxdpec/josh-investigates-klimawandel-staffel-1-ep-1',
         'info_dict': {
-            'id': 'local_playlist-f99b626bdfe13568579a',
+            'id': '15907dc3-ec3c-11e8-a442-0e40cf2fc285',
-            'ext': 'flv',
+            'ext': 'mp4',
-            'title': 'Episode_the-mindy-project_shows_season-4_episode-3_full-episode_part1',
+            'title': 'Josh Investigates',
+            'description': 'Steht uns das Ende der Welt bevor?',
         },
-        'params': {
-            # rtmp download
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1074-workaholics',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1727-the-mindy-project/bonus',
-        'only_matching': True,
     }]
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'
+    _GEO_COUNTRIES = ['DE']
 
-    def _real_extract(self, url):
+    def _get_feed_query(self, uri):
-        video_id = self._match_id(url)
+        return {
-
+            'accountOverride': 'intl.mtvi.com',
-        webpage = self._download_webpage(url, video_id)
+            'arcEp': 'web.cc.tv',
-
+            'ep': 'b9032c3a',
-        mrss_url = self._search_regex(
+            'imageEp': 'web.cc.tv',
-            r'data-mrss=(["\'])(?P<url>(?:(?!\1).)+)\1',
+            'mgid': uri,
-            webpage, 'mrss url', group='url')
-
-        return self._get_videos_info_from_url(mrss_url, video_id)
-
-
-class ComedyCentralShortnameIE(InfoExtractor):
-    _VALID_URL = r'^:(?P<id>tds|thedailyshow|theopposition)$'
-    _TESTS = [{
-        'url': ':tds',
-        'only_matching': True,
-    }, {
-        'url': ':thedailyshow',
-        'only_matching': True,
-    }, {
-        'url': ':theopposition',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        shortcut_map = {
-            'tds': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'thedailyshow': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'theopposition': 'http://www.cc.com/shows/the-opposition-with-jordan-klepper/full-episodes',
         }
-        return self.url_result(shortcut_map[video_id])

youtube_dlc/extractor/common.py

@@ -336,9 +336,8 @@ class InfoExtractor(object):
     There must be a key "entries", which is a list, an iterable, or a PagedList
     object, each element of which is a valid dictionary by this specification.
 
-    Additionally, playlists can have "id", "title", "description", "uploader",
+    Additionally, playlists can have "id", "title", and any other relevent
-    "uploader_id", "uploader_url", "duration" attributes with the same semantics
+    attributes with the same semantics as videos (see above).
-    as videos (see above).
 
 
     _type "multi_video" indicates that there are multiple videos that
@@ -967,10 +966,11 @@ class InfoExtractor(object):
             urls, playlist_id=playlist_id, playlist_title=playlist_title)
 
     @staticmethod
-    def playlist_result(entries, playlist_id=None, playlist_title=None, playlist_description=None):
+    def playlist_result(entries, playlist_id=None, playlist_title=None, playlist_description=None, **kwargs):
         """Returns a playlist"""
         video_info = {'_type': 'playlist',
                       'entries': entries}
+        video_info.update(kwargs)
         if playlist_id:
             video_info['id'] = playlist_id
         if playlist_title:
@@ -1366,17 +1366,17 @@ class InfoExtractor(object):
     class FormatSort:
         regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
 
-        default = ('hidden', 'has_video', 'extractor', 'lang', 'quality',
+        default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
-                   'res', 'fps', 'codec', 'size', 'br', 'asr',
+                   'res', 'fps', 'codec:vp9', 'size', 'br', 'asr',
-                   'proto', 'ext', 'has_audio', 'source', 'format_id')
+                   'proto', 'ext', 'has_audio', 'source', 'format_id')  # These must not be aliases
 
         settings = {
             'vcodec': {'type': 'ordered', 'regex': True,
-                       'order': ['vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
+                       'order': ['av0?1', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
-            'protocol': {'type': 'ordered', 'regex': True,
+            'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
-                         'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
+                      'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
             'vext': {'type': 'ordered', 'field': 'video_ext',
                      'order': ('mp4', 'webm', 'flv', '', 'none'),
                      'order_free': ('webm', 'mp4', 'flv', '', 'none')},
@@ -1384,14 +1384,14 @@ class InfoExtractor(object):
                      'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                      'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
             'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
-            'extractor_preference': {'priority': True, 'type': 'extractor'},
+            'ie_pref': {'priority': True, 'type': 'extractor', 'field': 'extractor_preference'},
-            'has_video': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'has_audio': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'language_preference': {'priority': True, 'convert': 'ignore'},
+            'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
             'quality': {'priority': True, 'convert': 'float_none'},
             'filesize': {'convert': 'bytes'},
-            'filesize_approx': {'convert': 'bytes'},
+            'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
-            'format_id': {'convert': 'string'},
+            'id': {'convert': 'string', 'field': 'format_id'},
             'height': {'convert': 'float_none'},
             'width': {'convert': 'float_none'},
             'fps': {'convert': 'float_none'},
@@ -1399,32 +1399,42 @@ class InfoExtractor(object):
             'vbr': {'convert': 'float_none'},
             'abr': {'convert': 'float_none'},
             'asr': {'convert': 'float_none'},
-            'source_preference': {'convert': 'ignore'},
+            'source': {'convert': 'ignore', 'field': 'source_preference'},
+
             'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
-            'bitrate': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
+            'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
-            'filesize_estimate': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'filesize_approx')},
+            'size': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'fs_approx')},
-            'extension': {'type': 'combined', 'field': ('vext', 'aext')},
+            'ext': {'type': 'combined', 'field': ('vext', 'aext')},
-            'dimension': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},  # not named as 'resolution' because such a field exists
+            'res': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},
-            'res': {'type': 'alias', 'field': 'dimension'},
+
-            'ext': {'type': 'alias', 'field': 'extension'},
+            # Most of these exist only for compatibility reasons
-            'br': {'type': 'alias', 'field': 'bitrate'},
+            'dimension': {'type': 'alias', 'field': 'res'},
+            'resolution': {'type': 'alias', 'field': 'res'},
+            'extension': {'type': 'alias', 'field': 'ext'},
+            'bitrate': {'type': 'alias', 'field': 'br'},
             'total_bitrate': {'type': 'alias', 'field': 'tbr'},
             'video_bitrate': {'type': 'alias', 'field': 'vbr'},
             'audio_bitrate': {'type': 'alias', 'field': 'abr'},
             'framerate': {'type': 'alias', 'field': 'fps'},
-            'lang': {'type': 'alias', 'field': 'language_preference'},  # not named as 'language' because such a field exists
+            'language_preference': {'type': 'alias', 'field': 'lang'},  # not named as 'language' because such a field exists
-            'proto': {'type': 'alias', 'field': 'protocol'},
+            'protocol': {'type': 'alias', 'field': 'proto'},
-            'source': {'type': 'alias', 'field': 'source_preference'},
+            'source_preference': {'type': 'alias', 'field': 'source'},
-            'size': {'type': 'alias', 'field': 'filesize_estimate'},
+            'filesize_approx': {'type': 'alias', 'field': 'fs_approx'},
+            'filesize_estimate': {'type': 'alias', 'field': 'size'},
             'samplerate': {'type': 'alias', 'field': 'asr'},
             'video_ext': {'type': 'alias', 'field': 'vext'},
             'audio_ext': {'type': 'alias', 'field': 'aext'},
             'video_codec': {'type': 'alias', 'field': 'vcodec'},
             'audio_codec': {'type': 'alias', 'field': 'acodec'},
-            'video': {'type': 'alias', 'field': 'has_video'},
+            'video': {'type': 'alias', 'field': 'hasvid'},
-            'audio': {'type': 'alias', 'field': 'has_audio'},
+            'has_video': {'type': 'alias', 'field': 'hasvid'},
-            'extractor': {'type': 'alias', 'field': 'extractor_preference'},
+            'audio': {'type': 'alias', 'field': 'hasaud'},
-            'preference': {'type': 'alias', 'field': 'extractor_preference'}}
+            'has_audio': {'type': 'alias', 'field': 'hasaud'},
+            'extractor': {'type': 'alias', 'field': 'ie_pref'},
+            'preference': {'type': 'alias', 'field': 'ie_pref'},
+            'extractor_preference': {'type': 'alias', 'field': 'ie_pref'},
+            'format_id': {'type': 'alias', 'field': 'id'},
+        }
 
         _order = []
 
@@ -2254,7 +2264,7 @@ class InfoExtractor(object):
             })
         return entries
 
-    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, formats_dict={}, data=None, headers={}, query={}):
+    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             mpd_url, video_id,
             note=note or 'Downloading MPD manifest',
@@ -2268,10 +2278,9 @@ class InfoExtractor(object):
         mpd_base_url = base_url(urlh.geturl())
 
         return self._parse_mpd_formats(
-            mpd_doc, mpd_id=mpd_id, mpd_base_url=mpd_base_url,
+            mpd_doc, mpd_id, mpd_base_url, mpd_url)
-            formats_dict=formats_dict, mpd_url=mpd_url)
 
-    def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', formats_dict={}, mpd_url=None):
+    def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', mpd_url=None):
         """
         Parse formats from MPD manifest.
         References:
@@ -2550,15 +2559,7 @@ class InfoExtractor(object):
                         else:
                             # Assuming direct URL to unfragmented media.
                             f['url'] = base_url
-
+                        formats.append(f)
-                        # According to [1, 5.3.5.2, Table 7, page 35] @id of Representation
-                        # is not necessarily unique within a Period thus formats with
-                        # the same `format_id` are quite possible. There are numerous examples
-                        # of such manifests (see https://github.com/ytdl-org/youtube-dl/issues/15111,
-                        # https://github.com/ytdl-org/youtube-dl/issues/13919)
-                        full_info = formats_dict.get(representation_id, {}).copy()
-                        full_info.update(f)
-                        formats.append(full_info)
                     else:
                         self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
         return formats

youtube_dlc/extractor/cspan.py

@@ -8,11 +8,14 @@ from ..utils import (
     ExtractorError,
     extract_attributes,
     find_xpath_attr,
+    get_element_by_attribute,
     get_element_by_class,
     int_or_none,
     js_to_json,
     merge_dicts,
+    parse_iso8601,
     smuggle_url,
+    str_to_int,
     unescapeHTML,
 )
 from .senateisvp import SenateISVPIE
@@ -116,8 +119,30 @@ class CSpanIE(InfoExtractor):
                 jwsetup, video_id, require_title=False, m3u8_id='hls',
                 base_url=url)
             add_referer(info['formats'])
+            for subtitles in info['subtitles'].values():
+                for subtitle in subtitles:
+                    ext = determine_ext(subtitle['url'])
+                    if ext == 'php':
+                        ext = 'vtt'
+                    subtitle['ext'] = ext
             ld_info = self._search_json_ld(webpage, video_id, default={})
-            return merge_dicts(info, ld_info)
+            title = get_element_by_class('video-page-title', webpage) or \
+                self._og_search_title(webpage)
+            description = get_element_by_attribute('itemprop', 'description', webpage) or \
+                self._html_search_meta(['og:description', 'description'], webpage)
+            return merge_dicts(info, ld_info, {
+                'title': title,
+                'thumbnail': get_element_by_attribute('itemprop', 'thumbnailUrl', webpage),
+                'description': description,
+                'timestamp': parse_iso8601(get_element_by_attribute('itemprop', 'uploadDate', webpage)),
+                'location': get_element_by_attribute('itemprop', 'contentLocation', webpage),
+                'duration': int_or_none(self._search_regex(
+                    r'jwsetup\.seclength\s*=\s*(\d+);',
+                    webpage, 'duration', fatal=False)),
+                'view_count': str_to_int(self._search_regex(
+                    r"<span[^>]+class='views'[^>]*>([\d,]+)\s+Views</span>",
+                    webpage, 'views', fatal=False)),
+            })
 
         # Obsolete
         # We first look for clipid, because clipprog always appears before

youtube_dlc/extractor/dplay.py

@@ -17,7 +17,12 @@ from ..utils import (
 class DPlayIE(InfoExtractor):
     _VALID_URL = r'''(?x)https?://
         (?P<domain>
-            (?:www\.)?(?P<host>dplay\.(?P<country>dk|fi|jp|se|no))|
+            (?:www\.)?(?P<host>d
+                (?:
+                    play\.(?P<country>dk|fi|jp|se|no)|
+                    iscoveryplus\.(?P<plus_country>dk|es|fi|it|se|no)
+                )
+            )|
             (?P<subdomain_country>es|it)\.dplay\.com
         )/[^/]+/(?P<id>[^/]+/[^/?#]+)'''
 
@@ -126,6 +131,24 @@ class DPlayIE(InfoExtractor):
     }, {
         'url': 'https://www.dplay.jp/video/gold-rush/24086',
         'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.se/videos/nugammalt-77-handelser-som-format-sverige/nugammalt-77-handelser-som-format-sverige-101',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.dk/videoer/ted-bundy-mind-of-a-monster/ted-bundy-mind-of-a-monster',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.no/videoer/i-kongens-klr/sesong-1-episode-7',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.it/videos/biografie-imbarazzanti/luigi-di-maio-la-psicosi-di-stanislawskij',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.es/videos/la-fiebre-del-oro/temporada-8-episodio-1',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.discoveryplus.fi/videot/shifting-gears-with-aaron-kaufman/episode-16',
+        'only_matching': True,
     }]
 
     def _get_disco_api_info(self, url, display_id, disco_host, realm, country):
@@ -241,7 +264,7 @@ class DPlayIE(InfoExtractor):
         mobj = re.match(self._VALID_URL, url)
         display_id = mobj.group('id')
         domain = mobj.group('domain').lstrip('www.')
-        country = mobj.group('country') or mobj.group('subdomain_country')
+        country = mobj.group('country') or mobj.group('subdomain_country') or mobj.group('plus_country')
-        host = 'disco-api.' + domain if domain.startswith('dplay.') else 'eu2-prod.disco-api.com'
+        host = 'disco-api.' + domain if domain[0] == 'd' else 'eu2-prod.disco-api.com'
         return self._get_disco_api_info(
             url, display_id, host, 'dplay' + country, country)

youtube_dlc/extractor/egghead.py

@@ -12,7 +12,14 @@ from ..utils import (
 )
 
 
-class EggheadCourseIE(InfoExtractor):
+class EggheadBaseIE(InfoExtractor):
+    def _call_api(self, path, video_id, resource, fatal=True):
+        return self._download_json(
+            'https://app.egghead.io/api/v1/' + path,
+            video_id, 'Downloading %s JSON' % resource, fatal=fatal)
+
+
+class EggheadCourseIE(EggheadBaseIE):
     IE_DESC = 'egghead.io course'
     IE_NAME = 'egghead:course'
     _VALID_URL = r'https://egghead\.io/courses/(?P<id>[^/?#&]+)'
@@ -28,10 +35,9 @@ class EggheadCourseIE(InfoExtractor):
 
     def _real_extract(self, url):
         playlist_id = self._match_id(url)
-
+        series_path = 'series/' + playlist_id
-        lessons = self._download_json(
+        lessons = self._call_api(
-            'https://egghead.io/api/v1/series/%s/lessons' % playlist_id,
+            series_path + '/lessons', playlist_id, 'course lessons')
-            playlist_id, 'Downloading course lessons JSON')
 
         entries = []
         for lesson in lessons:
@@ -44,9 +50,8 @@ class EggheadCourseIE(InfoExtractor):
             entries.append(self.url_result(
                 lesson_url, ie=EggheadLessonIE.ie_key(), video_id=lesson_id))
 
-        course = self._download_json(
+        course = self._call_api(
-            'https://egghead.io/api/v1/series/%s' % playlist_id,
+            series_path, playlist_id, 'course', False) or {}
-            playlist_id, 'Downloading course JSON', fatal=False) or {}
 
         playlist_id = course.get('id')
         if playlist_id:
@@ -57,7 +62,7 @@ class EggheadCourseIE(InfoExtractor):
             course.get('description'))
 
 
-class EggheadLessonIE(InfoExtractor):
+class EggheadLessonIE(EggheadBaseIE):
     IE_DESC = 'egghead.io lesson'
     IE_NAME = 'egghead:lesson'
     _VALID_URL = r'https://egghead\.io/(?:api/v1/)?lessons/(?P<id>[^/?#&]+)'
@@ -74,7 +79,7 @@ class EggheadLessonIE(InfoExtractor):
             'upload_date': '20161209',
             'duration': 304,
             'view_count': 0,
-            'tags': ['javascript', 'free'],
+            'tags': 'count:2',
         },
         'params': {
             'skip_download': True,
@@ -88,8 +93,8 @@ class EggheadLessonIE(InfoExtractor):
     def _real_extract(self, url):
         display_id = self._match_id(url)
 
-        lesson = self._download_json(
+        lesson = self._call_api(
-            'https://egghead.io/api/v1/lessons/%s' % display_id, display_id)
+            'lessons/' + display_id, display_id, 'lesson')
 
         lesson_id = compat_str(lesson['id'])
         title = lesson['title']

youtube_dlc/extractor/extractors.py

@@ -46,7 +46,14 @@ from .alura import (
     AluraCourseIE
 )
 from .amcnetworks import AMCNetworksIE
-from .americastestkitchen import AmericasTestKitchenIE
+from .animelab import (
+    AnimeLabIE,
+    AnimeLabShowsIE,
+)
+from .americastestkitchen import (
+    AmericasTestKitchenIE,
+    AmericasTestKitchenSeasonIE,
+)
 from .animeondemand import AnimeOnDemandIE
 from .anvato import AnvatoIE
 from .aol import AolIE
@@ -59,6 +66,7 @@ from .appletrailers import (
     AppleTrailersIE,
     AppleTrailersSectionIE,
 )
+from .applepodcasts import ApplePodcastsIE
 from .archiveorg import ArchiveOrgIE
 from .arcpublishing import ArcPublishingIE
 from .arkena import ArkenaIE
@@ -82,6 +90,11 @@ from .atvat import ATVAtIE
 from .audimedia import AudiMediaIE
 from .audioboom import AudioBoomIE
 from .audiomack import AudiomackIE, AudiomackAlbumIE
+from .audius import (
+    AudiusIE,
+    AudiusTrackIE,
+    AudiusPlaylistIE
+)
 from .awaan import (
     AWAANIE,
     AWAANVideoIE,
@@ -104,14 +117,22 @@ from .bellmedia import BellMediaIE
 from .beatport import BeatportIE
 from .bet import BetIE
 from .bfi import BFIPlayerIE
+from .bfmtv import (
+    BFMTVIE,
+    BFMTVLiveIE,
+    BFMTVArticleIE,
+)
+from .bibeltv import BibelTVIE
 from .bigflix import BigflixIE
 from .bild import BildIE
 from .bilibili import (
     BiliBiliIE,
+    BiliBiliSearchIE,
     BiliBiliBangumiIE,
     BilibiliAudioIE,
     BilibiliAudioAlbumIE,
     BiliBiliPlayerIE,
+    BilibiliChannelIE,
 )
 from .biobiochiletv import BioBioChileTVIE
 from .bitchute import (
@@ -233,11 +254,8 @@ from .cnn import (
 )
 from .coub import CoubIE
 from .comedycentral import (
-    ComedyCentralFullEpisodesIE,
     ComedyCentralIE,
-    ComedyCentralShortnameIE,
     ComedyCentralTVIE,
-    ToshIE,
 )
 from .commonmistakes import CommonMistakesIE, UnicodeBOMIE
 from .commonprotocols import (
@@ -442,7 +460,10 @@ from .go import GoIE
 from .godtube import GodTubeIE
 from .golem import GolemIE
 from .googledrive import GoogleDriveIE
-from .googleplus import GooglePlusIE
+from .googlepodcasts import (
+    GooglePodcastsIE,
+    GooglePodcastsFeedIE,
+)
 from .googlesearch import GoogleSearchIE
 from .goshgay import GoshgayIE
 from .gputechconf import GPUTechConfIE
@@ -484,6 +505,10 @@ from .ign import (
     OneUPIE,
     PCMagIE,
 )
+from .iheart import (
+    IHeartRadioIE,
+    IHeartRadioPodcastIE,
+)
 from .imdb import (
     ImdbIE,
     ImdbListIE
@@ -533,7 +558,10 @@ from .karaoketv import KaraoketvIE
 from .karrierevideos import KarriereVideosIE
 from .keezmovies import KeezMoviesIE
 from .ketnet import KetnetIE
-from .khanacademy import KhanAcademyIE
+from .khanacademy import (
+    KhanAcademyIE,
+    KhanAcademyUnitIE,
+)
 from .kickstarter import KickStarterIE
 from .kinja import KinjaEmbedIE
 from .kinopoisk import KinoPoiskIE
@@ -656,6 +684,16 @@ from .microsoftvirtualacademy import (
     MicrosoftVirtualAcademyIE,
     MicrosoftVirtualAcademyCourseIE,
 )
+from .mildom import (
+    MildomIE,
+    MildomVodIE,
+    MildomUserVodIE,
+)
+from .minds import (
+    MindsIE,
+    MindsChannelIE,
+    MindsGroupIE,
+)
 from .ministrygrid import MinistryGridIE
 from .minoto import MinotoIE
 from .miomio import MioMioIE
@@ -1136,6 +1174,10 @@ from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
 from .sportdeutschland import SportDeutschlandIE
+from .spotify import (
+    SpotifyIE,
+    SpotifyShowIE,
+)
 from .spreaker import (
     SpreakerIE,
     SpreakerPageIE,
@@ -1244,6 +1286,10 @@ from .toutv import TouTvIE
 from .toypics import ToypicsUserIE, ToypicsIE
 from .traileraddict import TrailerAddictIE
 from .trilulilu import TriluliluIE
+from .trovo import (
+    TrovoIE,
+    TrovoVodIE,
+)
 from .trunews import TruNewsIE
 from .trutv import TruTVIE
 from .tube8 import Tube8IE
@@ -1262,6 +1308,7 @@ from .tv2 import (
     TV2IE,
     TV2ArticleIE,
     KatsomoIE,
+    MTVUutisetArticleIE,
 )
 from .tv2dk import (
     TV2DKIE,
@@ -1402,7 +1449,6 @@ from .vidme import (
     VidmeUserIE,
     VidmeUserLikesIE,
 )
-from .vidzi import VidziIE
 from .vier import VierIE, VierVideosIE
 from .viewlift import (
     ViewLiftIE,
@@ -1462,6 +1508,7 @@ from .vrv import (
     VRVSeriesIE,
 )
 from .vshare import VShareIE
+from .vtm import VTMIE
 from .medialaan import MedialaanIE
 from .vube import VubeIE
 from .vuclip import VuClipIE

youtube_dlc/extractor/franceculture.py

@@ -11,7 +11,7 @@ from ..utils import (
 
 class FranceCultureIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/emissions/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'http://www.franceculture.fr/emissions/carnet-nomade/rendez-vous-au-pays-des-geeks',
         'info_dict': {
             'id': 'rendez-vous-au-pays-des-geeks',
@@ -20,10 +20,14 @@ class FranceCultureIE(InfoExtractor):
             'title': 'Rendez-vous au pays des geeks',
             'thumbnail': r're:^https?://.*\.jpg$',
             'upload_date': '20140301',
-            'timestamp': 1393642916,
+            'timestamp': 1393700400,
             'vcodec': 'none',
         }
-    }
+    }, {
+        # no thumbnail
+        'url': 'https://www.franceculture.fr/emissions/la-recherche-montre-en-main/la-recherche-montre-en-main-du-mercredi-10-octobre-2018',
+        'only_matching': True,
+    }]
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
@@ -36,19 +40,19 @@ class FranceCultureIE(InfoExtractor):
                     </h1>|
                     <div[^>]+class="[^"]*?(?:title-zone-diffusion|heading-zone-(?:wrapper|player-button))[^"]*?"[^>]*>
                 ).*?
-                (<button[^>]+data-asset-source="[^"]+"[^>]+>)
+                (<button[^>]+data-(?:url|asset-source)="[^"]+"[^>]+>)
             ''',
             webpage, 'video data'))
 
-        video_url = video_data['data-asset-source']
+        video_url = video_data.get('data-url') or video_data['data-asset-source']
-        title = video_data.get('data-asset-title') or self._og_search_title(webpage)
+        title = video_data.get('data-asset-title') or video_data.get('data-diffusion-title') or self._og_search_title(webpage)
 
         description = self._html_search_regex(
             r'(?s)<div[^>]+class="intro"[^>]*>.*?<h2>(.+?)</h2>',
             webpage, 'description', default=None)
         thumbnail = self._search_regex(
             r'(?s)<figure[^>]+itemtype="https://schema.org/ImageObject"[^>]*>.*?<img[^>]+(?:data-dejavu-)?src="([^"]+)"',
-            webpage, 'thumbnail', fatal=False)
+            webpage, 'thumbnail', default=None)
         uploader = self._html_search_regex(
             r'(?s)<span class="author">(.*?)</span>',
             webpage, 'uploader', default=None)
@@ -64,6 +68,6 @@ class FranceCultureIE(InfoExtractor):
             'ext': ext,
             'vcodec': 'none' if ext == 'mp3' else None,
             'uploader': uploader,
-            'timestamp': int_or_none(video_data.get('data-asset-created-date')),
+            'timestamp': int_or_none(video_data.get('data-start-time')) or int_or_none(video_data.get('data-asset-created-date')),
             'duration': int_or_none(video_data.get('data-duration')),
         }

youtube_dlc/extractor/generic.py

@@ -131,6 +131,7 @@ from .gedi import GediEmbedsIE
 from .rcs import RCSEmbedsIE
 from .bitchute import BitChuteIE
 from .arcpublishing import ArcPublishingIE
+from .medialaan import MedialaanIE
 
 
 class GenericIE(InfoExtractor):
@@ -2224,6 +2225,20 @@ class GenericIE(InfoExtractor):
                 'duration': 1581,
             },
         },
+        {
+            # MyChannels SDK embed
+            # https://www.24kitchen.nl/populair/deskundige-dit-waarom-sommigen-gevoelig-zijn-voor-voedselallergieen
+            'url': 'https://www.demorgen.be/nieuws/burgemeester-rotterdam-richt-zich-in-videoboodschap-tot-relschoppers-voelt-het-goed~b0bcfd741/',
+            'md5': '90c0699c37006ef18e198c032d81739c',
+            'info_dict': {
+                'id': '194165',
+                'ext': 'mp4',
+                'title': 'Burgemeester Aboutaleb spreekt relschoppers toe',
+                'timestamp': 1611740340,
+                'upload_date': '20210127',
+                'duration': 159,
+            },
+        },
     ]
 
     def report_following_redirect(self, new_url):
@@ -2463,6 +2478,9 @@ class GenericIE(InfoExtractor):
         webpage = self._webpage_read_content(
             full_response, url, video_id, prefix=first_bytes)
 
+        if '<title>DPG Media Privacy Gate</title>' in webpage:
+            webpage = self._download_webpage(url, video_id)
+
         self.report_extraction(video_id)
 
         # Is it an RSS feed, a SMIL file, an XSPF playlist or a MPD manifest?
@@ -2594,6 +2612,11 @@ class GenericIE(InfoExtractor):
         if arc_urls:
             return self.playlist_from_matches(arc_urls, video_id, video_title, ie=ArcPublishingIE.ie_key())
 
+        mychannels_urls = MedialaanIE._extract_urls(webpage)
+        if mychannels_urls:
+            return self.playlist_from_matches(
+                mychannels_urls, video_id, video_title, ie=MedialaanIE.ie_key())
+
         # Look for embedded rtl.nl player
         matches = re.findall(
             r'<iframe[^>]+?src="((?:https?:)?//(?:(?:www|static)\.)?rtl\.nl/(?:system/videoplayer/[^"]+(?:video_)?)?embed[^"]+)"',

youtube_dlc/extractor/googledrive.py

@@ -7,6 +7,7 @@ from ..compat import compat_parse_qs
 from ..utils import (
     determine_ext,
     ExtractorError,
+    get_element_by_class,
     int_or_none,
     lowercase_escape,
     try_get,
@@ -237,7 +238,7 @@ class GoogleDriveIE(InfoExtractor):
                 if confirmation_webpage:
                     confirm = self._search_regex(
                         r'confirm=([^&"\']+)', confirmation_webpage,
-                        'confirmation code', fatal=False)
+                        'confirmation code', default=None)
                     if confirm:
                         confirmed_source_url = update_url_query(source_url, {
                             'confirm': confirm,
@@ -245,6 +246,11 @@ class GoogleDriveIE(InfoExtractor):
                         urlh = request_source_file(confirmed_source_url, 'confirmed source')
                         if urlh and urlh.headers.get('Content-Disposition'):
                             add_source_format(urlh)
+                    else:
+                        self.report_warning(
+                            get_element_by_class('uc-error-subcaption', confirmation_webpage)
+                            or get_element_by_class('uc-error-caption', confirmation_webpage)
+                            or 'unable to extract confirmation code')
 
         if not formats and reason:
             raise ExtractorError(reason, expected=True)

youtube_dlc/extractor/googleplus.py

@@ -1,73 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-import codecs
-
-from .common import InfoExtractor
-from ..utils import unified_strdate
-
-
-class GooglePlusIE(InfoExtractor):
-    IE_DESC = 'Google Plus'
-    _VALID_URL = r'https?://plus\.google\.com/(?:[^/]+/)*?posts/(?P<id>\w+)'
-    IE_NAME = 'plus.google'
-    _TEST = {
-        'url': 'https://plus.google.com/u/0/108897254135232129896/posts/ZButuJc6CtH',
-        'info_dict': {
-            'id': 'ZButuJc6CtH',
-            'ext': 'flv',
-            'title': '嘆きの天使 降臨',
-            'upload_date': '20120613',
-            'uploader': '井上ヨシマサ',
-        }
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        # Step 1, Retrieve post webpage to extract further information
-        webpage = self._download_webpage(url, video_id, 'Downloading entry webpage')
-
-        title = self._og_search_description(webpage).splitlines()[0]
-        upload_date = unified_strdate(self._html_search_regex(
-            r'''(?x)<a.+?class="o-U-s\s[^"]+"\s+style="display:\s*none"\s*>
-                    ([0-9]{4}-[0-9]{2}-[0-9]{2})</a>''',
-            webpage, 'upload date', fatal=False, flags=re.VERBOSE))
-        uploader = self._html_search_regex(
-            r'rel="author".*?>(.*?)</a>', webpage, 'uploader', fatal=False)
-
-        # Step 2, Simulate clicking the image box to launch video
-        DOMAIN = 'https://plus.google.com/'
-        video_page = self._search_regex(
-            r'<a href="((?:%s)?photos/.*?)"' % re.escape(DOMAIN),
-            webpage, 'video page URL')
-        if not video_page.startswith(DOMAIN):
-            video_page = DOMAIN + video_page
-
-        webpage = self._download_webpage(video_page, video_id, 'Downloading video page')
-
-        def unicode_escape(s):
-            decoder = codecs.getdecoder('unicode_escape')
-            return re.sub(
-                r'\\u[0-9a-fA-F]{4,}',
-                lambda m: decoder(m.group(0))[0],
-                s)
-
-        # Extract video links all sizes
-        formats = [{
-            'url': unicode_escape(video_url),
-            'ext': 'flv',
-            'width': int(width),
-            'height': int(height),
-        } for width, height, video_url in re.findall(
-            r'\d+,(\d+),(\d+),"(https?://[^.]+\.googleusercontent\.com.*?)"', webpage)]
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'uploader': uploader,
-            'upload_date': upload_date,
-            'formats': formats,
-        }

youtube_dlc/extractor/googlepodcasts.py

@@ -0,0 +1,88 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_podcast_url,
+    int_or_none,
+    try_get,
+    urlencode_postdata,
+)
+
+
+class GooglePodcastsBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://podcasts\.google\.com/feed/'
+
+    def _batch_execute(self, func_id, video_id, params):
+        return json.loads(self._download_json(
+            'https://podcasts.google.com/_/PodcastsUi/data/batchexecute',
+            video_id, data=urlencode_postdata({
+                'f.req': json.dumps([[[func_id, json.dumps(params), None, '1']]]),
+            }), transform_source=lambda x: self._search_regex(r'(?s)(\[.+\])', x, 'data'))[0][2])
+
+    def _extract_episode(self, episode):
+        return {
+            'id': episode[4][3],
+            'title': episode[8],
+            'url': clean_podcast_url(episode[13]),
+            'thumbnail': episode[2],
+            'description': episode[9],
+            'creator': try_get(episode, lambda x: x[14]),
+            'timestamp': int_or_none(episode[11]),
+            'duration': int_or_none(episode[12]),
+            'series': episode[1],
+        }
+
+
+class GooglePodcastsIE(GooglePodcastsBaseIE):
+    IE_NAME = 'google:podcasts'
+    _VALID_URL = GooglePodcastsBaseIE._VALID_URL_BASE + r'(?P<feed_url>[^/]+)/episode/(?P<id>[^/?&#]+)'
+    _TEST = {
+        'url': 'https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5ucHIub3JnLzM0NDA5ODUzOS9wb2RjYXN0LnhtbA/episode/MzBlNWRlN2UtOWE4Yy00ODcwLTk2M2MtM2JlMmUyNmViOTRh',
+        'md5': 'fa56b2ee8bd0703e27e42d4b104c4766',
+        'info_dict': {
+            'id': '30e5de7e-9a8c-4870-963c-3be2e26eb94a',
+            'ext': 'mp3',
+            'title': 'WWDTM New Year 2021',
+            'description': 'We say goodbye to 2020 with Christine Baranksi, Doug Jones, Jonna Mendez, and Kellee Edwards.',
+            'upload_date': '20210102',
+            'timestamp': 1609606800,
+            'duration': 2901,
+            'series': "Wait Wait... Don't Tell Me!",
+        }
+    }
+
+    def _real_extract(self, url):
+        b64_feed_url, b64_guid = re.match(self._VALID_URL, url).groups()
+        episode = self._batch_execute(
+            'oNjqVe', b64_guid, [b64_feed_url, b64_guid])[1]
+        return self._extract_episode(episode)
+
+
+class GooglePodcastsFeedIE(GooglePodcastsBaseIE):
+    IE_NAME = 'google:podcasts:feed'
+    _VALID_URL = GooglePodcastsBaseIE._VALID_URL_BASE + r'(?P<id>[^/?&#]+)/?(?:[?#&]|$)'
+    _TEST = {
+        'url': 'https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5ucHIub3JnLzM0NDA5ODUzOS9wb2RjYXN0LnhtbA',
+        'info_dict': {
+            'title': "Wait Wait... Don't Tell Me!",
+            'description': "NPR's weekly current events quiz. Have a laugh and test your news knowledge while figuring out what's real and what we've made up.",
+        },
+        'playlist_mincount': 20,
+    }
+
+    def _real_extract(self, url):
+        b64_feed_url = self._match_id(url)
+        data = self._batch_execute('ncqJEe', b64_feed_url, [b64_feed_url])
+
+        entries = []
+        for episode in (try_get(data, lambda x: x[1][0]) or []):
+            entries.append(self._extract_episode(episode))
+
+        feed = try_get(data, lambda x: x[3]) or []
+        return self.playlist_result(
+            entries, playlist_title=try_get(feed, lambda x: x[0]),
+            playlist_description=try_get(feed, lambda x: x[2]))

youtube_dlc/extractor/iheart.py

@@ -0,0 +1,97 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_html,
+    clean_podcast_url,
+    int_or_none,
+    str_or_none,
+)
+
+
+class IHeartRadioBaseIE(InfoExtractor):
+    def _call_api(self, path, video_id, fatal=True, query=None):
+        return self._download_json(
+            'https://api.iheart.com/api/v3/podcast/' + path,
+            video_id, fatal=fatal, query=query)
+
+    def _extract_episode(self, episode):
+        return {
+            'thumbnail': episode.get('imageUrl'),
+            'description': clean_html(episode.get('description')),
+            'timestamp': int_or_none(episode.get('startDate'), 1000),
+            'duration': int_or_none(episode.get('duration')),
+        }
+
+
+class IHeartRadioIE(IHeartRadioBaseIE):
+    IENAME = 'iheartradio'
+    _VALID_URL = r'(?:https?://(?:www\.)?iheart\.com/podcast/[^/]+/episode/(?P<display_id>[^/?&#]+)-|iheartradio:)(?P<id>\d+)'
+    _TEST = {
+        'url': 'https://www.iheart.com/podcast/105-behind-the-bastards-29236323/episode/part-one-alexander-lukashenko-the-dictator-70346499/?embed=true',
+        'md5': 'c8609c92c8688dcb69d8541042b8abca',
+        'info_dict': {
+            'id': '70346499',
+            'ext': 'mp3',
+            'title': 'Part One: Alexander Lukashenko: The Dictator of Belarus',
+            'description': 'md5:96cc7297b3a5a9ebae28643801c96fae',
+            'timestamp': 1597741200,
+            'upload_date': '20200818',
+        }
+    }
+
+    def _real_extract(self, url):
+        episode_id = self._match_id(url)
+        episode = self._call_api(
+            'episodes/' + episode_id, episode_id)['episode']
+        info = self._extract_episode(episode)
+        info.update({
+            'id': episode_id,
+            'title': episode['title'],
+            'url': clean_podcast_url(episode['mediaUrl']),
+        })
+        return info
+
+
+class IHeartRadioPodcastIE(IHeartRadioBaseIE):
+    IE_NAME = 'iheartradio:podcast'
+    _VALID_URL = r'https?://(?:www\.)?iheart(?:podcastnetwork)?\.com/podcast/[^/?&#]+-(?P<id>\d+)/?(?:[?#&]|$)'
+    _TESTS = [{
+        'url': 'https://www.iheart.com/podcast/1119-it-could-happen-here-30717896/',
+        'info_dict': {
+            'id': '30717896',
+            'title': 'It Could Happen Here',
+            'description': 'md5:5842117412a967eb0b01f8088eb663e2',
+        },
+        'playlist_mincount': 11,
+    }, {
+        'url': 'https://www.iheartpodcastnetwork.com/podcast/105-stuff-you-should-know-26940277',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        podcast_id = self._match_id(url)
+        path = 'podcasts/' + podcast_id
+        episodes = self._call_api(
+            path + '/episodes', podcast_id, query={'limit': 1000000000})['data']
+
+        entries = []
+        for episode in episodes:
+            episode_id = str_or_none(episode.get('id'))
+            if not episode_id:
+                continue
+            info = self._extract_episode(episode)
+            info.update({
+                '_type': 'url',
+                'id': episode_id,
+                'title': episode.get('title'),
+                'url': 'iheartradio:' + episode_id,
+                'ie_key': IHeartRadioIE.ie_key(),
+            })
+            entries.append(info)
+
+        podcast = self._call_api(path, podcast_id, False) or {}
+
+        return self.playlist_result(
+            entries, podcast_id, podcast.get('title'), podcast.get('description'))