Release 2021.02.15

Cookies for some new accounts doesn't work with age-gated videos without `has_verified=1`

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.24
+ [debug] yt-dlp version 2021.02.09
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.24
+ [debug] yt-dlp version 2021.02.09
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -55,9 +55,7 @@ jobs:
         asset_content_type: application/octet-stream
     - name: Get SHA2-256SUMS for youtube-dlc
       id: sha2_file
-      env:
+      run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc)"
-        SHA2: ${{ hashFiles('youtube-dlc') }}
-      run: echo "::set-output name=sha2_unix::$SHA2"
     - name: Install dependencies for pypi
       run: |
         python -m pip install --upgrade pip
@@ -75,6 +73,9 @@ jobs:
 
     runs-on: windows-latest
 
+    outputs:
+      sha2_windows: ${{ steps.sha2_file_win.outputs.sha2_windows }}
+
     needs: build_unix
 
     steps:
@@ -84,14 +85,14 @@ jobs:
       with:
           python-version: '3.8'
     - name: Install Requirements
-      run: pip install pyinstaller mutagen
+      run: pip install pyinstaller mutagen pycryptodome
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script
-      run: python devscripts/pyinst.py 64
+      run: python pyinst.py 64
     - name: Upload youtube-dlc.exe Windows binary
       id: upload-release-windows
       uses: actions/upload-release-asset@v1
@@ -104,14 +105,15 @@ jobs:
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for youtube-dlc.exe
       id: sha2_file_win
-      env:
+      run: echo "::set-output name=sha2_windows::$(certUtil -hashfile dist\youtube-dlc.exe SHA256 | findstr -v :)"
-        SHA2_win: ${{ hashFiles('dist/youtube-dlc.exe') }}
-      run: echo "::set-output name=sha2_windows::$SHA2_win"
 
   build_windows32:
 
     runs-on: windows-latest
 
+    outputs:
+      sha2_windows32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
+
     needs: [build_unix, build_windows]
 
     steps:
@@ -121,15 +123,17 @@ jobs:
       with:
           python-version: '3.4.4'
           architecture: 'x86'
+    - name: Install VS libs
+      run: choco install vcexpress2010
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen
+      run: pip install pyinstaller==3.5 mutagen pycryptodome
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script for 32 Bit
-      run: python devscripts/pyinst.py 32
+      run: python pyinst.py 32
     - name: Upload Executable youtube-dlc_x86.exe
       id: upload-release-windows32
       uses: actions/upload-release-asset@v1
@@ -142,9 +146,7 @@ jobs:
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for youtube-dlc_x86.exe
       id: sha2_file_win32
-      env:
+      run: echo "::set-output name=sha2_windows32::$(certUtil -hashfile dist\youtube-dlc_x86.exe SHA256 | findstr -v :)"
-        SHA2_win32: ${{ hashFiles('dist/youtube-dlc_x86.exe') }}
-      run: echo "::set-output name=sha2_windows32::$SHA2_win32"
     - name: Make SHA2-256SUMS file
       env:
         SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}

.github/workflows/quick-test.yml

@@ -2,7 +2,7 @@ name: Quick Test
 on: [push, pull_request]
 jobs:
   tests:
-    name: Core Tests
+    name: Core Test
     if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -17,6 +17,7 @@ MANIFEST
 test/local_parameters.json
 .coverage
 cover/
+secrets/
 updates_key.pem
 *.egg-info
 .tox
@@ -54,9 +55,19 @@ youtube-dlc
 *.swf
 *.part
 *.ytdl
+*.frag
+*.frag.urls
+*.aria2
 *.swp
 *.ogg
 *.opus
+*.info.json
+*.live_chat.json
+*.jpg
+*.png
+*.webp
+*.annotations.xml
+*.description
 
 # Config
 *.conf

CONTRIBUTORS

@@ -17,3 +17,4 @@ alxnull
 FelixFrog
 Zocker1999NET
 nao20010128nao
+shirt-dev

Changelog.md

@@ -8,7 +8,7 @@
 * Change "Merged with youtube-dl" version in Readme.md if needed
 * Commit to master as `Release <version>`
 * Push to origin/release - build task will now run
-* Update version.py using devscripts\update-version.py
+* Update version.py using devscripts\update-version.py (be wary of timezones)
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
 * Push to origin/master
@@ -17,8 +17,72 @@
 -->
 
 
+### 2021.02.15
+* **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
+* [niconico] Improved extraction and support encrypted/SMILE movies
+* Fix HLS AES-128 with multiple keys in external downloaders
+* [youtube_live_chat] Fix by using POST API
+* [rumble] Add support for video page
+* Option to allow downloading unplayable video formats (`--allow-unplayable-formats`)
+* [ExtractAudio] Don't re-encode when file is already in a common audio format
+* Change optional dependency to `pycryptodome`
+* [youtube] Fix search continuations
+* [youtube] Fix for new accounts
+* Improve build/updater:
+    * Fix SHA256 calculation in build and implement hash checking for updater
+    * Exit immediately in windows once the update process starts
+    * Fix updater for `x86.exe`
+    * Updater looks for both `yt-dlp` and `youtube-dlc` in releases for future-proofing
+* Fix issue with unicode filenames in aria2c
+* Fix `allow_playlist_files` not being correctly passed through
+* Fix for empty HTTP head requests
+* Fix `get_executable_path` in UNIX
+* [sponskrub] Print ffmpeg output and errors to terminal
+* `__real_download` should be false when ffmpeg unavailable and no download
+* Show `exe`/`zip`/`source` and 32/64bit in verbose message
+
+
+### 2021.02.09
+* **aria2c support for DASH/HLS**: by [shirt](https://github.com/shirt-dev)
+* **Implement Updater** (`-U`) by [shirt](https://github.com/shirt-dev)
+* [youtube] Fix comment extraction
+* [youtube_live_chat] Improve extraction
+* [youtube] Fix for channel URLs sometimes not downloading all pages
+* [aria2c] Changed default arguments to `--console-log-level=warn --summary-interval=0 --file-allocation=none -x16 -j16 -s16`
+* Add fallback for thumbnails
+* [embedthumbnail] Keep original thumbnail after conversion if write_thumbnail given
+* [embedsubtitle] Keep original subtitle after conversion if write_subtitles given
+* [pyinst.py] Move back to root dir
+* [youtube] Simplified renderer parsing and bugfixes
+* [movefiles] Fix compatibility with python2
+* [remuxvideo] Fix validation of conditional remux
+* [sponskrub] Don't raise error when the video does not exist
+* [documentation] Crypto is an optional dependency
+
+
+### 2021.02.04
+* **Merge youtube-dl:** Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
+* **Date/time formatting in output template:**
+    * You can use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
+* **Multiple output templates:**
+    * Separate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
+    * The allowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
+* [youtube] More metadata extraction for channel/playlist URLs (channel, uploader, thumbnail, tags)
+* New option `--no-write-playlist-metafiles` to prevent writing playlist metadata files
+* [audius] Fix extractor
+* [youtube_live_chat] Fix `parse_yt_initial_data` and add `fragment_retries`
+* [postprocessor] Raise errors correctly
+* [metadatafromtitle] Fix bug when extracting data from numeric fields
+* Fix issue with overwriting files
+* Fix "Default format spec" appearing in quiet mode
+* [FormatSort] Allow user to prefer av01 over vp9 (The default is still vp9)
+* [FormatSort] fix bug where `quality` had more priority than `hasvid`
+* [pyinst] Automatically detect python architecture and working directory
+* Strip out internal fields such as `_filename` from infojson
+
+
 ### 2021.01.29
-* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by @animelover1984 and @bbepis
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
     * Add `--get-comments`
     * [youtube] Extract comments
     * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
@@ -58,7 +122,7 @@
     * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
     * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
 * Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
-* [mildom] Add extractor by @nao20010128nao
+* [mildom] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
 * Warn when using old style `--external-downloader-args` and `--post-processor-args`
 * Fix `--no-overwrite` when using `--write-link`
 * [sponskrub] Output `unrecognized argument` error message correctly
@@ -89,7 +153,7 @@
 
 ### 2021.01.14
 * Added option `--break-on-reject`
-* [roosterteeth.com] Fix for bonus episodes by @Zocker1999NET
+* [roosterteeth.com] Fix for bonus episodes by [Zocker1999NET](https://github.com/Zocker1999NET)
 * [tiktok] Fix for when share_info is empty
 * [EmbedThumbnail] Fix bug due to incorrect function name
 * [documentation] Changed sponskrub links to point to [pukkandan/sponskrub](https://github.com/pukkandan/SponSkrub) since I am now providing both linux and windows releases
@@ -98,18 +162,18 @@
 
 
 ### 2021.01.12
-* [roosterteeth.com] Add subtitle support by @samiksome
+* [roosterteeth.com] Add subtitle support by [samiksome](https://github.com/samiksome)
-* Added `--force-overwrites`, `--no-force-overwrites` by @alxnull
+* Added `--force-overwrites`, `--no-force-overwrites` by [alxnull](https://github.com/alxnull)
 * Changed fork name to `yt-dlp`
-* Fix typos by @FelixFrog
+* Fix typos by [FelixFrog](https://github.com/FelixFrog)
 * [ci] Option to skip
 * [changelog] Added unreleased changes in blackjack4494/yt-dlc
 
 
 ### 2021.01.10
-* [archive.org] Fix extractor and add support for audio and playlists by @wporr
+* [archive.org] Fix extractor and add support for audio and playlists by [wporr](https://github.com/wporr)
-* [Animelab] Added by @mariuszskon
+* [Animelab] Added by [mariuszskon](https://github.com/mariuszskon)
-* [youtube:search] Fix view_count by @ohnonot
+* [youtube:search] Fix view_count by [ohnonot](https://github.com/ohnonot)
 * [youtube] Show if video is embeddable in info
 * Update version badge automatically in README
 * Enable `test_youtube_search_matching`
@@ -118,11 +182,11 @@
 
 ### 2021.01.09
 * [youtube] Fix bug in automatic caption extraction
-* Add `post_hooks` to YoutubeDL by @alexmerkel
+* Add `post_hooks` to YoutubeDL by [alexmerkel](https://github.com/alexmerkel)
-* Batch file enumeration improvements by @glenn-slayden
+* Batch file enumeration improvements by [glenn-slayden](https://github.com/glenn-slayden)
-* Stop immediately when reaching `--max-downloads` by @glenn-slayden
+* Stop immediately when reaching `--max-downloads` by [glenn-slayden](https://github.com/glenn-slayden)
-* Fix incorrect ANSI sequence for restoring console-window title by @glenn-slayden
+* Fix incorrect ANSI sequence for restoring console-window title by [glenn-slayden](https://github.com/glenn-slayden)
-* Kill child processes when yt-dlc is killed by @Unrud
+* Kill child processes when yt-dlc is killed by [Unrud](https://github.com/Unrud)
 
 
 ### 2021.01.08
@@ -132,11 +196,11 @@
 
 
 ### 2021.01.07-1
-* [Akamai] fix by @nixxo
+* [Akamai] fix by [nixxo](https://github.com/nixxo)
-* [Tiktok] merge youtube-dl tiktok extractor by @GreyAlien502
+* [Tiktok] merge youtube-dl tiktok extractor by [GreyAlien502](https://github.com/GreyAlien502)
-* [vlive] add support for playlists by @kyuyeunk
+* [vlive] add support for playlists by [kyuyeunk](https://github.com/kyuyeunk)
-* [youtube_live_chat] make sure playerOffsetMs is positive by @siikamiika
+* [youtube_live_chat] make sure playerOffsetMs is positive by [siikamiika](https://github.com/siikamiika)
-* Ignore extra data streams in ffmpeg by @jbruchon
+* Ignore extra data streams in ffmpeg by [jbruchon](https://github.com/jbruchon)
 * Allow passing different arguments to different postprocessors using `--postprocessor-args`
 * Deprecated `--sponskrub-args`. The same can now be done using `--postprocessor-args "sponskrub:<args>"`
 * [CI] Split tests into core-test and full-test
@@ -166,15 +230,15 @@
     * Changed video format sorting to show video only files and video+audio files together.
     * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
     * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
+* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
-* Added `--force-download-archive` (`--force-write-archive`) by @h-h-h-h
+* Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
 * Added `--list-formats-as-table`,  `--list-formats-old`
 * **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
     * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
     * Renamed: `--write-subs`, `--no-write-subs`, `--no-write-auto-subs`, `--write-auto-subs`. Note that these can still be used without the ending "s"
 * Relaxed validation for format filters so that any arbitrary field can be used
-* Fix for embedding thumbnail in mp3 by @pauldubois98
+* Fix for embedding thumbnail in mp3 by [pauldubois98](https://github.com/pauldubois98) ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
 * Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
 * **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
     * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
@@ -191,7 +255,7 @@
     * Redirect channel home to /video
     * Print youtube's warning message
     * Multiple pages are handled better for feeds
-* Add --break-on-existing by @gergesh
+* Add --break-on-existing by [gergesh](https://github.com/gergesh)
 * Pre-check video IDs in the archive before downloading
 * [bitwave.tv] New extractor
 * [Gedi] Add extractor

Makefile

@@ -4,7 +4,7 @@ man: README.txt youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtub
 
 
 clean:
-	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
+	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec *.frag *.frag.urls *.frag.aria2 CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
 

README.md

@@ -2,7 +2,9 @@
 
 [![Release version](https://img.shields.io/github/v/release/pukkandan/yt-dlp?color=brightgreen&label=Release)](https://github.com/pukkandan/yt-dlp/releases/latest)
 [![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
-[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)  
+[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)
+[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord)](https://discord.gg/S75JaBna)  
+
 [![Commits](https://img.shields.io/github/commit-activity/m/pukkandan/yt-dlp?label=commits)](https://github.com/pukkandan/yt-dlp/commits)
 [![Last Commit](https://img.shields.io/github/last-commit/pukkandan/yt-dlp/master)](https://github.com/pukkandan/yt-dlp/commits)
 [![Downloads](https://img.shields.io/github/downloads/pukkandan/yt-dlp/total)](https://github.com/pukkandan/yt-dlp/releases/latest)
@@ -54,22 +56,24 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl v2021.01.24.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl v2021.02.10**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/pukkandan/yt-dlp/pull/31) for details.
 
 * **Youtube improvements**:
-    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and supports downloading multiple pages of content
-    * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
+    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs works correctly
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
 
+* **Aria2c with HLS/DASH**: You can use aria2c as the external downloader for DASH(mpd) and HLS(m3u8) formats. No more slow ffmpeg/native downloads
+
 * **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina, rumble
 
 * **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
 
-* **Multiple paths**: You can give different paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [`--paths`](https://github.com/pukkandan/yt-dlp/#:~:text=-P,%20--paths%20TYPE:PATH) for details
+* **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [`--paths`](https://github.com/pukkandan/yt-dlp/#:~:text=-P,%20--paths%20TYPE:PATH) for details
 
 <!-- Relative link doesn't work for "#:~:text=" -->
 
@@ -77,7 +81,9 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Other new options**: `--parse-metadata`, `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
 
-* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, `%(duration_string)s` in `-o`, faster archive checking, more [format selection options](#format-selection) etc
+* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc
+
+* **Self-updater**: The releases can be updated using `youtube-dlc -U`
 
 
 See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes
@@ -91,22 +97,23 @@ If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the
 # INSTALLATION
 
 You can install yt-dlp using one of the following methods:
+* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest) (recommended method)
 * Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
-* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest)
 * Use pip+git: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp.git@release`
 * Install master branch: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp`
 
 ### UPDATE
-`-U` does not work. Simply repeat the install process to update.
+Starting from version `2021.02.09`, you can use `youtube-dlc -U` to update if you are using the provided release.
+If you are using `pip`, simply re-run the same command that was used to install the program.
 
 ### COMPILE
 
 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen for embedding thumbnail in opus/ogg files)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen and pycryptodome)
 
-    python -m pip install --upgrade pyinstaller mutagen
+    python -m pip install --upgrade pyinstaller mutagen pycryptodome
 
-For the 64bit version, run `py devscripts\pyinst.py 64` using 64bit python3. Similarly, to install 32bit version, run `py devscripts\pyinst.py 32` using 32bit python (preferably 3)
+Once you have all the necessary dependancies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
 
 You can also build the executable without any version info or metadata by using:
 
@@ -134,9 +141,9 @@ Then simply type this
 ## General Options:
     -h, --help                       Print this help text and exit
     --version                        Print program version and exit
-    -U, --update                     [BROKEN] Update this program to latest
+    -U, --update                     Update this program to latest version. Make
-                                     version. Make sure that you have sufficient
+                                     sure that you have sufficient permissions
-                                     permissions (run with sudo if needed)
+                                     (run with sudo if needed)
     -i, --ignore-errors              Continue on download errors, for example to
                                      skip unavailable videos in a playlist
                                      (default) (Alias: --no-abort-on-error)
@@ -333,16 +340,16 @@ Then simply type this
                                      comments and ignored
     -P, --paths TYPE:PATH            The paths where the files should be
                                      downloaded. Specify the type of file and
-                                     the path separated by a colon ":"
+                                     the path separated by a colon ":". All the
-                                     (supported: description|annotation|subtitle
+                                     same types as --output are supported.
-                                     |infojson|thumbnail). Additionally, you can
+                                     Additionally, you can also provide "home"
-                                     also provide "home" and "temp" paths. All
+                                     and "temp" paths. All intermediary files
-                                     intermediary files are first downloaded to
+                                     are first downloaded to the temp path and
-                                     the temp path and then the final files are
+                                     then the final files are moved over to the
-                                     moved over to the home path after download
+                                     home path after download is finished. This
-                                     is finished. Note that this option is
+                                     option is ignored if --output is an
-                                     ignored if --output is an absolute path
+                                     absolute path
-    -o, --output TEMPLATE            Output filename template, see "OUTPUT
+    -o, --output [TYPE:]TEMPLATE     Output filename template, see "OUTPUT
                                      TEMPLATE" for details
     --output-na-placeholder TEXT     Placeholder value for unavailable meta
                                      fields in output filename template
@@ -359,9 +366,11 @@ Then simply type this
                                      This option includes --no-continue
     --no-force-overwrites            Do not overwrite the video, but overwrite
                                      related files (default)
-    -c, --continue                   Resume partially downloaded files (default)
+    -c, --continue                   Resume partially downloaded files/fragments
-    --no-continue                    Restart download of partially downloaded
+                                     (default)
-                                     files from beginning
+    --no-continue                    Do not resume partially downloaded
+                                     fragments. If the file is unfragmented,
+                                     restart download of the entire file
     --part                           Use .part files instead of writing directly
                                      into output file (default)
     --no-part                        Do not use .part files - write directly
@@ -374,10 +383,16 @@ Then simply type this
                                      file
     --no-write-description           Do not write video description (default)
     --write-info-json                Write video metadata to a .info.json file
+                                     (this may contain personal information)
     --no-write-info-json             Do not write video metadata (default)
     --write-annotations              Write video annotations to a
                                      .annotations.xml file
     --no-write-annotations           Do not write video annotations (default)
+    --write-playlist-metafiles       Write playlist metadata in addition to the
+                                     video metadata when using --write-info-json,
+                                     --write-description etc. (default)
+    --no-write-playlist-metafiles    Do not write playlist metadata when using
+                                     --write-info-json, --write-description etc.
     --get-comments                   Retrieve video comments to be placed in the
                                      .info.json file
     --load-info-json FILE            JSON file containing the video information
@@ -453,10 +468,6 @@ Then simply type this
                                      files in the current directory to debug
                                      problems
     --print-traffic                  Display sent and read HTTP traffic
-    -C, --call-home                  [Broken] Contact the youtube-dlc server for
-                                     debugging
-    --no-call-home                   Do not contact the youtube-dlc server for
-                                     debugging (default)
 
 ## Workarounds:
     --encoding ENCODING              Force the specified encoding (experimental)
@@ -529,6 +540,11 @@ Then simply type this
                                      bestvideo+bestaudio), output to given
                                      container format. One of mkv, mp4, ogg,
                                      webm, flv. Ignored if no merge is required
+    --allow-unplayable-formats       Allow unplayable formats to be listed and
+                                     downloaded. All video postprocessing will
+                                     also be turned off
+    --no-allow-unplayable-formats    Do not allow unplayable formats to be
+                                     listed or downloaded (default)
 
 ## Subtitle Options:
     --write-subs                     Write subtitle file
@@ -746,7 +762,11 @@ The `-o` option is used to indicate a template for the output file names while `
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
-The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dlc -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Allowed names along with sequence type are:
+The basic usage of `-o` is not to set any template arguments when downloading a single file, like in `youtube-dlc -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Date/time fields can also be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it inside the parantheses seperated from the field name using a `>`. For example, `%(duration>%H-%M-%S)s`.
+
+Additionally, you can set different output templates for the various metadata files seperately from the general output template by specifying the type of file followed by the template seperated by a colon ":". The different filetypes supported are `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
+
+The available fields are:
 
  - `id` (string): Video identifier
  - `title` (string): Video title
@@ -853,7 +873,7 @@ If you are using an output template inside a Windows batch file then you must es
 
 #### Output template examples
 
-Note that on Windows you may need to use double quotes instead of single.
+Note that on Windows you need to use double quotes instead of single.
 
 ```bash
 $ youtube-dlc --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc
@@ -865,14 +885,17 @@ youtube-dlc_test_video_.mp4          # A simple file name
 # Download YouTube playlist videos in separate directory indexed by video order in a playlist
 $ youtube-dlc -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re
 
+# Download YouTube playlist videos in seperate directories according to their uploaded year
+$ youtube-dlc -o '%(upload_date>%Y)s/%(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re
+
 # Download all playlists of YouTube channel/user keeping each playlist in separate directory:
 $ youtube-dlc -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists
 
 # Download Udemy course keeping each chapter in separate directory under MyVideos directory in your home
-$ youtube-dlc -u user -p password -o '~/MyVideos/%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/
+$ youtube-dlc -u user -p password -P '~/MyVideos' -o '%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/
 
 # Download entire series season keeping each series and each season in separate directory under C:/MyVideos
-$ youtube-dlc -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617
+$ youtube-dlc -P "C:/MyVideos" -o "%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617
 
 # Stream the video being downloaded to stdout
 $ youtube-dlc -o - BaW_jenozKc
@@ -958,7 +981,7 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `quality`: The quality of the format. This is a metadata field available in some websites
  - `source`: Preference of the source as given by the extractor
  - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `vcodec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
  - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
  - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
@@ -979,7 +1002,7 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
 
 Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec:vp9,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 

devscripts/update-version.py

@@ -8,7 +8,7 @@ from datetime import datetime
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 old_version = locals()['__version__']
 
-old_version_list = old_version.replace('-', '.').split(".", 4)
+old_version_list = old_version.split(".", 4)
 
 old_ver = '.'.join(old_version_list[:3])
 old_rev = old_version_list[3] if len(old_version_list) > 3 else ''

docs/supportedsites.md

@@ -1,6 +1,5 @@
 # Supported sites
  - **1tv**: Первый канал
- - **1up.com**
  - **20min**
  - **220.ro**
  - **23video**
@@ -80,8 +79,9 @@
  - **AudioBoom**
  - **audiomack**
  - **audiomack:album**
- - **Audius**
+ - **Audius**: Audius.co
- - **audius:playlist**
+ - **audius:playlist**: Audius.co playlists
+ - **audius:track**: Audius track ID or API link. Prepend with "audius:"
  - **AWAAN**
  - **awaan:live**
  - **awaan:season**
@@ -393,6 +393,8 @@
  - **HungamaSong**
  - **Hypem**
  - **ign.com**
+ - **IGNArticle**
+ - **IGNVideo**
  - **IHeartRadio**
  - **iheartradio:podcast**
  - **imdb**: Internet Movie Database trailers
@@ -558,6 +560,7 @@
  - **mtv:video**
  - **mtvjapan**
  - **mtvservices:embedded**
+ - **MTVUutisetArticle**
  - **MuenchenTV**: münchen.tv
  - **mva**: Microsoft Virtual Academy videos
  - **mva:course**: Microsoft Virtual Academy courses
@@ -699,7 +702,6 @@
  - **parliamentlive.tv**: UK parliament videos
  - **Patreon**
  - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
- - **pcmag**
  - **PearVideo**
  - **PeerTube**
  - **People**
@@ -1089,7 +1091,6 @@
  - **vidme**
  - **vidme:user**
  - **vidme:user:likes**
- - **Vidzi**
  - **vier**: vier.be and vijf.be
  - **vier:videos**
  - **viewlift**
@@ -1134,6 +1135,7 @@
  - **vrv**
  - **vrv:series**
  - **VShare**
+ - **VTM**
  - **VTXTV**
  - **vube**: Vube.com
  - **VuClip**
@@ -1216,9 +1218,9 @@
  - **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
  - **youtube:playlist**: YouTube.com playlists
  - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
- - **youtube:search**: YouTube.com searches
+ - **youtube:search**: YouTube.com searches, "ytsearch" keyword
  - **youtube:search:date**: YouTube.com searches, newest videos first, "ytsearchdate" keyword
- - **youtube:search_url**: YouTube.com searches, "ytsearch" keyword
+ - **youtube:search_url**: YouTube.com search URLs
  - **youtube:subscriptions**: YouTube.com subscriptions feed, ":ytsubs" for short (requires authentication)
  - **youtube:tab**: YouTube.com tab
  - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)

pyinst.py

@@ -1,5 +1,10 @@
+#!/usr/bin/env python
+# coding: utf-8
+
 from __future__ import unicode_literals
 import sys
+# import os
+import platform
 
 from PyInstaller.utils.win32.versioninfo import (
     VarStruct, VarFileInfo, StringStruct, StringTable,
@@ -7,18 +12,21 @@ from PyInstaller.utils.win32.versioninfo import (
 )
 import PyInstaller.__main__
 
+arch = sys.argv[1] if len(sys.argv) > 1 else platform.architecture()[0][:2]
+assert arch in ('32', '64')
+print('Building %sbit version' % arch)
+_x86 = '_x86' if arch == '32' else ''
 
-assert len(sys.argv) > 1 and sys.argv[1] in ("32", "64")
+FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
-_x86 = "_x86" if sys.argv[1] == "32" else ""
-
-FILE_DESCRIPTION = 'Media Downloader%s' % (" (32 Bit)" if _x86 else '')
-SHORT_URLS = {"32": "git.io/JUGsM", "64": "git.io/JLh7K"}
 
+# root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+# print('Changing working directory to %s' % root_dir)
+# os.chdir(root_dir)
 
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 VERSION = locals()['__version__']
 
-VERSION_LIST = VERSION.replace('-', '.').split('.')
+VERSION_LIST = VERSION.split('.')
 VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
 
 print('Version: %s%s' % (VERSION, _x86))
@@ -38,21 +46,21 @@ VERSION_FILE = VSVersionInfo(
     kids=[
         StringFileInfo([
             StringTable(
-                "040904B0", [
+                '040904B0', [
-                    StringStruct("Comments", "Youtube-dlc%s Command Line Interface." % _x86),
+                    StringStruct('Comments', 'Youtube-dlc%s Command Line Interface.' % _x86),
-                    StringStruct("CompanyName", "pukkandan@gmail.com"),
+                    StringStruct('CompanyName', 'https://github.com/pukkandan/yt-dlp'),
-                    StringStruct("FileDescription", FILE_DESCRIPTION),
+                    StringStruct('FileDescription', FILE_DESCRIPTION),
-                    StringStruct("FileVersion", VERSION),
+                    StringStruct('FileVersion', VERSION),
-                    StringStruct("InternalName", "youtube-dlc%s" % _x86),
+                    StringStruct('InternalName', 'youtube-dlc%s' % _x86),
                     StringStruct(
-                        "LegalCopyright",
+                        'LegalCopyright',
-                        "pukkandan@gmail.com | UNLICENSE",
+                        'pukkandan@gmail.com | UNLICENSE',
                     ),
-                    StringStruct("OriginalFilename", "youtube-dlc%s.exe" % _x86),
+                    StringStruct('OriginalFilename', 'youtube-dlc%s.exe' % _x86),
-                    StringStruct("ProductName", "Youtube-dlc%s" % _x86),
+                    StringStruct('ProductName', 'Youtube-dlc%s' % _x86),
-                    StringStruct("ProductVersion", "%s%s | %s" % (VERSION, _x86, SHORT_URLS[sys.argv[1]])),
+                    StringStruct('ProductVersion', '%s%s' % (VERSION, _x86)),
                 ])]),
-        VarFileInfo([VarStruct("Translation", [0, 1200])])
+        VarFileInfo([VarStruct('Translation', [0, 1200])])
     ]
 )
 
@@ -64,6 +72,7 @@ PyInstaller.__main__.run([
     '--exclude-module=test',
     '--exclude-module=ytdlp_plugins',
     '--hidden-import=mutagen',
+    '--hidden-import=Crypto',
     'youtube_dlc/__main__.py',
 ])
 SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

requirements.txt

@@ -1 +1,2 @@
 mutagen
+pycryptodome

setup.py

@@ -20,7 +20,7 @@ LONG_DESCRIPTION = '\n\n'.join((
     '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
     open("README.md", "r", encoding="utf-8").read()))
 
-REQUIREMENTS = ['mutagen']
+REQUIREMENTS = ['mutagen', 'pycryptodome']
 
 
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':

test/test_youtube_chapters.py

@@ -1,275 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-from __future__ import unicode_literals
-
-# Allow direct execution
-import os
-import sys
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from test.helper import expect_value
-from youtube_dlc.extractor import YoutubeIE
-
-
-class TestYoutubeChapters(unittest.TestCase):
-
-    _TEST_CASES = [
-        (
-            # https://www.youtube.com/watch?v=A22oy8dFjqc
-            # pattern: 00:00 - <title>
-            '''This is the absolute ULTIMATE experience of Queen's set at LIVE AID, this is the best video mixed to the absolutely superior stereo radio broadcast. This vastly superior audio mix takes a huge dump on all of the official mixes. Best viewed in 1080p. ENJOY! ***MAKE SURE TO READ THE DESCRIPTION***<br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+36);return false;">00:36</a> - Bohemian Rhapsody<br /><a href="#" onclick="yt.www.watch.player.seekTo(02*60+42);return false;">02:42</a> - Radio Ga Ga<br /><a href="#" onclick="yt.www.watch.player.seekTo(06*60+53);return false;">06:53</a> - Ay Oh!<br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+34);return false;">07:34</a> - Hammer To Fall<br /><a href="#" onclick="yt.www.watch.player.seekTo(12*60+08);return false;">12:08</a> - Crazy Little Thing Called Love<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+03);return false;">16:03</a> - We Will Rock You<br /><a href="#" onclick="yt.www.watch.player.seekTo(17*60+18);return false;">17:18</a> - We Are The Champions<br /><a href="#" onclick="yt.www.watch.player.seekTo(21*60+12);return false;">21:12</a> - Is This The World We Created...?<br /><br />Short song analysis:<br /><br />- "Bohemian Rhapsody": Although it's a short medley version, it's one of the best performances of the ballad section, with Freddie nailing the Bb4s with the correct studio phrasing (for the first time ever!).<br /><br />- "Radio Ga Ga": Although it's missing one chorus, this is one of - if not the best - the best versions ever, Freddie nails all the Bb4s and sounds very clean! Spike Edney's Roland Jupiter 8 also really shines through on this mix, compared to the DVD releases!<br /><br />- "Audience Improv": A great improv, Freddie sounds strong and confident. You gotta love when he sustains that A4 for 4 seconds!<br /><br />- "Hammer To Fall": Despite missing a verse and a chorus, it's a strong version (possibly the best ever). Freddie sings the song amazingly, and even ad-libs a C#5 and a C5! Also notice how heavy Brian's guitar sounds compared to the thin DVD mixes - it roars!<br /><br />- "Crazy Little Thing Called Love": A great version, the crowd loves the song, the jam is great as well! Only downside to this is the slight feedback issues.<br /><br />- "We Will Rock You": Although cut down to the 1st verse and chorus, Freddie sounds strong. He nails the A4, and the solo from Dr. May is brilliant!<br /><br />- "We Are the Champions": Perhaps the high-light of the performance - Freddie is very daring on this version, he sustains the pre-chorus Bb4s, nails the 1st C5, belts great A4s, but most importantly: He nails the chorus Bb4s, in all 3 choruses! This is the only time he has ever done so! It has to be said though, the last one sounds a bit rough, but that's a side effect of belting high notes for the past 18 minutes, with nodules AND laryngitis!<br /><br />- "Is This The World We Created... ?": Freddie and Brian perform a beautiful version of this, and it is one of the best versions ever. It's both sad and hilarious that a couple of BBC engineers are talking over the song, one of them being completely oblivious of the fact that he is interrupting the performance, on live television... Which was being televised to almost 2 billion homes.<br /><br /><br />All rights go to their respective owners!<br />-----Copyright Disclaimer Under Section 107 of the Copyright Act 1976, allowance is made for fair use for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing. Non-profit, educational or personal use tips the balance in favor of fair use''',
-            1477,
-            [{
-                'start_time': 36,
-                'end_time': 162,
-                'title': 'Bohemian Rhapsody',
-            }, {
-                'start_time': 162,
-                'end_time': 413,
-                'title': 'Radio Ga Ga',
-            }, {
-                'start_time': 413,
-                'end_time': 454,
-                'title': 'Ay Oh!',
-            }, {
-                'start_time': 454,
-                'end_time': 728,
-                'title': 'Hammer To Fall',
-            }, {
-                'start_time': 728,
-                'end_time': 963,
-                'title': 'Crazy Little Thing Called Love',
-            }, {
-                'start_time': 963,
-                'end_time': 1038,
-                'title': 'We Will Rock You',
-            }, {
-                'start_time': 1038,
-                'end_time': 1272,
-                'title': 'We Are The Champions',
-            }, {
-                'start_time': 1272,
-                'end_time': 1477,
-                'title': 'Is This The World We Created...?',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=ekYlRhALiRQ
-            # pattern: <num>. <title> 0:00
-            '1.  Those Beaten Paths of Confusion <a href="#" onclick="yt.www.watch.player.seekTo(0*60+00);return false;">0:00</a><br />2.  Beyond the Shadows of Emptiness & Nothingness <a href="#" onclick="yt.www.watch.player.seekTo(11*60+47);return false;">11:47</a><br />3.  Poison Yourself...With Thought <a href="#" onclick="yt.www.watch.player.seekTo(26*60+30);return false;">26:30</a><br />4.  The Agents of Transformation <a href="#" onclick="yt.www.watch.player.seekTo(35*60+57);return false;">35:57</a><br />5.  Drowning in the Pain of Consciousness <a href="#" onclick="yt.www.watch.player.seekTo(44*60+32);return false;">44:32</a><br />6.  Deny the Disease of Life <a href="#" onclick="yt.www.watch.player.seekTo(53*60+07);return false;">53:07</a><br /><br />More info/Buy: http://crepusculonegro.storenvy.com/products/257645-cn-03-arizmenda-within-the-vacuum-of-infinity<br /><br />No copyright is intended. The rights to this video are assumed by the owner and its affiliates.',
-            4009,
-            [{
-                'start_time': 0,
-                'end_time': 707,
-                'title': '1. Those Beaten Paths of Confusion',
-            }, {
-                'start_time': 707,
-                'end_time': 1590,
-                'title': '2. Beyond the Shadows of Emptiness & Nothingness',
-            }, {
-                'start_time': 1590,
-                'end_time': 2157,
-                'title': '3. Poison Yourself...With Thought',
-            }, {
-                'start_time': 2157,
-                'end_time': 2672,
-                'title': '4. The Agents of Transformation',
-            }, {
-                'start_time': 2672,
-                'end_time': 3187,
-                'title': '5. Drowning in the Pain of Consciousness',
-            }, {
-                'start_time': 3187,
-                'end_time': 4009,
-                'title': '6. Deny the Disease of Life',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=WjL4pSzog9w
-            # pattern: 00:00 <title>
-            '<a href="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" class="yt-uix-servicelink  " data-target-new-window="True" data-servicelink="CDAQ6TgYACITCNf1raqT2dMCFdRjGAod_o0CBSj4HQ" data-url="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" rel="nofollow noopener" target="_blank">https://arizmenda.bandcamp.com/merch/...</a><br /><br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+00);return false;">00:00</a> Christening Unborn Deformities <br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+08);return false;">07:08</a> Taste of Purity<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+16);return false;">16:16</a> Sculpting Sins of a Universal Tongue<br /><a href="#" onclick="yt.www.watch.player.seekTo(24*60+45);return false;">24:45</a> Birth<br /><a href="#" onclick="yt.www.watch.player.seekTo(31*60+24);return false;">31:24</a> Neves<br /><a href="#" onclick="yt.www.watch.player.seekTo(37*60+55);return false;">37:55</a> Libations in Limbo',
-            2705,
-            [{
-                'start_time': 0,
-                'end_time': 428,
-                'title': 'Christening Unborn Deformities',
-            }, {
-                'start_time': 428,
-                'end_time': 976,
-                'title': 'Taste of Purity',
-            }, {
-                'start_time': 976,
-                'end_time': 1485,
-                'title': 'Sculpting Sins of a Universal Tongue',
-            }, {
-                'start_time': 1485,
-                'end_time': 1884,
-                'title': 'Birth',
-            }, {
-                'start_time': 1884,
-                'end_time': 2275,
-                'title': 'Neves',
-            }, {
-                'start_time': 2275,
-                'end_time': 2705,
-                'title': 'Libations in Limbo',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=o3r1sn-t3is
-            # pattern: <title> 00:00 <note>
-            'Download this show in MP3: <a href="http://sh.st/njZKK" class="yt-uix-servicelink  " data-url="http://sh.st/njZKK" data-target-new-window="True" data-servicelink="CDAQ6TgYACITCK3j8_6o2dMCFVDCGAoduVAKKij4HQ" rel="nofollow noopener" target="_blank">http://sh.st/njZKK</a><br /><br />Setlist:<br />I-E-A-I-A-I-O <a href="#" onclick="yt.www.watch.player.seekTo(00*60+45);return false;">00:45</a><br />Suite-Pee <a href="#" onclick="yt.www.watch.player.seekTo(4*60+26);return false;">4:26</a>  (Incomplete)<br />Attack <a href="#" onclick="yt.www.watch.player.seekTo(5*60+31);return false;">5:31</a> (First live performance since 2011)<br />Prison Song <a href="#" onclick="yt.www.watch.player.seekTo(8*60+42);return false;">8:42</a><br />Know <a href="#" onclick="yt.www.watch.player.seekTo(12*60+32);return false;">12:32</a> (First live performance since 2011)<br />Aerials <a href="#" onclick="yt.www.watch.player.seekTo(15*60+32);return false;">15:32</a><br />Soldier Side - Intro <a href="#" onclick="yt.www.watch.player.seekTo(19*60+13);return false;">19:13</a><br />B.Y.O.B. <a href="#" onclick="yt.www.watch.player.seekTo(20*60+09);return false;">20:09</a><br />Soil <a href="#" onclick="yt.www.watch.player.seekTo(24*60+32);return false;">24:32</a><br />Darts <a href="#" onclick="yt.www.watch.player.seekTo(27*60+48);return false;">27:48</a><br />Radio/Video <a href="#" onclick="yt.www.watch.player.seekTo(30*60+38);return false;">30:38</a><br />Hypnotize <a href="#" onclick="yt.www.watch.player.seekTo(35*60+05);return false;">35:05</a><br />Temper <a href="#" onclick="yt.www.watch.player.seekTo(38*60+08);return false;">38:08</a> (First live performance since 1999)<br />CUBErt <a href="#" onclick="yt.www.watch.player.seekTo(41*60+00);return false;">41:00</a><br />Needles <a href="#" onclick="yt.www.watch.player.seekTo(42*60+57);return false;">42:57</a><br />Deer Dance <a href="#" onclick="yt.www.watch.player.seekTo(46*60+27);return false;">46:27</a><br />Bounce <a href="#" onclick="yt.www.watch.player.seekTo(49*60+38);return false;">49:38</a><br />Suggestions <a href="#" onclick="yt.www.watch.player.seekTo(51*60+25);return false;">51:25</a><br />Psycho <a href="#" onclick="yt.www.watch.player.seekTo(53*60+52);return false;">53:52</a><br />Chop Suey! <a href="#" onclick="yt.www.watch.player.seekTo(58*60+13);return false;">58:13</a><br />Lonely Day <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+01*60+15);return false;">1:01:15</a><br />Question! <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+04*60+14);return false;">1:04:14</a><br />Lost in Hollywood <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+08*60+10);return false;">1:08:10</a><br />Vicinity of Obscenity  <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+13*60+40);return false;">1:13:40</a>(First live performance since 2012)<br />Forest <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+16*60+17);return false;">1:16:17</a><br />Cigaro <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+20*60+02);return false;">1:20:02</a><br />Toxicity <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+23*60+57);return false;">1:23:57</a>(with Chino Moreno)<br />Sugar <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+27*60+53);return false;">1:27:53</a>',
-            5640,
-            [{
-                'start_time': 45,
-                'end_time': 266,
-                'title': 'I-E-A-I-A-I-O',
-            }, {
-                'start_time': 266,
-                'end_time': 331,
-                'title': 'Suite-Pee (Incomplete)',
-            }, {
-                'start_time': 331,
-                'end_time': 522,
-                'title': 'Attack (First live performance since 2011)',
-            }, {
-                'start_time': 522,
-                'end_time': 752,
-                'title': 'Prison Song',
-            }, {
-                'start_time': 752,
-                'end_time': 932,
-                'title': 'Know (First live performance since 2011)',
-            }, {
-                'start_time': 932,
-                'end_time': 1153,
-                'title': 'Aerials',
-            }, {
-                'start_time': 1153,
-                'end_time': 1209,
-                'title': 'Soldier Side - Intro',
-            }, {
-                'start_time': 1209,
-                'end_time': 1472,
-                'title': 'B.Y.O.B.',
-            }, {
-                'start_time': 1472,
-                'end_time': 1668,
-                'title': 'Soil',
-            }, {
-                'start_time': 1668,
-                'end_time': 1838,
-                'title': 'Darts',
-            }, {
-                'start_time': 1838,
-                'end_time': 2105,
-                'title': 'Radio/Video',
-            }, {
-                'start_time': 2105,
-                'end_time': 2288,
-                'title': 'Hypnotize',
-            }, {
-                'start_time': 2288,
-                'end_time': 2460,
-                'title': 'Temper (First live performance since 1999)',
-            }, {
-                'start_time': 2460,
-                'end_time': 2577,
-                'title': 'CUBErt',
-            }, {
-                'start_time': 2577,
-                'end_time': 2787,
-                'title': 'Needles',
-            }, {
-                'start_time': 2787,
-                'end_time': 2978,
-                'title': 'Deer Dance',
-            }, {
-                'start_time': 2978,
-                'end_time': 3085,
-                'title': 'Bounce',
-            }, {
-                'start_time': 3085,
-                'end_time': 3232,
-                'title': 'Suggestions',
-            }, {
-                'start_time': 3232,
-                'end_time': 3493,
-                'title': 'Psycho',
-            }, {
-                'start_time': 3493,
-                'end_time': 3675,
-                'title': 'Chop Suey!',
-            }, {
-                'start_time': 3675,
-                'end_time': 3854,
-                'title': 'Lonely Day',
-            }, {
-                'start_time': 3854,
-                'end_time': 4090,
-                'title': 'Question!',
-            }, {
-                'start_time': 4090,
-                'end_time': 4420,
-                'title': 'Lost in Hollywood',
-            }, {
-                'start_time': 4420,
-                'end_time': 4577,
-                'title': 'Vicinity of Obscenity (First live performance since 2012)',
-            }, {
-                'start_time': 4577,
-                'end_time': 4802,
-                'title': 'Forest',
-            }, {
-                'start_time': 4802,
-                'end_time': 5037,
-                'title': 'Cigaro',
-            }, {
-                'start_time': 5037,
-                'end_time': 5273,
-                'title': 'Toxicity (with Chino Moreno)',
-            }, {
-                'start_time': 5273,
-                'end_time': 5640,
-                'title': 'Sugar',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=PkYLQbsqCE8
-            # pattern: <num> - <title> [<latinized title>] 0:00:00
-            '''Затемно (Zatemno) is an Obscure Black Metal Band from Russia.<br /><br />"Во прах (Vo prakh)'' Into The Ashes", Debut mini-album released may 6, 2016, by Death Knell Productions<br />Released on 6 panel digipak CD, limited to 100 copies only<br />And digital format on Bandcamp<br /><br />Tracklist<br /><br />1 - Во прах [Vo prakh] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+00*60+00);return false;">0:00:00</a><br />2 - Искупление [Iskupleniye] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+08*60+10);return false;">0:08:10</a><br />3 - Из серпов луны...[Iz serpov luny] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+14*60+30);return false;">0:14:30</a><br /><br />Links:<br /><a href="https://deathknellprod.bandcamp.com/album/--2" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://deathknellprod.bandcamp.com/album/--2" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://deathknellprod.bandcamp.com/a...</a><br /><a href="https://www.facebook.com/DeathKnellProd/" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://www.facebook.com/DeathKnellProd/" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://www.facebook.com/DeathKnellProd/</a><br /><br /><br />I don't have any right about this artifact, my only intention is to spread the music of the band, all rights are reserved to the Затемно (Zatemno) and his producers, Death Knell Productions.<br /><br />------------------------------------------------------------------<br /><br />Subscribe for more videos like this.<br />My link: <a href="https://web.facebook.com/AttackOfTheDragons" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://web.facebook.com/AttackOfTheDragons" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://web.facebook.com/AttackOfTheD...</a>''',
-            1138,
-            [{
-                'start_time': 0,
-                'end_time': 490,
-                'title': '1 - Во прах [Vo prakh]',
-            }, {
-                'start_time': 490,
-                'end_time': 870,
-                'title': '2 - Искупление [Iskupleniye]',
-            }, {
-                'start_time': 870,
-                'end_time': 1138,
-                'title': '3 - Из серпов луны...[Iz serpov luny]',
-            }]
-        ),
-        (
-            # https://www.youtube.com/watch?v=xZW70zEasOk
-            # time point more than duration
-            '''● LCS Spring finals: Saturday and Sunday from <a href="#" onclick="yt.www.watch.player.seekTo(13*60+30);return false;">13:30</a> outside the venue! <br />● PAX East: Fri, Sat & Sun - more info in tomorrows video on the main channel!''',
-            283,
-            []
-        ),
-    ]
-
-    def test_youtube_chapters(self):
-        for description, duration, expected_chapters in self._TEST_CASES:
-            ie = YoutubeIE()
-            expect_value(
-                self, ie._extract_chapters_from_description(description, duration),
-                expected_chapters, None)
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_youtube_signature.py

@@ -19,55 +19,46 @@ from youtube_dlc.compat import compat_str, compat_urlretrieve
 _TESTS = [
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vflHOr_nV.js',
-        'js',
         86,
         '>=<;:/.-[+*)(\'&%$#"!ZYX0VUTSRQPONMLKJIHGFEDCBA\\yxwvutsrqponmlkjihgfedcba987654321',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vfldJ8xgI.js',
-        'js',
         85,
         '3456789a0cdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRS[UVWXYZ!"#$%&\'()*+,-./:;<=>?@',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vfle-mVwz.js',
-        'js',
         90,
         ']\\[@?>=<;:/.-,+*)(\'&%$#"hZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjiagfedcb39876',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl0Cbn9e.js',
-        'js',
         84,
         'O1I3456789abcde0ghijklmnopqrstuvwxyzABCDEFGHfJKLMN2PQRSTUVW@YZ!"#$%&\'()*+,-./:;<=',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js',
-        'js',
         '2ACFC7A61CA478CD21425E5A57EBD73DDC78E22A.2094302436B2D377D14A3BBA23022D023B8BC25AA',
         'A52CB8B320D22032ABB3A41D773D2B6342034902.A22E87CDD37DBE75A5E52412DC874AC16A7CFCA2',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflBb0OQx.js',
-        'js',
         84,
         '123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ0STUVWXYZ!"#$%&\'()*+,@./:;<=>'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl9FYC6l.js',
-        'js',
         83,
         '123456789abcdefghijklmnopqr0tuvwxyzABCDETGHIJKLMNOPQRS>UVWXYZ!"#$%&\'()*+,-./:;<=F'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflCGk6yw/html5player.js',
-        'js',
         '4646B5181C6C3020DF1D9C7FCFEA.AD80ABF70C39BD369CCCAE780AFBB98FA6B6CB42766249D9488C288',
         '82C8849D94266724DC6B6AF89BBFA087EACCD963.B93C07FBA084ACAEFCF7C9D1FD0203C6C1815B6B'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js',
-        'js',
         '312AA52209E3623129A412D56A40F11CB0AF14AE.3EE09501CB14E3BCDC3B2AE808BF3F1D14E7FBF12',
         '112AA5220913623229A412D56A40F11CB0AF14AE.3EE0950FCB14EEBCDC3B2AE808BF331D14E7FBF3',
     )
@@ -78,6 +69,10 @@ class TestPlayerInfo(unittest.TestCase):
     def test_youtube_extract_player_info(self):
         PLAYER_URLS = (
             ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/en_US/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/fr_FR/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-en_US.vflset/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-de_DE.vflset/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-tablet-en_US.vflset/base.js', '64dddad9'),
             # obsolete
             ('https://www.youtube.com/yts/jsbin/player_ias-vfle4-e03/en_US/base.js', 'vfle4-e03'),
             ('https://www.youtube.com/yts/jsbin/player_ias-vfl49f_g4/en_US/base.js', 'vfl49f_g4'),
@@ -86,13 +81,9 @@ class TestPlayerInfo(unittest.TestCase):
             ('https://www.youtube.com/yts/jsbin/player-en_US-vflaxXRn1/base.js', 'vflaxXRn1'),
             ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js', 'vflXGBaUN'),
             ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js', 'vflKjOTVq'),
-            ('http://s.ytimg.com/yt/swfbin/watch_as3-vflrEm9Nq.swf', 'vflrEm9Nq'),
-            ('https://s.ytimg.com/yts/swfbin/player-vflenCdZL/watch_as3.swf', 'vflenCdZL'),
         )
         for player_url, expected_player_id in PLAYER_URLS:
-            expected_player_type = player_url.split('.')[-1]
+            player_id = YoutubeIE._extract_player_info(player_url)
-            player_type, player_id = YoutubeIE._extract_player_info(player_url)
-            self.assertEqual(player_type, expected_player_type)
             self.assertEqual(player_id, expected_player_id)
 
 
@@ -104,13 +95,13 @@ class TestSignature(unittest.TestCase):
             os.mkdir(self.TESTDATA_DIR)
 
 
-def make_tfunc(url, stype, sig_input, expected_sig):
+def make_tfunc(url, sig_input, expected_sig):
     m = re.match(r'.*-([a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$', url)
     assert m, '%r should follow URL format' % url
     test_id = m.group(1)
 
     def test_func(self):
-        basename = 'player-%s.%s' % (test_id, stype)
+        basename = 'player-%s.js' % test_id
         fn = os.path.join(self.TESTDATA_DIR, basename)
 
         if not os.path.exists(fn):
@@ -118,22 +109,16 @@ def make_tfunc(url, stype, sig_input, expected_sig):
 
         ydl = FakeYDL()
         ie = YoutubeIE(ydl)
-        if stype == 'js':
+        with io.open(fn, encoding='utf-8') as testf:
-            with io.open(fn, encoding='utf-8') as testf:
+            jscode = testf.read()
-                jscode = testf.read()
+        func = ie._parse_sig_js(jscode)
-            func = ie._parse_sig_js(jscode)
-        else:
-            assert stype == 'swf'
-            with open(fn, 'rb') as testf:
-                swfcode = testf.read()
-            func = ie._parse_sig_swf(swfcode)
         src_sig = (
             compat_str(string.printable[:sig_input])
             if isinstance(sig_input, int) else sig_input)
         got_sig = func(src_sig)
         self.assertEqual(got_sig, expected_sig)
 
-    test_func.__name__ = str('test_signature_' + stype + '_' + test_id)
+    test_func.__name__ = str('test_signature_js_' + test_id)
     setattr(TestSignature, test_func.__name__, test_func)
 
 

youtube_dlc/YoutubeDL.py

@@ -27,6 +27,7 @@ import traceback
 import random
 
 from string import ascii_letters
+from zipimport import zipimporter
 
 from .compat import (
     compat_basestring,
@@ -49,6 +50,7 @@ from .utils import (
     date_from_str,
     DateRange,
     DEFAULT_OUTTMPL,
+    OUTTMPL_TYPES,
     determine_ext,
     determine_protocol,
     DOT_DESKTOP_LINK_TEMPLATE,
@@ -61,6 +63,7 @@ from .utils import (
     ExistingVideoReached,
     expand_path,
     ExtractorError,
+    float_or_none,
     format_bytes,
     format_field,
     formatSeconds,
@@ -91,6 +94,7 @@ from .utils import (
     sanitized_Request,
     std_headers,
     str_or_none,
+    strftime_or_none,
     subtitles_filename,
     to_high_limit_path,
     UnavailableVideoError,
@@ -172,15 +176,21 @@ class YoutubeDL(object):
     forcejson:         Force printing info_dict as JSON.
     dump_single_json:  Force printing the info_dict of the whole playlist
                        (or video) as a single JSON line.
-    force_write_download_archive: Force writing download archive regardless of
+    force_write_download_archive: Force writing download archive regardless
-                       'skip_download' or 'simulate'.
+                       of 'skip_download' or 'simulate'.
     simulate:          Do not download the video files.
     format:            Video format code. see "FORMAT SELECTION" for more details.
-    format_sort:       How to sort the video formats. see "Sorting Formats" for more details.
+    allow_unplayable_formats:   Allow unplayable formats to be extracted and downloaded.
-    format_sort_force: Force the given format_sort. see "Sorting Formats" for more details.
+    format_sort:       How to sort the video formats. see "Sorting Formats"
-    allow_multiple_video_streams:   Allow multiple video streams to be merged into a single file
+                       for more details.
-    allow_multiple_audio_streams:   Allow multiple audio streams to be merged into a single file
+    format_sort_force: Force the given format_sort. see "Sorting Formats"
-    outtmpl:           Template for output names.
+                       for more details.
+    allow_multiple_video_streams:   Allow multiple video streams to be merged
+                       into a single file
+    allow_multiple_audio_streams:   Allow multiple audio streams to be merged
+                       into a single file
+    outtmpl:           Dictionary of templates for output names. Allowed keys
+                       are 'default' and the keys of OUTTMPL_TYPES (in utils.py)
     outtmpl_na_placeholder: Placeholder for unavailable meta fields.
     restrictfilenames: Do not allow "&" and spaces in file names
     trim_file_name:    Limit length of filename (extension excluded)
@@ -206,6 +216,8 @@ class YoutubeDL(object):
                        unless writeinfojson is also given
     writeannotations:  Write the video annotations to a .annotations.xml file
     writethumbnail:    Write the thumbnail image to a file
+    allow_playlist_files: Whether to write playlists' description, infojson etc
+                       also to disk when using the 'write*' options
     write_all_thumbnails:  Write all thumbnail formats to files
     writelink:         Write an internet shortcut file, depending on the
                        current platform (.url/.webloc/.desktop)
@@ -446,6 +458,9 @@ class YoutubeDL(object):
                 self.report_warning('--merge-output-format will be ignored since --remux-video or --recode-video is given')
             self.params['merge_output_format'] = self.params['final_ext']
 
+        if 'overwrites' in self.params and self.params['overwrites'] is None:
+            del self.params['overwrites']
+
         check_deprecated('autonumber_size', '--autonumber-size', 'output template with %(autonumber)0Nd, where N in the number of digits')
         check_deprecated('autonumber', '--auto-number', '-o "%(autonumber)s-%(title)s.%(ext)s"')
         check_deprecated('usetitle', '--title', '-o "%(title)s-%(id)s.%(ext)s"')
@@ -487,10 +502,7 @@ class YoutubeDL(object):
                 'Set the LC_ALL environment variable to fix this.')
             self.params['restrictfilenames'] = True
 
-        if isinstance(params.get('outtmpl'), bytes):
+        self.outtmpl_dict = self.parse_outtmpl()
-            self.report_warning(
-                'Parameter outtmpl is bytes, but should be a unicode string. '
-                'Put  from __future__ import unicode_literals  at the top of your code file or consider switching to Python 3.x.')
 
         self._setup_opener()
 
@@ -722,15 +734,33 @@ class YoutubeDL(object):
     def report_file_delete(self, file_name):
         """Report that existing file will be deleted."""
         try:
-            self.to_screen('Deleting already existent file %s' % file_name)
+            self.to_screen('Deleting existing file %s' % file_name)
         except UnicodeEncodeError:
-            self.to_screen('Deleting already existent file')
+            self.to_screen('Deleting existing file')
 
-    def prepare_filename(self, info_dict, warn=False):
+    def parse_outtmpl(self):
-        """Generate the output filename."""
+        outtmpl_dict = self.params.get('outtmpl', {})
+        if not isinstance(outtmpl_dict, dict):
+            outtmpl_dict = {'default': outtmpl_dict}
+        outtmpl_dict.update({
+            k: v for k, v in DEFAULT_OUTTMPL.items()
+            if not outtmpl_dict.get(k)})
+        for key, val in outtmpl_dict.items():
+            if isinstance(val, bytes):
+                self.report_warning(
+                    'Parameter outtmpl is bytes, but should be a unicode string. '
+                    'Put  from __future__ import unicode_literals  at the top of your code file or consider switching to Python 3.x.')
+        return outtmpl_dict
+
+    def _prepare_filename(self, info_dict, tmpl_type='default'):
         try:
             template_dict = dict(info_dict)
 
+            template_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
+                formatSeconds(info_dict['duration'], '-')
+                if info_dict.get('duration', None) is not None
+                else None)
+
             template_dict['epoch'] = int(time.time())
             autonumber_size = self.params.get('autonumber_size')
             if autonumber_size is None:
@@ -751,9 +781,11 @@ class YoutubeDL(object):
             template_dict = dict((k, v if isinstance(v, compat_numeric_types) else sanitize(k, v))
                                  for k, v in template_dict.items()
                                  if v is not None and not isinstance(v, (list, tuple, dict)))
-            template_dict = collections.defaultdict(lambda: self.params.get('outtmpl_na_placeholder', 'NA'), template_dict)
+            na = self.params.get('outtmpl_na_placeholder', 'NA')
+            template_dict = collections.defaultdict(lambda: na, template_dict)
 
-            outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
+            outtmpl = self.outtmpl_dict.get(tmpl_type, self.outtmpl_dict['default'])
+            force_ext = OUTTMPL_TYPES.get(tmpl_type)
 
             # For fields playlist_index and autonumber convert all occurrences
             # of %(field)s to %(field)0Nd for backward compatibility
@@ -769,27 +801,45 @@ class YoutubeDL(object):
                     r'%%(\1)0%dd' % field_size_compat_map[mobj.group('field')],
                     outtmpl)
 
+            # As of [1] format syntax is:
+            #  %[mapping_key][conversion_flags][minimum_width][.precision][length_modifier]type
+            # 1. https://docs.python.org/2/library/stdtypes.html#string-formatting
+            FORMAT_RE = r'''(?x)
+                (?<!%)
+                %
+                \({0}\)  # mapping key
+                (?:[#0\-+ ]+)?  # conversion flags (optional)
+                (?:\d+)?  # minimum field width (optional)
+                (?:\.\d+)?  # precision (optional)
+                [hlL]?  # length modifier (optional)
+                (?P<type>[diouxXeEfFgGcrs%])  # conversion type
+            '''
+
+            numeric_fields = list(self._NUMERIC_FIELDS)
+
+            # Format date
+            FORMAT_DATE_RE = FORMAT_RE.format(r'(?P<key>(?P<field>\w+)>(?P<format>.+?))')
+            for mobj in re.finditer(FORMAT_DATE_RE, outtmpl):
+                conv_type, field, frmt, key = mobj.group('type', 'field', 'format', 'key')
+                if key in template_dict:
+                    continue
+                value = strftime_or_none(template_dict.get(field), frmt, na)
+                if conv_type in 'crs':  # string
+                    value = sanitize(field, value)
+                else:  # number
+                    numeric_fields.append(key)
+                    value = float_or_none(value, default=None)
+                if value is not None:
+                    template_dict[key] = value
+
             # Missing numeric fields used together with integer presentation types
             # in format specification will break the argument substitution since
             # string NA placeholder is returned for missing fields. We will patch
             # output template for missing fields to meet string presentation type.
-            for numeric_field in self._NUMERIC_FIELDS:
+            for numeric_field in numeric_fields:
                 if numeric_field not in template_dict:
-                    # As of [1] format syntax is:
-                    #  %[mapping_key][conversion_flags][minimum_width][.precision][length_modifier]type
-                    # 1. https://docs.python.org/2/library/stdtypes.html#string-formatting
-                    FORMAT_RE = r'''(?x)
-                        (?<!%)
-                        %
-                        \({0}\)  # mapping key
-                        (?:[#0\-+ ]+)?  # conversion flags (optional)
-                        (?:\d+)?  # minimum field width (optional)
-                        (?:\.\d+)?  # precision (optional)
-                        [hlL]?  # length modifier (optional)
-                        [diouxXeEfFgGcrs%]  # conversion type
-                    '''
                     outtmpl = re.sub(
-                        FORMAT_RE.format(numeric_field),
+                        FORMAT_RE.format(re.escape(numeric_field)),
                         r'%({0})s'.format(numeric_field), outtmpl)
 
             # expand_path translates '%%' into '%' and '$$' into '$'
@@ -805,6 +855,9 @@ class YoutubeDL(object):
             # title "Hello $PATH", we don't want `$PATH` to be expanded.
             filename = expand_path(outtmpl).replace(sep, '') % template_dict
 
+            if force_ext is not None:
+                filename = replace_extension(filename, force_ext, template_dict.get('ext'))
+
             # https://github.com/blackjack4494/youtube-dlc/issues/85
             trim_file_name = self.params.get('trim_file_name', False)
             if trim_file_name:
@@ -822,25 +875,28 @@ class YoutubeDL(object):
                 filename = encodeFilename(filename, True).decode(preferredencoding())
             filename = sanitize_path(filename)
 
-            if warn and not self.__prepare_filename_warned:
-                if not self.params.get('paths'):
-                    pass
-                elif filename == '-':
-                    self.report_warning('--paths is ignored when an outputting to stdout')
-                elif os.path.isabs(filename):
-                    self.report_warning('--paths is ignored since an absolute path is given in output template')
-                self.__prepare_filename_warned = True
-
             return filename
         except ValueError as err:
             self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
             return None
 
-    def prepare_filepath(self, filename, dir_type=''):
+    def prepare_filename(self, info_dict, dir_type='', warn=False):
-        if filename == '-':
+        """Generate the output filename."""
-            return filename
         paths = self.params.get('paths', {})
         assert isinstance(paths, dict)
+        filename = self._prepare_filename(info_dict, dir_type or 'default')
+
+        if warn and not self.__prepare_filename_warned:
+            if not paths:
+                pass
+            elif filename == '-':
+                self.report_warning('--paths is ignored when an outputting to stdout')
+            elif os.path.isabs(filename):
+                self.report_warning('--paths is ignored since an absolute path is given in output template')
+            self.__prepare_filename_warned = True
+        if filename == '-' or not filename:
+            return filename
+
         homepath = expand_path(paths.get('home', '').strip())
         assert isinstance(homepath, compat_str)
         subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
@@ -992,10 +1048,6 @@ class YoutubeDL(object):
         self.add_extra_info(ie_result, {
             'extractor': ie.IE_NAME,
             'webpage_url': url,
-            'duration_string': (
-                formatSeconds(ie_result['duration'], '-')
-                if ie_result.get('duration', None) is not None
-                else None),
             'webpage_url_basename': url_basename(url),
             'extractor_key': ie.ie_key(),
         })
@@ -1015,10 +1067,7 @@ class YoutubeDL(object):
             extract_flat = self.params.get('extract_flat', False)
             if ((extract_flat == 'in_playlist' and 'playlist' in extra_info)
                     or extract_flat is True):
-                self.__forced_printings(
+                self.__forced_printings(ie_result, self.prepare_filename(ie_result), incomplete=True)
-                    ie_result,
-                    self.prepare_filepath(self.prepare_filename(ie_result)),
-                    incomplete=True)
                 return ie_result
 
         if result_type == 'video':
@@ -1108,54 +1157,53 @@ class YoutubeDL(object):
         # We process each entry in the playlist
         playlist = ie_result.get('title') or ie_result.get('id')
         self.to_screen('[download] Downloading playlist: %s' % playlist)
-        ie_copy = {
-            'playlist': playlist,
-            'playlist_id': ie_result.get('id'),
-            'playlist_title': ie_result.get('title'),
-            'playlist_uploader': ie_result.get('uploader'),
-            'playlist_uploader_id': ie_result.get('uploader_id'),
-            'playlist_index': 0
-        }
-        ie_copy.update(dict(ie_result))
 
-        def ensure_dir_exists(path):
+        if self.params.get('allow_playlist_files', True):
-            return make_dir(path, self.report_error)
+            ie_copy = {
+                'playlist': playlist,
+                'playlist_id': ie_result.get('id'),
+                'playlist_title': ie_result.get('title'),
+                'playlist_uploader': ie_result.get('uploader'),
+                'playlist_uploader_id': ie_result.get('uploader_id'),
+                'playlist_index': 0
+            }
+            ie_copy.update(dict(ie_result))
 
-        if self.params.get('writeinfojson', False):
+            def ensure_dir_exists(path):
-            infofn = replace_extension(
+                return make_dir(path, self.report_error)
-                self.prepare_filepath(self.prepare_filename(ie_copy), 'infojson'),
-                'info.json', ie_result.get('ext'))
-            if not ensure_dir_exists(encodeFilename(infofn)):
-                return
-            if self.params.get('overwrites', True) and os.path.exists(encodeFilename(infofn)):
-                self.to_screen('[info] Playlist metadata is already present')
-            else:
-                self.to_screen('[info] Writing playlist metadata as JSON to: ' + infofn)
-                playlist_info = dict(ie_result)
-                playlist_info.pop('entries')
-                try:
-                    write_json_file(self.filter_requested_info(playlist_info), infofn)
-                except (OSError, IOError):
-                    self.report_error('Cannot write playlist metadata to JSON file ' + infofn)
 
-        if self.params.get('writedescription', False):
+            if self.params.get('writeinfojson', False):
-            descfn = replace_extension(
+                infofn = self.prepare_filename(ie_copy, 'pl_infojson')
-                self.prepare_filepath(self.prepare_filename(ie_copy), 'description'),
+                if not ensure_dir_exists(encodeFilename(infofn)):
-                'description', ie_result.get('ext'))
-            if not ensure_dir_exists(encodeFilename(descfn)):
-                return
-            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(descfn)):
-                self.to_screen('[info] Playlist description is already present')
-            elif ie_result.get('description') is None:
-                self.report_warning('There\'s no playlist description to write.')
-            else:
-                try:
-                    self.to_screen('[info] Writing playlist description to: ' + descfn)
-                    with io.open(encodeFilename(descfn), 'w', encoding='utf-8') as descfile:
-                        descfile.write(ie_result['description'])
-                except (OSError, IOError):
-                    self.report_error('Cannot write playlist description file ' + descfn)
                     return
+                if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(infofn)):
+                    self.to_screen('[info] Playlist metadata is already present')
+                else:
+                    playlist_info = dict(ie_result)
+                    # playlist_info['entries'] = list(playlist_info['entries'])  # Entries is a generator which shouldnot be resolved here
+                    del playlist_info['entries']
+                    self.to_screen('[info] Writing playlist metadata as JSON to: ' + infofn)
+                    try:
+                        write_json_file(self.filter_requested_info(playlist_info), infofn)
+                    except (OSError, IOError):
+                        self.report_error('Cannot write playlist metadata to JSON file ' + infofn)
+
+            if self.params.get('writedescription', False):
+                descfn = self.prepare_filename(ie_copy, 'pl_description')
+                if not ensure_dir_exists(encodeFilename(descfn)):
+                    return
+                if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(descfn)):
+                    self.to_screen('[info] Playlist description is already present')
+                elif ie_result.get('description') is None:
+                    self.report_warning('There\'s no playlist description to write.')
+                else:
+                    try:
+                        self.to_screen('[info] Writing playlist description to: ' + descfn)
+                        with io.open(encodeFilename(descfn), 'w', encoding='utf-8') as descfile:
+                            descfile.write(ie_result['description'])
+                    except (OSError, IOError):
+                        self.report_error('Cannot write playlist description file ' + descfn)
+                        return
 
         playlist_results = []
 
@@ -1341,7 +1389,7 @@ class YoutubeDL(object):
             and (
                 not can_merge()
                 or info_dict.get('is_live', False)
-                or self.params.get('outtmpl', DEFAULT_OUTTMPL) == '-'))
+                or self.outtmpl_dict['default'] == '-'))
 
         return (
             'best/bestvideo+bestaudio'
@@ -1852,7 +1900,7 @@ class YoutubeDL(object):
         if req_format is None:
             req_format = self._default_format_spec(info_dict, download=download)
             if self.params.get('verbose'):
-                self._write_string('[debug] Default format spec: %s\n' % req_format)
+                self.to_screen('[debug] Default format spec: %s' % req_format)
 
         format_selector = self.build_format_selector(req_format)
 
@@ -2003,10 +2051,10 @@ class YoutubeDL(object):
 
         info_dict = self.pre_process(info_dict)
 
-        filename = self.prepare_filename(info_dict, warn=True)
+        info_dict['_filename'] = full_filename = self.prepare_filename(info_dict, warn=True)
-        info_dict['_filename'] = full_filename = self.prepare_filepath(filename)
+        temp_filename = self.prepare_filename(info_dict, 'temp')
-        temp_filename = self.prepare_filepath(filename, 'temp')
         files_to_move = {}
+        skip_dl = self.params.get('skip_download', False)
 
         # Forced printings
         self.__forced_printings(info_dict, full_filename, incomplete=False)
@@ -2018,7 +2066,7 @@ class YoutubeDL(object):
             # Do nothing else if in simulate mode
             return
 
-        if filename is None:
+        if full_filename is None:
             return
 
         def ensure_dir_exists(path):
@@ -2030,9 +2078,7 @@ class YoutubeDL(object):
             return
 
         if self.params.get('writedescription', False):
-            descfn = replace_extension(
+            descfn = self.prepare_filename(info_dict, 'description')
-                self.prepare_filepath(filename, 'description'),
-                'description', info_dict.get('ext'))
             if not ensure_dir_exists(encodeFilename(descfn)):
                 return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(descfn)):
@@ -2049,9 +2095,7 @@ class YoutubeDL(object):
                     return
 
         if self.params.get('writeannotations', False):
-            annofn = replace_extension(
+            annofn = self.prepare_filename(info_dict, 'annotation')
-                self.prepare_filepath(filename, 'annotation'),
-                'annotations.xml', info_dict.get('ext'))
             if not ensure_dir_exists(encodeFilename(annofn)):
                 return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(annofn)):
@@ -2087,10 +2131,11 @@ class YoutubeDL(object):
             # ie = self.get_info_extractor(info_dict['extractor_key'])
             for sub_lang, sub_info in subtitles.items():
                 sub_format = sub_info['ext']
-                sub_filename = subtitles_filename(temp_filename, sub_lang, sub_format, info_dict.get('ext'))
+                sub_fn = self.prepare_filename(info_dict, 'subtitle')
-                sub_filename_final = subtitles_filename(
+                sub_filename = subtitles_filename(
-                    self.prepare_filepath(filename, 'subtitle'),
+                    temp_filename if not skip_dl else sub_fn,
                     sub_lang, sub_format, info_dict.get('ext'))
+                sub_filename_final = subtitles_filename(sub_fn, sub_lang, sub_format, info_dict.get('ext'))
                 if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(sub_filename)):
                     self.to_screen('[info] Video subtitle %s.%s is already present' % (sub_lang, sub_format))
                     files_to_move[sub_filename] = sub_filename_final
@@ -2124,10 +2169,10 @@ class YoutubeDL(object):
                                                 (sub_lang, error_to_compat_str(err)))
                             continue
 
-        if self.params.get('skip_download', False):
+        if skip_dl:
             if self.params.get('convertsubtitles', False):
                 # subconv = FFmpegSubtitlesConvertorPP(self, format=self.params.get('convertsubtitles'))
-                filename_real_ext = os.path.splitext(filename)[1][1:]
+                filename_real_ext = os.path.splitext(full_filename)[1][1:]
                 filename_wo_ext = (
                     os.path.splitext(full_filename)[0]
                     if filename_real_ext == info_dict['ext']
@@ -2142,14 +2187,12 @@ class YoutubeDL(object):
                 else:
                     try:
                         self.post_process(full_filename, info_dict, files_to_move)
-                    except (PostProcessingError) as err:
+                    except PostProcessingError as err:
-                        self.report_error('postprocessing: %s' % str(err))
+                        self.report_error('Postprocessing: %s' % str(err))
                         return
 
         if self.params.get('writeinfojson', False):
-            infofn = replace_extension(
+            infofn = self.prepare_filename(info_dict, 'infojson')
-                self.prepare_filepath(filename, 'infojson'),
-                'info.json', info_dict.get('ext'))
             if not ensure_dir_exists(encodeFilename(infofn)):
                 return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(infofn)):
@@ -2161,11 +2204,14 @@ class YoutubeDL(object):
                 except (OSError, IOError):
                     self.report_error('Cannot write video metadata to JSON file ' + infofn)
                     return
-            info_dict['__infojson_filepath'] = infofn
+            info_dict['__infojson_filename'] = infofn
 
-        thumbdir = os.path.dirname(self.prepare_filepath(filename, 'thumbnail'))
+        thumbfn = self.prepare_filename(info_dict, 'thumbnail')
-        for thumbfn in self._write_thumbnails(info_dict, temp_filename):
+        thumb_fn_temp = temp_filename if not skip_dl else thumbfn
-            files_to_move[thumbfn] = os.path.join(thumbdir, os.path.basename(thumbfn))
+        for thumb_ext in self._write_thumbnails(info_dict, thumb_fn_temp):
+            thumb_filename_temp = replace_extension(thumb_fn_temp, thumb_ext, info_dict.get('ext'))
+            thumb_filename = replace_extension(thumbfn, thumb_ext, info_dict.get('ext'))
+            files_to_move[thumb_filename_temp] = info_dict['__thumbnail_filename'] = thumb_filename
 
         # Write internet shortcut files
         url_link = webloc_link = desktop_link = False
@@ -2218,7 +2264,7 @@ class YoutubeDL(object):
 
         # Download
         must_record_download_archive = False
-        if not self.params.get('skip_download', False):
+        if not skip_dl:
             try:
 
                 def existing_file(*filepaths):
@@ -2247,13 +2293,15 @@ class YoutubeDL(object):
                 if info_dict.get('requested_formats') is not None:
                     downloaded = []
                     merger = FFmpegMergerPP(self)
-                    if not merger.available:
+                    if self.params.get('allow_unplayable_formats'):
-                        postprocessors = []
+                        self.report_warning(
-                        self.report_warning('You have requested multiple '
+                            'You have requested merging of multiple formats '
-                                            'formats but ffmpeg is not installed.'
+                            'while also allowing unplayable formats to be downloaded. '
-                                            ' The formats won\'t be merged.')
+                            'The formats won\'t be merged to prevent data corruption.')
-                    else:
+                    elif not merger.available:
-                        postprocessors = [merger]
+                        self.report_warning(
+                            'You have requested merging of multiple formats but ffmpeg is not installed. '
+                            'The formats won\'t be merged.')
 
                     def compatible_formats(formats):
                         # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2293,22 +2341,25 @@ class YoutubeDL(object):
                     full_filename = correct_ext(full_filename)
                     temp_filename = correct_ext(temp_filename)
                     dl_filename = existing_file(full_filename, temp_filename)
+                    info_dict['__real_download'] = False
                     if dl_filename is None:
                         for f in requested_formats:
                             new_info = dict(info_dict)
                             new_info.update(f)
                             fname = prepend_extension(
-                                self.prepare_filepath(self.prepare_filename(new_info), 'temp'),
+                                self.prepare_filename(new_info, 'temp'),
                                 'f%s' % f['format_id'], new_info['ext'])
                             if not ensure_dir_exists(fname):
                                 return
                             downloaded.append(fname)
                             partial_success, real_download = dl(fname, new_info)
+                            info_dict['__real_download'] = info_dict['__real_download'] or real_download
                             success = success and partial_success
-                        info_dict['__postprocessors'] = postprocessors
+                        if merger.available and not self.params.get('allow_unplayable_formats'):
-                        info_dict['__files_to_merge'] = downloaded
+                            info_dict['__postprocessors'].append(merger)
-                        # Even if there were no downloads, it is being merged only now
+                            info_dict['__files_to_merge'] = downloaded
-                        info_dict['__real_download'] = True
+                            # Even if there were no downloads, it is being merged only now
+                            info_dict['__real_download'] = True
                 else:
                     # Just a single file
                     dl_filename = existing_file(full_filename, temp_filename)
@@ -2328,7 +2379,7 @@ class YoutubeDL(object):
                 self.report_error('content too short (expected %s bytes and served %s)' % (err.expected, err.downloaded))
                 return
 
-            if success and filename != '-':
+            if success and full_filename != '-':
                 # Fixup content
                 fixup_policy = self.params.get('fixup')
                 if fixup_policy is None:
@@ -2391,8 +2442,8 @@ class YoutubeDL(object):
 
                 try:
                     self.post_process(dl_filename, info_dict, files_to_move)
-                except (PostProcessingError) as err:
+                except PostProcessingError as err:
-                    self.report_error('postprocessing: %s' % str(err))
+                    self.report_error('Postprocessing: %s' % str(err))
                     return
                 try:
                     for ph in self._post_hooks:
@@ -2410,7 +2461,7 @@ class YoutubeDL(object):
 
     def download(self, url_list):
         """Download a given list of URLs."""
-        outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
+        outtmpl = self.outtmpl_dict['default']
         if (len(url_list) > 1
                 and outtmpl != '-'
                 and '%' not in outtmpl
@@ -2458,16 +2509,14 @@ class YoutubeDL(object):
 
     @staticmethod
     def filter_requested_info(info_dict):
+        fields_to_remove = ('requested_formats', 'requested_subtitles')
         return dict(
             (k, v) for k, v in info_dict.items()
-            if k not in ['requested_formats', 'requested_subtitles'])
+            if (k[0] != '_' or k == '_type') and k not in fields_to_remove)
 
     def run_pp(self, pp, infodict, files_to_move={}):
         files_to_delete = []
-        try:
+        files_to_delete, infodict = pp.run(infodict)
-            files_to_delete, infodict = pp.run(infodict)
-        except PostProcessingError as e:
-            self.report_error(e.msg)
         if not files_to_delete:
             return files_to_move, infodict
 
@@ -2495,12 +2544,13 @@ class YoutubeDL(object):
         """Run all the postprocessors on the given file."""
         info = dict(ie_info)
         info['filepath'] = filename
+        info['__files_to_move'] = {}
 
         for pp in ie_info.get('__postprocessors', []) + self._pps['normal']:
             files_to_move, info = self.run_pp(pp, info, files_to_move)
-        info = self.run_pp(MoveFilesAfterDownloadPP(self, files_to_move), info, files_to_move)[1]
+        info = self.run_pp(MoveFilesAfterDownloadPP(self, files_to_move), info)[1]
         for pp in self._pps['aftermove']:
-            files_to_move, info = self.run_pp(pp, info, {})
+            info = self.run_pp(pp, info, {})[1]
 
     def _make_archive_id(self, info_dict):
         video_id = info_dict.get('id')
@@ -2640,7 +2690,7 @@ class YoutubeDL(object):
                     '|',
                     format_field(f, 'filesize', ' %s', func=format_bytes) + format_field(f, 'filesize_approx', '~%s', func=format_bytes),
                     format_field(f, 'tbr', '%4dk'),
-                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n"),
+                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n").replace('niconico_', ''),
                     '|',
                     format_field(f, 'vcodec', default='unknown').replace('none', ''),
                     format_field(f, 'vbr', '%4dk'),
@@ -2721,7 +2771,12 @@ class YoutubeDL(object):
                 self.get_encoding()))
         write_string(encoding_str, encoding=None)
 
-        self._write_string('[debug] yt-dlp version %s\n' % __version__)
+        source = (
+            '(exe)' if hasattr(sys, 'frozen')
+            else '(zip)' if isinstance(globals().get('__loader__'), zipimporter)
+            else '(source)' if os.path.basename(sys.argv[0]) == '__main__.py'
+            else '')
+        self._write_string('[debug] yt-dlp version %s %s\n' % (__version__, source))
         if _LAZY_LOADER:
             self._write_string('[debug] Lazy loading extractors enabled\n')
         if _PLUGIN_CLASSES:
@@ -2748,8 +2803,10 @@ class YoutubeDL(object):
                 return impl_name + ' version %d.%d.%d' % sys.pypy_version_info[:3]
             return impl_name
 
-        self._write_string('[debug] Python version %s (%s) - %s\n' % (
+        self._write_string('[debug] Python version %s (%s %s) - %s\n' % (
-            platform.python_version(), python_implementation(),
+            platform.python_version(),
+            python_implementation(),
+            platform.architecture()[0],
             platform_name()))
 
         exe_versions = FFmpegPostProcessor.get_versions(self)
@@ -2851,25 +2908,22 @@ class YoutubeDL(object):
             encoding = preferredencoding()
         return encoding
 
-    def _write_thumbnails(self, info_dict, filename):
+    def _write_thumbnails(self, info_dict, filename):  # return the extensions
-        if self.params.get('writethumbnail', False):
+        write_all = self.params.get('write_all_thumbnails', False)
-            thumbnails = info_dict.get('thumbnails')
+        thumbnails = []
-            if thumbnails:
+        if write_all or self.params.get('writethumbnail', False):
-                thumbnails = [thumbnails[-1]]
-        elif self.params.get('write_all_thumbnails', False):
             thumbnails = info_dict.get('thumbnails') or []
-        else:
+        multiple = write_all and len(thumbnails) > 1
-            thumbnails = []
 
         ret = []
-        for t in thumbnails:
+        for t in thumbnails[::1 if write_all else -1]:
             thumb_ext = determine_ext(t['url'], 'jpg')
-            suffix = '_%s' % t['id'] if len(thumbnails) > 1 else ''
+            suffix = '%s.' % t['id'] if multiple else ''
-            thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
+            thumb_display_id = '%s ' % t['id'] if multiple else ''
-            t['filename'] = thumb_filename = replace_extension(filename + suffix, thumb_ext, info_dict.get('ext'))
+            t['filename'] = thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
 
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
-                ret.append(thumb_filename)
+                ret.append(suffix + thumb_ext)
                 self.to_screen('[%s] %s: Thumbnail %sis already present' %
                                (info_dict['extractor'], info_dict['id'], thumb_display_id))
             else:
@@ -2879,10 +2933,12 @@ class YoutubeDL(object):
                     uf = self.urlopen(t['url'])
                     with open(encodeFilename(thumb_filename), 'wb') as thumbf:
                         shutil.copyfileobj(uf, thumbf)
-                    ret.append(thumb_filename)
+                    ret.append(suffix + thumb_ext)
                     self.to_screen('[%s] %s: Writing thumbnail %sto: %s' %
                                    (info_dict['extractor'], info_dict['id'], thumb_display_id, thumb_filename))
                 except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                     self.report_warning('Unable to download thumbnail "%s": %s' %
                                         (t['url'], error_to_compat_str(err)))
+            if ret and not write_all:
+                break
         return ret

youtube_dlc/__init__.py

@@ -15,7 +15,6 @@ import sys
 
 from .options import (
     parseOpts,
-    _remux_formats,
 )
 from .compat import (
     compat_getpass,
@@ -24,7 +23,6 @@ from .compat import (
 from .utils import (
     DateRange,
     decodeOption,
-    DEFAULT_OUTTMPL,
     DownloadError,
     ExistingVideoReached,
     expand_path,
@@ -33,11 +31,12 @@ from .utils import (
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
+    REMUX_EXTENSIONS,
+    render_table,
     SameFileError,
     setproctitle,
     std_headers,
     write_string,
-    render_table,
 )
 from .update import update_self
 from .downloader import (
@@ -211,13 +210,12 @@ def _real_main(argv=None):
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
     if opts.recodevideo is not None:
-        if opts.recodevideo not in _remux_formats:
+        if opts.recodevideo not in REMUX_EXTENSIONS:
             parser.error('invalid video recode format specified')
-    if opts.remuxvideo and opts.recodevideo:
-        opts.remuxvideo = None
-        write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
     if opts.remuxvideo is not None:
-        if opts.remuxvideo not in _remux_formats:
+        opts.remuxvideo = opts.remuxvideo.replace(' ', '')
+        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
+        if not re.match(remux_regex, opts.remuxvideo):
             parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
         if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
@@ -232,23 +230,21 @@ def _real_main(argv=None):
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
 
-    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
+    outtmpl = opts.outtmpl
-    # this was the old behaviour if only --all-sub was given.
+    if not outtmpl:
-    if opts.allsubtitles and not opts.writeautomaticsub:
+        outtmpl = {'default': (
-        opts.writesubtitles = True
+            '%(title)s-%(id)s-%(format)s.%(ext)s' if opts.format == '-1' and opts.usetitle
-
+            else '%(id)s-%(format)s.%(ext)s' if opts.format == '-1'
-    outtmpl = ((opts.outtmpl is not None and opts.outtmpl)
+            else '%(autonumber)s-%(title)s-%(id)s.%(ext)s' if opts.usetitle and opts.autonumber
-               or (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s')
+            else '%(title)s-%(id)s.%(ext)s' if opts.usetitle
-               or (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s')
+            else '%(id)s.%(ext)s' if opts.useid
-               or (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s')
+            else '%(autonumber)s-%(id)s.%(ext)s' if opts.autonumber
-               or (opts.usetitle and '%(title)s-%(id)s.%(ext)s')
+            else None)}
-               or (opts.useid and '%(id)s.%(ext)s')
+    outtmpl_default = outtmpl.get('default')
-               or (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s')
+    if outtmpl_default is not None and not os.path.splitext(outtmpl_default)[1] and opts.extractaudio:
-               or DEFAULT_OUTTMPL)
-    if not os.path.splitext(outtmpl)[1] and opts.extractaudio:
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
-                     ' template'.format(outtmpl))
+                     ' template'.format(outtmpl_default))
 
     for f in opts.format_sort:
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
@@ -266,6 +262,40 @@ def _real_main(argv=None):
     any_printing = opts.print_json
     download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
 
+    def report_conflict(arg1, arg2):
+        write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
+    if opts.remuxvideo and opts.recodevideo:
+        report_conflict('--recode-video', '--remux-video')
+        opts.remuxvideo = False
+    if opts.allow_unplayable_formats:
+        if opts.extractaudio:
+            report_conflict('--allow-unplayable-formats', '--extract-audio')
+            opts.extractaudio = False
+        if opts.remuxvideo:
+            report_conflict('--allow-unplayable-formats', '--remux-video')
+            opts.remuxvideo = False
+        if opts.recodevideo:
+            report_conflict('--allow-unplayable-formats', '--recode-video')
+            opts.recodevideo = False
+        if opts.addmetadata:
+            report_conflict('--allow-unplayable-formats', '--add-metadata')
+            opts.addmetadata = False
+        if opts.embedsubtitles:
+            report_conflict('--allow-unplayable-formats', '--embed-subs')
+            opts.embedsubtitles = False
+        if opts.embedthumbnail:
+            report_conflict('--allow-unplayable-formats', '--embed-thumbnail')
+            opts.embedthumbnail = False
+        if opts.xattrs:
+            report_conflict('--allow-unplayable-formats', '--xattrs')
+            opts.xattrs = False
+        if opts.fixup and opts.fixup.lower() not in ('never', 'ignore'):
+            report_conflict('--allow-unplayable-formats', '--fixup')
+        opts.fixup = 'never'
+        if opts.sponskrub:
+            report_conflict('--allow-unplayable-formats', '--sponskrub')
+        opts.sponskrub = False
+
     # PostProcessors
     postprocessors = []
     if opts.metafromfield:
@@ -307,9 +337,17 @@ def _real_main(argv=None):
             'format': opts.convertsubtitles,
         })
     if opts.embedsubtitles:
+        already_have_subtitle = opts.writesubtitles
         postprocessors.append({
             'key': 'FFmpegEmbedSubtitle',
+            'already_have_subtitle': already_have_subtitle
         })
+        if not already_have_subtitle:
+            opts.writesubtitles = True
+    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
+    # this was the old behaviour if only --all-sub was given.
+    if opts.allsubtitles and not opts.writeautomaticsub:
+        opts.writesubtitles = True
     if opts.embedthumbnail:
         already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
         postprocessors.append({
@@ -341,16 +379,23 @@ def _real_main(argv=None):
             'when': 'aftermove'
         })
 
-    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
+    def report_args_compat(arg, name):
+        write_string(
+            'WARNING: %s given without specifying name. The arguments will be given to all %s\n' % (arg, name),
+            out=sys.stderr)
     if 'default' in opts.external_downloader_args:
-        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
+        report_args_compat('--external-downloader-args', 'external downloaders')
 
     if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
-        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
+        report_args_compat('--post-processor-args', 'post-processors')
         opts.postprocessor_args.setdefault('sponskrub', [])
         opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
 
-    audio_ext = opts.audioformat if (opts.extractaudio and opts.audioformat != 'best') else None
+    final_ext = (
+        opts.recodevideo
+        or (opts.remuxvideo in REMUX_EXTENSIONS) and opts.remuxvideo
+        or (opts.extractaudio and opts.audioformat != 'best') and opts.audioformat
+        or None)
 
     match_filter = (
         None if opts.match_filter is None
@@ -382,6 +427,7 @@ def _real_main(argv=None):
         'simulate': opts.simulate or any_getting,
         'skip_download': opts.skip_download,
         'format': opts.format,
+        'allow_unplayable_formats': opts.allow_unplayable_formats,
         'format_sort': opts.format_sort,
         'format_sort_force': opts.format_sort_force,
         'allow_multiple_video_streams': opts.allow_multiple_video_streams,
@@ -413,13 +459,14 @@ def _real_main(argv=None):
         'playlistreverse': opts.playlist_reverse,
         'playlistrandom': opts.playlist_random,
         'noplaylist': opts.noplaylist,
-        'logtostderr': opts.outtmpl == '-',
+        'logtostderr': outtmpl_default == '-',
         'consoletitle': opts.consoletitle,
         'nopart': opts.nopart,
         'updatetime': opts.updatetime,
         'writedescription': opts.writedescription,
         'writeannotations': opts.writeannotations,
         'writeinfojson': opts.writeinfojson or opts.getcomments,
+        'allow_playlist_files': opts.allow_playlist_files,
         'getcomments': opts.getcomments,
         'writethumbnail': opts.writethumbnail,
         'write_all_thumbnails': opts.write_all_thumbnails,
@@ -471,7 +518,7 @@ def _real_main(argv=None):
         'extract_flat': opts.extract_flat,
         'mark_watched': opts.mark_watched,
         'merge_output_format': opts.merge_output_format,
-        'final_ext': opts.recodevideo or opts.remuxvideo or audio_ext,
+        'final_ext': final_ext,
         'postprocessors': postprocessors,
         'fixup': opts.fixup,
         'source_address': opts.source_address,
@@ -502,16 +549,22 @@ def _real_main(argv=None):
     }
 
     with YoutubeDL(ydl_opts) as ydl:
-        # Update version
+        actual_use = len(all_urls) or opts.load_info_filename
-        if opts.update_self:
-            update_self(ydl.to_screen, opts.verbose, ydl._opener)
 
         # Remove cache dir
         if opts.rm_cachedir:
             ydl.cache.remove()
 
+        # Update version
+        if opts.update_self:
+            # If updater returns True, exit. Required for windows
+            if update_self(ydl.to_screen, opts.verbose, ydl._opener):
+                if actual_use:
+                    parser.error('The program must exit for the update to complete')
+                sys.exit()
+
         # Maybe do nothing
-        if (len(all_urls) < 1) and (opts.load_info_filename is None):
+        if not actual_use:
             if opts.update_self or opts.rm_cachedir:
                 sys.exit()
 

youtube_dlc/downloader/__init__.py

@@ -1,23 +1,33 @@
 from __future__ import unicode_literals
 
+from ..utils import (
+    determine_protocol,
+)
+
+
+def _get_real_downloader(info_dict, protocol=None, *args, **kwargs):
+    info_copy = info_dict.copy()
+    if protocol:
+        info_copy['protocol'] = protocol
+    return get_suitable_downloader(info_copy, *args, **kwargs)
+
+
+# Some of these require _get_real_downloader
 from .common import FileDownloader
+from .dash import DashSegmentsFD
 from .f4m import F4mFD
 from .hls import HlsFD
 from .http import HttpFD
 from .rtmp import RtmpFD
-from .dash import DashSegmentsFD
 from .rtsp import RtspFD
 from .ism import IsmFD
+from .niconico import NiconicoDmcFD
 from .youtube_live_chat import YoutubeLiveChatReplayFD
 from .external import (
     get_external_downloader,
     FFmpegFD,
 )
 
-from ..utils import (
-    determine_protocol,
-)
-
 PROTOCOL_MAP = {
     'rtmp': RtmpFD,
     'm3u8_native': HlsFD,
@@ -27,11 +37,12 @@ PROTOCOL_MAP = {
     'f4m': F4mFD,
     'http_dash_segments': DashSegmentsFD,
     'ism': IsmFD,
+    'niconico_dmc': NiconicoDmcFD,
     'youtube_live_chat_replay': YoutubeLiveChatReplayFD,
 }
 
 
-def get_suitable_downloader(info_dict, params={}):
+def get_suitable_downloader(info_dict, params={}, default=HttpFD):
     """Get the downloader class that can handle the info dict."""
     protocol = determine_protocol(info_dict)
     info_dict['protocol'] = protocol
@@ -45,16 +56,17 @@ def get_suitable_downloader(info_dict, params={}):
         if ed.can_download(info_dict):
             return ed
 
-    if protocol.startswith('m3u8') and info_dict.get('is_live'):
+    if protocol.startswith('m3u8'):
-        return FFmpegFD
+        if info_dict.get('is_live'):
+            return FFmpegFD
+        elif _get_real_downloader(info_dict, 'frag_urls', params, None):
+            return HlsFD
+        elif params.get('hls_prefer_native') is True:
+            return HlsFD
+        elif params.get('hls_prefer_native') is False:
+            return FFmpegFD
 
-    if protocol == 'm3u8' and params.get('hls_prefer_native') is True:
+    return PROTOCOL_MAP.get(protocol, default)
-        return HlsFD
-
-    if protocol == 'm3u8_native' and params.get('hls_prefer_native') is False:
-        return FFmpegFD
-
-    return PROTOCOL_MAP.get(protocol, HttpFD)
 
 
 __all__ = [

youtube_dlc/downloader/common.py

@@ -332,7 +332,7 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            not self.params.get('overwrites', True)
+            not self.params.get('overwrites', subtitle)
             and os.path.exists(encodeFilename(filename))
         )
 

youtube_dlc/downloader/dash.py

@@ -1,6 +1,8 @@
 from __future__ import unicode_literals
 
+from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
+
 from ..compat import compat_urllib_error
 from ..utils import (
     DownloadError,
@@ -20,31 +22,42 @@ class DashSegmentsFD(FragmentFD):
         fragments = info_dict['fragments'][:1] if self.params.get(
             'test', False) else info_dict['fragments']
 
+        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+
         ctx = {
             'filename': filename,
             'total_frags': len(fragments),
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        if real_downloader:
+            self._prepare_external_frag_download(ctx)
+        else:
+            self._prepare_and_start_frag_download(ctx)
 
         fragment_retries = self.params.get('fragment_retries', 0)
         skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
 
+        fragment_urls = []
         frag_index = 0
         for i, fragment in enumerate(fragments):
             frag_index += 1
             if frag_index <= ctx['fragment_index']:
                 continue
+            fragment_url = fragment.get('url')
+            if not fragment_url:
+                assert fragment_base_url
+                fragment_url = urljoin(fragment_base_url, fragment['path'])
+
+            if real_downloader:
+                fragment_urls.append(fragment_url)
+                continue
+
             # In DASH, the first segment contains necessary headers to
             # generate a valid MP4 file, so always abort for the first segment
             fatal = i == 0 or not skip_unavailable_fragments
             count = 0
             while count <= fragment_retries:
                 try:
-                    fragment_url = fragment.get('url')
-                    if not fragment_url:
-                        assert fragment_base_url
-                        fragment_url = urljoin(fragment_base_url, fragment['path'])
                     success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
                     if not success:
                         return False
@@ -75,6 +88,16 @@ class DashSegmentsFD(FragmentFD):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False
 
-        self._finish_frag_download(ctx)
+        if real_downloader:
-
+            info_copy = info_dict.copy()
+            info_copy['url_list'] = fragment_urls
+            fd = real_downloader(self.ydl, self.params)
+            # TODO: Make progress updates work without hooking twice
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
+            success = fd.real_download(filename, info_copy)
+            if not success:
+                return False
+        else:
+            self._finish_frag_download(ctx)
         return True

youtube_dlc/downloader/external.py

@@ -6,6 +6,12 @@ import subprocess
 import sys
 import time
 
+try:
+    from Crypto.Cipher import AES
+    can_decrypt_frag = True
+except ImportError:
+    can_decrypt_frag = False
+
 from .common import FileDownloader
 from ..compat import (
     compat_setenv,
@@ -18,15 +24,20 @@ from ..utils import (
     cli_bool_option,
     cli_configuration_args,
     encodeFilename,
+    error_to_compat_str,
     encodeArgument,
     handle_youtubedl_headers,
     check_executable,
     is_outdated_version,
     process_communicate_or_kill,
+    sanitized_Request,
+    sanitize_open,
 )
 
 
 class ExternalFD(FileDownloader):
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps')
+
     def real_download(self, filename, info_dict):
         self.report_destination(filename)
         tmpfilename = self.temp_name(filename)
@@ -79,7 +90,7 @@ class ExternalFD(FileDownloader):
 
     @classmethod
     def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps')
+        return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
 
     @classmethod
     def can_download(cls, info_dict):
@@ -109,8 +120,53 @@ class ExternalFD(FileDownloader):
         _, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
             self.to_stderr(stderr.decode('utf-8', 'replace'))
+
+        if 'url_list' in info_dict:
+            file_list = []
+            for [i, url] in enumerate(info_dict['url_list']):
+                tmpsegmentname = '%s_%s.frag' % (tmpfilename, i)
+                file_list.append(tmpsegmentname)
+            key_list = info_dict.get('key_list')
+            decrypt_info = None
+            dest, _ = sanitize_open(tmpfilename, 'wb')
+            for i, file in enumerate(file_list):
+                src, _ = sanitize_open(file, 'rb')
+                if key_list:
+                    decrypt_info = next((x for x in key_list if x['INDEX'] == i), decrypt_info)
+                    if decrypt_info['METHOD'] == 'AES-128':
+                        iv = decrypt_info.get('IV')
+                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                        encrypted_data = src.read()
+                        decrypted_data = AES.new(
+                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
+                        dest.write(decrypted_data)
+                    else:
+                        fragment_data = src.read()
+                        dest.write(fragment_data)
+                else:
+                    fragment_data = src.read()
+                    dest.write(fragment_data)
+                src.close()
+            dest.close()
+            if not self.params.get('keep_fragments', False):
+                for file_path in file_list:
+                    try:
+                        os.remove(file_path)
+                    except OSError as ose:
+                        self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
+                try:
+                    file_path = '%s.frag.urls' % tmpfilename
+                    os.remove(file_path)
+                except OSError as ose:
+                    self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
+
         return p.returncode
 
+    def _prepare_url(self, info_dict, url):
+        headers = info_dict.get('http_headers')
+        return sanitized_Request(url, None, headers) if headers else url
+
 
 class CurlFD(ExternalFD):
     AVAILABLE_OPT = '-V'
@@ -186,15 +242,17 @@ class WgetFD(ExternalFD):
 
 class Aria2cFD(ExternalFD):
     AVAILABLE_OPT = '-v'
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'frag_urls')
 
     def _make_cmd(self, tmpfilename, info_dict):
         cmd = [self.exe, '-c']
-        cmd += self._configuration_args([
-            '--min-split-size', '1M', '--max-connection-per-server', '4'])
         dn = os.path.dirname(tmpfilename)
+        if 'url_list' not in info_dict:
+            cmd += ['--out', os.path.basename(tmpfilename)]
+        verbose_level_args = ['--console-log-level=warn', '--summary-interval=0']
+        cmd += self._configuration_args(['--file-allocation=none', '-x16', '-j16', '-s16'] + verbose_level_args)
         if dn:
             cmd += ['--dir', dn]
-        cmd += ['--out', os.path.basename(tmpfilename)]
         if info_dict.get('http_headers') is not None:
             for key, val in info_dict['http_headers'].items():
                 cmd += ['--header', '%s: %s' % (key, val)]
@@ -202,7 +260,22 @@ class Aria2cFD(ExternalFD):
         cmd += self._option('--all-proxy', 'proxy')
         cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
         cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
-        cmd += ['--', info_dict['url']]
+        cmd += ['--auto-file-renaming=false']
+        if 'url_list' in info_dict:
+            cmd += verbose_level_args
+            cmd += ['--uri-selector', 'inorder', '--download-result=hide']
+            url_list_file = '%s.frag.urls' % tmpfilename
+            url_list = []
+            for [i, url] in enumerate(info_dict['url_list']):
+                tmpsegmentname = '%s_%s.frag' % (os.path.basename(tmpfilename), i)
+                url_list.append('%s\n\tout=%s' % (url, tmpsegmentname))
+            stream, _ = sanitize_open(url_list_file, 'wb')
+            stream.write('\n'.join(url_list).encode('utf-8'))
+            stream.close()
+
+            cmd += ['-i', url_list_file]
+        else:
+            cmd += ['--', info_dict['url']]
         return cmd
 
 
@@ -221,9 +294,7 @@ class HttpieFD(ExternalFD):
 
 
 class FFmpegFD(ExternalFD):
-    @classmethod
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
-    def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
 
     @classmethod
     def available(cls):
@@ -233,7 +304,7 @@ class FFmpegFD(ExternalFD):
         url = info_dict['url']
         ffpp = FFmpegPostProcessor(downloader=self)
         if not ffpp.available:
-            self.report_error('m3u8 download detected but ffmpeg could not be found. Please install one.')
+            self.report_error('m3u8 download detected but ffmpeg could not be found. Please install')
             return False
         ffpp.check_version()
 

youtube_dlc/downloader/f4m.py

@@ -267,13 +267,14 @@ class F4mFD(FragmentFD):
         media = doc.findall(_add_ns('media'))
         if not media:
             self.report_error('No media found')
-        for e in (doc.findall(_add_ns('drmAdditionalHeader'))
+        if not self.params.get('allow_unplayable_formats'):
-                  + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
+            for e in (doc.findall(_add_ns('drmAdditionalHeader'))
-            # If id attribute is missing it's valid for all media nodes
+                      + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
-            # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
+                # If id attribute is missing it's valid for all media nodes
-            if 'id' not in e.attrib:
+                # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
-                self.report_error('Missing ID in f4m DRM')
+                if 'id' not in e.attrib:
-        media = remove_encrypted_media(media)
+                    self.report_error('Missing ID in f4m DRM')
+            media = remove_encrypted_media(media)
         if not media:
             self.report_error('Unsupported DRM')
         return media

youtube_dlc/downloader/fragment.py

@@ -95,11 +95,12 @@ class FragmentFD(FileDownloader):
         frag_index_stream.write(json.dumps({'downloader': downloader}))
         frag_index_stream.close()
 
-    def _download_fragment(self, ctx, frag_url, info_dict, headers=None):
+    def _download_fragment(self, ctx, frag_url, info_dict, headers=None, request_data=None):
         fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
         fragment_info_dict = {
             'url': frag_url,
             'http_headers': headers or info_dict.get('http_headers'),
+            'request_data': request_data,
         }
         success = ctx['dl'].download(fragment_filename, fragment_info_dict)
         if not success:
@@ -277,3 +278,24 @@ class FragmentFD(FileDownloader):
             'status': 'finished',
             'elapsed': elapsed,
         })
+
+    def _prepare_external_frag_download(self, ctx):
+        if 'live' not in ctx:
+            ctx['live'] = False
+        if not ctx['live']:
+            total_frags_str = '%d' % ctx['total_frags']
+            ad_frags = ctx.get('ad_frags', 0)
+            if ad_frags:
+                total_frags_str += ' (not including %d ad)' % ad_frags
+        else:
+            total_frags_str = 'unknown (live)'
+        self.to_screen(
+            '[%s] Total fragments: %s' % (self.FD_NAME, total_frags_str))
+
+        tmpfilename = self.temp_name(ctx['filename'])
+
+        # Should be initialized before ytdl file check
+        ctx.update({
+            'tmpfilename': tmpfilename,
+            'fragment_index': 0,
+        })

youtube_dlc/downloader/hls.py

@@ -8,6 +8,7 @@ try:
 except ImportError:
     can_decrypt_frag = False
 
+from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
 from .external import FFmpegFD
 
@@ -28,9 +29,8 @@ class HlsFD(FragmentFD):
     FD_NAME = 'hlsnative'
 
     @staticmethod
-    def can_download(manifest, info_dict):
+    def can_download(manifest, info_dict, allow_unplayable_formats=False):
-        UNSUPPORTED_FEATURES = (
+        UNSUPPORTED_FEATURES = [
-            r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
             # r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [2]
 
             # Live streams heuristic does not always work (e.g. geo restricted to Germany
@@ -49,7 +49,11 @@ class HlsFD(FragmentFD):
             # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
             # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
             # 5. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.5
-        )
+        ]
+        if not allow_unplayable_formats:
+            UNSUPPORTED_FEATURES += [
+                r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
+            ]
         check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
         is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
         check_results.append(can_decrypt_frag or not is_aes128_enc)
@@ -65,7 +69,7 @@ class HlsFD(FragmentFD):
         man_url = urlh.geturl()
         s = urlh.read().decode('utf-8', 'ignore')
 
-        if not self.can_download(s, info_dict):
+        if not self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')):
             if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
                 self.report_error('pycrypto not found. Please install it.')
                 return False
@@ -73,10 +77,13 @@ class HlsFD(FragmentFD):
                 'hlsnative has detected features it does not support, '
                 'extraction will be delegated to ffmpeg')
             fd = FFmpegFD(self.ydl, self.params)
-            for ph in self._progress_hooks:
+            # TODO: Make progress updates work without hooking twice
-                fd.add_progress_hook(ph)
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
+        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+
         def is_ad_fragment_start(s):
             return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
                     or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
@@ -85,6 +92,8 @@ class HlsFD(FragmentFD):
             return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
                     or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
 
+        fragment_urls = []
+
         media_frags = 0
         ad_frags = 0
         ad_frag_next = False
@@ -109,7 +118,10 @@ class HlsFD(FragmentFD):
             'ad_frags': ad_frags,
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        if real_downloader:
+            self._prepare_external_frag_download(ctx)
+        else:
+            self._prepare_and_start_frag_download(ctx)
 
         fragment_retries = self.params.get('fragment_retries', 0)
         skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
@@ -122,6 +134,7 @@ class HlsFD(FragmentFD):
         i = 0
         media_sequence = 0
         decrypt_info = {'METHOD': 'NONE'}
+        key_list = []
         byte_range = {}
         frag_index = 0
         ad_frag_next = False
@@ -140,6 +153,11 @@ class HlsFD(FragmentFD):
                         else compat_urlparse.urljoin(man_url, line))
                     if extra_query:
                         frag_url = update_url_query(frag_url, extra_query)
+
+                    if real_downloader:
+                        fragment_urls.append(frag_url)
+                        continue
+
                     count = 0
                     headers = info_dict.get('http_headers', {})
                     if byte_range:
@@ -168,6 +186,7 @@ class HlsFD(FragmentFD):
                         self.report_error(
                             'giving up after %s fragment retries' % fragment_retries)
                         return False
+
                     if decrypt_info['METHOD'] == 'AES-128':
                         iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                         decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
@@ -197,6 +216,10 @@ class HlsFD(FragmentFD):
                             decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
                         if decrypt_url != decrypt_info['URI']:
                             decrypt_info['KEY'] = None
+                    key_data = decrypt_info.copy()
+                    key_data['INDEX'] = frag_index
+                    key_list.append(key_data)
+
                 elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
                     media_sequence = int(line[22:])
                 elif line.startswith('#EXT-X-BYTERANGE'):
@@ -211,6 +234,17 @@ class HlsFD(FragmentFD):
                 elif is_ad_fragment_end(line):
                     ad_frag_next = False
 
-        self._finish_frag_download(ctx)
+        if real_downloader:
-
+            info_copy = info_dict.copy()
+            info_copy['url_list'] = fragment_urls
+            info_copy['key_list'] = key_list
+            fd = real_downloader(self.ydl, self.params)
+            # TODO: Make progress updates work without hooking twice
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
+            success = fd.real_download(filename, info_copy)
+            if not success:
+                return False
+        else:
+            self._finish_frag_download(ctx)
         return True

youtube_dlc/downloader/http.py

@@ -27,6 +27,7 @@ from ..utils import (
 class HttpFD(FileDownloader):
     def real_download(self, filename, info_dict):
         url = info_dict['url']
+        request_data = info_dict.get('request_data', None)
 
         class DownloadContext(dict):
             __getattr__ = dict.get
@@ -101,7 +102,7 @@ class HttpFD(FileDownloader):
                 range_end = ctx.data_len - 1
             has_range = range_start is not None
             ctx.has_range = has_range
-            request = sanitized_Request(url, None, headers)
+            request = sanitized_Request(url, request_data, headers)
             if has_range:
                 set_range(request, range_start, range_end)
             # Establish connection
@@ -152,7 +153,7 @@ class HttpFD(FileDownloader):
                     try:
                         # Open the connection again without the range header
                         ctx.data = self.ydl.urlopen(
-                            sanitized_Request(url, None, headers))
+                            sanitized_Request(url, request_data, headers))
                         content_length = ctx.data.info()['Content-Length']
                     except (compat_urllib_error.HTTPError, ) as err:
                         if err.code < 500 or err.code >= 600:

youtube_dlc/downloader/niconico.py

@@ -0,0 +1,54 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import threading
+
+from .common import FileDownloader
+from ..downloader import _get_real_downloader
+from ..extractor.niconico import NiconicoIE
+from ..compat import compat_urllib_request
+
+
+class NiconicoDmcFD(FileDownloader):
+    """ Downloading niconico douga from DMC with heartbeat """
+
+    FD_NAME = 'niconico_dmc'
+
+    def real_download(self, filename, info_dict):
+        self.to_screen('[%s] Downloading from DMC' % self.FD_NAME)
+
+        ie = NiconicoIE(self.ydl)
+        info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
+
+        fd = _get_real_downloader(info_dict, params=self.params)(self.ydl, self.params)
+
+        success = download_complete = False
+        timer = [None]
+
+        heartbeat_lock = threading.Lock()
+        heartbeat_url = heartbeat_info_dict['url']
+        heartbeat_data = heartbeat_info_dict['data']
+        heartbeat_interval = heartbeat_info_dict.get('interval', 30)
+        self.to_screen('[%s] Heartbeat with %s second interval...' % (self.FD_NAME, heartbeat_interval))
+
+        def heartbeat():
+            try:
+                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data.encode())
+            except Exception:
+                self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
+
+            with heartbeat_lock:
+                if not download_complete:
+                    timer[0] = threading.Timer(heartbeat_interval, heartbeat)
+                    timer[0].start()
+
+        try:
+            heartbeat()
+            success = fd.real_download(filename, info_dict)
+        finally:
+            if heartbeat_lock:
+                with heartbeat_lock:
+                    timer[0].cancel()
+                    download_complete = True
+
+            return success

youtube_dlc/downloader/youtube_live_chat.py

@@ -1,9 +1,14 @@
 from __future__ import division, unicode_literals
 
-import re
 import json
 
 from .fragment import FragmentFD
+from ..compat import compat_urllib_error
+from ..utils import (
+    try_get,
+    RegexNotFoundError,
+)
+from ..extractor.youtube import YoutubeBaseInfoExtractor as YT_BaseIE
 
 
 class YoutubeLiveChatReplayFD(FragmentFD):
@@ -15,6 +20,7 @@ class YoutubeLiveChatReplayFD(FragmentFD):
         video_id = info_dict['video_id']
         self.to_screen('[%s] Downloading live chat' % self.FD_NAME)
 
+        fragment_retries = self.params.get('fragment_retries', 0)
         test = self.params.get('test', False)
 
         ctx = {
@@ -23,19 +29,53 @@ class YoutubeLiveChatReplayFD(FragmentFD):
             'total_frags': None,
         }
 
-        def dl_fragment(url):
+        ie = YT_BaseIE(self.ydl)
-            headers = info_dict.get('http_headers', {})
-            return self._download_fragment(ctx, url, info_dict, headers)
 
-        def parse_yt_initial_data(data):
+        def dl_fragment(url, data=None, headers=None):
-            window_patt = b'window\\["ytInitialData"\\]\\s*=\\s*(.*?)(?<=});'
+            http_headers = info_dict.get('http_headers', {})
-            var_patt = b'var\\s+ytInitialData\\s*=\\s*(.*?)(?<=});'
+            if headers:
-            for patt in window_patt, var_patt:
+                http_headers = http_headers.copy()
+                http_headers.update(headers)
+            return self._download_fragment(ctx, url, info_dict, http_headers, data)
+
+        def download_and_parse_fragment(url, frag_index, request_data):
+            count = 0
+            while count <= fragment_retries:
                 try:
-                    raw_json = re.search(patt, data).group(1)
+                    success, raw_fragment = dl_fragment(url, request_data, {'content-type': 'application/json'})
-                    return json.loads(raw_json)
+                    if not success:
-                except AttributeError:
+                        return False, None, None
-                    continue
+                    try:
+                        data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+                    except RegexNotFoundError:
+                        data = None
+                    if not data:
+                        data = json.loads(raw_fragment)
+                    live_chat_continuation = try_get(
+                        data,
+                        lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
+                    offset = continuation_id = None
+                    processed_fragment = bytearray()
+                    for action in live_chat_continuation.get('actions', []):
+                        if 'replayChatItemAction' in action:
+                            replay_chat_item_action = action['replayChatItemAction']
+                            offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
+                        processed_fragment.extend(
+                            json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
+                    if offset is not None:
+                        continuation_id = try_get(
+                            live_chat_continuation,
+                            lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
+                    self._append_fragment(ctx, processed_fragment)
+
+                    return True, continuation_id, offset
+                except compat_urllib_error.HTTPError as err:
+                    count += 1
+                    if count <= fragment_retries:
+                        self.report_retry_fragment(err, frag_index, count, fragment_retries)
+            if count > fragment_retries:
+                self.report_error('giving up after %s fragment retries' % fragment_retries)
+                return False, None, None
 
         self._prepare_and_start_frag_download(ctx)
 
@@ -43,55 +83,41 @@ class YoutubeLiveChatReplayFD(FragmentFD):
             'https://www.youtube.com/watch?v={}'.format(video_id))
         if not success:
             return False
-        data = parse_yt_initial_data(raw_fragment)
+        try:
-        continuation_id = data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
+            data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+        except RegexNotFoundError:
+            return False
+        continuation_id = try_get(
+            data,
+            lambda x: x['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation'])
         # no data yet but required to call _append_fragment
         self._append_fragment(ctx, b'')
 
-        first = True
+        ytcfg = ie._extract_ytcfg(video_id, raw_fragment.decode('utf-8', 'replace'))
-        offset = None
+
+        if not ytcfg:
+            return False
+        api_key = try_get(ytcfg, lambda x: x['INNERTUBE_API_KEY'])
+        innertube_context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'])
+        if not api_key or not innertube_context:
+            return False
+        url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+
+        frag_index = offset = 0
         while continuation_id is not None:
-            data = None
+            frag_index += 1
-            if first:
+            request_data = {
-                url = 'https://www.youtube.com/live_chat_replay?continuation={}'.format(continuation_id)
+                'context': innertube_context,
-                success, raw_fragment = dl_fragment(url)
+                'continuation': continuation_id,
-                if not success:
+            }
-                    return False
+            if frag_index > 1:
-                data = parse_yt_initial_data(raw_fragment)
+                request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
-            else:
+            success, continuation_id, offset = download_and_parse_fragment(
-                url = ('https://www.youtube.com/live_chat_replay/get_live_chat_replay'
+                url, frag_index, json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n')
-                       + '?continuation={}'.format(continuation_id)
+            if not success:
-                       + '&playerOffsetMs={}'.format(max(offset - 5000, 0))
+                return False
-                       + '&hidden=false'
+            if test:
-                       + '&pbj=1')
-                success, raw_fragment = dl_fragment(url)
-                if not success:
-                    return False
-                data = json.loads(raw_fragment)['response']
-
-            first = False
-            continuation_id = None
-
-            live_chat_continuation = data['continuationContents']['liveChatContinuation']
-            offset = None
-            processed_fragment = bytearray()
-            if 'actions' in live_chat_continuation:
-                for action in live_chat_continuation['actions']:
-                    if 'replayChatItemAction' in action:
-                        replay_chat_item_action = action['replayChatItemAction']
-                        offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
-                    processed_fragment.extend(
-                        json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
-                try:
-                    continuation_id = live_chat_continuation['continuations'][0]['liveChatReplayContinuationData']['continuation']
-                except KeyError:
-                    continuation_id = None
-
-            self._append_fragment(ctx, processed_fragment)
-
-            if test or offset is None:
                 break
 
         self._finish_frag_download(ctx)
-
         return True

youtube_dlc/extractor/abcnews.py

@@ -1,14 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import calendar
 import re
-import time
 
 from .amp import AMPIE
 from .common import InfoExtractor
-from .youtube import YoutubeIE
+from ..utils import (
-from ..compat import compat_urlparse
+    parse_duration,
+    parse_iso8601,
+    try_get,
+)
 
 
 class AbcNewsVideoIE(AMPIE):
@@ -18,8 +19,8 @@ class AbcNewsVideoIE(AMPIE):
                         (?:
                             abcnews\.go\.com/
                             (?:
-                                [^/]+/video/(?P<display_id>[0-9a-z-]+)-|
+                                (?:[^/]+/)*video/(?P<display_id>[0-9a-z-]+)-|
-                                video/embed\?.*?\bid=
+                                video/(?:embed|itemfeed)\?.*?\bid=
                             )|
                             fivethirtyeight\.abcnews\.go\.com/video/embed/\d+/
                         )
@@ -36,6 +37,8 @@ class AbcNewsVideoIE(AMPIE):
             'description': 'George Stephanopoulos goes one-on-one with Iranian Foreign Minister Dr. Javad Zarif.',
             'duration': 180,
             'thumbnail': r're:^https?://.*\.jpg$',
+            'timestamp': 1380454200,
+            'upload_date': '20130929',
         },
         'params': {
             # m3u8 download
@@ -47,6 +50,12 @@ class AbcNewsVideoIE(AMPIE):
     }, {
         'url': 'http://abcnews.go.com/2020/video/2020-husband-stands-teacher-jail-student-affairs-26119478',
         'only_matching': True,
+    }, {
+        'url': 'http://abcnews.go.com/video/itemfeed?id=46979033',
+        'only_matching': True,
+    }, {
+        'url': 'https://abcnews.go.com/GMA/News/video/history-christmas-story-67894761',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -67,28 +76,23 @@ class AbcNewsIE(InfoExtractor):
     _VALID_URL = r'https?://abcnews\.go\.com/(?:[^/]+/)+(?P<display_id>[0-9a-z-]+)/story\?id=(?P<id>\d+)'
 
     _TESTS = [{
-        'url': 'http://abcnews.go.com/Blotter/News/dramatic-video-rare-death-job-america/story?id=10498713#.UIhwosWHLjY',
+        # Youtube Embeds
+        'url': 'https://abcnews.go.com/Entertainment/peter-billingsley-child-actor-christmas-story-hollywood-power/story?id=51286501',
         'info_dict': {
-            'id': '10505354',
+            'id': '51286501',
-            'ext': 'flv',
+            'title': "Peter Billingsley: From child actor in 'A Christmas Story' to Hollywood power player",
-            'display_id': 'dramatic-video-rare-death-job-america',
+            'description': 'Billingsley went from a child actor to Hollywood power player.',
-            'title': 'Occupational Hazards',
-            'description': 'Nightline investigates the dangers that lurk at various jobs.',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'upload_date': '20100428',
-            'timestamp': 1272412800,
         },
-        'add_ie': ['AbcNewsVideo'],
+        'playlist_count': 5,
     }, {
         'url': 'http://abcnews.go.com/Entertainment/justin-timberlake-performs-stop-feeling-eurovision-2016/story?id=39125818',
         'info_dict': {
             'id': '38897857',
             'ext': 'mp4',
-            'display_id': 'justin-timberlake-performs-stop-feeling-eurovision-2016',
             'title': 'Justin Timberlake Drops Hints For Secret Single',
             'description': 'Lara Spencer reports the buzziest stories of the day in "GMA" Pop News.',
-            'upload_date': '20160515',
+            'upload_date': '20160505',
-            'timestamp': 1463329500,
+            'timestamp': 1462442280,
         },
         'params': {
             # m3u8 download
@@ -100,49 +104,55 @@ class AbcNewsIE(InfoExtractor):
     }, {
         'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
         'only_matching': True,
+    }, {
+        # inline.type == 'video'
+        'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        story_id = self._match_id(url)
-        display_id = mobj.group('display_id')
+        webpage = self._download_webpage(url, story_id)
-        video_id = mobj.group('id')
+        story = self._parse_json(self._search_regex(
+            r"window\['__abcnews__'\]\s*=\s*({.+?});",
+            webpage, 'data'), story_id)['page']['content']['story']['everscroll'][0]
+        article_contents = story.get('articleContents') or {}
 
-        webpage = self._download_webpage(url, video_id)
+        def entries():
-        video_url = self._search_regex(
+            featured_video = story.get('featuredVideo') or {}
-            r'window\.abcnvideo\.url\s*=\s*"([^"]+)"', webpage, 'video URL')
+            feed = try_get(featured_video, lambda x: x['video']['feed'])
-        full_video_url = compat_urlparse.urljoin(url, video_url)
+            if feed:
+                yield {
+                    '_type': 'url',
+                    'id': featured_video.get('id'),
+                    'title': featured_video.get('name'),
+                    'url': feed,
+                    'thumbnail': featured_video.get('images'),
+                    'description': featured_video.get('description'),
+                    'timestamp': parse_iso8601(featured_video.get('uploadDate')),
+                    'duration': parse_duration(featured_video.get('duration')),
+                    'ie_key': AbcNewsVideoIE.ie_key(),
+                }
 
-        youtube_url = YoutubeIE._extract_url(webpage)
+            for inline in (article_contents.get('inlines') or []):
+                inline_type = inline.get('type')
+                if inline_type == 'iframe':
+                    iframe_url = try_get(inline, lambda x: x['attrs']['src'])
+                    if iframe_url:
+                        yield self.url_result(iframe_url)
+                elif inline_type == 'video':
+                    video_id = inline.get('id')
+                    if video_id:
+                        yield {
+                            '_type': 'url',
+                            'id': video_id,
+                            'url': 'http://abcnews.go.com/video/embed?id=' + video_id,
+                            'thumbnail': inline.get('imgSrc') or inline.get('imgDefault'),
+                            'description': inline.get('description'),
+                            'duration': parse_duration(inline.get('duration')),
+                            'ie_key': AbcNewsVideoIE.ie_key(),
+                        }
 
-        timestamp = None
+        return self.playlist_result(
-        date_str = self._html_search_regex(
+            entries(), story_id, article_contents.get('headline'),
-            r'<span[^>]+class="timestamp">([^<]+)</span>',
+            article_contents.get('subHead'))
-            webpage, 'timestamp', fatal=False)
-        if date_str:
-            tz_offset = 0
-            if date_str.endswith(' ET'):  # Eastern Time
-                tz_offset = -5
-                date_str = date_str[:-3]
-            date_formats = ['%b. %d, %Y', '%b %d, %Y, %I:%M %p']
-            for date_format in date_formats:
-                try:
-                    timestamp = calendar.timegm(time.strptime(date_str.strip(), date_format))
-                except ValueError:
-                    continue
-            if timestamp is not None:
-                timestamp -= tz_offset * 3600
-
-        entry = {
-            '_type': 'url_transparent',
-            'ie_key': AbcNewsVideoIE.ie_key(),
-            'url': full_video_url,
-            'id': video_id,
-            'display_id': display_id,
-            'timestamp': timestamp,
-        }
-
-        if youtube_url:
-            entries = [entry, self.url_result(youtube_url, ie=YoutubeIE.ie_key())]
-            return self.playlist_result(entries)
-
-        return entry

youtube_dlc/extractor/adn.py

@@ -26,6 +26,7 @@ from ..utils import (
     strip_or_none,
     try_get,
     unified_strdate,
+    urlencode_postdata,
 )
 
 
@@ -51,9 +52,12 @@ class ADNIE(InfoExtractor):
         }
     }
 
+    _NETRC_MACHINE = 'animedigitalnetwork'
     _BASE_URL = 'http://animedigitalnetwork.fr'
     _API_BASE_URL = 'https://gw.api.animedigitalnetwork.fr/'
     _PLAYER_BASE_URL = _API_BASE_URL + 'player/'
+    _HEADERS = {}
+    _LOGIN_ERR_MESSAGE = 'Unable to log in'
     _RSA_KEY = (0x9B42B08905199A5CCE2026274399CA560ECB209EE9878A708B1C0812E1BB8CB5D1FB7441861147C1A1F2F3A0476DD63A9CAC20D3E983613346850AA6CB38F16DC7D720FD7D86FC6E5B3D5BBC72E14CD0BF9E869F2CEA2CCAD648F1DCE38F1FF916CEFB2D339B64AA0264372344BC775E265E8A852F88144AB0BD9AA06C1A4ABB, 65537)
     _POS_ALIGN_MAP = {
         'start': 1,
@@ -129,19 +133,42 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
             }])
         return subtitles
 
+    def _real_initialize(self):
+        username, password = self._get_login_info()
+        if not username:
+            return
+        try:
+            access_token = (self._download_json(
+                self._API_BASE_URL + 'authentication/login', None,
+                'Logging in', self._LOGIN_ERR_MESSAGE, fatal=False,
+                data=urlencode_postdata({
+                    'password': password,
+                    'rememberMe': False,
+                    'source': 'Web',
+                    'username': username,
+                })) or {}).get('accessToken')
+            if access_token:
+                self._HEADERS = {'authorization': 'Bearer ' + access_token}
+        except ExtractorError as e:
+            message = None
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
+                resp = self._parse_json(
+                    e.cause.read().decode(), None, fatal=False) or {}
+                message = resp.get('message') or resp.get('code')
+            self.report_warning(message or self._LOGIN_ERR_MESSAGE)
+
     def _real_extract(self, url):
         video_id = self._match_id(url)
         video_base_url = self._PLAYER_BASE_URL + 'video/%s/' % video_id
         player = self._download_json(
             video_base_url + 'configuration', video_id,
-            'Downloading player config JSON metadata')['player']
+            'Downloading player config JSON metadata',
+            headers=self._HEADERS)['player']
         options = player['options']
 
         user = options['user']
         if not user.get('hasAccess'):
-            raise ExtractorError(
+            self.raise_login_required()
-                'This video is only available for paying users', expected=True)
-            # self.raise_login_required() # FIXME: Login is not implemented
 
         token = self._download_json(
             user.get('refreshTokenUrl') or (self._PLAYER_BASE_URL + 'refresh/token'),
@@ -188,8 +215,7 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                 message = error.get('message')
                 if e.cause.code == 403 and error.get('code') == 'player-bad-geolocation-country':
                     self.raise_geo_restricted(msg=message)
-                else:
+                raise ExtractorError(message)
-                    raise ExtractorError(message)
         else:
             raise ExtractorError('Giving up retrying')
 

youtube_dlc/extractor/aenetworks.py

@@ -252,7 +252,7 @@ class AENetworksShowIE(AENetworksListBaseIE):
     _TESTS = [{
         'url': 'http://www.history.com/shows/ancient-aliens',
         'info_dict': {
-            'id': 'SH012427480000',
+            'id': 'SERIES1574',
             'title': 'Ancient Aliens',
             'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
         },

youtube_dlc/extractor/amp.py

@@ -8,6 +8,7 @@ from ..utils import (
     int_or_none,
     mimetype2ext,
     parse_iso8601,
+    unified_timestamp,
     url_or_none,
 )
 
@@ -88,7 +89,7 @@ class AMPIE(InfoExtractor):
 
         self._sort_formats(formats)
 
-        timestamp = parse_iso8601(item.get('pubDate'), ' ') or parse_iso8601(item.get('dc-date'))
+        timestamp = unified_timestamp(item.get('pubDate'), ' ') or parse_iso8601(item.get('dc-date'))
 
         return {
             'id': video_id,

youtube_dlc/extractor/audius.py

@@ -76,6 +76,7 @@ class AudiusBaseIE(InfoExtractor):
 
 class AudiusIE(AudiusBaseIE):
     _VALID_URL = r'''(?x)https?://(?:www\.)?(?:audius\.co/(?P<uploader>[\w\d-]+)(?!/album|/playlist)/(?P<title>\S+))'''
+    IE_DESC = 'Audius.co'
     _TESTS = [
         {
             # URL from Chrome address bar which replace backslash to forward slash
@@ -124,11 +125,15 @@ class AudiusIE(AudiusBaseIE):
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        uploader, title, track_id = mobj.groups()
+        track_id = try_get(mobj, lambda x: x.group('track_id'))
         if track_id is None:
+            title = mobj.group('title')
+            # uploader = mobj.group('uploader')
             url = self._prepare_url(url, title)
             track_data = self._resolve_url(url, title)
         else:  # API link
+            title = None
+            # uploader = None
             track_data = self._api_request('/tracks/%s' % track_id, track_id)
 
         if not isinstance(track_data, dict):
@@ -167,9 +172,26 @@ class AudiusIE(AudiusBaseIE):
         }
 
 
+class AudiusTrackIE(AudiusIE):
+    _VALID_URL = r'''(?x)(?:audius:)(?:https?://(?:www\.)?.+/v1/tracks/)?(?P<track_id>\w+)'''
+    IE_NAME = 'audius:track'
+    IE_DESC = 'Audius track ID or API link. Prepend with "audius:"'
+    _TESTS = [
+        {
+            'url': 'audius:9RWlo',
+            'only_matching': True
+        },
+        {
+            'url': 'audius:http://discoveryprovider.audius.prod-us-west-2.staked.cloud/v1/tracks/9RWlo',
+            'only_matching': True
+        },
+    ]
+
+
 class AudiusPlaylistIE(AudiusBaseIE):
     _VALID_URL = r'https?://(?:www\.)?audius\.co/(?P<uploader>[\w\d-]+)/(?:album|playlist)/(?P<title>\S+)'
     IE_NAME = 'audius:playlist'
+    IE_DESC = 'Audius.co playlists'
     _TEST = {
         'url': 'https://audius.co/test_acc/playlist/test-playlist-22910',
         'info_dict': {
@@ -189,14 +211,15 @@ class AudiusPlaylistIE(AudiusBaseIE):
             if not track_id:
                 raise ExtractorError('Unable to get track ID from playlist')
             entries.append(self.url_result(
-                '%s%s/tracks/%s' % (self._API_BASE, self._API_V, track_id),
+                'audius:%s' % track_id,
-                ie=AudiusIE.ie_key(), video_id=track_id))
+                ie=AudiusTrackIE.ie_key(), video_id=track_id))
         return entries
 
     def _real_extract(self, url):
         self._select_api_base()
         mobj = re.match(self._VALID_URL, url)
-        uploader, title = mobj.groups()
+        title = mobj.group('title')
+        # uploader = mobj.group('uploader')
         url = self._prepare_url(url, title)
         playlist_response = self._resolve_url(url, title)
 

youtube_dlc/extractor/awaan.py

@@ -48,6 +48,7 @@ class AWAANBaseIE(InfoExtractor):
             'duration': int_or_none(video_data.get('duration')),
             'timestamp': parse_iso8601(video_data.get('create_time'), ' '),
             'is_live': is_live,
+            'uploader_id': video_data.get('user_id'),
         }
 
 
@@ -107,6 +108,7 @@ class AWAANLiveIE(AWAANBaseIE):
             'title': 're:Dubai Al Oula [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
             'upload_date': '20150107',
             'timestamp': 1420588800,
+            'uploader_id': '71',
         },
         'params': {
             # m3u8 download

youtube_dlc/extractor/azmedien.py

@@ -47,7 +47,7 @@ class AZMedienIE(InfoExtractor):
         'url': 'https://www.telebaern.tv/telebaern-news/montag-1-oktober-2018-ganze-sendung-133531189#video=0_7xjo9lf1',
         'only_matching': True
     }]
-    _API_TEMPL = 'https://www.%s/api/pub/gql/%s/NewsArticleTeaser/cb9f2f81ed22e9b47f4ca64ea3cc5a5d13e88d1d'
+    _API_TEMPL = 'https://www.%s/api/pub/gql/%s/NewsArticleTeaser/a4016f65fe62b81dc6664dd9f4910e4ab40383be'
     _PARTNER_ID = '1719221'
 
     def _real_extract(self, url):

youtube_dlc/extractor/bleacherreport.py

@@ -90,13 +90,19 @@ class BleacherReportCMSIE(AMPIE):
     _VALID_URL = r'https?://(?:www\.)?bleacherreport\.com/video_embed\?id=(?P<id>[0-9a-f-]{36}|\d{5})'
     _TESTS = [{
         'url': 'http://bleacherreport.com/video_embed?id=8fd44c2f-3dc5-4821-9118-2c825a98c0e1&library=video-cms',
-        'md5': '2e4b0a997f9228ffa31fada5c53d1ed1',
+        'md5': '670b2d73f48549da032861130488c681',
         'info_dict': {
             'id': '8fd44c2f-3dc5-4821-9118-2c825a98c0e1',
-            'ext': 'flv',
+            'ext': 'mp4',
             'title': 'Cena vs. Rollins Would Expose the Heavyweight Division',
             'description': 'md5:984afb4ade2f9c0db35f3267ed88b36e',
+            'upload_date': '20150723',
+            'timestamp': 1437679032,
+
         },
+        'expected_warnings': [
+            'Unable to download f4m manifest'
+        ]
     }]
 
     def _real_extract(self, url):

youtube_dlc/extractor/bravotv.py

@@ -12,7 +12,7 @@ from ..utils import (
 
 
 class BravoTVIE(AdobePassIE):
-    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?P<req_id>bravotv|oxygen)\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'https://www.bravotv.com/top-chef/season-16/episode-15/videos/the-top-chef-season-16-winner-is',
         'md5': 'e34684cfea2a96cd2ee1ef3a60909de9',
@@ -28,10 +28,13 @@ class BravoTVIE(AdobePassIE):
     }, {
         'url': 'http://www.bravotv.com/below-deck/season-3/ep-14-reunion-part-1',
         'only_matching': True,
+    }, {
+        'url': 'https://www.oxygen.com/in-ice-cold-blood/season-2/episode-16/videos/handling-the-horwitz-house-after-the-murder-season-2',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
+        site, display_id = re.match(self._VALID_URL, url).groups()
         webpage = self._download_webpage(url, display_id)
         settings = self._parse_json(self._search_regex(
             r'<script[^>]+data-drupal-selector="drupal-settings-json"[^>]*>({.+?})</script>', webpage, 'drupal settings'),
@@ -53,11 +56,14 @@ class BravoTVIE(AdobePassIE):
                 tp_path = release_pid = tve['release_pid']
             if tve.get('entitlement') == 'auth':
                 adobe_pass = settings.get('tve_adobe_auth', {})
+                if site == 'bravotv':
+                    site = 'bravo'
                 resource = self._get_mvpd_resource(
-                    adobe_pass.get('adobePassResourceId', 'bravo'),
+                    adobe_pass.get('adobePassResourceId') or site,
                     tve['title'], release_pid, tve.get('rating'))
                 query['auth'] = self._extract_mvpd_auth(
-                    url, release_pid, adobe_pass.get('adobePassRequestorId', 'bravo'), resource)
+                    url, release_pid,
+                    adobe_pass.get('adobePassRequestorId') or site, resource)
         else:
             shared_playlist = settings['ls_playlist']
             account_pid = shared_playlist['account_pid']

youtube_dlc/extractor/brightcove.py

@@ -478,11 +478,12 @@ class BrightcoveNewIE(AdobePassIE):
             container = source.get('container')
             ext = mimetype2ext(source.get('type'))
             src = source.get('src')
+            skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
             # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
-            if container == 'WVM' or source.get('key_systems'):
+            if skip_unplayable and (container == 'WVM' or source.get('key_systems')):
                 num_drm_sources += 1
                 continue
-            elif ext == 'ism':
+            elif ext == 'ism' and skip_unplayable:
                 continue
             elif ext == 'm3u8' or container == 'M2TS':
                 if not src:
@@ -546,7 +547,8 @@ class BrightcoveNewIE(AdobePassIE):
                 error = errors[0]
                 raise ExtractorError(
                     error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
-            if sources and num_drm_sources == len(sources):
+            if (not self._downloader.params.get('allow_unplayable_formats')
+                    and sources and num_drm_sources == len(sources)):
                 raise ExtractorError('This video is DRM protected.', expected=True)
 
         self._sort_formats(formats)

youtube_dlc/extractor/ccma.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import datetime
 import re
 
 from .common import InfoExtractor
@@ -8,8 +9,8 @@ from ..utils import (
     clean_html,
     int_or_none,
     parse_duration,
-    parse_iso8601,
     parse_resolution,
+    try_get,
     url_or_none,
 )
 
@@ -24,8 +25,9 @@ class CCMAIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'L\'espot de La Marató de TV3',
             'description': 'md5:f12987f320e2f6e988e9908e4fe97765',
-            'timestamp': 1470918540,
+            'timestamp': 1478608140,
-            'upload_date': '20160811',
+            'upload_date': '20161108',
+            'age_limit': 0,
         }
     }, {
         'url': 'http://www.ccma.cat/catradio/alacarta/programa/el-consell-de-savis-analitza-el-derbi/audio/943685/',
@@ -35,8 +37,24 @@ class CCMAIE(InfoExtractor):
             'ext': 'mp3',
             'title': 'El Consell de Savis analitza el derbi',
             'description': 'md5:e2a3648145f3241cb9c6b4b624033e53',
-            'upload_date': '20171205',
+            'upload_date': '20170512',
-            'timestamp': 1512507300,
+            'timestamp': 1494622500,
+            'vcodec': 'none',
+            'categories': ['Esports'],
+        }
+    }, {
+        'url': 'http://www.ccma.cat/tv3/alacarta/crims/crims-josep-tallada-lespereu-me-capitol-1/video/6031387/',
+        'md5': 'b43c3d3486f430f3032b5b160d80cbc3',
+        'info_dict': {
+            'id': '6031387',
+            'ext': 'mp4',
+            'title': 'Crims - Josep Talleda, l\'"Espereu-me" (capítol 1)',
+            'description': 'md5:7cbdafb640da9d0d2c0f62bad1e74e60',
+            'timestamp': 1582577700,
+            'upload_date': '20200224',
+            'subtitles': 'mincount:4',
+            'age_limit': 16,
+            'series': 'Crims',
         }
     }]
 
@@ -72,17 +90,27 @@ class CCMAIE(InfoExtractor):
 
         informacio = media['informacio']
         title = informacio['titol']
-        durada = informacio.get('durada', {})
+        durada = informacio.get('durada') or {}
         duration = int_or_none(durada.get('milisegons'), 1000) or parse_duration(durada.get('text'))
-        timestamp = parse_iso8601(informacio.get('data_emissio', {}).get('utc'))
+        tematica = try_get(informacio, lambda x: x['tematica']['text'])
+
+        timestamp = None
+        data_utc = try_get(informacio, lambda x: x['data_emissio']['utc'])
+        try:
+            timestamp = datetime.datetime.strptime(
+                data_utc, '%Y-%d-%mT%H:%M:%S%z').timestamp()
+        except TypeError:
+            pass
 
         subtitles = {}
-        subtitols = media.get('subtitols', {})
+        subtitols = media.get('subtitols') or []
-        if subtitols:
+        if isinstance(subtitols, dict):
-            sub_url = subtitols.get('url')
+            subtitols = [subtitols]
+        for st in subtitols:
+            sub_url = st.get('url')
             if sub_url:
                 subtitles.setdefault(
-                    subtitols.get('iso') or subtitols.get('text') or 'ca', []).append({
+                    st.get('iso') or st.get('text') or 'ca', []).append({
                         'url': sub_url,
                     })
 
@@ -97,6 +125,16 @@ class CCMAIE(InfoExtractor):
                     'height': int_or_none(imatges.get('alcada')),
                 }]
 
+        age_limit = None
+        codi_etic = try_get(informacio, lambda x: x['codi_etic']['id'])
+        if codi_etic:
+            codi_etic_s = codi_etic.split('_')
+            if len(codi_etic_s) == 2:
+                if codi_etic_s[1] == 'TP':
+                    age_limit = 0
+                else:
+                    age_limit = int_or_none(codi_etic_s[1])
+
         return {
             'id': media_id,
             'title': title,
@@ -106,4 +144,9 @@ class CCMAIE(InfoExtractor):
             'thumbnails': thumbnails,
             'subtitles': subtitles,
             'formats': formats,
+            'age_limit': age_limit,
+            'alt_title': informacio.get('titol_complet'),
+            'episode_number': int_or_none(informacio.get('capitol')),
+            'categories': [tematica] if tematica else None,
+            'series': informacio.get('programa'),
         }

youtube_dlc/extractor/cda.py

@@ -95,8 +95,11 @@ class CDAIE(InfoExtractor):
         if 'Ten film jest dostępny dla użytkowników premium' in webpage:
             raise ExtractorError('This video is only available for premium users.', expected=True)
 
+        if re.search(r'niedostępn[ey] w(?:&nbsp;|\s+)Twoim kraju\s*<', webpage):
+            self.raise_geo_restricted()
+
         need_confirm_age = False
-        if self._html_search_regex(r'(<form[^>]+action="/a/validatebirth")',
+        if self._html_search_regex(r'(<form[^>]+action="[^"]*/a/validatebirth[^"]*")',
                                    webpage, 'birthday validate form', default=None):
             webpage = self._download_age_confirm_page(
                 url, video_id, note='Confirming age')

youtube_dlc/extractor/ceskatelevize.py

@@ -147,7 +147,8 @@ class CeskaTelevizeIE(InfoExtractor):
                 is_live = item.get('type') == 'LIVE'
                 formats = []
                 for format_id, stream_url in item.get('streamUrls', {}).items():
-                    if 'drmOnly=true' in stream_url:
+                    if (not self._downloader.params.get('allow_unplayable_formats')
+                            and 'drmOnly=true' in stream_url):
                         continue
                     if 'playerType=flash' in stream_url:
                         stream_formats = self._extract_m3u8_formats(

youtube_dlc/extractor/common.py

@@ -336,9 +336,8 @@ class InfoExtractor(object):
     There must be a key "entries", which is a list, an iterable, or a PagedList
     object, each element of which is a valid dictionary by this specification.
 
-    Additionally, playlists can have "id", "title", "description", "uploader",
+    Additionally, playlists can have "id", "title", and any other relevent
-    "uploader_id", "uploader_url", "duration" attributes with the same semantics
+    attributes with the same semantics as videos (see above).
-    as videos (see above).
 
 
     _type "multi_video" indicates that there are multiple videos that
@@ -967,15 +966,16 @@ class InfoExtractor(object):
             urls, playlist_id=playlist_id, playlist_title=playlist_title)
 
     @staticmethod
-    def playlist_result(entries, playlist_id=None, playlist_title=None, playlist_description=None):
+    def playlist_result(entries, playlist_id=None, playlist_title=None, playlist_description=None, **kwargs):
         """Returns a playlist"""
         video_info = {'_type': 'playlist',
                       'entries': entries}
+        video_info.update(kwargs)
         if playlist_id:
             video_info['id'] = playlist_id
         if playlist_title:
             video_info['title'] = playlist_title
-        if playlist_description:
+        if playlist_description is not None:
             video_info['description'] = playlist_description
         return video_info
 
@@ -1366,13 +1366,13 @@ class InfoExtractor(object):
     class FormatSort:
         regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
 
-        default = ('hidden', 'has_video', 'extractor', 'lang', 'quality',
+        default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
-                   'res', 'fps', 'codec', 'size', 'br', 'asr',
+                   'res', 'fps', 'codec:vp9', 'size', 'br', 'asr',
-                   'proto', 'ext', 'has_audio', 'source', 'format_id')
+                   'proto', 'ext', 'has_audio', 'source', 'format_id')  # These must not be aliases
 
         settings = {
             'vcodec': {'type': 'ordered', 'regex': True,
-                       'order': ['vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
+                       'order': ['av0?1', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
             'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
@@ -2264,7 +2264,7 @@ class InfoExtractor(object):
             })
         return entries
 
-    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, formats_dict={}, data=None, headers={}, query={}):
+    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             mpd_url, video_id,
             note=note or 'Downloading MPD manifest',
@@ -2278,10 +2278,9 @@ class InfoExtractor(object):
         mpd_base_url = base_url(urlh.geturl())
 
         return self._parse_mpd_formats(
-            mpd_doc, mpd_id=mpd_id, mpd_base_url=mpd_base_url,
+            mpd_doc, mpd_id, mpd_base_url, mpd_url)
-            formats_dict=formats_dict, mpd_url=mpd_url)
 
-    def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', formats_dict={}, mpd_url=None):
+    def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', mpd_url=None):
         """
         Parse formats from MPD manifest.
         References:
@@ -2359,6 +2358,8 @@ class InfoExtractor(object):
                         extract_Initialization(segment_template)
             return ms_info
 
+        skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
+
         mpd_duration = parse_duration(mpd_doc.get('mediaPresentationDuration'))
         formats = []
         for period in mpd_doc.findall(_add_ns('Period')):
@@ -2368,11 +2369,11 @@ class InfoExtractor(object):
                 'timescale': 1,
             })
             for adaptation_set in period.findall(_add_ns('AdaptationSet')):
-                if is_drm_protected(adaptation_set):
+                if skip_unplayable and is_drm_protected(adaptation_set):
                     continue
                 adaption_set_ms_info = extract_multisegment_info(adaptation_set, period_ms_info)
                 for representation in adaptation_set.findall(_add_ns('Representation')):
-                    if is_drm_protected(representation):
+                    if skip_unplayable and is_drm_protected(representation):
                         continue
                     representation_attrib = adaptation_set.attrib.copy()
                     representation_attrib.update(representation.attrib)
@@ -2560,15 +2561,7 @@ class InfoExtractor(object):
                         else:
                             # Assuming direct URL to unfragmented media.
                             f['url'] = base_url
-
+                        formats.append(f)
-                        # According to [1, 5.3.5.2, Table 7, page 35] @id of Representation
-                        # is not necessarily unique within a Period thus formats with
-                        # the same `format_id` are quite possible. There are numerous examples
-                        # of such manifests (see https://github.com/ytdl-org/youtube-dl/issues/15111,
-                        # https://github.com/ytdl-org/youtube-dl/issues/13919)
-                        full_info = formats_dict.get(representation_id, {}).copy()
-                        full_info.update(f)
-                        formats.append(full_info)
                     else:
                         self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
         return formats
@@ -2594,7 +2587,10 @@ class InfoExtractor(object):
          1. [MS-SSTR]: Smooth Streaming Protocol,
             https://msdn.microsoft.com/en-us/library/ff469518.aspx
         """
-        if ism_doc.get('IsLive') == 'TRUE' or ism_doc.find('Protection') is not None:
+        if ism_doc.get('IsLive') == 'TRUE':
+            return []
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and ism_doc.find('Protection') is not None):
             return []
 
         duration = int(ism_doc.attrib['Duration'])

youtube_dlc/extractor/crackle.py

@@ -103,7 +103,7 @@ class CrackleIE(InfoExtractor):
 
             formats = []
             for e in media['MediaURLs']:
-                if e.get('UseDRM') is True:
+                if not self._downloader.params.get('allow_unplayable_formats') and e.get('UseDRM') is True:
                     continue
                 format_url = url_or_none(e.get('Path'))
                 if not format_url:

youtube_dlc/extractor/egghead.py

@@ -12,7 +12,14 @@ from ..utils import (
 )
 
 
-class EggheadCourseIE(InfoExtractor):
+class EggheadBaseIE(InfoExtractor):
+    def _call_api(self, path, video_id, resource, fatal=True):
+        return self._download_json(
+            'https://app.egghead.io/api/v1/' + path,
+            video_id, 'Downloading %s JSON' % resource, fatal=fatal)
+
+
+class EggheadCourseIE(EggheadBaseIE):
     IE_DESC = 'egghead.io course'
     IE_NAME = 'egghead:course'
     _VALID_URL = r'https://egghead\.io/courses/(?P<id>[^/?#&]+)'
@@ -28,10 +35,9 @@ class EggheadCourseIE(InfoExtractor):
 
     def _real_extract(self, url):
         playlist_id = self._match_id(url)
-
+        series_path = 'series/' + playlist_id
-        lessons = self._download_json(
+        lessons = self._call_api(
-            'https://egghead.io/api/v1/series/%s/lessons' % playlist_id,
+            series_path + '/lessons', playlist_id, 'course lessons')
-            playlist_id, 'Downloading course lessons JSON')
 
         entries = []
         for lesson in lessons:
@@ -44,9 +50,8 @@ class EggheadCourseIE(InfoExtractor):
             entries.append(self.url_result(
                 lesson_url, ie=EggheadLessonIE.ie_key(), video_id=lesson_id))
 
-        course = self._download_json(
+        course = self._call_api(
-            'https://egghead.io/api/v1/series/%s' % playlist_id,
+            series_path, playlist_id, 'course', False) or {}
-            playlist_id, 'Downloading course JSON', fatal=False) or {}
 
         playlist_id = course.get('id')
         if playlist_id:
@@ -57,7 +62,7 @@ class EggheadCourseIE(InfoExtractor):
             course.get('description'))
 
 
-class EggheadLessonIE(InfoExtractor):
+class EggheadLessonIE(EggheadBaseIE):
     IE_DESC = 'egghead.io lesson'
     IE_NAME = 'egghead:lesson'
     _VALID_URL = r'https://egghead\.io/(?:api/v1/)?lessons/(?P<id>[^/?#&]+)'
@@ -74,7 +79,7 @@ class EggheadLessonIE(InfoExtractor):
             'upload_date': '20161209',
             'duration': 304,
             'view_count': 0,
-            'tags': ['javascript', 'free'],
+            'tags': 'count:2',
         },
         'params': {
             'skip_download': True,
@@ -88,8 +93,8 @@ class EggheadLessonIE(InfoExtractor):
     def _real_extract(self, url):
         display_id = self._match_id(url)
 
-        lesson = self._download_json(
+        lesson = self._call_api(
-            'https://egghead.io/api/v1/lessons/%s' % display_id, display_id)
+            'lessons/' + display_id, display_id, 'lesson')
 
         lesson_id = compat_str(lesson['id'])
         title = lesson['title']

youtube_dlc/extractor/extractors.py

@@ -90,7 +90,11 @@ from .atvat import ATVAtIE
 from .audimedia import AudiMediaIE
 from .audioboom import AudioBoomIE
 from .audiomack import AudiomackIE, AudiomackAlbumIE
-from .audius import AudiusIE, AudiusPlaylistIE
+from .audius import (
+    AudiusIE,
+    AudiusTrackIE,
+    AudiusPlaylistIE
+)
 from .awaan import (
     AWAANIE,
     AWAANVideoIE,
@@ -498,8 +502,8 @@ from .hungama import (
 from .hypem import HypemIE
 from .ign import (
     IGNIE,
-    OneUPIE,
+    IGNVideoIE,
-    PCMagIE,
+    IGNArticleIE,
 )
 from .iheart import (
     IHeartRadioIE,
@@ -1304,6 +1308,7 @@ from .tv2 import (
     TV2IE,
     TV2ArticleIE,
     KatsomoIE,
+    MTVUutisetArticleIE,
 )
 from .tv2dk import (
     TV2DKIE,
@@ -1444,7 +1449,6 @@ from .vidme import (
     VidmeUserIE,
     VidmeUserLikesIE,
 )
-from .vidzi import VidziIE
 from .vier import VierIE, VierVideosIE
 from .viewlift import (
     ViewLiftIE,
@@ -1504,6 +1508,7 @@ from .vrv import (
     VRVSeriesIE,
 )
 from .vshare import VShareIE
+from .vtm import VTMIE
 from .medialaan import MedialaanIE
 from .vube import VubeIE
 from .vuclip import VuClipIE

youtube_dlc/extractor/generic.py

@@ -130,7 +130,9 @@ from .kinja import KinjaEmbedIE
 from .gedi import GediEmbedsIE
 from .rcs import RCSEmbedsIE
 from .bitchute import BitChuteIE
+from .rumble import RumbleEmbedIE
 from .arcpublishing import ArcPublishingIE
+from .medialaan import MedialaanIE
 
 
 class GenericIE(InfoExtractor):
@@ -2224,6 +2226,20 @@ class GenericIE(InfoExtractor):
                 'duration': 1581,
             },
         },
+        {
+            # MyChannels SDK embed
+            # https://www.24kitchen.nl/populair/deskundige-dit-waarom-sommigen-gevoelig-zijn-voor-voedselallergieen
+            'url': 'https://www.demorgen.be/nieuws/burgemeester-rotterdam-richt-zich-in-videoboodschap-tot-relschoppers-voelt-het-goed~b0bcfd741/',
+            'md5': '90c0699c37006ef18e198c032d81739c',
+            'info_dict': {
+                'id': '194165',
+                'ext': 'mp4',
+                'title': 'Burgemeester Aboutaleb spreekt relschoppers toe',
+                'timestamp': 1611740340,
+                'upload_date': '20210127',
+                'duration': 159,
+            },
+        },
     ]
 
     def report_following_redirect(self, new_url):
@@ -2463,6 +2479,9 @@ class GenericIE(InfoExtractor):
         webpage = self._webpage_read_content(
             full_response, url, video_id, prefix=first_bytes)
 
+        if '<title>DPG Media Privacy Gate</title>' in webpage:
+            webpage = self._download_webpage(url, video_id)
+
         self.report_extraction(video_id)
 
         # Is it an RSS feed, a SMIL file, an XSPF playlist or a MPD manifest?
@@ -2594,6 +2613,11 @@ class GenericIE(InfoExtractor):
         if arc_urls:
             return self.playlist_from_matches(arc_urls, video_id, video_title, ie=ArcPublishingIE.ie_key())
 
+        mychannels_urls = MedialaanIE._extract_urls(webpage)
+        if mychannels_urls:
+            return self.playlist_from_matches(
+                mychannels_urls, video_id, video_title, ie=MedialaanIE.ie_key())
+
         # Look for embedded rtl.nl player
         matches = re.findall(
             r'<iframe[^>]+?src="((?:https?:)?//(?:(?:www|static)\.)?rtl\.nl/(?:system/videoplayer/[^"]+(?:video_)?)?embed[^"]+)"',
@@ -3315,6 +3339,13 @@ class GenericIE(InfoExtractor):
             return self.playlist_from_matches(
                 bitchute_urls, video_id, video_title, ie=BitChuteIE.ie_key())
 
+        rumble_urls = RumbleEmbedIE._extract_urls(webpage)
+        if len(rumble_urls) == 1:
+            return self.url_result(rumble_urls[0], RumbleEmbedIE.ie_key())
+        if rumble_urls:
+            return self.playlist_from_matches(
+                rumble_urls, video_id, video_title, ie=RumbleEmbedIE.ie_key())
+
         # Look for HTML5 media
         entries = self._parse_html5_media_entries(url, webpage, video_id, m3u8_id='hls')
         if entries:

youtube_dlc/extractor/globo.py

@@ -96,7 +96,7 @@ class GloboIE(InfoExtractor):
         video = self._download_json(
             'http://api.globovideos.com/videos/%s/playlist' % video_id,
             video_id)['videos'][0]
-        if video.get('encrypted') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('encrypted') is True:
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         title = video['title']

youtube_dlc/extractor/googledrive.py

@@ -7,6 +7,7 @@ from ..compat import compat_parse_qs
 from ..utils import (
     determine_ext,
     ExtractorError,
+    get_element_by_class,
     int_or_none,
     lowercase_escape,
     try_get,
@@ -237,7 +238,7 @@ class GoogleDriveIE(InfoExtractor):
                 if confirmation_webpage:
                     confirm = self._search_regex(
                         r'confirm=([^&"\']+)', confirmation_webpage,
-                        'confirmation code', fatal=False)
+                        'confirmation code', default=None)
                     if confirm:
                         confirmed_source_url = update_url_query(source_url, {
                             'confirm': confirm,
@@ -245,6 +246,11 @@ class GoogleDriveIE(InfoExtractor):
                         urlh = request_source_file(confirmed_source_url, 'confirmed source')
                         if urlh and urlh.headers.get('Content-Disposition'):
                             add_source_format(urlh)
+                    else:
+                        self.report_warning(
+                            get_element_by_class('uc-error-subcaption', confirmation_webpage)
+                            or get_element_by_class('uc-error-caption', confirmation_webpage)
+                            or 'unable to extract confirmation code')
 
         if not formats and reason:
             raise ExtractorError(reason, expected=True)

youtube_dlc/extractor/hotstar.py

@@ -141,7 +141,7 @@ class HotStarIE(HotStarBaseIE):
 
         title = video_data['title']
 
-        if video_data.get('drmProtected'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drmProtected'):
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         headers = {'Referer': url}

youtube_dlc/extractor/ign.py

@@ -3,230 +3,255 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from ..compat import (
+    compat_parse_qs,
+    compat_urllib_parse_urlparse,
+)
 from ..utils import (
+    HEADRequest,
+    determine_ext,
     int_or_none,
     parse_iso8601,
+    strip_or_none,
+    try_get,
 )
 
 
-class IGNIE(InfoExtractor):
+class IGNBaseIE(InfoExtractor):
+    def _call_api(self, slug):
+        return self._download_json(
+            'http://apis.ign.com/{0}/v3/{0}s/slug/{1}'.format(self._PAGE_TYPE, slug), slug)
+
+
+class IGNIE(IGNBaseIE):
     """
     Extractor for some of the IGN sites, like www.ign.com, es.ign.com de.ign.com.
     Some videos of it.ign.com are also supported
     """
 
-    _VALID_URL = r'https?://.+?\.ign\.com/(?:[^/]+/)?(?P<type>videos|show_videos|articles|feature|(?:[^/]+/\d+/video))(/.+)?/(?P<name_or_id>.+)'
+    _VALID_URL = r'https?://(?:.+?\.ign|www\.pcmag)\.com/videos/(?:\d{4}/\d{2}/\d{2}/)?(?P<id>[^/?&#]+)'
     IE_NAME = 'ign.com'
+    _PAGE_TYPE = 'video'
 
-    _API_URL_TEMPLATE = 'http://apis.ign.com/video/v3/videos/%s'
+    _TESTS = [{
-    _EMBED_RE = r'<iframe[^>]+?["\']((?:https?:)?//.+?\.ign\.com.+?/embed.+?)["\']'
+        'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
-
+        'md5': 'd2e1586d9987d40fad7867bf96a018ea',
-    _TESTS = [
+        'info_dict': {
-        {
+            'id': '8f862beef863986b2785559b9e1aa599',
-            'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
+            'ext': 'mp4',
-            'md5': 'febda82c4bafecd2d44b6e1a18a595f8',
+            'title': 'The Last of Us Review',
-            'info_dict': {
+            'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
-                'id': '8f862beef863986b2785559b9e1aa599',
+            'timestamp': 1370440800,
-                'ext': 'mp4',
+            'upload_date': '20130605',
-                'title': 'The Last of Us Review',
+            'tags': 'count:9',
-                'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
+        }
-                'timestamp': 1370440800,
+    }, {
-                'upload_date': '20130605',
+        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
-                'uploader_id': 'cberidon@ign.com',
+        'md5': 'f1581a6fe8c5121be5b807684aeac3f6',
-            }
+        'info_dict': {
-        },
+            'id': 'ee10d774b508c9b8ec07e763b9125b91',
-        {
+            'ext': 'mp4',
-            'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
+            'title': 'What\'s New Now: Is GoGo Snooping on Your Data?',
-            'info_dict': {
+            'description': 'md5:817a20299de610bd56f13175386da6fa',
-                'id': '100-little-things-in-gta-5-that-will-blow-your-mind',
+            'timestamp': 1420571160,
-            },
+            'upload_date': '20150106',
-            'playlist': [
+            'tags': 'count:4',
-                {
+        }
-                    'info_dict': {
+    }, {
-                        'id': '5ebbd138523268b93c9141af17bec937',
+        'url': 'https://www.ign.com/videos/is-a-resident-evil-4-remake-on-the-way-ign-daily-fix',
-                        'ext': 'mp4',
+        'only_matching': True,
-                        'title': 'GTA 5 Video Review',
+    }]
-                        'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
-                        'timestamp': 1379339880,
-                        'upload_date': '20130916',
-                        'uploader_id': 'danieljkrupa@gmail.com',
-                    },
-                },
-                {
-                    'info_dict': {
-                        'id': '638672ee848ae4ff108df2a296418ee2',
-                        'ext': 'mp4',
-                        'title': '26 Twisted Moments from GTA 5 in Slow Motion',
-                        'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
-                        'timestamp': 1386878820,
-                        'upload_date': '20131212',
-                        'uploader_id': 'togilvie@ign.com',
-                    },
-                },
-            ],
-            'params': {
-                'skip_download': True,
-            },
-        },
-        {
-            'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
-            'md5': '618fedb9c901fd086f6f093564ef8558',
-            'info_dict': {
-                'id': '078fdd005f6d3c02f63d795faa1b984f',
-                'ext': 'mp4',
-                'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
-                'description': 'Brian and Jared explore Michel Ancel\'s captivating new preview.',
-                'timestamp': 1408047180,
-                'upload_date': '20140814',
-                'uploader_id': 'jamesduggan1990@gmail.com',
-            },
-        },
-        {
-            'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
-            'only_matching': True,
-        },
-        {
-            'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
-            'only_matching': True,
-        },
-        {
-            # videoId pattern
-            'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
-            'only_matching': True,
-        },
-    ]
-
-    def _find_video_id(self, webpage):
-        res_id = [
-            r'"video_id"\s*:\s*"(.*?)"',
-            r'class="hero-poster[^"]*?"[^>]*id="(.+?)"',
-            r'data-video-id="(.+?)"',
-            r'<object id="vid_(.+?)"',
-            r'<meta name="og:image" content=".*/(.+?)-(.+?)/.+.jpg"',
-            r'videoId&quot;\s*:\s*&quot;(.+?)&quot;',
-            r'videoId["\']\s*:\s*["\']([^"\']+?)["\']',
-        ]
-        return self._search_regex(res_id, webpage, 'video id', default=None)
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        display_id = self._match_id(url)
-        name_or_id = mobj.group('name_or_id')
+        video = self._call_api(display_id)
-        page_type = mobj.group('type')
+        video_id = video['videoId']
-        webpage = self._download_webpage(url, name_or_id)
+        metadata = video['metadata']
-        if page_type != 'video':
+        title = metadata.get('longTitle') or metadata.get('title') or metadata['name']
-            multiple_urls = re.findall(
-                r'<param name="flashvars"[^>]*value="[^"]*?url=(https?://www\.ign\.com/videos/.*?)["&]',
-                webpage)
-            if multiple_urls:
-                entries = [self.url_result(u, ie='IGN') for u in multiple_urls]
-                return {
-                    '_type': 'playlist',
-                    'id': name_or_id,
-                    'entries': entries,
-                }
-
-        video_id = self._find_video_id(webpage)
-        if not video_id:
-            return self.url_result(self._search_regex(
-                self._EMBED_RE, webpage, 'embed url'))
-        return self._get_video_info(video_id)
-
-    def _get_video_info(self, video_id):
-        api_data = self._download_json(
-            self._API_URL_TEMPLATE % video_id, video_id)
 
         formats = []
-        m3u8_url = api_data['refs'].get('m3uUrl')
+        refs = video.get('refs') or {}
+
+        m3u8_url = refs.get('m3uUrl')
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', 'm3u8_native',
                 m3u8_id='hls', fatal=False))
-        f4m_url = api_data['refs'].get('f4mUrl')
+
+        f4m_url = refs.get('f4mUrl')
         if f4m_url:
             formats.extend(self._extract_f4m_formats(
                 f4m_url, video_id, f4m_id='hds', fatal=False))
-        for asset in api_data['assets']:
+
+        for asset in (video.get('assets') or []):
+            asset_url = asset.get('url')
+            if not asset_url:
+                continue
             formats.append({
-                'url': asset['url'],
+                'url': asset_url,
-                'tbr': asset.get('actual_bitrate_kbps'),
+                'tbr': int_or_none(asset.get('bitrate'), 1000),
-                'fps': asset.get('frame_rate'),
+                'fps': int_or_none(asset.get('frame_rate')),
                 'height': int_or_none(asset.get('height')),
                 'width': int_or_none(asset.get('width')),
             })
+
+        mezzanine_url = try_get(video, lambda x: x['system']['mezzanineUrl'])
+        if mezzanine_url:
+            formats.append({
+                'ext': determine_ext(mezzanine_url, 'mp4'),
+                'format_id': 'mezzanine',
+                'preference': 1,
+                'url': mezzanine_url,
+            })
+
         self._sort_formats(formats)
 
-        thumbnails = [{
+        thumbnails = []
-            'url': thumbnail['url']
+        for thumbnail in (video.get('thumbnails') or []):
-        } for thumbnail in api_data.get('thumbnails', [])]
+            thumbnail_url = thumbnail.get('url')
+            if not thumbnail_url:
+                continue
+            thumbnails.append({
+                'url': thumbnail_url,
+            })
 
-        metadata = api_data['metadata']
+        tags = []
+        for tag in (video.get('tags') or []):
+            display_name = tag.get('displayName')
+            if not display_name:
+                continue
+            tags.append(display_name)
 
         return {
-            'id': api_data.get('videoId') or video_id,
+            'id': video_id,
-            'title': metadata.get('longTitle') or metadata.get('name') or metadata.get['title'],
+            'title': title,
-            'description': metadata.get('description'),
+            'description': strip_or_none(metadata.get('description')),
             'timestamp': parse_iso8601(metadata.get('publishDate')),
             'duration': int_or_none(metadata.get('duration')),
-            'display_id': metadata.get('slug') or video_id,
+            'display_id': display_id,
-            'uploader_id': metadata.get('creator'),
             'thumbnails': thumbnails,
             'formats': formats,
+            'tags': tags,
         }
 
 
-class OneUPIE(IGNIE):
+class IGNVideoIE(InfoExtractor):
-    _VALID_URL = r'https?://gamevideos\.1up\.com/(?P<type>video)/id/(?P<name_or_id>.+)\.html'
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:[a-z]{2}/)?[^/]+/(?P<id>\d+)/(?:video|trailer)/'
-    IE_NAME = '1up.com'
-
     _TESTS = [{
-        'url': 'http://gamevideos.1up.com/video/id/34976.html',
+        'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
-        'md5': 'c9cc69e07acb675c31a16719f909e347',
+        'md5': 'dd9aca7ed2657c4e118d8b261e5e9de1',
         'info_dict': {
-            'id': '34976',
+            'id': 'e9be7ea899a9bbfc0674accc22a36cc8',
             'ext': 'mp4',
-            'title': 'Sniper Elite V2 - Trailer',
+            'title': 'How Hitman Aims to Be Different Than Every Other Stealth Game - NYCC 2015',
-            'description': 'md5:bf0516c5ee32a3217aa703e9b1bc7826',
+            'description': 'Taking out assassination targets in Hitman has never been more stylish.',
-            'timestamp': 1313099220,
+            'timestamp': 1444665600,
-            'upload_date': '20110811',
+            'upload_date': '20151012',
-            'uploader_id': 'IGN',
         }
+    }, {
+        'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
+        'only_matching': True,
+    }, {
+        # Youtube embed
+        'url': 'https://me.ign.com/ar/ratchet-clank-rift-apart/144327/trailer/embed',
+        'only_matching': True,
+    }, {
+        # Twitter embed
+        'url': 'http://adria.ign.com/sherlock-season-4/9687/trailer/embed',
+        'only_matching': True,
+    }, {
+        # Vimeo embed
+        'url': 'https://kr.ign.com/bic-2018/3307/trailer/embed',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id = self._match_id(url)
-        result = super(OneUPIE, self)._real_extract(url)
+        req = HEADRequest(url.rsplit('/', 1)[0] + '/embed')
-        result['id'] = mobj.group('name_or_id')
+        url = self._request_webpage(req, video_id).geturl()
-        return result
+        ign_url = compat_parse_qs(
+            compat_urllib_parse_urlparse(url).query).get('url', [None])[0]
+        if ign_url:
+            return self.url_result(ign_url, IGNIE.ie_key())
+        return self.url_result(url)
 
 
-class PCMagIE(IGNIE):
+class IGNArticleIE(IGNBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?pcmag\.com/(?P<type>videos|article2)(/.+)?/(?P<name_or_id>.+)'
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:articles(?:/\d{4}/\d{2}/\d{2})?|(?:[a-z]{2}/)?feature/\d+)/(?P<id>[^/?&#]+)'
-    IE_NAME = 'pcmag'
+    _PAGE_TYPE = 'article'
-
-    _EMBED_RE = r'iframe\.setAttribute\("src",\s*__util.objToUrlString\("http://widgets\.ign\.com/video/embed/content\.html?[^"]*url=([^"]+)["&]'
-
     _TESTS = [{
-        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
+        'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
-        'md5': '212d6154fd0361a2781075f1febbe9ad',
         'info_dict': {
-            'id': 'ee10d774b508c9b8ec07e763b9125b91',
+            'id': '524497489e4e8ff5848ece34',
-            'ext': 'mp4',
+            'title': '100 Little Things in GTA 5 That Will Blow Your Mind',
-            'title': '010615_What\'s New Now: Is GoGo Snooping on Your Data?',
+        },
-            'description': 'md5:a7071ae64d2f68cc821c729d4ded6bb3',
+        'playlist': [
-            'timestamp': 1420571160,
+            {
-            'upload_date': '20150106',
+                'info_dict': {
-            'uploader_id': 'cozzipix@gmail.com',
+                    'id': '5ebbd138523268b93c9141af17bec937',
-        }
+                    'ext': 'mp4',
+                    'title': 'GTA 5 Video Review',
+                    'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
+                    'timestamp': 1379339880,
+                    'upload_date': '20130916',
+                },
+            },
+            {
+                'info_dict': {
+                    'id': '638672ee848ae4ff108df2a296418ee2',
+                    'ext': 'mp4',
+                    'title': '26 Twisted Moments from GTA 5 in Slow Motion',
+                    'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
+                    'timestamp': 1386878820,
+                    'upload_date': '20131212',
+                },
+            },
+        ],
+        'params': {
+            'playlist_items': '2-3',
+            'skip_download': True,
+        },
     }, {
-        'url': 'http://www.pcmag.com/article2/0,2817,2470156,00.asp',
+        'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
-        'md5': '94130c1ca07ba0adb6088350681f16c1',
         'info_dict': {
-            'id': '042e560ba94823d43afcb12ddf7142ca',
+            'id': '53ee806780a81ec46e0790f8',
-            'ext': 'mp4',
+            'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
-            'title': 'HTC\'s Weird New Re Camera - What\'s New Now',
+        },
-            'description': 'md5:53433c45df96d2ea5d0fda18be2ca908',
+        'playlist_count': 2,
-            'timestamp': 1412953920,
+    }, {
-            'upload_date': '20141010',
+        # videoId pattern
-            'uploader_id': 'chris_snyder@pcmag.com',
+        'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
-        }
+        'only_matching': True,
+    }, {
+        # Youtube embed
+        'url': 'https://www.ign.com/articles/2021-mvp-named-in-puppy-bowl-xvii',
+        'only_matching': True,
+    }, {
+        # IMDB embed
+        'url': 'https://www.ign.com/articles/2014/08/07/sons-of-anarchy-final-season-trailer',
+        'only_matching': True,
+    }, {
+        # Facebook embed
+        'url': 'https://www.ign.com/articles/2017/09/20/marvels-the-punisher-watch-the-new-trailer-for-the-netflix-series',
+        'only_matching': True,
+    }, {
+        # Brightcove embed
+        'url': 'https://www.ign.com/articles/2016/01/16/supergirl-goes-flying-with-martian-manhunter-in-new-clip',
+        'only_matching': True,
     }]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        article = self._call_api(display_id)
+
+        def entries():
+            media_url = try_get(article, lambda x: x['mediaRelations'][0]['media']['metadata']['url'])
+            if media_url:
+                yield self.url_result(media_url, IGNIE.ie_key())
+            for content in (article.get('content') or []):
+                for video_url in re.findall(r'(?:\[(?:ignvideo\s+url|youtube\s+clip_id)|<iframe[^>]+src)="([^"]+)"', content):
+                    yield self.url_result(video_url)
+
+        return self.playlist_result(
+            entries(), article.get('articleId'),
+            strip_or_none(try_get(article, lambda x: x['metadata']['headline'])))

youtube_dlc/extractor/ivi.py

@@ -163,7 +163,10 @@ class IviIE(InfoExtractor):
         for f in result.get('files', []):
             f_url = f.get('url')
             content_format = f.get('content_format')
-            if not f_url or '-MDRM-' in content_format or '-FPS-' in content_format:
+            if not f_url:
+                continue
+            if (not self._downloader.params.get('allow_unplayable_formats')
+                    and ('-MDRM-' in content_format or '-FPS-' in content_format)):
                 continue
             formats.append({
                 'url': f_url,

youtube_dlc/extractor/kaltura.py

@@ -309,7 +309,7 @@ class KalturaIE(InfoExtractor):
             if f.get('fileExt') == 'chun':
                 continue
             # DRM-protected video, cannot be decrypted
-            if f.get('fileExt') == 'wvm':
+            if not self._downloader.params.get('allow_unplayable_formats') and f.get('fileExt') == 'wvm':
                 continue
             if not f.get('fileExt'):
                 # QT indicates QuickTime; some videos have broken fileExt

youtube_dlc/extractor/limelight.py

@@ -96,7 +96,9 @@ class LimelightBaseIE(InfoExtractor):
         urls = []
         for stream in pc_item.get('streams', []):
             stream_url = stream.get('url')
-            if not stream_url or stream.get('drmProtected') or stream_url in urls:
+            if not stream_url or stream_url in urls:
+                continue
+            if not self._downloader.params.get('allow_unplayable_formats') and stream.get('drmProtected'):
                 continue
             urls.append(stream_url)
             ext = determine_ext(stream_url)

youtube_dlc/extractor/medialaan.py

@@ -2,268 +2,113 @@ from __future__ import unicode_literals
 
 import re
 
-from .gigya import GigyaBaseIE
+from .common import InfoExtractor
-
-from ..compat import compat_str
 from ..utils import (
+    extract_attributes,
     int_or_none,
-    parse_duration,
+    mimetype2ext,
-    try_get,
+    parse_iso8601,
-    unified_timestamp,
 )
 
 
-class MedialaanIE(GigyaBaseIE):
+class MedialaanIE(InfoExtractor):
     _VALID_URL = r'''(?x)
                     https?://
-                        (?:www\.|nieuws\.)?
                         (?:
-                            (?P<site_id>vtm|q2|vtmkzoom)\.be/
+                            (?:embed\.)?mychannels.video/embed/|
-                            (?:
+                            embed\.mychannels\.video/(?:s(?:dk|cript)/)?production/|
-                                video(?:/[^/]+/id/|/?\?.*?\baid=)|
+                            (?:www\.)?(?:
-                                (?:[^/]+/)*
+                                (?:
-                            )
+                                    7sur7|
+                                    demorgen|
+                                    hln|
+                                    joe|
+                                    qmusic
+                                )\.be|
+                                (?:
+                                    [abe]d|
+                                    bndestem|
+                                    destentor|
+                                    gelderlander|
+                                    pzc|
+                                    tubantia|
+                                    volkskrant
+                                )\.nl
+                            )/video/(?:[^/]+/)*[^/?&#]+~p
                         )
-                        (?P<id>[^/?#&]+)
+                        (?P<id>\d+)
                     '''
-    _NETRC_MACHINE = 'medialaan'
-    _APIKEY = '3_HZ0FtkMW_gOyKlqQzW5_0FHRC7Nd5XpXJZcDdXY4pk5eES2ZWmejRW5egwVm4ug-'
-    _SITE_TO_APP_ID = {
-        'vtm': 'vtm_watch',
-        'q2': 'q2',
-        'vtmkzoom': 'vtmkzoom',
-    }
     _TESTS = [{
-        # vod
+        'url': 'https://www.bndestem.nl/video/de-terugkeer-van-ally-de-aap-en-wie-vertrekt-er-nog-bij-nac~p193993',
-        'url': 'http://vtm.be/video/volledige-afleveringen/id/vtm_20170219_VM0678361_vtmwatch',
         'info_dict': {
-            'id': 'vtm_20170219_VM0678361_vtmwatch',
+            'id': '193993',
             'ext': 'mp4',
-            'title': 'Allemaal Chris afl. 6',
+            'title': 'De terugkeer van Ally de Aap en wie vertrekt er nog bij NAC?',
-            'description': 'md5:4be86427521e7b07e0adb0c9c554ddb2',
+            'timestamp': 1611663540,
-            'timestamp': 1487533280,
+            'upload_date': '20210126',
-            'upload_date': '20170219',
+            'duration': 238,
-            'duration': 2562,
-            'series': 'Allemaal Chris',
-            'season': 'Allemaal Chris',
-            'season_number': 1,
-            'season_id': '256936078124527',
-            'episode': 'Allemaal Chris afl. 6',
-            'episode_number': 6,
-            'episode_id': '256936078591527',
         },
         'params': {
             'skip_download': True,
         },
-        'skip': 'Requires account credentials',
     }, {
-        # clip
+        'url': 'https://www.gelderlander.nl/video/kanalen/degelderlander~c320/series/snel-nieuws~s984/noodbevel-in-doetinchem-politie-stuurt-mensen-centrum-uit~p194093',
-        'url': 'http://vtm.be/video?aid=168332',
-        'info_dict': {
-            'id': '168332',
-            'ext': 'mp4',
-            'title': '"Veronique liegt!"',
-            'description': 'md5:1385e2b743923afe54ba4adc38476155',
-            'timestamp': 1489002029,
-            'upload_date': '20170308',
-            'duration': 96,
-        },
-    }, {
-        # vod
-        'url': 'http://vtm.be/video/volledige-afleveringen/id/257107153551000',
         'only_matching': True,
     }, {
-        # vod
+        'url': 'https://embed.mychannels.video/sdk/production/193993?options=TFTFF_default',
-        'url': 'http://vtm.be/video?aid=163157',
         'only_matching': True,
     }, {
-        # vod
+        'url': 'https://embed.mychannels.video/script/production/193993',
-        'url': 'http://www.q2.be/video/volledige-afleveringen/id/2be_20170301_VM0684442_q2',
         'only_matching': True,
     }, {
-        # clip
+        'url': 'https://embed.mychannels.video/production/193993',
-        'url': 'http://vtmkzoom.be/k3-dansstudio/een-nieuw-seizoen-van-k3-dansstudio',
         'only_matching': True,
     }, {
-        # http/s redirect
+        'url': 'https://mychannels.video/embed/193993',
-        'url': 'https://vtmkzoom.be/video?aid=45724',
+        'only_matching': True,
-        'info_dict': {
-            'id': '257136373657000',
-            'ext': 'mp4',
-            'title': 'K3 Dansstudio Ushuaia afl.6',
-        },
-        'params': {
-            'skip_download': True,
-        },
-        'skip': 'Requires account credentials',
     }, {
-        # nieuws.vtm.be
+        'url': 'https://embed.mychannels.video/embed/193993',
-        'url': 'https://nieuws.vtm.be/stadion/stadion/genk-nog-moeilijk-programma',
         'only_matching': True,
     }]
 
-    def _real_initialize(self):
+    @staticmethod
-        self._logged_in = False
+    def _extract_urls(webpage):
-
+        entries = []
-    def _login(self):
+        for element in re.findall(r'(<div[^>]+data-mychannels-type="video"[^>]*>)', webpage):
-        username, password = self._get_login_info()
+            mychannels_id = extract_attributes(element).get('data-mychannels-id')
-        if username is None:
+            if mychannels_id:
-            self.raise_login_required()
+                entries.append('https://mychannels.video/embed/' + mychannels_id)
-
+        return entries
-        auth_data = {
-            'APIKey': self._APIKEY,
-            'sdk': 'js_6.1',
-            'format': 'json',
-            'loginID': username,
-            'password': password,
-        }
-
-        auth_info = self._gigya_login(auth_data)
-
-        self._uid = auth_info['UID']
-        self._uid_signature = auth_info['UIDSignature']
-        self._signature_timestamp = auth_info['signatureTimestamp']
-
-        self._logged_in = True
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        production_id = self._match_id(url)
-        video_id, site_id = mobj.group('id', 'site_id')
+        production = self._download_json(
+            'https://embed.mychannels.video/sdk/production/' + production_id,
+            production_id, query={'options': 'UUUU_default'})['productions'][0]
+        title = production['title']
 
-        webpage = self._download_webpage(url, video_id)
+        formats = []
-
+        for source in (production.get('sources') or []):
-        config = self._parse_json(
+            src = source.get('src')
-            self._search_regex(
+            if not src:
-                r'videoJSConfig\s*=\s*JSON\.parse\(\'({.+?})\'\);',
+                continue
-                webpage, 'config', default='{}'), video_id,
+            ext = mimetype2ext(source.get('type'))
-            transform_source=lambda s: s.replace(
+            if ext == 'm3u8':
-                '\\\\', '\\').replace(r'\"', '"').replace(r"\'", "'"))
+                formats.extend(self._extract_m3u8_formats(
-
+                    src, production_id, 'mp4', 'm3u8_native',
-        vod_id = config.get('vodId') or self._search_regex(
+                    m3u8_id='hls', fatal=False))
-            (r'\\"vodId\\"\s*:\s*\\"(.+?)\\"',
-             r'"vodId"\s*:\s*"(.+?)"',
-             r'<[^>]+id=["\']vod-(\d+)'),
-            webpage, 'video_id', default=None)
-
-        # clip, no authentication required
-        if not vod_id:
-            player = self._parse_json(
-                self._search_regex(
-                    r'vmmaplayer\(({.+?})\);', webpage, 'vmma player',
-                    default=''),
-                video_id, transform_source=lambda s: '[%s]' % s, fatal=False)
-            if player:
-                video = player[-1]
-                if video['videoUrl'] in ('http', 'https'):
-                    return self.url_result(video['url'], MedialaanIE.ie_key())
-                info = {
-                    'id': video_id,
-                    'url': video['videoUrl'],
-                    'title': video['title'],
-                    'thumbnail': video.get('imageUrl'),
-                    'timestamp': int_or_none(video.get('createdDate')),
-                    'duration': int_or_none(video.get('duration')),
-                }
             else:
-                info = self._parse_html5_media_entries(
+                formats.append({
-                    url, webpage, video_id, m3u8_id='hls')[0]
+                    'ext': ext,
-                info.update({
+                    'url': src,
-                    'id': video_id,
-                    'title': self._html_search_meta('description', webpage),
-                    'duration': parse_duration(self._html_search_meta('duration', webpage)),
                 })
-        # vod, authentication required
+        self._sort_formats(formats)
-        else:
-            if not self._logged_in:
-                self._login()
 
-            settings = self._parse_json(
+        return {
-                self._search_regex(
+            'id': production_id,
-                    r'jQuery\.extend\(Drupal\.settings\s*,\s*({.+?})\);',
+            'title': title,
-                    webpage, 'drupal settings', default='{}'),
+            'formats': formats,
-                video_id)
+            'thumbnail': production.get('posterUrl'),
-
+            'timestamp': parse_iso8601(production.get('publicationDate'), ' '),
-            def get(container, item):
+            'duration': int_or_none(production.get('duration')) or None,
-                return try_get(
+        }
-                    settings, lambda x: x[container][item],
-                    compat_str) or self._search_regex(
-                    r'"%s"\s*:\s*"([^"]+)' % item, webpage, item,
-                    default=None)
-
-            app_id = get('vod', 'app_id') or self._SITE_TO_APP_ID.get(site_id, 'vtm_watch')
-            sso = get('vod', 'gigyaDatabase') or 'vtm-sso'
-
-            data = self._download_json(
-                'http://vod.medialaan.io/api/1.0/item/%s/video' % vod_id,
-                video_id, query={
-                    'app_id': app_id,
-                    'user_network': sso,
-                    'UID': self._uid,
-                    'UIDSignature': self._uid_signature,
-                    'signatureTimestamp': self._signature_timestamp,
-                })
-
-            formats = self._extract_m3u8_formats(
-                data['response']['uri'], video_id, entry_protocol='m3u8_native',
-                ext='mp4', m3u8_id='hls')
-
-            self._sort_formats(formats)
-
-            info = {
-                'id': vod_id,
-                'formats': formats,
-            }
-
-            api_key = get('vod', 'apiKey')
-            channel = get('medialaanGigya', 'channel')
-
-            if api_key:
-                videos = self._download_json(
-                    'http://vod.medialaan.io/vod/v2/videos', video_id, fatal=False,
-                    query={
-                        'channels': channel,
-                        'ids': vod_id,
-                        'limit': 1,
-                        'apikey': api_key,
-                    })
-                if videos:
-                    video = try_get(
-                        videos, lambda x: x['response']['videos'][0], dict)
-                    if video:
-                        def get(container, item, expected_type=None):
-                            return try_get(
-                                video, lambda x: x[container][item], expected_type)
-
-                        def get_string(container, item):
-                            return get(container, item, compat_str)
-
-                        info.update({
-                            'series': get_string('program', 'title'),
-                            'season': get_string('season', 'title'),
-                            'season_number': int_or_none(get('season', 'number')),
-                            'season_id': get_string('season', 'id'),
-                            'episode': get_string('episode', 'title'),
-                            'episode_number': int_or_none(get('episode', 'number')),
-                            'episode_id': get_string('episode', 'id'),
-                            'duration': int_or_none(
-                                video.get('duration')) or int_or_none(
-                                video.get('durationMillis'), scale=1000),
-                            'title': get_string('episode', 'title'),
-                            'description': get_string('episode', 'text'),
-                            'timestamp': unified_timestamp(get_string(
-                                'publication', 'begin')),
-                        })
-
-            if not info.get('title'):
-                info['title'] = try_get(
-                    config, lambda x: x['videoConfig']['title'],
-                    compat_str) or self._html_search_regex(
-                    r'\\"title\\"\s*:\s*\\"(.+?)\\"', webpage, 'title',
-                    default=None) or self._og_search_title(webpage)
-
-        if not info.get('description'):
-            info['description'] = self._html_search_regex(
-                r'<div[^>]+class="field-item\s+even">\s*<p>(.+?)</p>',
-                webpage, 'description', default=None)
-
-        return info

youtube_dlc/extractor/niconico.py

@@ -1,25 +1,25 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import datetime
+import re
-import functools
 import json
-import math
+import datetime
 
 from .common import InfoExtractor
+from ..postprocessor.ffmpeg import FFmpegPostProcessor
 from ..compat import (
     compat_parse_qs,
     compat_urllib_parse_urlparse,
 )
 from ..utils import (
-    determine_ext,
     dict_get,
     ExtractorError,
-    float_or_none,
-    InAdvancePagedList,
     int_or_none,
+    float_or_none,
+    OnDemandPagedList,
     parse_duration,
     parse_iso8601,
+    PostProcessingError,
     remove_start,
     try_get,
     unified_timestamp,
@@ -191,37 +191,87 @@ class NiconicoIE(InfoExtractor):
             self._downloader.report_warning('unable to log in: bad username or password')
         return login_ok
 
-    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
+    def _get_heartbeat_info(self, info_dict):
-        def yesno(boolean):
-            return 'yes' if boolean else 'no'
 
-        session_api_data = api_data['video']['dmcInfo']['session_api']
+        video_id, video_src_id, audio_src_id = info_dict['url'].split(':')[1].split('/')
-        session_api_endpoint = session_api_data['urls'][0]
 
-        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
+        # Get video webpage for API data.
+        webpage, handle = self._download_webpage_handle(
+            'http://www.nicovideo.jp/watch/' + video_id, video_id)
+
+        api_data = self._parse_json(self._html_search_regex(
+            'data-api-data="([^"]+)"', webpage,
+            'API data', default='{}'), video_id)
+
+        session_api_data = try_get(api_data, lambda x: x['video']['dmcInfo']['session_api'])
+        session_api_endpoint = try_get(session_api_data, lambda x: x['urls'][0])
+
+        # ping
+        self._download_json(
+            'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
+            query={'t': try_get(api_data, lambda x: x['video']['dmcInfo']['tracking_id'])},
+            headers={
+                'Origin': 'https://www.nicovideo.jp',
+                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
+                'X-Frontend-Id': '6',
+                'X-Frontend-Version': '0'
+            })
+
+        yesno = lambda x: 'yes' if x else 'no'
+
+        # m3u8 (encryption)
+        if 'encryption' in try_get(api_data, lambda x: x['video']['dmcInfo']) or {}:
+            protocol = 'm3u8'
+            session_api_http_parameters = {
+                'parameters': {
+                    'hls_parameters': {
+                        'encryption': {
+                            'hls_encryption_v1': {
+                                'encrypted_key': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['encrypted_key']),
+                                'key_uri': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['key_uri'])
+                            }
+                        },
+                        'transfer_preset': '',
+                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
+                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                        'segment_duration': 6000
+                    }
+                }
+            }
+        # http
+        else:
+            protocol = 'http'
+            session_api_http_parameters = {
+                'parameters': {
+                    'http_output_download_parameters': {
+                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
+                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                    }
+                }
+            }
 
         session_response = self._download_json(
             session_api_endpoint['url'], video_id,
             query={'_format': 'json'},
             headers={'Content-Type': 'application/json'},
-            note='Downloading JSON metadata for %s' % format_id,
+            note='Downloading JSON metadata for %s' % info_dict['format_id'],
             data=json.dumps({
                 'session': {
                     'client_info': {
-                        'player_id': session_api_data['player_id'],
+                        'player_id': session_api_data.get('player_id'),
                     },
                     'content_auth': {
-                        'auth_type': session_api_data['auth_types'][session_api_data['protocols'][0]],
+                        'auth_type': try_get(session_api_data, lambda x: x['auth_types'][session_api_data['protocols'][0]]),
-                        'content_key_timeout': session_api_data['content_key_timeout'],
+                        'content_key_timeout': session_api_data.get('content_key_timeout'),
                         'service_id': 'nicovideo',
-                        'service_user_id': session_api_data['service_user_id']
+                        'service_user_id': session_api_data.get('service_user_id')
                     },
-                    'content_id': session_api_data['content_id'],
+                    'content_id': session_api_data.get('content_id'),
                     'content_src_id_sets': [{
                         'content_src_ids': [{
                             'src_id_to_mux': {
-                                'audio_src_ids': [audio_quality['id']],
+                                'audio_src_ids': [audio_src_id],
-                                'video_src_ids': [video_quality['id']],
+                                'video_src_ids': [video_src_id],
                             }
                         }]
                     }],
@@ -229,52 +279,78 @@ class NiconicoIE(InfoExtractor):
                     'content_uri': '',
                     'keep_method': {
                         'heartbeat': {
-                            'lifetime': session_api_data['heartbeat_lifetime']
+                            'lifetime': session_api_data.get('heartbeat_lifetime')
                         }
                     },
-                    'priority': session_api_data['priority'],
+                    'priority': session_api_data.get('priority'),
                     'protocol': {
                         'name': 'http',
                         'parameters': {
-                            'http_parameters': {
+                            'http_parameters': session_api_http_parameters
-                                'parameters': {
-                                    'http_output_download_parameters': {
-                                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
-                                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
-                                    }
-                                }
-                            }
                         }
                     },
-                    'recipe_id': session_api_data['recipe_id'],
+                    'recipe_id': session_api_data.get('recipe_id'),
                     'session_operation_auth': {
                         'session_operation_auth_by_signature': {
-                            'signature': session_api_data['signature'],
+                            'signature': session_api_data.get('signature'),
-                            'token': session_api_data['token'],
+                            'token': session_api_data.get('token'),
                         }
                     },
                     'timing_constraint': 'unlimited'
                 }
             }).encode())
 
-        resolution = video_quality.get('resolution', {})
+        info_dict['url'] = session_response['data']['session']['content_uri']
+        info_dict['protocol'] = protocol
+
+        # get heartbeat info
+        heartbeat_info_dict = {
+            'url': session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT',
+            'data': json.dumps(session_response['data']),
+            # interval, convert milliseconds to seconds, then halve to make a buffer.
+            'interval': float_or_none(session_api_data.get('heartbeat_lifetime'), scale=2000),
+        }
+
+        return info_dict, heartbeat_info_dict
+
+    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
+        def parse_format_id(id_code):
+            mobj = re.match(r'''(?x)
+                    (?:archive_)?
+                    (?:(?P<codec>[^_]+)_)?
+                    (?:(?P<br>[\d]+)kbps_)?
+                    (?:(?P<res>[\d+]+)p_)?
+                ''', '%s_' % id_code)
+            return mobj.groupdict() if mobj else {}
+
+        protocol = 'niconico_dmc'
+        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
+        vdict = parse_format_id(video_quality['id'])
+        adict = parse_format_id(audio_quality['id'])
+        resolution = video_quality.get('resolution', {'height': vdict.get('res')})
 
         return {
-            'url': session_response['data']['session']['content_uri'],
+            'url': '%s:%s/%s/%s' % (protocol, video_id, video_quality['id'], audio_quality['id']),
             'format_id': format_id,
             'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
-            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
+            'vcodec': vdict.get('codec'),
-            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
+            'acodec': adict.get('codec'),
-            'height': resolution.get('height'),
+            'vbr': float_or_none(video_quality.get('bitrate'), 1000) or float_or_none(vdict.get('br')),
-            'width': resolution.get('width'),
+            'abr': float_or_none(audio_quality.get('bitrate'), 1000) or float_or_none(adict.get('br')),
+            'height': int_or_none(resolution.get('height', vdict.get('res'))),
+            'width': int_or_none(resolution.get('width')),
+            'quality': -2 if 'low' in format_id else -1,  # Default quality value is -1
+            'protocol': protocol,
+            'http_headers': {
+                'Origin': 'https://www.nicovideo.jp',
+                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
+            }
         }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        # Get video webpage. We are not actually interested in it for normal
+        # Get video webpage for API data.
-        # cases, but need the cookies in order to be able to download the
-        # info webpage
         webpage, handle = self._download_webpage_handle(
             'http://www.nicovideo.jp/watch/' + video_id, video_id)
         if video_id.startswith('so'):
@@ -284,80 +360,134 @@ class NiconicoIE(InfoExtractor):
             'data-api-data="([^"]+)"', webpage,
             'API data', default='{}'), video_id)
 
-        def _format_id_from_url(video_url):
+        def get_video_info_web(items):
-            return 'economy' if video_real_url.endswith('low') else 'normal'
+            return dict_get(api_data['video'], items)
+
+        # Get video info
+        video_info_xml = self._download_xml(
+            'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
+            video_id, note='Downloading video info page')
+
+        def get_video_info_xml(items):
+            if not isinstance(items, list):
+                items = [items]
+            for item in items:
+                ret = xpath_text(video_info_xml, './/' + item)
+                if ret:
+                    return ret
+
+        if get_video_info_xml('error'):
+            error_code = get_video_info_xml('code')
+
+            if error_code == 'DELETED':
+                raise ExtractorError('The video has been deleted.',
+                                     expected=True)
+            elif error_code == 'NOT_FOUND':
+                raise ExtractorError('The video is not found.',
+                                     expected=True)
+            elif error_code == 'COMMUNITY':
+                self.to_screen('%s: The video is community members only.' % video_id)
+            else:
+                raise ExtractorError('%s reports error: %s' % (self.IE_NAME, error_code))
+
+        # Start extracting video formats
+        formats = []
+
+        # Get HTML5 videos info
+        try:
+            dmc_info = api_data['video']['dmcInfo']
+        except KeyError:
+            raise ExtractorError('The video can\'t downloaded.',
+                                 expected=True)
+
+        quality_info = dmc_info.get('quality')
+        for audio_quality in quality_info.get('audios') or {}:
+            for video_quality in quality_info.get('videos') or {}:
+                if not audio_quality.get('available') or not video_quality.get('available'):
+                    continue
+                formats.append(self._extract_format_for_quality(
+                    api_data, video_id, audio_quality, video_quality))
+
+        # Get flv/swf info
+        video_real_url = try_get(api_data, lambda x: x['video']['smileInfo']['url'])
+        is_economy = video_real_url.endswith('low')
+
+        if is_economy:
+            self.report_warning('Site is currently in economy mode! You will only have access to lower quality streams')
+
+        # Invoking ffprobe to determine resolution
+        pp = FFmpegPostProcessor(self._downloader)
+        cookies = self._get_cookies('https://nicovideo.jp').output(header='', sep='; path=/; domain=nicovideo.jp;\n')
+
+        self.to_screen('%s: %s' % (video_id, 'Checking smile format with ffprobe'))
 
         try:
-            video_real_url = api_data['video']['smileInfo']['url']
+            metadata = pp.get_metadata_object(video_real_url, ['-cookies', cookies])
-        except KeyError:  # Flash videos
+        except PostProcessingError as err:
-            # Get flv info
+            raise ExtractorError(err.msg, expected=True)
-            flv_info_webpage = self._download_webpage(
-                'http://flapi.nicovideo.jp/api/getflv/' + video_id + '?as3=1',
-                video_id, 'Downloading flv info')
 
-            flv_info = compat_parse_qs(flv_info_webpage)
+        v_stream = a_stream = {}
-            if 'url' not in flv_info:
-                if 'deleted' in flv_info:
-                    raise ExtractorError('The video has been deleted.',
-                                         expected=True)
-                elif 'closed' in flv_info:
-                    raise ExtractorError('Niconico videos now require logging in',
-                                         expected=True)
-                elif 'error' in flv_info:
-                    raise ExtractorError('%s reports error: %s' % (
-                        self.IE_NAME, flv_info['error'][0]), expected=True)
-                else:
-                    raise ExtractorError('Unable to find video URL')
 
-            video_info_xml = self._download_xml(
+        # Some complex swf files doesn't have video stream (e.g. nm4809023)
-                'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
+        for stream in metadata['streams']:
-                video_id, note='Downloading video info page')
+            if stream['codec_type'] == 'video':
+                v_stream = stream
+            elif stream['codec_type'] == 'audio':
+                a_stream = stream
 
-            def get_video_info(items):
+        # Community restricted videos seem to have issues with the thumb API not returning anything at all
-                if not isinstance(items, list):
+        filesize = int(
-                    items = [items]
+            (get_video_info_xml('size_high') if not is_economy else get_video_info_xml('size_low'))
-                for item in items:
+            or metadata['format']['size']
-                    ret = xpath_text(video_info_xml, './/' + item)
+        )
-                    if ret:
+        extension = (
-                        return ret
+            get_video_info_xml('movie_type')
+            or 'mp4' if 'mp4' in metadata['format']['format_name'] else metadata['format']['format_name']
+        )
 
-            video_real_url = flv_info['url'][0]
+        # 'creation_time' tag on video stream of re-encoded SMILEVIDEO mp4 files are '1970-01-01T00:00:00.000000Z'.
+        timestamp = (
+            parse_iso8601(get_video_info_web('first_retrieve'))
+            or unified_timestamp(get_video_info_web('postedDateTime'))
+        )
+        metadata_timestamp = (
+            parse_iso8601(try_get(v_stream, lambda x: x['tags']['creation_time']))
+            or timestamp if extension != 'mp4' else 0
+        )
 
-            extension = get_video_info('movie_type')
+        # According to compconf, smile videos from pre-2017 are always better quality than their DMC counterparts
-            if not extension:
+        smile_threshold_timestamp = parse_iso8601('2016-12-08T00:00:00+09:00')
-                extension = determine_ext(video_real_url)
 
-            formats = [{
+        is_source = timestamp < smile_threshold_timestamp or metadata_timestamp > 0
+
+        # If movie file size is unstable, old server movie is not source movie.
+        if filesize > 1:
+            formats.append({
                 'url': video_real_url,
+                'format_id': 'smile' if not is_economy else 'smile_low',
+                'format_note': 'SMILEVIDEO source' if not is_economy else 'SMILEVIDEO low quality',
                 'ext': extension,
-                'format_id': _format_id_from_url(video_real_url),
+                'container': extension,
-            }]
+                'vcodec': v_stream.get('codec_name'),
-        else:
+                'acodec': a_stream.get('codec_name'),
-            formats = []
+                # Some complex swf files doesn't have total bit rate metadata (e.g. nm6049209)
+                'tbr': int_or_none(metadata['format'].get('bit_rate'), scale=1000),
+                'vbr': int_or_none(v_stream.get('bit_rate'), scale=1000),
+                'abr': int_or_none(a_stream.get('bit_rate'), scale=1000),
+                'height': int_or_none(v_stream.get('height')),
+                'width': int_or_none(v_stream.get('width')),
+                'source_preference': 5 if not is_economy else -2,
+                'quality': 5 if is_source and not is_economy else None,
+                'filesize': filesize
+            })
 
-            dmc_info = api_data['video'].get('dmcInfo')
+        if len(formats) == 0:
-            if dmc_info:  # "New" HTML5 videos
+            raise ExtractorError('Unable to find video info.')
-                quality_info = dmc_info['quality']
-                for audio_quality in quality_info['audios']:
-                    for video_quality in quality_info['videos']:
-                        if not audio_quality['available'] or not video_quality['available']:
-                            continue
-                        formats.append(self._extract_format_for_quality(
-                            api_data, video_id, audio_quality, video_quality))
 
-                self._sort_formats(formats)
+        self._sort_formats(formats)
-            else:  # "Old" HTML5 videos
-                formats = [{
-                    'url': video_real_url,
-                    'ext': 'mp4',
-                    'format_id': _format_id_from_url(video_real_url),
-                }]
-
-            def get_video_info(items):
-                return dict_get(api_data['video'], items)
 
         # Start extracting information
-        title = get_video_info('title')
+        title = get_video_info_web('originalTitle')
         if not title:
             title = self._og_search_title(webpage, default=None)
         if not title:
@@ -372,14 +502,13 @@ class NiconicoIE(InfoExtractor):
         video_detail = watch_api_data.get('videoDetail', {})
 
         thumbnail = (
-            get_video_info(['thumbnail_url', 'thumbnailURL'])
+            self._html_search_regex(r'<meta property="og:image" content="([^"]+)">', webpage, 'thumbnail data', default=None)
+            or get_video_info_web(['thumbnail_url', 'largeThumbnailURL', 'thumbnailURL'])
             or self._html_search_meta('image', webpage, 'thumbnail', default=None)
             or video_detail.get('thumbnail'))
 
-        description = get_video_info('description')
+        description = get_video_info_web('description')
 
-        timestamp = (parse_iso8601(get_video_info('first_retrieve'))
-                     or unified_timestamp(get_video_info('postedDateTime')))
         if not timestamp:
             match = self._html_search_meta('datePublished', webpage, 'date published', default=None)
             if match:
@@ -389,7 +518,7 @@ class NiconicoIE(InfoExtractor):
                 video_detail['postedAt'].replace('/', '-'),
                 delimiter=' ', timezone=datetime.timedelta(hours=9))
 
-        view_count = int_or_none(get_video_info(['view_counter', 'viewCount']))
+        view_count = int_or_none(get_video_info_web(['view_counter', 'viewCount']))
         if not view_count:
             match = self._html_search_regex(
                 r'>Views: <strong[^>]*>([^<]+)</strong>',
@@ -398,7 +527,7 @@ class NiconicoIE(InfoExtractor):
                 view_count = int_or_none(match.replace(',', ''))
         view_count = view_count or video_detail.get('viewCount')
 
-        comment_count = (int_or_none(get_video_info('comment_num'))
+        comment_count = (int_or_none(get_video_info_web('comment_num'))
                          or video_detail.get('commentCount')
                          or try_get(api_data, lambda x: x['thread']['commentCount']))
         if not comment_count:
@@ -409,19 +538,19 @@ class NiconicoIE(InfoExtractor):
                 comment_count = int_or_none(match.replace(',', ''))
 
         duration = (parse_duration(
-            get_video_info('length')
+            get_video_info_web('length')
             or self._html_search_meta(
                 'video:duration', webpage, 'video duration', default=None))
             or video_detail.get('length')
-            or get_video_info('duration'))
+            or get_video_info_web('duration'))
 
-        webpage_url = get_video_info('watch_url') or url
+        webpage_url = get_video_info_web('watch_url') or url
 
         # Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
         # in the JSON, which will cause None to be returned instead of {}.
         owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
-        uploader_id = get_video_info(['ch_id', 'user_id']) or owner.get('id')
+        uploader_id = get_video_info_web(['ch_id', 'user_id']) or owner.get('id')
-        uploader = get_video_info(['ch_name', 'user_nickname']) or owner.get('nickname')
+        uploader = get_video_info_web(['ch_name', 'user_nickname']) or owner.get('nickname')
 
         return {
             'id': video_id,
@@ -456,60 +585,45 @@ class NiconicoPlaylistIE(InfoExtractor):
         'url': 'https://www.nicovideo.jp/user/805442/mylist/27411728',
         'only_matching': True,
     }]
-    _PAGE_SIZE = 100
-
-    def _call_api(self, list_id, resource, query):
-        return self._download_json(
-            'https://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
-            'Downloading %s JSON metatdata' % resource, query=query,
-            headers={'X-Frontend-Id': 6})['data']['mylist']
-
-    def _parse_owner(self, item):
-        owner = item.get('owner') or {}
-        if owner:
-            return {
-                'uploader': owner.get('name'),
-                'uploader_id': owner.get('id'),
-            }
-        return {}
-
-    def _fetch_page(self, list_id, page):
-        page += 1
-        items = self._call_api(list_id, 'page %d' % page, {
-            'page': page,
-            'pageSize': self._PAGE_SIZE,
-        })['items']
-        for item in items:
-            video = item.get('video') or {}
-            video_id = video.get('id')
-            if not video_id:
-                continue
-            count = video.get('count') or {}
-            get_count = lambda x: int_or_none(count.get(x))
-            info = {
-                '_type': 'url',
-                'id': video_id,
-                'title': video.get('title'),
-                'url': 'https://www.nicovideo.jp/watch/' + video_id,
-                'description': video.get('shortDescription'),
-                'duration': int_or_none(video.get('duration')),
-                'view_count': get_count('view'),
-                'comment_count': get_count('comment'),
-                'ie_key': NiconicoIE.ie_key(),
-            }
-            info.update(self._parse_owner(video))
-            yield info
 
     def _real_extract(self, url):
         list_id = self._match_id(url)
-        mylist = self._call_api(list_id, 'list', {
+        webpage = self._download_webpage(url, list_id)
-            'pageSize': 1,
+
-        })
+        header = self._parse_json(self._html_search_regex(
-        entries = InAdvancePagedList(
+            r'data-common-header="([^"]+)"', webpage,
-            functools.partial(self._fetch_page, list_id),
+            'webpage header'), list_id)
-            math.ceil(mylist['totalItemCount'] / self._PAGE_SIZE),
+        frontendId = header.get('initConfig').get('frontendId')
-            self._PAGE_SIZE)
+        frontendVersion = header.get('initConfig').get('frontendVersion')
-        result = self.playlist_result(
+
-            entries, list_id, mylist.get('name'), mylist.get('description'))
+        def get_page_data(pagenum, pagesize):
-        result.update(self._parse_owner(mylist))
+            return self._download_json(
-        return result
+                'http://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
+                query={'page': 1 + pagenum, 'pageSize': pagesize},
+                headers={
+                    'X-Frontend-Id': frontendId,
+                    'X-Frontend-Version': frontendVersion,
+                }).get('data').get('mylist')
+
+        data = get_page_data(0, 1)
+        title = data.get('name')
+        description = data.get('description')
+        uploader = data.get('owner').get('name')
+        uploader_id = data.get('owner').get('id')
+
+        def pagefunc(pagenum):
+            data = get_page_data(pagenum, 25)
+            return ({
+                '_type': 'url',
+                'url': 'http://www.nicovideo.jp/watch/' + item.get('watchId'),
+            } for item in data.get('items'))
+
+        return {
+            '_type': 'playlist',
+            'id': list_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': uploader_id,
+            'entries': OnDemandPagedList(pagefunc, 25),
+        }

youtube_dlc/extractor/ninecninemedia.py

@@ -36,7 +36,8 @@ class NineCNineMediaIE(InfoExtractor):
                 '$include': '[HasClosedCaptions]',
             })
 
-        if try_get(content_package, lambda x: x['Constraints']['Security']['Type']):
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and try_get(content_package, lambda x: x['Constraints']['Security']['Type'])):
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         manifest_base_url = content_package_url + 'manifest.'

youtube_dlc/extractor/ninenow.py

@@ -66,7 +66,7 @@ class NineNowIE(InfoExtractor):
 
         video_data = common_data['video']
 
-        if video_data.get('drm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drm'):
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         brightcove_id = video_data.get('brightcoveId') or 'ref:' + video_data['referenceId']

youtube_dlc/extractor/npo.py

@@ -246,7 +246,7 @@ class NPOIE(NPOBaseIE):
                 })
 
         if not formats:
-            if drm:
+            if not self._downloader.params.get('allow_unplayable_formats') and drm:
                 raise ExtractorError('This video is DRM protected.', expected=True)
             return
 

youtube_dlc/extractor/pornhub.py

@@ -22,11 +22,15 @@ from ..utils import (
     orderedSet,
     remove_quotes,
     str_to_int,
+    update_url_query,
+    urlencode_postdata,
     url_or_none,
 )
 
 
 class PornHubBaseIE(InfoExtractor):
+    _NETRC_MACHINE = 'pornhub'
+
     def _download_webpage_handle(self, *args, **kwargs):
         def dl(*args, **kwargs):
             return super(PornHubBaseIE, self)._download_webpage_handle(*args, **kwargs)
@@ -52,6 +56,66 @@ class PornHubBaseIE(InfoExtractor):
 
         return webpage, urlh
 
+    def _real_initialize(self):
+        self._logged_in = False
+
+    def _login(self, host):
+        if self._logged_in:
+            return
+
+        site = host.split('.')[0]
+
+        # Both sites pornhub and pornhubpremium have separate accounts
+        # so there should be an option to provide credentials for both.
+        # At the same time some videos are available under the same video id
+        # on both sites so that we have to identify them as the same video.
+        # For that purpose we have to keep both in the same extractor
+        # but under different netrc machines.
+        username, password = self._get_login_info(netrc_machine=site)
+        if username is None:
+            return
+
+        login_url = 'https://www.%s/%slogin' % (host, 'premium/' if 'premium' in host else '')
+        login_page = self._download_webpage(
+            login_url, None, 'Downloading %s login page' % site)
+
+        def is_logged(webpage):
+            return any(re.search(p, webpage) for p in (
+                r'class=["\']signOut',
+                r'>Sign\s+[Oo]ut\s*<'))
+
+        if is_logged(login_page):
+            self._logged_in = True
+            return
+
+        login_form = self._hidden_inputs(login_page)
+
+        login_form.update({
+            'username': username,
+            'password': password,
+        })
+
+        response = self._download_json(
+            'https://www.%s/front/authenticate' % host, None,
+            'Logging in to %s' % site,
+            data=urlencode_postdata(login_form),
+            headers={
+                'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
+                'Referer': login_url,
+                'X-Requested-With': 'XMLHttpRequest',
+            })
+
+        if response.get('success') == '1':
+            self._logged_in = True
+            return
+
+        message = response.get('message')
+        if message is not None:
+            raise ExtractorError(
+                'Unable to login: %s' % message, expected=True)
+
+        raise ExtractorError('Unable to log in')
+
 
 class PornHubIE(PornHubBaseIE):
     IE_DESC = 'PornHub and Thumbzilla'
@@ -163,12 +227,20 @@ class PornHubIE(PornHubBaseIE):
     }, {
         'url': 'https://www.pornhubpremium.com/view_video.php?viewkey=ph5e4acdae54a82',
         'only_matching': True,
+    }, {
+        # Some videos are available with the same id on both premium
+        # and non-premium sites (e.g. this and the following test)
+        'url': 'https://www.pornhub.com/view_video.php?viewkey=ph5f75b0f4b18e3',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.pornhubpremium.com/view_video.php?viewkey=ph5f75b0f4b18e3',
+        'only_matching': True,
     }]
 
     @staticmethod
     def _extract_urls(webpage):
         return re.findall(
-            r'<iframe[^>]+?src=["\'](?P<url>(?:https?:)?//(?:www\.)?pornhub\.(?:com|net|org)/embed/[\da-z]+)',
+            r'<iframe[^>]+?src=["\'](?P<url>(?:https?:)?//(?:www\.)?pornhub(?:premium)?\.(?:com|net|org)/embed/[\da-z]+)',
             webpage)
 
     def _extract_count(self, pattern, webpage, name):
@@ -180,12 +252,7 @@ class PornHubIE(PornHubBaseIE):
         host = mobj.group('host') or 'pornhub.com'
         video_id = mobj.group('id')
 
-        if 'premium' in host:
+        self._login(host)
-            if not self._downloader.params.get('cookiefile'):
-                raise ExtractorError(
-                    'PornHub Premium requires authentication.'
-                    ' You may want to use --cookies.',
-                    expected=True)
 
         self._set_cookie(host, 'age_verified', '1')
 
@@ -405,6 +472,10 @@ class PornHubIE(PornHubBaseIE):
 
 
 class PornHubPlaylistBaseIE(PornHubBaseIE):
+    def _extract_page(self, url):
+        return int_or_none(self._search_regex(
+            r'\bpage=(\d+)', url, 'page', default=None))
+
     def _extract_entries(self, webpage, host):
         # Only process container div with main playlist content skipping
         # drop-down menu that uses similar pattern for videos (see
@@ -422,26 +493,6 @@ class PornHubPlaylistBaseIE(PornHubBaseIE):
                 container))
         ]
 
-    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        host = mobj.group('host')
-        playlist_id = mobj.group('id')
-
-        webpage = self._download_webpage(url, playlist_id)
-
-        entries = self._extract_entries(webpage, host)
-
-        playlist = self._parse_json(
-            self._search_regex(
-                r'(?:playlistObject|PLAYLIST_VIEW)\s*=\s*({.+?});', webpage,
-                'playlist', default='{}'),
-            playlist_id, fatal=False)
-        title = playlist.get('title') or self._search_regex(
-            r'>Videos\s+in\s+(.+?)\s+[Pp]laylist<', webpage, 'title', fatal=False)
-
-        return self.playlist_result(
-            entries, playlist_id, title, playlist.get('description'))
-
 
 class PornHubUserIE(PornHubPlaylistBaseIE):
     _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net|org))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/?#&]+))(?:[?#&]|/(?!videos)|$)'
@@ -463,14 +514,27 @@ class PornHubUserIE(PornHubPlaylistBaseIE):
     }, {
         'url': 'https://www.pornhub.com/model/zoe_ph?abc=1',
         'only_matching': True,
+    }, {
+        # Unavailable via /videos page, but available with direct pagination
+        # on pornstar page (see [1]), requires premium
+        # 1. https://github.com/ytdl-org/youtube-dl/issues/27853
+        'url': 'https://www.pornhubpremium.com/pornstar/sienna-west',
+        'only_matching': True,
+    }, {
+        # Same as before, multi page
+        'url': 'https://www.pornhubpremium.com/pornstar/lily-labeau',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
         user_id = mobj.group('id')
+        videos_url = '%s/videos' % mobj.group('url')
+        page = self._extract_page(url)
+        if page:
+            videos_url = update_url_query(videos_url, {'page': page})
         return self.url_result(
-            '%s/videos' % mobj.group('url'), ie=PornHubPagedVideoListIE.ie_key(),
+            videos_url, ie=PornHubPagedVideoListIE.ie_key(), video_id=user_id)
-            video_id=user_id)
 
 
 class PornHubPagedPlaylistBaseIE(PornHubPlaylistBaseIE):
@@ -483,32 +547,55 @@ class PornHubPagedPlaylistBaseIE(PornHubPlaylistBaseIE):
                 <button[^>]+\bid=["\']moreDataBtn
             ''', webpage) is not None
 
-    def _real_extract(self, url):
+    def _entries(self, url, host, item_id):
-        mobj = re.match(self._VALID_URL, url)
+        page = self._extract_page(url)
-        host = mobj.group('host')
-        item_id = mobj.group('id')
 
-        page = int_or_none(self._search_regex(
+        VIDEOS = '/videos'
-            r'\bpage=(\d+)', url, 'page', default=None))
 
-        entries = []
+        def download_page(base_url, num, fallback=False):
-        for page_num in (page, ) if page is not None else itertools.count(1):
+            note = 'Downloading page %d%s' % (num, ' (switch to fallback)' if fallback else '')
+            return self._download_webpage(
+                base_url, item_id, note, query={'page': num})
+
+        def is_404(e):
+            return isinstance(e.cause, compat_HTTPError) and e.cause.code == 404
+
+        base_url = url
+        has_page = page is not None
+        first_page = page if has_page else 1
+        for page_num in (first_page, ) if has_page else itertools.count(first_page):
             try:
-                webpage = self._download_webpage(
+                try:
-                    url, item_id, 'Downloading page %d' % page_num,
+                    webpage = download_page(base_url, page_num)
-                    query={'page': page_num})
+                except ExtractorError as e:
+                    # Some sources may not be available via /videos page,
+                    # trying to fallback to main page pagination (see [1])
+                    # 1. https://github.com/ytdl-org/youtube-dl/issues/27853
+                    if is_404(e) and page_num == first_page and VIDEOS in base_url:
+                        base_url = base_url.replace(VIDEOS, '')
+                        webpage = download_page(base_url, page_num, fallback=True)
+                    else:
+                        raise
             except ExtractorError as e:
-                if isinstance(e.cause, compat_HTTPError) and e.cause.code == 404:
+                if is_404(e) and page_num != first_page:
                     break
                 raise
             page_entries = self._extract_entries(webpage, host)
             if not page_entries:
                 break
-            entries.extend(page_entries)
+            for e in page_entries:
+                yield e
             if not self._has_more(webpage):
                 break
 
-        return self.playlist_result(orderedSet(entries), item_id)
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        host = mobj.group('host')
+        item_id = mobj.group('id')
+
+        self._login(host)
+
+        return self.playlist_result(self._entries(url, host, item_id), item_id)
 
 
 class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):

youtube_dlc/extractor/prosiebensat1.py

@@ -34,7 +34,7 @@ class ProSiebenSat1BaseIE(InfoExtractor):
                 'ids': clip_id,
             })[0]
 
-        if video.get('is_protected') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('is_protected') is True:
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         formats = []

youtube_dlc/extractor/rtbf.py

@@ -125,7 +125,7 @@ class RTBFIE(InfoExtractor):
                 })
 
         mpd_url = data.get('urlDash')
-        if not data.get('drm') and mpd_url:
+        if mpd_url and (self._downloader.params.get('allow_unplayable_formats') or not data.get('drm')):
             formats.extend(self._extract_mpd_formats(
                 mpd_url, media_id, mpd_id='dash', fatal=False))
 

youtube_dlc/extractor/rumble.py

@@ -1,6 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import re
+
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
@@ -28,6 +30,14 @@ class RumbleEmbedIE(InfoExtractor):
         'only_matching': True,
     }]
 
+    @staticmethod
+    def _extract_urls(webpage):
+        return [
+            mobj.group('url')
+            for mobj in re.finditer(
+                r'(?:<(?:script|iframe)[^>]+\bsrc=|["\']embedUrl["\']\s*:\s*)["\'](?P<url>%s)' % RumbleEmbedIE._VALID_URL,
+                webpage)]
+
     def _real_extract(self, url):
         video_id = self._match_id(url)
         video = self._download_json(

youtube_dlc/extractor/ruutu.py

@@ -200,8 +200,8 @@ class RuutuIE(InfoExtractor):
                 return node.get('value')
 
         if not formats:
-            drm = xpath_text(video_xml, './Clip/DRM', default=None)
+            if (not self._downloader.params.get('allow_unplayable_formats')
-            if drm:
+                    and xpath_text(video_xml, './Clip/DRM', default=None)):
                 raise ExtractorError('This video is DRM protected.', expected=True)
             ns_st_cds = pv('ns_st_cds')
             if ns_st_cds != 'free':

youtube_dlc/extractor/shahid.py

@@ -111,7 +111,7 @@ class ShahidIE(ShahidBaseIE):
         playout = self._call_api(
             'playout/url/' + video_id, video_id)['playout']
 
-        if playout.get('drm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and playout.get('drm'):
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         formats = self._extract_m3u8_formats(playout['url'], video_id, 'mp4')

youtube_dlc/extractor/sonyliv.py

@@ -75,7 +75,7 @@ class SonyLIVIE(InfoExtractor):
         video_id = self._match_id(url)
         content = self._call_api(
             '1.5', 'IN/CONTENT/VIDEOURL/VOD/' + video_id, video_id)
-        if content.get('isEncrypted'):
+        if not self._downloader.params.get('allow_unplayable_formats') and content.get('isEncrypted'):
             raise ExtractorError('This video is DRM protected.', expected=True)
         dash_url = content['videoURL']
         headers = {

youtube_dlc/extractor/svt.py

@@ -255,8 +255,10 @@ class SVTPlayIE(SVTPlayBaseIE):
             svt_id = self._search_regex(
                 (r'<video[^>]+data-video-id=["\']([\da-zA-Z-]+)',
                  r'["\']videoSvtId["\']\s*:\s*["\']([\da-zA-Z-]+)',
+                 r'["\']videoSvtId\\?["\']\s*:\s*\\?["\']([\da-zA-Z-]+)',
                  r'"content"\s*:\s*{.*?"id"\s*:\s*"([\da-zA-Z-]+)"',
-                 r'["\']svtId["\']\s*:\s*["\']([\da-zA-Z-]+)'),
+                 r'["\']svtId["\']\s*:\s*["\']([\da-zA-Z-]+)',
+                 r'["\']svtId\\?["\']\s*:\s*\\?["\']([\da-zA-Z-]+)'),
                 webpage, 'video id')
 
         info_dict = self._extract_by_video_id(svt_id, webpage)

youtube_dlc/extractor/toggle.py

@@ -154,7 +154,8 @@ class ToggleIE(InfoExtractor):
                 })
         if not formats:
             for meta in (info.get('Metas') or []):
-                if meta.get('Key') == 'Encryption' and meta.get('Value') == '1':
+                if (not self._downloader.params.get('allow_unplayable_formats')
+                        and meta.get('Key') == 'Encryption' and meta.get('Value') == '1'):
                     raise ExtractorError(
                         'This video is DRM protected.', expected=True)
             # Most likely because geo-blocked

youtube_dlc/extractor/toutv.py

@@ -74,7 +74,7 @@ class TouTvIE(RadioCanadaIE):
             })
         # IsDrm does not necessarily mean the video is DRM protected (see
         # https://github.com/ytdl-org/youtube-dl/issues/13994).
-        if metadata.get('IsDrm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and metadata.get('IsDrm'):
             self.report_warning('This video is probably DRM protected.', path)
         video_id = metadata['IdMedia']
         details = metadata['Details']

youtube_dlc/extractor/tv2.py

@@ -20,7 +20,7 @@ from ..utils import (
 
 class TV2IE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?tv2\.no/v/(?P<id>\d+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'http://www.tv2.no/v/916509/',
         'info_dict': {
             'id': '916509',
@@ -33,7 +33,7 @@ class TV2IE(InfoExtractor):
             'view_count': int,
             'categories': list,
         },
-    }
+    }]
     _API_DOMAIN = 'sumo.tv2.no'
     _PROTOCOLS = ('HDS', 'HLS', 'DASH')
     _GEO_COUNTRIES = ['NO']
@@ -42,6 +42,12 @@ class TV2IE(InfoExtractor):
         video_id = self._match_id(url)
         api_base = 'http://%s/api/web/asset/%s' % (self._API_DOMAIN, video_id)
 
+        asset = self._download_json(
+            api_base + '.json', video_id,
+            'Downloading metadata JSON')['asset']
+        title = asset.get('subtitle') or asset['title']
+        is_live = asset.get('live') is True
+
         formats = []
         format_urls = []
         for protocol in self._PROTOCOLS:
@@ -81,7 +87,8 @@ class TV2IE(InfoExtractor):
                 elif ext == 'm3u8':
                     if not data.get('drmProtected'):
                         formats.extend(self._extract_m3u8_formats(
-                            video_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                            video_url, video_id, 'mp4',
+                            'm3u8' if is_live else 'm3u8_native',
                             m3u8_id=format_id, fatal=False))
                 elif ext == 'mpd':
                     formats.extend(self._extract_mpd_formats(
@@ -99,11 +106,6 @@ class TV2IE(InfoExtractor):
             raise ExtractorError('This video is DRM protected.', expected=True)
         self._sort_formats(formats)
 
-        asset = self._download_json(
-            api_base + '.json', video_id,
-            'Downloading metadata JSON')['asset']
-        title = asset['title']
-
         thumbnails = [{
             'id': thumbnail.get('@type'),
             'url': thumbnail.get('url'),
@@ -112,7 +114,7 @@ class TV2IE(InfoExtractor):
         return {
             'id': video_id,
             'url': video_url,
-            'title': title,
+            'title': self._live_title(title) if is_live else title,
             'description': strip_or_none(asset.get('description')),
             'thumbnails': thumbnails,
             'timestamp': parse_iso8601(asset.get('createTime')),
@@ -120,6 +122,7 @@ class TV2IE(InfoExtractor):
             'view_count': int_or_none(asset.get('views')),
             'categories': asset.get('keywords', '').split(','),
             'formats': formats,
+            'is_live': is_live,
         }
 
 
@@ -168,13 +171,13 @@ class TV2ArticleIE(InfoExtractor):
 
 
 class KatsomoIE(TV2IE):
-    _VALID_URL = r'https?://(?:www\.)?(?:katsomo|mtv)\.fi/(?:#!/)?(?:[^/]+/[0-9a-z-]+-\d+/[0-9a-z-]+-|[^/]+/\d+/[^/]+/)(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?(?:katsomo|mtv(uutiset)?)\.fi/(?:sarja/[0-9a-z-]+-\d+/[0-9a-z-]+-|(?:#!/)?jakso/(?:\d+/[^/]+/)?|video/prog)(?P<id>\d+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'https://www.mtv.fi/sarja/mtv-uutiset-live-33001002003/lahden-pelicans-teki-kovan-ratkaisun-ville-nieminen-pihalle-1181321',
         'info_dict': {
             'id': '1181321',
             'ext': 'mp4',
-            'title': 'MTV Uutiset Live',
+            'title': 'Lahden Pelicans teki kovan ratkaisun – Ville Nieminen pihalle',
             'description': 'Päätöksen teki Pelicansin hallitus.',
             'timestamp': 1575116484,
             'upload_date': '20191130',
@@ -186,7 +189,60 @@ class KatsomoIE(TV2IE):
             # m3u8 download
             'skip_download': True,
         },
-    }
+    }, {
+        'url': 'http://www.katsomo.fi/#!/jakso/33001005/studio55-fi/658521/jukka-kuoppamaki-tekee-yha-lauluja-vaikka-lentokoneessa',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.mtvuutiset.fi/video/prog1311159',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.katsomo.fi/#!/jakso/1311159',
+        'only_matching': True,
+    }]
     _API_DOMAIN = 'api.katsomo.fi'
     _PROTOCOLS = ('HLS', 'MPD')
     _GEO_COUNTRIES = ['FI']
+
+
+class MTVUutisetArticleIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)mtvuutiset\.fi/artikkeli/[^/]+/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://www.mtvuutiset.fi/artikkeli/tallaisia-vaurioita-viking-amorellassa-on-useamman-osaston-alla-vetta/7931384',
+        'info_dict': {
+            'id': '1311159',
+            'ext': 'mp4',
+            'title': 'Viking Amorellan matkustajien evakuointi on alkanut – tältä operaatio näyttää laivalla',
+            'description': 'Viking Amorellan matkustajien evakuointi on alkanut – tältä operaatio näyttää laivalla',
+            'timestamp': 1600608966,
+            'upload_date': '20200920',
+            'duration': 153.7886666,
+            'view_count': int,
+            'categories': list,
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        },
+    }, {
+        # multiple Youtube embeds
+        'url': 'https://www.mtvuutiset.fi/artikkeli/50-vuotta-subarun-vastaiskua/6070962',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        article_id = self._match_id(url)
+        article = self._download_json(
+            'http://api.mtvuutiset.fi/mtvuutiset/api/json/' + article_id,
+            article_id)
+
+        def entries():
+            for video in (article.get('videos') or []):
+                video_type = video.get('videotype')
+                video_url = video.get('url')
+                if not (video_url and video_type in ('katsomo', 'youtube')):
+                    continue
+                yield self.url_result(
+                    video_url, video_type.capitalize(), video.get('video_id'))
+
+        return self.playlist_result(
+            entries(), article_id, article.get('title'), article.get('description'))

youtube_dlc/extractor/tv4.py

@@ -17,7 +17,7 @@ class TV4IE(InfoExtractor):
             tv4\.se/(?:[^/]+)/klipp/(?:.*)-|
             tv4play\.se/
             (?:
-                (?:program|barn)/(?:[^/]+/|(?:[^\?]+)\?video_id=)|
+                (?:program|barn)/(?:(?:[^/]+/){1,2}|(?:[^\?]+)\?video_id=)|
                 iframe/video/|
                 film/|
                 sport/|
@@ -65,6 +65,10 @@ class TV4IE(InfoExtractor):
         {
             'url': 'http://www.tv4play.se/program/farang/3922081',
             'only_matching': True,
+        },
+        {
+            'url': 'https://www.tv4play.se/program/nyheterna/avsnitt/13315940',
+            'only_matching': True,
         }
     ]
 

youtube_dlc/extractor/tvnow.py

@@ -69,7 +69,7 @@ class TVNowBaseIE(InfoExtractor):
             if formats:
                 break
         else:
-            if info.get('isDrm'):
+            if not self._downloader.params.get('allow_unplayable_formats') and info.get('isDrm'):
                 raise ExtractorError(
                     'Video %s is DRM protected' % video_id, expected=True)
             if info.get('geoblocked'):

youtube_dlc/extractor/urplay.py

@@ -42,8 +42,8 @@ class URPlayIE(InfoExtractor):
         url = url.replace('skola.se/Produkter', 'play.se/program')
         webpage = self._download_webpage(url, video_id)
         urplayer_data = self._parse_json(self._html_search_regex(
-            r'data-react-class="components/Player/Player"[^>]+data-react-props="({.+?})"',
+            r'data-react-class="routes/Product/components/ProgramContainer/ProgramContainer"[^>]+data-react-props="({.+?})"',
-            webpage, 'urplayer data'), video_id)['currentProduct']
+            webpage, 'urplayer data'), video_id)['accessibleEpisodes'][0]
         episode = urplayer_data['title']
 
         host = self._download_json('http://streaming-loadbalancer.ur.se/loadbalancer.json', video_id)['redirect']

youtube_dlc/extractor/vidio.py

@@ -4,7 +4,13 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..utils import int_or_none
+from ..utils import (
+    int_or_none,
+    parse_iso8601,
+    str_or_none,
+    strip_or_none,
+    try_get,
+)
 
 
 class VidioIE(InfoExtractor):
@@ -21,57 +27,63 @@ class VidioIE(InfoExtractor):
             'thumbnail': r're:^https?://.*\.jpg$',
             'duration': 149,
             'like_count': int,
+            'uploader': 'TWELVE Pic',
+            'timestamp': 1444902800,
+            'upload_date': '20151015',
+            'uploader_id': 'twelvepictures',
+            'channel': 'Cover Music Video',
+            'channel_id': '280236',
+            'view_count': int,
+            'dislike_count': int,
+            'comment_count': int,
+            'tags': 'count:4',
         },
     }, {
         'url': 'https://www.vidio.com/watch/77949-south-korea-test-fires-missile-that-can-strike-all-of-the-north',
         'only_matching': True,
     }]
 
+    def _real_initialize(self):
+        self._api_key = self._download_json(
+            'https://www.vidio.com/auth', None, data=b'')['api_key']
+
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id, display_id = re.match(self._VALID_URL, url).groups()
-        video_id, display_id = mobj.group('id', 'display_id')
+        data = self._download_json(
+            'https://api.vidio.com/videos/' + video_id, display_id, headers={
+                'Content-Type': 'application/vnd.api+json',
+                'X-API-KEY': self._api_key,
+            })
+        video = data['videos'][0]
+        title = video['title'].strip()
 
-        webpage = self._download_webpage(url, display_id)
-
-        title = self._og_search_title(webpage)
-
-        m3u8_url, duration, thumbnail = [None] * 3
-
-        clips = self._parse_json(
-            self._html_search_regex(
-                r'data-json-clips\s*=\s*(["\'])(?P<data>\[.+?\])\1',
-                webpage, 'video data', default='[]', group='data'),
-            display_id, fatal=False)
-        if clips:
-            clip = clips[0]
-            m3u8_url = clip.get('sources', [{}])[0].get('file')
-            duration = clip.get('clip_duration')
-            thumbnail = clip.get('image')
-
-        m3u8_url = m3u8_url or self._search_regex(
-            r'data(?:-vjs)?-clip-hls-url=(["\'])(?P<url>(?:(?!\1).)+)\1',
-            webpage, 'hls url', group='url')
         formats = self._extract_m3u8_formats(
-            m3u8_url, display_id, 'mp4', entry_protocol='m3u8_native')
+            data['clips'][0]['hls_url'], display_id, 'mp4', 'm3u8_native')
         self._sort_formats(formats)
 
-        duration = int_or_none(duration or self._search_regex(
+        get_first = lambda x: try_get(data, lambda y: y[x + 's'][0], dict) or {}
-            r'data-video-duration=(["\'])(?P<duration>\d+)\1', webpage,
+        channel = get_first('channel')
-            'duration', fatal=False, group='duration'))
+        user = get_first('user')
-        thumbnail = thumbnail or self._og_search_thumbnail(webpage)
+        username = user.get('username')
-
+        get_count = lambda x: int_or_none(video.get('total_' + x))
-        like_count = int_or_none(self._search_regex(
-            (r'<span[^>]+data-comment-vote-count=["\'](\d+)',
-             r'<span[^>]+class=["\'].*?\blike(?:__|-)count\b.*?["\'][^>]*>\s*(\d+)'),
-            webpage, 'like count', fatal=False))
 
         return {
             'id': video_id,
             'display_id': display_id,
             'title': title,
-            'description': self._og_search_description(webpage),
+            'description': strip_or_none(video.get('description')),
-            'thumbnail': thumbnail,
+            'thumbnail': video.get('image_url_medium'),
-            'duration': duration,
+            'duration': int_or_none(video.get('duration')),
-            'like_count': like_count,
+            'like_count': get_count('likes'),
             'formats': formats,
+            'uploader': user.get('name'),
+            'timestamp': parse_iso8601(video.get('created_at')),
+            'uploader_id': username,
+            'uploader_url': 'https://www.vidio.com/@' + username if username else None,
+            'channel': channel.get('name'),
+            'channel_id': str_or_none(channel.get('id')),
+            'view_count': get_count('view_count'),
+            'dislike_count': get_count('dislikes'),
+            'comment_count': get_count('comments'),
+            'tags': video.get('tag_list'),
         }

youtube_dlc/extractor/viki.py

@@ -315,7 +315,7 @@ class VikiIE(VikiBaseIE):
                 # Despite CODECS metadata in m3u8 all video-only formats
                 # are actually video+audio
                 for f in m3u8_formats:
-                    if '_drm/index_' in f['url']:
+                    if not self._downloader.params.get('allow_unplayable_formats') and '_drm/index_' in f['url']:
                         continue
                     if f.get('acodec') == 'none' and f.get('vcodec') != 'none':
                         f['acodec'] = None

youtube_dlc/extractor/vlive.py

@@ -125,7 +125,7 @@ class VLiveIE(VLiveBaseIE):
                 headers={'Referer': 'https://www.vlive.tv/'}, query=query)
         except ExtractorError as e:
             if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                self.raise_login_required(json.loads(e.cause.read().decode())['message'])
+                self.raise_login_required(json.loads(e.cause.read().decode('utf-8'))['message'])
             raise
 
     def _real_extract(self, url):

youtube_dlc/extractor/vtm.py

@@ -0,0 +1,62 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    parse_iso8601,
+    try_get,
+)
+
+
+class VTMIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?vtm\.be/([^/?&#]+)~v(?P<id>[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})'
+    _TEST = {
+        'url': 'https://vtm.be/gast-vernielt-genkse-hotelkamer~ve7534523-279f-4b4d-a5c9-a33ffdbe23e1',
+        'md5': '37dca85fbc3a33f2de28ceb834b071f8',
+        'info_dict': {
+            'id': '192445',
+            'ext': 'mp4',
+            'title': 'Gast vernielt Genkse hotelkamer',
+            'timestamp': 1611060180,
+            'upload_date': '20210119',
+            'duration': 74,
+            # TODO: fix url _type result processing
+            # 'series': 'Op Interventie',
+        }
+    }
+
+    def _real_extract(self, url):
+        uuid = self._match_id(url)
+        video = self._download_json(
+            'https://omc4vm23offuhaxx6hekxtzspi.appsync-api.eu-west-1.amazonaws.com/graphql',
+            uuid, query={
+                'query': '''{
+  getComponent(type: Video, uuid: "%s") {
+    ... on Video {
+      description
+      duration
+      myChannelsVideo
+      program {
+        title
+      }
+      publishedAt
+      title
+    }
+  }
+}''' % uuid,
+            }, headers={
+                'x-api-key': 'da2-lz2cab4tfnah3mve6wiye4n77e',
+            })['data']['getComponent']
+
+        return {
+            '_type': 'url',
+            'id': uuid,
+            'title': video.get('title'),
+            'url': 'http://mychannels.video/embed/%d' % video['myChannelsVideo'],
+            'description': video.get('description'),
+            'timestamp': parse_iso8601(video.get('publishedAt')),
+            'duration': int_or_none(video.get('duration')),
+            'series': try_get(video, lambda x: x['program']['title']),
+            'ie_key': 'Medialaan',
+        }

youtube_dlc/extractor/vvvvid.py

@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from .youtube import YoutubeIE
 from ..utils import (
     ExtractorError,
     int_or_none,
@@ -47,6 +48,22 @@ class VVVVIDIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
+    }, {
+        # video_type == 'video/youtube'
+        'url': 'https://www.vvvvid.it/show/404/one-punch-man/406/486683/trailer',
+        'md5': '33e0edfba720ad73a8782157fdebc648',
+        'info_dict': {
+            'id': 'RzmFKUDOUgw',
+            'ext': 'mp4',
+            'title': 'Trailer',
+            'upload_date': '20150906',
+            'description': 'md5:a5e802558d35247fee285875328c0b80',
+            'uploader_id': 'BandaiVisual',
+            'uploader': 'BANDAI NAMCO Arts Channel',
+        },
+        'params': {
+            'skip_download': True,
+        },
     }, {
         'url': 'https://www.vvvvid.it/show/434/perche-dovrei-guardarlo-di-dario-moccia/437/489048',
         'only_matching': True
@@ -154,12 +171,13 @@ class VVVVIDIE(InfoExtractor):
                     if season_number:
                         info['season_number'] = int(season_number)
 
-        for quality in ('_sd', ''):
+        video_type = video_data.get('video_type')
+        is_youtube = False
+        for quality in ('', '_sd'):
             embed_code = video_data.get('embed_info' + quality)
             if not embed_code:
                 continue
             embed_code = ds(embed_code)
-            video_type = video_data.get('video_type')
             if video_type in ('video/rcs', 'video/kenc'):
                 if video_type == 'video/kenc':
                     kenc = self._download_json(
@@ -172,19 +190,28 @@ class VVVVIDIE(InfoExtractor):
                     if kenc_message:
                         embed_code += '?' + ds(kenc_message)
                 formats.extend(self._extract_akamai_formats(embed_code, video_id))
+            elif video_type == 'video/youtube':
+                info.update({
+                    '_type': 'url_transparent',
+                    'ie_key': YoutubeIE.ie_key(),
+                    'url': embed_code,
+                })
+                is_youtube = True
+                break
             else:
                 formats.extend(self._extract_wowza_formats(
                     'http://sb.top-ix.org/videomg/_definst_/mp4:%s/playlist.m3u8' % embed_code, video_id))
             metadata_from_url(embed_code)
 
-        self._sort_formats(formats)
+        if not is_youtube:
+            self._sort_formats(formats)
+            info['formats'] = formats
 
         metadata_from_url(video_data.get('thumbnail'))
         info.update(self._extract_common_video_info(video_data))
         info.update({
             'id': video_id,
             'title': title,
-            'formats': formats,
             'duration': int_or_none(video_data.get('length')),
             'series': video_data.get('show_title'),
             'season_id': season_id,

youtube_dlc/extractor/wakanim.py

@@ -41,12 +41,13 @@ class WakanimIE(InfoExtractor):
         m3u8_url = urljoin(url, self._search_regex(
             r'file\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage, 'm3u8 url',
             group='url'))
-        # https://docs.microsoft.com/en-us/azure/media-services/previous/media-services-content-protection-overview#streaming-urls
+        if not self._downloader.params.get('allow_unplayable_formats'):
-        encryption = self._search_regex(
+            # https://docs.microsoft.com/en-us/azure/media-services/previous/media-services-content-protection-overview#streaming-urls
-            r'encryption%3D(c(?:enc|bc(?:s-aapl)?))',
+            encryption = self._search_regex(
-            m3u8_url, 'encryption', default=None)
+                r'encryption%3D(c(?:enc|bc(?:s-aapl)?))',
-        if encryption and encryption in ('cenc', 'cbcs-aapl'):
+                m3u8_url, 'encryption', default=None)
-            raise ExtractorError('This video is DRM protected.', expected=True)
+            if encryption in ('cenc', 'cbcs-aapl'):
+                raise ExtractorError('This video is DRM protected.', expected=True)
 
         formats = self._extract_m3u8_formats(
             m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',

youtube_dlc/extractor/xhamster.py

@@ -11,11 +11,14 @@ from ..utils import (
     dict_get,
     extract_attributes,
     ExtractorError,
+    float_or_none,
     int_or_none,
     parse_duration,
+    str_or_none,
     try_get,
     unified_strdate,
     url_or_none,
+    urljoin,
 )
 
 
@@ -146,36 +149,89 @@ class XHamsterIE(InfoExtractor):
             video = initials['videoModel']
             title = video['title']
             formats = []
-            for format_id, formats_dict in video['sources'].items():
+            format_urls = set()
+            format_sizes = {}
+            sources = try_get(video, lambda x: x['sources'], dict) or {}
+            for format_id, formats_dict in sources.items():
                 if not isinstance(formats_dict, dict):
                     continue
+                download_sources = try_get(sources, lambda x: x['download'], dict) or {}
+                for quality, format_dict in download_sources.items():
+                    if not isinstance(format_dict, dict):
+                        continue
+                    format_sizes[quality] = float_or_none(format_dict.get('size'))
                 for quality, format_item in formats_dict.items():
                     if format_id == 'download':
                         # Download link takes some time to be generated,
                         # skipping for now
                         continue
-                        if not isinstance(format_item, dict):
+                    format_url = format_item
-                            continue
-                        format_url = format_item.get('link')
-                        filesize = int_or_none(
-                            format_item.get('size'), invscale=1000000)
-                    else:
-                        format_url = format_item
-                        filesize = None
                     format_url = url_or_none(format_url)
-                    if not format_url:
+                    if not format_url or format_url in format_urls:
                         continue
+                    format_urls.add(format_url)
                     formats.append({
                         'format_id': '%s-%s' % (format_id, quality),
                         'url': format_url,
                         'ext': determine_ext(format_url, 'mp4'),
                         'height': get_height(quality),
-                        'filesize': filesize,
+                        'filesize': format_sizes.get(quality),
                         'http_headers': {
                             'Referer': urlh.geturl(),
                         },
                     })
-            self._sort_formats(formats)
+            xplayer_sources = try_get(
+                initials, lambda x: x['xplayerSettings']['sources'], dict)
+            if xplayer_sources:
+                hls_sources = xplayer_sources.get('hls')
+                if isinstance(hls_sources, dict):
+                    for hls_format_key in ('url', 'fallback'):
+                        hls_url = hls_sources.get(hls_format_key)
+                        if not hls_url:
+                            continue
+                        hls_url = urljoin(url, hls_url)
+                        if not hls_url or hls_url in format_urls:
+                            continue
+                        format_urls.add(hls_url)
+                        formats.extend(self._extract_m3u8_formats(
+                            hls_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                            m3u8_id='hls', fatal=False))
+                standard_sources = xplayer_sources.get('standard')
+                if isinstance(standard_sources, dict):
+                    for format_id, formats_list in standard_sources.items():
+                        if not isinstance(formats_list, list):
+                            continue
+                        for standard_format in formats_list:
+                            if not isinstance(standard_format, dict):
+                                continue
+                            for standard_format_key in ('url', 'fallback'):
+                                standard_url = standard_format.get(standard_format_key)
+                                if not standard_url:
+                                    continue
+                                standard_url = urljoin(url, standard_url)
+                                if not standard_url or standard_url in format_urls:
+                                    continue
+                                format_urls.add(standard_url)
+                                ext = determine_ext(standard_url, 'mp4')
+                                if ext == 'm3u8':
+                                    formats.extend(self._extract_m3u8_formats(
+                                        standard_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                                        m3u8_id='hls', fatal=False))
+                                    continue
+                                quality = (str_or_none(standard_format.get('quality'))
+                                           or str_or_none(standard_format.get('label'))
+                                           or '')
+                                formats.append({
+                                    'format_id': '%s-%s' % (format_id, quality),
+                                    'url': standard_url,
+                                    'ext': ext,
+                                    'height': get_height(quality),
+                                    'filesize': format_sizes.get(quality),
+                                    'http_headers': {
+                                        'Referer': standard_url,
+                                    },
+                                })
+            self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
 
             categories_list = video.get('categories')
             if isinstance(categories_list, list):

youtube_dlc/extractor/zype.py

@@ -87,11 +87,16 @@ class ZypeIE(InfoExtractor):
                 r'(["\'])(?P<url>(?:(?!\1).)+\.m3u8(?:(?!\1).)*)\1',
                 body, 'm3u8 url', group='url', default=None)
             if not m3u8_url:
-                source = self._parse_json(self._search_regex(
+                source = self._search_regex(
-                    r'(?s)sources\s*:\s*\[\s*({.+?})\s*\]', body,
+                    r'(?s)sources\s*:\s*\[\s*({.+?})\s*\]', body, 'source')
-                    'source'), video_id, js_to_json)
+
-                if source.get('integration') == 'verizon-media':
+                def get_attr(key):
-                    m3u8_url = 'https://content.uplynk.com/%s.m3u8' % source['id']
+                    return self._search_regex(
+                        r'\b%s\s*:\s*([\'"])(?P<val>(?:(?!\1).)+)\1' % key,
+                        source, key, group='val')
+
+                if get_attr('integration') == 'verizon-media':
+                    m3u8_url = 'https://content.uplynk.com/%s.m3u8' % get_attr('id')
             formats = self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls')
             text_tracks = self._search_regex(

youtube_dlc/options.py

@@ -16,15 +16,14 @@ from .compat import (
 from .utils import (
     expand_path,
     get_executable_path,
+    OUTTMPL_TYPES,
     preferredencoding,
+    REMUX_EXTENSIONS,
     write_string,
 )
 from .version import __version__
 
 
-_remux_formats = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
-
-
 def _hide_login_info(opts):
     PRIVATE_OPTS = set(['-p', '--password', '-u', '--username', '--video-password', '--ap-password', '--ap-username'])
     eqre = re.compile('^(?P<key>' + ('|'.join(re.escape(po) for po in PRIVATE_OPTS)) + ')=.+$')
@@ -154,7 +153,7 @@ def parseOpts(overrideArguments=None):
     general.add_option(
         '-U', '--update',
         action='store_true', dest='update_self',
-        help='[BROKEN] Update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)')
+        help='Update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)')
     general.add_option(
         '-i', '--ignore-errors', '--no-abort-on-error',
         action='store_true', dest='ignoreerrors', default=True,
@@ -520,6 +519,16 @@ def parseOpts(overrideArguments=None):
             'If a merge is required (e.g. bestvideo+bestaudio), '
             'output to given container format. One of mkv, mp4, ogg, webm, flv. '
             'Ignored if no merge is required'))
+    video_format.add_option(
+        '--allow-unplayable-formats',
+        action='store_true', dest='allow_unplayable_formats', default=False,
+        help=(
+            'Allow unplayable formats to be listed and downloaded. '
+            'All video postprocessing will also be turned off'))
+    video_format.add_option(
+        '--no-allow-unplayable-formats',
+        action='store_false', dest='allow_unplayable_formats',
+        help='Do not allow unplayable formats to be listed or downloaded (default)')
 
     subtitles = optparse.OptionGroup(parser, 'Subtitle Options')
     subtitles.add_option(
@@ -811,11 +820,13 @@ def parseOpts(overrideArguments=None):
     verbosity.add_option(
         '-C', '--call-home',
         dest='call_home', action='store_true', default=False,
-        help='[Broken] Contact the youtube-dlc server for debugging')
+        # help='[Broken] Contact the youtube-dlc server for debugging')
+        help=optparse.SUPPRESS_HELP)
     verbosity.add_option(
         '--no-call-home',
         dest='call_home', action='store_false',
-        help='Do not contact the youtube-dlc server for debugging (default)')
+        # help='Do not contact the youtube-dlc server for debugging (default)')
+        help=optparse.SUPPRESS_HELP)
 
     filesystem = optparse.OptionGroup(parser, 'Filesystem Options')
     filesystem.add_option(
@@ -831,19 +842,23 @@ def parseOpts(overrideArguments=None):
         metavar='TYPE:PATH', dest='paths', default={}, type='str',
         action='callback', callback=_dict_from_multiple_values_options_callback,
         callback_kwargs={
-            'allowed_keys': 'home|temp|config|description|annotation|subtitle|infojson|thumbnail',
+            'allowed_keys': 'home|temp|%s' % '|'.join(OUTTMPL_TYPES.keys()),
             'process': lambda x: x.strip()},
         help=(
             'The paths where the files should be downloaded. '
-            'Specify the type of file and the path separated by a colon ":" '
+            'Specify the type of file and the path separated by a colon ":". '
-            '(supported: description|annotation|subtitle|infojson|thumbnail). '
+            'All the same types as --output are supported. '
             'Additionally, you can also provide "home" and "temp" paths. '
             'All intermediary files are first downloaded to the temp path and '
             'then the final files are moved over to the home path after download is finished. '
-            'Note that this option is ignored if --output is an absolute path'))
+            'This option is ignored if --output is an absolute path'))
     filesystem.add_option(
         '-o', '--output',
-        dest='outtmpl', metavar='TEMPLATE',
+        metavar='[TYPE:]TEMPLATE', dest='outtmpl', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': '|'.join(OUTTMPL_TYPES.keys()),
+            'default_key': 'default', 'process': lambda x: x.strip()},
         help='Output filename template, see "OUTPUT TEMPLATE" for details')
     filesystem.add_option(
         '--output-na-placeholder',
@@ -892,11 +907,13 @@ def parseOpts(overrideArguments=None):
     filesystem.add_option(
         '-c', '--continue',
         action='store_true', dest='continue_dl', default=True,
-        help='Resume partially downloaded files (default)')
+        help='Resume partially downloaded files/fragments (default)')
     filesystem.add_option(
         '--no-continue',
         action='store_false', dest='continue_dl',
-        help='Restart download of partially downloaded files from beginning')
+        help=(
+            'Do not resume partially downloaded fragments. '
+            'If the file is unfragmented, restart download of the entire file'))
     filesystem.add_option(
         '--part',
         action='store_false', dest='nopart', default=False,
@@ -924,7 +941,7 @@ def parseOpts(overrideArguments=None):
     filesystem.add_option(
         '--write-info-json',
         action='store_true', dest='writeinfojson', default=False,
-        help='Write video metadata to a .info.json file')
+        help='Write video metadata to a .info.json file (this may contain personal information)')
     filesystem.add_option(
         '--no-write-info-json',
         action='store_false', dest='writeinfojson',
@@ -937,6 +954,18 @@ def parseOpts(overrideArguments=None):
         '--no-write-annotations',
         action='store_false', dest='writeannotations',
         help='Do not write video annotations (default)')
+    filesystem.add_option(
+        '--write-playlist-metafiles',
+        action='store_true', dest='allow_playlist_files', default=True,
+        help=(
+            'Write playlist metadata in addition to the video metadata '
+            'when using --write-info-json, --write-description etc. (default)'))
+    filesystem.add_option(
+        '--no-write-playlist-metafiles',
+        action='store_false', dest='allow_playlist_files',
+        help=(
+            'Do not write playlist metadata when using '
+            '--write-info-json, --write-description etc.'))
     filesystem.add_option(
         '--get-comments',
         action='store_true', dest='getcomments', default=False,
@@ -1023,7 +1052,7 @@ def parseOpts(overrideArguments=None):
             'Remux the video into another container if necessary (currently supported: %s). '
             'If target container does not support the video/audio codec, remuxing will fail. '
             'You can specify multiple rules; eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
-            'and anything else to mkv.' % '|'.join(_remux_formats)))
+            'and anything else to mkv.' % '|'.join(REMUX_EXTENSIONS)))
     postproc.add_option(
         '--recode-video',
         metavar='FORMAT', dest='recodevideo', default=None,

youtube_dlc/postprocessor/embedthumbnail.py

@@ -10,9 +10,9 @@ import base64
 
 try:
     import mutagen
-    _has_mutagen = True
+    has_mutagen = True
 except ImportError:
-    _has_mutagen = False
+    has_mutagen = False
 
 from .ffmpeg import FFmpegPostProcessor
 
@@ -47,7 +47,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             self.to_screen('There aren\'t any thumbnails to embed')
             return [], info
 
-        thumbnail_filename = info['thumbnails'][-1]['filename']
+        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filename']
 
         if not os.path.exists(encodeFilename(thumbnail_filename)):
             self.report_warning('Skipping embedding the thumbnail because the file is missing.')
@@ -66,7 +66,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                 self.to_screen('Correcting extension to webp and escaping path for thumbnail "%s"' % thumbnail_filename)
                 thumbnail_webp_filename = replace_extension(thumbnail_filename, 'webp')
                 os.rename(encodeFilename(thumbnail_filename), encodeFilename(thumbnail_webp_filename))
-                thumbnail_filename = thumbnail_webp_filename
+                original_thumbnail = thumbnail_filename = thumbnail_webp_filename
                 thumbnail_ext = 'webp'
 
         # Convert unsupported thumbnail formats to JPEG (see #25687, #25717)
@@ -78,9 +78,9 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
             self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
             self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
-            os.remove(encodeFilename(escaped_thumbnail_filename))
             thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
             # Rename back to unescaped for further processing
+            os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
             os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
             thumbnail_filename = thumbnail_jpg_filename
             thumbnail_ext = 'jpg'
@@ -152,8 +152,10 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                     success = False
 
         elif info['ext'] in ['ogg', 'opus']:
-            if not _has_mutagen:
+            if not has_mutagen:
                 raise EmbedThumbnailPPError('module mutagen was not found. Please install using `python -m pip install mutagen`')
+            self.to_screen('Adding thumbnail to "%s"' % filename)
+
             size_regex = r',\s*(?P<w>\d+)x(?P<h>\d+)\s*[,\[]'
             size_result = self.run_ffmpeg(thumbnail_filename, thumbnail_filename, ['-hide_banner'])
             mobj = re.search(size_regex, size_result)
@@ -181,5 +183,11 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
         if success and temp_filename != filename:
             os.remove(encodeFilename(filename))
             os.rename(encodeFilename(temp_filename), encodeFilename(filename))
-        files_to_delete = [] if self._already_have_thumbnail else [thumbnail_filename]
+
+        files_to_delete = [thumbnail_filename]
+        if self._already_have_thumbnail:
+            info['__files_to_move'][original_thumbnail] = replace_extension(
+                info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:])
+            if original_thumbnail == thumbnail_filename:
+                files_to_delete = []
         return files_to_delete, info

youtube_dlc/postprocessor/ffmpeg.py

@@ -61,7 +61,7 @@ class FFmpegPostProcessor(PostProcessor):
 
     def check_version(self):
         if not self.available:
-            raise FFmpegPostProcessorError('ffmpeg not found. Please install one.')
+            raise FFmpegPostProcessorError('ffmpeg not found. Please install')
 
         required_version = '10-0' if self.basename == 'avconv' else '1.0'
         if is_outdated_version(
@@ -165,7 +165,7 @@ class FFmpegPostProcessor(PostProcessor):
 
     def get_audio_codec(self, path):
         if not self.probe_available and not self.available:
-            raise PostProcessingError('ffprobe and ffmpeg not found. Please install one.')
+            raise PostProcessingError('ffprobe and ffmpeg not found. Please install')
         try:
             if self.probe_available:
                 cmd = [
@@ -221,8 +221,7 @@ class FFmpegPostProcessor(PostProcessor):
 
         cmd += opts
         cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
-        if self._downloader.params.get('verbose', False):
+        self.write_debug('ffprobe command line: %s' % shell_quote(cmd))
-            self._downloader.to_screen('[debug] ffprobe command line: %s' % shell_quote(cmd))
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
         stdout, stderr = p.communicate()
         return json.loads(stdout.decode('utf-8', 'replace'))
@@ -261,7 +260,7 @@ class FFmpegPostProcessor(PostProcessor):
         stdout, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
             stderr = stderr.decode('utf-8', 'replace').strip()
-            if self._downloader.params.get('verbose', False):
+            if self.get_param('verbose', False):
                 self.report_error(stderr)
             raise FFmpegPostProcessorError(stderr.split('\n')[-1])
         self.try_utime(out_path, oldest_mtime, oldest_mtime)
@@ -281,6 +280,8 @@ class FFmpegPostProcessor(PostProcessor):
 
 
 class FFmpegExtractAudioPP(FFmpegPostProcessor):
+    COMMON_AUDIO_EXTENSIONS = ('wav', 'flac', 'm4a', 'aiff', 'mp3', 'ogg', 'mka', 'opus', 'wma')
+
     def __init__(self, downloader=None, preferredcodec=None, preferredquality=None, nopostoverwrites=False):
         FFmpegPostProcessor.__init__(self, downloader)
         if preferredcodec is None:
@@ -302,6 +303,10 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
 
     def run(self, information):
         path = information['filepath']
+        orig_ext = information['ext']
+
+        if self._preferredcodec == 'best' and orig_ext in self.COMMON_AUDIO_EXTENSIONS:
+            self.to_screen('Skipping audio extraction since the file is already in a common audio format')
 
         filecodec = self.get_audio_codec(path)
         if filecodec is None:
@@ -442,6 +447,10 @@ class FFmpegVideoConvertorPP(FFmpegPostProcessor):
 
 
 class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
+    def __init__(self, downloader=None, already_have_subtitle=False):
+        super(FFmpegEmbedSubtitlePP, self).__init__(downloader)
+        self._already_have_subtitle = already_have_subtitle
+
     def run(self, information):
         if information['ext'] not in ('mp4', 'webm', 'mkv'):
             self.to_screen('Subtitles can only be embedded in mp4, webm or mkv files')
@@ -501,7 +510,8 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
         os.remove(encodeFilename(filename))
         os.rename(encodeFilename(temp_filename), encodeFilename(filename))
 
-        return sub_filenames, information
+        files_to_delete = [] if self._already_have_subtitle else sub_filenames
+        return files_to_delete, information
 
 
 class FFmpegMetadataPP(FFmpegPostProcessor):
@@ -578,7 +588,7 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
                 in_filenames.append(metadata_filename)
                 options.extend(['-map_metadata', '1'])
 
-        if '__infojson_filepath' in info and info['ext'] in ('mkv', 'mka'):
+        if '__infojson_filename' in info and info['ext'] in ('mkv', 'mka'):
             old_stream, new_stream = self.get_stream_number(
                 filename, ('tags', 'mimetype'), 'application/json')
             if old_stream is not None:
@@ -586,7 +596,7 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
                 new_stream -= 1
 
             options.extend([
-                '-attach', info['__infojson_filepath'],
+                '-attach', info['__infojson_filename'],
                 '-metadata:s:%d' % new_stream, 'mimetype=application/json'
             ])
 

youtube_dlc/postprocessor/metadatafromfield.py

@@ -4,6 +4,7 @@ import re
 
 from .common import PostProcessor
 from ..compat import compat_str
+from ..utils import str_or_none
 
 
 class MetadataFromFieldPP(PostProcessor):
@@ -48,8 +49,12 @@ class MetadataFromFieldPP(PostProcessor):
             if field not in info:
                 self.report_warning('Video doesnot have a %s' % field)
                 continue
+            data_to_parse = str_or_none(info[field])
+            if data_to_parse is None:
+                self.report_warning('Field %s cannot be parsed' % field)
+                continue
             self.write_debug('Searching for r"%s" in %s' % (regex, field))
-            match = re.search(regex, info[field])
+            match = re.search(regex, data_to_parse)
             if match is None:
                 self.report_warning('Could not interpret video %s as "%s"' % (field, dictn['format']))
                 continue

youtube_dlc/postprocessor/movefilesafterdownload.py

@@ -4,11 +4,11 @@ import shutil
 
 from .common import PostProcessor
 from ..utils import (
+    decodeFilename,
     encodeFilename,
     make_dir,
     PostProcessingError,
 )
-from ..compat import compat_str
 
 
 class MoveFilesAfterDownloadPP(PostProcessor):
@@ -25,12 +25,13 @@ class MoveFilesAfterDownloadPP(PostProcessor):
         dl_path, dl_name = os.path.split(encodeFilename(info['filepath']))
         finaldir = info.get('__finaldir', dl_path)
         finalpath = os.path.join(finaldir, dl_name)
-        self.files_to_move[info['filepath']] = finalpath
+        self.files_to_move.update(info['__files_to_move'])
+        self.files_to_move[info['filepath']] = decodeFilename(finalpath)
 
+        make_newfilename = lambda old: decodeFilename(os.path.join(finaldir, os.path.basename(encodeFilename(old))))
         for oldfile, newfile in self.files_to_move.items():
             if not newfile:
-                newfile = os.path.join(finaldir, os.path.basename(encodeFilename(oldfile)))
+                newfile = make_newfilename(oldfile)
-            oldfile, newfile = compat_str(oldfile), compat_str(newfile)
             if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
                 continue
             if not os.path.exists(encodeFilename(oldfile)):
@@ -39,7 +40,7 @@ class MoveFilesAfterDownloadPP(PostProcessor):
             if os.path.exists(encodeFilename(newfile)):
                 if self.get_param('overwrites', True):
                     self.report_warning('Replacing existing file "%s"' % newfile)
-                    os.path.remove(encodeFilename(newfile))
+                    os.remove(encodeFilename(newfile))
                 else:
                     self.report_warning(
                         'Cannot move file "%s" out of temporary directory since "%s" already exists. '
@@ -49,5 +50,5 @@ class MoveFilesAfterDownloadPP(PostProcessor):
             self.to_screen('Moving file "%s" to "%s"' % (oldfile, newfile))
             shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
 
-        info['filepath'] = compat_str(finalpath)
+        info['filepath'] = finalpath
         return [], info

youtube_dlc/postprocessor/sponskrub.py

@@ -12,6 +12,7 @@ from ..utils import (
     str_or_none,
     PostProcessingError,
     prepend_extension,
+    process_communicate_or_kill,
 )
 
 
@@ -43,6 +44,10 @@ class SponSkrubPP(PostProcessor):
         if self.path is None:
             return [], information
 
+        filename = information['filepath']
+        if not os.path.exists(encodeFilename(filename)):  # no download
+            return [], information
+
         if information['extractor_key'].lower() != 'youtube':
             self.to_screen('Skipping sponskrub since it is not a YouTube video')
             return [], information
@@ -58,7 +63,6 @@ class SponSkrubPP(PostProcessor):
             if not information.get('__real_download', False):
                 self.report_warning('If sponskrub is run multiple times, unintended parts of the video could be cut out.')
 
-        filename = information['filepath']
         temp_filename = prepend_extension(filename, self._temp_ext)
         if os.path.exists(encodeFilename(temp_filename)):
             os.remove(encodeFilename(temp_filename))
@@ -72,8 +76,9 @@ class SponSkrubPP(PostProcessor):
         cmd = [encodeArgument(i) for i in cmd]
 
         self.write_debug('sponskrub command line: %s' % shell_quote(cmd))
-        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+        pipe = None if self.get_param('verbose') else subprocess.PIPE
-        stdout, stderr = p.communicate()
+        p = subprocess.Popen(cmd, stdout=pipe)
+        stdout = process_communicate_or_kill(p)[0]
 
         if p.returncode == 0:
             os.remove(encodeFilename(filename))
@@ -82,9 +87,7 @@ class SponSkrubPP(PostProcessor):
         elif p.returncode == 3:
             self.to_screen('No segments in the SponsorBlock database')
         else:
-            msg = stderr.decode('utf-8', 'replace').strip() or stdout.decode('utf-8', 'replace').strip()
+            msg = stdout.decode('utf-8', 'replace').strip() if stdout else ''
-            self.write_debug(msg, prefix=False)
+            msg = msg.split('\n')[0 if msg.lower().startswith('unrecognised') else -1]
-            line = 0 if msg[:12].lower() == 'unrecognised' else -1
-            msg = msg.split('\n')[line]
             raise PostProcessingError(msg if msg else 'sponskrub failed with error code %s' % p.returncode)
         return [], information

youtube_dlc/update.py

@@ -5,6 +5,7 @@ import json
 import traceback
 import hashlib
 import os
+import platform
 import subprocess
 import sys
 from zipimport import zipimporter
@@ -15,6 +16,7 @@ from .utils import encode_compat_str
 from .version import __version__
 
 
+'''  # Not signed
 def rsa_verify(message, signature, key):
     from hashlib import sha256
     assert isinstance(message, bytes)
@@ -27,82 +29,79 @@ def rsa_verify(message, signature, key):
         return False
     expected = b'0001' + (byte_size - len(asn1) // 2 - 3) * b'ff' + b'00' + asn1
     return expected == signature
+'''
 
 
 def update_self(to_screen, verbose, opener):
-    """Update the program file with the latest version from the repository"""
+    """
+    Update the program file with the latest version from the repository
+    Returns whether the program should terminate
+    """
 
-    return to_screen('Update is currently broken.\nVisit  https://github.com/pukkandan/yt-dlp/releases/latest  to get the latest version')
+    JSON_URL = 'https://api.github.com/repos/pukkandan/yt-dlp/releases/latest'
 
-    UPDATE_URL = 'https://blackjack4494.github.io//update/'
+    def calc_sha256sum(path):
-    VERSION_URL = UPDATE_URL + 'LATEST_VERSION'
-    JSON_URL = UPDATE_URL + 'versions.json'
-    UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537)
-
-    def sha256sum():
         h = hashlib.sha256()
         b = bytearray(128 * 1024)
         mv = memoryview(b)
-        with open(os.path.realpath(sys.executable), 'rb', buffering=0) as f:
+        with open(os.path.realpath(path), 'rb', buffering=0) as f:
             for n in iter(lambda: f.readinto(mv), 0):
                 h.update(mv[:n])
         return h.hexdigest()
 
-    to_screen('Current Build Hash %s' % sha256sum())
+    to_screen('Current Build Hash %s' % calc_sha256sum(sys.executable))
 
     if not isinstance(globals().get('__loader__'), zipimporter) and not hasattr(sys, 'frozen'):
-        to_screen('It looks like you installed youtube-dlc with a package manager, pip, setup.py or a tarball. Please use that to update.')
+        to_screen('It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update.')
-        return
-
-    # compiled file.exe can find itself by
-    # to_screen(os.path.basename(sys.executable))
-    # and path to py or exe
-    # to_screen(os.path.realpath(sys.executable))
-
-    # Check if there is a new version
-    try:
-        newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
-    except Exception:
-        if verbose:
-            to_screen(encode_compat_str(traceback.format_exc()))
-        to_screen('ERROR: can\'t find the current version. Please try again later.')
-        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
-        return
-    if newversion == __version__:
-        to_screen('youtube-dlc is up-to-date (' + __version__ + ')')
         return
 
     # Download and check versions info
     try:
-        versions_info = opener.open(JSON_URL).read().decode('utf-8')
+        version_info = opener.open(JSON_URL).read().decode('utf-8')
-        versions_info = json.loads(versions_info)
+        version_info = json.loads(version_info)
     except Exception:
         if verbose:
             to_screen(encode_compat_str(traceback.format_exc()))
         to_screen('ERROR: can\'t obtain versions info. Please try again later.')
-        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+        to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
         return
-    if 'signature' not in versions_info:
-        to_screen('ERROR: the versions file is not signed or corrupted. Aborting.')
-        return
-    signature = versions_info['signature']
-    del versions_info['signature']
-    if not rsa_verify(json.dumps(versions_info, sort_keys=True).encode('utf-8'), signature, UPDATES_RSA_KEY):
-        to_screen('ERROR: the versions file signature is invalid. Aborting.')
-        return
-
-    version_id = versions_info['latest']
 
     def version_tuple(version_str):
         return tuple(map(int, version_str.split('.')))
+
+    version_id = version_info['tag_name']
     if version_tuple(__version__) >= version_tuple(version_id):
-        to_screen('youtube-dlc is up to date (%s)' % __version__)
+        to_screen('yt-dlp is up to date (%s)' % __version__)
         return
 
     to_screen('Updating to version ' + version_id + ' ...')
-    version = versions_info['versions'][version_id]
 
-    print_notes(to_screen, versions_info['versions'])
+    version_labels = {
+        'zip_3': '',
+        'zip_2': '',
+        # 'zip_2': '_py2',
+        'exe_64': '.exe',
+        'exe_32': '_x86.exe',
+    }
+
+    def get_bin_info(bin_or_exe, version):
+        label = version_labels['%s_%s' % (bin_or_exe, version)]
+        return next(
+            (i for i in version_info['assets']
+                if i['name'] in ('yt-dlp%s' % label, 'youtube-dlc%s' % label)), {})
+
+    def get_sha256sum(bin_or_exe, version):
+        label = version_labels['%s_%s' % (bin_or_exe, version)]
+        urlh = next(
+            (i for i in version_info['assets']
+                if i['name'] in ('SHA2-256SUMS')), {}).get('browser_download_url')
+        if not urlh:
+            return None
+        hash_data = opener.open(urlh).read().decode('utf-8')
+        hashes = list(map(lambda x: x.split(':'), hash_data.splitlines()))
+        return next(
+            (i[1] for i in hashes
+                if i[0] in ('yt-dlp%s' % label, 'youtube-dlc%s' % label)), None)
 
     # sys.executable is set to the full pathname of the exe-file for py2exe
     # though symlinks are not followed so that we need to do this manually
@@ -113,7 +112,7 @@ def update_self(to_screen, verbose, opener):
         to_screen('ERROR: no write permissions on %s' % filename)
         return
 
-    # Py2EXE
+    # PyInstaller
     if hasattr(sys, 'frozen'):
         exe = filename
         directory = os.path.dirname(exe)
@@ -122,19 +121,20 @@ def update_self(to_screen, verbose, opener):
             return
 
         try:
-            urlh = opener.open(version['exe'][0])
+            arch = platform.architecture()[0][:2]
+            url = get_bin_info('exe', arch).get('browser_download_url')
+            if not url:
+                to_screen('ERROR: unable to fetch updates')
+                to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
+                return
+            urlh = opener.open(url)
             newcontent = urlh.read()
             urlh.close()
-        except (IOError, OSError):
+        except (IOError, OSError, StopIteration):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
             to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
-            return
-
-        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
-        if newcontent_hash != version['exe'][1]:
-            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
             return
 
         try:
@@ -146,20 +146,33 @@ def update_self(to_screen, verbose, opener):
             to_screen('ERROR: unable to write the new version')
             return
 
+        expected_sum = get_sha256sum('exe', arch)
+        if not expected_sum:
+            to_screen('WARNING: no hash information found for the release')
+        elif calc_sha256sum(exe + '.new') != expected_sum:
+            to_screen('ERROR: unable to verify the new executable')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
+            try:
+                os.remove(exe + '.new')
+            except OSError:
+                to_screen('ERROR: unable to remove corrupt download')
+            return
+
         try:
-            bat = os.path.join(directory, 'youtube-dlc-updater.bat')
+            bat = os.path.join(directory, 'yt-dlp-updater.cmd')
             with io.open(bat, 'w') as batfile:
                 batfile.write('''
-@echo off
+@(
-echo Waiting for file handle to be closed ...
+    echo.Waiting for file handle to be closed ...
-ping 127.0.0.1 -n 5 -w 1000 > NUL
+    ping 127.0.0.1 -n 5 -w 1000 > NUL
-move /Y "%s.new" "%s" > NUL
+    move /Y "%s.new" "%s" > NUL
-echo Updated youtube-dlc to version %s.
+    echo.Updated yt-dlp to version %s.
-start /b "" cmd /c del "%%~f0"&exit /b"
+)
-                \n''' % (exe, exe, version_id))
+@start /b "" cmd /c del "%%~f0"&exit /b
+                ''' % (exe, exe, version_id))
 
             subprocess.Popen([bat])  # Continues to run in the background
-            return  # Do not show premature success messages
+            return True  # Exit app
         except (IOError, OSError):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
@@ -169,33 +182,51 @@ start /b "" cmd /c del "%%~f0"&exit /b"
     # Zip unix package
     elif isinstance(globals().get('__loader__'), zipimporter):
         try:
-            urlh = opener.open(version['bin'][0])
+            py_ver = platform.python_version()[0]
+            url = get_bin_info('zip', py_ver).get('browser_download_url')
+            if not url:
+                to_screen('ERROR: unable to fetch updates')
+                to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
+                return
+            urlh = opener.open(url)
             newcontent = urlh.read()
             urlh.close()
-        except (IOError, OSError):
+        except (IOError, OSError, StopIteration):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
             to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
-            return
-
-        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
-        if newcontent_hash != version['bin'][1]:
-            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
             return
 
         try:
-            with open(filename, 'wb') as outf:
+            with open(filename + '.new', 'wb') as outf:
                 outf.write(newcontent)
         except (IOError, OSError):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
+            to_screen('ERROR: unable to write the new version')
+            return
+
+        expected_sum = get_sha256sum('zip', py_ver)
+        if expected_sum and calc_sha256sum(filename + '.new') != expected_sum:
+            to_screen('ERROR: unable to verify the new zip')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
+            try:
+                os.remove(filename + '.new')
+            except OSError:
+                to_screen('ERROR: unable to remove corrupt zip')
+            return
+
+        try:
+            os.rename(filename + '.new', filename)
+        except OSError:
             to_screen('ERROR: unable to overwrite current version')
             return
 
-    to_screen('Updated youtube-dlc. Restart youtube-dlc to use the new version.')
+    to_screen('Updated yt-dlp. Restart youtube-dlc to use the new version.')
 
 
+'''  # UNUSED
 def get_notes(versions, fromVersion):
     notes = []
     for v, vdata in sorted(versions.items()):
@@ -210,3 +241,4 @@ def print_notes(to_screen, versions, fromVersion=__version__):
         to_screen('PLEASE NOTE:')
         for note in notes:
             to_screen(note)
+'''

youtube_dlc/utils.py

@@ -50,6 +50,7 @@ from .compat import (
     compat_html_entities_html5,
     compat_http_client,
     compat_integer_types,
+    compat_numeric_types,
     compat_kwargs,
     compat_os_name,
     compat_parse_qs,
@@ -1714,6 +1715,8 @@ KNOWN_EXTENSIONS = (
     'wav',
     'f4f', 'f4m', 'm3u8', 'smil')
 
+REMUX_EXTENSIONS = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
+
 # needed for sanitizing filenames in restricted mode
 ACCENT_CHARS = dict(zip('ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ',
                         itertools.chain('AAAAAA', ['AE'], 'CEEEEIIIIDNOOOOOOO', ['OE'], 'UUUUUY', ['TH', 'ss'],
@@ -2606,6 +2609,8 @@ class YoutubeDLHandler(compat_urllib_request.HTTPHandler):
 
     @staticmethod
     def deflate(data):
+        if not data:
+            return data
         try:
             return zlib.decompress(data, -zlib.MAX_WBITS)
         except zlib.error:
@@ -3673,6 +3678,18 @@ def url_or_none(url):
     return url if re.match(r'^(?:(?:https?|rt(?:m(?:pt?[es]?|fp)|sp[su]?)|mms|ftps?):)?//', url) else None
 
 
+def strftime_or_none(timestamp, date_format, default=None):
+    datetime_object = None
+    try:
+        if isinstance(timestamp, compat_numeric_types):  # unix timestamp
+            datetime_object = datetime.datetime.utcfromtimestamp(timestamp)
+        elif isinstance(timestamp, compat_str):  # assume YYYYMMDD
+            datetime_object = datetime.datetime.strptime(timestamp, '%Y%m%d')
+        return datetime_object.strftime(date_format)
+    except (ValueError, TypeError, AttributeError):
+        return default
+
+
 def parse_duration(s):
     if not isinstance(s, compat_basestring):
         return None
@@ -4156,7 +4173,18 @@ def qualities(quality_ids):
     return q
 
 
-DEFAULT_OUTTMPL = '%(title)s [%(id)s].%(ext)s'
+DEFAULT_OUTTMPL = {
+    'default': '%(title)s [%(id)s].%(ext)s',
+}
+OUTTMPL_TYPES = {
+    'subtitle': None,
+    'thumbnail': None,
+    'description': 'description',
+    'annotation': 'annotations.xml',
+    'infojson': 'info.json',
+    'pl_description': 'description',
+    'pl_infojson': 'info.json',
+}
 
 
 def limit_length(s, length):
@@ -4666,9 +4694,7 @@ def cli_configuration_args(params, arg_name, key, default=[], exe=None):  # retu
         return default, False
     assert isinstance(argdict, dict)
 
-    assert isinstance(key, compat_str)
     key = key.lower()
-
     args = exe_args = None
     if exe is not None:
         assert isinstance(exe, compat_str)
@@ -5910,7 +5936,7 @@ def make_dir(path, to_screen=None):
 
 def get_executable_path():
     path = os.path.dirname(sys.argv[0])
-    if os.path.abspath(sys.argv[0]) != os.path.abspath(sys.executable):  # Not packaged
+    if os.path.basename(sys.argv[0]) == '__main__':  # Running from source
         path = os.path.join(path, '..')
     return os.path.abspath(path)
 

youtube_dlc/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.01.24'
+__version__ = '2021.02.09'