Release 2021.02.19

https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d

Except: [kakao] improve info extraction and detect geo restriction
https://github.com/ytdl-org/youtube-dl/commit/d8085580f63ad3b146a31712ff76cf41d5a4558a

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.02.09
+ [debug] yt-dlp version 2021.02.16
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.02.09
+ [debug] yt-dlp version 2021.02.16
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -55,8 +55,11 @@ jobs:
         asset_content_type: application/octet-stream
     - name: Get SHA2-256SUMS for youtube-dlc
       id: sha2_file
-      run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc)"
+      run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc | awk '{print $1}')"
     - name: Install dependencies for pypi
+      env:
+        PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+      if: "env.PYPI_TOKEN != ''"
       run: |
         python -m pip install --upgrade pip
         pip install setuptools wheel twine
@@ -64,6 +67,7 @@ jobs:
       env:
         TWINE_USERNAME: __token__
         TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      if: "env.TWINE_PASSWORD != ''"
       run: |
         rm -rf dist/*
         python setup.py sdist bdist_wheel
@@ -84,6 +88,8 @@ jobs:
       uses: actions/setup-python@v2
       with:
           python-version: '3.8'
+    - name: Upgrade pip and enable wheel support
+      run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
       run: pip install pyinstaller mutagen pycryptodome
     - name: Bump version
@@ -105,7 +111,7 @@ jobs:
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for youtube-dlc.exe
       id: sha2_file_win
-      run: echo "::set-output name=sha2_windows::$(certUtil -hashfile dist\youtube-dlc.exe SHA256 | findstr -v :)"
+      run: echo "::set-output name=sha2_windows::$((Get-FileHash dist\youtube-dlc.exe -Algorithm SHA256).Hash.ToLower())"
 
   build_windows32:
 
@@ -123,10 +129,10 @@ jobs:
       with:
           python-version: '3.4.4'
           architecture: 'x86'
-    - name: Install VS libs
+    - name: Upgrade pip and enable wheel support
-      run: choco install vcexpress2010
+      run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen pycryptodome
+      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
@@ -146,7 +152,7 @@ jobs:
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for youtube-dlc_x86.exe
       id: sha2_file_win32
-      run: echo "::set-output name=sha2_windows32::$(certUtil -hashfile dist\youtube-dlc_x86.exe SHA256 | findstr -v :)"
+      run: echo "::set-output name=sha2_windows32::$((Get-FileHash dist\youtube-dlc_x86.exe -Algorithm SHA256).Hash.ToLower())"
     - name: Make SHA2-256SUMS file
       env:
         SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
@@ -169,16 +175,3 @@ jobs:
         asset_path: ./SHA2-256SUMS
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
-
-#  update_version_badge:
-#    runs-on: ubuntu-latest
-#    needs: build_unix
-#    steps:
-#    - name: Create Version Badge
-#      uses: schneegans/dynamic-badges-action@v1.0.0
-#      with:
-#        auth: ${{ secrets.GIST_TOKEN }}
-#        gistID: c69cb23c3c5b3316248e52022790aa57
-#        filename: version.json
-#        label: Version
-#        message: ${{ needs.build_unix.outputs.ytdlc_version }}

.gitignore

@@ -36,6 +36,7 @@ README.txt
 # Binary
 youtube-dl
 youtube-dlc
+youtube-dlc.zip
 *.exe
 
 # Downloaded

CONTRIBUTORS

@@ -17,4 +17,7 @@ alxnull
 FelixFrog
 Zocker1999NET
 nao20010128nao
-shirt-dev
+shirt-dev
+kurumigi
+tsukumi
+bbepis

Changelog.md

@@ -8,7 +8,7 @@
 * Change "Merged with youtube-dl" version in Readme.md if needed
 * Commit to master as `Release <version>`
 * Push to origin/release - build task will now run
-* Update version.py using devscripts\update-version.py (be wary of timezones)
+* Update version.py using devscripts\update-version.py
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
 * Push to origin/master
@@ -17,25 +17,50 @@
 -->
 
 
+### 2021.02.19
+* **Merge youtube-dl:** Upto [commit/cf2dbec](https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d) (except kakao)
+* [viki] Fix extractor
+* [niconico] Extract `channel` and `channel_id` by [kurumigi](https://github.com/kurumigi)
+* [youtube] Multiple page support for hashtag URLs
+* [youtube] Add more invidious instances
+* [youtube] Fix comment extraction when comment text is empty
+* Option `--windows-filenames` to force use of windows compatible filenames
+* [ExtractAudio] Bugfix
+* Don't raise `parser.error` when exiting for update
+* [MoveFiles] Fix for when merger can't run
+* Changed `--trim-file-name` to `--trim-filenames` to be similar to related options
+* Format Sort improvements:
+    * Prefer `vp9.2` more than other `vp9` codecs
+    * Remove forced priority of `quality`
+    * Remove unnecessary `field_preference` and misuse of `preference` from extractors
+* Build improvements:
+    * Fix hash output by [shirt](https://github.com/shirt-dev)
+    * Lock python package versions for x86 and use `wheels` by [shirt](https://github.com/shirt-dev)
+    * Exclude `vcruntime140.dll` from UPX by [jbruchon](https://github.com/jbruchon)
+    * Set version number based on UTC time, not local time
+    * Publish on PyPi only if token is set
+* [documentation] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
+
+
 ### 2021.02.15
 * **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
-* [niconico] Improved extraction and support encrypted/SMILE movies
+* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
-* Fix HLS AES-128 with multiple keys in external downloaders
+* Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
-* [youtube_live_chat] Fix by using POST API
+* [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
 * [rumble] Add support for video page
-* Option to allow downloading unplayable video formats (`--allow-unplayable-formats`)
+* Option `--allow-unplayable-formats` to allow downloading unplayable video formats
 * [ExtractAudio] Don't re-encode when file is already in a common audio format
-* Change optional dependency to `pycryptodome`
 * [youtube] Fix search continuations
 * [youtube] Fix for new accounts
-* Improve build/updater:
+* Improve build/updater: by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
     * Fix SHA256 calculation in build and implement hash checking for updater
     * Exit immediately in windows once the update process starts
     * Fix updater for `x86.exe`
     * Updater looks for both `yt-dlp` and `youtube-dlc` in releases for future-proofing
-* Fix issue with unicode filenames in aria2c
+    * Change optional dependency to `pycryptodome`
+* Fix issue with unicode filenames in aria2c by [shirt](https://github.com/shirt-dev)
 * Fix `allow_playlist_files` not being correctly passed through
-* Fix for empty HTTP head requests
+* Fix for empty HTTP head requests by [shirt](https://github.com/shirt-dev)
 * Fix `get_executable_path` in UNIX
 * [sponskrub] Print ffmpeg output and errors to terminal
 * `__real_download` should be false when ffmpeg unavailable and no download
@@ -82,7 +107,7 @@
 
 
 ### 2021.01.29
-* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
     * Add `--get-comments`
     * [youtube] Extract comments
     * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE

README.md

@@ -177,7 +177,6 @@ Then simply type this
                                      containing directory
     --flat-playlist                  Do not extract the videos of a playlist,
                                      only list them
-    --flat-videos                    Do not resolve the video urls
     --no-flat-playlist               Extract the videos of a playlist
     --mark-watched                   Mark videos watched (YouTube only)
     --no-mark-watched                Do not mark videos watched
@@ -361,6 +360,12 @@ Then simply type this
                                      filenames
     --no-restrict-filenames          Allow Unicode characters, "&" and spaces in
                                      filenames (default)
+    --windows-filenames              Force filenames to be windows compatible
+    --no-windows-filenames           Make filenames windows compatible only if
+                                     using windows (default)
+    --trim-filenames LENGTH          Limit the filename length (excluding
+                                     extension) to the specified number of
+                                     characters
     -w, --no-overwrites              Do not overwrite any files
     --force-overwrites               Overwrite all video and metadata files.
                                      This option includes --no-continue
@@ -411,8 +416,6 @@ Then simply type this
                                      may change
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
-    --trim-file-name LENGTH          Limit the filename length (extension
-                                     excluded)
 
 ## Thumbnail Images:
     --write-thumbnail                Write thumbnail image to disk
@@ -516,8 +519,12 @@ Then simply type this
     --no-audio-multistreams          Only one audio stream is downloaded for
                                      each output file (default)
     --all-formats                    Download all available video formats
-    --prefer-free-formats            Prefer free video formats over non-free
+    --prefer-free-formats            Prefer video formats with free containers
-                                     formats of same quality
+                                     over non-free ones of same quality. Use
+                                     with "-S ext" to strictly prefer free
+                                     containers irrespective of quality
+    --no-prefer-free-formats         Don't give any special preference to free
+                                     containers (default)
     -F, --list-formats               List all available formats of requested
                                      videos
     --list-formats-as-table          Present the output of -F in tabular form
@@ -978,10 +985,10 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `hasaud`: Gives priority to formats that has a audio stream
  - `ie_pref`: The format preference as given by the extractor
  - `lang`: Language preference as given by the extractor
- - `quality`: The quality of the format. This is a metadata field available in some websites
+ - `quality`: The quality of the format as given by the extractor
  - `source`: Preference of the source as given by the extractor
  - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
  - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
  - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
@@ -1000,9 +1007,9 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `br`: Equivalent to using `tbr,vbr,abr`
  - `asr`: Audio sample rate in Hz
 
-Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
+Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec:vp9,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 

devscripts/update-version.py

@@ -13,7 +13,7 @@ old_version_list = old_version.split(".", 4)
 old_ver = '.'.join(old_version_list[:3])
 old_rev = old_version_list[3] if len(old_version_list) > 3 else ''
 
-ver = datetime.now().strftime("%Y.%m.%d")
+ver = datetime.utcnow().strftime("%Y.%m.%d")
 rev = str(int(old_rev or 0) + 1) if old_ver == ver else ''
 
 VERSION = '.'.join((ver, rev)) if rev else ver

docs/supportedsites.md

@@ -5,6 +5,7 @@
  - **23video**
  - **24video**
  - **3qsdn**: 3Q SDN
+ - **3sat**
  - **4tube**
  - **56.com**
  - **5min**
@@ -222,6 +223,7 @@
  - **curiositystream**
  - **curiositystream:collection**
  - **CWTV**
+ - **DagelijkseKost**: dagelijksekost.een.be
  - **DailyMail**
  - **dailymotion**
  - **dailymotion:playlist**
@@ -244,6 +246,7 @@
  - **DiscoveryGo**
  - **DiscoveryGoPlaylist**
  - **DiscoveryNetworksDe**
+ - **DiscoveryPlus**
  - **DiscoveryVR**
  - **Disney**
  - **dlive:stream**
@@ -370,6 +373,7 @@
  - **HentaiStigma**
  - **hetklokhuis**
  - **hgtv.com:show**
+ - **HGTVDe**
  - **HiDive**
  - **HistoricFilms**
  - **history:player**
@@ -854,6 +858,9 @@
  - **ShahidShow**
  - **Shared**: shared.sx
  - **ShowRoomLive**
+ - **simplecast**
+ - **simplecast:episode**
+ - **simplecast:podcast**
  - **Sina**
  - **sky.it**
  - **sky:news**
@@ -1231,6 +1238,7 @@
  - **ZattooLive**
  - **ZDF-3sat**
  - **ZDFChannel**
+ - **Zhihu**
  - **zingmp3**: mp3.zing.vn
  - **zoom**
  - **Zype**

pyinst.py

@@ -73,6 +73,7 @@ PyInstaller.__main__.run([
     '--exclude-module=ytdlp_plugins',
     '--hidden-import=mutagen',
     '--hidden-import=Crypto',
+    '--upx-exclude=vcruntime140.dll',
     'youtube_dlc/__main__.py',
 ])
 SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

test/test_youtube_lists.py

@@ -12,6 +12,7 @@ from test.helper import FakeYDL
 
 from youtube_dlc.extractor import (
     YoutubePlaylistIE,
+    YoutubeTabIE,
     YoutubeIE,
 )
 
@@ -57,14 +58,22 @@ class TestYoutubeLists(unittest.TestCase):
         entries = result['entries']
         self.assertEqual(len(entries), 100)
 
-    def test_youtube_flat_playlist_titles(self):
+    def test_youtube_flat_playlist_extraction(self):
         dl = FakeYDL()
         dl.params['extract_flat'] = True
-        ie = YoutubePlaylistIE(dl)
+        ie = YoutubeTabIE(dl)
-        result = ie.extract('https://www.youtube.com/playlist?list=PL-KKIb8rvtMSrAO9YFbeM6UQrAqoFTUWv')
+        result = ie.extract('https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc')
         self.assertIsPlaylist(result)
-        for entry in result['entries']:
+        entries = list(result['entries'])
-            self.assertTrue(entry.get('title'))
+        self.assertTrue(len(entries) == 1)
+        video = entries[0]
+        self.assertEqual(video['_type'], 'url_transparent')
+        self.assertEqual(video['ie_key'], 'Youtube')
+        self.assertEqual(video['id'], 'BaW_jenozKc')
+        self.assertEqual(video['url'], 'BaW_jenozKc')
+        self.assertEqual(video['title'], 'youtube-dl test video "\'/\\ä↭𝕐')
+        self.assertEqual(video['duration'], 10)
+        self.assertEqual(video['uploader'], 'Philipp Hagemeister')
 
 
 if __name__ == '__main__':

youtube_dlc/YoutubeDL.py

@@ -189,11 +189,15 @@ class YoutubeDL(object):
                        into a single file
     allow_multiple_audio_streams:   Allow multiple audio streams to be merged
                        into a single file
+    paths:             Dictionary of output paths. The allowed keys are 'home'
+                       'temp' and the keys of OUTTMPL_TYPES (in utils.py)
     outtmpl:           Dictionary of templates for output names. Allowed keys
-                       are 'default' and the keys of OUTTMPL_TYPES (in utils.py)
+                       are 'default' and the keys of OUTTMPL_TYPES (in utils.py).
+                       A string a also accepted for backward compatibility
     outtmpl_na_placeholder: Placeholder for unavailable meta fields.
     restrictfilenames: Do not allow "&" and spaces in file names
     trim_file_name:    Limit length of filename (extension excluded)
+    windowsfilenames:  Force the filenames to be windows compatible
     ignoreerrors:      Do not stop on download errors
                        (Default True when running youtube-dlc,
                        but False when directly accessing YoutubeDL class)
@@ -868,13 +872,6 @@ class YoutubeDL(object):
                     sub_ext = fn_groups[-2]
                 filename = '.'.join(filter(None, [fn_groups[0][:trim_file_name], sub_ext, ext]))
 
-            # Temporary fix for #4787
-            # 'Treat' all problem characters by passing filename through preferredencoding
-            # to workaround encoding issues with subprocess on python2 @ Windows
-            if sys.version_info < (3, 0) and sys.platform == 'win32':
-                filename = encodeFilename(filename, True).decode(preferredencoding())
-            filename = sanitize_path(filename)
-
             return filename
         except ValueError as err:
             self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
@@ -901,7 +898,14 @@ class YoutubeDL(object):
         assert isinstance(homepath, compat_str)
         subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
         assert isinstance(subdir, compat_str)
-        return sanitize_path(os.path.join(homepath, subdir, filename))
+        path = os.path.join(homepath, subdir, filename)
+
+        # Temporary fix for #4787
+        # 'Treat' all problem characters by passing filename through preferredencoding
+        # to workaround encoding issues with subprocess on python2 @ Windows
+        if sys.version_info < (3, 0) and sys.platform == 'win32':
+            path = encodeFilename(path, True).decode(preferredencoding())
+        return sanitize_path(path, force=self.params.get('windowsfilenames'))
 
     def _match_entry(self, info_dict, incomplete):
         """ Returns None if the file should be downloaded """
@@ -2360,6 +2364,9 @@ class YoutubeDL(object):
                             info_dict['__files_to_merge'] = downloaded
                             # Even if there were no downloads, it is being merged only now
                             info_dict['__real_download'] = True
+                        else:
+                            for file in downloaded:
+                                files_to_move[file] = None
                 else:
                     # Just a single file
                     dl_filename = existing_file(full_filename, temp_filename)
@@ -2713,8 +2720,6 @@ class YoutubeDL(object):
                 if f.get('preference') is None or f['preference'] >= -1000]
             header_line = ['format code', 'extension', 'resolution', 'note']
 
-        # if len(formats) > 1:
-        #     table[-1][-1] += (' ' if table[-1][-1] else '') + '(best)'
         self.to_screen(
             '[info] Available formats for %s:\n%s' % (info_dict['id'], render_table(
                 header_line,

youtube_dlc/__init__.py

@@ -440,6 +440,7 @@ def _real_main(argv=None):
         'autonumber_size': opts.autonumber_size,
         'autonumber_start': opts.autonumber_start,
         'restrictfilenames': opts.restrictfilenames,
+        'windowsfilenames': opts.windowsfilenames,
         'ignoreerrors': opts.ignoreerrors,
         'force_generic_extractor': opts.force_generic_extractor,
         'ratelimit': opts.ratelimit,
@@ -560,7 +561,7 @@ def _real_main(argv=None):
             # If updater returns True, exit. Required for windows
             if update_self(ydl.to_screen, opts.verbose, ydl._opener):
                 if actual_use:
-                    parser.error('The program must exit for the update to complete')
+                    sys.exit('ERROR: The program must exit for the update to complete')
                 sys.exit()
 
         # Maybe do nothing

youtube_dlc/extractor/adobetv.py

@@ -66,7 +66,7 @@ class AdobeTVBaseIE(InfoExtractor):
                 if original_filename.startswith('s3://') and not s3_extracted:
                     formats.append({
                         'format_id': 'original',
-                        'preference': 1,
+                        'quality': 1,
                         'url': original_filename.replace('s3://', 'https://s3.amazonaws.com/'),
                     })
                     s3_extracted = True

youtube_dlc/extractor/alura.py

@@ -67,7 +67,7 @@ class AluraIE(InfoExtractor):
                             f['height'] = int('720' if m.group('res') == 'hd' else '480')
                 formats.extend(video_format)
 
-            self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+            self._sort_formats(formats)
 
             return {
                 'id': video_id,

youtube_dlc/extractor/aol.py

@@ -125,7 +125,7 @@ class AolIE(YahooIE):
                         'height': int_or_none(qs.get('h', [None])[0]),
                     })
                 formats.append(f)
-        self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
+        self._sort_formats(formats)
 
         return {
             'id': video_id,

youtube_dlc/extractor/aparat.py

@@ -72,8 +72,7 @@ class AparatIE(InfoExtractor):
                             r'(\d+)[pP]', label or '', 'height',
                             default=None)),
                     })
-        self._sort_formats(
+        self._sort_formats(formats)
-            formats, field_preference=('height', 'width', 'tbr', 'format_id'))
 
         info = self._search_json_ld(webpage, video_id, default={})
 

youtube_dlc/extractor/arcpublishing.py

@@ -129,10 +129,6 @@ class ArcPublishingIE(InfoExtractor):
                 if all([f.get('acodec') == 'none' for f in m3u8_formats]):
                     continue
                 for f in m3u8_formats:
-                    if f.get('acodec') == 'none':
-                        f['preference'] = -40
-                    elif f.get('vcodec') == 'none':
-                        f['preference'] = -50
                     height = f.get('height')
                     if not height:
                         continue
@@ -150,10 +146,9 @@ class ArcPublishingIE(InfoExtractor):
                     'height': int_or_none(s.get('height')),
                     'filesize': int_or_none(s.get('filesize')),
                     'url': s_url,
-                    'preference': -1,
+                    'quality': -10,
                 })
-        self._sort_formats(
+        self._sort_formats(formats)
-            formats, ('preference', 'width', 'height', 'vbr', 'filesize', 'tbr', 'ext', 'format_id'))
 
         subtitles = {}
         for subtitle in (try_get(video, lambda x: x['subtitles']['urls'], list) or []):

youtube_dlc/extractor/ard.py

@@ -324,20 +324,42 @@ class ARDIE(InfoExtractor):
 
         formats = []
         for a in video_node.findall('.//asset'):
+            file_name = xpath_text(a, './fileName', default=None)
+            if not file_name:
+                continue
+            format_type = a.attrib.get('type')
+            format_url = url_or_none(file_name)
+            if format_url:
+                ext = determine_ext(file_name)
+                if ext == 'm3u8':
+                    formats.extend(self._extract_m3u8_formats(
+                        format_url, display_id, 'mp4', entry_protocol='m3u8_native',
+                        m3u8_id=format_type or 'hls', fatal=False))
+                    continue
+                elif ext == 'f4m':
+                    formats.extend(self._extract_f4m_formats(
+                        update_url_query(format_url, {'hdcore': '3.7.0'}),
+                        display_id, f4m_id=format_type or 'hds', fatal=False))
+                    continue
             f = {
-                'format_id': a.attrib['type'],
+                'format_id': format_type,
-                'width': int_or_none(a.find('./frameWidth').text),
+                'width': int_or_none(xpath_text(a, './frameWidth')),
-                'height': int_or_none(a.find('./frameHeight').text),
+                'height': int_or_none(xpath_text(a, './frameHeight')),
-                'vbr': int_or_none(a.find('./bitrateVideo').text),
+                'vbr': int_or_none(xpath_text(a, './bitrateVideo')),
-                'abr': int_or_none(a.find('./bitrateAudio').text),
+                'abr': int_or_none(xpath_text(a, './bitrateAudio')),
-                'vcodec': a.find('./codecVideo').text,
+                'vcodec': xpath_text(a, './codecVideo'),
-                'tbr': int_or_none(a.find('./totalBitrate').text),
+                'tbr': int_or_none(xpath_text(a, './totalBitrate')),
             }
-            if a.find('./serverPrefix').text:
+            server_prefix = xpath_text(a, './serverPrefix', default=None)
-                f['url'] = a.find('./serverPrefix').text
+            if server_prefix:
-                f['playpath'] = a.find('./fileName').text
+                f.update({
+                    'url': server_prefix,
+                    'playpath': file_name,
+                })
             else:
-                f['url'] = a.find('./fileName').text
+                if not format_url:
+                    continue
+                f['url'] = format_url
             formats.append(f)
         self._sort_formats(formats)
 

youtube_dlc/extractor/arte.py

@@ -150,7 +150,6 @@ class ArteTVIE(ArteTVBaseIE):
 
             format = {
                 'format_id': format_id,
-                'preference': -10 if f.get('videoFormat') == 'M3U8' else None,
                 'language_preference': lang_pref,
                 'format_note': '%s, %s' % (f.get('versionCode'), f.get('versionLibelle')),
                 'width': int_or_none(f.get('width')),
@@ -168,7 +167,9 @@ class ArteTVIE(ArteTVBaseIE):
 
             formats.append(format)
 
-        self._sort_formats(formats)
+        # For this extractor, quality only represents the relative quality
+        # with respect to other formats with the same resolution
+        self._sort_formats(formats, ('res', 'quality'))
 
         return {
             'id': player_info.get('VID') or video_id,

youtube_dlc/extractor/beatport.py

@@ -69,12 +69,10 @@ class BeatportIE(InfoExtractor):
                 'vcodec': 'none',
             }
             if ext == 'mp3':
-                fmt['preference'] = 0
                 fmt['acodec'] = 'mp3'
                 fmt['abr'] = 96
                 fmt['asr'] = 44100
             elif ext == 'mp4':
-                fmt['preference'] = 1
                 fmt['acodec'] = 'aac'
                 fmt['abr'] = 96
                 fmt['asr'] = 44100

youtube_dlc/extractor/bilibili.py

@@ -203,7 +203,7 @@ class BiliBiliIE(InfoExtractor):
                     formats.append({
                         'url': backup_url,
                         # backup URLs have lower priorities
-                        'preference': -2 if 'hd.mp4' in backup_url else -3,
+                        'quality': -2 if 'hd.mp4' in backup_url else -3,
                     })
 
                 for a_format in formats:

youtube_dlc/extractor/bokecc.py

@@ -23,7 +23,7 @@ class BokeCCBaseIE(InfoExtractor):
         formats = [{
             'format_id': format_id,
             'url': quality.find('./copy').attrib['playurl'],
-            'preference': int(quality.attrib['value']),
+            'quality': int(quality.attrib['value']),
         } for quality in info_xml.findall('./video/quality')]
 
         self._sort_formats(formats)

youtube_dlc/extractor/bpb.py

@@ -47,7 +47,7 @@ class BpbIE(InfoExtractor):
             quality = 'high' if '_high' in video_url else 'low'
             formats.append({
                 'url': video_url,
-                'preference': 10 if quality == 'high' else 0,
+                'quality': 10 if quality == 'high' else 0,
                 'format_note': quality,
                 'format_id': '%s-%s' % (quality, determine_ext(video_url)),
             })

youtube_dlc/extractor/cammodels.py

@@ -82,7 +82,7 @@ class CamModelsIE(InfoExtractor):
                     f.update({
                         'ext': 'mp4',
                         # hls skips fragments, preferring rtmp
-                        'preference': -1,
+                        'quality': -10,
                     })
                 else:
                     continue

youtube_dlc/extractor/canalplus.py

@@ -89,7 +89,7 @@ class CanalplusIE(InfoExtractor):
                     # the secret extracted from ya function in http://player.canalplus.fr/common/js/canalPlayer.js
                     'url': format_url + '?secret=pqzerjlsmdkjfoiuerhsdlfknaes',
                     'format_id': format_id,
-                    'preference': preference(format_id),
+                    'quality': preference(format_id),
                 })
         self._sort_formats(formats)
 

youtube_dlc/extractor/canvas.py

@@ -7,19 +7,21 @@ from .common import InfoExtractor
 from .gigya import GigyaBaseIE
 from ..compat import compat_HTTPError
 from ..utils import (
-    extract_attributes,
     ExtractorError,
-    strip_or_none,
+    clean_html,
+    extract_attributes,
     float_or_none,
+    get_element_by_class,
     int_or_none,
     merge_dicts,
     str_or_none,
+    strip_or_none,
     url_or_none,
 )
 
 
 class CanvasIE(InfoExtractor):
-    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza)/assets/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza|dako)/assets/(?P<id>[^/?#&]+)'
     _TESTS = [{
         'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
         'md5': '68993eda72ef62386a15ea2cf3c93107',
@@ -332,3 +334,51 @@ class VrtNUIE(GigyaBaseIE):
             'display_id': display_id,
             'season_number': int_or_none(page.get('episode_season')),
         })
+
+
+class DagelijkseKostIE(InfoExtractor):
+    IE_DESC = 'dagelijksekost.een.be'
+    _VALID_URL = r'https?://dagelijksekost\.een\.be/gerechten/(?P<id>[^/?#&]+)'
+    _TEST = {
+        'url': 'https://dagelijksekost.een.be/gerechten/hachis-parmentier-met-witloof',
+        'md5': '30bfffc323009a3e5f689bef6efa2365',
+        'info_dict': {
+            'id': 'md-ast-27a4d1ff-7d7b-425e-b84f-a4d227f592fa',
+            'display_id': 'hachis-parmentier-met-witloof',
+            'ext': 'mp4',
+            'title': 'Hachis parmentier met witloof',
+            'description': 'md5:9960478392d87f63567b5b117688cdc5',
+            'thumbnail': r're:^https?://.*\.jpg$',
+            'duration': 283.02,
+        },
+        'expected_warnings': ['is not a supported codec'],
+    }
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+
+        title = strip_or_none(get_element_by_class(
+            'dish-metadata__title', webpage
+        ) or self._html_search_meta(
+            'twitter:title', webpage))
+
+        description = clean_html(get_element_by_class(
+            'dish-description', webpage)
+        ) or self._html_search_meta(
+            ('description', 'twitter:description', 'og:description'),
+            webpage)
+
+        video_id = self._html_search_regex(
+            r'data-url=(["\'])(?P<id>(?:(?!\1).)+)\1', webpage, 'video id',
+            group='id')
+
+        return {
+            '_type': 'url_transparent',
+            'url': 'https://mediazone.vrt.be/api/v1/dako/assets/%s' % video_id,
+            'ie_key': CanvasIE.ie_key(),
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+        }

youtube_dlc/extractor/ccma.py

@@ -1,12 +1,14 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import calendar
 import datetime
 import re
 
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
+    extract_timezone,
     int_or_none,
     parse_duration,
     parse_resolution,
@@ -97,8 +99,9 @@ class CCMAIE(InfoExtractor):
         timestamp = None
         data_utc = try_get(informacio, lambda x: x['data_emissio']['utc'])
         try:
-            timestamp = datetime.datetime.strptime(
+            timezone, data_utc = extract_timezone(data_utc)
-                data_utc, '%Y-%d-%mT%H:%M:%S%z').timestamp()
+            timestamp = calendar.timegm((datetime.datetime.strptime(
+                data_utc, '%Y-%d-%mT%H:%M:%S') - timezone).timetuple())
         except TypeError:
             pass
 

youtube_dlc/extractor/cctv.py

@@ -162,7 +162,7 @@ class CCTVIE(InfoExtractor):
                         'url': video_url,
                         'format_id': 'http',
                         'quality': quality,
-                        'preference': -1,
+                        'source_preference': -10
                     })
 
         hls_url = try_get(data, lambda x: x['hls_url'], compat_str)

youtube_dlc/extractor/common.py

@@ -1367,12 +1367,12 @@ class InfoExtractor(object):
         regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
 
         default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
-                   'res', 'fps', 'codec:vp9', 'size', 'br', 'asr',
+                   'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
                    'proto', 'ext', 'has_audio', 'source', 'format_id')  # These must not be aliases
 
         settings = {
             'vcodec': {'type': 'ordered', 'regex': True,
-                       'order': ['av0?1', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
+                       'order': ['av0?1', 'vp0?9.2', 'vp0?9', '[hx]265|he?vc?', '[hx]264|avc', 'vp0?8', 'mp4v|h263', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
             'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
@@ -1384,11 +1384,11 @@ class InfoExtractor(object):
                      'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                      'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
             'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
-            'ie_pref': {'priority': True, 'type': 'extractor', 'field': 'extractor_preference'},
+            'ie_pref': {'priority': True, 'type': 'extractor'},
             'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
             'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
+            'lang': {'priority': True, 'convert': 'ignore', 'type': 'extractor', 'field': 'language_preference'},
-            'quality': {'priority': True, 'convert': 'float_none'},
+            'quality': {'convert': 'float_none', 'type': 'extractor'},
             'filesize': {'convert': 'bytes'},
             'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
             'id': {'convert': 'string', 'field': 'format_id'},
@@ -1399,7 +1399,7 @@ class InfoExtractor(object):
             'vbr': {'convert': 'float_none'},
             'abr': {'convert': 'float_none'},
             'asr': {'convert': 'float_none'},
-            'source': {'convert': 'ignore', 'field': 'source_preference'},
+            'source': {'convert': 'ignore', 'type': 'extractor', 'field': 'source_preference'},
 
             'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
             'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
@@ -1469,13 +1469,12 @@ class InfoExtractor(object):
             elif conversion == 'bytes':
                 return FileDownloader.parse_bytes(value)
             elif conversion == 'order':
-                order_free = self._get_field_setting(field, 'order_free')
+                order_list = (self._use_free_order and self._get_field_setting(field, 'order_free')) or self._get_field_setting(field, 'order')
-                order_list = order_free if order_free and self._use_free_order else self._get_field_setting(field, 'order')
                 use_regex = self._get_field_setting(field, 'regex')
                 list_length = len(order_list)
                 empty_pos = order_list.index('') if '' in order_list else list_length + 1
                 if use_regex and value is not None:
-                    for (i, regex) in enumerate(order_list):
+                    for i, regex in enumerate(order_list):
                         if regex and re.match(regex, value):
                             return list_length - i
                     return list_length - empty_pos  # not in list
@@ -1544,7 +1543,7 @@ class InfoExtractor(object):
         def print_verbose_info(self, to_screen):
             to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
             if self._sort_extractor:
-                to_screen('[debug] Sort order given by extractor: %s' % ','.join(self._sort_extractor))
+                to_screen('[debug] Sort order given by extractor: %s' % ', '.join(self._sort_extractor))
             to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
                 '+' if self._get_field_setting(field, 'reverse') else '', field,
                 '%s%s(%s)' % ('~' if self._get_field_setting(field, 'closest') else ':',
@@ -1561,7 +1560,7 @@ class InfoExtractor(object):
             if type == 'extractor':
                 maximum = self._get_field_setting(field, 'max')
                 if value is None or (maximum is not None and value >= maximum):
-                    value = 0
+                    value = -1
             elif type == 'boolean':
                 in_list = self._get_field_setting(field, 'in_list')
                 not_in_list = self._get_field_setting(field, 'not_in_list')
@@ -1694,7 +1693,7 @@ class InfoExtractor(object):
         self.to_screen(msg)
         time.sleep(timeout)
 
-    def _extract_f4m_formats(self, manifest_url, video_id, preference=None, f4m_id=None,
+    def _extract_f4m_formats(self, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
                              transform_source=lambda s: fix_xml_ampersands(s).strip(),
                              fatal=True, m3u8_id=None, data=None, headers={}, query={}):
         manifest = self._download_xml(
@@ -1709,10 +1708,10 @@ class InfoExtractor(object):
             return []
 
         return self._parse_f4m_formats(
-            manifest, manifest_url, video_id, preference=preference, f4m_id=f4m_id,
+            manifest, manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
             transform_source=transform_source, fatal=fatal, m3u8_id=m3u8_id)
 
-    def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, f4m_id=None,
+    def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
                            transform_source=lambda s: fix_xml_ampersands(s).strip(),
                            fatal=True, m3u8_id=None):
         if not isinstance(manifest, compat_etree_Element) and not fatal:
@@ -1777,7 +1776,7 @@ class InfoExtractor(object):
                 ext = determine_ext(manifest_url)
                 if ext == 'f4m':
                     f4m_formats = self._extract_f4m_formats(
-                        manifest_url, video_id, preference=preference, f4m_id=f4m_id,
+                        manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
                         transform_source=transform_source, fatal=fatal)
                     # Sometimes stream-level manifest contains single media entry that
                     # does not contain any quality metadata (e.g. http://matchtv.ru/#live-player).
@@ -1797,7 +1796,7 @@ class InfoExtractor(object):
                 elif ext == 'm3u8':
                     formats.extend(self._extract_m3u8_formats(
                         manifest_url, video_id, 'mp4', preference=preference,
-                        m3u8_id=m3u8_id, fatal=fatal))
+                        quality=quality, m3u8_id=m3u8_id, fatal=fatal))
                     continue
             formats.append({
                 'format_id': format_id,
@@ -1810,22 +1809,24 @@ class InfoExtractor(object):
                 'height': height,
                 'vcodec': vcodec,
                 'preference': preference,
+                'quality': quality,
             })
         return formats
 
-    def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, m3u8_id=None):
+    def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, quality=None, m3u8_id=None):
         return {
             'format_id': '-'.join(filter(None, [m3u8_id, 'meta'])),
             'url': m3u8_url,
             'ext': ext,
             'protocol': 'm3u8',
             'preference': preference - 100 if preference else -100,
+            'quality': quality,
             'resolution': 'multiple',
             'format_note': 'Quality selection URL',
         }
 
     def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
-                              entry_protocol='m3u8', preference=None,
+                              entry_protocol='m3u8', preference=None, quality=None,
                               m3u8_id=None, note=None, errnote=None,
                               fatal=True, live=False, data=None, headers={},
                               query={}):
@@ -1843,10 +1844,10 @@ class InfoExtractor(object):
 
         return self._parse_m3u8_formats(
             m3u8_doc, m3u8_url, ext=ext, entry_protocol=entry_protocol,
-            preference=preference, m3u8_id=m3u8_id, live=live)
+            preference=preference, quality=quality, m3u8_id=m3u8_id, live=live)
 
     def _parse_m3u8_formats(self, m3u8_doc, m3u8_url, ext=None,
-                            entry_protocol='m3u8', preference=None,
+                            entry_protocol='m3u8', preference=None, quality=None,
                             m3u8_id=None, live=False):
         if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
             return []
@@ -1884,6 +1885,7 @@ class InfoExtractor(object):
                 'ext': ext,
                 'protocol': entry_protocol,
                 'preference': preference,
+                'quality': quality,
             }]
 
         groups = {}
@@ -1912,6 +1914,7 @@ class InfoExtractor(object):
                     'ext': ext,
                     'protocol': entry_protocol,
                     'preference': preference,
+                    'quality': quality,
                 }
                 if media_type == 'AUDIO':
                     f['vcodec'] = 'none'
@@ -1971,6 +1974,7 @@ class InfoExtractor(object):
                     'fps': float_or_none(last_stream_inf.get('FRAME-RATE')),
                     'protocol': entry_protocol,
                     'preference': preference,
+                    'quality': quality,
                 }
                 resolution = last_stream_inf.get('RESOLUTION')
                 if resolution:
@@ -2678,7 +2682,7 @@ class InfoExtractor(object):
                 })
         return formats
 
-    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None):
+    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None, quality=None):
         def absolute_url(item_url):
             return urljoin(base_url, item_url)
 
@@ -2701,7 +2705,7 @@ class InfoExtractor(object):
                 formats = self._extract_m3u8_formats(
                     full_url, video_id, ext='mp4',
                     entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id,
-                    preference=preference, fatal=False)
+                    preference=preference, quality=quality, fatal=False)
             elif ext == 'mpd':
                 is_plain_url = False
                 formats = self._extract_mpd_formats(

youtube_dlc/extractor/coub.py

@@ -87,7 +87,7 @@ class CoubIE(InfoExtractor):
                     'filesize': int_or_none(item.get('size')),
                     'vcodec': 'none' if kind == 'audio' else None,
                     'quality': quality_key(quality),
-                    'preference': preference_key(HTML5),
+                    'source_preference': preference_key(HTML5),
                 })
 
         iphone_url = file_versions.get(IPHONE, {}).get('url')
@@ -95,7 +95,7 @@ class CoubIE(InfoExtractor):
             formats.append({
                 'url': iphone_url,
                 'format_id': IPHONE,
-                'preference': preference_key(IPHONE),
+                'source_preference': preference_key(IPHONE),
             })
 
         mobile_url = file_versions.get(MOBILE, {}).get('audio_url')
@@ -103,7 +103,7 @@ class CoubIE(InfoExtractor):
             formats.append({
                 'url': mobile_url,
                 'format_id': '%s-audio' % MOBILE,
-                'preference': preference_key(MOBILE),
+                'source_preference': preference_key(MOBILE),
             })
 
         self._sort_formats(formats)

youtube_dlc/extractor/crunchyroll.py

@@ -473,15 +473,11 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                 stream.get('url'), video_id, stream.get('format'),
                 audio_lang, hardsub_lang)
             for f in vrv_formats:
-                if not hardsub_lang:
+                f['language_preference'] = 1 if audio_lang == language else 0
-                    f['preference'] = 1
+                f['quality'] = (
-                language_preference = 0
+                    1 if not hardsub_lang
-                if audio_lang == language:
+                    else 0 if hardsub_lang == language
-                    language_preference += 1
+                    else -1)
-                if hardsub_lang == language:
-                    language_preference += 1
-                if language_preference:
-                    f['language_preference'] = language_preference
             formats.extend(vrv_formats)
         if not formats:
             available_fmts = []
@@ -571,7 +567,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                         'ext': 'flv',
                     })
                     formats.append(format_info)
-        self._sort_formats(formats, ('preference', 'language_preference', 'height', 'width', 'tbr', 'fps'))
+        self._sort_formats(formats)
 
         metadata = self._call_rpc_api(
             'VideoPlayer_GetMediaMetadata', video_id,

youtube_dlc/extractor/dispeak.py

@@ -82,6 +82,7 @@ class DigitallySpeakingIE(InfoExtractor):
                 'play_path': remove_end(audio.get('url'), '.flv'),
                 'ext': 'flv',
                 'vcodec': 'none',
+                'quality': 1,
                 'format_id': audio.get('code'),
             })
         slide_video_path = xpath_text(metadata, './slideVideo', fatal=True)
@@ -91,7 +92,6 @@ class DigitallySpeakingIE(InfoExtractor):
             'ext': 'flv',
             'format_note': 'slide deck video',
             'quality': -2,
-            'preference': -2,
             'format_id': 'slides',
             'acodec': 'none',
         })
@@ -102,7 +102,6 @@ class DigitallySpeakingIE(InfoExtractor):
             'ext': 'flv',
             'format_note': 'speaker video',
             'quality': -1,
-            'preference': -1,
             'format_id': 'speaker',
         })
         return formats

youtube_dlc/extractor/dplay.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .common import InfoExtractor
@@ -10,11 +11,13 @@ from ..utils import (
     ExtractorError,
     float_or_none,
     int_or_none,
+    strip_or_none,
     unified_timestamp,
 )
 
 
 class DPlayIE(InfoExtractor):
+    _PATH_REGEX = r'/(?P<id>[^/]+/[^/?#]+)'
     _VALID_URL = r'''(?x)https?://
         (?P<domain>
             (?:www\.)?(?P<host>d
@@ -24,7 +27,7 @@ class DPlayIE(InfoExtractor):
                 )
             )|
             (?P<subdomain_country>es|it)\.dplay\.com
-        )/[^/]+/(?P<id>[^/]+/[^/?#]+)'''
+        )/[^/]+''' + _PATH_REGEX
 
     _TESTS = [{
         # non geo restricted, via secure api, unsigned download hls URL
@@ -151,56 +154,79 @@ class DPlayIE(InfoExtractor):
         'only_matching': True,
     }]
 
+    def _process_errors(self, e, geo_countries):
+        info = self._parse_json(e.cause.read().decode('utf-8'), None)
+        error = info['errors'][0]
+        error_code = error.get('code')
+        if error_code == 'access.denied.geoblocked':
+            self.raise_geo_restricted(countries=geo_countries)
+        elif error_code in ('access.denied.missingpackage', 'invalid.token'):
+            raise ExtractorError(
+                'This video is only available for registered users. You may want to use --cookies.', expected=True)
+        raise ExtractorError(info['errors'][0]['detail'], expected=True)
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['Authorization'] = 'Bearer ' + self._download_json(
+            disco_base + 'token', display_id, 'Downloading token',
+            query={
+                'realm': realm,
+            })['data']['attributes']['token']
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        streaming = self._download_json(
+            disco_base + 'playback/videoPlaybackInfo/' + video_id,
+            video_id, headers=headers)['data']['attributes']['streaming']
+        streaming_list = []
+        for format_id, format_dict in streaming.items():
+            streaming_list.append({
+                'type': format_id,
+                'url': format_dict.get('url'),
+            })
+        return streaming_list
+
     def _get_disco_api_info(self, url, display_id, disco_host, realm, country):
         geo_countries = [country.upper()]
         self._initialize_geo_bypass({
             'countries': geo_countries,
         })
         disco_base = 'https://%s/' % disco_host
-        token = self._download_json(
-            disco_base + 'token', display_id, 'Downloading token',
-            query={
-                'realm': realm,
-            })['data']['attributes']['token']
         headers = {
             'Referer': url,
-            'Authorization': 'Bearer ' + token,
         }
-        video = self._download_json(
+        self._update_disco_api_headers(headers, disco_base, display_id, realm)
-            disco_base + 'content/videos/' + display_id, display_id,
+        try:
-            headers=headers, query={
+            video = self._download_json(
-                'fields[channel]': 'name',
+                disco_base + 'content/videos/' + display_id, display_id,
-                'fields[image]': 'height,src,width',
+                headers=headers, query={
-                'fields[show]': 'name',
+                    'fields[channel]': 'name',
-                'fields[tag]': 'name',
+                    'fields[image]': 'height,src,width',
-                'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
+                    'fields[show]': 'name',
-                'include': 'images,primaryChannel,show,tags'
+                    'fields[tag]': 'name',
-            })
+                    'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
+                    'include': 'images,primaryChannel,show,tags'
+                })
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
+                self._process_errors(e, geo_countries)
+            raise
         video_id = video['data']['id']
         info = video['data']['attributes']
         title = info['name'].strip()
         formats = []
         try:
-            streaming = self._download_json(
+            streaming = self._download_video_playback_info(
-                disco_base + 'playback/videoPlaybackInfo/' + video_id,
+                disco_base, video_id, headers)
-                display_id, headers=headers)['data']['attributes']['streaming']
         except ExtractorError as e:
             if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                info = self._parse_json(e.cause.read().decode('utf-8'), display_id)
+                self._process_errors(e, geo_countries)
-                error = info['errors'][0]
-                error_code = error.get('code')
-                if error_code == 'access.denied.geoblocked':
-                    self.raise_geo_restricted(countries=geo_countries)
-                elif error_code == 'access.denied.missingpackage':
-                    self.raise_login_required()
-                raise ExtractorError(info['errors'][0]['detail'], expected=True)
             raise
-        for format_id, format_dict in streaming.items():
+        for format_dict in streaming:
             if not isinstance(format_dict, dict):
                 continue
             format_url = format_dict.get('url')
             if not format_url:
                 continue
+            format_id = format_dict.get('type')
             ext = determine_ext(format_url)
             if format_id == 'dash' or ext == 'mpd':
                 formats.extend(self._extract_mpd_formats(
@@ -248,7 +274,7 @@ class DPlayIE(InfoExtractor):
             'id': video_id,
             'display_id': display_id,
             'title': title,
-            'description': info.get('description'),
+            'description': strip_or_none(info.get('description')),
             'duration': float_or_none(info.get('videoDuration'), 1000),
             'timestamp': unified_timestamp(info.get('publishStart')),
             'series': series,
@@ -268,3 +294,75 @@ class DPlayIE(InfoExtractor):
         host = 'disco-api.' + domain if domain[0] == 'd' else 'eu2-prod.disco-api.com'
         return self._get_disco_api_info(
             url, display_id, host, 'dplay' + country, country)
+
+
+class DiscoveryPlusIE(DPlayIE):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.com/video' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.com/video/property-brothers-forever-home/food-and-family',
+        'info_dict': {
+            'id': '1140794',
+            'display_id': 'property-brothers-forever-home/food-and-family',
+            'ext': 'mp4',
+            'title': 'Food and Family',
+            'description': 'The brothers help a Richmond family expand their single-level home.',
+            'duration': 2583.113,
+            'timestamp': 1609304400,
+            'upload_date': '20201230',
+            'creator': 'HGTV',
+            'series': 'Property Brothers: Forever Home',
+            'season_number': 1,
+            'episode_number': 1,
+        },
+        'skip': 'Available for Premium users',
+    }]
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['x-disco-client'] = 'WEB:UNKNOWN:dplus_us:15.0.0'
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        return self._download_json(
+            disco_base + 'playback/v3/videoPlaybackInfo',
+            video_id, headers=headers, data=json.dumps({
+                'deviceInfo': {
+                    'adBlocker': False,
+                },
+                'videoId': video_id,
+                'wisteriaProperties': {
+                    'platform': 'desktop',
+                },
+            }).encode('utf-8'))['data']['attributes']['streaming']
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'us1-prod-direct.discoveryplus.com', 'go', 'us')
+
+
+class HGTVDeIE(DPlayIE):
+    _VALID_URL = r'https?://de\.hgtv\.com/sendungen' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://de.hgtv.com/sendungen/tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette/',
+        'info_dict': {
+            'id': '151205',
+            'display_id': 'tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette',
+            'ext': 'mp4',
+            'title': 'Wer braucht schon eine Toilette',
+            'description': 'md5:05b40a27e7aed2c9172de34d459134e2',
+            'duration': 1177.024,
+            'timestamp': 1595705400,
+            'upload_date': '20200725',
+            'creator': 'HGTV',
+            'series': 'Tiny House - klein, aber oho',
+            'season_number': 3,
+            'episode_number': 3,
+        },
+        'params': {
+            'format': 'bestvideo',
+        },
+    }]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'eu1-prod.disco-api.com', 'hgtv', 'de')

youtube_dlc/extractor/dreisat.py

@@ -0,0 +1,193 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    unified_strdate,
+    xpath_text,
+    determine_ext,
+    float_or_none,
+    ExtractorError,
+)
+
+
+class DreiSatIE(InfoExtractor):
+    IE_NAME = '3sat'
+    _GEO_COUNTRIES = ['DE']
+    _VALID_URL = r'https?://(?:www\.)?3sat\.de/mediathek/(?:(?:index|mediathek)\.php)?\?(?:(?:mode|display)=[^&]+&)*obj=(?P<id>[0-9]+)'
+    _TESTS = [
+        {
+            'url': 'http://www.3sat.de/mediathek/index.php?mode=play&obj=45918',
+            'md5': 'be37228896d30a88f315b638900a026e',
+            'info_dict': {
+                'id': '45918',
+                'ext': 'mp4',
+                'title': 'Waidmannsheil',
+                'description': 'md5:cce00ca1d70e21425e72c86a98a56817',
+                'uploader': 'SCHWEIZWEIT',
+                'uploader_id': '100000210',
+                'upload_date': '20140913'
+            },
+            'params': {
+                'skip_download': True,  # m3u8 downloads
+            }
+        },
+        {
+            'url': 'http://www.3sat.de/mediathek/mediathek.php?mode=play&obj=51066',
+            'only_matching': True,
+        },
+    ]
+
+    def _parse_smil_formats(self, smil, smil_url, video_id, namespace=None, f4m_params=None, transform_rtmp_url=None):
+        param_groups = {}
+        for param_group in smil.findall(self._xpath_ns('./head/paramGroup', namespace)):
+            group_id = param_group.get(self._xpath_ns(
+                'id', 'http://www.w3.org/XML/1998/namespace'))
+            params = {}
+            for param in param_group:
+                params[param.get('name')] = param.get('value')
+            param_groups[group_id] = params
+
+        formats = []
+        for video in smil.findall(self._xpath_ns('.//video', namespace)):
+            src = video.get('src')
+            if not src:
+                continue
+            bitrate = int_or_none(self._search_regex(r'_(\d+)k', src, 'bitrate', None)) or float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
+            group_id = video.get('paramGroup')
+            param_group = param_groups[group_id]
+            for proto in param_group['protocols'].split(','):
+                formats.append({
+                    'url': '%s://%s' % (proto, param_group['host']),
+                    'app': param_group['app'],
+                    'play_path': src,
+                    'ext': 'flv',
+                    'format_id': '%s-%d' % (proto, bitrate),
+                    'tbr': bitrate,
+                })
+        self._sort_formats(formats)
+        return formats
+
+    def extract_from_xml_url(self, video_id, xml_url):
+        doc = self._download_xml(
+            xml_url, video_id,
+            note='Downloading video info',
+            errnote='Failed to download video info')
+
+        status_code = xpath_text(doc, './status/statuscode')
+        if status_code and status_code != 'ok':
+            if status_code == 'notVisibleAnymore':
+                message = 'Video %s is not available' % video_id
+            else:
+                message = '%s returned error: %s' % (self.IE_NAME, status_code)
+            raise ExtractorError(message, expected=True)
+
+        title = xpath_text(doc, './/information/title', 'title', True)
+
+        urls = []
+        formats = []
+        for fnode in doc.findall('.//formitaeten/formitaet'):
+            video_url = xpath_text(fnode, 'url')
+            if not video_url or video_url in urls:
+                continue
+            urls.append(video_url)
+
+            is_available = 'http://www.metafilegenerator' not in video_url
+            geoloced = 'static_geoloced_online' in video_url
+            if not is_available or geoloced:
+                continue
+
+            format_id = fnode.attrib['basetype']
+            format_m = re.match(r'''(?x)
+                (?P<vcodec>[^_]+)_(?P<acodec>[^_]+)_(?P<container>[^_]+)_
+                (?P<proto>[^_]+)_(?P<index>[^_]+)_(?P<indexproto>[^_]+)
+            ''', format_id)
+
+            ext = determine_ext(video_url, None) or format_m.group('container')
+
+            if ext == 'meta':
+                continue
+            elif ext == 'smil':
+                formats.extend(self._extract_smil_formats(
+                    video_url, video_id, fatal=False))
+            elif ext == 'm3u8':
+                # the certificates are misconfigured (see
+                # https://github.com/ytdl-org/youtube-dl/issues/8665)
+                if video_url.startswith('https://'):
+                    continue
+                formats.extend(self._extract_m3u8_formats(
+                    video_url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id=format_id, fatal=False))
+            elif ext == 'f4m':
+                formats.extend(self._extract_f4m_formats(
+                    video_url, video_id, f4m_id=format_id, fatal=False))
+            else:
+                quality = xpath_text(fnode, './quality')
+                if quality:
+                    format_id += '-' + quality
+
+                abr = int_or_none(xpath_text(fnode, './audioBitrate'), 1000)
+                vbr = int_or_none(xpath_text(fnode, './videoBitrate'), 1000)
+
+                tbr = int_or_none(self._search_regex(
+                    r'_(\d+)k', video_url, 'bitrate', None))
+                if tbr and vbr and not abr:
+                    abr = tbr - vbr
+
+                formats.append({
+                    'format_id': format_id,
+                    'url': video_url,
+                    'ext': ext,
+                    'acodec': format_m.group('acodec'),
+                    'vcodec': format_m.group('vcodec'),
+                    'abr': abr,
+                    'vbr': vbr,
+                    'tbr': tbr,
+                    'width': int_or_none(xpath_text(fnode, './width')),
+                    'height': int_or_none(xpath_text(fnode, './height')),
+                    'filesize': int_or_none(xpath_text(fnode, './filesize')),
+                    'protocol': format_m.group('proto').lower(),
+                })
+
+        geolocation = xpath_text(doc, './/details/geolocation')
+        if not formats and geolocation and geolocation != 'none':
+            self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
+
+        self._sort_formats(formats)
+
+        thumbnails = []
+        for node in doc.findall('.//teaserimages/teaserimage'):
+            thumbnail_url = node.text
+            if not thumbnail_url:
+                continue
+            thumbnail = {
+                'url': thumbnail_url,
+            }
+            thumbnail_key = node.get('key')
+            if thumbnail_key:
+                m = re.match('^([0-9]+)x([0-9]+)$', thumbnail_key)
+                if m:
+                    thumbnail['width'] = int(m.group(1))
+                    thumbnail['height'] = int(m.group(2))
+            thumbnails.append(thumbnail)
+
+        upload_date = unified_strdate(xpath_text(doc, './/details/airtime'))
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': xpath_text(doc, './/information/detail'),
+            'duration': int_or_none(xpath_text(doc, './/details/lengthSec')),
+            'thumbnails': thumbnails,
+            'uploader': xpath_text(doc, './/details/originChannelTitle'),
+            'uploader_id': xpath_text(doc, './/details/originChannelId'),
+            'upload_date': upload_date,
+            'formats': formats,
+        }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        details_url = 'http://www.3sat.de/mediathek/xmlservice/web/beitragsDetails?id=%s' % video_id
+        return self.extract_from_xml_url(video_id, details_url)

youtube_dlc/extractor/drtv.py

@@ -242,7 +242,7 @@ class DRTVIE(InfoExtractor):
                     elif target == 'HLS':
                         formats.extend(self._extract_m3u8_formats(
                             uri, video_id, 'mp4', entry_protocol='m3u8_native',
-                            preference=preference, m3u8_id=format_id,
+                            quality=preference, m3u8_id=format_id,
                             fatal=False))
                     else:
                         bitrate = link.get('Bitrate')
@@ -254,7 +254,7 @@ class DRTVIE(InfoExtractor):
                             'tbr': int_or_none(bitrate),
                             'ext': link.get('FileFormat'),
                             'vcodec': 'none' if kind == 'AudioResource' else None,
-                            'preference': preference,
+                            'quality': preference,
                         })
             subtitles_list = asset.get('SubtitlesList') or asset.get('Subtitleslist')
             if isinstance(subtitles_list, list):

youtube_dlc/extractor/espn.py

@@ -154,7 +154,7 @@ class ESPNIE(OnceIE):
                         'tbr': int(mobj.group(3)),
                     })
                 if source_id == 'mezzanine':
-                    f['preference'] = 1
+                    f['quality'] = 1
                 formats.append(f)
 
         links = clip.get('links', {})

youtube_dlc/extractor/extractors.py

@@ -182,6 +182,7 @@ from .canvas import (
     CanvasIE,
     CanvasEenIE,
     VrtNUIE,
+    DagelijkseKostIE,
 )
 from .carambatv import (
     CarambaTVIE,
@@ -309,7 +310,12 @@ from .douyutv import (
     DouyuShowIE,
     DouyuTVIE,
 )
-from .dplay import DPlayIE
+from .dplay import (
+    DPlayIE,
+    DiscoveryPlusIE,
+    HGTVDeIE,
+)
+from .dreisat import DreiSatIE
 from .drbonanza import DRBonanzaIE
 from .drtuber import DrTuberIE
 from .drtv import (
@@ -1107,6 +1113,11 @@ from .shared import (
     VivoIE,
 )
 from .showroomlive import ShowRoomLiveIE
+from .simplecast import (
+    SimplecastIE,
+    SimplecastEpisodeIE,
+    SimplecastPodcastIE,
+)
 from .sina import SinaIE
 from .sixplay import SixPlayIE
 from .skyit import (
@@ -1165,11 +1176,6 @@ from .spike import (
     BellatorIE,
     ParamountNetworkIE,
 )
-from .storyfire import (
-    StoryFireIE,
-    StoryFireUserIE,
-    StoryFireSeriesIE,
-)
 from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
@@ -1193,6 +1199,11 @@ from .srgssr import (
 from .srmediathek import SRMediathekIE
 from .stanfordoc import StanfordOpenClassroomIE
 from .steam import SteamIE
+from .storyfire import (
+    StoryFireIE,
+    StoryFireUserIE,
+    StoryFireSeriesIE,
+)
 from .streamable import StreamableIE
 from .streamcloud import StreamcloudIE
 from .streamcz import StreamCZIE
@@ -1652,6 +1663,7 @@ from .zattoo import (
     ZattooLiveIE,
 )
 from .zdf import ZDFIE, ZDFChannelIE
+from .zhihu import ZhihuIE
 from .zingmp3 import ZingMp3IE
 from .zoom import ZoomIE
 from .zype import ZypeIE

youtube_dlc/extractor/facebook.py

@@ -619,7 +619,7 @@ class FacebookIE(InfoExtractor):
                         formats.append({
                             'format_id': '%s_%s_%s' % (format_id, quality, src_type),
                             'url': src,
-                            'preference': preference,
+                            'quality': preference,
                         })
             extract_dash_manifest(f[0], formats)
             subtitles_src = f[0].get('subtitles_src')

youtube_dlc/extractor/firsttv.py

@@ -104,7 +104,7 @@ class FirstTVIE(InfoExtractor):
                     'tbr': tbr,
                     'source_preference': quality(f.get('name')),
                     # quality metadata of http formats may be incorrect
-                    'preference': -1,
+                    'preference': -10,
                 })
             # m3u8 URL format is reverse engineered from [1] (search for
             # master.m3u8). dashEdges (that is currently balancer-vod.1tv.ru)

youtube_dlc/extractor/flickr.py

@@ -88,7 +88,7 @@ class FlickrIE(InfoExtractor):
                 formats.append({
                     'format_id': stream_type,
                     'url': stream['_content'],
-                    'preference': preference(stream_type),
+                    'quality': preference(stream_type),
                 })
             self._sort_formats(formats)
 

youtube_dlc/extractor/generic.py

@@ -133,6 +133,7 @@ from .bitchute import BitChuteIE
 from .rumble import RumbleEmbedIE
 from .arcpublishing import ArcPublishingIE
 from .medialaan import MedialaanIE
+from .simplecast import SimplecastIE
 
 
 class GenericIE(InfoExtractor):
@@ -2240,6 +2241,15 @@ class GenericIE(InfoExtractor):
                 'duration': 159,
             },
         },
+        {
+            # Simplecast player embed
+            'url': 'https://www.bio.org/podcast',
+            'info_dict': {
+                'id': 'podcast',
+                'title': 'I AM BIO Podcast | BIO',
+            },
+            'playlist_mincount': 52,
+        },
     ]
 
     def report_following_redirect(self, new_url):
@@ -2794,6 +2804,12 @@ class GenericIE(InfoExtractor):
             return self.playlist_from_matches(
                 matches, video_id, video_title, getter=unescapeHTML, ie='FunnyOrDie')
 
+        # Look for Simplecast embeds
+        simplecast_urls = SimplecastIE._extract_urls(webpage)
+        if simplecast_urls:
+            return self.playlist_from_matches(
+                simplecast_urls, video_id, video_title)
+
         # Look for BBC iPlayer embed
         matches = re.findall(r'setPlaylist\("(https?://www\.bbc\.co\.uk/iplayer/[^/]+/[\da-z]{8})"\)', webpage)
         if matches:

youtube_dlc/extractor/go.py

@@ -236,7 +236,7 @@ class GoIE(AdobePassIE):
                 if re.search(r'(?:/mp4/source/|_source\.mp4)', asset_url):
                     f.update({
                         'format_id': ('%s-' % format_id if format_id else '') + 'SOURCE',
-                        'preference': 1,
+                        'quality': 1,
                     })
                 else:
                     mobj = re.search(r'/(\d+)x(\d+)/', asset_url)

youtube_dlc/extractor/hearthisat.py

@@ -115,7 +115,7 @@ class HearThisAtIE(InfoExtractor):
                     'vcodec': 'none',
                     'ext': ext,
                     'url': download_url,
-                    'preference': 2,  # Usually better quality
+                    'quality': 2,  # Usually better quality
                 })
         self._sort_formats(formats)
 

youtube_dlc/extractor/ign.py

@@ -100,7 +100,7 @@ class IGNIE(IGNBaseIE):
             formats.append({
                 'ext': determine_ext(mezzanine_url, 'mp4'),
                 'format_id': 'mezzanine',
-                'preference': 1,
+                'quality': 1,
                 'url': mezzanine_url,
             })
 

youtube_dlc/extractor/imgur.py

@@ -72,7 +72,7 @@ class ImgurIE(InfoExtractor):
                 gif_json, video_id, transform_source=js_to_json)
             formats.append({
                 'format_id': 'gif',
-                'preference': -10,
+                'preference': -10,  # gifs are worse than videos
                 'width': width,
                 'height': height,
                 'ext': 'gif',

youtube_dlc/extractor/iqiyi.py

@@ -373,7 +373,7 @@ class IqiyiIE(InfoExtractor):
                     'url': stream['m3utx'],
                     'format_id': vd,
                     'ext': 'mp4',
-                    'preference': self._FORMATS_MAP.get(vd, -1),
+                    'quality': self._FORMATS_MAP.get(vd, -1),
                     'protocol': 'm3u8_native',
                 })
 

youtube_dlc/extractor/kuwo.py

@@ -49,7 +49,7 @@ class KuwoBaseIE(InfoExtractor):
                     'url': song_url,
                     'format_id': file_format['format'],
                     'format': file_format['format'],
-                    'preference': file_format['preference'],
+                    'quality': file_format['preference'],
                     'abr': file_format.get('abr'),
                 })
 

youtube_dlc/extractor/leeco.py

@@ -185,7 +185,7 @@ class LeIE(InfoExtractor):
                     f['height'] = int_or_none(format_id[:-1])
 
                 formats.append(f)
-        self._sort_formats(formats, ('height', 'quality', 'format_id'))
+        self._sort_formats(formats, ('res', 'quality'))
 
         publish_time = parse_iso8601(self._html_search_regex(
             r'发布时间&nbsp;([^<>]+) ', page, 'publish time', default=None),

youtube_dlc/extractor/lifenews.py

@@ -201,7 +201,7 @@ class LifeEmbedIE(InfoExtractor):
             formats.append({
                 'url': original_url,
                 'format_id': determine_ext(original_url, None),
-                'preference': 1,
+                'quality': 1,
             })
 
         playlist = self._parse_json(

youtube_dlc/extractor/limelight.py

@@ -175,7 +175,7 @@ class LimelightBaseIE(InfoExtractor):
                 formats.append({
                     'url': media_url,
                     'format_id': format_id,
-                    'preference': -1,
+                    'quality': -10,
                     'ext': ext,
                 })
 

youtube_dlc/extractor/linkedin.py

@@ -124,7 +124,10 @@ class LinkedInLearningIE(LinkedInLearningBaseIE):
                 streaming_url, video_slug, 'mp4',
                 'm3u8_native', m3u8_id='hls', fatal=False))
 
-        self._sort_formats(formats, ('width', 'height', 'source_preference', 'tbr', 'abr'))
+        # It seems like this would be correctly handled by default
+        # However, unless someone can confirm this, the old
+        # behaviour is being kept as-is
+        self._sort_formats(formats, ('res', 'source_preference'))
 
         return {
             'id': self._get_video_id(video_data, course_slug, video_slug),

youtube_dlc/extractor/liveleak.py

@@ -143,7 +143,7 @@ class LiveLeakIE(InfoExtractor):
                         formats.append({
                             'format_id': format_id,
                             'url': orig_url,
-                            'preference': 1,
+                            'quality': 1,
                         })
             self._sort_formats(formats)
             info_dict['formats'] = formats

youtube_dlc/extractor/livestream.py

@@ -84,7 +84,7 @@ class LivestreamIE(InfoExtractor):
                 'format_id': 'smil_%d' % tbr,
                 'ext': 'flv',
                 'tbr': tbr,
-                'preference': -1000,
+                'preference': -1000,  # Strictly inferior than all other formats?
             })
         return formats
 

youtube_dlc/extractor/massengeschmacktv.py

@@ -67,7 +67,7 @@ class MassengeschmackTVIE(InfoExtractor):
                 'vcodec': 'none' if format_id.startswith('Audio') else None,
             })
 
-        self._sort_formats(formats, ('width', 'height', 'filesize', 'tbr'))
+        self._sort_formats(formats)
 
         return {
             'id': episode,

youtube_dlc/extractor/mdr.py

@@ -137,11 +137,11 @@ class MDRIE(InfoExtractor):
                 if ext == 'm3u8':
                     formats.extend(self._extract_m3u8_formats(
                         video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                        preference=0, m3u8_id='HLS', fatal=False))
+                        quality=1, m3u8_id='HLS', fatal=False))
                 elif ext == 'f4m':
                     formats.extend(self._extract_f4m_formats(
                         video_url + '?hdcore=3.7.0&plugin=aasp-3.7.0.39.44', video_id,
-                        preference=0, f4m_id='HDS', fatal=False))
+                        quality=1, f4m_id='HDS', fatal=False))
                 else:
                     media_type = xpath_text(asset, './mediaType', 'media type', default='MP4')
                     vbr = int_or_none(xpath_text(asset, './bitrateVideo', 'vbr'), 1000)

youtube_dlc/extractor/mediasite.py

@@ -206,7 +206,7 @@ class MediasiteIE(InfoExtractor):
             # disprefer 'secondary' streams
             if stream_type != 0:
                 for fmt in stream_formats:
-                    fmt['preference'] = -1
+                    fmt['quality'] = -10
 
             thumbnail_url = Stream.get('ThumbnailUrl')
             if thumbnail_url:

youtube_dlc/extractor/msn.py

@@ -132,7 +132,7 @@ class MSNIE(InfoExtractor):
                         'width': int_or_none(file_.get('width')),
                         'height': int_or_none(file_.get('height')),
                         'vbr': int_or_none(self._search_regex(r'_(\d+)\.mp4', format_url, 'vbr', default=None)),
-                        'preference': 1 if format_id == '1001' else None,
+                        'quality': 1 if format_id == '1001' else None,
                     })
             self._sort_formats(formats)
 

youtube_dlc/extractor/muenchentv.py

@@ -61,7 +61,7 @@ class MuenchenTVIE(InfoExtractor):
                 'tbr': int_or_none(s.get('label')),
                 'ext': 'mp4',
                 'format_id': format_id,
-                'preference': -100 if '.smil' in s['file'] else 0,
+                'preference': -100 if '.smil' in s['file'] else 0,  # Strictly inferior than all other formats?
             })
         self._sort_formats(formats)
 

youtube_dlc/extractor/nba.py

@@ -303,7 +303,7 @@ class NBABaseIE(NBACVPBaseIE):
                 formats.append({
                     'format_id': 'source',
                     'url': source_url,
-                    'preference': 1,
+                    'quality': 1,
                 })
 
             m3u8_url = video.get('m3u8')

youtube_dlc/extractor/niconico.py

@@ -546,11 +546,29 @@ class NiconicoIE(InfoExtractor):
 
         webpage_url = get_video_info_web('watch_url') or url
 
+        # for channel movie and community movie
+        channel_id = try_get(
+            api_data,
+            (lambda x: x['channel']['globalId'],
+             lambda x: x['community']['globalId']))
+        channel = try_get(
+            api_data,
+            (lambda x: x['channel']['name'],
+             lambda x: x['community']['name']))
+
         # Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
         # in the JSON, which will cause None to be returned instead of {}.
         owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
-        uploader_id = get_video_info_web(['ch_id', 'user_id']) or owner.get('id')
+        uploader_id = (
-        uploader = get_video_info_web(['ch_name', 'user_nickname']) or owner.get('nickname')
+            get_video_info_web(['ch_id', 'user_id'])
+            or owner.get('id')
+            or channel_id
+        )
+        uploader = (
+            get_video_info_web(['ch_name', 'user_nickname'])
+            or owner.get('nickname')
+            or channel
+        )
 
         return {
             'id': video_id,
@@ -561,6 +579,8 @@ class NiconicoIE(InfoExtractor):
             'uploader': uploader,
             'timestamp': timestamp,
             'uploader_id': uploader_id,
+            'channel': channel,
+            'channel_id': channel_id,
             'view_count': view_count,
             'comment_count': comment_count,
             'duration': duration,

youtube_dlc/extractor/ninegag.py

@@ -2,10 +2,11 @@ from __future__ import unicode_literals
 
 from .common import InfoExtractor
 from ..utils import (
-    determine_ext,
     ExtractorError,
+    determine_ext,
     int_or_none,
     try_get,
+    unescapeHTML,
     url_or_none,
 )
 
@@ -14,7 +15,7 @@ class NineGagIE(InfoExtractor):
     IE_NAME = '9gag'
     _VALID_URL = r'https?://(?:www\.)?9gag\.com/gag/(?P<id>[^/?&#]+)'
 
-    _TEST = {
+    _TESTS = [{
         'url': 'https://9gag.com/gag/ae5Ag7B',
         'info_dict': {
             'id': 'ae5Ag7B',
@@ -29,7 +30,11 @@ class NineGagIE(InfoExtractor):
             'dislike_count': int,
             'comment_count': int,
         }
-    }
+    }, {
+        # HTML escaped title
+        'url': 'https://9gag.com/gag/av5nvyb',
+        'only_matching': True,
+    }]
 
     def _real_extract(self, url):
         post_id = self._match_id(url)
@@ -43,7 +48,7 @@ class NineGagIE(InfoExtractor):
                 'The given url does not contain a video',
                 expected=True)
 
-        title = post['title']
+        title = unescapeHTML(post['title'])
 
         duration = None
         formats = []

youtube_dlc/extractor/noco.py

@@ -183,7 +183,7 @@ class NocoIE(InfoExtractor):
                         'filesize': int_or_none(fmt.get('filesize')),
                         'format_note': qualities[format_id].get('quality_name'),
                         'quality': qualities[format_id].get('priority'),
-                        'preference': preference,
+                        'language_preference': preference,
                     })
 
         self._sort_formats(formats)

youtube_dlc/extractor/npo.py

@@ -425,7 +425,7 @@ class NPOIE(NPOBaseIE):
                         stream_url, video_id, fatal=False)
                     # f4m downloader downloads only piece of live stream
                     for f4m_format in f4m_formats:
-                        f4m_format['preference'] = -1
+                        f4m_format['preference'] = -5
                     formats.extend(f4m_formats)
                 elif stream_type == 'hls':
                     formats.extend(self._extract_m3u8_formats(

youtube_dlc/extractor/ntvde.py

@@ -62,7 +62,7 @@ class NTVDeIE(InfoExtractor):
             m3u8_url = compat_urlparse.urljoin('http://video.n-tv.de', vdata['videoM3u8'])
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, ext='mp4', entry_protocol='m3u8_native',
-                preference=0, m3u8_id='hls', fatal=False))
+                quality=1, m3u8_id='hls', fatal=False))
         self._sort_formats(formats)
 
         return {

youtube_dlc/extractor/nytimes.py

@@ -72,7 +72,7 @@ class NYTimesBaseIE(InfoExtractor):
                     'tbr': int_or_none(video.get('bitrate'), 1000) or None,
                     'ext': ext,
                 })
-        self._sort_formats(formats, ('height', 'width', 'filesize', 'tbr', 'fps', 'format_id'))
+        self._sort_formats(formats)
 
         thumbnails = []
         for image in video_data.get('images', []):

youtube_dlc/extractor/ora.py

@@ -55,7 +55,7 @@ class OraTVIE(InfoExtractor):
                 formats.append({
                     'url': http_template % q,
                     'format_id': q,
-                    'preference': preference(q),
+                    'quality': preference(q),
                 })
             self._sort_formats(formats)
         else:

youtube_dlc/extractor/picarto.py

@@ -78,7 +78,7 @@ class PicartoIE(InfoExtractor):
                         update_url_query(
                             'https://%s/hls/%s/index.m3u8'
                             % (edge_ep, channel_id), params),
-                        channel_id, 'mp4', preference=preference,
+                        channel_id, 'mp4', quality=preference,
                         m3u8_id='-'.join(format_id), fatal=False))
                     continue
                 elif tech_type == 'video/mp4' or tech_label == 'MP4':
@@ -88,7 +88,7 @@ class PicartoIE(InfoExtractor):
                             'https://%s/mp4/%s.mp4' % (edge_ep, channel_id),
                             params),
                         'format_id': '-'.join(format_id),
-                        'preference': preference,
+                        'quality': preference,
                     })
                 else:
                     # rtmp format does not seem to work

youtube_dlc/extractor/pinterest.py

@@ -54,8 +54,7 @@ class PinterestBaseIE(InfoExtractor):
                         'height': int_or_none(format_dict.get('height')),
                         'duration': duration,
                     })
-            self._sort_formats(
+            self._sort_formats(formats)
-                formats, field_preference=('height', 'width', 'tbr', 'format_id'))
 
         description = data.get('description') or data.get('description_html') or data.get('seo_description')
         timestamp = unified_timestamp(data.get('created_at'))

youtube_dlc/extractor/playtvak.py

@@ -150,7 +150,7 @@ class PlaytvakIE(InfoExtractor):
                 ext = 'mp4'
                 # Some streams have mp3 audio which does not play
                 # well with ffmpeg filter aac_adtstoasc
-                preference = -1
+                preference = -10
             elif format_ == 'adobe':  # f4m manifest fails with 404 in 80% of requests
                 continue
             else:  # Other formats not supported yet

youtube_dlc/extractor/qqmusic.py

@@ -121,7 +121,7 @@ class QQMusicIE(InfoExtractor):
                        % (details['prefix'], mid, details['ext'], vkey, guid),
                 'format': format_id,
                 'format_id': format_id,
-                'preference': details['preference'],
+                'quality': details['preference'],
                 'abr': details.get('abr'),
             })
         self._check_formats(formats, mid)

youtube_dlc/extractor/radiofrance.py

@@ -43,7 +43,7 @@ class RadioFranceIE(InfoExtractor):
                 'format_id': fm[0],
                 'url': fm[1],
                 'vcodec': 'none',
-                'preference': i,
+                'quality': i,
             }
             for i, fm in
             enumerate(re.findall(r"([a-z0-9]+)\s*:\s*'([^']+)'", formats_str))

youtube_dlc/extractor/reddit.py

@@ -40,7 +40,7 @@ class RedditIE(InfoExtractor):
             'https://v.redd.it/%s/DASHPlaylist.mpd' % video_id, video_id,
             mpd_id='dash', fatal=False))
 
-        self._sort_formats(formats, ('height', 'width'))
+        self._sort_formats(formats)
 
         return {
             'id': video_id,

youtube_dlc/extractor/rtl2.py

@@ -93,7 +93,7 @@ class RTL2IE(InfoExtractor):
                 'flash_version': 'LNX 11,2,202,429',
                 'rtmp_conn': rtmp_conn,
                 'no_resume': True,
-                'preference': 1,
+                'quality': 1,
             })
 
         m3u8_url = video_info.get('streamurl_hls')

youtube_dlc/extractor/rtp.py

@@ -45,7 +45,7 @@ class RTPIE(InfoExtractor):
             if file_key:
                 formats.append({
                     'url': 'https://cdn-ondemand.rtp.pt' + file_key,
-                    'preference': 1,
+                    'quality': 1,
                 })
             self._sort_formats(formats)
         else:

youtube_dlc/extractor/rutv.py

@@ -180,11 +180,11 @@ class RUTVIE(InfoExtractor):
                         'rtmp_live': True,
                         'ext': 'flv',
                         'vbr': int(quality),
-                        'preference': preference,
+                        'quality': preference,
                     }
                 elif transport == 'm3u8':
                     formats.extend(self._extract_m3u8_formats(
-                        url, video_id, 'mp4', preference=preference, m3u8_id='hls'))
+                        url, video_id, 'mp4', quality=preference, m3u8_id='hls'))
                     continue
                 else:
                     fmt = {

youtube_dlc/extractor/sendtonews.py

@@ -80,7 +80,9 @@ class SendtoNewsIE(InfoExtractor):
                     'format_id': '%s-%d' % (determine_protocol(f), tbr),
                     'tbr': tbr,
                 })
-            self._sort_formats(info_dict['formats'], ('tbr', 'height', 'width', 'format_id'))
+            # 'tbr' was explicitly set to be prefered over 'height' originally,
+            # So this is being kept unless someone can confirm this is unnecessary
+            self._sort_formats(info_dict['formats'], ('tbr', 'res'))
 
             thumbnails = []
             if video.get('thumbnailUrl'):

youtube_dlc/extractor/simplecast.py

@@ -0,0 +1,160 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_podcast_url,
+    int_or_none,
+    parse_iso8601,
+    strip_or_none,
+    try_get,
+    urlencode_postdata,
+)
+
+
+class SimplecastBaseIE(InfoExtractor):
+    _UUID_REGEX = r'[\da-f]{8}-(?:[\da-f]{4}-){3}[\da-f]{12}'
+    _API_BASE = 'https://api.simplecast.com/'
+
+    def _call_api(self, path_tmpl, video_id):
+        return self._download_json(
+            self._API_BASE + path_tmpl % video_id, video_id)
+
+    def _call_search_api(self, resource, resource_id, resource_url):
+        return self._download_json(
+            'https://api.simplecast.com/%ss/search' % resource, resource_id,
+            data=urlencode_postdata({'url': resource_url}))
+
+    def _parse_episode(self, episode):
+        episode_id = episode['id']
+        title = episode['title'].strip()
+        audio_file = episode.get('audio_file') or {}
+        audio_file_url = audio_file.get('url') or episode.get('audio_file_url') or episode['enclosure_url']
+
+        season = episode.get('season') or {}
+        season_href = season.get('href')
+        season_id = None
+        if season_href:
+            season_id = self._search_regex(
+                r'https?://api.simplecast.com/seasons/(%s)' % self._UUID_REGEX,
+                season_href, 'season id', default=None)
+
+        webpage_url = episode.get('episode_url')
+        channel_url = None
+        if webpage_url:
+            channel_url = self._search_regex(
+                r'(https?://[^/]+\.simplecast\.com)',
+                webpage_url, 'channel url', default=None)
+
+        return {
+            'id': episode_id,
+            'display_id': episode.get('slug'),
+            'title': title,
+            'url': clean_podcast_url(audio_file_url),
+            'webpage_url': webpage_url,
+            'channel_url': channel_url,
+            'series': try_get(episode, lambda x: x['podcast']['title']),
+            'season_number': int_or_none(season.get('number')),
+            'season_id': season_id,
+            'thumbnail': episode.get('image_url'),
+            'episode_id': episode_id,
+            'episode_number': int_or_none(episode.get('number')),
+            'description': strip_or_none(episode.get('description')),
+            'timestamp': parse_iso8601(episode.get('published_at')),
+            'duration': int_or_none(episode.get('duration')),
+            'filesize': int_or_none(audio_file.get('size') or episode.get('audio_file_size')),
+        }
+
+
+class SimplecastIE(SimplecastBaseIE):
+    IE_NAME = 'simplecast'
+    _VALID_URL = r'https?://(?:api\.simplecast\.com/episodes|player\.simplecast\.com)/(?P<id>%s)' % SimplecastBaseIE._UUID_REGEX
+    _COMMON_TEST_INFO = {
+        'display_id': 'errant-signal-chris-franklin-new-wave-video-essays',
+        'id': 'b6dc49a2-9404-4853-9aa9-9cfc097be876',
+        'ext': 'mp3',
+        'title': 'Errant Signal - Chris Franklin & New Wave Video Essays',
+        'episode_number': 1,
+        'episode_id': 'b6dc49a2-9404-4853-9aa9-9cfc097be876',
+        'description': 'md5:34752789d3d2702e2d2c975fbd14f357',
+        'season_number': 1,
+        'season_id': 'e23df0da-bae4-4531-8bbf-71364a88dc13',
+        'series': 'The RE:BIND.io Podcast',
+        'duration': 5343,
+        'timestamp': 1580979475,
+        'upload_date': '20200206',
+        'webpage_url': r're:^https?://the-re-bind-io-podcast\.simplecast\.com/episodes/errant-signal-chris-franklin-new-wave-video-essays',
+        'channel_url': r're:^https?://the-re-bind-io-podcast\.simplecast\.com$',
+    }
+    _TESTS = [{
+        'url': 'https://api.simplecast.com/episodes/b6dc49a2-9404-4853-9aa9-9cfc097be876',
+        'md5': '8c93be7be54251bf29ee97464eabe61c',
+        'info_dict': _COMMON_TEST_INFO,
+    }, {
+        'url': 'https://player.simplecast.com/b6dc49a2-9404-4853-9aa9-9cfc097be876',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _extract_urls(webpage):
+        return re.findall(
+            r'''(?x)<iframe[^>]+src=["\']
+                (
+                    https?://(?:embed\.simplecast\.com/[0-9a-f]{8}|
+                    player\.simplecast\.com/%s
+                ))''' % SimplecastBaseIE._UUID_REGEX, webpage)
+
+    def _real_extract(self, url):
+        episode_id = self._match_id(url)
+        episode = self._call_api('episodes/%s', episode_id)
+        return self._parse_episode(episode)
+
+
+class SimplecastEpisodeIE(SimplecastBaseIE):
+    IE_NAME = 'simplecast:episode'
+    _VALID_URL = r'https?://(?!api\.)[^/]+\.simplecast\.com/episodes/(?P<id>[^/?&#]+)'
+    _TEST = {
+        'url': 'https://the-re-bind-io-podcast.simplecast.com/episodes/errant-signal-chris-franklin-new-wave-video-essays',
+        'md5': '8c93be7be54251bf29ee97464eabe61c',
+        'info_dict': SimplecastIE._COMMON_TEST_INFO,
+    }
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        episode = self._call_search_api(
+            'episode', mobj.group(1), mobj.group(0))
+        return self._parse_episode(episode)
+
+
+class SimplecastPodcastIE(SimplecastBaseIE):
+    IE_NAME = 'simplecast:podcast'
+    _VALID_URL = r'https?://(?!(?:api|cdn|embed|feeds|player)\.)(?P<id>[^/]+)\.simplecast\.com(?!/episodes/[^/?&#]+)'
+    _TESTS = [{
+        'url': 'https://the-re-bind-io-podcast.simplecast.com',
+        'playlist_mincount': 33,
+        'info_dict': {
+            'id': '07d28d26-7522-42eb-8c53-2bdcfc81c43c',
+            'title': 'The RE:BIND.io Podcast',
+        },
+    }, {
+        'url': 'https://the-re-bind-io-podcast.simplecast.com/episodes',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        subdomain = self._match_id(url)
+        site = self._call_search_api('site', subdomain, url)
+        podcast = site['podcast']
+        podcast_id = podcast['id']
+        podcast_title = podcast.get('title')
+
+        def entries():
+            episodes = self._call_api('podcasts/%s/episodes', podcast_id)
+            for episode in (episodes.get('collection') or []):
+                info = self._parse_episode(episode)
+                info['series'] = podcast_title
+                yield info
+
+        return self.playlist_result(entries(), podcast_id, podcast_title)

youtube_dlc/extractor/sina.py

@@ -99,7 +99,7 @@ class SinaIE(InfoExtractor):
                 formats.append({
                     'format_id': quality_id,
                     'url': update_url_query(file_api, {'vid': file_id}),
-                    'preference': preference(quality_id),
+                    'quality': preference(quality_id),
                     'ext': 'mp4',
                 })
             self._sort_formats(formats)

youtube_dlc/extractor/soundcloud.py

@@ -422,7 +422,7 @@ class SoundcloudIE(InfoExtractor):
                         'ext': urlhandle_detect_ext(urlh) or 'mp3',
                         'filesize': int_or_none(urlh.headers.get('Content-Length')),
                         'url': format_url,
-                        'preference': 10,
+                        'quality': 10,
                     })
 
         def invalid_url(url):

youtube_dlc/extractor/spankbang.py

@@ -129,7 +129,7 @@ class SpankBangIE(InfoExtractor):
                     format_url = format_url[0]
                 extract_format(format_id, format_url)
 
-        self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'tbr', 'format_id'))
+        self._sort_formats(formats)
 
         info = self._search_json_ld(webpage, video_id, default={})
 

youtube_dlc/extractor/spankwire.py

@@ -108,7 +108,7 @@ class SpankwireIE(InfoExtractor):
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
                 m3u8_id='hls', fatal=False))
-        self._sort_formats(formats, ('height', 'tbr', 'width', 'format_id'))
+        self._sort_formats(formats)
 
         view_count = str_to_int(video.get('viewed'))
 

youtube_dlc/extractor/srgssr.py

@@ -89,7 +89,7 @@ class SRGSSRIE(InfoExtractor):
                     formats.append({
                         'format_id': format_id,
                         'url': asset_url,
-                        'preference': preference(quality),
+                        'quality': preference(quality),
                         'ext': 'flv' if protocol == 'RTMP' else None,
                     })
         self._sort_formats(formats)

youtube_dlc/extractor/storyfire.py

@@ -1,255 +1,151 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import itertools
+import functools
+
 from .common import InfoExtractor
+from ..utils import (
+    # HEADRequest,
+    int_or_none,
+    OnDemandPagedList,
+    smuggle_url,
+)
 
 
-class StoryFireIE(InfoExtractor):
+class StoryFireBaseIE(InfoExtractor):
-    _VALID_URL = r'(?:(?:https?://(?:www\.)?storyfire\.com/video-details)|(?:https://storyfire.app.link))/(?P<id>[^/\s]+)'
+    _VALID_URL_BASE = r'https?://(?:www\.)?storyfire\.com/'
-    _TESTS = [{
+
+    def _call_api(self, path, video_id, resource, query=None):
+        return self._download_json(
+            'https://storyfire.com/app/%s/%s' % (path, video_id), video_id,
+            'Downloading %s JSON metadata' % resource, query=query)
+
+    def _parse_video(self, video):
+        title = video['title']
+        vimeo_id = self._search_regex(
+            r'https?://player\.vimeo\.com/external/(\d+)',
+            video['vimeoVideoURL'], 'vimeo id')
+
+        # video_url = self._request_webpage(
+        #    HEADRequest(video['vimeoVideoURL']), video_id).geturl()
+        # formats = []
+        # for v_url, suffix in [(video_url, '_sep'), (video_url.replace('/sep/video/', '/video/'), '')]:
+        #    formats.extend(self._extract_m3u8_formats(
+        #        v_url, video_id, 'mp4', 'm3u8_native',
+        #        m3u8_id='hls' + suffix, fatal=False))
+        #    formats.extend(self._extract_mpd_formats(
+        #        v_url.replace('.m3u8', '.mpd'), video_id,
+        #        mpd_id='dash' + suffix, fatal=False))
+        # self._sort_formats(formats)
+
+        uploader_id = video.get('hostID')
+
+        return {
+            '_type': 'url_transparent',
+            'id': vimeo_id,
+            'title': title,
+            'description': video.get('description'),
+            'url': smuggle_url(
+                'https://player.vimeo.com/video/' + vimeo_id, {
+                    'http_headers': {
+                        'Referer': 'https://storyfire.com/',
+                    }
+                }),
+            # 'formats': formats,
+            'thumbnail': video.get('storyImage'),
+            'view_count': int_or_none(video.get('views')),
+            'like_count': int_or_none(video.get('likesCount')),
+            'comment_count': int_or_none(video.get('commentsCount')),
+            'duration': int_or_none(video.get('videoDuration')),
+            'timestamp': int_or_none(video.get('publishDate')),
+            'uploader': video.get('username'),
+            'uploader_id': uploader_id,
+            'uploader_url': 'https://storyfire.com/user/%s/video' % uploader_id if uploader_id else None,
+            'episode_number': int_or_none(video.get('episodeNumber') or video.get('episode_number')),
+        }
+
+
+class StoryFireIE(StoryFireBaseIE):
+    _VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'video-details/(?P<id>[0-9a-f]{24})'
+    _TEST = {
         'url': 'https://storyfire.com/video-details/5df1d132b6378700117f9181',
-        'md5': '560953bfca81a69003cfa5e53ac8a920',
+        'md5': 'caec54b9e4621186d6079c7ec100c1eb',
         'info_dict': {
-            'id': '5df1d132b6378700117f9181',
+            'id': '378954662',
             'ext': 'mp4',
             'title': 'Buzzfeed Teaches You About Memes',
             'uploader_id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
             'timestamp': 1576129028,
-            'description': 'Mocking Buzzfeed\'s meme lesson. Reuploaded from YouTube because of their new policies',
+            'description': 'md5:0b4e28021548e144bed69bb7539e62ea',
             'uploader': 'whang!',
             'upload_date': '20191212',
+            'duration': 418,
+            'view_count': int,
+            'like_count': int,
+            'comment_count': int,
         },
-        'params': {'format': 'bestvideo'}  # There are no merged formats in the playlist.
+        'params': {
-    }, {
+            'skip_download': True,
-        'url': 'https://storyfire.app.link/5GxAvWOQr8',  # Alternate URL format, with unrelated short ID
-        'md5': '7a2dc6d60c4889edfed459c620fe690d',
-        'info_dict': {
-            'id': '5f1e11ecd78a57b6c702001d',
-            'ext': 'm4a',
-            'title': 'Weird Nintendo Prototype Leaks',
-            'description': 'A stream taking a look at some weird Nintendo Prototypes with Luigi in Mario 64 and weird Yoshis',
-            'timestamp': 1595808576,
-            'upload_date': '20200727',
-            'uploader': 'whang!',
-            'uploader_id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
         },
-        'params': {'format': 'bestaudio'}  # Verifying audio extraction
+        'expected_warnings': ['Unable to download JSON metadata']
-
-    }]
-
-    _aformats = {
-        'audio-medium-audio': {'acodec': 'aac', 'abr': 125, 'preference': -10},
-        'audio-high-audio': {'acodec': 'aac', 'abr': 254, 'preference': -1},
     }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        video = self._call_api(
-
+            'generic/video-detail', video_id, 'video')['video']
-        # Extracting the json blob is mandatory to proceed with extraction.
+        return self._parse_video(video)
-        jsontext = self._html_search_regex(
-            r'<script id="__NEXT_DATA__" type="application/json">(.+?)</script>',
-            webpage, 'json_data')
-
-        json = self._parse_json(jsontext, video_id)
-
-        # The currentVideo field in the json is mandatory
-        # because it contains the only link to the m3u playlist
-        video = json['props']['initialState']['video']['currentVideo']
-        videourl = video['vimeoVideoURL']  # Video URL is mandatory
-
-        # Extract other fields from the json in an error tolerant fashion
-        # ID may be incorrect (on short URL format), correct it.
-        parsed_id = video.get('_id')
-        if parsed_id:
-            video_id = parsed_id
-
-        title = video.get('title')
-        description = video.get('description')
-
-        thumbnail = video.get('storyImage')
-        views = video.get('views')
-        likes = video.get('likesCount')
-        comments = video.get('commentsCount')
-        duration = video.get('videoDuration')
-        publishdate = video.get('publishDate')  # Apparently epoch time, day only
-
-        uploader = video.get('username')
-        uploader_id = video.get('hostID')
-        # Construct an uploader URL
-        uploader_url = None
-        if uploader_id:
-            uploader_url = "https://storyfire.com/user/%s/video" % uploader_id
-
-        # Collect root playlist to determine formats
-        formats = self._extract_m3u8_formats(
-            videourl, video_id, 'mp4', 'm3u8_native')
-
-        # Modify formats to fill in missing information about audio codecs
-        for format in formats:
-            aformat = self._aformats.get(format['format_id'])
-            if aformat:
-                format['acodec'] = aformat['acodec']
-                format['abr'] = aformat['abr']
-                format['preference'] = aformat['preference']
-                format['ext'] = 'm4a'
-
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': description,
-            'ext': "mp4",
-            'url': videourl,
-            'formats': formats,
-
-            'thumbnail': thumbnail,
-            'view_count': views,
-            'like_count': likes,
-            'comment_count': comments,
-            'duration': duration,
-            'timestamp': publishdate,
-
-            'uploader': uploader,
-            'uploader_id': uploader_id,
-            'uploader_url': uploader_url,
-
-        }
 
 
-class StoryFireUserIE(InfoExtractor):
+class StoryFireUserIE(StoryFireBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?storyfire\.com/user/(?P<id>[^/\s]+)/video'
+    _VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'user/(?P<id>[^/]+)/video'
-    _TESTS = [{
+    _TEST = {
-        'url': 'https://storyfire.com/user/ntZAJFECERSgqHSxzonV5K2E89s1/video',
-        'info_dict': {
-            'id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
-            'title': 'whang!',
-        },
-        'playlist_mincount': 18
-    }, {
         'url': 'https://storyfire.com/user/UQ986nFxmAWIgnkZQ0ftVhq4nOk2/video',
         'info_dict': {
             'id': 'UQ986nFxmAWIgnkZQ0ftVhq4nOk2',
-            'title': 'McJuggerNuggets',
         },
-        'playlist_mincount': 143
+        'playlist_mincount': 151,
+    }
+    _PAGE_SIZE = 20
 
-    }]
+    def _fetch_page(self, user_id, page):
-
+        videos = self._call_api(
-    # Generator for fetching playlist items
+            'publicVideos', user_id, 'page %d' % (page + 1), {
-    def _enum_videos(self, baseurl, user_id, firstjson):
+                'skip': page * self._PAGE_SIZE,
-        totalVideos = int(firstjson['videosCount'])
+            })['videos']
-        haveVideos = 0
+        for video in videos:
-        json = firstjson
+            yield self._parse_video(video)
-
-        for page in itertools.count(1):
-            for video in json['videos']:
-                id = video['_id']
-                url = "https://storyfire.com/video-details/%s" % id
-                haveVideos += 1
-                yield {
-                    '_type': 'url',
-                    'id': id,
-                    'url': url,
-                    'ie_key': 'StoryFire',
-
-                    'title': video.get('title'),
-                    'description': video.get('description'),
-                    'view_count': video.get('views'),
-                    'comment_count': video.get('commentsCount'),
-                    'duration': video.get('videoDuration'),
-                    'timestamp': video.get('publishDate'),
-                }
-            # Are there more pages we could fetch?
-            if haveVideos < totalVideos:
-                pageurl = baseurl + ("%i" % haveVideos)
-                json = self._download_json(pageurl, user_id,
-                                           note='Downloading page %s' % page)
-
-                # Are there any videos in the new json?
-                videos = json.get('videos')
-                if not videos or len(videos) == 0:
-                    break  # no videos
-
-            else:
-                break  # We have fetched all the videos, stop
 
     def _real_extract(self, url):
         user_id = self._match_id(url)
-
+        entries = OnDemandPagedList(functools.partial(
-        baseurl = "https://storyfire.com/app/publicVideos/%s?skip=" % user_id
+            self._fetch_page, user_id), self._PAGE_SIZE)
-
+        return self.playlist_result(entries, user_id)
-        # Download first page to ensure it can be downloaded, and get user information if available.
-        firstpage = baseurl + "0"
-        firstjson = self._download_json(firstpage, user_id)
-
-        title = None
-        videos = firstjson.get('videos')
-        if videos and len(videos):
-            title = videos[1].get('username')
-
-        return {
-            '_type': 'playlist',
-            'entries': self._enum_videos(baseurl, user_id, firstjson),
-            'id': user_id,
-            'title': title,
-        }
 
 
-class StoryFireSeriesIE(InfoExtractor):
+class StoryFireSeriesIE(StoryFireBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?storyfire\.com/write/series/stories/(?P<id>[^/\s]+)'
+    _VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'write/series/stories/(?P<id>[^/?&#]+)'
     _TESTS = [{
         'url': 'https://storyfire.com/write/series/stories/-Lq6MsuIHLODO6d2dDkr/',
         'info_dict': {
             'id': '-Lq6MsuIHLODO6d2dDkr',
         },
-        'playlist_mincount': 13
+        'playlist_mincount': 13,
     }, {
         'url': 'https://storyfire.com/write/series/stories/the_mortal_one/',
         'info_dict': {
             'id': 'the_mortal_one',
         },
-        'playlist_count': 0  # This playlist has entries, but no videos.
+        'playlist_count': 0,
-    }, {
-        'url': 'https://storyfire.com/write/series/stories/story_time',
-        'info_dict': {
-            'id': 'story_time',
-        },
-        'playlist_mincount': 10
     }]
 
-    # Generator for returning playlist items
+    def _extract_videos(self, stories):
-    # This object is substantially different than the one in the user videos page above
+        for story in stories.values():
-    def _enum_videos(self, jsonlist):
+            if story.get('hasVideo'):
-        for video in jsonlist:
+                yield self._parse_video(story)
-            id = video['_id']
-            if video.get('hasVideo'):  # Boolean element
-                url = "https://storyfire.com/video-details/%s" % id
-                yield {
-                    '_type': 'url',
-                    'id': id,
-                    'url': url,
-                    'ie_key': 'StoryFire',
-
-                    'title': video.get('title'),
-                    'description': video.get('description'),
-                    'view_count': video.get('views'),
-                    'likes_count': video.get('likesCount'),
-                    'comment_count': video.get('commentsCount'),
-                    'duration': video.get('videoDuration'),
-                    'timestamp': video.get('publishDate'),
-                }
 
     def _real_extract(self, url):
-        list_id = self._match_id(url)
+        series_id = self._match_id(url)
-
+        stories = self._call_api(
-        listurl = "https://storyfire.com/app/seriesStories/%s/list" % list_id
+            'seriesStories', series_id, 'series stories')
-        json = self._download_json(listurl, list_id)
+        return self.playlist_result(self._extract_videos(stories), series_id)
-
-        return {
-            '_type': 'playlist',
-            'entries': self._enum_videos(json),
-            'id': list_id
-        }

youtube_dlc/extractor/tele13.py

@@ -70,7 +70,7 @@ class Tele13IE(InfoExtractor):
                     formats.append({
                         'url': format_url,
                         'format_id': f.get('label'),
-                        'preference': preference(f.get('label')),
+                        'quality': preference(f.get('label')),
                         'ext': ext,
                     })
                 urls.append(format_url)

youtube_dlc/extractor/telemb.py

@@ -57,7 +57,7 @@ class TeleMBIE(InfoExtractor):
                     'app': rtmp.group('app'),
                     'player_url': 'http://p.jwpcdn.com/6/10/jwplayer.flash.swf',
                     'page_url': 'http://www.telemb.be',
-                    'preference': -1,
+                    'preference': -10,
                 })
             formats.append(fmt)
         self._sort_formats(formats)

youtube_dlc/extractor/threeqsdn.py

@@ -133,12 +133,10 @@ class ThreeQSDNIE(InfoExtractor):
                         'vcodec': 'none' if height == 0 else None,
                         'width': width,
                     })
-        for f in formats:
+        # It seems like this would be correctly handled by default
-            if f.get('acodec') == 'none':
+        # However, unless someone can confirm this, the old
-                f['preference'] = -40
+        # behaviour is being kept as-is
-            elif f.get('vcodec') == 'none':
+        self._sort_formats(formats, ('res', 'source_preference'))
-                f['preference'] = -50
-        self._sort_formats(formats, ('preference', 'width', 'height', 'source_preference', 'tbr', 'vbr', 'abr', 'ext', 'format_id'))
 
         subtitles = {}
         for subtitle in (config.get('subtitles') or []):

youtube_dlc/extractor/udemy.py

@@ -405,7 +405,7 @@ class UdemyIE(InfoExtractor):
                 if f.get('url'):
                     formats.append(f)
 
-        self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+        self._sort_formats(formats)
 
         return {
             'id': video_id,

youtube_dlc/extractor/umg.py

@@ -91,7 +91,7 @@ class UMGDeIE(InfoExtractor):
         if not formats:
             for format_id in (867, 836, 940):
                 add_m3u8_format(format_id)
-        self._sort_formats(formats, ('width', 'height', 'filesize', 'tbr'))
+        self._sort_formats(formats)
 
         return {
             'id': video_id,

youtube_dlc/extractor/uol.py

@@ -110,7 +110,6 @@ class UOLIE(InfoExtractor):
                 'format_id': format_id,
                 'url': f_url,
                 'quality': quality(format_id),
-                'preference': -1,
             })
         self._sort_formats(formats)
 

youtube_dlc/extractor/urort.py

@@ -44,7 +44,7 @@ class UrortIE(InfoExtractor):
                 'ext': f['FileType'],
                 'format_id': '%s-%s' % (f['FileType'], f.get('Quality', '')),
                 'url': 'http://p3urort.blob.core.windows.net/tracks/%s' % f['FileRef'],
-                'preference': 3 if f['FileType'] == 'mp3' else 2,
+                'quality': 3 if f['FileType'] == 'mp3' else 2,
             } for f in s['Files']]
             self._sort_formats(formats)
             e = {

youtube_dlc/extractor/videopress.py

@@ -4,21 +4,22 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_str
 from ..utils import (
     determine_ext,
     float_or_none,
+    int_or_none,
     parse_age_limit,
     qualities,
     random_birthday,
-    try_get,
     unified_timestamp,
     urljoin,
 )
 
 
 class VideoPressIE(InfoExtractor):
-    _VALID_URL = r'https?://videopress\.com/embed/(?P<id>[\da-zA-Z]+)'
+    _ID_REGEX = r'[\da-zA-Z]{8}'
+    _PATH_REGEX = r'video(?:\.word)?press\.com/embed/'
+    _VALID_URL = r'https?://%s(?P<id>%s)' % (_PATH_REGEX, _ID_REGEX)
     _TESTS = [{
         'url': 'https://videopress.com/embed/kUJmAcSf',
         'md5': '706956a6c875873d51010921310e4bc6',
@@ -36,35 +37,36 @@ class VideoPressIE(InfoExtractor):
         # 17+, requires birth_* params
         'url': 'https://videopress.com/embed/iH3gstfZ',
         'only_matching': True,
+    }, {
+        'url': 'https://video.wordpress.com/embed/kUJmAcSf',
+        'only_matching': True,
     }]
 
     @staticmethod
     def _extract_urls(webpage):
         return re.findall(
-            r'<iframe[^>]+src=["\']((?:https?://)?videopress\.com/embed/[\da-zA-Z]+)',
+            r'<iframe[^>]+src=["\']((?:https?://)?%s%s)' % (VideoPressIE._PATH_REGEX, VideoPressIE._ID_REGEX),
             webpage)
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
         query = random_birthday('birth_year', 'birth_month', 'birth_day')
+        query['fields'] = 'description,duration,file_url_base,files,height,original,poster,rating,title,upload_date,width'
         video = self._download_json(
             'https://public-api.wordpress.com/rest/v1.1/videos/%s' % video_id,
             video_id, query=query)
 
         title = video['title']
 
-        def base_url(scheme):
+        file_url_base = video.get('file_url_base') or {}
-            return try_get(
+        base_url = file_url_base.get('https') or file_url_base.get('http')
-                video, lambda x: x['file_url_base'][scheme], compat_str)
-
-        base_url = base_url('https') or base_url('http')
 
         QUALITIES = ('std', 'dvd', 'hd')
         quality = qualities(QUALITIES)
 
         formats = []
-        for format_id, f in video['files'].items():
+        for format_id, f in (video.get('files') or {}).items():
             if not isinstance(f, dict):
                 continue
             for ext, path in f.items():
@@ -75,12 +77,14 @@ class VideoPressIE(InfoExtractor):
                         'ext': determine_ext(path, ext),
                         'quality': quality(format_id),
                     })
-        original_url = try_get(video, lambda x: x['original'], compat_str)
+        original_url = video.get('original')
         if original_url:
             formats.append({
                 'url': original_url,
                 'format_id': 'original',
                 'quality': len(QUALITIES),
+                'width': int_or_none(video.get('width')),
+                'height': int_or_none(video.get('height')),
             })
         self._sort_formats(formats)
 

youtube_dlc/extractor/vidme.py

@@ -181,6 +181,7 @@ class VidmeIE(InfoExtractor):
                     'url': format_url,
                     'width': int_or_none(f.get('width')),
                     'height': int_or_none(f.get('height')),
+                    # Clips should never be prefered over full video
                     'preference': 0 if f.get('type', '').endswith(
                         'clip') else 1,
                 })

youtube_dlc/extractor/viewlift.py

@@ -134,7 +134,7 @@ class ViewLiftEmbedIE(ViewLiftBaseIE):
         if hls_url:
             formats.extend(self._extract_m3u8_formats(
                 hls_url, film_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
-        self._sort_formats(formats, ('height', 'tbr', 'format_id'))
+        self._sort_formats(formats)
 
         info = {
             'id': film_id,

youtube_dlc/extractor/viki.py

@@ -22,6 +22,7 @@ from ..utils import (
     parse_iso8601,
     sanitized_Request,
     std_headers,
+    try_get,
 )
 
 
@@ -31,7 +32,7 @@ class VikiBaseIE(InfoExtractor):
     _API_URL_TEMPLATE = 'https://api.viki.io%s&sig=%s'
 
     _APP = '100005a'
-    _APP_VERSION = '2.2.5.1428709186'
+    _APP_VERSION = '6.0.0'
     _APP_SECRET = 'MM_d*yP@`&1@]@!AVrXf_o-HVEnoTnm$O-ti4[G~$JDI/Dc-&piU&z&5.;:}95=Iad'
 
     _GEO_BYPASS = False
@@ -42,7 +43,7 @@ class VikiBaseIE(InfoExtractor):
     _ERRORS = {
         'geo': 'Sorry, this content is not available in your region.',
         'upcoming': 'Sorry, this content is not yet available.',
-        # 'paywall': 'paywall',
+        'paywall': 'Sorry, this content is only available to Viki Pass Plus subscribers',
     }
 
     def _prepare_call(self, path, timestamp=None, post_data=None):
@@ -63,14 +64,25 @@ class VikiBaseIE(InfoExtractor):
 
     def _call_api(self, path, video_id, note, timestamp=None, post_data=None):
         resp = self._download_json(
-            self._prepare_call(path, timestamp, post_data), video_id, note)
+            self._prepare_call(path, timestamp, post_data),
+            video_id, note,
+            headers={
+                'x-client-user-agent': std_headers['User-Agent'],
+                'x-viki-as-id': self._APP,
+                'x-viki-app-ver': self._APP_VERSION,
+            })
 
         error = resp.get('error')
         if error:
             if error == 'invalid timestamp':
                 resp = self._download_json(
                     self._prepare_call(path, int(resp['current_timestamp']), post_data),
-                    video_id, '%s (retry)' % note)
+                    video_id, '%s (retry)' % note,
+                    headers={
+                        'x-client-user-agent': std_headers['User-Agent'],
+                        'x-viki-as-id': self._APP,
+                        'x-viki-app-ver': self._APP_VERSION,
+                    })
                 error = resp.get('error')
             if error:
                 self._raise_error(resp['error'])
@@ -83,11 +95,13 @@ class VikiBaseIE(InfoExtractor):
             expected=True)
 
     def _check_errors(self, data):
-        for reason, status in data.get('blocking', {}).items():
+        for reason, status in (data.get('blocking') or {}).items():
             if status and reason in self._ERRORS:
                 message = self._ERRORS[reason]
                 if reason == 'geo':
                     self.raise_geo_restricted(msg=message)
+                elif reason == 'paywall':
+                    self.raise_login_required(message)
                 raise ExtractorError('%s said: %s' % (
                     self.IE_NAME, message), expected=True)
 
@@ -132,13 +146,19 @@ class VikiIE(VikiBaseIE):
         'info_dict': {
             'id': '1023585v',
             'ext': 'mp4',
-            'title': 'Heirs Episode 14',
+            'title': 'Heirs - Episode 14',
-            'uploader': 'SBS',
+            'uploader': 'SBS Contents Hub',
-            'description': 'md5:c4b17b9626dd4b143dcc4d855ba3474e',
+            'timestamp': 1385047627,
             'upload_date': '20131121',
             'age_limit': 13,
+            'duration': 3570,
+            'episode_number': 14,
+        },
+        'params': {
+            'format': 'bestvideo',
         },
         'skip': 'Blocked in the US',
+        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
     }, {
         # clip
         'url': 'http://www.viki.com/videos/1067139v-the-avengers-age-of-ultron-press-conference',
@@ -154,7 +174,8 @@ class VikiIE(VikiBaseIE):
             'uploader': 'Arirang TV',
             'like_count': int,
             'age_limit': 0,
-        }
+        },
+        'skip': 'Sorry. There was an error loading this video',
     }, {
         'url': 'http://www.viki.com/videos/1048879v-ankhon-dekhi',
         'info_dict': {
@@ -172,7 +193,7 @@ class VikiIE(VikiBaseIE):
     }, {
         # episode
         'url': 'http://www.viki.com/videos/44699v-boys-over-flowers-episode-1',
-        'md5': '94e0e34fd58f169f40c184f232356cfe',
+        'md5': '0a53dc252e6e690feccd756861495a8c',
         'info_dict': {
             'id': '44699v',
             'ext': 'mp4',
@@ -184,6 +205,10 @@ class VikiIE(VikiBaseIE):
             'uploader': 'group8',
             'like_count': int,
             'age_limit': 13,
+            'episode_number': 1,
+        },
+        'params': {
+            'format': 'bestvideo',
         },
         'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
     }, {
@@ -210,7 +235,7 @@ class VikiIE(VikiBaseIE):
     }, {
         # non-English description
         'url': 'http://www.viki.com/videos/158036v-love-in-magic',
-        'md5': 'adf9e321a0ae5d0aace349efaaff7691',
+        'md5': '41faaba0de90483fb4848952af7c7d0d',
         'info_dict': {
             'id': '158036v',
             'ext': 'mp4',
@@ -221,6 +246,10 @@ class VikiIE(VikiBaseIE):
             'title': 'Love In Magic',
             'age_limit': 13,
         },
+        'params': {
+            'format': 'bestvideo',
+        },
+        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
     }]
 
     def _real_extract(self, url):
@@ -230,29 +259,27 @@ class VikiIE(VikiBaseIE):
             'https://www.viki.com/api/videos/' + video_id,
             video_id, 'Downloading video JSON', headers={
                 'x-client-user-agent': std_headers['User-Agent'],
-                'x-viki-app-ver': '4.0.57',
+                'x-viki-as-id': self._APP,
+                'x-viki-app-ver': self._APP_VERSION,
             })
         video = resp['video']
 
         self._check_errors(video)
 
         title = self.dict_selection(video.get('titles', {}), 'en', allow_fallback=False)
+        episode_number = int_or_none(video.get('number'))
         if not title:
-            title = 'Episode %d' % video.get('number') if video.get('type') == 'episode' else video.get('id') or video_id
+            title = 'Episode %d' % episode_number if video.get('type') == 'episode' else video.get('id') or video_id
-            container_titles = video.get('container', {}).get('titles', {})
+            container_titles = try_get(video, lambda x: x['container']['titles'], dict) or {}
             container_title = self.dict_selection(container_titles, 'en')
             title = '%s - %s' % (container_title, title)
 
         description = self.dict_selection(video.get('descriptions', {}), 'en')
 
-        duration = int_or_none(video.get('duration'))
+        like_count = int_or_none(try_get(video, lambda x: x['likes']['count']))
-        timestamp = parse_iso8601(video.get('created_at'))
-        uploader = video.get('author')
-        like_count = int_or_none(video.get('likes', {}).get('count'))
-        age_limit = parse_age_limit(video.get('rating'))
 
         thumbnails = []
-        for thumbnail_id, thumbnail in video.get('images', {}).items():
+        for thumbnail_id, thumbnail in (video.get('images') or {}).items():
             thumbnails.append({
                 'id': thumbnail_id,
                 'url': thumbnail.get('url'),
@@ -263,7 +290,12 @@ class VikiIE(VikiBaseIE):
             # New way to fetch subtitles
             new_video = self._download_json(
                 'https://www.viki.com/api/videos/%s' % video_id, video_id,
-                'Downloading new video JSON to get subtitles', fatal=False)
+                'Downloading new video JSON to get subtitles', fatal=False,
+                headers={
+                    'x-client-user-agent': std_headers['User-Agent'],
+                    'x-viki-as-id': self._APP,
+                    'x-viki-app-ver': self._APP_VERSION,
+                })
             for sub in new_video.get('streamSubtitles').get('dash'):
                 subtitles[sub.get('srclang')] = [{
                     'ext': 'vtt',
@@ -272,7 +304,7 @@ class VikiIE(VikiBaseIE):
                 }]
         except AttributeError:
             # fall-back to the old way if there isn't a streamSubtitles attribute
-            for subtitle_lang, _ in video.get('subtitle_completions', {}).items():
+            for subtitle_lang, _ in (video.get('subtitle_completions') or {}).items():
                 subtitles[subtitle_lang] = [{
                     'ext': subtitles_format,
                     'url': self._prepare_call(
@@ -283,13 +315,15 @@ class VikiIE(VikiBaseIE):
             'id': video_id,
             'title': title,
             'description': description,
-            'duration': duration,
+            'duration': int_or_none(video.get('duration')),
-            'timestamp': timestamp,
+            'timestamp': parse_iso8601(video.get('created_at')),
-            'uploader': uploader,
+            'uploader': video.get('author'),
+            'uploader_url': video.get('author_url'),
             'like_count': like_count,
-            'age_limit': age_limit,
+            'age_limit': parse_age_limit(video.get('rating')),
             'thumbnails': thumbnails,
             'subtitles': subtitles,
+            'episode_number': episode_number,
         }
 
         formats = []
@@ -383,7 +417,7 @@ class VikiChannelIE(VikiBaseIE):
         'info_dict': {
             'id': '50c',
             'title': 'Boys Over Flowers',
-            'description': 'md5:ecd3cff47967fe193cff37c0bec52790',
+            'description': 'md5:804ce6e7837e1fd527ad2f25420f4d59',
         },
         'playlist_mincount': 71,
     }, {
@@ -394,6 +428,7 @@ class VikiChannelIE(VikiBaseIE):
             'description': 'md5:05bf5471385aa8b21c18ad450e350525',
         },
         'playlist_count': 127,
+        'skip': 'Page not found',
     }, {
         'url': 'http://www.viki.com/news/24569c-showbiz-korea',
         'only_matching': True,

youtube_dlc/extractor/vimeo.py

@@ -116,7 +116,8 @@ class VimeoBaseInfoExtractor(InfoExtractor):
     def _vimeo_sort_formats(self, formats):
         # Bitrates are completely broken. Single m3u8 may contain entries in kbps and bps
         # at the same time without actual units specified. This lead to wrong sorting.
-        self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'tbr', 'format_id'))
+        # But since yt-dlp prefers 'res,fps' anyway, 'field_preference' is not needed
+        self._sort_formats(formats)
 
     def _parse_config(self, config, video_id):
         video_data = config['video']
@@ -178,16 +179,9 @@ class VimeoBaseInfoExtractor(InfoExtractor):
             formats.append({
                 'format_id': 'live-archive-source',
                 'url': live_archive_source_url,
-                'preference': 1,
+                'quality': 10,
             })
 
-        # Reduntant code! This is already done in common.py
-        # for f in formats:
-        #     if f.get('vcodec') == 'none':
-        #         f['preference'] = -50
-        #     elif f.get('acodec') == 'none':
-        #         f['preference'] = -40
-
         subtitles = {}
         text_tracks = config['request'].get('text_tracks')
         if text_tracks:
@@ -227,10 +221,12 @@ class VimeoBaseInfoExtractor(InfoExtractor):
             'is_live': is_live,
         }
 
-    def _extract_original_format(self, url, video_id):
+    def _extract_original_format(self, url, video_id, unlisted_hash=None):
+        query = {'action': 'load_download_config'}
+        if unlisted_hash:
+            query['unlisted_hash'] = unlisted_hash
         download_data = self._download_json(
-            url, video_id, fatal=False,
+            url, video_id, fatal=False, query=query,
-            query={'action': 'load_download_config'},
             headers={'X-Requested-With': 'XMLHttpRequest'})
         if download_data:
             source_file = download_data.get('source_file')
@@ -250,7 +246,7 @@ class VimeoBaseInfoExtractor(InfoExtractor):
                             'height': int_or_none(source_file.get('height')),
                             'filesize': parse_filesize(source_file.get('size')),
                             'format_id': source_name,
-                            'preference': 1,
+                            'quality': 1,
                         }
 
 
@@ -510,6 +506,11 @@ class VimeoIE(VimeoBaseInfoExtractor):
         {
             'url': 'https://vimeo.com/160743502/abd0e13fb4',
             'only_matching': True,
+        },
+        {
+            # requires passing unlisted_hash(a52724358e) to load_download_config request
+            'url': 'https://vimeo.com/392479337/a52724358e',
+            'only_matching': True,
         }
         # https://gettingthingsdone.com/workflowmap/
         # vimeo embed with check-password page protected by Referer header
@@ -674,7 +675,8 @@ class VimeoIE(VimeoBaseInfoExtractor):
             if config.get('view') == 4:
                 config = self._verify_player_video_password(redirect_url, video_id, headers)
 
-        vod = config.get('video', {}).get('vod', {})
+        video = config.get('video') or {}
+        vod = video.get('vod') or {}
 
         def is_rented():
             if '>You rented this title.<' in webpage:
@@ -734,7 +736,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
         formats = []
 
         source_format = self._extract_original_format(
-            'https://vimeo.com/' + video_id, video_id)
+            'https://vimeo.com/' + video_id, video_id, video.get('unlisted_hash'))
         if source_format:
             formats.append(source_format)
 

youtube_dlc/extractor/vzaar.py

@@ -70,7 +70,7 @@ class VzaarIE(InfoExtractor):
             f = {
                 'url': source_url,
                 'format_id': 'http',
-                'preference': 1,
+                'quality': 1,
             }
             if 'audio' in source_url:
                 f.update({