Revision history for StreamFinder 1.00 Date/time First version, released on an unsuspecting world. 1.01 2018/08/10 - Fixed issue with some Facebook videos on Youtube. - Removed Tunein as a valid site (can no longer extract streams). 1.02 2019/01/31 - Make StreamFinder::Youtube the default and handle pretty much anything youtube-dl handles, ie. brighteon, vimeo, etc. 1.03 2019/08/15 - Fix StreamFinder::Reciva now failing to connect to server (added wget as fallthrough as LWP::UserAgent and curl fail with unsupported SSL protocol errors). - Fix StreamFinder.IHeartRadio now failing to grab images / icons. - Strip off "?autoplay=true" from StreamFinder::Youtube urls. - Minor doc fixes. 1.04 NOT RELEASED - Make debug option work for all stream types, and allow specifying as either "debug", "-debug", "debug => 1", or "-debug => 1" in parameter list for new() (ie. my $station = new StreamFinder($url, -debug => 1); - Better handle ".m3u8" streams returned by brighteon.com (Youtube). 1.05 2019/09/03 - Add StreamFinder::BannedVideo (https://banned.video/) to handle their video streams (which youtube-dl doesn't currently handle). 1.10 2019/09/07 - Add user-configuration files (~/.config/StreamFinder/config, ~/.config/StreamFinder//config (see docs). User can now specify / override most LWP::UserAgent options, ie. "agent", "timeout", etc. in the new configuration files. User must create these files, if needed with a text editor. - Greatly reduce falling back to wget by adding default user-agent and checking for wget availability. Specify an "agent" => "your agent" line in the user-configuration file to change. - Add "-random" and "-noplaylists" options to $station->getURL() (see docs) - StreamFinder::BannedVideo: Add "-keep" option to better control type and order of streams returned (see docs). - StreamFinder::Facebook: Depreciated ~/.config/.fbdata for new user-configuration files. - StreamFinder::IHeartRadio: Add podcast-playing capability. - StreamFinder::IHeartRadio: Depreciated old stream order options, replacing them with new -keep and -skip arguments and configuration file entries. - StreamFinder::RadioNet: (https://www.radio.net/) ADDED - StreamFinder::Tunein: RESTORED (now uses StreamFinder::Youtube for stream fetching), but adds metadata loading (which youtube-dl does not currently fetch for these streams. - StreamFinder::Youtube: Better handle "chunky" brighteon.com .m3u8 video streams. Also add "-notitle" option, streamlining use by StreamFinder::Tunein (see docs). 1.11 2019/09/07 - StreamFinder::Youtube: bugfix for brighteon.com m3u8 video URLs. - Docs: fix a cpl. typos, actually include the updated README file. 1.12 2019/09/10 - StreamFinder::Tunein: Add podcast-playing capability. (Will look for "playUrl" before falling back to youtube-dl for stream). - StreamFinder::Reciva: Fix to make LWP::Useragent fetch reciva URLs using the ANCIENT TLSv1 to allow to work without falling back to wget. See the StreamFinder::Reciva docs for creating a one-line custom config file to do this - I didn't make it the default since they could FIX this at ANY time, which would break things! - Add URI::Escape and HTML::Entities as DEPENDENCIES - in order to ensure that the returned stream titles are un-escaped. - Fix $station->getID('fcc') to work properly (return the station's FCC call-letters for sights and stations that support it. - Set $station->{artist} and $station->{year} where applicable. - Minor doc updates. - Fix some sight-specific submodules to properly accept just a station / video ID rather than a full URL (see docs). - Add "desc" option go getTitle() to fetch station's (long) description (for sites that provide that field), otherwise, the title is returned. 1.13 2019/09/11 - Fix missing (recently-added) file (RadioNet.pm) in MANIFEST file. - More doc. fixes, mostly to the individual submodules. 1.20 2019/09/16 - Removed usage and all references to WWW::Youtube::Download - This module no longer fetches the "author" (Youtube user) nor does anything else we can't more easily do manually. - StreamFinder::Apple: (podcasts.apple.com) ADDED! - StreamFinder::Brighteon: (brighteon.com) ADDED (split off from StreamFinder::Youtube). - StreamFinder::Vimeo: (vimeo.com) ADDED (split off from StreamFinder::Youtube. We separated these for better support - ie. ability to get the "author" fields and access some additional streams. We also can often avoid having to call youtube-dl. - StreamFinder::Youtube: Removed "notitle" option, added "fast" option, and add "description" field fetch to youtube-dl. Fixed url-fetching order to call youtube-dl FIRST to prevent infinite stall if called with an already-valid stream-url (Fauxdacious mediaplayer). StreamFinder::IHeartRadio - Fix sometimes-corrupted ID field. StreamFinder::Reciva - Fix failure to always capture ID field if given wrongly-formatted full url. - Add $station->{'genre'} field to IHeartRadio, RadioNet, Reciva, and Tunein (the sites that support it). - Various small code and doc. tweaks / cleanups. - DEBUG was accidently left set to ON, now turned OFF by default! 1.21 2019/09/18 - Significant documentation improvements & polishing. - StreamFinder::BannedVideo: Fix incorrect IDs returned. - StreamFinder::Tunein: (new()) - Fix mishandled URL if only station-ID specified (non-full URL). - StreamFinder::Apple: (new()) - Handle if only podcast-ID specified. 1.22 2019/10/07 - Minor bugfixes, namely handling double-quotes in titles in some submodules. - Minor doc. improvements. 1.23 2019/11/02 - StreamFinder::BannedVideo: Fix to handle change in their site - to execute extra step now needed for infowarsmedia.com videos and make the retrieved mp4 stream the "best" stream. 1.24 2019/11/06 - StreamFinder::Youtube: Eliminate express page scan for metadata for non-Youtube sites processed to prevent ocassional infinite loop caused by attempting to scan a media stream instead of a website. - Unescape HTML entities in the Genre string in Reciva & Tunein. - Minor doc. improvements. 1.25 2019/11/08 - StreamFinder::Brighteon: Fix another site-change that caused blank titles to be returned, and broaden title and description searching ability. 1.26 2019/11/19 - Correctly handle "HOME" directory in M#-Windows. - StreamFinder::IHeartRadio - Handle non-US site URLs. Change default "keep" priority to "secure_shoutcast, shoutcast, secure, any" for easier use by Fauxdacious. 1.27 2020/04/23 - Added StreamFinder::Spreaker for https://widget.spreaker.com podcasts. - Minor doc. improvements. 1.28 2020/04/25 - Fix failure to include StreamFinder::Spreaker in MANIFEST and to update change log. 1.29 2020/07/02 - Split out video's channel page (that was previously appended onto the "artist" field) into a new, separate "albumartist" field. This affects the video sites: Brighteon, Vimeo, and Youtube. (Old way was: artist:="Artist - Artist's video channel URL"). These are now put into separate fields. - Fix .pls playlists returned by some Tunein stations by stripping off some garbage Tunein appends in the format: "stream.pls?DIST=TuneIn&TGT=TuneIn&maxServers=..." and more accurately obtain title, artist, and album fields for stations vs podcasts. - Fix issue causing some vimeo videos to fail to fetch stream. - Clean up some "dirty" Spreaker titles (convert invalid characters). - Marked StreamFinder::Radionomy depreciated - site appears defunct. 1.30 2020/07/05 - Add "-nopls" option to getURL() to parse ".pls" playlists and return only a single entry, like "-noplaylists", but pass others (".m3u" and ".m3u8") through unchanged. This needed because most ".m3u*" are playable "HLS" streams, whereas ".pls" playlists are actually playlists and many players will not play them as-is. - StreamFinder::RadioNet: Fix issue with no streams being fetched for some stations & clean up titles a bit. - StreamFinder::IHeartRadio: Add year to podcasts, fix some title and description issues. - StreamFinder::Spreaker: Fix some streams not working, fix "-debug" option to work. 1.31 2020/08/10 - Add StreamFinder::Blogger (www.blogger.com) for Blogger videos. - Add StreamFinder::Bitchute (www.bitchute.com) for Bitchute videos. 1.32 2020/10/16 - Remove StreamFinder::BannedVideo as banned.video has become so paranoid as to add sophisticated anti-bot/scraping detection that I have been unable to figure out how to circumvent. NOTE/RANT: They claim to be pro freedom / anti-big tech spying, etc. but their site has become unusable without third-party / Google javascripts, and other malware nasties we and our users are trying to avoid for our own safety! Most of their videos can now also be found on Bitchute anyway! - StreamFinder::Youtube: Fix minor bug that sometimes returned no streams. 1.33 2020/11/01 - StreamFinder::Apple: Fix some icon images not being downloaded. - Add StreamFinder::SermonAudio. - Remove depreciated StreamFinder::Radionomy (site now defunct). 1.34 2020/11/02 - Add StreamFinder::Castbox (https://castbox.fm podcasts) - StreamFinder::Spreaker: Enhance to accept podcast page URLs (like other podcast-ish modules rather then just the bare widget.spreaker... URLs. These pages also have artist and albumartist fields! Also, properly return the total no. of streams found ($podcast->count()). - "internationalize" several sites (eliminate the ".com" from the required matching regices (allow for ".eu", ".net", etc. to match"). 1.35 2020/11/04 - Fix regex glitch breaking the sites we "internationalized" in v1.34. 1.36 2020/11/16 - StreamFinder::Apple: Return correct description for podcast episodes. - StreamFinder::Spreaker: Fix minor title and image issues. - StreamFinder::Youtube: (For now): convert www.youtube.com to youtube.be to get around youtube.com seeming to block youtube-dl. 1.37 2020/11/25 - Add StreamFinder::Rumble for videos on rumble.com - StreamFinder::Castbox: Return full description for podcast episodes. 1.38 2021/01/01 - StreamFinder::IHeartRadio: Handle streamless custom radio-station urls, ie. "https://station.iheart.com" (revealed by CPAN bug# 133982). These reference the correct "live" url that contains the steams! Also fix description, artist and albumartist fields failing to return values for (some?) podcasts. - StreamFinder::Rumble: Strip the podcast ID field to just it's unique ID. - StreamFinder::Reciva will likely be removed next release after 01/30/2021 due to that site's announcement to go offline after that date! 1.40 2021/01/04 - Add StreamFinder::Google for Google (podcasts.google.com) podcasts. (closes CPAN feature# 133993). 1.41 2021/01/06 - StreamFinder::Google, StreamFinder::Castbox, StreamFinder::Spreaker, and StreamFinder::IHeartRadio: Add ability to accept a podcast / channel page without a specific episode and return the first (latest) episode for that page. NOTE: StreamFinder::Apple already does this but returns all episode streams in a list with the first episode the first stream, but the metadata returned is for the Apple podcast page, and left this way for backward compatability. 1.42 2021/04/11 - StreamFinder::Rumble: Fix failure to find any streams (Rumble chgd. their html slightly), and Add "-keep" option to limit / order the streams fetched (default: mp4,webm,any) - see updated docs. - StreamFinder::Apple: Remove music.apple.com (music samples no longer scrapable? (Much more complicated and I never use it - you only ever could get short sample clips - not full songs anyway! Also, misc. fixes/tweaks, also see last item below: - StreamFinder::Castbox: Added Album field. - StreamFinder::Spreaker: Fix failure to fetch anything (Spreaker chgd. their html a bit!). Also, added Album field, and additional check for possible AlbumArtist url. - StreamFinder::Google: Misc. fixes/tweaks, also see below: - Apple, Castbox, and Google: Add ability to fetch the first (latest?) podcast, and return all podcasts as an extended MP3 playlist (new "-playlist" option to the $station->get() function) - for podcast author's main page urls. 1.43 2021/04/11 - StreamFinder::Rumble: Fix doc glitch in SYNOPSIS section specifying the "-keep" option as a hash reference (must actually be an array reference, causing user to not get any streams if using the SYNOPSIS example as a program. Also fixed this in StreamFinder::IHeartRadio! - StreamFinder::Spreaker: Fix failure to fetch icon image for some podcasts. 1.44 2021/04/19 - StreamFinder::Rumble: If only a podcast "ID" is given, check to see if it's either the long ID for the html page or the short one used for the embed page and append the appropriate url (prev. only the long html page ID string was supported. Either way, the correct Rumble ID will be returned. - Add some default values for "genre", such as "Podcast" for Spreaker, Castbox, Apple, and "Video" for Youtube, Vimeo, Brighteon and Bitchute. - Initialize some variables that should be. - StreamFinder::Apple: Better checks for album, genre and year. - Misc. doc-fixes all over. 1.45 2021/05/04 - Add StreamFinder::Anystream - as a "fall-back" module to search any specified webpage URL for streams (streams are matched by searching for URLs matching specific known stream types, ie. ".mp3", ".ogg", etc. (The default "-keep" list is: "mp3,ogg,flac,mp4,m4a,mpd,m3u8,m3u,pls"). - Added StreamFinder option "-omit" to specify submodules that you have installed to NOT be used, ie. -omit => "Anystream,Youtube". 1.46 2021/05/05 - Oops - add StreamFinder::Anystream to MANIFEST. - Add & document new -secure option (all) to allow user to limit streams returned to (secure) https streems. - Minor doc. cleanups. 1.47 2021/05/10 - Fix potential infinite loop if unrecognized option arg. passed in. - StreamFinder::Youtube - Add fallback ability to check html page for embedded iframe containing a valid StreamFinder page, and new option: "-noiframes" to not do this (if set to 1/true). 1.48 2021/05/19 - StreamFinder::Youtube, StreamFinder::Anystream - Only look for streams in actual HTML pages, NOT media URLs - to prevent potentially very long or even infinite delay trying to fetch the URL. - StreamFinder::Spreaker - Unescape artist and album fields, remove some extraineous debug output. 1.50 2021/05/22 - Refactor all modules to combine all common methods into a separate internal module: StreamFinder::_Class to greatly simplify and eliminate redundant code. This should make creating new site modules much simpler (now most if not all functions other than "new()" should not be required in site modules unless it's necessary to override! - StreamFinder::Tunein - Add "-notrim" option to NOT trim extra "ad", misc. stuff from stream URLs (previously, and by default URLs are "trimmed" removing this stuff (after and including the first "?"). - StreamFinder::Tunein, StreamFinder::IHeartRadio - Attempt to grab full description of podcasts (may include HTML, ie. links). - Misc. minor code cleanups found while converting.