Revision history for CPAN-Testers-Data-Generator ================================================ 1.21 2015-08-26 - uploads table now has a unique id as primary key. - cpanstats, release_data and release_summary tables now include an uploadid field and index. 1.20 2015-04-19 - fixed tail log searching. - ensure we are parsing a CPAN::Testers::Report object. - added examples/metabase-get.pl to check GUID entries. 1.19 2015-04-17 - fixed date/time timezone bug (thanks Slaven). 1.18 2015-04-17 - added insertion of PASS report data into passreports table. - added build-pass-reports.pl script to examples, for emergencies :) - maintain history of Bucket changes. 1.17 2014-09-11 - fixed license fields in META.json to be lists. 1.16 2014-05-15 - move to using DateTime to calculate dates and times. - fixed distribution name in META. - fixed _get_createdate guid pattern matching. 1.15 2014-03-23 - finally upgraded to Metabase-1.003. - extended test suite. 1.14 2014-03-22 - arch os no longer needs to decode JSON. 1.13 2014-03-20 - enable mysql_auto_reconnect. - updates from live fixes. 1.12 2014-03-19 - specify the end timestamp when getting the guids in a date range. - store complete report (using Sereal) in db as metabase.fact. Also retrieve and de-reference facts hash from 'fact' field, as 'report' field will be dropped once the 'fact' field is populated for all reports (including historical ones). - rollback end timestamp (for now), until storage bug fixed. - expand test suite. - store perl6 reports with type=6. Amended to ignore these for the moment, as this needs to be determined by different metadata. - remove SQLite references. 1.11 2013-09-08 - added Data::FlexSerializer functionality to compress reports. - added missing prerequisite, Metabase::Resource. 1.10 2013-05-12 - fixed skip test count. - fixed cleanup of directory removal. 1.09 2013-04-14 - date clarification for lookups. - additional logging. - added LICENSE file. - added cleanup process in test suite. - added Test::Database to test suite. 1.08 2012-12-15 - fixes to make sure we move on if no reports processed. 1.07 2012-12-06 - updated README - rewritten generate method. 1.06 2012-11-18 - removed JSON & RSS Recent code. - fixed dates. 1.05 2012-11-13 - parse message fix. - disabled SQLite updates. - new script: bin/cpanstats-sqlite (v0.01) - script updates: bin/cpanstats (v1.05) bin/cpanstats-select (v0.12) bin/cpanstats-update (v0.10) bin/cpanstats-delete (v0.15) 1.04 2012-10-23 - script updates: bin/cpanstats (v1.04) - CHANGES => Changes. - added tail feature. 1.03 2012-10-21 - spelling fixes. (thanks to Florian Schlichting & Gregor Herrmann). - AWS values are now configure settings, that can be overridden on the command line. - compensate for Amazon SimpleDB's brokeness with dates. - added parse function for specific GUIDs. - change file dates changed to meet W3CDTF standards. - META tests no longer require specific versions. - reworked Makefile.PL for clarity. - License updated to Artistic License v2.0. - added minimum perl version (5.006). - added missing prerequisite. 1.02 2012-04-01 - reworked regenerate functionality and script. - reworked reparse functionality and script to use Metabase API. - added handling of localonly and check options. - updated script: bin/cpanstats (v1.02) 1.01 2012-01-22 - generator now builds a JSON and RSS file of the last n reports. - added id to page_request insert (to speed up page builder). - reworked regenerator to allow a file of guid/date ranges. - update regenerate option and documentation: bin/cpanstats (v1.01) - added mappings for old badly formatted distributions. 1.00 2010-11-14 - redesigned to use Metabase. All NNTP references removed. - removed ignore and nostore options. - added nonstop option. - added error report mail alert, to highlight problems quickly - improved validation of metabase results, due to the search sometimes returning older reports! - improved retrieval of tester fact - scripts removed: bin/articles-verify bin/articles-view bin/cpanstats-reparse bin/cpanstats-verify examples/cpanstats-convert - added guid command line options and/or output: bin/cpanstats-select (v0.11) bin/cpanstats-update (v0.09) bin/cpanstats-delete (v0.14) - major update of all SQL tables and databases examples/cpanstats-createdb (v0.02) - added META.json - added regenerate() and retrieve_report() methods. - added regenerate option and documentation: bin/cpanstats (v1.00) - various data amendments, including _check_arch_os(), as the metabase uses the raw data. - new perl_version table incorporated to speed up perl version lookups. 0.41 2010-03-18 - fixes to change the 'id' (was NNTP ID) to an auto incremental field. - reworked logic to better fit latest changes. - added repository to META.yml. - documentation updates. 0.40 2010-02-02 - fixes to accommodate GUID changes. - added support for 'type' field. 0.39 2010-01-30 - major update to include guid for Metabase/CT2.0. 0.38 2009-10-19 - added functionality to update the osname table. - added cpanstats DB schema to examples directory. 0.37 2009-06-07 - added page request functionality. - added release summary functionality. - due to the recent expansion, the 12generate.t test script now needs more complex logic. The intention is to use Test::Database for this, but unfortunately T::D isn't stable enough to use yet. As such this test script is now effectively disabled. 0.36 2009-02-16 - clean up of pre-requisites. - CPAN::Testers::Data::Generator::Article abstracted into its own distribution CPAN-Testers-Common-Article. - fixed scripts due to above: bin/cpanstats (v0.07) bin/cpanstats-reparse (v0.10) 0.35 2009-01-29 - updated patterns when extracting perl version (Article.pm). - updated patterns when extracting osname (Article.pm). - updated cannot connect to NNTP server error message. 0.34 2009-01-12 - updated Copyright info, minor fixes and added 'Getopt::ArgvFile', if not already used: bin/articles-verify (v0.03) bin/articles-view (v0.03) bin/cpanstats (v0.06) bin/cpanstats-delete (v0.13) bin/cpanstats-reparse (v0.09) bin/cpanstats-select (v0.10) bin/cpanstats-update (v0.08) bin/cpanstats-verify (v0.24) - added cpanstats-verify.sh to examples. - added missing log file command line option to bin/cpanstats (v0.06). - added missing prerequisites (thanks to CPANTS) 0.33 2008-12-29 - updated help text to bin/cpanstats-verify (v0.23) - fix to stop existing records being reinserted. 0.32 2008-12-19 - added progress option to bin/cpanstats-verify (v0.22) - fixes to Article.pm to attempt to correct missing osnames and osvers. - added articles scripts that have been used behind the scenes for sometime: bin/articles-verify (v0.02) bin/articles-view (v0.02) 0.31 2008-12-18 - name change to fit below the CPAN::Testers namespace. - added epoch to date extraction from article. - added filename reference from CPAN-DistnameInfo. - added ability to provide SQL string in bin/cpanstats-select. - added better command line option parsing to bin/cpanstats-reparse. - convert to use CPAN::Testers::Common::DBUtils. - added MySQL cpanstats database to codebase. - new scripts added to examples to handle database creation: examples/cpanstats-createdb (v0.01) examples/cpanstats-convert (v0.01) - update to bin scripts with latest code changes: bin/cpanstats (v0.05) bin/cpanstats-delete (v0.12) bin/cpanstats-reparse (v0.08) bin/cpanstats-select (v0.09) bin/cpanstats-update (v0.07) bin/cpanstats-verify (v0.21) 0.30 2008-10-01 - ensure upload pattern matching is for the end of the string. - add more option validation in bin/cpanstats-verify (v0.20). 0.29 2008-09-19 - ignore any uploads that are not in a supported archive format used for testing. 0.28 2008-09-18 - added SQLite pragma 'auto_vacuum' to database creation. - created an _init function to generate db/nntp objects in one place - add --localonly option to help screen in bin/cpanstats-verify (v0.19) - fixed test script to correctly do cmp_ok :) 0.27 2008-09-11 - added 'grade' option to search in bin/cpanstats-select (v0.08) - fixed help text in bin/cpanstats-update (v0.06) - fixed test bug in 12generate.t (I can't count!) 0.26 2008-09-02 - abstracted all database functionality out of Generator.pm into Database.pm, including database creation. All scripts updated too. - database links changed to reference CPAN Testers development site. - corrections to the database schema in POD - several POD updates across all files, including bin/cpanstats (v0.04) bin/cpanstats-delete (v0.11) bin/cpanstats-reparse (v0.07) bin/cpanstats-select (v0.07) bin/cpanstats-update (v0.05) bin/cpanstats-verify (v0.18) - fixed dependency in cpanstats - consistently provided help|h and version|V options to all scripts - fix to support old form of PAUSE uploads - abstracted reparse code into Generator.pm from cpanstats-reparse - more tests 0.25 2008-08-28 - added POD about bugs and see also sections. - fixed POD in scripts. - added author to upload metadata, to save lookups elsewhere. - renamed several supporting scripts: delstats => bin/cpanstats-delete (v0.10) selectstats => bin/cpanstats-select (v0.06) upstats => bin/cpanstats-update (v0.04) readstats => bin/cpanstats-verify (v0.17) - added further functionality to cpanstats-select. - added bin/cpanstats-reparse (v0.06) to distribution - added nostore option to not retain old articles in the articles database. 0.24 2008-08-19 - added bin/readstats (v0.16), bin/delstats (v0.09), bin/selectstats (v0.05) and bin/upstats (v0.03) from the non-CPAN cpanstats distribution. - updated all scripts to use an options hash rather than individual variables. - added new Database.pm module to handle all local DB interaction. - fixed test bug in t/12generate.t (thanks to Slaven Rezic). 0.23 2008-08-18 - new maintainer: Barbie. - major overhaul to draw in CPAN Testers Statistics code. - added version to Article.pm. - added ability to decode QuotedPrintable and Base64 (Article.pm) - added functionality to parse the upload articles as well as the report articles (Article.pm). - added more reliable subject parsing. - added more reliable Perl -V parsing. - testers.db now renamed to cpanstats.db. - news.db now renamed to articles.db. - field names changed: status => state, distribution => dist, archname => platform. - new fields added: postdate,tester. - fixed prerequisites in modules, Makefile.PL and META.yml. - bin/cpan_www_testers_generate replaced with bin/cpanstats.pl - added more test samples - added more tests 0.22 2005-02-03T15:27:41 - now includes perl + OS version in the testers.db - use SQLite 3.0 - bin/cpan_www_testers_generate expanded to give more flexibility for execution - added Makefile.PL (as per TODO) - added tests - added more POD - (thanks to Barbie) 0.21 2004-03-07T18:31:35 - the NNTP cache now uses SQLite instead of DB_File - rename Changes to CHANGES 0.20 2003-08-13T21:52:06 - make sure to HTML escape everything - design update, now with CSS - Fix bug where reports are use A/AM/AMS/Crypt-TEA-1.22.tar.gz instead of Crypt-TEA-1.22.tar.gz - added a TODO - sort using version, not Sort::Version - renamed to CPAN::WWW::Testers::Generator - included HISTORY section by Elaine Ashton - moved examples/generate.pl to an installed script, bin/cpan_www_testers_generate 0.10 2003-07-31T23:09:25 - initial release