Revision history for C and Perl extension of nfdump. 1.30 2020-12-22 - fixed issue with building ffilter code on FreeBSD - fixed issue when no exporters are configured - removed memory leask (https://github.com/VUTBR/libnf/issues/14) 1.29 2020-12-20 - updated to nfdump 1.6.22 - c: fixed issue with creating exporter map - nfdupp: fixes issue with reporting wrong number of output records - nfdump: duriation is now printed with units (s,m,h) - nfdump: tcp flags are displayed in default output - nfdump: fixed issue when -O flags was zeroing last 8 bits in src and dst address - perl: updated info in perl documentation 1.28 2020-04-10 - updated to nfdump 1.6.20 - code cleanup 1.27 2020-04-07 - after a few years stalled development the library is going to be maintained and sponsored by NetX Networks a.s. - source code moved from original https://github.com/VUTBR/libnf to https://github.com/NETX-AS/libnf - updated README - un-submoduled ffilter code to avoid nigtmare with git submodules - backported patch https://github.com/VUTBR/libnf/pull/24 - updated to nfdump 1.6.18 - removed support for legacy nadump V0 format - some test files in perl/t/ dir converted from V0 format 1.26 2018-06-xx - never released - updated to nfdump 1.6.17 - c: better processing of list of exporters - c: ringbuf support to handle data beween libnf instancec in real time - c: new functions lnf_ring_init, lnf_ring_info, lnf_ring_read, lnf_ring_write, lnf_ring_free - c: new examples examples/lnf_ex13_ring_write.c, examples/lnf_ex14_ring_read.c - nfdumpp: new shm (ringbuf) output format (-o shm) - nfdumpp: new --loop-read option for continuos reading of input file(s) - nfdumpp: new --shm-read option for read data from shm (ringbuf) - perl: perl module is not separate code but part of libnf repository 1.25 2016-12-04 - libnf and modules moved to separate repositories (libnf, libnf-ffilter, libnf-perl) command git submodule update --remote --merge must be used to update to latest submodule reference -c: some leaks in memheap removed -c: versioning of libnf.so added (issue #5, thanks to Tomas Cejka) -c: added new API function lnf_mem_clean(lnf_mem_t *lnf_mem) -c: added exmaple lnf_ex11_memclean.c -c: new function lnf_rec_get_raw/lnf_rec_set_raw -c: new example examples/lnf_ex12_rec_raw.c -c: new function lnf_filter_ffilter_ptr -c: fix of RPM build (issue #5, thanks to Tomas Cejka) -perl: added new test 53-libnf-filter.t -nfdumpp: handles properly error messages for libnf filter -nfdumpp: ability ro read binnary format from stdin (nfdumpp -r -) 1.24 2016-06-21 -perl: better thread testing (with more detailed results) 1.23 2016-06-20 -c: fixed build on FreeBSD 1.22 2016-06-18 -c: adjusted to nfdump-1.6.15 1.21 2016-05-28 -nffdump: invalid mutex initialisation 1.20 2016-05-09 -removed several warnings -c: new field LNF_FLD_INET_FAMILY containing either AF_INET or AF_INET6 -perl: new field inetfamily -perl: new function family2txt and txt2family -nfdumpp: removed bug where nfdumpp produces corupted output file -nfdumpp: added -n option to limit aggregated outputs 1.19 2016-02-03 -perl: removed dependency on lbz 1.18 2016-01-29 -perl: new variables $flow->{NUM_OF_FIELDS} and $flow->{NAME} can be accesed Zangerl, Report #110170) -c: adjusted to nfdump-1.6.14 -c: two new lnf_open flags LNF_COMP_LZO and LNF_COMP_BZ2 -c: two new info options LNF_INFO_LZO_COMPRESSED and LNF_INFO_BZ2_COMPRESSED -c: new ./configure option to build with static or dynam BZ2 support -c: separation of new filter code. The new filter can also be used in other projects (include/ffliter.h and src/ffiltler, thank to Lukas Hutak for cooperation). -c: removed memory leak in lnf_lookup_map and lnf_close (tnaks for Lukas Hutak) -c: fixed size of some fields (patch submitted by Lukas Hutak) -c: API change - tha value of LNF_AGGR_KEY is 0x0008 -c: API change - new LNF_AGGR_AUTO for autodetection of aggregation mode -nfdumpp: new -w and -o options -nfdumpp: new modular output system -nfdumpp: invalid dispaly of duration field -nfdumpp: multiple outputs when multiple -A switch is used Examples: nfdumpp -R DIR -A proto -O bytes -A srcvlan -O bps nfdumpp -R DIR -A ip -O pkts -A port -O bytes -A proto nfdumpp -R DIR -A ip/16 -O bytes -A ip/24 1.17 2015-10-27 -perl: added test t/09-instances.t -perl: test for calculated items in test t/08-aggreg-sort.t -perl: new option CompatMode -c: time fields can be user as aggregation key with alligned time -c: new lnf_mem option LNF_OPT_COMP_STATSCMP -c: sort support for calculated fields -c: removed bug when thread key was nor released after lnf_mem release -c: LNF_CALC_*PS items returns LNF_ERR_NAN when division by 0 is detected -c: new flag LNF_APPEND in lnf_open for open file in append mode (not supported by nfdump format) -c: LNF_FLD_INFO_SIZE can be returned by lnf_fld_info -c: added LNF__T with basic types definitions used by libnf -c: removed bug when memheap crashes after mixing pair and non pair fields (Thanks to Jan Wrona) -c: new fields LNF_FLD_INGRESS_ACL and LNF_FLD_EGRESS_ACL representing all fields *_ACL_ID, *_ACE_ID, *_XACE_ID in one structure lnf_acl_t -nfdumpp: redesigned command options 1.16 2015-09-07 -c: added license and copyright information -c: added field event (thanks to Artem Buglak) -perl: module allows to sort records without aggregation (list mode) -c: bug when sorting same values (Thanks to Jan Wrona) -c: libnf-info is compiled statically now (Thanks to Lukas Hutak) 1.15 2015-07-26 - c: new function lnf_mem_setopt for configuring some lnf_mem options - c: lnf_mem_t can be switched into list mode via lnf_mem_setopt(LNF_OPT_LISTMODE, ...); In this mode there is no aggregation of items; mode can be used for sorting records - c: improved merge_threads (removed deadlock) - c: new function lnf_mem_lookup_c and lnf_mem_lookup_raw_c - c: new example examples/lnf_ex09_memlookup.c - c: new example examples/lnf_ex10_listmode.c - c: new option -l in examples/lnf_ex04_threads.c - nfdumpp: sorting without aggregation - c: removed binary and .o objects from original nfdump sources in distribution package (Bug #105944) 1.14 2015-07-18 - c: new way to work with data in lnf_mem, new functions lnf_mem_first_c, lnf_mem_next_c, lnf_mem_read_c, lnf_mem_read_raw_c - c: functions lnf_mem_read, lnf_mem_read_raw, lnf_mem_read_reset marked as deprecated - c: examples redesigned to use new lnf_mem functions - c: libnf.h was not installed by default (thanks to Tomas Cejka) - c: removed memory release bug in merge_threads - perl: added new test 52-libnf-examples.t to test examples - perl: t/08-aggreg-sort.t was missing in distribution package - perl: Socket6 module is not required starting with perl 5.14.0 http://search.cpan.org/~jesse/perl-5.14.0/pod/perldelta.pod#Improved_IPv6_support - perl: buggy implementation of inet_pton in Ubuntu 1.13 2015-07-08 - perl: module requires use of threads - perl: tests that depends on threads are executed only on systems where thread support is enabled 1.12 2015-07-05 - c: removed __thread directive from source codes - c: redesign of thread patch support nfdump-thread.patch and nfdump-leak.patch patch merged into one file 1.11 2015-07-04 - nfdumpp: added basic comparison test for threads - perl: removed some issues with thread support in perl 1.10 2015-06-30 - Project entered into close cooperation with CESNET - Czech National Research Network - c: thread support enabled by default (--enable-threads) - c: aggregation unit supports statistics - c: new pair fields LNF_FLD_PAIR_PORT|ADDR|AS|IF|VLAN - c: examples/lnf_ex08_statistics added - perl: Net::NfDump->info returns proper data - perl: invalid handling of LNF_UINT16 (Bug #105472) - perl: nfasnupd do not require Geo::IP module 1.09 2015-05-28 - perl: removed memory leak (Bug #104294) - perl: improved test 1.08 2015-05-27 - c: doc update - c: EXPERIMENTAL very early stage of new filter code independent of the original nfdump filter code. New function lnf_filter_init_v1 or lnf_filter_init_v2 is used to initialize either old (nfdump) or new (libnf) filter. By default lnf_filter_init calls the old one filter. - c: new error code LNF_ERR_OTHER_MSG for errors where lnf_error buffer is set - c: _LIBNF_H macro added in the libnf.h file - c: new function lnf_mem_read_reset - set cursor to first record - c: nfdumpp uses by default only 70% of available threads - c: nfdumpp checks -O option - c: removed bug - libnf fails after attempt to read empty lnf_mem - c: removed memory leak in original nfdump in LZO - c: initial number of buckets in hash table increased to 500000 - c: new functions lnf_mem_read_raw, lnf_mem_write_raw added - c: examples/lnf_ex06_readreset added - c: examples/lnf_ex05_memtrans added - c: examples/lnf_ex04_threads option -n added - c: bug in bpd and pps calculation - perl: removed memory leak in perl API (Bug #104294) - perl: nfasnupd accepts -c with additional prefix database 1.07 2015-03-17 - c: nfdumpp utility added - c: added calculated items - duration, bps, pps, bpp - c: lnf_fld_info added - c: lnf_fld_parse added - c: lnf_mem_fastaggr added - c: lnf_info extended - c: performance update (15-20%) - c: thread support for filter operations (not released to public) - perl: static library removed from build tree (Bug #101544) - perl: nf_asn_geo_update and download_asn_db removed - perl: bin/nfasnupd - ASn updater added (replacement for nf_asn_geo_update) - perl: doc update 1.06 2014-12-06 - c: lnf_rec_fget lnf_rec_fset performance update (15%) - c: added src/fields.c,h - code related to fields is now separated - perl: build system update 1.05 2014-12-05 - c: build system update - c: added --enable-threads for thread support; disabled by default - c: flex, bison is not needed for building - c: adjusted to nfdump 1.6.13 1.04 2014-11-17 - perl: added example examples/example2.pl - perl: added test 08-aggreg-sort.t - perl: aggregation and sorting added into Net::NfDump (Aggreg => 1 and SortBy => attributes) - c: lnf_field_t structure extended with default sort and aggregation order - c: added example lnf_ex04_threads.c - c: threads support added for parallel processing flow records in aggregation. Code not released for public, the public version works properly only in single thread environment. - c: added example lnf_ex03_aggreg.c - c: added lnf_mem object with aggregation and sorting capability - c: lnf_rec_fget, lnf_rec_fset increased performance (30%) Thanks to Vasek Bartos. 1.03 2014-06-23 - c: headers includes in the examples - c: code preparation for aggregation and sorting - perl: test for pure read without record processing (-G) 1.02 2014-06-18 - c: added structure lnf_brec1_t containing most commonly used records - c: examples updated to use lnf_brec1_t - c: examples updated to be usable for benchmarks - perl: updated build system (more resistant) - perl: benchmarks of c code added as the part of perl testing 1.01 2014-06-15 - source code strictly separated into autonomous package (libnf-xxx, universal C API) and perl Net::NfDump module that is placed on top of the C API - redesigned build system - prepared for new features (aggregation, sorting, ...) - changes inspired by Vasek Bartos and his libnfdump code 0.14 2014-04-24 - examples updated to use last version of Net::IP::LPM 0.13 2014-04-20 - adjusted to nfdump 1.6.12 - support for new record type introduced by nfdump 1.6.12 - transparently reads record types from previous nfdump - created files are always with the new record type - !! files created by this version mus be read by nfdump >= 1.6.12 - support for new fields (ingress_vrfid, egress_vrfid, event_flag block_start, block_end, block_step, block_size) - flow_start item changed to event_time - fw_event removed (use fw_xevent instead) - doc extended with info about aggregation 0.11 2013-11-26 - adjusted to nfdump 1.6.11 0.10 2013-09-12 - decreased test length 0.09 2013-08-13 - adjusted to nfdump 1.6.10p1 0.08 2013-06-06 - removed bug in TimeWindowStart, thanks to Pavel Sejnoha 0.07 2013-05-21 - removed bug in examples, thanks to James Deaton - updated to nfdump 1.6.10 0.05 2013-05-03 - language corrections 0.05 2013-03-16 - silently skip known (but not supported) blocks - example added (update AS info) 0.04 2013-03-03 - the first stable version 0.03_02 2013-03-03 - the version of nfdump updated to 1.6.9 0.03_01 2013-02-28 - NEL, NSEL support added - a few bugs removed 0.02_05 2013-02-26 - example1.pl added - documentation updated 0.02_04 2013-02-24 - a few bugs removed - clonerow added - InputFiles and Fields can by handled as a reference to list or comma separated list 0.02_01 2013-02-11 - hasharray functions added - performance optimization (4x faster) 0.01_06 2013-01-20 - nfdump source codes added into distribution - fixes for libnf on OpenBSD apllied 0.01_04 2013-01-15 - extra compilation of nfdump utility 0.01_04 2013-01-14 - removed dependency on lex/flex during compilation 0.01_03 2013-01-12 - fixes for multiplaform compilation 0.01_02 2013-01-10 - documentation update 0.01_01 2012-11-30 - initial version