feat: better compression bandwidth - #198

Open

mterenzi wants to merge 18 commits into

farsightsec:nextfrom

mterenzi:realtime-compression

Commits on Mar 3, 2026

Fix aliasing violations in nmsg

Rafael Vanoni
authored and
rafaelvanoni
committed
f968420

Commits on Apr 15, 2026

Make json-c build dependency mandatory
regalk13
committed
384c36a

Commits on Apr 22, 2026

print calculated crc in host byte order for mismatch warning debug message
reedjc
committed
dae8047

Commits on May 29, 2026

nmsg --readpres hang for base email and logline and linkpair

Rafael Vanoni
authored and
rafaelvanoni
committed
00c3bfe
fix loop in _nmsg_strbuf_expand()

Rafael Vanoni
authored and
rafaelvanoni
committed
e0833f6

Commits on Jun 1, 2026

check return of alloc/dup() calls; check max buf size and fix offset in zlib inflate()
Rafael Vanoni
committed
307e00b

Commits on Jun 3, 2026

remove unused libmy functions/files

Rafael Vanoni
authored and
rafaelvanoni
committed
4cc5f38

Commits on Jun 4, 2026

fix closing docbook tag
reedjc
committed
6740971

Commits on Jun 9, 2026

fix docbook for end of sentence period outside of para
reedjc
committed
037cb96

Commits on Jun 22, 2026

Add support for single nmsg payload kafka input/output sources via nmsgp: proto.
shw700
committed
72e0803

Commits on Jul 9, 2026

Handle fragmented containers in zmq, kafka inputs
cmikk
committed
188ef14

Commits on Aug 21, 2026

Add --zasync: compress nmsg file containers on a background thread, one in flight
Maximilian Terenzi
committed
c1cf638
Widen --zasync to a compressor pool; add a committer that writes in ticket order
Maximilian Terenzi
committed
711cfdb
Move the compressor pool to output_async.c; spawn workers on demand, add tests
Maximilian Terenzi
committed
8847deb
Size the reorder ring and worker ceiling per pool; rebuild on a ceiling change
Maximilian Terenzi
committed
5a3bece
Cull compressors idle past --zcull, never below --zmin; take the MRU worker
Maximilian Terenzi
committed
aee2c8a
Fix use-after-free on so_pool and a message lost on a pool error; pick workers by index
Maximilian Terenzi
committed
bdc19ef
Audit: fix self-cull with culling off, a stuck flush, and --zcull/--zmin parsing
Maximilian Terenzi
committed
c5076cc