Icod.Grep 1.6.0: performance and scalability

[uniblab]

Implements T6 — Performance and Scalability for the Icod.Grep 1.6.0 release from merged main / 1.5.0 at 423c0e9623100492fa01b6e4d14c183761d111d7.

Governing principle

Measure first. Optimize second. Preserve behavior always.

All quantitative acceptance decisions are based on repeated BenchmarkDotNet comparisons on the established physical Windows reference host. GitHub-hosted Windows/Linux/macOS runs remain correctness and benchmark-smoke gates rather than sources for narrow percentage claims.

Completed T6 work

T6.0 — benchmark foundation and immutable 1.5.0 baseline

Established deterministic command, filesystem, and record-reader benchmarks; physical-host metadata; ABBA comparison tooling; hosted benchmark smokes; and the retained 1.5.0 reference report.

T6.2 / CommandFramework R2.5 consumer integration

Integrated Icod.CommandFramework 2.2.0 prepared byte input so managed BRE/ERE matching prepares each searched record once and reuses that representation across repeated offsets, multiple patterns, and -o enumeration.

Representative physical improvements versus 1.5.0 included:

• BRE ASCII sparse allocation: 604.60 MB → ~41 MB;
• BRE UTF-8 sparse allocation: 462.87 MB → ~29 MB;
• BRE long-line allocation: 3,005.86 MB → ~214 MB;
• large-file allocation: 4,836.02 MB → ~329 MB.

T6.1 — fixed-string scalability — closed

Added an immutable Aho-Corasick-style matcher for the conservative case-sensitive multi-pattern -F subset while preserving fallback behavior for -i, -w, -x, and empty patterns.

Physical results included:

• fixed-100: roughly 63× faster than 1.5.0;
• fixed-1000: roughly 341× faster in the focused physical run;
• full-suite residual validation found no material collateral regression.

Closure report: Icod.Grep-1.6.0-T6.1-Closure-and-Residual-Report.md.

T6.4 — binary probing — closed

Seekable binary probing now reads bounded 8 KiB chunks up to the GNU-compatible probe window, stops immediately on NUL/EOF, uses pooled storage, and restores the original stream position exactly. Non-seekable prefix replay remains unchanged.

Representative physical results:

• many-small-files: ~16% faster, ~8 MB less allocation;
• recursive-tree: ~14% faster, ~8 MB less allocation.

Closure report: Icod.Grep-1.6.0-T6.4-Binary-Probe-Closure-Report.md.

T6.3 — record pipeline and very-large-record scalability — closed

Candidate 1 removed Grep's redundant copy of each already-owned ByteRecord. Candidate 2 then bypassed the shared materializing compatibility reader inside Grep and consumes independently owned record segments directly, only reassembling records that actually span segments.

Representative Candidate 2 improvements over Candidate 1:

• long-line: ~5.9% faster, ~8 MB less allocation;
• large-file: ~8.0% faster, ~29 MB less allocation;
• fixed single-pattern: ~14.5% faster, ~47.6% less allocation;
• PCRE lookbehind: ~12.6% faster, ~49% less allocation.

Shared CommandFramework record-reader controls remained unchanged.

Closure report: Icod.Grep-1.6.0-T6.3-Closure-and-Residual-Report.md.

T6.6 — filesystem / many-file scalability — closed

Eliminated redundant System.IO.FileStream read buffering beneath Grep's own bounded segmented-reader buffer. Traversal ordering and concurrency semantics remain unchanged.

Physical results:

• many-small-files allocation: 34.62 MB → 18.53 MB (-46.5%), elapsed time effectively flat;
• recursive-tree allocation: 34.91 MB → 18.81 MB (-46.1%), about 15.5% faster;
• large-file allocation unchanged, with no throughput regression.

Closure report: Icod.Grep-1.6.0-T6.6-Filesystem-Closure-Report.md.

Current tranche — T6.5 output, formatting, color, and context allocation

The existing benchmark suite was intentionally output-light, so T6.5 first adds deterministic output-heavy measurements before changing production output code.

Current benchmark head c0fcffb5d529691891c727fbf9816f2e8d14ec74 adds:

• dense selected-record output;
• -H -n -b prefix-heavy output;
• dense -o output;
• forced-color output;
• context output; and
• line-buffered output.

Each workload validates its expected output before BenchmarkDotNet measures it. PR CI is green across Windows, Linux, macOS, benchmark smokes, package smokes, and archive smokes.

The next gate is the physical pre-optimization T6.5 reference measurement on these new workloads. No output-path optimization will be accepted before that baseline identifies the dominant allocation costs.

Remaining T6 work

1. T6.5 — output/formatting/color/context allocation;
2. T6.7 — PCRE-specific profiling, only where the residual justifies library-specific work;
3. T6.8 — stress/resource/scalability limits;
4. T6.9 — release closure, final whole-suite measurement, documentation, and release-readiness audit.

Version

The development and package version remain 1.6.0.