CI test harness for detection rules against `Atomic Red Team`

[rabbitstack]

Motivation

Fibratus's detection rules currently have no automated, continuous validation against real attack technique execution. As a Detection-as-Code practice, rules should be tested the same way code is, automatically, on every change, with a measurable pass/fail signal rather than validated ad hoc. This also surfaces rule regressions early: a refactor to the rules engine or an event schema change could silently break existing detections without anyone noticing until a real incident.

Expectation

A GitHub Actions workflow that, on every push to master, builds and runs the latest Fibratus, executes a curated set of Atomic Red Team techniques (including new atomics specific to process injection and privilege escalation not yet covered upstream), checks which techniques triggered a matching Fibratus alert, and reports coverage, failing the build if coverage drops below a defined threshold.

Proposal

• Atomics source. Use the fork at rabbitstack/atomic-red-team rather than upstream redcanaryco/atomic-red-team, since new atomics need to be authored there to cover process injection and privilege escalation techniques not currently present upstream
• Workflow trigger. Run on every push to master
• Pipeline steps
  1. Build Fibratus from the latest master
  2. Install it. Check for the operational status of the Fibratus service
  3. Clone the rabbitstack/atomic-red-team fork
  4. Execute the selected/tagged atomics relevant to the covered techniques, with jitter between executions (randomized delay) so techniques don't fire all at once, avoiding event bursts that could mask timing-sensitive detections or overwhelm the pipeline
  5. Parse the Fibratus log file for generated alerts after the run
  6. Match fired alerts against the executed atomics/techniques (by technique ID) to compute a coverage percentage
  7. Generate a report summarizing matched vs. unmatched rules and overall coverage %
  8. Fail the workflow if coverage falls below a configured threshold
  9. Upload the test harness results (report, raw Fibratus logs, per-technique pass/fail) as GitHub Actions artifacts

Additional context

• Per-technique mapping file. Maintain an explicit mapping between atomic test IDs (or MITRE ATT&CK technique/sub-technique IDs) and the Fibratus rule(s) expected to fire, so the harness knows which alert should correspond to which atomic rather than just checking "did any alert fire"
• False-positive tracking, not just coverage. Alongside true-positive coverage %, consider tracking whether benign/unrelated activity during the run triggers unexpected alerts, to catch overly broad rules
• Flaky/timing-sensitive atomics. Some atomics may require retries or longer observation windows before the corresponding event surfaces (e.g. async telemetry, delayed persistence triggers). Consider a per-technique timeout/retry policy rather than a single fixed wait
• Historical trend, not just pass/fail. Persisting coverage % over time (e.g. as a badge or trend chart from uploaded artifacts) would make regressions visible even when they don't cross the failure threshold
• Windows Defender/AV interference. Defender or other AV on the runner could intercept/quarantine atomics before Fibratus ever observes them