Skip to content

Detection quality & safety: verification identity, precision/recall gate, ReDoS-proofing, source-map expansion - #1

Merged
azmolhaque merged 8 commits into
mainfrom
claude/new-session-thsmic
Jul 18, 2026
Merged

Detection quality & safety: verification identity, precision/recall gate, ReDoS-proofing, source-map expansion#1
azmolhaque merged 8 commits into
mainfrom
claude/new-session-thsmic

Conversation

@azmolhaque

@azmolhaque azmolhaque commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

A measured, tested capability pass over SecretNode, grounded in a fresh audit vs 2026 secret-scanning SOTA (TruffleHog/Gitleaks). Each change is an independent, shippable slice behind tests. The test suite grows 145 → 167, all green; ruff clean. Full analysis in docs/TECHNICAL-AUDIT-AND-ROADMAP.md.

  • R1 — Verification enrichment. A verified credential now yields a short, non-sensitive identity/scope label (GitHub @acct + token scopes, Stripe account + LIVE/charges, Slack workspace/user, OpenAI org, npm/GitLab/Telegram handle, SendGrid send-scope, Mailgun domain count) — the concrete blast radius an attacker inherits. Surfaced in HTML (live access: …), CSV (verified_detail column), and SARIF (appended after the literal [VERIFIED ACTIVE] token + a property). Backward-compatible: verify_finding() keeps its string API; new verify_finding_detailed() returns VerifyResult. Detail is derived identity only — never the secret itself; defensive extraction; fails closed.
  • R2 — FP/FN benchmark harness. backend/bench/ — a labelled corpus (12 synthetic, correctly-shaped positives + 15 placeholders/examples/high-entropy noise), a make bench precision/recall/F1 report, and a pytest CI gate (test_bench.py) that fails the build on a precision/recall regression. Current: precision 1.000 · recall 1.000 · F1 1.000 · 0 false positives. All corpus secrets are synthetic (deterministic, not live).
  • R3 — Regex ReDoS safety. A per-pattern match cap (MAX_MATCHES_PER_PATTERN) plus an automated gate: empirical wall-clock fuzz over all 54 detectors × 17 adversarial 50 KB inputs (no pattern may exceed 0.75s), a static nested-quantifier guard, and a cap-engagement test.
  • R4 — SARIF full detector catalog. The SARIF driver now advertises every detector as a rule (help text, CWE, severity) even on a clean scan, built from the live registry.
  • R5 — Source-map surface expansion. Decode a source map's embedded original source (sourcesContent) and scan it as real code with precise per-file attribution (app.js.map → src/config.js), instead of the raw .map JSON when source is present — which also removes the map's high-entropy mappings VLQ blob as a false-positive source. Env-tunable (SCAN_SOURCEMAP_CONTENT, MAX_SOURCEMAP_SOURCES); fully defensive.

Type of change

  • Bug fix
  • New secret pattern / detector
  • New feature
  • Docs / tooling

Checklist

  • ruff check backend/ passes
  • pytest passes (added/updated tests for the change) — 167 tests, all green (+22 this PR)
  • New detectors include severity, cwe, and remediation — n/a (no new detectors here)
  • No real credentials, live targets, or exploitation code introduced — corpus secrets are synthetic/deterministic
  • Scanner remains passive (no use of discovered credentials, no writes to targets) — verification stays off-by-default, read-only, fails closed

🤖 Generated with Claude Code

Generated by Claude Code

claude added 8 commits July 17, 2026 23:57
@claude
SARIF: advertise full detector catalog + add technical audit/roadmap
6eaa038 
R4: generate_sarif_report now emits the complete detector catalog as SARIF
reportingDescriptors (help text, CWE, security-severity) built from the live
registry — so CI consumers (GitHub code scanning) always have full rule
metadata, even on a clean scan. Results still resolve to described rules;
unknown types keep the finding-derived fallback. +2 tests (suite 145->147).

Adds docs/TECHNICAL-AUDIT-AND-ROADMAP.md: honest capability/gap analysis vs
2026 SOTA (TruffleHog/Gitleaks) and a sequenced, tested enhancement roadmap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R1: verification enrichment — capture identity/scope of live credentials
3f3bcfa 
A verified credential now also yields a short, non-sensitive identity/scope
label (GitHub @acct + token scopes, Stripe account + LIVE/charges, Slack
workspace/user, OpenAI org, npm/GitLab/Telegram handle, SendGrid send-scope,
Mailgun domain count) — the concrete blast radius an attacker inherits.

- verifier.py: verifiers return (active, detail); new VerifyResult +
  verify_finding_detailed(); verify_finding() kept as string-only wrapper
  (backward compatible). Detail extraction is defensive and never includes
  the secret value; fails closed.
- scanner.py: ValidatedFinding gains verified_detail; scan flow captures it.
- report.py: HTML ('live access: …'), CSV (verified_detail column), and SARIF
  (identity appended after the literal [VERIFIED ACTIVE] token + property).
- +7 tests. Suite 147 -> 154 green; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R2: FP/FN benchmark harness — measurable precision/recall + CI gate
c0fac2b 
Adds backend/bench/: a labelled corpus (12 synthetic, correctly-shaped
high-entropy positives + 15 placeholders/examples/high-entropy noise) and a
runner that measures the deterministic detection layer (extract_secrets:
regex + placeholder allowlist + entropy gate + base64 pass — no network/AI)
and reports precision / recall / F1 / type-accuracy with per-miss diagnostics.

- make bench: human-readable precision/recall report.
- test_bench.py: CI gate — fails the build if precision/recall/F1 < 0.95 or any
  placeholder/noise is flagged (brand value: zero false positives).
- Current: precision 1.000 · recall 1.000 · F1 1.000 · 0 FP.
- All corpus secrets are SYNTHETIC (deterministically generated, not live).
  The harness caught a malformed OpenAI test key, confirming the detector
  correctly requires the real 'T3BlbkFJ' key structure.

Suite 154 -> 158 green; ruff clean. Marks R1+R2 done in the roadmap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R3: regex ReDoS safety — match cap + automated backtracking gate
166d6a0 
- scanner: MAX_MATCHES_PER_PATTERN cap in _scan_text bounds the matches
  examined for any single detector on any single text (defence-in-depth
  against a crafted match-flood blob).
- test_regex_safety.py: empirical wall-clock fuzz over all 54 detectors x 17
  adversarial 50KB inputs (no pattern may exceed 0.75s — proves no catastrophic
  backtracking), a static nested-quantifier guard that gates future pattern
  additions, and a cap-engagement test.

Suite 158 -> 161 green; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R5: source-map surface expansion — scan decoded original source
e751064 
Decode a source map's embedded original source (sourcesContent) and scan it as
real code with precise per-file attribution (app.js.map -> src/config.js).

- scanner: extract_sourcemap_sources() (defensive JSON decode, bounded by
  MAX_SOURCEMAP_SOURCES) + looks_like_sourcemap(); run_scan now scans a map's
  decoded sources INSTEAD of the raw .map JSON when sourcesContent is present —
  better attribution, catches secrets escaped in the raw JSON, and removes the
  map's high-entropy 'mappings' VLQ blob as a false-positive source. Falls back
  to raw-body scan when a map has no usable sourcesContent.
- env: SCAN_SOURCEMAP_CONTENT (default true), MAX_SOURCEMAP_SOURCES (200).
- +6 tests; end-to-end loop verified (single attributed finding, no dup).

Note: inline JSON (__NEXT_DATA__) and HTML comments were already covered (the
full HTML body is scanned wholesale), so they are not reclaimed as 'new'.

Suite 161 -> 167 green; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R6: verifier expansion — 8 more live-verification providers (9 -> 17)
3309b07 
Add read-only whoami/validate verifiers for existing detectors that previously
returned 'unsupported': Cloudflare (tokens/verify), DigitalOcean (/account),
Datadog (/validate), Notion (/users/me), Linear (GraphQL viewer), Figma (/me),
Postman (/me), Doppler (/me). Each extracts R1 identity where available, is
strictly read-only, and fails closed.

Chosen over adding new detectors on purpose: this widens the verification-first
differentiator with zero new false-positive risk (no new regexes). README
provider list updated to match. +6 tests.

Suite green; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R9: executive-summary report — verification evidence + precision stat…
169c674 
…ement

Turn the engine work into a client-ready deliverable. generate_html_report now:
- leads with a verification-EVIDENCE callout: each verified-active credential
  with its R1 identity/scope shown as 'confirmed live access' (the strongest
  client signal — proven live, not shape-matched). Only rendered when there is
  live proof.
- adds a 'Verified Active' KPI tile (alert-styled when > 0).
- adds an honest 'Detection quality' statement: verification-first + a labelled
  benchmark corpus with a precision/recall gate in CI.

All additive; clean scans render no callout. +4 tests. Suite green; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@claude
R8: passive attack-surface breadth — HTTP security-posture checks
20bfee6 
First step from secret-scanner toward the fuller attack-surface scanner the
brand promises. New posture.py analyses the target root's own response for
security misconfigurations — pure passive analysis, no exploitation, no
third-party calls, fails closed:
  - missing/weak headers: HSTS, CSP, X-Content-Type-Options, clickjacking
    (X-Frame-Options / CSP frame-ancestors), Referrer-Policy, Permissions-Policy
  - software version disclosure (Server / X-Powered-By)
  - insecure cookies (missing Secure / HttpOnly)
Each PostureFinding carries severity/CWE/evidence/remediation.

- scanner: one passive GET on the root feeds analyze_security_headers();
  result['posture_findings'] populated (SCAN_HTTP_POSTURE, default true).
- report: dedicated 'Security Posture & Misconfigurations' HTML section +
  'Posture Issues' KPI tile — so even a clean credential scan returns
  actionable ASM findings.
- +11 tests (pure analyzer, fetch path, report section). Suite green; ruff clean.

Follow-up (deferred, external network): CT-log subdomains, dangling-DNS,
posture in CSV/SARIF.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
@azmolhaque
azmolhaque merged commit 623ea1a into main Jul 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@azmolhaque @claude