P0 epic: AST and static-analysis accuracy roadmap

[shaggitza]

Priority

P0 — top engineering priority. This epic tracks the complete five-part AST/static-analysis accuracy audit.

Why this is first priority

The checked-in verification artifact reports mypy normalized TP=3, FP=0, FN=68 (4.23% recall), with most false negatives attributed to discovery before propagation. The artifact may predate current source, but the audit reproduced concrete false negatives, false positives, established-empty inventories, and benchmark-integrity risks.

The objective is not to claim complete Python runtime reachability. It is to maximize accuracy within declared static subsets while making ambiguity, unsupported behavior, and incompleteness explicit.

Workstreams

Correctness and inventory foundations

• P0: Fix baseline/target diff mapping, renames, paths, and endpoint lifecycle #274 — dual baseline/target diff mapping, renames, paths, lifecycle, and unresolved accounting
• P0: Make secure AST extraction ordered, control-flow aware, and fail closed #275 — ordered control-flow-aware secure AST extraction
• P0: Restore FastAPI include_router and route-subclass discovery #276 — FastAPI include_router and route-subclass compatibility
• P0: Preserve route assembly and qualified dependency provenance #277 — route assembly and qualified dependency provenance
• P0: Make accuracy benchmarks fail closed and measure real incremental latency #283 — fail-closed benchmarks and truthful incremental measurements

Semantic propagation and framework execution

• P0: Add CFG-aware execution semantics and deferred-body tracking #278 — CFG-aware reachability and deferred bodies
• P0: Add bounded callable points-to and conservative virtual dispatch #279 — callable points-to and conservative virtual dispatch
• P0: Scope and complete lifecycle, middleware, error, and background surfaces #280 — selected-app framework surfaces, lifecycle, errors, middleware, and background tasks

Isolation, effects, and extensibility

• P0: Isolate runtime extraction imports across packages and projects #281 — runtime import isolation
• P0: Correct effect confidence, resource identity, and preset coverage #282 — effect confidence, resource identity, argument flow, and presets
• P0: Build a shared hybrid evidence graph and source inventory #284 — shared hybrid evidence graph, source inventory, and optional semantic providers

Architecture decision

Retain CPython AST for framework registration/source ranges and mypy for typed calls. Add dual snapshots, bounded CFG/callable/receiver analysis, and a versioned evidence graph. SCIP, Pyright, tree-sitter, LibCST, CodeQL, and runtime traces are complementary providers or differential oracles, not wholesale replacements.

Program-level gates

• Supported generated typed DAG subset reaches 100% precision and recall.
• Unrelated/dead-code controls produce zero HIGH/MEDIUM candidates.
• Unsupported dynamic registration produces conditional/unavailable, never established-empty inventory.
• First real-world milestone: normalized recall >=20%, precision >=90%, no new negative-control false positive.
• Propagation milestone: recall >=50%, precision >=90%, every repository non-zero recall.
• Fresh prediction-blind release corpus: recall >=80%, precision >=90%, per-repository recall >=70%, resolved coverage >=95%.
• HIGH direct evidence precision remains 100%; LOW remains report-only.
• Incremental p95 <=30 seconds for the documented medium-project class, with cold build and peak RSS reported separately.

Validation policy

• Controlled/property-based call-graph and FastAPI program generation.
• Baseline/target delete-add, rename, and move metamorphic tests.
• Secure AST versus sandboxed runtime route-census differential tests.
• mypy versus optional SCIP/Pyright definition/call-target disagreements stored as review fixtures.
• Raw exact and normalized real-world metrics, confidence tiers, discovery/propagation attribution, unresolved coverage, and negative-control specificity.
• Missing and not_evaluable labels must never be treated as negatives.

Explicit limits

Dynamic imports, reflection, plugins, monkeypatching, descriptors, native extensions, runtime configuration, and unobserved production behavior remain fundamental static-analysis limits. Runtime observations may confirm edges; their absence cannot prove non-reachability.

Related roadmap/issues: #20, #97, #99, #100, #101, #104, #107, #109, #110, #144, #148.

[shaggitza]

Implementation started: #286 merged as 0023a96 with the first #274 correctness tranche. CI is green across lint/mypy, Python 3.10–3.12, coverage, and package build. Next foundation target is current-FastAPI route inventory compatibility (#276) or baseline/target snapshot separation (#274).

[shaggitza]

Completed #276 in PR #287 (45db2b0). Runtime extraction now supports FastAPI 0.139+ lazy route contexts, nested/reused includes, mounted routers, root mounts, HTTP/WebSocket route subclasses, effective metadata, fail-closed unknown shapes, minimum FastAPI CI, and secure/runtime differential coverage. Full GitHub Actions passed on Python 3.10–3.12, minimum FastAPI 0.100.0, lint, mypy, coverage, and package build.

[shaggitza]

Completed runtime import isolation in #288 (merged as bf1adad). Runtime discovery now uses a bounded fresh POSIX subprocess, supports package/namespace import contexts, prevents cross-project module contamination and stale bytecode reuse, cleans process groups on every exit path, and documents the trusted-only/non-hermetic boundary. CI passed on Python 3.10–3.12, FastAPI 0.100, lint, mypy, coverage, and package build.

[shaggitza]

P0 benchmark-integrity tranche merged in #289 (d06ca41). Issue #283 remains open: fail-closed schema/provenance/coverage reporting and honest cold timing are complete, while real incremental backend telemetry and remaining release/attestation gates are deferred. All CI checks passed; frozen benchmark artifacts were untouched.

[shaggitza]

Progress: #290 merged (bdca8b7), completing phase 1 of #275. Native secure-AST discovery now preserves configured-root identity, models source-ordered destructive uncertainty, and fails closed across direct mutations, dynamic execution, binding/control-flow changes, aliases, class globals, imported-router snapshots, and Python 3.10–3.12 AST differences. #275 remains open for ordered-transfer consolidation, factory/bootstrap migration, custom surfaces, and public inventory-strength propagation. CI run 30282484104 passed.

[shaggitza]

P0 progress: #291 merged (c50da8c) and completed issue #275 phase 2. Custom surfaces now use fail-closed ordered control-flow/binding semantics across match, try/except*, loops, with-statements, comprehensions, short-circuit expressions, dotted imports, exact aliases, and statement fallthrough. Full CI passed on Python 3.10–3.12. #275 remains open for its final consolidation/report-strength phases.

[shaggitza]

P0 progress: #292 merged (95bddca) and completed issue #275 phase 3. Secure analysis reports now retain whole-inventory strength across all public formats; custom roots and bootstrap entrypoints fail closed; startup activation evidence survives JSON/YAML. Full CI passed across Python 3.10–3.12. #275 remains open for eager-header, startup-state, bounded-evaluation, and consolidation work.