Codex/builder packet retrieval phase5

CHANGELOG.md

@@ -2,6 +2,25 @@
 
 ## Unreleased
 
+### Builder Packet & Repository Code Retrieval — vNext.10.2 Phase 5
+
+- Added a deterministic `BuilderPacketCompiler` that turns an explicitly
+  Secondary-selected Objective WorkUnit into a small implementation packet by
+  reusing `RepositoryContextIndex`, current worktree bytes, section extraction,
+  bounded structural expansion, and durable `ContextSelectionPlan` records.
+- Added automatic target, test, direct-dependency, and nearby reference-pattern
+  selection with per-section repository/path/hash/range/reason provenance,
+  verified dependency summaries, trusted verification hints, bounded prior
+  failure evidence, multi-repository identity, and stable semantic packet hashes.
+- Added honest `INSUFFICIENT_CONTEXT`, `AMBIGUOUS_TARGET`, and model-returned
+  `NEEDS_MORE_CONTEXT` outcomes. These outcomes create no edit candidate and do
+  not grant the direct model shell, Git, filesystem, repository browsing, or
+  other agent authority.
+- Added deterministic fake-model, stale-index, budget/sectioning,
+  protected-path, multi-repository, hash-stability, and large-fixture coverage,
+  plus a gated real local-model qualification. Default strong-builder routing
+  and explicit-only Secondary selection remain unchanged.
+
 ### Secondary Objective Builder — vNext.10.2 Phase 4
 
 - Added an explicit-only, provider-neutral `SecondaryModelInference` and

contracts/context-contract.json

@@ -141,6 +141,7 @@
     "MODULE_PROXIMITY",
     "PRIOR_TASK_RELEVANCE",
     "PROGRESSIVE_EXPANSION",
+    "REFERENCE_PATTERN",
     "SYMBOL_MATCH",
     "TEST_SOURCE_PAIR",
     "TOKEN_OVERLAP"

contracts/orchestration-contract.json

@@ -111,6 +111,11 @@
     "MODEL_DIAGNOSIS",
     "PROVIDER_SIGNAL"
   ],
+  "builderPacketCompilationFailures": [
+    "AMBIGUOUS_TARGET",
+    "INSUFFICIENT_CONTEXT",
+    "STALE_SOURCE_CONTEXT"
+  ],
   "computeLocalities": [
     "LOCAL",
     "REMOTE",
@@ -1043,6 +1048,7 @@
   ],
   "secondaryBuilderAttemptStatuses": [
     "CANDIDATE_READY",
+    "CONTEXT_INSUFFICIENT",
     "EDITS_APPLIED",
     "FAILED",
     "INFERENCE_COMPLETED",
@@ -1051,12 +1057,14 @@
     "VERIFICATION_FAILED"
   ],
   "secondaryBuilderFailures": [
+    "AMBIGUOUS_TARGET",
     "APPLY_FAILURE",
     "CANCELLED",
     "CONTEXT_TOO_LARGE",
     "EMPTY_EDIT_SET",
     "FORBIDDEN_EDIT",
     "INFERENCE_UNAVAILABLE",
+    "INSUFFICIENT_CONTEXT",
     "INVALID_STRUCTURED_OUTPUT",
     "STALE_APPROVED_PROJECTION",
     "STALE_SOURCE_CONTEXT",

contracts/schema-versions.json

@@ -7,12 +7,13 @@
   "attemptRecord": "1.0.0",
   "autonomyTelemetry": "1.0.0",
   "browserScenario": "1.0.0",
+  "builderPacketCompilation": "1.0.0",
   "candidateArtifact": "1.1.0",
   "closureLedger": "1.0.0",
   "contextMetrics": "1.0.0",
   "contextPackage": "1.0.0",
   "contextProjection": "1.0.0",
-  "contextSelectionPlan": "1.0.0",
+  "contextSelectionPlan": "1.1.0",
   "contractConflict": "1.0.0",
   "controlPlaneRepair": "1.0.0",
   "dogfoodRun": "1.0.0",
@@ -60,9 +61,9 @@
   "runnerContextCapabilities": "1.0.0",
   "runnerOutput": "1.0.0",
   "schedulingDecision": "1.0.0",
-  "secondaryBuilderAttempt": "1.0.0",
-  "secondaryBuilderPacket": "1.0.0",
-  "secondaryBuilderResult": "1.0.0",
+  "secondaryBuilderAttempt": "1.1.0",
+  "secondaryBuilderPacket": "1.1.0",
+  "secondaryBuilderResult": "1.1.0",
   "specIntakeApproval": "1.0.0",
   "specIntakeDelta": "1.0.0",
   "specIntakeGrounding": "1.0.0",

docs/README.md

@@ -156,6 +156,9 @@ the intent in the evening, run one command, read the report in the morning.
 - [Secondary Objective Builder](orchestration/secondary-objective-builder.md)
   — explicit direct-model edits inside the existing governed worktree and
   candidate lifecycle; capability without automatic routing.
+- [Builder Packet compilation](orchestration/builder-packet-compilation.md)
+  — deterministic, fresh, bounded source/test/dependency retrieval for an
+  explicitly selected Secondary Builder; context quality without routing.
 - [Context projection](orchestration/context-projection.md) — share truth,
   not context: what each isolated worker sees, hashed and staleness-checked.
 - [Sub-agent isolation](orchestration/subagent-isolation.md) — worker

docs/local-model.md

@@ -7,6 +7,9 @@ implementation still reaches the configured large agent; the explicit-only
 [Secondary Objective Builder](orchestration/secondary-objective-builder.md)
 can also use this endpoint for bounded worktree edits. One managed server serves
 every logical role — roles are prompts, not processes.
+Its [Builder Packet compiler](orchestration/builder-packet-compilation.md)
+selects fresh source and tests first, so the direct model needs no repository
+tools for ordinary bounded implementation work.
 
 The local model is a **worker**, never an authority: its answers are
 schema-validated structured proposals that deterministic policy accepts,

docs/orchestration/builder-packet-compilation.md

@@ -0,0 +1,148 @@
+# Builder Packet compilation
+
+Phase 5 of vNext.10.2 prepares the implementation evidence a bounded direct
+model needs. It does not choose a model. An Objective WorkUnit reaches this
+path only after a caller explicitly selects the Secondary Objective Builder.
+
+```text
+WorkUnit + ContextProjection + verified dependency candidates
+                              + trusted verification configuration
+        ↓ deterministic query
+existing RepositoryContextIndex metadata
+        ↓ ranking + bounded structural expansion
+fresh bytes from the isolated Objective worktree
+        ↓ sectioning + category/character budgets
+small, explainable SecondaryBuilderPacket
+        ↓ one provider-neutral inference request
+normal worktree → verification → candidate lifecycle
+```
+
+## Compiler boundary
+
+`BuilderPacketCompiler` is provider-neutral and independently testable.
+`SecondaryBuilderContextCompiler` is the managed implementation. Compilation
+does not invoke an LLM, run tests, execute commands, or grant repository
+tools. It receives the WorkUnit, approved `ContextProjection`, attempt and
+baseline identity, verified dependency summaries, bounded prior failure
+evidence, trusted verification hints, and a capability-neutral budget.
+
+The compiler returns either a packet or an honest `INSUFFICIENT_CONTEXT` /
+`AMBIGUOUS_TARGET` result. It does not guess a target and it does not decide
+whether a strong model should run instead.
+
+## Existing index and deterministic retrieval
+
+There is no second search engine. Phase 5 reuses the context package's
+`RepositoryContextIndex`, retrieval query, ranking, section extraction,
+freshness, budget, and `ContextSelectionPlan` contracts. No vector database,
+embedding model, semantic search service, or model reranker is required.
+
+The query is derived from durable WorkUnit evidence: title, goal, expected
+artifacts and areas, contract/acceptance text, named paths and symbols,
+verified dependency paths, prior selected paths, and bounded failure output.
+Ranking preserves the existing hierarchy: explicit and failure references
+are mandatory; symbol and test/source evidence outrank dependency and module
+proximity; token overlap is only a hint. Nearby same-module implementations
+with the same suffix/convention or shared imports may receive the bounded
+`REFERENCE_PATTERN` reason. They are labeled as examples, never requirements.
+
+Default expansion stops at `ADJACENT_DEPENDENCIES`: target source, direct
+dependencies/dependents, and paired tests. The API exposes the existing
+closed expansion levels for a later bounded widening decision; Phase 5 does
+not implement an agentic search loop.
+
+## Freshness and worktree state
+
+The canonical repository index is reusable metadata. For an Objective
+attempt, the compiler overlays only paths changed between the canonical
+checkout and the isolated worktree, including already-verified dependency
+patches. This keeps dependency interfaces current without scanning the full
+worktree. Unfinished sibling worktrees are never an input.
+
+Only selected files are materialized. Selection re-reads current worktree
+bytes and compares SHA-256 hashes. Each packet section carries the complete
+file hash plus a hash of the exact supplied whole file or section. Execution
+checks those hashes again immediately before inference. A stale cache may be
+refreshed, but stale source is never presented as current.
+
+## Packet contents and budgets
+
+Approved truth stays in `ContextProjection`; repository evidence does not
+duplicate or replace it. Packet 1.1 contains:
+
+- objective and WorkUnit identity, approved requirements, constraints, ADRs,
+  decisions, and verified prior-work summaries;
+- explicit targets;
+- current implementation source or deterministic source sections;
+- relevant tests as a first-class section;
+- clearly labeled reference patterns;
+- verified dependency-candidate summaries and changed paths;
+- bounded prior failure evidence and trusted verification hints;
+- retrieval-plan references, expansion level, retrieval metrics, and quality
+  facts.
+
+Category and character limits preserve targets and tests before low-value
+siblings. Large files use deterministic declaration/line-centred sections;
+small files may be included whole. Generated, vendor, ignored, binary,
+credential-shaped, protected, unreadable, and oversized artifacts retain the
+existing index exclusions. Compilation never runs verification; hints come
+only from configured commands and actual verification remains after edits.
+
+## Explainability and semantic identity
+
+Every source, test, and reference item records repository identity, path,
+selection reason, file hash, section hash, optional line range, and symbols.
+The exact metadata plan is stored under the existing job-scoped context plan
+store; source bodies remain repository-derived and are not copied into a new
+cache.
+
+`contentHash`/`packetHash` canonically bind WorkUnit and approved-truth
+identity, selected repositories/paths/hashes/ranges, dependency evidence,
+verification hints, and packet schema. `createdAt` and observational timing
+metrics are excluded, so recompiling unchanged logical inputs has the same
+semantic identity.
+
+## Multi-repository behavior
+
+Repository identity namespaces every path. A justified cross-repository
+WorkUnit may supply multiple bounded indexes and roots, and its packet can
+therefore distinguish `backend:src/client.ts` from
+`frontend:src/client.ts`. Secondary repositories must be explicitly supplied
+and justified; compilation does not make all discovered repositories
+available automatically. Edit authority remains with the primary isolated
+worktree.
+
+## Context insufficiency
+
+Missing explicit targets, unresolved verified dependencies, protected or
+unmaterializable target source, and equal plausible targets stop compilation.
+During inference the model may also return `NEEDS_MORE_CONTEXT` with bounded
+reasons and no edits. Both paths create no candidate. Phase 6 may use recorded
+quality facts for eligibility; Phase 7 may decide widening, repair, or
+fallback. Neither policy exists in Phase 5.
+
+Metrics record indexed files considered, ranked candidates, selected files
+and sections, source/test characters, reference/dependency counts, budget
+use, mandatory references retained, expansion depth, stale entries,
+selection duration, and index reuse. Quality facts record target resolution,
+ambiguity, tests, verification hints, reference patterns, dependency
+completeness, budget utilization, and structural sufficiency. They are facts,
+not routing eligibility.
+
+## Qualification
+
+Deterministic tests cover explicit target retrieval, dependency/test
+expansion, reference patterns, budget pressure, sectioning, stale indexes,
+missing and ambiguous targets, protected paths, multi-repo identity, stable
+hashes, `NEEDS_MORE_CONTEXT`, large-index boundedness, and the automatic
+compiler → Secondary Builder → verified candidate flow.
+
+The opt-in real llama.cpp/Qwen qualification now compiles the target source
+automatically before inference and records packet metrics. CI skips it unless
+the local server/model environment variables documented in
+[Secondary Objective Builder](secondary-objective-builder.md#qualification)
+are present.
+
+Builder Packet compilation does not decide which model should execute the
+task. Phase 6 eligibility, Phase 7 routing/repair/fallback, LLM Gateway,
+Vector RAG, and OpenMind are not implemented here.

docs/orchestration/secondary-objective-builder.md

@@ -5,7 +5,7 @@ Objective runtime. A bounded direct model can propose concrete file edits for
 one build WorkUnit, while SpecBridge retains every authority-bearing action:
 
 ```text
-ContextProjection (approved truth) + explicit bounded source context
+ContextProjection (approved truth) + bounded repository context
         ↓ one SecondaryModelInference request
 strict CREATE/REPLACE proposal
         ↓ SpecBridge validates and writes
@@ -17,8 +17,9 @@ normal candidate → evaluation → aggregation → single-writer integration
 Phase 4 creates capability. It does **not** automatically route Objective work
 to the secondary backend. With no explicit `secondaryObjectiveBuilder`
 selection, BUILDER behavior remains the existing large-agent path. Automatic
-packet compilation, eligibility, routing, repair, and fallback belong to later
-phases.
+eligibility, routing, repair, and fallback belong to later phases. Phase 5
+adds deterministic [Builder Packet compilation](builder-packet-compilation.md)
+without changing this explicit-only selection rule.
 
 ## Not an Agent Harness
 
@@ -43,12 +44,15 @@ heuristically.
 It carries WorkUnit and Objective identity, goal, expected artifacts/areas,
 acceptance criteria, projected contracts, ADRs and approved decisions,
 constitution constraints, verified dependency evidence, fixed forbidden
-changes, verification names, and explicitly prepared source files.
+changes, trusted verification names, and compiler-selected source sections,
+tests, dependency evidence, and reference patterns.
 
 `ContextProjection` remains approved durable truth. Source is deliberately a
-separate `sourceContext` list of worktree-relative path, SHA-256 content hash,
-and bounded current UTF-8 content. Phase 4 callers choose those files
-explicitly; there is no automatic repository retrieval or target inference.
+separate repository context with repository identity, path, selection reason,
+whole-file hash, exact section hash/range, and bounded current UTF-8 content.
+Legacy Phase 4 callers may still provide `sourceContext` explicitly. When an
+explicit Secondary selection omits it, Phase 5 compiles the packet from the
+WorkUnit and existing `RepositoryContextIndex`.
 Immediately before inference, SpecBridge reloads approved Mission truth and
 checks projection freshness, then re-reads every source file and checks its
 hash and bytes. Stale approved truth or source fails the attempt before the
@@ -105,7 +109,10 @@ it is not rebuilt merely because the process restarted.
 
 Failure kinds distinguish inference unavailability, timeout, cancellation,
 invalid structured output, empty edits, forbidden edits, stale approved/source
-context, application failure, verification failure, and an oversized context.
+context, insufficient or ambiguous retrieval, application failure,
+verification failure, and an oversized context. A model may return the
+structured `NEEDS_MORE_CONTEXT` status; this records an insufficient attempt
+and creates no candidate.
 Phase 4 performs no secondary repair retry and no automatic large-model
 fallback.
 

integrations/claude-code-plugin/specbridge/dist/checksums.json

@@ -7,12 +7,12 @@
       "bytes": 155994
     },
     "cli.cjs": {
-      "sha256": "6379044267e76482b59da2c3b57ce97fe5f758815d10a9fefa0c8267bb53e9a9",
-      "bytes": 5588127
+      "sha256": "8bc205156d51810ec082b0e9aa3be5d2f12c307706e680796ca4909360dee2e7",
+      "bytes": 5634305
     },
     "mcp-server.cjs": {
-      "sha256": "6b8a8ce0992a5cefa9e15aadbe2e0aa22a97bf341c0db8c8a599a5ba9a8024d5",
-      "bytes": 3644860
+      "sha256": "3854958225b757a1e0a2197471faf308fbf4a97ab58c51507114e7009748aea5",
+      "bytes": 3654373
     }
   }
 }