diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f8db23..e4aad9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,42 @@ and Python distribution versions follow PEP 440. ## [Unreleased] +## [0.5.0] - 2026-08-24 + +### Added + +- Added immutable, model-free repository discovery application contracts for + deterministic candidate preparation, read-only expansion, verified source + reads, and canonical context packaging. +- Added the generic ContextForge bridge protocol v1 ADR, normative JSON Schema, + persistent workspace-bound JSON-RPC 2.0 service over bounded UTF-8 NDJSON + stdio, and a runnable generic client example. +- Added explicit protocol negotiation and compatibility failures for missing or + unsupported v1 handshakes, cooperative JSON-RPC request cancellation, typed + source-identity drift, deterministic preparation/selection identities, and + bounded concurrent requests with serialized responses. +- Added bridge architecture, CLI, troubleshooting, security, stream, + cancellation, read-only, repository-truth, and source-verification guidance. + +### Changed + +- Kept model-assisted discovery on the existing provider-backed path while + sharing its snapshot, mode, ranking, stale-index, path, and budget semantics + with the new public preparation API. +- Stabilized the public model-free expansion DTO around closed operations and + explicit result fields. Removed the unused competing bridge envelope and + stopped JSON-RPC responses from serializing internal model actions, tool + names, steps, request echoes, or whole application DTOs. +- Kept the read-only MCP server independent and unchanged while adding the + separate bridge transport. + +### Security + +- Defined bridge v1 as a trusted-local integration transport rather than an + authentication, sandbox, network, or multi-tenant boundary. The fixed + workspace permits verified reads only: no provider/network access, shell, + arbitrary subprocess, source/Git/index mutation, or disk artifact publish. + ## [0.4.2.post1] - 2026-08-21 ### Fixed diff --git a/README.md b/README.md index d363f97..c490370 100644 --- a/README.md +++ b/README.md @@ -28,8 +28,8 @@ commands.

> [!IMPORTANT] -> ContextForge is pre-alpha software. Version `0.4.2.post1` is the corrected -> first public-release candidate. Discovery benchmarking is experimental and +> ContextForge is pre-alpha software. Version `0.5.0` adds the stable generic +> local bridge v1. Discovery benchmarking is experimental and > its results should be reviewed alongside the recorded provider, model, > configuration, and source snapshot. @@ -183,12 +183,16 @@ mutating operations. | `contextforge diagnostics config [PATH]` | Explain effective configuration | Read-only | | `contextforge diagnostics provider [PATH]` | Show provider policy without probing it | Read-only | | `contextforge mcp serve [PATH]` | Run the local read-only stdio MCP server | Read-only session | +| `contextforge bridge --stdio --workspace PATH` | Run persistent JSON-RPC bridge v1 | Verified read-only workspace session | | `contextforge benchmark discovery PATH` | Run manifest-driven discovery benchmarks | Repository/index read-only; experimental | Global diagnostic options are `--log-level`, `--log-format`, `--log-file`, repeatable `--log-component`, `--no-log-file`, `--no-color`, and `-v`/`-vv`. Detailed syntax, defaults, streams, side effects, mistakes, and examples are in the [Wiki CLI reference](https://github.com/waterflane/ContextForge/wiki/CLI-Overview). +The local integration contract is documented in the +[bridge v1 guide](docs/guides/bridge.md), with a runnable +[generic client](examples/generic_bridge_client.py). ## Configuration diff --git a/ROADMAP.md b/ROADMAP.md index 41ec208..04d3bf1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -60,6 +60,17 @@ This roadmap describes broad milestones only. It does not promise dates. - [ ] Publish the reviewed signed `v0.4.2.post1` release through TestPyPI and PyPI. +## v0.5: Generic local integration bridge (release prepared) + +- [x] Expose immutable model-free candidate, expansion, verified-read, and + package application contracts without changing model-assisted discovery. +- [x] Add a persistent workspace-bound JSON-RPC 2.0 bridge over UTF-8 NDJSON + stdio with explicit protocol v1 negotiation and cancellation. +- [x] Keep repository truth, source identity verification, path policy, and + read-only guarantees inside ContextForge while consumers own model selection. +- [x] Keep MCP independent and preserve its existing read-only protocol. +- [ ] Publish or tag 0.5.0 after review and explicit release approval. + ## Later - Full multi-root workspaces. @@ -67,8 +78,8 @@ This roadmap describes broad milestones only. It does not promise dates. - Optional supplementary retrieval strategies that do not replace complete allowed-tree/text access. - Graphical workspace review UI. -- External coding-agent integrations that consume handoffs under their own - authority. +- Richer external integrations that consume bridge results and handoffs under + their own authority. Autonomous edits, coding-agent orchestration, Git worktree management, and Git mutation are not part of the current roadmap. diff --git a/SECURITY.md b/SECURITY.md index 06d4ffd..266b419 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,8 +6,8 @@ ContextForge is pre-1.0 software. Security fixes target the latest release. | Version | Supported | | --- | --- | -| Latest `0.4.x` release | Yes | -| `0.3.x` and earlier | No | +| Latest `0.5.x` release | Yes | +| `0.4.x` and earlier | No | ## Reporting a vulnerability @@ -98,3 +98,22 @@ discovery executor as in-process callers. It advertises only read-only tools and resources: no sampling, remote transport, subscriptions, source/index mutation, shell/process execution, Git mutation, or agent orchestration. Protocol output is isolated on stdout; diagnostics use stderr. + +The ContextForge bridge is also a local stdio transport for trusted +integrations, not a network service, sandbox, authorization layer, or +multi-tenant boundary. The process inherits the invoking user's ability to read +the bound workspace. Only launch clients you trust with repository excerpts, +candidate metadata, index provenance, and context packages. Do not expose the +bridge through a socket, web endpoint, shared pipe, privilege boundary, or +untrusted process broker without adding authentication, authorization, +isolation, and transport limits outside ContextForge. + +Bridge v1 is model-free and has no network, provider, shell, subprocess, +source-write, Git-mutation, or index-mutation capability. Its package operation +returns an in-memory artifact and does not publish a file. `snapshot`, +`expected_snapshot_digest`, per-file SHA-256 identities, stable source reads, +and `SOURCE_IDENTITY_CHANGED` prevent stale or substituted excerpts from being +accepted as current repository truth. They detect changes; they do not turn a +hostile same-account process into a safe peer. Stdout is reserved for bounded +JSON-RPC frames and bounded diagnostics use stderr; both can contain sensitive +repository-derived information and must be protected accordingly. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md index 2c0f5a2..7625c9e 100644 --- a/docs/RELEASE_CHECKLIST.md +++ b/docs/RELEASE_CHECKLIST.md @@ -35,13 +35,16 @@ uv run pytest uv build uv run twine check dist/* uv run check-wheel-contents dist/*.whl +uv run python scripts/validate_distribution.py dist ``` Inspect wheel and sdist contents. They may contain only package source, `pyproject.toml`, README, LICENSE, NOTICE, CHANGELOG, required packaging metadata, and Hatch's backend-required `.gitignore` build-control file. They must not contain tests, Wiki files, planning notes, GitHub metadata, caches, -local state, logs, reports, credentials, or heavy images. +local state, logs, reports, credentials, temporary bridge files, local paths, +or development artifacts. Both archives must contain the complete +`contextforge.bridge` package. Install the wheel and sdist in separate clean Python 3.12 and 3.13 environments. Verify only package metadata, importability, and version entry points: @@ -51,8 +54,14 @@ python -c "import contextforge" contextforge --version ctxf --version python -m contextforge --version +contextforge bridge --help ``` +Run an interactive subprocess smoke test that negotiates bridge protocol `1.0`, +captures a real `snapshot`, and performs clean `shutdown`. Confirm every stdout +line is JSON-RPC, stderr is separate, and the bound fixture workspace is byte-for-byte +unchanged. Run the independent MCP stdio tests in the full pytest suite. + Release validation must not require Ollama, LM Studio, a remote model API, an API server, or model-backed smoke tests. Tests use structural-only behavior or the deterministic fake provider. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 60f0b03..015472b 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -27,6 +27,8 @@ The project is organized as a modular monolith with these boundaries: - `repositories`: repository and language analysis adapters; - `storage`: storage adapters; - `models`: model-provider adapters; +- `bridge`: trusted-local JSON-RPC/NDJSON transport over the model-free + application boundary; - `cli`: command-line interface; - `api`: local HTTP API; - future IDE integrations outside the core. @@ -61,7 +63,8 @@ a deterministic fake, and a local Ollama adapter. Incremental model-assisted [file and symbol semantic analysis](semantic-analysis.md) stores interpretations separately from source facts. Bounded hierarchical [repository architecture and feature maps](repository-maps.md) preserve the -same facts-versus-interpretation boundary. Model-guided repository discovery is +same facts-versus-interpretation boundary. Model-free and model-assisted +repository discovery are documented in [Repository discovery](repository-discovery.md), and its review-to-package integration and pure prompt compiler are documented in [Context handoffs and prompt compilation](context-handoffs.md). ContextForge @@ -75,7 +78,12 @@ decisions, request budgets, and causal errors for CLI and future interfaces without turning progress refreshes into logs. Thin Typer commands expose index and context workflows, while a bounded read-only MCP adapter exposes the same core APIs without shell, source-write, Git-mutation, -or index-mutation capabilities. +or index-mutation capabilities. The independent +[generic bridge v1](../guides/bridge.md) is a second local stdio adapter. It +binds one workspace, keeps repository truth and source verification inside +ContextForge, and lets the consumer own model selection and orchestration. MCP +and bridge do not depend on one another and neither introduces transport logic +into the Python core. ## Excluded from v0.1.0 @@ -97,3 +105,5 @@ repository intelligence, bounded discovery, handoffs, and read-only MCP are implemented in v0.4. Version 0.4.1 adds maintenance-level progress, diagnostics, nested-ignore, and CLI usability improvements; it does not claim future GUI, remote transport, orchestration, or source-mutation functionality. +Version 0.5.0 adds the trusted-local, model-free, read-only bridge v1 without +changing the existing model-assisted discovery or MCP semantics. diff --git a/docs/architecture/repository-discovery.md b/docs/architecture/repository-discovery.md index 16b8b9a..f066db1 100644 --- a/docs/architecture/repository-discovery.md +++ b/docs/architecture/repository-discovery.md @@ -1,4 +1,4 @@ -# Model-guided repository discovery +# Repository discovery ## Implemented boundary @@ -60,19 +60,48 @@ contracts are documented in the ## Public API -The main entry point is asynchronous: +The model-free application boundary exposes four synchronous operations over +immutable DTOs: + +```python +prepared = prepare_discovery_candidates(snapshot, request) +expanded = expand_discovery(snapshot, prepared, expansion) +verified = read_verified_context(snapshot, prepared, selection) +package = package_verified_context(snapshot, verified) +``` + +Candidate preparation and expansion reuse the same mode-aware knowledge loader, +ranking, path policy, budgets, index freshness checks, and verified source tools +as model-assisted discovery. They never require or invoke a `ModelProvider`. +Prepared and verified DTOs contain portable paths and source identities but no +repository root, filesystem handle, mutable executor, or `DiscoverySession`. +The public expansion DTO names a closed `operation` and returns explicit +`ok`, `code`, `data`, truncation, progress, and budget fields. Internal +model-assisted `action_id`, `tool_name`, step counters, and +`DiscoveryObservation` objects do not cross this application boundary. + +The existing model-assisted entry point remains asynchronous: ```python record = await discover_repository(snapshot, provider, request) ``` `DiscoverySession` exposes the same lifecycle for callers that need in-progress -state. Core closed models include `DiscoveryRequest`, `DiscoveryMode`, +state inside trusted Python integrations. It is not a bridge DTO. Core closed +models include `DiscoveryRequest`, `DiscoveryMode`, `DiscoveryAction`, `DiscoveryObservation`, `DiscoveryState`, `DiscoveryBudget`, `DiscoveryCandidate`, `SelectionReason`, `FinalContextSelection`, `CompletenessWarning`, and `DiscoveryRunRecord`. Typed failures carry a run record whose `final_selection` is always absent. +The trusted-local stdio adapter is described by +[ADR-002](../decisions/002-generic-context-bridge-protocol-v1.md), the +[bridge guide](../guides/bridge.md), and the +[protocol v1 JSON Schema](../schemas/contextforge-bridge-v1.schema.json). +JSON-RPC, NDJSON framing, protocol negotiation, concurrency, cancellation, and +process lifetime remain entirely in `contextforge.bridge`, not in the +application DTOs or discovery core. + ## Tool and security boundary `DISCOVERY_TOOL_SCHEMAS` publishes closed input schemas for repository overview, diff --git a/docs/decisions/002-generic-context-bridge-protocol-v1.md b/docs/decisions/002-generic-context-bridge-protocol-v1.md new file mode 100644 index 0000000..60c7d35 --- /dev/null +++ b/docs/decisions/002-generic-context-bridge-protocol-v1.md @@ -0,0 +1,145 @@ +# ADR-002: Generic ContextForge bridge protocol v1 + +## Status + +Accepted and implemented for the ContextForge 0.5.0 foundation. + +## Context + +Local agents, IDE integrations, MCP adapters, and external harnesses need the +same repository discovery capabilities without importing ContextForge's mutable +`DiscoverySession` or forcing ContextForge to choose or call their model. +Duplicating repository scanning, index freshness checks, path authorization, or +source reads in each integration would create competing definitions of +repository truth and weaken the existing security boundary. + +The bridge is a persistent workspace-bound process started with +`contextforge bridge --stdio --workspace PATH`. It uses JSON-RPC 2.0 over +UTF-8 NDJSON and retains only bounded preparation state in memory. + +## Decision + +ContextForge defines a generic bridge protocol v1 around four operations: + +- `prepare_discovery_candidates` produces deterministic, bounded candidates; +- `expand_discovery` performs one closed read-only evidence operation; +- `read_verified_context` verifies a caller-owned candidate selection; +- `package_verified_context` re-verifies and builds the canonical context + package. + +The Python application API uses frozen, closed Pydantic DTOs. It never returns +`DiscoverySession`, `DiscoveryKnowledge`, `DiscoveryToolExecutor`, filesystem +handles, mutable budget trackers, or internal model-action observations. The +bridge projects those DTOs into an independent closed JSON-RPC contract; it does +not serialize application models wholesale. Any local harness can consume it, +but no message, operation, or Python type is named for or coupled to a specific +harness. + +## Ownership of repository truth + +ContextForge owns repository truth for every operation. An in-process caller +identifies a repository root or passes an already verified `ProjectSnapshot`; +a bridge client is restricted to the workspace fixed when the process starts. +ContextForge scans according to its existing ignore/protection policy, produces +portable case-sensitive paths and source identities, and authorizes all reads +against that inventory. Index records and model-generated semantics are hints; +they never override current source. + +The integration owns task orchestration and candidate choice. It may use any +model, no model, or user input, but can select only identifiers issued by the +preparation it supplies back to ContextForge. + +## Snapshot and index pinning + +Candidate preparation includes a `source_snapshot_digest`, optional immutable +`index_generation_id`, and deterministic `preparation_id`. Later operations +must present that preparation. ContextForge re-scans or checks the supplied +snapshot and rejects the operation if repository identities, the pinned index +generation, mode semantics, request constraints, candidates, or warnings no +longer match. + +`fresh` ignores persistent semantic and repository-wide maps. `indexed` +requires current structural records. `hybrid` uses current indexed records and +fills structural gaps from current source. Stale records are excluded, stale +global maps are not used, and stale paths are disclosed. + +## Source verification and budgets + +Candidate metadata carries portable paths, byte sizes, and SHA-256 source +identities. Expansion reads reuse the closed discovery tool schemas and stable +reader. Final reads are all-or-nothing, re-authorize exact snapshot membership, +reject links and traversal, verify size and hash, decode strict UTF-8, and apply +file/source/result/context budgets. Package construction re-reads source and +checks the produced block identities against the verified DTO. + +Budget usage is explicit and immutable at the public boundary. A caller carries +forward the latest usage between otherwise stateless expansion messages. A +caller cannot lower already charged preparation usage. + +## Cancellation + +Protocol v1 uses the JSON-RPC request `id` as its cancellation identity and +defines `$/cancelRequest` with that target ID. Cancellation is cooperative and +all-or-nothing: no cancelled read or package operation returns a partial +successful artifact. The bridge maps request IDs to application cancellation +signals. Cancellation is never permission to mutate source or the index. + +## Transport isolation + +Protocol v1 exposes `hello`, `status`, `snapshot`, `discover`, `expand`, `read`, +`package`, `$/cancelRequest`, and `shutdown`. Standard input is bounded UTF-8 +NDJSON and standard output contains only serialized JSON-RPC response frames. +Diagnostics are bounded and written only to standard error. Independent +requests may run concurrently, while all response writes are serialized. + +Repository-sensitive methods require `expected_snapshot_digest` after +`snapshot`. Drift is returned as typed `SOURCE_IDENTITY_CHANGED`, never as a +partial result. Cancellation IDs are JSON-RPC request IDs and cancellation is +carried into the application operation through its cooperative event. + +The process is a trusted-local integration transport. It provides no network +listener, authentication, authorization, encryption, tenant isolation, or +sandbox boundary. It inherits the invoking user's read access, so only trusted +local consumers may launch it. Stdout responses and stderr diagnostics are +sensitive local streams and must not be exposed through an untrusted broker. + +The adapter never exposes shell execution, source writes, Git mutation, +arbitrary subprocesses, path-policy bypasses, or direct index mutation. Package +results remain in memory and are returned in the response. + +## Model ownership + +Model selection remains outside the Python bridge. ContextForge cannot know an +integration's latency, cost, privacy, routing, or capability policy. Candidate +preparation and evidence expansion are deterministic and require no +`ModelProvider`. Existing model-assisted `discover_repository` remains a +compatible in-process workflow and continues to own its provider lifecycle. + +## Compatibility and versioning + +Protocol version `1.0` is independent of the Python package version and of the +discovery/context package schema versions. The client must call `hello` with an +explicit version before repository work. Missing negotiation and unsupported +versions are typed failures, and `hello` reports the selected and supported +versions. V1 messages are closed: unknown fields and unknown operations are +rejected. Within v1, new optional fields may be added only when old readers can +safely ignore them through a negotiated minor-version capability; otherwise +the protocol major version changes. +Required-field changes, changed path or budget semantics, weaker verification, +or changed success/failure meaning require a new major version. + +Serialized DTOs use deterministic UTF-8 JSON with sorted keys, no non-finite +numbers, and a final LF when persisted. Request and response IDs are correlation +metadata, not repository or source identities. Error responses are structured, +safe, and never share a result payload. + +## Consequences + +- Harnesses and IDEs can orchestrate discovery without a Python-side model. +- Existing model-assisted discovery semantics remain unchanged. +- Repository truth, immutable index generations, verification, and path policy + remain centralized. +- Stateful transport concerns and richer working-set/session behavior remain + explicitly deferred. +- Read-only MCP remains a separate adapter with its own protocol and lifecycle; + bridge availability does not change MCP methods or discovery semantics. diff --git a/docs/guides/bridge.md b/docs/guides/bridge.md new file mode 100644 index 0000000..d32b908 --- /dev/null +++ b/docs/guides/bridge.md @@ -0,0 +1,163 @@ +# Generic bridge v1 + +ContextForge bridge v1 is a persistent, workspace-bound, model-free service for +trusted local integrations. Start it as a child process: + +```bash +contextforge bridge --stdio --workspace /path/to/repository +``` + +It is not a remote API, sandbox, session manager, model gateway, or agent +orchestrator. The consumer owns model selection, prompts, retries, working-set +state, and candidate choice. ContextForge retains repository truth: scanning, +ignore/protection policy, current index provenance, path authorization, source +identity, verified reads, budgets, and canonical package construction. + +The transport is JSON-RPC 2.0 over UTF-8 NDJSON. Each request and response is +one JSON object followed by LF. Stdout contains response frames only. Bounded +diagnostics go to stderr; a client must never parse stderr as protocol data. +Repository excerpts and packages on stdout, and operational details on stderr, +can be sensitive. + +## Required handshake and versioning + +Protocol versioning is independent of the ContextForge package version and the +context-package schema. The first application request must be: + +```json +{"jsonrpc":"2.0","id":1,"method":"hello","params":{"protocol_version":"1.0","client_name":"example"}} +``` + +The response reports `protocol_version`, `supported_protocol_versions`, exact +method/operation capabilities, limits, and read-only policy. A client must +compare the selected version with the version it implements before continuing. +Missing negotiation returns `PROTOCOL_NEGOTIATION_REQUIRED`; an unsupported +version returns `INCOMPATIBLE_PROTOCOL_VERSION` and the supported list. + +V1 requests and parameter objects are closed. Unknown methods, fields, and +operations fail rather than being ignored. Compatible optional evolution +requires explicit minor-version negotiation. Required fields, changed source or +budget semantics, weaker verification, or changed success/failure meaning +require a new major version. The normative frame schema is +[`contextforge-bridge-v1.schema.json`](../schemas/contextforge-bridge-v1.schema.json). + +## Repository flow + +1. Call `snapshot` and retain its `snapshot_digest`. +2. Pass that value as `expected_snapshot_digest` to every repository-sensitive + `discover`, `expand`, `read`, or `package` request. +3. Give the consumer or its selected model the deterministic candidate metadata. +4. Optionally call one of the closed `expand` operations to obtain more bounded + evidence. +5. Send caller-selected candidate IDs to `read` or `package`. Including the + candidate `path` and `source_sha256` adds explicit identity assertions. + +Example discovery request: + +```json +{"jsonrpc":"2.0","id":3,"method":"discover","params":{"expected_snapshot_digest":"<64 hex characters>","task":"Locate the configuration loader","mode":"hybrid"}} +``` + +`discover` is model-free: `model_provider_used` is always `false`. Ranking is +deterministic for the pinned repository/index state. The consumer may use any +model, no model, or human input to select candidates. ContextForge never accepts +an arbitrary path as a substitute for a prepared candidate. + +The public `expand` operations in v1 are `symbol`, `text`, `callers`, +`importers`, and `related_tests`. Their `data` is bounded structured evidence. +No internal discovery action IDs, tool names, mutable executors, or model-side +session state are part of the bridge contract. + +## Method contract + +| Method | Required parameters | Result purpose | +| --- | --- | --- | +| `hello` | `protocol_version` | Negotiated version, package version, capabilities, workspace identity, and policy | +| `status` | none | Current readiness, source drift, and read-only index status | +| `snapshot` | none | New authoritative digest and bounded inventory summary | +| `discover` | `expected_snapshot_digest`, `task` | Deterministic candidates and preparation identity; never a model call | +| `expand` | digest, preparation ID, operation | One bounded read-only evidence result and cumulative usage | +| `read` | digest, preparation ID, non-empty items | All-or-nothing verified source excerpts and selection identity | +| `package` | digest, preparation ID, non-empty items | Re-verified canonical `ContextPackage` schema 1 in memory | +| `$/cancelRequest` | target request `id` | Cooperative cancellation notification or optional acknowledgement | +| `shutdown` | none | Stop accepting work and terminate after the response | + +Every params object is closed and may also carry `timeout_ms` from 1 through +900,000, except cancellation, which accepts only its target `id`. Discover +defaults to `hybrid`; optional pinned/excluded paths must be unique, sorted, +portable relative paths. Read/package items must be unique and sorted by +candidate ID; line ranges are one-based, inclusive, sorted, and disjoint. See +the normative schema for every budget and response field. + +JSON-RPC standard errors retain their numeric meaning. ContextForge also puts a +stable uppercase typed code in `error.data.code`. Integration-relevant v1 codes +include `PROTOCOL_NEGOTIATION_REQUIRED`, `INCOMPATIBLE_PROTOCOL_VERSION`, +`SNAPSHOT_REQUIRED`, `SOURCE_IDENTITY_CHANGED`, `UNKNOWN_PREPARATION`, +`UNKNOWN_CANDIDATE`, `REQUEST_CANCELLED`, `REQUEST_TIMEOUT`, +`DUPLICATE_REQUEST_ID`, `SHUTTING_DOWN`, `MESSAGE_TOO_LARGE`, +`RESOURCE_LIMIT_EXCEEDED`, `INVALID_SOURCE_RANGE`, +`APPLICATION_REQUEST_REJECTED`, and `INTERNAL_ERROR`. Errors never share a +`result` payload, and internal exceptions or local paths are not returned. + +## Verified source and identity changes + +Index records, semantic summaries, rankings, and consumer/model output are +hints. Current repository source is authoritative. Every source excerpt is +re-authorized against the snapshot and re-read through the stable reader. +ContextForge verifies regular-file identity, portable exact-case path, size, +SHA-256, strict UTF-8 decoding, line ranges, and effective byte/file budgets. +Reads and packages are all-or-nothing; partial successful source is never +returned. + +`expected_snapshot_digest` is an optimistic concurrency and identity guard. If +the repository differs from the snapshot captured by `snapshot`, the operation +fails with JSON-RPC code `-32001` and typed code +`SOURCE_IDENTITY_CHANGED`. Discard prepared candidates and excerpts, call +`snapshot` again, and restart discovery. Do not retry the stale request by +silently replacing its digest. + +## Cancellation and shutdown + +Independent requests may run concurrently and responses can arrive out of +order. Correlate them by JSON-RPC `id`. Response writes are serialized. +Cancellation is cooperative and uses the target request ID: + +```json +{"jsonrpc":"2.0","method":"$/cancelRequest","params":{"id":7}} +``` + +The target completes normally if it won the race, or fails with typed +`REQUEST_CANCELLED`. Cancellation does not return partial read/package output +and grants no mutation capability. A cancellation notification has no response; +include its own JSON-RPC `id` only when a `{"cancelled": true|false}` response is +needed. + +Send `shutdown` after outstanding work is resolved, wait for its response, then +close stdin and wait for the child process. Clean stdin EOF also stops the +bridge. Abrupt process termination is safe with respect to repository and index +state because bridge operations are read-only. + +Shutdown and clean EOF use the same bounded drain. The bridge first stops +accepting work, signals every active request's cooperative cancellation event, +and waits at most 5 seconds. Any request task still pending then receives direct +asyncio cancellation and gets at most another 0.1 seconds for cleanup. After +that 5.1-second maximum drain budget, the bridge detaches any remaining task and +does not wait for it again. These internal v1 limits are fixed rather than CLI +configurable. A timed-out request cannot return a partial success, and the +shutdown response remains a normal serialized JSON-RPC frame. + +## Security and read-only boundary + +Run the bridge only as a child process of a trusted local consumer. It inherits +the user's filesystem read authority and intentionally returns repository data. +It provides no authentication, authorization, encryption, tenant isolation, or +protection from another hostile process running as the same user. Do not expose +stdio through a network or untrusted broker without an external security layer. + +Bridge v1 cannot write repository source, mutate Git, invoke a shell or arbitrary +subprocess, call a model/provider, access external data, mutate the index, or +publish artifacts to disk. `package` returns the canonical package in memory. +The workspace is fixed at process start and requests cannot replace it. + +See the runnable [generic bridge client](../../examples/generic_bridge_client.py) +and [troubleshooting guide](troubleshooting.md). diff --git a/docs/guides/cli.md b/docs/guides/cli.md index 2b6aaca..9bd4d1d 100644 --- a/docs/guides/cli.md +++ b/docs/guides/cli.md @@ -37,6 +37,20 @@ result while progress and logs use stderr. See [Discovery output and benchmarks](discovery.md) for the canonical-result, renderer, benchmark, warning, counter, and repeatability contracts. +## Local bridge + +```bash +contextforge bridge --stdio --workspace /path/to/repository +``` + +`--stdio` is required in protocol v1. The bridge reads one UTF-8 JSON-RPC 2.0 +request per stdin line and writes one response per stdout line. Stdout is +protocol-only; bounded diagnostics use stderr. The client must negotiate +protocol `1.0` with `hello` before repository requests. The workspace is fixed +for the process lifetime and all bridge capabilities are model-free and +read-only. See the [bridge v1 guide](bridge.md) for the method, cancellation, +snapshot, security, and shutdown contracts. + ## Read-only diagnostics ```bash diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 4451d5b..64f7405 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -32,3 +32,52 @@ leaving the lower ContextForge override in place. The budget record decomposes system, user, source, selected index, schema, requested output, protocol overhead, and safety margin so the total can be reproduced numerically without exposing the prompt. + +## Bridge exits, hangs, or returns no parseable response + +Confirm the installed command and protocol help first: + +```bash +contextforge --version +contextforge bridge --help +``` + +Start with `--stdio` and send exactly one UTF-8 JSON object plus LF per request. +The first application request must be `hello` with +`{"protocol_version":"1.0"}`. `PROTOCOL_NEGOTIATION_REQUIRED` means a +repository request arrived before a successful hello. +`INCOMPATIBLE_PROTOCOL_VERSION` means the client must stop and use one of the +reported `supported_protocol_versions`; do not guess compatibility from the +Python package version. + +Read responses only from stdout and correlate by JSON-RPC `id`; concurrent +responses may be out of order. Read stderr separately for bounded diagnostics. +Do not combine stderr into stdout, add terminal prompts to stdin, pretty-print a +request across lines, or buffer a request without its final LF. The maximum v1 +frame size is reported by `hello` and oversized frames fail with +`MESSAGE_TOO_LARGE`. + +## Bridge reports SOURCE_IDENTITY_CHANGED + +The repository no longer matches the digest returned by `snapshot`, or a +selected candidate's optional `path`/`source_sha256` assertion changed. Discard +the preparation and any excerpts derived from it, call `snapshot` again, then +repeat discovery and selection. Do not substitute the new digest into an old +request: candidates and ranges belong to the old repository truth. + +For reproducible runs, pause formatters, generators, checkout operations, and +other processes that rewrite the workspace. The bridge detects identity drift +but is not a sandbox against a hostile same-account writer. + +## Cancellation did not stop immediately + +`$/cancelRequest` is cooperative and targets the JSON-RPC request `id`. The +operation may finish before cancellation is observed. A cancelled operation +returns `REQUEST_CANCELLED` and never a partial successful read or package. +After shutdown begins, new work fails with `SHUTTING_DOWN`. + +## Bridge cannot find a preparation + +`UNKNOWN_PREPARATION` means the ID belongs to another bridge process/snapshot or +was evicted from the bounded in-memory cache. Call `snapshot` and `discover` +again in the same process. Preparation state is intentionally not persisted. diff --git a/docs/schemas/contextforge-bridge-v1.schema.json b/docs/schemas/contextforge-bridge-v1.schema.json new file mode 100644 index 0000000..2d09966 --- /dev/null +++ b/docs/schemas/contextforge-bridge-v1.schema.json @@ -0,0 +1,441 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://contextforge.dev/schemas/contextforge-bridge-v1.schema.json", + "title": "ContextForge trusted-local bridge protocol v1", + "description": "Closed JSON-RPC 2.0 frames transported as one UTF-8 JSON object plus LF per stdio message.", + "oneOf": [ + { "$ref": "#/$defs/request" }, + { "$ref": "#/$defs/success" }, + { "$ref": "#/$defs/failure" } + ], + "$defs": { + "id": { + "oneOf": [ + { "type": "string", "minLength": 1, "maxLength": 200 }, + { "type": "integer" } + ] + }, + "sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "portablePath": { + "type": "string", + "minLength": 1, + "maxLength": 32768, + "pattern": "^(?!/)(?![A-Za-z]:)(?!.*(?:^|/)\\.\\.(?:/|$))(?!.*\\\\).+$" + }, + "timeout": { "type": "integer", "minimum": 1, "maximum": 900000 }, + "budget": { + "type": "object", + "additionalProperties": false, + "properties": { + "max_steps": { "type": "integer", "minimum": 1, "maximum": 100 }, + "max_model_calls": { "type": "integer", "minimum": 1, "maximum": 100 }, + "max_files_read": { "type": "integer", "minimum": 1, "maximum": 1000 }, + "max_source_bytes": { "type": "integer", "minimum": 1, "maximum": 16777216 }, + "max_tool_result_bytes": { "type": "integer", "minimum": 1, "maximum": 16777216 }, + "max_context_bytes": { "type": "integer", "minimum": 1, "maximum": 10485760 }, + "max_context_files": { "type": "integer", "minimum": 1, "maximum": 1000 }, + "max_preselected_candidates": { "type": "integer", "minimum": 0, "maximum": 100 }, + "timeout_seconds": { "type": "number", "exclusiveMinimum": 0, "maximum": 900 }, + "repeated_action_warning": { "type": "integer", "minimum": 2, "maximum": 4 }, + "repeated_action_limit": { "type": "integer", "minimum": 3, "maximum": 10 } + } + }, + "budgetUsage": { + "type": "object", + "additionalProperties": false, + "required": [ + "steps", "model_calls", "model_generations", "repair_generations", + "provider_discovery_calls", "provider_capability_calls", + "transport_attempts", "total_provider_http_calls", + "provider_http_calls", "files_read", "source_bytes", + "tool_result_bytes", "context_bytes", "context_files" + ], + "properties": { + "steps": { "type": "integer", "minimum": 0 }, + "model_calls": { "type": "integer", "minimum": 0 }, + "model_generations": { "type": "integer", "minimum": 0 }, + "repair_generations": { "type": "integer", "minimum": 0 }, + "provider_discovery_calls": { "type": "integer", "minimum": 0 }, + "provider_capability_calls": { "type": "integer", "minimum": 0 }, + "transport_attempts": { "type": "integer", "minimum": 0 }, + "total_provider_http_calls": { "type": "integer", "minimum": 0 }, + "provider_http_calls": { "type": "integer", "minimum": 0 }, + "files_read": { "type": "integer", "minimum": 0 }, + "source_bytes": { "type": "integer", "minimum": 0 }, + "tool_result_bytes": { "type": "integer", "minimum": 0 }, + "context_bytes": { "type": "integer", "minimum": 0 }, + "context_files": { "type": "integer", "minimum": 0 } + } + }, + "lineRange": { + "type": "object", + "additionalProperties": false, + "required": ["start_line", "end_line"], + "properties": { + "start_line": { "type": "integer", "minimum": 1 }, + "end_line": { "type": "integer", "minimum": 1 } + } + }, + "selectionItem": { + "type": "object", + "additionalProperties": false, + "required": ["candidate_id"], + "properties": { + "candidate_id": { "type": "string", "minLength": 1, "maxLength": 200 }, + "path": { "$ref": "#/$defs/portablePath" }, + "source_sha256": { "$ref": "#/$defs/sha256" }, + "ranges": { "type": "array", "items": { "$ref": "#/$defs/lineRange" } } + } + }, + "request": { + "oneOf": [ + { "$ref": "#/$defs/helloRequest" }, + { "$ref": "#/$defs/statusRequest" }, + { "$ref": "#/$defs/snapshotRequest" }, + { "$ref": "#/$defs/discoverRequest" }, + { "$ref": "#/$defs/expandRequest" }, + { "$ref": "#/$defs/readRequest" }, + { "$ref": "#/$defs/packageRequest" }, + { "$ref": "#/$defs/cancelRequest" }, + { "$ref": "#/$defs/shutdownRequest" } + ] + }, + "helloRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "hello" }, + "params": { + "type": "object", "additionalProperties": false, + "required": ["protocol_version"], + "properties": { + "protocol_version": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+$", "maxLength": 32 }, + "client_name": { "type": "string", "minLength": 1, "maxLength": 200 }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "statusRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "status" }, + "params": { + "type": "object", "additionalProperties": false, + "properties": { + "expected_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "snapshotRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "snapshot" }, + "params": { + "type": "object", "additionalProperties": false, + "properties": { "timeout_ms": { "$ref": "#/$defs/timeout" } } + } + } + }, + "discoverRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "discover" }, + "params": { + "type": "object", "additionalProperties": false, + "required": ["expected_snapshot_digest", "task"], + "properties": { + "expected_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "task": { "type": "string", "minLength": 1, "maxLength": 20000 }, + "mode": { "enum": ["fresh", "indexed", "hybrid"] }, + "pinned_paths": { "type": "array", "items": { "$ref": "#/$defs/portablePath" }, "uniqueItems": true }, + "excluded_paths": { "type": "array", "items": { "$ref": "#/$defs/portablePath" }, "uniqueItems": true }, + "strict": { "type": "boolean" }, + "budget": { "$ref": "#/$defs/budget" }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "expandRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "expand" }, + "params": { + "type": "object", "additionalProperties": false, + "required": ["expected_snapshot_digest", "preparation_id", "operation"], + "properties": { + "expected_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "preparation_id": { "$ref": "#/$defs/sha256" }, + "operation": { "enum": ["symbol", "text", "callers", "importers", "related_tests"] }, + "arguments": { "type": "object" }, + "budget_usage": { "$ref": "#/$defs/budgetUsage" }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "readRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "read" }, + "params": { + "type": "object", "additionalProperties": false, + "required": ["expected_snapshot_digest", "preparation_id", "items"], + "properties": { + "expected_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "preparation_id": { "$ref": "#/$defs/sha256" }, + "items": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/selectionItem" } }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "packageRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "package" }, + "params": { + "type": "object", "additionalProperties": false, + "required": ["expected_snapshot_digest", "preparation_id", "items"], + "properties": { + "expected_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "preparation_id": { "$ref": "#/$defs/sha256" }, + "items": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/selectionItem" } }, + "include_tree": { "type": "boolean" }, + "timeout_ms": { "$ref": "#/$defs/timeout" } + } + } + } + }, + "cancelRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "$/cancelRequest" }, + "params": { + "type": "object", "additionalProperties": false, "required": ["id"], + "properties": { + "id": { "$ref": "#/$defs/id" } + } + } + } + }, + "shutdownRequest": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "method", "params"], + "properties": { + "jsonrpc": { "const": "2.0" }, "id": { "$ref": "#/$defs/id" }, + "method": { "const": "shutdown" }, + "params": { + "type": "object", "additionalProperties": false, + "properties": { "timeout_ms": { "$ref": "#/$defs/timeout" } } + } + } + }, + "helloResult": { + "type": "object", "additionalProperties": false, + "required": ["protocol_version", "contextforge_version", "supported_protocol_versions", "capabilities", "workspace", "policy"], + "properties": { + "protocol_version": { "const": "1.0" }, + "contextforge_version": { "type": "string" }, + "supported_protocol_versions": { "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true }, + "capabilities": { "type": "object" }, + "workspace": { + "type": "object", "additionalProperties": false, + "required": ["identity"], + "properties": { "identity": { "$ref": "#/$defs/sha256" } } + }, + "policy": { "type": "object" } + } + }, + "statusResult": { + "type": "object", "additionalProperties": false, + "required": ["ready", "snapshot_digest", "current_snapshot_digest", "source_identity_changed", "index"], + "properties": { + "ready": { "type": "boolean" }, + "snapshot_digest": { "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] }, + "current_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "source_identity_changed": { "type": "boolean" }, + "index": { "type": "object" } + } + }, + "snapshotResult": { + "type": "object", "additionalProperties": false, + "required": ["snapshot_digest", "file_count", "source_bytes", "languages"], + "properties": { + "snapshot_digest": { "$ref": "#/$defs/sha256" }, + "file_count": { "type": "integer", "minimum": 0 }, + "source_bytes": { "type": "integer", "minimum": 0 }, + "languages": { "type": "object", "additionalProperties": { "type": "integer", "minimum": 0 } } + } + }, + "candidate": { + "type": "object", "additionalProperties": false, + "required": ["candidate_id", "path", "language", "rank", "score", "ranking_signals", "source_sha256", "source_size_bytes", "evidence_origin", "structural_evidence", "semantic_evidence"], + "properties": { + "candidate_id": { "type": "string", "minLength": 1, "maxLength": 200 }, + "path": { "$ref": "#/$defs/portablePath" }, + "language": { "type": "string" }, + "rank": { "type": "integer", "minimum": 1 }, + "score": { "type": "number", "minimum": 0 }, + "ranking_signals": { "type": "array", "minItems": 1, "maxItems": 10, "items": { "type": "string" } }, + "source_sha256": { "$ref": "#/$defs/sha256" }, + "source_size_bytes": { "type": "integer", "minimum": 0 }, + "evidence_origin": { "enum": ["snapshot", "fresh", "indexed", "hybrid"] }, + "structural_evidence": { "type": "boolean" }, + "semantic_evidence": { "type": "boolean" } + } + }, + "discoverResult": { + "type": "object", "additionalProperties": false, + "required": ["preparation_id", "source_snapshot_digest", "mode", "index_generation_id", "index_status", "candidates", "total_candidate_count", "stale_index_paths", "warnings", "budget_usage", "config_digest", "model_provider_used"], + "properties": { + "preparation_id": { "$ref": "#/$defs/sha256" }, + "source_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "mode": { "enum": ["fresh", "indexed", "hybrid"] }, + "index_generation_id": { "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] }, + "index_status": { "enum": ["not_used", "unavailable", "current", "stale"] }, + "candidates": { "type": "array", "items": { "$ref": "#/$defs/candidate" } }, + "total_candidate_count": { "type": "integer", "minimum": 0 }, + "stale_index_paths": { "type": "array", "items": { "$ref": "#/$defs/portablePath" }, "uniqueItems": true }, + "warnings": { "type": "array", "items": { "type": "object" } }, + "budget_usage": { "$ref": "#/$defs/budgetUsage" }, + "config_digest": { "$ref": "#/$defs/sha256" }, + "model_provider_used": { "const": false } + } + }, + "expandResult": { + "type": "object", "additionalProperties": false, + "required": ["preparation_id", "operation", "ok", "code", "data", "truncated", "made_progress", "budget_usage"], + "properties": { + "preparation_id": { "$ref": "#/$defs/sha256" }, + "operation": { "enum": ["symbol", "text", "callers", "importers", "related_tests"] }, + "ok": { "type": "boolean" }, + "code": { "type": "string" }, + "data": { "type": "object" }, + "truncated": { "type": "boolean" }, + "made_progress": { "type": "boolean" }, + "budget_usage": { "$ref": "#/$defs/budgetUsage" } + } + }, + "verifiedBlock": { + "type": "object", "additionalProperties": false, + "required": ["start_line", "end_line", "text", "line_count", "size_bytes", "sha256"], + "properties": { + "start_line": { "oneOf": [{ "type": "integer", "minimum": 1 }, { "type": "null" }] }, + "end_line": { "oneOf": [{ "type": "integer", "minimum": 1 }, { "type": "null" }] }, + "text": { "type": "string" }, + "line_count": { "type": "integer", "minimum": 0 }, + "size_bytes": { "type": "integer", "minimum": 0 }, + "sha256": { "$ref": "#/$defs/sha256" } + } + }, + "verifiedFile": { + "type": "object", "additionalProperties": false, + "required": ["candidate_id", "path", "language", "source_size_bytes", "source_sha256", "source_line_count", "blocks", "included_line_count", "included_content_bytes"], + "properties": { + "candidate_id": { "type": "string" }, + "path": { "$ref": "#/$defs/portablePath" }, + "language": { "oneOf": [{ "type": "string" }, { "type": "null" }] }, + "source_size_bytes": { "type": "integer", "minimum": 0 }, + "source_sha256": { "$ref": "#/$defs/sha256" }, + "source_line_count": { "type": "integer", "minimum": 0 }, + "blocks": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/verifiedBlock" } }, + "included_line_count": { "type": "integer", "minimum": 0 }, + "included_content_bytes": { "type": "integer", "minimum": 0 } + } + }, + "readResult": { + "type": "object", "additionalProperties": false, + "required": ["preparation_id", "selection_id", "source_snapshot_digest", "files", "budget_usage"], + "properties": { + "preparation_id": { "$ref": "#/$defs/sha256" }, + "selection_id": { "$ref": "#/$defs/sha256" }, + "source_snapshot_digest": { "$ref": "#/$defs/sha256" }, + "files": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/verifiedFile" } }, + "budget_usage": { "$ref": "#/$defs/budgetUsage" } + } + }, + "packageResult": { + "type": "object", "additionalProperties": false, + "required": ["selection_id", "package"], + "properties": { + "selection_id": { "$ref": "#/$defs/sha256" }, + "package": { + "type": "object", + "description": "Canonical ContextPackage schema 1, which is independently closed and verified.", + "required": ["schema_version", "title", "project", "tree", "files", "statistics"], + "properties": { "schema_version": { "const": 1 } } + } + } + }, + "cancelResult": { + "type": "object", "additionalProperties": false, "required": ["cancelled"], + "properties": { "cancelled": { "type": "boolean" } } + }, + "shutdownResult": { + "type": "object", "additionalProperties": false, "required": ["shutdown"], + "properties": { "shutdown": { "const": true } } + }, + "success": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "result"], + "properties": { + "jsonrpc": { "const": "2.0" }, + "id": { "$ref": "#/$defs/id" }, + "result": { + "oneOf": [ + { "$ref": "#/$defs/helloResult" }, + { "$ref": "#/$defs/statusResult" }, + { "$ref": "#/$defs/snapshotResult" }, + { "$ref": "#/$defs/discoverResult" }, + { "$ref": "#/$defs/expandResult" }, + { "$ref": "#/$defs/readResult" }, + { "$ref": "#/$defs/packageResult" }, + { "$ref": "#/$defs/cancelResult" }, + { "$ref": "#/$defs/shutdownResult" } + ] + } + } + }, + "failure": { + "type": "object", "additionalProperties": false, + "required": ["jsonrpc", "id", "error"], + "properties": { + "jsonrpc": { "const": "2.0" }, + "id": { "oneOf": [{ "$ref": "#/$defs/id" }, { "type": "null" }] }, + "error": { + "type": "object", "additionalProperties": false, + "required": ["code", "message", "data"], + "properties": { + "code": { "type": "integer" }, + "message": { "type": "string", "minLength": 1, "maxLength": 2000 }, + "data": { + "type": "object", "required": ["code"], + "properties": { + "code": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]{0,127}$" } + } + } + } + } + } + } + } +} diff --git a/examples/generic_bridge_client.py b/examples/generic_bridge_client.py new file mode 100644 index 0000000..6a65f8a --- /dev/null +++ b/examples/generic_bridge_client.py @@ -0,0 +1,128 @@ +"""Minimal synchronous client for the trusted-local ContextForge bridge v1.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any, TextIO, cast + +PROTOCOL_VERSION = "1.0" + + +class BridgeError(RuntimeError): + """Safe JSON-RPC error returned by the bridge.""" + + +class BridgeClient: + def __init__(self, workspace: Path) -> None: + self._process = subprocess.Popen( + [ + sys.executable, + "-m", + "contextforge", + "bridge", + "--stdio", + "--workspace", + str(workspace), + ], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=None, + text=True, + encoding="utf-8", + ) + self._stdin = cast(TextIO, self._process.stdin) + self._stdout = cast(TextIO, self._process.stdout) + self._next_id = 1 + + def request(self, method: str, params: dict[str, Any]) -> dict[str, Any]: + request_id = self._next_id + self._next_id += 1 + frame = { + "jsonrpc": "2.0", + "id": request_id, + "method": method, + "params": params, + } + self._stdin.write(json.dumps(frame, separators=(",", ":")) + "\n") + self._stdin.flush() + line = self._stdout.readline() + if not line: + raise BridgeError("bridge closed stdout before returning a response") + response = json.loads(line) + if response.get("jsonrpc") != "2.0" or response.get("id") != request_id: + raise BridgeError("unexpected JSON-RPC response correlation") + if "error" in response: + error = response["error"] + typed_code = error.get("data", {}).get("code", "UNKNOWN") + raise BridgeError(f"{typed_code}: {error.get('message', 'bridge error')}") + return cast(dict[str, Any], response["result"]) + + def close(self) -> None: + if self._process.poll() is None: + self.request("shutdown", {}) + self._stdin.close() + returncode = self._process.wait(timeout=10) + if returncode != 0: + raise BridgeError(f"bridge exited with status {returncode}") + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Read one verified candidate through ContextForge bridge v1." + ) + parser.add_argument("workspace", type=Path) + parser.add_argument("task") + arguments = parser.parse_args() + + client = BridgeClient(arguments.workspace.resolve(strict=True)) + try: + hello = client.request( + "hello", + {"protocol_version": PROTOCOL_VERSION, "client_name": "generic-example"}, + ) + if hello.get("protocol_version") != PROTOCOL_VERSION: + raise BridgeError("bridge selected an incompatible protocol version") + + snapshot = client.request("snapshot", {}) + digest = snapshot["snapshot_digest"] + discovery = client.request( + "discover", + { + "expected_snapshot_digest": digest, + "task": arguments.task, + "mode": "hybrid", + }, + ) + candidates = discovery["candidates"] + if not candidates: + raise BridgeError("discovery returned no selectable candidates") + + # A real consumer may ask its own model or user to choose. ContextForge + # remains model-free and verifies whichever issued candidate is selected. + selected = candidates[0] + verified = client.request( + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": discovery["preparation_id"], + "items": [ + { + "candidate_id": selected["candidate_id"], + "path": selected["path"], + "source_sha256": selected["source_sha256"], + } + ], + }, + ) + print(json.dumps(verified, ensure_ascii=False, indent=2)) + finally: + client.close() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/smoke_distribution.py b/scripts/smoke_distribution.py index 056bf74..69f3e8a 100644 --- a/scripts/smoke_distribution.py +++ b/scripts/smoke_distribution.py @@ -3,15 +3,25 @@ from __future__ import annotations from importlib.metadata import distribution +from importlib.util import find_spec import contextforge from contextforge._metadata import __version__ +from contextforge.bridge import ( + BRIDGE_PROTOCOL_VERSION, + SUPPORTED_BRIDGE_PROTOCOL_VERSIONS, + BridgeServer, +) installed = distribution("contextforge-repo") assert installed.metadata["Name"] == "contextforge-repo" assert installed.metadata["Version"] == __version__ assert installed.metadata["License-Expression"] == "Apache-2.0" assert contextforge.__version__ == __version__ +assert BRIDGE_PROTOCOL_VERSION == "1.0" +assert SUPPORTED_BRIDGE_PROTOCOL_VERSIONS == ("1.0",) +assert BridgeServer.__module__ == "contextforge.bridge.server" +assert find_spec("contextforge.protocol") is None scripts = { entry.name: entry.value diff --git a/scripts/validate_distribution.py b/scripts/validate_distribution.py index b17407b..8874af2 100644 --- a/scripts/validate_distribution.py +++ b/scripts/validate_distribution.py @@ -14,18 +14,34 @@ VERSION_PATTERN = re.compile(r'^__version__ = "(?P[^"]+)"$', re.MULTILINE) FORBIDDEN_PARTS = { ".contextforge", + ".agents", + ".coverage", ".env", ".git", ".github", ".mypy_cache", ".pytest_cache", ".ruff_cache", + ".uv-cache", + ".venv", + ".review-fixture", "__pycache__", + "build", "docs", "htmlcov", "tests", + "dist", "wiki", } +FORBIDDEN_CONTENT_PATTERNS = { + "workspace absolute path": re.compile( + rb"C:[\\/]+Programming[\\/]+Projects[\\/]+ContextForge", re.IGNORECASE + ), + "developer home path": re.compile(rb"C:[\\/]+Users[\\/]+water", re.IGNORECASE), + "private key": re.compile(rb"BEGIN (?:RSA |OPENSSH |EC )?PRIVATE KEY"), + "AWS access key": re.compile(rb"AKIA[0-9A-Z]{16}"), + "provider credential": re.compile(rb"sk-[A-Za-z0-9_-]{20,}"), +} def _source_version(root: Path) -> str: @@ -65,6 +81,14 @@ def _has_forbidden_part(path: PurePosixPath) -> bool: ) +def _validate_content(archive_name: str, member_name: str, payload: bytes) -> None: + for description, pattern in FORBIDDEN_CONTENT_PATTERNS.items(): + if pattern.search(payload): + raise ValueError( + f"{archive_name}: {description} found in packaged {member_name}" + ) + + def _validate_wheel(path: Path, version: str) -> None: with zipfile.ZipFile(path) as archive: names = [PurePosixPath(name) for name in archive.namelist()] @@ -80,6 +104,18 @@ def _validate_wheel(path: Path, version: str) -> None: raise ValueError(f"{path.name}: unexpected wheel path {name}") if _has_forbidden_part(name): raise ValueError(f"{path.name}: forbidden wheel path {name}") + if name.parts and not str(name).endswith("/"): + _validate_content(path.name, str(name), archive.read(str(name))) + required_bridge = { + PurePosixPath("contextforge/bridge/__init__.py"), + PurePosixPath("contextforge/bridge/models.py"), + PurePosixPath("contextforge/bridge/protocol.py"), + PurePosixPath("contextforge/bridge/server.py"), + } + if not required_bridge.issubset(names): + raise ValueError(f"{path.name}: bridge package is incomplete") + if PurePosixPath("contextforge/protocol.py") in names: + raise ValueError(f"{path.name}: obsolete bridge envelope was packaged") def _validate_sdist(path: Path, version: str) -> None: @@ -124,6 +160,24 @@ def _validate_sdist(path: Path, version: str) -> None: raise ValueError(f"{path.name}: unexpected sdist path {relative}") if _has_forbidden_part(relative): raise ValueError(f"{path.name}: forbidden sdist path {relative}") + if member.isfile(): + member_file = archive.extractfile(member) + if member_file is None: + raise ValueError(f"{path.name}: could not read {relative}") + _validate_content(path.name, str(relative), member_file.read()) + relative_names = { + PurePosixPath(*PurePosixPath(member.name).parts[1:]) for member in members + } + required_bridge = { + PurePosixPath("src/contextforge/bridge/__init__.py"), + PurePosixPath("src/contextforge/bridge/models.py"), + PurePosixPath("src/contextforge/bridge/protocol.py"), + PurePosixPath("src/contextforge/bridge/server.py"), + } + if not required_bridge.issubset(relative_names): + raise ValueError(f"{path.name}: bridge package is incomplete") + if PurePosixPath("src/contextforge/protocol.py") in relative_names: + raise ValueError(f"{path.name}: obsolete bridge envelope was packaged") def main() -> None: diff --git a/src/contextforge/_metadata.py b/src/contextforge/_metadata.py index ccef8fd..6c51f2e 100644 --- a/src/contextforge/_metadata.py +++ b/src/contextforge/_metadata.py @@ -3,4 +3,4 @@ APP_NAME = "ContextForge" APP_SLUG = "contextforge" APP_TAGLINE = "Build bounded, reviewable repository context for coding agents." -__version__ = "0.4.2.post1" +__version__ = "0.5.0" diff --git a/src/contextforge/application.py b/src/contextforge/application.py index 8e1ce08..99559a0 100644 --- a/src/contextforge/application.py +++ b/src/contextforge/application.py @@ -23,6 +23,12 @@ discover_repository, render_context_suggestion, ) +from contextforge.discovery.application import ( + expand_discovery, + package_verified_context, + prepare_discovery_candidates, + read_verified_context, +) from contextforge.filesystem import FileTooLargeError, StableReadError, read_file_stably from contextforge.git import GitDiffRequest from contextforge.handoff import ( @@ -1400,8 +1406,12 @@ def _reject_json_constant(value: str) -> None: "canonical_json", "clean_repository_index", "create_automatic_handoff", + "expand_discovery", "inspect_repository_index", "load_task_handoff", + "package_verified_context", + "prepare_discovery_candidates", + "read_verified_context", "render_context_suggestion", "render_handoff_review", "suggest_repository_context", diff --git a/src/contextforge/bridge/__init__.py b/src/contextforge/bridge/__init__.py new file mode 100644 index 0000000..ed25065 --- /dev/null +++ b/src/contextforge/bridge/__init__.py @@ -0,0 +1,20 @@ +"""Persistent generic ContextForge bridge protocol v1.""" + +from .protocol import BRIDGE_PROTOCOL_VERSION, SUPPORTED_BRIDGE_PROTOCOL_VERSIONS +from .server import ( + JSONRPC_VERSION, + MAX_JSONRPC_MESSAGE_BYTES, + BridgeServer, + run_stdio_bridge, + serve_stdio_bridge, +) + +__all__ = [ + "BRIDGE_PROTOCOL_VERSION", + "SUPPORTED_BRIDGE_PROTOCOL_VERSIONS", + "JSONRPC_VERSION", + "MAX_JSONRPC_MESSAGE_BYTES", + "BridgeServer", + "run_stdio_bridge", + "serve_stdio_bridge", +] diff --git a/src/contextforge/bridge/models.py b/src/contextforge/bridge/models.py new file mode 100644 index 0000000..1ffea60 --- /dev/null +++ b/src/contextforge/bridge/models.py @@ -0,0 +1,160 @@ +"""Closed parameter models for the persistent JSON-RPC bridge.""" + +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +from contextforge.core.validation import Sha256, validate_portable_relative_path +from contextforge.discovery import ( + DiscoveryBudget, + DiscoveryBudgetUsage, + DiscoveryLineRange, + DiscoveryMode, +) + + +class BridgeParams(BaseModel): + """Strict base for all method parameters.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + timeout_ms: int | None = Field(default=None, ge=1, le=900_000, strict=True) + + +class HelloParams(BridgeParams): + protocol_version: str = Field( + min_length=1, max_length=32, pattern=r"^[0-9]+\.[0-9]+$" + ) + client_name: str | None = Field(default=None, min_length=1, max_length=200) + + +class StatusParams(BridgeParams): + expected_snapshot_digest: Sha256 | None = None + + +class SnapshotParams(BridgeParams): + pass + + +class DiscoverParams(BridgeParams): + expected_snapshot_digest: Sha256 + task: str = Field(min_length=1, max_length=20_000) + mode: DiscoveryMode = DiscoveryMode.HYBRID + pinned_paths: tuple[str, ...] = () + excluded_paths: tuple[str, ...] = () + strict: bool = False + budget: DiscoveryBudget = Field(default_factory=DiscoveryBudget) + + @field_validator("mode", mode="before") + @classmethod + def validate_mode(cls, value: object) -> object: + if isinstance(value, str): + return DiscoveryMode(value) + return value + + @field_validator("pinned_paths", "excluded_paths") + @classmethod + def validate_paths(cls, value: tuple[str, ...]) -> tuple[str, ...]: + paths = tuple(validate_portable_relative_path(path) for path in value) + if paths != tuple(sorted(set(paths))): + raise ValueError("paths must be unique and in canonical order") + return paths + + +ExpansionOperation = Literal[ + "symbol", + "text", + "callers", + "importers", + "related_tests", +] + + +class ExpandParams(BridgeParams): + expected_snapshot_digest: Sha256 + preparation_id: Sha256 + operation: ExpansionOperation + arguments: dict[str, Any] = Field(default_factory=dict) + budget_usage: DiscoveryBudgetUsage = Field(default_factory=DiscoveryBudgetUsage) + + +class BridgeSelectionItem(BaseModel): + """One candidate selection with optional identity assertions.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + candidate_id: str = Field(min_length=1, max_length=200) + path: str | None = None + source_sha256: Sha256 | None = None + ranges: tuple[DiscoveryLineRange, ...] = () + + @field_validator("path") + @classmethod + def validate_path(cls, value: str | None) -> str | None: + return None if value is None else validate_portable_relative_path(value) + + @model_validator(mode="after") + def validate_ranges(self) -> BridgeSelectionItem: + previous_end = 0 + for item in self.ranges: + if item.start_line <= previous_end: + raise ValueError("ranges must be sorted and disjoint") + previous_end = item.end_line + return self + + +class ReadParams(BridgeParams): + expected_snapshot_digest: Sha256 + preparation_id: Sha256 + items: tuple[BridgeSelectionItem, ...] = Field(min_length=1) + + @field_validator("items") + @classmethod + def validate_items( + cls, value: tuple[BridgeSelectionItem, ...] + ) -> tuple[BridgeSelectionItem, ...]: + identifiers = tuple(item.candidate_id for item in value) + if identifiers != tuple(sorted(set(identifiers))): + raise ValueError("selection items must be unique and in canonical order") + return value + + +class PackageParams(ReadParams): + include_tree: bool = True + + +class CancelParams(BaseModel): + model_config = ConfigDict(extra="forbid", strict=True) + + id: str | int + + @field_validator("id") + @classmethod + def validate_id(cls, value: str | int) -> str | int: + if isinstance(value, bool) or ( + isinstance(value, str) and (not value or len(value) > 200) + ): + raise ValueError("id must be a bounded non-empty string or integer") + return value + + +class ShutdownParams(BridgeParams): + pass + + +__all__ = [ + "BridgeParams", + "BridgeSelectionItem", + "CancelParams", + "DiscoverParams", + "ExpandParams", + "ExpansionOperation", + "HelloParams", + "PackageParams", + "ReadParams", + "ShutdownParams", + "SnapshotParams", + "StatusParams", +] diff --git a/src/contextforge/bridge/protocol.py b/src/contextforge/bridge/protocol.py new file mode 100644 index 0000000..eb35e23 --- /dev/null +++ b/src/contextforge/bridge/protocol.py @@ -0,0 +1,8 @@ +"""Stable version metadata for the ContextForge bridge protocol.""" + +from typing import Final, Literal + +BRIDGE_PROTOCOL_VERSION: Final[Literal["1.0"]] = "1.0" +SUPPORTED_BRIDGE_PROTOCOL_VERSIONS: Final[tuple[Literal["1.0"], ...]] = ("1.0",) + +__all__ = ["BRIDGE_PROTOCOL_VERSION", "SUPPORTED_BRIDGE_PROTOCOL_VERSIONS"] diff --git a/src/contextforge/bridge/server.py b/src/contextforge/bridge/server.py new file mode 100644 index 0000000..c2f8288 --- /dev/null +++ b/src/contextforge/bridge/server.py @@ -0,0 +1,1080 @@ +"""Persistent bounded JSON-RPC 2.0 service over NDJSON streams.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +import os +import unicodedata +from collections import OrderedDict +from dataclasses import dataclass +from pathlib import Path +from typing import Any, BinaryIO, TextIO + +from pydantic import BaseModel, ValidationError + +from contextforge._metadata import __version__ +from contextforge.application import inspect_repository_index +from contextforge.context import ( + ContextLimitError, + ContextReaderError, + FileChangedError, + InvalidLineRangeError, +) +from contextforge.core.validation import validate_portable_relative_path +from contextforge.discovery import ( + DiscoveryExpansionOperation, + DiscoveryExpansionRequest, + DiscoveryRequest, + DiscoverySelection, + DiscoverySelectionItem, +) +from contextforge.discovery.application import ( + DiscoveryApplicationError, + DiscoveryPreparationMismatchError, + expand_discovery, + package_verified_context, + prepare_discovery_candidates, + read_verified_context, +) +from contextforge.discovery.models import DiscoveryCandidatePreparation +from contextforge.discovery.tools import TOOL_INPUT_MODELS +from contextforge.intelligence import ( + calculate_source_snapshot_digest, + canonical_json_bytes, +) +from contextforge.repositories import ProjectSnapshot, ScanOptions, scan_repository + +from .models import ( + BridgeSelectionItem, + CancelParams, + DiscoverParams, + ExpandParams, + ExpansionOperation, + HelloParams, + PackageParams, + ReadParams, + ShutdownParams, + SnapshotParams, + StatusParams, +) +from .protocol import BRIDGE_PROTOCOL_VERSION, SUPPORTED_BRIDGE_PROTOCOL_VERSIONS + +JSONRPC_VERSION = "2.0" +MAX_JSONRPC_MESSAGE_BYTES = 1024 * 1024 +MAX_DIAGNOSTIC_BYTES = 64 * 1024 +MAX_PREPARATIONS = 128 +DEFAULT_SHUTDOWN_TIMEOUT_SECONDS = 5.0 +FORCED_CANCELLATION_TIMEOUT_SECONDS = 0.1 + +PARSE_ERROR = -32700 +INVALID_REQUEST = -32600 +METHOD_NOT_FOUND = -32601 +INVALID_PARAMS = -32602 +INTERNAL_ERROR = -32603 +REQUEST_CANCELLED = -32800 +SOURCE_IDENTITY_CHANGED = -32001 +MESSAGE_TOO_LARGE = -32002 +DUPLICATE_REQUEST_ID = -32003 +REQUEST_TIMEOUT = -32004 +SHUTTING_DOWN = -32005 +INCOMPATIBLE_PROTOCOL_VERSION = -32006 +PROTOCOL_NEGOTIATION_REQUIRED = -32007 + +_METHOD_MODELS: dict[str, type[BaseModel]] = { + "hello": HelloParams, + "status": StatusParams, + "snapshot": SnapshotParams, + "discover": DiscoverParams, + "expand": ExpandParams, + "read": ReadParams, + "package": PackageParams, + "$/cancelRequest": CancelParams, + "shutdown": ShutdownParams, +} + +_EXPANSION_TOOLS: dict[ExpansionOperation, DiscoveryExpansionOperation] = { + "symbol": "search_symbols", + "text": "search_text", + "callers": "find_callers", + "importers": "find_importers", + "related_tests": "find_related_tests", +} + + +class BridgeFault(RuntimeError): + """One safe typed JSON-RPC failure.""" + + def __init__( + self, + rpc_code: int, + typed_code: str, + message: str, + *, + data: dict[str, Any] | None = None, + ) -> None: + super().__init__(message) + self.rpc_code = rpc_code + self.typed_code = typed_code + self.message = message + self.data = data or {} + + +@dataclass(slots=True) +class _ActiveRequest: + cancellation: asyncio.Event + task: asyncio.Task[None] + + +class _SerializedWriter: + def __init__(self, stream: BinaryIO) -> None: + self._stream = stream + self._lock = asyncio.Lock() + + async def write(self, frame: dict[str, Any]) -> None: + payload = ( + json.dumps( + frame, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + + b"\n" + ) + async with self._lock: + await asyncio.to_thread(self._write, payload) + + def _write(self, payload: bytes) -> None: + self._stream.write(payload) + self._stream.flush() + + +class _BoundedDiagnostics: + def __init__(self, stream: TextIO | None) -> None: + self._stream = stream + self._remaining = MAX_DIAGNOSTIC_BYTES + + def write(self, message: str) -> None: + if self._stream is None or self._remaining <= 0: + return + safe = message.replace("\r", " ").replace("\n", " ")[:2000] + "\n" + encoded = safe.encode("utf-8")[: self._remaining] + self._remaining -= len(encoded) + self._stream.write(encoded.decode("utf-8", errors="ignore")) + self._stream.flush() + + +class BridgeServer: + """Workspace-bound persistent ContextForge bridge protocol v1 server.""" + + def __init__( + self, + workspace: str | Path, + *, + shutdown_timeout_seconds: float = DEFAULT_SHUTDOWN_TIMEOUT_SECONDS, + ) -> None: + root = Path(workspace).expanduser().resolve(strict=True) + if not root.is_dir(): + raise NotADirectoryError(str(root)) + if not math.isfinite(shutdown_timeout_seconds) or shutdown_timeout_seconds <= 0: + raise ValueError("shutdown_timeout_seconds must be finite and positive") + self.workspace = root + self.shutdown_timeout_seconds = shutdown_timeout_seconds + self.workspace_identity = _workspace_identity(root) + self.config_digest = hashlib.sha256( + canonical_json_bytes( + {"scan_options": ScanOptions().model_dump(mode="json")} + ) + ).hexdigest() + self._snapshot_digest: str | None = None + self._preparations: OrderedDict[str, DiscoveryCandidatePreparation] = ( + OrderedDict() + ) + self._active: dict[tuple[str, str | int], _ActiveRequest] = {} + self._writer: _SerializedWriter | None = None + self._diagnostics = _BoundedDiagnostics(None) + self._shutting_down = False + self._protocol_negotiated = False + + async def serve( + self, + input_stream: BinaryIO, + output_stream: BinaryIO, + error_stream: TextIO | None = None, + ) -> None: + """Serve bounded NDJSON until shutdown or a clean input EOF.""" + + self._writer = _SerializedWriter(output_stream) + self._diagnostics = _BoundedDiagnostics(error_stream) + try: + while not self._shutting_down: + line, oversized = await _read_bounded_line(input_stream) + if line is None: + break + if oversized: + await self._write_error( + None, + BridgeFault( + MESSAGE_TOO_LARGE, + "MESSAGE_TOO_LARGE", + "JSON-RPC message exceeds the configured byte limit.", + data={"max_message_bytes": MAX_JSONRPC_MESSAGE_BYTES}, + ), + ) + continue + frame = self._decode_frame(line) + if isinstance(frame, BridgeFault): + await self._write_error(None, frame) + continue + if frame.get("method") == "shutdown": + await self._accept_request(frame, inline=True) + break + await self._accept_request(frame, inline=False) + finally: + self._begin_shutdown() + await self._drain_active_requests() + + def _begin_shutdown(self) -> None: + """Stop new work and request cooperative cancellation of active work.""" + + self._shutting_down = True + current = asyncio.current_task() + for request in tuple(self._active.values()): + if request.task is not current: + request.cancellation.set() + + async def _drain_active_requests(self) -> None: + """Drain active work within the fixed graceful-shutdown budget.""" + + current = asyncio.current_task() + tasks = { + request.task + for request in tuple(self._active.values()) + if request.task is not current and not request.task.done() + } + if not tasks: + return + done, pending = await asyncio.wait( + tasks, + timeout=self.shutdown_timeout_seconds, + ) + for task in done: + self._consume_task_result(task) + if not pending: + self._forget_active_tasks(tasks) + return + self._diagnostics.write( + f"bridge shutdown deadline expired with {len(pending)} active request(s)" + ) + for task in pending: + task.cancel() + done, pending = await asyncio.wait( + pending, + timeout=FORCED_CANCELLATION_TIMEOUT_SECONDS, + ) + for task in done: + self._consume_task_result(task) + for task in pending: + task.cancel() + task.add_done_callback(self._consume_task_result) + self._forget_active_tasks(tasks) + + def _consume_task_result(self, task: asyncio.Task[None]) -> None: + """Consume a shutdown outcome and retain a safe unexpected-failure signal.""" + + if task.cancelled(): + return + if task.exception() is not None: + self._diagnostics.write( + "bridge active request ended unexpectedly during shutdown" + ) + + def _forget_active_tasks(self, tasks: set[asyncio.Task[None]]) -> None: + """Detach drained or abandoned requests from the bridge lifecycle.""" + + for key, request in tuple(self._active.items()): + if request.task in tasks: + self._active.pop(key, None) + + def _decode_frame(self, line: bytes) -> dict[str, Any] | BridgeFault: + try: + value = json.loads( + line.decode("utf-8"), + parse_constant=lambda value: (_ for _ in ()).throw(ValueError(value)), + ) + except (UnicodeDecodeError, json.JSONDecodeError, ValueError): + return BridgeFault(PARSE_ERROR, "MALFORMED_JSON", "Malformed JSON input.") + if not isinstance(value, dict): + return BridgeFault( + INVALID_REQUEST, + "INVALID_REQUEST", + "JSON-RPC request must be an object.", + ) + return value + + async def _accept_request(self, frame: dict[str, Any], *, inline: bool) -> None: + try: + method, request_id, params = _validate_envelope(frame) + except BridgeFault as exc: + await self._write_error(_safe_frame_id(frame), exc) + return + + if self._shutting_down: + await self._write_error( + request_id, + BridgeFault( + SHUTTING_DOWN, + "SHUTTING_DOWN", + "Bridge shutdown has started.", + ), + ) + return + + if method == "$/cancelRequest": + await self._cancel_request(params, request_id) + return + + assert request_id is not None + key = _id_key(request_id) + if key in self._active: + await self._write_error( + request_id, + BridgeFault( + DUPLICATE_REQUEST_ID, + "DUPLICATE_REQUEST_ID", + "A request with this ID is already active.", + ), + ) + return + + cancellation = asyncio.Event() + task = asyncio.create_task( + self._process_request(method, request_id, params, cancellation) + ) + self._active[key] = _ActiveRequest(cancellation=cancellation, task=task) + if inline: + await task + + async def _cancel_request( + self, raw_params: dict[str, Any], request_id: str | int | None + ) -> None: + try: + params = CancelParams.model_validate_json( + json.dumps(raw_params, ensure_ascii=False, allow_nan=False) + ) + except ValidationError as exc: + if request_id is not None: + await self._write_validation_error(request_id, exc) + return + active = self._active.get(_id_key(params.id)) + if active is not None: + active.cancellation.set() + if request_id is not None: + await self._write_result(request_id, {"cancelled": active is not None}) + + async def _process_request( + self, + method: str, + request_id: str | int, + raw_params: dict[str, Any], + cancellation: asyncio.Event, + ) -> None: + negotiated_protocol = False + try: + model = _METHOD_MODELS[method] + try: + params = model.model_validate_json( + json.dumps(raw_params, ensure_ascii=False, allow_nan=False) + ) + except ValidationError as exc: + await self._write_validation_error(request_id, exc) + return + if method == "hello": + hello = _require_type(params, HelloParams) + if hello.protocol_version not in SUPPORTED_BRIDGE_PROTOCOL_VERSIONS: + raise BridgeFault( + INCOMPATIBLE_PROTOCOL_VERSION, + "INCOMPATIBLE_PROTOCOL_VERSION", + "The requested bridge protocol version is not supported.", + data={ + "requested_protocol_version": hello.protocol_version, + "supported_protocol_versions": list( + SUPPORTED_BRIDGE_PROTOCOL_VERSIONS + ), + }, + ) + negotiated_protocol = True + elif method != "shutdown" and not self._protocol_negotiated: + raise BridgeFault( + PROTOCOL_NEGOTIATION_REQUIRED, + "PROTOCOL_NEGOTIATION_REQUIRED", + "Call hello with a supported protocol_version first.", + data={ + "supported_protocol_versions": list( + SUPPORTED_BRIDGE_PROTOCOL_VERSIONS + ) + }, + ) + timeout_ms = getattr(params, "timeout_ms", None) + operation = self._dispatch(method, params, cancellation) + if timeout_ms is None: + result = await operation + else: + try: + result = await asyncio.wait_for( + operation, timeout=timeout_ms / 1000 + ) + except TimeoutError: + cancellation.set() + raise BridgeFault( + REQUEST_TIMEOUT, + "REQUEST_TIMEOUT", + "The request exceeded its caller-supplied timeout.", + ) from None + if cancellation.is_set(): + raise asyncio.CancelledError + if negotiated_protocol: + self._protocol_negotiated = True + await self._write_result(request_id, result) + except asyncio.CancelledError: + await self._write_error( + request_id, + BridgeFault( + REQUEST_CANCELLED, + "REQUEST_CANCELLED", + "The request was cancelled.", + ), + ) + except BridgeFault as exc: + await self._write_error(request_id, exc) + except (DiscoveryPreparationMismatchError, FileChangedError): + await self._write_error( + request_id, + BridgeFault( + SOURCE_IDENTITY_CHANGED, + "SOURCE_IDENTITY_CHANGED", + "Repository source identity changed during the operation.", + ), + ) + except ContextLimitError: + await self._write_error( + request_id, + BridgeFault( + INVALID_PARAMS, + "RESOURCE_LIMIT_EXCEEDED", + "The requested source excerpt exceeds an effective limit.", + ), + ) + except (ContextReaderError, InvalidLineRangeError): + await self._write_error( + request_id, + BridgeFault( + INVALID_PARAMS, + "INVALID_SOURCE_RANGE", + "The requested source path or line range is invalid.", + ), + ) + except DiscoveryApplicationError: + await self._write_error( + request_id, + BridgeFault( + INVALID_PARAMS, + "APPLICATION_REQUEST_REJECTED", + "The application operation rejected the request.", + ), + ) + except Exception: + self._diagnostics.write("bridge request failed with an internal error") + await self._write_error( + request_id, + BridgeFault( + INTERNAL_ERROR, + "INTERNAL_ERROR", + "ContextForge could not complete the request.", + ), + ) + finally: + self._active.pop(_id_key(request_id), None) + + async def _dispatch( + self, method: str, raw: BaseModel, cancellation: asyncio.Event + ) -> dict[str, Any]: + if cancellation.is_set(): + raise asyncio.CancelledError + if method == "hello": + return self._hello() + if method == "status": + return await self._status(_require_type(raw, StatusParams), cancellation) + if method == "snapshot": + return await self._snapshot(cancellation) + if method == "discover": + return await self._discover( + _require_type(raw, DiscoverParams), cancellation + ) + if method == "expand": + return await self._expand(_require_type(raw, ExpandParams), cancellation) + if method == "read": + return await self._read(_require_type(raw, ReadParams), cancellation) + if method == "package": + return await self._package(_require_type(raw, PackageParams), cancellation) + if method == "shutdown": + self._begin_shutdown() + return {"shutdown": True} + raise BridgeFault( + METHOD_NOT_FOUND, + "METHOD_NOT_FOUND", + "The requested method is not supported.", + ) + + def _hello(self) -> dict[str, Any]: + return { + "protocol_version": BRIDGE_PROTOCOL_VERSION, + "supported_protocol_versions": list(SUPPORTED_BRIDGE_PROTOCOL_VERSIONS), + "contextforge_version": __version__, + "capabilities": { + "methods": [ + "hello", + "status", + "snapshot", + "discover", + "expand", + "read", + "package", + "$/cancelRequest", + "shutdown", + ], + "expansion_operations": sorted(_EXPANSION_TOOLS), + "model_free_discovery": True, + "cooperative_cancellation": True, + "concurrent_requests": True, + "serialized_responses": True, + "max_message_bytes": MAX_JSONRPC_MESSAGE_BYTES, + }, + "workspace": { + "identity": self.workspace_identity, + }, + "policy": { + "repository_access": "read_only_verified_snapshot", + "external_data": "disabled", + "portable_paths_only": True, + "source_writes": False, + "index_mutation": False, + "shell": False, + "subprocess_execution": False, + }, + } + + async def _status( + self, params: StatusParams, cancellation: asyncio.Event + ) -> dict[str, Any]: + snapshot = await asyncio.to_thread(scan_repository, self.workspace) + if cancellation.is_set(): + raise asyncio.CancelledError + digest = calculate_source_snapshot_digest(snapshot) + report = await asyncio.to_thread( + inspect_repository_index, + self.workspace, + provider_configuration=None, + ) + if cancellation.is_set(): + raise asyncio.CancelledError + return { + "ready": not self._shutting_down, + "snapshot_digest": self._snapshot_digest, + "current_snapshot_digest": digest, + "source_identity_changed": ( + params.expected_snapshot_digest is not None + and params.expected_snapshot_digest != digest + ), + "index": { + "initialized": report.initialized, + "index_schema": report.index_schema, + "active_generation_id": report.active_generation_id, + "indexed_files": report.indexed_files, + "stale_files": list(report.stale_files), + "failed_files": list(report.failed_files), + "deleted_records": list(report.deleted_records), + "added_files": list(report.added_files), + "changed_files": list(report.changed_files), + "provider_id": report.provider_id, + "model_id": report.model_id, + "prompt_versions": list(report.prompt_versions), + "global_maps": { + "overview": report.overview_status, + "architecture": report.architecture_status, + "features": report.feature_status, + }, + "lock_status": report.lock_status, + }, + } + + async def _snapshot(self, cancellation: asyncio.Event) -> dict[str, Any]: + snapshot = await asyncio.to_thread(scan_repository, self.workspace) + if cancellation.is_set(): + raise asyncio.CancelledError + digest = calculate_source_snapshot_digest(snapshot) + if digest != self._snapshot_digest: + self._preparations.clear() + self._snapshot_digest = digest + return { + "snapshot_digest": digest, + "file_count": len(snapshot.files), + "source_bytes": sum(item.size_bytes for item in snapshot.files), + "languages": dict(sorted(snapshot.summary.languages.items())), + } + + async def _discover( + self, params: DiscoverParams, cancellation: asyncio.Event + ) -> dict[str, Any]: + snapshot = await self._verified_snapshot( + params.expected_snapshot_digest, cancellation + ) + request = DiscoveryRequest( + task=params.task, + mode=params.mode, + pinned_paths=params.pinned_paths, + excluded_paths=params.excluded_paths, + strict=params.strict, + budget=params.budget, + ) + preparation = await asyncio.to_thread( + prepare_discovery_candidates, + snapshot, + request, + cancellation=cancellation, + ) + self._remember_preparation(preparation) + return { + "preparation_id": preparation.preparation_id, + "source_snapshot_digest": preparation.source_snapshot_digest, + "mode": preparation.mode.value, + "index_generation_id": preparation.index_generation_id, + "index_status": preparation.index_status, + "candidates": [ + item.model_dump(mode="json") for item in preparation.candidates + ], + "total_candidate_count": preparation.total_candidate_count, + "stale_index_paths": list(preparation.stale_index_paths), + "warnings": [item.model_dump(mode="json") for item in preparation.warnings], + "budget_usage": preparation.budget_usage.model_dump(mode="json"), + "config_digest": self.config_digest, + "model_provider_used": False, + } + + async def _expand( + self, params: ExpandParams, cancellation: asyncio.Event + ) -> dict[str, Any]: + snapshot = await self._verified_snapshot( + params.expected_snapshot_digest, cancellation + ) + preparation = self._require_preparation(params.preparation_id) + tool_name = _EXPANSION_TOOLS[params.operation] + self._validate_expansion_arguments(tool_name, params.arguments) + expansion = DiscoveryExpansionRequest( + preparation_id=preparation.preparation_id, + operation=tool_name, + arguments=params.arguments, + budget_usage=params.budget_usage, + ) + result = await asyncio.to_thread( + expand_discovery, + snapshot, + preparation, + expansion, + cancellation=cancellation, + ) + return { + "preparation_id": result.preparation_id, + "operation": params.operation, + "ok": result.ok, + "code": result.code, + "data": result.data, + "truncated": result.truncated, + "made_progress": result.made_progress, + "budget_usage": result.budget_usage.model_dump(mode="json"), + } + + async def _read( + self, params: ReadParams, cancellation: asyncio.Event + ) -> dict[str, Any]: + snapshot = await self._verified_snapshot( + params.expected_snapshot_digest, cancellation + ) + preparation = self._require_preparation(params.preparation_id) + selection = self._selection(preparation, params.items) + verified = await asyncio.to_thread( + read_verified_context, + snapshot, + preparation, + selection, + cancellation=cancellation, + ) + return { + "preparation_id": verified.preparation_id, + "selection_id": self._selection_id(preparation, params.items), + "source_snapshot_digest": verified.source_snapshot_digest, + "files": [item.model_dump(mode="json") for item in verified.files], + "budget_usage": verified.budget_usage.model_dump(mode="json"), + } + + async def _package( + self, params: PackageParams, cancellation: asyncio.Event + ) -> dict[str, Any]: + snapshot = await self._verified_snapshot( + params.expected_snapshot_digest, cancellation + ) + preparation = self._require_preparation(params.preparation_id) + selection = self._selection(preparation, params.items) + verified = await asyncio.to_thread( + read_verified_context, + snapshot, + preparation, + selection, + cancellation=cancellation, + ) + package = await asyncio.to_thread( + package_verified_context, + snapshot, + verified, + include_tree=params.include_tree, + cancellation=cancellation, + ) + return { + "selection_id": self._selection_id(preparation, params.items), + "package": package.model_dump(mode="json"), + } + + async def _verified_snapshot( + self, expected: str, cancellation: asyncio.Event + ) -> ProjectSnapshot: + if self._snapshot_digest is None: + raise BridgeFault( + INVALID_PARAMS, + "SNAPSHOT_REQUIRED", + "Call snapshot before repository-sensitive methods.", + ) + snapshot = await asyncio.to_thread(scan_repository, self.workspace) + if cancellation.is_set(): + raise asyncio.CancelledError + current = calculate_source_snapshot_digest(snapshot) + if expected != self._snapshot_digest or current != expected: + raise BridgeFault( + SOURCE_IDENTITY_CHANGED, + "SOURCE_IDENTITY_CHANGED", + "Repository source identity differs from the expected snapshot.", + data={ + "expected_snapshot_digest": expected, + "current_snapshot_digest": current, + }, + ) + return snapshot + + def _require_preparation( + self, preparation_id: str + ) -> DiscoveryCandidatePreparation: + preparation = self._preparations.get(preparation_id) + if preparation is None: + raise BridgeFault( + INVALID_PARAMS, + "UNKNOWN_PREPARATION", + "The preparation is not available in this bridge process.", + ) + self._preparations.move_to_end(preparation_id) + return preparation + + def _remember_preparation(self, preparation: DiscoveryCandidatePreparation) -> None: + self._preparations[preparation.preparation_id] = preparation + self._preparations.move_to_end(preparation.preparation_id) + while len(self._preparations) > MAX_PREPARATIONS: + self._preparations.popitem(last=False) + + def _selection( + self, + preparation: DiscoveryCandidatePreparation, + items: tuple[BridgeSelectionItem, ...], + ) -> DiscoverySelection: + candidates = {item.candidate_id: item for item in preparation.candidates} + selected: list[DiscoverySelectionItem] = [] + for item in items: + candidate = candidates.get(item.candidate_id) + if candidate is None: + raise BridgeFault( + INVALID_PARAMS, + "UNKNOWN_CANDIDATE", + "A selected candidate was not returned by discover.", + ) + if item.path is not None and item.path != candidate.path: + raise BridgeFault( + SOURCE_IDENTITY_CHANGED, + "SOURCE_IDENTITY_CHANGED", + "Selected path does not match the prepared candidate.", + ) + if ( + item.source_sha256 is not None + and item.source_sha256 != candidate.source_sha256 + ): + raise BridgeFault( + SOURCE_IDENTITY_CHANGED, + "SOURCE_IDENTITY_CHANGED", + "Selected source identity does not match the prepared candidate.", + ) + selected.append( + DiscoverySelectionItem( + candidate_id=item.candidate_id, + ranges=item.ranges, + ) + ) + return DiscoverySelection( + preparation_id=preparation.preparation_id, + items=tuple(selected), + ) + + def _selection_id( + self, + preparation: DiscoveryCandidatePreparation, + items: tuple[BridgeSelectionItem, ...], + ) -> str: + candidates = {item.candidate_id: item for item in preparation.candidates} + normalized_ranges = [ + { + "path": candidates[item.candidate_id].path, + "ranges": [ + [line_range.start_line, line_range.end_line] + for line_range in item.ranges + ], + } + for item in items + ] + normalized_task = unicodedata.normalize( + "NFC", + preparation.task.replace("\r\n", "\n").replace("\r", "\n").strip(), + ) + payload = { + "workspace": self.workspace_identity, + "snapshot_digest": preparation.source_snapshot_digest, + "config_digest": self.config_digest, + "task_digest": hashlib.sha256(normalized_task.encode("utf-8")).hexdigest(), + "discovery_mode": preparation.mode.value, + "effective_budget": preparation.budget.model_dump(mode="json"), + "selected_ranges": normalized_ranges, + } + return hashlib.sha256(canonical_json_bytes(payload)).hexdigest() + + def _validate_expansion_arguments( + self, tool_name: str, arguments: dict[str, Any] + ) -> None: + model = TOOL_INPUT_MODELS[tool_name] + try: + model.model_validate(arguments) + for key in ("path", "path_prefix"): + value = arguments.get(key) + if value is not None: + validate_portable_relative_path(value) + except (ValidationError, ValueError) as exc: + raise BridgeFault( + INVALID_PARAMS, + "INVALID_PARAMS", + "Expansion arguments are invalid.", + data={"validation": _validation_details(exc)}, + ) from None + + async def _write_validation_error( + self, request_id: str | int, exc: ValidationError + ) -> None: + await self._write_error( + request_id, + BridgeFault( + INVALID_PARAMS, + "INVALID_PARAMS", + "Method parameters are invalid.", + data={"validation": _validation_details(exc)}, + ), + ) + + async def _write_result( + self, request_id: str | int, result: dict[str, Any] + ) -> None: + await self._require_writer().write( + {"jsonrpc": JSONRPC_VERSION, "id": request_id, "result": result} + ) + + async def _write_error( + self, request_id: str | int | None, fault: BridgeFault + ) -> None: + data = {"code": fault.typed_code, **fault.data} + await self._require_writer().write( + { + "jsonrpc": JSONRPC_VERSION, + "id": request_id, + "error": { + "code": fault.rpc_code, + "message": fault.message, + "data": data, + }, + } + ) + + def _require_writer(self) -> _SerializedWriter: + if self._writer is None: + raise RuntimeError("bridge writer is not initialized") + return self._writer + + +async def serve_stdio_bridge( + workspace: str | Path, + input_stream: BinaryIO, + output_stream: BinaryIO, + error_stream: TextIO | None = None, +) -> None: + """Create and serve one workspace-bound stdio bridge.""" + + await BridgeServer(workspace).serve(input_stream, output_stream, error_stream) + + +def run_stdio_bridge( + workspace: str | Path, + input_stream: BinaryIO, + output_stream: BinaryIO, + error_stream: TextIO | None = None, +) -> None: + """Run stdio without re-awaiting requests abandoned by bounded shutdown.""" + + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + try: + loop.run_until_complete( + serve_stdio_bridge(workspace, input_stream, output_stream, error_stream) + ) + finally: + pending = asyncio.all_tasks(loop) + for task in pending: + task.cancel() + # The bridge has already applied both bounded cancellation phases. + # Loop closure is intentional for these lifecycle-abandoned tasks. + task_with_lifecycle_flag: Any = task + task_with_lifecycle_flag._log_destroy_pending = False + loop.close() + asyncio.set_event_loop(None) + + +async def _read_bounded_line(stream: BinaryIO) -> tuple[bytes | None, bool]: + chunk = await asyncio.to_thread(stream.readline, MAX_JSONRPC_MESSAGE_BYTES + 1) + if not chunk: + return None, False + oversized = len(chunk) > MAX_JSONRPC_MESSAGE_BYTES + if oversized and not chunk.endswith(b"\n"): + while True: + remainder = await asyncio.to_thread( + stream.readline, MAX_JSONRPC_MESSAGE_BYTES + 1 + ) + if not remainder or remainder.endswith(b"\n"): + break + return chunk, oversized + + +def _validate_envelope( + frame: dict[str, Any], +) -> tuple[str, str | int | None, dict[str, Any]]: + allowed = {"jsonrpc", "method", "params", "id"} + if set(frame) - allowed or frame.get("jsonrpc") != JSONRPC_VERSION: + raise BridgeFault( + INVALID_REQUEST, + "INVALID_REQUEST", + "Invalid JSON-RPC request envelope.", + ) + method = frame.get("method") + if not isinstance(method, str) or not method: + raise BridgeFault( + INVALID_REQUEST, + "INVALID_REQUEST", + "JSON-RPC method must be a non-empty string.", + ) + if method not in _METHOD_MODELS: + raise BridgeFault( + METHOD_NOT_FOUND, + "METHOD_NOT_FOUND", + "The requested method is not supported.", + ) + request_id = frame.get("id") + if request_id is not None and not _valid_rpc_id(request_id): + raise BridgeFault( + INVALID_REQUEST, + "INVALID_REQUEST", + "JSON-RPC id must be a string or integer.", + ) + if method != "$/cancelRequest" and request_id is None: + raise BridgeFault( + INVALID_REQUEST, + "INVALID_REQUEST", + "Requests require a non-null correlation id.", + ) + if "params" not in frame: + raise BridgeFault( + INVALID_PARAMS, + "INVALID_PARAMS", + "Method parameters must be present as an object.", + ) + params = frame["params"] + if not isinstance(params, dict): + raise BridgeFault( + INVALID_PARAMS, + "INVALID_PARAMS", + "Method parameters must be an object.", + ) + return method, request_id, params + + +def _safe_frame_id(frame: dict[str, Any]) -> str | int | None: + value = frame.get("id") + if not _valid_rpc_id(value): + return None + return value + + +def _valid_rpc_id(value: object) -> bool: + if isinstance(value, bool): + return False + if isinstance(value, str): + return 0 < len(value) <= 200 + return isinstance(value, int) + + +def _id_key(value: str | int) -> tuple[str, str | int]: + return ("string", value) if isinstance(value, str) else ("integer", value) + + +def _workspace_identity(root: Path) -> str: + value = unicodedata.normalize("NFC", os.path.normcase(str(root))).replace("\\", "/") + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def _require_type[T: BaseModel](value: BaseModel, expected: type[T]) -> T: + if not isinstance(value, expected): + raise TypeError("unexpected bridge parameter model") + return value + + +def _validation_details(exc: ValidationError | ValueError) -> list[dict[str, Any]]: + if isinstance(exc, ValidationError): + return [ + { + "location": [str(item) for item in error["loc"]], + "message": error["msg"], + "type": error["type"], + } + for error in exc.errors(include_url=False, include_input=False) + ][:20] + return [{"location": [], "message": str(exc), "type": "value_error"}] + + +__all__ = [ + "BRIDGE_PROTOCOL_VERSION", + "JSONRPC_VERSION", + "MAX_JSONRPC_MESSAGE_BYTES", + "BridgeServer", + "run_stdio_bridge", + "serve_stdio_bridge", +] diff --git a/src/contextforge/cli/main.py b/src/contextforge/cli/main.py index 9655b1f..f7840c3 100644 --- a/src/contextforge/cli/main.py +++ b/src/contextforge/cli/main.py @@ -3,11 +3,12 @@ import sys from enum import StrEnum from pathlib import Path -from typing import Annotated, Never +from typing import Annotated, BinaryIO, Never, cast import typer from contextforge._metadata import APP_NAME, __version__ +from contextforge.bridge import run_stdio_bridge from contextforge.cli.benchmark_commands import benchmark_app from contextforge.cli.context_commands import context_app from contextforge.cli.diagnostics_commands import diagnostics_app @@ -161,7 +162,8 @@ def cli( log_format=None if log_format is None else log_format.value, log_file=log_file, component_filter=selected_components, - no_log_file=no_log_file or ctx.invoked_subcommand == "benchmark", + no_log_file=no_log_file + or ctx.invoked_subcommand in {"benchmark", "bridge"}, no_color=no_color, verbosity=verbose, ) @@ -268,6 +270,50 @@ def doctor() -> None: typer.echo(f"Log level: {settings.log_level}") +@app.command() +def bridge( + stdio: Annotated[ + bool, + typer.Option( + "--stdio", + help=( + "Required in v1. Read UTF-8 NDJSON requests from stdin and write " + "only JSON-RPC 2.0 responses to stdout." + ), + ), + ] = False, + workspace: Annotated[ + Path, + typer.Option( + "--workspace", + help=( + "Repository root to bind for the lifetime of this verified " + "read-only local integration session." + ), + exists=True, + file_okay=False, + dir_okay=True, + resolve_path=True, + ), + ] = Path("."), +) -> None: + """Run trusted-local, model-free ContextForge bridge protocol v1. + + The client must negotiate protocol 1.0 with hello before repository calls. + Stdout is protocol-only; bounded diagnostics use stderr. The bridge never + writes source or index state and never selects or invokes a model. + """ + + if not stdio: + _exit_with_error("bridge v1 requires --stdio", code=2) + input_stream = cast(BinaryIO, getattr(sys.stdin, "buffer", sys.stdin)) + output_stream = cast(BinaryIO, getattr(sys.stdout, "buffer", sys.stdout)) + try: + run_stdio_bridge(workspace, input_stream, output_stream, sys.stderr) + except (FileNotFoundError, NotADirectoryError, OSError, ValueError) as exc: + _exit_with_error(str(exc), code=1) + + @app.command() def scan( path: Annotated[ diff --git a/src/contextforge/discovery/__init__.py b/src/contextforge/discovery/__init__.py index ced73f4..14e98d1 100644 --- a/src/contextforge/discovery/__init__.py +++ b/src/contextforge/discovery/__init__.py @@ -1,7 +1,18 @@ -"""Public model-guided repository discovery API.""" +"""Public model-free and model-assisted repository discovery API.""" +from contextforge.discovery.application import ( + DISCOVERY_EXPANSION_OPERATIONS, + DiscoveryApplicationError, + DiscoveryPreparationMismatchError, + DiscoverySelectionError, + expand_discovery, + package_verified_context, + prepare_discovery_candidates, + read_verified_context, +) from contextforge.discovery.completeness import review_completeness from contextforge.discovery.models import ( + DISCOVERY_APPLICATION_SCHEMA_VERSION, DISCOVERY_SCHEMA_VERSION, CompletenessWarning, DiscoveryAction, @@ -9,16 +20,26 @@ DiscoveryBudget, DiscoveryBudgetUsage, DiscoveryCandidate, + DiscoveryCandidatePreparation, DiscoveryCandidateRecord, + DiscoveryExpansionOperation, + DiscoveryExpansionRequest, + DiscoveryExpansionResult, DiscoveryLineRange, DiscoveryMode, DiscoveryObservation, DiscoveryRequest, DiscoveryRunRecord, + DiscoverySelection, + DiscoverySelectionItem, DiscoveryState, FinalContextSelection, IndexedContextSelection, + PreparedDiscoveryCandidate, SelectionReason, + VerifiedContext, + VerifiedContextBlock, + VerifiedContextFile, ) from contextforge.discovery.renderers import ( DiscoveryRenderError, @@ -48,6 +69,8 @@ ) __all__ = [ + "DISCOVERY_APPLICATION_SCHEMA_VERSION", + "DISCOVERY_EXPANSION_OPERATIONS", "DISCOVERY_SCHEMA_VERSION", "DISCOVERY_SYSTEM_INSTRUCTIONS", "DISCOVERY_TOOL_SCHEMAS", @@ -55,22 +78,31 @@ "CompletenessWarning", "DiscoveryAction", "DiscoveryActionBatch", + "DiscoveryApplicationError", "DiscoveryBudget", "DiscoveryBudgetUsage", "DiscoveryCancelledError", "DiscoveryCandidate", + "DiscoveryCandidatePreparation", "DiscoveryCandidateRecord", "DiscoveryError", + "DiscoveryExpansionRequest", + "DiscoveryExpansionOperation", + "DiscoveryExpansionResult", "DiscoveryKnowledge", "DiscoveryLimitError", "DiscoveryLineRange", "DiscoveryMode", "DiscoveryObservation", "DiscoveryProtocolError", + "DiscoveryPreparationMismatchError", "DiscoveryRenderError", "DiscoveryRequest", "DiscoveryResultFormat", "DiscoveryRunRecord", + "DiscoverySelection", + "DiscoverySelectionError", + "DiscoverySelectionItem", "DiscoverySession", "DiscoverySourceChangedError", "DiscoveryState", @@ -78,12 +110,20 @@ "DiscoveryUnavailableError", "FinalContextSelection", "IndexedContextSelection", + "PreparedDiscoveryCandidate", "GitDiffProvider", "GitDiffResult", "SelectionReason", "ToolBudgetExceededError", "ToolBudgetTracker", + "VerifiedContext", + "VerifiedContextBlock", + "VerifiedContextFile", "discover_repository", + "expand_discovery", + "package_verified_context", + "prepare_discovery_candidates", + "read_verified_context", "review_completeness", "render_context_suggestion", ] diff --git a/src/contextforge/discovery/application.py b/src/contextforge/discovery/application.py new file mode 100644 index 0000000..1df4006 --- /dev/null +++ b/src/contextforge/discovery/application.py @@ -0,0 +1,501 @@ +"""Public model-free repository discovery application workflows.""" + +from __future__ import annotations + +import asyncio +import hashlib +from pathlib import Path +from typing import Any, Literal + +from pydantic import BaseModel + +from contextforge.context import ( + ContextBuildOptions, + ContextPackage, + ContextSelection, + LineRange, + LineRangeRequest, + ReaderLimits, + build_context_package, + read_selected_text_files, +) +from contextforge.discovery.models import ( + CompletenessWarning, + DiscoveryBudgetUsage, + DiscoveryCandidatePreparation, + DiscoveryExpansionOperation, + DiscoveryExpansionRequest, + DiscoveryExpansionResult, + DiscoveryMode, + DiscoveryRequest, + DiscoverySelection, + PreparedDiscoveryCandidate, + VerifiedContext, + VerifiedContextBlock, + VerifiedContextFile, +) +from contextforge.discovery.session import DiscoverySession +from contextforge.discovery.tools import DiscoveryToolExecutor, ToolBudgetTracker +from contextforge.intelligence import ( + calculate_source_snapshot_digest, + canonical_json_bytes, +) +from contextforge.repositories import ProjectSnapshot, scan_repository + +DiscoverySource = ProjectSnapshot | str | Path + +DISCOVERY_EXPANSION_OPERATIONS: tuple[DiscoveryExpansionOperation, ...] = ( + "get_repository_overview", + "list_tree", + "search_index", + "search_symbols", + "search_text", + "get_file_summary", + "get_symbol_summary", + "find_imports", + "find_importers", + "find_references", + "find_callers", + "find_related_tests", + "read_file", + "read_lines", + "get_context_budget", +) + + +class DiscoveryApplicationError(RuntimeError): + """Base failure at the immutable model-free application boundary.""" + + +class DiscoveryPreparationMismatchError(DiscoveryApplicationError): + """Raised when a DTO is not pinned to the supplied repository snapshot.""" + + +class DiscoverySelectionError(DiscoveryApplicationError): + """Raised when a caller selection is outside a prepared candidate set.""" + + +def prepare_discovery_candidates( + source: DiscoverySource, + request: DiscoveryRequest, + *, + cancellation: asyncio.Event | None = None, +) -> DiscoveryCandidatePreparation: + """Return deterministic mode-aware candidates without invoking a model.""" + + _raise_if_cancelled(cancellation) + snapshot = _snapshot(source) + session, _ = _prepare_runtime(snapshot, request) + knowledge = session._require_knowledge() + files = {item.path: item for item in snapshot.files} + manifest_paths = ( + set() + if knowledge.manifest is None + else {item.path for item in knowledge.manifest.files} + ) + candidates = tuple( + PreparedDiscoveryCandidate( + candidate_id=record.candidate_id, + path=record.path, + language=record.language, + rank=record.rank, + score=record.score, + ranking_signals=record.ranking_signals, + source_sha256=files[record.path].sha256, + source_size_bytes=files[record.path].size_bytes, + evidence_origin=_evidence_origin( + request.mode, + record.path, + manifest_paths=manifest_paths, + structural_paths=set(knowledge.code_maps), + ), + structural_evidence=record.path in knowledge.code_maps, + semantic_evidence=record.path in knowledge.semantic_analyses, + ) + for record in session._preselected_candidates + ) + warnings = list(session.warnings) + if knowledge.stale_index_paths: + warnings.append( + CompletenessWarning( + code="stale-index-coverage", + message=( + "Stale index paths were excluded from indexed evidence; " + "current snapshot identities remain authoritative." + ), + related_paths=knowledge.stale_index_paths[:50], + confidence=1.0, + ) + ) + values: dict[str, Any] = { + "task": request.task, + "mode": request.mode, + "pinned_paths": request.pinned_paths, + "excluded_paths": request.excluded_paths, + "strict": request.strict, + "source_snapshot_digest": session.snapshot_digest, + "index_generation_id": ( + knowledge.manifest.generation_id if knowledge.manifest is not None else None + ), + "index_status": _index_status(request.mode, knowledge), + "candidates": candidates, + "total_candidate_count": len(session._ranked_candidates), + "stale_index_paths": knowledge.stale_index_paths, + "warnings": _canonical_warnings(warnings), + "budget": request.budget, + "budget_usage": session.budget.usage(), + } + preparation_id = hashlib.sha256( + canonical_json_bytes(_json_value(values)) + ).hexdigest() + _raise_if_cancelled(cancellation) + return DiscoveryCandidatePreparation( + preparation_id=preparation_id, + **values, + ) + + +def expand_discovery( + source: DiscoverySource, + preparation: DiscoveryCandidatePreparation, + expansion: DiscoveryExpansionRequest, + *, + cancellation: asyncio.Event | None = None, +) -> DiscoveryExpansionResult: + """Execute one bounded deterministic evidence lookup against repository truth.""" + + if expansion.preparation_id != preparation.preparation_id: + raise DiscoveryPreparationMismatchError( + "expansion does not reference the supplied preparation" + ) + _raise_if_cancelled(cancellation) + snapshot = _snapshot(source) + current = prepare_discovery_candidates(snapshot, _request(preparation)) + _require_same_preparation(preparation, current) + session, executor = _prepare_runtime(snapshot, _request(preparation)) + usage = _merged_usage(preparation.budget_usage, expansion.budget_usage) + _restore_budget(session.budget, usage) + if session.budget.steps >= preparation.budget.max_steps: + raise DiscoveryApplicationError("maximum discovery steps exceeded") + session.budget.steps += 1 + action_id = hashlib.sha256( + canonical_json_bytes( + {"operation": expansion.operation, "arguments": expansion.arguments} + ) + ).hexdigest()[:32] + observation = executor.execute( + step=session.budget.steps, + action_id=action_id, + tool_name=expansion.operation, + arguments=expansion.arguments, + ) + _raise_if_cancelled(cancellation) + return DiscoveryExpansionResult( + preparation_id=preparation.preparation_id, + operation=expansion.operation, + ok=observation.ok, + code=observation.code, + data=observation.data, + truncated=observation.truncated, + made_progress=observation.made_progress, + budget_usage=session.budget.usage(), + ) + + +def read_verified_context( + source: DiscoverySource, + preparation: DiscoveryCandidatePreparation, + selection: DiscoverySelection, + *, + cancellation: asyncio.Event | None = None, +) -> VerifiedContext: + """Read a caller selection all-or-nothing and verify every source identity.""" + + if selection.preparation_id != preparation.preparation_id: + raise DiscoveryPreparationMismatchError( + "selection does not reference the supplied preparation" + ) + _raise_if_cancelled(cancellation) + snapshot = _snapshot(source) + _require_snapshot(preparation, snapshot) + candidates = {item.candidate_id: item for item in preparation.candidates} + files = {item.path: item for item in snapshot.files} + selected_files = [] + line_ranges: list[LineRangeRequest] = [] + for item in selection.items: + candidate = candidates.get(item.candidate_id) + if candidate is None: + raise DiscoverySelectionError( + f"candidate is outside the prepared set: {item.candidate_id}" + ) + project_file = files.get(candidate.path) + if project_file is None or project_file.sha256 != candidate.source_sha256: + raise DiscoveryPreparationMismatchError( + f"prepared source identity is stale: {candidate.path}" + ) + selected_files.append(project_file) + line_ranges.extend( + LineRangeRequest( + path=candidate.path, + range=LineRange(value.start_line, value.end_line), + ) + for value in item.ranges + ) + if len(selected_files) > preparation.budget.max_context_files: + raise DiscoveryApplicationError("maximum context files exceeded") + usage = _merged_usage(preparation.budget_usage, selection.budget_usage) + tracker = ToolBudgetTracker(preparation.budget) + _restore_budget(tracker, usage) + for project_file in selected_files: + tracker.charge_read(project_file.size_bytes) + verified = read_selected_text_files( + snapshot, + tuple(selected_files), + line_ranges=tuple(line_ranges), + limits=ReaderLimits( + max_files=preparation.budget.max_context_files, + max_source_bytes=preparation.budget.max_source_bytes, + max_content_bytes=preparation.budget.max_context_bytes, + ), + ) + total_bytes = sum(item.included_content_bytes for item in verified) + if total_bytes > preparation.budget.max_context_bytes: + raise DiscoveryApplicationError("maximum context bytes exceeded") + tracker.context_files = len(verified) + tracker.context_bytes = total_bytes + candidate_by_path = { + candidates[item.candidate_id].path: item.candidate_id + for item in selection.items + } + result_files = tuple( + VerifiedContextFile( + candidate_id=candidate_by_path[item.project_file.path], + path=item.project_file.path, + language=item.project_file.language, + source_size_bytes=item.project_file.size_bytes, + source_sha256=item.project_file.sha256, + source_line_count=item.source_line_count, + blocks=tuple( + VerifiedContextBlock( + start_line=( + block.line_range.start if block.line_range is not None else None + ), + end_line=( + block.line_range.end if block.line_range is not None else None + ), + text=block.text, + line_count=block.line_count, + size_bytes=block.size_bytes, + sha256=block.sha256, + ) + for block in item.blocks + ), + included_line_count=item.included_line_count, + included_content_bytes=item.included_content_bytes, + ) + for item in verified + ) + _raise_if_cancelled(cancellation) + return VerifiedContext( + preparation_id=preparation.preparation_id, + task=preparation.task, + mode=preparation.mode, + source_snapshot_digest=preparation.source_snapshot_digest, + index_generation_id=preparation.index_generation_id, + files=result_files, + budget_usage=tracker.usage(), + ) + + +def package_verified_context( + source: DiscoverySource, + verified: VerifiedContext, + *, + include_tree: bool = True, + cancellation: asyncio.Event | None = None, +) -> ContextPackage: + """Re-verify and package one immutable verified-context DTO.""" + + _raise_if_cancelled(cancellation) + snapshot = _snapshot(source) + if calculate_source_snapshot_digest(snapshot) != verified.source_snapshot_digest: + raise DiscoveryPreparationMismatchError( + "verified context does not match the current repository snapshot" + ) + ranges = tuple( + LineRangeRequest( + path=item.path, + range=LineRange(block.start_line, block.end_line), + ) + for item in verified.files + for block in item.blocks + if block.start_line is not None and block.end_line is not None + ) + package = build_context_package( + snapshot, + ContextBuildOptions( + title=verified.task, + selection=ContextSelection( + exact_paths=tuple(item.path for item in verified.files), + line_ranges=ranges, + ), + include_tree=include_tree, + max_files=max(len(verified.files), 1), + max_source_bytes_per_file=max( + max(item.source_size_bytes for item in verified.files), 1 + ), + max_total_content_bytes=max( + sum(item.included_content_bytes for item in verified.files), 1 + ), + ), + ) + expected = tuple( + ( + item.path, + item.source_sha256, + tuple( + (block.start_line, block.end_line, block.sha256) + for block in item.blocks + ), + ) + for item in verified.files + ) + actual = tuple( + ( + item.path, + item.source_sha256, + tuple( + (block.start_line, block.end_line, block.sha256) + for block in item.blocks + ), + ) + for item in package.files + ) + if actual != expected: + raise DiscoveryPreparationMismatchError( + "verified content changed before package construction" + ) + _raise_if_cancelled(cancellation) + return package + + +def _prepare_runtime( + snapshot: ProjectSnapshot, request: DiscoveryRequest +) -> tuple[DiscoverySession, DiscoveryToolExecutor]: + session = DiscoverySession(snapshot, None, request) + executor, _ = session.prepare_read_only_tools() + return session, executor + + +def _snapshot(source: DiscoverySource) -> ProjectSnapshot: + if isinstance(source, ProjectSnapshot): + return source + return scan_repository(source) + + +def _request(preparation: DiscoveryCandidatePreparation) -> DiscoveryRequest: + return DiscoveryRequest( + task=preparation.task, + mode=preparation.mode, + pinned_paths=preparation.pinned_paths, + excluded_paths=preparation.excluded_paths, + strict=preparation.strict, + budget=preparation.budget, + ) + + +def _require_snapshot( + preparation: DiscoveryCandidatePreparation, snapshot: ProjectSnapshot +) -> None: + if calculate_source_snapshot_digest(snapshot) != preparation.source_snapshot_digest: + raise DiscoveryPreparationMismatchError( + "preparation does not match the current repository snapshot" + ) + + +def _require_same_preparation( + expected: DiscoveryCandidatePreparation, + actual: DiscoveryCandidatePreparation, +) -> None: + if expected != actual: + raise DiscoveryPreparationMismatchError( + "repository or pinned index changed after candidate preparation" + ) + + +def _evidence_origin( + mode: DiscoveryMode, + path: str, + *, + manifest_paths: set[str], + structural_paths: set[str], +) -> Literal["snapshot", "fresh", "indexed", "hybrid"]: + if path not in structural_paths: + return "snapshot" + if mode is DiscoveryMode.FRESH: + return "fresh" + if path in manifest_paths: + return "indexed" + return "hybrid" + + +def _index_status( + mode: DiscoveryMode, knowledge: Any +) -> Literal["not_used", "unavailable", "current", "stale"]: + if mode is DiscoveryMode.FRESH: + return "not_used" + if knowledge.manifest is None: + return "unavailable" + if knowledge.stale_index_paths: + return "stale" + return "current" + + +def _canonical_warnings( + warnings: list[CompletenessWarning], +) -> tuple[CompletenessWarning, ...]: + unique = {(item.code, item.path, item.related_paths): item for item in warnings} + return tuple( + unique[key] + for key in sorted(unique, key=lambda item: (item[0], item[1] or "", item[2])) + ) + + +def _json_value(value: Any) -> Any: + if isinstance(value, BaseModel): + return value.model_dump(mode="json") + if isinstance(value, dict): + return {key: _json_value(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [_json_value(item) for item in value] + return value + + +def _merged_usage( + baseline: DiscoveryBudgetUsage, supplied: DiscoveryBudgetUsage +) -> DiscoveryBudgetUsage: + left = baseline.model_dump() + right = supplied.model_dump() + return DiscoveryBudgetUsage(**{key: max(left[key], right[key]) for key in left}) + + +def _restore_budget(tracker: ToolBudgetTracker, usage: DiscoveryBudgetUsage) -> None: + for name, value in usage.model_dump().items(): + setattr(tracker, name, value) + + +def _raise_if_cancelled(cancellation: asyncio.Event | None) -> None: + if cancellation is not None and cancellation.is_set(): + raise asyncio.CancelledError + + +__all__ = [ + "DISCOVERY_EXPANSION_OPERATIONS", + "DiscoveryApplicationError", + "DiscoveryPreparationMismatchError", + "DiscoverySelectionError", + "expand_discovery", + "package_verified_context", + "prepare_discovery_candidates", + "read_verified_context", +] diff --git a/src/contextforge/discovery/models.py b/src/contextforge/discovery/models.py index bab421b..835395c 100644 --- a/src/contextforge/discovery/models.py +++ b/src/contextforge/discovery/models.py @@ -2,6 +2,7 @@ from __future__ import annotations +import hashlib import math import re from enum import StrEnum @@ -12,6 +13,25 @@ from contextforge.core.validation import Sha256, validate_portable_relative_path DISCOVERY_SCHEMA_VERSION: Literal[1] = 1 +DISCOVERY_APPLICATION_SCHEMA_VERSION: Literal[1] = 1 + +DiscoveryExpansionOperation = Literal[ + "get_repository_overview", + "list_tree", + "search_index", + "search_symbols", + "search_text", + "get_file_summary", + "get_symbol_summary", + "find_imports", + "find_importers", + "find_references", + "find_callers", + "find_related_tests", + "read_file", + "read_lines", + "get_context_budget", +] NonNegativeInt = Annotated[int, Field(ge=0, strict=True)] PositiveInt = Annotated[int, Field(gt=0, strict=True)] @@ -264,6 +284,272 @@ def validate_record_path(cls, value: str) -> str: return validate_portable_relative_path(value) +class PreparedDiscoveryCandidate(DiscoveryModel): + """Portable deterministic candidate offered to an external orchestrator.""" + + candidate_id: str + path: str + language: str + rank: PositiveInt + score: float = Field(ge=0.0, allow_inf_nan=False) + ranking_signals: tuple[str, ...] = Field(min_length=1, max_length=10) + source_sha256: Sha256 + source_size_bytes: NonNegativeInt + evidence_origin: Literal["snapshot", "fresh", "indexed", "hybrid"] + structural_evidence: bool + semantic_evidence: bool + + @field_validator("candidate_id") + @classmethod + def validate_id(cls, value: str) -> str: + if not _IDENTIFIER.fullmatch(value): + raise ValueError("candidate_id must be a bounded portable identifier") + return value + + @field_validator("path") + @classmethod + def validate_candidate_path(cls, value: str) -> str: + return validate_portable_relative_path(value) + + +class DiscoveryCandidatePreparation(DiscoveryModel): + """Immutable model-free discovery input pinned to repository truth.""" + + schema_version: Literal[1] = DISCOVERY_APPLICATION_SCHEMA_VERSION + preparation_id: Sha256 + task: str = Field(min_length=1, max_length=20_000) + mode: DiscoveryMode + pinned_paths: tuple[str, ...] = () + excluded_paths: tuple[str, ...] = () + strict: bool = False + source_snapshot_digest: Sha256 + index_generation_id: Sha256 | None = None + index_status: Literal["not_used", "unavailable", "current", "stale"] + candidates: tuple[PreparedDiscoveryCandidate, ...] + total_candidate_count: NonNegativeInt + stale_index_paths: tuple[str, ...] = () + warnings: tuple[CompletenessWarning, ...] = () + budget: DiscoveryBudget + budget_usage: DiscoveryBudgetUsage + + @field_validator("stale_index_paths") + @classmethod + def validate_stale_paths(cls, value: tuple[str, ...]) -> tuple[str, ...]: + paths = tuple(validate_portable_relative_path(path) for path in value) + if paths != tuple(sorted(set(paths))): + raise ValueError("stale index paths must be unique and canonical") + return paths + + @field_validator("pinned_paths", "excluded_paths") + @classmethod + def validate_manual_paths(cls, value: tuple[str, ...]) -> tuple[str, ...]: + paths = tuple(validate_portable_relative_path(path) for path in value) + if paths != tuple(sorted(set(paths))): + raise ValueError("manual paths must be unique and canonical") + return paths + + @model_validator(mode="after") + def validate_candidates(self) -> DiscoveryCandidatePreparation: + identifiers = tuple(item.candidate_id for item in self.candidates) + ranks = tuple(item.rank for item in self.candidates) + if len(identifiers) != len(set(identifiers)): + raise ValueError("prepared candidate IDs must be unique") + if ranks != tuple(sorted(ranks)): + raise ValueError("prepared candidates must use canonical rank order") + if self.total_candidate_count < len(self.candidates): + raise ValueError("total candidate count cannot be below serialized count") + return self + + +class DiscoveryExpansionRequest(DiscoveryModel): + """One stateless read-only evidence operation over a preparation.""" + + schema_version: Literal[1] = DISCOVERY_APPLICATION_SCHEMA_VERSION + preparation_id: Sha256 + operation: DiscoveryExpansionOperation + arguments: dict[str, Any] = Field(default_factory=dict) + budget_usage: DiscoveryBudgetUsage = Field(default_factory=DiscoveryBudgetUsage) + + +class DiscoveryExpansionResult(DiscoveryModel): + """Immutable result of one deterministic discovery expansion.""" + + schema_version: Literal[1] = DISCOVERY_APPLICATION_SCHEMA_VERSION + preparation_id: Sha256 + operation: DiscoveryExpansionOperation + ok: bool + code: str + data: dict[str, Any] = Field(default_factory=dict) + truncated: bool = False + made_progress: bool = True + budget_usage: DiscoveryBudgetUsage + + @field_validator("code") + @classmethod + def validate_code(cls, value: str) -> str: + if not _IDENTIFIER.fullmatch(value): + raise ValueError("result code must be bounded and portable") + return value + + +class DiscoverySelectionItem(DiscoveryModel): + """One prepared candidate selected by a caller, optionally by line range.""" + + candidate_id: str + ranges: tuple[DiscoveryLineRange, ...] = () + + @field_validator("candidate_id") + @classmethod + def validate_id(cls, value: str) -> str: + if not _IDENTIFIER.fullmatch(value): + raise ValueError("candidate_id must be a bounded portable identifier") + return value + + @field_validator("ranges") + @classmethod + def validate_ranges( + cls, value: tuple[DiscoveryLineRange, ...] + ) -> tuple[DiscoveryLineRange, ...]: + previous_end = 0 + for item in value: + if item.start_line <= previous_end: + raise ValueError("selection ranges must be sorted and disjoint") + previous_end = item.end_line + return value + + +class DiscoverySelection(DiscoveryModel): + """Caller-owned selection over a specific deterministic preparation.""" + + schema_version: Literal[1] = DISCOVERY_APPLICATION_SCHEMA_VERSION + preparation_id: Sha256 + items: tuple[DiscoverySelectionItem, ...] = Field(min_length=1) + budget_usage: DiscoveryBudgetUsage = Field(default_factory=DiscoveryBudgetUsage) + + @field_validator("items") + @classmethod + def validate_items( + cls, value: tuple[DiscoverySelectionItem, ...] + ) -> tuple[DiscoverySelectionItem, ...]: + identifiers = tuple(item.candidate_id for item in value) + if identifiers != tuple(sorted(set(identifiers))): + raise ValueError("selection items must be unique and canonical") + return value + + +class VerifiedContextBlock(DiscoveryModel): + """One verified canonical source block.""" + + start_line: PositiveInt | None = None + end_line: PositiveInt | None = None + text: str + line_count: NonNegativeInt + size_bytes: NonNegativeInt + sha256: Sha256 + + @model_validator(mode="after") + def validate_bounds(self) -> VerifiedContextBlock: + if (self.start_line is None) != (self.end_line is None): + raise ValueError("verified block bounds must both be set or absent") + if ( + self.start_line is not None + and self.end_line is not None + and self.end_line < self.start_line + ): + raise ValueError("verified block end must not precede start") + if "\r" in self.text: + raise ValueError("verified text must use canonical LF newlines") + encoded = self.text.encode("utf-8") + if self.size_bytes != len(encoded): + raise ValueError("verified block size does not match UTF-8 content") + if self.sha256 != hashlib.sha256(encoded).hexdigest(): + raise ValueError("verified block hash does not match content") + line_count = ( + 0 + if not self.text + else self.text.count("\n") + (not self.text.endswith("\n")) + ) + if self.line_count != line_count: + raise ValueError("verified block line count does not match content") + if ( + self.start_line is not None + and self.end_line is not None + and self.line_count != self.end_line - self.start_line + 1 + ): + raise ValueError("verified block line count does not match range") + return self + + +class VerifiedContextFile(DiscoveryModel): + """One snapshot-owned file whose selected content was re-read and verified.""" + + candidate_id: str + path: str + language: str | None = None + source_size_bytes: NonNegativeInt + source_sha256: Sha256 + source_line_count: NonNegativeInt + blocks: tuple[VerifiedContextBlock, ...] = Field(min_length=1) + included_line_count: NonNegativeInt + included_content_bytes: NonNegativeInt + + @field_validator("path") + @classmethod + def validate_file_path(cls, value: str) -> str: + return validate_portable_relative_path(value) + + @field_validator("candidate_id") + @classmethod + def validate_candidate_id(cls, value: str) -> str: + if not _IDENTIFIER.fullmatch(value): + raise ValueError("candidate_id must be a bounded portable identifier") + return value + + @model_validator(mode="after") + def validate_content(self) -> VerifiedContextFile: + if self.included_line_count != sum(item.line_count for item in self.blocks): + raise ValueError("included line count does not match verified blocks") + if self.included_content_bytes != sum(item.size_bytes for item in self.blocks): + raise ValueError("included byte count does not match verified blocks") + if self.included_content_bytes > self.source_size_bytes: + raise ValueError("verified content cannot exceed source size") + ranged = any(item.start_line is not None for item in self.blocks) + if ranged and any(item.start_line is None for item in self.blocks): + raise ValueError("verified file cannot mix full and ranged blocks") + if not ranged and len(self.blocks) != 1: + raise ValueError("verified full source requires exactly one block") + return self + + +class VerifiedContext(DiscoveryModel): + """All-or-nothing verified source content ready for deterministic packaging.""" + + schema_version: Literal[1] = DISCOVERY_APPLICATION_SCHEMA_VERSION + preparation_id: Sha256 + task: str = Field(min_length=1, max_length=20_000) + mode: DiscoveryMode + source_snapshot_digest: Sha256 + index_generation_id: Sha256 | None = None + files: tuple[VerifiedContextFile, ...] = Field(min_length=1) + budget_usage: DiscoveryBudgetUsage + + @model_validator(mode="after") + def validate_files(self) -> VerifiedContext: + identifiers = tuple(item.candidate_id for item in self.files) + paths = tuple(item.path for item in self.files) + if len(identifiers) != len(set(identifiers)): + raise ValueError("verified candidate IDs must be unique") + if paths != tuple(sorted(set(paths))): + raise ValueError("verified paths must be unique and canonical") + if self.budget_usage.context_files != len(self.files): + raise ValueError("context file usage does not match verified files") + if self.budget_usage.context_bytes != sum( + item.included_content_bytes for item in self.files + ): + raise ValueError("context byte usage does not match verified files") + return self + + class CompletenessWarning(DiscoveryModel): """Advisory missing-context or static-analysis limitation.""" @@ -446,6 +732,7 @@ def validate_terminal_state(self) -> DiscoveryRunRecord: __all__ = [ + "DISCOVERY_APPLICATION_SCHEMA_VERSION", "DISCOVERY_SCHEMA_VERSION", "CompletenessWarning", "DiscoveryAction", @@ -453,14 +740,24 @@ def validate_terminal_state(self) -> DiscoveryRunRecord: "DiscoveryBudget", "DiscoveryBudgetUsage", "DiscoveryCandidate", + "DiscoveryCandidatePreparation", "DiscoveryCandidateRecord", + "DiscoveryExpansionRequest", + "DiscoveryExpansionOperation", + "DiscoveryExpansionResult", "DiscoveryLineRange", "DiscoveryMode", "DiscoveryObservation", "DiscoveryRequest", "DiscoveryRunRecord", + "DiscoverySelection", + "DiscoverySelectionItem", "DiscoveryState", "IndexedContextSelection", + "PreparedDiscoveryCandidate", "FinalContextSelection", "SelectionReason", + "VerifiedContext", + "VerifiedContextBlock", + "VerifiedContextFile", ] diff --git a/src/contextforge/discovery/session.py b/src/contextforge/discovery/session.py index 719bf7d..4d48d5f 100644 --- a/src/contextforge/discovery/session.py +++ b/src/contextforge/discovery/session.py @@ -184,7 +184,7 @@ class DiscoverySession: def __init__( self, snapshot: ProjectSnapshot, - provider: ModelProvider, + provider: ModelProvider | None, request: DiscoveryRequest, *, git_diff_provider: GitDiffProvider | None = None, @@ -390,7 +390,7 @@ async def run(self) -> DiscoveryRunRecord: "candidate_count_after_filtering": len(self._ranked_candidates), "candidate_count_after_ranking": len(self._ranked_candidates), "complete_index_considered_for_one_request": False, - "synthesis_provider": self.provider.provider_id, + "synthesis_provider": self._require_provider().provider_id, "synthesis_mode": "provider", }, ) @@ -1345,7 +1345,7 @@ async def _request_actions(self) -> tuple[DiscoveryAction, ...]: total_deadline = asyncio.timeout(remaining) try: async with total_deadline: - response = await self.provider.complete_structured( + response = await self._require_provider().complete_structured( request, cancellation=self.cancellation ) except TimeoutError as exc: @@ -2148,6 +2148,11 @@ def _require_executor(self) -> DiscoveryToolExecutor: raise RuntimeError("discovery tools are not initialized") return self._executor + def _require_provider(self) -> ModelProvider: + if self.provider is None: + raise RuntimeError("model-assisted discovery requires a ModelProvider") + return self.provider + async def discover_repository( snapshot: ProjectSnapshot, diff --git a/src/contextforge/mcp/foundation.py b/src/contextforge/mcp/foundation.py index 200b278..b1919bb 100644 --- a/src/contextforge/mcp/foundation.py +++ b/src/contextforge/mcp/foundation.py @@ -212,7 +212,7 @@ def __init__( # required later for the explicitly named suggest_context tool. session = DiscoverySession( self.snapshot, - provider, # type: ignore[arg-type] + provider, request, git_diff_provider=self.git_provider, ) diff --git a/tests/test_bridge.py b/tests/test_bridge.py new file mode 100644 index 0000000..51cecb3 --- /dev/null +++ b/tests/test_bridge.py @@ -0,0 +1,1142 @@ +import asyncio +import gc +import io +import json +import queue +import subprocess +import sys +import threading +import time +from pathlib import Path +from typing import Any, cast + +import pytest +from pydantic import ValidationError + +import contextforge.bridge.server as bridge_module +from contextforge.bridge import MAX_JSONRPC_MESSAGE_BYTES, BridgeServer +from contextforge.bridge.models import ( + BridgeSelectionItem, + CancelParams, + DiscoverParams, + ReadParams, +) + + +def test_bridge_protocol_schema_is_closed_and_matches_v1() -> None: + root = Path(__file__).resolve().parents[1] + schema = json.loads( + (root / "docs/schemas/contextforge-bridge-v1.schema.json").read_text( + encoding="utf-8" + ) + ) + + assert schema["$defs"]["helloRequest"]["properties"]["params"]["required"] == [ + "protocol_version" + ] + expand = schema["$defs"]["expandRequest"]["properties"]["params"] + assert expand["additionalProperties"] is False + assert "action_id" not in expand["properties"] + assert "tool_name" not in expand["properties"] + assert schema["$defs"]["discoverResult"]["additionalProperties"] is False + + +class _QueueInput: + def __init__(self) -> None: + self._lines: queue.Queue[bytes] = queue.Queue() + + def send(self, frame: dict[str, Any] | bytes) -> None: + if isinstance(frame, bytes): + self._lines.put(frame) + else: + self._lines.put( + json.dumps(frame, ensure_ascii=False, separators=(",", ":")).encode() + + b"\n" + ) + + def close(self) -> None: + self._lines.put(b"") + + def readline(self, size: int = -1) -> bytes: + del size + return self._lines.get(timeout=5) + + +class _RecordingOutput: + def __init__(self) -> None: + self._condition = threading.Condition() + self.chunks: list[bytes] = [] + self.concurrent_write = False + self._writing = False + + def write(self, value: bytes) -> int: + with self._condition: + if self._writing: + self.concurrent_write = True + self._writing = True + time.sleep(0.002) + with self._condition: + self.chunks.append(value) + self._writing = False + self._condition.notify_all() + return len(value) + + def flush(self) -> None: + pass + + def wait(self, count: int) -> list[dict[str, Any]]: + deadline = time.monotonic() + 10 + with self._condition: + while len(self.chunks) < count: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise AssertionError("timed out waiting for bridge frames") + self._condition.wait(remaining) + chunks = list(self.chunks[:count]) + return [cast(dict[str, Any], json.loads(chunk)) for chunk in chunks] + + +class _Harness: + def __init__( + self, + workspace: Path, + *, + shutdown_timeout_seconds: float | None = None, + ) -> None: + self.input = _QueueInput() + self.output = _RecordingOutput() + self.stderr = io.StringIO() + self.server = ( + BridgeServer(workspace) + if shutdown_timeout_seconds is None + else BridgeServer( + workspace, + shutdown_timeout_seconds=shutdown_timeout_seconds, + ) + ) + self.task: asyncio.Task[None] | None = None + + async def start(self, *, negotiated: bool = True) -> None: + self.server._protocol_negotiated = negotiated + self.task = asyncio.create_task( + self.server.serve( + cast(Any, self.input), cast(Any, self.output), self.stderr + ) + ) + + async def response(self, count: int) -> list[dict[str, Any]]: + return await asyncio.to_thread(self.output.wait, count) + + async def close(self) -> None: + self.input.close() + assert self.task is not None + await self.task + + +def _request( + request_id: str | int, method: str, params: dict[str, Any] | None = None +) -> dict[str, Any]: + return { + "jsonrpc": "2.0", + "id": request_id, + "method": method, + "params": params or {}, + } + + +async def _snapshot(harness: _Harness, response_count: int = 1) -> str: + harness.input.send(_request("snapshot", "snapshot")) + response = (await harness.response(response_count))[-1] + return cast(str, response["result"]["snapshot_digest"]) + + +async def _discover( + harness: _Harness, digest: str, response_count: int = 2 +) -> dict[str, Any]: + harness.input.send( + _request( + "discover", + "discover", + { + "expected_snapshot_digest": digest, + "task": "Find alpha", + "mode": "fresh", + }, + ) + ) + return (await harness.response(response_count))[-1] + + +def test_bridge_handshake_protocol_purity_and_shutdown(tmp_path: Path) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send(_request("hello", "hello", {"protocol_version": "1.0"})) + hello = (await harness.response(1))[0] + assert hello["jsonrpc"] == "2.0" + assert hello["result"]["protocol_version"] == "1.0" + assert hello["result"]["supported_protocol_versions"] == ["1.0"] + assert hello["result"]["capabilities"]["model_free_discovery"] is True + assert hello["result"]["policy"]["source_writes"] is False + assert "shell" in hello["result"]["policy"] + + harness.input.send(_request("shutdown", "shutdown")) + frames = await harness.response(2) + assert frames[-1]["result"] == {"shutdown": True} + assert harness.task is not None + await harness.task + assert harness.stderr.getvalue() == "" + assert all(chunk.endswith(b"\n") for chunk in harness.output.chunks) + assert all( + json.loads(chunk)["jsonrpc"] == "2.0" for chunk in harness.output.chunks + ) + + asyncio.run(exercise()) + + +def test_bridge_requires_compatible_protocol_negotiation(tmp_path: Path) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start(negotiated=False) + + harness.input.send(_request("early", "snapshot")) + early = (await harness.response(1))[-1] + assert early["error"]["data"]["code"] == "PROTOCOL_NEGOTIATION_REQUIRED" + + harness.input.send(_request("missing", "hello")) + missing = (await harness.response(2))[-1] + assert missing["error"]["data"]["code"] == "INVALID_PARAMS" + + harness.input.send( + _request("incompatible", "hello", {"protocol_version": "2.0"}) + ) + incompatible = (await harness.response(3))[-1] + assert incompatible["error"]["data"] == { + "code": "INCOMPATIBLE_PROTOCOL_VERSION", + "requested_protocol_version": "2.0", + "supported_protocol_versions": ["1.0"], + } + + harness.input.send(_request("compatible", "hello", {"protocol_version": "1.0"})) + compatible = (await harness.response(4))[-1] + assert compatible["result"]["protocol_version"] == "1.0" + + harness.input.send(_request("snapshot", "snapshot")) + snapshot = (await harness.response(5))[-1] + assert len(snapshot["result"]["snapshot_digest"]) == 64 + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_rejects_malformed_oversized_unknown_and_invalid_requests( + tmp_path: Path, +) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send(b"{broken\n") + harness.input.send(b"{" + b"x" * MAX_JSONRPC_MESSAGE_BYTES + b"}\n") + harness.input.send(_request("unknown", "run_shell")) + harness.input.send(_request("invalid", "discover", {"task": "x"})) + frames = await harness.response(4) + codes = [frame["error"]["data"]["code"] for frame in frames] + assert codes == [ + "MALFORMED_JSON", + "MESSAGE_TOO_LARGE", + "METHOD_NOT_FOUND", + "INVALID_PARAMS", + ] + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_handles_duplicate_and_concurrent_requests_with_serial_writes( + tmp_path: Path, monkeypatch: Any +) -> None: + original_scan = cast(Any, bridge_module).scan_repository + started = threading.Event() + release = threading.Event() + calls = 0 + calls_lock = threading.Lock() + + def controlled_scan(path: Path) -> Any: + nonlocal calls + with calls_lock: + calls += 1 + current = calls + if current == 1: + started.set() + assert release.wait(5) + return original_scan(path) + + monkeypatch.setattr(bridge_module, "scan_repository", controlled_scan) + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send(_request("same", "status")) + assert await asyncio.to_thread(started.wait, 5) + harness.input.send(_request("same", "status")) + harness.input.send(_request("other", "status")) + duplicate = (await harness.response(1))[0] + assert duplicate["error"]["data"]["code"] == "DUPLICATE_REQUEST_ID" + release.set() + frames = await harness.response(3) + assert {frame["id"] for frame in frames[1:]} == {"same", "other"} + assert not harness.output.concurrent_write + assert all(len(chunk.splitlines()) == 1 for chunk in harness.output.chunks) + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_cancellation_reaches_application_operation( + tmp_path: Path, monkeypatch: Any +) -> None: + (tmp_path / "alpha.py").write_text("def alpha():\n pass\n", encoding="utf-8") + started = threading.Event() + observed = threading.Event() + + def cancellable_prepare( + source: Any, request: Any, *, cancellation: asyncio.Event | None = None + ) -> Any: + del source, request + assert cancellation is not None + started.set() + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + if cancellation.is_set(): + observed.set() + raise asyncio.CancelledError + time.sleep(0.001) + raise AssertionError("application operation did not receive cancellation") + + monkeypatch.setattr( + bridge_module, "prepare_discovery_candidates", cancellable_prepare + ) + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + digest = await _snapshot(harness) + harness.input.send( + _request( + "slow", + "discover", + { + "expected_snapshot_digest": digest, + "task": "alpha", + "mode": "fresh", + }, + ) + ) + assert await asyncio.to_thread(started.wait, 5) + harness.input.send( + { + "jsonrpc": "2.0", + "method": "$/cancelRequest", + "params": {"id": "slow"}, + } + ) + response = (await harness.response(2))[-1] + assert response["id"] == "slow" + assert response["error"]["data"]["code"] == "REQUEST_CANCELLED" + assert observed.is_set() + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_shutdown_cooperatively_cancels_active_operation( + tmp_path: Path, monkeypatch: Any +) -> None: + (tmp_path / "alpha.py").write_text("alpha = 1\n", encoding="utf-8") + started = threading.Event() + observed = threading.Event() + + def cooperative_prepare( + source: Any, request: Any, *, cancellation: asyncio.Event | None = None + ) -> Any: + del source, request + assert cancellation is not None + started.set() + while not cancellation.is_set(): + time.sleep(0.001) + observed.set() + raise asyncio.CancelledError + + monkeypatch.setattr( + bridge_module, "prepare_discovery_candidates", cooperative_prepare + ) + + async def exercise() -> None: + harness = _Harness(tmp_path, shutdown_timeout_seconds=0.2) + await harness.start() + digest = await _snapshot(harness) + harness.input.send( + _request( + "slow", + "discover", + { + "expected_snapshot_digest": digest, + "task": "alpha", + "mode": "fresh", + }, + ) + ) + assert await asyncio.to_thread(started.wait, 5) + started_at = time.monotonic() + harness.input.send(_request("shutdown", "shutdown")) + assert harness.task is not None + await asyncio.wait_for(harness.task, timeout=1) + elapsed = time.monotonic() - started_at + frames = [json.loads(chunk) for chunk in harness.output.chunks] + shutdown = next(frame for frame in frames if frame["id"] == "shutdown") + cancelled = next(frame for frame in frames if frame["id"] == "slow") + assert shutdown["result"] == {"shutdown": True} + assert cancelled["error"]["data"]["code"] == "REQUEST_CANCELLED" + assert "result" not in cancelled + assert observed.is_set() + assert elapsed < 0.5 + assert harness.stderr.getvalue() == "" + assert all( + json.loads(chunk)["jsonrpc"] == "2.0" for chunk in harness.output.chunks + ) + + asyncio.run(exercise()) + + +def test_bridge_shutdown_is_bounded_when_task_ignores_first_cancellation( + tmp_path: Path, +) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path, shutdown_timeout_seconds=0.02) + await harness.start() + cooperative = asyncio.Event() + first_task_cancel = asyncio.Event() + second_task_cancel = asyncio.Event() + blocker = asyncio.Event() + + async def stubborn_operation() -> None: + try: + await blocker.wait() + except asyncio.CancelledError: + first_task_cancel.set() + try: + await blocker.wait() + except asyncio.CancelledError: + second_task_cancel.set() + raise + + task = asyncio.create_task(stubborn_operation()) + active_type = cast(Any, bridge_module)._ActiveRequest + harness.server._active[("str", "stubborn")] = active_type( + cancellation=cooperative, + task=task, + ) + await asyncio.sleep(0) + started_at = time.monotonic() + harness.input.send(_request("shutdown", "shutdown")) + shutdown = (await harness.response(1))[0] + assert harness.task is not None + await asyncio.wait_for(harness.task, timeout=0.5) + elapsed = time.monotonic() - started_at + await asyncio.wait_for(second_task_cancel.wait(), timeout=0.5) + with pytest.raises(asyncio.CancelledError): + await task + assert shutdown["result"] == {"shutdown": True} + assert cooperative.is_set() + assert first_task_cancel.is_set() + assert task.done() + assert elapsed < 0.5 + stderr = harness.stderr.getvalue() + assert "shutdown deadline expired" in stderr + assert "Traceback" not in stderr + assert len(harness.output.chunks) == 1 + assert json.loads(harness.output.chunks[0])["jsonrpc"] == "2.0" + + asyncio.run(exercise()) + + +def test_bridge_eof_cleanup_is_bounded_for_non_cooperative_task( + tmp_path: Path, +) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path, shutdown_timeout_seconds=0.02) + await harness.start() + cooperative = asyncio.Event() + first_task_cancel = asyncio.Event() + second_task_cancel = asyncio.Event() + blocker = asyncio.Event() + + async def stubborn_operation() -> None: + try: + await blocker.wait() + except asyncio.CancelledError: + first_task_cancel.set() + try: + await blocker.wait() + except asyncio.CancelledError: + second_task_cancel.set() + raise + + task = asyncio.create_task(stubborn_operation()) + active_type = cast(Any, bridge_module)._ActiveRequest + harness.server._active[("str", "stubborn-eof")] = active_type( + cancellation=cooperative, + task=task, + ) + await asyncio.sleep(0) + started_at = time.monotonic() + harness.input.close() + assert harness.task is not None + await asyncio.wait_for(harness.task, timeout=0.5) + elapsed = time.monotonic() - started_at + await asyncio.wait_for(second_task_cancel.wait(), timeout=0.5) + with pytest.raises(asyncio.CancelledError): + await task + assert cooperative.is_set() + assert first_task_cancel.is_set() + assert task.done() + assert elapsed < 0.5 + assert harness.output.chunks == [] + stderr = harness.stderr.getvalue() + assert "shutdown deadline expired" in stderr + assert "Traceback" not in stderr + + asyncio.run(exercise()) + + +def test_bridge_sync_runner_does_not_reawait_abandoned_task( + tmp_path: Path, monkeypatch: Any, capsys: Any +) -> None: + cancellation_count = 0 + retained: list[asyncio.Task[None]] = [] + + async def fake_serve( + workspace: Any, + input_stream: Any, + output_stream: Any, + error_stream: Any, + ) -> None: + del workspace, input_stream, output_stream, error_stream + + async def repeatedly_non_cooperative() -> None: + nonlocal cancellation_count + blocker = asyncio.Event() + while True: + try: + await blocker.wait() + except asyncio.CancelledError: + cancellation_count += 1 + + task = asyncio.create_task(repeatedly_non_cooperative()) + retained.append(task) + await asyncio.sleep(0) + task.cancel() + await asyncio.sleep(0) + task.cancel() + await asyncio.sleep(0) + + monkeypatch.setattr(bridge_module, "serve_stdio_bridge", fake_serve) + started_at = time.monotonic() + bridge_module.run_stdio_bridge(tmp_path, io.BytesIO(), io.BytesIO(), io.StringIO()) + elapsed = time.monotonic() - started_at + assert cancellation_count == 2 + assert not retained[0].done() + assert elapsed < 0.5 + retained.clear() + gc.collect() + captured = capsys.readouterr() + assert "Task was destroyed" not in captured.err + assert "Traceback" not in captured.err + + +def test_bridge_discover_expand_read_and_package_are_verified_and_in_memory( + tmp_path: Path, +) -> None: + source = tmp_path / "alpha.py" + source.write_text("def alpha():\n return 1\n", encoding="utf-8") + initial = source.read_bytes() + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + digest = await _snapshot(harness) + discovered = await _discover(harness, digest) + result = discovered["result"] + assert result["model_provider_used"] is False + assert { + "schema_version", + "task", + "pinned_paths", + "excluded_paths", + "strict", + "budget", + }.isdisjoint(result) + assert [item["rank"] for item in result["candidates"]] == sorted( + item["rank"] for item in result["candidates"] + ) + candidate = result["candidates"][0] + preparation_id = result["preparation_id"] + + harness.input.send( + _request( + "expand", + "expand", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "operation": "text", + "arguments": {"query": "alpha"}, + }, + ) + ) + expanded = (await harness.response(3))[-1] + assert expanded["result"]["operation"] == "text" + assert expanded["result"]["ok"] is True + assert "observation" not in expanded["result"] + assert "tool_name" not in expanded["result"] + + item = { + "candidate_id": candidate["candidate_id"], + "path": candidate["path"], + "source_sha256": candidate["source_sha256"], + "ranges": [{"start_line": 1, "end_line": 1}], + } + harness.input.send( + _request( + "read", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [item], + }, + ) + ) + read = (await harness.response(4))[-1] + assert read["result"]["files"][0]["path"] == "alpha.py" + assert read["result"]["files"][0]["blocks"][0]["text"] == "def alpha():\n" + assert {"schema_version", "task", "mode", "index_generation_id"}.isdisjoint( + read["result"] + ) + + harness.input.send( + _request( + "package", + "package", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [item], + }, + ) + ) + packaged = (await harness.response(5))[-1] + assert packaged["result"]["selection_id"] == read["result"]["selection_id"] + package = packaged["result"]["package"] + assert package["files"][0]["source_sha256"] == candidate["source_sha256"] + assert source.read_bytes() == initial + assert not (tmp_path / ".contextforge").exists() + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_rejects_path_traversal_source_mismatch_and_snapshot_drift( + tmp_path: Path, +) -> None: + source = tmp_path / "alpha.py" + source.write_text("alpha = 1\n", encoding="utf-8") + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + digest = await _snapshot(harness) + discovered = await _discover(harness, digest) + result = discovered["result"] + candidate = result["candidates"][0] + preparation_id = result["preparation_id"] + + harness.input.send( + _request( + "traversal", + "expand", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "operation": "importers", + "arguments": {"path": "../secret"}, + }, + ) + ) + traversal = (await harness.response(3))[-1] + assert traversal["error"]["data"]["code"] == "INVALID_PARAMS" + + harness.input.send( + _request( + "mismatch", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [ + { + "candidate_id": candidate["candidate_id"], + "path": candidate["path"], + "source_sha256": "0" * 64, + } + ], + }, + ) + ) + mismatch = (await harness.response(4))[-1] + assert mismatch["error"]["data"]["code"] == "SOURCE_IDENTITY_CHANGED" + + harness.input.send( + _request( + "path-mismatch", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [ + { + "candidate_id": candidate["candidate_id"], + "path": "different.py", + } + ], + }, + ) + ) + path_mismatch = (await harness.response(5))[-1] + assert path_mismatch["error"]["data"]["code"] == "SOURCE_IDENTITY_CHANGED" + + source.write_text("alpha = 2\n", encoding="utf-8") + harness.input.send( + _request( + "drift", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [{"candidate_id": candidate["candidate_id"]}], + }, + ) + ) + drift = (await harness.response(6))[-1] + assert drift["error"]["data"]["code"] == "SOURCE_IDENTITY_CHANGED" + assert "result" not in drift + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_clean_eof_produces_no_spurious_frame(tmp_path: Path) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + await harness.close() + assert harness.output.chunks == [] + assert harness.stderr.getvalue() == "" + + asyncio.run(exercise()) + + +def test_bridge_cli_stdio_keeps_stdout_protocol_only(tmp_path: Path) -> None: + process = subprocess.Popen( + [ + sys.executable, + "-m", + "contextforge", + "bridge", + "--stdio", + "--workspace", + str(tmp_path), + ], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + encoding="utf-8", + ) + assert process.stdin is not None + assert process.stdout is not None + output = [] + for frame in ( + _request("hello", "hello", {"protocol_version": "1.0"}), + _request("snapshot", "snapshot"), + _request("shutdown", "shutdown"), + ): + process.stdin.write(json.dumps(frame) + "\n") + process.stdin.flush() + output.append(json.loads(process.stdout.readline())) + process.stdin.close() + returncode = process.wait(timeout=10) + assert process.stderr is not None + stderr = process.stderr.read() + + assert returncode == 0, stderr + assert [frame["id"] for frame in output] == ["hello", "snapshot", "shutdown"] + assert output[0]["result"]["protocol_version"] == "1.0" + assert len(output[1]["result"]["snapshot_digest"]) == 64 + assert all(frame["jsonrpc"] == "2.0" for frame in output) + assert stderr == "" + + +def test_bridge_validates_jsonrpc_envelopes_and_cancel_params(tmp_path: Path) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + invalid_frames: list[dict[str, Any]] = [ + {"jsonrpc": "1.0", "id": "version", "method": "hello", "params": {}}, + {"jsonrpc": "2.0", "id": "extra", "method": "hello", "params": {}, "x": 1}, + {"jsonrpc": "2.0", "id": "method", "method": 1, "params": {}}, + {"jsonrpc": "2.0", "id": [], "method": "hello", "params": {}}, + {"jsonrpc": "2.0", "method": "hello", "params": {}}, + {"jsonrpc": "2.0", "id": "params", "method": "hello", "params": []}, + {"jsonrpc": "2.0", "id": "missing-params", "method": "snapshot"}, + { + "jsonrpc": "2.0", + "id": "x" * 201, + "method": "snapshot", + "params": {}, + }, + ] + for frame in invalid_frames: + harness.input.send(frame) + harness.input.send(b"[]\n") + frames = await harness.response(len(invalid_frames) + 1) + assert all("error" in frame for frame in frames) + assert frames[0]["id"] == "version" + assert frames[3]["id"] is None + + harness.input.send(_request("cancel", "$/cancelRequest", {"id": "not-active"})) + cancelled = (await harness.response(10))[-1] + assert cancelled["result"] == {"cancelled": False} + harness.input.send(_request("bad-cancel", "$/cancelRequest", {"id": True})) + invalid_cancel = (await harness.response(11))[-1] + assert invalid_cancel["error"]["data"]["code"] == "INVALID_PARAMS" + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_repository_error_paths_are_typed_and_all_or_nothing( + tmp_path: Path, +) -> None: + (tmp_path / "alpha.py").write_text("alpha = 1\n", encoding="utf-8") + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send( + _request( + "premature", + "discover", + { + "expected_snapshot_digest": "0" * 64, + "task": "alpha", + "mode": "fresh", + }, + ) + ) + premature = (await harness.response(1))[-1] + assert premature["error"]["data"]["code"] == "SNAPSHOT_REQUIRED" + + digest = await _snapshot(harness, 2) + harness.input.send( + _request( + "unknown-preparation", + "expand", + { + "expected_snapshot_digest": digest, + "preparation_id": "0" * 64, + "operation": "text", + "arguments": {"query": "alpha"}, + }, + ) + ) + unknown = (await harness.response(3))[-1] + assert unknown["error"]["data"]["code"] == "UNKNOWN_PREPARATION" + + harness.input.send( + _request( + "discover-limited", + "discover", + { + "expected_snapshot_digest": digest, + "task": "alpha", + "mode": "fresh", + "budget": {"max_steps": 1, "max_context_bytes": 1}, + }, + ) + ) + discovered = (await harness.response(4))[-1]["result"] + candidate = discovered["candidates"][0] + preparation_id = discovered["preparation_id"] + + harness.input.send( + _request( + "unknown-candidate", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [{"candidate_id": "missing"}], + }, + ) + ) + unknown_candidate = (await harness.response(5))[-1] + assert unknown_candidate["error"]["data"]["code"] == "UNKNOWN_CANDIDATE" + + harness.input.send( + _request( + "invalid-range", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [ + { + "candidate_id": candidate["candidate_id"], + "ranges": [{"start_line": 1, "end_line": 999}], + } + ], + }, + ) + ) + invalid_range = (await harness.response(6))[-1] + assert invalid_range["error"]["data"]["code"] == "INVALID_SOURCE_RANGE" + + harness.input.send( + _request( + "limit", + "read", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "items": [{"candidate_id": candidate["candidate_id"]}], + }, + ) + ) + limited = (await harness.response(7))[-1] + assert limited["error"]["data"]["code"] == "RESOURCE_LIMIT_EXCEEDED" + + harness.input.send( + _request( + "expand-once", + "expand", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "operation": "text", + "arguments": {"query": "alpha"}, + }, + ) + ) + expanded = (await harness.response(8))[-1]["result"] + harness.input.send( + _request( + "expand-over-budget", + "expand", + { + "expected_snapshot_digest": digest, + "preparation_id": preparation_id, + "operation": "text", + "arguments": {"query": "alpha"}, + "budget_usage": expanded["budget_usage"], + }, + ) + ) + rejected = (await harness.response(9))[-1] + assert rejected["error"]["data"]["code"] == "APPLICATION_REQUEST_REJECTED" + + harness.input.send( + _request( + "status", + "status", + {"expected_snapshot_digest": "0" * 64}, + ) + ) + status = (await harness.response(10))[-1] + assert status["result"]["source_identity_changed"] is True + assert {"schema_version", "repository_identity"}.isdisjoint( + status["result"]["index"] + ) + await harness.close() + + asyncio.run(exercise()) + + +def test_bridge_timeout_and_internal_failure_are_safe( + tmp_path: Path, monkeypatch: Any +) -> None: + original_scan = cast(Any, bridge_module).scan_repository + + async def timeout_exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + digest = await _snapshot(harness) + + def slow_scan(path: Path) -> Any: + time.sleep(0.05) + return original_scan(path) + + monkeypatch.setattr(bridge_module, "scan_repository", slow_scan) + harness.input.send( + _request( + "timeout", + "discover", + { + "expected_snapshot_digest": digest, + "task": "alpha", + "mode": "fresh", + "timeout_ms": 1, + }, + ) + ) + timed_out = (await harness.response(2))[-1] + assert timed_out["error"]["data"]["code"] == "REQUEST_TIMEOUT" + await harness.close() + + asyncio.run(timeout_exercise()) + + def broken_scan(path: Path) -> Any: + del path + raise RuntimeError("absolute secret must not escape") + + monkeypatch.setattr(bridge_module, "scan_repository", broken_scan) + + async def failure_exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send(_request("failure", "status")) + failure = (await harness.response(1))[-1] + assert failure["error"]["data"]["code"] == "INTERNAL_ERROR" + assert "secret" not in json.dumps(failure) + assert "internal error" in harness.stderr.getvalue() + await harness.close() + + asyncio.run(failure_exercise()) + + +def test_bridge_parameter_models_reject_noncanonical_values() -> None: + with pytest.raises(ValidationError): + DiscoverParams.model_validate_json( + json.dumps( + { + "expected_snapshot_digest": "0" * 64, + "task": "x", + "pinned_paths": ["z.py", "a.py"], + } + ) + ) + with pytest.raises(ValidationError): + BridgeSelectionItem.model_validate( + { + "candidate_id": "candidate", + "path": "..\\secret", + } + ) + with pytest.raises(ValidationError): + ReadParams.model_validate_json( + json.dumps( + { + "expected_snapshot_digest": "0" * 64, + "preparation_id": "0" * 64, + "items": [ + {"candidate_id": "b"}, + {"candidate_id": "a"}, + ], + } + ) + ) + with pytest.raises(ValidationError): + CancelParams.model_validate({"id": True}) + with pytest.raises(ValidationError): + BridgeSelectionItem.model_validate_json( + json.dumps( + { + "candidate_id": "candidate", + "path": None, + "ranges": [ + {"start_line": 1, "end_line": 2}, + {"start_line": 2, "end_line": 3}, + ], + } + ) + ) + + +def test_bridge_low_level_resource_guards(tmp_path: Path) -> None: + regular_file = tmp_path / "file.txt" + regular_file.write_text("not a workspace", encoding="utf-8") + with pytest.raises(NotADirectoryError): + BridgeServer(regular_file) + + unstarted = BridgeServer(tmp_path) + with pytest.raises(RuntimeError, match="writer"): + unstarted._require_writer() + + diagnostics_type = cast(Any, bridge_module)._BoundedDiagnostics + diagnostics = diagnostics_type(None) + diagnostics.write("ignored") + target = io.StringIO() + diagnostics = diagnostics_type(target) + diagnostics._remaining = 3 + diagnostics.write("abcdef") + assert len(target.getvalue().encode("utf-8")) == 3 + diagnostics.write("ignored after exhaustion") + + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + harness.input.send(b"x" * (MAX_JSONRPC_MESSAGE_BYTES + 1)) + harness.input.send(b"discarded-tail\n") + harness.input.send( + b'{"jsonrpc":"2.0","id":"nan","method":"hello","params":{"timeout_ms":NaN}}\n' + ) + harness.input.send( + { + "jsonrpc": "2.0", + "method": "$/cancelRequest", + "params": {"id": True}, + } + ) + harness.input.send( + { + "jsonrpc": "2.0", + "method": "$/cancelRequest", + "params": {"id": "not-active"}, + } + ) + harness.input.send(_request("shutdown", "shutdown")) + frames = await harness.response(3) + assert [frame["error"]["data"]["code"] for frame in frames[:2]] == [ + "MESSAGE_TOO_LARGE", + "MALFORMED_JSON", + ] + assert frames[-1]["result"] == {"shutdown": True} + assert harness.task is not None + await harness.task + + asyncio.run(exercise()) + + +def test_bridge_rejects_new_work_after_shutdown_starts(tmp_path: Path) -> None: + async def exercise() -> None: + harness = _Harness(tmp_path) + await harness.start() + await asyncio.sleep(0) + harness.server._shutting_down = True + await harness.server._accept_request(_request("late", "hello"), inline=False) + late = (await harness.response(1))[-1] + assert late["error"]["data"]["code"] == "SHUTTING_DOWN" + harness.input.close() + assert harness.task is not None + await harness.task + + asyncio.run(exercise()) diff --git a/tests/test_discovery_application.py b/tests/test_discovery_application.py new file mode 100644 index 0000000..d8aed4b --- /dev/null +++ b/tests/test_discovery_application.py @@ -0,0 +1,318 @@ +import asyncio +import inspect +import json +from pathlib import Path +from typing import Any, cast + +import pytest +from pydantic import ValidationError + +from contextforge.application import ( + expand_discovery, + package_verified_context, + prepare_discovery_candidates, + read_verified_context, +) +from contextforge.context import SelectedFileTooLargeError +from contextforge.discovery import ( + DiscoveryBudget, + DiscoveryCandidatePreparation, + DiscoveryExpansionRequest, + DiscoveryMode, + DiscoveryRequest, + DiscoverySelection, + DiscoverySelectionItem, + DiscoveryUnavailableError, + discover_repository, +) +from contextforge.intelligence import ( + acquire_index_lock, + build_structural_index, + load_manifest, +) +from contextforge.models import ( + FakeModelProvider, + ModelRequest, + ProviderConfiguration, +) +from contextforge.repositories import ProjectSnapshot, scan_repository + + +def _snapshot(root: Path, files: dict[str, str]) -> ProjectSnapshot: + for path, content in files.items(): + destination = root.joinpath(*path.split("/")) + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_text(content, encoding="utf-8", newline="") + return scan_repository(root) + + +def _index(snapshot: ProjectSnapshot) -> None: + with acquire_index_lock(snapshot.root, "application-contract") as lock: + build_structural_index(snapshot, lock) + + +def _selection(preparation: DiscoveryCandidatePreparation) -> DiscoverySelection: + return DiscoverySelection( + preparation_id=preparation.preparation_id, + items=( + DiscoverySelectionItem( + candidate_id=min( + preparation.candidates, + key=lambda item: item.candidate_id, + ).candidate_id + ), + ), + ) + + +@pytest.mark.parametrize( + ("mode", "build_index", "expected_status", "expected_origin"), + [ + (DiscoveryMode.FRESH, False, "not_used", "fresh"), + (DiscoveryMode.INDEXED, True, "current", "indexed"), + (DiscoveryMode.HYBRID, True, "current", "indexed"), + (DiscoveryMode.HYBRID, False, "unavailable", "hybrid"), + ], +) +def test_model_free_candidate_preparation_preserves_mode_semantics( + tmp_path: Path, + mode: DiscoveryMode, + build_index: bool, + expected_status: str, + expected_origin: str, +) -> None: + snapshot = _snapshot( + tmp_path, + {"src/service.py": "def serve():\n return 1\n", "README.md": "service\n"}, + ) + if build_index: + _index(snapshot) + + preparation = prepare_discovery_candidates( + snapshot, + DiscoveryRequest(task="Find service", mode=mode), + ) + + assert preparation.mode is mode + assert preparation.index_status == expected_status + assert preparation.candidates + assert {item.evidence_origin for item in preparation.candidates} == { + expected_origin + } + assert preparation.budget_usage.model_calls == 0 + assert all(item.source_sha256 for item in preparation.candidates) + assert "provider" not in inspect.signature(prepare_discovery_candidates).parameters + assert ( + preparation.index_generation_id == load_manifest(tmp_path).generation_id + if build_index + else preparation.index_generation_id is None + ) + if mode is DiscoveryMode.HYBRID and not build_index: + assert any( + item.code == "hybrid-index-unavailable" for item in preparation.warnings + ) + + +def test_preparation_is_immutable_and_serializes_deterministically( + tmp_path: Path, +) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n", "b.py": "B = 2\n"}) + request = DiscoveryRequest(task="Find A", mode=DiscoveryMode.FRESH) + first = prepare_discovery_candidates(snapshot, request) + second = prepare_discovery_candidates(snapshot, request) + + assert first == second + assert first.model_dump_json() == second.model_dump_json() + assert json.dumps( + first.model_dump(mode="json"), sort_keys=True, separators=(",", ":") + ) == json.dumps( + second.model_dump(mode="json"), sort_keys=True, separators=(",", ":") + ) + with pytest.raises(ValidationError): + first.task = "changed" + with pytest.raises(ValidationError): + first.candidates[0].path = "changed.py" + + +def test_model_free_preparation_honors_cooperative_cancellation(tmp_path: Path) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n"}) + cancellation = asyncio.Event() + cancellation.set() + with pytest.raises(asyncio.CancelledError): + prepare_discovery_candidates( + snapshot, + DiscoveryRequest(task="A", mode=DiscoveryMode.FRESH), + cancellation=cancellation, + ) + + +def test_expansion_reuses_path_policy_and_returns_verified_source_identity( + tmp_path: Path, +) -> None: + snapshot = _snapshot(tmp_path, {"pkg/a.py": "def alpha():\n return 1\n"}) + preparation = prepare_discovery_candidates( + snapshot, DiscoveryRequest(task="alpha", mode=DiscoveryMode.FRESH) + ) + candidate = preparation.candidates[0] + result = expand_discovery( + snapshot, + preparation, + DiscoveryExpansionRequest( + preparation_id=preparation.preparation_id, + operation="read_file", + arguments={"path": candidate.path}, + ), + ) + invalid = expand_discovery( + snapshot, + preparation, + DiscoveryExpansionRequest( + preparation_id=preparation.preparation_id, + operation="read_file", + arguments={"path": "../secret"}, + ), + ) + + assert result.ok + assert result.data["source_sha256"] == candidate.source_sha256 + assert result.budget_usage.model_calls == 0 + assert invalid.code == "invalid_input" + assert "action_id" not in result.model_dump() + assert "tool_name" not in result.model_dump() + assert "step" not in result.model_dump() + + +def test_expansion_carries_budget_usage_and_enforces_step_limit(tmp_path: Path) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n"}) + preparation = prepare_discovery_candidates( + snapshot, + DiscoveryRequest( + task="A", + mode=DiscoveryMode.FRESH, + budget=DiscoveryBudget(max_steps=1), + ), + ) + first = expand_discovery( + snapshot, + preparation, + DiscoveryExpansionRequest( + preparation_id=preparation.preparation_id, + operation="get_repository_overview", + ), + ) + assert first.budget_usage.steps == 1 + with pytest.raises(RuntimeError, match="steps"): + expand_discovery( + snapshot, + preparation, + DiscoveryExpansionRequest( + preparation_id=preparation.preparation_id, + operation="get_repository_overview", + budget_usage=first.budget_usage, + ), + ) + + +def test_verified_read_and_package_are_pinned_and_budgeted(tmp_path: Path) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n", "b.py": "B = 2\n"}) + preparation = prepare_discovery_candidates( + snapshot, DiscoveryRequest(task="Find A", mode=DiscoveryMode.FRESH) + ) + selection = _selection(preparation) + verified = read_verified_context(snapshot, preparation, selection) + package = package_verified_context(snapshot, verified) + + assert verified.source_snapshot_digest == preparation.source_snapshot_digest + assert verified.files[0].source_sha256 in { + item.source_sha256 for item in preparation.candidates + } + assert package.files[0].source_sha256 == verified.files[0].source_sha256 + assert package.files[0].blocks[0].text == verified.files[0].blocks[0].text + assert package.title == preparation.task + + limited = prepare_discovery_candidates( + snapshot, + DiscoveryRequest( + task="Find A", + mode=DiscoveryMode.FRESH, + budget=DiscoveryBudget(max_context_bytes=1), + ), + ) + with pytest.raises(SelectedFileTooLargeError): + read_verified_context(snapshot, limited, _selection(limited)) + + +def test_source_changes_invalidate_preparation_and_verified_context( + tmp_path: Path, +) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n"}) + preparation = prepare_discovery_candidates( + snapshot, DiscoveryRequest(task="A", mode=DiscoveryMode.FRESH) + ) + (tmp_path / "a.py").write_text("A = 2\n", encoding="utf-8", newline="") + current = scan_repository(tmp_path) + + with pytest.raises(RuntimeError, match="snapshot"): + read_verified_context(current, preparation, _selection(preparation)) + + +def test_stale_index_is_rejected_or_disclosed_by_mode(tmp_path: Path) -> None: + snapshot = _snapshot(tmp_path, {"a.py": "A = 1\n", "b.py": "B = 1\n"}) + _index(snapshot) + (tmp_path / "b.py").write_text("B = 2\n", encoding="utf-8", newline="") + current = scan_repository(tmp_path) + + hybrid = prepare_discovery_candidates( + current, DiscoveryRequest(task="A", mode=DiscoveryMode.HYBRID) + ) + assert hybrid.index_status == "stale" + assert hybrid.stale_index_paths == ("b.py",) + assert any(item.code == "stale-index-coverage" for item in hybrid.warnings) + + stale_only = _snapshot(tmp_path / "only", {"only.py": "A = 1\n"}) + _index(stale_only) + (stale_only.root / "only.py").write_text("A = 2\n", encoding="utf-8", newline="") + with pytest.raises(DiscoveryUnavailableError): + prepare_discovery_candidates( + scan_repository(stale_only.root), + DiscoveryRequest(task="A", mode=DiscoveryMode.INDEXED), + ) + + +def test_existing_model_assisted_discovery_remains_compatible(tmp_path: Path) -> None: + snapshot = _snapshot(tmp_path, {"service.py": "def serve():\n return 1\n"}) + _index(snapshot) + + def responder(request: ModelRequest, _: int) -> str: + records = cast( + list[dict[str, Any]], request.trusted_code_map_facts["candidates"] + ) + return json.dumps( + { + "schema_version": 1, + "candidate_ids": [records[0]["candidate_id"]], + "summary": "Selected service implementation.", + } + ) + + provider = FakeModelProvider( + ProviderConfiguration( + provider_id="fake", + endpoint="fake://offline", + model_id="discovery-v1", + timeout_seconds=2, + retry_limit=0, + ), + responder=responder, + ) + result = asyncio.run( + discover_repository( + snapshot, + provider, + DiscoveryRequest(task="Find service", mode=DiscoveryMode.INDEXED), + ) + ) + + assert result.status == "complete" + assert result.final_selection is not None + assert result.budget_usage.model_calls == 1