feat-016: grounded documentation generation (ckg docs …)#158
Open
kjoshi07 wants to merge 11 commits into
Open
Conversation
Graduate FA-006 (grounded documentation generation) into a feat-016 spec targeting 0.7.0. Scope: full Phase 1 (all four descriptive doc types — ai-context, architecture, component, design) with grounding + citations + draft/promote gate + dirty-aware `ckg docs update` + opt-in `ckg docs sync` flywheel. ADR drafts (Phase 3) and CI-hardened flywheel (Phase 2) explicitly deferred. Add the tracker row (feat-016, accepted, deps 006/007/010/012/004) and a 0.7.0 milestone entry. Design doc (design-016) is next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
design-016: how docgen works — docs as a grounded projection of the graph. Approved decisions baked in: (1) writer is a multi-turn agentforge.Agent loop over a read-only, provenance-floored ckg toolset (grounding enforced at the tool boundary, not a fixed pack); (2) citations are a structured footnote block verified against the captured ProvenanceSet; (3) ai-context is draft-only to output_root; (4) a doc_lang_version stamps the template/citation contract. Reuse over reinvention: agentforge.Agent (wraps BudgetPolicy) + the feat-008 Tool instances + KnowledgeIngestor (feat-010) for opt-in sync + DirtySet "docs" consumer for staleness. No locked-kinds change (ADR-0005) — docs are files + a sidecar manifest. 8-chunk plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The LLM-free metadata core for grounded doc generation: - docgen/types.py — DocType, DocTarget, SymbolRef, GroundedFact, GroundedPack, ProvenanceSet (the citation trust boundary + staleness key), Footnote, DocArtifact; DOC_LANG_VERSION="1.0". - docgen/errors.py — DocgenError hierarchy (Ungrounded/BadCitation/ DocDisabled/PromoteRequired). - docgen/manifest.py — sidecar output_root/.ckg-docs.json (mirrors the .ckg/dirty.json pattern): typed round-trip, upsert/get/all/remove/ promote, content_sha drift key. `stale` is computed, never serialized. - docgen/staleness.py — pure join between a doc's source_ids and the DirtySet "docs" cursor; is_stale / stale_docs. - config DocGenConfig(_Block, KEY="docgen") — auto-discovered; validates component_granularity. - DirtySet: "docs" added to DEFAULT_CONSUMERS (generated docs go stale like summaries; indexing already fans changed symbols to it). Framework-adjacent but this chunk imports only core (no agentforge yet). Full gate: 1114 passed / 69 skipped / 94.43% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deterministic trust boundary — graph facts in, verified citations out, no LLM: - docgen/recipes/base.py — Recipe ABC (async seed -> GroundedPack) + RECIPES registry + register/get_recipe. Add a doc type = a subclass + one register call; nothing shared changes. - docgen/recipes/architecture.py — the architecture seed: top-PageRank symbols (feat-007) + routes/models topology (feat-011) as citable >=parsed facts; repo summary (llm) + DI services as non-citable notes (anti-echo-chamber in the seed). - docgen/citations.py — verify_citations: parse GFM footnote defs + per-section markers, prove every footnote resolves into the ProvenanceSet (BadCitationError on a fabricated/dangling cite), flag uncited sections (UngroundedError under require_citations), rewrite the References block to human-facing links. Pure text-in/text-out. Tests: 11 new (citation verifier over crafted markdown incl. bad/dangling/ ungrounded cases + refs-block isolation; architecture recipe over a real tiny Kuzu index — real symbol ids/paths, source=parsed, no llm leak). Full gate: 1124 passed / 69 skipped / 94.37% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The multi-turn agent loop, proven end to end and hermetic: - docgen/toolset.py — grounded_tools (the read-only feat-008 Tool instances, standalone) + capture_refs: scan tool-observation JSON for citable facts (id + kind + name + path + span), applying the >=parsed provenance floor (llm items dropped — anti-echo-chamber at the tool boundary, tool-agnostic, no second store handle). - docgen/templates/ — Template objects (section skeleton) + SYSTEM_PROMPT carrying the grounding/citation contract the verifier enforces. Architecture template. (Python objects, not .md.tmpl — no artifact packaging.) - docgen/runner.py — AgentDocRunner.compose: builds agentforge.Agent (strategy=react, tools, budget_usd, max_iterations) over the seed task, runs it, captures the ProvenanceSet (seed refs U tool-result refs), wraps budget/iteration trip (BudgetExceeded/GuardrailViolation -> DocgenError). agentforge imported lazily. First non-test framework-Agent consumer in the repo. - docgen/generator.py — DocGenerator.generate: seed -> compose -> verify citations -> write stamped draft under output_root -> manifest upsert. Hermetic CI via agentforge.testing.MockLLMClient (scripted tool-calls + final body, no creds). Config: docgen.provider defaults to a resolvable framework provider (anthropic), model_ref() builds the Agent model string. [BUG-CARRIED, same chunk] citation parser took split()[0] of a footnote def — but symbol ids are 5 SPACE-joined fields, so it grabbed only "ckg". Fixed: the whole remainder of a `[^fN]: <id>` line is the id. Tests: capture floor (llm dropped, malformed skipped, dedupe) + full generate via MockLLMClient (real Agent + real tools -> draft written, manifest recorded, links rewritten; fabricated citation -> BadCitationError; budget trip -> DocgenError). Full gate: 1132 passed / 69 skipped / 94.37% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pure additions behind the recipe/template seams — no shared-code edits:
- recipes/_common.py — shared graph-based, provenance-floored (>= parsed)
seed helpers: symbol_facts (structural symbols under a path scope via
GraphQuery min_source=PARSED) + decision_facts (ADRs) + node_ref.
- recipes/ai_context.py — orient an AI assistant: central symbols +
decisions; repo summary as framing.
- recipes/component.py — one module: structural symbols under scope +
routes/models rooted there; file summaries as framing.
- recipes/design.py — subsystem how+why: scoped symbols + governing
decisions (highest-synthesis type; leans on require_citations + promote).
- templates/{ai_context,component,design}.py — section skeletons; all four
registered.
Tests: seed grounding per type over a real index (ai-context central
symbols; component scoped to a path incl. empty-for-unknown-scope; design
scoped); all four registered in RECIPES + TEMPLATES; build_task embeds
seed ids/sections/non-citable notes; DocDisabled on an unregistered type.
Full gate: 1140 passed / 69 skipped / 94.39% cov; ruff + mypy strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The freshness + review surface on DocGenerator:
- update() — drain DirtySet("docs"), regenerate only the docs whose
source_ids intersect the dirty set (stale_docs), then mark the covered
symbols clean. Mirrors CodeGraph.summarize's drain->run->mark_clean
(feat-004 reuse); DirtySet rooted via resolve_root(repo, StoreConfig).
- list_docs() — every doc with `stale` computed (dirty source, moved index
commit, or bumped doc_lang_version).
- diff(path) — unified diff of the on-disk file vs a fresh regeneration
(what update would change); DocgenError on an unknown path.
- promote(path) — flip draft -> accepted (human gate).
- generate() refactored to share _compose() with diff.
- DocArtifact gains `scope` (persisted) so a doc can be regenerated with
its original target; manifest + promote carry it through.
Tests via a repeating _ConstLLM (MockLLMClient exhausts across the multiple
composes update/diff need): update regenerates only dirty docs and clears
the cursor; list flags stale after a source dirties; promote flips + errors
on unknown path; diff is empty for a fresh doc and surfaces a hand-edit.
Full gate: 1145 passed / 69 skipped / 94.40% cov; ruff + mypy strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- CodeGraph facade (framework layer, ADR-0001): docs_generate / docs_update / docs_list / docs_diff / docs_promote, each building a DocGenerator lazily from DocGenConfig; enabled gate on the write verbs; optional `model` seam (bring-your-own LLMClient) for hermetic tests + advanced callers. DocGenConfig gains `enabled`. - cli.py: `ckg docs` nested subparser (generate/update/list/diff/promote), modeled on `ckg ci`. generate takes --type/--all/--scope/--budget-usd; list --format table|json (reuses cli_format); diff/promote take a positional doc path (repo via --path). DocgenError -> stderr + exit 2. Tests: generate requires --type/--all; refuses when docgen.enabled:false; list surfaces a seeded doc (JSON); promote flips + shows accepted on relist; diff on an unknown doc exits 2. CLI tests are sync (main owns the loop) with async fixtures seeding via a const LLMClient. Full gate: 1150 passed / 69 skipped / 94.20% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`ckg docs sync` re-ingests ACCEPTED docs into the graph so retrieval gets richer — opt-in (docgen.round_trip) and gated on promotion: - DocGenerator.sync: per accepted doc, build a DocChunk node tagged Provenance.llm with DESCRIBES edges to its footnote symbols (already validated real symbols — the citation graph IS the linkage), upsert, and embed the text when embed is enabled. Refuses when round_trip is off (DocgenError) or docs are unpromoted (PromoteRequired). - Built in docgen (not KnowledgeIngestor.ingest, whose doc pass GCs repo-wide and would wipe real ingested docs) — a targeted, GC-free path. - THE anti-echo-chamber guarantee is provenance: synced docs are Source.LLM, and recipes/tools floor at >= parsed, so a synced doc can inform search but is never a citable ground-truth fact. - CodeGraph.docs_sync facade + `ckg docs sync` CLI (refuses read-only stores via _refuse_write_if_read_only — sync writes the graph). Tests: refuses round_trip off; refuses unpromoted; creates an llm-sourced DocChunk with DESCRIBES to the cited symbol AND a >=parsed floor excludes it (the guarantee, proven); embeds via the fake embedder without creds. Full gate: 1154 passed / 69 skipped / 94.09% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ckg_status reports docgen.{enabled, types, doc_lang_version} (parity with
the query surface), tested.
- Guide 14 (grounded doc generation) + guides index row.
- CHANGELOG [Unreleased]: feat-016 entry.
- feat-016 spec Implementation-status: the 8-chunk build, approved design
decisions, and documented deviations (capture from RunResult.steps not a
proxy; Python templates; sync builds DocChunk directly using footnotes for
DESCRIBES; provider defaults to anthropic; facade model seam; the
[BUG-CARRIED] space-in-symbol-id citation fix).
Full gate: 1154 passed / 69 skipped / 94.10% cov; ruff + mypy strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated end-to-end on this repo: indexed src/agentforge_graph (192 files, 2210 nodes) and generated its own architecture doc with a real Anthropic model — a genuine, useful, grounded doc with 28 real citations. Three refinements the hermetic tests couldn't surface: - Economy over exhaustion: the framework Agent caps total tokens (~200k, not constructor-overridable), and a long ReAct loop of large tool observations exhausts it on a big repo. max_iterations 24 -> 12 + the system prompt now tells the model to lean on the rich seed and make only a few targeted tool calls. - Citations pruned, not fatal; grounding is doc-level. A citation is valid only if used inline AND resolving into the provenance set; fabricated ids, dangling markers, and unused defs are pruned (marker stripped, def dropped) — never fatal (real models mis-cite occasionally). require_citations becomes a doc-level gate (>=1 valid citation, refuse pure hallucination); per-section gaps are REPORTED (ungrounded_sections) for the reviewer, not failed on. Integrity guarantee preserved (every PUBLISHED citation is real); the promote gate carries per-section adequacy. The strict per-section rule in design-016 proved impractical live. - Preamble stripped: text before the first heading (model meta-commentary like "Let me compile the document:") is dropped. feat-016 spec Implementation-status updated with the calibrations + a follow-up (docgen-specific tool response_token_cap to extend the loop budget on large repos). Full gate: 1156 passed / 69 skipped / 94.09% cov; ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Graduates FA-006 into feat-016 (target 0.7.0): documentation as a cited projection of the graph.
What ships (Phase 1)
ckg docs generate/update/list/diff/promote/syncfor the four descriptive doc types — ai-context (CLAUDE.md/AGENTS.md), architecture, component, design. A realagentforge.Agentcomposes each doc while pulling facts from the read-only ckg tools; every published claim is attributed to a real symbol, verified against the run's provenance set (the tool boundary).docgen.output_rootand never overwrite human docs; a doc is a draft untilckg docs promote(the human gate). Citations are verified — fabricated/dangling ones are pruned; grounding is enforced doc-level (require_citations) with per-section gaps reported for review.ckg docs updateregenerates only docs whose source symbols changed — reuses the feat-004DirtySet.ckg docs sync(default off) re-ingests accepted docs asllm-sourcedDocChunknodes thatDESCRIBEStheir cited symbols. Anti-echo-chamber by provenance — recipes/tools ground at≥ parsed, so a synced doc can inform search but is never a citable ground-truth fact.docgen:config block;ckg_statusreportsdocgen.{enabled, types, doc_lang_version}. Guide 14.Design
design-016(accepted). Writer is a multi-turnagentforge.Agentloop over a read-only, provenance-floored toolset — grounding at the tool boundary, not a fixed pack. Reuses the framework Agent +BudgetPolicy+ the feat-008Toolinstances; no new locked kinds (ADR-0005). Built in 8 chunks + a live-dogfood calibration commit.Validation
agentforge.testing.MockLLMClientdrives the real Agent loop over the real tools — generation, capture, citation verification, update, sync, CLI — all creds-free.max_iterations24→12 + prompt guidance, vs the framework Agent's ~200k token cap); citations pruned-not-fatal with doc-level grounding; preamble stripping.Gate
1156 passed / 69 skipped / 94.09% cov; ruff + mypy strict clean throughout (per chunk).
🤖 Generated with Claude Code