Skip to content

fix(sdd): stop the planning artifact growing with steps × spec-size - #99

Open
grushikhin wants to merge 1 commit into
masterfrom
fix/sdd-planning-artifact-bloat
Open

fix(sdd): stop the planning artifact growing with steps × spec-size#99
grushikhin wants to merge 1 commit into
masterfrom
fix/sdd-planning-artifact-bloat

Conversation

@grushikhin

Copy link
Copy Markdown

What prompted this

Running /sdd:plan-task with default settings (plus one --human-in-the-loop gate) on a real migration produced an 8282-line task file. The input spec was 3758 lines; the pipeline added 4524.

Measured, not estimated:

Section Lines Share
## Implementation Process 6374 77%
→ of which #### Verification blocks 4509 54%
→ actual step bodies 1865 23%
## Acceptance Criteria 816 10%
## Architecture 699 8%

Step bodies are fine — 77 lines per step. The multiplier is steps × per-step spec-size, and two blocks inside every spec were byte-identical across all 19 steps:

Subsection Lines Occurrences
##### Project Guidelines Alignment 342 19 of 19
**Regular Checks:** 260 19 of 19

That is 602 lines of pure duplication, against CONTRIBUTING.md's own principle — "Minimal token footprint — every token counts", "Prompts should be concise".

Changes

# Change File
1 qa-engineer writes the spec to a sidecar <task-basename>.verification.md, one ### Step N section per step. Task file gets a one-line **Verification:** pointer per step plus the existing Verification Summary table. agents/qa-engineer.md
2 Regular Checks and Project Guidelines Alignment emitted once in the sidecar preamble; steps reference them by name and list exceptions only. agents/qa-engineer.md
3 implement-task resolves the spec location — sidecar when present, inline #### Verification otherwise. skills/implement-task/SKILL.md
4 Verification level None made reachable; Panel narrowed. agents/qa-engineer.md
5 tech-lead bounds step count, not only step size. agents/tech-lead.md
6 --fast no longer includes verifications. skills/plan-task/SKILL.md

Why a sidecar

The verification spec has exactly one consumer, reading exactly one step: a judge scoring that step. The task file is read end-to-end by humans and by every other agent in the pipeline. Inlining made the plan carry 4509 lines that no reader of the plan needs. The judge's input is unchanged — it is told the sidecar path and the ### Step N heading, so it now loads ~190 lines instead of scanning 8282.

Why None and Panel were recalibrated

None was reachable only for "mkdir, delete, JSON update", so effectively every real step received a full spec plus an agent pair. It now also applies when the step's outcome is fully decided by a deterministic gate — if the step is "add a third tsc invocation to the typecheck script" and CI failing is the whole test, a judge only restates the gate's verdict. The guard against over-using it is stated explicitly: could this step pass every gate and still be wrong? If yes, a judge is warranted. Panel now additionally requires a failure mode no deterministic gate can catch.

Why step count needed a bound

tech-lead already bounds step size ("no step larger than Large") and forbids trivial steps — but nothing bounded count, so a ~40-file migration decomposed into 19 steps, each carrying an impl+judge pair. Adds per-effort target bands with a hard ceiling, and names the diagnostic symptom of over-splitting: ordering invariants that exist only because a slice was split ("repoint the import in step 4 before step 6 deletes the module"). The run that prompted this generated five such invariants plus two pages defending them; merging the steps deletes them for free.

Why --fast was a trap

Its alias included the verifications stage, so it reduced judge rounds while leaving output size untouched — the opposite of what a user reaching for --fast expects. Removed from the alias, opt in explicitly. --one-shot already excluded it and was the only flag that actually shrank the artifact.

Backward compatibility

No migration needed. implement-task applies a resolution rule before Phase 2:

Condition Where the spec is read from
Sidecar exists Its ### Step N section + shared preamble
Sidecar absent, task file has inline #### Verification Those inline sections
Neither Every step treated as level None, stated in the summary

Task files already written by qa-engineer 3.1.2 keep working untouched; the sidecar wins when both are present.

Expected effect

On the task that prompted this: 8282 → ~3800 lines in the task file, with the verification spec intact in a sidecar of comparable size to what it replaced minus the 602 duplicated lines. Combined with the step-count bound, that same task would plan as ~8–12 steps rather than 19, cutting agent pairs proportionally.

Not included

Trimming the per-step spec itself — 5 score anchors to 3, and dropping Test Cases to Cover in favour of test_matrix + coverage_map. Both are defensible cuts (~another 1600 lines on the measured task) but they change what a judge evaluates, not merely where it lives, so they belong in a separate change with its own review.

Verification

  • just set-version sdd 3.2.0 and just set-marketplace-version 3.6.0 per CLAUDE.md (not edited by hand).
  • Docs updated: docs/plugins/sdd/{plan-task,implement-task,usage-examples}.md. just sync-* not run — those targets copy only README.md, which needed no change.
  • Repo-wide grep for #### Verification re-checked; remaining hits are the Verification Level Determination heading and the intentional backward-compatibility references.

⚠️ Not yet exercised end-to-end with plugins/customaize-agent:test-prompt — worth a run before merge, since changes 1–3 form a contract between two skills.

🤖 Generated with Claude Code

A real 19-step task planned with default settings produced an 8282-line task
file. Measured composition: 6374 lines in Implementation Process, of which 4509
were `#### Verification` blocks against 1865 lines of actual step content. 54% of
the artifact was material no reader of the plan needs, and two blocks were
byte-identical across all 19 steps (Project Guidelines Alignment 342 lines,
Regular Checks 260 lines). This contradicts the marketplace's own stated
principle that every token counts.

Five changes:

- qa-engineer writes the verification spec to a sidecar
  `<task-basename>.verification.md` with one `### Step N` section per step; the
  task file gets a one-line `**Verification:**` pointer plus the existing
  Verification Summary table. A judge scores one step, so it never needed the
  whole file inline.
- qa-engineer emits Regular Checks and Project Guidelines Alignment once, in the
  sidecar preamble; steps reference them by name and list exceptions only.
- implement-task resolves the spec location: sidecar when present, inline
  `#### Verification` otherwise, so task files written by the previous version
  keep working unchanged.
- Verification level `None` was reachable only for mkdir/delete/config, so every
  real step got a full spec. It now also covers steps whose outcome is fully
  decided by a deterministic gate, and Panel is reserved for HIGH criticality
  with a failure mode no gate can catch.
- tech-lead bounds step *count*, not only step size. Only size was bounded, so a
  ~40-file migration decomposed into 19 steps, each carrying an agent pair. Adds
  per-effort target bands with a hard ceiling, and flags ordering invariants that
  exist solely because one slice was split.

Also: `--fast` included the `verifications` stage, so it cut judge rounds while
leaving the artifact just as large. Removed from the alias; opt in explicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
### Phase 6: Define Verifications

`qa-engineer` agent (opus) adds LLM-as-Judge verification sections with custom rubrics, thresholds, and verification levels (None, Single Judge, Panel of 2, or Per-Item) for each implementation step.
`qa-engineer` agent (opus) writes LLM-as-Judge verification specs with custom rubrics, thresholds, and verification levels (None, Single Judge, Panel of 2, or Per-Item) for each implementation step. The specs go into a sidecar file next to the task — `<task-basename>.verification.md`, one `### Step N` section each — and the task file gets a one-line `**Verification:**` pointer per step plus the `## Verification Summary` table.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest another way to make it. I have 2 options in mind:

  • Remove from main task file all steps description, and create .spec/subtasks/*.md files. Each per task. Then main task will contain only checeklist for whole task verification. Majority of tasks "steps" is hard to verify properly, they may not yet contain tests written. So probably verification better to place not steps level, but at overral task level.
  • Alternatively, I thinking about combining all plan agents to a single one, that will perform majority of planing, and produce much shorter task spec. -> maybe there meaning to make something like that just in SADD plugin and remove SDD at all. Something like /plan-do-judge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants