Ahe explore agentic lift

[Enforcer03]

Summary by cubic

Adds pluggable lift strategies (heuristic, llm_assisted, agentic), durable guard-verdict logging for the evolve loop, and a new hdp.redteam suite to generate and replay adversarial edits against the guard.

• New Features

  • Lift strategies: hdp.engine.lift.lift(..., strategy='heuristic'|'llm_assisted'|'agentic', llm=...). Default remains heuristic. CLI accepts --strategy.
  • Uncertainty sentinel: hdp.engine.adapters.uncertain.Uncertain marks fields (e.g., blast_radius) for LLM refinement without changing heuristic output. Adapters updated to emit/resolve it.
  • Agentic lift: backend-agnostic mechanics in hdp.engine.adapters.agentic let an LLM read the repo and propose the HDP document.
  • Evolve loop logging: evolve.log plus JSONL guard-audit per decision; concise per-iteration summaries with tqdm postfix. Optional warn-and-retry on denied edits.
  • Red team: hdp.redteam proposes candidate edits via an injected llm, replays them across guard engines/modes, measures outcomes by tier, and renders Markdown reports. Includes code-aware triage and evolve-run replay.
  • Examples and tooling: demo notebook, experiment runner script, and comprehensive $0 tests for new strategies, logging, and red-team flows.

• Migration

  • No breaking changes. lift() defaults to heuristic and matches prior output.
  • To use LLM paths, pass strategy and an llm callable to lift(). The CLI accepts the flag but requires supplying llm from Python.
  • Evolve runs now write human-readable logs and a JSONL audit in the run workdir. For red-team evolve-run replay, run with guard interaction set to monitor to capture post.hdp.

^{Written for commit 9651490. Summary will update on new commits.}

[cubic-dev-ai]

27 issues found across 26 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="hdp/engine/lift/__init__.py">

<violation number="1" location="hdp/engine/lift/__init__.py:35">
P2: Existing positional `target` callers now fail with `TypeError`; retain `target` as positional-or-keyword and make only new strategy arguments keyword-only.</violation>
</file>

<file name="hdp/engine/adapters/agentic.py">

<violation number="1" location="hdp/engine/adapters/agentic.py:28">
P1: Agentic lift sends a harness `.env` (including API keys) to the configured LLM because hidden files have an empty suffix and are not excluded. Excluding hidden files before reading prevents documented local credentials from leaving the repository.</violation>
</file>

<file name="hdp/redteam/report.py">

<violation number="1" location="hdp/redteam/report.py:154">
P2: LLM-supplied candidate text can inject new rows/headings into the Markdown report, making guard results misleading; escape or normalize Markdown control characters before rendering candidate fields.</violation>
</file>

<file name="hdp/engine/adapters/nexau.py">

<violation number="1" location="hdp/engine/adapters/nexau.py:439">
P2: In llm_assisted mode a tool's Uncertain blast_radius can be refined by the LLM to a value higher than the governance ceiling, leaving the lifted document internally inconsistent. `max_blast`/`governance.blast_radius` and `audit.log_all_tool_calls` are computed earlier via `resolved(blast)` (the conservative default, usually "session"), and `_llm_refine` only rewrites the per-component `blast_radius` sentinel; the ceiling/audit fields are never recomputed afterward. So a tool whose blast_radius the LLM resolves to "system"/"external" ends up sitting under a "session" governance ceiling with `log_all_tool_calls: false` — undermining the safety invariant that governance.blast_radius is an upper bound and that audit logging tracks the effective scope. The same pattern is present in openharness.py and mini_swe_agent.py. Consider recomputing the ceiling/audit from the post-refinement tool blast radii (or capping each refined per-tool blast_radius at the governance ceiling) so heuristic and llm_assisted outputs stay mutually consistent.</violation>

<violation number="2" location="hdp/engine/adapters/nexau.py:465">
P1: Agentic lift accepts manifests with embedded `file` components but writes none of those files, so a schema-valid result cannot be generated into a NexAU harness (`FileNotFoundError` on `read_embedded`). Materialize validated relative embedded files from `harness_dir` into `out_dir`, or reject proposals that reference files not created in the output.</violation>
</file>

<file name="hdp/engine/adapters/openharness.py">

<violation number="1" location="hdp/engine/adapters/openharness.py:482">
P1: LLM-assisted lift can emit a tool above `governance.blast_radius`, because the ceiling is calculated before this assignment using the sentinel default. Recompute the ceiling/audit after refinement (or constrain accepted answers) so output satisfies the guard's component ceiling invariant.</violation>

<violation number="2" location="hdp/engine/adapters/openharness.py:503">
P1: Agentic lift can disclose `.env` secrets to the LLM because broad repository collection includes dotfiles with an empty suffix. Exclude sensitive dotfiles/credential paths before assembling the prompt.</violation>

<violation number="3" location="hdp/engine/adapters/openharness.py:506">
P2: Agentic lift does not enforce its target or supplied metadata, so valid LLM output can be written with a different backend or ignored overrides. Validate these values after parsing and fail loud when they differ from `self.target`, requested id/version/base model.</violation>
</file>

<file name="hdp/engine/tests/test_loop.py">

<violation number="1" location="hdp/engine/tests/test_loop.py:92">
P3: This new module-level `_illegal_proposer` duplicates the existing inline `illegal_proposer` in `test_loop_guard_rolls_back_illegal_edit` byte-for-byte (module-level copy only adds type annotations). Both tamper `verification[0]` and declare the identical `chg-bad`/`tb2-verifier` change, so any future change to the tamper target or declared change (e.g. the guard deny reason) must be made in two places and they can drift. Consider promoting the existing local helper to module level and reusing it, or have the new helper accept the tamper component as a parameter.</violation>
</file>

<file name="hdp/redteam/llm_gen.py">

<violation number="1" location="hdp/redteam/llm_gen.py:173">
P1: A candidate containing an absolute path or `../../` can delete or overwrite files outside the temporary replay copy before guard evaluation; validate `remove_files` and `embedded` paths resolve beneath `dest` before applying either operation.</violation>

<violation number="2" location="hdp/redteam/llm_gen.py:234">
P2: Each candidate is re-materialized (full `copytree` + doctoring) once per (engine, mode) pair — up to 4 identical copies for every candidate — and the reconcile engine ignores `mode`, so the reconcile 'enforce' and 'review' iterations are redundant with identical outputs. For the malformed-candidate path this also emits four identical failure rows. Materializing once per candidate and branching per engine (only the atomic path needs a fresh copy, since it replaces the tree) would avoid the redundant work.</violation>
</file>

<file name="hdp/redteam/regen.py">

<violation number="1" location="hdp/redteam/regen.py:47">
P3: The live HTTP call has no timeout, so a stalled or black-holed connection to the LLM endpoint blocks `python -m hdp.redteam.regen` forever instead of failing loudly as the module's own docstring promises. Pass an explicit `timeout=` to `urlopen` (and ideally a clear retry/error path) so a hung network call surfaces a message rather than a silent hang.</violation>

<violation number="2" location="hdp/redteam/regen.py:50">
P3: When the model refuses (message content is `null`, e.g. a safety filter), `str(None)` produces the string "None" that flows into `propose_candidates`, surfacing as a confusing "LLM reply contained no JSON object" error rather than a clear "model returned no content" message. Guard the content before returning so the failure reason is accurate and actionable.</violation>
</file>

<file name="hdp/engine/adapters/mini_swe_agent.py">

<violation number="1" location="hdp/engine/adapters/mini_swe_agent.py:434">
P1: Agentic lift can return a schema-valid HDP document whose file-backed components were never written, so generating its harness later raises `FileNotFoundError`; materialize approved component files from the source harness (or reject dangling `file` references) before returning.</violation>
</file>

<file name="hdp/redteam/triage.py">

<violation number="1" location="hdp/redteam/triage.py:181">
P2: Path-backed executable refs are never resolved, so valid components can be shortlisted but reviewed without their actual implementation. Read `ref.path` relative to the HDP directory before falling back to dotted bindings.</violation>
</file>

<file name="hdp/engine/loop.py">

<violation number="1" location="hdp/engine/loop.py:180">
P2: A new, nonexistent `log_dir` aborts `evolve()` before its first iteration because neither log directory is created. Create `log_dir` before initializing the logger so the optional output location works like `workdir`.</violation>

<violation number="2" location="hdp/engine/loop.py:249">
P2: Under `warn_and_retry`, denied attempts that eventually recover are recorded in guard_audit.jsonl (every attempt is audited) but are never reflected in the loop's own guard-verdict telemetry: `run.log('guard_denied')`, the `cum_denied` progress-bar postfix, and `IterationResult.guard_denied` all read `len(report.denied)` from only the final attempt. So when a retry succeeds, the iteration reports 0 denials even though denials happened, making the audit and the loop counters disagree. Consider accumulating the denied count across all attempts inside the retry loop (or intentionally tracking final-vs-cumulative separately) so the persisted telemetry and round metrics match the audit trail.</violation>
</file>

<file name="hdp/examples/hdp_demo.ipynb">

<violation number="1" location="hdp/examples/hdp_demo.ipynb:36">
P3: The first code cell was committed with its failure output persisted: the saved execution result is an `AssertionError` ("Run this notebook from the repo root.") with a full traceback. Anyone opening this demo notebook sees the first cell marked as errored, which undercuts the demo's purpose (it also suggests the notebook was saved from a non-root working directory). Clear the cell output and re-run/commit the notebook from the repo root so the saved state is green.</violation>

<violation number="2" location="hdp/examples/hdp_demo.ipynb:111">
P2: The claimed round-trip validity gate reports success even after dropped, added, or modified files because it never validates those conditions; make the cell assert the same identity contract as the test, including semantic manifest equality.</violation>

<violation number="3" location="hdp/examples/hdp_demo.ipynb:132">
P1: Cell 4 (Section 4 · Port) imports `from hdp.engine.port import audit, port, PortCoverageError`, but no `hdp/engine/port` module exists anywhere in the repository (verified: `find hdp -iname '*port*'` returns nothing and `rg 'PortCoverageError|def audit'` matches nothing). Running this cell crashes with `ModuleNotFoundError`, so the notebook's port demonstration — the centerpiece of Section 4 — can never execute. Either ship the `hdp.engine.port` module (audit/port/PortCoverageError) as part of this PR, or remove/replace this cell until the feature exists.</violation>
</file>

<file name="hdp/redteam/evolve_replay.py">

<violation number="1" location="hdp/redteam/evolve_replay.py:64">
P1: Valid runs using `evolution.manifest_dir` are measured as missing-manifest denials because replay ignores the configured directory. Resolve the directory from `doc.hdp` like `track._manifest_dir` before loading each iteration manifest.</violation>
</file>

<file name="scripts/run_experiments.sh">

<violation number="1" location="scripts/run_experiments.sh:73">
P3: `--proportion` without a value exits with an unbound-variable shell error; validate that an argument follows the option and show usage.</violation>

<violation number="2" location="scripts/run_experiments.sh:86">
P2: Small positive proportions can run only reconcile and produce no A/B comparison because `ceil(P*N)` can be odd; round selected count up to a complete pair before iterating.</violation>
</file>

<file name="hdp/engine/tests/test_lift_strategies.py">

<violation number="1" location="hdp/engine/tests/test_lift_strategies.py:36">
P3: This regression guard can't actually guard a regression: both compared documents are produced by the same new code path (default strategy IS 'heuristic', and lift() uses the same `if strategy == "heuristic"` branch), so they are byte-identical by construction no matter what the sentinel machinery does. To truly pin the deterministic mapping, compare against a committed golden hdp.yaml fixture (or assert the concrete expected field values a snapshot of the old output produced) rather than a second lift of the same seed.</violation>
</file>

<file name="hdp/redteam/tests/test_redteam_replay.py">

<violation number="1" location="hdp/redteam/tests/test_redteam_replay.py:1">
P3: The module docstring starts with a literal `$0` — a stray shell-template artifact. Remove it so the docstring reads cleanly.</violation>

<violation number="2" location="hdp/redteam/tests/test_redteam_replay.py:80">
P2: The blanket `tree_unchanged is True` check over all atomic denied outcomes will break if any fixture candidate fails to load: `run_candidates`'s load-failure branch already emits a denied (`tier="structural"`) atomic outcome whose `tree_unchanged` is `_tree_hash(new_dir) == old_hash`, which is False whenever the materialized edit altered the tree before failing validation. That row is not a real atomic tree-restore, so the invariant assertion is unsound here; filter it out or drop it.</violation>

<violation number="3" location="hdp/redteam/tests/test_redteam_replay.py:122">
P3: `assert len(cands) >= 20` hard-codes a minimum that nothing in the pipeline guarantees: `propose_candidates` (and regen.py) return exactly whatever the LLM returns, with no count enforcement, so a freshly regenerated fixture with fewer than 20 candidates would make this test fail. Loosen the assertion or enforce the count at generation time.</violation>
</file>

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Agentic lift sends a harness .env (including API keys) to the configured LLM because hidden files have an empty suffix and are not excluded. Excluding hidden files before reading prevents documented local credentials from leaving the repository.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/engine/adapters/agentic.py, line 28:

<comment>Agentic lift sends a harness `.env` (including API keys) to the configured LLM because hidden files have an empty suffix and are not excluded. Excluding hidden files before reading prevents documented local credentials from leaving the repository.</comment>

<file context>
@@ -0,0 +1,79 @@
+    for path in sorted(harness_dir.rglob("*")):
+        if len(out) >= max_files:
+            break
+        if not path.is_file() or path.suffix.lower() not in _TEXT_SUFFIXES:
+            continue
+        if any(part in _SKIP_DIRS for part in path.relative_to(harness_dir).parts):
</file context>

[cubic-dev-ai]

P1: Agentic lift accepts manifests with embedded file components but writes none of those files, so a schema-valid result cannot be generated into a NexAU harness (FileNotFoundError on read_embedded). Materialize validated relative embedded files from harness_dir into out_dir, or reject proposals that reference files not created in the output.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/engine/adapters/nexau.py, line 465:

<comment>Agentic lift accepts manifests with embedded `file` components but writes none of those files, so a schema-valid result cannot be generated into a NexAU harness (`FileNotFoundError` on `read_embedded`). Materialize validated relative embedded files from `harness_dir` into `out_dir`, or reject proposals that reference files not created in the output.</comment>

<file context>
@@ -406,7 +429,38 @@ def lift(
+
+        HDPManifest.model_validate(doc_dict)  # raise on an invalid agentic proposal
+        out_dir.mkdir(parents=True, exist_ok=True)
+        _write_yaml(out_dir / "hdp.yaml", doc_dict)
+        return load(out_dir)
</file context>

[cubic-dev-ai]

P1: LLM-assisted lift can emit a tool above governance.blast_radius, because the ceiling is calculated before this assignment using the sentinel default. Recompute the ceiling/audit after refinement (or constrain accepted answers) so output satisfies the guard's component ceiling invariant.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/engine/adapters/openharness.py, line 482:

<comment>LLM-assisted lift can emit a tool above `governance.blast_radius`, because the ceiling is calculated before this assignment using the sentinel default. Recompute the ceiling/audit after refinement (or constrain accepted answers) so output satisfies the guard's component ceiling invariant.</comment>

<file context>
@@ -456,6 +472,38 @@ def _enumerate_tools(permission: dict) -> list[tuple[str, bool | None, str | Non
+        for container, key, unc in list(iter_uncertain(doc_dict)):
+            answer = (llm(refine_prompt(str(key), unc)) or "").strip()
+            if answer and (not unc.candidates or answer in unc.candidates):
+                container[key] = answer
         return doc_dict
+
</file context>

[cubic-dev-ai]

P1: Agentic lift can disclose .env secrets to the LLM because broad repository collection includes dotfiles with an empty suffix. Exclude sensitive dotfiles/credential paths before assembling the prompt.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/engine/adapters/openharness.py, line 503:

<comment>Agentic lift can disclose `.env` secrets to the LLM because broad repository collection includes dotfiles with an empty suffix. Exclude sensitive dotfiles/credential paths before assembling the prompt.</comment>

<file context>
@@ -456,6 +472,38 @@ def _enumerate_tools(permission: dict) -> list[tuple[str, bool | None, str | Non
+
+        harness_dir = Path(harness_dir).resolve()
+        out_dir = Path(out_dir).resolve()
+        prompt = build_agentic_prompt(harness_dir, self.target, meta_id, version, base_model)
+        doc_dict = parse_agentic_doc(llm(prompt))
+
</file context>

[cubic-dev-ai]

P1: A candidate containing an absolute path or ../../ can delete or overwrite files outside the temporary replay copy before guard evaluation; validate remove_files and embedded paths resolve beneath dest before applying either operation.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/redteam/llm_gen.py, line 173:

<comment>A candidate containing an absolute path or `../../` can delete or overwrite files outside the temporary replay copy before guard evaluation; validate `remove_files` and `embedded` paths resolve beneath `dest` before applying either operation.</comment>

<file context>
@@ -0,0 +1,258 @@
+        if idx >= 0:
+            del seq[idx]
+        for rel in candidate.remove_files or []:
+            fp = dest / rel
+            if fp.is_file():
+                fp.unlink()
</file context>

[cubic-dev-ai]

P3: --proportion without a value exits with an unbound-variable shell error; validate that an argument follows the option and show usage.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/run_experiments.sh, line 73:

<comment>`--proportion` without a value exits with an unbound-variable shell error; validate that an argument follows the option and show usage.</comment>

<file context>
@@ -0,0 +1,184 @@
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --proportion) PROPORTION="$2"; shift 2 ;;
+        --dry-run)    DRY_RUN="--dry-run"; shift ;;
+        --stop-on-error) STOP_ON_ERROR=true; shift ;;
</file context>

[cubic-dev-ai]

P3: This regression guard can't actually guard a regression: both compared documents are produced by the same new code path (default strategy IS 'heuristic', and lift() uses the same if strategy == "heuristic" branch), so they are byte-identical by construction no matter what the sentinel machinery does. To truly pin the deterministic mapping, compare against a committed golden hdp.yaml fixture (or assert the concrete expected field values a snapshot of the old output produced) rather than a second lift of the same seed.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/engine/tests/test_lift_strategies.py, line 36:

<comment>This regression guard can't actually guard a regression: both compared documents are produced by the same new code path (default strategy IS 'heuristic', and lift() uses the same `if strategy == "heuristic"` branch), so they are byte-identical by construction no matter what the sentinel machinery does. To truly pin the deterministic mapping, compare against a committed golden hdp.yaml fixture (or assert the concrete expected field values a snapshot of the old output produced) rather than a second lift of the same seed.</comment>

<file context>
@@ -0,0 +1,191 @@
+    default_doc = lift(SEED, tmp_path / "default.hdp")                       # no strategy arg
+    heuristic_doc = lift(SEED, tmp_path / "heuristic.hdp", strategy="heuristic")
+
+    assert (tmp_path / "default.hdp" / "hdp.yaml").read_bytes() == \
+           (tmp_path / "heuristic.hdp" / "hdp.yaml").read_bytes()
+    assert {c.id for _l, c in default_doc.components()} == \
</file context>

[cubic-dev-ai]

P3: assert len(cands) >= 20 hard-codes a minimum that nothing in the pipeline guarantees: propose_candidates (and regen.py) return exactly whatever the LLM returns, with no count enforcement, so a freshly regenerated fixture with fewer than 20 candidates would make this test fail. Loosen the assertion or enforce the count at generation time.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/redteam/tests/test_redteam_replay.py, line 122:

<comment>`assert len(cands) >= 20` hard-codes a minimum that nothing in the pipeline guarantees: `propose_candidates` (and regen.py) return exactly whatever the LLM returns, with no count enforcement, so a freshly regenerated fixture with fewer than 20 candidates would make this test fail. Loosen the assertion or enforce the count at generation time.</comment>

<file context>
@@ -0,0 +1,142 @@
+        pytest.skip("no cached fixture; run `python -m hdp.redteam.regen` to generate one")
+    old = _doc(tmp_path)
+    cands = [EditCandidate.from_dict(c) for c in json.loads(fx.read_text(encoding="utf-8"))]
+    assert len(cands) >= 20
+    report = run_candidates(old, cands)
+    for o in report.outcomes:
</file context>

[cubic-dev-ai]

P3: The module docstring starts with a literal $0 — a stray shell-template artifact. Remove it so the docstring reads cleanly.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/redteam/tests/test_redteam_replay.py, line 1:

<comment>The module docstring starts with a literal `$0` — a stray shell-template artifact. Remove it so the docstring reads cleanly.</comment>

<file context>
@@ -0,0 +1,142 @@
+"""$0, CI-safe tests for hdp.redteam — no live LLM call in any path.
+
+Covers: inline replay + tree-hash invariant on denied atomic outcomes, propose_candidates with
</file context>

[cubic-dev-ai]

P3: The first code cell was committed with its failure output persisted: the saved execution result is an AssertionError ("Run this notebook from the repo root.") with a full traceback. Anyone opening this demo notebook sees the first cell marked as errored, which undercuts the demo's purpose (it also suggests the notebook was saved from a non-root working directory). Clear the cell output and re-run/commit the notebook from the repo root so the saved state is green.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At hdp/examples/hdp_demo.ipynb, line 36:

<comment>The first code cell was committed with its failure output persisted: the saved execution result is an `AssertionError` ("Run this notebook from the repo root.") with a full traceback. Anyone opening this demo notebook sees the first cell marked as errored, which undercuts the demo's purpose (it also suggests the notebook was saved from a non-root working directory). Clear the cell output and re-run/commit the notebook from the repo root so the saved state is green.</comment>

<file context>
@@ -0,0 +1,210 @@
+   "outputs": [
+    {
+     "ename": "AssertionError",
+     "evalue": "Run this notebook from the repo root.",
+     "output_type": "error",
+     "traceback": [
</file context>