Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 158 additions & 0 deletions .github/workflows/e2e-cli-tier2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
name: E2E CLI Tests (Tier 2)

# Tier 2 runtimes are best-effort (see README "Supported surface"). They are not
# a PR gate; this schedule exists so they cannot rot silently between releases.
on:
schedule:
# Mondays 06:00 UTC
- cron: "0 6 * * 1"
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1

jobs:
e2e-cli-tier2:
name: E2E CLI Tier 2 - ${{ matrix.cli }} (${{ matrix.os }})
runs-on: ${{ matrix.os }}
environment: e2e-cli
strategy:
fail-fast: false
matrix:
cli: [gemini, copilot]
os: [ubuntu-24.04, macos-latest]

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install system dependencies (Linux)
if: runner.os == 'Linux'
run: |
sudo apt-get update
sudo apt-get install -y protobuf-compiler libclang-dev

- name: Install system dependencies (macOS)
if: runner.os == 'macOS'
run: |
brew install protobuf llvm
echo "LIBCLANG_PATH=$(brew --prefix llvm)/lib" >> $GITHUB_ENV

- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
toolchain: "1.97"

- name: Cache cargo registry
uses: Swatinem/rust-cache@v2
with:
shared-key: "e2e-cli-${{ matrix.os }}"

- name: Build daemon and ingest binaries
run: cargo build -p memory-daemon -p memory-ingest

- name: Install bats-core (Linux)
if: runner.os == 'Linux'
run: |
sudo apt-get install -y bats

- name: Install bats-core (macOS)
if: runner.os == 'macOS'
run: |
brew install bats-core

- name: Install bats helper libraries
run: |
mkdir -p tests/cli/lib
git clone --depth 1 https://github.com/bats-core/bats-support.git tests/cli/lib/bats-support
git clone --depth 1 https://github.com/bats-core/bats-assert.git tests/cli/lib/bats-assert

- name: Verify jq is available
run: jq --version

- name: Run bats tests
id: bats_run
continue-on-error: true
env:
BATS_LIB_PATH: tests/cli/lib
MEMORY_DAEMON_BIN: target/debug/memory-daemon
MEMORY_INGEST_BIN: target/debug/memory-ingest
run: |
mkdir -p tests/cli/.runs
if [ -d "tests/cli/${{ matrix.cli }}" ]; then
bats --report-formatter junit --output tests/cli/.runs tests/cli/${{ matrix.cli }}/ 2>&1 | tee e2e-cli-results.txt
else
echo "No tests found for ${{ matrix.cli }} — skipping"
echo "::notice::No bats tests found for ${{ matrix.cli }}, skipping"
exit 0
fi

- name: Upload JUnit XML report
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-tier2-${{ matrix.cli }}-${{ matrix.os }}
path: tests/cli/.runs/report.xml
if-no-files-found: ignore
retention-days: 14

- name: Upload failure artifacts
if: failure() || steps.bats_run.outcome == 'failure'
uses: actions/upload-artifact@v4
with:
name: failure-artifacts-tier2-${{ matrix.cli }}-${{ matrix.os }}
path: |
tests/cli/.runs/
e2e-cli-results.txt
if-no-files-found: ignore
retention-days: 7

- name: Report summary
if: always()
run: |
echo "## E2E CLI Results (Tier 2): ${{ matrix.cli }} (${{ matrix.os }})" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f e2e-cli-results.txt ]; then
echo '```' >> $GITHUB_STEP_SUMMARY
tail -20 e2e-cli-results.txt >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
else
echo "No test results file found." >> $GITHUB_STEP_SUMMARY
fi

- name: Check bats test result
if: always() && steps.bats_run.outcome == 'failure'
run: |
echo "Bats tests failed for ${{ matrix.cli }}"
exit 1

matrix-report:
name: CLI Matrix Report (Tier 2)
needs: [e2e-cli-tier2]
if: always()
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Download all JUnit artifacts
uses: actions/download-artifact@v4
with:
path: junit-reports
pattern: junit-tier2-*
merge-multiple: false

- name: Generate matrix report
run: |
chmod +x scripts/cli-matrix-report.sh
scripts/cli-matrix-report.sh junit-reports "gemini copilot" >> $GITHUB_STEP_SUMMARY

- name: Upload matrix report
if: always()
uses: actions/upload-artifact@v4
with:
name: cli-matrix-report-tier2
path: junit-reports/
if-no-files-found: ignore
retention-days: 14
13 changes: 8 additions & 5 deletions .github/workflows/e2e-cli.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,27 @@
name: E2E CLI Tests
name: E2E CLI Tests (Tier 1)

# Tier 1 runtimes are the supported surface (see README "Supported surface").
# They gate every PR. Tier 2 runtimes run on a schedule in e2e-cli-tier2.yml.
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1

jobs:
e2e-cli:
name: E2E CLI - ${{ matrix.cli }} (${{ matrix.os }})
name: E2E CLI Tier 1 - ${{ matrix.cli }} (${{ matrix.os }})
runs-on: ${{ matrix.os }}
environment: e2e-cli
strategy:
fail-fast: false
matrix:
cli: [claude-code, gemini, opencode, copilot, codex]
cli: [claude-code, codex]
os: [ubuntu-24.04, macos-latest]

steps:
Expand Down Expand Up @@ -109,7 +112,7 @@ jobs:
- name: Report summary
if: always()
run: |
echo "## E2E CLI Results: ${{ matrix.cli }} (${{ matrix.os }})" >> $GITHUB_STEP_SUMMARY
echo "## E2E CLI Results (Tier 1): ${{ matrix.cli }} (${{ matrix.os }})" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f e2e-cli-results.txt ]; then
echo '```' >> $GITHUB_STEP_SUMMARY
Expand All @@ -126,7 +129,7 @@ jobs:
exit 1

matrix-report:
name: CLI Matrix Report
name: CLI Matrix Report (Tier 1)
needs: [e2e-cli]
if: always()
runs-on: ubuntu-24.04
Expand Down
3 changes: 2 additions & 1 deletion .planning/MILESTONES.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@

**Known Gaps:**

- OC-01–06: OpenCode converter is a stub (deferred — OpenCode runtime format still evolving)
- OC-01–06: OpenCode converter is a stub (deferred — OpenCode runtime format still evolving).
**Closed in v3.1 Phase 57 by removing the stub**; OpenCode is no longer a supported runtime.

**Stats:**

Expand Down
7 changes: 6 additions & 1 deletion .planning/PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,12 @@ Agent Memory implements a layered cognitive architecture:

### Known Gaps (v2.7)

- OC-01–06: OpenCode converter is a stub (methods return empty). Deferred to v3.0.
- OC-01–06: RESOLVED-BY-REMOVAL in v3.1 Phase 57. The OpenCode converter was a
stub whose methods returned empty, so `memory-installer --agent opencode`
reported success and wrote no files. Rather than carry the gap further, the
converter, the `Runtime::OpenCode` variant, its bats suite, and the archived
plugin directory were deleted, and OpenCode is documented as not supported.
The runtime-agnostic `memory-ingest --agent opencode` path is unaffected.

### Deferred / Future

Expand Down
12 changes: 8 additions & 4 deletions .planning/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
- ✅ **v2.6 Cognitive Retrieval** — Phases 39-44 (shipped 2026-03-16)
- ✅ **v2.7 Multi-Runtime Portability** — Phases 45-50 (shipped 2026-03-22)
- **v3.0 Competitive Parity & Benchmarks** — Phases 51-53 + Phase 51.5 (in progress; Phase 51.5 merged 2026-04-28)
- **v3.1 Make It True** — Phases 54-58 (in progress; Phases 54, 54.5, 55 merged 2026-08-30, Phase 56 executing)
- **v3.1 Make It True** — Phases 54-58 (in progress; Phases 54, 54.5, 55, 56 merged 2026-08-30, Phase 57 executing)

## Phases

Expand Down Expand Up @@ -279,15 +279,19 @@ Close the claim/reality gap, then open the shop window. No new capabilities.
- [x] 55-01: Split setup vs query (64.6s was toc_build)
- [x] 55-02: Honest percentiles (n≥10 / n≥30)

### Phase 56: Honest Benchmarks (3/3 plans) — IN EXECUTION 2026-08-30
### Phase 56: Honest Benchmarks (3/3 plans) — COMPLETE 2026-08-30 (PR #34)

- [x] 56-01: Custom harness (real recall@k, content compression, fail-loud, isolation, ≥25 fixtures)
- [x] 56-02: LOCOMO adapter v2 (real schema, mock vs llm-judge)
- [x] 56-03: Smoke artifacts + HOLD comparison marketing

### Phase 57: Shop Window & Positioning (0/3)
### Phase 57: Shop Window & Positioning (3/3 plans) — IN EXECUTION 2026-08-30

- [x] 57-01: Repo hygiene (root README, LICENSE, repository URL)
- [x] 57-02: Positioning writeup vs Mem0 / Zep / MemMachine / Letta
- [x] 57-03: Scope trim — Tier 1/Tier 2 surface; OpenCode stub deleted

### Phase 58: Launch (side quest)

*Updated: 2026-08-30 — Phase 54.5 merged (#35); Phase 56 Honest Benchmarks in execution*
*Updated: 2026-08-30 — Phase 56 merged (#34); Phase 57 Shop Window in execution*

25 changes: 15 additions & 10 deletions .planning/STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ gsd_state_version: 1.0
milestone_name: Make It True
status: in_progress
stopped_at: null
last_updated: "2026-08-30T20:45:00.000Z"
last_activity: 2026-08-30 — Phase 54.5 merged (#35); Phase 56 Honest Benchmarks rebasing
last_updated: "2026-08-30T22:30:00.000Z"
last_activity: 2026-08-30 — Phase 56 merged (#34); Phase 57 Shop Window & Positioning in execution
progress:
total_phases: 6
completed_phases: 3
completed_phases: 4
total_plans: 14
completed_plans: 11
percent: 79
Expand All @@ -20,29 +20,31 @@ progress:
See: .planning/PROJECT.md (updated 2026-03-22)

**Core value:** Agent can answer "what were we talking about last week?" without scanning everything
**Current focus:** v3.1 Phase 56Honest Benchmarks (real recall@k, locomo10.json schema, HOLD comparison marketing)
**Current focus:** v3.1 Phase 57Shop Window & Positioning (root README, LICENSE, positioning writeup, supported-surface tiering)

## Current Position

Phase: 56 of 58 (Honest Benchmarks)
Plan: 01-03 implemented on `feature/phase-56-honest-benchmarks` (PR #34)
Status: Phase 54 + 54.5 + 55 merged; Phase 56 rebase onto #35
Last activity: 2026-08-30 — #35 merged (clippy pin); rebasing #34
Phase: 57 of 58 (Shop Window & Positioning)
Plan: 01-03 implemented on `claude/phase-54-toolchain-drift-3k4fer`
Status: Phases 54, 54.5, 55, 56 merged; Phase 57 in review
Last activity: 2026-08-30 — #34 merged; Phase 57 README/LICENSE/positioning/scope-trim

Progress: [████████░░] ~79% (11/14 plans; Phase 56 of 54-58)
Progress: [████████░░] 11/14 plans merged; Phase 57's 3 plans are implemented and in review
(Phase 58 is a side quest, not a GSD phase)

## Out-of-band Work

### Open PRs

| PR | What | Status |
|---|---|---|
| #34 | Phase 56 Honest Benchmarks | Open; rebasing onto #35 |
| _(none open)_ | | |

### Recently Merged

| PR | What | Merged |
|---|---|---|
| #34 | Phase 56 Honest Benchmarks | 2026-08-30 |
| #35 | Phase 54.5 truth leaks + rustc 1.97 pin | 2026-08-30 |
| #33 | Phase 55 Performance Truth | 2026-08-30 |
| #32 | Phase 54 Integration Truth | 2026-08-30 |
Expand All @@ -59,3 +61,6 @@ Progress: [████████░░] ~79% (11/14 plans; Phase 56 of 54-58)
- Phase 55: split setup vs query in `perf_bench`; p90/p99 withheld below 10/30 samples
- Warm = one setup + N query samples; cold = new store per iteration
- Phase 56: substring metric is `context_hit_rate`; HOLD LOCOMO comparison marketing until `locomo_llm_judge` artifact exists
- Phase 57 tiering: Tier 1 = Claude Code + Codex CLI (PR gate); Tier 2 = Gemini + Copilot (weekly schedule)
- Phase 57: OpenCode removed rather than archived — a converter whose methods return empty is a false success, not a gap
- Phase 57: no comparative benchmark claim ships while the only committed results are mock-backend / mock-judge
32 changes: 32 additions & 0 deletions .planning/phases/57-shop-window/57-CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Phase 57: Shop Window & Positioning

**Gathered:** 2026-08-30
**Status:** In execution
**Source:** docs/plans/phase-57-shop-window-plan.md

Make the public face of the repo match the reality Phases 54–56 established:
a root README with an honest status table, a LICENSE, a positioning writeup
that leads with the structural differences and declines to make a benchmark
comparison it cannot back, and a supported-surface trim that deletes the
OpenCode stub instead of shipping empty methods.

## What was already true before this phase

- No root `README.md` — the GitHub landing page was empty
- No `LICENSE` file, despite `license = "MIT"` in `Cargo.toml`
- `workspace.package.repository` pointed at `spillwave/agent-memory`, not the
actual remote `SpillwaveSolutions/agent-memory`
- `docs/README.md` advertised "Passive capture from Claude Code, OpenCode,
Gemini CLI hooks" and a "Plugin (TypeScript)" OpenCode adapter; the OpenCode
converter's methods all returned empty and `plugins/memory-opencode-plugin/`
contained only an archived README
- Five bats CLI suites gated every PR, one of them for a runtime with no
working converter

## Constraints carried in

- **Benchmark gate (Phase 56):** the only committed results are mock-backend
and mock-judge, so no comparative accuracy claim may appear anywhere public
- **Execution-evidence rule (v3.1 process change):** the quickstart is a
run-dependent requirement, so it must be verified by actually executing it
and committing the transcript — not by the README's existence
Loading
Loading