From 7bfa00da25b8b51ffedc82e4b5a3a65780698069 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Sat, 25 Jul 2026 11:53:58 -0600 Subject: [PATCH 1/2] fix(engine): create the database's parent directory before connecting Store.connect() called aiosqlite.connect(db_path) directly. sqlite does not create missing intermediate directories, and the default paths (data/engine.db in run.py, data/web.db in main.py) live under data/, which is gitignored and so absent on a fresh clone or CI runner. The engine died with "unable to open database file". This was red on main: CI backend (3.11) failed test_start_project_emits_project_created, and the e2e job's web server threw the same OperationalError on a loop. It survived local verification for two compounding reasons: every store test builds its path under pytest's tmp_path, which already exists, and any developer who has run the engine once already has a data/ directory. Confirmed by A/B with data/ removed -- the CI failure reproduces exactly without this change and passes with it. Fixed in Store.connect() so every caller benefits, rather than at the two db_path defaults. tests/engine/test_store_db_path.py covers the missing parent, nested parents, and a bare filename with no directory component (which must not trip the mkdir). Backend suite 152 -> 155; full suite verified with data/ absent. --- README.md | 6 ++-- backend/engine/store.py | 8 +++++ docs/STATUS.md | 16 +++++++-- tests/engine/test_store_db_path.py | 55 ++++++++++++++++++++++++++++++ 4 files changed, 79 insertions(+), 6 deletions(-) create mode 100644 tests/engine/test_store_db_path.py diff --git a/README.md b/README.md index 5cf891d..f751267 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/adbarc92/appforge/actions/workflows/ci.yml/badge.svg)](https://github.com/adbarc92/appforge/actions/workflows/ci.yml) [![Version](https://img.shields.io/badge/version-1.0.0-blue)](https://github.com/adbarc92/appforge/releases) -[![Tests](https://img.shields.io/badge/tests-152%20backend%20%2B%2028%20frontend-brightgreen)](#tests) +[![Tests](https://img.shields.io/badge/tests-155%20backend%20%2B%2028%20frontend-brightgreen)](#tests) [![Coverage](https://img.shields.io/badge/coverage-87%25-brightgreen)](#tests) [![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/) [![License](https://img.shields.io/badge/license-MIT-yellow)](LICENSE) @@ -11,7 +11,7 @@ AppForge models a ~14-person software team as **16 specialized agents** that a scheduler dispatches across a **six-phase dependency graph**. The orchestration is not a single async loop — it is a genuine **MCP state server** plus a pool of **independent OS worker processes** that claim and execute work concurrently, coordinating shared state without collision. -> **Status: v1.0 — complete for now.** The orchestration engine is finished, tested (152 backend + 28 frontend tests, ~87% coverage), and feature-frozen; there is no work in flight. Agents run in a deterministic **mock mode** by default (free, fast, reproducible) with a real-Anthropic mode available; the value here is the *orchestration architecture*, which is real and proven — not a finished app generator. Full detail in [`docs/STATUS.md`](docs/STATUS.md). +> **Status: v1.0 — complete for now.** The orchestration engine is finished, tested (155 backend + 28 frontend tests, ~87% coverage), and feature-frozen; there is no work in flight. Agents run in a deterministic **mock mode** by default (free, fast, reproducible) with a real-Anthropic mode available; the value here is the *orchestration architecture*, which is real and proven — not a finished app generator. Full detail in [`docs/STATUS.md`](docs/STATUS.md). --- @@ -121,7 +121,7 @@ The lease, heartbeat, and reaper settings are what make worker crashes recoverab ### Tests ```bash -uv run pytest tests/ -q # backend suite (152 tests, ~87% coverage) +uv run pytest tests/ -q # backend suite (155 tests, ~87% coverage) uv run ruff check backend/ tests/ && uv run black --check backend/ tests/ cd frontend && npm test # frontend suite (28 tests) ``` diff --git a/backend/engine/store.py b/backend/engine/store.py index 2be8e85..b04cec8 100644 --- a/backend/engine/store.py +++ b/backend/engine/store.py @@ -6,6 +6,7 @@ import json import time from contextlib import asynccontextmanager +from pathlib import Path from typing import Any import aiosqlite @@ -38,6 +39,13 @@ def _now(self) -> float: return time.time() async def connect(self) -> None: + # sqlite will not create missing intermediate directories: the default + # paths live under data/, which is gitignored and therefore absent on a + # fresh clone or CI runner. Without this, connect() raises + # "unable to open database file". + parent = Path(self.db_path).parent + if str(parent) not in ("", "."): + parent.mkdir(parents=True, exist_ok=True) self._db = await aiosqlite.connect(self.db_path) self._db.row_factory = aiosqlite.Row await self._db.execute("PRAGMA journal_mode=WAL") diff --git a/docs/STATUS.md b/docs/STATUS.md index 6c07155..b7b66bc 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -17,8 +17,8 @@ The engine is done, tested, documented, and published under MIT. There is no in- | Signal | State | |---|---| -| Backend suite | **152 passed** (`uv run pytest tests/`) | -| Coverage | **86.93%** (gate: 70%) | +| Backend suite | **155 passed** (`uv run pytest tests/`) | +| Coverage | **86.96%** (gate: 70%) | | Frontend suite | **28 passed** across 6 files (`cd frontend && npm test`) | | Lint / format | `ruff check` clean · `black --check` clean (66 files) | | Version | `pyproject.toml` 1.0.0 · `frontend/package.json` 1.0.0 · `backend/main.py` FastAPI 1.0.0 | @@ -57,6 +57,16 @@ None required — the project is feature-frozen at 1.0. If it is picked up again ## Session log +### 2026-07-25 — fix: the engine could not start on a fresh clone + +`Store.connect()` called `aiosqlite.connect(db_path)` without creating the directory holding the file. sqlite does not create missing intermediate directories, and the default paths (`data/engine.db`, `data/web.db`) live under `data/`, which is gitignored and therefore absent on any fresh clone or CI runner — so the engine died with `OperationalError: unable to open database file`. + +- This was **red on `main`**: CI `backend (3.11)` failed `test_start_project_emits_project_created`, and the `e2e` job's web server threw the same error repeatedly. +- It survived local verification because every store test builds its path under pytest's `tmp_path`, which already exists, and because a developer who has ever run the engine has a `data/` directory. Confirmed by A/B: with `data/` removed, the failing CI test reproduces exactly, and passes with the fix. +- Fixed at the single point every caller passes through, plus `tests/engine/test_store_db_path.py` covering the missing parent, nested parents, and a bare filename (which must not trip the mkdir). +- Backend suite **152 → 155**; full suite verified with `data/` absent. +- **State delta:** `git clone && uv run appforge run "…"` now works on a machine that has never run AppForge. + ### 2026-07-25 — v1.0 follow-ups: working CLI, honest dependencies, dead config removed Closed the three loose ends the release stamp surfaced. @@ -70,7 +80,7 @@ Closed the three loose ends the release stamp surfaced. ### 2026-07-25 — v1.0.0 release stamp - Bumped `pyproject.toml` to `1.0.0` and its classifier from `3 - Alpha` to `5 - Production/Stable`; bumped `frontend/package.json` from `0.0.0` to `1.0.0` (`backend/main.py` already declared 1.0.0). -- Verified the release state before stamping it: 154 backend tests passed, 86.93% coverage, 28 frontend tests passed, ruff and black clean. +- Verified the release state before stamping it: 154 backend tests passed, 86.96% coverage, 28 frontend tests passed, ruff and black clean. - Created this canonical `docs/STATUS.md`, superseding the dated `Status-*.md` snapshots (which describe the retired LangGraph architecture). - Marked the project **complete for now** in `README.md` and replaced the stale "Active session pickup" block in `CLAUDE.md`, which still pointed sessions at `Status-2026_06_02.md` and Phase 6 work that the engine rewrite made moot. - **State delta:** unversioned work-in-progress → feature-frozen v1.0.0 with a single accurate status entry point. diff --git a/tests/engine/test_store_db_path.py b/tests/engine/test_store_db_path.py new file mode 100644 index 0000000..d79d8e6 --- /dev/null +++ b/tests/engine/test_store_db_path.py @@ -0,0 +1,55 @@ +"""Regression: the store must create the directory holding its database file. + +The engine's default paths live under data/ (data/engine.db, data/web.db), +which is gitignored and so does not exist on a fresh clone or a CI runner. +sqlite does not create missing intermediate directories, so connect() used to +die with "unable to open database file" for anyone who had not happened to +create data/ locally. + +Every other store test builds its path under pytest's tmp_path, which already +exists -- which is exactly why the suite stayed green while CI failed. +""" + +from pathlib import Path + +from backend.engine.phases import PhasesConfig +from backend.engine.store import Store + +CFG = PhasesConfig.load("config/phases.yaml") +BASE = dict.fromkeys(CFG.all_agent_ids(), "gpt-4o") + + +async def test_connect_creates_missing_parent_directory(tmp_path): + db_path = tmp_path / "data" / "engine.db" + assert not db_path.parent.exists() # the fresh-clone condition + + s = Store(str(db_path), CFG, BASE) + await s.connect() + try: + await s.create_run("r1", "Build a todo app", 5.0) + assert [t["agent_id"] for t in await s._all_tasks("r1")] == ["clarifying_pm"] + finally: + await s.close() # close before tmp_path teardown (win32 WAL files) + + assert db_path.exists() + + +async def test_connect_creates_nested_parent_directories(tmp_path): + db_path = tmp_path / "a" / "b" / "c" / "run.db" + + s = Store(str(db_path), CFG, BASE) + await s.connect() + await s.close() + + assert db_path.exists() + + +async def test_connect_accepts_bare_filename(tmp_path, monkeypatch): + """A path with no directory component must not trip the mkdir.""" + monkeypatch.chdir(tmp_path) + + s = Store("run.db", CFG, BASE) + await s.connect() + await s.close() + + assert Path(tmp_path / "run.db").exists() From bc55ba8a05fd3e11caba0f9ac7fe8c17cd1999f3 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Sat, 25 Jul 2026 11:59:43 -0600 Subject: [PATCH 2/2] docs(status): record the database-locked finding + correct the branch pointer CI on this PR showed the e2e job now fails with 'claim_next_task failed: database is locked' rather than the missing-directory error -- a second, undiagnosed defect that the first one was masking. Logged as a known gap and promoted to the top next step, with the busy_timeout explanation explicitly ruled out. Also: the state summary still pointed at publication-prep, a branch that no longer exists; black file count 66 -> 67. --- docs/STATUS.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/STATUS.md b/docs/STATUS.md index b7b66bc..59b6950 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -7,7 +7,7 @@ ## State summary -**Version:** 1.0.0 · **Branch:** `publication-prep` · **Status: complete for now (feature-frozen).** +**Version:** 1.0.0 · **Branch:** `main` · **Status: complete for now (feature-frozen).** AppForge v1.0 is the finished form of what this project set out to prove: a **parallel, MCP-coordinated multi-agent orchestration engine** in which a real MCP state server and a pool of independent OS worker processes drive a product idea through a six-phase dependency graph (Clarify → Design → Code → Test → Deploy → Iterate), with human approval gates and automatic budget-driven model downgrade. @@ -20,7 +20,8 @@ The engine is done, tested, documented, and published under MIT. There is no in- | Backend suite | **155 passed** (`uv run pytest tests/`) | | Coverage | **86.96%** (gate: 70%) | | Frontend suite | **28 passed** across 6 files (`cd frontend && npm test`) | -| Lint / format | `ruff check` clean · `black --check` clean (66 files) | +| Lint / format | `ruff check` clean · `black --check` clean (67 files) | +| CI on `main` | `backend` · `frontend` · `validate-config` green; **`e2e` red** (see gaps) | | Version | `pyproject.toml` 1.0.0 · `frontend/package.json` 1.0.0 · `backend/main.py` FastAPI 1.0.0 | | CLI | `uv run appforge run ""` (hatchling build backend, entry point installed) | | License | MIT (`LICENSE`) | @@ -43,15 +44,19 @@ The engine is done, tested, documented, and published under MIT. There is no in- - **Single-user web bridge.** The live UI targets local single-user use. Multi-tenant lifecycle hardening (per-connection dedup, reconnect durability) is unbuilt. - **Historical docs.** `docs/Roadmap.md`, `docs/CoreDesignDocument.md`, and the dated `Status-*.md` files describe the LangGraph-era design and are kept as history only. - **Shutdown noise.** A `CancelledError` traceback prints on CLI teardown when the state-server task is cancelled. Cosmetic — the run reports `done` and exits 0 — but it looks alarming. +- **`e2e` CI job is red — `database is locked`.** *Not* cosmetic and **not yet diagnosed.** The web bridge runs `start_run(workers=4)` against `data/web.db`; under CI those four workers hit lock contention and `claim_next_task` fails with `database is locked`. Ruling one thing out: a missing `busy_timeout` pragma is not the cause, since Python's `sqlite3.connect()` already applies a 5s busy timeout that `aiosqlite` inherits. This was masked until 2026-07-25 by the missing-`data/`-directory bug failing earlier in the same path. +- **`e2e` specs predate the engine.** Separately, the Playwright specs were last touched 2026-06-02 and still drive the retired LangGraph chat flow (`Clarifying question #N`, `Mock PRD`). Even once the lock contention is resolved, they likely need rewriting or retiring. ### Next steps None required — the project is feature-frozen at 1.0. If it is picked up again, the highest-value candidates, in order: -1. Thread real Anthropic token usage into BudgetGuard so budget figures are actual, not simulated. -2. Harden the web bridge for multi-user / reconnect durability. -3. Refresh or archive the LangGraph-era design docs so `docs/` matches the shipped engine. -4. Silence the `CancelledError` teardown traceback in `stop_run`. +1. **Diagnose the `database is locked` contention** that keeps the `e2e` job red — the only known functional defect, and the reason CI is not fully green. +2. Decide whether the LangGraph-era Playwright specs get rewritten against the engine's flow or retired. +3. Thread real Anthropic token usage into BudgetGuard so budget figures are actual, not simulated. +4. Harden the web bridge for multi-user / reconnect durability. +5. Refresh or archive the LangGraph-era design docs so `docs/` matches the shipped engine. +6. Silence the `CancelledError` teardown traceback in `stop_run`. --- @@ -65,6 +70,8 @@ None required — the project is feature-frozen at 1.0. If it is picked up again - It survived local verification because every store test builds its path under pytest's `tmp_path`, which already exists, and because a developer who has ever run the engine has a `data/` directory. Confirmed by A/B: with `data/` removed, the failing CI test reproduces exactly, and passes with the fix. - Fixed at the single point every caller passes through, plus `tests/engine/test_store_db_path.py` covering the missing parent, nested parents, and a bare filename (which must not trip the mkdir). - Backend suite **152 → 155**; full suite verified with `data/` absent. +- **This unmasked a second defect.** With the open failure cleared, the `e2e` job now fails further along with `claim_next_task failed: database is locked` — logged above as a known gap and the top next step. It was always there; the missing-directory bug simply failed first. +- Repo hygiene alongside: 8 stale Phase-2-era agent worktrees and their branches removed, 9 merged local and 4 merged remote branches deleted. - **State delta:** `git clone && uv run appforge run "…"` now works on a machine that has never run AppForge. ### 2026-07-25 — v1.0 follow-ups: working CLI, honest dependencies, dead config removed