fix(engine): create the database's parent directory before connecting

README.md

@@ -2,7 +2,7 @@
 
 [![CI](https://github.com/adbarc92/appforge/actions/workflows/ci.yml/badge.svg)](https://github.com/adbarc92/appforge/actions/workflows/ci.yml)
 [![Version](https://img.shields.io/badge/version-1.0.0-blue)](https://github.com/adbarc92/appforge/releases)
-[![Tests](https://img.shields.io/badge/tests-152%20backend%20%2B%2028%20frontend-brightgreen)](#tests)
+[![Tests](https://img.shields.io/badge/tests-155%20backend%20%2B%2028%20frontend-brightgreen)](#tests)
 [![Coverage](https://img.shields.io/badge/coverage-87%25-brightgreen)](#tests)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/)
 [![License](https://img.shields.io/badge/license-MIT-yellow)](LICENSE)
@@ -11,7 +11,7 @@
 
 AppForge models a ~14-person software team as **16 specialized agents** that a scheduler dispatches across a **six-phase dependency graph**. The orchestration is not a single async loop — it is a genuine **MCP state server** plus a pool of **independent OS worker processes** that claim and execute work concurrently, coordinating shared state without collision.
 
-> **Status: v1.0 — complete for now.** The orchestration engine is finished, tested (152 backend + 28 frontend tests, ~87% coverage), and feature-frozen; there is no work in flight. Agents run in a deterministic **mock mode** by default (free, fast, reproducible) with a real-Anthropic mode available; the value here is the *orchestration architecture*, which is real and proven — not a finished app generator. Full detail in [`docs/STATUS.md`](docs/STATUS.md).
+> **Status: v1.0 — complete for now.** The orchestration engine is finished, tested (155 backend + 28 frontend tests, ~87% coverage), and feature-frozen; there is no work in flight. Agents run in a deterministic **mock mode** by default (free, fast, reproducible) with a real-Anthropic mode available; the value here is the *orchestration architecture*, which is real and proven — not a finished app generator. Full detail in [`docs/STATUS.md`](docs/STATUS.md).
 
 ---
 
@@ -121,7 +121,7 @@ The lease, heartbeat, and reaper settings are what make worker crashes recoverab
 ### Tests
 
 ```bash
-uv run pytest tests/ -q          # backend suite (152 tests, ~87% coverage)
+uv run pytest tests/ -q          # backend suite (155 tests, ~87% coverage)
 uv run ruff check backend/ tests/ && uv run black --check backend/ tests/
 cd frontend && npm test          # frontend suite (28 tests)
 ```

backend/engine/store.py

@@ -6,6 +6,7 @@
 import json
 import time
 from contextlib import asynccontextmanager
+from pathlib import Path
 from typing import Any
 
 import aiosqlite
@@ -38,6 +39,13 @@ def _now(self) -> float:
         return time.time()
 
     async def connect(self) -> None:
+        # sqlite will not create missing intermediate directories: the default
+        # paths live under data/, which is gitignored and therefore absent on a
+        # fresh clone or CI runner. Without this, connect() raises
+        # "unable to open database file".
+        parent = Path(self.db_path).parent
+        if str(parent) not in ("", "."):
+            parent.mkdir(parents=True, exist_ok=True)
         self._db = await aiosqlite.connect(self.db_path)
         self._db.row_factory = aiosqlite.Row
         await self._db.execute("PRAGMA journal_mode=WAL")

docs/STATUS.md

@@ -7,7 +7,7 @@
 
 ## State summary
 
-**Version:** 1.0.0 · **Branch:** `publication-prep` · **Status: complete for now (feature-frozen).**
+**Version:** 1.0.0 · **Branch:** `main` · **Status: complete for now (feature-frozen).**
 
 AppForge v1.0 is the finished form of what this project set out to prove: a **parallel, MCP-coordinated multi-agent orchestration engine** in which a real MCP state server and a pool of independent OS worker processes drive a product idea through a six-phase dependency graph (Clarify → Design → Code → Test → Deploy → Iterate), with human approval gates and automatic budget-driven model downgrade.
 
@@ -17,10 +17,11 @@ The engine is done, tested, documented, and published under MIT. There is no in-
 
 | Signal | State |
 |---|---|
-| Backend suite | **152 passed** (`uv run pytest tests/`) |
-| Coverage | **86.93%** (gate: 70%) |
+| Backend suite | **155 passed** (`uv run pytest tests/`) |
+| Coverage | **86.96%** (gate: 70%) |
 | Frontend suite | **28 passed** across 6 files (`cd frontend && npm test`) |
-| Lint / format | `ruff check` clean · `black --check` clean (66 files) |
+| Lint / format | `ruff check` clean · `black --check` clean (67 files) |
+| CI on `main` | `backend` · `frontend` · `validate-config` green; **`e2e` red** (see gaps) |
 | Version | `pyproject.toml` 1.0.0 · `frontend/package.json` 1.0.0 · `backend/main.py` FastAPI 1.0.0 |
 | CLI | `uv run appforge run "<idea>"` (hatchling build backend, entry point installed) |
 | License | MIT (`LICENSE`) |
@@ -43,20 +44,36 @@ The engine is done, tested, documented, and published under MIT. There is no in-
 - **Single-user web bridge.** The live UI targets local single-user use. Multi-tenant lifecycle hardening (per-connection dedup, reconnect durability) is unbuilt.
 - **Historical docs.** `docs/Roadmap.md`, `docs/CoreDesignDocument.md`, and the dated `Status-*.md` files describe the LangGraph-era design and are kept as history only.
 - **Shutdown noise.** A `CancelledError` traceback prints on CLI teardown when the state-server task is cancelled. Cosmetic — the run reports `done` and exits 0 — but it looks alarming.
+- **`e2e` CI job is red — `database is locked`.** *Not* cosmetic and **not yet diagnosed.** The web bridge runs `start_run(workers=4)` against `data/web.db`; under CI those four workers hit lock contention and `claim_next_task` fails with `database is locked`. Ruling one thing out: a missing `busy_timeout` pragma is not the cause, since Python's `sqlite3.connect()` already applies a 5s busy timeout that `aiosqlite` inherits. This was masked until 2026-07-25 by the missing-`data/`-directory bug failing earlier in the same path.
+- **`e2e` specs predate the engine.** Separately, the Playwright specs were last touched 2026-06-02 and still drive the retired LangGraph chat flow (`Clarifying question #N`, `Mock PRD`). Even once the lock contention is resolved, they likely need rewriting or retiring.
 
 ### Next steps
 
 None required — the project is feature-frozen at 1.0. If it is picked up again, the highest-value candidates, in order:
 
-1. Thread real Anthropic token usage into BudgetGuard so budget figures are actual, not simulated.
-2. Harden the web bridge for multi-user / reconnect durability.
-3. Refresh or archive the LangGraph-era design docs so `docs/` matches the shipped engine.
-4. Silence the `CancelledError` teardown traceback in `stop_run`.
+1. **Diagnose the `database is locked` contention** that keeps the `e2e` job red — the only known functional defect, and the reason CI is not fully green.
+2. Decide whether the LangGraph-era Playwright specs get rewritten against the engine's flow or retired.
+3. Thread real Anthropic token usage into BudgetGuard so budget figures are actual, not simulated.
+4. Harden the web bridge for multi-user / reconnect durability.
+5. Refresh or archive the LangGraph-era design docs so `docs/` matches the shipped engine.
+6. Silence the `CancelledError` teardown traceback in `stop_run`.
 
 ---
 
 ## Session log
 
+### 2026-07-25 — fix: the engine could not start on a fresh clone
+
+`Store.connect()` called `aiosqlite.connect(db_path)` without creating the directory holding the file. sqlite does not create missing intermediate directories, and the default paths (`data/engine.db`, `data/web.db`) live under `data/`, which is gitignored and therefore absent on any fresh clone or CI runner — so the engine died with `OperationalError: unable to open database file`.
+
+- This was **red on `main`**: CI `backend (3.11)` failed `test_start_project_emits_project_created`, and the `e2e` job's web server threw the same error repeatedly.
+- It survived local verification because every store test builds its path under pytest's `tmp_path`, which already exists, and because a developer who has ever run the engine has a `data/` directory. Confirmed by A/B: with `data/` removed, the failing CI test reproduces exactly, and passes with the fix.
+- Fixed at the single point every caller passes through, plus `tests/engine/test_store_db_path.py` covering the missing parent, nested parents, and a bare filename (which must not trip the mkdir).
+- Backend suite **152 → 155**; full suite verified with `data/` absent.
+- **This unmasked a second defect.** With the open failure cleared, the `e2e` job now fails further along with `claim_next_task failed: database is locked` — logged above as a known gap and the top next step. It was always there; the missing-directory bug simply failed first.
+- Repo hygiene alongside: 8 stale Phase-2-era agent worktrees and their branches removed, 9 merged local and 4 merged remote branches deleted.
+- **State delta:** `git clone && uv run appforge run "…"` now works on a machine that has never run AppForge.
+
 ### 2026-07-25 — v1.0 follow-ups: working CLI, honest dependencies, dead config removed
 
 Closed the three loose ends the release stamp surfaced.
@@ -70,7 +87,7 @@ Closed the three loose ends the release stamp surfaced.
 ### 2026-07-25 — v1.0.0 release stamp
 
 - Bumped `pyproject.toml` to `1.0.0` and its classifier from `3 - Alpha` to `5 - Production/Stable`; bumped `frontend/package.json` from `0.0.0` to `1.0.0` (`backend/main.py` already declared 1.0.0).
-- Verified the release state before stamping it: 154 backend tests passed, 86.93% coverage, 28 frontend tests passed, ruff and black clean.
+- Verified the release state before stamping it: 154 backend tests passed, 86.96% coverage, 28 frontend tests passed, ruff and black clean.
 - Created this canonical `docs/STATUS.md`, superseding the dated `Status-*.md` snapshots (which describe the retired LangGraph architecture).
 - Marked the project **complete for now** in `README.md` and replaced the stale "Active session pickup" block in `CLAUDE.md`, which still pointed sessions at `Status-2026_06_02.md` and Phase 6 work that the engine rewrite made moot.
 - **State delta:** unversioned work-in-progress → feature-frozen v1.0.0 with a single accurate status entry point.

tests/engine/test_store_db_path.py

@@ -0,0 +1,55 @@
+"""Regression: the store must create the directory holding its database file.
+
+The engine's default paths live under data/ (data/engine.db, data/web.db),
+which is gitignored and so does not exist on a fresh clone or a CI runner.
+sqlite does not create missing intermediate directories, so connect() used to
+die with "unable to open database file" for anyone who had not happened to
+create data/ locally.
+
+Every other store test builds its path under pytest's tmp_path, which already
+exists -- which is exactly why the suite stayed green while CI failed.
+"""
+
+from pathlib import Path
+
+from backend.engine.phases import PhasesConfig
+from backend.engine.store import Store
+
+CFG = PhasesConfig.load("config/phases.yaml")
+BASE = dict.fromkeys(CFG.all_agent_ids(), "gpt-4o")
+
+
+async def test_connect_creates_missing_parent_directory(tmp_path):
+    db_path = tmp_path / "data" / "engine.db"
+    assert not db_path.parent.exists()  # the fresh-clone condition
+
+    s = Store(str(db_path), CFG, BASE)
+    await s.connect()
+    try:
+        await s.create_run("r1", "Build a todo app", 5.0)
+        assert [t["agent_id"] for t in await s._all_tasks("r1")] == ["clarifying_pm"]
+    finally:
+        await s.close()  # close before tmp_path teardown (win32 WAL files)
+
+    assert db_path.exists()
+
+
+async def test_connect_creates_nested_parent_directories(tmp_path):
+    db_path = tmp_path / "a" / "b" / "c" / "run.db"
+
+    s = Store(str(db_path), CFG, BASE)
+    await s.connect()
+    await s.close()
+
+    assert db_path.exists()
+
+
+async def test_connect_accepts_bare_filename(tmp_path, monkeypatch):
+    """A path with no directory component must not trip the mkdir."""
+    monkeypatch.chdir(tmp_path)
+
+    s = Store("run.db", CFG, BASE)
+    await s.connect()
+    await s.close()
+
+    assert Path(tmp_path / "run.db").exists()