Publish v0.1 benchmark results and analytics

[brunovicco]

Problem

RAGForge had completed benchmark evidence but no explicitly published result, stable read-only API, or analytical interface for reviewing and communicating the findings.

The project documentation also did not reflect the complete ten-strategy benchmark matrix or provide a qualified recommendation based on the final cost-controlled evaluation.

Solution

Publish the deterministic v0.1 benchmark run and expose its aggregate results through a validated repository boundary, a read-only FastAPI API, and a bilingual Streamlit dashboard.

The published recommendation identifies SAC as the best-balanced strategy for this benchmark sample while preserving the limitations and reproducibility evidence of the evaluation.

Main changes

• Publish run 20260726T185553Z, including:
  • immutable evidence manifests and checksums;
  • per-question evaluation evidence;
  • aggregate results and replay cache;
  • deterministic sampling metadata for 60 stratified questions from the 194-question test split.
• Add an explicit experiments/published-runs.json publication catalog.
• Add application contracts and a filesystem adapter for loading only cataloged, completed runs.
• Validate run identifiers, JSON size, paths, schemas, manifest status, and identity consistency.
• Add a read-only FastAPI API:
  • GET /health
  • GET /api/v1/benchmark-runs
  • GET /api/v1/benchmark-runs/{run_id}
• Add an offline Streamlit analytical dashboard with:
  • Portuguese and English interfaces;
  • strategy comparison table and quality chart;
  • explanations for all ten retrieval techniques;
  • definitions and interpretation guidance for all nine reported metrics;
  • reproducibility and evidence references.
• Recommend SAC as the primary v0.1 strategy based on its balanced retrieval and answer-quality profile.
• Update the Docker entrypoint to run the published-results API.
• Update Make targets, architecture documentation, and English and Portuguese READMEs.

Test evidence

• Full quality gate passed.
• 528 tests passed; 20 integration tests deselected.
• Test coverage: 82.69%.
• Ruff lint and formatting checks passed.
• Mypy strict typing passed for 208 source files.
• Architecture and governance validations passed.
• Bandit reported no security issues.
• pip-audit reported no new known vulnerabilities; two documented exceptions remain ignored.
• Streamlit rendering was exercised in Portuguese and English with no runtime exceptions.
• Published-run checksums, event chain, and manifest references verify successfully.
• Staged diff passes git diff --cached --check.

Security and data impact

• The API is read-only and exposes only explicitly cataloged benchmark aggregates.
• API responses do not expose prompts, per-question model responses, replay-cache contents, or provider credentials.
• Repository paths are resolved and constrained to the project root.
• JSON documents have explicit size limits and strict Pydantic validation.
• Unknown and malformed run identifiers return stable error contracts.
• The published evidence contains benchmark outputs for reproducibility but no provider credentials or production personal data.
• No database migration or mutable external service is introduced.

Operational and rollout impact

• The container now starts the FastAPI application with Uvicorn on port 8000.
• Run the API with make api.
• Run the offline dashboard with make dashboard.
• Both applications depend on the versioned publication catalog and local benchmark files.
• Neither application makes provider calls when displaying published results.
• Existing live benchmark execution remains unchanged.

Risks and follow-ups

• The recommendation is based on a deterministic sample of 60 questions, not the complete 194-question test split.
• Independent judge metrics remain qualified until the ADR-0007 human-calibration gate is completed.
• Semantic citation auditing was disabled for this cost-controlled run.
• Provider pricing was not configured, so the report does not include an authoritative cost estimate.
• The published replay cache and evidence add significant repository size.
• The live Arena, adaptive router, and corrective workflow remain planned follow-up work.