Skip to content

Run the live smoke harness as a release gate, not by memory #161

Description

@yuanhao

Second of the trustworthiness items. #160 did the first.

Why

examples/release_smoke.rs found a real bug on its first run — tools with zero parameters were uncallable on Anthropic, in shipped releases. examples/long_horizon.rs then found the LlmCompaction fallback behaviour that became #150.

Both currently run when someone remembers. That is the same class of problem the price audit had before it was written into the release checklist: a check that depends on memory is a check you do not have.

Why not per-PR CI

Per-PR is the wrong granularity. Per-release is right: a few cents, a few minutes, once per version.

Proposal

A manual-dispatch GitHub Action (workflow_dispatch, optionally also on push to release/**) that runs:

cargo run --example release_smoke                      # ANTHROPIC_API_KEY
SMOKE_MODEL=deepseek cargo run --example release_smoke # DEEPSEEK_API_KEY

Both already exit non-zero on failure, so they gate correctly as-is.

DeepSeek matters as a second target: it exercises the OpenAI-compat implementation, which is a separate SSE parser and tool-call accumulator from Anthropic's. The zero-arg bug lived in exactly that layer.

Acceptance

  • Workflow exists and can be dispatched against a release branch.
  • Keys stored as repository secrets; the job is skipped rather than failed when absent, so forks are not broken.
  • memory/release.md's checklist references it as a numbered step alongside the price audit.
  • Timeout set — a hung run should fail the gate, not hang the queue.

Not in scope

long_horizon is not a gate. It is a diagnostic: slower, and its interesting output is a judgement call rather than a pass/fail. Keep it manual.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions