Skip to content

test(agents): evaluate GPT-6 Astra async execution backend #1662

Description

@groupthinking

Decision

TEST — add GPT-6 Astra only as an optional, disabled Agent Factory execution backend/model and benchmark it against the native path and Google Antigravity.

Verified change (September 7, 2026)

Primary sources:

The official contract identifies gpt-6-astra and documents:

  • asynchronous function/custom tool calls using async: true, with results returned under the original call_id;
  • mid-turn steering over the Responses API WebSocket while preserving completed work;
  • reasoning configuration changes without rewriting the cached prompt prefix;
  • structured outputs, programmatic tool calling, multi-agent orchestration, and computer use;
  • a 1,050,000-token context window and 128,000 maximum output;
  • $10/M input and $50/M output pricing, with 2x input/cache and 1.5x output rates for requests above 272K input tokens;
  • text/image input, but no audio or video input.

Why this belongs in Agent Factory

Async tool calls can keep independent work moving while EventRelay executes a long-running tool, and mid-turn steering can map user corrections into an active run without discarding completed work. Those capabilities could shorten the current SEE → TAKE ACTION → RECORD → REVIEW loop.

This is a potential backend, not a replacement for Agent Factory's policy, provenance, receipts, or Video Pack.

Bounded implementation

  • Add a provider-neutral interface for pending tool calls: stable run ID, provider call_id, tool name, serialized input digest, state, deadlines, result digest, and receipt locator.
  • Add a fixture/mock Astra adapter using the Responses API contract; live calls remain disabled by default.
  • Persist pending calls before dispatch and make result submission idempotent.
  • Allow independent ready work to continue while one tool is pending; prove no duplicate execution after retry/restart.
  • Map mid-turn steering to an append-only EventRelay instruction/update event that preserves the original request and completed-step receipts.
  • Record every reasoning configuration_update in the execution receipt.
  • Enforce explicit tool allowlists, approval boundaries, origin isolation, and immutable/audited skill and AGENTS.md inputs.
  • Enforce per-run input/output token caps and route or reject requests above 272K input tokens unless separately approved.
  • Reject unsupported direct audio/video inputs; use the existing Video Pack handoff for video-derived evidence.
  • Treat provider safety stops as terminal/blocked receipts, never as successful completion.
  • Produce a machine-readable comparison against native Agent Factory and Antigravity: completion quality, wall time, serialized idle time, duplicate-tool rate, receipt completeness, token usage, estimated cost, and policy denials.

Fixture acceptance tests

  • A slow tool does not block an independent tool or independent answer segment.
  • A restart between dispatch and result preserves exactly-once application semantics.
  • A late, duplicated, or mismatched call_id result is rejected and recorded.
  • Steering changes only unfinished work and preserves completed receipts.
  • Token caps and the >272K pricing boundary fail closed.
  • Direct video/audio inputs fail closed and point to Video Pack.
  • Tool, skill, and repository-instruction injection attempts cannot expand authorization.
  • Mocked provider errors, safety stops, timeout, and cancellation produce durable failure receipts.

Safety boundary

No merge, deployment, live provider call, billing change, or paid benchmark is included. A live run requires a separate capped usage approval after fixture tests pass.

Estimated effort: 1–2 engineering days for the fixture adapter and benchmark harness; 2–4 hours plus provider usage for a later capped live comparison.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions