Skip to content

eval: add inventory coding task coverage - #6

Merged
CuSO41108 merged 1 commit into
mainfrom
codex/eval-inventory-cases
Jul 19, 2026
Merged

eval: add inventory coding task coverage#6
CuSO41108 merged 1 commit into
mainfrom
codex/eval-inventory-cases

Conversation

@CuSO41108

Copy link
Copy Markdown
Owner

Summary

  • add an isolated inventory_service fixture with passing seed tests
  • add seven deterministic coding tasks covering boundaries, new APIs, atomic batches, compatibility, formatting, and code/test/docs changes
  • expand the built-in Eval v2 suite from 28 to 35 cases

Verification

  • python -m unittest tests.unit.test_evals tests.unit.test_eval_runtime_integration tests.unit.test_openai_compatible -v (39 passed)
  • python -m unittest discover -s evals/fixtures/inventory_service/tests -v (4 passed)
  • git diff --check

Local live-model Smoke

  • inventory_low_stock_boundary_001: passed all task, behavior, file-change, and trajectory checks
  • inventory_reserve_many_001: implementation and independent verification passed, but the Agent hit max_tool_rounds_exceeded before self-verification/final response
  • inventory_document_default_threshold_001: all code/test/README verification passed, but the Agent hit the same process limit before self-verification/final response

The two failures are retained as capability evidence; their Oracles were not weakened. No credentials, model configuration, or live artifacts are included.

Copilot AI review requested due to automatic review settings July 19, 2026 10:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@CuSO41108
CuSO41108 merged commit 2e90ea8 into main Jul 19, 2026
2 checks passed
@CuSO41108
CuSO41108 deleted the codex/eval-inventory-cases branch July 19, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants