Add source-backed evaluator CLI flows

[wuman001]

Summary

• Add source-backed evaluator CLI flows for task, answer, and trajectory evaluation.
• Run task inputs through the default CLI main agent when --agent is omitted, then judge the produced state.
• Support trajectory replay, generated trajectory evaluation, answer veto gating, and keep local eval artifacts out of Git.

Test Plan

• pytest tests/docs/test_evaluator_report_docs.py tests/evaluations/test_evaluation_input_sources.py tests/core/test_evaluator_runtime.py tests/core/test_evaluator_top_level_command.py tests/test_slash_commands.py -q
• openspec validate aworld-cli-evaluator-source-run-2026-06-10 --strict
• python -m py_compile aworld/evaluations/sources.py aworld/evaluations/report.py aworld-cli/src/aworld_cli/evaluator_runtime.py aworld-cli/src/aworld_cli/commands/evaluation_cmd.py aworld-cli/src/aworld_cli/top_level_commands/evaluator_cmd.py
• conda run -n aworld_env aworld-cli evaluator --input ~/Documents/task_input.jsonl --kind task --judge-agent eval/trajectory_evaluator/answer_quality_agent.md --out-dir eval/trajectory_evaluator/reports

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive framework-owned evaluation input source layer, state adapters, and a replay runtime harness to support replaying existing outputs (such as task+answer files and AWorld trajectory logs) without re-executing targets. It also adds rollout-owning runtime composition, user simulators, step-level rewards, retry wrappers, environment isolation, and independent repeated trials with pass@k/pass^k metrics. On the CLI side, a source-backed evaluator mode is added along with report schema export, validation, and lifecycle hooks. The review feedback highlights a potential crash in StateCheckGrader.grade due to unhandled exceptions during non-numeric comparisons, a local import that should be moved to the top of the file per PEP 8, and several broken documentation links containing absolute local file paths.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The _compare_values function attempts to convert values to float for comparison operators like >, <, etc. If the actual value from the state is a non-numeric string, this will raise a ValueError which is not handled in this try...except block. This could crash the evaluation for certain state check graders. Consider wrapping the call to _compare_values in its own try...except block to handle potential ValueError or TypeError exceptions gracefully.

        try:
            actual = _resolve_path(sources[self.source], self.path)
            try:
                passed = _compare_values(actual, self.op, self.expected)
                reason = "matched" if passed else f"expected {self.expected!r}, got {actual!r}"
            except (ValueError, TypeError):
                passed = False
                reason = f"incompatible types for comparison: expected comparable to {self.expected!r}, got {actual!r}"
        except KeyError:
            actual = None
            passed = False
            reason = f"missing path: {'.'.join(self.path)}"

[gemini-code-assist]

This from ... import ... statement is inside the run_evaluator_cli function. According to PEP 8, imports should usually be at the top of the file for clarity and to avoid potential issues with repeated imports. If this local import is intended to avoid a circular dependency or for a significant performance optimization, it would be good to add a comment explaining why. Otherwise, please move it to the top of the file for better code organization and readability.

[gemini-code-assist]

The links on lines 110 and 170 use absolute file paths from your local machine (e.g., /Users/wuman/...). These will result in broken links for other users and in the rendered documentation. Please replace them with relative paths.

[gemini-code-assist]

The links in this section use absolute file paths from your local machine. This will cause them to be broken for other users and in the rendered documentation. Please update them to use relative paths.