feat(dea): add autopush option to model_env and update documentation - #1
Open
graceqi-g wants to merge 364 commits into
Open
feat(dea): add autopush option to model_env and update documentation#1graceqi-g wants to merge 364 commits into
graceqi-g wants to merge 364 commits into
Conversation
…ove host PATH dependency
…date subprocess calls to use stdin=DEVNULL
…e to missing usage data
…ion and support v1.0.5+ binary changes
… pairing and latency tracking logic
…test suite and configuration datasets feat: add AgyCliGenerator support to evaluator and models, including test suite and configuration datasets
…face for CLI-driven models and simplify AgentEvaluator logic
… in testing documentation
… and v1.0.5 compatibility
…n details and redundant verification instructions.
… fields in agy cli
…upport-agy refactor: replace if-chain with dictionary dispatch for model generators
…ons in agy_cli_agent_testing.md
* feat: opt-in function-calling for the Gemini SDK judge Adds a tools/ package (Tool dataclass, registry) and a fetch_url tool that the GeminiGenerator can invoke via the google.genai function-calling loop. Tools are opt-in per judge via a `tools:` YAML list; configs without the key take the existing single-shot codepath unchanged. The fetch_url tool restricts to public HTTPS hosts (SSRF guard via ipaddress on each resolved IP), times out at 10s, caps responses at 50KB, and strips HTML to text. Tool errors are returned as `Error: ...` strings so the model can react rather than the judge crashing. The retry/backoff loop on rate-limit errors is extracted into a single _call_generate_content helper shared by both the single-shot and tool-loop paths. The tool loop is bounded at MAX_TOOL_ITERATIONS=5. Tested: 12 unit tests for fetch_url (scheme, SSRF, size cap, HTML strip, HTTP error, timeout, URL error); 6 unit tests for the Gemini loop (single-shot preserved, unknown tool fails fast, no-call-emit returns text, tool invoked + FunctionResponse threaded back, iteration cap, tool-exception-as-error-string); existing binaryrubricscorer tests still pass. Claude judge tool support deferred to a follow-up. * docs: add judge_tools.md covering opt-in tool use, fetch_url, and how to add tools Documents the feature added in the previous commit: activation via the tools: YAML key, the fetch_url tool behavior and constraints, an end-to-end example wiring it to BinaryRubricScorer for a Beam-version rubric, security notes (HTTPS-only, SSRF guard, known redirect/TOCTOU gaps), and a recipe for adding new tools. * address review bot nits: log HTML extraction failures, drop unused pytest import, assert retry-loop unreachable github-code-quality bot flagged three items on GoogleCloudPlatform#409: 1. fetch_url.py: empty 'except: pass' on HTML extraction now logs at debug with exc_info, with a comment that fallback to raw text is intentional. 2. tools_fetch_url_test.py: unused 'import pytest' removed. 3. gemini.py: _call_generate_content fell off the retry loop with an implicit None return. The loop always returns or raises, but the invariant is now explicit via a raise RuntimeError after the loop so static analyzers do not flag the fallthrough. * docs: add architecture documentation including external dependency graph mapping
…gy_output_format refactor(agy_cli): parse stream-json events instead of transcript logs
…atform#516) Co-authored-by: James Nguyen <jamesamn@google.com>
…nteract Add missing parameter to Interact
…elease-please--branches--main--components--evalbench chore(main): release 1.12.1
- Adds ignore_extra_skills config flag to SkillsTrajectoryMatcher (defaults to False for backward compatibility). - When ignore_extra_skills is set to true, evaluates pure skill coverage (|expected & actual| / |expected|) without penalizing extra activated skills. - Returns 100.0% score when expected_skills is empty or omitted. - Adds comprehensive unit test suite in evalbench/test/skillstrajectorymatcher_test.py.
…atcher - Renamed config option to allow_extra_skills (defaulting to False for backward compatibility). - When allow_extra_skills is set to true, evaluates pure skill coverage (|expected & actual| / |expected|) without penalizing extra activated skills. - When expected_skills is empty, returns 100.0% if actual is empty or if allow_extra_skills is True. - Expanded unit tests in evalbench/test/skillstrajectorymatcher_test.py.
…dd-ignore-extra-skills-config Add allow_extra_skills config option to SkillsTrajectoryMatcher
…cp-readability-gemini-3.1 feat(mcp_readability): move style judge to gemini-3.1-pro with truncation handling
…_name The BigQuery reporter hardcoded its dataset to <project>.evalbench, so every run's configs/results/scores/summary landed in the same four tables. Add an optional reporting.bigquery.table_name key that overrides the dataset segment (defaulting to "evalbench"), letting a run write to <project>.<table_name> to isolate its output. The dashboard query is templated on the same name so its Looker link keeps working against the configured dataset.
…ix_timeout Support timeout configuration for Antigravity (agy) CLI generator
…elease-please--branches--main--components--evalbench chore(main): release 1.13.0
…ix_credentials Fix GCP credential issues in Codex CLI generator for local MCP tools
…eat/bq-configurable-dataset feat: make BigQuery dataset name configurable via reporting.table_name
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for setting
model_env: autopushin DataEngineeringAgent generator configs and updates the documentation.