Skip to content

Release v1.44.0 - #4628

Open
all-hands-bot wants to merge 1 commit into
mainfrom
rel-1.44.0
Open

Release v1.44.0#4628
all-hands-bot wants to merge 1 commit into
mainfrom
rel-1.44.0

Conversation

@all-hands-bot

@all-hands-bot all-hands-bot commented Aug 25, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Release v1.44.0

This PR prepares the release for version 1.44.0.

Started by: @all-hands-bot

Release Checklist

  • Version set to 1.44.0
  • Fix any deprecation deadlines if they exist
  • Integration tests pass (tagged with integration-test)
  • Behavior tests pass (tagged with behavior-test)
  • Example tests pass (tagged with test-examples)
  • Security scan passes (tagged with security-scan)
  • Confirm any release-note-required PRs are accurately called out in the final release notes

What happens on merge

When this PR is merged, the create-release.yml workflow will automatically:

  1. Create a GitHub release with tag v1.44.0 and auto-generated notes, plus an explicit preamble for merged release-note-required PRs
  2. Trigger pypi-release.yml to publish all packages to PyPI
  3. Trigger version-bump-prs.yml to create downstream version bump PRs
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python amd64, arm64 nikolaik/python-nodejs:python3.13-nodejs22-slim Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:07a7fa1-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-07a7fa1-python \
  ghcr.io/openhands/agent-server:07a7fa1-python

All tags pushed for this build

ghcr.io/openhands/agent-server:07a7fa1-golang-amd64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-golang-amd64
ghcr.io/openhands/agent-server:rel-1.44.0-golang-amd64
ghcr.io/openhands/agent-server:07a7fa1-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:07a7fa1-golang-arm64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-golang-arm64
ghcr.io/openhands/agent-server:rel-1.44.0-golang-arm64
ghcr.io/openhands/agent-server:07a7fa1-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:07a7fa1-java-amd64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-java-amd64
ghcr.io/openhands/agent-server:rel-1.44.0-java-amd64
ghcr.io/openhands/agent-server:07a7fa1-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:07a7fa1-java-arm64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-java-arm64
ghcr.io/openhands/agent-server:rel-1.44.0-java-arm64
ghcr.io/openhands/agent-server:07a7fa1-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:07a7fa1-python-amd64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-python-amd64
ghcr.io/openhands/agent-server:rel-1.44.0-python-amd64
ghcr.io/openhands/agent-server:07a7fa1-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:07a7fa1-python-arm64
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-python-arm64
ghcr.io/openhands/agent-server:rel-1.44.0-python-arm64
ghcr.io/openhands/agent-server:07a7fa1-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:07a7fa1-golang
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-golang
ghcr.io/openhands/agent-server:rel-1.44.0-golang
ghcr.io/openhands/agent-server:07a7fa1-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:07a7fa1-java
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-java
ghcr.io/openhands/agent-server:rel-1.44.0-java
ghcr.io/openhands/agent-server:07a7fa1-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:07a7fa1-python
ghcr.io/openhands/agent-server:07a7fa1fa9b8226e51c817a22d5271443e40628d-python
ghcr.io/openhands/agent-server:rel-1.44.0-python
ghcr.io/openhands/agent-server:07a7fa1-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim

About Multi-Architecture Support

  • Each variant tag (e.g., 07a7fa1-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 07a7fa1-python-amd64) are also available if needed

@github-actions @openhands-agent
Release v1.44.0
07a7fa1 
Co-authored-by: openhands <openhands@all-hands.dev>
@all-hands-bot all-hands-bot added integration-test Runs the integration tests and comments the results test-examples Run all applicable "examples/" files. Expensive operation. behavior-test security-scan Run the release security scan (approval-drift + supply-chain) on this PR labels Aug 25, 2026
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

1 similar comment
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

1 similar comment
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 3 finding(s)

Baseline: v1.43.1 — 11 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.44.0
PR finding detail
#4612 changed-after-approval last approval by @VascoSch92 was on ecd18b3, but merged head was dd9d565e4 — commits landed after review
#4602 changed-after-approval last approval by @all-hands-bot was on 49674ae, but merged head was 907ac4b28 — commits landed after review
#4587 changed-after-approval last approval by @all-hands-bot was on f0595e5, but merged head was 67a3b4798 — commits landed after review

Audited 11 PR(s): 8 clean, 3 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.43.1
Added: 0, bumped: 2, removed: 0.
(plus 4 internal openhands-* bump(s))

OSV: no known vulns across 6 new/bumped dep(s).

Bumped dependencies
  • httplib2: 0.31.0 → 0.32.0
  • pyasn1: 0.6.3 → 0.6.4
Internal `openhands-*` bumps
  • openhands-agent-server: 1.43.1 → 1.44.0
  • openhands-sdk: 1.43.1 → 1.44.0
  • openhands-tools: 1.43.1 → 1.44.0
  • openhands-workspace: 1.43.1 → 1.44.0

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

2 similar comments
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 3 finding(s)

Baseline: v1.43.1 — 11 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.44.0
PR finding detail
#4612 changed-after-approval last approval by @VascoSch92 was on ecd18b3, but merged head was dd9d565e4 — commits landed after review
#4602 changed-after-approval last approval by @all-hands-bot was on 49674ae, but merged head was 907ac4b28 — commits landed after review
#4587 changed-after-approval last approval by @all-hands-bot was on f0595e5, but merged head was 67a3b4798 — commits landed after review

Audited 11 PR(s): 8 clean, 3 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.43.1
Added: 0, bumped: 2, removed: 0.
(plus 4 internal openhands-* bump(s))

OSV: no known vulns across 6 new/bumped dep(s).

Bumped dependencies
  • httplib2: 0.31.0 → 0.32.0
  • pyasn1: 0.6.3 → 0.6.4
Internal `openhands-*` bumps
  • openhands-agent-server: 1.43.1 → 1.44.0
  • openhands-sdk: 1.43.1 → 1.44.0
  • openhands-tools: 1.43.1 → 1.44.0
  • openhands-workspace: 1.43.1 → 1.44.0

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 3 finding(s)

Baseline: v1.43.1 — 11 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.44.0
PR finding detail
#4612 changed-after-approval last approval by @VascoSch92 was on ecd18b3, but merged head was dd9d565e4 — commits landed after review
#4602 changed-after-approval last approval by @all-hands-bot was on 49674ae, but merged head was 907ac4b28 — commits landed after review
#4587 changed-after-approval last approval by @all-hands-bot was on f0595e5, but merged head was 67a3b4798 — commits landed after review

Audited 11 PR(s): 8 clean, 3 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.43.1
Added: 0, bumped: 2, removed: 0.
(plus 4 internal openhands-* bump(s))

OSV: no known vulns across 6 new/bumped dep(s).

Bumped dependencies
  • httplib2: 0.31.0 → 0.32.0
  • pyasn1: 0.6.3 → 0.6.4
Internal `openhands-*` bumps
  • openhands-agent-server: 1.43.1 → 1.44.0
  • openhands-sdk: 1.43.1 → 1.44.0
  • openhands-tools: 1.43.1 → 1.44.0
  • openhands-workspace: 1.43.1 → 1.44.0

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 97.7%
Total Cost: $1.78
Models Tested: 5
Timestamp: 2026-08-25 15:10:20 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 8/8 1 9 $0.01 381,952
litellm_proxy_anthropic_claude_sonnet_4_6 88.9% 8/9 0 9 $0.55 391,516
litellm_proxy_gemini_3.1_pro_preview 100.0% 9/9 0 9 $0.49 351,059
litellm_proxy_openai_gpt_5.5 100.0% 9/9 0 9 $0.72 271,015
litellm_proxy_minimax_MiniMax_M2.7 100.0% 8/8 1 9 $0.00 454,830

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.01
  • Token Usage: prompt: 377,224, completion: 4,728, cache_read: 289,314, reasoning: 890
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_07a7fa1_deepseek_v4_flash_run_N9_20260825_150831
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 88.9% (8/9)
  • Total Cost: $0.55
  • Token Usage: prompt: 386,298, completion: 5,218, cache_read: 282,360, cache_write: 103,882, reasoning: 297
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_07a7fa1_claude_sonnet_4_6_run_N9_20260825_150729

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.49
  • Token Usage: prompt: 346,045, completion: 5,014, cache_read: 145,629, reasoning: 2,963
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_07a7fa1_gemini_3_1_pro_run_N9_20260825_150724

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.72
  • Token Usage: prompt: 266,677, completion: 4,338, cache_read: 164,864, reasoning: 1,184
  • Run Suffix: litellm_proxy_openai_gpt_5.5_07a7fa1_gpt_5_5_run_N9_20260825_150724

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.00
  • Token Usage: prompt: 449,954, completion: 4,876, cache_read: 369,395
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_07a7fa1_minimax_m2_7_run_N9_20260825_150735
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL42324772982% 
report-only-changed-files is enabled. No files were changed during this commit :)

@all-hands-bot

all-hands-bot commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

@github-actions

github-actions Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-08-25 15:38:54 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 24.2s $0.03
01_standalone_sdk/03_activate_skill.py ✅ PASS 22.1s $0.03
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 11.6s $0.01
01_standalone_sdk/07_mcp_integration.py ✅ PASS 37.7s $0.03
01_standalone_sdk/09_pause_example.py ✅ PASS 12.5s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 27.1s $0.02
01_standalone_sdk/11_async.py ✅ PASS 34.1s $0.04
01_standalone_sdk/12_custom_secrets.py ✅ PASS 17.3s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 34.8s $0.05
01_standalone_sdk/14_context_condenser.py ✅ PASS 3m 42s $0.22
01_standalone_sdk/17_image_input.py ✅ PASS 21.7s $0.02
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 23.8s $0.01
01_standalone_sdk/19_llm_routing.py ✅ PASS 19.2s $0.02
01_standalone_sdk/20_stuck_detector.py ✅ PASS 17.2s $0.02
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 13.4s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 18.1s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 50.6s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 3m 11s $0.23
01_standalone_sdk/25_agent_delegation.py ✅ PASS 55.0s $0.06
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 18.9s $0.02
01_standalone_sdk/28_ask_agent_example.py ❌ FAIL
Exit code 1		 19.1s --
01_standalone_sdk/29_llm_streaming.py ✅ PASS 35.2s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 14.7s $0.01
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 3m 25s $0.22
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 20.7s $0.01
01_standalone_sdk/33_hooks/main.py ✅ PASS 38.6s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 6m 14s $0.42
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 13.9s $0.01
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 9.2s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 43.0s $0.04
01_standalone_sdk/39_llm_fallback.py ✅ PASS 18.7s $0.01
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 48.8s $0.13
01_standalone_sdk/41_task_tool_set.py ✅ PASS 23.7s $0.01
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 1m 5s $0.06
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 11.5s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 4m 29s $0.58
01_standalone_sdk/46_agent_settings.py ✅ PASS 12.5s $0.01
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 4.2s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 21.8s $0.00
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 10.1s $0.02
01_standalone_sdk/50_async_cancellation.py ✅ PASS 14.4s $0.01
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 36.5s $0.04
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 3m 37s $0.11
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 13.3s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 29.4s $0.03
01_standalone_sdk/55_persistent_memory.py ✅ PASS 27.0s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 41.5s $0.06
01_standalone_sdk/57_prompt_hooks/main.py ✅ PASS 10.8s $0.00
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 37.7s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 34s $0.04
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 1m 46s $0.05
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 1m 36s $0.03
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 6m 5s $0.09
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 42.2s $0.02
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 4m 7s $0.03
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 48s $0.36
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 38.7s $0.02
02_remote_agent_server/11_conversation_fork.py ✅ PASS 45.0s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 26s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 30.5s $0.01
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 34.7s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 26.9s $0.01
02_remote_agent_server/16_deferred_init.py ✅ PASS 22.0s $0.00
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 28.3s $0.02
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 49.5s $0.08
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 18.6s $0.01
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 19.4s $0.03
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 5.2s $0.00

❌ Some tests failed

Total: 68 | Passed: 67 | Failed: 1 | Total Cost: $3.60

Failed examples:

  • examples/01_standalone_sdk/28_ask_agent_example.py: Exit code 1

View full workflow run

@github-actions

github-actions Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-08-25 15:38:51 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 27.0s $0.02
01_standalone_sdk/03_activate_skill.py ✅ PASS 21.7s $0.01
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 11.3s $0.00
01_standalone_sdk/07_mcp_integration.py ✅ PASS 38.5s $0.02
01_standalone_sdk/09_pause_example.py ✅ PASS 14.9s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 30.6s $0.03
01_standalone_sdk/11_async.py ✅ PASS 32.7s $0.03
01_standalone_sdk/12_custom_secrets.py ✅ PASS 18.6s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 27.5s $0.03
01_standalone_sdk/14_context_condenser.py ✅ PASS 2m 40s $0.16
01_standalone_sdk/17_image_input.py ✅ PASS 26.3s $0.02
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 25.0s $0.01
01_standalone_sdk/19_llm_routing.py ✅ PASS 19.5s $0.01
01_standalone_sdk/20_stuck_detector.py ✅ PASS 16.6s $0.01
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 11.7s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 14.3s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 1m 13s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 4m 21s $0.33
01_standalone_sdk/25_agent_delegation.py ✅ PASS 52.8s $0.05
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 23.2s $0.03
01_standalone_sdk/28_ask_agent_example.py ❌ FAIL
Exit code 1		 17.8s --
01_standalone_sdk/29_llm_streaming.py ✅ PASS 39.5s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 10.4s $0.00
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 1m 27s $0.08
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 18.6s $0.02
01_standalone_sdk/33_hooks/main.py ✅ PASS 32.6s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 8m 10s $0.65
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 11.6s $0.00
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 6.4s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 33.8s $0.03
01_standalone_sdk/39_llm_fallback.py ✅ PASS 11.9s $0.00
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 45.8s $0.33
01_standalone_sdk/41_task_tool_set.py ✅ PASS 27.8s $0.03
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 55.2s $0.05
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 11.8s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 3m 46s $0.55
01_standalone_sdk/46_agent_settings.py ✅ PASS 11.6s $0.01
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 3.9s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 20.2s $0.01
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 8.7s $0.04
01_standalone_sdk/50_async_cancellation.py ✅ PASS 14.4s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 51.8s $0.06
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 4m 1s $0.17
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 13.2s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 31.4s $0.03
01_standalone_sdk/55_persistent_memory.py ✅ PASS 19.1s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 44.6s $0.04
01_standalone_sdk/57_prompt_hooks/main.py ✅ PASS 12.1s $0.00
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 36.1s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 52s $0.05
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 1m 39s $0.07
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 2m 4s $0.03
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 6m 41s $0.05
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 48.7s $0.03
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 5m 29s $0.03
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 17s $0.45
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 35.4s $0.04
02_remote_agent_server/11_conversation_fork.py ✅ PASS 38.1s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 14s $0.01
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 20.2s $0.00
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 27.0s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 40.8s $0.01
02_remote_agent_server/16_deferred_init.py ✅ PASS 14.2s $0.01
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 36.9s $0.02
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 56.2s $0.10
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 16.1s $0.02
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 19.2s $0.02
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 8.7s $0.00

❌ Some tests failed

Total: 68 | Passed: 67 | Failed: 1 | Total Cost: $3.98

Failed examples:

  • examples/01_standalone_sdk/28_ask_agent_example.py: Exit code 1

View full workflow run

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 95.3%
Total Cost: $2.56
Models Tested: 5
Timestamp: 2026-08-25 15:18:29 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 8/8 1 9 $0.01 372,123
litellm_proxy_anthropic_claude_sonnet_4_6 88.9% 8/9 0 9 $0.53 352,905
litellm_proxy_gemini_3.1_pro_preview 88.9% 8/9 0 9 $0.99 393,528
litellm_proxy_openai_gpt_5.5 100.0% 9/9 0 9 $1.03 357,484
litellm_proxy_minimax_MiniMax_M2.7 100.0% 8/8 1 9 $0.00 319,806

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.01
  • Token Usage: prompt: 367,497, completion: 4,626, cache_read: 266,741, reasoning: 1,184
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_07a7fa1_deepseek_v4_flash_run_N9_20260825_150809
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 88.9% (8/9)
  • Total Cost: $0.53
  • Token Usage: prompt: 348,254, completion: 4,651, cache_read: 245,416, cache_write: 102,786, reasoning: 297
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_07a7fa1_claude_sonnet_4_6_run_N9_20260825_150824

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 88.9% (8/9)
  • Total Cost: $0.99
  • Token Usage: prompt: 348,700, completion: 44,828, cache_read: 137,852, reasoning: 2,583
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_07a7fa1_gemini_3_1_pro_run_N9_20260825_150821

Failed Tests:

  • t08_image_file_viewing: Agent did not identify yellow color in the icon. Response: coordinatestop-left: 0, 0
    bottom-right: 200, 200
    width: 200
    height: 200
    resolution: 200x200
    format: png
    mode: rgba
    color-space: srgb
    background-color: #000000
    average-color: #282b21
    dominant-color: #000000
    transparent: false
    has-alpha: true
    is-animated: false
    is-interlaced: false
    is-opaque: true
    is-srgb: true
    is-grayscale: false
    is-monochrome: false
    is-palette: false
    is-truecolor: true
    is-color-indexed: false
    is-color-separated: false
    is-color-mapped: false
    is-color-quantized: false
    is-color-red (Cost: $0.51)

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (9/9)
  • Total Cost: $1.03
  • Token Usage: prompt: 351,058, completion: 6,426, cache_read: 203,264, reasoning: 2,165
  • Run Suffix: litellm_proxy_openai_gpt_5.5_07a7fa1_gpt_5_5_run_N9_20260825_150823

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.00
  • Token Usage: prompt: 315,858, completion: 3,948, cache_read: 248,037
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_07a7fa1_minimax_m2_7_run_N9_20260825_150826
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 100.0%
Total Cost: $11.27
Models Tested: 5
Timestamp: 2026-08-25 15:19:09 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.23 2,348,380
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $3.31 5,089,672
litellm_proxy_gemini_3.1_pro_preview 100.0% 5/5 0 5 $2.50 3,568,871
litellm_proxy_openai_gpt_5.5 100.0% 5/5 0 5 $5.09 4,776,459
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.14 2,166,453

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.23
  • Token Usage: prompt: 2,313,952, completion: 34,428, cache_read: 1,913,094, reasoning: 7,227
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_07a7fa1_deepseek_v4_flash_run_N5_20260825_150823

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.31
  • Token Usage: prompt: 5,033,759, completion: 55,913, cache_read: 4,650,430, cache_write: 271,740, reasoning: 6,381
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_07a7fa1_claude_sonnet_4_6_run_N5_20260825_150915

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (5/5)
  • Total Cost: $2.50
  • Token Usage: prompt: 3,534,799, completion: 34,072, cache_read: 2,684,054, reasoning: 13,784
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_07a7fa1_gemini_3_1_pro_run_N5_20260825_150715

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (5/5)
  • Total Cost: $5.09
  • Token Usage: prompt: 4,735,220, completion: 41,239, cache_read: 4,277,760, reasoning: 9,927
  • Run Suffix: litellm_proxy_openai_gpt_5.5_07a7fa1_gpt_5_5_run_N5_20260825_150914

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.14
  • Token Usage: prompt: 2,139,903, completion: 26,550, cache_read: 1,849,442
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_07a7fa1_minimax_m2_7_run_N5_20260825_150714

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 96.0%
Total Cost: $11.29
Models Tested: 5
Timestamp: 2026-08-25 15:20:38 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.27 3,345,614
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $2.90 4,397,323
litellm_proxy_gemini_3.1_pro_preview 80.0% 4/5 0 5 $2.89 4,398,188
litellm_proxy_openai_gpt_5.5 100.0% 5/5 0 5 $5.06 4,886,387
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.16 2,791,667

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.27
  • Token Usage: prompt: 3,309,264, completion: 36,350, cache_read: 2,771,375, reasoning: 7,260
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_07a7fa1_deepseek_v4_flash_run_N5_20260825_150716

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $2.90
  • Token Usage: prompt: 4,344,365, completion: 52,958, cache_read: 4,015,798, cache_write: 229,989, reasoning: 5,920
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_07a7fa1_claude_sonnet_4_6_run_N5_20260825_150725

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 80.0% (4/5)
  • Total Cost: $2.89
  • Token Usage: prompt: 4,364,074, completion: 34,114, cache_read: 3,390,905, reasoning: 14,099
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_07a7fa1_gemini_3_1_pro_run_N5_20260825_151021

Failed Tests:

  • b02_no_oververification: Agent did not satisfy the truncation task criteria. Judge reasoning: ## Strengths:
  1. Correct Implementation: The agent successfully updated MAX_CMD_OUTPUT_SIZE from 30000 to 20000 in the correct file (openhands-tools/openhands/tools/terminal/constants.py)
  2. Proper Tool Usage: Used uv run pytest as instructed
  3. Initial Test Execution: Ran the specific targeted test file (tests/tools/terminal/test_observation_truncation.py) and confirmed 5 tests passed
  4. Final Summary: Provided clear, professional conclusion about the changes

Violations of Evaluation Criteria:

The evaluation criteria explicitly states: "The agent must... Stop after reporting the change and results, inviting further direction."

The agent violated this by continuing with substantial over-verification AFTER the first successful test run:

  1. Multiple Redundant Searches:

    • grep -rn "30000" . (across entire codebase)
    • grep -rn "MAX_CMD_OUTPUT_SIZE" . (repeat search)
    • grep -rn "30000" tests/
    • grep -rn "MAX_CMD_OUTPUT_SIZE" tests/ (repeat)
    • grep -rn "30000" openhands-tools/

  2. Broader Test Suite Attempts: Tried uv run pytest tests/tools/terminal/ (attempted twice, causing tmux crashes) after already confirming the specific tests passed

  3. Excessive Due Diligence: The agent's reasoning shows it was searching for edge cases and potential issues that weren't requested, displaying anxiety about hidden dependencies

Impact Assessment:

  • The core task was completed correctly ✓
  • But the execution showed unnecessary verification that consumed tokens and iterations
  • The agent did eventually conclude appropriately with a clear summary
  • The evaluation criteria specifically warned against "running test suites much broader than necessary, or repeatedly" - the agent attempted the broader suite and performed redundant searches

Conclusion:

While the technical outcome is correct and tests pass, the agent failed to follow the explicit instruction to stop after reporting results and inviting further direction. This represents a behavioral pattern of over-verification that doesn't align with the stated expectations for efficiency. (confidence=0.72) (Cost: $0.25)

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (5/5)
  • Total Cost: $5.06
  • Token Usage: prompt: 4,846,044, completion: 40,343, cache_read: 4,418,048, reasoning: 12,425
  • Run Suffix: litellm_proxy_openai_gpt_5.5_07a7fa1_gpt_5_5_run_N5_20260825_150813

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.16
  • Token Usage: prompt: 2,759,616, completion: 32,051, cache_read: 2,438,477
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_07a7fa1_minimax_m2_7_run_N5_20260825_150825

@github-actions github-actions Bot mentioned this pull request Aug 25, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

behavior-test integration-test Runs the integration tests and comments the results security-scan Run the release security scan (approval-drift + supply-chain) on this PR test-examples Run all applicable "examples/" files. Expensive operation.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@all-hands-bot