Repository navigation
Release v1.54.0 - #5609
Merged
Merged
Release v1.54.0#5609
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
all-hands-bot
added
integration-test
Contributor
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
Contributor
🔒 Release Security Scan🔒 Approval drift (time-of-check vs time-of-use)❌ 7 finding(s) Baseline:
Audited 21 PR(s): 14 clean, 7 flagged, 0 un-auditable. 📦 Supply-chain dependency diff✅ no findings Baseline: OSV: no known vulns across 6 new/bumped dep(s). Bumped dependencies
Internal `openhands-*` bumps
Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed. |
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Contributor
|
Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly. |
Contributor
🧪 Integration Tests ResultsOverall Success Rate: 100.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
Coverage Report •
|
Contributor
🔄 Running Examples with
|
| Example | Status | Duration | Cost |
|---|---|---|---|
| 01_standalone_sdk/02_custom_tools.py | ✅ PASS | 32.1s | $0.03 |
| 01_standalone_sdk/03_activate_skill.py | ✅ PASS | 23.4s | $0.03 |
| 01_standalone_sdk/05_use_llm_registry.py | ✅ PASS | 10.7s | $0.01 |
| 01_standalone_sdk/07_mcp_integration.py | ✅ PASS | 35.2s | $0.03 |
| 01_standalone_sdk/09_pause_example.py | ✅ PASS | 13.3s | $0.01 |
| 01_standalone_sdk/10_persistence.py | ✅ PASS | 26.0s | $0.02 |
| 01_standalone_sdk/11_async.py | ✅ PASS | 28.8s | $0.03 |
| 01_standalone_sdk/12_custom_secrets.py | ✅ PASS | 13.1s | $0.01 |
| 01_standalone_sdk/13_get_llm_metrics.py | ✅ PASS | 28.8s | $0.04 |
| 01_standalone_sdk/14_context_condenser.py | ✅ PASS | 3m 3s | $0.18 |
| 01_standalone_sdk/17_image_input.py | ✅ PASS | 23.3s | $0.02 |
| 01_standalone_sdk/18_send_message_while_processing.py | ✅ PASS | 29.7s | $0.02 |
| 01_standalone_sdk/19_llm_routing.py | ✅ PASS | 16.5s | $0.02 |
| 01_standalone_sdk/20_stuck_detector.py | ✅ PASS | 18.4s | $0.02 |
| 01_standalone_sdk/21_generate_extraneous_conversation_costs.py | ✅ PASS | 11.5s | $0.00 |
| 01_standalone_sdk/22_anthropic_thinking.py | ✅ PASS | 18.7s | $0.01 |
| 01_standalone_sdk/23_responses_reasoning.py | ✅ PASS | 38.8s | $0.01 |
| 01_standalone_sdk/24_planning_agent_workflow.py | ✅ PASS | 4m 32s | $0.29 |
| 01_standalone_sdk/25_agent_delegation.py | ✅ PASS | 1m 26s | $0.07 |
| 01_standalone_sdk/26_custom_visualizer.py | ✅ PASS | 21.4s | $0.03 |
| 01_standalone_sdk/28_ask_agent_example.py | ✅ PASS | 29.9s | $0.04 |
| 01_standalone_sdk/29_llm_streaming.py | ✅ PASS | 36.7s | $0.02 |
| 01_standalone_sdk/30_tom_agent.py | ✅ PASS | 10.5s | $0.01 |
| 01_standalone_sdk/31_iterative_refinement.py | ✅ PASS | 3m 55s | $0.22 |
| 01_standalone_sdk/32_configurable_security_policy.py | ✅ PASS | 21.4s | $0.02 |
| 01_standalone_sdk/33_hooks/main.py | ✅ PASS | 37.9s | $0.04 |
| 01_standalone_sdk/34_critic_example.py | ✅ PASS | 24.1s | $0.02 |
| 01_standalone_sdk/36_event_json_to_openai_messages.py | ✅ PASS | 12.3s | $0.01 |
| 01_standalone_sdk/37_llm_profile_store/main.py | ✅ PASS | 12.8s | $0.00 |
| 01_standalone_sdk/38_browser_session_recording.py | ✅ PASS | 47.7s | $0.04 |
| 01_standalone_sdk/39_llm_fallback.py | ✅ PASS | 13.0s | $0.01 |
| 01_standalone_sdk/40_acp_agent_example.py | ✅ PASS | 54.4s | $0.37 |
| 01_standalone_sdk/41_task_tool_set.py | ✅ PASS | 27.0s | $0.03 |
| 01_standalone_sdk/42_file_based_subagents.py | ✅ PASS | 47.3s | $0.05 |
| 01_standalone_sdk/44_model_switching_in_convo.py | ✅ PASS | 11.0s | $0.01 |
| 01_standalone_sdk/45_parallel_tool_execution.py | ✅ PASS | 3m 18s | $0.38 |
| 01_standalone_sdk/46_agent_settings.py | ✅ PASS | 12.4s | $0.01 |
| 01_standalone_sdk/47_defense_in_depth_security.py | ✅ PASS | 4.4s | $0.00 |
| 01_standalone_sdk/48_conversation_fork.py | ✅ PASS | 22.5s | $0.01 |
| 01_standalone_sdk/49_switch_llm_tool.py | ✅ PASS | 8.8s | $0.04 |
| 01_standalone_sdk/50_async_cancellation.py | ✅ PASS | 15.2s | $0.00 |
| 01_standalone_sdk/51_agent_hooks/main.py | ✅ PASS | 42.7s | $0.06 |
| 01_standalone_sdk/52_dynamic_workflow.py | ✅ PASS | 1m 56s | $0.14 |
| 01_standalone_sdk/53_client_defined_tools.py | ✅ PASS | 14.2s | $0.01 |
| 01_standalone_sdk/54_goal_completion_loop.py | ✅ PASS | 32.1s | $0.03 |
| 01_standalone_sdk/55_persistent_memory.py | ✅ PASS | 19.9s | $0.02 |
| 01_standalone_sdk/56_structured_output.py | ✅ PASS | 50.0s | $0.06 |
| 01_standalone_sdk/57_prompt_hooks/main.py | ✅ PASS | 12.8s | $0.00 |
| 01_standalone_sdk/58_ask_oracle_tool/main.py | ✅ PASS | 16.0s | $0.01 |
| 01_standalone_sdk/59_route_task_to_model.py | ✅ PASS | 12.8s | $0.05 |
| 02_remote_agent_server/01_convo_with_local_agent_server.py | ✅ PASS | 39.9s | $0.02 |
| 02_remote_agent_server/02_convo_with_docker_sandboxed_server.py | ✅ PASS | 1m 21s | $0.03 |
| 02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py | ✅ PASS | 2m 47s | $0.15 |
| 02_remote_agent_server/04_convo_with_api_sandboxed_server.py | ✅ PASS | 1m 15s | $0.03 |
| 02_remote_agent_server/06_custom_tool/main.py | ✅ PASS | 5m 40s | $0.05 |
| 02_remote_agent_server/07_convo_with_cloud_workspace.py | ❌ FAIL Exit code 1 |
37.0s | -- |
| 02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py | ✅ PASS | 3m 46s | $0.02 |
| 02_remote_agent_server/09_acp_agent_with_remote_runtime.py | ✅ PASS | 1m 45s | $0.45 |
| 02_remote_agent_server/10_cloud_workspace_share_credentials.py | ❌ FAIL Exit code 1 |
35.6s | -- |
| 02_remote_agent_server/11_conversation_fork.py | ✅ PASS | 49.9s | $0.00 |
| 02_remote_agent_server/12_settings_and_secrets_api.py | ✅ PASS | 2m 23s | $0.01 |
| 02_remote_agent_server/13_workspace_get_llm.py | ✅ PASS | 32.5s | $0.01 |
| 02_remote_agent_server/14_client_defined_tools.py | ✅ PASS | 40.4s | $0.02 |
| 02_remote_agent_server/15_openai_compatible_gateway.py | ✅ PASS | 23.5s | $0.02 |
| 02_remote_agent_server/16_deferred_init.py | ✅ PASS | 24.1s | $0.01 |
| 04_llm_specific_tools/01_gpt5_apply_patch_preset.py | ✅ PASS | 38.5s | $0.03 |
| 04_llm_specific_tools/02_gemini_file_tools.py | ✅ PASS | 47.2s | $0.09 |
| 05_skills_and_plugins/01_loading_agentskills/main.py | ✅ PASS | 17.0s | $0.02 |
| 05_skills_and_plugins/02_loading_plugins/main.py | ✅ PASS | 22.1s | $0.03 |
| 05_skills_and_plugins/04_mixed_marketplace_skills/main.py | ✅ PASS | 9.8s | $0.00 |
❌ Some tests failed
Total: 70 | Passed: 68 | Failed: 2 | Total Cost: $3.59
Failed examples:
- examples/02_remote_agent_server/07_convo_with_cloud_workspace.py: Exit code 1
- examples/02_remote_agent_server/10_cloud_workspace_share_credentials.py: Exit code 1
Co-authored-by: openhands <openhands@all-hands.dev>
Member
|
Merged current main (c980344) into this release branch and regenerated the TypeScript contract in 298227f. The earlier agent-server-api failure occurred because CI tested the synthetic merge containing the new InstalledCanvasExtensionResponse.requested_ref field while the prepared contract predated that main change. Generation followed by check:agent-server-api now passes, as do all 44 canvas_extensions_router tests and applicable pre-commit hooks. Fresh CI is running. The approval-drift security gate remains unresolved; no gate has been bypassed. Publication and merging remain pending user approval. This comment was generated by an AI agent (OpenHands) on behalf of the user. |
Contributor
🧪 Behavior Tests ResultsOverall Success Rate: 96.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
Failed Tests:
What went well:
What went wrong (over-verification):
Critical instruction violation: The agent's behavior suggests excessive caution/thoroughness rather than appropriate scoping and confidence in the straightforward change made. (confidence=0.68) (Cost: $0.27) litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
|
Contributor
|
Test Suites Triggered
Results will be posted here when complete. |
Member
|
Release checklist follow-up:
Approval-drift triage (the security gate remains blocked; no waiver or approval is being manufactured):
The current audit compares exact approved and merged head SHAs, so explanatory comments alone do not turn this gate green. A maintainer must review the drift and determine the supported resolution; the guard has not been weakened. This audit comment was generated by an AI agent (OpenHands) on behalf of the user. |
Contributor
🧪 Integration Tests ResultsOverall Success Rate: 92.6% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
Skipped Tests:
Failed Tests:
litellm_proxy_gemini_3.1_pro_preview
Skipped Tests:
Failed Tests:
What the agent did:
Critical issues:
Positive aspects:
Verdict: While the agent took a reasonable initial approach and provided explanations, it failed to progress beyond the search phase and became stuck in unproductive repetition. The actual version bumping task was never attempted. (confidence=0.75) (Cost: $0.27)
litellm_proxy_anthropic_claude_sonnet_4_6
Failed Tests:
litellm_proxy_deepseek_deepseek_v4_flash
Skipped Tests:
Failed Tests:
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
Failed Tests:
|
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Member
|
Merged main into the release branch in 3ba7766, then regenerated the source-derived TypeScript API contract after CI identified drift. The generated change is only the ProfileDetailResponse documentation reflecting the merged X-Expose-Secrets behavior. Source OpenAPI export, generate:agent-server-api, check:agent-server-api, and git diff --check passed locally. Fresh PR CI will validate the updated head. Release integration/behavior/examples and image security validation remain pending; this does not mark those gates passed. Graham reports he personally reviewed the previously flagged approval drift and found it acceptable; the automated audit has not been bypassed or reclassified by this comment. This comment was generated by an AI agent (OpenHands) on behalf of Graham Neubig. |
7 tasks
Co-authored-by: openhands <openhands@all-hands.dev>
Member
|
Combined-candidate validation: locally merged release head 13a6ade with #5670 (9079947), #5671 (5f05244), and #5675 (0eaa7a3) in an isolated detached worktree; no release-branch changes or merges of these PRs were pushed.
Full examples CI on #5670 and review of the three focused PRs are still pending. Final release-head validation is required after reviewed fixes land. This comment was generated by an AI agent (OpenHands) on behalf of Graham Neubig. |
Co-authored-by: allhands-bot <allhands-bot@users.noreply.github.com> (cherry picked from commit cd17bd8)
Co-authored-by: openhands <openhands@all-hands.dev>
- keep_first=2 so the first condensation retains the user instruction and the second command still runs. - max_iterations 10 -> 20: a condensation step also counts as an iteration, so the happy path already uses ~4 and the old cap risked MaxIterationsReached. - max_tokens 5000 -> 6000 and payloads 1000 -> 2000 reps so one tool result clears the threshold by a wide margin across the CI model matrix. - verify_result now requires at least two condensations so the redesign's second cycle is actually asserted. Co-authored-by: openhands <openhands@all-hands.dev>
neubig
added
behavior-test
and removed
test-examples
Contributor
|
Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly. |
neubig
added
test-examples
Member
|
Release status update after merging current main. Merged main into the release branch (6 commits: #5673, #5182 security analysis event, #5683, #5670 file-editor newlines, #5579 ACP model discovery, #5671 c02 hard-reset). The merge was clean (no conflicts). Regenerated the Agent Server TypeScript contract ( Release gates re-dispatched on the final head
Remaining blockers to merge: the three label suites to go green and a human disposition on the approval-drift findings. No gate has been bypassed. This comment was generated by an AI agent (OpenHands) on behalf of @gneubig. |
Contributor
🧪 Integration Tests ResultsOverall Success Rate: 100.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
Contributor
🔄 Running Examples with
|
| Example | Status | Duration | Cost |
|---|---|---|---|
| 01_standalone_sdk/02_custom_tools.py | ✅ PASS | 26.8s | $0.03 |
| 01_standalone_sdk/03_activate_skill.py | ✅ PASS | 24.0s | $0.03 |
| 01_standalone_sdk/05_use_llm_registry.py | ✅ PASS | 10.8s | $0.00 |
| 01_standalone_sdk/07_mcp_integration.py | ✅ PASS | 1m 6s | $0.07 |
| 01_standalone_sdk/09_pause_example.py | ✅ PASS | 16.9s | $0.01 |
| 01_standalone_sdk/10_persistence.py | ✅ PASS | 1m 6s | $0.06 |
| 01_standalone_sdk/11_async.py | ✅ PASS | 34.2s | $0.04 |
| 01_standalone_sdk/12_custom_secrets.py | ✅ PASS | 14.0s | $0.01 |
| 01_standalone_sdk/13_get_llm_metrics.py | ✅ PASS | 33.1s | $0.04 |
| 01_standalone_sdk/14_context_condenser.py | ✅ PASS | 3m 0s | $0.19 |
| 01_standalone_sdk/17_image_input.py | ✅ PASS | 24.9s | $0.05 |
| 01_standalone_sdk/18_send_message_while_processing.py | ✅ PASS | 19.6s | $0.01 |
| 01_standalone_sdk/19_llm_routing.py | ✅ PASS | 17.5s | $0.02 |
| 01_standalone_sdk/20_stuck_detector.py | ✅ PASS | 15.5s | $0.02 |
| 01_standalone_sdk/21_generate_extraneous_conversation_costs.py | ✅ PASS | 15.5s | $0.00 |
| 01_standalone_sdk/22_anthropic_thinking.py | ✅ PASS | 16.2s | $0.01 |
| 01_standalone_sdk/23_responses_reasoning.py | ✅ PASS | 40.4s | $0.02 |
| 01_standalone_sdk/24_planning_agent_workflow.py | ✅ PASS | 6m 38s | $0.45 |
| 01_standalone_sdk/25_agent_delegation.py | ✅ PASS | 1m 24s | $0.07 |
| 01_standalone_sdk/26_custom_visualizer.py | ✅ PASS | 21.1s | $0.03 |
| 01_standalone_sdk/28_ask_agent_example.py | ✅ PASS | 33.1s | $0.03 |
| 01_standalone_sdk/29_llm_streaming.py | ✅ PASS | 34.3s | $0.02 |
| 01_standalone_sdk/30_tom_agent.py | ✅ PASS | 11.2s | $0.01 |
| 01_standalone_sdk/31_iterative_refinement.py | ✅ PASS | 1m 22s | $0.08 |
| 01_standalone_sdk/32_configurable_security_policy.py | ✅ PASS | 23.6s | $0.02 |
| 01_standalone_sdk/33_hooks/main.py | ✅ PASS | 42.8s | $0.04 |
| 01_standalone_sdk/34_critic_example.py | ✅ PASS | 31.3s | $0.02 |
| 01_standalone_sdk/36_event_json_to_openai_messages.py | ✅ PASS | 10.6s | $0.00 |
| 01_standalone_sdk/37_llm_profile_store/main.py | ✅ PASS | 11.7s | $0.00 |
| 01_standalone_sdk/38_browser_session_recording.py | ✅ PASS | 54.2s | $0.04 |
| 01_standalone_sdk/39_llm_fallback.py | ✅ PASS | 11.7s | $0.01 |
| 01_standalone_sdk/40_acp_agent_example.py | ✅ PASS | 1m 4s | $0.40 |
| 01_standalone_sdk/41_task_tool_set.py | ✅ PASS | 25.9s | $0.02 |
| 01_standalone_sdk/42_file_based_subagents.py | ✅ PASS | 37.9s | $0.04 |
| 01_standalone_sdk/44_model_switching_in_convo.py | ✅ PASS | 11.1s | $0.01 |
| 01_standalone_sdk/45_parallel_tool_execution.py | ✅ PASS | 4m 12s | $0.49 |
| 01_standalone_sdk/46_agent_settings.py | ✅ PASS | 14.0s | $0.00 |
| 01_standalone_sdk/47_defense_in_depth_security.py | ✅ PASS | 4.1s | $0.00 |
| 01_standalone_sdk/48_conversation_fork.py | ✅ PASS | 21.5s | $0.00 |
| 01_standalone_sdk/49_switch_llm_tool.py | ✅ PASS | 8.5s | $0.04 |
| 01_standalone_sdk/50_async_cancellation.py | ✅ PASS | 14.2s | $0.00 |
| 01_standalone_sdk/51_agent_hooks/main.py | ✅ PASS | 47.5s | $0.04 |
| 01_standalone_sdk/52_dynamic_workflow.py | ✅ PASS | 3m 22s | $0.11 |
| 01_standalone_sdk/53_client_defined_tools.py | ✅ PASS | 15.7s | $0.01 |
| 01_standalone_sdk/54_goal_completion_loop.py | ✅ PASS | 35.3s | $0.04 |
| 01_standalone_sdk/55_persistent_memory.py | ✅ PASS | 18.2s | $0.02 |
| 01_standalone_sdk/56_structured_output.py | ✅ PASS | 51.0s | $0.07 |
| 01_standalone_sdk/57_prompt_hooks/main.py | ✅ PASS | 16.1s | $0.00 |
| 01_standalone_sdk/58_ask_oracle_tool/main.py | ✅ PASS | 24.2s | $0.01 |
| 01_standalone_sdk/59_route_task_to_model.py | ✅ PASS | 9.3s | $0.02 |
| 02_remote_agent_server/01_convo_with_local_agent_server.py | ✅ PASS | 38.3s | $0.02 |
| 02_remote_agent_server/02_convo_with_docker_sandboxed_server.py | ✅ PASS | 1m 8s | $0.03 |
| 02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py | ✅ PASS | 2m 7s | $0.14 |
| 02_remote_agent_server/04_convo_with_api_sandboxed_server.py | ✅ PASS | 1m 18s | $0.04 |
| 02_remote_agent_server/06_custom_tool/main.py | ✅ PASS | 6m 6s | $0.06 |
| 02_remote_agent_server/07_convo_with_cloud_workspace.py | ✅ PASS | 45.3s | $0.04 |
| 02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py | ✅ PASS | 4m 13s | $0.03 |
| 02_remote_agent_server/09_acp_agent_with_remote_runtime.py | ✅ PASS | 1m 52s | $0.17 |
| 02_remote_agent_server/10_cloud_workspace_share_credentials.py | ✅ PASS | 30.4s | $0.00 |
| 02_remote_agent_server/11_conversation_fork.py | ✅ PASS | 39.2s | $0.00 |
| 02_remote_agent_server/12_settings_and_secrets_api.py | ✅ PASS | 2m 16s | $0.01 |
| 02_remote_agent_server/13_workspace_get_llm.py | ✅ PASS | 26.4s | $0.01 |
| 02_remote_agent_server/14_client_defined_tools.py | ✅ PASS | 38.7s | $0.02 |
| 02_remote_agent_server/15_openai_compatible_gateway.py | ✅ PASS | 22.0s | $0.02 |
| 02_remote_agent_server/16_deferred_init.py | ✅ PASS | 22.9s | $0.00 |
| 04_llm_specific_tools/01_gpt5_apply_patch_preset.py | ✅ PASS | 25.4s | $0.02 |
| 04_llm_specific_tools/02_gemini_file_tools.py | ✅ PASS | 36.8s | $0.08 |
| 05_skills_and_plugins/01_loading_agentskills/main.py | ✅ PASS | 16.3s | $0.02 |
| 05_skills_and_plugins/02_loading_plugins/main.py | ✅ PASS | 30.0s | $0.03 |
| 05_skills_and_plugins/04_mixed_marketplace_skills/main.py | ✅ PASS | 7.3s | $0.00 |
✅ All tests passed!
Total: 70 | Passed: 70 | Failed: 0 | Total Cost: $3.51
Contributor
🧪 Behavior Tests ResultsOverall Success Rate: 96.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
Failed Tests:
However, the agent violated the explicit evaluation criteria by creating redundant/unrequested files. During the development process, the agent created
The criteria specification is unambiguous: only litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
|
Contributor
🧪 Behavior Tests ResultsOverall Success Rate: 96.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
Failed Tests:
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
|
…ism' into rel-1.54.0
neubig
added
integration-test
Contributor
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
neubig
removed
the
test-examples
Contributor
|
Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly. |
neubig
added
test-examples
Contributor
🧪 Integration Tests ResultsOverall Success Rate: 100.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
Contributor
🔄 Running Examples with
|
| Example | Status | Duration | Cost |
|---|---|---|---|
| 01_standalone_sdk/02_custom_tools.py | ✅ PASS | 27.3s | $0.03 |
| 01_standalone_sdk/03_activate_skill.py | ✅ PASS | 21.7s | $0.03 |
| 01_standalone_sdk/05_use_llm_registry.py | ✅ PASS | 10.1s | $0.01 |
| 01_standalone_sdk/07_mcp_integration.py | ✅ PASS | 43.3s | $0.03 |
| 01_standalone_sdk/09_pause_example.py | ✅ PASS | 12.6s | $0.01 |
| 01_standalone_sdk/10_persistence.py | ✅ PASS | 30.7s | $0.03 |
| 01_standalone_sdk/11_async.py | ✅ PASS | 33.9s | $0.04 |
| 01_standalone_sdk/12_custom_secrets.py | ✅ PASS | 12.5s | $0.01 |
| 01_standalone_sdk/13_get_llm_metrics.py | ✅ PASS | 22.6s | $0.02 |
| 01_standalone_sdk/14_context_condenser.py | ✅ PASS | 3m 5s | $0.17 |
| 01_standalone_sdk/17_image_input.py | ✅ PASS | 19.7s | $0.02 |
| 01_standalone_sdk/18_send_message_while_processing.py | ✅ PASS | 20.1s | $0.01 |
| 01_standalone_sdk/19_llm_routing.py | ✅ PASS | 16.9s | $0.02 |
| 01_standalone_sdk/20_stuck_detector.py | ✅ PASS | 18.7s | $0.03 |
| 01_standalone_sdk/21_generate_extraneous_conversation_costs.py | ✅ PASS | 10.9s | $0.00 |
| 01_standalone_sdk/22_anthropic_thinking.py | ✅ PASS | 23.7s | $0.02 |
| 01_standalone_sdk/23_responses_reasoning.py | ✅ PASS | 44.9s | $0.02 |
| 01_standalone_sdk/24_planning_agent_workflow.py | ✅ PASS | 5m 11s | $0.33 |
| 01_standalone_sdk/25_agent_delegation.py | ✅ PASS | 1m 1s | $0.06 |
| 01_standalone_sdk/26_custom_visualizer.py | ✅ PASS | 21.0s | $0.03 |
| 01_standalone_sdk/28_ask_agent_example.py | ✅ PASS | 39.2s | $0.04 |
| 01_standalone_sdk/29_llm_streaming.py | ✅ PASS | 44.9s | $0.02 |
| 01_standalone_sdk/30_tom_agent.py | ✅ PASS | 14.8s | $0.02 |
| 01_standalone_sdk/31_iterative_refinement.py | ✅ PASS | 3m 49s | $0.21 |
| 01_standalone_sdk/32_configurable_security_policy.py | ✅ PASS | 19.5s | $0.02 |
| 01_standalone_sdk/33_hooks/main.py | ✅ PASS | 30.2s | $0.04 |
| 01_standalone_sdk/34_critic_example.py | ✅ PASS | 24.8s | $0.02 |
| 01_standalone_sdk/36_event_json_to_openai_messages.py | ✅ PASS | 10.7s | $0.00 |
| 01_standalone_sdk/37_llm_profile_store/main.py | ✅ PASS | 4.4s | $0.00 |
| 01_standalone_sdk/38_browser_session_recording.py | ✅ PASS | 52.4s | $0.03 |
| 01_standalone_sdk/39_llm_fallback.py | ✅ PASS | 11.7s | $0.01 |
| 01_standalone_sdk/40_acp_agent_example.py | ✅ PASS | 51.0s | $0.33 |
| 01_standalone_sdk/41_task_tool_set.py | ✅ PASS | 22.3s | $0.03 |
| 01_standalone_sdk/42_file_based_subagents.py | ✅ PASS | 40.0s | $0.05 |
| 01_standalone_sdk/44_model_switching_in_convo.py | ✅ PASS | 8.5s | $0.00 |
| 01_standalone_sdk/45_parallel_tool_execution.py | ✅ PASS | 2m 43s | $0.36 |
| 01_standalone_sdk/46_agent_settings.py | ✅ PASS | 11.8s | $0.01 |
| 01_standalone_sdk/47_defense_in_depth_security.py | ✅ PASS | 3.4s | $0.00 |
| 01_standalone_sdk/48_conversation_fork.py | ✅ PASS | 19.8s | $0.01 |
| 01_standalone_sdk/49_switch_llm_tool.py | ✅ PASS | 8.1s | $0.04 |
| 01_standalone_sdk/50_async_cancellation.py | ✅ PASS | 14.4s | $0.00 |
| 01_standalone_sdk/51_agent_hooks/main.py | ✅ PASS | 37.4s | $0.05 |
| 01_standalone_sdk/52_dynamic_workflow.py | ✅ PASS | 4m 19s | $0.12 |
| 01_standalone_sdk/53_client_defined_tools.py | ✅ PASS | 17.2s | $0.01 |
| 01_standalone_sdk/54_goal_completion_loop.py | ✅ PASS | 28.1s | $0.02 |
| 01_standalone_sdk/55_persistent_memory.py | ✅ PASS | 17.1s | $0.02 |
| 01_standalone_sdk/56_structured_output.py | ✅ PASS | 42.3s | $0.06 |
| 01_standalone_sdk/57_prompt_hooks/main.py | ✅ PASS | 12.3s | $0.00 |
| 01_standalone_sdk/58_ask_oracle_tool/main.py | ✅ PASS | 16.8s | $0.01 |
| 01_standalone_sdk/59_route_task_to_model.py | ✅ PASS | 8.9s | $0.05 |
| 02_remote_agent_server/01_convo_with_local_agent_server.py | ✅ PASS | 38.6s | $0.02 |
| 02_remote_agent_server/02_convo_with_docker_sandboxed_server.py | ✅ PASS | 1m 21s | $0.04 |
| 02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py | ✅ PASS | 2m 15s | $0.10 |
| 02_remote_agent_server/04_convo_with_api_sandboxed_server.py | ✅ PASS | 1m 46s | $0.06 |
| 02_remote_agent_server/06_custom_tool/main.py | ✅ PASS | 5m 36s | $0.06 |
| 02_remote_agent_server/07_convo_with_cloud_workspace.py | ✅ PASS | 40.4s | $0.05 |
| 02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py | ✅ PASS | 3m 32s | $0.03 |
| 02_remote_agent_server/09_acp_agent_with_remote_runtime.py | ✅ PASS | 1m 7s | $0.41 |
| 02_remote_agent_server/10_cloud_workspace_share_credentials.py | ✅ PASS | 30.6s | $0.00 |
| 02_remote_agent_server/11_conversation_fork.py | ✅ PASS | 46.4s | $0.00 |
| 02_remote_agent_server/12_settings_and_secrets_api.py | ✅ PASS | 2m 20s | $0.01 |
| 02_remote_agent_server/13_workspace_get_llm.py | ✅ PASS | 24.1s | $0.01 |
| 02_remote_agent_server/14_client_defined_tools.py | ✅ PASS | 23.5s | $0.02 |
| 02_remote_agent_server/15_openai_compatible_gateway.py | ✅ PASS | 18.2s | $0.02 |
| 02_remote_agent_server/16_deferred_init.py | ✅ PASS | 13.3s | $0.00 |
| 04_llm_specific_tools/01_gpt5_apply_patch_preset.py | ✅ PASS | 18.3s | $0.02 |
| 04_llm_specific_tools/02_gemini_file_tools.py | ✅ PASS | 48.5s | $0.10 |
| 05_skills_and_plugins/01_loading_agentskills/main.py | ✅ PASS | 19.7s | $0.02 |
| 05_skills_and_plugins/02_loading_plugins/main.py | ✅ PASS | 23.4s | $0.03 |
| 05_skills_and_plugins/04_mixed_marketplace_skills/main.py | ✅ PASS | 4.7s | $0.00 |
✅ All tests passed!
Total: 70 | Passed: 70 | Failed: 0 | Total Cost: $3.50
Contributor
🧪 Behavior Tests ResultsOverall Success Rate: 100.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_minimax_MiniMax_M2.7
|
Member
|
Release gates are green on the current head #5675 (c04 token-condenser determinism) merged into the release branch. The integration-test change ( Label-gated suites on
Only remaining red check: Everything an agent can move is green; merging #5609 now needs a human call on the approval-drift gate. This comment was generated by an AI agent (OpenHands) on behalf of @gneubig. |
neubig
approved these changes
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.54.0
This PR prepares the release for version 1.54.0.
Started by: @neubig
Release Checklist
integration-test) — release integration suite passed on prior head e2947c6; needs revalidation after merging main. The ordinary TypeScript integration check already passes on current head, but is not a substitute for this release suite.behavior-test) — release behavior suite passed on prior head e2947c6; needs revalidation on current head.test-examples) — previous run: 68/70 passed; both failures were HTTP 500 from Cloud/api/v1/sandboxes. A fresh current-head example run has been dispatched; still awaiting a pass.security-scan) — blocked by approval drift. Supply-chain dependency scan passed, but the approval-drift audit flagged fix(agent-server): tolerate state saves while zipping a trajectory #5538, fix(agent-server): omit acp/ credential dir from download-trajectory zip #5551, refactor(sdk): replace settings and tool metadata probing with typed APIs #5357, feat(agent-server): archive runtimes of long-inactive Docker conversations #5429, feat(agent-server): shed gitignored workspace dirs of stopped runtimes above a disk budget #5407, and feat(agent-server): reclaim Docker runtime caches and disable core dumps #5263. Requires resolution/review, not a checkbox override.release-note-requiredPRs are accurately called out in the final release notes — verified the release workflow preamble query returns exactly #5503 since v1.53.0; its accurate release-note entry is staged below. Publication is still pending.Checklist audited against head
298227f40b7319a0f0373aa9bcf72837005418a3. Unchecked items are outstanding release gates, not omitted bookkeeping. Nothing has been merged or published.This checklist was updated by an AI agent (OpenHands) on behalf of the user.
Verified release-note-required entry
What happens on merge
When this PR is merged, the
create-release.ymlworkflow will automatically:v1.54.0and auto-generated notes, plus an explicit preamble for mergedrelease-note-requiredPRsv1.54.0version-bump-prs.ymlafter successful PyPI publication🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:e81e8ea-pythonRun
All tags pushed for this build
About Multi-Architecture Support
e81e8ea-python) is a multi-arch manifest supporting both amd64 and arm64e81e8ea-python-amd64) are also available if needed