Skip to content

test(security): verify mixed-batch analyzer verdicts by action ID - #5741

Open
safal207 wants to merge 1 commit into
OpenHands:mainfrom
safal207:tests/security-analysis-mixed-batch
Open

safal207 wants to merge 1 commit into
OpenHands:mainfrom
safal207:tests/security-analysis-mixed-batch

Conversation

@safal207

Copy link
Copy Markdown

HUMAN:


AGENT:

Why

The existing three-action batch test checks the sets of action IDs, but uniform analyzer results let incorrectly associated values pass. Strengthen it so moving one action's risk or details to another action ID fails the test. This is the focused coverage follow-up requested in Arian's comment after #5182.

Summary

  • Use LOW/HIGH/MEDIUM analyzer verdicts and distinct tool_call_id detail markers in the existing batch test.
  • Compare both complete dictionaries against expectations keyed by the actual ActionEvent.id, retain the acting model's LOW self-label invariant, and close the conversation in finally.
  • Change only tests/sdk/agent/test_security_analysis_event.py (+35/-8). No production behavior or public API changes.

Issue Number

Refs #4259; tests-only follow-up to #5182.

How to Test

Validated on Windows LIS, Python 3.13.16, with frozen workspace dependencies at base 7bd668c9ab74f62c85745cd95adb503f53e7228b plus this patch. Set OH_PERSISTENCE_DIR to an isolated writable directory before importing the SDK.

uv run --frozen pre-commit run --files tests/sdk/agent/test_security_analysis_event.py
uv run --frozen pytest tests/sdk/agent/test_security_analysis_event.py
uv run --frozen pytest tests/sdk/security tests/sdk/agent/test_security_analysis_event.py tests/sdk/agent/test_extract_security_risk.py tests/sdk/agent/test_security_policy_integration.py tests/sdk/agent/test_response_dispatch.py tests/sdk/agent/test_async_secret_masking.py
  • All applicable pre-commit hooks passed.
  • Test file: 11 passed; fresh process after both negative controls: 11 passed.
  • Focused suite: 923 passed, 7 skipped, 10 xfailed.
  • Two separate process-local controls rotate only emitted event risk values or only detail values, preserving ID keys, the other dictionary, the analyzer, and policy input. Each produces 10 passed and exactly one expected failure, at the corresponding dictionary assertion. The former key-only predicates still pass on each altered event. Both controls are removed before the restored run; no production files are edited.

Video/Screenshots

No visual changes. Local evidence includes command logs, JUnit XML, before/after file hashes, and the replayable negative-control script. These artifacts are prepared separately from the native tests-only patch.

Design Doc

Not applicable to this small coverage change.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

This validates the Agent.step path with mocked completion and a no-op tool. It establishes test sensitivity to value-to-ID permutations; it does not establish a production authorization defect or expand confirmation enforcement. Full CI, live-model execution, and restart durability were not run for this contribution.

Co-authored-by: openhands <openhands@all-hands.dev>
@safal207
safal207 marked this pull request as ready for review October 13, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant