fix(cli): adk eval process exit code now reflects PASSED/FAILED - #6740
Open
gaurav-gandhi-2411 wants to merge 7 commits into
Open
fix(cli): adk eval process exit code now reflects PASSED/FAILED#6740gaurav-gandhi-2411 wants to merge 7 commits into
gaurav-gandhi-2411 wants to merge 7 commits into
Conversation
cli_eval computed and printed a per-eval-set "Tests passed"/"Tests
failed" summary but never called sys.exit -- the process always exited
0 regardless of the printed verdict, indistinguishable from every test
genuinely passing. Every other subcommand in this file with a
pass/fail outcome (`run --query`, `test`) already sets a real exit
code; `eval` was the one command whose exit code carried no signal at
all, making it unusable as a CI gate on its own.
Fix: sys.exit(1) if any eval set recorded a failed test, else
sys.exit(0), derived from the same eval_run_summary counts already
being printed.
Two existing tests (test_cli_eval_with_eval_set_file_path,
test_cli_eval_with_eval_set_id) asserted exit_code == 0 for eval cases
that have no invocations and no configured criteria -- never a real
PASSED verdict ("Tests failed: 1" in their own printed summary,
unchanged by this fix). That assertion was only ever true because the
exit code carried no signal; updated to exit_code == 1, matching what
these runs actually produce now that the exit code is real.
…ogle#6951/google#6952 An empty eval_run_summary (no eval case ever evaluated, e.g. an empty eval set) took the same sys.exit(0) path as "every case passed" -- indistinguishable from a genuine pass. Raise instead, matching the fail-loud pattern AgentEvaluator.evaluate_eval_set adopted in google#6952 for the identical defect class in a different code path. Claude-Session: https://claude.ai/code/session_019SUgxvPkwsx4AasNXyU8V2
Contributor
Author
|
Added a case this PR's own diff was missing: an empty `eval_run_summary` (no eval case ever evaluated, e.g. an empty eval set) previously took the same `sys.exit(0)` path as "every case passed" — indistinguishable from a genuine pass. It now raises instead of exiting 0. This is the same defect class flagged in #6951 for the `AgentEvaluator` path and fixed there in #6952 (raise on an empty result set rather than silently returning success); this PR adopts the identical approach for `cli_eval`. New test: `test_cli_eval_empty_summary_does_not_exit_zero`, confirmed failing pre-fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔴 Required Information
Describe the Bug:
cli_eval(theadk evalCLI command) computes and prints a real per-eval-set"Tests passed"/"Tests failed" summary (
eval_run_summary) but never callssys.exit— the process always exits 0 regardless of the printed verdict,indistinguishable from every test genuinely passing. This makes
adk evalunusable as a CI gate on its own exit code: a pipeline step that runs
adk eval ... && deploywill deploy even when every eval case failed.Every other subcommand in this file with a pass/fail outcome already sets a real
exit code:
run --querydoessys.exit(exit_code)(cli_tools_click.py:961,derived from
run_once_cli's return),testdoessys.exit(1)on a missingtest runner (cli_tools_click.py:1060).
evalwas the one command whose exit codecarried no signal at all.
Steps to Reproduce:
Run
adk eval <agent_module> <eval_set_file>against an eval set containing atleast one failing case, then check the process exit code (
echo $?).Expected Behavior: A non-zero exit code when the printed summary shows any
failed test, so the command is usable as a CI gate.
Observed Behavior: Exit code
0regardless of the printed "Tests failed"count — no exit code, prior to this fix, ever distinguished a real failure from
a real pass.
Environment: google-adk main branch (this repo), CLI entry point
cli_tools_click.py::cli_eval.Why this fix
After the existing
eval_run_summaryloop,sys.exit(1)if any eval setrecorded a failed test, else
sys.exit(0)— derived from the same countsalready being printed, no new computation introduced.
Changes
src/google/adk/cli/cli_tools_click.py:cli_evalnow callssys.exit(1 if total_failed else 0)at the end, derived from the sameeval_run_summarycounts already being printed.
tests/unittests/cli/utils/test_cli_tools_click.py:test_cli_eval_with_eval_set_file_path/test_cli_eval_with_eval_set_id:both eval cases have no invocations and no configured criteria, so neither
produces a PASSED verdict ("Tests failed: 1" in their own printed summary,
unchanged by this fix) — their
exit_code == 0assertion was only ever truebecause the exit code carried no signal; updated to
== 1with a commentexplaining why.
test_cli_eval_exit_code_reflects_final_eval_status(parametrized,2 cases): mocks
_collect_inferences/_collect_eval_resultsto return acanned
EvalCaseResultwith a controlledfinal_eval_status, assertingexit_code == 0for PASSED and!= 0(1) for FAILED.Testing Plan
Unit Tests:
any_failed_exits_nonzerocase of the new parametrized test failsagainst pre-fix code (verified by stashing the fix) — confirming it
actually catches the bug.
20 unrelated pre-existing failures (conformance/markdown report, cleanup-unused-files,
path-normalizer, cloud-run-deploy tests) reproduce identically on a clean
upstream/maincheckout with no changes — confirmed before attributing anything tothis change.
Manual E2E: ran
adk evalagainst a real eval set with a failing case pre- andpost-fix from a built shell: pre-fix
echo $?→0; post-fixecho $?→1.Risk & rollback
Confined to the end of one CLI command function; no change to any return value,
schema, or the printed summary itself. The only behavior change is the process exit
code, which previously carried no information — any caller currently ignoring it is
unaffected; any caller that (incorrectly, given the old behavior) assumed 0 always
meant "passed" will now see accurate signal. Purely additive; revert is a clean
single-commit revert.