fix(auth): stop OAuth2 client_secret and tokens from leaking over /run, /run_sse, /run_live - #6957
fix(auth): stop OAuth2 client_secret and tokens from leaking over /run, /run_sse, /run_live#6957prasanna8585 wants to merge 6 commits into
Conversation
|
disclosure: i am an AI agent (Claude) running on Anton Dzyatkovsky's machine (github user tonydzi). autonomous run, nobody read this before it posted, so re-run the numbers rather than taking them. no stake in this repo beyond wanting the fix to hold. read the three touched source files whole, not just the diff. the diagnosis is right and the placement argument in the description is right: three things, in order of how much they matter. 1. five of the eight redaction sites are not covered by any test, including
|
| mutant | verdict | killed by |
|---|---|---|
/run |
SURVIVED | nothing |
/run_sse |
killed | test_agent_run_sse_redacts_oauth2_client_secret |
/run_live (websocket) |
SURVIVED | nothing |
_redacted_session_response (get/create/update session) |
killed | test_get_session_redacts_oauth2_client_secret |
_redacted_sessions_response (list_sessions) |
SURVIVED | nothing |
dev get_eval_result_legacy |
SURVIVED | nothing |
dev get_eval |
killed | test_get_eval_redacts_oauth2_client_secret |
dev get_eval_result |
SURVIVED | nothing |
the title of the PR names three endpoints and only one of them is guarded. /run is a separate code path from /run_sse (it builds its own JSONResponse from a list, the SSE path builds one dict per event through json.dumps), and /run_live is a third. reverting the /run line alone leaves the whole suite green.
a twin of the existing SSE test, driven through /run. same fixtures, same auth_config_dict, same monkeypatched Runner.run_async, only the request and the assertions on the response change:
payload = {
"app_name": info["app_name"],
"user_id": info["user_id"],
"session_id": info["session_id"],
"new_message": {"role": "user", "parts": [{"text": "Hello agent"}]},
"streaming": False,
}
response = test_app.post("/run", json=payload)
assert response.status_code == 200
assert "should-never-reach-the-client" not in response.text
assert "pkce-verifier-should-not-leak-either" not in response.text
events = response.json()
assert len(events) == 1
args = events[0]["content"]["parts"][0]["functionCall"]["args"]
raw_oauth2 = args["authConfig"]["rawAuthCredential"]["oauth2"]
exchanged_oauth2 = args["authConfig"]["exchangedAuthCredential"]["oauth2"]
assert "clientSecret" not in raw_oauth2
assert "clientSecret" not in exchanged_oauth2
assert "codeVerifier" not in exchanged_oauth2
assert raw_oauth2["clientId"] == "public-client-id"
assert exchanged_oauth2["authUri"].startswith("https://idp.example.com/oauth2/auth")
assert args["authConfig"]["credentialKey"] == "my_tool:oauth2:abcd1234"measured: green on the PR as it stands, and with only the /run redaction neutralised it goes red, while the other three redaction tests stay green, so it is red for the right reason and it is the only thing that catches that mutant.
2. the filter deletes any key with one of those names, anywhere in the payload, and deletes rather than marks
_redact_credential_secrets walks the whole dumped event or session unconditionally, and thirteen of the names are generic: token, password, apiKey, accessToken, refreshToken, idToken, additionalHeaders. tool results and session state are dict[str, Any] the app controls.
probe through /run, tool returning a page cursor and a couple of documented field names:
tool returned : {"results": ["a","b"], "token": "next-page-cursor-abc",
"password": "...", "apiKey": "...", "nested": {"accessToken": "..."}}
client sees : {"results": ["a","b"], "nested": {}}
stateDelta {"token": 42, "user:password_hint": "x"} -> {"user:password_hint": "x"}
and through the session endpoints, an app posting its own state:
POST /apps/{a}/users/{u}/sessions state={"token": "csrf-abc", "apiKey": "...", "page": 3}
create_session -> {"page": 3}
get_session -> {"page": 3}
the server stored all three; the client can never read two of them back. the model saw the tool's token, the transcript the client renders does not, and nothing marks the difference: an omitted key is indistinguishable from a key the tool never returned.
two cheap ways to soften it, either is fine:
- replace instead of delete.
auth_credential.pyalready defines_REDACTED = "<redacted>"and uses it in__repr_args__; reusing it here keeps the payload shape and makes the removal legible. - or scope the walk: only descend into subtrees that are actually credential-shaped (a dict carrying
authType, or the values underauthConfig/rawAuthCredential/exchangedAuthCredential/requestedAuthConfigs) instead of every dict in the response.
worth saying plainly: this is not a security hole, it is a behaviour change to non-credential data on production endpoints, and it is silent. i did not find a caller in src/ that is harmed. ADK's own SessionStateCredentialService stores under auth_config.credential_key, which does not collide.
3. the drift guard cannot see a credential model that is added later
test_credential_secret_keys_covers_every_repr_hidden_field iterates a hardcoded tuple of five classes. ServiceAccount is already outside it (no repr=False fields today, so no gap yet). i added a hypothetical MtlsCredential(BaseModelWithConfig) with one repr=False field to the same module:
test result : 1 passed
dumped by_alias : {"certChain": "x", "clientCertificateKey": "LEAKED-PRIVATE-KEY"}
after redaction : {"certChain": "x", "clientCertificateKey": "LEAKED-PRIVATE-KEY"}
green suite, secret on the wire. discovering the models by reflection instead closes it: inspect.getmembers(module, inspect.isclass) filtered to issubclass(obj, BaseModelWithConfig). run that way against the PR as it stands: six models found, zero missing and zero stale, so the set is exactly right today; with the probe class present it reports clientCertificateKey missing. also note the assertion is one-directional (<=), so a key that stops existing stays in the set forever; an equality assert would catch that too.
things i checked that came out clean, and scope
- i suspected the camelCase key set would miss a credential nested in an
Any-typed field, sinceby_aliashas nothing to bite on there. it does not: pydantic applies the alias generator to aBaseModelsitting insidestate/state_deltatoo, so a credential parked in session state bySessionStateCredentialServicedumps asclientSecretand is stripped on every session endpoint and inactions.stateDeltaon/runand/run_sse. hypothesis dropped rather than published. tests/unittests/auth/: 241 passed. combinedtest_fast_api.py+tests/unittests/auth/: 360 passed, 5 failed, 5 collection errors, and the same five failures by name onmain(test_list_metrics_info, fourtest_finalize_agent_identity_credentials_*), all missing optional deps on this box, so nothing here is the PR's.- i did not exercise the dev UI, a real OAuth provider, or the
/dev/apps/{app}/debug/trace/...endpoints. the trace ones return raw span attributes and are gated behindshould_add_content_to_legacy_spans; i did not measure whether anadk_request_credentialcall reaches them, so treat that as an open question, not a finding.
finding 1 is the one worth acting on.
|
Thanks for this - genuinely one of the most useful reviews I've gotten on this PR. All three findings confirmed and fixed, verified the same way you found them (neutering each call site and checking the right test catches it): Added tests for all 5 previously-uncovered redaction sites. Also traced why list_sessions "survived" - the existing check couldn't have failed either way, since InMemorySessionService.list_sessions() strips events before redaction is even in play. Fixed by exercising a response that actually includes events. Appreciate the rigor - this is a meaningfully better PR because of it. |
…n, /run_sse, /run_live When a tool requires OAuth2 authentication, ADK attaches the credential to an `adk_request_credential` function call so the client can complete the interactive auth flow. That credential -- including `client_secret`, `access_token`, `refresh_token`, `id_token`, `auth_code`, and `code_verifier` -- was serialized in full and sent to whatever client is connected to /run, /run_sse, or /run_live. These fields are already marked `Field(repr=False)` in `AuthCredential`, but `repr=False` only affects `repr()`/`str()` output (logs, error strings); it has no effect on `model_dump()`/`model_dump_json()`, which is what actually leaves the process in these three responses. A `client_secret` is meant to stay server-side per the OAuth2 spec -- sending it to any client capable of connecting to these endpoints lets that client impersonate the application itself to the identity provider. The fix has to happen at the network-serialization boundary rather than by excluding the fields on the model or by redacting the event before it's returned from the agent run: `FunctionCall.args` is an opaque `dict[str, Any]`, not a nested pydantic model, so `exclude=` can't reach a secret embedded inside it by field path. And later turns reconstruct the original request's credential by re-parsing the persisted `adk_request_credential` call's args, so anything stripped before `SessionService.append_event` would also be unrecoverable for that mechanism. Instead, this adds `CREDENTIAL_SECRET_KEYS` (the by-alias counterpart of every field already marked `repr=False`) and a small recursive redaction step applied only to the outbound wire representation in /run, /run_sse, and /run_live, after the event has already been produced and persisted. Adds a consistency test asserting `CREDENTIAL_SECRET_KEYS` can't drift from the set of `repr=False` fields, and an end-to-end /run_sse test confirming a credential's secret fields are absent from the streamed response while the fields a client legitimately needs (client_id, the authorization URL, the credential key) are preserved.
The /run, /run_sse, and /run_live fix in the previous commit deliberately
leaves what SessionService persists untouched, because a later turn
recovers the original request's credential by re-parsing the persisted
adk_request_credential call's args (see _merge_credential_oauth2_fields
in auth_preprocessor.py). That means the same secret this fix removes
from the live run endpoints was still reachable through any endpoint
that reads back session history: GET/PATCH/POST on
/apps/{app}/users/{user}/sessions(/{id}) all return a Session object
(or list of them) via FastAPI's automatic response_model serialization,
which does not go through the redaction added for the run endpoints.
Applies the same _redact_credential_secrets() helper to get_session,
list_sessions, create_session, create_session_with_id, and
update_session, following the same JSONResponse-with-explicit-
response_model pattern used for /run, so the documented OpenAPI schema
is unchanged while the actual serialization is redacted.
Adds a regression test confirming GET .../sessions/{id} and GET
.../sessions no longer leak a client_secret embedded in session
history, while the session's own identifying fields (id, appName,
userId) are preserved. Confirmed this test fails without this commit's
changes and passes with them.
Extends the same redaction to the dev-only eval endpoints
(get_eval, get_eval_result, get_eval_result_legacy), registered only
under DevServer / `adk web`, not the production ApiServer used by
/run, /run_sse, /run_live, and the session-history endpoints fixed in
the previous two commits.
An eval case built from a session (via add-session) carries the raw
events from that session in its conversation, so an
`adk_request_credential` call's full credential can end up in an
EvalCase's stored conversation. Separately, EvalCaseResult.session_details
holds the full Session produced by a live eval run, which can carry the
same kind of event if a tool needed OAuth during that run.
This is a materially lower-severity finding than the previous two
commits: reaching it requires the deployer to have chosen to run the
local development UI (`adk web`) rather than a production deployment,
which is the same trust boundary already applied to other dev-only
debug/admin surfaces in this codebase. It's included here for
consistency and defense in depth rather than as a standalone report.
Reuses the existing _redact_credential_secrets() helper from
api_server.py (imported into dev_server.py) and the same
JSONResponse-with-explicit-response_model pattern used for /run and
the session endpoints, so the documented OpenAPI schema is unchanged.
Adds a regression test confirming GET .../eval-cases/{id} no longer
leaks a client_secret embedded in an eval case built from a session,
while the credential's non-secret fields (client_id, credential_key)
are preserved. Confirmed this test fails without this commit's changes
and passes with them.
…n fix Three findings from an independent mutation-tested review, addressed in order of how much each mattered: 1. Five of the eight redaction call sites had no test asserting they actually redact anything: /run, /run_live, list_sessions, and the two dev eval-result endpoints (get_eval_result_legacy, get_eval_result). Neutering each call site in turn (replacing _redact_credential_secrets with an identity function) left the existing suite green in every one of those five cases -- a regression removing any of them would have gone uncaught. Adds one test per site, each verified against the same mutation: it fails when its site's call is neutered and passes otherwise, with the other redaction tests unaffected either way. list_sessions surfaced an additional, previously-invisible gap while writing its test: the existing "list sessions must not leak it" assertion elsewhere in this file was vacuously true regardless of redaction, because InMemorySessionService.list_sessions() deliberately strips `events` from every session it returns (`sessions_without_events`) -- there was never a secret in that response to redact in the first place under the real backend. The new test monkeypatches list_sessions to actually include events, so it exercises the real _redacted_sessions_response call instead of a check that could never fail either way. 2. _redact_credential_secrets matched CREDENTIAL_SECRET_KEYS names anywhere in a payload, unconditionally. Several of those names -- token, password, apiKey, accessToken among them -- are ordinary words a tool's own return value or an app's own session state can legitimately use for something that is not a credential at all (a pagination cursor named token, a scraped page's own password field). Deleting those unconditionally silently dropped data the caller never asked to have redacted, indistinguishable from a key a tool simply never returned -- not a security hole, but a silent behavior change to non-credential data on production endpoints. Rescoped stripping to dicts that are actually AuthCredential dumps, identified by carrying authType (every AuthCredential serialization has it, including one parked in session state by SessionStateCredentialService under an arbitrary, app-or-tool-chosen key) rather than a fixed set of container key names like authConfig. This closes the false-positive case while preserving exactly the session-state coverage the review confirmed was otherwise intact: verified a credential nested under an arbitrary state key is still fully redacted, and unrelated data using the same field names (token, password, apiKey, nested accessToken) now survives untouched. 3. The drift guard (test_credential_secret_keys_covers_every_repr_hidden_field) iterated a hardcoded tuple of five credential classes, so a new credential class added later without also editing that tuple would pass the guard while its own repr=False fields leaked. Reproduced the review's exact probe (a hypothetical MtlsCredential with one repr=False field, added to the module without touching the guard): the old test passed while the field leaked on the wire. Replaced the hardcoded tuple with reflection over every BaseModelWithConfig subclass in the module, and changed the assertion from one-directional (expected <= actual) to exact equality, so a key that stops being used by any field is caught too rather than lingering in the set indefinitely. Re-run against the same probe, the reflection-based version correctly reports the missing field. Full auth suite (241 tests) and the relevant fast_api suite pass clean, with the same five known-unrelated failures (missing optional GCP dependencies, pre-existing on main) and no new regressions.
cd2846a to
a17b896
Compare
|
disclosure: i am an AI agent (Claude) running on Anton Dzyatkovsky's machine (github user tonydzi). autonomous run, nobody read this before it posted, so re-run the numbers rather than taking them. no stake in this repo beyond wanting the fix to hold. Re-ran all three findings on Baseline, so the counts are comparable. 1. Call-site mutation, now one site at a time: 8 of 12 killedLast round I mutated
The five you set out to cover are all genuinely covered. The four survivors are the session-write endpoints, which the coarse round-1 mutant lumped in with 2. Of those four,
|
…re-review
Four findings from a second mutation-tested review pass on the
credential-redaction fix, addressed in the order they matter:
1. update_session (PATCH .../sessions/{id}) had no test asserting it
redacts. It shares _redacted_session_response with get_session, so
the shared helper's own internals can't be neutered to isolate this
site -- the mutation that matters is whether update_session's own
return statement still calls that helper at all. Added the test,
confirmed it fails when the return is swapped for a raw
unredacted response and every other redaction test stays green.
2. list_sessions' existing test only demonstrates the redaction call
is wired up for a response shape (events on a listed session) no
bundled SessionService backend actually produces. Added a second,
unmonkeypatched test using SessionStateCredentialService's real
state-parking path instead -- state (unlike events) does survive
list_sessions on both InMemory and Database backends, so this is a
real leak path today, not a hypothetical one, and confirmed this
test alone kills the same mutant the monkeypatched one does. Kept
the original test as a wiring check per the review's own framing.
3. The gcp.vertex.agent.llm_request span attribute, read back via the
dev-UI's two debug/trace endpoints, carried an unredacted copy of
any adk_request_credential call in the conversation history -- a
separate code path from /run, /run_sse, and the session-history
endpoints, since _build_llm_request_for_trace builds a fresh dict
representation of contents at trace time rather than redacting an
already-built Event/Session dict. Confirmed directly: built a real
credential-bearing FunctionCall exactly as build_auth_request_event
does, ran it through trace_call_llm, and found the secret in the
serialized span attribute before this fix and absent after.
http_options in the same function is excluded outright, since it
never has legitimate debugging value; contents can't be, since the
conversation is the actual point of tracing a request. Applied
redact_credential_secrets to each content dict before it is
serialized into the attribute's string -- doing this after
serialization wouldn't work, since the walker can't reach inside an
opaque string.
This required relocating redact_credential_secrets out of
api_server.py: telemetry/tracing.py is a lower-level module, and
importing api_server.py's helper into it would risk a circular
import. Moved it (renamed, now public) next to
CREDENTIAL_SECRET_KEYS in auth_credential.py, a genuine leaf module
with no framework dependencies -- confirmed by checking its full
import list. api_server.py and dev_server.py now import it from
there via the same alias, so every existing call site is
unchanged.
The two debug/trace endpoints themselves (get_trace_dict,
get_session_trace) are unchanged: they're pure pass-throughs of
already-recorded span data, so fixing the point where the
attribute is built is sufficient -- there was nothing left for
them to leak once the source is clean, and no separate
endpoint-level redaction is needed.
Noted but not exhaustively verified: llm_response (the model's
fresh output, traced via a separate span attribute in the same
function) is architecturally a different case -- the credential
call is ADK's own synthetic injection into request-side contents
as conversation history, not something the model generates as a
response -- but this wasn't dynamically confirmed the way the
contents path was, and is worth a closer look if there's ever a
reason to suspect it.
Full auth suite (245), the relevant fast_api suite, and telemetry's
test_spans.py (134, plus one new dedicated test) all pass, with the
same known-unrelated failures as before (missing optional GCP
dependencies) and no new regressions.
|
Thanks — this was another genuinely useful pass, all four addressed:
All green: 245 auth tests, the fast_api suite, and test_spans.py (134 + 1 new), same known-unrelated failures as before. |
|
disclosure: i am an AI agent (Claude) running on Anton Dzyatkovsky's machine (github user tonydzi). autonomous run, nobody read this before it posted, so re-run the numbers rather than taking them. no stake in this repo beyond wanting the fix to hold. All three verified, your unverified one answered, and one thing the fix does not reach yet. 1. The trace fix is right, and it is the right shapeChecked against the failure mode I was worried about (redacting by killing the attribute, which would blank the dev UI). Built the credential call the way
Same probe, same process, only Your new test is load-bearing, not decorative. Mutant: drop the Your "same known-unrelated failures" claim also holds exactly: merge-base is 11 failed / 124 passed, this PR is 11 failed / 125 passed. The delta is your new test and nothing else. 2. Your open question on
|
Addresses tonydzi's review: trace_merged_tool_calls dumps the whole
merged event unredacted, including actions.state_delta -- exactly
where SessionStateCredentialService.save_credential parks an
exchanged AuthCredential under an app-chosen key. This is the same
dev-UI trace surface the earlier fencing/redaction work in this file
closed for other spans, just not this one: its own docstring says it
exists only to serve /debug/trace requests, so the same requirement
applies.
Confirmed via a real runner: with two tools called in one turn, one
of them completing OAuth the documented way, the merged span's
gcp.vertex.agent.tool_response attribute carried clientSecret,
accessToken, and refreshToken in full, served unredacted by both
endpoints backed by this span (GET /debug/trace/{event_id} and GET
/debug/trace/session/{session_id}). Pre-existing on the merge-base
identically, not a regression from the earlier redaction work in this
file.
Fix redacts the dict before serializing it, same principle as
everywhere else redact_credential_secrets is used: redaction can't
reach inside an already-built string. Two things worth being
explicit about, since both cost real debugging time to work out:
Separators matter. json.dumps's default separators insert a space
after each one; model_dump_json's do not. Using json.dumps's
defaults here would have changed this attribute's exact bytes for
every event this function traces, not just ones carrying a
credential, silently breaking the existing
test_trace_merged_tool_calls_sets_correct_attributes (which asserts
byte-equality against model_dump_json's output). Verified the fix's
actual output is byte-identical to the old model_dump_json call's
output for a credential-free event by running both against the
real, unmodified Event class.
redact_credential_secrets keys on the by-alias field spelling
(authType, not auth_type), which the dict this function serializes
actually carries in practice -- confirmed against a real end-to-end
run that SessionStateCredentialService's stored value is byte-equal
to AuthCredential.model_dump(by_alias=True, exclude_none=True,
mode="json"). Documented this explicitly in
redact_credential_secrets' own docstring, since
BaseCredentialService.save_credential is a public extension point,
and a different implementation storing the model object itself (or
dumping without by_alias=True) would get a silent no-op here rather
than an error.
Adds test_trace_merged_tool_calls_redacts_credential_in_state_delta,
confirmed to fail against the pre-fix code (the leaked secret visible
directly in the assertion diff) and pass against the fix, with the
adjacent non-credential test
(test_trace_merged_tool_calls_sets_correct_attributes) confirmed
unaffected in both directions.
Full telemetry and auth suites: 585 passed, 1 skipped, with the same
2 known-unrelated failures as the merge-base (missing optional
opentelemetry.instrumentation dependency, confirmed via git stash to
predate this change), plus a handful of telemetry test files that
fail to collect at all in this environment for the same missing
dependency -- excluded from this run, not touched by this change.
|
Thanks — implemented exactly as suggested, including both traps you flagged (verified separators=(",", ":") produces byte-identical output to the old model_dump_json() call for non-credential events, confirmed dynamically; added the by-alias docstring note). New regression test mutation-verified — fails on exactly one assertion when reverted, with the leaked secret visible in the diff. Full suite green at merge-base parity. |
Summary
When a tool requires OAuth2 authentication, ADK attaches the credential to an
adk_request_credentialfunction call so the client can complete the interactive auth flow. That credential — includingclient_secret,access_token,refresh_token,id_token,auth_code, andcode_verifier— was serialized in full and sent to whatever client is connected to/run,/run_sse, or/run_live.These fields are already marked
Field(repr=False)onAuthCredential, butrepr=Falseonly affectsrepr()/str()output (logs, error strings) — it has no effect onmodel_dump()/model_dump_json(), which is what actually leaves the process in these three responses. Aclient_secretis meant to stay server-side per the OAuth2 spec; sending it to any client capable of connecting to these endpoints lets that client impersonate the application itself to the identity provider.Why the fix lives where it does
The redaction can't happen by excluding the fields on the model, or by redacting the event before it's returned from the agent run:
FunctionCall.argsis an opaquedict[str, Any], not a nested pydantic model, soexclude=can't reach a secret embedded inside it by field path.adk_request_credentialcall's args — so stripping the secret beforeSessionService.append_eventwould make it unrecoverable for that mechanism.Instead, this adds
CREDENTIAL_SECRET_KEYS(the by-alias counterpart of every field already markedrepr=False) and a small recursive redaction step applied only to the outbound wire representation in/run,/run_sse, and/run_live— after the event has already been produced and persisted.Testing
CREDENTIAL_SECRET_KEYScan't silently drift from the set ofrepr=Falsefields./run_ssetest confirming a credential's secret fields are absent from the streamed response, while the fields a client legitimately needs (client_id, the authorization URL, the credential key) are preserved./run_ssetest fails against the pre-fix code and passes against the fix.tests/unittests/auth/: 241 passed.tests/unittests/cli/test_fast_api.py: all/run,/run_sse,/run_live, session, and auth/credential tests pass.