fix(auth): handle PermissionError on workload certificates to avoid startup hang and crash#17568
Conversation
There was a problem hiding this comment.
Code Review
This pull request improves error handling when checking and accessing agent identity certificates. Specifically, it updates _is_certificate_file_ready to propagate PermissionError while catching other OSError exceptions, and handles PermissionError in get_agent_identity_certificate_path by logging a warning and falling back to unbound tokens. Corresponding unit tests have been added to verify these scenarios. There are no review comments, so I have no additional feedback to provide.
There was a problem hiding this comment.
Thank you for this contribution!
System-level / Off-diff Finding:
Graceful OSError Handling in get_and_parse_agent_identity_certificate()
get_and_parse_agent_identity_certificate() (around line 208) does not handle exceptions when reading the certificate file. Because os.path.getsize only requires directory traversal / metadata search permissions, _is_certificate_file_ready can evaluate to True even for a file that the current process does not have read permissions to open (e.g., owned by root with 600). When open(cert_path, "rb") is called, it will crash with a raw PermissionError.
Wrap the file-read in a try...except OSError block to gracefully fail-closed with a controlled RefreshError:
try:
with open(cert_path, "rb") as cert_file:
cert_bytes = cert_file.read()
except OSError as e:
raise exceptions.RefreshError(
f"Failed to read agent identity certificate file at {cert_path}: {e}. "
"Token binding protection is failing. You can turn off this protection by setting "
f"{environment_vars.GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES} to false "
"to fall back to unbound tokens."
) from eThere was a problem hiding this comment.
I think we may want to catch here too. I think it's possible for get_agent_identity_certificate_path to work correctly but then have .read to raise a PermissionError since .stat and .read require different permissions
There was a problem hiding this comment.
Done! I have wrapped the file read block in a try...except OSError block to catch read-level permissions errors and fall back to unbound tokens.
| if not path: | ||
| return False | ||
| try: | ||
| return os.path.getsize(path) > 0 |
There was a problem hiding this comment.
nit: the PR description mentions using os.stat , but the code uses os.path.getsize . Functionally equivalent since getsize uses stat under the hood, but just pointing it out!
There was a problem hiding this comment.
The new commit uses os.stat directly. I'll double check the PR description after making all the changes requested by reviewers to ensure it's still valid. Thanks for pointing it out :)
…tartup hang and crash In sandboxed environments (such as when running the gcloud CLI within a snap package manager sandbox), AppArmor rules can block reading the workload credentials directory or certificate files, raising a PermissionError. Previously, `os.path.exists()` caught this PermissionError and returned `False`. The library interpreted this as the files not being ready yet (e.g. due to a startup race) and entered a 30-second retry loop before failing with a RefreshError. This change: 1. Updates `_is_certificate_file_ready` to use `os.stat` and propagate `PermissionError`. 2. Catches `PermissionError` immediately during certificate lookup, logging a warning and falling back to unbound tokens without retrying or crashing. 3. Adds corresponding unit tests.
022d2ae to
cea7183
Compare
cea7183 to
51b41f3
Compare
🤖 I have created a release *beep* *boop* --- <details><summary>db-dtypes: 1.7.1</summary> ## [1.7.1](db-dtypes-v1.7.0...db-dtypes-v1.7.1) (2026-07-07) ### Bug Fixes * avoid deprecated unitless operations for NumPy 2.5 compatibility ([#17589](#17589)) ([d0b2abc](d0b2abc)) </details> <details><summary>gapic-generator: 1.37.0</summary> ## [1.37.0](gapic-generator-v1.36.0...gapic-generator-v1.37.0) (2026-07-07) ### Features * implement native PEP 0810 lazy loading ([#17591](#17591)) ([8a1270c](8a1270c)) ### Bug Fixes * **deps:** bump google-api-core to 2.25.0 ([#17599](#17599)) ([8b359e2](8b359e2)) * **tests:** add --cov-append to gapic-generator and proto-plus to preserve monorepo coverage ([#17603](#17603)) ([2ddcf4d](2ddcf4d)) </details> <details><summary>google-auth: 2.55.2</summary> ## [2.55.2](google-auth-v2.55.1...google-auth-v2.55.2) (2026-07-07) ### Bug Fixes * **auth:** Agentic Identites mTLS gaps fix _is_mtls and SslCredentials. ([#17387](#17387)) ([7bfa41a](7bfa41a)) * **auth:** align mTLS discovery and enforce fail-fast transport configuration. ([#17470](#17470)) ([f492d3d](f492d3d)) * **auth:** handle PermissionError on workload certificates to avoid startup hang and crash ([#17568](#17568)) ([f538ad8](f538ad8)) </details> <details><summary>google-cloud-agentregistry: 0.1.0</summary> ## 0.1.0 (2026-07-07) ### Features * **google/cloud/agentregistry/v1:** add google-cloud-agentregistry ([#17565](#17565)) ([f479800](f479800)) </details> <details><summary>google-cloud-biglake-hive: 0.3.1</summary> ## [0.3.1](google-cloud-biglake-hive-v0.3.0...google-cloud-biglake-hive-v0.3.1) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-bigquery: 3.42.2</summary> ## [3.42.2](google-cloud-bigquery-v3.42.1...google-cloud-bigquery-v3.42.2) (2026-07-07) ### Bug Fixes * **bigquery:** avoid SSLError retry loop ([#17489](#17489)) ([8248d8e](8248d8e)) * include amended user agent in read client ([#17592](#17592)) ([c43caee](c43caee)) </details> <details><summary>google-cloud-binary-authorization: 1.18.0</summary> ## [1.18.0](google-cloud-binary-authorization-v1.17.0...google-cloud-binary-authorization-v1.18.0) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-chronicle: 0.6.2</summary> ## [0.6.2](google-cloud-chronicle-v0.6.1...google-cloud-chronicle-v0.6.2) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-dataform: 0.11.2</summary> ## [0.11.2](google-cloud-dataform-v0.11.1...google-cloud-dataform-v0.11.2) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-dataproc: 5.30.0</summary> ## [5.30.0](google-cloud-dataproc-v5.29.0...google-cloud-dataproc-v5.30.0) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-datastore: 2.26.0</summary> ## [2.26.0](google-cloud-datastore-v2.25.0...google-cloud-datastore-v2.26.0) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-dialogflow: 2.50.0</summary> ## [2.50.0](google-cloud-dialogflow-v2.49.0...google-cloud-dialogflow-v2.50.0) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-kms: 3.15.0</summary> ## [3.15.0](google-cloud-kms-v3.14.0...google-cloud-kms-v3.15.0) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-logging: 3.16.1</summary> ## [3.16.1](google-cloud-logging-v3.16.0...google-cloud-logging-v3.16.1) (2026-07-07) ### Documentation * **logging:** fix StructuredLogHandler docstring parameter name ([#17625](#17625)) ([32862f0](32862f0)), closes [#17604](#17604) </details> <details><summary>google-cloud-modelarmor: 0.7.1</summary> ## [0.7.1](google-cloud-modelarmor-v0.7.0...google-cloud-modelarmor-v0.7.1) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-network-security: 0.13.3</summary> ## [0.13.3](google-cloud-network-security-v0.13.2...google-cloud-network-security-v0.13.3) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-saasplatform-saasservicemgmt: 0.7.1</summary> ## [0.7.1](google-cloud-saasplatform-saasservicemgmt-v0.7.0...google-cloud-saasplatform-saasservicemgmt-v0.7.1) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-securesourcemanager: 0.6.1</summary> ## [0.6.1](google-cloud-securesourcemanager-v0.6.0...google-cloud-securesourcemanager-v0.6.1) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>google-cloud-storage: 3.12.1</summary> ## [3.12.1](google-cloud-storage-v3.12.0...google-cloud-storage-v3.12.1) (2026-07-07) ### Bug Fixes * **storage:** log occasional (1 in 5 million) additional bytes received from GCS in read path ([#17423](#17423)) ([335c12f](335c12f)) </details> <details><summary>google-cloud-support: 0.5.1</summary> ## [0.5.1](google-cloud-support-v0.5.0...google-cloud-support-v0.5.1) (2026-07-07) ### Features * update googleapis and regenerate ([#17635](#17635)) ([9638879](9638879)) </details> <details><summary>proto-plus: 1.28.1</summary> ## [1.28.1](proto-plus-v1.28.0...proto-plus-v1.28.1) (2026-07-07) ### Bug Fixes * **tests:** add --cov-append to gapic-generator and proto-plus to preserve monorepo coverage ([#17603](#17603)) ([2ddcf4d](2ddcf4d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
In sandboxed environments (such as when running the gcloud CLI within a snap package manager sandbox), AppArmor rules can block reading the workload credentials directory or certificate files, raising a PermissionError.
Previously,
os.path.exists()caught this PermissionError and returnedFalse. The library interpreted this as the files not being ready yet (e.g. due to a startup race) and entered a 30-second retry loop before failing with a RefreshError.This change:
_is_certificate_file_readyto useos.statand propagatePermissionError.PermissionErrorimmediately during certificate lookup, logging a warning and falling back to unbound tokens without retrying or crashing.Fixes b/522957573