v36.0.0 regression: load_incluster_config() sends requests as system:anonymous on GKE

What happened

After upgrading from kubernetes==35.0.0 to kubernetes==36.0.0, all API calls
made via load_incluster_config() are rejected with 403 Forbidden and
User "system:anonymous", resulting in requests being treated as unauthenticated.

Downgrading back to kubernetes==35.0.0 immediately resolves the issue with
zero other changes.

Environment

Kubernetes version: GKE v1.32.4
Python client version: kubernetes==36.0.0 (broken), kubernetes==35.0.0 (working)
Python version: 3.11
Container image: python:3.11-slim
Runtime environment: Python application running inside a Docker container on GKE
Authentication method: config.load_incluster_config() using mounted ServiceAccount token
Cloud: Google Kubernetes Engine (GKE) with Workload Identity enabled

How to reproduce

Run a pod on GKE with Workload Identity enabled on the node pool
Use kubernetes==36.0.0
Call config.load_incluster_config() followed by any API call:

from kubernetes import client, config

config.load_incluster_config()
v1 = client.CoreV1Api()
v1.list_pod_for_all_namespaces(watch=False)  # 403 — system:anonymous

What you expected to happen

Requests should authenticate as the pod's ServiceAccount, same as v35.0.0.

What actually happened

(403)
Reason: Forbidden
HTTP response body:
{
  "message": "pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
}

Anything else we need to know?

Additional validation performed:

ServiceAccount token exists correctly at:
/var/run/secrets/kubernetes.io/serviceaccount/token
kubectl auth can-i confirms RBAC permissions are valid
Raw authenticated requests using the mounted token succeed
Only the Python Kubernetes client in v36.0.0 fails

Confirmed workaround:

kubernetes==35.0.0

This is a duplicate, a fix has been merged and a release is going to be cut soon

Thanks for looking into this.

While I understand this likely stems from the same underlying regression in v36.0.0 (the BearerToken vs authorization key change), I believe the observed behavior here is different enough to be worth highlighting separately.

Unlike the 401 Unauthorized reported in #2584 and #2591, my cluster setup results in a 403 Forbidden with system:anonymous. This appears to happen in environments like GKE where a missing token is treated as an anonymous request rather than being rejected immediately as unauthorized.

So while the root cause may be the same, I think this issue provides a useful additional data point for the 403 system:anonymous scenario, which might not be immediately obvious from the existing reports.

If this is already covered by the fix in PR #2585, feel free to close this as a duplicate — I just wanted to make sure this specific behavior is captured.

https://github.com/kubernetes-client/python/releases/tag/v36.0.1 has been released, it should fix the issue. Could you please give it a try?

@yliaog

@yliaog it looks like this works but it appears to break some downstream libraries.

nolar/kopf#1307

@markafarrell

@markafarrell thanks for confirming the fix.

please see

Breaking Change from upgrading openapi generator to v6.6.0:
https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md#breaking-change-from-upgrading-openapi-generator-to-v660

@starboyhassan

@starboyhassan could you please try out the new release https://github.com/kubernetes-client/python/releases/tag/v36.0.1 to see if the particular case you saw is fixed ?

We have the same issue when using the ansible kubernetes.core.k8s module with 36.0.0 and 36.0.1 with openshift. The 36.0.1 update has not resolved the issue. Using 35.0.0 works

same issue, using k8s1.35.

@LTek-online

@LTek-online @MountainGG

could you check if you are having the same problem as nolar/kopf#1307 ? if yes, then could you fix it on your side?

https://github.com/kubernetes-client/python/releases/tag/v36.0.2 has been released, please check if it helps fix the problem.

@yliaog

@yliaog 36.0.2 does not fix the the issue with nolar/kopf#1307

kopf calls config.get_api_key_with_prefix('authorization')

36.0.2 only fixes when accessing via Configuration.auth_settings()

@yliaog

@yliaog Tested the python kubernetes==36.0.2 module with the ansible module kubernetes.core.k8s and it works. 36.0.0 and 36.0.1 don't.

FYI https://github.com/nolar/kopf/releases/tag/1.44.6
This version of kopf includes support >= v36.0.1

Pinned kubernetes==35.0.0 to keep the operator image on the last verified working Kubernetes Python client version.

We compared the images:

previous working image: kubernetes==35.0.0
rebuilt failing image: kubernetes==36.0.2

With 36.0.2, Kopf logs authentication as successful, but Kubernetes API requests are handled as system:anonymous.
Although that issue is marked fixed upstream, we still reproduce the behavior with 36.0.2 in our operator image, so we pin to 35.0.0 until a newer client version is verified in our cluster.

Pinned kubernetes==35.0.0 to keep the operator image on the last verified working Kubernetes Python client version.

We compared the images:

previous working image: kubernetes==35.0.0

rebuilt failing image: kubernetes==36.0.2

With 36.0.2, Kopf logs authentication as successful, but Kubernetes API requests are handled as system:anonymous.
Although that issue is marked fixed upstream, we still reproduce the behavior with 36.0.2 in our operator image, so we pin to 35.0.0 until a newer client version is verified in our cluster.

You also need kopf >= 1.44.6

If you upgrade kopf to the version the new version of kubernetes should work as expected.

Pinned kubernetes==35.0.0 to keep the operator image on the last verified working Kubernetes Python client version.
We compared the images:

previous working image: kubernetes==35.0.0

rebuilt failing image: kubernetes==36.0.2

With 36.0.2, Kopf logs authentication as successful, but Kubernetes API requests are handled as system:anonymous.
Although that issue is marked fixed upstream, we still reproduce the behavior with 36.0.2 in our operator image, so we pin to 35.0.0 until a newer client version is verified in our cluster.

You also need kopf >= 1.44.6

If you upgrade kopf to the version the new version of kubernetes should work as expected.

Thank a lot!
Confirm that it works with
kubernetes==36.0.2
kopf == 1.44.6

Confirmed this is still broken in 36.0.2, outside of kopf/ansible too — same system:anonymous symptom with a plain manual Configuration() for an external cluster.

Repro:

from kubernetes import client

configuration = client.Configuration()
configuration.api_key["authorization"] = "<token>"
configuration.api_key_prefix["authorization"] = "Bearer"

print(configuration.auth_settings())
# v34.1.0: value = 'Bearer <token>'   (correct)
# v36.0.2: value = '<token>'          (missing "Bearer " prefix)

#2604 fixed the token lookup in auth_settings() by calling get_api_key_with_prefix('BearerToken', alias='authorization'), but get_api_key_with_prefix only applies alias to the key lookup, not the prefix one:

key = self.api_key.get(identifier, self.api_key.get(alias) if alias is not None else None)
if key:
    prefix = self.api_key_prefix.get(identifier)  # never checks `alias`

So anyone who sets api_key_prefix['authorization'] (the old convention) gets a bearer token with no "Bearer " prefix. GKE treats that as anonymous. Same fix pattern should apply to the prefix lookup:

prefix = self.api_key_prefix.get(identifier, self.api_key_prefix.get(alias) if alias is not None else None)

Pinning to <36.0.0 (we're on 34.1.0) works around it :\

Apply alias fallback to prefix lookup in get_api_key_with_prefix

Navigation Menu

Repository navigation

v36.0.0 regression: load_incluster_config() sends requests as system:anonymous on GKE #2592

What happened

Environment

How to reproduce

What you expected to happen

What actually happened

Anything else we need to know?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

v36.0.0 regression: load_incluster_config() sends requests as system:anonymous on GKE #2592

Description

What happened

Environment

How to reproduce

What you expected to happen

What actually happened

Anything else we need to know?

Activity

aojea commented on May 22, 2026

starboyhassan commented on May 22, 2026

yliaog commented on May 26, 2026

markafarrell commented on May 27, 2026

yliaog commented on May 27, 2026

yliaog commented on May 27, 2026

LTek-online commented on May 28, 2026

MountainGG commented on May 28, 2026

yliaog commented on May 28, 2026

yliaog commented on Jun 1, 2026

markafarrell commented on Jun 2, 2026

LTek-online commented on Jun 2, 2026

markafarrell commented on Jun 4, 2026

anastasiyatolstikenableu commented on Jun 30, 2026

markafarrell commented on Jun 30, 2026

anastasiyatolstikenableu commented on Jul 1, 2026

HaimLC commented on Jul 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions