default helm install and argo submit results in permissions failure #3592
Description
Describe the bug
What happened? What did you expect to happen?
After installing the Argo Workflows Helm Chart and following the instructions in the helm install notes, one gets a permissions failure using the default configuration:
Name: hello-world-8xgsq
Namespace: default
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Error
Message: wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User "system:serviceaccount:default:default" cannot create resource "workflowtaskresults" in API group "argoproj.io" in the namespace "default"
Conditions:
PodRunning False
Completed True
Created: Mon Nov 10 08:18:23 -0500 (20 seconds ago)
Started: Mon Nov 10 08:18:23 -0500 (20 seconds ago)
Finished: Mon Nov 10 08:18:43 -0500 (now)
Duration: 20 seconds
Progress: 0/1
ResourcesDuration: 0s*(1 cpu),4s*(100Mi memory)
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ hello-world-8xgsq hello-world hello-world-8xgsq 8s wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User "system:serviceaccount:default:default" cannot create resource "workflowtaskresults" in API group "argoproj.io" in the namespace "default"
I would have expected the default install configuration to at least include permissions/rolebinding to properly execute the argo submit instructions returned in the helm install notes.
Related helm chart
argo-workflows
Helm chart version
Helm Chart 0.45.27, Argo Workflows v3.7.3
To Reproduce
·> kind create cluster -n demo
Creating cluster "demo" ...
✓ Ensuring node image (kindest/node:v1.33.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-demo"
You can now use your cluster with:
kubectl cluster-info --context kind-demo
Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
·> helm install -n argo-workflows argo-wf argo/argo-workflows --create-namespace
NAME: argo-wf
LAST DEPLOYED: Mon Nov 10 08:17:15 2025
NAMESPACE: argo-workflows
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get Argo Server external IP/domain by running:
kubectl --namespace argo-workflows get services -o wide | grep argo-wf-argo-workflows-server
2. Submit the hello-world workflow by running:
argo submit https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml --watch
·> kubectl --namespace argo-workflows get services -o wide | grep argo-wf-argo-workflows-server
argo-wf-argo-workflows-server ClusterIP 10.96.215.6 <none> 2746/TCP 59s app.kubernetes.io/instance=argo-wf,app.kubernetes.io/name=argo-workflows-server
·> argo submit https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml --watch
eventually results in:
Name: hello-world-8xgsq
Namespace: default
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Error
Message: wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User "system:serviceaccount:default:default" cannot create resource "workflowtaskresults" in API group "argoproj.io" in the namespace "default"
Conditions:
PodRunning False
Completed True
Created: Mon Nov 10 08:18:23 -0500 (20 seconds ago)
Started: Mon Nov 10 08:18:23 -0500 (20 seconds ago)
Finished: Mon Nov 10 08:18:43 -0500 (now)
Duration: 20 seconds
Progress: 0/1
ResourcesDuration: 0s*(1 cpu),4s*(100Mi memory)
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ hello-world-8xgsq hello-world hello-world-8xgsq 8s wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User "system:serviceaccount:default:default" cannot create resource "workflowtaskresults" in API group "argoproj.io" in the namespace "default"
I would have expected the default install configuration to at least include permissions/rolebinding to properly execute the argo submit instructions returned in the helm install notes.
Version(s)
Helm Chart 0.45.27, Argo Workflows v3.7.3
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
kind create cluster -n demo
helm install -n argo-workflows argo-wf argo/argo-workflows --create-namespace
kubectl --namespace argo-workflows get services -o wide | grep argo-wf-argo-workflows-server
argo submit https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml --watchLogs from the workflow controller
·> kubectl logs -n argo-workflows argo-wf-argo-workflows-workflow-controller-74b5bbb888-drtl9
time="2025-11-10T13:17:19Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2025-11-10T13:17:19Z" level=info msg="cron config" cronSyncPeriod=10s
time="2025-11-10T13:17:19Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
time="2025-11-10T13:17:19.390Z" level=info msg="not enabling pprof debug endpoints"
time="2025-11-10T13:17:19.395Z" level=info msg="Configuration updated"
time="2025-11-10T13:17:19.395Z" level=info msg="Persistence configuration disabled"
time="2025-11-10T13:17:19.395Z" level=info executorImage="quay.io/argoproj/argoexec:v3.7.3" executorImagePullPolicy= managedNamespace=
time="2025-11-10T13:17:19.395Z" level=info msg="Starting Prometheus metrics exporter"
time="2025-11-10T13:17:19.396Z" level=info msg="Leader election is turned off. Running in single-instance mode"
time="2025-11-10T13:17:19.396Z" level=info msg="starting leading" id=single-instance
time="2025-11-10T13:17:19.396Z" level=info msg="DB migration is disabled"
time="2025-11-10T13:17:19.396Z" level=info msg="Starting Workflow Controller" defaultRequeueTime=10s version=v3.7.3
time="2025-11-10T13:17:19.396Z" level=info msg="Generating Self Signed TLS Certificates for Telemetry Servers"
time="2025-11-10T13:17:19.396Z" level=info msg="Current Worker Numbers" cronWorkflowWorkers=8 podCleanup=4 workflowArchive=8 workflowTtlWorkers=4 workflowWorkers=32
time="2025-11-10T13:17:19.396Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2025-11-10T13:17:19.398Z" level=info msg="Watching task results" labelSelector="!workflows.argoproj.io/controller-instanceid,workflows.argoproj.io/workflow"
time="2025-11-10T13:17:19.398Z" level=info msg=Plugins executorPlugins=false
time="2025-11-10T13:17:19.398Z" level=info msg="Sync manager ttl" syncLimitCacheTTL=0s
time="2025-11-10T13:17:19.398Z" level=info msg="Sync manager initialized" dbConfigured=false
time="2025-11-10T13:17:19.405Z" level=info msg="Manager initialized successfully"
time="2025-11-10T13:17:19.406Z" level=info msg="Received Workflow Controller config map argo-workflows/argo-wf-argo-workflows-workflow-controller-configmap update"
time="2025-11-10T13:17:19.407Z" level=info msg="Configuration updated"
time="2025-11-10T13:17:19.407Z" level=info msg="Persistence configuration disabled"
time="2025-11-10T13:17:19.407Z" level=info executorImage="quay.io/argoproj/argoexec:v3.7.3" executorImagePullPolicy= managedNamespace=
time="2025-11-10T13:17:20.509Z" level=info msg="Performing periodic GC" periodicity=5m0s
time="2025-11-10T13:17:20.509Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2025-11-10T13:17:20.509Z" level=info msg="Starting CronWorkflow controller"
time="2025-11-10T13:17:20.511Z" level=info msg="Starting workflow garbage collector controller (retentionWorkers 4)"
time="2025-11-10T13:17:20.511Z" level=info msg="Started workflow garbage collection"
I1110 13:17:20.511399 1 shared_informer.go:535] "Warning: the sharedIndexInformer has started, run more than once is not allowed"
time="2025-11-10T13:18:23.140Z" level=info msg="Processing workflow" Phase= ResourceVersion=682 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.145Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2025-11-10T13:18:23.145Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.145Z" level=info msg="Updated phase -> Running" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.145Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.146Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.146Z" level=info msg="Pod node hello-world-8xgsq initialized Pending" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.146Z" level=warning msg="couldn't get boundaryTemplate through nodeName hello-world-8xgsq" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.160Z" level=info msg="add pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:23.160Z" level=info msg="Created pod: hello-world-8xgsq (hello-world-8xgsq)" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.160Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.160Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.160Z" level=info msg="Workflow to be dehydrated" Workflow Size=1054
time="2025-11-10T13:18:23.166Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:23.171Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=686 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.177Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=686 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.177Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.177Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.177Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.177Z" level=info msg="Workflow to be dehydrated" Workflow Size=1350
time="2025-11-10T13:18:23.186Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=688 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.188Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=688 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.188Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:23.188Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.188Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.188Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.189Z" level=info msg="Workflow to be dehydrated" Workflow Size=1365
time="2025-11-10T13:18:23.194Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=690 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.196Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=690 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.196Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.196Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:23.196Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:28.002Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:30.016Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:31.021Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:32.028Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:33.161Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=690 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:33.161Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:33.162Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:18:33.162Z" level=warning msg="marking its taskResult as completed since wait container did not exit normally" exitCode=64 namespace=default nodeID=hello-world-8xgsq reason=Error workflow=hello-world-8xgsq
time="2025-11-10T13:18:33.162Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:33.162Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:33.162Z" level=info msg="Workflow to be dehydrated" Workflow Size=1354
time="2025-11-10T13:18:33.167Z" level=info msg="update pod event" pod=hello-world-8xgsq
time="2025-11-10T13:18:33.167Z" level=info msg="cleaning up pod" action=terminateContainers key=default/hello-world-8xgsq/terminateContainers namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:18:33.167Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:18:33.169Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=720 workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.168Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=720 namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.168Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.168Z" level=info msg="Pod failed: wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:default:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"default\"" displayName=hello-world-8xgsq namespace=default pod=hello-world-8xgsq templateName=hello-world workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=warning msg="marking its taskResult as completed since wait container did not exit normally" exitCode=64 namespace=default nodeID=hello-world-8xgsq reason=Error workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg=reconcileAgentPod namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg="Updated message -> wait: Error (exit code 64): workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:default:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"default\"" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg="Marking workflow completed" namespace=default workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.169Z" level=info msg="Workflow to be dehydrated" Workflow Size=1865
time="2025-11-10T13:18:43.172Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=737 workflow=hello-world-8xgsq
time="2025-11-10T13:18:43.178Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:18:43.183Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/hello-world-8xgsq/labelPodCompleted namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:18:43.194Z" level=info msg="delete pod event" pod=hello-world-8xgsq
time="2025-11-10T13:19:03.167Z" level=info msg="cleaning up pod" action=killContainers key=default/hello-world-8xgsq/killContainers namespace=default podName=hello-world-8xgsq
time="2025-11-10T13:22:19.391Z" level=info msg="Alloc=6278 TotalAlloc=16033 Sys=29013 NumGC=7 Goroutines=202"
time="2025-11-10T13:27:19.391Z" level=info msg="Alloc=6144 TotalAlloc=16684 Sys=29013 NumGC=10 Goroutines=202"
Logs from in your workflow's wait container
none. pod never started.
Expected behavior
I would have expected the default install configuration to at least include permissions/rolebinding to properly execute the argo submit instructions returned in the helm install notes.
Screenshots
No response
Additional context
Originally submitted as a bug on argo-workflows project: argoproj/argo-workflows#15016