Skip to content

Scaling issues with cks cluster #14122

Open
Bug
Open
Scaling issues with cks cluster#14122
Bug
Labels
bugcomponent:kubernetes

Description

@kiranchavala
kiranchavala
opened on Sep 10, 2026

problem

Scaling issues with cks cluster

versions

ACS 4.23, 4.22
Hypervisor : KVM
CKS : 1.36

The steps to reproduce the bug

Steps to reproduce the issue

  1. Create 2 service offerings

  2. create kubernetescluster name=cks-npe zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e controlnodes=1 size=2 nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 nodeofferings[1].node=worker nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 hypervisor=KVM

Check the database

mysql> SELECT name, state, service_offering_id, control_node_service_offering_id,        worker_node_service_offering_id, etcd_node_service_offering_id, etcd_node_count FROM cloud.kubernetes_cluster WHERE name='cks-npe' AND removed IS NULL;
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| name    | state   | service_offering_id | control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id | etcd_node_count |
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| cks-npe | Running |                NULL |                               26 |                              27 |                          NULL |            NULL |
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
1 row in set (0.00 sec)
  1. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0

exception


(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T10:41:08+0000",
  "created": "2026-09-10T10:41:08+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "034171ef-25d7-4243-bbd7-6524431309af",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}

Logs


root@Cloudstack-423:/home/ubuntu# cat   /var/log/cloudstack/management/management-server.log |grep -i "logid:034171ef"
2026-09-10 10:41:08,704 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T10:56:08Z\",\"sessionkey\":\"v1xKdYPWziwzZfJE-1T4fxIgyyI\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"serviceofferingid\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9114\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\",\\\"interface com.cloud.offering.ServiceOffering\\\":\\\"d7892a54-8604-44ab-8892-dcd1357571a0\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1430,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"034171ef-25d7-4243-bbd7-6524431309af"}
2026-09-10 10:41:08,706 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430, ctx-e097f2dc]) (logid:034171ef) Account [Account [{"accountName":"admin","id":2,"uuid":"47dc9e4c-83f6-11f1-aed8-bc24112069f3"}]] has access to resource.
2026-09-10 10:41:08,735 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Complete async job-1430, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Publish async job-1430 complete on message bus
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Wake up jobs related to job-1430
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Update db status for job-1430
2026-09-10 10:41:08,738 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Wake up jobs joined with job-1430 and disjoin all subjobs created from job- 1430
2026-09-10 10:41:08,751 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1430
2026-09-10 10:41:08,751 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Remove job-1430 from job monitoring
  1. Scale the cks cluster by providing only the size parameter
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T10:57:23+0000",
  "created": "2026-09-10T10:57:23+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "0f928699-0ce2-47de-b9db-eeb8401c406d",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 0f928699-0ce2-47de-b9db-eeb8401c406d

Logs

root@Cloudstack-423:/home/ubuntu# cat     /var/log/cloudstack/management/management-server.log |grep -i "logid:0f928699"
2026-09-10 10:57:23,386 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:12:23Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9118\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1431,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"0f928699-0ce2-47de-b9db-eeb8401c406d"}
2026-09-10 10:57:23,410 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) Scaling Kubernetes cluster KubernetesCluster {"id":7,"name":"cks-npe","uuid":"7944a42f-fcf2-4f4e-932a-e7b59e0724b1"}
2026-09-10 10:57:23,412 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD
2026-09-10 10:57:23,412 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) Revoking launch permission for systemVM Template
2026-09-10 10:57:23,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Complete async job-1431, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD"}
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Publish async job-1431 complete on message bus
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Wake up jobs related to job-1431
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Update db status for job-1431
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Wake up jobs joined with job-1431 and disjoin all subjobs created from job- 1431
2026-09-10 10:57:23,445 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1431
2026-09-10 10:57:23,445 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Remove job-1431 from job monitoring
  1. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 nodeofferings[0].node=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0

(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 nodeofferings[0].node=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T11:04:52+0000",
  "created": "2026-09-10T11:04:51+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "8dabeecf-0c6a-4771-b2d9-502865ffd944",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 8dabeecf-0c6a-4771-b2d9-502865ffd944

logs


root@Cloudstack-423:/home/ubuntu# cat     /var/log/cloudstack/management/management-server.log |grep -i "logid:8dabeecf"
2026-09-10 11:04:51,999 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:19:51Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"nodeofferings[0].offering\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"nodeofferings[0].node\":\"etcd\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9121\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1432,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8dabeecf-0c6a-4771-b2d9-502865ffd944"}
2026-09-10 11:04:52,015 DEBUG [c.c.k.c.KubernetesServiceHelperImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432, ctx-75ed24fb]) (logid:8dabeecf) Node Type: 'etcd' should use Service Offering ID: 'd7892a54-8604-44ab-8892-dcd1357571a0'
2026-09-10 11:04:52,027 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Complete async job-1432, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Publish async job-1432 complete on message bus
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Wake up jobs related to job-1432
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Update db status for job-1432
2026-09-10 11:04:52,029 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Wake up jobs joined with job-1432 and disjoin all subjobs created from job- 1432
2026-09-10 11:04:52,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1432
2026-09-10 11:04:52,044 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Remove job-1432 from job monitoring
  1. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 nodeofferings[0].key=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 nodeofferings[0].key=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T11:07:43+0000",
  "created": "2026-09-10T11:07:43+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "051ee202-e648-438e-bd19-f1aa885044f4",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 051ee202-e648-438e-bd19-f1aa885044f4

logs

root@Cloudstack-423:/home/ubuntu# cat     /var/log/cloudstack/management/management-server.log |grep -i "logid:051ee202"
2026-09-10 11:07:43,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:22:43Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"nodeofferings[0].key\":\"etcd\",\"nodeofferings[0].offering\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9124\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1433,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"051ee202-e648-438e-bd19-f1aa885044f4"}
2026-09-10 11:07:43,564 ERROR [c.c.k.c.KubernetesServiceHelperImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433, ctx-6bdd9a94]) (logid:051ee202) Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Complete async job-1433, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'"}
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Publish async job-1433 complete on message bus
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Wake up jobs related to job-1433
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Update db status for job-1433
2026-09-10 11:07:43,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Wake up jobs joined with job-1433 and disjoin all subjobs created from job- 1433
2026-09-10 11:07:43,585 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1433
2026-09-10 11:07:43,585 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Remove job-1433 from job monitoring
  1. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 autoscalingenabled=true minsize=3 maxsize=5 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0

(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 autoscalingenabled=true minsize=3 maxsize=5 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T11:12:07+0000",
  "created": "2026-09-10T11:12:07+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3



root@Cloudstack-423:/home/ubuntu# cat     /var/log/cloudstack/management/management-server.log |grep -i "logid:1f5ab1ce"
2026-09-10 11:12:07,190 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:27:07Z\",\"autoscalingenabled\":\"true\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"minsize\":\"3\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"serviceofferingid\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9127\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"maxsize\":\"5\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\",\\\"interface com.cloud.offering.ServiceOffering\\\":\\\"d7892a54-8604-44ab-8892-dcd1357571a0\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1434,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3"}
2026-09-10 11:12:07,197 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434, ctx-17662367]) (logid:1f5ab1ce) Account [Account [{"accountName":"admin","id":2,"uuid":"47dc9e4c-83f6-11f1-aed8-bc24112069f3"}]] has access to resource.
2026-09-10 11:12:07,225 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 11:12:07,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Complete async job-1434, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Publish async job-1434 complete on message bus
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Wake up jobs related to job-1434
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Update db status for job-1434
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Wake up jobs joined with job-1434 and disjoin all subjobs created from job- 1434
2026-09-10 11:12:07,250 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1434
2026-09-10 11:12:07,250 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Remove job-1434 from job monitoring
  1. Create a cks cluster with etcd node
create kubernetescluster name=cks-etcd3 \
  zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd \
  kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 \
  networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e \
  controlnodes=1 size=2 etcdnodes=1 \
  nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 \
  nodeofferings[1].node=worker  nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 \
  nodeofferings[2].node=etcd    nodeofferings[2].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 \
  hypervisor=KVM

mysql> SELECT name, state, service_offering_id, control_node_service_offering_id,        worker_node_service_offering_id, etcd_node_service_offering_id, etcd_node_count FROM cloud.kubernetes_cluster WHERE name='cks-etcd3' AND removed IS NULL;
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| name      | state   | service_offering_id | control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id | etcd_node_count |
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| cks-etcd3 | Running |                NULL |                               26 |                              27 |                            26 |               1 |
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
1 row in set (0.00 sec)


(localcloud) 🐱 > scale kubernetescluster id=bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da size=3
{
  "account": "admin",
  "accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2026-09-10T11:45:41+0000",
  "created": "2026-09-10T11:45:41+0000",
  "domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
  "domainpath": "ROOT",
  "jobid": "3ac3beef-852d-46a7-be6b-04c956006e9d",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 3ac3beef-852d-46a7-be6b-04c956006e9d


root@Cloudstack-423:/home/ubuntu# cat      /var/log/cloudstack/management/management-server.log |grep -i "logid:3ac3beef"
2026-09-10 11:45:41,445 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T12:00:41Z\",\"sessionkey\":\"-6T60SNJxljbmS-Daml5a3Wkwmo\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9271\",\"id\":\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1464,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"3ac3beef-852d-46a7-be6b-04c956006e9d"}
2026-09-10 11:45:41,465 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Scaling Kubernetes cluster KubernetesCluster {"id":9,"name":"cks-etcd3","uuid":"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da"}
2026-09-10 11:45:41,469 WARN  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found
2026-09-10 11:45:41,469 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Revoking launch permission for systemVM Template
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Complete async job-1464, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found"}
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Publish async job-1464 complete on message bus
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Wake up jobs related to job-1464
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Update db status for job-1464
2026-09-10 11:45:41,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Wake up jobs joined with job-1464 and disjoin all subjobs created from job- 1464
2026-09-10 11:45:41,507 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1464
2026-09-10 11:45:41,507 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Remove job-1464 from job monitoring

What to do about it?

Scaling should work without any exception message

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugcomponent:kubernetes

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions