Can I disable HPA while the argo rollout is progressing?

[sungho-rpls]

HPA scales out replicas in my environment because of high CPU usage while the argo rollout progresses. High CPU usage for startup time is acceptable for me, but too many replicas by HPA cause problems with latency, resource usage, and database connections. Is it possible to disable HPA scaling when the argo rollout is progressing?

[kostis-codefresh]

If you are doing blue/green deployment use the previewReplicaCount replica to lock down preview pods.

If you are doing canary deployments use setCanaryScale to lock down canary pods.

Would that work for you?

[florianmutter]

Possibly related:

• Argo doesn't support maxSurge: 0% correctly #2239
• rollout doesn't honor canary.maxSurge and maxUnavailable settings when steps filed is defined  #3284
• Progressive option for workloadRef should take MaxSurge and maxUnavailable into account #4260

[florianmutter]

Here is an observation of ours: It seems that shifting traffic with istio does not happen as fast as stable pods are shut down if dynamicStableScale is enabled. You can see in the following graph that CPU usage of the remaining stable pods doubles because rollout shuts half of them down (weight set to 50%) before the traffic was shifted to the canary pods.

Blue line is number of stable pods, yellow line number of canary pods. The black line marks where canary pods are ready and stable pods are shut down. CPU usage increases because 90% of traffic is still routed to stable pods. This could be something that does contribute to the HPA scaling up 1 minute later.

Is there a way to make rollout scale down stable slower?

[kostis-codefresh]

But is this a problem of Argo Rollouts or Istio?

What does your Rollout look like? Do you have separate setCanaryScale/setWeight directives?

[GurayCetin]

As three steps; setWeight %10 -> %50 -> %80

spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: datadog-error-rate
                clusterScope: true
        - setWeight: 50
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: datadog-error-rate
                clusterScope: true
        - setWeight: 80
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: datadog-error-rate
                clusterScope: true

[rave2101]

@GurayCetin were you able to find any work around for this issue?