Experiments Integration With Rollouts: Pods Spun Up By Experiments Also Get Selected By The Service For The Rollout Pods

[satyamsareen007]

Summary

We are using the basic canary strategy, i.e no traffic router. Our use case is to run tests against the canary version, and if the results look good, then start with the rollout and start diverting traffic to the canary version.

We have a service with selector: app:microservice that selects the rollout pods and forwards real customer traffic.

When we tried integrating experiments with the rollout as the first step, with the following test config:

spec:
  replicas: 5
  strategy:
      steps:
      - experiment:
          duration: 10m
          templates:
          - name: canary-vesion-experiment
            specRef: canary
            service:
              name: canary-service-experiment
      - setWeight: 20
      - pause: {duration: 120}
      - setWeight: 100

The pods spun up by the experiment also have the label app:microservice, which makes our original service also select the experiment pods along with the rollout pods, thus actual customer traffic also flows to the experiment pods, which is not desirable.

Is there a way we can spin up an experiment without our original service selecting those pods? So that the experiment can run in isolation and doesn't affect the real customer traffic.

I know there is a standalone experiments resource as well, but is there a way we can mention the standalone experiments resource as the first step in the rollout and when it succeeds, the other steps of setting canary weight start to execute.

How can we make sure the rollout triggers once the standalone experiments resource has succeeded and not proceed with the rollout if it fails?

Or if there is a way we can update the pods labels in the inline way decalring the experiments so that the original service doesn't select the experiment pods.

Use Cases

Run tests against the ephemeral canary replica using experiments while the original/main service, accepting the real customer traffic, does not select pods spun by the experiment. if the experiment succeeds, proceed with the rollout and if it fails, don't proceed with the rollout/rollout enters a degraded state.

If a standalone experiment resource is way to go, how can we mention the experiment in the rollout as the first step.

---

Message from the maintainers:

Need this enhancement? Give it a 👍. We prioritize the issues with the most 👍.