Create automation to keep Azure Pipelines variables in sync

[lbussell]

Problem

Azure Pipelines variables configured in the ADO GUI can drift from expected values over time due to manual changes, accidental modifications, or incomplete setup.
Currently there's no automated way to detect when pipeline variables are out of sync with expected defaults.

Example: The MCR status portal has a temporary outage causing us to set waitForIngestionEnabled to false instead of true. It is easy for this change to be forgotten and not reverted back to the expected value.

Proposed Solution

Create a new tool and pipeline that:

1. Reads expected pipeline variable configurations from JSON files in the repo
2. Queries Azure DevOps (via Azure SDK/.NET) to get the current pipeline variables
3. Compares expected vs actual values to detect drift
4. Files a GitHub issue when drift is detected

Configuration

Imagine a JSON metadata file co-located with each the pipeline definition:

• Pipeline: eng/pipelines/dotnet-buildtools-image-builder-official.yml
• Metadata: eng/pipelines/dotnet-buildtools-image-builder-official.metadata.json

{
  "appliesTo": [
    {
      "name": "dotnet-buildtools-image-builder-official",
      "organization": "dnceng",
      "project": "internal",
      "pipelineId": 367
    },
    // Could optionally apply to multiple pipelines
  ],
  "variables": [
    {
      "name": "imageBuilder.pathArgs",
      "value": "",
      "allowOverride": true
    },
    {
      "name": "someOtherVariable",
      "allowOverride": true
      // Absence of value means the variable should exist but the value can be anything
    },
    {
      "name": "system.debug",
      "value": "false",
      "allowOverride": true
    },
    // etc...
  ]
}

Out of scope

• Auto-remediation of drift (setting variables back to expected values)
• Secret variables

[lbussell]

Initial design/MVP:

• New dotnet tool produced by this repo. Publish to dnceng package feed just like the ImageBuilder.Models package.
• Commands:
  • init - create baselines for all pipelines in the repo
    • Option to store in one file (.pipelines.json?) or alongside each pipeline (my-pipeline.yml -> my-pipeline.metadata.json)
    • The user can then manually update/edit/remove pipelines or individual variables as needed after this.
  • variables ./my-pipeline.yml - view variables for a pipeline
  • variables validate ./my-pipeline.yml
    • Reach out to Azure DevOps to see what variables are set and compare it against the local baseline (using a json format similar to what's described above)
    • If the pipeline isn't in sync, then print specific remediation steps to the console
    • Return status code 1 or 0 depending on whether a given pipeline is in sync

Next steps:

• File issues with specific remediation steps
• One-line command to automatically re-sync a pipeline's variables