Add automated tests

[rfay]

We need automated tests for this.

• Many will need to be end-to-end, requiring an authentication technique
• Each template will need to be tested

[rfay]

Claude suggestions:

Research: Automated Testing Approaches for coder-ddev and code.ddev.com

I've researched testing patterns across the Coder ecosystem and adjacent projects (coder/registry, DDEV add-ons, terraform test framework). Here's a comprehensive breakdown.

---

Current State

The repo has some testing infrastructure already but no CI pipelines:

• make test — runs ddev --version, docker --version, node --version inside the built image. Validates binary presence only, not functional behavior.
• drupal-core/scripts/test-issue-branches.sh — manual script that tests Drupal issue branch + composer resolution across D10/D11/D12. Runs locally, not in CI.
• drupal-core/scripts/create-test-workspaces.sh — creates a matrix of test Coder workspaces (5 configurations covering D10/D11/D12 + issue forks). Has --create, --check, --delete modes. Also manual, not in CI.
• No .github/workflows/ — no automated CI runs on push/PR.
• bats-core, bats-support, bats-assert are already installed in the Docker image (via Homebrew) — unused.

---

Testing Layers

A comprehensive approach needs four layers. Each has different tooling and infrastructure requirements.

Layer 1: Static Validation (no live systems required)

terraform validate and terraform fmt --check should run on every PR. These catch HCL syntax errors and formatting issues.

# .github/workflows/validate.yml
- uses: hashicorp/setup-terraform@v3
- run: terraform init -backend=false
  working-directory: user-defined-web
- run: terraform validate
  working-directory: user-defined-web
- run: terraform fmt -check -recursive

Also useful: tflint for opinionated Terraform linting (deprecated resource arguments, required provider versions, etc.).

Layer 2: Terraform Unit Tests (no live Coder needed)

Terraform's native test framework (added in v1.6, mocking in v1.7) supports .tftest.hcl files that can run against mock providers. This is extremely useful for validating variable validation rules, required resource structure, and output correctness without requiring a live Coder deployment.

Example user-defined-web/tests/validate.tftest.hcl:

mock_provider "coder" {}
mock_provider "docker" {}

variables {
  workspace_image_registry = "index.docker.io/ddev/coder-ddev"
  image_version = "test"
}

run "workspace_creates_required_resources" {
  command = plan
  assert {
    condition     = length(docker_container.workspace) > 0
    error_message = "workspace container resource must be created"
  }
}

run "rejects_empty_image_version" {
  command = plan
  variables {
    image_version = ""
  }
  expect_failures = [var.image_version]
}

# Run all template tests
terraform test -verbose

This approach is used by the coder/registry modules and runs in ~seconds on any Linux/macOS machine.

Layer 3: Shell Script Unit Tests (Docker required, no Sysbox)

The startup script logic in template.tf (the inline startup_script) can be extracted into a testable shell script and tested with bats-core, which is already installed in the image. The DDEV project uses this pattern extensively for their add-on tests (ddev/github-action-add-on-test).

What can be tested with bats (mocking external binaries that aren't available in a normal CI runner):

• DDEV global_config.yaml is correctly placed at ~/.ddev/global_config.yaml
• The startup script sets expected environment variables (DDEV_NONINTERACTIVE, DDEV_NO_INSTRUMENTATION, etc.)
• File copy logic from /home/coder-files/ to /home/coder/ works
• WELCOME.txt is present after initialization
• npm global bin path is in $PATH
• Git SSH wrapper is configured correctly

Example tests/startup.bats:

#!/usr/bin/env bats
load 'test_helper/bats-support/load'
load 'test_helper/bats-assert/load'

setup() {
  export HOME="$(mktemp -d)"
  mkdir -p "$HOME/.ddev" /home/coder-files/.ddev
  cp image/scripts/.ddev/global_config.yaml /home/coder-files/.ddev/
}

@test "ddev global_config.yaml is placed correctly" {
  run bash -c "source startup_functions.sh && setup_ddev_config"
  assert_success
  assert [ -f "$HOME/.ddev/global_config.yaml" ]
}

@test "WELCOME.txt is present after init" {
  run bash -c "source startup_functions.sh && copy_home_files"
  assert_success
  assert [ -f "$HOME/WELCOME.txt" ]
}

This requires refactoring the inline startup script into sourced functions — which would also improve maintainability of the template itself.

Layer 4: Integration Tests (live Coder + Sysbox host required)

This is the most valuable but also most constrained layer. It requires:

• A running Coder deployment (code.ddev.com or a dedicated test instance)
• A machine with Sysbox installed as the Coder node (standard GitHub Actions runners don't support Sysbox)
• A Coder machine token for CI authentication

The canonical Coder CI pattern uses the official coder/setup-action GitHub Action:

# .github/workflows/integration-test.yml
name: Integration Tests
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  test-user-defined-web:
    runs-on: [self-hosted, sysbox]  # Requires self-hosted runner with Sysbox
    env:
      TEMPLATE: user-defined-web
      WORKSPACE: ci-test-${{ github.run_id }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: coder/setup-action@v1
        with:
          access_url: ${{ vars.CODER_URL }}
          coder_session_token: ${{ secrets.CODER_SESSION_TOKEN }}

      # Push template as inactive (won't affect existing workspaces)
      - run: |
          cp VERSION $TEMPLATE/VERSION
          coder templates push $TEMPLATE --activate=false \
            --name ci-${{ github.sha }} --yes \
            --variable workspace_image_registry=index.docker.io/ddev/coder-ddev
          
      # Create a test workspace from this specific version
      - run: |
          coder create $WORKSPACE \
            --template $TEMPLATE \
            --template-version ci-${{ github.sha }} \
            --yes

      # Wait for startup and verify
      - run: coder ssh $WORKSPACE --wait=yes -- ddev --version
      - run: coder ssh $WORKSPACE --wait=yes -- docker ps
      - run: coder ssh $WORKSPACE --wait=yes -- bash -c "test -f ~/.ddev/global_config.yaml"
      - run: coder ssh $WORKSPACE --wait=yes -- bash -c "ddev config --project-type=php --docroot=web && ddev start"

      # Promote on success
      - run: |
          coder templates versions promote \
            --template $TEMPLATE \
            --template-version ci-${{ github.sha }}

      - if: always()
        run: coder delete $WORKSPACE --yes

The key insight from Coder's docs on template testing: push with --activate=false, test against that specific version, then promote only on success. This prevents broken templates from affecting existing workspaces.

Authentication setup: Create a dedicated CI user and machine token:

coder users create --email ci@code.ddev.com --username ci-bot --login-type none
coder tokens create --user ci-bot --lifetime 8760h
# Store the token as CODER_SESSION_TOKEN in GitHub Actions secrets

---

What Tests to Write, Per Template

user-defined-web

Test	Layer	Verifies
terraform validate	Static	HCL syntax is valid
Variable validation (mock provider)	Terraform unit	Required variables enforced
make test (binary versions)	Image	ddev, docker, node present
startup script copies home files	Shell (bats)	/home/coder-files/ → /home/coder/ works
ddev --version after startup	Integration	DDEV is functional in workspace
docker ps after startup	Integration	DinD daemon is running
~/.ddev/global_config.yaml exists	Integration	DDEV config is initialized
ddev config && ddev start (PHP)	Integration	DDEV can start a project

drupal-core

This template has the most complex logic (issue branch detection, seed DB import, Composer resolution). The existing test-issue-branches.sh and create-test-workspaces.sh scripts should be the basis for CI tests here.

Test	Layer	Verifies
terraform validate	Static	HCL syntax valid
Issue branch parameter validation	Terraform unit	Branch format validated
Drupal project cloned and configured	Integration	~/drupal/web/index.php exists
Seed DB imported successfully	Integration	ddev drush status shows DB connected
Issue fork branches resolvable	Integration	Composer resolves issue branch
Works across D10, D11, D12	Integration	Matrix across drupal_version values
DDEV site accessible via port-forward	Integration	HTTP 200 from ddev describe URL

For the Drupal issue branch tests specifically, a matrix strategy is ideal:

strategy:
  matrix:
    drupal_version: ["10", "11", "12"]
    issue_branch: ["0", "3001234-fix-something"]
  fail-fast: false

freeform

Lighter template. Primarily needs the Traefik/routing behavior verified.

Test	Layer	Verifies
terraform validate	Static	HCL valid
Container starts and agent connects	Integration	Workspace reaches running state
Traefik routing works	Integration	Coder port-forward returns HTTP response

---

The Sysbox Constraint

The hardest part of full integration testing is that Sysbox cannot run on standard GitHub Actions runners. Sysbox requires kernel-level changes (installation on the host via .deb package) and cannot be installed at CI runtime without a privileged runner, which GitHub doesn't offer.

Options:

1. Self-hosted GitHub Actions runner on code.ddev.com's Sysbox-capable node — registers as runs-on: [self-hosted, sysbox]. This is the cleanest path. The runner creates real workspaces on the production or staging Coder deployment.
2. Scheduled tests on a dedicated staging Coder instance — run integration tests on a schedule (nightly) rather than on every PR. Use workflow_dispatch + a cron trigger. Keeps production clean.
3. Separate "canary" workspace — keep a persistent test workspace up, run verification checks against it on a schedule via coder ssh.
4. GitHub Actions with alternative DinD — for tests that don't need the full Sysbox stack (e.g., testing that DDEV works with a regular Docker daemon), use ddev/github-action-setup-ddev in a standard ubuntu-latest runner.

---

Suggested Phased Implementation

Phase 1 (no infrastructure needed): Add .github/workflows/validate.yml with terraform validate and terraform fmt --check for all three templates. Add terraform test with mock providers for variable validation.

Phase 2 (Docker only): Extract startup script into testable functions. Write bats tests for file initialization, DDEV config placement, PATH setup. Run in standard ubuntu-latest with Docker available.

Phase 3 (self-hosted runner): Set up a self-hosted GitHub Actions runner on the Sysbox-capable host. Add integration test workflow that creates/verifies/deletes test workspaces. Start with user-defined-web, add drupal-core matrix tests second.

Phase 4 (drupal-core specific): Move test-issue-branches.sh logic into the integration test workflow. Run the existing workspace creation matrix automatically on drupal-core template changes.

---

References

• Coder: Test Templates Through CI/CD
• Coder: Change Management / GitOps
• coder/setup-action
• coder/registry CONTRIBUTING.md — TypeScript + Terraform test patterns
• coder/terraform-provider-coderd — Terraform-native template management
• Terraform Language Tests
• bats-core
• ddev/github-action-setup-ddev
• ddev/github-action-add-on-test — DDEV bats testing patterns