Consider: Encrypt secrets in pillar files at rest

[jpmckinney]

[Drafted by Claude]

Production credentials (database passwords, API keys, etc.) are stored in plaintext in the private pillar repository. This means:

• Anyone with read access to the repository has every production credential.
• A leaked personal access token, or a compromised laptop with a clone, exposes all secrets at once.
• Secrets persist in git history, so removing one requires rotation, not just deletion.
• GitHub itself is in the trust boundary: a GitHub account or infrastructure compromise is a credential compromise.

Since we deploy with salt-ssh (no master), pillar rendering happens on the machine running run.py, so encryption at rest in git protects the repository without changing the deployment trust model — whoever can run salt-ssh already has root SSH to all targets.

Proposal

Encrypt secret values with sops using age keys.

sops encrypts at the value level, so pillar files keep readable keys (db_password: ENC[...]) — diffs and PR review stay meaningful. age keypairs are single short strings (no GPG keyring), and multi-recipient encryption means each deployer's key can decrypt; offboarding is re-encrypting without that key (plus rotating what they could read).

• Add .sops.yaml listing deployers' age public keys and a path regex matching secret pillar files.
• Each deployer keeps their age private key in ~/.config/sops/age/keys.txt (outside the repo).
• Editing workflow: sops edit pillar/private/foo.sls.

Integration with run.py

run.py currently ends with os.execvp("salt-ssh", sys.argv), introduced in 8c6857f to keep salt-ssh in the foreground and propagate its exit code (the prior subprocess.Popen detached and always exited 0). execvp replaces the process, so the wrapper can't clean up decrypted files after salt-ssh finishes.

Options:

1. Decrypt in the wrapper. Before invoking salt-ssh, decrypt matching pillar files into a gitignored directory; replace os.execvp with subprocess.run(["salt-ssh", *sys.argv[1:]]) + sys.exit(returncode) inside try/finally that deletes the decrypted files. Preserves the foreground/exit-code semantics of 8c6857f while allowing cleanup.
2. sops exec-file. Let sops own the temp-file lifecycle and keep execvp untouched. Awkward with multiple secret pillar files.
3. Salt's GPG renderer instead of sops. Zero wrapper changes (#!yaml|gpg header, gpg_keydir in Saltfile), but whole-value GPG blobs, no meaningful diffs, and GPG key management for every deployer.

Option 1 is preferred.

Migration

1. Generate age keys for each deployer; commit .sops.yaml.
2. Encrypt existing secret pillar files with sops encrypt --in-place.
3. Update run.py per option 1; document setup in the deploy docs.
4. (TBD) Rotate all credentials currently in git history — they remain exposed to anyone who has ever cloned the repository, and encryption going forward doesn't retroactively protect them.

Non-goals

• Protecting the deployment path itself: salt-ssh copies rendered pillar data to targets over SSH regardless. The threat addressed here is repository/GitHub exposure.
• Vault or cloud secret managers: dynamic credentials and audit logs aren't worth running another critical service at our scale.