Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ jobs:
- run: uv run ruff format --check
- run: uv run pytest
- run: uv build
- run: uv tool install dist/*.whl && uv tool run db-cutover --help
- run: uvx --from ./dist/*.whl db-cutover --help
- run: test -f "$(uvx --from ./dist/*.whl db-cutover skill)/SKILL.md"

postgres-integration:
runs-on: ubuntu-latest
Expand Down
61 changes: 61 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Release

on:
push:
tags: ['v*']

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
with:
python-version: '3.11'
- name: Check the tag matches the package version
run: |
version="$(uv version --short)"
if [ "v$version" != "$GITHUB_REF_NAME" ]; then
echo "tag $GITHUB_REF_NAME does not match version $version" >&2
exit 1
fi
- run: uv sync --frozen --group dev
- run: uv run pytest
- run: uv build
- name: Check the built package installs and includes the skill
run: |
uvx --from ./dist/*.whl db-cutover --version
test -f "$(uvx --from ./dist/*.whl db-cutover skill)/SKILL.md"
- name: Write the release notes from the changelog
run: |
version="${GITHUB_REF_NAME#v}"
awk -v v="$version" '$1 == "##" { on = ($2 == v) ; next } on' CHANGELOG.md > release-notes.md
test -s release-notes.md
- uses: actions/upload-artifact@v4
with:
name: release
path: |
dist/
release-notes.md

publish:
needs: build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/project/db-cutover/
permissions:
id-token: write # PyPI trusted publishing
contents: write # the GitHub release
steps:
- uses: actions/download-artifact@v4
with:
name: release
- uses: astral-sh/setup-uv@v6
- run: uv publish --trusted-publishing always dist/*
- run: gh release create "$GITHUB_REF_NAME" dist/* --repo "$GITHUB_REPOSITORY" --title "$GITHUB_REF_NAME" --notes-file release-notes.md --verify-tag
env:
GH_TOKEN: ${{ github.token }}
21 changes: 19 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# db_cutover_orchestrator
# db-cutover

`db-cutover`: a Python CLI for zero-downtime blue/green PostgreSQL cutovers on RDS (logical replication plus a
PgBouncer switch). `README.md` covers use, `docs/procedure.md` the steps of a cutover, `docs/design.md` the
Expand All @@ -12,6 +12,9 @@ reasoning. Read `SECURITY.md` before handling operational configuration or logs.
`kubectl port-forward`. `shell.py`: aws, kubectl, pg_dump, psql. `output.py`: log lines, checks, JSON.
- `commands/`: `inspect.py` (read-only and long-running), `blue.py`, `replication.py`, `switch.py` (cutover,
finish), `terraform.py` (green-tf, promote-tf).
- `skills/db-cutover/`: the [Agent Skill](https://agentskills.io/specification) for agents that operate the tool
(`SKILL.md` plus `references/`). The wheel includes it as `db_cutover/skills/db-cutover`, and `db-cutover skill`
prints or installs it. It is written for an agent running a cutover; this file is for an agent changing the code.

## Rules

Expand All @@ -29,4 +32,18 @@ reasoning. Read `SECURITY.md` before handling operational configuration or logs.
- Logic that can be pure (sequence planning, HCL edits, ini parsing) stays pure and has tests in `tests/`.
- Long-running commands print one line per change, not per poll, so they work as background monitors.
- Before committing: `uv run pytest`, `uv run ruff check .`, `uv run ruff format --check .` (CI runs the same).
- When behaviour changes, update `README.md`, `docs/procedure.md` and `docs/design.md` in the same commit.
- When behaviour changes, update `README.md`, `docs/procedure.md`, `docs/design.md` and the skill in the same
commit. A new command needs a line in `skills/db-cutover/references/commands.md` (a test checks).
- The skill's frontmatter has only `name`, `description`, `license` and `metadata`. Keep `SKILL.md` under 500
lines with the rules first, and write standing instructions for "you" (the agent) and "the operator".

## Releasing

1. Set the version in `pyproject.toml`, `src/db_cutover/__init__.py`, the skill's `metadata.version` and the
README's install command, run `uv lock`, and give `CHANGELOG.md` a `## <version> (<date>)` section. A test
checks that they agree.
2. Merge to `main`, then tag that commit `v<version>` and push the tag.
3. The `Release` workflow builds the package, checks the tag against the version, publishes to PyPI with trusted
publishing (no token: PyPI trusts this repository's `release.yml` in the `pypi` environment) and creates the
GitHub release from the changelog section. PyPI never accepts the same version twice, so a mistake needs a new
version.
10 changes: 9 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
# Changelog

## Unreleased
## 0.2.0 (2026-11-16)

The first release on PyPI.

- Install with `uv tool install db-cutover` or `pipx install db-cutover`. `db-cutover --version` prints the version.
- Add an agent skill (`skills/db-cutover`) for planning, rehearsing and running a cutover with an operator: each
command and its dry run, the stop points, the rehearsal checklist and recovery. It ships inside the package:
`db-cutover skill` prints its directory and `db-cutover skill --install DIR` copies it into an agent's skills
directory.
- README: who the tool is for, a rehearsal walkthrough, and its guarantees and limits.
- Stop and require recovery after a partial or unknown PgBouncer reload. Never resume mixed BLUE/GREEN targets.
Report failed resumes and post-switch verification failures with unsuccessful exit status.
- Require `--execute` for `probe`, `wait-sync`, and `pgbouncer resume`. `verify` is read-only by default;
Expand Down
134 changes: 108 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,30 @@ and switching application traffic through PgBouncer. The application keeps using
At the switch, PgBouncer holds new queries while in-flight transactions finish, then sends queued work to the new
instance.

This repository contains the Python CLI, an [example configuration](examples/config.toml), a
[step-by-step procedure](docs/procedure.md), and [design notes](docs/design.md). It is intended for a human operator
working with a coding agent. Each command has a narrow purpose and every command that changes database,
infrastructure, or PgBouncer state requires `--execute`.
`db-cutover` is the Python CLI that runs each step. It comes with an [example configuration](examples/config.toml),
a [step-by-step procedure](docs/procedure.md), [design notes](docs/design.md) and an
[agent skill](skills/db-cutover/SKILL.md). It is made for a human operator working with a coding agent: each command
has a narrow purpose, and every command that changes database, infrastructure or PgBouncer state is a dry run until
it gets `--execute`.

**Release candidate:** failure-path tests and PostgreSQL replication tests do not replace a rehearsal on your
application's RDS and PgBouncer deployment. Read the requirements below before planning a cutover.

## Who it's for

Teams that run PostgreSQL on RDS behind PgBouncer in Kubernetes and need a major version upgrade (or a move to a new
instance) without a maintenance window. It suits you if:

- you want to choose the moment of the switch and see every check before it, instead of handing the upgrade to a
managed service;
- your application already connects through PgBouncer with transaction pooling, or can be made to;
- someone on the team can provision an RDS instance and change a Kubernetes ConfigMap.

It is not for databases older than PostgreSQL 16, setups without PgBouncer, or schemas that rely on large objects,
materialized views, unlogged tables or foreign tables. For those, look at
[RDS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) or
`pg_upgrade` with a maintenance window.

## Requirements

- PostgreSQL **16 or newer**, running on RDS. BLUE is the existing primary; GREEN is a separate instance at the
Expand All @@ -33,17 +49,17 @@ application's RDS and PgBouncer deployment. Read the requirements below before p

## Install

Install Python 3.11+ and [uv](https://docs.astral.sh/uv/). Clone the repository and use its locked dependencies:
`db-cutover` needs Python 3.11 or newer. Install it as a standalone tool with [uv](https://docs.astral.sh/uv/) or
[pipx](https://pipx.pypa.io/):

```sh
git clone https://github.com/DocSpring/db-cutover.git
cd db-cutover
uv sync --frozen
uv run db-cutover --help
uv tool install db-cutover==0.2.0 # or: pipx install db-cutover==0.2.0
db-cutover --version
db-cutover --help
```

For an operational run, check out a reviewed release tag or commit and keep it fixed throughout the migration.
A standalone installation can use `uv tool install .` from that checkout.
Pin the version and keep it fixed from the first rehearsal to the end of the migration. To run from source instead,
clone the repository at a release tag and use its locked dependencies (`uv sync --frozen`, then `uv run db-cutover`).

The machine running the tool also needs `aws`, `kubectl`, and matching, current `pg_dump`/`psql` clients at least as
new as **both** database servers. `op` is optional for 1Password references; Terraform is optional for the two HCL
Expand All @@ -54,11 +70,30 @@ and supply credentials through environment variables or a secret manager. Never
schema dumps, reports, Terraform state or logs. `cmd:` secret references execute shell commands: treat configs as
trusted executable input.

## Working with an agent

The package includes an [Agent Skill](https://agentskills.io/specification) that teaches a coding agent (Claude
Code, Codex, Cursor and others) to plan, rehearse and run a cutover with you: what each command does, which ones
need `--execute`, the rehearsal checklist, the points where it must stop for your decision, and recovery.

```sh
db-cutover skill --install .claude/skills # or .agents/skills, or your agent's skills directory
npx skills add DocSpring/db-cutover # the same skill, straight from this repository
```

Give the agent the config path, the approved versions, the traffic controls and the limits of what it may execute.
Have it run each changing command without `--execute` first, explain the output, and follow your agreed execution
policy. A failed gate is a stop condition.

Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
stopped; quiet output alone is not proof that a monitor is still alive.

## Workflow

Read [the procedure](docs/procedure.md) for the provisioning, permissions, traffic controls and recovery steps
between these commands. In the examples below, `C` is a private config path and `TARGET_VERSION` is the RDS engine
version you have selected and tested. Prefix commands with `uv run` if you used the checkout installation.
version you have selected and tested. Prefix commands with `uv run` if you run from a source checkout.

```sh
db-cutover "$C" preflight
Expand Down Expand Up @@ -92,19 +127,64 @@ Start the probe after `prepare-blue`; stop it before `finish` drops its table. `
read-only; `--execute` adds a marker write to test replication. Dry runs can connect, read data, create local state
files, and consume database resources. They do not pause clients or modify remote state.

## Working with an agent

Give the agent [AGENTS.md](AGENTS.md), the config path, the approved versions, the traffic controls and the
operator's authorization boundaries. Have it run each changing command without `--execute` first, explain the
output, and follow the operator's agreed execution policy. A failed gate is a stop condition.

Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
stopped; quiet output alone is not proof that a monitor is still alive.

Before RELOAD, a cutover failure attempts to resume BLUE and reports any recovery failure. After RELOAD starts,
unknown or mixed targets **remain paused** and the command fails. An operator must align every pod to GREEN before
resuming. Never interpret replication or a sequence gap as permission to write to both databases.
## Rehearse first

Rehearse twice: the whole procedure on staging, then the pause on production.

1. **Build a staging setup that matches production** where it matters: the same PgBouncer image, pool mode and
ConfigMap mount, the same PostgreSQL source and target versions and extensions, and the application with its
real timeouts. Put traffic on it.
2. **Run every step there, including a real switch**, with `watch` and `probe --execute` running. Write down how
long the copy took, how much WAL BLUE kept, and the pause the switch reports.
3. **Cause the failures on purpose.** Leave a direct writer connected, hold a transaction open on BLUE, press
Ctrl-C during a rehearsal's pause, restart a PgBouncer pod after the ConfigMap names GREEN. Each one should stop
the switch and leave traffic on BLUE. Practise the [recovery steps](docs/procedure.md#recovery) for the last one.
4. **On production, rehearse the pause** once replication has caught up and the direct writers are stopped:

```sh
db-cutover "$C" cutover --execute --rehearse
```

This pauses live traffic, runs every check the real switch runs, and resumes on BLUE. Its report gives the pause
in milliseconds. Compare it with every timeout in the request path before you go on.

`--rehearse` never reloads PgBouncer, so only the staging switch tests RELOAD, GREEN under the application and
recovery from a partial switch. The skill's
[rehearsal checklist](skills/db-cutover/references/rehearsal.md) lists each thing to check and record.

## Guarantees and limits

What the tool does:

- **Nothing changes without `--execute`.** A dry run reads, checks and reports. It can connect, read data, create
local state files and use database resources, but it never pauses clients or changes remote state.
- **No switch without proof that GREEN is current.** With every PgBouncer pod paused, the tool checks that no
client session is left on BLUE, writes a marker on BLUE and waits for it on GREEN, and compares the highest ID of
every table before any pod reloads.
- **A failure before RELOAD leaves BLUE as primary.** The tool resumes traffic on BLUE, including on Ctrl-C, and
reports a resume it could not confirm.
- **It never resumes pods that point at different databases.** After RELOAD starts, mixed or unknown targets
**remain paused** and the command fails. An operator aligns every pod to GREEN before resuming.
- **It keeps secrets and row data out of its output.** PostgreSQL diagnostics, query text and health response
bodies are withheld. State files go in a private directory.

What it can't do:

- **Promise zero failed requests.** Queries wait during the pause. Requests succeed only if every timeout in their
path is longer than the pause and recovery works. An RDS reboot can take much longer than a switch.
- **Fence clients it doesn't know about.** Anything that connects without PgBouncer has to be stopped and kept
away by you. The tool checks for sessions; it can't prevent new ones.
- **Close the window between the ConfigMap change and the switch.** A PgBouncer pod restarted in that window
serves GREEN early. The tool detects it before switching but can't stop it.
- **Go back to BLUE once GREEN has taken writes.** Replication runs one way. Never read replication or a sequence
gap as permission to write to both databases. After the switch, fix forward on GREEN.
- **Survive its own process being killed.** After SIGKILL or a lost connection during a switch, an operator has to
read the state from the pods.
- **Prove the data is identical.** `verify` compares schemas, counts and sampled rows. It is not a full comparison.
- **Copy everything.** Roles, grants, extensions, database settings and other databases need separate preparation.
Sequence values are advanced, not replicated.

The [design notes](docs/design.md) give the reasoning, and [SECURITY.md](SECURITY.md) the limits of the tests.

## Development

Expand All @@ -118,4 +198,6 @@ uv build

The default suite uses no cloud credentials. [Integration tests](tests/integration/README.md) use disposable
PostgreSQL 16 and 17 containers with synthetic data. CI runs both suites. See [SECURITY.md](SECURITY.md) for data
handling and release checks. Licensed under [MIT](LICENSE).
handling and release checks, and [AGENTS.md](AGENTS.md) for the code's rules and how to release.

Built by [DocSpring](https://docspring.com). Licensed under [MIT](LICENSE).
31 changes: 29 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,30 @@ version = "0.2.0"
description = "Agent-friendly tools for zero-downtime blue/green PostgreSQL cutovers on RDS with logical replication and a PgBouncer switch"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
license = "MIT"
license-files = ["LICENSE"]
authors = [{ name = "DocSpring, Inc.", email = "support@docspring.com" }]
keywords = ["postgresql", "rds", "pgbouncer", "logical-replication", "blue-green", "upgrade", "zero-downtime"]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: System Administrators",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: Database",
"Topic :: System :: Systems Administration",
]
dependencies = ["psycopg[binary]>=3.2,<4"]

[project.urls]
Repository = "https://github.com/DocSpring/db-cutover"
Changelog = "https://github.com/DocSpring/db-cutover/blob/main/CHANGELOG.md"
Issues = "https://github.com/DocSpring/db-cutover/issues"
Documentation = "https://github.com/DocSpring/db-cutover/blob/main/docs/procedure.md"

[project.scripts]
Expand All @@ -18,12 +37,20 @@ db-cutover = "db_cutover.cli:main"
dev = ["pytest>=8", "ruff>=0.6"]

[build-system]
requires = ["hatchling>=1.25"]
requires = ["hatchling>=1.27"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/db_cutover"]

# The agent skill lives at the repository root, where `npx skills add DocSpring/db-cutover` finds it, and ships
# inside the package, where `db-cutover skill` finds it.
[tool.hatch.build.targets.wheel.force-include]
"skills/db-cutover" = "db_cutover/skills/db-cutover"

[tool.hatch.build.targets.sdist]
include = ["/src", "/skills", "/tests", "/docs", "/examples", "/CHANGELOG.md", "/SECURITY.md", "/AGENTS.md"]

[tool.ruff]
line-length = 120
target-version = "py311"
Expand Down
Loading
Loading