feat(migrate): add migration command suite - #479
Draft
royanger wants to merge 33 commits into
Draft
Conversation
…`.settings` `clerk migrate run` wrote a `.settings` file into the working directory to remember what it last imported. The CLI cannot gitignore that on the user's behalf, so it lands inside the repository being migrated — and it carried the Firebase signer key, a secret, as plaintext JSON. That state now lives in the `migrations` section of the CLI's own config file, keyed by project through `getProjectKey()` (linked profile, then git remote, then directory). This is the shape `clerk webhooks listen` already uses for its relay token, so `migrations` sits beside `relay` with the same accessor pair. The Firebase hash parameters are dropped from persistence rather than moved: remembering a secret writes it to disk wherever the file lives. They now fall back to `CLERK_FIREBASE_SIGNER_KEY`, `CLERK_FIREBASE_SALT_SEPARATOR`, `CLERK_FIREBASE_ROUNDS` and `CLERK_FIREBASE_MEM_COST`, so a repeat run still need not re-type four flags, and `.env.local` is already gitignored. No migration path for existing `.settings` files: `clerk migrate` is unreleased, so nothing in the wild has one.
…v vars the way the secret key does Two related gaps. **The env vars did not work from a file.** `CLERK_FIREBASE_*`, `AUTH0_*` and the `*_DB_URL` vars were read straight off `process.env`, which the shipped binary never populates from a `.env` file — it is compiled with `--no-compile-autoload-dotenv`. Only an exported shell variable reached them. The secret key avoided this by parsing the project's env files itself. That lookup is now shared: `findKeyInProject` moves out of `keyless-target.ts` into `lib/dotenv.ts` as `findEnvValue`, and every migration value goes through it. Each resolution reports its source, so `--verbose` names the file a value came from. Left on `process.env`: `CLERK_MIGRATE_RATE_LIMIT`, `CLERK_MIGRATE_CONCURRENCY_LIMIT` and `FIREBASE_AUTH_EMULATOR_HOST` — runtime knobs rather than project config, and `resolveLimits` is sync on a hot path. **There was no way to see or change what a run would pick up.** `clerk migrate settings` lists every setting with its value and its source, `set` changes one, `clear` forgets them. Values are split by what they are, not by which command wrote them: project state to the CLI config, credentials to `.env.clerk-migrate`. That file is the migration's own rather than the app's `.env.local`, because a Firebase signer key is of no use to the application being migrated and does not belong in the file its developers read daily. It is added to `.gitignore` on creation — reusing `ensureGitignoreEntry`, promoted from `keyless.ts` to `lib/git.ts` for the second caller — and deleted when `clear` removes its last value. Credentials are redacted everywhere they are shown, `--json` included.
…e column alignment
The list showed seven kebab-case identifiers and nothing else, so `file` and
`skip-unsupported-providers` read as jargon rather than as anything a user could
act on. A description column now carries the prose the registry already held.
The names stay kebab-case on purpose: each one is identical to the `migrate run`
flag it backs, so `firebase-signer-key` and `--firebase-signer-key` are one knob
reached two ways rather than two spellings to learn. Sentence case belongs in
the description, which is where it now is.
Also fixes the alignment. `column()` pads to the visible width before colouring;
the previous code coloured first and then hand-compensated for the escape bytes
with `dim("—").length - 1`, which only held for unset rows — any row with a
value pulled `SOURCE` and everything after it out of line.
…` to a trailing parenthetical Matches how the README heading already scopes it (`--skip-unsupported-providers (Supabase)`), and leaves the description reading as one sentence rather than a label plus a colon.
`migrate logs list|clean|convert` and `migrate transformers list` printed flush-left with no intro/outro frame, while `migrate run`, `migrate delete` and every `migrate export` already wrapped — as do the pre-existing commands they mirror (`apps list`, `mcp list`, `unlink`, `config pull`). Follows `commands/mcp/list.ts`: the `--json` early return stays outside the gutter so machine-readable output is unchanged, and only the human path wraps. `withGutter` no-ops outside human mode, so agent output is untouched, and it turns a cancelled prompt into `└ Paused` rather than `└ Failed` — which `logs clean` and `logs convert` both needed.
…d icon conventions
Five formatting mismatches between `migrate` and every command around it.
**Summaries escaped the gutter.** `run` and `delete` printed their final
summary through `log.raw`, the un-prefixed channel meant for machine-readable
output, so it landed flush-left and broke the `┌ … │ … └` frame. `log.info`
runs each line through `applyPrefix`, so only the channel changes.
**Spinners lost the `...` convention.** All 39 pre-existing `withSpinner` call
sites end their message in `...` and none passes a done-message — the
vanishing ellipsis *is* the completion signal. Migrate had inverted both
halves. Adds the ellipsis to every message and `spinner.update()` string, and
drops the four done-message arguments so the stop text derives from the
message like everywhere else.
**No Next steps blocks.** Every comparable command closes with one. Adds
`MIGRATE_DONE`, `MIGRATE_DELETE` and `MIGRATE_EXPORT`, wired through the
gutter's `setNextSteps`. `reportExport` now returns the steps rather than
hand-rolling a `dim("Next: …")` line, which keeps the six export modules'
call shape intact. An export of zero users returns none — there is nothing to
import — so `setNextSteps` ignores an empty list instead of rendering a
header with no bullets under it.
**`user(s)` pluralization.** The CLI's form is `${n} thing${n === 1 ? "" : "s"}`.
Migrate used `user(s)`/`row(s)` in ten places while using the correct form in
others.
**`●`/`○` status icons**, which appear nowhere else in the codebase, become
the established `✓`/`✗`/`!` vocabulary from `doctor` and `init`.
`MultiSelectPrompt` has always bound `a` to toggle every option, but clack's instruction footer never listed it and takes no override — so a genuinely useful key stayed undiscoverable unless each call site spelled it out in its own message, which is worse: it is a property of the prompt, not of any one question. `MULTISELECT_INSTRUCTIONS` is the only seam, and it is read fresh on every render. Inserted second-to-last so `Enter: confirm` stays where readers expect it. `i` (invert) is left out deliberately: it is rarely what anyone wants, and a four-item legend stops being scannable. The test lives outside `prompts.test.ts`, which mocks the whole module. What is worth checking is that the real clack export is still a live array read at render time — an upgrade that froze it, replaced it, or rendered a copy would drop `a: all` silently and nothing else in the suite would notice.
The report already knew which instance settings would cost users; acting on it meant leaving the CLI for the dashboard. A human run now offers one selectable change per flagged row, before the import confirmation, and writes the selection as a single `PATCH` of the instance config document — the same document `clerk config patch` writes. These are offers, not corrections. A flagged setting is not a wrong setting: an instance that genuinely requires an email address is configured exactly as its owner intended, and fixing the export may well be the right answer. Nothing is preselected, and selecting nothing continues to the import with the instance untouched. The redraw after a write is computed from the write, not from a second settings fetch. Clerk's Frontend API is eventually consistent, so a `/v1/environment` read issued this soon after routinely still reports the pre-write settings and would redraw every row the operator just cleared. The offer then repeats while anything is still flagged: applying one change routinely leaves others worth making, so reaching the second never costs a second run of the command. Email and phone take two writes rather than one — they are verifiable attributes, and Clerk rejects one that is on with no way to verify it, while switching it off empties `verification_strategies`. To make the offer answerable, the report itself now leads with **outcomes** rather than per-field coverage: each user is classified once, into the worst outcome that applies to them, so the ✗/⚠/✓ totals add up to the file. A required identifier rejects a user outright; a required password does not, because the import sends `skip_password_requirement`. The field rows below no longer restate user counts, which read as contradicting that block. "If you import them, this applies to them too" names what is masked behind a rejection. A user who is not being created cannot lose a field, so a setting affecting only rejected users costs nothing today — right up until the requirement rejecting them is relaxed, at which point all of it lands at once. Surfacing it up front collapses apply → re-check → discover → apply into one decision. Stands down with a warning rather than a failed run when the instance cannot be resolved, and for keyless applications, whose Backend API has no route for any of these settings.
`run` is registered `isDefault`, so `clerk migrate --transformer clerk --file users.json` has always worked and is the shorter spelling. Every example, error message, next-step line and README invocation now uses it. `clerk migrate run` stays addressable — scripts and older docs use it — but nothing points there. The group's own help follows `clerk config`: options stay on the subcommands, and `migrate --help` is a list of subcommands plus examples covering each one. `transformers` had no examples block at all; it does now. Two fixes this turned up: - A partial `CLERK_FIREBASE_*` set left in `.env.clerk-migrate` failed *every* subsequent run, including a Supabase one that never asked for Firebase. That was a regression from routing those values through the env file: previously only explicit flags could trigger the all-four-or-nothing check. A partial set that came from saved config is now warned about and ignored; a partial set that came from flags still fails, because there the user did ask. - `readme.test.ts` resolved a documented command to its group and read only that group's options, so every `clerk migrate --transformer …` example looked like it used a flag the binary rejects. It now follows the default subcommand, the same way Commander does. `migrate delete`'s description said "in this directory"; the record it reads has been keyed by project since the `.settings` removal.
…rmer is firebase `migrate run` is one command serving every platform, so a `CLERK_FIREBASE_SIGNER_KEY` left in `.env.clerk-migrate` after a Firebase migration was in scope for whatever ran next. A complete leftover set was resolved and passed along on a Supabase import; a partial one failed that import outright, naming four `--firebase-*` flags the user had not used and did not need. The gate now sits before the lookup rather than being a filter after it: any transformer but `firebase` returns immediately, without reading the environment, the env files, or even its own flags. Nothing downstream misused the value — only the Firebase transformer reads it off `TransformContext` — but resolving it at all is what let stale config warn and fail unrelated runs. The previous fix only covered the partial case, and did it transformer-blind. Moved to `lib/firebase-hash.ts` so the wizard can resolve after the platform is picked without importing from `run.ts`, which imports the wizard. That also keeps the interactive path: choosing Firebase with all four already set skips the prompt, choosing anything else never looks. The per-platform export commands need no equivalent gate — `migrate export auth0` reads `AUTH0_*` and nothing else, because there the command *is* the platform. `migrate run` is the only one that spans them.
`clerk migrate export` writes real user records to ./exports, and the Firebase export is driven by a service account key people download into the checkout. Neither belongs in the repository, and both are one `git add -A` away from it.
`setMigrationEntry` mutated the in-memory config and returned without writing it, so nothing recorded what the last import did. `clerk migrate delete` reads that entry to find the users to undo, and with it never written the undo path had nothing to work from.
…clerk-secret-key `run` was registered `isDefault`, so `clerk migrate` on its own meant "import". That reads fine until `migrate export` sits beside it: one direction is implied by the bare group name and the other has to be spelled out. Both are named now, and bare `clerk migrate` prints help. The `--clerk-secret-key` alias goes with it. It was carried over from the standalone migration tool, but `migrate` ships new in this CLI — there is no released spelling to stay compatible with, so there is nothing to deprecate.
Two exports of the same platform used to write the same path, so the second silently overwrote the first. Filenames now carry a local `YYYYMMDD-HHmm` stamp, and every export settles its destination *before* it starts — one prompt, prefilled with the proposed path, so Enter accepts it. Asked up front on purpose: coming back to a long export stalled on a prompt, with every user held in memory and nothing on disk, is the worse half of that trade. `--output` is an answer already given, and agent mode takes the proposal without asking.
Dashboards hand out `postgres://user:[YOUR-PASSWORD]@host/db` and people paste their real password in verbatim. A `#`, `@` or `/` in it makes the whole string unparseable, here and later inside `Bun.SQL` — and the prompt is masked, so the paste that failed is not even visible to check. `normalizeConnectionString` percent-encodes the userinfo when the raw string will not parse, splitting on the LAST `@` so an unencoded one inside the password does not end the userinfo early. Strings that already parse are returned untouched, so a correctly encoded password is never double-encoded.
Every other resolver in the CLI answers "where do I operate?" with the linked project, silently. For an export that default is actively dangerous: the linked instance is normally the migration's *destination*, so taking it without asking is how a run exports an instance and imports it straight back into itself. So a resolved instance is no longer taken silently — the account's instances are offered, one flat row each (`my-app - Production instance (ins_…)`) rather than an application picker followed by an instance picker. An application is not what an export reads from; an instance is, and dev and prod are different user pools. The resolved application's instances lead the list, so taking one is still a single Enter. `--secret-key` still names an instance outright and runs unquestioned.
`export firebase` without `--service-account` exited with a usage error, which is a dead end in the interactive picker: choose Firebase, get told to re-run with a flag. It now prompts, the way `export supabase` prompts for its connection string. The answer can be a path to the downloaded file *or* the key's JSON pasted whole, so a key kept in a password manager or a CI secret never has to be written to disk. Prompted as a password, since the JSON carries a private key. Agent mode has nobody to ask, so it still names the flag.
A section heading and a leading sentence, matching how `--help` lays out its own sections, with each description wrapped rather than run off the edge. Width is capped at 80 columns, not merely measured, so two runs of the same command lay out the same way on different terminals. A backticked span is never broken across lines: `log.info` pairs backticks per line, so a split span leaves an unmatched backtick on each and colours the wrong half of both. No gutter — this reads a static registry, it does not run anything — and no dimmed text, which the descriptions are the whole point of.
`migrate import` now writes `import-<timestamp>.log` and `migrate delete` writes `delete-<timestamp>.log`, so a listing points at the command behind each line. The old `migration-` and `user-deletion-` names, written by the standalone tool and earlier CLI builds, still classify and convert. `migrate logs list` leads with the filename (what `logs convert` and `logs clean` talk about), renders the UTC stamp in the reader's own zone, prints the log directory relative to the cwd, and closes with a fixed legend of every kind rather than only the ones present.
`[REDACTED]` was local to `lib/users.ts`; `clerk migrate settings` needs the same string so a withheld credential reads identically wherever the CLI declines to show one.
…ngs list Firebase hands its scrypt parameters over as `base64_signer_key`, `rounds` and friends, and every guide — Clerk's own standalone script included — tells you to paste them into `.env` under those names. Those spellings, and their `FIREBASE_` prefixed forms, now resolve as aliases behind the `CLERK_FIREBASE_*` variables, read from one registry shared by the listing and the import. `migrate settings list` gains the orientation lines, count and next-steps block the CLI's other listings carry, attributes an environment value to the env file it actually came from (Bun loads `.env.local` before the CLI runs, so "`ROUNDS` env var" named nothing the reader could edit), names the alias alongside the file, and withholds credentials as `[REDACTED]` rather than a head-and-tail truncation.
Bun reports every connection-level failure — DNS, refused, no route — as a bare `Error` reading "Unable to connect. Is the computer able to access the url?". It names neither the host nor what wanted it, and the global handler could only render it as `unexpected_error`. Connection failures now surface as a `CliError` naming the host, under the new `network_unreachable` code. Everything else, an aborted request included, is left exactly as thrown.
Migration logs are the only record of which users landed and which failed, and `migrate delete` reads them to undo a run — so where they go is worth one question, asked before the first log file is written. `import`, `export` and `delete` now start at `startLogging()`, which settles the directory (`CLERK_MIGRATE_LOG_DIR`, then the saved `log-dir`, then `./logs`) and asks a human who has chosen neither. The answer is saved under the new `log-dir` setting, so the question is asked once per project and never again; `-y`, agent mode and a non-TTY take `./logs` and save nothing, leaving the question open for the first interactive run. `logs list|clean|convert` resolve the directory without ever asking: they are read-only, and "where should logs go?" is not a question to put in front of someone who asked to see the logs they already have. `log-dir` is the first setting kept in the config that also answers to an environment variable, so `settings list` checks the environment for a config setting too — a listing that showed the remembered path while the run read another is the one thing the source column exists to prevent.
`clerk migrate settings clear <name>` forgets a single setting and leaves the rest of the project alone; with no name it still clears both stores, as before. Both stores are cleared either way, because a setting can sit in either and `log-dir` can sit in both — clearing half of one is worse than clearing none, since the command would report the setting gone while the next run still read it. An `env` value goes under every spelling the setting answers to, so dropping `CLERK_FIREBASE_ROUNDS` no longer leaves a bare `ROUNDS` behind to win the next resolution. `.choices()` rejects an unknown name before the action runs, so the friendly "Unknown setting" errors inside `set.ts` and `clear.ts` were unreachable from the CLI and a one-character miss got back only the list of eight names. The argument's parser now names the near miss first — `logs-dir` suggests `log-dir` — while leaving whether a value is allowed to Commander.
…dev user limit
`DEV_USER_LIMIT` was 500 and enforced: an import of more users into a
development instance was refused outright. Both halves were wrong. The limit
a development instance is created with is 100, Clerk raises it per instance
on request, and the real value (`max_allowed_users`) is served by no public
API — so the number can never be known to be this instance's, and refusing
blocked imports the destination would happily accept.
The import now reads the live user count from `GET /v1/users/count`,
measures the file against the headroom that implies, and warns when it does
not fit — naming what the instance already holds and roughly how many users
will be rejected. A human is asked whether to continue; `-y` and agent mode
proceed on the warning alone. The final prompt then restates the split
("Import 1 user and expect 1 to fail?") rather than a number the instance
will not take.
The summary's error breakdown gains notes for the two errors that read as
account-level restrictions and are not: blocked SMS countries (a per-instance
blocklist, with development instances pointed at Clerk's test numbers and
production at the Dashboard setting) and the user quota. Both messages point
at "contact support", which is the wrong first move for most readers. After a
partial import the next steps now lead with the grep that names which users
failed and why, since the breakdown only counts each error.
Without this the first complaint came from deep inside the secret-key chain, which resolves the linked profile before it ever asks for a token — so a signed-out operator in an unlinked directory was told to run `clerk link`, a command that would only turn around and ask them to sign in. Both failures landed after the wizard had already walked them through picking a platform and a file. `migrate import` now checks for somewhere to import *into* first, mirroring `resolveBapiSecretKey`: `--secret-key`, `--app`, `CLERK_SECRET_KEY` and an unclaimed accountless application each name the destination on their own. A human gets the same sign-in-then-link flow `clerk link` already runs; an agent, which can answer neither a browser login nor an application picker, gets an error naming whichever half is missing.
`--db-url "libsql://app-org.turso.io"` fell through to the SQLite default and `bun:sqlite` tried to open a local file by that name. A libsql URL now routes to the server's HTTP pipeline endpoint instead: `bun:sqlite` only opens local files, and `@libsql/client` ships native optional dependencies that do not survive `bun build --compile`, so the wire protocol is fewer lines than the dependency would be. The client reports itself as `sqlite`, since that is the dialect — nothing downstream branches differently. The token comes from `?authToken=` on the URL, the form the Turso CLI prints, or from `TURSO_AUTH_TOKEN`/`LIBSQL_AUTH_TOKEN`; a self-hosted sqld with auth disabled needs neither. Redaction covers the query parameter as well as userinfo, so a token cannot reach an error message or `--verbose` output, and a 401 is explained rather than left as a bare status.
🦋 Changeset detectedLatest commit: 4bf8048 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| await fetchInstanceSettings("sk_test_abc"); | ||
|
|
||
| const urls = mockFetch.mock.calls.map(([input]) => String(input)); | ||
| expect(urls.every((url) => !url.includes("satellite.example.com"))).toBe(true); |
royanger
force-pushed
the
ra/integrate-migration-tool-into-cli
branch
from
September 11, 2026 04:15
67dda1a to
4bf8048
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.