fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)

## Problem

The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.

## Fix: change the encoding, not the structure

Mint a **26-char lowercase base32hex** run id:

```
run_<24-char base32hex core><region char><version char>
      [ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```

- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.

DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.

## Residency discriminator: length → version char

`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.

## Scope / verification

- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.

## Open decisions (flagged, not silently chosen)

1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.


---

## ⚠️ Required migration — deploy in lockstep

This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):

1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
   - key `runOpsMintKsuid` → `runOpsMintKind`
   - value `"ksuid"` → `"runOpsId"`

Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: d-cs

.github/workflows/e2e.yml

@@ -42,6 +42,8 @@ jobs:
       - name: 📥 Download deps
         run: pnpm install --frozen-lockfile --filter trigger.dev...
 
+      # Prisma clients generate concurrently and share one engine binary in the
+      # pnpm store; the generate script retries the transient Windows EPERM race.
       - name: 📀 Generate Prisma Client
         run: pnpm run generate
 

apps/webapp/app/env.server.ts

@@ -1727,10 +1727,10 @@ const EnvironmentSchema = z
     RUN_REPLICATION_LEGACY_ORIGIN_GENERATION: z.coerce.number().int().default(0),
     RUN_REPLICATION_NEW_ORIGIN_GENERATION: z.coerce.number().int().default(1),
 
-    // Run-ops KSUID mint cutover — per-env, canary-first, OFF by default.
-    // Even when on, an env mints KSUID only if its per-org runOpsMintKsuid flag is
-    // "ksuid" AND isSplitEnabled() is true. Cache mirrors REALTIME_BACKEND_FLAG_CACHE_*.
-    RUN_OPS_MINT_KSUID_ENABLED: BoolEnv.default(false),
+    // Run-ops id mint cutover — per-env, canary-first, OFF by default.
+    // Even when on, an env mints run-ops ids only if its per-org runOpsMintKind flag is
+    // "runOpsId" AND isSplitEnabled() is true. Cache mirrors REALTIME_BACKEND_FLAG_CACHE_*.
+    RUN_OPS_MINT_ENABLED: BoolEnv.default(false),
     RUN_OPS_MINT_FLAG_CACHE_TTL_MS: z.coerce.number().int().default(30_000),
     RUN_OPS_MINT_FLAG_CACHE_MAX_ENTRIES: z.coerce.number().int().default(10_000),
 

apps/webapp/app/presenters/v3/ApiBatchResultsPresenter.server.ts

@@ -131,7 +131,7 @@ export class ApiBatchResultsPresenter extends BasePresenter {
   }
 
   // Split: resolve the batch row new-first then off the legacy READ REPLICA only (a batch id may
-  // be cuid or ksuid, and a cuid-shaped id can still have been backfilled onto NEW, so id-shape
+  // be cuid or run-ops id, and a cuid-shaped id can still have been backfilled onto NEW, so id-shape
   // residency is not authoritative for the row — the new-first-then-legacy probe is), then
   // hydrate every member run independently via the per-run read-through primitive.
   async #callSplit(

apps/webapp/app/presenters/v3/ApiWaitpointPresenter.server.ts

@@ -42,7 +42,7 @@ export class ApiWaitpointPresenter extends BasePresenter {
       // Public waitpoint retrieve. Split on: new run-ops client first, then the LEGACY
       // RUN-OPS READ REPLICA ONLY on a new-probe miss — never the legacy primary.
       // Split off (single-DB / self-host): one plain waitpoint.findFirst against the replica
-      // (passthrough). The waitpointId is the residency-classifiable KSUID id (the route
+      // (passthrough). The waitpointId is the residency-classifiable run-ops id (the route
       // pre-decodes the friendlyId via WaitpointId.toId).
       const hydrate = (client: PrismaReplicaClient) =>
         client.waitpoint.findFirst({

apps/webapp/app/presenters/v3/BatchListPresenter.server.ts

@@ -101,7 +101,7 @@ export class BatchListPresenter extends BasePresenter {
       }
     }
 
-    // codepoint comparator (NEVER localeCompare): BatchTaskRun.id is ASCII (cuid or ksuid).
+    // codepoint comparator (NEVER localeCompare): BatchTaskRun.id is ASCII (cuid or run-ops id).
     const sign = direction === "forward" ? 1 : -1; // forward => DESC; backward => ASC
     return Array.from(byId.values())
       .sort((a, b) => (a.id < b.id ? sign : a.id > b.id ? -sign : 0))

apps/webapp/app/presenters/v3/BatchPresenter.server.ts

@@ -79,7 +79,7 @@ export class BatchPresenter extends BasePresenter {
     const batchResult = await readThrough<BatchRow>({
       // The read-through key; here it is the batch friendlyId. A cuid-shaped batch friendlyId
       // classifies as LEGACY and the read-through probes both stores (new first, then legacy
-      // replica); a ksuid-shaped one (cut-over orgs) classifies as NEW and reads only the new
+      // replica); a run-ops-shaped one (cut-over orgs) classifies as NEW and reads only the new
       // store — either way the row is found on the DB that owns it.
       runId: batchId,
       environmentId,

apps/webapp/app/routes/api.v1.runs.$runFriendlyId.input-streams.wait.ts

@@ -78,7 +78,7 @@ const { action, loader } = createActionApiRoute(
         }
       }
 
-      // Create the waitpoint. Co-locate it with the owning run (run-ops split) so a ksuid
+      // Create the waitpoint. Co-locate it with the owning run (run-ops split) so a run-ops id
       // run's input-stream waitpoint lands on the run's DB and its block edge resolves.
       const result = await engine.createManualWaitpoint({
         runId: run.id,

apps/webapp/app/routes/api.v1.runs.$runFriendlyId.session-streams.wait.ts

@@ -99,7 +99,7 @@ const { action, loader } = createActionApiRoute(
         }
       }
 
-      // Create the waitpoint. Co-locate it with the owning run (run-ops split) so a ksuid
+      // Create the waitpoint. Co-locate it with the owning run (run-ops split) so a run-ops id
       // run's session-stream waitpoint lands on the run's DB and its block edge resolves.
       const result = await engine.createManualWaitpoint({
         runId: run.id,

apps/webapp/app/routes/engine.v1.runs.$runFriendlyId.wait.duration.ts

@@ -42,7 +42,7 @@ const { action } = createActionApiRoute(
         : undefined;
 
       const { waitpoint } = await engine.createDateTimeWaitpoint({
-        // Co-locate the waitpoint with the run that blocks on it (run-ops split): a ksuid run lives
+        // Co-locate the waitpoint with the run that blocks on it (run-ops split): a run-ops run lives
         // on the dedicated DB, but the minted waitpoint id is always a cuid, so without the run id
         // the waitpoint would route to the control-plane DB and the block edge would never resolve.
         runId: run.id,

apps/webapp/app/runEngine/concerns/idempotencyResidency.server.test.ts

@@ -16,9 +16,9 @@ function makeDeps(over: Partial<ResolveIdempotencyClientDeps>): ResolveIdempoten
     fallbackClient: FALLBACK,
     newClient: NEW_CLIENT,
     legacyClient: LEGACY_CLIENT,
-    resolveMintKind: async () => "ksuid",
+    resolveMintKind: async () => "runOpsId",
     classify: (id) => {
-      if (id.length === 27) return "NEW";
+      if (id.length === 26 && id[25] === "1") return "NEW";
       if (id.length === 25) return "LEGACY";
       throw new Error(`unclassifiable: ${id.length}`);
     },
@@ -38,10 +38,10 @@ describe("resolveIdempotencyDedupClient", () => {
     expect(client).toBe(FALLBACK);
   });
 
-  it("routes a root run to the NEW client when the env mints ksuid", async () => {
+  it("routes a root run to the NEW client when the env mints run-ops ids", async () => {
     const client = await resolveIdempotencyDedupClient(
       { environmentForMint: env, parentRunFriendlyId: undefined },
-      makeDeps({ resolveMintKind: async () => "ksuid" })
+      makeDeps({ resolveMintKind: async () => "runOpsId" })
     );
     expect(client).toBe(NEW_CLIENT);
   });
@@ -54,10 +54,10 @@ describe("resolveIdempotencyDedupClient", () => {
     expect(client).toBe(LEGACY_CLIENT);
   });
 
-  it("routes a child to the NEW client when the ksuid parent is NEW-resident", async () => {
-    const ksuidParent = RunId.toFriendlyId("a".repeat(27));
+  it("routes a child to the NEW client when the run-ops parent is NEW-resident", async () => {
+    const runOpsParent = RunId.toFriendlyId("a".repeat(24) + "01");
     const client = await resolveIdempotencyDedupClient(
-      { environmentForMint: env, parentRunFriendlyId: ksuidParent },
+      { environmentForMint: env, parentRunFriendlyId: runOpsParent },
       makeDeps({ resolveMintKind: async () => "cuid" }) // mint flag must NOT win for a child
     );
     expect(client).toBe(NEW_CLIENT);
@@ -67,7 +67,7 @@ describe("resolveIdempotencyDedupClient", () => {
     const cuidParent = RunId.toFriendlyId("b".repeat(25));
     const client = await resolveIdempotencyDedupClient(
       { environmentForMint: env, parentRunFriendlyId: cuidParent },
-      makeDeps({ resolveMintKind: async () => "ksuid" }) // mint flag must NOT win for a child
+      makeDeps({ resolveMintKind: async () => "runOpsId" }) // mint flag must NOT win for a child
     );
     expect(client).toBe(LEGACY_CLIENT);
   });