perf(webapp,run-engine): cut CPU on the engine-facing worker-action routes (#4746)

Cuts CPU on the `engine/v1/worker-actions/*` routes a managed supervisor
calls, and adds the benchmark harness the numbers come from.

Measured on a local stack: **on-CPU per completed run 9.07ms → 6.59ms
(−27%)**, busy fraction 45.6% → 33.8%, with every worker-action p50 down
23–27%. Load was 5,000 runs / 24 virtual supervisors / 90s window /
30,120 requests / 0 errors.

Query-count work from the same investigation is deliberately **not**
here — it will follow as a separate PR.

## The three changes

**1. Split the event-loop monitor in two (~14% of on-CPU, plus ~5pp of
GC).**

`eventLoopMonitor.server.ts` installs a global `async_hooks` hook:
`init` writes a `Map` entry for *every* async resource the process
creates, `before` calls `process.hrtime()` and `context.active()` on
every one. Enabling any async hook also puts V8 on the slow path for
promise instrumentation process-wide. `EVENT_LOOP_MONITOR_ENABLED`
defaulted to `"1"`, so this was the shipping configuration.

The blocked-loop detector is now opt-in (`EVENT_LOOP_MONITOR_ENABLED`,
default `0`). The event-loop *utilization* gauge — a single interval
timer with no per-request cost — moves to its own flag
(`EVENT_LOOP_UTILIZATION_MONITOR_ENABLED`, default `1`) and stays on, so
the useful half survives without the expensive half.

A/B under identical load:

| | monitor on | monitor off | change |
|---|---|---|---|
| on-CPU per run | 9.08ms | 7.25ms | −20% |
| GC self time | 9.80% | 5.05% | −4.75pp |
| dequeue p50 | 76.6ms | 62.8ms | −18% |
| attempts/start p50 | 56.3ms | 43.5ms | −23% |

**2. Bucket route matching by first static path segment (10.4% → 3.9% of
on-CPU).**

`patches/@remix-run__router@1.23.3.patch` already memoized flattened
branches and compiled path regexes. What remained was the linear scan:
`matchRouteBranch` walked the ranked branch list calling `matchPath` per
branch across 521 route files, so every worker-action request paid a
scan proportional to the whole route table.

Branches are now indexed by their lowercased leading segment, with one
always-considered list for branches whose leading segment is dynamic,
splat or optional (and for root/pathless paths). A request walks only
its own bucket merged with that list. Route-matching self time dropped
64% (3.6s → 1.3s over a 90s window).

Ordering is preserved exactly: both lists hold indexes into the already
rank-sorted branch array and are walked in ascending-index order, so the
first match found is the same branch the full scan would have found.
Bucketing lowercases on both sides, so case-insensitive matching still
resolves and `caseSensitive: true` routes are still rejected by
`matchPath` itself. A pathname whose own leading segment can't be
bucketed falls back to the full scan.

Verified equivalent to the unpatched matcher over 20,050 pathnames
(literal, dynamic, splat, optional, case variants, basenames,
percent-encoded) with zero mismatches.
`apps/webapp/test/routeMatchingPatch.test.ts` pins the matching
semantics rather than the optimisation, so it still passes without the
patch.

**3. Demote per-heartbeat and per-dequeue `info` logs to `debug`.**

These are the two highest-rate engine calls and each wrote a synchronous
structured log line on every request. Synchronous `console` writes can
block the loop when stdout backs up, which costs more than the ~1.3% CPU
share suggests.

## The harness

Two benchmarks, neither in the default suite (they run for minutes,
attach the V8 profiler, and report numbers rather than assert on them).
See `apps/webapp/test/bench/README.md`.

- `apps/webapp/test/bench/engineHttp.bench.test.ts` — spawns a real
webapp against throwaway Postgres/Redis containers, seeds a production
environment with a promoted managed deployment, and drives a closed-loop
supervisor pool through the full lifecycle. Profiling runs over CDP
rather than `--cpu-prof` so it covers only the measured window instead
of being swamped by boot, and `performance.eventLoopUtilization()` is
sampled *inside* the webapp process.
-
`internal-packages/run-engine/src/engine/bench/runEngineLifecycle.bench.test.ts`
— drives `RunEngine` directly, profiling enqueue and lifecycle
separately so engine cost isn't mixed with request-stack overhead.
- `apps/webapp/test/bench/analyzeProfile.ts` — dependency-free
`.cpuprofile` analyzer that symbolicates through the build's source maps
and ranks CPU by package, self time and total time. Percentages are
shares of on-CPU time (V8's `(idle)`/`(program)` excluded).

`startWebapp` gains `overrideEnv`, applied after the worker-disable
defaults, so the HTTP bench can re-enable the run engine worker that
drains the master queue into the worker queues a supervisor dequeues
from.

The local OTel collector gains a traces pipeline. It only defined a
metrics pipeline, so pointing `INTERNAL_OTEL_TRACE_EXPORTER_URL` at it
locally failed and the webapp silently fell back to the console span
logger.

## Configuration

For operators upgrading:

- `EVENT_LOOP_MONITOR_ENABLED` (now defaults to `0`) — the
per-async-resource blocked-loop detector. Set to `1` to restore the
previous behaviour and keep emitting `event-loop-blocked` spans.
- `EVENT_LOOP_UTILIZATION_MONITOR_ENABLED` (new, defaults to `1`) — the
`nodejs.event_loop.utilization` gauge. Unchanged in behaviour; it just
has its own flag now so it survives turning the detector off.

## Notes for review

- `pnpm-lock.yaml` changes only because the router patch content
changed, which changes its patch hash.
- One thing the profile ruled out: with a real OTLP collector receiving
spans, tracing costs ~1.7% of on-CPU at 100% sampling and ~0.8% at the
production rate. Span shipping is not a hidden cost, so nothing here
touches it.
- Caveats on the numbers: a laptop, not production hardware, so DB and
Redis *latency* are unrepresentative (client-side CPU is what's ranked);
single webapp process; throughput varies ~5% run to run, which is why
the claims rest on on-CPU per run rather than req/s.

## Verification

- 20,050-pathname router equivalence check vs the unpatched matcher,
zero mismatches
- `apps/webapp/test/routeMatchingPatch.test.ts` (12 cases) passes
- webapp e2e smoke suite (68 tests) passes through the patched router
- run-engine suites covering the snapshot/attempt paths pass
- `typecheck`, `format`, `lint`, `knip` clean

Author: ericallam

.gitignore

@@ -87,3 +87,6 @@ ailogger-output.log
 observability-map.json
 
 .claude/worktrees/
+
+# CPU benchmark artifacts (profiles + summaries)
+.bench/

.server-changes/reduce-webapp-cpu-on-worker-routes.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Cut webapp CPU usage by about a quarter on the routes that workers call most, freeing headroom at the same request rate. Detailed event-loop blocking traces are no longer recorded by default, because producing them was itself a large part of that cost.

apps/webapp/app/entry.server.tsx

@@ -18,7 +18,7 @@ import type { OperatingSystemPlatform } from "./components/primitives/OperatingS
 import { OperatingSystemContextProvider } from "./components/primitives/OperatingSystemProvider";
 import { assertRunOpsSplitSentinel, Prisma } from "./db.server";
 import { env } from "./env.server";
-import { eventLoopMonitor } from "./eventLoopMonitor.server";
+import { eventLoopMonitor, eventLoopUtilizationMonitor } from "./eventLoopMonitor.server";
 import { logger } from "./services/logger.server";
 import { buildImgSrcDirective, parseCspImageOrigins, withImgSrc } from "./utils/cspImageOrigins";
 import { singleton } from "./utils/singleton";
@@ -360,6 +360,10 @@ if (env.EVENT_LOOP_MONITOR_ENABLED === "1") {
   eventLoopMonitor.enable();
 }
 
+if (env.EVENT_LOOP_UTILIZATION_MONITOR_ENABLED === "1") {
+  eventLoopUtilizationMonitor.enable();
+}
+
 if (remoteBuildsEnabled()) {
   console.log("🏗️  Remote builds enabled");
 } else {

apps/webapp/app/env.server.ts

@@ -995,7 +995,8 @@ const EnvironmentSchema = z
 
     CENTS_PER_RUN: z.coerce.number().default(0),
 
-    EVENT_LOOP_MONITOR_ENABLED: z.string().default("1"),
+    EVENT_LOOP_MONITOR_ENABLED: z.string().default("0"),
+    EVENT_LOOP_UTILIZATION_MONITOR_ENABLED: z.string().default("1"),
     MAXIMUM_LIVE_RELOADING_EVENTS: z.coerce.number().int().default(1000),
     MAXIMUM_TRACE_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(25_000),
     MAXIMUM_TRACE_DETAILED_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(10_000),

apps/webapp/app/eventLoopMonitor.server.ts

@@ -89,25 +89,51 @@ function after(asyncId: number) {
   }
 }
 
+/**
+ * Per-async-resource blocked-loop detection. This is the expensive half: the
+ * hook fires for every async resource the process creates, and enabling any
+ * async hook also puts V8 on the slow path for promise instrumentation
+ * process-wide. On a request-heavy instance it costs roughly a seventh of all
+ * on-CPU time, which is why it is opt-in rather than on by default.
+ */
 export const eventLoopMonitor = singleton("eventLoopMonitor", () => {
   const hook = createHook({ init, before, after, destroy });
 
-  let stopEventLoopUtilizationMonitoring: () => void;
-
   return {
     enable: () => {
       console.log("🥸  Initializing event loop monitor");
 
       hook.enable();
-
-      stopEventLoopUtilizationMonitoring = startEventLoopUtilizationMonitoring();
     },
     disable: () => {
       console.log("🥸  Disabling event loop monitor");
 
       hook.disable();
+    },
+  };
+});
+
+/**
+ * The cheap half: a single interval timer reading `eventLoopUtilization()`.
+ * It costs nothing per request, so it stays on by default and is what a
+ * high-traffic instance should rely on when the async hook is too expensive.
+ */
+export const eventLoopUtilizationMonitor = singleton("eventLoopUtilizationMonitor", () => {
+  let stop: (() => void) | undefined;
 
-      stopEventLoopUtilizationMonitoring?.();
+  return {
+    enable: () => {
+      if (stop) {
+        return;
+      }
+
+      console.log("🥸  Initializing event loop utilization monitor");
+
+      stop = startEventLoopUtilizationMonitoring();
+    },
+    disable: () => {
+      stop?.();
+      stop = undefined;
     },
   };
 });

apps/webapp/package.json

@@ -25,7 +25,8 @@
     "upload:sourcemaps": "bash ./upload-sourcemaps.sh",
     "test": "vitest --no-file-parallelism",
     "test:perf": "vitest --config ./vitest.perf.config.ts --run",
-    "eval:dev": "evalite watch"
+    "eval:dev": "evalite watch",
+    "test:bench": "vitest --config ./vitest.bench.config.ts --run"
   },
   "dependencies": {
     "@ai-sdk/openai": "^3.0.0",

apps/webapp/test/bench/README.md

@@ -0,0 +1,91 @@
+# Engine CPU benchmarks
+
+Two benchmarks for the paths the production engine service spends its CPU in, plus a
+`.cpuprofile` analyzer. Neither runs in CI: they take minutes, attach the V8 profiler, and
+report numbers rather than assert on them.
+
+| bench | what it covers | where |
+| --- | --- | --- |
+| `engineHttp.bench.test.ts` | the full request stack for `engine/v1/worker-actions/*` | `apps/webapp` |
+| `runEngineLifecycle.bench.test.ts` | run-engine and run-queue with no HTTP in the way | `internal-packages/run-engine` |
+
+Artifacts (profiles + JSON summaries) land in `.bench/` at the repo root, which is gitignored.
+
+## HTTP bench
+
+Measures what a managed supervisor actually does: dequeue, start attempt, heartbeat,
+read latest snapshot, complete attempt. Needs a built webapp.
+
+```bash
+pnpm run build --filter webapp
+cd apps/webapp
+pnpm run test:bench
+```
+
+It spawns a real webapp against throwaway Postgres and Redis containers, seeds a production
+environment with a promoted managed deployment, fills the worker queue over the public
+trigger API, then drives a closed-loop supervisor pool for the measured window.
+
+The webapp is spawned with `--inspect` and profiled over CDP, so the profile covers only the
+measured window rather than boot. Event-loop utilization is sampled **inside** the webapp
+process over the same connection.
+
+Knobs:
+
+| var | default | meaning |
+| --- | --- | --- |
+| `BENCH_RUNS` | 1200 | runs queued before the window opens |
+| `BENCH_SUPERVISORS` | 16 | concurrent virtual supervisors |
+| `BENCH_HEARTBEATS` | 2 | heartbeats per run |
+| `BENCH_DURATION_MS` | 60000 | measured window |
+| `BENCH_SAMPLING_INTERVAL_US` | 200 | V8 sampling interval |
+| `BENCH_PROFILE_NAME` | `engine-http` | artifact basename |
+| `BENCH_EXTRA_ENV` | — | JSON merged into the webapp's env |
+| `BENCH_OUT_DIR` | `<repo>/.bench` | artifact directory |
+
+`BENCH_EXTRA_ENV` plus `BENCH_PROFILE_NAME` is how you A/B a single flag:
+
+```bash
+BENCH_RUNS=5000 BENCH_SUPERVISORS=24 BENCH_DURATION_MS=90000 \
+  BENCH_PROFILE_NAME=engine-http-no-elm \
+  BENCH_EXTRA_ENV='{"EVENT_LOOP_MONITOR_ENABLED":"0"}' \
+  pnpm run test:bench
+```
+
+Run the same size for both arms and compare `on-cpu ms per completed run` rather than
+throughput: throughput on a laptop moves ~5% run to run, on-CPU per unit of work is far
+steadier.
+
+## Run-engine bench
+
+No HTTP, no webapp: drives `RunEngine` directly so engine and queue costs are not mixed with
+request-stack overhead. Profiles two phases separately, because blending them hides which one
+owns a hot frame.
+
+```bash
+cd internal-packages/run-engine
+pnpm run test:bench
+```
+
+Knobs: `BENCH_RUNS`, `BENCH_CONSUMERS`, `BENCH_HEARTBEATS`, `BENCH_CONCURRENCY_LIMIT`,
+`BENCH_SAMPLING_INTERVAL_US`, `BENCH_OUT_DIR`.
+
+The driver shares a process with the code under measurement, so its own cost is in the
+profile. It is a thin await loop and appears under its own frames rather than smeared across
+engine frames.
+
+## Analyzing a profile
+
+```bash
+pnpm --filter webapp exec tsx test/bench/analyzeProfile.ts .bench/engine-http.cpuprofile --top 30
+```
+
+Three views: CPU by bucket (which package owns the cycles), hottest frames by self time (what
+to go fix), and hottest frames by total time (entry points, and a check that the load
+exercised the route mix you intended). Frames are symbolicated through the build's source
+maps, so bundled chunks report as the source files they came from.
+
+Percentages are shares of **on-CPU** time, with V8's `(idle)` and `(program)` excluded. A
+share of wall clock would make everything look cheap whenever the bench was IO-bound.
+
+`--json <path>` writes the full analysis for diffing two runs.

apps/webapp/test/bench/analyzeProfile.ts

@@ -0,0 +1,66 @@
+#!/usr/bin/env tsx
+/**
+ * Ranks where a `.cpuprofile` spent its cycles.
+ *
+ *   pnpm --filter webapp exec tsx test/bench/analyzeProfile.ts <profile> [--top 40] [--json out.json]
+ *
+ * `--root` overrides the repo root used to make source paths relative and to
+ * find the build's source maps; it defaults to the repo containing this file.
+ */
+import { readFileSync, writeFileSync } from "node:fs";
+import { resolve } from "node:path";
+import { analyzeProfile, formatAnalysis, type CpuProfile } from "./lib/profileAnalysis";
+
+function parseArgs(argv: string[]): {
+  profilePath?: string;
+  top: number;
+  json?: string;
+  root: string;
+} {
+  const here = typeof __dirname === "string" ? __dirname : import.meta.dirname;
+
+  const defaults = {
+    top: 30,
+    root: resolve(here, "..", "..", "..", ".."),
+  };
+
+  let profilePath: string | undefined;
+  let top = defaults.top;
+  let json: string | undefined;
+  let root = defaults.root;
+
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i]!;
+    if (arg === "--top") {
+      const raw = argv[++i];
+      const parsed = Number(raw);
+      if (!Number.isFinite(parsed) || parsed <= 0) {
+        console.error(`--top expects a positive number, got "${raw ?? ""}"`);
+        process.exit(1);
+      }
+      top = parsed;
+    } else if (arg === "--json") json = argv[++i];
+    else if (arg === "--root") root = resolve(argv[++i]!);
+    else if (!arg.startsWith("--")) profilePath = arg;
+  }
+
+  return { profilePath, top, json, root };
+}
+
+const { profilePath, top, json, root } = parseArgs(process.argv.slice(2));
+
+if (!profilePath) {
+  console.error("usage: analyzeProfile.ts <path-to-.cpuprofile> [--top N] [--json out.json]");
+  process.exit(1);
+}
+
+const profile = JSON.parse(readFileSync(profilePath, "utf8")) as CpuProfile;
+const analysis = analyzeProfile(profile, root);
+
+console.log(`\n=== ${profilePath} ===`);
+console.log(formatAnalysis(analysis, top));
+
+if (json) {
+  writeFileSync(json, JSON.stringify(analysis, null, 2));
+  console.log(`\nwrote ${json}`);
+}