WIP: Collect and normalize engine metrics with a MetricMapping

[dennis-upbound]

Description of your changes

Towards #269, implementing the collection and normalization halves of the design in #363.

MetricMapping landed as a data resource that nothing consumed. Applying one reported Ready and changed no behaviour, because the collector the design hands mappings to did not exist. Building it turned up three more things the design assumed were already true, so the commits are ordered by what unblocked the next.

The tree did not build. flake.nix lists compose-metric-mapping and the uv2nix resolver builds its package set from uv.lock, which had no such workspace member, so nix failed at evaluation before any build ran. That also blocked schema regeneration, which the function depends on.

The engine's serving port was unnamed on every backend, so anything scraping it had to match a number. Under prefill/decode that number is wrong — the pd-sidecar takes 8000 and the engine moves to its own port, so a target matching 8000 reports the proxy's metrics as the engine's. It is now named http on the Standalone, llm-d and decode paths alike.

Nothing stamped an engine-type label. modelplane.ai/engine existed only in the MetricMapping XRD's own description, so the selector could never match. engines[].type is new on ModelDeployment and ModelReplica, copied down beside phase, and stamped on serving pods. It is free-form rather than an enum so a platform team can map a forked engine without a Modelplane release, and an engine declaring no type is scraped under its native names rather than renamed by a guess.

Then the collector. compose-serving-stack reads every MetricMapping as a required resource and renders each into OTTL: one statement per rename and per added label, each gated on the engine the mapping selects. The gate keeps this label-driven rather than name-driven, so two mappings can rename the same source name and an unmapped engine's series pass through untouched.

What is verified

Run end to end on a real GKE cluster with vLLM 0.23.0 serving Qwen2.5-0.5B on an NVIDIA L4, and on the local two-cluster e2e harness.

On GKE the engine published 359 metric lines and the collector re-exposed the mapped ones renamed, every series carrying the engine label:

modelplane_requests_waiting{engine="vllm",instance="10.1.3.19:8000",job="modelplane-engines",model_name="qwen2.5-0.5b"}
modelplane_requests_running{...}
modelplane_time_to_first_token_{bucket,sum,count}{...}
modelplane_request_latency_{bucket,sum,count}{...}

The rename is in place, not a duplication: all four mapped source names are absent from the exported set afterwards. The remaining 308 unmapped lines passed through. nix run .#e2e -- --verify also passes, asserting the same thing against a mock engine that serves vLLM-named metrics.

The affected checks pass: test-/ty-compose-serving-stack, the same pair for compose-model-replica and compose-metric-mapping, uv-lock, and the repo-wide python check. uv-lock and both compose-metric-mapping checks were failing before this.

Four defects the unit tests could not catch

Worth reading, because all four passed a green test suite. Each test asserted what a builder returned; none asserted the function did anything.

_otel_release() was defined and never called. compose() had no collector step and never requested the mappings — request was not even imported. The feature did not exist, and every builder test passed.

The collector's Service exposed nothing. ports.prom-export set containerPort without servicePort, and the chart builds the Service from servicePort.

The Service name was derived from the release name, so nothing could address the collector reliably. It is pinned with fullnameOverride, the way the Prometheus release in the same function already does it.

The collector was composed but never listed in mark_readiness, so Crossplane reported it unready on every reconcile. That blocked the ServingStack, which blocked BackendReady, which meant an InferenceCluster could never reach Ready — on any cluster, with or without a MetricMapping. The local e2e passed straight through it, because that assertion reads the collector's exporter directly and a collector that works but is never reported as working looks identical to success. Only a real cluster, where something downstream depends on that readiness, surfaced it.

Tests were added for the last one in both directions, and I confirmed it fails with the fix reverted rather than passing vacuously.

A behaviour the design got slightly wrong

The design says an unmapped engine "still gets scraped under its native names". The collector's exporter replaces : with _, so vLLM's vllm:gpu_cache_usage_perc is published as vllm_gpu_cache_usage_perc. Passthrough keeps the name but not the punctuation. That makes the upgrade note two changes rather than one, and it is now written down in the docs page instead of implied.

Docs

collecting-engine-metrics.md told an operator to hand-write a PodMonitor, port-forward the workload Prometheus and query vllm:num_requests_running. All three are now wrong. The PodMonitor manifest is deleted and the page says to delete an existing one, since applying it alongside the composed collector scrapes every engine twice.

Still open

The EPP half of #363 is untouched. Its metrics port does not exist either — the container exposes grpc on 9002 and grpc-health on 9003, its args carry no metrics flag, and I have not confirmed --metrics-endpoint-auth=false exists on llm-d-router-endpoint-picker:v0.9.0.

compose-serving-stack hardcodes modelplane.ai/engine, modelplane_ai_serving and http by value, because a composition function cannot import another's package. compose-model-replica owns all three, so this is a fourth cross-function naming contract in this codebase and I have added to a problem rather than fixed one.

engines[].type is a property named type inside an OpenAPI schema. Legal, and ugly to read in the XRD. I took the trade because engines: [{name: qwen, type: vllm}] is what people read, but it is a one-word change to engineType if that is the wrong call.

I have:

• Read and followed Modelplane's contribution process.
• Run the affected checks and made sure they pass. Not the full nix flake check — the cross-arch image builds fail locally on arm64 for reasons unrelated to this change.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

🤖 Generated with Claude Code

[dennis-upbound]

Closing this one too, on the same reasoning: it implements #363, which is still under review, so the same "are we aligned on direction yet" question applies.

The branch dennis/metrics-poc stays, and the description above records what it proved against vLLM 0.23.0 on a real GKE cluster along with four defects worth keeping visible, because all four passed a green test suite — most importantly that the collector was composed but never marked ready, which blocked BackendReady and meant an InferenceCluster could never reach Ready on any cluster, whether or not a MetricMapping existed.

Two findings from it are already in #363 rather than sitting only here: the exporter rewrites : to _, so passthrough keeps an unmapped engine's metric name but not its punctuation, and the EPP half of the design has no metrics port to scrape yet.