Compose the serving stack from build-time component lists - #428
Draft
haarchri wants to merge 20 commits into
Draft
Compose the serving stack from build-time component lists#428haarchri wants to merge 20 commits into
haarchri wants to merge 20 commits into
Conversation
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
… staleness check Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Collaborator
Author
|
Nebius Standard: Nebius Dynamo: |
Collaborator
Author
|
AWS Standard: |
negz
reviewed
Sep 4, 2026
negz
left a comment
negz
left a comment
Collaborator
There was a problem hiding this comment.
Looks good so far, no real issues I can spot.
There was a problem hiding this comment.
🟡 Changes recommended
Cloud GPU configuration mismatches can prevent DRA operation or double-allocate GPUs, and initial reconciliation can report readiness before the stack exists.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Moves serving-stack composition to fixed build-time component lists, with AICR-generated cloud stacks and hand-written fallbacks.
Changes:
- Adds component models, cloud/stack catalogs, dependency ordering, and readiness handling.
- Adds the pinned AICR generator, Nix app, and freshness check.
- Removes runtime version overrides and adds required cloud selection.
File summaries
| File | Description |
|---|---|
apis/inferenceclusters/definition.yaml |
Updates the Vultr Kubernetes default. |
apis/servingstacks/definition.yaml |
Adds cloud selection and removes version overrides. |
apis/vultrclusters/definition.yaml |
Updates the VKE version default. |
docs/manifests/reference/servingstacks.yaml |
Updates the ServingStack example. |
docs/manifests/reference/vultrclusters.yaml |
Updates the Vultr example. |
flake.nix |
Exposes AICR and the stack generator. |
functions/compose-inference-cluster/function/fn.py |
Propagates cluster source to ServingStack. |
functions/compose-inference-cluster/tests/test_fn.py |
Updates composition expectations. |
functions/compose-serving-stack/function/fn.py |
Renders and orders component lists. |
functions/compose-serving-stack/function/gateway.py |
Renders per-cluster gateway resources. |
functions/compose-serving-stack/function/stacks/__init__.py |
Joins cloud, common, and stack components. |
functions/compose-serving-stack/function/stacks/clouds/__init__.py |
Defines the cloud package. |
functions/compose-serving-stack/function/stacks/clouds/existing.py |
Defines the Existing-cluster stack. |
functions/compose-serving-stack/function/stacks/clouds/generated/__init__.py |
Defines generated stack packages. |
functions/compose-serving-stack/function/stacks/clouds/generated/aicr/__init__.py |
Documents AICR-generated stacks. |
functions/compose-serving-stack/function/stacks/clouds/generated/aicr/aks.py |
Adds the generated AKS stack. |
functions/compose-serving-stack/function/stacks/clouds/generated/aicr/eks.py |
Adds the generated EKS stack. |
functions/compose-serving-stack/function/stacks/clouds/generated/aicr/gke.py |
Adds the generated GKE stack. |
functions/compose-serving-stack/function/stacks/clouds/nebius.py |
Defines the Nebius stack. |
functions/compose-serving-stack/function/stacks/clouds/vultr.py |
Defines the Vultr stack. |
functions/compose-serving-stack/function/stacks/common.py |
Defines shared serving components. |
functions/compose-serving-stack/function/stacks/components.py |
Defines component data types. |
functions/compose-serving-stack/function/stacks/crds/gaie.yaml |
Vendors GAIE CRDs. |
functions/compose-serving-stack/function/stacks/crds/modelexpress.yaml |
Vendors ModelExpress CRDs. |
functions/compose-serving-stack/function/stacks/dynamo.py |
Defines Dynamo-specific components. |
functions/compose-serving-stack/function/stacks/standard.py |
Defines Standard-specific components. |
functions/compose-serving-stack/generate.py |
Generates cloud stacks from AICR. |
functions/compose-serving-stack/tests/test_fn.py |
Tests rendering, ordering, and inventories. |
functions/compose-serving-stack/tests/test_stacks.py |
Tests component-list invariants. |
functions/compose-vultr-cluster/tests/test_fn.py |
Updates the expected VKE version. |
nix/aicr.nix |
Packages the pinned AICR CLI. |
nix/apps.nix |
Adds the stack-generation app. |
nix/checks.nix |
Adds generated-stack freshness checking. |
pyproject.toml |
Configures lint exceptions for generated code. |
schemas/.lock.json |
Refreshes the API schema lock. |
schemas/python/models/ai/modelplane/inferencecluster/v1alpha1.py |
Updates the generated Vultr default. |
schemas/python/models/ai/modelplane/infrastructure/servingstack/v1alpha1.py |
Reflects the new ServingStack API. |
schemas/python/models/ai/modelplane/infrastructure/vultrcluster/v1alpha1.py |
Updates the generated VKE default. |
Review details
Suppressed comments (2)
functions/compose-serving-stack/generate.py:400
- AKS GPU pools are tainted
nvidia.com/gpu=true:NoSchedule(compose-aks-cluster/function/fn.py:103), but this generatesEqualtolerations for the valuepresent. The generated NFD worker, GPU Operator daemonsets, and DRA kubelet plugin therefore do not tolerate Modelplane's GPU nodes and cannot publish ResourceSlices. Use the same taint value as the cluster composition, then regenerate the checked-in AKS stack.
"aks": ["--accelerated-node-toleration", "nvidia.com/gpu=present:NoSchedule"],
functions/compose-serving-stack/generate.py:575
- Selecting the GKE
draprofile requires GPU pools to carrygke-no-default-nvidia-gpu-device-plugin=trueand usegpu-driver-version=disabled, as this embedded profile states. The current GKE composition instead setsgpuDriverVersion="DEFAULT"and onlycloud.google.com/gke-nvidia-gpu-dra-driver=true(compose-gke-cluster/function/fn.py:380-388). This leaves GKE's managed device plugin advertising the same GPUs as the DRA driver, so two independent allocators can assign one device. Update the GKE pool/driver setup to satisfy this profile before selecting it.
if cloud == "gke":
return ["--data", str(catalog), "--profile", "gpuStack=dra"]
- Files reviewed: 31/38 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Collaborator
Author
|
local e2e test is green too: |
Collaborator
Author
|
Azure Standard: |
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Collaborator
Author
|
AKS model with standard stack: |
Collaborator
Author
|
AKS Dynamo: |
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Collaborator
Author
|
GKE Standard: |
Collaborator
Author
|
GKE Standard Model Response: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Implements the serving stack generation design (#422).
The stack is now a list of components fixed at build time.
nix run .#stacksresolves NVIDIA AICR recipes for EKS, AKS and GKE and writes them into the function's stacks package; the generator embeds Modelplane's catalog fork, pins the aicr release it is synced against, and fails closed on anything unclassified, a new upstream component, a managed values path that stopped landing, a Helm null tombstone. Nebius, Vultr and Existing are hand-written files in the same shape, so the format has no dependency on any generator. The function joins the XR's cloud and stack and renders what it gets: teardown Usages and install ordering both derive from each component's depends_on edges, and readiness from what was rendered. A stacks-current flake check regenerates in the sandbox and diffs against the tree, so a stale or hand-edited generated file fails CI.Breaking: spec.versions, spec.standard, spec.dynamo and spec.nvidiaDriverRoot are removed, and spec.cloud (set by the cluster composition) is required. The kube-prometheus-stack and DRA driver composed resources are renamed, so an upgraded cluster recreates those releases, and the DRA driver moves to the nvidia-dra-driver namespace. The generated clouds gain GPU Operator, NVSentinel, nodewright and the Prometheus operator components at the versions AICR validated; the hand-written clouds state the same pins for the components both halves share.
Validated by the unit and flake checks and on a live EKS, AKS, GKE, Nebius and Vultr Cluster (Standard and Dynamo), E2E via Kind for InferenceCluster Readiness + Model Serving
Fixes #
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.