JupyterHub on GCP: transparent Cloud Build images + run-scoped Dask Gateway clusters (LCR-174)#163
Conversation
…lusters (LCR-174) On a lightcone JupyterHub deployment (hub-deploy, dask-cloudbuild branch) `lc build` and `lc run` now work with zero configuration: - New `kubernetes` container runtime: the Dask worker pod *is* the container, so recipes run unwrapped and Containerfile specs resolve to registry refs (`$LIGHTCONE_REGISTRY/lc-<project>:<hash>`) — the same content-addressed identity as local tags, spelled for a registry. - engine/cloudbuild.py: build backend selected by the deployment's env contract (LIGHTCONE_REGISTRY + LIGHTCONE_BUILD_BUCKET). Freshness is one registry HEAD; a build tars the staged (hashed) context to the build bucket and runs through GCP Cloud Build as the deployment's build SA. Auth = Workload Identity via the metadata server; pure urllib, no SDK, no git remote needed. - cluster_for_run grows a Gateway branch: create a run-scoped cluster with the project image, adapt 1..jobs, wait (bounded) for the first worker, and cull it on exit — create/cull per run is what keeps the image fresh, since a Gateway cluster's image is fixed at creation. The executor rejoins the cluster by name through the Gateway API (gateway:// can't be dialled by a bare Client). - Rule output now returns through the Dask future result and is printed by the driver — the only channel that reaches the terminal from Gateway worker pods — which also retires the cross-node stdout flock. Bootstrap failures (no sentinel output) forward a bounded raw tail. - Site detection by env markers (DASK_GATEWAY__ADDRESS): runtime kubernetes, scratch = $HOME (the NFS volume shared with workers; no separate scratch space on the hub). - Gateway snakemake invocation: --latency-wait 120 (NFS sync, LCR-177) and --shared-fs-usage without software-deployment so workers spawn their own `python`, not the driver's sys.executable. - `lc init` on the hub scaffolds a worker-capable Containerfile (dask/dask-gateway/snakemake/lightcone-cli pinned to the ambient hub versions, uid-1000 user) — LCR-176 Part A. - Manifests additionally record the pod-reported LIGHTCONE_WORKER_IMAGE as execution ground truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lightcone-cli | a45e432 | Commit Preview URL Branch Preview URL |
Jul 22 2026, 09:42 PM |
The deployment injects DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE (the notebook image) as the client's ambient default; lc run's explicit image kwarg must beat it or every cluster would run the notebook image instead of the lc-build one. dask-gateway's _submit applies config defaults first, then kwargs — a regression test now locks that in (dask-gateway added to the dev group). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
✅ Eval Results
Graders✅ spec_valid (1.00) Full output |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
Spawned snakemake job commands carry no --directory; remote executors are expected to cd themselves (the official kubernetes executor does the same). Local/SLURM workers inherit the driver's cwd by accident, but Gateway worker pods start in the image's WORKDIR (e.g. /app) — the child snakemake resolved relative paths there and died on a read-only .snakemake. Found in the live e2e test on the lightcone hub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
On a kubernetes deployment `lc run` hashes the registry ref into each manifest's code_version, but the status walker resolved Containerfile specs to local-store tags — so every freshly materialized output read as stale. New container.runtime_registry() is the single source of truth for which spelling of the image identity a runtime uses; the Snakefile generator and status walker both go through it. Found in the live e2e test on the lightcone hub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
…ronment` option The deployment's cluster-options handler no longer needs any lightcone-specific injection: lc run reads the gateway's declared options and passes, through the stock `environment` option, the DASK_DISTRIBUTED__WORKER__RESOURCES__* scheduling contract (mirrored from the declared worker_cores/worker_memory), the driver's HOME/USER/LOGNAME (passwd-less uid-1000 images crash getpass.getuser() without them), and LIGHTCONE_WORKER_IMAGE (manifest ground truth, now covering the deployment-default-image case too). Ambient environment defaults declared by the deployment are preserved underneath. This makes the hub's dask-gateway config near-stock 2i2c daskhub — the matching hub-deploy simplification removes the whole env block from its options handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2
|
Streamlining update ( Validated on the deployed hub after rolling the near-stock config: forced full re-run of 🤖 Generated with Claude Code |
| * ``--shared-fs-usage`` without ``software-deployment`` — with it | ||
| (the default) the child snakemake command embeds the *driver's* | ||
| ``sys.executable``, a path that doesn't exist inside the worker | ||
| image; without it, workers invoke plain ``python`` from their own | ||
| PATH. Everything else stays shared: the NFS home carries the | ||
| project, sources, and snakemake persistence. |
There was a problem hiding this comment.
Shouldn't this 'shared-fs-usage' without software-deployment be the default? does it break anything in the other execution backends if we have it everywhere?
There was a problem hiding this comment.
Yes — it's now unconditional (93a2117). I checked snakemake's source: dropping software-deployment from --shared-fs-usage only changes the spawned job command from embedding the driver's sys.executable to invoking plain python (plus skipping --scheduler-solver-path/--conda-base-path, which we don't use). That's equally correct on local and SLURM, where workers inherit the driver's activated environment — the one scenario that changes is running lc outside an activated env (e.g. a bare pipx shim with no venv on PATH), which the project-venv convention already rules out. Validated with a fresh lc init → lc run → lc verify locally.
There was a problem hiding this comment.
Flagging an interaction with the plugin de-bundling PR (#161) before we resolve this thread — the two PRs make opposite assumptions about the project venv, and this change tightens the coupling.
Making --shared-fs-usage exclude software-deployment makes the activated environment load-bearing for rule execution on the local and SLURM backends, in two places:
- Driver: the top-level command is a bare
snakemakeconsole-script, sosnakemakemust be on PATH. - Spawned jobs: the dask executor now spawns plain
python -m snakemake …, sopythonon the worker's PATH must be able to importsnakemake+ the dask executor plugin.
Both resolve today only because lc init installs lightcone-cli (hence the whole snakemake/dask stack) into the project .venv, which the session-start hook activates. That's the "project-venv convention rules it out" guarantee cited above — and this change now leans on it for spawned jobs too, not just the driver. The SLURM branch already hard-requires it (dask_cluster.py aborts if dask isn't on PATH inside the allocation's activated env).
The tension: #161 (de-bundling) moves to an empty project venv with a global lc. If that lands, the local/SLURM path here breaks — the driver's snakemake isn't on PATH (a global uv tool install exposes only lc/astra), and spawned python -m snakemake hits ModuleNotFoundError. The hub/Gateway path is unaffected: worker pods get the stack from the image via requirements.txt (added by this PR), never from the venv.
So whichever of #163/#161 merges second has to reconcile this. If we do drop the venv install, the options are roughly:
- Invoke snakemake through lc's own interpreter (
[sys.executable, "-m", "snakemake", …]) and re-addsoftware-deploymentso spawned jobs use the same interpreter — clean, but partially reverts this thread and reintroduces a per-backend branch for the gateway (where the driver's interpreter path doesn't exist in the worker image). - Keep a stack-bearing env on PATH some other way (a global env exposing
snakemake/dask), leaving the empty project venv fine.
Not blocking this PR — just flagging so we resolve this thread with eyes open rather than silently.
Generated by Claude Code
…affold, uniform invocation - dask-gateway moves from the [gateway] extra (and dev group) into the main dependencies; the ImportError fallback in the gateway branch is gone with it. - `lc init` scaffolds one Containerfile everywhere: no on-hub branching, no baked-in uid/USER (pod identity is deployment config — hub-deploy sets runAsUser on cluster pods), no per-dist worker-stack pins. requirements.txt instead pins lightcone-cli itself, which now carries the whole execution stack (snakemake, dask, distributed, dask-gateway) as normal dependencies. - The snakemake invocation is identical on every backend: --latency-wait is dropped (no longer observed on the hub), and --shared-fs-usage minus software-deployment is now unconditional — spawned jobs run plain `python` from the worker's own environment on all backends (worker image on the gateway; the driver's activated env locally and on SLURM). The `gateway` parameter of _build_snakemake_cmd is gone. - site_registry: keep scratch_root=$HOME on the hub but document why it can't be omitted (the tempdir fallback is pod-local, so worker-side .snakemake metadata would be lost and every run would look stale). Validated end-to-end locally: lc init → lc run (rules spawn bare `python -m snakemake` through the dask executor) → lc status / lc verify green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live run on the redeployed hub failed with `getpwuid(): uid not found: 1000`: notebook pods don't necessarily export USER/LOGNAME (their own passwd entry covers getpass there), so the forward-if-set approach provisioned nothing — and the now environment-agnostic worker image has no passwd entry for the pod uid, so the child snakemake crashed at getpass.getuser(). Always set USER/LOGNAME in the provisioned worker environment, deriving the name on the driver via getpass (env first, passwd fallback; "jovyan" as the last resort). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Review round addressed in The first live run on the redeployed hub caught one gap — notebook pods don't export 🤖 Generated with Claude Code |
Developer Certificate of OriginThis PR has been approved. Before it can be merged, all contributors must sign the Developer Certificate of Origin. Statusnone yet
How to signPost the following comment exactly as written:
|
2 participants
Summary
Implements the Kubernetes/JupyterHub-on-GCP PRD (LCR-174) against the hub-deploy
dask-cloudbuilddeployment: on a lightcone hub,lc runmakes sure the project image is up to date (building through GCP Cloud Build when needed), creates a run-scoped Dask Gateway cluster with that image, runs the pipeline in worker pods, and culls the cluster when the run finishes — the exact same zero-configuration UX as on a laptop or a SLURM allocation. Everything keys off the env contract the deployment injects into user pods (DASK_GATEWAY__*,LIGHTCONE_REGISTRY,LIGHTCONE_BUILD_BUCKET,LIGHTCONE_BUILD_SERVICE_ACCOUNT); nothing is configured by the user.Design
One new runtime, not a new subsystem. The spec already routes everything through two seams — the container runtime (
wrap_recipe, image resolution) and the cluster context (cluster_for_run) — so the k8s integration is a value flowing through existing seams:kubernetesruntime (engine/container.py): the worker pod is the container.wrap_recipeis a passthrough (no nested containers), and Containerfile specs resolve to registry refs. One content-addressed identity everywhere:image_identity()yields the digest that spellslc-<project>-<hash>in a local store and$LIGHTCONE_REGISTRY/lc-<project>:<hash>in the registry — socode_version, staleness, and reruns work identically on every backend.engine/cloudbuild.py, ~400 lines): selected when the deployment contract is present. Freshness is a single registry HEAD on the content-addressed ref — unchanged files never rebuild, never even upload. A build tars the staged build context (the exact file set the tag hashes), uploads it to the deployment's GCS bucket, and submits a Cloud Build job that runs as the deployment's least-privilege build SA. Auth is the pod's Workload Identity via the metadata server — pureurllib, no SDK dependency, no git remote, no stored credentials. Failures surface the build-log tail.cluster_for_run(engine/dask_cluster.py): create a cluster with the project image as theimagecluster option, scale adaptively1..--jobs, wait (bounded, default 600 s) for the first worker so an unpullable image fails loudly instead of hanging at zero workers, and shut the cluster down on exit — success or failure. Create/cull per run is PRD decision Add Dagster execution layer for reproducible analysis pipelines #1 and is also what makes image updates seamless: a Gateway cluster's image is fixed at creation. A startup assertion refuses clusters whose workers don't advertise thecpus/memoryresource contract (the other silent-hang mode).environmentoption — theDASK_DISTRIBUTED__WORKER__RESOURCES__*scheduling contract mirrored from the declaredworker_cores/worker_memory, the driver'sHOME/USER/LOGNAME(passwd-less uid-1000 images crashgetpass.getuser()without them), andLIGHTCONE_WORKER_IMAGE(covering the deployment-default-image case too), preserving any deployment-declared environment defaults underneath. The hub's options handler reduces to the stock 2i2c daskhub shape (cores/memory/image/environment) plus the per-user home mount — zero lightcone-specific injection server-side (hub-deploye10088c).gateway://schedulers can't be dialled by a bareClient, solc runhands the executor the cluster name (LIGHTCONE_GATEWAY_CLUSTER) and it rejoins through the authenticated Gateway API as a guest (shutdown_on_close=False)._run_shellnow returns(exit_code, sentinel_block)and the driver prints each finished rule's block — the one channel that works uniformly across LocalCluster threads, srun workers, and Gateway pods. This retires the cross-node stdoutflockentirely (one less NFS/DVS concern), and a child snakemake that dies before the rule body (missing package in the worker image) forwards a bounded raw tail instead of vanishing (PRD architecture bullet / LCR-175 semantics). Legacy bare-intresults from older workers are still accepted (version skew on image-based deployments). The executor also prefixes every spawned job withcd <workdir>— spawned snakemake commands carry no--directory, and Gateway worker pods start in the image'sWORKDIR, not the project (local/SLURM workers only worked by inheriting the driver's cwd).engine/site_registry.py): a pod's hostname is noise;DASK_GATEWAY__ADDRESSis the signal. Thejupyterhubsite declarescontainer_runtime: kubernetesandscratch_root: $HOME— no separate scratch space on the hub; home is the NFS volume shared with every worker pod, so driver and workers see one consistent snakemake state.--shared-fs-usageminussoftware-deploymentis unconditional, so spawned jobs always run plainpythonfrom the worker's own environment (the worker image on the gateway; the driver's activated env locally/SLURM) instead of embedding the driver'ssys.executable. No gateway-specific flags remain (--latency-waitdropped — the NFS latency was not reproducible in recent tests).lc initwrites the same Containerfile everywhere;requirements.txtpinslightcone-cli, which carries the whole execution stack (snakemake, dask, distributed, dask-gateway — now a normal dependency, no[gateway]extra). No uid/USER is baked into the image: cluster pods run as the notebook uid via the deployment'ssecurityContext(hub-deployc8dc6cd), andgetpass.getuser()works through the lc-provisionedUSER/LOGNAME/HOME(LCR-176 Part A).code_versionhashes the registry ref (Containerfile edit → new ref → rerun), and a new additiveworker_imagefield recordsLIGHTCONE_WORKER_IMAGE— provisioned into every pod by lc itself — as ground truth of what actually executed.A spec declaring several distinct containers is rejected on this backend with guidance (the worker pod image is fixed per run — LCR-176 note); a project with no containers runs on the deployment's default image.
Out of scope (per PRD)
lc runcalls).Relation to #162
Same PRD, different mechanism: #162 built through the BinderHub service (git-ref-based, requires committed/pushed state and a GitHub connect flow). This PR targets the newer Cloud-Build-based deployment (hub-deploy
dask-cloudbuild), where builds are git-free tarball submissions — which removes the GitHub coupling and most of the surface area (~1.1k src lines vs ~4k).Testing
just lintclean (ruff + strict mypy), 388 tests pass (+55 new: Cloud Build control flow against mocked HTTP seams incl. failure-tail and force-rebuild, gateway create/cull lifecycle incl. failure-path culling and zero-worker timeout, self-provisioned worker environment, an image-override regression test against the real dask-gateway client (lc's explicit image must beat the ambientDASK_GATEWAY__CLUSTER__OPTIONS__IMAGEdefault), executor rendezvous-by-name / workdir-prefix / legacy-result unpacking, sentinel-block forwarding, env-marker site detection, hub scratch resolution, runtime-aware status resolution, kubernetes-runtime Snakefile generation, environment-agnosticlc initscaffold).union2.1_lcdm_fit, user pod → Cloud Build cached-image check → run-scoped Gateway cluster with the project image → all four rules executed in worker pods with per-rule output on the driver terminal →lc verifygreen, manifests recordingworker_imageground truth → cluster culled on exit). The live test surfaced and fixed three integration bugs: gateway pods must use the moderndask scheduler/dask workersubcommands (deployment-side, hub-deploybfec4ab), the executor mustcdinto the workdir for spawned jobs (c7b18e3), and the status walker must resolve image identities runtime-aware (50508b9). A second live validation (forced full re-run) was done after the near-stock streamlining (3533b25+ hub-deploye10088c): all rules on workers,lc status/lc verifygreen, manifests carrying the lc-provisionedworker_image, cluster culled — and the run automatically picked up a changed build context (new content hash → Cloud Build → fresh cluster on the new image).Refs: LCR-174 (PRD), LCR-176 (Part A + build path).
🤖 Generated with Claude Code
https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2