Flow D — Lambda MicroVM Agent Sandbox (KRO RGD + ACK, label-branched, suspend/resume) - #41
Open
elamaran11 wants to merge 54 commits into
Open
Conversation
Flow D is a second Flow-A substrate: AWS Lambda MicroVM via the ACK lambdamicrovms controller, composed by a single KRO ResourceGraphDefinition. Mirrors the flow-a-sandbox-capability.md style. Covers substrate architecture (KRO RGD over ACK primitives), the platform/app ownership split inside one RGD, the RuntimeClass-marked bridge shim (claim -> pod -> MicroVM), and the GA migration path (self-managed controller -> Managed ACK adopts it, RGD unchanged). Flow C is reserved for other work; this substrate is Flow D.
…ctory README New §4.5 'Flow D — Lambda MicroVM substrate (alternative to Flow A)': KRO RGD over ACK primitives (Managed KRO + Managed ACK for GA iam/s3 + self-managed lambdamicrovms pre-GA controller), the platform-owned/app-owned split encoded in the MicrovmImage/Microvm CRDs, the RuntimeClass-marked bridge shim, and disabled-by-default GitOps delivery. Adds the TOC entry and a 'second substrate' note to the Two-flows-at-a-glance section. Reuses dark-factory-coder image.
…isabled) New addons.yaml entry installing the pre-GA Lambda MicroVM ACK controller (oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart v0.1.1) that reconciles the MicrovmImage + Microvm CRDs. Self-managed because Managed ACK only bundles GA controllers; coexists with Managed ACK (GA iam/s3) since CRD groups differ. enabled:false (Flow D dormant), hub-only via alwaysSelector, sync-wave 0 so CRDs+controller precede the MicrovmSandbox RGD (wave 2). Pod-identity auth (no IRSA annotation), cluster install scope.
templates/50-rgd-microvm-sandbox.yaml: one kro.run/v1alpha1 ResourceGraphDefinition that expands a single MicrovmSandbox CR into S3 Bucket + build/exec IAM Roles (GA -> Managed ACK) + MicrovmImage + Microvm (pre-GA -> self-managed ACK). The platform/app ownership split is encoded in the schema (spec.image.* platform-owned, spec.run.* app-owned); status surfaces microvmID/state for the bridge to mirror. RGD is controller-install-agnostic (unchanged when lambdamicrovms goes GA). values.yaml: new microvm: block (enabled:false, region, apiGroup, bridgeImage, idle/connector defaults; cluster-specific ARNs left blank for the overlay). Verified: helm template renders 0 RGD when disabled, 1 valid RGD (5 resources: bucket/buildRole/execRole/image/microvm) when enabled.
…eClass shim) templates/51-sandboxtemplate-microvm.yaml: the shim that keeps the Agent-Sandbox UX while the coder runs in a remote Lambda MicroVM. A SandboxTemplate (<templateName>-microvm) whose pod is a thin BRIDGE on a normal Auto-Mode node (no kata runtimeClass/nodeSelector) — it idles until a claim injects DF_ISSUE_NUMBER (same as Flow A), applies a MicrovmSandbox (KRO) CR, waits for Microvm RUNNING, mirrors lifecycle (pod Running <-> Microvm RUNNING), and deletes the CR on exit (-> TerminateMicrovm). Ships its ServiceAccount + least-priv Role/RoleBinding (microvmsandboxes only) + the bridge.sh ConfigMap. Both substrates coexist; a claim picks one by which template it references. Verified: helm template renders 5 objects when enabled, 0 when disabled; bridge.sh passes sh -n.
Hub per-cluster overlay gets a microvm: block (enabled:false) alongside the kata nodepool block. Cluster/account-specific ARNs (baseImageARN, codeArtifactUri, ingress/egress connectors) left blank with guidance to fill before enabling. Flow D stays fully dormant on the hub until the Managed KRO + Managed ACK capabilities and the self-managed lambdamicrovms controller are in place. Verified full-chart render with the overlay: 0 Flow D objects when dormant (existing Kata objects intact), clean inventory when enabled (1 RGD + 2 SandboxTemplates + bridge SA/Role/RoleBinding/ConfigMap), valid YAML in both modes.
…into Flow D The ACK GA-services-list reference and the pre-GA-vs-GA / self-managed rationale belong with the consumer (Flow D), not the platform capability-enablement doc. Add the authoritative ACK community services (GA list) link backing the 'GA upstream' claim, and clarify the ownership split: the Managed KRO + Managed ACK CAPABILITIES are enabled in appmod-blueprints (EKS Capability toggle, see its docs/EKS-Capabilities-KRO-ACK-Setup.md); THIS repo owns the self-managed lambdamicrovms controller + the KRO MicrovmSandbox RGD + the sandbox shim.
Align the MicrovmSandbox RGD with the actual lambda-microvms API (verified against the live service + AWS docs in us-west-2): - IAM trust principal: lambda.amazonaws.com (+ sts:TagSession) — NOT the guessed lambdamicrovms.amazonaws.com. buildRole gets inline ECR-read + S3-read + logs (Lambda pulls the coder image from private ECR at build). - Drop network connectors entirely: Lambda MicroVMs have PUBLIC egress by default, which is all the coder needs (git/gh/registry). Ingress/VPC-egress connectors are optional add-ons (the latter created via a different service, aws lambda-core) — omitted from v1. Removes ingress/egressConnectorARN from schema + both resources. - codeArtifactUri: document it accepts an ECR image URI or S3 path (Flow D uses the arm64 dark-factory-coder ECR image directly). - base image ARN format corrected to arn:aws:lambda:<region>:aws:microvm-image:al2023-1 (ARM_64-only) in values guidance. Verified: helm template renders 5 resources, 0 connector refs, correct principal.
Set the hub Flow D overlay to the confirmed live values: - baseImageARN: arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1 (verified available; ARM_64-only). - codeArtifactUri: dark-factory-coder:v0.2.3-arm64 ECR URI (accepts ECR directly; arm64 tag to be built — MicroVM is ARM_64-only, existing amd64 tags untouched). - Drop the connector fields (MicroVMs default to public egress). Still enabled:false (dormant) until the ack-lambdamicrovms controller is synced.
…ifecycle)
Implements the REQUIRED declarative suspend/resume: Sandbox.spec.operatingMode has
{Running,Suspended} but the ACK Microvm CR has no suspend field (suspend/resume are
imperative SDK ops the controller doesn't reconcile). Close the gap with a tiny
always-on reconcile loop (templates/52) — pure shim, no virtual-kubelet, no new image:
- microvm-lifecycle Deployment (alpine/k8s, ConfigMap reconcile.sh, matches
pool-manager pattern): watches Sandboxes labeled substrate=lambda-microvm; on
operatingMode Running->Suspended calls suspend-microvm, Suspended->Running calls
resume-microvm, by microvmID resolved from the MicrovmSandbox status. Idempotent
via a last-mode annotation. + SA/Role/RoleBinding (pod-identity auth).
- Bridge (51) made suspend-aware: labels its MicrovmSandbox agent-sandbox.io/sandbox
for correlation; preStop checks operatingMode and, if Suspended, writes a marker so
cleanup() KEEPS the MicrovmSandbox (VM survives suspend); real teardown only on
claim end. Create is idempotent (resume-or-create). +sandboxes get RBAC.
- values: microvm.lifecycle.intervalSeconds (15s).
Robust vs preStop-only: survives pod/node loss; resume needs no live pod. Verified:
0 objects disabled; 10 when enabled; reconcile.sh + bridge.sh pass sh -n.
Add §D.3a (diagram doc) + a Suspend/resume subsection (README §4.5) describing the microvm-lifecycle reconcile loop that translates Sandbox.operatingMode -> Lambda MicroVM suspend/resume-microvm, why it's needed (ACK Microvm has no suspend field), and why a reconcile loop beats preStop hooks (robust to pod/node loss). Matches the templates/52 implementation.
Per reviewer request: the coder now writes a concise, reviewer-facing description of WHAT it changed (and why) to artifacts/description.md as part of its run, and the PR body renders it as a '### 📝 Changes' section ABOVE the existing verification block. Falls back to a neutral line if absent so the body is never empty. - claude path: description instruction appended to the prompt. - kiro path: same instruction appended to SPEC.md (kiro reads --spec). - PR body: new Changes section placed before the <!-- dark-factory:status --> marker, so status.js (which rewrites from the marker down) preserves it on every update. node --check passes.
Two fixes for the 'security review waits ~15 min while the PR already shows a result' problem: 1. Early-exit: the code-review job's status field flips to COMPLETED long AFTER the AWS Security Agent App bot has already posted its findings on the PR. So in addition to polling status, each iteration now probes list-findings; the moment it returns a well-formed result (findingsSummaries present) we proceed — no waiting for the status flip. Reuses those findings for the report (no re-fetch). 2. Graceful timeout: this step is advisory (the App bot posts the authoritative result regardless), so exceeding the poll timeout now posts a neutral 'pending' status + a pointer to the bot comment, instead of a misleading red 'error'. This also fixes the stale sticky board: the sticky-status step depends on security-agent (runs once, after it), so a fast security step means the board is rewritten with accurate rows within seconds instead of ~15 min later. sh -n passes.
… substrate One pipeline, substrate chosen by label (per design decision): - sensor: accept BOTH dark-factory (Kata) + darkfactory-lambda (MicroVM) labels; pass the firing label to df-run as the trigger-label param (index 6). - df-run: new trigger-label param (default dark-factory); claim-sandbox takes a warm-pool input; the DAG computes it via an Argo expression (darkfactory-lambda -> coder-warmpool-microvm, else coder-warmpool). Everything downstream (coder, holdout, devops, security, PR, merge, teardown) is identical. - agent-sandbox: add the Lambda-MicroVM SandboxWarmPool (claim.warmPoolRef is required, so Flow D needs its own pool) referencing the -microvm SandboxTemplate; microvm.warmPool values (name, targetIdle). dark-factory: warmPool.lambdaName. Verified: df-run + sensor render to valid YAML; warm-pool expression resolves correctly; lambda warmpool renders only when microvm.enabled (0 when disabled).
Flow D visible suspend/resume via Sandbox.operatingMode (per design decision):
- New microvm-suspend DAG step (gated microvmSuspend.enabled + lambda substrate +
PR exists): after drive-coder, flips Sandbox.spec.operatingMode=Suspended so the
agent-sandbox microvm-lifecycle controller calls suspend-microvm — compute freed
while the gates run (coder is idle during review). No-op for Kata.
- microvm-set-mode script template: patches operatingMode, best-effort verifies the
MicrovmSandbox microvmID (advisory). RBAC: workflow SA gets sandboxes 'patch'.
- Resume: happens naturally — df-iterate (PR comment) needs the VM, and the MicroVM
idlePolicy.autoResumeEnabled resumes on next request; the lifecycle controller
also resumes on operatingMode=Running.
- values: microvmSuspend {enabled, image}.
Verified: full chart renders valid YAML; suspend step present when enabled, absent
when disabled.
elamaran11
changed the title
Applied field-level corrections from the lambdamicrovms-controller reference
(mmeckes/lambdamicrovms-controller, 06-kro example + CRD source):
- codeArtifact.uri is S3-ONLY (s3://bucket/key of a zip w/ Dockerfile), NOT an
ECR image URI. Fixed the misleading 'ECR or S3' comments in the RGD schema +
values (the Dockerfile inside MAY pull ECR base layers — build role keeps ecr:Get*).
- image.readyWhen: gate on status.state == CREATED||UPDATED so the Microvm never
launches from a half-built/failed image (KRO holds the instance until ready).
- MicrovmImage.cpuConfigurations: [{architecture: ARM_64}] — the sole supported
arch; make it explicit rather than relying on a default.
- MicrovmImage.logging.cloudWatch.logGroup: /aws/lambda/microvms/<name>-image so
CREATE_FAILED build output is retrievable (aws logs tail ...).
- status.microvmState now reads Microvm.status.state (PENDING/RUNNING/SUSPENDED/
...) not conditions[0].status; added imageState passthrough.
- build role S3 policy scoped to the actual bucket via CEL ref (bucket.spec.name)
+ ListBucket, and logs scoped to the microvms log-group prefix.
Design unchanged: Sandbox CRDs → ACK Lambda MicroVM for create/delete + the
microvm-lifecycle shim for suspend/resume (imperative ops, not controller-managed
— confirmed by the reference). Still gated microvm.enabled=false; render-verified.
…bda chart
Clean folder separation — Flow D (Lambda MicroVM) is now a standalone chart instead
of files dumped into the Kata agent-sandbox chart:
gitops/addons/charts/agent-sandbox-lambda/
Chart.yaml, values.yaml, templates/_helpers.tpl
templates/image/10-rgd-and-image.yaml — KRO RGD (image-only) + the ONE platform
MicrovmSandbox instance (built ONCE)
templates/shim/20-bridge-sandboxtemplate.yaml — bridge: reads image handoff,
RunMicrovm (SDK), TerminateMicrovm
templates/shim/30-microvm-lifecycle.yaml — suspend/resume controller (SDK)
- Moved the 3 microvm templates out of agent-sandbox (Kata chart) → now Kata-only.
- Moved the microvm: values block out of agent-sandbox/values.yaml + the hub overlay
into agent-sandbox-lambda/{values.yaml, clusters/hub overlay}. Corrected stale docs
(codeArtifact is S3-only, not ECR).
- New gated addon entry agent-sandbox-lambda (hub-only, sync-wave 2); dormant until
microvm.enabled=true.
ARCHITECTURE (per design): KRO builds the platform image ONCE; the shim runs/suspends/
resumes/terminates the per-session VM imperatively (ACK doesn't reconcile those). RGD no
longer contains a Microvm resource. Renders: Kata chart microvm-free; Lambda chart 0
disabled / 13 enabled. Flow D only — dark-factory (PR #32) untouched.
… across review loop) Implements the required Lambda coder lifecycle: coder codes → SUSPEND; findings + fix → RESUME the SAME VM (keeps context); terminate ONLY at final exit (merge). - df-run onExit teardown is now SUBSTRATE-AWARE (script, not a blind resource delete): Kata deletes the claim (frees pool); LAMBDA KEEPS the claim + suspended VM so the same VM survives the whole review→fix→re-review loop (df-iterate resumes it). Without this, df-run's exit deleted the claim → orphaned/terminated the suspended VM. - df-merge-teardown adds a microvm-terminate step (gated microvmSuspend.enabled) that resolves the per-session VM id from the Sandbox annotation and TerminateMicrovm on merge — the ONLY place the Lambda VM is destroyed. Ordered before the claim delete so the bridge's suspend-marker can't keep a leaked VM. - microvm-set-mode verify reads the VM id from the Sandbox annotation (post image-only RGD refactor — no per-session MicrovmSandbox status anymore). Flow-D-branch changes to the dark-factory template (base of PR #41); gated by microvmSuspend.enabled so Flow B (Kata, PR #32) is unaffected. Renders clean both ways.
…d/resume/terminate lifecycle Aligns the Flow D docs with the refactor: KRO builds the image ONCE (RGD has no Microvm); the shim runs the per-session VM imperatively (RunMicrovm/suspend/resume/ TerminateMicrovm); the coder VM PERSISTS suspended across the review→fix loop and is terminated only at merge. Updated the composition table, platform/app split, shim description, the suspend/resume/terminate lifecycle section, and the delivery section (now the standalone agent-sandbox-lambda chart with image/ + shim/ subfolders). NOTE: diagrams/flow-d-microvm-sandbox.md ASCII still shows the old per-claim model — follow-up.
Per design: everything via Managed ACK + KRO, self-managed ACK only for the pre-GA lambdamicrovms controller — NO Crossplane, NO Terraform, NO CLI. - Controller bootstrap IAM is now an ACK iam.services.k8s.aws Role + ACK eks.services.k8s.aws PodIdentityAssociation (templates/shim/00-controller-pod-identity.yaml) — same Managed-ACK mechanism as the RGD's build/exec roles. Role trusts pods.eks.amazonaws.com; inline policy = lambda:*Microvm* + iam:PassRole (lambda). Bridge + lifecycle SAs get their own PodIdentityAssociations reusing the role. All 4 objects server-dry-run VERIFIED against the live ACK CRDs on the hub. - Enabled ack-lambdamicrovms controller addon (enabled:true) + microvm.enabled=true in the hub overlay (accountId + podIdentity set). Downstream (S3, build/exec roles, MicrovmImage) stays in the KRO RGD via ACK. Flow D only (PR #41); dark-factory (PR #32) untouched.
…ambda-microvm-sandbox # Conflicts: # examples/dark-factory/coder/entrypoint.js # gitops/addons/charts/dark-factory/scripts/security-agent.sh # gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml
…rking structure) After #45 was reverted (PR back to old flat-addons + working ACK structure), reset flow-d to the pre-#45-merge state and re-merged the CURRENT #32 (47b8908, contains all the colleague's still-valid Flow B work minus the reverted hub-sandbox commit). Then re-applied the Flow D df-run orchestration (dropped when I took their df-run in the merge): substrate-branched claim (lambda warm pool on darkfactory-lambda), microvm-suspend DAG step + microvm-set-mode template (suspend after coder pushes), substrate-aware teardown (Kata deletes claim; Lambda keeps suspended VM until merge). df-merge-teardown already has microvm-terminate. Renders clean (default + microvmSuspend).
elamaran11
force-pushed
the
flow-d-lambda-microvm-sandbox
branch
from
August 1, 2026 23:24
fabe855 to
ba514ac
Compare
added 5 commits
August 2, 2026 21:02
The MicrovmSandbox instance failed dry-run ('CRD not found') and blocked the whole
app sync (nothing applied, incl the RGD that GENERATES that CRD — chicken-and-egg
within the app). RGD → sync-wave -1 (KRO generates the CRD first); instance → wave 1 +
SkipDryRunOnMissingResource so the first pass doesn't block. ArgoCD retries/selfHeal
converge.
The coder Kata VM resolves via public DNS (8.8.8.8/1.1.1.1) since a Kata
guest can't use in-cluster CoreDNS and EKS Auto Mode runs no kube-dns
Service. coder-sandbox-egress only permitted :53 to namespaceSelector{}
(in-cluster), so name resolution silently failed — the coder crashed with
EAI_AGAIN api.github.com even though :443 egress worked, and no PR was ever
pushed. Add a :53 egress rule scoped to the resolver /32s (networkPolicy.
dnsResolvers), tighter than the existing :443 public allow.
ttlStrategy.secondsAfterCompletion was 3600, so runs were GC'd an hour after finishing — and there's no workflow DB archive on this cluster, so they were lost for good (Friday's runs gone). Bump to 604800 (7d) via new argo.workflowTtlSecondsAfterCompletion value across df-run/df-iterate/ df-merge-teardown so demo+debug history survives a work week.
Two fixes so the MicrovmSandbox RGD actually reconciles on the hub:
1. apiGroup kro.run (was sandbox.agents.x-k8s.io): EKS Managed KRO's
controller only watches the kro.run group — an RGD on any other group
stays state=Inactive ('cache sync timeout ... microvmsandboxes').
Verified by group-probe on the hub. Generated CRD is microvmsandboxes.
kro.run; kind (MicrovmSandbox) unchanged.
2. New 40-kro-graph-rbac.yaml: KRO runs as the cluster's KRO capability
role (EKS access entry, session KRO). AmazonEKSKROPolicy grants kro.run
but NOT CRUD on the ACK children the graph creates, so reconcile hit
'forbidden: cannot get buckets'. Grant a ClusterRole scoped to exactly
the 3 child groups (s3 buckets, iam roles, lambdamicrovms images/vms)
bound to that identity (microvm.kroCapability values).
The Kata substrate showed microvm-suspend as phase=Error (type=Skipped):
Argo substitutes the trigger-label value inline, so the unquoted
'dark-factory == darkfactory-lambda' parsed as arithmetic on bare
identifiers and errored ('Failed to evaluate when expression'). Harmless
(the step is correctly skipped on Kata and the run still completes) but it
surfaces a spurious Error node. Quote both operands so it evaluates as a
clean string comparison and skips silently on non-lambda runs.
added 27 commits
August 3, 2026 11:50
microvmSuspend.enabled defaulted true, so the df-run DAG always rendered a microvm-suspend task — on a Kata (Flow B) run it's runtime-skipped but still appears as a node in the graph, an irrelevant/confusing Lambda-MicroVM step in a pipeline with no MicroVM. Default it false so the step isn't rendered at all unless a cluster runs the Lambda substrate; enable per-cluster overlay for darkfactory-lambda.
…e condition CreateMicrovmImage/RunMicrovm was denied iam:PassRole on coder-microvm-build: the statement conditioned on iam:PassedToService=lambda.amazonaws.com, but the Lambda MicroVM sub-service passes to a different principal (verified: sim allowed for lambda.amazonaws.com yet the live API denied; microvms/microvm.lambda.amazonaws.com also implicitDeny). Replace the brittle service condition with a Resource scope to arn:...:role/*-microvm-build + *-microvm-exec — net-tighter (only these two purpose- built roles can be passed) and principal-agnostic. Post-change sim: allowed.
…uild After PassRole was fixed, CreateMicrovmImage next failed on lambda:PassNetworkConnector for the AWS-managed INTERNET_EGRESS connector (the default egress attached when the RGD sets no explicit connectors). Add lambda:PassNetworkConnector + List/GetNetworkConnector scoped to arn:aws:lambda:<region>:*:network-connector:*.
The overlay codeArtifactUri named dark-factory-microvm-artifacts-<acct>-<region>, but
the RGD creates the artifact bucket as ${image.name}-microvm-artifacts (coder-microvm-
artifacts). The image build read from a bucket the RGD never provisioned → NoSuchBucket
/ CREATE_FAILED. Point it at s3://coder-microvm-artifacts/coder-v0.2.5-arm64.zip.
…rity
Flow B's SandboxClaim injects env (DF_ISSUE_NUMBER, DF_REPO, ...) into a container
named 'coder' — the same contract as the Kata substrate. The microvm bridge template
named its container 'bridge', so the operator rejected every claim targeting the
microvm warmpool ('target container coder not found in template'). Rename bridge->coder
(still runs bridge.sh -> RunMicrovm) so the Lambda substrate is transparent to df-run.
The bridge carries role=coder so the shared coder-sandbox-egress policy applies — which denies the service+VPC CIDRs on :443 to isolate untrusted coder code from the control plane. But the bridge (trusted platform code) MUST reach the API server to read the MicrovmSandbox image handoff + annotate the Sandbox; without it kubectl hangs and it never RunMicrovm's. Add an additive NetworkPolicy selecting only the bridge's distinct substrate=lambda-microvm label, allowing :443 to the apiserver (svc ClusterIP + VPC endpoint CIDRs). Kata coder isolation is untouched (different selector).
…ovms) alpine/k8s:1.31.0 ships aws-cli 1.34, which does NOT know the pre-GA lambda-microvms service — 'aws lambda-microvms run-microvm' just printed the service list, so the bridge never launched a VM. Switch bridgeImage to public.ecr.aws/aws-cli/aws-cli:latest (glibc, always-current v2, has lambda-microvms — same image the security-agent uses) and fetch a static kubectl at start in bridge.sh + reconcile.sh (that image has no kubectl).
…/k8s) The hub overlay pinned bridgeImage: alpine/k8s:1.31.0, overriding the chart default — so the bridge/lifecycle kept the aws-cli that lacks lambda-microvms. Align the overlay to public.ecr.aws/aws-cli/aws-cli:latest.
RunMicrovm failed 'retrieving credentials from container-role: connect timeout http://169.254.170.23/v1/credentials' — the bridge gets its AWS creds from the Pod Identity agent at link-local 169.254.170.23, but the shared coder-egress policy denies 169.254.0.0/16 (IMDS block for untrusted coder code). Allow ONLY the Pod Identity /32:80 for the bridge selector; IMDS (169.254.169.254) stays denied, Kata coder untouched.
…smatch)
The bridge annotates the owning Sandbox with the microvm-id so the lifecycle controller
can suspend/resume it. But SANDBOX_NAME fell back to df-${DF_ISSUE_NUMBER} (e.g. df-9999),
while agent-sandbox names the Sandbox after the CLAIM (e.g. df-issue-smoke-d) — so the
annotate silently failed, suspend couldn't find the VM, and pod teardown TERMINATED it
instead. Set SANDBOX_NAME from the pod's own metadata.name (pod==Sandbox name); fix the
preStop fallback too.
Turned it off earlier to keep the Kata graph clean, but Flow D REQUIRES the microvm-suspend step (suspend-after-code). It's when-gated + now skips cleanly on Kata (quoted operands → Skipped, not Error), so a both-substrate hub should have it on. A Kata-only cluster can set it false.
Substrate + lifecycle are proven E2E, but running the actual coder in the VM is an application re-architecture, not wiring: the /run lifecycle hook has a 30s timeout and the MicroVM model is a long-running HTTP service (request/response), while the coder is a one-shot 5-15min batch job. runHookPayload is an HTTP body to a /run endpoint the app must SERVE (a SecretKeyReference on the Microvm CR) — NOT env/file. Document the async /run wrapper approach + VPC egress connector (bootstrap Job) + Bifrost internal NLB + runHookPayload secret + SG rules, with open questions, before building.
Make the coder actually run inside the MicroVM, Bedrock-direct (no Bifrost/EKS-network dependency): - entrypoint.js: branch LLM transport on USE_BEDROCK. Kata (default)=Bifrost shim; Lambda=CLAUDE_CODE_USE_BEDROCK=1 + real Bedrock model id, creds from the exec role. - examples/dark-factory/coder-microvm/: hook-server.js serves the MicroVM lifecycle hooks on :8080; /run materializes runHookPayload (issue ctx + gh token) and BACKGROUND- spawns entrypoint.js (the 30s run-hook can't hold a 5-15min coder run), returning fast. Dockerfile FROM the arm64 coder image + overlays the updated entrypoint.js (no ECR rebuild). - RGD: MicrovmImage hooks (ready+run+suspend/resume/terminate) ENABLED; exec role gets bedrock:InvokeModel*. See docs/dark-factory/flow-d-coder-in-microvm-design.md.
…RunMicrovm The bridge now builds a JSON payload (issue number/repo/branch/title + GitHub token + region) and passes it via --run-hook-payload on run-microvm (CLI form is a plain ≤16KB string). hook-server.js's /run receives it, writes /tmp/secrets/gh-token + DF_* env, and background-spawns the Bedrock-direct coder. Mounts the dark-factory-github secret (gh-token) read-only at /etc/df. No Bifrost key — Bedrock-direct via the exec role.
The bridge image (aws-cli v2) has no node — the node -e payload builder crashed the bridge right after reading the handoff (CrashLoopBackOff, before RunMicrovm). Switch to python3 (present in the image) + json.dumps for safe escaping of the token/title.
The multi-line python at column 0 broke out of the bridge.sh: | block scalar → ArgoCD ComparisonError 'could not find expected :'. Collapse to one line so all script content stays indented under the block scalar.
Bedrock-direct async design built + deployed: hook-server runs in the VM (verified via CloudWatch 'listening on :8080'), bridge passes runHookPayload, VM launches RUNNING — but the /run hook never fires so the coder doesn't start. Confirmed cause: runHookPayload is a SecretKeyReference delivered via the declarative Microvm CR (controller-driven), NOT the imperative run-microvm --run-hook-payload CLI the bridge uses. Correct path documented: per-session Microvm CR + Secret. hook-server/artifact/image/IAM/Bedrock are done + verified.
Switch the bridge from the imperative run-microvm CLI (which never fires /run) to the
declarative path the runHookPayload SecretKeyReference actually requires: write a payload
Secret + create a Microvm CR (imageIdentifier, executionRoleARN, runHookPayload{name,key},
idlePolicy); the self-managed controller reconciles it and delivers the payload to the /run
hook → hook-server background-spawns the Bedrock-direct coder. Teardown deletes the CR
(controller terminates the VM); suspend keeps it. Manifests built as JSON via python3 (no
heredoc — a column-0 EOF breaks the bridge.sh block scalar). Bridge RBAC += microvms +
secrets CRUD.
…ilent Declarative Microvm CR path works (CR RUNNING, microvmID populated, clean CR-delete teardown, 0 orphans) but the /run hook produces no runtime coder output in CloudWatch. Ruled out: payload mechanism, image hooks (v2.0), IAM, bridge crash, YAML. Open frontier: whether the service invokes /run against the in-VM hook-server + why no logs.
Probed the live VM: GET / -> ok, POST /run -> coder started. Whole chain functional (issue->workflow->bridge->Microvm CR->VM->hook-server->/run->coder-spawn->clean teardown). Two isolated gaps: (1) service doesn't auto-invoke /run on launch (had to POST manually); (2) runtime logs don't reach CloudWatch (build logs only) — blinded prior runs.
…connector Close the 2 E2E gaps: (1) the service's internal /run auto-fire wasn't reliably starting the coder — the bridge now waits for RUNNING+endpoint, mints an auth token, and POSTs the payload to /run on the VM endpoint (deterministic, matches the reference run_session; verified by probe that this starts the coder). Needs ingressNetworkConnectors:ALL_INGRESS on the Microvm to reach the endpoint. (2) runtime logs — add Microvm.spec.logging.cloudWatch (logStream runtime-<mvm>) so the coder's stdout is visible (image logging was build-only).
The bridge mints an auth token to POST /run on the VM endpoint, but the controller/bridge role lacked lambda:CreateMicrovmAuthToken (+ShellAuthToken) — token mint failed AccessDenied, so /run was never driven. Add both verbs to the controller inline policy (reused by the bridge via Pod Identity).
Runtime CloudWatch routing is unreliable on this runtime, so make the coder run OBSERVABLE
directly: hook-server captures the coder's stdout/stderr to /tmp/coder.log + tracks state
(running/done/exited:N/spawn-error), exposes GET /status {coderState, log tail}. The bridge
polls /status in its hold loop and echoes it — so the coder run is visible in the bridge pod
logs (kubectl logs) even without CloudWatch.
Overwriting the same S3 key doesn't change the URI, so the controller never rebuilds (stayed v2.0 without /status). New key -r2 → URI change → rebuild with the /status observability + auth-token fixes.
The /status route + log-capture correlated with a MicrovmImage build hung 2h+ on the ready hook (pre-GA controller). Revert hook-server to the minimal known-good shape that built v2.0 cleanly: trivial synchronous /run that detached-spawns the coder, no /status. Point codeArtifactUri at a fresh key (r3) to force a clean rebuild.
…ility) No CloudWatch runtime routing + no shell, so /run captures the coder's stdout/stderr to /tmp/coder.log and /logs returns it (read over the HTTP token). This is how we finally SEE why the coder isn't producing a PR. Artifact r4.
…orkspace) ROOT CAUSE (finally seen via /logs): the coder crashed immediately — 'EACCES: permission denied, mkdir /workspace/artifacts' at entrypoint.js:398 — because the MicroVM rootfs is read-only with no /workspace volume (unlike Kata). Point WORKSPACE at the writable tmpfs. Artifact r5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flow D — Lambda MicroVM Agent Sandbox (+ bundled Flow B improvements)
Adds Flow D: a second Agent-Sandbox substrate where the coder runs in an AWS Lambda MicroVM instead of a Kata pod. One pipeline, substrate chosen by label. Base branch:
dark-factory-autonomous-agent-coding-pattern(PR #32).Flow D — substrate
MicrovmSandboxRGD ties MicrovmImage + Microvm + IAM roles + S3 bucket (one CR). API corrected to the reallambda-microvmsservice (verified live; ARM_64-only; public egress default → no connectors; ECR codeArtifact).ack-lambdamicrovmscontroller addon (pre-GA; Managed ACK can't bundle it) — disabled by default.lambda-microvmSandboxTemplate + bridge (RuntimeClass shim) + its SandboxWarmPool.al2023-1, arm64 coder ECR URI).Flow D — pipeline (one pipeline, label-branched)
dark-factory(Kata) +darkfactory-lambda(MicroVM); passes the label to df-run.claimbinds the Kata pool or the MicroVM pool based on the label; everything downstream identical.Flow D — suspend/resume (required)
microvm-lifecyclecontroller reconcilesSandbox.operatingMode→suspend/resume-microvm(the ACK Microvm CR has no suspend field).microvm-suspenddf-run step: after the coder pushes, suspend the MicroVM while gates run (compute freed); resumes on iterate / next request.Bundled Flow B improvements (deployed + verified on PR #56)
Verification
helm templateclean; scriptssh -n/node --check.Platform prereq (separate, merged): appmod-blueprints #796/#800 enabled Managed KRO + ACK on the hub.
Generated with Claude Code.