Skip to content

K8s deployment #9

Open
Open

Description

@xavieralmendros-aily

Summary

Zeroboot's current deployment tooling (deploy/deploy.sh + systemd) targets bare-metal
or standalone VM hosts. There is no documented path for running Zeroboot inside a
Kubernetes cluster, which is where most production AI workloads live today.

The gap

To run Zeroboot in k8s, the following is needed but undocumented:

  1. Which EC2/cloud instance types expose /dev/kvm — e.g. AWS t3 (burstable) does
    NOT, while c5/m5/c6i (Nitro non-burstable) do. This is non-obvious and a common
    blocker.
  2. A KVM device plugin setup — pods need /dev/kvm access without privileged: true.
    The kubevirt device plugin pattern works but isn't mentioned anywhere.
  3. An official container image — the deploy script copies a compiled binary over SSH.
    There's no Dockerfile or published image, so teams must build their own, bundling
    Firecracker + the zeroboot binary + vmlinux + rootfs.ext4.
  4. PersistentVolume guidance — the template snapshot (/var/lib/zeroboot) must survive
    pod restarts. Without a PVC, every restart triggers a 15s re-snapshot.
  5. Helm chart or reference manifests — Deployment, Service, PVC, and node
    affinity/toleration patterns for pinning Zeroboot to KVM-capable nodes.

Why this matters

Container orchestration (Kubernetes, ECS) is the standard runtime for production AI agent
systems. Teams evaluating Zeroboot as a sandbox backend for agent workloads will hit all
of the above blockers on day one. A k8s deployment guide (or even a reference Helm chart)
would significantly lower the adoption barrier.

Suggested additions

  • docs/KUBERNETES.md — step-by-step guide covering node requirements, device plugin,
    PVC setup, and reference manifests
  • Dockerfile — builds an image with Firecracker + zeroboot binary + kernel + a base rootfs
  • Published image on GHCR or Docker Hub
  • Note in README linking to the k8s guide alongside the existing systemd deployment

Activity

  1. added 2 commits that reference this issue on Mar 24, 2026
    b27e58f
    9571c64
  2. chaosreload commented on Mar 24, 2026

    @chaosreload

    Hi! I have opened a PR to address all the items listed here: #13

    The PR includes:

    • Dockerfile — multi-stage build with Firecracker bundled; vmlinux/rootfs mounted via PVC
    • deploy/k8s/ — namespace, PVC, Deployment (KVM device plugin, podAntiAffinity, health probes), Service, and HPA based on zeroboot_concurrent_forks
    • docker/entrypoint.sh — handles snapshot creation on first boot, skips if already present on PVC (no more 15s re-snapshot on restart)
    • docs/KUBERNETES.md — EC2 instance selection guide (c8i/m8i/r8i recommended for nested virt on non-metal sizes), KVM device plugin setup, autoscaling with custom metrics, Prometheus ServiceMonitor config

    Happy to incorporate any feedback!

  3. added a commit that references this issue on Mar 25, 2026
    cd14c2e
  4. congwang-mk commented on Apr 1, 2026

    @congwang-mk

    One of the reasons Kubernetes deployment is complex here is the /dev/kvm dependency — you need specific instance types, device plugins, and privileged access.

    If the goal is sandboxing agent workloads in k8s, sandlock avoids this entirely. It's pure userspace (Landlock + seccomp), so it:

    • Runs in any Linux container — no device plugins, no /dev/kvm, no privileged mode
    • Installs via pip install sandlock
    • Provides network allowlisting, resource limits, filesystem isolation, and port virtualization

    The tradeoff is isolation granularity (syscall-level vs hardware-level), but for most AI agent sandboxing use cases, it's sufficient and dramatically simpler to deploy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions