diff --git a/.github/aw/agent-runtime-instructions.md b/.github/aw/agent-runtime-instructions.md new file mode 100644 index 00000000000..9482ae32f69 --- /dev/null +++ b/.github/aw/agent-runtime-instructions.md @@ -0,0 +1,51 @@ +--- +description: Choose and configure agent runtimes for GitHub Agentic Workflows. +disable-model-invocation: true +--- + +# Agent Runtime Instructions + +Use these instructions when creating or updating workflows that mention Docker, gVisor, Docker sbx, ARC DinD, self-hosted runners, or `sandbox.agent.runtime-install`. + +## Runtime fields + +- Omit `sandbox.agent.runtime` for the default Docker agent runtime. +- Set `sandbox.agent.runtime: gvisor` only when the runner has a local Docker daemon and can install or already has `runsc`. +- Set `sandbox.agent.runtime: docker-sbx` only when the runner supports KVM-backed microVMs. +- Do not set `sandbox.agent.runtime: docker`; Docker is selected by omitting the field. +- Do not set `sandbox.agent.runtime: sbx`; `sbx` is only a bounded-query runtime name. +- Set `runner.topology: arc-dind` for ARC or equivalent Kubernetes runners that use a Docker-in-Docker sidecar. This is a runner topology, not an agent runtime. + +## Compatibility + +- Do not combine `runner.topology: arc-dind` with `sandbox.agent.runtime: gvisor` or `sandbox.agent.runtime: docker-sbx`. +- ARC DinD workflows must be rootless: do not add `sudo`, `apt-get install`, or other host package bootstrap steps. +- Docker sbx requires KVM and normally does not work on ARC DinD because the sbx daemon must run on the runner host. + +## `runtime-install` + +- `sandbox.agent.runtime-install` defaults to `true` for gVisor and Docker sbx provisioning. +- Set `runtime-install: false` only when the runner image or pod is pre-provisioned with the runtime and required daemon or policy. +- When any imported workflow sets `runtime-install: false`, false wins during import merging. +- With `runtime-install: false`, gh-aw skips generated runtime checks and setup, so the runner must already satisfy those prerequisites. + +## gVisor guidance + +- gVisor uses `runsc` for the agent container while AWF infrastructure containers continue to use Docker. +- The generated gVisor installer may use `sudo`, but do not set `sandbox.agent.sudo: true` merely for gVisor. +- Use gVisor when stronger kernel isolation is needed and the workload is compatible with gVisor syscall behavior. + +## Docker sbx guidance + +- Docker sbx runs the agent in a KVM-backed microVM and requires a KVM-capable Linux runner. +- With runtime installation enabled, set `sandbox.agent.sudo: true` because gh-aw installs `docker-sbx`, adjusts `/dev/kvm`, starts the sbx daemon, authenticates CLIs, pulls the template, and runs a smoke test. +- Docker sbx requires both `DOCKER_USERNAME` and `DOCKER_PAT` Actions secrets. `DOCKER_PAT` must be a Docker Hub personal access token that can authenticate Docker Hub pulls for the sandbox template. +- `DOCKER_USERNAME` and `DOCKER_PAT` remain required even with `runtime-install: false`, because compiled workflows refresh sbx credentials immediately before agent execution. +- Do not use Docker sbx for workflows triggered from untrusted forks unless the trigger and credential model safely provide those secrets. + +## ARC DinD guidance + +- Use `runner.topology: arc-dind` when `DOCKER_HOST` points to a DinD sidecar such as `tcp://localhost:2375` or `tcp://dind:2375`. +- Ensure the runner container and DinD sidecar share `/home/runner/_work`. +- Use a daemon-visible tool cache path such as `/tmp/gh-aw/tool-cache`, not `/opt/hostedtoolcache`. +- If the Docker socket is bind-mounted at a nonstandard path, set `GH_AW_DOCKER_SOCK_PATH`. Set `GH_AW_DOCKER_SOCK_GID` only when group detection with `stat` fails. diff --git a/.github/aw/create-agentic-workflow.md b/.github/aw/create-agentic-workflow.md index 185f54eef3e..716812fbc25 100644 --- a/.github/aw/create-agentic-workflow.md +++ b/.github/aw/create-agentic-workflow.md @@ -29,6 +29,7 @@ Load these topic files only when relevant: - [report.md](report.md) for reporting output structure and recurring report lifecycle - [release-workflow.md](release-workflow.md) for release workflows that build, test, publish a GitHub release, and generate release highlights - [linter-workflows.md](linter-workflows.md) for mining, refining, or applying custom linter rules +- [agent-runtime-instructions.md](agent-runtime-instructions.md) when choosing or debugging Docker, gVisor, Docker sbx, ARC DinD, self-hosted runners, or `sandbox.agent.runtime-install` ## Modes diff --git a/.github/aw/syntax-agentic.md b/.github/aw/syntax-agentic.md index c161e88e98b..1d407e6e669 100644 --- a/.github/aw/syntax-agentic.md +++ b/.github/aw/syntax-agentic.md @@ -315,7 +315,8 @@ description: Agentic workflow specific frontmatter fields for GitHub Agentic Wor ``` - **`sandbox.agent.sudo`** (boolean) controls whether AWF runs in root mode. Default is `false`: AWF runs rootless in network-isolation egress mode (`--network-isolation`), with MCP sidecars attached as bridge containers on the internal `awf-net` network. Set `sudo: true` for the legacy root mode; in strict mode explicit `sudo: true` is an error (warning otherwise). - - **`sandbox.agent.runtime`** (string) selects an extra-isolation container runtime for the agent: `gvisor` (runs under gVisor's `runsc` for kernel-level isolation) or `docker-sbx` (Docker sbx microVM with KVM hypervisor-level isolation; needs `DOCKER_PAT`/`DOCKER_USERNAME` secrets and a KVM-capable runner). Both require `sudo: true` and are incompatible with `runner.topology: arc-dind`. + - **`sandbox.agent.runtime`** (string) selects an extra-isolation container runtime for the agent: `gvisor` (runs under gVisor's `runsc` for kernel-level isolation) or `docker-sbx` (Docker sbx microVM with KVM hypervisor-level isolation). Both are incompatible with `runner.topology: arc-dind`. gVisor's generated host installer uses `sudo`, but `sandbox.agent.sudo` should remain false. Docker sbx requires `sudo: true`, `DOCKER_PAT`/`DOCKER_USERNAME` secrets, and a KVM-capable runner when runtime installation is enabled. + - **`sandbox.agent.runtime-install`** (boolean) controls generated gVisor or Docker sbx provisioning and defaults to `true`. Set it to `false` only when the runner is pre-provisioned; Docker sbx credential refresh still runs. False wins when imported workflows merge this field. See [agent-runtime-instructions.md](agent-runtime-instructions.md) for requirements and troubleshooting. - **Strict mode**: `sandbox.agent` blocks without an explicit `id: awf` are rejected in strict mode. Any non-nil, non-disabled agent config without `id`/`type` defaults to AWF at runtime. - **`tools:`** - Tool configuration for the coding agent (`github`, `agentic-workflows`, `edit`, `web-fetch`, `web-search`, `bash`, `playwright`, custom MCP server names, plus `timeout`/`startup-timeout`/`cli-proxy`). See [syntax-tools-imports.md](syntax-tools-imports.md#tool-configuration) for the full schema (GitHub `mode`/`toolsets`/integrity fields, bash allowlist decision rule, Playwright CLI mode). diff --git a/.github/aw/update-agentic-workflow.md b/.github/aw/update-agentic-workflow.md index 765b3c9263d..018586ff6f1 100644 --- a/.github/aw/update-agentic-workflow.md +++ b/.github/aw/update-agentic-workflow.md @@ -22,6 +22,7 @@ Load these additional files only when relevant: - [visual-regression.md](visual-regression.md) - [serena-tool.md](serena-tool.md) - [linter-workflows.md](linter-workflows.md) +- [agent-runtime-instructions.md](agent-runtime-instructions.md) for changes involving Docker, gVisor, Docker sbx, ARC DinD, self-hosted runners, or `sandbox.agent.runtime-install` ## Scope diff --git a/.github/skills/agentic-workflows/SKILL.md b/.github/skills/agentic-workflows/SKILL.md index 995e0a670cc..742a125b032 100644 --- a/.github/skills/agentic-workflows/SKILL.md +++ b/.github/skills/agentic-workflows/SKILL.md @@ -16,6 +16,7 @@ Repository overlay (optional): Read only the files you need: Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/action-container-substitutions.md` +- `.github/aw/agent-runtime-instructions.md` - `.github/aw/agentic-chat.md` - `.github/aw/agentic-workflows-mcp.md` - `.github/aw/asciicharts.md` diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs index 524497c133c..29fcf1a8717 100644 --- a/docs/astro.config.mjs +++ b/docs/astro.config.mjs @@ -441,6 +441,7 @@ export default defineConfig({ { label: 'Safe Outputs (Ephemerals)', link: '/reference/ephemerals/' }, { label: 'Safe Outputs (Footers)', link: '/reference/footers/' }, { label: 'Sandbox', link: '/reference/sandbox/' }, + { label: 'Sandbox (Agent Runtimes)', link: '/reference/agent-runtimes/' }, { label: 'Self-Hosted Runners', link: '/reference/self-hosted-runners/' }, { label: 'Sandbox (MCP Gateway)', link: '/reference/mcp-gateway/' }, { label: 'Sandbox (Network Access)', link: '/reference/network/' }, diff --git a/docs/src/content/docs/reference/agent-runtimes.md b/docs/src/content/docs/reference/agent-runtimes.md new file mode 100644 index 00000000000..c087d40ce22 --- /dev/null +++ b/docs/src/content/docs/reference/agent-runtimes.md @@ -0,0 +1,360 @@ +--- +title: Agent Runtime Selection +description: Choose and configure Docker, gVisor, Docker sbx, or ARC DinD for an agentic workflow, with runner requirements and troubleshooting guidance. +sidebar: + order: 1340 +--- + +Agentic workflows use AWF (Agent Workflow Firewall) to run the agent in an isolated environment. The environment can use the runner's standard Docker runtime, gVisor, or Docker sbx. ARC DinD is a runner topology that changes how the standard Docker environment is reached; it is not another value of `sandbox.agent.runtime`. + +Use this page when selecting a runtime, writing workflow frontmatter, provisioning a runner, or diagnosing a runtime setup failure. + +## Runtime and topology fields + +These similarly named fields control different layers: + +| Field | Purpose | Values covered here | +| --- | --- | --- | +| `sandbox.agent.runtime` | Selects the isolation backend for the main agent | `gvisor`, `docker-sbx`, or omitted for Docker | +| `sandbox.agent.runtime-install` | Controls whether gh-aw installs and prepares gVisor or Docker sbx | `true` by default; `false` for a pre-provisioned runner | +| `runner.topology` | Describes how the runner reaches Docker | `arc-dind`, or omitted for a local Docker daemon | +| `tools.github.bounded-queries.runtime` | Selects the backend for bounded-query scripts only | `docker`, `gvisor`, `sbx` | +| `runtimes` | Installs language toolchains such as Node.js, Python, and Go | Unrelated to agent isolation | + +> [!IMPORTANT] +> Do not set `sandbox.agent.runtime: sbx`. The agent runtime value is `docker-sbx`. The shorter `sbx` value is used only by `tools.github.bounded-queries.runtime` and internally by AWF. + +## Choose a runtime + +| Choice | Isolation boundary | Runner requirements | Main tradeoff | +| --- | --- | --- | --- | +| Docker | Linux namespaces, cgroups, and the host kernel | Linux and a usable Docker daemon | Fastest and most compatible, but the agent shares the host kernel | +| gVisor | A `runsc` user-space kernel between the agent and host kernel | Local Docker daemon, `sudo`, systemd, and access to gVisor downloads | Stronger kernel isolation with syscall compatibility and performance overhead | +| Docker sbx | A KVM-backed microVM for the agent | KVM, nested virtualization, `sudo`, apt, Docker Hub credentials, and local Docker | Strongest boundary here, but has the most setup cost and platform constraints | +| ARC DinD | Standard Docker agent container in a DinD sidecar | ARC or equivalent Kubernetes runner with a privileged DinD sidecar and shared work volume | Supports Kubernetes runner fleets, but adds split-filesystem and daemon-connectivity complexity | + +Apply this selection order: + +1. Use **ARC DinD** when the runner is an ARC pod or another Kubernetes runner whose Docker daemon is a DinD sidecar. Do not combine it with gVisor or Docker sbx. +2. Otherwise, use **Docker sbx** when the user requires a hardware-virtualized boundary and the runner exposes working KVM. +3. Otherwise, use **gVisor** when untrusted agent code warrants a smaller host-kernel attack surface and the workload is compatible with `runsc`. +4. Use the default **Docker** runtime when compatibility, startup time, or runner portability is more important than an additional kernel or VM boundary. + +If the user's requirement is unclear, prefer Docker. Do not select a stronger runtime until the runner prerequisites are known to be available. + +## Requirements shared by all choices + +The main agent job requires a Linux runner. macOS and Windows runners are not supported. The runner must have enough CPU, memory, and disk for the agent, AWF, the MCP gateway, proxy containers, and any configured MCP servers. + +Docker must be reachable by the runner user. On a conventional runner, this normally means that `/var/run/docker.sock` exists and the runner user can access it. Verify the baseline before investigating a specialized runtime: + +```bash +docker version +docker info +docker compose version +docker run --rm hello-world +``` + +The runner also needs outbound HTTPS access to GitHub, the selected AI provider, `ghcr.io`, and the domains required by setup steps and `network.allowed`. See [Self-Hosted Runners](/gh-aw/reference/self-hosted-runners/) for the complete runner baseline. + +Compile after changing frontmatter: + +```bash +gh aw compile +``` + +Compilation catches unsupported field values and known incompatible combinations before a workflow runs. + +## Docker + +Docker is the default. Leave `sandbox.agent.runtime` unset: + +```aw wrap +--- +on: issues +sandbox: + agent: + id: awf +--- + +Investigate this issue. +``` + +The entire `sandbox` block may be omitted when its defaults are sufficient. AWF still provides network isolation and proxy enforcement; "Docker" does not mean that the agent runs without a sandbox. + +### Docker runner requirements + +A conventional Docker runner needs: + +- A Linux Docker Engine that the runner user can access. +- Docker Compose support. +- A daemon on the same filesystem as the runner, unless `runner.topology: arc-dind` is configured. +- Outbound access to pull the pinned AWF, proxy, gateway, and MCP images. + +For a self-hosted runner, avoid a remote TCP Docker daemon unless the environment is intentionally configured as a split-daemon topology. Bind-mount source paths are resolved by the daemon, not by the client, so a remote daemon that cannot see the runner workspace causes missing-workspace and mount errors. + +### Docker tradeoffs + +Docker has the lowest startup overhead and the broadest compatibility with build tools, debuggers, filesystem operations, and syscalls. The agent container shares the runner host's kernel, so a kernel vulnerability presents a larger escape surface than gVisor or Docker sbx. + +### Docker troubleshooting + +**`permission denied` while connecting to Docker:** Confirm the socket path and permissions with `docker context show`, `printf '%s\n' "${DOCKER_HOST:-}"`, and `stat /var/run/docker.sock`. Add the runner user to the correct socket group as part of runner provisioning rather than changing permissions in workflow steps. + +**`Cannot connect to the Docker daemon`:** Confirm the daemon is running and that `DOCKER_HOST` points to the intended endpoint. If `DOCKER_HOST` is a `tcp://` address for a DinD sidecar, configure `runner.topology: arc-dind`. + +**The agent sees an empty workspace:** The Docker daemon cannot resolve the runner-side bind-mount source. Use a local daemon, or configure the runner as ARC DinD with a shared work volume. + +**Image pull failures:** Check registry reachability, rate limits, proxy configuration, and any repository-level `container_pins` substitutions. + +## gVisor + +gVisor runs only the agent container under the `runsc` OCI runtime. AWF's infrastructure containers continue to use standard Docker. + +```aw wrap +--- +on: issues +sandbox: + agent: + id: awf + runtime: gvisor +--- + +Investigate this issue. +``` + +### gVisor runner requirements + +The generated setup step: + +1. Detects `x86_64` or `aarch64` with `uname -m`. +2. Downloads pinned `runsc` and `containerd-shim-runsc-v1` binaries and their SHA-512 files from `storage.googleapis.com/gvisor`. +3. Verifies both checksums. +4. Uses `sudo` to install the binaries under `/usr/local/bin`. +5. Runs `sudo runsc install` and `sudo systemctl restart docker`. +6. Verifies the runtime with `docker run --rm --runtime=runsc hello-world`. + +The runner therefore needs: + +- A supported Linux architecture and a Docker Engine managed by systemd. +- Passwordless, non-interactive `sudo` for the runner user. +- Permission to modify Docker's runtime configuration and restart Docker. +- Outbound HTTPS access to `storage.googleapis.com/gvisor` and the registry serving `hello-world`. +- AWF `v0.27.30` or newer. The repository default is newer; this matters when `firewall.version` or `sandbox.agent.version` is pinned. + +> [!IMPORTANT] +> Host-level `sudo` is required by the generated gVisor installation step, but `sandbox.agent.sudo: true` is not required. Leave that field omitted or false to retain AWF's default network-isolation mode. Setting it to true changes the agent security mode and is rejected in strict mode. + +Set `runtime-install: false` when the runner image already contains a working, Docker-registered `runsc` runtime: + +```aw wrap +--- +sandbox: + agent: + id: awf + runtime: gvisor + runtime-install: false +--- +``` + +This skips the generated download, checksum, installation, Docker registration, restart, and smoke-test step. The runner no longer needs workflow-time `sudo`, systemd, or access to the gVisor download host, but `docker info` must already list `runsc`. If any imported workflow sets `runtime-install: false`, false wins during import merging. + +gVisor cannot be combined with `runner.topology: arc-dind`. The generated installer must register `runsc` with the same Docker daemon that starts the agent and restart that daemon through systemd. An ARC runner cannot perform those operations against its DinD sidecar. + +### gVisor tradeoffs + +gVisor substantially reduces the host-kernel syscall surface exposed to agent code. It does not require KVM and is lighter than a microVM. + +The user-space kernel adds CPU, syscall, filesystem, and network overhead. Some low-level workloads can fail because they depend on an unsupported syscall, privileged operation, device, kernel module, eBPF behavior, unusual `/proc` or `/sys` semantics, or exact host-kernel behavior. Prefer Docker for kernel-sensitive build and test workloads unless the stronger boundary is required. + +### gVisor troubleshooting + +**Download or checksum failure:** Confirm access to `storage.googleapis.com/gvisor`, inspect proxy or TLS interception, and verify that the runner architecture is reported as `x86_64` or `aarch64`. + +**`sudo` prompts or fails:** Provision passwordless `sudo` for the runner service account. Setting `sandbox.agent.sudo: true` does not grant host permissions and will not fix the installer. + +**`systemctl: command not found` or Docker is not a systemd service:** The runner image is incompatible with the generated installer. Use a conventional systemd-based runner, pre-provision and maintain gVisor outside the workflow only if the generated setup remains compatible, or select Docker. + +**`unknown or invalid runtime name: runsc`:** Run `docker info` and confirm `runsc` appears in the runtimes map. Check the output of `runsc install`, confirm Docker was restarted rather than reloaded, and confirm the workflow is not using a different or remote Docker daemon. + +**The runtime installs but the agent still uses Docker:** Remove an AWF pin older than `v0.27.30`, or update `firewall.version` or `sandbox.agent.version` to `v0.27.30` or newer and recompile. + +**Tests fail only under gVisor:** Treat this as a compatibility issue if the failure involves syscalls, devices, namespaces, tracing, eBPF, or kernel-specific filesystem behavior. Reproduce with `docker run --runtime=runsc ...` and compare it with the same image under standard Docker. + +## Docker sbx + +Docker sbx runs the agent in a KVM-backed microVM. The MCP gateway, Squid proxy, API proxy, and other infrastructure containers remain on the Docker host. + +```aw wrap +--- +on: issues +sandbox: + agent: + id: awf + runtime: docker-sbx + sudo: true +--- + +Investigate this issue. +``` + +Add these Actions secrets to the repository or organization: + +| Secret | Purpose | +| --- | --- | +| `DOCKER_USERNAME` | Docker Hub account used by both the Docker and sbx CLIs | +| `DOCKER_PAT` | Docker Hub personal access token used to pull the sandbox template | + +`DOCKER_PAT` is required for Docker sbx, including when `runtime-install: false`, because the compiled workflow refreshes sbx credentials immediately before agent execution. Use a Docker Hub personal access token rather than a password, and make sure it can pull `docker/sandbox-templates:shell-docker`. + +### Docker sbx runner requirements + +The runner needs: + +- Linux with the KVM module loaded and `/dev/kvm` exposed. +- Nested virtualization when the runner itself is a virtual machine. +- Passwordless, non-interactive `sudo`. +- An apt-based distribution on which the official Docker repository and `docker-sbx` package can be installed. +- Docker Engine and the Docker CLI. +- Docker Hub credentials with access to `docker/sandbox-templates:shell-docker`. +- Outbound HTTPS access to `get.docker.com`, Docker's apt repository, and Docker Hub. +- AWF `v0.27.30` or newer. + +The compiler generates fail-fast KVM and secret checks, installs `docker-sbx`, changes `/dev/kvm` permissions, starts the sbx daemon, authenticates both CLIs, initializes an allow-all sbx policy, pulls the template, and runs a create/exec/remove smoke test. It refreshes sbx credentials again immediately before AWF starts the agent. + +`sandbox.agent.sudo: true` is mandatory when runtime installation is enabled. Unlike gVisor, the compiler treats this as an explicit Docker sbx installation requirement. AWF still enables network isolation for Docker sbx; the field permits the required installation path rather than disabling egress enforcement for the microVM. + +Set `runtime-install: false` when the runner image already has Docker sbx, a working sbx daemon and policy, KVM access, and the required template: + +```aw wrap +--- +sandbox: + agent: + id: awf + runtime: docker-sbx + runtime-install: false +--- +``` + +This skips the generated KVM check, secret check, package installation, daemon setup, template pull, and pre-flight smoke test. It also removes the compile-time requirement for `sandbox.agent.sudo: true`. The runner must already satisfy those checks; gh-aw does not verify them when installation is disabled. The credential-refresh step still runs immediately before agent execution, so `DOCKER_USERNAME` and `DOCKER_PAT` remain required. If any imported workflow sets `runtime-install: false`, false wins during import merging. + +Docker sbx cannot be combined with `runner.topology: arc-dind`. ARC DinD normally does not expose nested KVM, and the sbx daemon must run on the runner host rather than inside the DinD sidecar. + +### Docker sbx tradeoffs + +Docker sbx provides the strongest isolation boundary among these choices because agent code runs behind a hardware-virtualized guest kernel. It is appropriate when agent code is highly untrusted and the runner platform supports KVM. + +It has the highest cold-start cost, consumes more memory and disk, requires Docker Hub credentials, changes `/dev/kvm` permissions for the job, and adds an sbx daemon and template lifecycle. The microVM boundary can also expose path, networking, CLI installation, and terminal behavior differences. For example, gh-aw omits TTY mode for Docker sbx because sbx TTY execution can terminate long-running sessions prematurely. + +### Docker sbx troubleshooting + +**`KVM kernel module is not loaded` or `/dev/kvm is missing`:** The runner does not provide hardware virtualization. Enable nested virtualization and pass `/dev/kvm` through to the runner, or select gVisor or Docker. Frontmatter cannot add KVM capability. With `runtime-install: false`, this generated check is skipped, but sbx execution still fails if KVM is unavailable. + +**Permission denied for `/dev/kvm`:** With runtime installation enabled, confirm the runner can execute passwordless `sudo` and that its security policy permits the generated `chmod 666 /dev/kvm`. With `runtime-install: false`, gh-aw does not run that `chmod`, so provision the runner image or pod so the runner user can access `/dev/kvm` before the workflow starts. If neither access model is acceptable, do not use Docker sbx. + +**`DOCKER_PAT` or `DOCKER_USERNAME` is empty:** Define both Actions secrets in the scope available to the workflow. Secrets are not passed to workflows triggered from untrusted forks, so Docker sbx is unsuitable for such runs unless the trigger and credential model are changed safely. + +**`Unable to locate package docker-sbx`:** Confirm the runner is apt-based, `curl https://get.docker.com` can configure the Docker repository, and the package is available for the runner architecture and distribution. + +**The sbx daemon does not become ready:** Inspect `/tmp/sbx-daemon.log`, then run `sbx daemon status`. Check stale daemon processes, policy state, KVM access, and whether the runner permits the daemon to create its required resources. With `runtime-install: false`, gh-aw does not start or initialize the daemon; runner provisioning must do so. + +**`user is not authenticated to Docker`:** Confirm the PAT is valid for Docker Hub and both `docker login` and `sbx login` succeed. Current compiled workflows refresh sbx credentials immediately before execution; upgrade gh-aw and recompile if the refresh step is absent from the lock file. + +**Template pull fails:** Test `docker pull docker/sandbox-templates:shell-docker` with the same credentials. Check Docker Hub access, account entitlements, proxy behavior, and rate limits. + +**Pre-flight smoke test fails:** The generated test creates `test-sandbox-direct`, executes `uname -a`, and removes it. Inspect the first failing `sbx create`, `sbx exec`, or `sbx stop` command before investigating AWF because the failure is below the workflow firewall layer. + +**The CLI is missing inside the microVM:** Upgrade gh-aw and recompile. Docker sbx requires engine CLIs to be staged under `${RUNNER_TEMP}/gh-aw/engine-cli`, which is visible to the microVM. + +## ARC with Docker-in-Docker + +ARC DinD describes a split-daemon runner: the GitHub Actions runner is one container and Docker runs in a privileged sidecar. The agent still uses standard Docker, so omit `sandbox.agent.runtime`. + +```aw wrap +--- +on: issues +runs-on: arc-runner-set +runner: + topology: arc-dind +--- + +Investigate this issue. +``` + +Do not set `runtime: docker`, `runtime: gvisor`, or `runtime: docker-sbx` in this configuration. There is no explicit `docker` value for `sandbox.agent.runtime`. + +### ARC DinD runner requirements + +The ARC or equivalent Kubernetes pod needs: + +- `containerMode.type="dind"` or an equivalent privileged Docker sidecar. Kubernetes container mode is not supported. +- A shared `/home/runner/_work` volume between the runner and DinD sidecar. +- `DOCKER_HOST` set to the sidecar's `tcp://` endpoint. +- An unprivileged runner container; only the DinD sidecar needs `privileged: true`. +- No `sudo`, `apt install`, or other root-requiring commands in `steps`, `pre-steps`, `pre-agent-steps`, or `post-steps`. +- AWF `v0.27.20` or newer. +- A tool cache on a daemon-visible shared path, such as `/tmp/gh-aw/tool-cache`, rather than `/opt/hostedtoolcache`. + +The compiler uses `runner.topology: arc-dind` to enable sysroot staging, shared-volume paths, chroot identity, log relocation, network isolation, and tool-cache checks. At runtime, the generated workflow also inspects `DOCKER_HOST` and passes the Docker endpoint to AWF. + +If a custom `copilot-setup-steps` job installs the Copilot CLI on a runner container with `allowPrivilegeEscalation: false`, invoke `install_copilot_cli.sh --rootless`. This installs the CLI under `~/.local/bin` instead of using `sudo`. + +See [How to run GitHub Copilot coding agent on ARC with Docker-in-Docker](/gh-aw/guides/arc-dind-copilot-agent/) for the runner scale-set setup. + +### ARC DinD tradeoffs + +ARC DinD makes agentic workflows available to ephemeral Kubernetes runner fleets without granting privilege to the runner container. The privileged DinD sidecar remains a significant infrastructure trust boundary. + +The separate runner and daemon filesystems make paths, tool caches, sockets, and logs more complex than on a conventional Docker host. DinD also adds image caching and storage overhead. Choose ARC DinD because the runner platform requires it, not as an isolation upgrade over Docker. + +### ARC DinD troubleshooting + +**Compilation rejects `sudo` or `apt-get install`:** Move system packages into the runner image or DinD image. ARC DinD workflows are validated as rootless and must not bootstrap host packages during the job. + +**`Docker daemon is not accessible` from the MCP gateway:** If a Unix socket is mounted at a nonstandard path, set `GH_AW_DOCKER_SOCK_PATH` in the runner pod. The gateway derives the socket group ID with `stat` when possible; set `GH_AW_DOCKER_SOCK_GID` only when that detection fails or the socket is not visible during setup. See [Docker socket override for split-daemon topologies](/gh-aw/reference/self-hosted-runners/#docker-socket-override-for-split-daemon-topologies). + +**The agent sees an empty workspace or mount source does not exist:** Confirm both containers share `/home/runner/_work`, `GITHUB_WORKSPACE` is under that volume, and `runner.topology: arc-dind` was present when the lock file was compiled. + +**`RUNNER_TOOL_CACHE is under /opt` warning or setup tools are missing:** Set `RUNNER_TOOL_CACHE=/tmp/gh-aw/tool-cache` in the runner pod and re-run. `/opt/hostedtoolcache` is normally visible only to the runner container. + +**`spawn /usr/local/bin/copilot ENOENT` or an engine CLI is missing:** Upgrade gh-aw and recompile. Current workflows stage activated engine binaries into `${RUNNER_TEMP}/gh-aw/bin`, which is visible to the DinD daemon. + +**Proxy DNS failure such as `getaddrinfo EAI_AGAIN`:** Docker containers created by the DinD daemon do not automatically use Kubernetes service discovery. Make the proxy reachable by IP or configure DNS forwarding from the Docker network to cluster DNS. + +**Logs are missing from `/tmp/gh-aw`:** ARC DinD writes logs under `$RUNNER_TEMP/gh-aw/sandbox/firewall/logs/` because `$RUNNER_TEMP` is on the shared work volume. + +## Bounded-query runtime names + +Bounded queries have a separate runtime setting: + +```aw wrap +--- +tools: + github: + bounded-queries: + runtime: gvisor # docker, gvisor, or sbx +--- +``` + +This setting affects only bounded-query scripts. It does not configure the main agent and does not provision the host for `sandbox.agent.runtime`. + +The bounded-query `sbx` backend is experimental and capability-gated. AWF performs a fail-closed host preflight and does not fall back to Docker or gVisor. Do not choose it merely because the main agent uses `docker-sbx`. + +## Debug in dependency order + +Runtime failures are easiest to isolate from the runner upward: + +1. Verify the runner operating system, architecture, disk, memory, and required privilege. +2. Verify Docker independently with `docker version`, `docker info`, `docker compose version`, and `docker run --rm hello-world`. +3. Verify the specialized backend independently: `docker run --runtime=runsc ...` for gVisor, `sbx create` and `sbx exec` for Docker sbx, or Docker API access through `DOCKER_HOST` for ARC DinD. +4. Confirm the frontmatter uses the correct field and value, then run `gh aw compile`. +5. Inspect the generated lock file for the expected setup and pre-flight steps. +6. Inspect AWF logs. Conventional runners use `/tmp/gh-aw/sandbox/firewall/logs/`; ARC DinD uses `$RUNNER_TEMP/gh-aw/sandbox/firewall/logs/`. + +Enable compiler diagnostics when generated configuration is unexpected: + +```bash +DEBUG=workflow:* gh aw compile 2>debug.log +``` + +Do not compensate for a missing host capability with extra agent mounts, manual AWF arguments, or `sandbox.agent.sudo`. Fix the runner prerequisite or select a compatible runtime. diff --git a/pkg/cli/data/agentic_workflows_fallback_aw_files.json b/pkg/cli/data/agentic_workflows_fallback_aw_files.json index 7d1f03c07d6..ef0f0e3b030 100644 --- a/pkg/cli/data/agentic_workflows_fallback_aw_files.json +++ b/pkg/cli/data/agentic_workflows_fallback_aw_files.json @@ -1,5 +1,6 @@ [ "action-container-substitutions.md", + "agent-runtime-instructions.md", "agentic-chat.md", "agentic-workflows-mcp.md", "asciicharts.md",