Skip to content

packages: add containerd-2.2#862

Merged
ytsssun merged 3 commits intobottlerocket-os:developfrom
ytsssun:containerd-22/core-kit
Mar 27, 2026
Merged

packages: add containerd-2.2#862
ytsssun merged 3 commits intobottlerocket-os:developfrom
ytsssun:containerd-22/core-kit

Conversation

@ytsssun
Copy link
Copy Markdown
Contributor

@ytsssun ytsssun commented Mar 12, 2026

Issue number:

Closes #806

Description of changes:

Introduces containerd 2.2.2 as a new package, replacing containerd 2.1 for all 34 variants.

Upstream changelogs:

Patch changes:

  • Dropped 1002-bump-google-golang-org-grpc.patch — upstream 2.2.2 ships gRPC v1.78.0+ natively
  • Renumbered remaining patches:
    • 1001 (io_uring revert) — still needed, io_uring syscalls blocked in seccomp defaults
    • 1002 (credential fallback) — was 1003, transfer service credential fallback
    • 1003 (hosts-dir) — was 1004, ctr default hosts-dir

Other changes:

  • Updated clarify.toml — removed stale goyaml.v2/* entries
  • Config templates unchanged — already using hosts.toml-based registry config

Testing done:

1. Binary versions (verified on running node):

bash-5.2# containerd --version
containerd github.com/containerd/containerd/v2 2.2.2+bottlerocket 301b2dac98f15c27117da5c8af12118a041a31d9
bash-5.2# ctr --version
ctr github.com/containerd/containerd/v2 2.2.2+bottlerocket

2. Conformance testing: 20/20 unique variant/arch combos passing

Phase Arch Instance Variants Tested Result
A x86_64 m5.large aws-k8s-1.33, 1.33-fips, 1.34, 1.34-fips, 1.35, 1.35-fips ✅ All passing
B aarch64 m6g.large aws-k8s-1.33, 1.33-fips, 1.34, 1.34-fips, 1.35, 1.35-fips ✅ All passing
C x86_64 g5.xlarge aws-k8s-1.29-nvidia-fips, 1.33-nvidia, 1.34-nvidia, 1.35-nvidia ✅ All passing
D aarch64 g5g.2xlarge aws-k8s-1.29-nvidia-fips, 1.33-nvidia, 1.34-nvidia, 1.35-nvidia ✅ All passing
Full conformance results per variant/arch
Variant Arch K8s Version Tests Instance
aws-k8s-1.33 x86_64 1.33 419/419 m5.large
aws-k8s-1.33-fips x86_64 1.33 419/419 m5.large
aws-k8s-1.34 x86_64 1.34 424/424 m5.large
aws-k8s-1.34-fips x86_64 1.34 424/424 m5.large
aws-k8s-1.35 x86_64 1.35 441/441 m5.large
aws-k8s-1.35-fips x86_64 1.35 441/441 m5.large
aws-k8s-1.33 aarch64 1.33 419/419 m6g.large
aws-k8s-1.33-fips aarch64 1.33 419/419 m6g.large
aws-k8s-1.34 aarch64 1.34 424/424 m6g.large
aws-k8s-1.34-fips aarch64 1.34 424/424 m6g.large
aws-k8s-1.35 aarch64 1.35 441/441 m6g.large
aws-k8s-1.35-fips aarch64 1.35 441/441 m6g.large
aws-k8s-1.29-nvidia-fips x86_64 1.29 388/388 g5.xlarge
aws-k8s-1.33-nvidia x86_64 1.33 419/419 g5.xlarge
aws-k8s-1.34-nvidia x86_64 1.34 424/424 g5.xlarge
aws-k8s-1.35-nvidia x86_64 1.35 441/441 g5.xlarge
aws-k8s-1.29-nvidia-fips aarch64 1.29 388/388 g5g.2xlarge
aws-k8s-1.33-nvidia aarch64 1.33 419/419 g5g.2xlarge
aws-k8s-1.34-nvidia aarch64 1.34 424/424 g5g.2xlarge
aws-k8s-1.35-nvidia aarch64 1.35 441/441 g5g.2xlarge

3. SubPath test (bottlerocket#4755): Kubernetes 1.35 enables OCI image volumes by default, but containerd 2.1 silently ignores the subPath field in volumeMounts for these volumes — mounting the entire image filesystem regardless of the subPath value. We built a test OCI image with a known directory structure, deployed the same pod manifest on both the GA AMI (containerd 2.1) and our custom build (containerd 2.2), and confirmed that 2.1 mounts the full image while 2.2 correctly scopes the mount to only the specified subdirectory. Verified with containerd 2.2.1; low risk for 2.2.2 (no relevant changes in the image volume path).

Full test steps and output

Test image pushed to ECR (<ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/subpath-test:latest):

FROM public.ecr.aws/docker/library/busybox:latest
RUN mkdir -p /data/subdir-a /data/subdir-b && \
    echo "content-from-subdir-a" > /data/subdir-a/file-a.txt && \
    echo "content-from-subdir-b" > /data/subdir-b/file-b.txt && \
    echo "content-from-root" > /root-file.txt

Pod manifest (same for both tests), mounting with subPath: data/subdir-a:

apiVersion: v1
kind: Pod
metadata:
  name: subpath-test
spec:
  containers:
  - name: checker
    image: public.ecr.aws/docker/library/busybox:latest
    command: ["sleep", "3600"]
    volumeMounts:
    - name: test-image-vol
      mountPath: /mnt/test
      subPath: data/subdir-a
  volumes:
  - name: test-image-vol
    image:
      reference: <ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/subpath-test:latest

containerd 2.1 (aws-k8s-1.35 GA AMI) — subPath ignored ❌

The entire image filesystem is mounted at /mnt/test:

$ kubectl exec subpath-test -- ls -la /mnt/test/
total 24
drwxr-xr-x    1 root     root             6 Mar 16 21:41 .
drwxr-xr-x    3 root     root            18 Mar 16 21:41 ..
drwxr-xr-x    2 root     root         12288 Sep 26  2024 bin
drwxr-xr-x    4 root     root            38 Mar 11 23:33 data
drwxr-xr-x    2 root     root             6 Sep 26  2024 dev
drwxr-xr-x    3 root     root           100 Sep 26  2024 etc
drwxr-xr-x    2 nobody   nobody           6 Sep 26  2024 home
drwxr-xr-x    2 root     root          4096 Sep 26  2024 lib
lrwxrwxrwx    1 root     root             3 Sep 26  2024 lib64 -> lib
drwxr-xr-x    2 root     root             6 Mar 11 23:33 proc
drwx------    2 root     root             6 Sep 26  2024 root
-rw-r--r--    1 root     root            18 Mar 11 23:33 root-file.txt
drwxr-xr-x    2 root     root             6 Mar 11 23:33 sys
drwxrwxrwt    2 root     root             6 Sep 26  2024 tmp
drwxr-xr-x    4 root     root            29 Sep 26  2024 usr
drwxr-xr-x    4 root     root            30 Sep 26  2024 var

$ kubectl exec subpath-test -- ls -la /mnt/test/data
total 0
drwxr-xr-x    4 root     root            38 Mar 11 23:33 .
drwxr-xr-x    1 root     root             6 Mar 16 21:41 ..
drwxr-xr-x    2 root     root            24 Mar 11 23:33 subdir-a
drwxr-xr-x    2 root     root            24 Mar 11 23:33 subdir-b

containerd 2.2 (custom aws-k8s-1.35 build) — subPath working ✅

Only the contents of data/subdir-a are mounted at /mnt/test:

$ kubectl exec subpath-test -- ls -la /mnt/test/
total 4
drwxr-xr-x    2 root     root            24 Mar 11 23:33 .
drwxr-xr-x    3 root     root            18 Mar 16 21:22 ..
-rw-r--r--    1 root     root            22 Mar 11 23:33 file-a.txt

$ kubectl exec subpath-test -- cat /mnt/test/file-a.txt
content-from-subdir-a

$ kubectl exec subpath-test -- ls /mnt/test/root-file.txt
ls: /mnt/test/root-file.txt: No such file or directory

$ kubectl exec subpath-test -- ls /mnt/test/subdir-b
ls: /mnt/test/subdir-b: No such file or directory

$ kubectl exec subpath-test -- find /mnt/test -type f
/mnt/test/file-a.txt

4. Image pull performance: Tested CRI pull times on m5d.4xlarge with ephemeral NVMe storage, pulling docker.io/pytorch/pytorch:2.3.0-cuda12.1-cudnn8-devel (8.5GB compressed).

Configuration crictl pull ctr pull vs 2.1 baseline
Containerd 2.1.6 (default) 1m34.7s 85.4s
Containerd 2.2.2 (enable parallel unpack) 1m8.1s 66.3s 28% faster
SOCI parallel-pull-unpack (on 2.1.6) 1m4.3s 84.7s 32% faster
Test environment and configuration

All tests on m5d.4xlarge (16 vCPU, 64GB RAM, 2x300GB NVMe SSD).
Ephemeral NVMe storage bound via bootstrap command to /var/lib/containerd, /var/lib/kubelet, /var/log/pods.
All three configurations use 20 max concurrent downloads.

Containerd 2.1.6 (baseline): Concurrent downloads only, no parallel unpack.

[plugins.'io.containerd.transfer.v1.local']
max_concurrent_downloads = 20
concurrent_layer_fetch_buffer = 16777216

Containerd 2.2.2: Concurrent downloads + parallel unpack (5 concurrent unpacks).

[plugins.'io.containerd.transfer.v1.local']
max_concurrent_downloads = 20
concurrent_layer_fetch_buffer = 16777216
max_concurrent_unpacks = 5

SOCI parallel-pull-unpack (on containerd 2.1.6): SOCI manages both downloads and unpacks via the CRI path.
Ephemeral storage also bound to /var/lib/soci-snapshotter (SOCI's unpack working directory).

[settings.container-runtime]
snapshotter = "soci"

[settings.container-runtime-plugins.soci-snapshotter]
pull-mode = "parallel-pull-unpack"

[settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]
concurrent-download-chunk-size = "8mb"
discard-unpacked-layers = true
max-concurrent-downloads = 20
max-concurrent-downloads-per-image = 5
max-concurrent-unpacks = 5
max-concurrent-unpacks-per-image = 4

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Comment thread packages/containerd-2.1/containerd-2.1.spec
@ytsssun ytsssun force-pushed the containerd-22/core-kit branch from 5e92c50 to 4d712bc Compare March 20, 2026 21:43
@henry118
Copy link
Copy Markdown

@ytsssun We should carry containerd/containerd#13125 as part of this work.

@ytsssun
Copy link
Copy Markdown
Contributor Author

ytsssun commented Mar 25, 2026

@ytsssun We should carry containerd/containerd#13125 as part of this work.

Sounds good. Will that patch land in a new release soon? If not I can add a patch for this. @henry118

@henry118
Copy link
Copy Markdown

@ytsssun We should carry containerd/containerd#13125 as part of this work.

Sounds good. Will that patch land in a new release soon? If not I can add a patch for this. @henry118

I think it will be part of v2.2.3. We'll have to carry it by ourselves for 2.2.2

ytsssun added 3 commits March 26, 2026 22:21
Signed-off-by: Yutong Sun <yutongsu@amazon.com>
Bump the priority epoch for containerd 2.1 from 0 to 1 to establish
the correct precedence among containerd packages. A higher epoch
means the package is preferred when resolving the generic 'containerd'
dependency.

The resulting priority order is:
  containerd 1.7 (epoch 2) > containerd 2.1 (epoch 1) > containerd 2.2 (epoch 0)

Signed-off-by: Yutong Sun <yutongsu@amazon.com>
…el unpack

Carry containerd/containerd#13125 (cherry-pick of #13115 to release/2.2).
Fixes whiteouts being ignored when parallel unpack is used with overlayfs
by converting bind mounts to overlay mounts before applying diffs.

Signed-off-by: Yutong Sun <yutongsu@amazon.com>
@KCSesh
Copy link
Copy Markdown
Contributor

KCSesh commented Mar 26, 2026

LGTM will wait for patch

@ytsssun ytsssun force-pushed the containerd-22/core-kit branch from 4d712bc to 1313a25 Compare March 26, 2026 23:28
@ytsssun
Copy link
Copy Markdown
Contributor Author

ytsssun commented Mar 26, 2026

Pushed patch per @henry118 , see the commit 1313a25

@arnaldo2792 arnaldo2792 self-requested a review March 27, 2026 18:52
Copy link
Copy Markdown

@henry118 henry118 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

containerd patch lgtm

@ytsssun
Copy link
Copy Markdown
Contributor Author

ytsssun commented Mar 27, 2026

Verified the fix on a running aws-k8s-1.35 x86_64 node (containerd 2.2.2+bottlerocket).
Tested using the upstream reproduction image (docker.io/ajhalaney/whiteout:latest):

bash-5.2# ctr run --rm --snapshotter overlayfs docker.io/ajhalaney/whiteout:latest test
bash-5.2# echo $?
0

Snapshot inspection confirms the whiteout character device is present:

-rw-r--r--. 1 root root    0 Mar 13 20:28 .../snapshots/53/fs/this-will-be-deleted
c---------. 1 root root 0, 0 Mar 27 06:34 .../snapshots/54/fs/this-will-be-deleted

Without the patch, snapshot would be missing the c--------- device.

@ytsssun ytsssun merged commit a4afada into bottlerocket-os:develop Mar 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade to containerd 2.2

5 participants