Skip to content

THRIFT-6237: Shrink the MSVC Docker image - #3844

Merged
Jens-G merged 1 commit into
apache:masterfrom
slachiewicz:THRIFT-6237
Sep 14, 2026
Merged

Jens-G merged 1 commit into
apache:masterfrom
slachiewicz:THRIFT-6237

Conversation

@slachiewicz

@slachiewicz slachiewicz commented Sep 11, 2026

Copy link
Copy Markdown
Member

Follow-up to THRIFT-5952. The MSVC CI image is 6.3 GiB compressed and every run pulls all of it, because the windows-2025 runners pre-cache no container images. The last full rebuild took 53 minutes against the 60 minute step timeout. Measurements and the full list of findings are in THRIFT-6237.

What the diff does not show:

  • The .NET Framework SDK base (4.0 GiB compressed) contributed only the dotnet CLI. C++ compiles with the separately installed VS 2022 Build Tools (MSVC 19.44 in the CMake log), and all lib/netstd test projects target net10.0. The .NET 10 SDK image is 2.7 GiB.
  • The one thing the C++ build still needed from a Framework SDK is the 4.7.2 reference assemblies for lib/cpp/test/testgencppCLI.vcxproj (C++/CLI). Without them cmake --build fails with MSB3644 and build.bat skips ctest, so CI reported a missing LastTest.log rather than a test result. The targeting pack is now a Build Tools component.
  • The SDK major must stay in step with the highest TargetFramework in lib/netstd: an SDK 10 image cannot build a net11.0 target. A comment next to FROM says so.
  • jdk8 (377 MiB) was never used: the CMake summary reports the Java library OFF because Gradle is missing.
  • cmd treats && next after IF ... or FOR ... DO as part of that command's body. The SETX calls that followed the cleanup loops therefore ran once per loop iteration, or only when the directory existed. The loops are now parenthesised and placed after the environment writes, and PATH is written to the machine-scope registry key, which has no 1024 character SETX limit.
  • The workflow hashed only the Dockerfile for the image tag, so editing the two COPY'd appveyor scripts never produced a new image.
  • The Pull cached image step ended in Error: Process completed with exit code 1 on every build-from-scratch run: the runner exits a pwsh step with the last native command's exit code, which after a manifest unknown pull is 1, and continue-on-error hid it. The step now exits 0 on that path and continue-on-error is dropped, so a real pull failure fails the job at the pull rather than at the container build with no image.
  • OpenSSL moves to 3.6.4. slproweb no longer serves the 3.6.2 installer, so master's Dockerfile currently fails to build at that step (fork run).
  • Node.js is pinned to the LTS line (24.20.0), a change from whatever the unpinned nodejs package resolved to when the current image was built.

Verified: upstream run 34623301653 and fork run 34623297376 → image builds from scratch and all tests pass. The pull_request run skips the GHCR push, so the first master run after merge rebuilds and pushes the image.

Step Before Now
Build Docker image 53 min 29 min
Build and test inside container 18 min
  • Did you create an Apache Jira ticket?
  • If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
  • Did you squash your changes to a single commit?
  • Did you do your best to avoid breaking changes?
  • If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

This change was created with AI assistance.

@mergeable mergeable Bot added build and general CI cmake, automake and build system changes github_actions Pull requests that update GitHub Actions code labels Sep 11, 2026
@slachiewicz

Copy link
Copy Markdown
Member Author

@Jens-G there where some changes to what .net versions we support (i don't have any expierience in this area)

@Jens-G

Jens-G commented Sep 11, 2026

Copy link
Copy Markdown
Member

there where some changes to what .net versions we support

Not sure what the quesiton is, but try to answer the best I can:

  • Policy is to support what's not EOL, i.e. around November I am going to add net11 and drop 7 and 8.
  • Plus netstandard2.x for compatibility.

In the past we had two different packages, the original NET Framework one, from which the new dotnet one emerged as a separte library package. These finally got consolidated into what we have today, essentially dropping native NET Framework in favour of netstandard2.x.

So it might be well that the NET Framework you are about to remove is some relict from that time.

Does that answer the question?

@slachiewicz

Copy link
Copy Markdown
Member Author

Yes, that answers it, thanks. The Framework base was exactly that relic: the C++ build uses the separately installed Build Tools, and the only thing still needed from a Framework SDK is the 4.7.2 targeting pack for the C++/CLI test project, which is now a Build Tools component.

One coupling to note for November: every lib/netstd project tops out at net10.0, and the SDK 10 image cannot build a net11 target. When you add net11, the FROM line in build/docker/msvc/Dockerfile must move to mcr.microsoft.com/dotnet/sdk:11.0 in the same change.

Branch rewritten: the two commits are squashed into one, rebased on current master (which picks up the TNonblockingServerTest fix from THRIFT-6244 that caused the AppVeyor failure here), and a comment next to FROM records the SDK/TargetFramework coupling. No other change to the diff.

This comment was created with AI assistance.

@slachiewicz
slachiewicz marked this pull request as ready for review September 13, 2026 20:12
Client: build

Base the image on the .NET 10 SDK image instead of the .NET Framework SDK
image: the C++ build uses the separately installed VS 2022 Build Tools, and
lib/netstd only needs the dotnet CLI (all test projects target net10.0).
Drop jdk8, which CMake never used (Java library OFF, Gradle missing), the
Boost debug and static-runtime variants, the OpenSSL debug libraries and
the redundant curl package, and pin the Chocolatey packages. Node.js moves
to the LTS line.

Swapping the base image drops the .NET Framework reference assemblies that
lib/cpp/test/testgencppCLI.vcxproj (C++/CLI) needs, so cmake --build failed
with MSB3644. The 4.7.2 targeting pack is added as a Build Tools component
instead of relying on the base image for it.

PATH is extended in the machine-scope registry key instead of through
user-scope SETX, which re-embeds the expanded PATH on every call and
truncates at 1024 characters. Cleanup FOR and IF commands are wrapped in
parentheses: cmd treats everything after them, including "&& next", as
their body, so the SETX calls that followed them ran inside the loop, or
only when the directory existed.

OpenSSL moves to 3.6.4 because slproweb no longer serves the 3.6.2
installer, which made the previous Dockerfile fail to build.

The workflow hashes the two copied appveyor scripts together with the
Dockerfile so that editing them produces a new image tag.

The "Pull cached image" step no longer ends in "Process completed with
exit code 1" on the build-from-scratch path: the runner exits a pwsh step
with the last native command's exit code, which after a "manifest unknown"
pull is 1, and continue-on-error hid it. The step now exits 0 there and
continue-on-error is dropped, so a real pull failure fails the job at the
pull instead of at the container build with no image.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@slachiewicz

Copy link
Copy Markdown
Member Author

Rewritten again (5da8aa7): the Pull cached image step ended in Error: Process completed with exit code 1 on every build-from-scratch run. A pwsh step exits with the last native command's exit code, which after a manifest unknown pull is 1, and continue-on-error masked it. The step now exits 0 on that path and continue-on-error is dropped, so a real pull failure fails the job at the pull instead of at the container build. The image build itself is unchanged; the previous run on 698b56b was green (image build 30 min, tests 19 min).

This comment was created with AI assistance.

@Jens-G

Jens-G commented Sep 14, 2026

Copy link
Copy Markdown
Member

Let's see the net11 RTM first. I'm not going to invest time before that.

@Jens-G
Jens-G merged commit 2b4a296 into apache:master Sep 14, 2026
102 checks passed
@Jens-G

Jens-G commented Sep 14, 2026

Copy link
Copy Markdown
Member

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

@slachiewicz
slachiewicz deleted the THRIFT-6237 branch September 15, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build and general CI cmake, automake and build system changes github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants