Skip to content

Cap every build workflow job at 60 minutes - #3816

Open
slachiewicz wants to merge 1 commit into
apache:masterfrom
slachiewicz:ci-job-timeouts
Open

Cap every build workflow job at 60 minutes#3816
slachiewicz wants to merge 1 commit into
apache:masterfrom
slachiewicz:ci-job-timeouts

Conversation

@slachiewicz

@slachiewicz slachiewicz commented Sep 7, 2026

Copy link
Copy Markdown
Member

Only lib-python and lib-python-macos carried a timeout-minutes, so every other job in the build workflow runs to the 6-hour GitHub default when it hangs. lib-d hung in make check on 7 September and held a runner for 47 minutes.

The cap is 60 rather than 30 because the apt mirror, not the work, sets the ceiling. Over the 296 build.yml runs that concluded success or failure between 2026-08-25 and 2026-09-13 (24,228 successful jobs):

Threshold Successful jobs above it Runs affected
30 min 16 14 of 296
45 min 1 1
60 min 0 0

All 16 are the same shape: 28 to 42 minutes inside an apt-get install step downloading the libboost-all-dev closure from azure.archive.ubuntu.com, then under 5 minutes for everything else. The longest was cross-test (php, …) at 45.1 minutes in run 34603753680. Per-job budgets do not help here, since the slow step is shared by nearly every job and the compute part is never the problem. 60 fails none of those runs and still frees a stuck runner six times sooner than the default.

The two python jobs keep the 20 they already had. lib-python runs the same apt step, so its 20 has the same exposure, but that value predates this PR.

Verified: parsing build.yml after the change lists all 18 jobs with a timeout-minutes.

This change was created with AI assistance (Claude Opus 5).

@mergeable mergeable Bot added the github_actions Pull requests that update GitHub Actions code label Sep 7, 2026
@slachiewicz
slachiewicz marked this pull request as draft September 7, 2026 17:23
@slachiewicz

Copy link
Copy Markdown
Member Author

30 minutes is still a lot and if You have suggestion what values apply to each module - i can adjust

@Jens-G

Jens-G commented Sep 11, 2026

Copy link
Copy Markdown
Member

Code review

No blocking issues found. Checked for bugs and CLAUDE.md compliance.

One suggestion, below the bar for an issue but verified:

  • 30 minutes also cuts off runs that currently pass. The table in the description covers the last five green master runs. Looking at build.yml runs from 2026-09-02 to 2026-09-11 instead, several jobs succeeded after more than 30 minutes because an apt-get step stalled on the package mirror:

    • cross-test (rb, java,kotlin): 34 min, 33 of them in "Install openssl and certificates" (run 34139215123, push to master)
    • lib-java-kotlin: 37 min, 33 of them in "Install dependencies" (run 33613699571)
    • lib-erl (26.2, 3.27.0): 41 min, nearly all of it in "Install dependencies" (run 34137352657)

    With this change those runs fail and need a re-run. These numbers may help with the per-job values you asked about.

runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:

🤖 Generated with Claude Code

Client: build

A hung job otherwise runs to the 6-hour GitHub default before anything
notices. lib-d hung in make check on 2026-09-07 and held a runner for 47
minutes.

The value is set by the apt mirror, not by the work. Of the 296 build.yml
runs that concluded between 2026-08-25 and 2026-09-13, 14 had a job that
succeeded only after more than 30 minutes: 16 jobs in all, each spending
28 to 42 minutes inside an apt-get install step and under 5 minutes on
everything else, the longest at 45 minutes overall. A 30-minute cap would
have failed each of those runs; 60 fails none of them and still frees a
stuck runner six times sooner than the default. lib-python and
lib-python-macos keep the 20 they already had.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@slachiewicz slachiewicz changed the title Cap every build workflow job at 30 minutes Cap every build workflow job at 60 minutes Sep 13, 2026
@slachiewicz

Copy link
Copy Markdown
Member Author

Thanks, that was the right check. I extended it to the 296 build.yml runs that concluded success or failure between 2026-08-25 and 2026-09-13: 16 successful jobs in 14 runs exceeded 30 minutes, all of them 28 to 42 minutes in an apt-get install step, none over 45.1. So 30 would have cost a re-run on about one run in twenty. Pushed 92a9675 with 60 instead, which clears every observed case and still cuts a hang from 360 minutes to 60. Per-job values do not buy anything, since the slow step is the same libboost-all-dev download in nearly every job; trimming that closure is a separate change worth measuring; cross-test runs prebuilt cpp binaries, so it likely needs only the boost runtime libraries.

This comment was created with AI assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants