Skip to content

NO-ISSUE: Increase greenboot and robot timeouts for release scenarios - #7319

Closed
agullon wants to merge 2 commits into
openshift:mainfrom
agullon:NO-ISSUE-release-timeouts-main
Closed

NO-ISSUE: Increase greenboot and robot timeouts for release scenarios#7319
agullon wants to merge 2 commits into
openshift:mainfrom
agullon:NO-ISSUE-release-timeouts-main

Conversation

@agullon

@agullon agullon commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Increase greenboot healthcheck timeout from 600s (10 min) to 1200s (20 min) for release scenarios
  • Increase Robot Framework test execution timeout from the CI-configured 45m to 60m for release scenarios
  • Both changes are scoped to release scenarios only via ci_phase_boot_and_test.sh

Context

Release scenarios running upgrade paths with LVMS workloads followed by full standard suites
were hitting timeout limits under I/O contention on x86 (c5.metal, 4750 Mbps EBS bandwidth)
when many VMs boot and pull container images from the mirror registry in parallel. The ARM
instance (m7g.metal, 20000 Mbps EBS) has ~4.2x more I/O bandwidth and doesn't hit these limits.

Specific failures observed in PR #7297:

  • el96-y2@el98-lrel@lvms-standard1/2: greenboot healthcheck failed — pods not ready within 600s
  • el98-y1@el98-lrel@lvms-standard: Robot Framework killed by SIGTERM at 45m (44/52 tests had passed)

These timeouts are ceilings, not floors — greenboot polls and exits immediately when pods are ready,
and robot finishes whenever tests complete. The happy-path duration is unaffected.

The el10 lvms-standard scenarios and the optional scenario already set these values per-scenario.
This change centralizes them for all release scenarios in ci_phase_boot_and_test.sh.

Counterpart of #7318 for release-5.0.

Test plan

  • Verify release CI jobs pass with the new timeouts
  • Confirm non-release scenarios (presubmits, periodics, c2cc) are unaffected

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Standardized release-scenario timing by applying a 20-minute system startup allowance and a 60-minute test execution limit.
    • Removed scenario-specific timeout overrides so release tests use the shared timing configuration consistently.

Release scenarios running upgrade paths with LVMS workloads followed
by full standard suites were hitting timeout limits under I/O contention
on x86 (c5.metal, 4750 Mbps EBS) when many VMs boot and pull images in
parallel. Increase greenboot healthcheck timeout from 600s to 1200s and
robot framework timeout from 30m (CI-overridden to 45m) to 60m for
release scenarios only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 8b6f075e-8259-4ced-bff4-a2c1e17e5ae3

📥 Commits

Reviewing files that changed from the base of the PR and between 7e4127e and 722d7fc.

📒 Files selected for processing (5)
  • test/scenarios-bootc/el10/releases/el102-lrel@optional-sigstore.sh
  • test/scenarios-bootc/el10/releases/el102-lrel@optional.sh
  • test/scenarios-bootc/el10/releases/el102@rpm-standard.sh
  • test/scenarios-bootc/el9/releases/el98-lrel@optional-sigstore.sh
  • test/scenarios-bootc/el9/releases/el98-lrel@optional.sh
💤 Files with no reviewable changes (5)

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.


Walkthrough

Release scenarios now receive GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m from the CI phase script. Scenario-specific timeout assignments were removed.

Changes

Release scenario timeout configuration

Layer / File(s) Summary
Centralize release timeouts
test/bin/ci_phase_boot_and_test.sh, test/scenarios-bootc/el10/releases/el102@rpm-standard.sh
The release-scenario branch exports GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m. The RPM-standard scenario no longer sets a local test timeout.
Remove duplicate optional-scenario overrides
test/scenarios-bootc/el10/releases/*, test/scenarios-bootc/el9/releases/*
Optional scenarios no longer define local Greenboot and test execution timeout overrides.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 722d7

Release scenarios receive higher centralized timeout ceilings while non-release scenarios remain unchanged. No current merge-blocking risk is identified.

Suggested labels: ready-for-human-review

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS — The pull request changes only CI/scenario shell scripts. The added lines export GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m; the other changes remove redundant timeout assignments…
Test Structure And Quality ✅ Passed The pull request changes six shell scripts only. The diff from the base commit contains no Go files, Ginkgo It blocks, or Ginkgo assertions. The added lines in test/bin/ci_phase_boot_and_test.sh o…
Microshift Test Compatibility ✅ Passed PASS — The pull request changes six shell scripts only. The diff adds timeout exports and removes redundant scenario overrides; it adds no Ginkgo tests or test cases using OpenShift APIs or unsupporte…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The pull request changes only test/bin/ci_phase_boot_and_test.sh and five release scenario shell scripts. The diff adds timeout exports and removes redundant timeout assignments. It adds no Gi…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only CI/scenario shell scripts. It adds release-scoped timeout environment variables and removes redundant per-scenario timeout overrides. The available commit range con…
Ote Binary Stdout Contract ✅ Passed PASS: The full PR range changes only six shell scenario/CI files. The only additions are GREENBOOT_TIMEOUT and TEST_EXECUTION_TIMEOUT exports; the other changes remove shell variable assignments. …
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request changes only six shell scripts. It adds centralized timeout exports and removes redundant per-scenario assignments. It adds no Ginkgo tests, IPv4-specific logic, or external con…
No-Weak-Crypto ✅ Passed PASS: The pull request adds only GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m in the release branch of ci_phase_boot_and_test.sh, and removes redundant timeout assignments from five rel…
Container-Privileges ✅ Passed PASS. The PR changes only timeout environment variables in test/bin/ci_phase_boot_and_test.sh and removes redundant timeout assignments from release scenario scripts. The PR diff adds no `privileged…
No-Sensitive-Data-In-Logs ✅ Passed PASS: The PR adds only release-scenario timeout exports and explanatory comments, and removes duplicate timeout assignments. The added values are 1200 and 60m; they contain no sensitive data. Exis…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: increasing Greenboot and Robot Framework timeouts for release scenarios. It is concise and specific.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files.

Full details: Stable And Deterministic Test Names

Explanation

PASS — The pull request changes only CI/scenario shell scripts. The added lines export GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m; the other changes remove redundant timeout assignments. No It(), Describe(), Context(), or When() test titles were added or modified, and no dynamic test-name value was introduced.

Full details: Test Structure And Quality

Explanation

The pull request changes six shell scripts only. The diff from the base commit contains no Go files, Ginkgo It blocks, or Ginkgo assertions. The added lines in test/bin/ci_phase_boot_and_test.sh only export release-scenario timeout variables before invoking scenario.sh; they do not introduce setup, cleanup, waits, or assertions. The Ginkgo test-structure check is therefore not applicable.

Full details: Microshift Test Compatibility

Explanation

PASS — The pull request changes six shell scripts only. The diff adds timeout exports and removes redundant scenario overrides; it adds no Ginkgo tests or test cases using OpenShift APIs or unsupported MicroShift features.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request changes only test/bin/ci_phase_boot_and_test.sh and five release scenario shell scripts. The diff adds timeout exports and removes redundant timeout assignments. It adds no Ginkgo e2e tests (It, Describe, Context, or When) and changes no Go test files. The SNO compatibility check is therefore not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request changes only CI/scenario shell scripts. It adds release-scoped timeout environment variables and removes redundant per-scenario timeout overrides. The available commit range contains no deployment manifests, operator/controller code, or topology scheduling constraints such as affinity, node selectors, tolerations, replica logic, topology spread, or PDBs. The topology-aware scheduling check is therefore not applicable.

Full details: Ote Binary Stdout Contract

Explanation

PASS: The full PR range changes only six shell scenario/CI files. The only additions are GREENBOOT_TIMEOUT and TEST_EXECUTION_TIMEOUT exports; the other changes remove shell variable assignments. No OTE Go binary code, main()/suite setup, logging configuration, or new stdout write was introduced. The existing shell echo statements are outside the stated OTE process-level code scope.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS: The pull request changes only six shell scripts. It adds centralized timeout exports and removes redundant per-scenario assignments. It adds no Ginkgo tests, IPv4-specific logic, or external connectivity code, so this check is not applicable.

Full details: No-Weak-Crypto

Explanation

PASS: The pull request adds only GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m in the release branch of ci_phase_boot_and_test.sh, and removes redundant timeout assignments from five release scenario files. The PR diff introduces no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom cryptography, or secret/token comparisons.

Full details: Container-Privileges

Explanation

PASS. The PR changes only timeout environment variables in test/bin/ci_phase_boot_and_test.sh and removes redundant timeout assignments from release scenario scripts. The PR diff adds no privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation settings. It also changes no container or Kubernetes manifest. The existing sudo command is unchanged and does not match the stated container privilege conditions.

Full details: No-Sensitive-Data-In-Logs

Explanation

PASS: The PR adds only release-scenario timeout exports and explanatory comments, and removes duplicate timeout assignments. The added values are 1200 and 60m; they contain no sensitive data. Existing tee and bash -x logging is unchanged. No passwords, tokens, API keys, PII, session IDs, hostnames, or customer data are introduced by the diff.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from eslutsky and jogeo September 3, 2026 08:12
@coderabbitai coderabbitai Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 3, 2026
@openshift-ci

openshift-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agullon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2026
GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m are now set
centrally in ci_phase_boot_and_test.sh for all release scenarios.
Remove the redundant per-scenario overrides.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@agullon agullon changed the title Increase greenboot and robot timeouts for release scenarios NO-ISSUE: Increase greenboot and robot timeouts for release scenarios Sep 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@agullon: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

  • Increase greenboot healthcheck timeout from 600s (10 min) to 1200s (20 min) for release scenarios
  • Increase Robot Framework test execution timeout from the CI-configured 45m to 60m for release scenarios
  • Both changes are scoped to release scenarios only via ci_phase_boot_and_test.sh

Context

Release scenarios running upgrade paths with LVMS workloads followed by full standard suites
were hitting timeout limits under I/O contention on x86 (c5.metal, 4750 Mbps EBS bandwidth)
when many VMs boot and pull container images from the mirror registry in parallel. The ARM
instance (m7g.metal, 20000 Mbps EBS) has ~4.2x more I/O bandwidth and doesn't hit these limits.

Specific failures observed in PR #7297:

  • el96-y2@el98-lrel@lvms-standard1/2: greenboot healthcheck failed — pods not ready within 600s
  • el98-y1@el98-lrel@lvms-standard: Robot Framework killed by SIGTERM at 45m (44/52 tests had passed)

These timeouts are ceilings, not floors — greenboot polls and exits immediately when pods are ready,
and robot finishes whenever tests complete. The happy-path duration is unaffected.

The el10 lvms-standard scenarios and the optional scenario already set these values per-scenario.
This change centralizes them for all release scenarios in ci_phase_boot_and_test.sh.

Counterpart of #7318 for release-5.0.

Test plan

  • Verify release CI jobs pass with the new timeouts
  • Confirm non-release scenarios (presubmits, periodics, c2cc) are unaffected

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
  • Standardized release-scenario timing by applying a 20-minute system startup allowance and a 60-minute test execution limit.
  • Removed scenario-specific timeout overrides so release tests use the shared timing configuration consistently.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 3, 2026
@openshift-ci

openshift-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

@agullon: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@agullon

agullon commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Consolidated into #7326

@agullon agullon closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants