Add optional s390x e2e CI for ClusterResourceOverride operator - #83095
Add optional s390x e2e CI for ClusterResourceOverride operator#83095vinayakray19 wants to merge 12 commits into
Conversation
Introduce a separate main__s390x variant that provisions libvirt-s390x-vpn-oz on OCP 5.0.0-ec.5, installs the operator via OLM (optional IDMS), creates the CR, and runs a placeholder test step. Co-authored-by: Cursor <cursoragent@cursor.com>
Use the pipeline src image so the ClusterResourceOverride Go e2e suite runs against the OLM-installed operator, skipping upgrade-only tests by default. Co-authored-by: Cursor <cursoragent@cursor.com>
After OLM install, patch the CSV and Deployment to use the s390x quay.io images so the job exercises those builds without a custom pull secret. Co-authored-by: Cursor <cursoragent@cursor.com>
openshift-ci
Bot
added
the
needs-ok-to-test
|
Hi @vinayakray19. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds s390x CI support for the ClusterResourceOverride admission operator. The change defines build configuration, IBM Z workflow execution, OLM installation, image mirroring and overrides, e2e testing, diagnostics, and cleanup. Changess390x ClusterResourceOverride CI
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CI as s390x CI configuration
participant Workflow as cluster-resource-override-s390x workflow
participant Installer as install-operator step
participant Tests as s390x e2e runner
participant Cleanup as s390x cleanup step
CI->>Workflow: Starts IBM Z UPI/libvirt workflow
Workflow->>Installer: Runs operator installation
Installer->>Tests: Provides the configured operator namespace
Tests->>Cleanup: Completes the test phase
Cleanup->>Cleanup: Deletes test resources and namespace
Suggested labels: Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: vinayakray19 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/ok-to-test |
openshift-ci
Bot
added
ok-to-test
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-commands.sh`:
- Line 11: Update the ImageDigestMirrorSet deletion command using CRO_IDMS_NAME
to include the same --timeout=60s option as the adjacent oc delete commands,
while preserving --ignore-not-found and the existing failure-tolerant behavior.
In
`@ci-operator/step-registry/cluster-resource-override/s390x/cluster-resource-override-s390x-workflow.yaml`:
- Around line 13-16: Reorder the post steps so the `gather` chain runs before
`cluster-resource-override-s390x-cleanup`, preserving
`ipi-conf-debug-kdump-gather-logs` as appropriate. Place
`cluster-resource-override-s390x-cleanup` after diagnostics and immediately
before `upi-libvirt-cleanup-post`.
In
`@ci-operator/step-registry/cluster-resource-override/s390x/run-tests/cluster-resource-override-s390x-run-tests-commands.sh`:
- Line 18: Update the KUBECTL setup in the test script to resolve oc into a
separate assignment before exporting it, so a failed lookup is not masked and
the script cannot continue with an empty value passed to make e2e.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 17c69068-e59b-411f-9ec3-831f52ed1602
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/cluster-resource-override-admission-operator/openshift-cluster-resource-override-admission-operator-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (20)
ci-operator/config/openshift/cluster-resource-override-admission-operator/openshift-cluster-resource-override-admission-operator-main__s390x.yamlci-operator/step-registry/cluster-resource-override/OWNERSci-operator/step-registry/cluster-resource-override/s390x/OWNERSci-operator/step-registry/cluster-resource-override/s390x/cleanup/OWNERSci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-commands.shci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-ref.metadata.jsonci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-ref.yamlci-operator/step-registry/cluster-resource-override/s390x/cluster-resource-override-s390x-workflow.metadata.jsonci-operator/step-registry/cluster-resource-override/s390x/cluster-resource-override-s390x-workflow.yamlci-operator/step-registry/cluster-resource-override/s390x/install-operator/OWNERSci-operator/step-registry/cluster-resource-override/s390x/install-operator/cluster-resource-override-s390x-install-operator-commands.shci-operator/step-registry/cluster-resource-override/s390x/install-operator/cluster-resource-override-s390x-install-operator-ref.metadata.jsonci-operator/step-registry/cluster-resource-override/s390x/install-operator/cluster-resource-override-s390x-install-operator-ref.yamlci-operator/step-registry/cluster-resource-override/s390x/run-tests/OWNERSci-operator/step-registry/cluster-resource-override/s390x/run-tests/cluster-resource-override-s390x-run-tests-commands.shci-operator/step-registry/cluster-resource-override/s390x/run-tests/cluster-resource-override-s390x-run-tests-ref.metadata.jsonci-operator/step-registry/cluster-resource-override/s390x/run-tests/cluster-resource-override-s390x-run-tests-ref.yamlci-operator/step-registry/cluster-resource-override/s390x/test/OWNERSci-operator/step-registry/cluster-resource-override/s390x/test/cluster-resource-override-s390x-test-chain.metadata.jsonci-operator/step-registry/cluster-resource-override/s390x/test/cluster-resource-override-s390x-test-chain.yaml
| oc delete subscription "${CRO_SUBSCRIPTION_NAME}" -n "${CRO_NAMESPACE}" --ignore-not-found --timeout=60s || true | ||
| oc delete csv --all -n "${CRO_NAMESPACE}" --ignore-not-found --timeout=60s || true | ||
| oc delete operatorgroup --all -n "${CRO_NAMESPACE}" --ignore-not-found --timeout=60s || true | ||
| oc delete imagedigestmirrorset "${CRO_IDMS_NAME}" --ignore-not-found || true |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="ci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-commands.sh"
echo "== file exists and relevant lines =="
if [ -f "$file" ]; then
nl -ba "$file" | sed -n '1,80p'
else
echo "missing: $file"
fi
echo
echo "== nearby oc delete timeout patterns in same directory =="
rg -n --fixed-strings 'oc delete|timeout=' ci-operator/step-registry/cluster-resource-override/s390x/cleanup || true
echo
echo "== oc availability/version if present =="
command -v oc || true
if command -v oc >/dev/null 2>&1; then
oc version --client >/dev/null 2>&1 && oc version --client || true
oc delete --help 2>/dev/null | rg -n 'Usage:|\s--timeout|Timeout period|0s|Default' | head -30 || true
fi
echo
echo "== any imagedigestmirrorset delete commands in repo =="
rg -n 'imagedigestmirrorset|ImageDigestMirrorSet' .Repository: openshift/release
Length of output: 232
🌐 Web query:
OpenShift oc delete timeout flag documentation
💡 Result:
In the OpenShift command-line interface (oc), the --timeout flag is used with the oc delete command to specify the length of time to wait before giving up on a deletion operation [1][2]. Key details regarding the --timeout flag: - Function: It defines the maximum duration the client will wait for the deletion process to complete [1][2]. - Default Behavior: If set to 0 (which is the default), the system determines the timeout based on the size of the object being deleted [1][2]. - Configuration: When specified, it accepts time durations (e.g., 30s, 5m). Additionally, the oc delete command supports other related flags for managing deletion behavior: - --grace-period: Defines the number of seconds allowed for the resource to terminate gracefully [2]. - --force: When combined with --grace-period=0, this allows for immediate removal of resources, bypassing standard graceful deletion processes [3][2]. - --request-timeout: A separate flag that specifies the time to wait before giving up on a single server request [1][2]. For further details on specific command usage, you can run oc delete --help in your terminal, which provides the most accurate and context-specific documentation for your version of the OpenShift CLI.
Citations:
- 1: https://manpages.opensuse.org/Tumbleweed/oc/oc-delete.1.en.html
- 2: https://manpages.opensuse.org/Tumbleweed/oc/oc-replace.1.en.html
- 3: https://docs.okd.io/latest/cli_reference/openshift_cli/developer-cli-commands.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="ci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-commands.sh"
echo "== file exists =="
ls -l "$file" || true
echo
echo "== relevant lines =="
sed -n '1,80p' "$file"
echo
echo "== oc delete timeout patterns in same directory =="
grep -n -E 'oc delete|timeout=' "$file" | sed -n '1,80p' || true
echo
echo "== imagedigestmirrorset references in repo =="
grep -RIn -E 'imagedigestmirrorset|ImageDigestMirrorSet' . | sed -n '1,120p' || trueRepository: openshift/release
Length of output: 20739
Bound the ImageDigestMirrorSet delete.
The adjacent oc delete commands use --timeout=60s; this one does not. A stalled deletion can leave the cleanup step blocked until the step timeout instead of returning quickly. Add the same timeout pattern.
Proposed fix
-oc delete imagedigestmirrorset "${CRO_IDMS_NAME}" --ignore-not-found || true
+oc delete imagedigestmirrorset "${CRO_IDMS_NAME}" --ignore-not-found --timeout=60s || true📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| oc delete imagedigestmirrorset "${CRO_IDMS_NAME}" --ignore-not-found || true | |
| oc delete imagedigestmirrorset "${CRO_IDMS_NAME}" --ignore-not-found --timeout=60s || true |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/cluster-resource-override/s390x/cleanup/cluster-resource-override-s390x-cleanup-commands.sh`
at line 11, Update the ImageDigestMirrorSet deletion command using CRO_IDMS_NAME
to include the same --timeout=60s option as the adjacent oc delete commands,
while preserving --ignore-not-found and the existing failure-tolerant behavior.
Source: MCP tools
| post: | ||
| - ref: cluster-resource-override-s390x-cleanup | ||
| - ref: ipi-conf-debug-kdump-gather-logs | ||
| - chain: gather |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Run diagnostics before component cleanup.
cluster-resource-override-s390x-cleanup deletes ${CRO_NAMESPACE} before gather runs. A failed install or test then loses the operator Pods, events, and logs that gather must collect.
Move cluster-resource-override-s390x-cleanup after gather and before upi-libvirt-cleanup-post.
Based on the downstream cleanup snippet, the cleanup step deletes the operator namespace before the workflow reaches gather.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/cluster-resource-override/s390x/cluster-resource-override-s390x-workflow.yaml`
around lines 13 - 16, Reorder the post steps so the `gather` chain runs before
`cluster-resource-override-s390x-cleanup`, preserving
`ipi-conf-debug-kdump-gather-logs` as appropriate. Place
`cluster-resource-override-s390x-cleanup` after diagnostics and immediately
before `upi-libvirt-cleanup-post`.
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Match the Kuadrant s390x release pin so install pulls latest 4.22 s390x nightlies instead of 5.0.0-ec.5. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
build07 lacks VPN reachability to the IBM OZ libvirt pool; match other libvirt-s390x-vpn jobs on build12. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Add the prowgen job-release label, strip metadata EOF newlines, and split KUBECTL export so required release PR checks and rehearse can proceed. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: your |
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Return the libvirt job to the EC.5 candidate release and capture deployment/pod/event diagnostics when catalog or image-patch rollouts fail. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Wire cro-operator/cro-operand from ocp-s390x:5.0 into the install step via dependencies so e2e patches ART s390x images after OLM subscribe. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
5.0 ocp-s390x CRO tags are missing; keep the cluster on 5.0.0-ec.5 and source operator/operand images from the 4.22 s390x ART tags instead. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop ocp-s390x cro-operator/cro-operand base images; leave image overrides optional so the job uses redhat-operators CSV images on 5.0.0-ec.5. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep catalog subscribe on 5.0.0-ec.5, then patch operator/operand to the public tdudhade quay builds for the rollout under test. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: |
1 similar comment
|
@vinayakray19: |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Use the multi/s390x nightly candidate streams instead of the 5.0.0-ec.5 release pin. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2e |
|
@vinayakray19: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@vinayakray19: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary
main__s390xci-operator variant foropenshift/cluster-resource-override-admission-operatorwith an on-demand/test s390x-e2ejob (optional: true,always_run: false).libvirt-s390x-vpn-ozusing the existing UPI/libvirt VPN workflow steps.quay.io/tdudhade/...:0.0.5); no custom pull secret required.make e2e), skipping upgrade-only tests by default, then cleans up.Test plan
/pj-rehearse pull-ci-openshift-cluster-resource-override-admission-operator-main-s390x-e2emake e2epasses (upgrade tests skipped)Made with Cursor
Summary by CodeRabbit
Adds an optional
main__s390xCI variant for the ClusterResourceOverride admission operator.libvirt-s390x-vpn-oz.ocp-s390x:5.0payload images and supports s390x operator and operand image overrides.make e2ewith upgrade-only tests skipped by default.