From e80b975b12ef99af6291472d177d90f0bb30743a Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:07 +0200 Subject: [PATCH 1/7] NO-ISSUE: Use .example TLD instead of .local in hostname test The .local TLD is reserved for mDNS (RFC 6762) and can cause DNS interference with OVN initialization on systems with Avahi or systemd-resolved, contributing to healthcheck timeouts after hostname changes. Co-Authored-By: Claude Opus 4.6 (1M context) pre-commit.check-secrets: ENABLED --- test/suites/standard1/hostname.robot | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/suites/standard1/hostname.robot b/test/suites/standard1/hostname.robot index 661341dabe..2cb792b4e8 100644 --- a/test/suites/standard1/hostname.robot +++ b/test/suites/standard1/hostname.robot @@ -13,7 +13,7 @@ Test Tags restart slow *** Variables *** -${NEW_HOSTNAME} microshift.local +${NEW_HOSTNAME} microshift-test.example ${OLD_HOSTNAME} ${EMPTY} @@ -29,7 +29,7 @@ Verify Local Host Name Should Contain ${hostname} standard Verify Local Host Name Resolution - [Documentation] Verify correct host name resolution through mDNS + [Documentation] Verify MicroShift restarts correctly after a hostname change [Setup] Configure New Hostname Named Deployment Should Be Available router-default timeout=${DEFAULT_WAIT_TIMEOUT} ns=openshift-ingress From f015692ec655fa3a44ea55e780e7344633ddaace Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:09 +0200 Subject: [PATCH 2/7] test: disable journald rate limiting in logging test The "Case Insensitive Log Levels" test fails on ARM when testing TraceAll because klog verbosity 10 generates ~144K journal messages, exceeding journald's default rate limit of 10K messages per 30 seconds. The suppressed messages include the startup config dump line that the test greps for, making the assertion impossible to satisfy. Instead of globally disabling rate limiting in all test VMs, scope the fix to the logging suite: disable rate limiting in suite setup and re-enable it in teardown. The drop-in write and journald restart are wrapped in `bash -c` so the redirection and restart run as root under SSHLibrary's sudo=True (the shell that expands `>` would otherwise run unprivileged), and `mkdir -p` keeps the keyword self-contained rather than relying on kickstart provisioning of /etc/systemd/journald.conf.d. Co-Authored-By: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) pre-commit.check-secrets: ENABLED --- test/resources/systemd.resource | 24 ++++++++++++++++++++++++ test/suites/configuration2/logging.robot | 3 +++ 2 files changed, 27 insertions(+) diff --git a/test/resources/systemd.resource b/test/resources/systemd.resource index 1ab890b014..9293088837 100644 --- a/test/resources/systemd.resource +++ b/test/resources/systemd.resource @@ -85,3 +85,27 @@ Systemctl Daemon Reload ... sudo=True return_stdout=True return_stderr=True return_rc=True Log Many ${stdout} ${stderr} Should Be Equal As Integers 0 ${rc} + +Disable Journal Rate Limiting + [Documentation] Disable journald rate limiting by writing a drop-in that + ... sets RateLimitBurst=0, then restarting the journal service. + ${stdout} ${stderr} ${rc}= Execute Command + ... bash -c "mkdir -p /etc/systemd/journald.conf.d && printf '[Journal]\nRateLimitBurst=0\n' > /etc/systemd/journald.conf.d/disable-ratelimit.conf && systemctl restart systemd-journald" + ... sudo=True + ... return_stdout=True + ... return_stderr=True + ... return_rc=True + Log Many ${stdout} ${stderr} + Should Be Equal As Integers 0 ${rc} + +Enable Journal Rate Limiting + [Documentation] Re-enable default journald rate limiting by removing the + ... drop-in created by Disable Journal Rate Limiting. + ${stdout} ${stderr} ${rc}= Execute Command + ... bash -c "rm -f /etc/systemd/journald.conf.d/disable-ratelimit.conf && systemctl restart systemd-journald" + ... sudo=True + ... return_stdout=True + ... return_stderr=True + ... return_rc=True + Log Many ${stdout} ${stderr} + Should Be Equal As Integers 0 ${rc} diff --git a/test/suites/configuration2/logging.robot b/test/suites/configuration2/logging.robot index 7a404d98a1..963ef4a17b 100644 --- a/test/suites/configuration2/logging.robot +++ b/test/suites/configuration2/logging.robot @@ -4,6 +4,7 @@ Documentation Tests for case-insensitive log level parsing Resource ../../resources/common.resource Resource ../../resources/microshift-config.resource Resource ../../resources/microshift-process.resource +Resource ../../resources/systemd.resource Library ../../resources/journalctl.py Suite Setup Setup @@ -29,10 +30,12 @@ Setup Check Required Env Variables Login MicroShift Host Setup Kubeconfig + Disable Journal Rate Limiting Teardown [Documentation] Test suite teardown Remove Drop In MicroShift Config 10-loglevel + Enable Journal Rate Limiting Restart MicroShift Logout MicroShift Host Remove Kubeconfig From 8241d75f19bacd07ccb968b71676c01ca780ec0f Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:11 +0200 Subject: [PATCH 3/7] fix: wait for namespace deletion to complete in RF teardowns The Remove Namespace keyword ran `oc delete namespace` under Run With Kubeconfig's 300s process timeout with allow_fail defaulting to False. On ARM dual-stack, namespace garbage collection (pod teardown plus finalizers, slowed by OVN reconciliation saturating the CPU) can exceed 300s, so the oc process was killed, returned non-zero, and cascaded into a suite teardown failure that retroactively marked all passing tests as failed. Delete with allow_fail=True (matching the pattern already used across the suites, e.g. standard1/kustomize.robot) so a slow or killed cleanup does not fail the teardown, then poll until the namespace is actually gone. The poll asserts a genuine NotFound rather than any non-zero exit code (which a transient API error could also produce), so a subsequent test can reuse the namespace name without a "being deleted" collision, bounded to 5m. Co-Authored-By: Claude Opus 4.6 (1M context) pre-commit.check-secrets: ENABLED --- test/resources/kubeconfig.resource | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/test/resources/kubeconfig.resource b/test/resources/kubeconfig.resource index af77860d0d..8b997b7c3e 100644 --- a/test/resources/kubeconfig.resource +++ b/test/resources/kubeconfig.resource @@ -92,6 +92,21 @@ Create Random Namespace RETURN test-${rand} Remove Namespace - [Documentation] Removes the given namespace. + [Documentation] Removes the given namespace and waits until it is actually + ... gone so a subsequent test can safely reuse the name. The delete + ... tolerates failure (allow_fail) so a slow deletion killed at the Run + ... With Kubeconfig process timeout under load does not fail the teardown. [Arguments] ${ns} - Run With Kubeconfig oc delete namespace ${ns} + Run With Kubeconfig oc delete namespace ${ns} allow_fail=True + Wait Until Keyword Succeeds 5m 5s Namespace Should Not Exist ${ns} + +Namespace Should Not Exist + [Documentation] Fails unless the namespace is reported NotFound, so that a + ... transient API error (which also exits non-zero) is not mistaken for a + ... successful deletion. + [Arguments] ${ns} + ${stdout} ${rc}= Run With Kubeconfig oc get namespace ${ns} + ... allow_fail=True return_rc=True + Should Not Be Equal As Integers ${rc} 0 Namespace ${ns} still exists + Should Contain ${stdout} NotFound + ... msg=oc get namespace ${ns} failed for a reason other than NotFound: ${stdout} From 214dbdd7e5db6737d534b50a851834774ae9076a Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:14 +0200 Subject: [PATCH 4/7] test: centralize release scenario timeouts Release scenarios running upgrade paths with LVMS workloads followed by full standard suites were hitting timeout limits under I/O contention when many VMs boot and pull images in parallel. Set the greenboot healthcheck timeout to 1200s (from 600s) and the robot framework timeout to 60m centrally in ci_phase_boot_and_test.sh for all release scenarios, and remove the now-redundant per-scenario overrides so the value lives in a single place. Co-Authored-By: Claude Opus 4.6 (1M context) pre-commit.check-secrets: ENABLED --- test/bin/ci_phase_boot_and_test.sh | 9 ++++++--- .../el10/releases/el102-lrel@optional-sigstore.sh | 9 --------- .../scenarios-bootc/el10/releases/el102-lrel@optional.sh | 9 --------- .../el10/releases/el102-y1@el102-lrel@lvms-standard.sh | 3 --- .../el10/releases/el102-y1@el102-lrel@rpm-standard.sh | 2 -- test/scenarios-bootc/el10/releases/el102@rpm-standard.sh | 3 --- .../el10/releases/el98-y1@el102-lrel@lvms-standard.sh | 3 --- .../el10/releases/el98-y2@el102-lrel@lvms-standard.sh | 3 --- .../el9/releases/el98-lrel@optional-sigstore.sh | 9 --------- test/scenarios-bootc/el9/releases/el98-lrel@optional.sh | 9 --------- 10 files changed, 6 insertions(+), 53 deletions(-) diff --git a/test/bin/ci_phase_boot_and_test.sh b/test/bin/ci_phase_boot_and_test.sh index 79bd22e8c6..8bcc995222 100755 --- a/test/bin/ci_phase_boot_and_test.sh +++ b/test/bin/ci_phase_boot_and_test.sh @@ -116,9 +116,12 @@ elif [[ "${SCENARIO_SOURCES}" =~ .*releases.* ]]; then # hypervisor only ever holds the still-running scenarios' VMs. jobs_arg="-j 20" scenario_action="create-run-shutdown" - - # Give release scenarios a longer per-scenario execution timeout since - # many now run both the standard1 and standard2 suites. + # Release scenarios run upgrade paths with LVMS workloads followed by + # full standard suites (many now run both standard1 and standard2), which + # need more time than the default 30m robot timeout and 600s greenboot + # healthcheck — especially under I/O contention when many VMs boot and + # pull images in parallel. + export GREENBOOT_TIMEOUT=1200 export TEST_EXECUTION_TIMEOUT=60m fi diff --git a/test/scenarios-bootc/el10/releases/el102-lrel@optional-sigstore.sh b/test/scenarios-bootc/el10/releases/el102-lrel@optional-sigstore.sh index e657720bb7..fd25080fd0 100644 --- a/test/scenarios-bootc/el10/releases/el102-lrel@optional-sigstore.sh +++ b/test/scenarios-bootc/el10/releases/el102-lrel@optional-sigstore.sh @@ -2,15 +2,6 @@ # Sourced from scenario.sh and uses functions defined there. -# Each optional suite restarts MicroShift with its own kustomizePaths config, -# adding ~10 minutes of restart overhead to the total execution time. -# shellcheck disable=SC2034 # used elsewhere -TEST_EXECUTION_TIMEOUT=60m - -# shellcheck disable=SC2034 # used elsewhere -# Increase greenboot timeout for optional packages (more services to start) -GREENBOOT_TIMEOUT=1200 - # Enable container signature verification for current release images, # including the optional components. # These are ec / rc / z-stream, thus must all to be signed. diff --git a/test/scenarios-bootc/el10/releases/el102-lrel@optional.sh b/test/scenarios-bootc/el10/releases/el102-lrel@optional.sh index 9a62cfccff..a17d90753d 100644 --- a/test/scenarios-bootc/el10/releases/el102-lrel@optional.sh +++ b/test/scenarios-bootc/el10/releases/el102-lrel@optional.sh @@ -2,15 +2,6 @@ # Sourced from scenario.sh and uses functions defined there. -# Each optional suite restarts MicroShift with its own kustomizePaths config, -# adding ~10 minutes of restart overhead to the total execution time. -# shellcheck disable=SC2034 # used elsewhere -TEST_EXECUTION_TIMEOUT=60m - -# shellcheck disable=SC2034 # used elsewhere -# Increase greenboot timeout for optional packages (more services to start) -GREENBOOT_TIMEOUT=1200 - # Redefine network-related settings to use the dedicated network bridge VM_BRIDGE_IP="$(get_vm_bridge_ip "${VM_MULTUS_NETWORK}")" # shellcheck disable=SC2034 # used elsewhere diff --git a/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@lvms-standard.sh b/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@lvms-standard.sh index 1dcdccbad0..420a560401 100644 --- a/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@lvms-standard.sh +++ b/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@lvms-standard.sh @@ -6,9 +6,6 @@ # ensure MicroShift is upgraded before running validation tests export TEST_RANDOMIZATION=none -# Add extra timeout because it runs both standard1 and standard2 suites. -export TEST_EXECUTION_TIMEOUT=60m - start_image="rhel102-bootc-brew-y1-with-optional" dest_image="rhel102-bootc-brew-lrel-optional" diff --git a/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@rpm-standard.sh b/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@rpm-standard.sh index aaf86293ee..244e80bb70 100644 --- a/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@rpm-standard.sh +++ b/test/scenarios-bootc/el10/releases/el102-y1@el102-lrel@rpm-standard.sh @@ -13,8 +13,6 @@ export SKIP_GREENBOOT=true # did not want to spend the resources on a new VM. export TEST_RANDOMIZATION=none -export TEST_EXECUTION_TIMEOUT=60m - scenario_create_vms() { exit_if_brew_rpms_not_found diff --git a/test/scenarios-bootc/el10/releases/el102@rpm-standard.sh b/test/scenarios-bootc/el10/releases/el102@rpm-standard.sh index 816814dbfb..aaa0b7da23 100644 --- a/test/scenarios-bootc/el10/releases/el102@rpm-standard.sh +++ b/test/scenarios-bootc/el10/releases/el102@rpm-standard.sh @@ -13,9 +13,6 @@ export SKIP_GREENBOOT=true # did not want to spend the resources on a new VM. export TEST_RANDOMIZATION=none -# Add extra timeout because it run both standard1 and standard2 suites. -export TEST_EXECUTION_TIMEOUT=60m - scenario_create_vms() { exit_if_brew_rpms_not_found diff --git a/test/scenarios-bootc/el10/releases/el98-y1@el102-lrel@lvms-standard.sh b/test/scenarios-bootc/el10/releases/el98-y1@el102-lrel@lvms-standard.sh index 0891c22b2a..6723a273ee 100644 --- a/test/scenarios-bootc/el10/releases/el98-y1@el102-lrel@lvms-standard.sh +++ b/test/scenarios-bootc/el10/releases/el98-y1@el102-lrel@lvms-standard.sh @@ -6,9 +6,6 @@ # ensure MicroShift is upgraded before running validation tests export TEST_RANDOMIZATION=none -# Add extra timeout because it runs both standard1 and standard2 suites. -export TEST_EXECUTION_TIMEOUT=60m - start_image="rhel98-bootc-brew-y1-with-optional" dest_image="rhel102-bootc-brew-lrel-optional" diff --git a/test/scenarios-bootc/el10/releases/el98-y2@el102-lrel@lvms-standard.sh b/test/scenarios-bootc/el10/releases/el98-y2@el102-lrel@lvms-standard.sh index 46da7aaff9..492e2c47fc 100644 --- a/test/scenarios-bootc/el10/releases/el98-y2@el102-lrel@lvms-standard.sh +++ b/test/scenarios-bootc/el10/releases/el98-y2@el102-lrel@lvms-standard.sh @@ -6,9 +6,6 @@ # ensure MicroShift is upgraded before running validation tests export TEST_RANDOMIZATION=none -# Add extra timeout because it runs both standard1 and standard2 suites. -export TEST_EXECUTION_TIMEOUT=60m - start_image="rhel98-bootc-brew-y2-with-optional" dest_image="rhel102-bootc-brew-lrel-optional" diff --git a/test/scenarios-bootc/el9/releases/el98-lrel@optional-sigstore.sh b/test/scenarios-bootc/el9/releases/el98-lrel@optional-sigstore.sh index fc078e23c9..e13d318a43 100644 --- a/test/scenarios-bootc/el9/releases/el98-lrel@optional-sigstore.sh +++ b/test/scenarios-bootc/el9/releases/el98-lrel@optional-sigstore.sh @@ -2,15 +2,6 @@ # Sourced from scenario.sh and uses functions defined there. -# Each optional suite restarts MicroShift with its own kustomizePaths config, -# adding ~10 minutes of restart overhead to the total execution time. -# shellcheck disable=SC2034 # used elsewhere -TEST_EXECUTION_TIMEOUT=60m - -# shellcheck disable=SC2034 # used elsewhere -# Increase greenboot timeout for optional packages (more services to start) -GREENBOOT_TIMEOUT=1200 - # Enable container signature verification for current release images, # including the optional components. # These are ec / rc / z-stream, thus must all to be signed. diff --git a/test/scenarios-bootc/el9/releases/el98-lrel@optional.sh b/test/scenarios-bootc/el9/releases/el98-lrel@optional.sh index a4d3782b8e..2d271f649d 100644 --- a/test/scenarios-bootc/el9/releases/el98-lrel@optional.sh +++ b/test/scenarios-bootc/el9/releases/el98-lrel@optional.sh @@ -2,15 +2,6 @@ # Sourced from scenario.sh and uses functions defined there. -# Each optional suite restarts MicroShift with its own kustomizePaths config, -# adding ~10 minutes of restart overhead to the total execution time. -# shellcheck disable=SC2034 # used elsewhere -TEST_EXECUTION_TIMEOUT=60m - -# shellcheck disable=SC2034 # used elsewhere -# Increase greenboot timeout for optional packages (more services to start) -GREENBOOT_TIMEOUT=1200 - # Redefine network-related settings to use the dedicated network bridge VM_BRIDGE_IP="$(get_vm_bridge_ip "${VM_MULTUS_NETWORK}")" # shellcheck disable=SC2034 # used elsewhere From 51e2816afc8daac71f1adf171bca31003b75f5bb Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:16 +0200 Subject: [PATCH 5/7] fix: restore ginkgo VM disk to 30GB to avoid VG exhaustion #7298 reduced release-scenario VM disks from 30GB to 20GB. That is safe for the lvms-standard/standard scenarios (they create a single 1Gi PVC), but the ginkgo scenario runs the full storage spec suite which requests several 1Gi PVCs concurrently. At 20GB the topolvm data VG only has ~420MiB free, so 7 storage specs fail with: ResourceExhausted ... no enough space left on VG: free=440401920, requested=1073741824 (arm-el10) and on x86 el10 the same undersized VM shows etcd ReadIndex latency and apiserver TLS-handshake flaps from I/O contention. Restore only this scenario to --vm_disksize 30; the other nine 20GB scenarios stay as-is since a single 1Gi PVC fits comfortably. Co-Authored-By: Claude Opus 4.8 (1M context) pre-commit.check-secrets: ENABLED --- test/scenarios-bootc/el10/releases/el102-lrel@ginkgo-tests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/scenarios-bootc/el10/releases/el102-lrel@ginkgo-tests.sh b/test/scenarios-bootc/el10/releases/el102-lrel@ginkgo-tests.sh index 1d37084ffd..34c16bd3a0 100644 --- a/test/scenarios-bootc/el10/releases/el102-lrel@ginkgo-tests.sh +++ b/test/scenarios-bootc/el10/releases/el102-lrel@ginkgo-tests.sh @@ -11,7 +11,7 @@ scenario_create_vms() { fi prepare_kickstart host1 kickstart-bootc.ks.template "${start_image}" - launch_vm rhel102-bootc --vm_vcpus 4 + launch_vm rhel102-bootc --vm_disksize 30 --vm_vcpus 4 } scenario_remove_vms() { From c81071ebe005334e0c07fa24bb697c17a7bb8bd1 Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 12:57:19 +0200 Subject: [PATCH 6/7] fix: allowlist benign cert-manager SA race in log-scan forbidden check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On a fresh/clean start the standard2 Log Scan test intermittently fails "Should Not Find Forbidden" with: pods cert-manager-cainjector-... is forbidden: error looking up service account cert-manager/cert-manager-cainjector: serviceaccount ... not found Investigation: the cainjector/controller/webhook Deployments and their ServiceAccounts are created dynamically by the cert-manager operator, not by MicroShift's static manifests. The kube-controller-manager ReplicaSet controller can briefly attempt to create a pod before its ServiceAccount is observed, logging this transient "forbidden" and retrying it away once the SA lands. The workloads become ready (MicroShift healthcheck passes), so this is a benign eventual-consistency startup race, not a MicroShift manifest-ordering bug — the ordering is the operator's, not ours. Add a scoped known-exceptions allowlist to the journalctl log-scan helper and register this single pattern, so genuine "forbidden" regressions still fail while this benign race is ignored. While here, make the test itself easier to read: rename the per-boot keyword to "Boot And Scan Journal", move the repeated journal-cursor capture into it, and split the assertions into "Scan Boot Journal". The two calls now read as "clean first boot" (forbidden check skipped, since a clean boot logs the benign race above) and "restart" (must be forbidden-free). Co-Authored-By: Claude Opus 4.8 (1M context) pre-commit.check-secrets: ENABLED --- test/resources/journalctl.py | 36 ++++++++++++++++--- test/suites/standard2/log-scan.robot | 53 +++++++++++++++++++--------- 2 files changed, 68 insertions(+), 21 deletions(-) diff --git a/test/resources/journalctl.py b/test/resources/journalctl.py index d5fa2663b9..e2cb1e3879 100644 --- a/test/resources/journalctl.py +++ b/test/resources/journalctl.py @@ -1,6 +1,7 @@ from robot.libraries.BuiltIn import BuiltIn import libostree +import re import time _log = BuiltIn().log @@ -27,28 +28,55 @@ def get_journal_cursor(unit="microshift") -> str: return cursor -def get_log_output_with_pattern(cursor: str, pattern: str, unit="microshift") -> tuple[str, int]: +def get_log_output_with_pattern(cursor: str, pattern: str, unit="microshift", exceptions=None) -> tuple[str, int]: """ Get the logs since the cursor matching the pattern and return the log content and exit code. Optional argument `unit` may be used to specify a systemd unit other than microshift, for example microshift-observability.service. Note that this function ignores case when matching the pattern. + + Optional argument `exceptions` is a list of regular expressions describing + known-benign log lines to ignore. Matching lines are dropped from the + result; if every matched line is an exception, the return code is + downgraded to 1 (no relevant match) so callers treat it as "not found". """ stdout, rc = libostree.remote_sudo_rc( f"journalctl -u {unit} --cursor='{cursor}' --no-pager --case-sensitive=false --grep '{pattern}'" ) + if exceptions and rc == 0: + matched = [line for line in stdout.splitlines() if line.strip()] + remaining = [ + line for line in matched + if not any(re.search(exc, line, re.IGNORECASE) for exc in exceptions) + ] + dropped = len(matched) - len(remaining) + if dropped: + # Make the suppression visible: a benign exception that turns + # persistent (e.g. a ServiceAccount that never lands) would + # otherwise be masked silently. + BuiltIn().log( + f"Ignored {dropped} known-benign '{pattern}' line(s) via exceptions", "WARN" + ) + if not remaining: + # All matches were known-benign exceptions; report no relevant match. + rc = 1 + stdout = "\n".join(remaining) BuiltIn().log(f"log lines matching '{pattern}':\n{stdout}") return stdout, rc -def pattern_should_not_appear_in_log_output(cursor, pattern, unit="microshift", retries=30, wait=10): - """Get the logs since the cursor and verify that the pattern does not appear.""" +def pattern_should_not_appear_in_log_output(cursor, pattern, unit="microshift", retries=30, wait=10, exceptions=None): + """Get the logs since the cursor and verify that the pattern does not appear. + + Optional argument `exceptions` is a list of regular expressions describing + known-benign log lines that should not count as a match. + """ # The grep argument causes journalctl to exit with an error if the # pattern is not found, therefore we want the return code to be 1, # indicating that there was no match. for attempt in range(1, retries + 2): - stdout, rc = get_log_output_with_pattern(cursor, pattern, unit) + stdout, rc = get_log_output_with_pattern(cursor, pattern, unit, exceptions) if rc == 1 or attempt > retries: BuiltIn().should_be_equal_as_integers(rc, 1) return diff --git a/test/suites/standard2/log-scan.robot b/test/suites/standard2/log-scan.robot index 1631f00d1e..9cc8b864d3 100644 --- a/test/suites/standard2/log-scan.robot +++ b/test/suites/standard2/log-scan.robot @@ -13,25 +13,34 @@ Test Tags restart slow *** Variables *** -${CURSOR} ${EMPTY} # The journal cursor before restarting MicroShift +${CURSOR} ${EMPTY} # Journal cursor for the current boot; set by Boot And Scan Journal + +# Known-benign "forbidden" log lines to ignore during the log scan. +# On a fresh/clean start the cert-manager operator creates the cainjector, +# controller and webhook Deployments and their ServiceAccounts dynamically. +# The kube-controller-manager ReplicaSet controller can briefly try to create +# a pod before the matching ServiceAccount is observed, logging a transient +# "forbidden: error looking up service account" that it retries away once the +# SA lands. These pods are operator-managed (not MicroShift manifests), so the +# ordering is not under MicroShift's control and this is a benign startup race. +@{FORBIDDEN_EXCEPTIONS} +... is forbidden: error looking up service account cert-manager/[^ ]+: serviceaccount .* not found *** Test Cases *** Log Scan - [Documentation] Run log scan tests in a specific order. - # Clean up and enable MicroShift to start from scratch + [Documentation] Scan the journal of a clean first boot and then of a + ... restart. Both boots are checked, but the "forbidden" check runs only + ... on the restart: a clean first boot logs a benign "forbidden" while + ... components initialize (see ${FORBIDDEN_EXCEPTIONS}), whereas a restart + ... must be free of it. Cleanup MicroShift --all --keep-images Enable MicroShift - ${cursor}= Get Journal Cursor - VAR ${CURSOR}= ${cursor} scope=SUITE - # Start, stop and check logs after clean startup - Start Stop And Check Logs check_forbidden=False - - ${cursor}= Get Journal Cursor - VAR ${CURSOR}= ${cursor} scope=SUITE - # Restart, stop and check logs - Start Stop And Check Logs + # Clean first boot: skip the forbidden check. + Boot And Scan Journal check_forbidden=False + # Restart: forbidden messages must not reappear. + Boot And Scan Journal *** Keywords *** @@ -48,16 +57,25 @@ Teardown Logout MicroShift Host Remove Kubeconfig -Start Stop And Check Logs - [Documentation] Start, wait until initialized, stop and check for errors. +Boot And Scan Journal + [Documentation] Record the journal cursor, start MicroShift, wait until it + ... is initialized, stop it, and scan this boot's journal for wanted and + ... unwanted messages. [Arguments] ${check_forbidden}=True + ${cursor}= Get Journal Cursor + VAR ${CURSOR}= ${cursor} scope=SUITE Start MicroShift Setup Kubeconfig Wait For MicroShift Healthcheck Success Stop MicroShift - # Note: The 'forbidden' messages appear on clean startup + Scan Boot Journal check_forbidden=${check_forbidden} + +Scan Boot Journal + [Documentation] Assert this boot's journal contains the expected readiness + ... messages and none of the unwanted ones. + [Arguments] ${check_forbidden}=True IF ${check_forbidden} Should Not Find Forbidden Should Not Find Cannot Patch Resource Services Should Not Timeout When Stopping @@ -65,8 +83,9 @@ Start Stop And Check Logs Should Find MicroShift Is Ready Should Not Find Forbidden - [Documentation] Logs should not say "forbidden" - Pattern Should Not Appear In Log Output ${CURSOR} forbidden + [Documentation] Logs should not say "forbidden", excluding known-benign + ... startup races listed in ${FORBIDDEN_EXCEPTIONS}. + Pattern Should Not Appear In Log Output ${CURSOR} forbidden exceptions=${FORBIDDEN_EXCEPTIONS} Should Not Find Cannot Patch Resource [Documentation] Logs should not say "cannot patch resource" From 3a3c241a39992db3ea9179529940295c2d79e0f3 Mon Sep 17 00:00:00 2001 From: Alejandro Gullon Date: Tue, 8 Sep 2026 22:07:01 +0200 Subject: [PATCH 7/7] test: harden flaky statefulset and kserve waits Two release-scenario tests raced asynchronous resource creation: - otp-workloads/statefulset-pvc: `oc wait pod/hello-statefulset-0 --for=condition=Ready` ran immediately after creating the StatefulSet, before its controller created pod-0, so `oc wait` on the named pod failed with NotFound. Wait for the pod to exist first (reusing Wait Until Resource Exists) before checking readiness. - ai-model-serving/ai-model-serving-online: the ServingRuntime was applied with a bare `oc apply` before the kserve validating webhook had endpoints, so the apply was rejected ("no endpoints available for service kserve-webhook-server-service"). Retry the apply until the webhook is serving, mirroring the retry already used for the InferenceService rollout. Co-Authored-By: Claude Opus 4.8 (1M context) pre-commit.check-secrets: ENABLED --- test/suites/ai-model-serving/ai-model-serving-online.robot | 5 ++++- test/suites/otp-workloads/statefulset-pvc.robot | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/test/suites/ai-model-serving/ai-model-serving-online.robot b/test/suites/ai-model-serving/ai-model-serving-online.robot index 36f0fe9202..013350e081 100644 --- a/test/suites/ai-model-serving/ai-model-serving-online.robot +++ b/test/suites/ai-model-serving/ai-model-serving-online.robot @@ -48,7 +48,10 @@ Deploy OpenVINO Serving Runtime ... /usr/lib/microshift/manifests.d/050-microshift-ai-model-serving-runtimes/ovms-kserve.yaml ... ${OVMS_KSERVE_MANIFEST} Local Command Should Work sed -i "s,image: ovms-image,image: ${ovms_image}," "${OVMS_KSERVE_MANIFEST}" - Oc Apply -n ${NAMESPACE} -f ${OVMS_KSERVE_MANIFEST} + # Retry until the kserve validating webhook has endpoints; a bare apply + # fails if the webhook pod is not serving yet. + Wait Until Keyword Succeeds 15x 2s + ... Oc Apply -n ${NAMESPACE} -f ${OVMS_KSERVE_MANIFEST} Deploy OpenVINO Resnet Model [Documentation] Deploys InferenceService object to create Deployment and Service to serve the model. diff --git a/test/suites/otp-workloads/statefulset-pvc.robot b/test/suites/otp-workloads/statefulset-pvc.robot index 0b08d05bd8..2cae706d9a 100644 --- a/test/suites/otp-workloads/statefulset-pvc.robot +++ b/test/suites/otp-workloads/statefulset-pvc.robot @@ -24,6 +24,10 @@ Custom Label For PVC In StatefulSets ... OCP-28018 [Setup] Create StatefulSet Resources + # The StatefulSet controller creates pod-0 asynchronously; wait for it to + # exist before checking readiness, since `oc wait` on a named pod fails + # immediately if the pod is not created yet. + Wait Until Resource Exists pod ${POD_NAME} ns=${NAMESPACE} Named Pod Should Be Ready ${POD_NAME} ns=${NAMESPACE} timeout=5m Wait Until Keyword Succeeds 60s 5s ... PVC Should Have Label ${PVC_NAME} ${NAMESPACE} app hello-pod