Skip to content

WIP: [Storage] [CCLM] Post-CCLM validation tests, Windows VM tests#3926

Closed
jpeimer wants to merge 7 commits intoRedHatQE:mainfrom
jpeimer:cclm_write_to_remote_vm_add_win_add_post_cclm_validations
Closed

WIP: [Storage] [CCLM] Post-CCLM validation tests, Windows VM tests#3926
jpeimer wants to merge 7 commits intoRedHatQE:mainfrom
jpeimer:cclm_write_to_remote_vm_add_win_add_post_cclm_validations

Conversation

@jpeimer
Copy link
Copy Markdown
Contributor

@jpeimer jpeimer commented Feb 19, 2026

Short description:
  • Add post-CCLM validation tests
  • Add Windows VM tests
More details:
  • CCLM multiple VMs
  • Make storage class configurable
  • Check boot time before and after CCLM for Fedora and RHEL VMs
  • Allow connection to the VM Console in the remote cluster with the kubeconfig

Jira: https://issues.redhat.com/browse/CNV-60016

What this PR does / why we need it:
Which issue(s) this PR fixes:
Special notes for reviewer:

Assisted by Claude AI and Cursor AI

jira-ticket:

Summary by CodeRabbit

  • Tests

    • Expanded cross-cluster live-migration coverage: multi‑VM and Windows VTPM flows, snapshots/restores, file persistence checks, boot-time verification, and fixtures for remote cluster provisioning and mapping.
  • Refactor

    • Consolidated VM and storage helpers into shared utilities; added boot-time and lifecycle helpers; console and storage helpers now accept an optional kubeconfig for remote operations.
  • Chores

    • Adjusted timeout fixture handling and reorganized storage-class constants and imports.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Feb 19, 2026

📝 Walkthrough

Walkthrough

Adds cross-cluster live-migration (CCLM) tests and fixtures with remote kubeconfig propagation, relocates storage-class constants and helpers to shared modules, moves dv_wait_timeout to parent tests, and extends utilities (artifactory, console, storage) to accept optional client/kubeconfig parameters for remote operations.

Changes

Cohort / File(s) Summary
Top-level storage tests
tests/storage/conftest.py, tests/storage/constants.py, tests/storage/utils.py
Added dv_wait_timeout fixture (class scope) and imported TIMEOUT_30MIN; moved storage-class constants (STORAGE_CLASS_A/B, NO_STORAGE_CLASS_FAILURE_MESSAGE) and implemented shared helpers check_file_in_vm() and get_storage_class_for_storage_migration().
CCLM fixtures, utils & tests
tests/storage/cross_cluster_live_migration/conftest.py, tests/storage/cross_cluster_live_migration/constants.py, tests/storage/cross_cluster_live_migration/utils.py, tests/storage/cross_cluster_live_migration/test_cclm.py
Added remote-cluster kubeconfig fixture and many remote/local storage-class and VM provisioning fixtures (including DV and Windows VTPM flows); added boot/file lifecycle fixtures; refactored verify_compute_live_migration_after_cclm to accept local_vms and added boot-time parsing/verification, VM lifecycle helpers, and new test classes/methods for multi-VM and Windows VTPM scenarios.
Storage-migration module cleanup & tests
tests/storage/storage_migration/conftest.py, tests/storage/storage_migration/constants.py, tests/storage/storage_migration/utils.py, tests/storage/storage_migration/test_storage_class_migration.py
Removed local dv_wait_timeout and storage-migration constants/helpers (migrated to parent module); updated imports in tests to reference shared constants/helpers.
Utilities & unit tests
utilities/artifactory.py, utilities/console.py, utilities/storage.py, utilities/unittests/test_artifactory.py, utilities/unittests/test_console.py
Added optional `client: DynamicClient

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.62% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description check ✅ Passed The description includes key details and covers most required sections, though some template sections (What this PR does/why we need it, Which issue(s) this PR fixes, jira-ticket URL) are empty or incomplete.
Title check ✅ Passed The PR title directly describes the main changes: adding post-CCLM validation tests and Windows VM tests, which aligns with the changeset's core purpose.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-virtualization-qe-bot-4
Copy link
Copy Markdown

Report bugs in Issues

Welcome! 🎉

This pull request will be automatically processed with the following features:

🔄 Automatic Actions

  • Reviewer Assignment: Reviewers are automatically assigned based on the OWNERS file in the repository root
  • Size Labeling: PR size labels (XS, S, M, L, XL, XXL) are automatically applied based on changes
  • Issue Creation: A tracking issue is created for this PR and will be closed when the PR is merged or closed
  • Branch Labeling: Branch-specific labels are applied to track the target branch
  • Auto-verification: Auto-verified users have their PRs automatically marked as verified
  • Labels: Enabled categories: branch, can-be-merged, cherry-pick, has-conflicts, hold, needs-rebase, size, verified, wip

📋 Available Commands

PR Status Management

  • /wip - Mark PR as work in progress (adds WIP: prefix to title)
  • /wip cancel - Remove work in progress status
  • /hold - Block PR merging (approvers only)
  • /hold cancel - Unblock PR merging
  • /verified - Mark PR as verified
  • /verified cancel - Remove verification status
  • /reprocess - Trigger complete PR workflow reprocessing (useful if webhook failed or configuration changed)
  • /regenerate-welcome - Regenerate this welcome message

Review & Approval

  • /lgtm - Approve changes (looks good to me)
  • /approve - Approve PR (approvers only)
  • /assign-reviewers - Assign reviewers based on OWNERS file
  • /assign-reviewer @username - Assign specific reviewer
  • /check-can-merge - Check if PR meets merge requirements

Testing & Validation

  • /retest tox - Run Python test suite with tox
  • /retest build-container - Rebuild and test container image
  • /retest verify-bugs-are-open - verify-bugs-are-open
  • /retest all - Run all available tests

Container Operations

  • /build-and-push-container - Build and push container image (tagged with PR number)
    • Supports additional build arguments: /build-and-push-container --build-arg KEY=value

Cherry-pick Operations

  • /cherry-pick <branch> - Schedule cherry-pick to target branch when PR is merged
    • Multiple branches: /cherry-pick branch1 branch2 branch3

Label Management

  • /<label-name> - Add a label to the PR
  • /<label-name> cancel - Remove a label from the PR

✅ Merge Requirements

This PR will be automatically approved when the following conditions are met:

  1. Approval: /approve from at least one approver
  2. LGTM Count: Minimum 2 /lgtm from reviewers
  3. Status Checks: All required status checks must pass
  4. No Blockers: No WIP, hold, conflict labels
  5. Verified: PR must be marked as verified (if verification is enabled)

📊 Review Process

Approvers and Reviewers

Approvers:

  • dshchedr
  • jpeimer
  • myakove
  • rnetser
  • vsibirsk

Reviewers:

  • Ahmad-Hafe
  • RoniKishner
  • dalia-frank
  • dshchedr
  • duyanyan
  • geetikakay
  • josemacassan
  • jpeimer
  • kgoldbla
  • kshvaika
  • rnetser
  • stesrn
  • vsibirsk
Available Labels
  • hold
  • verified
  • wip
  • lgtm
  • approve

💡 Tips

  • WIP Status: Use /wip when your PR is not ready for review
  • Verification: The verified label is automatically removed on each new commit
  • Cherry-picking: Cherry-pick labels are processed when the PR is merged
  • Container Builds: Container images are automatically tagged with the PR number
  • Permission Levels: Some commands require approver permissions
  • Auto-verified Users: Certain users have automatic verification and merge privileges

For more information, please refer to the project documentation or contact the maintainers.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (7)
utilities/artifactory.py (1)

86-147: ⚠️ Potential issue | 🟡 Minor

MEDIUM: Document side effects for public utilities that create resources.

Both functions can create/deploy resources; add a Side effects section to the Google‑style docstrings.

Proposed fix
 def get_artifactory_secret(
@@
     Returns:
         Secret: The Artifactory Secret resource object.
 
     Raises:
         KeyError: If ARTIFACTORY_USER or ARTIFACTORY_TOKEN environment variables are not set.
+
+    Side effects:
+        Creates and deploys a Secret in the target namespace if it does not exist.
     """
@@
 def get_artifactory_config_map(
@@
     Returns:
         ConfigMap: The Artifactory ConfigMap resource object containing the TLS certificate.
@@
     Raises:
         KeyError: If server_url is not found in py_config.
         OSError: If SSL connection to the server fails.
+
+    Side effects:
+        Creates and deploys a ConfigMap in the target namespace if it does not exist.
     """

As per coding guidelines "Google-format docstrings REQUIRED for all public functions with non-obvious return values OR side effects".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@utilities/artifactory.py` around lines 86 - 147, Add a "Side effects" section
to the Google-style docstrings for get_artifactory_secret and
get_artifactory_config_map that clearly states these functions may create and
deploy Kubernetes resources (Secret named by ARTIFACTORY_SECRET_NAME via
Secret.deploy and ConfigMap "artifactory-configmap" via ConfigMap.deploy), and
mention any environment/ config dependencies and possible exceptions (KeyError
when ARTIFACTORY_USER/ARTIFACTORY_TOKEN or py_config["server_url"] are missing,
and OSError/SSL side effects from ssl.get_server_certificate). Include brief
guidance about permissions needed to create resources in the namespace and that
existing resources will be returned if present.
utilities/console.py (1)

127-133: ⚠️ Potential issue | 🟡 Minor

MEDIUM: Quote kubeconfig paths to avoid command parsing failures.

If the kubeconfig path contains spaces or shell metacharacters, the virtctl command can misparse. Quoting the path makes command construction robust.

Proposed fix
+from shlex import quote
@@
-        if self.kubeconfig:
-            cmd += f" --kubeconfig {self.kubeconfig}"
+        if self.kubeconfig:
+            cmd += f" --kubeconfig {quote(self.kubeconfig)}"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@utilities/console.py` around lines 127 - 133, The constructed command in
_generate_cmd uses self.kubeconfig unquoted which can break when the path
contains spaces or shell metacharacters; update _generate_cmd to quote the
kubeconfig when appending (e.g., use shlex.quote(self.kubeconfig) or wrap in
double quotes) so the --kubeconfig argument is always passed as a single token;
reference the virtctl_str, self.vm.name, self.vm.namespace and self.kubeconfig
variables when making the change.
tests/storage/cross_cluster_live_migration/conftest.py (1)

473-544: 🧹 Nitpick | 🔵 Trivial

MEDIUM: Unused remote_cluster_kubeconfig parameter — use @pytest.mark.usefixtures instead.

The remote_cluster_kubeconfig parameter at lines 477, 499, and 521 is declared as a fixture dependency but never used in the function body (Ruff ARG001). It's there to ensure the kubeconfig file exists before VM creation, which is a dependency-only concern.

The idiomatic way to express "I need this fixture's side-effect but not its value" is @pytest.mark.usefixtures. This removes the Ruff warning and makes the intent explicit:

Example for vm_for_cclm_from_template_with_data_source
 `@pytest.fixture`(scope="class")
+@pytest.mark.usefixtures("remote_cluster_kubeconfig")
 def vm_for_cclm_from_template_with_data_source(
     remote_admin_client,
     remote_cluster_source_test_namespace,
     remote_cluster_rhel10_data_source,
-    remote_cluster_kubeconfig,
     remote_cluster_source_storage_class,
 ):

Apply the same pattern to vm_for_cclm_with_instance_type and vm_for_cclm_from_template_with_dv.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/storage/cross_cluster_live_migration/conftest.py` around lines 473 -
544, The three fixtures vm_for_cclm_from_template_with_data_source,
vm_for_cclm_with_instance_type, and vm_for_cclm_from_template_with_dv declare an
unused parameter remote_cluster_kubeconfig; remove remote_cluster_kubeconfig
from each function signature and instead add
`@pytest.mark.usefixtures`("remote_cluster_kubeconfig") directly above each
fixture definition so the kubeconfig side-effect runs but the unused-argument
warning is eliminated; keep all other parameters and behavior unchanged.
tests/storage/storage_migration/utils.py (2)

53-56: ⚠️ Potential issue | 🟡 Minor

MEDIUM: Assertion message is missing a separator between the two f-strings.

The two f-string literals on lines 54-55 are implicitly concatenated with no space or newline between them. The output will read "…{'pvc1': 'sc-x'}Doesn't match…" — making the message difficult to parse during test triage. This matters because assertion messages are the primary debugging signal for CI failures.

Proposed fix
     assert not failed_pvc_storage_check, (
-        f"Failed PVC storage class check. PVC storage class: {failed_pvc_storage_check}"
-        f"Doesn't match expected target storage class: {target_storage_class}"
+        f"Failed PVC storage class check. PVC storage class: {failed_pvc_storage_check}. "
+        f"Doesn't match expected target storage class: {target_storage_class}"
     )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/storage/storage_migration/utils.py` around lines 53 - 56, The assertion
message concatenates two f-strings without a separator causing the PVC dict and
the following sentence to run together; update the assertion in
tests/storage/storage_migration/utils.py (the line using
failed_pvc_storage_check and target_storage_class) to include a clear separator
(e.g., add a space or newline between the two f-strings or merge them into a
single f-string) so the message reads "...PVC storage class:
{failed_pvc_storage_check} Doesn't match expected target storage class:
{target_storage_class}".

84-87: 🧹 Nitpick | 🔵 Trivial

LOW: Redundant .strip() on out.

Line 86 assigns out = ...strip(), then line 87 calls out.strip() again. The second .strip() is a no-op.

Proposed fix
 def verify_file_in_windows_vm(windows_vm: VirtualMachineForTests, file_name_with_path: str, file_content: str) -> None:
     cmd = shlex.split(f'powershell -command "Get-Content {file_name_with_path}"')
     out = run_ssh_commands(host=windows_vm.ssh_exec, commands=cmd)[0].strip()
-    assert out.strip() == file_content, f"'{out}' does not equal '{file_content}'"
+    assert out == file_content, f"'{out}' does not equal '{file_content}'"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/storage/storage_migration/utils.py` around lines 84 - 87, In
verify_file_in_windows_vm, remove the redundant second .strip() on out: out is
already stripped when assigned (in verify_file_in_windows_vm), so update the
assertion to compare out == file_content (or f"'{out}' does not equal
'{file_content}'") without calling out.strip() again; this fixes the no-op extra
call while keeping the same semantics.
tests/storage/cross_cluster_live_migration/test_cclm.py (2)

56-68: 🛠️ Refactor suggestion | 🟠 Major

MEDIUM: Dependency marker lacks a required # WHY comment.

The @pytest.mark.dependency at line 56 and all depends= references (lines 70, 77, 89, 94, 99) lack the required comment explaining why the dependency exists. The WHY here is straightforward — subsequent tests validate post-migration state and require the migration to have completed successfully — but the guideline is explicit.

Example:

# Dependency: post-migration tests require successful CCLM migration to validate state
`@pytest.mark.dependency`(name=f"{TESTS_CLASS_NAME_SEVERAL_VMS}::test_migrate_vm_from_remote_to_local_cluster")

As per coding guidelines: "When using @pytest.mark.dependency, a comment explaining WHY the dependency exists is REQUIRED."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/storage/cross_cluster_live_migration/test_cclm.py` around lines 56 -
68, Add a brief "# WHY" comment immediately above the `@pytest.mark.dependency`
decorator for test_migrate_vm_from_remote_to_local_cluster (and similarly for
all depends= references) that explains why the dependency is required (e.g.,
that subsequent tests validate post-migration state and therefore require the
CCLM migration to complete successfully); update the comment text near the
pytest.mark.dependency usage so it reads as a one-line rationale tied to the
decorator for clarity and compliance with the guideline.

25-33: ⚠️ Potential issue | 🟡 Minor

Add tier3 marker to module-level pytestmark—CCLM tests are complex, multi-cluster, and time-consuming.

Cross-cluster live migration tests involve multi-cluster coordination, expensive live migration operations, and Windows VM deployments, all of which are textbook characteristics of tier3 tests. The pytest configuration confirms tier3 is defined and actively used throughout the codebase for similar complex scenarios. Add the marker at module level to apply it consistently to both TestCCLMSeveralVMs and TestCCLMWindowsWithVTPM classes.

Proposed fix
 pytestmark = [
     pytest.mark.cclm,
     pytest.mark.remote_cluster,
+    pytest.mark.tier3,
     pytest.mark.usefixtures(
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/storage/cross_cluster_live_migration/test_cclm.py` around lines 25 -
33, The module-level pytestmark list is missing the tier3 marker; update the
pytestmark definition to include pytest.mark.tier3 so the marker applies to both
TestCCLMSeveralVMs and TestCCLMWindowsWithVTPM; locate the pytestmark variable
at the top of tests/storage/cross_cluster_live_migration/test_cclm.py and add
pytest.mark.tier3 alongside the existing pytest.mark.cclm and
pytest.mark.remote_cluster entries.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/storage/conftest.py`:
- Around line 592-594: The dv_wait_timeout fixture currently falls back to the
TIMEOUT_30MIN constant and uses a hasattr(request, "param") check; replace the
constant with an explicit numeric value (e.g., 1800) and simplify the logic to
use request.param.get("dv_wait_timeout", 1800) so the default is explicit; also
ensure any callers that pass this fixture value into functions use a named
parameter (e.g., timeout=...) instead of a positional multi-arg call.

In `@tests/storage/cross_cluster_live_migration/conftest.py`:
- Around line 153-171: Replace the single-letter file handle variable used when
writing the kubeconfig with a descriptive name and ensure cleanup remains same;
specifically, in the block that opens kubeconfig_path (currently using "f"),
rename the handle to something like "kubeconfig_file" (or similar) in the with
open(...) as ... statement and update its usage in
yaml.safe_dump(kubeconfig_dict, ...), keeping the existing cleanup try/except
that removes kubeconfig_path and temp_dir and logs via LOGGER.
- Line 148: The kubeconfig file creation writes the remote cluster bearer token
with default file permissions (via open(kubeconfig_path, "w")), making the token
world-readable; change the file creation to use secure permissions (e.g., open
the file descriptor via os.open with mode 0o600 and wrap it with os.fdopen, or
create it then immediately call os.chmod(kubeconfig_path, 0o600)) so
kubeconfig_path is only readable by the owner, and keep the code that writes the
"users": [{"name": user_name, "user": {"token": remote_cluster_auth_token}}]
block unchanged except for using the secured file handle or chmod.
- Around line 558-560: The fixture vms_boot_time_before_cclm currently depends
on vms_for_cclm and yields boot times, creating an implicit ordering dependency;
update its signature to depend on booted_vms_for_cclm instead of vms_for_cclm
and replace the yield with a direct return (no teardown) so it only runs after
VMs are actually booted: change def vms_boot_time_before_cclm(vms_for_cclm,
remote_cluster_kubeconfig) to def vms_boot_time_before_cclm(booted_vms_for_cclm,
remote_cluster_kubeconfig) and return the dict comprehension using
booted_vms_for_cclm.
- Around line 563-569: The fixtures booted_vms_for_cclm and
written_file_to_vms_before_cclm currently use yield but have no teardown; change
them to return the value instead of yielding so they are regular-returning
fixtures (keep the same parameters and scope and keep the running_vm calls and
any pre-setup logic), i.e., remove the generator semantics and ensure the
functions simply perform setup then return vms_for_cclm / the prepared value to
avoid implying a teardown block.

In `@tests/storage/cross_cluster_live_migration/test_cclm.py`:
- Around line 36-52: The parametrize calls use a comma-separated string for
argnames; change them to a tuple of positional argument names: replace the first
argument of pytest.mark.parametrize from "remote_cluster_source_storage_class,
local_cluster_target_storage_class, vms_for_cclm" to a tuple
('remote_cluster_source_storage_class', 'local_cluster_target_storage_class',
'vms_for_cclm') in the TestCCLMSeveralVMs parametrize and make the identical
change for the TestCCLMWindowsWithVTPM parametrize (i.e., update the
pytest.mark.parametrize argnames to a tuple form so pytest receives positional
argnames correctly).
- Around line 136-138: Add a pytest dependency marker to both
test_source_vms_can_be_deleted and test_source_windows_vms_can_be_deleted so
they only run if the migration test in the same test class completed
successfully; specifically, annotate each function with
`@pytest.mark.dependency`(depends=["<migration_test_name>"]) where
<migration_test_name> is the function name of the cross-cluster live migration
test in this class (replace with the actual migration test function name),
ensuring both cleanup tests depend on that migration test.

In `@tests/storage/cross_cluster_live_migration/utils.py`:
- Around line 137-156: Extract the duplicated assertion/reporting logic into a
shared helper like verify_vms_boot_time(local_vms, initial_boot_time,
boot_time_getter) that accepts a boot-time retrieval callable; move the
loop/compare/assert code from verify_vms_boot_time_after_migration and
verify_vms_boot_time_after_storage_migration into that helper, then change
verify_vms_boot_time_after_migration to call it with
get_vm_boot_time_via_console and change
verify_vms_boot_time_after_storage_migration to call it with get_vm_boot_time so
both reuse the same verification and reporting behavior.

In `@tests/storage/utils.py`:
- Around line 29-31: Replace the generic import "from utilities import console"
with an explicit import "from utilities.console import Console" to follow the
specific-import guideline (the file already imports
NO_STORAGE_CLASS_FAILURE_MESSAGE); then update all multi-argument pexpect expect
calls (e.g., vm_console.expect(file_name, timeout=TIMEOUT_20SEC)) to use named
parameters so the pattern is explicit—change them to
vm_console.expect(pattern=file_name, timeout=TIMEOUT_20SEC) (apply this to
instances referencing vm_console, TIMEOUT_20SEC, and file_name).

In `@utilities/storage.py`:
- Around line 627-642: The write_file function lacks type hints, a complete
Google docstring (Returns and Side effects), and should make stop_vm a
keyword-only parameter; update the signature to include type hints for vm
(VirtualMachine), filename (str), content (str), kubeconfig (Optional[str]) and
return type (None) and change stop_vm to be keyword-only (e.g., *, stop_vm: bool
= True); enhance the docstring to add Returns: None and Side effects: may
start/stop the VM and write to disk, and ensure the implementation uses a
try/finally to stop the VM when stop_vm is True (call vm.stop()) after using
console.Console(vm=vm, kubeconfig=kubeconfig) in write_file so the VM is started
if needed and always stopped when requested.

---

Outside diff comments:
In `@tests/storage/cross_cluster_live_migration/conftest.py`:
- Around line 473-544: The three fixtures
vm_for_cclm_from_template_with_data_source, vm_for_cclm_with_instance_type, and
vm_for_cclm_from_template_with_dv declare an unused parameter
remote_cluster_kubeconfig; remove remote_cluster_kubeconfig from each function
signature and instead add `@pytest.mark.usefixtures`("remote_cluster_kubeconfig")
directly above each fixture definition so the kubeconfig side-effect runs but
the unused-argument warning is eliminated; keep all other parameters and
behavior unchanged.

In `@tests/storage/cross_cluster_live_migration/test_cclm.py`:
- Around line 56-68: Add a brief "# WHY" comment immediately above the
`@pytest.mark.dependency` decorator for
test_migrate_vm_from_remote_to_local_cluster (and similarly for all depends=
references) that explains why the dependency is required (e.g., that subsequent
tests validate post-migration state and therefore require the CCLM migration to
complete successfully); update the comment text near the pytest.mark.dependency
usage so it reads as a one-line rationale tied to the decorator for clarity and
compliance with the guideline.
- Around line 25-33: The module-level pytestmark list is missing the tier3
marker; update the pytestmark definition to include pytest.mark.tier3 so the
marker applies to both TestCCLMSeveralVMs and TestCCLMWindowsWithVTPM; locate
the pytestmark variable at the top of
tests/storage/cross_cluster_live_migration/test_cclm.py and add
pytest.mark.tier3 alongside the existing pytest.mark.cclm and
pytest.mark.remote_cluster entries.

In `@tests/storage/storage_migration/utils.py`:
- Around line 53-56: The assertion message concatenates two f-strings without a
separator causing the PVC dict and the following sentence to run together;
update the assertion in tests/storage/storage_migration/utils.py (the line using
failed_pvc_storage_check and target_storage_class) to include a clear separator
(e.g., add a space or newline between the two f-strings or merge them into a
single f-string) so the message reads "...PVC storage class:
{failed_pvc_storage_check} Doesn't match expected target storage class:
{target_storage_class}".
- Around line 84-87: In verify_file_in_windows_vm, remove the redundant second
.strip() on out: out is already stripped when assigned (in
verify_file_in_windows_vm), so update the assertion to compare out ==
file_content (or f"'{out}' does not equal '{file_content}'") without calling
out.strip() again; this fixes the no-op extra call while keeping the same
semantics.

In `@utilities/artifactory.py`:
- Around line 86-147: Add a "Side effects" section to the Google-style
docstrings for get_artifactory_secret and get_artifactory_config_map that
clearly states these functions may create and deploy Kubernetes resources
(Secret named by ARTIFACTORY_SECRET_NAME via Secret.deploy and ConfigMap
"artifactory-configmap" via ConfigMap.deploy), and mention any environment/
config dependencies and possible exceptions (KeyError when
ARTIFACTORY_USER/ARTIFACTORY_TOKEN or py_config["server_url"] are missing, and
OSError/SSL side effects from ssl.get_server_certificate). Include brief
guidance about permissions needed to create resources in the namespace and that
existing resources will be returned if present.

In `@utilities/console.py`:
- Around line 127-133: The constructed command in _generate_cmd uses
self.kubeconfig unquoted which can break when the path contains spaces or shell
metacharacters; update _generate_cmd to quote the kubeconfig when appending
(e.g., use shlex.quote(self.kubeconfig) or wrap in double quotes) so the
--kubeconfig argument is always passed as a single token; reference the
virtctl_str, self.vm.name, self.vm.namespace and self.kubeconfig variables when
making the change.

Comment thread tests/storage/conftest.py
Comment thread tests/storage/cross_cluster_live_migration/conftest.py
Comment thread tests/storage/cross_cluster_live_migration/conftest.py
Comment thread tests/storage/cross_cluster_live_migration/conftest.py Outdated
Comment thread tests/storage/cross_cluster_live_migration/conftest.py Outdated
Comment thread tests/storage/cross_cluster_live_migration/test_cclm.py
Comment thread tests/storage/cross_cluster_live_migration/test_cclm.py
Comment thread tests/storage/cross_cluster_live_migration/utils.py
Comment thread tests/storage/utils.py
Comment thread utilities/storage.py Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@utilities/unittests/test_console.py`:
- Line 110: The test uses a bare assertion on console.cmd; update the assertion
to include a failure message that shows both expected and actual values so
failures are actionable — change the assertion that references console.cmd (the
existing line asserting "virtctl console test-vm -n test-namespace --kubeconfig
/path/to/kubeconfig") to use the two-argument assert form and include a
descriptive message containing the expected string and console.cmd so test
failures print the observed value.
- Around line 101-111: Add a new unit test that exercises the Console code path
where kubeconfig is provided but namespace is None: create a test (e.g.,
test_console_generate_cmd_with_kubeconfig_no_namespace) that sets
mock_vm.username and mock_vm.password, sets mock_vm.namespace = None, patches
"console.VIRTCTL" to "virtctl", constructs Console(vm=mock_vm,
kubeconfig="/path/to/kubeconfig"), and asserts Console.cmd equals the expected
string without a -n flag (i.e., includes "--kubeconfig /path/to/kubeconfig"
only). This targets the Console._generate_cmd/Console.cmd branch for kubeconfig
+ no namespace.

---

Duplicate comments:
In `@tests/storage/utils.py`:
- Around line 531-533: The defensive check on the schema-provided attribute
vm.ready should be removed or made explicit: either delete the if not vm.ready
branch and assume callers guarantee the VM is running, or add an explicit
parameter (e.g., start_if_not_ready: bool = False) to this helper and gate the
vm.start(wait=True) call on that parameter instead of checking vm.ready;
reference the vm.ready attribute and the vm.start(wait=True) call when making
the change so callers must opt into auto-start behavior.

In `@utilities/unittests/test_console.py`:
- Around line 103-104: The test assigns a real-looking credential to
mock_vm.password which triggers Ruff S105; update the assignment in the test
(the mock_vm.password attribute alongside mock_vm.username) to use a
non-credential placeholder such as "default-pass" (matching other uses in this
file) so the linter no longer flags a hardcoded password.

Comment thread utilities/unittests/test_console.py
Comment thread utilities/unittests/test_console.py
Copy link
Copy Markdown
Collaborator

@rnetser rnetser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pr is too big; please split to smaller PRs based on added functionality

Generate a kubeconfig file from the remote admin client credentials.
Returns the path to the generated kubeconfig file.
"""
# Extract cluster information from the client
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use get_client - pass the client config and temp_file_path

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it create a kubeconfig file on temp_file_path?
Doesn't seem to work that way. Keeping as is, as I don't yet understand how to get a kubeconfig file from get_client.

name="dv-fedora-imported-cclm",
namespace=remote_cluster_source_test_namespace.name,
source=REGISTRY_STR,
url=QUAY_FEDORA_CONTAINER_IMAGE,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use data source?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to test different types of VMs. We also have VMs that use DataSource.



@pytest.fixture(scope="class")
def vm_for_cclm_from_template_with_dv(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc the refered templated here is dataVolumeTemplate and not vm template, if so, the name is confusing

@pytest.fixture(scope="class")
def local_vms_after_cclm_migration(admin_client, namespace, vms_for_cclm):
"""
Create local VM references for VMs after CCLM migration.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the description is confusing - the fixture does not create anything but fetches vms from the cluster

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It creates python objects that reference the VMs in the local cluster.
If the description is confusing, I can rephrase.

namespace=remote_cluster_source_test_namespace.name,
storage_class=remote_cluster_source_storage_class,
source="http",
# Using WSL image to avoid the issue of the Windows VM not being able to boot
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a bug?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, and I don't have the capacity to debug it.
If you know which image is better to use in this test (and this image is not broken, not too old, etc - I will gladly use it).


@pytest.mark.dependency(depends=[f"{TESTS_CLASS_NAME_WINDOWS_WITH_VTPM}::test_migrate_windows_vm_with_vtpm"])
@pytest.mark.polarion("CNV-14337")
def test_snapshot_and_restore_windows_vms_after_cclm(self, unprivileged_client, local_vms_after_cclm_migration):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

raise ValueError(f"Could not determine boot time from output: {output}")


def get_vm_boot_time_via_console(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have utilities.virt.get_vm_boot_time -why not use it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing function is using ssh, but I can't ssh to a VM in the remote cluster

)

@pytest.mark.polarion("CNV-14334")
def test_source_vms_can_be_deleted(self, vms_for_cclm):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name is misleading - the test verifies that the vms are deleted

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test attempts to delete the VMs and verifies that this operation succeeded. I renamed the function to make it more clear.

assert not vms_failed_cleanup, f"Failed to clean up source VMs: {vms_failed_cleanup}"


def delete_file_in_vm(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is already a function to delete file, e.g tests.virt.cluster.vm_lifecycle.test_vm_data_persistency.delete_file
why not use it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Console connection shows more stable lately (we haven't seen console failures for a while)
  2. delete_file expects only RHEL / WIN, and calls run_os_command, I wouldn't want to use it.
def delete_file(vm, os):
    commands = {RHEL: f"rm {NEW_FILENAME}", WIN: f"del {NEW_FILENAME}"}
    run_os_command(vm=vm, command=commands[os])
    assert not grep_file(vm=vm, os=os)

Comment thread tests/storage/utils.py
assert actual_bus == expected_bus, f"Disk {volume.name} has bus '{actual_bus}' but expected '{expected_bus}'"


def check_file_in_vm(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is already utilities.oadp.check_file_in_running_vm - please re-use

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utilities.oadp.check_file_in_running_vm is different,
and I want to keep the storage utils separate from oadp.

@openshift-virtualization-qe-bot-3
Copy link
Copy Markdown
Contributor

/retest all

Auto-triggered: Files in this PR were modified by merged PR #4053.

Overlapping files

tests/storage/cross_cluster_live_migration/conftest.py
tests/storage/cross_cluster_live_migration/utils.py

Copy link
Copy Markdown
Contributor Author

@jpeimer jpeimer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rnetser as you asked, I'm splitting the PR. Here is the first one. I tried to address the comments there: #4108

raise ValueError(f"Could not determine boot time from output: {output}")


def get_vm_boot_time_via_console(
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing function is using ssh, but I can't ssh to a VM in the remote cluster

name="dv-fedora-imported-cclm",
namespace=remote_cluster_source_test_namespace.name,
source=REGISTRY_STR,
url=QUAY_FEDORA_CONTAINER_IMAGE,
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to test different types of VMs. We also have VMs that use DataSource.

@pytest.fixture(scope="class")
def local_vms_after_cclm_migration(admin_client, namespace, vms_for_cclm):
"""
Create local VM references for VMs after CCLM migration.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It creates python objects that reference the VMs in the local cluster.
If the description is confusing, I can rephrase.

Comment thread tests/storage/utils.py
assert actual_bus == expected_bus, f"Disk {volume.name} has bus '{actual_bus}' but expected '{expected_bus}'"


def check_file_in_vm(
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utilities.oadp.check_file_in_running_vm is different,
and I want to keep the storage utils separate from oadp.

assert not vms_failed_cleanup, f"Failed to clean up source VMs: {vms_failed_cleanup}"


def delete_file_in_vm(
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Console connection shows more stable lately (we haven't seen console failures for a while)
  2. delete_file expects only RHEL / WIN, and calls run_os_command, I wouldn't want to use it.
def delete_file(vm, os):
    commands = {RHEL: f"rm {NEW_FILENAME}", WIN: f"del {NEW_FILENAME}"}
    run_os_command(vm=vm, command=commands[os])
    assert not grep_file(vm=vm, os=os)

namespace=remote_cluster_source_test_namespace.name,
storage_class=remote_cluster_source_storage_class,
source="http",
# Using WSL image to avoid the issue of the Windows VM not being able to boot
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, and I don't have the capacity to debug it.
If you know which image is better to use in this test (and this image is not broken, not too old, etc - I will gladly use it).

Generate a kubeconfig file from the remote admin client credentials.
Returns the path to the generated kubeconfig file.
"""
# Extract cluster information from the client
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it create a kubeconfig file on temp_file_path?
Doesn't seem to work that way. Keeping as is, as I don't yet understand how to get a kubeconfig file from get_client.

@pytest.mark.dependency(depends=[f"{TESTS_CLASS_NAME_SEVERAL_VMS}::test_migrate_vm_from_remote_to_local_cluster"])
@pytest.mark.polarion("CNV-11997")
def test_snapshot_and_restore_vms_after_cclm(self, unprivileged_client, local_vms_after_cclm_migration):
for vm in local_vms_after_cclm_migration:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snapshot without restore makes no sense, but I can split it into two tests.

)

@pytest.mark.polarion("CNV-14334")
def test_source_vms_can_be_deleted(self, vms_for_cclm):
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test attempts to delete the VMs and verifies that this operation succeeded. I renamed the function to make it more clear.

@jpeimer
Copy link
Copy Markdown
Contributor Author

jpeimer commented Mar 9, 2026

/wip

@openshift-virtualization-qe-bot-5 openshift-virtualization-qe-bot-5 changed the title [Storage] [CCLM] Post-CCLM validation tests, Windows VM tests WIP: [Storage] [CCLM] Post-CCLM validation tests, Windows VM tests Mar 9, 2026
dshchedr pushed a commit that referenced this pull request Mar 19, 2026
##### Short description:
- Check VM boot_id before and after CCLM
- Allow connection to the VM Console in the remote cluster with the
kubeconfig
- Add post-CCLM validation tests

Jira: https://issues.redhat.com/browse/CNV-60016

##### More details:
Split from
#3926

##### What this PR does / why we need it:

##### Which issue(s) this PR fixes:

##### Special notes for reviewer:
Assisted by Claude AI and Cursor AI

##### jira-ticket:
<!-- full-ticket-url needs to be provided. This would add a link to the
pull request to the jira and close it when the pull request is merged
If the task is not tracked by a Jira ticket, just write "NONE".
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Expanded cross-cluster VM migration test coverage with additional VM
scenarios and state verification.
* Added post-migration validation for VM boot integrity and file
persistence.
  * New tests for VM state transitions and cleanup operations.

* **Refactor**
* Consolidated test constants for broader reusability across test
modules.
* Refactored VM file verification utilities for improved test
modularity.
  * Enhanced console utility support for remote cluster operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
jpeimer added a commit to jpeimer/openshift-virtualization-tests that referenced this pull request Mar 22, 2026
- Check VM boot_id before and after CCLM
- Allow connection to the VM Console in the remote cluster with the
kubeconfig
- Add post-CCLM validation tests

Jira: https://issues.redhat.com/browse/CNV-60016

Split from
RedHatQE#3926

Assisted by Claude AI and Cursor AI

<!-- full-ticket-url needs to be provided. This would add a link to the
pull request to the jira and close it when the pull request is merged
If the task is not tracked by a Jira ticket, just write "NONE".
-->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

* **Tests**
* Expanded cross-cluster VM migration test coverage with additional VM
scenarios and state verification.
* Added post-migration validation for VM boot integrity and file
persistence.
  * New tests for VM state transitions and cleanup operations.

* **Refactor**
* Consolidated test constants for broader reusability across test
modules.
* Refactored VM file verification utilities for improved test
modularity.
  * Enhanced console utility support for remote cluster operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@github-actions
Copy link
Copy Markdown

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale label Apr 11, 2026
@github-actions
Copy link
Copy Markdown

This PR was closed because it has not been updated in past 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants