Adding Robust destroy in cluster toolkit

[shubpal07]

Summary of Changes

The gcluster destroy command has been enhanced with a --robust flag to improve the reliability of resource cleanup, particularly for firewall rules.

The --robust flag introduces these key features:

1. Targeted Firewall Cleanup: Firewall rule cleanup is performed only for deployment groups that contain a network module (specifically, a module with source: "modules/network/vpc"). The cleanup for a group happens just before that specific group's resources are destroyed. This ensures firewalls are removed at the most appropriate time to avoid "resource in use" errors. The cleanup now uses the Google Cloud Go SDK instead of external gcloud calls.
2. Automatic Retry Mechanism: The entire destroy process for all selected groups is retried if any failure occurs.
   • Retry Count: The command will make up to 3 attempts to destroy the deployment successfully.
   • Purpose: This helps overcome transient issues or dependencies between resources that might not be immediately resolved in the cloud backend.

User Interaction

• Firewall Deletion Confirmation: When --robust is used and firewall rules are found for a group, the user is prompted to confirm their deletion [y/n], unless --auto-approve is also specified.
• Automated Mode: The --auto-approve flag suppresses all prompts, including firewall deletion confirmations, for automated workflows.

How to Use

Standard Robust Destroy (Interactive)

This will prompt you for confirmation before deleting firewall rules.

./gcluster destroy <DEPLOYMENT_DIRECTORY> --robust

Fully Automated Robust Destroy

This will automatically approve and delete all resources, including firewall rules, without any user prompts.

./gcluster destroy <DEPLOYMENT_DIRECTORY> --robust --auto-approve

Standard Destroy

The original destroy functionality remains unchanged. The robust logic is only triggered when the --robust flag is present.

# Standard interactive destroy
./gcluster destroy <DEPLOYMENT_DIRECTORY>

# Standard automated destroy
./gcluster destroy <DEPLOYMENT_DIRECTORY> --auto-approve

How to Test

1. Deploy a Cluster: First, create a sample deployment if you don't have one already.

   ./gcluster deploy examples/hpc-slurm.yaml

2. Build the Tool: Ensure you have the latest version of the gcluster binary with these changes.

   make gcluster

3. Run Robust Destroy: Execute the destroy command with the --robust flag on your deployment directory.

   ./gcluster destroy <YOUR_DEPLOYMENT_DIRECTORY> --robust

4. Verify the Prompt: The tool should list the firewall rules it intends to delete and wait for your input.

   • Enter y to confirm and watch the process continue.
   • (Optional) Re-deploy and run again, this time entering n to verify that the firewall cleanup is skipped.

5. Verify Automated Destroy: (Optional) If you want to test the non-interactive mode, re-deploy the cluster and run:

   ./gcluster destroy <YOUR_DEPLOYMENT_DIRECTORY> --robust --auto-approve

   The command should complete the entire destruction process without any prompts

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[shubpal07]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a --robust flag to the gcluster destroy command, adding functionality for targeted firewall cleanup and an automatic retry mechanism. The changes are well-structured, primarily within cmd/destroy.go. My review focuses on enhancing the robustness of the new error handling and resource cleanup logic to ensure the retry mechanism behaves as expected under failure conditions.

[sarthakag]

Can we make this the default behaviour for gcluster destroy command, instead of under the specific --robust flag? This would fix the incomplete destroy behaviour.

[cboneti]

Can we make this the default behaviour for gcluster destroy command, instead of under the specific --robust flag? This would fix the incomplete destroy behaviour.

I would prefer that we don't change the default yet. Let's test this extensively internally. Most external customers don't need this feature.

[cboneti]

Good new feature. Thanks!!