Disaster recovery in cloudstack

[TheKunalSen]

So I have built a cloudstack setup where I have a nfs server, 3 hosts and 1 management server in one zone (172.16.17.0/24)

And I have created a system gateway vm which is connected to a cloudflare tunnel and serving all my websites (routing the traffic into the internal 10.x.x.0/16 vms) through reverse proxy.

For in-zone ha if the host goes down the vms migrate to another host this is okay

But if I want to build a disaster recovery system where if this site goes down totally i still want my websites to be alive!

There are a couple methods I have thought
Which are

1. I build a site b setup almost same as this one and copy all the vms and all to that setup and setup a cloudflare tunnel load balancer which will load balance the traffic between two sites (note that the site b would be a new zone in that same management server which will be in a different public ip network 192.168.1.0/24)
   But in this case the problem is suppose I have 100 vms and I copied all of them manually to site b-zone-b and it's working, but if I create another vm on site-a-zone-a and I want that vm to replicate to site-b-zone-b, how is this possible cause the nfs storage would be different, the host ids would be different, the only thing which will be connected are the management servers.

2. If I try to add a site-b host which is on a different network on this zone then connect both the network on site to site vpn then the hosts gets added but the system vms on that host doesn't work or the agent states stays blank, even if i create another pod on that site -b network inside that site-a-zone-a the hosts gets added but the vms doesn't work properly, if this thing happens then host from different sites can migrate the vms between themselves.

What do you think what would be the best strategy for disaster recovery in cloudstack for me!

[Flpvoigt]

I would not put the two sites in the same CloudStack zone and expect CloudStack HA to replicate or migrate running VMs across the WAN. CloudStack HA is primarily for host/VM failure within a zone; it is not a complete-site DR mechanism.

For this topology, I would use:

1. Separate sites as separate zones (or separate regions if they have independent Management Server clusters), rather than adding site-B hosts to site-A's zone.
2. Replicate the VM data outside the hypervisor layer: use the storage platform's replication or a supported CloudStack Backup & Recovery provider. In CloudStack 4.22+, a cross-zone backup repository can be used to create a new instance in another zone; the repository must be reachable from the destination zone.
3. Replicate/copy the CloudStack database and secondary storage as part of the DR plan. Templates and snapshots alone are not enough if you need the whole environment and its metadata.
4. Replicate the application data as well (database replication, object storage, etc.). Cloudflare Load Balancing can direct traffic to site B, but it does not replicate VM state or application data.

A practical runbook is: take scheduled backups/snapshots in site A, copy them to a repository available in site B, keep equivalent networks/offerings/templates in site B, and restore/create the instances there during a site-A outage. Test the restore and DNS/Cloudflare failover regularly. I would avoid live migration across sites unless the hypervisor/storage vendor explicitly supports it with the required latency and shared storage guarantees.

References:

• https://docs.cloudstack.apache.org/en/latest/adminguide/backup_and_recovery.html
• https://docs.cloudstack.apache.org/en/latest/conceptsandterminology/choosing_deployment_architecture.html

[DRDAVIDBANNER]

Hi Kunal, I’m Pete, CTO at Sendense. This is a use case we handle with Sendense: agentless VM replication and disaster recovery between CloudStack sites.

Building on the suggestion above, I’d set this up as two separately operable sites, with site B having its own CloudStack management, compute, storage and networking. Sendense then keeps recovery copies of your workloads ready at the second site and provides the replication, testing and failover workflow.

The source and destination storage can also be different. You can keep your existing NFS storage at site A and use a different storage type at site B—for example, NFS at one site and block storage at the other. Each site can use the storage that suits its requirements and budget. Sendense handles protection and recovery across those environments, with no NFS workaround needed.

The different host IDs and storage locations you mentioned are handled through the destination configuration. Each site uses its own CloudStack resources, and the replica is prepared for the destination’s compute, storage and networks. You don’t need to reproduce the source host IDs or manually reconcile copied disk files.

For your setup, the workflow would look like this:

1. Configure replication to site B. Select the VMs you want to protect, their destination placement and the replication schedule.
2. Bring new VMs into the same workflow. As you add workloads at site A, add them to protection through Sendense, which manages their replication to the recovery site.
3. Test recovery while production stays running. Use an isolated test failover to boot the recovered VMs and check that the websites and their dependencies work.
4. Activate the recovery workloads when needed. Bring the services up at site B, then use your Cloudflare routing to direct traffic to the recovered environment.

For your websites, I’d include the database VMs and gateway/reverse proxy in the recovery plan, then test the complete path from Cloudflare through to the application. That gives you a practical way to verify the service will work from the second site.

There’s more on the workflow here: [Sendense disaster recovery](https://sendense.com/solutions/dr-at-scale).

Happy to walk through your two-site layout and show how this would work with your existing NFS setup and the storage you’re considering for site B.

[nickaccuweb]

@TheKunalSen

I would recommend going with your first option, using two separate CloudStack zones, one at each physical location.

Regarding Option 1:

CloudStack doesn't automatically replicate newly created VMs between zones just because they are managed by the same management server.

However, if you're using CloudStack 4.22 or later with KVM, there's a useful feature in the NAS Backup and Recovery plugin that allows you to create a new VM in another zone from an existing VM backup.

You can enable cross-zone instance creation on the backup repository, schedule VM backups, and replicate the backup data to Site B. The repository must remain accessible from the destination zone.

For your 100 VMs, you could use the CloudStack API to automate backup configuration for newly created VMs. This would reduce manual work, although it wouldn't provide continuous VM replication.

Regarding Option 2:

I wouldn't recommend extending the same zone across two physically separate sites just to achieve host-level failover.

Even if the hosts connect through a site-to-site VPN, live migration and system VM operations still depend on proper networking, storage accessibility, and latency. This could become unreliable during network interruptions.

A separate zone with its own hosts, primary storage, secondary storage, and network configuration would be a cleaner approach.

My recommended setup:

1. Configure Site A and Site B as separate CloudStack zones.
2. Keep independent NFS storage at both locations and replicate VM backups to Site B.
3. Use scheduled backups and CloudStack API automation to protect newly created VMs.
4. Deploy a reverse proxy and separate Cloudflare Tunnel at each site, using Cloudflare Load Balancing with health checks for automatic traffic failover.
5. Replicate application data and databases between sites so the standby applications have up-to-date data.
6. Protect the CloudStack management server and MySQL database as well. If both are hosted only at Site A, losing that site would prevent you from managing or restoring VMs until the management services are recovered.

One important point: VM backups and automatic website failover are two different things. If you want websites to remain available with minimal downtime, the critical application VMs should already be running at Site B, with their data synchronized. Otherwise, you'll need time to restore the VMs before Cloudflare can route traffic to them.

For your environment, I would start with an active-passive setup, test the backup restoration process, and then introduce application-level replication for the websites that require faster failover.

Also, just to clarify, 192.168.1.0/24 is a private IP range, not a public one.

Regards,
Nick R
AccuWeb.Cloud