What does Snapshots, Backups, and Incremental Snapshots mean in CloudStack

[daviftorres]

Dear CloudStack fellows,

Recently, Lucian mentioned the new 4.22 feature that "enables incremental volume snapshots" (https://www.youtube.com/watch?v=ZZi3CpX6u2I&t=680s).

I would like to open a discussion so we can collaboratively clarify the different types of snapshots and backups that CloudStack currently supports, and how they actually work.

First, there is an important conceptual distinction between snapshots and backups:

• Snapshots are primarily point-in-time rollback mechanisms. They rely on layers of changes, and usually do not involve copying the full dataset.
• Backups are copies of data intended for restore, retention, and disaster recovery.

From there, things become more nuanced:

• A full backup is a complete bit-for-bit copy of the original data.
• An incremental backup contains only the changes since the last backup, which helps save space and time.
  • Incremental backups often rely on snapshot mechanisms to track changes.
  • A synthetic full backup can later be built by applying all incremental layers on top of the initial full backup.

Where I am still unclear is how these concepts map into the Apache CloudStack ecosystem:

• Snapshots in CloudStack are stored on Secondary Storage alongside templates and images, which makes them feel closer to “copies” than pure rollback layers.
• NAS Backup appears similar to snapshots, but sends a volume copy to a separate repository.

For the sake of simplicity, I am focusing first on KVM + NFS but it is not limited to.

These distinctions matter, especially when SLAs, compliance requirements, and operational policies treat snapshots and backups very differently.

I encourage others to share their understanding and explanations. My goal is to consolidate this discussion and translate it into clearer documentation afterward.

Regards,

Davi Torres

[daviftorres]

We upgraded our Dev environment to 4.22, and we are testing instance and volume snapshots to visualize the incremental portion.

I can confirm that instance snapshots sometimes use QCOW2's internal snapshots.

Table vm_snapshots

Note: I had to include memory because the instance snapshots could (in that occasion) only be taken when the instance is in the Running state. Please correct me if I am wrong. This instance was deployed from an ISO.

Then, for no apparent reason on another instance that was deployed from a template to a unlinked sparse provisioning, it "decided" to take external snapshots.

It can also be observed on the XML Dump of the instance.

If you know what logic is used to define if the snapshot will be used for snapshots (internal or external), please comment below.

[chunkyen]

You can most certainly create instance snapshots while the instance is in a shutdown state. You can also create instance snapshot without memory.

[daviftorres]

You can most certainly create instance snapshots while the instance is in a shutdown state. You can also create instance snapshot without memory.

Dear @chunkyen , than you for your reply.

What you said is generally correct, but in practice it is not always true. There are cases where creating a snapshot without memory is not allowed, and the UI error message does not clearly explain why.

For example, in the instance below, snapshot creation only works if Snapshot memory is enabled:

However, on other instances, snapshot creation works fine without memory, for example:

Some additional context:

The instance that only allows snapshots with memory has qemu-guest-agent installed and was deployed from an ISO.

The instance that allows snapshots without memory was deployed from an out-of-the-box template and does not have qemu-guest-agent installed.

So while snapshots without memory are supported in general, there seems to be some internal logic or constraint that blocks it in certain cases, and the current error message does not make this clear.

Do you know what specifically causes this behavior?

[daviftorres]

Note: I had to include memory because the instance snapshots could (in that occasion) only be taken when the instance is in the Running state. Please correct me if I am wrong. This instance was deployed from an ISO.

Answering my own question, the fact that limited the Instance Snapshot without adding the memory was the missing setting rootDiskController = osdefault:

I managed to "fix" the ISO template by adding those settings in the table vm_template_details.

Besides the fix mentioned above, there is a limitation around mixing snapshot types. If a snapshot that includes memory already exists, the system does not allow creating a snapshot without memory. The reverse is also true.

My understanding is that disk-only snapshots use QCOW2 metadata, while disk-plus-memory snapshots use a chain-based approach. Because these rely on different underlying mechanisms, mixing them is not allowed, likely to prevent data corruption or loss. Like described in PR #11039

[daviftorres]

When working with volume snapshots, the behavior is very different from what one might expect.

Table snapshot_store_ref

Each consecutive snapshot creates a new file in Secondary Storage, and every one of these files has exactly the same size and the same hash.

Therefore, I conclude that volume snapshots are effectively standalone full backups that are sent to Secondary Storage each time.

Even the status uses the work Backup.

[gpordeus]

Hi! Yes, volume snapshots are backups with an unfortunate name.

Just pointing out you can also enable the configuration kvm.incremental.snapshot, which creates incremental chains of snapshot.delta.max size. (This would be the "incremental volume snapshots" mentioned in the first message)

[daviftorres]

Thank you @gpordeus . I do have it enabled and I am trying to understand how they work because labels and descriptions are not always clear. For example, in some occasions it does internal (leveraging QCOW2's metadata) and others it does external (chaining multiple QCOW2 files).

[gpordeus]

When using volume snapshots, there are internal pointers in the original volume (in the primary storage) that are used only to create the deltas, which are then backed up to the secondary storage. Would this be what you mean? There's more info about the workflow in #8907.

[daviftorres]

Moreover, in the snapshots table (see below), I found some references to different snapshot types (from 0 to 7, I guess).

And, I came across this documentation page that mentions snapshot types: https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

However, the information there is not detailed enough for me to fully understand the differences, so I would appreciate a more thorough explanation.

[slavkap]

@daviftorres,
MANUAL means that those are Volume snapshots initiated by the user
GROUP - those are instance snapshots without the memory. But there were some changes in the latest CS versions for NFS storage and such DB records could be missing.

[daviftorres]

Volume Snapshots (aka Instance Volume Snapshot)

When attached to a running instance, the button Take Snapshot of the volume is not available by default.

If the global setting Kvm snapshot enabled (kvm.snapshot.enabled) is set to true...

…the Take Snapshot button becomes available even while the VM is running.

I understand that taking a snapshot of a volume attached to a running VM is potentially can compromise filesystem and application-level data consistency. This issue is solved by capturing the memory with the disk into the snapshot.

There is a confusion caused but the name Instance Volume Snapshot because it is in fact a Volume Snapshot technically is a Backup. In summary, I recommend changing the name to just Backup or Volume Backup.

[daviftorres]

Instance Snapshot

Volumes in CloudStack can be linked or unlinked, and it’s important to understand the difference.

Linked Volumes

• These volumes are externally linked to the template image.
• Essentially, the volume acts as an overlay on top of the template, similar to an external snapshot backed by the template.

Example: Linked Volume deployed from Thin Provisioning offer.

Here, the instance’s volume file is layered on the template image.

Example: Linked Volume deployed from Thin Provisioning offer and 2 Instance Snapshots.

Unlinked Volumes

These volumes are standalone files, independent of any template.

Example: Unlinked Volume deployed from ISO

Example: Unlinked Volume deployed from Sparse Provisioning.

Example: Unlinked Volumes with Internal Snapshot

Note: Mixing internal and external snapshots is not recommended.

This is where the ROOT disk provisioning type is typically configured, at the Compute Offering:

And also at the Add Disk Offering creation:

Provisioning types explained

• Thin Provisioning - Linked Clone
  • Instances are created as linked clones.
  • The template cannot be removed while instances depend on it.
  • If the template becomes corrupted or inaccessible, ALL dependent instances will also fail.
• Sparse Provisioning - Full Copy (Lazy Thick)
  • A full copy of the original image is created (not linked).
  • Disk space is allocated, but zero blocks are written on demand.
  • Faster to create than Fat provisioning.
• Fat Provisioning - Full Copy (Eager Thick)
  • A full copy of the original image is created (not linked).
  • All disk blocks are fully zeroed at creation time.
  • Slower to create, but disk is fully initialized upfront.

[ingox]

@daviftorres there are pro and con for the different snapshot and backup options. Find them here:

Instance snapshot:

• is stored on the primary storage
• is no backup; just to freeze a certain state if you do changes
• is using qcow2 functionality
• does snapshot of the entire VM including all disks
• can include memory as well
• can have differential snapshots
  Volume snapshot:
• is copied to the secondary storage
• is individual to each volume
• can be differential
• you won't get a consistent snapshot across all volumes of a VM for the same point in time
• can be copied to other zones automatically
• can be restored in other zones without the need of template now
  NAS backup:
• does backup the entire VM to an external location
• can be restored to other zones with access to this backup storage
• does full backup only at the moment - differential backup is hopefully to come soon

I hopefully have covered most of the important criteria to be considered.

[davift]

Thank you @ingox , for your reply. This is pretty much in the lines that I want to consolidate knowledge.

[slavkap]

Just to add that for the block storage providers there could be some differences. From what I know
VM snapshots:
Linstor/StorPool/PowerFlex - don't include the memory and the consistency of the snapshots is handled by the storage plugins (there is no freezing of the VMs during instance snapshots).
Ceph - don't include the memory. The consistency is handled while the VM is frozen
Volume snapshots:
Linstor - from what I know it could keep the snapshot only on primary or only on secondary storage via global setting
StorPool - could be kept only on primary storage or on both primary/secondary storage. Are incremental
Ceph - can keep the snapshot on primary storage only or on secondary storage

For the rest of the storage plugins I don't have much experience

[bilalinamdar]

Can this be used for Linstor primary storage ?

[ingox]

@bilalinamdar What exactly to do you want to do on the Linstor storage?

[bilalinamdar]

@bilalinamdar What exactly to do you want to do on the Linstor storage?

I mean to ask this backup incremental feature works with NFS but will it work with linstor drbd?

[daviftorres]

I can't say much about Linstor/StorPool but these global settings appear on 4.22 when compared with 4.20.

storpool.snapshot.recovery.from.remote.check Storpool snapshot recovery from remote check
Minimal interval (in seconds) to check and recover StorPool snapshot from remote.
Default 300

use.storage.replication Use storage replication
For snapshot copy to another primary storage in a different zone. Supports only StorPool storage for now.
Default false

If you find answers, please share here.

[slavkap]

@bilalinamdar, @daviftorres, StorPool does not support backups through CloudStack for now (we have internal methods for backups). I think and Linstor does not support backups either (probably @rp- can confirm)
@daviftorres, both setting are used in the feature you mentioned and for now it is working only for StorPool

[ingox]

@bilalinamdar see this article from @rajujith:

https://www.shapeblue.com/nas-backup-and-recovery-plugin/

The NAS Backup and Recovery Plugin currently supports NFS (Network File System) and operates as a straightforward B&R recovery provider, making it both storage and vendor-agnostic. This enables administrators to do efficient backups of running instances, regardless of their source storage. It also can be used to create backups of stopped instances stored on both NFS and local primary storage. However, please note that restore operations are currently supported only for NFS and local primary storage.