Wednesday, 13 April 2016
Backup Exec causing Hyper-V checkpoint build up
Recently when deploying a new network for a client, we installed Backup Exec 15 to run backups of their Virtual Machines from Hyper-V. The backups were working fine, but I noticed that for one VM, there was an excessive amount of checkpoints created, around the time that BE backs up each night.
If you have a look at the type of checkpoint, you will notice these are ‘recovery’ checkpoints, and can’t simply be deleted (.AVHDX files).
This creates problems when you’re wanting to edit the Virtual Machine settings (especially the hard drives of the Virtual Machine), as it will say there’s a checkpoint associated with the machine and you can’t edit it. In my case, I was wanting to expand the File Server’s storage, but wasn’t able to until I sorted out the checkpoints.
The other VMs has the same problem (recovery checkpoints showing up), but there was only one per VM. After looking online, and having to log a support call with Backup Exec, they informed me that this is a feature bug of Service Pack 3. The following link shows their support article:
This article explains that you need to change the backup setting from ‘Faster Processing Method’, to ‘Standard Processing Method’. After doing this, and running a new backup (confirming that it worked etc), I was able to run the PowerShell commands to remove the additional checkpoints.
Get-VMSnapshot -VMName ‘VMName’ -ComputerName ‘HOST’ -SnapshotType Recovery | Remove- VMSnapshot
This command worked well for all the VMs, except APFP01, which had the multiple checkpoints. I got the following error message when I ran that command:
CATASTROPHIC FAILURE! – It’s not actually too bad. It just means that you’re not going to be able to automatically remove and merge with the command that I had mentioned above. To fix this issue, we’re going to have to manually merge each .avhdx file. Firstly let me note that you will have to power down the virtual server before doing this. To do this, you have to go into the location of the Virtual Machines, and find the .vhdx and .avhdx. You need to find the newest .avhdx file, and rename the file extension to .vhdx (remove the ‘a’). Once you’ve done that, click on Edit Disk in Hyper-V, find the .vhdx that you’ve just renamed, click Merge, then click Merge with Parent. This will take a while depending on the amount of data that you’ve got. Once that has finished, find the next newest .avhdx file, and rename etc. Repeat the same process and merge with parent disk. Keep doing this until you’ve merged all the .avhdx files, and you’re just left with your original .vhdx file.
Now once you have done this, you’re still going to have the checkpoints showing up in Hyper-V. I haven’t been able to remove these checkpoints from Hyper-V using any of the commands used within that Backup Exec KB article. To get the server back up and running, you will need to create a new Virtual Machine, and just set the drives to the existing .vhdx files. Once you boot up, this will run exactly the same as the previous server. You can then delete the old, problematic VM from Hyper-V. Just keep in mind that the Virtual Disks you’re using are still in the old folder, so don’t delete the folder from Hyper-V.
Something that I came across when I was doing this, was that the second drive didn’t attach to the server fully. It showed the C: drive and booted without problem etc, but the E: drive was missing. I just had to go into Disk Management and set the disk to Online, as it was offline for some reason.
After doing all of this, I have been able to expand the drives without issue, and there have been no further checkpoints created from Backup Exec (after changing that setting).
