XenServer 7.1 can't delete a virtual disk that failed a move between storage repositories, VM is broken / will not start

I've run into this issue more than a few times, so here's just enough info to get through this without blowing your virtual infrastructure apart. To avoid this issue in the future, avoid these common mistakes while transferring a VDI between host machines:

• Don't exit XenCenter while moving a VM between Xen hosts
• Don't unplug the switch that connects Xen hosts to a Storage Repository
• Don't shut down the Storage Repository or XenServer hosts during a virtual disk transfer
• Make sure each VM's virtual disks are named relating to its attached VM's (for sanity)

Now that we got that out of the way, lets talk about how Xen sees a VM and its virtual disk.

XenServer manages the host machine and connects the host's resources into a resource pool. This is usually the CPU, RAM and network for a virtual machine. Xen auto-assigns a Universally Unique Identifier or UUID to every virtual component so that it can keep track of all these resources without naming conflicts. While local storage is an option, a VM's virtual disk usually resides somewhere else such as a network-attached storage repository or SR. Once a SR is connected to Xen, the host server will connect to this SR and create files that act as virtual disks for VMs, just like an ISO file is used to create a CD or DVD. Xen calls these files Virtual Disk Images, or VDI. When Xen connects a VDI to a VM, it creates a disk configuration file known as a Virtual Block Device or VBD that tells the VM how to use the VDI as a disk.

When a VDI "disk" is being moved between Storage Repositories in a pool, its actually copying that VDI file from one storage location to another. Once it's copied from source to target and verified, Xen switches the VM to use the "disk" in the new target storage location, and the source VDI is deleted. Sometimes accidents happen, and the copy operation doesn't complete. The copy of the VDI on the new storage location is incomplete, while the source VDI on the original storage location is stuck under the control of the XenServer host that didn't finish the copy. Normal quick-fix operations (like restarting the toolstack) won't fix this issue, as Xen doesnt know what you want to keep! There is no real automatic answer when there could be valuable information in the broken copy and the source is no longer available.

Usually, the VM is in a broken state, and there could be several copies of the same VDI in various states, depending on how many times a move was attempted to "fix" the issue and how many pre-existing snapshots there are before the move. What do you keep? What to discard? The following instructions should help you make that decision.

• Identify the original VDI (or VDIs) prior to the move and reattach them to the VM
• Snapshot and Export the VM (make a backup)
• Move the VM to the new storage location
• Identify the snapshots of the VDI (or VDIs) and move them to new storage location
• Identify the VDI (or VDIs) that are locked to the Control Domain host server
• Remove the locks
• Delete the undesirable VDI (or VDIs) or move to a separate location

Identify the original VDI
Run these commands from the console window of a pool member host:

When a VDI is under control of the host server, the Name Label is temporarily changed to say Control domain on host:<host server name> The Name Label will be restored to the name of the VM after it is released.

xe vbd-list | grep -C3 "Control domain on host"

This will show you which virtual machines have stuck VDIs, and the corresponding UUID of each VDB will be listed as "UUID ( RO) : <UUID of VBD>". Copy the UUID of each listed VBD (connector object between VM and VDI) and use it to make the VBD inactive (it may already be inactive),

xe vbd-unplug uuid=<UUID of VBD>

Then break the connection between VDI and Control Domain host by destroying the disk config file:

xe vbd-destroy uuid=<UUID of VBD>

Now, reattach these VDI using XenCenter. Select the desired VM in the search column, then select the Storage tab. Under Virtual Disks, click the "Attach Disk" button and add the VDI with the same Name Label as the VM its being reattached to. Power up the VM and verify the system is functional.

Snapshot and Export the VM

Create a snapshot of the VM and then right-click the snapshot and select "Export to file..." Save the OFV export to your designated XenServer VM exports location, or to your desktop.

Optionally (for irreplaceable data), re-import the VM to make sure it boots up and verify the data is intact. In the upper lefthand corner of XenCenter, select File > Import and load the VM back into Xen.

Move the VM to the new storage location

Now that the VM has been exported safely,

Identify the snapshots of the VDI
This is pretty straight-forward. Just look in XenCenter in the storage repository, and any Virtual Disks that don't have a file size listed are snapshots.

Remove the locks