Error during retention pass or backup: Optimizing snapshot 'xxxxxxxx': Failed to locate N blob(s) required for <snapshot/xxxxxxxx>, unrestorable

Symptoms

Error during retention pass or backup:

CONFIG_TEXT: Optimizing snapshot 'xxxxxxxx': Failed to locate N blob(s) required for <snapshot/xxxxxxxx>, unrestorable

Cause

This error can happen if the storage provider or platform loses a file, a file was deleted manually from the Storage Vault, or if an exclusive lock was taken by a device for a dangerous operation, then the Storage Vault was manually unlocked while the job was running. 

In most cases, a retention job deleted some files after checking that no snapshot needed them, and the Vault was locked so the check was safe. Then a manual unlock was pushed by an admin, and another job re-used those files, unaware that the files were about to be deleted.

Resolution

The snapshots in the error messages are not restorable, or are partially restorable. Remove unrestorable snapshots from the Storage Vault. 

Snapshots can be deleted from a restore dialog. The IDs given in the error can be seen in the "delete multiple snapshots" sub-menu, allowing selection of all snapshots in the Storage Vault that give this error (<snapshot/xxxxxxxx>, unrestorable).

Data loss can be caused by unlocking a device while a job is running, or a device is asleep or suspended in the middle of a job. Avoid using the manual Storage Vault unlock unless:

• The Storage Vault supports safe unlock, or
• The admin unlocking the Vault can be certain that no active processes are running on any endpoint to the Storage Vault, and no device could be woken up and resume a retention pass.

If lock contention is a regular issue for the Storage Vault, consider breaking devices off into smaller Vaults. It is especially helpful to isolate consumer devices that sleep frequently, so they do not sleep with an active lock in place.