Error in reindex job: corrupt 0x00 byte in header

Symptoms

Error in reindex job: 

PRODUCT_ERROR: corrupt 0x00 byte in header

A similar error with a different header byte can also appear in a backup job, for example:

PRODUCT_ERROR: MarkState: Retried 1 times over 0:00: failed to read header: 0x4D: unknown header (created with new version of application?)

This error may occur in other job types.

Cause

The first bytes of Comet's encrypted files contain metadata needed to determine the encryption and compression used to create it.

0x00 is never used by Comet, and indicates that the file is corrupt. It is likely that the entire file is zero-filled.

The cause of corruption is likely a failure of the storage server.

The unknown header error with another byte, such as 0x4D, can occur due to data corruption in the Storage Vault. The resolution is the same in both cases.

Resolution

To keep backups running while you investigate the existing Storage Vault, create a new Storage Vault and redirect the schedules to it.

Delete all pack files reporting an unknown header, then start recovering your Storage Vault for data loss.

To delete all corrupt files in your Storage Vault, edit the path in line one and run:

Note that this is a very destructive command, ENSURE that you are running it only in your Storage Vault directory

cd /path/to/storage/vault/
find . -type f ! -name 'config' -print0 | while IFS= read -r -d '' f; do
  hash=$(sha256sum "$f" | cut -d' ' -f1)
  name=$(basename "$f")
  if [ "$hash" = "$name" ]; then
    echo "$name ok"
  else
    echo "[!!!] $f MISMATCH $hash — deleting"
    rm -f -- "$f"
  fi
done

Perform a re-index of the Storage Vault.

On a success, run a retention pass. 

Those two operations in that order perform a full referential integrity check, checking that each snapshots can reach its data.

Delete any snapshot that returns unrestorable, as it depended on the deleted pack(s).