How Restow proves your backups can be restored
A backup is only worth something if it comes back when you need it. A backup job that finished says that data was written, not that it can be read again. Restow therefore keeps two states apart: "backed up" and "proven restorable". A backup only counts as restorable once Restow has read it back and found it identical to what the backup recorded when it was made.
What a restore check is
When Restow makes a backup, it records a fingerprint of every item: a SHA-256 checksum that changes completely if even one byte changes. A restore check takes a random sample from the newest backup, reads it back through the same path a real restore uses, including decryption, and compares what comes back with those fingerprints. The check runs on the server and reads from your storage. It does not write anything into a mailbox or over the files on a machine.
How often it runs
- Mailboxes, OneDrive and IMAP accounts: every week (on Sunday at 03:00 in the recommended schedule, which you can change) and, while that schedule is on, after every backup. You can also start a check at any time. A check that could not complete is repeated a few minutes later, up to three times, and otherwise at the next scheduled check.
- Servers and clients: after every new good backup. If the server's check could not complete, it is tried again every hour until it does. The restore test on the machine is offered again after 1, 2, 4, 8, 16 and 24 hours, as long as the backup it tests is the machine's newest; after that, the next backup brings a new test.
- File shares: after every new restore point (since 0.4.0); the share's repository also gets a weekly restic check of 5 percent of its data.
- Proxmox VE VMs and containers (preview): within minutes of every new restore point (since 0.4.2, on by default and switchable for the installation), and the newest restore point of every guest at least every week. "Check now" in Recovery readiness starts a check by hand.
- Proxmox VE boot tests (preview): off by default; when a job turns them on, weekly on the newest restore point or after every backup, or by hand with "Boot test now".
- The storage itself: a sample every week and every stored file every month (see "The storage check" below), and for servers and clients a weekly check of one twentieth of each machine's backup data, so all of it is read over about five months.
What is checked for mailboxes and OneDrive
From the newest backup of each mailbox, OneDrive or IMAP account, Restow draws 20 mails, 20 files and a few calendar entries and contacts at random. Each one is read back and checked piece by piece: every stored piece has to decrypt and has to be the piece the backup names, and the whole item has to have the size and the fingerprint the backup recorded. If the storage check found damaged data that this backup uses, the backup counts as not restorable even when the sample did not touch it.
What is checked for servers and clients
After every good backup, the agent on the machine picks up to 20 random files that did not change while the backup ran and records their fingerprints. The Restow server then reads exactly those files back from the backup and compares the fingerprints. The agent also restores the same files once into a temporary folder on the machine, compares them there and deletes the copy, so the restore is proven on the machine as well. The agent reports what it found, file by file, and the Restow server judges that report by the same rule as its own check (below).
What is checked for SharePoint sites, file shares and Proxmox VE
A SharePoint site is checked like a OneDrive: Restow samples files of its document libraries and reads them back. For a file share, every new restore point is checked by reading up to 20 sampled files back and comparing them with what the backup recorded, and the share's repository gets a weekly restic check of 5 percent of its data; Recovery readiness rates a share by its newest restore point. For Proxmox VE (a preview), a job can also run boot tests, off by default: the guest's node restores the newest restore point as a new guest cut off from the network, starts it, waits for the guest agent and deletes it again. Green means the guest agent answered, yellow that the guest only started, red that the restore, the start or the boot failed; what runs inside the guest is not checked. Besides that, every new restore point of a guest is read back within minutes of its backup (a sample of its data blocks compared with the recorded SHA-256, for a container a restic check of a subset), and the optional monthly restore check of a Proxmox VE guest and the weekly check still exist.
What green, red and "Unverified" mean
- Ready (green): only after a restore check of the newest backup passed. Green is never shown for a backup that has merely finished.
- Attention (yellow): the backup can be restored, but something needs a look: the newest backup is two days old or older, there was nothing in it to check, a machine's backup was only partly complete, or the latest storage check could not read a pack this backup uses (a connection or permission error, which is not proven damage).
- Not restorable (red): there is evidence that a restore would not be complete today: an item came back different or missing, stored data is damaged without an intact copy, a pack could not be read in three storage checks in a row over at least 24 hours, or the newest backup is a week old or older, so recent data would be lost. Alerts report that last case as "Backup too old", not as a failed check. One rule applies to every mailbox, OneDrive, IMAP account, server and client: a check rates red only with evidence, that is data that is missing, that differs from what the backup recorded, or that does not decrypt or decode. For servers and clients that means a fingerprint that differs, or restic reporting that the file or the data it needs is missing or damaged.
- Unverified: a backup exists, but no check of exactly that backup has completed yet. A newer backup after a green check starts as "Unverified" again. A check that could not complete (the storage did not answer, timed out or throttled the check, the backup was busy, the check program could not start) rates nothing and sends no alert: in the list of runs it reads "Not completed, will be retried", neither red nor green, and it is repeated.
Two cases are rated differently since 0.4.7. An object removed from its source (a mailbox deleted in Microsoft 365, for example) is rated only by whether its newest kept backup restores, not by its age, because nothing backs it up any more and retention keeps that backup. And an object that Restow no longer backs up (account disabled, no licence, locked by Microsoft, or removed) first waits amber for an administrator's confirmation: no failure, but the tenant's readiness stays at least yellow, never green, until it is confirmed with "Yes, intended".
The storage check
Backups are stored in larger files called packs. When a pack is written, Restow records its size and fingerprint. The storage check reads packs again, on every repository that holds a copy, and compares them: a random sample every week and every pack every month. A damaged copy is repaired automatically from an intact copy on another location. A pack that is damaged everywhere marks every backup that uses it as not restorable at once, raises an alert, and later backups write that data again from the source, as far as the source still has it.
Since 0.4.2 the check tells damage from a read that failed. A pack it could not read for the moment (a download cut off mid-transfer, a reset connection, a timeout, the storage asking to slow down, or denied access) proves nothing about the data. The check reads it again, up to three times, and a pack that stays unreadable is listed as "temporarily unreadable" in Recovery readiness, Storage integrity, with the error, how many checks in a row could not read it and when. Backups that use it show "Attention", not "Not restorable". Every later check reads the flagged packs first, "Re-check these packs" reads only them again, and when they read intact again the finding ends. Only a pack that is proven damaged (missing, wrong size or checksum), or that could not be read in three checks in a row over at least 24 hours, makes the backups that use it not restorable.
What you see
- Recovery readiness: one table with every mailbox, OneDrive, IMAP account, server and client, its state, when it was last checked and why, with a report per check that lists what was read back.
- Alerts: rules that send an e-mail, a bell entry or a signed webhook when a check fails, needs attention or recovers, and when stored data is damaged.
- Overview and reports: the share of backups proven restorable and the verification history on the Overview (tab Statistics) and, with Business and Service Provider, scheduled summary reports.
- For IT service providers: with Service Provider, the provider view of the Overview across tenants, and the integration API reports the same states to your RMM or PSA, so a failed check can open a ticket.
Honest limits
- A sample is a sample. 20 mails out of 50,000 prove that the restore path and those mails work, not every single mail. The monthly storage check, which reads every stored pack, closes much of that gap, and a full check that reads every item of a backup can be requested when needed.
- The check proves that your backup can be read back intact. It does not restore into a live Microsoft 365 mailbox, OneDrive or IMAP account. A test restore into a separate target is not part of 0.1.0, so restore a test mailbox yourself from time to time.
- Microsoft 365 backup and restore have not yet been run against a real Microsoft 365 tenant in the release checks; they are covered by tests against a simulated Graph API.
- For servers and clients the sample only covers files that did not change during the backup. Whether a database dump is consistent depends on how it is made before the backup.
- A check finds damage; it does not prevent deletion. On S3 with Object Lock, immutable backups (off until an administrator turns them on per repository) make the storage itself refuse to delete locked data; a directory or an NFS share gets only a deletion delay, which is not immutability. Beyond that, protect the storage itself, for example with copies on a second location.
- The file share checks ran against Samba in the release checks, not against Windows Server, Synology, TrueNAS or real NFS exports; the boot tests, the restore check after every backup and the file restore of Proxmox VE guests ran against the simulated node of the test suites, not on a Proxmox VE cluster.
The technical details, with every checksum, sample size and rule as the code applies it, are in the documentation: how restorability is verified, in detail. See also the verification feature page.
Want the method behind this, for any tool? See how to prove that a backup can be restored.
Frequently asked
Does a finished backup job count as proof?
No. A backup job that finished only says that data was written. A backup counts as restorable once a restore check has read it back and found it identical to what the backup recorded. Until then it is shown as "Unverified", never as green.
Does the restore check write anything into my mailbox or onto my server?
No. The server reads the backup from your storage and compares it. For a server or client, the agent also restores a few files once into a temporary folder on the machine, compares them and deletes the copy. Nothing in a live mailbox or among the machine's existing files is touched.
What happens when a check cannot run?
It rates nothing, for mailboxes as for servers and clients. Red needs evidence: data that is missing, that differs from what the backup recorded, or that does not decrypt or decode. A check that could not complete for any other reason, for example because your storage did not answer (a network error, a timeout, throttling), the repository was busy or restic could not start, changes no rating and sends no alert; in the list of runs it reads "Not completed, will be retried". It is repeated: for mailboxes a few minutes later and otherwise at the next scheduled check, for the server's check of a machine every hour, and the restore test on the machine after 1, 2, 4, 8, 16 and 24 hours, as long as the backup it tests is the machine's newest. Such a check is never shown as red or green.
As an IT service provider, how do I see the state of all my customers?
Each tenant has its own Recovery readiness page. With the Service Provider edition the provider view of the Overview rolls the tenants up, alert rules can send a mail or a signed webhook into your ticket system when a check fails, and the integration API reports the same states to your RMM or PSA.