Server and client backup
Restow backs up Linux and macOS servers and client machines with a small agent that drives restic, into your own Restow instance. It is part of the Community edition, so every edition has it, and it is available since 0.1.0 (beta). The backups are file-level, stored deduplicated and encrypted, and they are checked the same way mailbox backups are: a backup that has never been read back is not counted as proven.
What it does
The agent is a small Go program for Linux (systemd, x86_64 and aarch64) and macOS 13 or newer (Intel and Apple Silicon). It backs up the folders you choose with restic 0.19.1 into a repository on your Restow instance, deduplicated, compressed and encrypted. It is file-level today: no disk images and no bare-metal restore yet. Restoring a whole machine means reinstalling the operating system and restoring the files; image backup with bare-metal restore is planned.
restic is the well-known open-source backup program behind the agent: fast, deduplicating, encrypted, with an open repository format. Restow ships its unmodified release, checked against pinned checksums, so a machine's backup is a regular restic repository that restic itself can open with the repository password, even without Restow. restic is an independent open-source project (BSD-2-Clause, restic.net); Restow is not affiliated with it.
Virtual machines and containers on Proxmox VE are not backed up with this agent but through Proxmox's own Backup Provider API, as a preview, see Proxmox VE backup.
Backup jobs for machines
Machines are backed up through backup jobs. A job says what is backed up, when, into which repository, for how long, and how often a restore check proves it, for many machines at once, with overrides per machine. For servers and clients you pick the folders to back up and exclude file types. You can limit how much bandwidth a backup may use, within a time window that you set. Hooks, bandwidth and schedule of a machine that is in a job are set in the job.
A machine backs up once it is in a job. Every machine without a job
is marked "Without backup", counted in a banner and offered "Create a
job" or "Add to job", and a newly enrolled machine waits for a job
instead of backing up on its own. Servers back up their application
data (/opt, /usr/local,
/var/lib) by default. A table of all machines shows the
host name, the person each machine is assigned to, the last backup and
the recovery readiness.
Since 0.3.5, for Linux machines, a notice in the job and in file
restore names the usual data folders (/opt,
/srv, /var/lib, /home and
others) that the job's paths leave out, with a button to add them.
Since 0.4.0, the agent also reports which of a fixed list of Linux
data folders (/opt, /usr/local,
/srv, /var/www, Docker and Podman volumes, the
PostgreSQL, MySQL and MongoDB folders under /var/lib)
exist and hold files, names only. The job editor then names the folders
the job's machines really have and the job leaves out, and a machine's page
has a button to add each such folder to the job. With older agents the notice of
0.3.5 stays. A machine's page also says what it runs as: container,
virtual machine or physical, with or without a desktop. A Linux
machine enrolled with a client token whose agent reports a container
or a virtual machine without a desktop gets the server profile, because
it runs services; the page says why.
Running jobs update on their own, without reloading the page, and a run opens a detail panel with charts of its throughput. The history lists every run, including the runs of the agents on your servers and clients, with filters by kind (backup, restore, restore check, export, import, maintenance) and by job.
Agent policies
Since 0.4.0, a few settings of every agent can be set once instead of per machine, on three levels: the installation (Installation, Agent policy), the tenant (the tenant page, Agents) and the backup job (the job editor). Each level may override a setting unless the installation locks it, and every view says where a value comes from and whether it is locked. The settings are:
- a backup window: weekdays and a time range, also across midnight, in the time zone of the job's schedule, in which scheduled backups, catch-ups and retries may start. A running backup continues, and "Back up now" ignores the window;
- catch up missed backups as soon as the machine is on again, or in the next backup window, optionally only up to a maximum age;
- the upload limit of a job that has none of its own (a job can also choose "Unlimited", which sets a policy's limit aside unless the installation locks it);
- only on AC power;
- the number of automatic retries, 0 to 10.
Without a policy the agents behave exactly as before. The page of a server or client shows the policy that applies to it, each setting with its source (default, installation, tenant, backup job or the machine itself) and whether it is locked, read-only, and why a due backup waits next to its next planned start: for the backup window, for an update to finish, for AC power, or to catch up in the next window. Every change is written to the audit log. The window, catch-up, retries and the waiting reason need agent 0.4.0; the upload limit and AC power reach older agents too.
While an update of Restow pauses new runs, the agent 0.4.0 starts no
scheduled backup, retry or catch-up and waits for the pause to end; the
wait does not count as a missed or failed backup. A "Back up now" request
stays queued on the server until the pause is over, also for older agents,
and restow-agent backup-now on the machine waits with a
message. restow-agent status shows the policy and why a due
backup waits. Agents older than 0.4.0 do not hold their own schedules
during an update: a backup such a machine starts by itself in that time
runs, and may fail if the API restarts under it.
How a machine is added
In the Restow web interface you add a new server or a new client.
Restow shows the install command and, separately, a one-time token
that is valid for 24 hours and works once. You run the command with
sudo on the machine; the script asks for the token and reads it
without showing it, so the token appears neither on the command line
nor in the process list. For unattended installs (RMM, scripts) the
script reads the token from a file only root can read. The install
script is served by your own instance: it downloads the agent and
restic from that instance, checks the maintainer's signature over
the release's checksum list and the SHA-256 of both binaries before
it runs anything, installs them in a folder only root can change
(/opt/restow-agent/bin on Linux,
/Library/Application Support/Restow/bin on macOS), sets
up the service (systemd or launchd), enrolls the machine and starts
it. Running it again repairs or upgrades in place.
The agent opens outbound HTTPS connections to your Restow instance and to nothing else, and it opens no port on the machine. On macOS you grant the agent Full Disk Access; without it, protected folders are skipped and the backup is reported as partial, with the reason in the run log.
Why the agent cannot destroy its own backups
Append-only is enforced by your Restow instance, not left to the agent's good behavior. The agent writes to a restic repository that the instance exposes in append-only mode: it can add backups but cannot delete or overwrite them. Restow's own end-to-end tests check that forget, prune, delete and overwrite with the agent's access are refused and the stored bytes stay unchanged. Retention, pruning and integrity checks run only on the Restow server.
Append-only protects against the machine. Against someone on the
Restow server or holding its storage key, the tenant's primary
repository can be made immutable since 0.4.0 (S3 with Object Lock): the
storage then locks the data, snapshots,
keys and config objects of every machine's
repository, but not restic's lock and index files. It is off until an
administrator turns it on, and the installation's default repository
cannot be made immutable. See
storage.
The agent never holds credentials for the storage target, such as S3 keys or mount paths. It knows only its own agent secret and the password of its own repository. Every machine has one repository and one random password; the server keeps that password encrypted with the tenant key, so an administrator can still restore when the machine itself is gone.
Restore
A restore on the machine always goes into a new folder and never over existing files (restore operators can do it too since 0.4.0): the target must not exist or must be empty, otherwise the run fails and nothing is touched. The alternative is to browse a backup in the Restow interface and download files or folders as a ZIP: a ZIP can hold up to 10,000 paths, and folders are browsed page by page ("Load more"). A restore task into a new folder on the machine takes up to 200 paths. Browsing, downloads and restore requests are written to the audit log. The repository is a plain restic repository, so a restore without Restow is possible with the repository password. An administrator can have Restow show that password (it asks for a recent sign-in and writes the access to the audit log). The password is also stored next to the repository, sealed with the tenant key, so with the installation's master key and the storage alone the standalone restore tool recovers it, without a running Restow. The general restore rules are on the restore page.
Proven restorable
After every backup the agent records the SHA-256 of up to 20 random files. Later the server reads those files back from the snapshot and compares the hashes; the result is green only when they match. That result feeds Recovery Readiness, the same status used for mailboxes, so a machine shows as proven only when its latest backup was read back. More on the approach is on the verification page.
Every new restore point of a machine is also judged for signs of ransomware: the amount of change, deletions, renamed extensions, known names, sampled files that look encrypted and data that neither compresses nor deduplicates. The activity report needs agent 0.4.0. A suspicious restore point pauses retention of that machine until an administrator resolves it. See ransomware detection.
Every week the server also checks each repository, reading one twentieth of the data per run, so the whole repository is read over roughly five months. Retention runs on the server as well; the default keeps 30 daily, 12 weekly and 12 monthly backups and can be changed per machine. The server decides what to delete from its own records of the backup runs, not from the times the machine writes into its snapshots.
Each machine has a storage budget, 2 TiB by default and adjustable per machine, and all machines of a tenant together 20 TiB. An upload that does not fit is refused while restores keep working, and an alert warns at 90 percent.
Stopping a backup, and updating the agent now
Since 0.4.2, Stop backup on a machine's page and in the run
drawer stops a running backup, and Withdraw backup takes back
a "Back up now" that has not started. Whoever may start a backup may stop
it; every stop asks first and is written to the audit log, with who asked.
The agent receives the order with its next contact (every minute while it
backs up, otherwise every five minutes), stops restic cleanly, so that no
snapshot and no lock are left behind, and a post-hook runs as usual. The run
ends as cancelled, not as failed: no failure alert, no
job.failed webhook, the last backup and the count of failed
backups stay as they were, and the next scheduled backup starts normally.
It needs agent 0.4.2; an older agent does not know the order and finishes
its backup.
Besides the self-update every six hours, Update to
<version> on a machine's page asks for the update at any time,
and sudo restow-agent update on the machine installs the offered
release at once, through the same signed download, restarts the service and
says what it did. While a backup, a restore or a restore test runs it
refuses, and the agent installs the update itself once idle. After the
server runs a new release, agents follow it while new runs wait, 15 minutes
at most; the Updates tab shows their progress and names the command for an
agent that was not reached.
Removing a machine
Since 0.4.5, a revoked machine can be removed from Restow, with or without its backups. Remove from Restow in the danger zone of the machine's page offers Remove from the list, keep the backups until they expire (the newest backup plus the longest rule of the machine's retention, read as time) or Delete the machine and its backups. Both need the hostname typed and a recent sign-in, and the machine has to be revoked first. The machine leaves every list, picker, dashboard and count and its backup job; its runs, reports and audit entries stay. The worker deletes the machine's restic repository on every target that holds files of it: the primary, its copies and retired primaries. Deleting at once is refused while a legal hold over the tenant or an unresolved suspicious restore point holds the backups, and the worker waits for such a hold when the backups expire. Objects that the storage still locks (S3 Object Lock, a directory's deletion delay) are deleted when their lock runs out; the machines page lists the removals that are still pending, with the date, the hold or the lock. Every step is audited.
Until the backups are deleted, the page of a removed machine offers them for browsing and the ZIP download (since 0.4.6), and the list of removed machines links "Restore files"; restoring onto the machine and every other action stay refused. An uninstall that waits for a machine that is off offers Revoke now, and a machine without contact for longer than a number of days that you set (30 by default, under Alerts of the machine) suggests revoking it.
Alerts
Restow raises an alert, through the same channels as every other job (notification bell, email, webhook), when:
- a server has been silent: by default no contact for more than 2 hours;
- a client has had no good backup for a while: by default 7 days, because a laptop is off at night;
- a backup is overdue (
backup.overdue): the alert rule takes the schedule of the job into account and can set a deadline of its own, from 24 to 720 hours; - a backup fails, or a restore on a machine fails;
- a restore test fails, or a repository check finds damage.
Warnings say why single items were not backed up and can be acknowledged with a note. Webhook alerts can also post to Discord, Slack or Microsoft Teams, see Microsoft 365 and IMAP backup.
Profiles and hooks
The Server profile backs up daily (the default time is 22:00). The
Client profile backs up when the machine can reach your Restow
instance, at most once every 4 hours, and can be limited in bandwidth
and set to run only on AC power (power detection is best effort).
Optional pre and post hooks run commands around a backup, for example
a database dump. They run as root, so the machine decides: hooks are
off on a new machine, and only root on the machine can allow them,
either only named scripts that root put into
/etc/restow-agent/hooks.d
(sudo restow-agent hooks scripts) or any command
(sudo restow-agent hooks any). Restow cannot change
that setting: it shows each machine's policy and refuses hooks the
machine would not run. Only with any can whoever changes
a machine's configuration in Restow run arbitrary commands as root on
it, so prefer scripts. Setting or changing a hook also
asks for a recent sign-in.
Stated limits, not hidden ones
- No Windows agent yet. It is planned, see the roadmap.
- No LVM, ZFS or btrfs snapshots yet, and no APFS snapshots on macOS. Consistency comes from the optional hooks (a database dump, for example); files that change while they are read can end up inconsistent in the snapshot.
- No mTLS yet: each agent authenticates with its own secret over HTTPS. A stolen secret does not allow deleting anything.
- While a machine of the tenant is active, Restow refuses to change the organization's primary storage target, because an active agent keeps writing to the old primary. To change it, revoke or uninstall the machines, change the primary target and enrol the machines again, which creates new repositories. Since 0.4.2 a "move" copies the repositories of the revoked machines to the new primary too, verified by SHA-256, and a machine's page names the repository its backups land in (and enrolling a machine warns when that is the server's own disk); with "keep", they stay on the previous target and remain browsable and downloadable. See storage.
- What the release checks cover: the agent's install script runs in the release checks only inside a Linux container. The macOS installer was tested on a Mac outside the release checks, and the launchd service itself has not been run for real yet.
- No image or bare-metal backups yet: the backup is file-based today. Image backup is planned.
- The agent runs as root, because it has to read every file it backs up.
-
The agent checks for updates a minute after it starts and then every
6 hours, and when the server asks (an update of the server, or "Update to
<version>"), and installs only
releases the maintainer signed: it checks the Ed25519 signature over
the release's checksum list with the key compiled into it, then the
SHA-256 of every file, so a compromised instance cannot make it run
a binary of its own. Automatic agent updates can be paused per
tenant. The first installation trusts the install script your
instance serves, as any
curl | shdoes: compare the release key fingerprint the script prints with the published one,SHA256:2LB7RPpIS0wqbcZ2NA8MjuQN7ZfFPqBX3CwrGFSu76o(Ed25519, also in the release notes). - Restow is a beta. Run it beside your existing backups until you have restored from it yourself.
Documentation
The administrator guide has the overview and the install steps, including the macOS permission.
Frequently asked
Does the agent need an inbound port on the machine?
No. The agent only opens outbound HTTPS connections to your own Restow instance and to nothing else, and it opens no listening port. Your instance never connects to the machine: tasks such as a restore request travel in the answer to the agent's regular check-in.
Can the agent delete or change its own backups?
No. Your Restow instance enforces append-only on the server side: the agent can add backups but cannot delete or overwrite them, and it never calls forget or prune. Retention and pruning run only on the Restow server. The agent also holds no credentials for the storage target, only its own agent secret and the password of its own repository. One limit is stated plainly: there is no client certificate yet, so a stolen agent secret (root on the machine) would allow writing new backups to that machine's repository and reading it, but not deleting anything.
Does Restow back up Windows machines?
Not yet. The agent ships for Linux and macOS, and has done since 0.1.0. A Windows agent is planned and listed on the roadmap; nothing Windows-specific is part of the current release, and the server refuses to enroll a Windows machine.
When does a machine start backing up?
As soon as it is in a backup job. A newly enrolled machine waits for a job instead of backing up on its own, and every machine without a job is marked "Without backup", counted in a banner and offered "Create a job" or "Add to job". Machines in no job do not count as protected anywhere, in the overview, the provider view, readiness or reports.
Does Restow tell me when a machine job leaves out the usual data folders?
For Linux machines, yes. Since 0.3.5 a notice in machine jobs and in file restore names the usual data folders that the job's paths leave out, with a button to add them. Since 0.4.0, with agent 0.4.0, the agent reports which of a fixed list of data folders (/opt, /usr/local, /srv, /var/www, Docker and Podman volumes, the PostgreSQL, MySQL and MongoDB folders under /var/lib) exist and hold files, names only, so the job editor names the folders the machine really has and the job leaves out, and the machine's page has a button to add each such folder to the job.
Can I limit when machines back up, and have missed backups caught up?
Yes, since 0.4.0, with agent policies on three levels (installation, tenant, backup job): a backup window in which scheduled backups, catch-ups and retries may start, catching up missed backups when the machine is on again or in the next window, an upload limit, only on AC power and the number of automatic retries. A running backup continues past the end of a window, and "Back up now" ignores it. The window, catch-up and retries need agent 0.4.0.
What does an agent do while Restow updates?
Agent 0.4.0 starts no scheduled backup, retry or catch-up while an update pauses new runs, waits for the pause to end, and does not count the wait as missed or failed. A "Back up now" request stays queued on the server until the pause is over. Agents older than 0.4.0 do not hold their own schedules during an update, so a backup such a machine starts by itself in that time may fail if the API restarts under it.
Can I stop a running backup of a machine?
Yes, since 0.4.2. "Stop backup" on the machine's page and in the run drawer stops the running backup, and "Withdraw backup" takes back a "Back up now" that has not started. The agent, from version 0.4.2, receives the order with its next contact, which is every minute while it backs up, stops restic cleanly (no snapshot and no lock is left behind), and the run ends as cancelled, not as failed: no failure alert, and it does not count as the last backup. Every stop asks first and is audited. An agent older than 0.4.2 finishes the backup.
How do I update the agent right now?
On the machine's page, "Update to <version>" asks for the update at any time, or run sudo restow-agent update on the machine (agent 0.4.2 or later, Linux and macOS). It fetches the release the server offers through the same signed download as the automatic self-update, restarts the service and says what it did; while a backup runs it refuses, and the agent updates itself once idle. After a server update, agents follow by themselves while new runs wait, 15 minutes at most; the Updates tab names the command that fits each agent that was not reached.
Does server and client backup cost extra?
It is part of the Community edition and therefore available in every edition. For what the editions cost, see the pricing page.
Read more in Learn: how daily, weekly and monthly retention works (grandfather-father-son) →
Read more in Learn: immutable backups, WORM and air gaps (append-only in context) →