BorgBackup: Securely backup your homelab offsite on a schedule

Self hosting is great, but with great power comes great responsibility - the responsibility of your data... and the backups of your data. You do have backups, right?

What is BorgBackup? #

BorgBackup is an open-source secure backup utility for Unix-like operating systems.

The backups are:

Secure and authenticated.
Compressed with either LZ4, zlib, LZMA or zstd.
Deduplicated. The same block of data will not be stored more than once.
Mountable with FUSE.

Prerequisites #

Access to an SSH server somewhere reliable. Could be a friend's homelab or one from here. 'Offsite' is key.
Disk space on that server equal to the amount of data you wish to backup with some headroom for future snapshots.
Your machine's root user's SSH public key authorized on that server so Borg can connect without human interaction.
That server's SSH host key trusted and stored in the root user's known_hosts file. To do this, run ssh-keyscan -p port hostname.tld then append the output to ~/.ssh/known_hosts.

ℹ️

While Borg can be used by itself, I like to use a nifty little helper called borgmatic to make configuration easier. borgmatic will read your configuration file and run borg accordingly so you don't have to remember borg commands. It will also prune your backups which borg does not perform automatically.

Installing Borg and borgmatic #

We'll assume you're running Ubuntu 20.04+. For other distros, refer to the official installation instructions here.

We'll also assume you're logged in as root, which is highly recommended. This is to allow borg to access system directories such as /etc, /var. Same applies when restoring. To correctly restore file permissions and set ownership, borg needs to be run as root.

 1## login as root
 2sudo su
 3## borg
 4add-apt-repository ppa:costamagnagianfranco/borgbackup
 5apt update apt install -y borgbackup
 6
 7## borgmatic
 8apt install -y python3-pip
 9pip3 install --upgrade borgmatic
10
11## logout and log back in as root
12## then verify installation
13borg -V
14borgmatic --version

Borgmatic will not be upgraded to newer versions automatically. Upgrade manually with pip3 occasionally to keep it up-to-date.

1pip3 install --upgrade borgmatic

Configuration #

Generating #

We'll generate a sample configuration file

1generate-borgmatic-config

This generates a sample configuration file at /etc/borgmatic/config.yaml.

You should edit the configuration file to suit your needs, as the generated values are only representative. All options are optional except where indicated, so feel free to ignore anything you don't need.

Mine for example looks like this:

 1location:
 2    source_directories:
 3        - /home
 4        - /etc
 5        - /root
 6        - /var/lib/docker/volumes
 7        - /crons
 8        - /certs
 9        - /mnt/nvme/prism
10        - /media/prismoriginals
11        - /mnt/nvme/jellyfin
12
13    repositories:
14        - ssh://user@host.tld:23/./borg ## offsite
15        - /mnt/backups/borg ## attached ext drive, redundant, optional
16
17    exclude_patterns:
18        - /home/*/.cache ## exclude caches
19        - /root/.launchpadlib
20        - /root/.vscode-server
21        - /root/.cache
22        - '*/.vim*.tmp' ## exclude tmps
23        - /mnt/nvme/jellyfin/logs ## exclude logs
24        - /mnt/nvme/jellyfin/cache
25
26    borgmatic_source_directory: /mnt/nvme/borgdata/.borgmatic
27    exclude_caches: true
28    keep_exclude_tags: false
29    exclude_if_present:
30        - .nobackup
31
32storage:
33    checkpoint_interval: 300 ## seconds
34    borg_base_directory: /mnt/nvme/borgdata ## will contain keyfile
35    encryption_passphrase: vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS ## use long pass
36    archive_name_format: "homelab-{now:%Y%m%d_%H%M}" ## use preferred format
37
38retention:
39    keep_hourly: 24
40    keep_daily: 7
41    keep_weekly: 2
42    keep_monthly: 1
43    prefix: "homelab-"
44
45consistency:
46    prefix: "homelab-"

⚠️

Securely store the encryption passphrase in your password manager. Losing this will render your backups inaccessible.

Correctly specify the SSH port number. In my setup, to exclude a new directory from being backed up, all I have to do is create a new file in that directory.

1touch /included_path/subdir_to_exclude/.nobackup

In my example, 24 hourly, 7 daily, 2 weekly and 1 monthly snapshots (aka archives) will be kept. I find this to work very well for me. Tune this as you wish.

If you're lost, you can add official comments back to your config file without losing your modifications. Then refer to the comments for help.

1generate-borgmatic-config -s /etc/borgmatic/config.yaml --overwrite

Validating the config file #

borgmatic comes with a handy tool to validate your config file:

1validate-borgmatic-config

Initializing the repository #

We'll be using the keyfile encryption option, which requires both this file and the passphrase to access backups.

1borgmatic -v2 init -e keyfile

⚠️

Securely store the keyfile or the contents of the keyfile in your password manager. Losing this will render your backups inaccessible.

In my example the keyfile is saved at /mnt/nvme/borgdata/.config/borg/keys

If you prefer not to deal with keyfiles, you can use the repokey option; the key will be stored on the backup server instead. This is still secure as your passphrase is still required to decrypt that key.

1borgmatic -v2 init -e repokey

Dry running your backup #

Let's test that borg can connect to the server and backup without issues.

1borgmatic -v2 --dry-run

If summary says successful, you're good to go

Scheduling a cronjob #

I like to run my backups, consistency checks and prunes every hour. You may want to tweak this to your taste.

Before we add a cronjob, we'll write a script to make future modifications easier. I like to store all of my cron scripts at /crons.

Create a file with your favorite editor at /crons/borg.sh with the following content:

1#!/bin/bash
2export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.local/bin
3
4borgmatic \
5    --verbosity -1 \
6    --syslog-verbosity 1 \
7    prune create check compact \
8    --stats

Then let's make that executable:

1chmod +x /crons/borg.sh

We're ready to add this to our crontab.

1## edit your crontab
2crontab -e
3
4## then add this line and save
530 * * * * /crons/borg.sh

This runs every hour at minute 30.

More information about cron schedule syntax here.

Listing archives #

1borgmatic list

Should return something like this:

1homelab-20230110_1531             Tue, 2023-01-10 15:31:47 [2d0958d602a0cc0056576186c12f5ee3f9db3dc8d8815c557bb526b344654da6]
2homelab-20230110_1632             Tue, 2023-01-10 16:32:37 [673002d0c57a7bbb86fdcafa5e17b672630f9431bafd601e38afc6066e200bd3]
3homelab-20230110_1733             Tue, 2023-01-10 17:33:11 [dc21a6bb7f40121865c6021b420db419bff527cba7a694b4ccf4ea15bf197066]
4homelab-20230110_1831             Tue, 2023-01-10 18:31:15 [5e0ee9cef2e1fb438259ec7e16f20833bce5c5754188c14ae67e2889504b4616]
5homelab-20230110_1932             Tue, 2023-01-10 19:32:09 [c35595bfcd08d0dea748a054938a4423ba75db999af0efc826face3515c22c16]
6homelab-20230110_2031             Tue, 2023-01-10 20:31:08 [1af4adc261063f5958d3b83ff844adaa63d3b96a17e607b153dd63a9d1b62db1]
7homelab-20230110_2132             Tue, 2023-01-10 21:32:24 [a72dcd39a4a3b228ba525dd4e71a63c95be513e50e797072bed4a02a95c3bbfa]
8

Manually backup single directory #

Sometimes it's handy to quickly archive a single directory. Since Borgmatic don't support this yet, we have to use borg directly.

Let's set environment variables for repo, passphrase and base directory.

1export BORG_REPO=ssh://user@host.tld:23/./borg
2export BORG_BASE_DIR=/mnt/nvme/borgdata
3export BORG_PASSPHRASE="vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS"

Then to backup /certs we simply run:

1borg -v create ::archive_name /certs

The variables can be added to .bashrc or .zshrc so they're loaded every time you log in.

1echo "export BORG_REPO=ssh://user@host.tld:23/./borg" >> .zshrc
2echo "BORG_BASE_DIR=/mnt/nvme/borgdata" >> .zshrc
3echo 'BORG_PASSPHRASE="vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS"' >> .zshrc

Restoring #

To restore the path /certs from an archive named homelab-20230110_1531 from the ssh://user@host.tld:23/./borg repository into /restore:

1mkdir /restore
2cd /restore
3borgmatic -v2 extract \
4    --repository ssh://user@host.tld:23/./borg \
5    --archive homelab-20230110_1531 \
6    --path certs

--path should not contain leading slash (similar to tar) and can be omitted if you want to extract the entire archive.

--repository can be omitted if you only have one repository specified in the borgmatic config file.

Mounting an archive #

To mount the path /certs from an archive named homelab-20230110_1531 from the ssh://user@host.tld:23/./borg repository at /mnt/borg:

1mkdir /mnt/borg
2borgmatic -v2 mount \
3    --repository ssh://user@host.tld:23/./borg \
4    --archive homelab-20230110_1531 \
5    --path certs \
6    --mount-point /mnt/borg

--path can be omitted if you want to mount the entire archive.

--repository can be omitted if you only have one repository specified in the borgmatic config file.

To unmount:

1borgmatic -v2 umount \
2    --mount-point /mnt/borg

Bonus: get notified when backups fail #

We can use a free online (self hostable!) service healthchecks.io to monitor backups and get notified if backups fail.

Grab yourself a free account, create a check, set the cron and the machine's timezone:

healthchecks

Copy the Ping URL and keep it handy, we'll need this later.

Open the file we created before that's in the crontab - /crons/borg.sh in an editor. We'll make a few modifications here so cron pings the URL at the start and end of the backup process and also sends the log to healthchecks.io.

 1export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.local/bin
 2
 3curl -m 10 --retry 5 \
 4    "https://your_ping_url/start" \
 5    2>/dev/null 1>/dev/null
 6
 7foo=$(borgmatic \
 8    --verbosity 1 \
 9    --syslog-verbosity -1 \
10    prune create check compact \
11    --stats 2>&1)
12
13curl -m 10 --retry 5 \
14    --data-raw "$foo" \
15    "https://hc-ping.com/your_ping_url" \
16    2>/dev/null 1>/dev/null

Event history on healthchecks.io looks something like this:

hc_events

Don't forget to logout as the root user!

Conclusion #

A backup that cannot be restored is as good as no backup. Please, test your backups! Restore, access some files, make sure they open. Spin up a VM, see if you can restore files there. A backup's no good if you cannot restore to a new machine.

Let's hope it never comes to that but until then, sleep well at night knowing your homelab is automatically backing itself offsite every day

#backup #borg #secure #encryption #automated #ssh