Gibdos Talks FOSS

Let The Borgs Do It

Today I finally finished my migration from my old Hetzner root server, which still ran my MailCow mailserver, to my netcup VPS. And since it's been quite a few years since I last set up my borg backup solution on Hetzner, I thought it would be a good idea to update and document my backup system.

So in this blog post, I will teach you how to:

Since I use Ubuntu Server on my VPS, the guide will focus on that. If you use a different Linux / OS, please check the installation instructions of borg and borgmatic. Since both programs are python-based, they run on practically any Linux distribution.

Automatic BackUp with Borgmatic

Requires
🤖 Borg Backup
🚀 Borgmatic
📦️ Hetzner Storage Box (or equivalent provider with borg support)

While borg can backup to any SSH capable server, it will gain a significant performance boost, if the target server also has borg installed. The quote from borg backup states:

Off-site backups
Borg can store data on any remote host accessible over SSH. If Borg is installed on the remote host, significant performance gains can be achieved compared to using a network file system (sshfs, NFS, …).

To prevent permission and file access problems, I will set up everything as root. So after connecting to your server, start with sudo -s to switch into root.

Setup

Install borgbackup with fuse3
Guide

# Install all dependencies
apt install python3 python3-dev python3-pip python3-virtualenv \
libacl1-dev \
libssl-dev \
liblz4-dev libzstd-dev libxxhash-dev \
build-essential pkg-config libfuse3-dev fuse3

# Install borgbackup with fuse3
pipx install borgbackup[pyfuse3]

# Check if /root/.local/bin is in your path with
echo $PATH
# If it is not in your $PATH
# Follow on-screen instruction to add it to $PATH

Install borgmatic

# Install borgmatic
pipx install borgmatic

# Check borgmatic installation
borgmatic --version

# Create default config.yaml
borgmatic config generate
# /etc/borgmatic/config.yaml

Setup Borg Repository

Setup Borgmatic with CONFIG.yaml

BackUp

Backup Script for Docker Container
If you want to backup all your docker container in /opt/docker/:

#!/bin/env bash

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nBacking up log files.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Make a copy of the logs files into logs_old folder
cp /PATH/TO/BACKUP_FOLDER/docker_backup.log /PATH/TO/BACKUP_FOLDER/logs_old/
cp /PATH/TO/BACKUP_FOLDER/borgmatic.log /PATH/TO/BACKUP_FOLDER/logs_old/

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nBacked up log files to logs_old.\nTruncating logs.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Empty the log files
truncate -s 0 /PATH/TO/BACKUP_FOLDER/docker_backup.log
truncate -s 0 /PATH/TO/BACKUP_FOLDER/borgmatic.log

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nTruncated log files.\nStopping docker container.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Run docker compose down on any compose.yml in any /opt/docker/FOLDER
for i in $(find /opt/docker/ -type f -name compose.yml); do
    /usr/bin/docker compose -f $i down &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log
done

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nStopped all docker containers.\nStarting borgmatic backup.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Run borgmatic to transfer your source folders to your storage server
/root/.local/bin/borgmatic create --verbosity 1 --list --stats &>> /PATH/TO/BACKUP_FOLDER/borgmatic.log

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nFinished borgmatic backup.\nStarting containers.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Run docker compose up -d on any compose.yml in any FOLDER inside /opt/docker/
for i in $(find /opt/docker/ -type f -name compose.yml); do
    /usr/bin/docker compose -f $i up -d &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log
done

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nStarted all docker containers.\n.Starting borgmatic prune.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Clean up old archives from your borg repo according to your retention policy
/root/.local/bin/borgmatic prune --list --stats >> /PATH/TO/BACKUP_FOLDER/borgmatic.log

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nBorg repo pruned.\n.Starting borgmatic compacting.\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

# Free up disk space from the removed archives
/root/.local/bin/borgmatic compact --progress &>> /PATH/TO/BACKUP_FOLDER/borgmatic.log

CURRENTDATE=`date +"%Y-%m-%d %T"`
echo -e $CURRENTDATE '\nBorg repo compacted.\n.Backup finished!\n' &>> /PATH/TO/BACKUP_FOLDER/docker_backup.log

Basic log rotation
To automate the log file clean up

#!/bin/env bash

# Deletes all log files in logs_old folder
rm -rf /PATH/TO/BACKUP_FOLDER/logs_old/*.log

Initial BackUp

Setup cronjob
With a cronjob, we can have our scripts run automatically at specific intervalls. To better understand the cron schema, use a cronjob generator.

Maintenance

Restore from Backups

Borg and Borgmatic make it very easy to quickly transfer the backed up files and folders back to your server, in case you need to restore some or all of them.

Mount your backups into a folder

Extract a single file or folder
If you want to transfer a single file or folder from your borg repo

# Search the latest archive for a file or folder
borgmatic --find FILE_OR_FOLDER_NAME --archive latest

# Extract a file or folder
borgmatic extract --archive latest --path PATH/TO/FOLDER_OR_FILE --destination /PATH/TO/DEST
# will extract to current folder without --destination
# Parent folders will be included, so files & folder will be at
# /PATH/TO/DEST/PATH/TO/FOLDER_OR_FILE

# --archive can also choose a specific archive
# Check the available archives with
borgmatic -l
# Use the archive name instead of latest

Full Restore (for migration / in emergencies)
If you want to transfer your entire borg backups

Additional Information

With borgmatic, you are not limited to a single config.yaml. You can easily copy the /etc/borgmatic/config.yaml and change the source_directories:, path:, etc.

If you want to use another config.yaml and create a second script, just add --config /PATH/TO/SECOND_CONF.yaml to any borgmatic command. (borgmatic --config /PATH/TO/SECOND_CONF.yaml -l, for example)

That's all for today. I hope this guide made data backup and restoration easier for some of you.