title	permalink
borgmatic	borgmatic/index.html

Overview

borgmatic is a simple Python wrapper script for the Borg backup software that initiates a backup, prunes any old backups according to a retention policy, and validates backups for consistency. The script supports specifying your settings in a declarative configuration file rather than having to put them all on the command-line, and handles common errors.

Here's an example config file:

location:
    # List of source directories to backup. Globs are expanded.
    source_directories:
        - /home
        - /etc
        - /var/log/syslog*

    # Paths to local or remote repositories.
    repositories:
        - user@backupserver:sourcehostname.borg

    # Any paths matching these patterns are excluded from backups.
    exclude_patterns:
        - /home/*/.cache

retention:
    # Retention policy for how many backups to keep in each category.
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6

consistency:
    # List of consistency checks to run: "repository", "archives", or both.
    checks:
        - repository
        - archives

borgmatic is hosted at https://torsion.org/borgmatic with source code available. It's also mirrored on GitHub for convenience.

Want to see borgmatic in action? Check out the screencast.

Installation

To get up and running, follow the Borg Quick Start to create a repository on a local or remote host. Note that if you plan to run borgmatic on a schedule with cron, and you encrypt your Borg repository with a passphrase instead of a key file, you'll either need to set the borgmatic encryption_passphrase configuration variable or set the BORG_PASSPHRASE environment variable. See the repository encryption section of the Quick Start for more info.

Alternatively, the passphrase can be specified programatically by setting either the borgmatic encryption_passcommand configuration variable or the BORG_PASSCOMMAND environment variable. See the Borg Security FAQ for more info.

If the repository is on a remote host, make sure that your local root user has key-based ssh access to the desired user account on the remote host.

To install borgmatic, run the following command to download and install it:

sudo pip3 install --upgrade borgmatic

Note that your pip binary may have a different name than "pip3". Make sure you're using Python 3, as borgmatic does not support Python 2.

Other ways to install

• A borgmatic Docker image based on Alpine Linux.
• Another borgmatic Docker image based on Alpine Linux, updated automatically whenever the Alpine image updates.
• A borgmatic package for Fedora.
• A borgmatic package for Arch Linux.
• A borgmatic package for OpenBSD.
  
  

Configuration

After you install borgmatic, generate a sample configuration file:

sudo generate-borgmatic-config

If that command is not found, then it may be installed in a location that's not in your system PATH. Try looking in /usr/local/bin/.

This generates a sample configuration file at /etc/borgmatic/config.yaml (by default). You should edit the file to suit your needs, as the values are just representative. All fields are optional except where indicated, so feel free to ignore anything you don't need.

You can also have a look at the full configuration schema for the authoritative set of all configuration options. This is handy if borgmatic has added new options since you originally created your configuration file.

Multiple configuration files

A more advanced usage is to create multiple separate configuration files and place each one in an /etc/borgmatic.d directory. For instance:

sudo mkdir /etc/borgmatic.d
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml

With this approach, you can have entirely different backup policies for different applications on your system. For instance, you may want one backup configuration for your database data directory, and a different configuration for your user home directories.

When you set up multiple configuration files like this, borgmatic will run each one in turn from a single borgmatic invocation. This includes, by default, the traditional /etc/borgmatic/config.yaml as well.

And if you need even more customizability, you can specify alternate configuration paths on the command-line with borgmatic's --config option. See borgmatic --help for more information.

Hooks

If you find yourself performing prepraration tasks before your backup runs, or cleanup work afterwards, borgmatic hooks may be of interest. They're simply shell commands that borgmatic executes for you at various points, and they're configured in the hooks section of your configuration file.

For instance, you can specify before_backup hooks to dump a database to file before backing it up, and specify after_backup hooks to delete the temporary file afterwards.

borgmatic hooks run once per configuration file. before_backup hooks run prior to backups of all repositories. after_backup hooks run afterwards, but not if an error occurs in a previous hook or in the backups themselves. And borgmatic runs on_error hooks if an error occurs.

An important security note about hooks: borgmatic executes all hook commands with the user permissions of borgmatic itself. So to prevent potential shell injection or privilege escalation, do not forget to set secure permissions (chmod 0700) on borgmatic configuration files and scripts invoked by hooks.

See the sample generated configuration file mentioned above for specifics about hook configuration syntax.

Upgrading

In general, all you should need to do to upgrade borgmatic is run the following:

sudo pip3 install --upgrade borgmatic

However, see below about special cases.

Upgrading from borgmatic 1.0.x

borgmatic changed its configuration file format in version 1.1.0 from INI-style to YAML. This better supports validation, and has a more natural way to express lists of values. To upgrade your existing configuration, first upgrade to the new version of borgmatic.

As of version 1.1.0, borgmatic no longer supports Python 2. If you were already running borgmatic with Python 3, then you can simply upgrade borgmatic in-place:

sudo pip3 install --upgrade borgmatic

But if you were running borgmatic with Python 2, uninstall and reinstall instead:

sudo pip uninstall borgmatic
sudo pip3 install borgmatic

The pip binary names for different versions of Python can differ, so the above commands may need some tweaking to work on your machine.

Once borgmatic is upgraded, run:

sudo upgrade-borgmatic-config

That will generate a new YAML configuration file at /etc/borgmatic/config.yaml (by default) using the values from both your existing configuration and excludes files. The new version of borgmatic will consume the YAML configuration file instead of the old one.

Upgrading from atticmatic

You can ignore this section if you're not an atticmatic user (the former name of borgmatic).

borgmatic only supports Borg now and no longer supports Attic. So if you're an Attic user, consider switching to Borg. See the Borg upgrade command for more information. Then, follow the instructions above about setting up your borgmatic configuration files.

If you were already using Borg with atticmatic, then you can easily upgrade from atticmatic to borgmatic. Simply run the following commands:

sudo pip3 uninstall atticmatic
sudo pip3 install borgmatic

That's it! borgmatic will continue using your /etc/borgmatic configuration files.

Usage

You can run borgmatic and start a backup simply by invoking it without arguments:

borgmatic

This will also prune any old backups as per the configured retention policy, and check backups for consistency problems due to things like file damage.

If you'd like to see the available command-line arguments, view the help:

borgmatic --help

Note that borgmatic prunes archives before creating an archive, so as to free up space for archiving. This means that when a borgmatic run finishes, there may still be prune-able archives. Not to worry, as they will get cleaned up at the start of the next run.

Verbosity

By default, the backup will proceed silently except in the case of errors. But if you'd like to to get additional information about the progress of the backup as it proceeds, use the verbosity option:

borgmatic --verbosity 1

Or, for even more progress spew:

borgmatic --verbosity 2

À la carte

If you want to run borgmatic with only pruning, creating, or checking enabled, the following optional flags are available:

borgmatic --prune
borgmatic --create
borgmatic --check

You can run with only one of these flags provided, or you can mix and match any number of them. This supports use cases like running consistency checks from a different cron job with a different frequency, or running pruning with a different verbosity level.

Additionally, borgmatic provides convenient flags for Borg's list and info functionality:

borgmatic --list
borgmatic --info

You can include an optional --json flag with --create, --list, or --info to get the output formatted as JSON.

Autopilot

If you want to run borgmatic automatically, say once a day, the you can configure a job runner to invoke it periodically.

cron

If you're using cron, download the sample cron file. Then, from the directory where you downloaded it:

sudo mv borgmatic /etc/cron.d/borgmatic
sudo chmod +x /etc/cron.d/borgmatic

You can modify the cron file if you'd like to run borgmatic more or less frequently.

systemd

If you're using systemd instead of cron to run jobs, download the sample systemd service file and the sample systemd timer file. Then, from the directory where you downloaded them:

sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
sudo systemctl enable borgmatic.timer
sudo systemctl start borgmatic.timer

Feel free to modify the timer file based on how frequently you'd like borgmatic to run.

Support and contributing

Issues

You've got issues? Or an idea for a feature enhancement? We've got an issue tracker. In order to create a new issue or comment on an issue, you'll need to login first. Note that you can login with an existing GitHub account if you prefer.

Other questions or comments? Contact mailto:witten@torsion.org.

Contributing

If you'd like to contribute to borgmatic development, please feel free to submit a Pull Request or open an issue first to discuss your idea. We also accept Pull Requests on GitHub, if that's more your thing. In general, contributions are very welcome. We don't bite!

Code style

Start with PEP 8. But then, apply the following deviations from it:

• For strings, prefer single quotes over double quotes.
• Limit all lines to a maximum of 100 characters.
• Use trailing commas within multiline values or argument lists.
• For multiline constructs, put opening and closing delimeters on lines separate from their contents.
• Within multiline constructs, use standard four-space indentation. Don't align indentation with an opening delimeter.

borgmatic code uses the Black code formatter and Flake8 code checker, so certain code style requirements will be enforced when running automated tests. See the Black and Flake8 documentation for more information.

Development

To get set up to hack on borgmatic, first clone master via HTTPS or SSH:

git clone https://projects.torsion.org/witten/borgmatic.git

Or:

git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git

Then, install borgmatic "editable" so that you can easily run borgmatic commands while you're hacking on them to make sure your changes work.

cd borgmatic/
pip3 install --editable --user .

Note that this will typically install the borgmatic commands into ~/.local/bin, which may or may not be on your PATH. There are other ways to install borgmatic editable as well, for instance into the system Python install (so without --user, as root), or even into a virtualenv. How or where you install borgmatic is up to you, but generally an editable install makes development and testing easier.

Running tests

Assuming you've cloned the borgmatic source code as described above, and you're in the borgmatic/ working copy, install tox, which is used for setting up testing environments:

sudo pip3 install tox

Finally, to actually run tests, run:

cd borgmatic
tox

Note that while running borgmatic itself only requires Python 3+, running borgmatic's tests require Python 3.6+.

If when running tests, you get an error from the Black code formatter about files that would be reformatted, you can ask Black to format them for you via the following:

tox -e black

End-to-end tests

borgmatic additionally includes some end-to-end tests that integration test with Borg for a few representative scenarios. These tests don't run by default because they're slow and require Docker. If you would like to run them, first install Docker, and make sure that it's executable by the current user. Then:

tox -e end-to-end

That builds a test container with your local borgmatic source, and runs end-to-end tests within it.

Troubleshooting

Broken pipe with remote repository

When running borgmatic on a large remote repository, you may receive errors like the following, particularly while "borg check" is validating backups for consistency:

    Write failed: Broken pipe
    borg: Error: Connection closed by remote host

This error can be caused by an ssh timeout, which you can rectify by adding the following to the ~/.ssh/config file on the client:

    Host *
        ServerAliveInterval 120

This should make the client keep the connection alive while validating backups.

libyaml compilation errors

borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally use a C YAML library (libyaml) if present. But if it's not installed, then when installing or upgrading borgmatic, you may see errors about compiling the YAML library. If so, not to worry. borgmatic should install and function correctly even without the C YAML library. And borgmatic won't be any faster with the C library present, so you don't need to go out of your way to install it.