New "borgmatic" command to support Borg backup software, a fork of Attic.

This commit is contained in:
Dan Helfman 2015-07-18 18:35:29 -07:00
parent bdce17ae71
commit f2f8503e77
13 changed files with 175 additions and 58 deletions

4
NEWS
View file

@ -1,3 +1,7 @@
0.1.0
* New "borgmatic" command to support Borg backup software, a fork of Attic.
0.0.7
* Flag for multiple levels of verbosity: some, and lots.

View file

@ -4,12 +4,13 @@ save_as: atticmatic/index.html
## Overview
atticmatic is a simple Python wrapper script for the [Attic backup
software](https://attic-backup.org/) that initiates a backup, prunes any old
backups according to a retention policy, and validates backups for
consistency. The script supports specifying your settings in a declarative
configuration file rather than having to put them all on the command-line, and
handles common errors.
atticmatic is a simple Python wrapper script for the
[Attic](https://attic-backup.org/) and
[Borg](https://borgbackup.github.io/borgbackup/) backup software that
initiates a backup, prunes any old backups according to a retention policy,
and validates backups for consistency. The script supports specifying your
settings in a declarative configuration file rather than having to put them
all on the command-line, and handles common errors.
Here's an example config file:
@ -17,7 +18,7 @@ Here's an example config file:
# Space-separated list of source directories to backup.
source_directories: /home /etc
# Path to local or remote Attic repository.
# Path to local or remote backup repository.
repository: user@backupserver:sourcehostname.attic
[retention]
@ -41,14 +42,14 @@ available](https://torsion.org/hg/atticmatic). It's also mirrored on
## Setup
To get up and running with Attic, follow the [Attic Quick
Start](https://attic-backup.org/quickstart.html) guide to create an Attic
To get up and running, follow the [Attic Quick
Start](https://attic-backup.org/quickstart.html) or the [Borg Quick
Start](https://borgbackup.github.io/borgbackup/quickstart.html) to create a
repository on a local or remote host. Note that if you plan to run atticmatic
on a schedule with cron, and you encrypt your attic repository with a
passphrase instead of a key file, you'll need to set the `ATTIC_PASSPHRASE`
environment variable. See [attic's repository encryption
documentation](https://attic-backup.org/quickstart.html#encrypted-repos) for
more info.
environment variable. See the repository encryption section of the Quick Start
for more info.
If the repository is on a remote host, make sure that your local root user has
key-based ssh access to the desired user account on the remote host.
@ -57,13 +58,19 @@ To install atticmatic, run the following command to download and install it:
sudo pip install --upgrade hg+https://torsion.org/hg/atticmatic
Then copy the following configuration files:
If you are using Attic, copy the following configuration files:
sudo cp sample/atticmatic.cron /etc/cron.d/atticmatic
sudo mkdir /etc/atticmatic/
sudo cp sample/config sample/excludes /etc/atticmatic/
Lastly, modify those files with your desired configuration.
If you are using Borg, copy the files like this instead:
sudo cp sample/atticmatic.cron /etc/cron.d/borgmatic
sudo mkdir /etc/borgmatic/
sudo cp sample/config sample/excludes /etc/borgmatic/
Lastly, modify the /etc files with your desired configuration.
## Usage
@ -73,6 +80,11 @@ arguments:
atticmatic
Or, if you're using Borg, use this command instead to make use of the Borg
backend:
borgmatic
This will also prune any old backups as per the configured retention policy,
and check backups for consistency problems due to things like file damage.

View file

View file

@ -0,0 +1,12 @@
from functools import partial
from atticmatic.backends import shared
# An atticmatic backend that supports Attic for actually handling backups.
COMMAND = 'attic'
create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

View file

@ -0,0 +1,13 @@
from functools import partial
from atticmatic.backends import shared
# An atticmatic backend that supports Borg for actually handling backups.
COMMAND = 'borg'
create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

View file

@ -6,10 +6,16 @@ import subprocess
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def create_archive(excludes_filename, verbosity, source_directories, repository):
# Common backend functionality shared by Attic and Borg. As the two backup
# commands diverge, these shared functions will likely need to be replaced
# with non-shared functions within atticmatic.backends.attic and
# atticmatic.backends.borg.
def create_archive(excludes_filename, verbosity, source_directories, repository, command):
'''
Given an excludes filename, a vebosity flag, a space-separated list of source directories, and
a local or remote repository path, create an attic archive.
Given an excludes filename, a vebosity flag, a space-separated list of source directories, a
local or remote repository path, and a command to run, create an attic archive.
'''
sources = tuple(source_directories.split(' '))
verbosity_flags = {
@ -17,8 +23,8 @@ def create_archive(excludes_filename, verbosity, source_directories, repository)
VERBOSITY_LOTS: ('--verbose', '--stats'),
}.get(verbosity, ())
command = (
'attic', 'create',
full_command = (
command, 'create',
'--exclude-from', excludes_filename,
'{repo}::{hostname}-{timestamp}'.format(
repo=repository,
@ -27,7 +33,7 @@ def create_archive(excludes_filename, verbosity, source_directories, repository)
),
) + sources + verbosity_flags
subprocess.check_call(command)
subprocess.check_call(full_command)
def _make_prune_flags(retention_config):
@ -52,18 +58,19 @@ def _make_prune_flags(retention_config):
)
def prune_archives(verbosity, repository, retention_config):
def prune_archives(verbosity, repository, retention_config, command):
'''
Given a verbosity flag, a local or remote repository path, and a retention config dict, prune
attic archives according the the retention policy specified in that configuration.
Given a verbosity flag, a local or remote repository path, a retention config dict, and a
command to run, prune attic archives according the the retention policy specified in that
configuration.
'''
verbosity_flags = {
VERBOSITY_SOME: ('--stats',),
VERBOSITY_LOTS: ('--verbose', '--stats'),
}.get(verbosity, ())
command = (
'attic', 'prune',
full_command = (
command, 'prune',
repository,
) + tuple(
element
@ -71,7 +78,7 @@ def prune_archives(verbosity, repository, retention_config):
for element in pair
) + verbosity_flags
subprocess.check_call(command)
subprocess.check_call(full_command)
DEFAULT_CHECKS = ('repository', 'archives')
@ -123,10 +130,10 @@ def _make_check_flags(checks):
)
def check_archives(verbosity, repository, consistency_config):
def check_archives(verbosity, repository, consistency_config, command):
'''
Given a verbosity flag, a local or remote repository path, and a consistency config dict, check
the contained attic archives for consistency.
Given a verbosity flag, a local or remote repository path, a consistency config dict, and a
command to run, check the contained attic archives for consistency.
If there are no consistency checks to run, skip running them.
'''
@ -139,12 +146,12 @@ def check_archives(verbosity, repository, consistency_config):
VERBOSITY_LOTS: ('--verbose',),
}.get(verbosity, ())
command = (
'attic', 'check',
full_command = (
command, 'check',
repository,
) + _make_check_flags(checks) + verbosity_flags
# Attic's check command spews to stdout even without the verbose flag. Suppress it.
# The check command spews to stdout even without the verbose flag. Suppress it.
stdout = None if verbosity_flags else open(os.devnull, 'w')
subprocess.check_call(command, stdout=stdout)
subprocess.check_call(full_command, stdout=stdout)

View file

@ -1,31 +1,34 @@
from __future__ import print_function
from argparse import ArgumentParser
from importlib import import_module
import os
from subprocess import CalledProcessError
import sys
from atticmatic.attic import check_archives, create_archive, prune_archives
from atticmatic.config import parse_configuration
DEFAULT_CONFIG_FILENAME = '/etc/atticmatic/config'
DEFAULT_EXCLUDES_FILENAME = '/etc/atticmatic/excludes'
DEFAULT_CONFIG_FILENAME_PATTERN = '/etc/{}/config'
DEFAULT_EXCLUDES_FILENAME_PATTERN = '/etc/{}/excludes'
def parse_arguments(*arguments):
def parse_arguments(command_name, *arguments):
'''
Parse the given command-line arguments and return them as an ArgumentParser instance.
Given the name of the command with which this script was invoked and command-line arguments,
parse the arguments and return them as an ArgumentParser instance. Use the command name to
determine the default configuration and excludes paths.
'''
parser = ArgumentParser()
parser.add_argument(
'-c', '--config',
dest='config_filename',
default=DEFAULT_CONFIG_FILENAME,
default=DEFAULT_CONFIG_FILENAME_PATTERN.format(command_name),
help='Configuration filename',
)
parser.add_argument(
'--excludes',
dest='excludes_filename',
default=DEFAULT_EXCLUDES_FILENAME,
default=DEFAULT_EXCLUDES_FILENAME_PATTERN.format(command_name),
help='Excludes filename',
)
parser.add_argument(
@ -37,15 +40,30 @@ def parse_arguments(*arguments):
return parser.parse_args(arguments)
def load_backend(command_name):
'''
Given the name of the command with which this script was invoked, return the corresponding
backend module responsible for actually dealing with backups.
'''
backend_name = {
'atticmatic': 'attic',
'borgmatic': 'borg',
}.get(command_name, 'attic')
return import_module('atticmatic.backends.{}'.format(backend_name))
def main():
try:
args = parse_arguments(*sys.argv[1:])
command_name = os.path.basename(sys.argv[0])
args = parse_arguments(command_name, *sys.argv[1:])
config = parse_configuration(args.config_filename)
repository = config.location['repository']
backend = load_backend(command_name)
create_archive(args.excludes_filename, args.verbosity, **config.location)
prune_archives(args.verbosity, repository, config.retention)
check_archives(args.verbosity, repository, config.consistency)
backend.create_archive(args.excludes_filename, args.verbosity, **config.location)
backend.prune_archives(args.verbosity, repository, config.retention)
backend.check_archives(args.verbosity, repository, config.consistency)
except (ValueError, IOError, CalledProcessError) as error:
print(error, file=sys.stderr)
sys.exit(1)

View file

@ -5,16 +5,19 @@ from nose.tools import assert_raises
from atticmatic import command as module
def test_parse_arguments_with_no_arguments_uses_defaults():
parser = module.parse_arguments()
COMMAND_NAME = 'foomatic'
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
def test_parse_arguments_with_no_arguments_uses_defaults():
parser = module.parse_arguments(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.verbosity == None
def test_parse_arguments_with_filename_arguments_overrides_defaults():
parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
parser = module.parse_arguments(COMMAND_NAME, '--config', 'myconfig', '--excludes', 'myexcludes')
assert parser.config_filename == 'myconfig'
assert parser.excludes_filename == 'myexcludes'
@ -22,10 +25,10 @@ def test_parse_arguments_with_filename_arguments_overrides_defaults():
def test_parse_arguments_with_verbosity_flag_overrides_default():
parser = module.parse_arguments('--verbosity', '1')
parser = module.parse_arguments(COMMAND_NAME, '--verbosity', '1')
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.verbosity == 1
@ -35,6 +38,6 @@ def test_parse_arguments_with_invalid_arguments_exits():
try:
with assert_raises(SystemExit):
module.parse_arguments('--posix-me-harder')
module.parse_arguments(COMMAND_NAME, '--posix-me-harder')
finally:
sys.stderr = original_stderr

View file

@ -2,7 +2,7 @@ from collections import OrderedDict
from flexmock import flexmock
from atticmatic import attic as module
from atticmatic.backends import shared as module
from atticmatic.tests.builtins import builtins_mock
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
@ -42,6 +42,7 @@ def test_create_archive_should_call_attic_with_parameters():
verbosity=None,
source_directories='foo bar',
repository='repo',
command='attic',
)
@ -55,6 +56,7 @@ def test_create_archive_with_verbosity_some_should_call_attic_with_stats_paramet
verbosity=VERBOSITY_SOME,
source_directories='foo bar',
repository='repo',
command='attic',
)
@ -68,6 +70,7 @@ def test_create_archive_with_verbosity_lots_should_call_attic_with_verbose_param
verbosity=VERBOSITY_LOTS,
source_directories='foo bar',
repository='repo',
command='attic',
)
@ -108,6 +111,7 @@ def test_prune_archives_should_call_attic_with_parameters():
verbosity=None,
repository='repo',
retention_config=retention_config,
command='attic',
)
@ -122,6 +126,7 @@ def test_prune_archives_with_verbosity_some_should_call_attic_with_stats_paramet
repository='repo',
verbosity=VERBOSITY_SOME,
retention_config=retention_config,
command='attic',
)
@ -136,6 +141,7 @@ def test_prune_archives_with_verbosity_lots_should_call_attic_with_verbose_param
repository='repo',
verbosity=VERBOSITY_LOTS,
retention_config=retention_config,
command='attic',
)
@ -193,6 +199,7 @@ def test_check_archives_should_call_attic_with_parameters():
verbosity=None,
repository='repo',
consistency_config=consistency_config,
command='attic',
)
@ -211,6 +218,7 @@ def test_check_archives_with_verbosity_some_should_call_attic_with_verbose_param
verbosity=VERBOSITY_SOME,
repository='repo',
consistency_config=consistency_config,
command='attic',
)
@ -229,6 +237,7 @@ def test_check_archives_with_verbosity_lots_should_call_attic_with_verbose_param
verbosity=VERBOSITY_LOTS,
repository='repo',
consistency_config=consistency_config,
command='attic',
)
@ -241,4 +250,5 @@ def test_check_archives_without_any_checks_should_bail():
verbosity=None,
repository='repo',
consistency_config=consistency_config,
command='attic',
)

View file

@ -0,0 +1,33 @@
from flexmock import flexmock
from atticmatic import command as module
def test_load_backend_with_atticmatic_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)
assert module.load_backend('atticmatic') == backend
def test_load_backend_with_unknown_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)
assert module.load_backend('unknownmatic') == backend
def test_load_backend_with_borgmatic_command_should_return_borg_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.borg')
.and_return(backend).once()
)
assert module.load_backend('borgmatic') == backend

View file

@ -2,7 +2,7 @@
# Space-separated list of source directories to backup.
source_directories: /home /etc
# Path to local or remote Attic repository.
# Path to local or remote repository.
repository: user@backupserver:sourcehostname.attic
[retention]

View file

@ -2,12 +2,17 @@ from setuptools import setup, find_packages
setup(
name='atticmatic',
version='0.0.7',
description='A wrapper script for Attic backup software that creates and prunes backups',
version='0.1.0',
description='A wrapper script for Attic/Borg backup software that creates and prunes backups',
author='Dan Helfman',
author_email='witten@torsion.org',
packages=find_packages(),
entry_points={'console_scripts': ['atticmatic = atticmatic.command:main']},
entry_points={
'console_scripts': [
'atticmatic = atticmatic.command:main',
'borgmatic = atticmatic.command:main',
]
},
tests_require=(
'flexmock',
'nose',