Merge branch 'master' into master
This commit is contained in:
commit
6b182c9d2d
17 changed files with 224 additions and 68 deletions
18
.drone.yml
18
.drone.yml
|
@ -14,6 +14,9 @@ services:
|
||||||
MYSQL_ROOT_PASSWORD: test
|
MYSQL_ROOT_PASSWORD: test
|
||||||
MYSQL_DATABASE: test
|
MYSQL_DATABASE: test
|
||||||
|
|
||||||
|
clone:
|
||||||
|
skip_verify: true
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: build
|
- name: build
|
||||||
image: alpine:3.9
|
image: alpine:3.9
|
||||||
|
@ -36,6 +39,9 @@ services:
|
||||||
MYSQL_ROOT_PASSWORD: test
|
MYSQL_ROOT_PASSWORD: test
|
||||||
MYSQL_DATABASE: test
|
MYSQL_DATABASE: test
|
||||||
|
|
||||||
|
clone:
|
||||||
|
skip_verify: true
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: build
|
- name: build
|
||||||
image: alpine:3.10
|
image: alpine:3.10
|
||||||
|
@ -58,6 +64,9 @@ services:
|
||||||
MYSQL_ROOT_PASSWORD: test
|
MYSQL_ROOT_PASSWORD: test
|
||||||
MYSQL_DATABASE: test
|
MYSQL_DATABASE: test
|
||||||
|
|
||||||
|
clone:
|
||||||
|
skip_verify: true
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: build
|
- name: build
|
||||||
image: alpine:3.13
|
image: alpine:3.13
|
||||||
|
@ -68,9 +77,14 @@ steps:
|
||||||
kind: pipeline
|
kind: pipeline
|
||||||
name: documentation
|
name: documentation
|
||||||
|
|
||||||
|
clone:
|
||||||
|
skip_verify: true
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: build
|
- name: build
|
||||||
image: plugins/docker
|
#image: plugins/docker
|
||||||
|
# Temporary work-around for https://github.com/drone-plugins/drone-docker/pull/327
|
||||||
|
image: techknowlogick/drone-docker
|
||||||
settings:
|
settings:
|
||||||
username:
|
username:
|
||||||
from_secret: docker_username
|
from_secret: docker_username
|
||||||
|
@ -80,5 +94,7 @@ steps:
|
||||||
dockerfile: docs/Dockerfile
|
dockerfile: docs/Dockerfile
|
||||||
|
|
||||||
trigger:
|
trigger:
|
||||||
|
repo:
|
||||||
|
- borgmatic-collective/borgmatic
|
||||||
branch:
|
branch:
|
||||||
- master
|
- master
|
||||||
|
|
28
NEWS
28
NEWS
|
@ -1,6 +1,30 @@
|
||||||
1.5.16.dev0
|
1.5.21.dev0
|
||||||
|
* Add support for old version (2.x) of jsonschema library.
|
||||||
|
|
||||||
|
1.5.20
|
||||||
|
* Re-release with correct version without dev0 tag.
|
||||||
|
|
||||||
|
1.5.19
|
||||||
|
* #387: Fix error when configured source directories are not present on the filesystem at the time
|
||||||
|
of backup. Now, Borg will complain, but the backup will still continue.
|
||||||
|
* #455: Mention changing borgmatic path in cron documentation.
|
||||||
|
* Update sample systemd service file with more granular read-only filesystem settings.
|
||||||
|
* Move Gitea and GitHub hosting from a personal namespace to an organization for better
|
||||||
|
collaboration with related projects.
|
||||||
|
* 1k ★s on GitHub!
|
||||||
|
|
||||||
|
1.5.18
|
||||||
|
* #389: Fix "message too long" error when logging to rsyslog.
|
||||||
|
* #440: Fix traceback that can occur when dumping a database.
|
||||||
|
|
||||||
|
1.5.17
|
||||||
|
* #437: Fix error when configuration file contains "umask" option.
|
||||||
|
* Remove test dependency on vim and /dev/urandom.
|
||||||
|
|
||||||
|
1.5.16
|
||||||
* #379: Suppress console output in sample crontab and systemd service files.
|
* #379: Suppress console output in sample crontab and systemd service files.
|
||||||
* #407: Fix syslog logging on FreeBSD.
|
* #407: Fix syslog logging on FreeBSD.
|
||||||
|
* #430: Fix hang when restoring a PostgreSQL "tar" format database dump.
|
||||||
* Better error messages! Switch the library used for validating configuration files (from pykwalify
|
* Better error messages! Switch the library used for validating configuration files (from pykwalify
|
||||||
to jsonschema).
|
to jsonschema).
|
||||||
* Link borgmatic Ansible role from installation documentation:
|
* Link borgmatic Ansible role from installation documentation:
|
||||||
|
@ -559,7 +583,7 @@
|
||||||
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
|
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
|
||||||
includes/excludes.
|
includes/excludes.
|
||||||
* Moved issue tracker from Taiga to integrated Gitea tracker at
|
* Moved issue tracker from Taiga to integrated Gitea tracker at
|
||||||
https://projects.torsion.org/witten/borgmatic/issues
|
https://projects.torsion.org/borgmatic-collective/borgmatic/issues
|
||||||
|
|
||||||
1.1.12
|
1.1.12
|
||||||
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"
|
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"
|
||||||
|
|
12
README.md
12
README.md
|
@ -106,7 +106,7 @@ development or hosting.
|
||||||
### Issues
|
### Issues
|
||||||
|
|
||||||
You've got issues? Or an idea for a feature enhancement? We've got an [issue
|
You've got issues? Or an idea for a feature enhancement? We've got an [issue
|
||||||
tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to
|
tracker](https://projects.torsion.org/borgmatic-collective/borgmatic/issues). In order to
|
||||||
create a new issue or comment on an issue, you'll need to [login
|
create a new issue or comment on an issue, you'll need to [login
|
||||||
first](https://projects.torsion.org/user/login). Note that you can login with
|
first](https://projects.torsion.org/user/login). Note that you can login with
|
||||||
an existing GitHub account if you prefer.
|
an existing GitHub account if you prefer.
|
||||||
|
@ -129,15 +129,15 @@ Other questions or comments? Contact
|
||||||
### Contributing
|
### Contributing
|
||||||
|
|
||||||
borgmatic [source code is
|
borgmatic [source code is
|
||||||
available](https://projects.torsion.org/witten/borgmatic) and is also mirrored
|
available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
|
||||||
on [GitHub](https://github.com/witten/borgmatic) for convenience.
|
on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
|
||||||
|
|
||||||
borgmatic is licensed under the GNU General Public License version 3 or any
|
borgmatic is licensed under the GNU General Public License version 3 or any
|
||||||
later version.
|
later version.
|
||||||
|
|
||||||
If you'd like to contribute to borgmatic development, please feel free to
|
If you'd like to contribute to borgmatic development, please feel free to
|
||||||
submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls)
|
submit a [Pull Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls)
|
||||||
or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
|
or open an [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) first
|
||||||
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
|
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
|
||||||
your thing. In general, contributions are very welcome. We don't bite!
|
your thing. In general, contributions are very welcome. We don't bite!
|
||||||
|
|
||||||
|
@ -145,5 +145,5 @@ Also, please check out the [borgmatic development
|
||||||
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
|
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
|
||||||
info on cloning source code, running tests, etc.
|
info on cloning source code, running tests, etc.
|
||||||
|
|
||||||
<a href="https://build.torsion.org/witten/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/witten/borgmatic/status.svg?ref=refs/heads/master)</a>
|
<a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/master)</a>
|
||||||
|
|
||||||
|
|
|
@ -6,14 +6,13 @@ permalink: security-policy/index.html
|
||||||
## Supported versions
|
## Supported versions
|
||||||
|
|
||||||
While we want to hear about security vulnerabilities in all versions of
|
While we want to hear about security vulnerabilities in all versions of
|
||||||
borgmatic, security fixes will only be made to the most recently released
|
borgmatic, security fixes are only made to the most recently released version.
|
||||||
version. It's not practical for our small volunteer effort to maintain
|
It's simply not practical for our small volunteer effort to maintain multiple
|
||||||
multiple different release branches and put out separate security patches for
|
release branches and put out separate security patches for each.
|
||||||
each.
|
|
||||||
|
|
||||||
## Reporting a vulnerability
|
## Reporting a vulnerability
|
||||||
|
|
||||||
If you find a security vulnerability, please [file a
|
If you find a security vulnerability, please [file a
|
||||||
ticket](https://torsion.org/borgmatic/#issues) or [send email
|
ticket](https://torsion.org/borgmatic/#issues) or [send email
|
||||||
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
|
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
|
||||||
back within a few days at most, and generally sooner.
|
back within a few days at most and generally sooner.
|
||||||
|
|
|
@ -44,13 +44,18 @@ def _expand_home_directories(directories):
|
||||||
return tuple(os.path.expanduser(directory) for directory in directories)
|
return tuple(os.path.expanduser(directory) for directory in directories)
|
||||||
|
|
||||||
|
|
||||||
def map_directories_to_devices(directories): # pragma: no cover
|
def map_directories_to_devices(directories):
|
||||||
'''
|
'''
|
||||||
Given a sequence of directories, return a map from directory to an identifier for the device on
|
Given a sequence of directories, return a map from directory to an identifier for the device on
|
||||||
which that directory resides. This is handy for determining whether two different directories
|
which that directory resides or None if the path doesn't exist.
|
||||||
are on the same filesystem (have the same device identifier).
|
|
||||||
|
This is handy for determining whether two different directories are on the same filesystem (have
|
||||||
|
the same device identifier).
|
||||||
'''
|
'''
|
||||||
return {directory: os.stat(directory).st_dev for directory in directories}
|
return {
|
||||||
|
directory: os.stat(directory).st_dev if os.path.exists(directory) else None
|
||||||
|
for directory in directories
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def deduplicate_directories(directory_devices):
|
def deduplicate_directories(directory_devices):
|
||||||
|
@ -82,6 +87,7 @@ def deduplicate_directories(directory_devices):
|
||||||
for parent in parents:
|
for parent in parents:
|
||||||
if (
|
if (
|
||||||
pathlib.PurePath(other_directory) == parent
|
pathlib.PurePath(other_directory) == parent
|
||||||
|
and directory_devices[directory] is not None
|
||||||
and directory_devices[other_directory] == directory_devices[directory]
|
and directory_devices[other_directory] == directory_devices[directory]
|
||||||
):
|
):
|
||||||
if directory in deduplicated:
|
if directory in deduplicated:
|
||||||
|
|
|
@ -135,12 +135,14 @@ properties:
|
||||||
type: string
|
type: string
|
||||||
description: |
|
description: |
|
||||||
Any paths matching these patterns are excluded from backups.
|
Any paths matching these patterns are excluded from backups.
|
||||||
Globs and tildes are expanded. Do not backslash spaces in
|
Globs and tildes are expanded. (Note however that a glob
|
||||||
path names. See the output of "borg help patterns" for more
|
pattern must either start with a glob or be an absolute
|
||||||
details.
|
path.) Do not backslash spaces in path names. See the output
|
||||||
|
of "borg help patterns" for more details.
|
||||||
example:
|
example:
|
||||||
- '*.pyc'
|
- '*.pyc'
|
||||||
- /home/*/.cache
|
- /home/*/.cache
|
||||||
|
- '*/.vim*.tmp'
|
||||||
- /etc/ssl
|
- /etc/ssl
|
||||||
- /home/user/path with spaces
|
- /home/user/path with spaces
|
||||||
exclude_from:
|
exclude_from:
|
||||||
|
@ -298,7 +300,7 @@ properties:
|
||||||
$borg_base_directory/.config/borg/keys
|
$borg_base_directory/.config/borg/keys
|
||||||
example: /path/to/base/config/keys
|
example: /path/to/base/config/keys
|
||||||
umask:
|
umask:
|
||||||
type: string
|
type: integer
|
||||||
description: Umask to be used for borg create. Defaults to 0077.
|
description: Umask to be used for borg create. Defaults to 0077.
|
||||||
example: 0077
|
example: 0077
|
||||||
lock_wait:
|
lock_wait:
|
||||||
|
@ -639,7 +641,7 @@ properties:
|
||||||
Password with which to connect to the database.
|
Password with which to connect to the database.
|
||||||
Omitting a password will only work if PostgreSQL
|
Omitting a password will only work if PostgreSQL
|
||||||
is configured to trust the configured username
|
is configured to trust the configured username
|
||||||
without a password, or you create a ~/.pgpass
|
without a password or you create a ~/.pgpass
|
||||||
file.
|
file.
|
||||||
example: trustsome1
|
example: trustsome1
|
||||||
format:
|
format:
|
||||||
|
@ -793,7 +795,7 @@ properties:
|
||||||
example:
|
example:
|
||||||
https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01
|
https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01
|
||||||
umask:
|
umask:
|
||||||
type: scalar
|
type: integer
|
||||||
description: |
|
description: |
|
||||||
Umask used when executing hooks. Defaults to the umask that
|
Umask used when executing hooks. Defaults to the umask that
|
||||||
borgmatic is run with.
|
borgmatic is run with.
|
||||||
|
|
|
@ -110,7 +110,10 @@ def parse_configuration(config_filename, schema_filename, overrides=None):
|
||||||
override.apply_overrides(config, overrides)
|
override.apply_overrides(config, overrides)
|
||||||
normalize.normalize(config)
|
normalize.normalize(config)
|
||||||
|
|
||||||
validator = jsonschema.Draft7Validator(schema)
|
try:
|
||||||
|
validator = jsonschema.Draft7Validator(schema)
|
||||||
|
except AttributeError:
|
||||||
|
validator = jsonschema.Draft4Validator(schema)
|
||||||
validation_errors = tuple(validator.iter_errors(config))
|
validation_errors = tuple(validator.iter_errors(config))
|
||||||
|
|
||||||
if validation_errors:
|
if validation_errors:
|
||||||
|
|
|
@ -59,11 +59,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
|
||||||
'''
|
'''
|
||||||
# Map from output buffer to sequence of last lines.
|
# Map from output buffer to sequence of last lines.
|
||||||
buffer_last_lines = collections.defaultdict(list)
|
buffer_last_lines = collections.defaultdict(list)
|
||||||
output_buffers = [
|
process_for_output_buffer = {
|
||||||
output_buffer_for_process(process, exclude_stdouts)
|
output_buffer_for_process(process, exclude_stdouts): process
|
||||||
for process in processes
|
for process in processes
|
||||||
if process.stdout or process.stderr
|
if process.stdout or process.stderr
|
||||||
]
|
}
|
||||||
|
output_buffers = list(process_for_output_buffer.keys())
|
||||||
|
|
||||||
# Log output for each process until they all exit.
|
# Log output for each process until they all exit.
|
||||||
while True:
|
while True:
|
||||||
|
@ -71,8 +72,23 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
|
||||||
(ready_buffers, _, _) = select.select(output_buffers, [], [])
|
(ready_buffers, _, _) = select.select(output_buffers, [], [])
|
||||||
|
|
||||||
for ready_buffer in ready_buffers:
|
for ready_buffer in ready_buffers:
|
||||||
|
ready_process = process_for_output_buffer.get(ready_buffer)
|
||||||
|
|
||||||
|
# The "ready" process has exited, but it might be a pipe destination with other
|
||||||
|
# processes (pipe sources) waiting to be read from. So as a measure to prevent
|
||||||
|
# hangs, vent all processes when one exits.
|
||||||
|
if ready_process and ready_process.poll() is not None:
|
||||||
|
for other_process in processes:
|
||||||
|
if (
|
||||||
|
other_process.poll() is None
|
||||||
|
and other_process.stdout
|
||||||
|
and other_process.stdout not in output_buffers
|
||||||
|
):
|
||||||
|
# Add the process's output to output_buffers to ensure it'll get read.
|
||||||
|
output_buffers.append(other_process.stdout)
|
||||||
|
|
||||||
line = ready_buffer.readline().rstrip().decode()
|
line = ready_buffer.readline().rstrip().decode()
|
||||||
if not line:
|
if not line or not ready_process:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Keep the last few lines of output in case the process errors, and we need the output for
|
# Keep the last few lines of output in case the process errors, and we need the output for
|
||||||
|
@ -123,9 +139,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
|
||||||
if not output_buffer:
|
if not output_buffer:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
remaining_output = output_buffer.read().rstrip().decode()
|
while True: # pragma: no cover
|
||||||
|
remaining_output = output_buffer.readline().rstrip().decode()
|
||||||
|
|
||||||
|
if not remaining_output:
|
||||||
|
break
|
||||||
|
|
||||||
if remaining_output: # pragma: no cover
|
|
||||||
logger.log(output_log_level, remaining_output)
|
logger.log(output_log_level, remaining_output)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -1,17 +1,5 @@
|
||||||
<h2>Improve this documentation</h2>
|
<h2>Improve this documentation</h2>
|
||||||
|
|
||||||
<p>Have an idea on how to make this documentation even better? Use our <a
|
<p>Have an idea on how to make this documentation even better? Use our <a
|
||||||
href="https://projects.torsion.org/witten/borgmatic/issues">issue tracker</a> to send your
|
href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues">issue tracker</a> to send your
|
||||||
feedback!</p>
|
feedback!</p>
|
||||||
|
|
||||||
<script>
|
|
||||||
document.getElementById('_page').value = window.location.href;
|
|
||||||
window.sk=window.sk||function(){(sk.q=sk.q||[]).push(arguments)};
|
|
||||||
|
|
||||||
sk('form', 'init', {
|
|
||||||
id: '1d536680ab96',
|
|
||||||
element: '#suggestion-form'
|
|
||||||
});
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<script defer src="https://js.statickit.com/statickit.js"></script>
|
|
||||||
|
|
|
@ -10,17 +10,17 @@ eleventyNavigation:
|
||||||
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
|
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://projects.torsion.org/witten/borgmatic.git
|
git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git
|
||||||
```
|
```
|
||||||
|
|
||||||
Or:
|
Or:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
|
git clone ssh://git@projects.torsion.org:3022/borgmatic-collective/borgmatic.git
|
||||||
```
|
```
|
||||||
|
|
||||||
Then, install borgmatic
|
Then, install borgmatic
|
||||||
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
|
"[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)"
|
||||||
so that you can run borgmatic commands while you're hacking on them to
|
so that you can run borgmatic commands while you're hacking on them to
|
||||||
make sure your changes work.
|
make sure your changes work.
|
||||||
|
|
||||||
|
@ -66,8 +66,6 @@ following:
|
||||||
tox -e black
|
tox -e black
|
||||||
```
|
```
|
||||||
|
|
||||||
Note that Black requires at minimum Python 3.6.
|
|
||||||
|
|
||||||
And if you get a complaint from the
|
And if you get a complaint from the
|
||||||
[isort](https://github.com/timothycrosley/isort) Python import orderer, you
|
[isort](https://github.com/timothycrosley/isort) Python import orderer, you
|
||||||
can ask isort to order your imports for you:
|
can ask isort to order your imports for you:
|
||||||
|
@ -118,7 +116,7 @@ See the Black, Flake8, and isort documentation for more information.
|
||||||
|
|
||||||
Each pull request triggers a continuous integration build which runs the test
|
Each pull request triggers a continuous integration build which runs the test
|
||||||
suite. You can view these builds on
|
suite. You can view these builds on
|
||||||
[build.torsion.org](https://build.torsion.org/witten/borgmatic), and they're
|
[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're
|
||||||
also linked from the commits list on each pull request.
|
also linked from the commits list on each pull request.
|
||||||
|
|
||||||
## Documentation development
|
## Documentation development
|
||||||
|
|
|
@ -28,7 +28,7 @@ sudo pip3 install --user --upgrade borgmatic
|
||||||
This installs borgmatic and its commands at the `/root/.local/bin` path.
|
This installs borgmatic and its commands at the `/root/.local/bin` path.
|
||||||
|
|
||||||
Your pip binary may have a different name than "pip3". Make sure you're using
|
Your pip binary may have a different name than "pip3". Make sure you're using
|
||||||
Python 3, as borgmatic does not support Python 2.
|
Python 3.6+, as borgmatic does not support Python 2.
|
||||||
|
|
||||||
The next step is to ensure that borgmatic's commands available are on your
|
The next step is to ensure that borgmatic's commands available are on your
|
||||||
system `PATH`, so that you can run borgmatic:
|
system `PATH`, so that you can run borgmatic:
|
||||||
|
@ -77,7 +77,7 @@ on a relatively dedicated system, then a global install can work out fine.
|
||||||
Besides the approaches described above, there are several other options for
|
Besides the approaches described above, there are several other options for
|
||||||
installing borgmatic:
|
installing borgmatic:
|
||||||
|
|
||||||
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/)
|
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files)
|
||||||
* [Docker base image](https://hub.docker.com/r/monachus/borgmatic/)
|
* [Docker base image](https://hub.docker.com/r/monachus/borgmatic/)
|
||||||
* [Debian](https://tracker.debian.org/pkg/borgmatic)
|
* [Debian](https://tracker.debian.org/pkg/borgmatic)
|
||||||
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
|
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
|
||||||
|
@ -250,7 +250,7 @@ that, you can configure a separate job runner to invoke it periodically.
|
||||||
### cron
|
### cron
|
||||||
|
|
||||||
If you're using cron, download the [sample cron
|
If you're using cron, download the [sample cron
|
||||||
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
|
file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic).
|
||||||
Then, from the directory where you downloaded it:
|
Then, from the directory where you downloaded it:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
@ -258,7 +258,10 @@ sudo mv borgmatic /etc/cron.d/borgmatic
|
||||||
sudo chmod +x /etc/cron.d/borgmatic
|
sudo chmod +x /etc/cron.d/borgmatic
|
||||||
```
|
```
|
||||||
|
|
||||||
You can modify the cron file if you'd like to run borgmatic more or less frequently.
|
If borgmatic is installed at a different location than
|
||||||
|
`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You
|
||||||
|
can also modify the cron file if you'd like to run borgmatic more or less
|
||||||
|
frequently.
|
||||||
|
|
||||||
### systemd
|
### systemd
|
||||||
|
|
||||||
|
@ -271,9 +274,9 @@ you may already have borgmatic systemd service and timer files. If so, you may
|
||||||
be able to skip some of the steps below.)
|
be able to skip some of the steps below.)
|
||||||
|
|
||||||
First, download the [sample systemd service
|
First, download the [sample systemd service
|
||||||
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
|
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
|
||||||
and the [sample systemd timer
|
and the [sample systemd timer
|
||||||
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
|
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
|
||||||
|
|
||||||
Then, from the directory where you downloaded them:
|
Then, from the directory where you downloaded them:
|
||||||
|
|
||||||
|
@ -294,7 +297,7 @@ borgmatic to run.
|
||||||
If you run borgmatic in macOS with launchd, you may encounter permissions
|
If you run borgmatic in macOS with launchd, you may encounter permissions
|
||||||
issues when reading files to backup. If that happens to you, you may be
|
issues when reading files to backup. If that happens to you, you may be
|
||||||
interested in an [unofficial work-around for Full Disk
|
interested in an [unofficial work-around for Full Disk
|
||||||
Access](https://projects.torsion.org/witten/borgmatic/issues/293).
|
Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293).
|
||||||
|
|
||||||
|
|
||||||
## Colored output
|
## Colored output
|
||||||
|
|
|
@ -32,13 +32,16 @@ RestrictSUIDSGID=yes
|
||||||
SystemCallArchitectures=native
|
SystemCallArchitectures=native
|
||||||
SystemCallFilter=@system-service
|
SystemCallFilter=@system-service
|
||||||
SystemCallErrorNumber=EPERM
|
SystemCallErrorNumber=EPERM
|
||||||
# Restrict write access
|
# To restrict write access further, change "ProtectSystem" to "strict" and uncomment
|
||||||
# Change to 'ProtectSystem=strict' and uncomment 'ProtectHome' to make the whole file
|
# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
|
||||||
# system read-only be default and uncomment 'ReadWritePaths' for the required write access.
|
# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
|
||||||
# Add local repositroy paths to the list of 'ReadWritePaths' like '-/mnt/my_backup_drive'.
|
# leaves most of the filesystem read-only to borgmatic.
|
||||||
ProtectSystem=full
|
ProtectSystem=full
|
||||||
# ProtectHome=read-only
|
# ReadWritePaths=-/mnt/my_backup_drive
|
||||||
# ReadWritePaths=-/root/.config/borg -/root/.cache/borg -/root/.borgmatic
|
# ReadOnlyPaths=-/var/lib/my_backup_source
|
||||||
|
# This will mount a tmpfs on top of /root and pass through needed paths
|
||||||
|
# ProtectHome=tmpfs
|
||||||
|
# BindPaths=-/root/.cache/borg -/root/.cache/borg -/root/.borgmatic
|
||||||
|
|
||||||
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW
|
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW
|
||||||
|
|
||||||
|
|
|
@ -38,7 +38,7 @@ twine upload -r pypi dist/borgmatic-*-py3-none-any.whl dist/borgmatic-*-py3-none
|
||||||
release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
|
release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
|
||||||
escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
|
escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
|
||||||
curl --silent --request POST \
|
curl --silent --request POST \
|
||||||
"https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" \
|
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases" \
|
||||||
--header "Authorization: token $projects_token" \
|
--header "Authorization: token $projects_token" \
|
||||||
--header "Accept: application/json" \
|
--header "Accept: application/json" \
|
||||||
--header "Content-Type: application/json" \
|
--header "Content-Type: application/json" \
|
||||||
|
|
|
@ -13,8 +13,8 @@ set -e
|
||||||
apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client
|
apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client
|
||||||
# If certain dependencies of black are available in this version of Alpine, install them.
|
# If certain dependencies of black are available in this version of Alpine, install them.
|
||||||
apk add --no-cache py3-typed-ast py3-regex || true
|
apk add --no-cache py3-typed-ast py3-regex || true
|
||||||
python3 -m pip install --upgrade pip==20.2.4 setuptools==50.3.2
|
python3 -m pip install --upgrade pip==21.3.1 setuptools==58.2.0
|
||||||
pip3 install tox==3.20.1
|
pip3 install tox==3.24.4
|
||||||
export COVERAGE_FILE=/tmp/.coverage
|
export COVERAGE_FILE=/tmp/.coverage
|
||||||
tox --workdir /tmp/.tox --sitepackages
|
tox --workdir /tmp/.tox --sitepackages
|
||||||
tox --workdir /tmp/.tox --sitepackages -e end-to-end
|
tox --workdir /tmp/.tox --sitepackages -e end-to-end
|
||||||
|
|
2
setup.py
2
setup.py
|
@ -1,6 +1,6 @@
|
||||||
from setuptools import find_packages, setup
|
from setuptools import find_packages, setup
|
||||||
|
|
||||||
VERSION = '1.5.16.dev0'
|
VERSION = '1.5.21.dev0'
|
||||||
|
|
||||||
|
|
||||||
setup(
|
setup(
|
||||||
|
|
|
@ -1,5 +1,6 @@
|
||||||
import logging
|
import logging
|
||||||
import subprocess
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from flexmock import flexmock
|
from flexmock import flexmock
|
||||||
|
@ -98,7 +99,7 @@ def test_log_outputs_kills_other_processes_when_one_errors():
|
||||||
process, 2, 'borg'
|
process, 2, 'borg'
|
||||||
).and_return(True)
|
).and_return(True)
|
||||||
other_process = subprocess.Popen(
|
other_process = subprocess.Popen(
|
||||||
['watch', 'true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
|
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
|
||||||
)
|
)
|
||||||
flexmock(module).should_receive('exit_code_indicates_error').with_args(
|
flexmock(module).should_receive('exit_code_indicates_error').with_args(
|
||||||
other_process, None, 'borg'
|
other_process, None, 'borg'
|
||||||
|
@ -123,6 +124,75 @@ def test_log_outputs_kills_other_processes_when_one_errors():
|
||||||
assert error.value.output
|
assert error.value.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_outputs_vents_other_processes_when_one_exits():
|
||||||
|
'''
|
||||||
|
Execute a command to generate a longish random string and pipe it into another command that
|
||||||
|
exits quickly. The test is basically to ensure we don't hang forever waiting for the exited
|
||||||
|
process to read the pipe, and that the string-generating process eventually gets vented and
|
||||||
|
exits.
|
||||||
|
'''
|
||||||
|
flexmock(module.logger).should_receive('log')
|
||||||
|
flexmock(module).should_receive('command_for_process').and_return('grep')
|
||||||
|
|
||||||
|
process = subprocess.Popen(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
'-c',
|
||||||
|
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
|
||||||
|
],
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
)
|
||||||
|
other_process = subprocess.Popen(
|
||||||
|
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
|
||||||
|
)
|
||||||
|
flexmock(module).should_receive('output_buffer_for_process').with_args(
|
||||||
|
process, (process.stdout,)
|
||||||
|
).and_return(process.stderr)
|
||||||
|
flexmock(module).should_receive('output_buffer_for_process').with_args(
|
||||||
|
other_process, (process.stdout,)
|
||||||
|
).and_return(other_process.stdout)
|
||||||
|
flexmock(process.stdout).should_call('readline').at_least().once()
|
||||||
|
|
||||||
|
module.log_outputs(
|
||||||
|
(process, other_process),
|
||||||
|
exclude_stdouts=(process.stdout,),
|
||||||
|
output_log_level=logging.INFO,
|
||||||
|
borg_local_path='borg',
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_outputs_does_not_error_when_one_process_exits():
|
||||||
|
flexmock(module.logger).should_receive('log')
|
||||||
|
flexmock(module).should_receive('command_for_process').and_return('grep')
|
||||||
|
|
||||||
|
process = subprocess.Popen(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
'-c',
|
||||||
|
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
|
||||||
|
],
|
||||||
|
stdout=None, # Specifically test the case of a process without stdout captured.
|
||||||
|
stderr=None,
|
||||||
|
)
|
||||||
|
other_process = subprocess.Popen(
|
||||||
|
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
|
||||||
|
)
|
||||||
|
flexmock(module).should_receive('output_buffer_for_process').with_args(
|
||||||
|
process, (process.stdout,)
|
||||||
|
).and_return(process.stderr)
|
||||||
|
flexmock(module).should_receive('output_buffer_for_process').with_args(
|
||||||
|
other_process, (process.stdout,)
|
||||||
|
).and_return(other_process.stdout)
|
||||||
|
|
||||||
|
module.log_outputs(
|
||||||
|
(process, other_process),
|
||||||
|
exclude_stdouts=(process.stdout,),
|
||||||
|
output_log_level=logging.INFO,
|
||||||
|
borg_local_path='borg',
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_log_outputs_truncates_long_error_output():
|
def test_log_outputs_truncates_long_error_output():
|
||||||
flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0
|
flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0
|
||||||
flexmock(module.logger).should_receive('log')
|
flexmock(module.logger).should_receive('log')
|
||||||
|
|
|
@ -60,6 +60,30 @@ def test_expand_home_directories_considers_none_as_no_directories():
|
||||||
assert paths == ()
|
assert paths == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_map_directories_to_devices_gives_device_id_per_path():
|
||||||
|
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
|
||||||
|
flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
|
||||||
|
|
||||||
|
device_map = module.map_directories_to_devices(('/foo', '/bar'))
|
||||||
|
|
||||||
|
assert device_map == {
|
||||||
|
'/foo': 55,
|
||||||
|
'/bar': 66,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_map_directories_to_devices_with_missing_path_does_not_error():
|
||||||
|
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
|
||||||
|
flexmock(module.os).should_receive('stat').with_args('/bar').and_raise(FileNotFoundError)
|
||||||
|
|
||||||
|
device_map = module.map_directories_to_devices(('/foo', '/bar'))
|
||||||
|
|
||||||
|
assert device_map == {
|
||||||
|
'/foo': 55,
|
||||||
|
'/bar': None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
'directories,expected_directories',
|
'directories,expected_directories',
|
||||||
(
|
(
|
||||||
|
@ -72,6 +96,7 @@ def test_expand_home_directories_considers_none_as_no_directories():
|
||||||
({'/root': 1, '/root/foo/': 1}, ('/root',)),
|
({'/root': 1, '/root/foo/': 1}, ('/root',)),
|
||||||
({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')),
|
({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')),
|
||||||
({'/root/foo': 1, '/root': 1}, ('/root',)),
|
({'/root/foo': 1, '/root': 1}, ('/root',)),
|
||||||
|
({'/root': None, '/root/foo': None}, ('/root', '/root/foo')),
|
||||||
({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')),
|
({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')),
|
||||||
({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)),
|
({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)),
|
||||||
({'/dup': 1, '/dup': 1}, ('/dup',)),
|
({'/dup': 1, '/dup': 1}, ('/dup',)),
|
||||||
|
|
Loading…
Reference in a new issue