Merge branch 'master' into master

This commit is contained in:
cadamswaite 2021-11-14 18:24:17 +00:00
commit 6b182c9d2d
17 changed files with 224 additions and 68 deletions

View file

@ -14,6 +14,9 @@ services:
MYSQL_ROOT_PASSWORD: test MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test MYSQL_DATABASE: test
clone:
skip_verify: true
steps: steps:
- name: build - name: build
image: alpine:3.9 image: alpine:3.9
@ -36,6 +39,9 @@ services:
MYSQL_ROOT_PASSWORD: test MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test MYSQL_DATABASE: test
clone:
skip_verify: true
steps: steps:
- name: build - name: build
image: alpine:3.10 image: alpine:3.10
@ -58,6 +64,9 @@ services:
MYSQL_ROOT_PASSWORD: test MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test MYSQL_DATABASE: test
clone:
skip_verify: true
steps: steps:
- name: build - name: build
image: alpine:3.13 image: alpine:3.13
@ -68,9 +77,14 @@ steps:
kind: pipeline kind: pipeline
name: documentation name: documentation
clone:
skip_verify: true
steps: steps:
- name: build - name: build
image: plugins/docker #image: plugins/docker
# Temporary work-around for https://github.com/drone-plugins/drone-docker/pull/327
image: techknowlogick/drone-docker
settings: settings:
username: username:
from_secret: docker_username from_secret: docker_username
@ -80,5 +94,7 @@ steps:
dockerfile: docs/Dockerfile dockerfile: docs/Dockerfile
trigger: trigger:
repo:
- borgmatic-collective/borgmatic
branch: branch:
- master - master

28
NEWS
View file

@ -1,6 +1,30 @@
1.5.16.dev0 1.5.21.dev0
* Add support for old version (2.x) of jsonschema library.
1.5.20
* Re-release with correct version without dev0 tag.
1.5.19
* #387: Fix error when configured source directories are not present on the filesystem at the time
of backup. Now, Borg will complain, but the backup will still continue.
* #455: Mention changing borgmatic path in cron documentation.
* Update sample systemd service file with more granular read-only filesystem settings.
* Move Gitea and GitHub hosting from a personal namespace to an organization for better
collaboration with related projects.
* 1k ★s on GitHub!
1.5.18
* #389: Fix "message too long" error when logging to rsyslog.
* #440: Fix traceback that can occur when dumping a database.
1.5.17
* #437: Fix error when configuration file contains "umask" option.
* Remove test dependency on vim and /dev/urandom.
1.5.16
* #379: Suppress console output in sample crontab and systemd service files. * #379: Suppress console output in sample crontab and systemd service files.
* #407: Fix syslog logging on FreeBSD. * #407: Fix syslog logging on FreeBSD.
* #430: Fix hang when restoring a PostgreSQL "tar" format database dump.
* Better error messages! Switch the library used for validating configuration files (from pykwalify * Better error messages! Switch the library used for validating configuration files (from pykwalify
to jsonschema). to jsonschema).
* Link borgmatic Ansible role from installation documentation: * Link borgmatic Ansible role from installation documentation:
@ -559,7 +583,7 @@
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed * #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
includes/excludes. includes/excludes.
* Moved issue tracker from Taiga to integrated Gitea tracker at * Moved issue tracker from Taiga to integrated Gitea tracker at
https://projects.torsion.org/witten/borgmatic/issues https://projects.torsion.org/borgmatic-collective/borgmatic/issues
1.1.12 1.1.12
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version" * #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"

View file

@ -106,7 +106,7 @@ development or hosting.
### Issues ### Issues
You've got issues? Or an idea for a feature enhancement? We've got an [issue You've got issues? Or an idea for a feature enhancement? We've got an [issue
tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to tracker](https://projects.torsion.org/borgmatic-collective/borgmatic/issues). In order to
create a new issue or comment on an issue, you'll need to [login create a new issue or comment on an issue, you'll need to [login
first](https://projects.torsion.org/user/login). Note that you can login with first](https://projects.torsion.org/user/login). Note that you can login with
an existing GitHub account if you prefer. an existing GitHub account if you prefer.
@ -129,15 +129,15 @@ Other questions or comments? Contact
### Contributing ### Contributing
borgmatic [source code is borgmatic [source code is
available](https://projects.torsion.org/witten/borgmatic) and is also mirrored available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
on [GitHub](https://github.com/witten/borgmatic) for convenience. on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
borgmatic is licensed under the GNU General Public License version 3 or any borgmatic is licensed under the GNU General Public License version 3 or any
later version. later version.
If you'd like to contribute to borgmatic development, please feel free to If you'd like to contribute to borgmatic development, please feel free to
submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls) submit a [Pull Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first or open an [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) first
to discuss your idea. We also accept Pull Requests on GitHub, if that's more to discuss your idea. We also accept Pull Requests on GitHub, if that's more
your thing. In general, contributions are very welcome. We don't bite! your thing. In general, contributions are very welcome. We don't bite!
@ -145,5 +145,5 @@ Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc. info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/witten/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/witten/borgmatic/status.svg?ref=refs/heads/master)</a> <a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/master)</a>

View file

@ -6,14 +6,13 @@ permalink: security-policy/index.html
## Supported versions ## Supported versions
While we want to hear about security vulnerabilities in all versions of While we want to hear about security vulnerabilities in all versions of
borgmatic, security fixes will only be made to the most recently released borgmatic, security fixes are only made to the most recently released version.
version. It's not practical for our small volunteer effort to maintain It's simply not practical for our small volunteer effort to maintain multiple
multiple different release branches and put out separate security patches for release branches and put out separate security patches for each.
each.
## Reporting a vulnerability ## Reporting a vulnerability
If you find a security vulnerability, please [file a If you find a security vulnerability, please [file a
ticket](https://torsion.org/borgmatic/#issues) or [send email ticket](https://torsion.org/borgmatic/#issues) or [send email
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
back within a few days at most, and generally sooner. back within a few days at most and generally sooner.

View file

@ -44,13 +44,18 @@ def _expand_home_directories(directories):
return tuple(os.path.expanduser(directory) for directory in directories) return tuple(os.path.expanduser(directory) for directory in directories)
def map_directories_to_devices(directories): # pragma: no cover def map_directories_to_devices(directories):
''' '''
Given a sequence of directories, return a map from directory to an identifier for the device on Given a sequence of directories, return a map from directory to an identifier for the device on
which that directory resides. This is handy for determining whether two different directories which that directory resides or None if the path doesn't exist.
are on the same filesystem (have the same device identifier).
This is handy for determining whether two different directories are on the same filesystem (have
the same device identifier).
''' '''
return {directory: os.stat(directory).st_dev for directory in directories} return {
directory: os.stat(directory).st_dev if os.path.exists(directory) else None
for directory in directories
}
def deduplicate_directories(directory_devices): def deduplicate_directories(directory_devices):
@ -82,6 +87,7 @@ def deduplicate_directories(directory_devices):
for parent in parents: for parent in parents:
if ( if (
pathlib.PurePath(other_directory) == parent pathlib.PurePath(other_directory) == parent
and directory_devices[directory] is not None
and directory_devices[other_directory] == directory_devices[directory] and directory_devices[other_directory] == directory_devices[directory]
): ):
if directory in deduplicated: if directory in deduplicated:

View file

@ -135,12 +135,14 @@ properties:
type: string type: string
description: | description: |
Any paths matching these patterns are excluded from backups. Any paths matching these patterns are excluded from backups.
Globs and tildes are expanded. Do not backslash spaces in Globs and tildes are expanded. (Note however that a glob
path names. See the output of "borg help patterns" for more pattern must either start with a glob or be an absolute
details. path.) Do not backslash spaces in path names. See the output
of "borg help patterns" for more details.
example: example:
- '*.pyc' - '*.pyc'
- /home/*/.cache - /home/*/.cache
- '*/.vim*.tmp'
- /etc/ssl - /etc/ssl
- /home/user/path with spaces - /home/user/path with spaces
exclude_from: exclude_from:
@ -298,7 +300,7 @@ properties:
$borg_base_directory/.config/borg/keys $borg_base_directory/.config/borg/keys
example: /path/to/base/config/keys example: /path/to/base/config/keys
umask: umask:
type: string type: integer
description: Umask to be used for borg create. Defaults to 0077. description: Umask to be used for borg create. Defaults to 0077.
example: 0077 example: 0077
lock_wait: lock_wait:
@ -639,7 +641,7 @@ properties:
Password with which to connect to the database. Password with which to connect to the database.
Omitting a password will only work if PostgreSQL Omitting a password will only work if PostgreSQL
is configured to trust the configured username is configured to trust the configured username
without a password, or you create a ~/.pgpass without a password or you create a ~/.pgpass
file. file.
example: trustsome1 example: trustsome1
format: format:
@ -793,7 +795,7 @@ properties:
example: example:
https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01 https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01
umask: umask:
type: scalar type: integer
description: | description: |
Umask used when executing hooks. Defaults to the umask that Umask used when executing hooks. Defaults to the umask that
borgmatic is run with. borgmatic is run with.

View file

@ -110,7 +110,10 @@ def parse_configuration(config_filename, schema_filename, overrides=None):
override.apply_overrides(config, overrides) override.apply_overrides(config, overrides)
normalize.normalize(config) normalize.normalize(config)
validator = jsonschema.Draft7Validator(schema) try:
validator = jsonschema.Draft7Validator(schema)
except AttributeError:
validator = jsonschema.Draft4Validator(schema)
validation_errors = tuple(validator.iter_errors(config)) validation_errors = tuple(validator.iter_errors(config))
if validation_errors: if validation_errors:

View file

@ -59,11 +59,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
''' '''
# Map from output buffer to sequence of last lines. # Map from output buffer to sequence of last lines.
buffer_last_lines = collections.defaultdict(list) buffer_last_lines = collections.defaultdict(list)
output_buffers = [ process_for_output_buffer = {
output_buffer_for_process(process, exclude_stdouts) output_buffer_for_process(process, exclude_stdouts): process
for process in processes for process in processes
if process.stdout or process.stderr if process.stdout or process.stderr
] }
output_buffers = list(process_for_output_buffer.keys())
# Log output for each process until they all exit. # Log output for each process until they all exit.
while True: while True:
@ -71,8 +72,23 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
(ready_buffers, _, _) = select.select(output_buffers, [], []) (ready_buffers, _, _) = select.select(output_buffers, [], [])
for ready_buffer in ready_buffers: for ready_buffer in ready_buffers:
ready_process = process_for_output_buffer.get(ready_buffer)
# The "ready" process has exited, but it might be a pipe destination with other
# processes (pipe sources) waiting to be read from. So as a measure to prevent
# hangs, vent all processes when one exits.
if ready_process and ready_process.poll() is not None:
for other_process in processes:
if (
other_process.poll() is None
and other_process.stdout
and other_process.stdout not in output_buffers
):
# Add the process's output to output_buffers to ensure it'll get read.
output_buffers.append(other_process.stdout)
line = ready_buffer.readline().rstrip().decode() line = ready_buffer.readline().rstrip().decode()
if not line: if not line or not ready_process:
continue continue
# Keep the last few lines of output in case the process errors, and we need the output for # Keep the last few lines of output in case the process errors, and we need the output for
@ -123,9 +139,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
if not output_buffer: if not output_buffer:
continue continue
remaining_output = output_buffer.read().rstrip().decode() while True: # pragma: no cover
remaining_output = output_buffer.readline().rstrip().decode()
if not remaining_output:
break
if remaining_output: # pragma: no cover
logger.log(output_log_level, remaining_output) logger.log(output_log_level, remaining_output)

View file

@ -1,17 +1,5 @@
<h2>Improve this documentation</h2> <h2>Improve this documentation</h2>
<p>Have an idea on how to make this documentation even better? Use our <a <p>Have an idea on how to make this documentation even better? Use our <a
href="https://projects.torsion.org/witten/borgmatic/issues">issue tracker</a> to send your href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues">issue tracker</a> to send your
feedback!</p> feedback!</p>
<script>
document.getElementById('_page').value = window.location.href;
window.sk=window.sk||function(){(sk.q=sk.q||[]).push(arguments)};
sk('form', 'init', {
id: '1d536680ab96',
element: '#suggestion-form'
});
</script>
<script defer src="https://js.statickit.com/statickit.js"></script>

View file

@ -10,17 +10,17 @@ eleventyNavigation:
To get set up to hack on borgmatic, first clone master via HTTPS or SSH: To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
```bash ```bash
git clone https://projects.torsion.org/witten/borgmatic.git git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git
``` ```
Or: Or:
```bash ```bash
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git git clone ssh://git@projects.torsion.org:3022/borgmatic-collective/borgmatic.git
``` ```
Then, install borgmatic Then, install borgmatic
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)" "[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)"
so that you can run borgmatic commands while you're hacking on them to so that you can run borgmatic commands while you're hacking on them to
make sure your changes work. make sure your changes work.
@ -66,8 +66,6 @@ following:
tox -e black tox -e black
``` ```
Note that Black requires at minimum Python 3.6.
And if you get a complaint from the And if you get a complaint from the
[isort](https://github.com/timothycrosley/isort) Python import orderer, you [isort](https://github.com/timothycrosley/isort) Python import orderer, you
can ask isort to order your imports for you: can ask isort to order your imports for you:
@ -118,7 +116,7 @@ See the Black, Flake8, and isort documentation for more information.
Each pull request triggers a continuous integration build which runs the test Each pull request triggers a continuous integration build which runs the test
suite. You can view these builds on suite. You can view these builds on
[build.torsion.org](https://build.torsion.org/witten/borgmatic), and they're [build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're
also linked from the commits list on each pull request. also linked from the commits list on each pull request.
## Documentation development ## Documentation development

View file

@ -28,7 +28,7 @@ sudo pip3 install --user --upgrade borgmatic
This installs borgmatic and its commands at the `/root/.local/bin` path. This installs borgmatic and its commands at the `/root/.local/bin` path.
Your pip binary may have a different name than "pip3". Make sure you're using Your pip binary may have a different name than "pip3". Make sure you're using
Python 3, as borgmatic does not support Python 2. Python 3.6+, as borgmatic does not support Python 2.
The next step is to ensure that borgmatic's commands available are on your The next step is to ensure that borgmatic's commands available are on your
system `PATH`, so that you can run borgmatic: system `PATH`, so that you can run borgmatic:
@ -77,7 +77,7 @@ on a relatively dedicated system, then a global install can work out fine.
Besides the approaches described above, there are several other options for Besides the approaches described above, there are several other options for
installing borgmatic: installing borgmatic:
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) * [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files)
* [Docker base image](https://hub.docker.com/r/monachus/borgmatic/) * [Docker base image](https://hub.docker.com/r/monachus/borgmatic/)
* [Debian](https://tracker.debian.org/pkg/borgmatic) * [Debian](https://tracker.debian.org/pkg/borgmatic)
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic) * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
@ -250,7 +250,7 @@ that, you can configure a separate job runner to invoke it periodically.
### cron ### cron
If you're using cron, download the [sample cron If you're using cron, download the [sample cron
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic). file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic).
Then, from the directory where you downloaded it: Then, from the directory where you downloaded it:
```bash ```bash
@ -258,7 +258,10 @@ sudo mv borgmatic /etc/cron.d/borgmatic
sudo chmod +x /etc/cron.d/borgmatic sudo chmod +x /etc/cron.d/borgmatic
``` ```
You can modify the cron file if you'd like to run borgmatic more or less frequently. If borgmatic is installed at a different location than
`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You
can also modify the cron file if you'd like to run borgmatic more or less
frequently.
### systemd ### systemd
@ -271,9 +274,9 @@ you may already have borgmatic systemd service and timer files. If so, you may
be able to skip some of the steps below.) be able to skip some of the steps below.)
First, download the [sample systemd service First, download the [sample systemd service
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service) file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
and the [sample systemd timer and the [sample systemd timer
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer). file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
Then, from the directory where you downloaded them: Then, from the directory where you downloaded them:
@ -294,7 +297,7 @@ borgmatic to run.
If you run borgmatic in macOS with launchd, you may encounter permissions If you run borgmatic in macOS with launchd, you may encounter permissions
issues when reading files to backup. If that happens to you, you may be issues when reading files to backup. If that happens to you, you may be
interested in an [unofficial work-around for Full Disk interested in an [unofficial work-around for Full Disk
Access](https://projects.torsion.org/witten/borgmatic/issues/293). Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293).
## Colored output ## Colored output

View file

@ -32,13 +32,16 @@ RestrictSUIDSGID=yes
SystemCallArchitectures=native SystemCallArchitectures=native
SystemCallFilter=@system-service SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM SystemCallErrorNumber=EPERM
# Restrict write access # To restrict write access further, change "ProtectSystem" to "strict" and uncomment
# Change to 'ProtectSystem=strict' and uncomment 'ProtectHome' to make the whole file # "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
# system read-only be default and uncomment 'ReadWritePaths' for the required write access. # paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
# Add local repositroy paths to the list of 'ReadWritePaths' like '-/mnt/my_backup_drive'. # leaves most of the filesystem read-only to borgmatic.
ProtectSystem=full ProtectSystem=full
# ProtectHome=read-only # ReadWritePaths=-/mnt/my_backup_drive
# ReadWritePaths=-/root/.config/borg -/root/.cache/borg -/root/.borgmatic # ReadOnlyPaths=-/var/lib/my_backup_source
# This will mount a tmpfs on top of /root and pass through needed paths
# ProtectHome=tmpfs
# BindPaths=-/root/.cache/borg -/root/.cache/borg -/root/.borgmatic
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

View file

@ -38,7 +38,7 @@ twine upload -r pypi dist/borgmatic-*-py3-none-any.whl dist/borgmatic-*-py3-none
release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')" release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')" escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
curl --silent --request POST \ curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" \ "https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases" \
--header "Authorization: token $projects_token" \ --header "Authorization: token $projects_token" \
--header "Accept: application/json" \ --header "Accept: application/json" \
--header "Content-Type: application/json" \ --header "Content-Type: application/json" \

View file

@ -13,8 +13,8 @@ set -e
apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client
# If certain dependencies of black are available in this version of Alpine, install them. # If certain dependencies of black are available in this version of Alpine, install them.
apk add --no-cache py3-typed-ast py3-regex || true apk add --no-cache py3-typed-ast py3-regex || true
python3 -m pip install --upgrade pip==20.2.4 setuptools==50.3.2 python3 -m pip install --upgrade pip==21.3.1 setuptools==58.2.0
pip3 install tox==3.20.1 pip3 install tox==3.24.4
export COVERAGE_FILE=/tmp/.coverage export COVERAGE_FILE=/tmp/.coverage
tox --workdir /tmp/.tox --sitepackages tox --workdir /tmp/.tox --sitepackages
tox --workdir /tmp/.tox --sitepackages -e end-to-end tox --workdir /tmp/.tox --sitepackages -e end-to-end

View file

@ -1,6 +1,6 @@
from setuptools import find_packages, setup from setuptools import find_packages, setup
VERSION = '1.5.16.dev0' VERSION = '1.5.21.dev0'
setup( setup(

View file

@ -1,5 +1,6 @@
import logging import logging
import subprocess import subprocess
import sys
import pytest import pytest
from flexmock import flexmock from flexmock import flexmock
@ -98,7 +99,7 @@ def test_log_outputs_kills_other_processes_when_one_errors():
process, 2, 'borg' process, 2, 'borg'
).and_return(True) ).and_return(True)
other_process = subprocess.Popen( other_process = subprocess.Popen(
['watch', 'true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT ['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
) )
flexmock(module).should_receive('exit_code_indicates_error').with_args( flexmock(module).should_receive('exit_code_indicates_error').with_args(
other_process, None, 'borg' other_process, None, 'borg'
@ -123,6 +124,75 @@ def test_log_outputs_kills_other_processes_when_one_errors():
assert error.value.output assert error.value.output
def test_log_outputs_vents_other_processes_when_one_exits():
'''
Execute a command to generate a longish random string and pipe it into another command that
exits quickly. The test is basically to ensure we don't hang forever waiting for the exited
process to read the pipe, and that the string-generating process eventually gets vented and
exits.
'''
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
[
sys.executable,
'-c',
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
other_process = subprocess.Popen(
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
process, (process.stdout,)
).and_return(process.stderr)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, (process.stdout,)
).and_return(other_process.stdout)
flexmock(process.stdout).should_call('readline').at_least().once()
module.log_outputs(
(process, other_process),
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
)
def test_log_outputs_does_not_error_when_one_process_exits():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
[
sys.executable,
'-c',
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
],
stdout=None, # Specifically test the case of a process without stdout captured.
stderr=None,
)
other_process = subprocess.Popen(
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
process, (process.stdout,)
).and_return(process.stderr)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, (process.stdout,)
).and_return(other_process.stdout)
module.log_outputs(
(process, other_process),
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
)
def test_log_outputs_truncates_long_error_output(): def test_log_outputs_truncates_long_error_output():
flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0 flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0
flexmock(module.logger).should_receive('log') flexmock(module.logger).should_receive('log')

View file

@ -60,6 +60,30 @@ def test_expand_home_directories_considers_none_as_no_directories():
assert paths == () assert paths == ()
def test_map_directories_to_devices_gives_device_id_per_path():
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
device_map = module.map_directories_to_devices(('/foo', '/bar'))
assert device_map == {
'/foo': 55,
'/bar': 66,
}
def test_map_directories_to_devices_with_missing_path_does_not_error():
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
flexmock(module.os).should_receive('stat').with_args('/bar').and_raise(FileNotFoundError)
device_map = module.map_directories_to_devices(('/foo', '/bar'))
assert device_map == {
'/foo': 55,
'/bar': None,
}
@pytest.mark.parametrize( @pytest.mark.parametrize(
'directories,expected_directories', 'directories,expected_directories',
( (
@ -72,6 +96,7 @@ def test_expand_home_directories_considers_none_as_no_directories():
({'/root': 1, '/root/foo/': 1}, ('/root',)), ({'/root': 1, '/root/foo/': 1}, ('/root',)),
({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')), ({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')),
({'/root/foo': 1, '/root': 1}, ('/root',)), ({'/root/foo': 1, '/root': 1}, ('/root',)),
({'/root': None, '/root/foo': None}, ('/root', '/root/foo')),
({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')), ({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')),
({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)), ({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)),
({'/dup': 1, '/dup': 1}, ('/dup',)), ({'/dup': 1, '/dup': 1}, ('/dup',)),