Refactor documentation into multiple separate pages for clarity and findability.
This commit is contained in:
parent
8b4ac0017b
commit
7e0e00d45d
10 changed files with 565 additions and 421 deletions
3
NEWS
3
NEWS
|
@ -1,3 +1,6 @@
|
||||||
|
1.2.16.dev0
|
||||||
|
* Refactor documentation into multiple separate pages for clarity and findability.
|
||||||
|
|
||||||
1.2.15
|
1.2.15
|
||||||
* #127: Remove date echo from schema example, as it's not a substitute for real logging.
|
* #127: Remove date echo from schema example, as it's not a substitute for real logging.
|
||||||
* #132: Leave exclude_patterns glob expansion to Borg, since doing it in borgmatic leads to
|
* #132: Leave exclude_patterns glob expansion to Borg, since doing it in borgmatic leads to
|
||||||
|
|
432
README.md
432
README.md
|
@ -54,338 +54,15 @@ href="https://asciinema.org/a/203761" target="_blank">screencast</a>.
|
||||||
<script src="https://asciinema.org/a/203761.js" id="asciicast-203761" async></script>
|
<script src="https://asciinema.org/a/203761.js" id="asciicast-203761" async></script>
|
||||||
|
|
||||||
|
|
||||||
## Installation
|
## How-to guides
|
||||||
|
|
||||||
To get up and running, first [install
|
* [Set up backups with borgmatic](docs/how-to/set-up-backups.md) \<-- *Start here!*
|
||||||
Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
|
* [Make per-application backups](docs/how-to/make-per-application-backups.md)
|
||||||
least version 1.1.
|
* [Deal with very large backups](docs/how-to/deal-with-very-large-backups.md)
|
||||||
|
* [Inspect your backups](docs/how-to/inspect-your-backups.md)
|
||||||
Then, run the following command to download and install borgmatic:
|
* [Run preparation steps before backups](docs/how-to/run-preparation-steps-before-backups.md)
|
||||||
|
* [Upgrade borgmatic](docs/how-to/upgrade.md)
|
||||||
```bash
|
* [Develop on borgmatic](docs/how-to/develop-on-borgmatic.md)
|
||||||
sudo pip3 install --upgrade borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that your pip binary may have a different name than "pip3". Make sure
|
|
||||||
you're using Python 3, as borgmatic does not support Python 2.
|
|
||||||
|
|
||||||
|
|
||||||
### Other ways to install
|
|
||||||
|
|
||||||
* [Docker image](https://hub.docker.com/r/monachus/borgmatic/)
|
|
||||||
* [Another Docker image](https://hub.docker.com/r/b3vis/borgmatic/)
|
|
||||||
* [Debian](https://tracker.debian.org/pkg/borgmatic)
|
|
||||||
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
|
|
||||||
* [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
|
|
||||||
* [Arch Linux](https://aur.archlinux.org/packages/borgmatic/)
|
|
||||||
* [OpenBSD](http://ports.su/sysutils/borgmatic)
|
|
||||||
* [openSUSE](https://software.opensuse.org/package/borgmatic)
|
|
||||||
|
|
||||||
<br><br>
|
|
||||||
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
After you install borgmatic, generate a sample configuration file:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo generate-borgmatic-config
|
|
||||||
```
|
|
||||||
|
|
||||||
If that command is not found, then it may be installed in a location that's
|
|
||||||
not in your system `PATH`. Try looking in `/usr/local/bin/`.
|
|
||||||
|
|
||||||
This generates a sample configuration file at /etc/borgmatic/config.yaml (by
|
|
||||||
default). You should edit the file to suit your needs, as the values are
|
|
||||||
representative. All fields are optional except where indicated, so feel free
|
|
||||||
to ignore anything you don't need.
|
|
||||||
|
|
||||||
You can also have a look at the [full configuration
|
|
||||||
schema](https://projects.torsion.org/witten/borgmatic/src/master/borgmatic/config/schema.yaml)
|
|
||||||
for the authoritative set of all configuration options. This is handy if
|
|
||||||
borgmatic has added new options since you originally created your
|
|
||||||
configuration file.
|
|
||||||
|
|
||||||
|
|
||||||
### Encryption
|
|
||||||
|
|
||||||
Note that if you plan to run borgmatic on a schedule with cron, and you
|
|
||||||
encrypt your Borg repository with a passphrase instead of a key file, you'll
|
|
||||||
either need to set the borgmatic `encryption_passphrase` configuration
|
|
||||||
variable or set the `BORG_PASSPHRASE` environment variable. See the
|
|
||||||
[repository encryption
|
|
||||||
section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
|
|
||||||
of the Quick Start for more info.
|
|
||||||
|
|
||||||
Alternatively, the passphrase can be specified programatically by setting
|
|
||||||
either the borgmatic `encryption_passcommand` configuration variable or the
|
|
||||||
`BORG_PASSCOMMAND` environment variable. See the [Borg Security
|
|
||||||
FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
|
|
||||||
for more info.
|
|
||||||
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Initialization
|
|
||||||
|
|
||||||
Before you can create backups with borgmatic, you first need to initialize a
|
|
||||||
Borg repository so you have a destination for your backup archives. (But skip
|
|
||||||
this step if you already have a Borg repository.) To create a repository, run
|
|
||||||
a command like the following:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --init --encryption repokey
|
|
||||||
```
|
|
||||||
|
|
||||||
This uses the borgmatic configuration file you created above to determine
|
|
||||||
which local or remote repository to create, and encrypts it with the
|
|
||||||
encryption passphrase specified there if one is provided. Read about [Borg
|
|
||||||
encryption
|
|
||||||
modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
|
|
||||||
for the menu of available encryption modes.
|
|
||||||
|
|
||||||
Also, optionally check out the [Borg Quick
|
|
||||||
Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
|
|
||||||
background about repository initialization.
|
|
||||||
|
|
||||||
Note that borgmatic skips repository initialization if the repository already
|
|
||||||
exists. This supports use cases like ensuring a repository exists prior to
|
|
||||||
performing a backup.
|
|
||||||
|
|
||||||
If the repository is on a remote host, make sure that your local user has
|
|
||||||
key-based SSH access to the desired user account on the remote host.
|
|
||||||
|
|
||||||
|
|
||||||
### Backups
|
|
||||||
|
|
||||||
You can run borgmatic and start a backup by invoking it without arguments:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
This will also prune any old backups as per the configured retention policy,
|
|
||||||
and check backups for consistency problems due to things like file damage.
|
|
||||||
|
|
||||||
If you'd like to see the available command-line arguments, view the help:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --help
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that borgmatic prunes archives *before* creating an archive, so as to
|
|
||||||
free up space for archiving. This means that when a borgmatic run finishes,
|
|
||||||
there may still be prune-able archives. Not to worry, as they will get cleaned
|
|
||||||
up at the start of the next run.
|
|
||||||
|
|
||||||
|
|
||||||
### Verbosity
|
|
||||||
|
|
||||||
By default, the backup will proceed silently except in the case of errors. But
|
|
||||||
if you'd like to to get additional information about the progress of the
|
|
||||||
backup as it proceeds, use the verbosity option:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --verbosity 1
|
|
||||||
```
|
|
||||||
|
|
||||||
Or, for even more progress spew:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --verbosity 2
|
|
||||||
```
|
|
||||||
|
|
||||||
### À la carte
|
|
||||||
|
|
||||||
If you want to run borgmatic with only pruning, creating, or checking enabled,
|
|
||||||
the following optional flags are available:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --prune
|
|
||||||
borgmatic --create
|
|
||||||
borgmatic --check
|
|
||||||
```
|
|
||||||
|
|
||||||
You can run with only one of these flags provided, or you can mix and match
|
|
||||||
any number of them. This supports use cases like running consistency checks
|
|
||||||
from a different cron job with a different frequency, or running pruning with
|
|
||||||
a different verbosity level.
|
|
||||||
|
|
||||||
Additionally, borgmatic provides convenient flags for Borg's
|
|
||||||
[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
|
|
||||||
[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
|
|
||||||
functionality:
|
|
||||||
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borgmatic --list
|
|
||||||
borgmatic --info
|
|
||||||
```
|
|
||||||
|
|
||||||
You can include an optional `--json` flag with `--create`, `--list`, or
|
|
||||||
`--info` to get the output formatted as JSON.
|
|
||||||
|
|
||||||
|
|
||||||
## Autopilot
|
|
||||||
|
|
||||||
If you want to run borgmatic automatically, say once a day, the you can
|
|
||||||
configure a job runner to invoke it periodically.
|
|
||||||
|
|
||||||
### cron
|
|
||||||
|
|
||||||
If you're using cron, download the [sample cron
|
|
||||||
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
|
|
||||||
Then, from the directory where you downloaded it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo mv borgmatic /etc/cron.d/borgmatic
|
|
||||||
sudo chmod +x /etc/cron.d/borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
You can modify the cron file if you'd like to run borgmatic more or less frequently.
|
|
||||||
|
|
||||||
### systemd
|
|
||||||
|
|
||||||
If you're using systemd instead of cron to run jobs, download the [sample
|
|
||||||
systemd service
|
|
||||||
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
|
|
||||||
and the [sample systemd timer
|
|
||||||
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
|
|
||||||
Then, from the directory where you downloaded them:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
|
|
||||||
sudo systemctl enable borgmatic.timer
|
|
||||||
sudo systemctl start borgmatic.timer
|
|
||||||
```
|
|
||||||
|
|
||||||
Feel free to modify the timer file based on how frequently you'd like
|
|
||||||
borgmatic to run.
|
|
||||||
|
|
||||||
|
|
||||||
## Advanced configuration
|
|
||||||
|
|
||||||
### Multiple configuration files
|
|
||||||
|
|
||||||
A more advanced usage is to create multiple separate configuration files and
|
|
||||||
place each one in an /etc/borgmatic.d directory. For instance:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo mkdir /etc/borgmatic.d
|
|
||||||
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
|
|
||||||
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
With this approach, you can have entirely different backup policies for
|
|
||||||
different applications on your system. For instance, you may want one backup
|
|
||||||
configuration for your database data directory, and a different configuration
|
|
||||||
for your user home directories.
|
|
||||||
|
|
||||||
When you set up multiple configuration files like this, borgmatic will run
|
|
||||||
each one in turn from a single borgmatic invocation. This includes, by
|
|
||||||
default, the traditional /etc/borgmatic/config.yaml as well.
|
|
||||||
|
|
||||||
And if you need even more customizability, you can specify alternate
|
|
||||||
configuration paths on the command-line with borgmatic's `--config` option.
|
|
||||||
See `borgmatic --help` for more information.
|
|
||||||
|
|
||||||
|
|
||||||
### Hooks
|
|
||||||
|
|
||||||
If you find yourself performing prepraration tasks before your backup runs, or
|
|
||||||
cleanup work afterwards, borgmatic hooks may be of interest. Hooks are
|
|
||||||
shell commands that borgmatic executes for you at various points, and they're
|
|
||||||
configured in the `hooks` section of your configuration file.
|
|
||||||
|
|
||||||
For instance, you can specify `before_backup` hooks to dump a database to file
|
|
||||||
before backing it up, and specify `after_backup` hooks to delete the temporary
|
|
||||||
file afterwards.
|
|
||||||
|
|
||||||
borgmatic hooks run once per configuration file. `before_backup` hooks run
|
|
||||||
prior to backups of all repositories. `after_backup` hooks run afterwards, but
|
|
||||||
not if an error occurs in a previous hook or in the backups themselves. And
|
|
||||||
borgmatic runs `on_error` hooks if an error occurs.
|
|
||||||
|
|
||||||
An important security note about hooks: borgmatic executes all hook commands
|
|
||||||
with the user permissions of borgmatic itself. So to prevent potential shell
|
|
||||||
injection or privilege escalation, do not forget to set secure permissions
|
|
||||||
(`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
|
|
||||||
|
|
||||||
See the sample generated configuration file mentioned above for specifics
|
|
||||||
about hook configuration syntax.
|
|
||||||
|
|
||||||
|
|
||||||
## Upgrading
|
|
||||||
|
|
||||||
In general, all you should need to do to upgrade borgmatic is run the
|
|
||||||
following:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo pip3 install --upgrade borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
See below about special cases.
|
|
||||||
|
|
||||||
|
|
||||||
### Upgrading from borgmatic 1.0.x
|
|
||||||
|
|
||||||
borgmatic changed its configuration file format in version 1.1.0 from
|
|
||||||
INI-style to YAML. This better supports validation, and has a more natural way
|
|
||||||
to express lists of values. To upgrade your existing configuration, first
|
|
||||||
upgrade to the new version of borgmatic.
|
|
||||||
|
|
||||||
As of version 1.1.0, borgmatic no longer supports Python 2. If you were
|
|
||||||
already running borgmatic with Python 3, then you can upgrade borgmatic
|
|
||||||
in-place:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo pip3 install --upgrade borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
But if you were running borgmatic with Python 2, uninstall and reinstall instead:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo pip uninstall borgmatic
|
|
||||||
sudo pip3 install borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
The pip binary names for different versions of Python can differ, so the above
|
|
||||||
commands may need some tweaking to work on your machine.
|
|
||||||
|
|
||||||
|
|
||||||
Once borgmatic is upgraded, run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo upgrade-borgmatic-config
|
|
||||||
```
|
|
||||||
|
|
||||||
That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
|
|
||||||
(by default) using the values from both your existing configuration and
|
|
||||||
excludes files. The new version of borgmatic will consume the YAML
|
|
||||||
configuration file instead of the old one.
|
|
||||||
|
|
||||||
|
|
||||||
### Upgrading from atticmatic
|
|
||||||
|
|
||||||
You can ignore this section if you're not an atticmatic user (the former name
|
|
||||||
of borgmatic).
|
|
||||||
|
|
||||||
borgmatic only supports Borg now and no longer supports Attic. So if you're
|
|
||||||
an Attic user, consider switching to Borg. See the [Borg upgrade
|
|
||||||
command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
|
|
||||||
for more information. Then, follow the instructions above about setting up
|
|
||||||
your borgmatic configuration files.
|
|
||||||
|
|
||||||
If you were already using Borg with atticmatic, then you can upgrade
|
|
||||||
from atticmatic to borgmatic by running the following commands:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo pip3 uninstall atticmatic
|
|
||||||
sudo pip3 install borgmatic
|
|
||||||
```
|
|
||||||
|
|
||||||
That's it! borgmatic will continue using your /etc/borgmatic configuration
|
|
||||||
files.
|
|
||||||
|
|
||||||
|
|
||||||
## Support and contributing
|
## Support and contributing
|
||||||
|
@ -409,95 +86,10 @@ or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
|
||||||
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
|
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
|
||||||
your thing. In general, contributions are very welcome. We don't bite!
|
your thing. In general, contributions are very welcome. We don't bite!
|
||||||
|
|
||||||
|
Also, please check out the [borgmatic development
|
||||||
|
how-to](docs/how-to/develop-on-borgmatic.md) for info on cloning source code,
|
||||||
|
running tests, etc.
|
||||||
|
|
||||||
### Code style
|
|
||||||
|
|
||||||
Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
|
|
||||||
the following deviations from it:
|
|
||||||
|
|
||||||
* For strings, prefer single quotes over double quotes.
|
|
||||||
* Limit all lines to a maximum of 100 characters.
|
|
||||||
* Use trailing commas within multiline values or argument lists.
|
|
||||||
* For multiline constructs, put opening and closing delimeters on lines
|
|
||||||
separate from their contents.
|
|
||||||
* Within multiline constructs, use standard four-space indentation. Don't align
|
|
||||||
indentation with an opening delimeter.
|
|
||||||
|
|
||||||
borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
|
|
||||||
formatter and [Flake8](http://flake8.pycqa.org/en/latest/) code checker, so
|
|
||||||
certain code style requirements will be enforced when running automated tests.
|
|
||||||
See the Black and Flake8 documentation for more information.
|
|
||||||
|
|
||||||
|
|
||||||
### Development
|
|
||||||
|
|
||||||
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://projects.torsion.org/witten/borgmatic.git
|
|
||||||
```
|
|
||||||
|
|
||||||
Or:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
|
|
||||||
```
|
|
||||||
|
|
||||||
Then, install borgmatic
|
|
||||||
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
|
|
||||||
so that you can run borgmatic commands while you're hacking on them to
|
|
||||||
make sure your changes work.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd borgmatic/
|
|
||||||
pip3 install --editable --user .
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that this will typically install the borgmatic commands into
|
|
||||||
`~/.local/bin`, which may or may not be on your PATH. There are other ways to
|
|
||||||
install borgmatic editable as well, for instance into the system Python
|
|
||||||
install (so without `--user`, as root), or even into a
|
|
||||||
[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
|
|
||||||
borgmatic is up to you, but generally an editable install makes development
|
|
||||||
and testing easier.
|
|
||||||
|
|
||||||
|
|
||||||
### Running tests
|
|
||||||
|
|
||||||
Assuming you've cloned the borgmatic source code as described above, and
|
|
||||||
you're in the `borgmatic/` working copy, install tox, which is used for
|
|
||||||
setting up testing environments:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo pip3 install tox
|
|
||||||
```
|
|
||||||
|
|
||||||
Finally, to actually run tests, run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd borgmatic
|
|
||||||
tox
|
|
||||||
```
|
|
||||||
|
|
||||||
If when running tests, you get an error from the
|
|
||||||
[Black](https://black.readthedocs.io/en/stable/) code formatter about files
|
|
||||||
that would be reformatted, you can ask Black to format them for you via the
|
|
||||||
following:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tox -e black
|
|
||||||
```
|
|
||||||
|
|
||||||
### End-to-end tests
|
|
||||||
|
|
||||||
borgmatic additionally includes some end-to-end tests that integration test
|
|
||||||
with Borg for a few representative scenarios. These tests don't run by default
|
|
||||||
because they're relatively slow and depend on Borg. If you would like to run
|
|
||||||
them:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tox -e end-to-end
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
|
72
docs/how-to/deal-with-very-large-backups.md
Normal file
72
docs/how-to/deal-with-very-large-backups.md
Normal file
|
@ -0,0 +1,72 @@
|
||||||
|
---
|
||||||
|
title: How to deal with very large backups
|
||||||
|
---
|
||||||
|
## Biggish data
|
||||||
|
|
||||||
|
Borg itself is great for efficiently de-duplicating data across successive
|
||||||
|
backup archives, even when dealing with very large repositories. However, you
|
||||||
|
may find that while borgmatic's default mode of "prune, create, and check"
|
||||||
|
works well on small repositories, it's not so great on larger ones. That's
|
||||||
|
because running the default consistency checks just takes a long time on large
|
||||||
|
repositories.
|
||||||
|
|
||||||
|
### A la carte actions
|
||||||
|
|
||||||
|
If you find yourself in this situation, you have some options. First, you can
|
||||||
|
run borgmatic's pruning, creating, or checking actions separately. For
|
||||||
|
instance, the the following optional flags are available:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --prune
|
||||||
|
borgmatic --create
|
||||||
|
borgmatic --check
|
||||||
|
```
|
||||||
|
|
||||||
|
You can run with only one of these flags provided, or you can mix and match
|
||||||
|
any number of them in a single borgmatic run. This supports approaches like
|
||||||
|
making backups with `--create` on a frequent schedule, while only running
|
||||||
|
expensive consistency checks with `--check` on a much less frequent basis from
|
||||||
|
a separate cron job.
|
||||||
|
|
||||||
|
### Consistency check configuration
|
||||||
|
|
||||||
|
Another option is to customize your consistency checks. The default
|
||||||
|
consistency checks run both full-repository checks and per-archive checks
|
||||||
|
within each repository.
|
||||||
|
|
||||||
|
But if you find that archive checks are just too slow, for example, you can
|
||||||
|
configure borgmatic to run repository checks only. Configure this in the
|
||||||
|
`consistency` section of borgmatic configuration:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
consistency:
|
||||||
|
checks:
|
||||||
|
- repository
|
||||||
|
```
|
||||||
|
|
||||||
|
If that's still too slow, you can disable consistency checks entirely,
|
||||||
|
either for a single repository or for all repositories.
|
||||||
|
|
||||||
|
Disabling all consistency checks looks like this:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
consistency:
|
||||||
|
checks:
|
||||||
|
- disabled
|
||||||
|
```
|
||||||
|
|
||||||
|
Or, if you have multiple repositories in your borgmatic configuration file,
|
||||||
|
you can keep running consistency checks, but only against a subset of the
|
||||||
|
repositories:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
consistency:
|
||||||
|
check_repositories:
|
||||||
|
- path/of/repository_to_check.borg
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [How to set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
|
||||||
|
* [borgmatic README](../../)
|
||||||
|
|
99
docs/how-to/develop-on-borgmatic.md
Normal file
99
docs/how-to/develop-on-borgmatic.md
Normal file
|
@ -0,0 +1,99 @@
|
||||||
|
---
|
||||||
|
title: How to develop on borgmatic
|
||||||
|
---
|
||||||
|
## Source code
|
||||||
|
|
||||||
|
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://projects.torsion.org/witten/borgmatic.git
|
||||||
|
```
|
||||||
|
|
||||||
|
Or:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, install borgmatic
|
||||||
|
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
|
||||||
|
so that you can run borgmatic commands while you're hacking on them to
|
||||||
|
make sure your changes work.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd borgmatic/
|
||||||
|
pip3 install --editable --user .
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that this will typically install the borgmatic commands into
|
||||||
|
`~/.local/bin`, which may or may not be on your PATH. There are other ways to
|
||||||
|
install borgmatic editable as well, for instance into the system Python
|
||||||
|
install (so without `--user`, as root), or even into a
|
||||||
|
[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
|
||||||
|
borgmatic is up to you, but generally an editable install makes development
|
||||||
|
and testing easier.
|
||||||
|
|
||||||
|
|
||||||
|
## Automated tests
|
||||||
|
|
||||||
|
Assuming you've cloned the borgmatic source code as described above, and
|
||||||
|
you're in the `borgmatic/` working copy, install tox, which is used for
|
||||||
|
setting up testing environments:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip3 install tox
|
||||||
|
```
|
||||||
|
|
||||||
|
Finally, to actually run tests, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd borgmatic
|
||||||
|
tox
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code formatting
|
||||||
|
|
||||||
|
If when running tests, you get an error from the
|
||||||
|
[Black](https://black.readthedocs.io/en/stable/) code formatter about files
|
||||||
|
that would be reformatted, you can ask Black to format them for you via the
|
||||||
|
following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tox -e black
|
||||||
|
```
|
||||||
|
|
||||||
|
### End-to-end tests
|
||||||
|
|
||||||
|
borgmatic additionally includes some end-to-end tests that integration test
|
||||||
|
with Borg for a few representative scenarios. These tests don't run by default
|
||||||
|
because they're relatively slow and depend on Borg. If you would like to run
|
||||||
|
them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tox -e end-to-end
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code style
|
||||||
|
|
||||||
|
Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
|
||||||
|
the following deviations from it:
|
||||||
|
|
||||||
|
* For strings, prefer single quotes over double quotes.
|
||||||
|
* Limit all lines to a maximum of 100 characters.
|
||||||
|
* Use trailing commas within multiline values or argument lists.
|
||||||
|
* For multiline constructs, put opening and closing delimeters on lines
|
||||||
|
separate from their contents.
|
||||||
|
* Within multiline constructs, use standard four-space indentation. Don't align
|
||||||
|
indentation with an opening delimeter.
|
||||||
|
|
||||||
|
borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
|
||||||
|
formatter and [Flake8](http://flake8.pycqa.org/en/latest/) code checker, so
|
||||||
|
certain code style requirements will be enforced when running automated tests.
|
||||||
|
See the Black and Flake8 documentation for more information.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [How to inspect your backups](../../docs/how-to/inspect-your-backups.md)
|
||||||
|
* [Support and contributing](../../#support-and-contributing)
|
||||||
|
* [borgmatic README](../../)
|
46
docs/how-to/inspect-your-backups.md
Normal file
46
docs/how-to/inspect-your-backups.md
Normal file
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
title: How to inspect your backups
|
||||||
|
---
|
||||||
|
## Backup progress
|
||||||
|
|
||||||
|
By default, borgmatic runs proceed silently except in the case of errors. But
|
||||||
|
if you'd like to to get additional information about the progress of the
|
||||||
|
backup as it proceeds, use the verbosity option:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --verbosity 1
|
||||||
|
```
|
||||||
|
|
||||||
|
This lists the files that borgmatic is archiving, which are those that are new
|
||||||
|
or changed since the last backup.
|
||||||
|
|
||||||
|
Or, for even more progress and debug spew:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --verbosity 2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Existing backups
|
||||||
|
|
||||||
|
Borgmatic provides convenient flags for Borg's
|
||||||
|
[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
|
||||||
|
[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
|
||||||
|
functionality:
|
||||||
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --list
|
||||||
|
borgmatic --info
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scripting borgmatic
|
||||||
|
|
||||||
|
To consume the output of borgmatic in other software, you can include an
|
||||||
|
optional `--json` flag with `--create`, `--list`, or `--info` to get the
|
||||||
|
output formatted as JSON.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [How to set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
|
||||||
|
* [borgmatic README](../../)
|
33
docs/how-to/make-per-application-backups.md
Normal file
33
docs/how-to/make-per-application-backups.md
Normal file
|
@ -0,0 +1,33 @@
|
||||||
|
---
|
||||||
|
title: How to make per-application backups
|
||||||
|
---
|
||||||
|
## Multiple backup configurations
|
||||||
|
|
||||||
|
You may find yourself wanting to create different backup policies for
|
||||||
|
different applications on your system. For instance, you may want one backup
|
||||||
|
configuration for your database data directory, and a different configuration
|
||||||
|
for your user home directories.
|
||||||
|
|
||||||
|
The way to accomplish that is pretty simple: Create multiple separate
|
||||||
|
configuration files and place each one in a `/etc/borgmatic.d/` directory. For
|
||||||
|
instance:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo mkdir /etc/borgmatic.d
|
||||||
|
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
|
||||||
|
sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
When you set up multiple configuration files like this, borgmatic will run
|
||||||
|
each one in turn from a single borgmatic invocation. This includes, by
|
||||||
|
default, the traditional `/etc/borgmatic/config.yaml` as well.
|
||||||
|
|
||||||
|
And if you need even more customizability, you can specify alternate
|
||||||
|
configuration paths on the command-line with borgmatic's `--config` option.
|
||||||
|
See `borgmatic --help` for more information.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [How to set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
|
||||||
|
* [borgmatic README](../../)
|
52
docs/how-to/run-preparation-steps-before-backups.md
Normal file
52
docs/how-to/run-preparation-steps-before-backups.md
Normal file
|
@ -0,0 +1,52 @@
|
||||||
|
---
|
||||||
|
title: How to run preparation steps before backups
|
||||||
|
---
|
||||||
|
## Preparation and cleanup hooks
|
||||||
|
|
||||||
|
If you find yourself performing prepraration tasks before your backup runs, or
|
||||||
|
cleanup work afterwards, borgmatic hooks may be of interest. Hooks are
|
||||||
|
shell commands that borgmatic executes for you at various points, and they're
|
||||||
|
configured in the `hooks` section of your configuration file.
|
||||||
|
|
||||||
|
For instance, you can specify `before_backup` hooks to dump a database to file
|
||||||
|
before backing it up, and specify `after_backup` hooks to delete the temporary
|
||||||
|
file afterwards. Here's an example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
hooks:
|
||||||
|
before_backup:
|
||||||
|
- dump-a-database /to/file.sql
|
||||||
|
after_backup:
|
||||||
|
- rm /to/file.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
borgmatic hooks run once per configuration file. `before_backup` hooks run
|
||||||
|
prior to backups of all repositories. `after_backup` hooks run afterwards, but
|
||||||
|
not if an error occurs in a previous hook or in the backups themselves.
|
||||||
|
|
||||||
|
|
||||||
|
## Error hooks
|
||||||
|
|
||||||
|
borgmatic also runs `on_error` hooks if an error occurs. Here's an example
|
||||||
|
configuration:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
hooks:
|
||||||
|
on_error:
|
||||||
|
- echo "Error while creating a backup."
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
An important security note about hooks: borgmatic executes all hook commands
|
||||||
|
with the user permissions of borgmatic itself. So to prevent potential shell
|
||||||
|
injection or privilege escalation, do not forget to set secure permissions
|
||||||
|
(`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
|
||||||
|
* [How to make per-application backups](../../docs/how-to/make-per-application-backups.md)
|
||||||
|
* [borgmatic README](../../)
|
167
docs/how-to/set-up-backups.md
Normal file
167
docs/how-to/set-up-backups.md
Normal file
|
@ -0,0 +1,167 @@
|
||||||
|
---
|
||||||
|
title: How to set up backups with borgmatic
|
||||||
|
---
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
To get up and running, first [install
|
||||||
|
Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
|
||||||
|
least version 1.1.
|
||||||
|
|
||||||
|
Then, run the following command to download and install borgmatic:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip3 install --upgrade borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that your pip binary may have a different name than "pip3". Make sure
|
||||||
|
you're using Python 3, as borgmatic does not support Python 2.
|
||||||
|
|
||||||
|
|
||||||
|
### Other ways to install
|
||||||
|
|
||||||
|
Along with the above process, you have several other options for installing
|
||||||
|
borgmatic:
|
||||||
|
|
||||||
|
* [Docker image](https://hub.docker.com/r/monachus/borgmatic/)
|
||||||
|
* [Another Docker image](https://hub.docker.com/r/b3vis/borgmatic/)
|
||||||
|
* [Debian](https://tracker.debian.org/pkg/borgmatic)
|
||||||
|
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
|
||||||
|
* [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
|
||||||
|
* [Arch Linux](https://aur.archlinux.org/packages/borgmatic/)
|
||||||
|
* [OpenBSD](http://ports.su/sysutils/borgmatic)
|
||||||
|
* [openSUSE](https://software.opensuse.org/package/borgmatic)
|
||||||
|
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
After you install borgmatic, generate a sample configuration file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo generate-borgmatic-config
|
||||||
|
```
|
||||||
|
|
||||||
|
If that command is not found, then it may be installed in a location that's
|
||||||
|
not in your system `PATH`. Try looking in `/usr/local/bin/`.
|
||||||
|
|
||||||
|
This generates a sample configuration file at /etc/borgmatic/config.yaml (by
|
||||||
|
default). You should edit the file to suit your needs, as the values are
|
||||||
|
representative. All fields are optional except where indicated, so feel free
|
||||||
|
to ignore anything you don't need.
|
||||||
|
|
||||||
|
You can also have a look at the [full configuration
|
||||||
|
schema](https://projects.torsion.org/witten/borgmatic/src/master/borgmatic/config/schema.yaml)
|
||||||
|
for the authoritative set of all configuration options. This is handy if
|
||||||
|
borgmatic has added new options since you originally created your
|
||||||
|
configuration file.
|
||||||
|
|
||||||
|
|
||||||
|
### Encryption
|
||||||
|
|
||||||
|
Note that if you plan to run borgmatic on a schedule with cron, and you
|
||||||
|
encrypt your Borg repository with a passphrase instead of a key file, you'll
|
||||||
|
either need to set the borgmatic `encryption_passphrase` configuration
|
||||||
|
variable or set the `BORG_PASSPHRASE` environment variable. See the
|
||||||
|
[repository encryption
|
||||||
|
section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
|
||||||
|
of the Borg Quick Start for more info.
|
||||||
|
|
||||||
|
Alternatively, you can specify the passphrase programatically by setting
|
||||||
|
either the borgmatic `encryption_passcommand` configuration variable or the
|
||||||
|
`BORG_PASSCOMMAND` environment variable. See the [Borg Security
|
||||||
|
FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
|
||||||
|
for more info.
|
||||||
|
|
||||||
|
|
||||||
|
## Initialization
|
||||||
|
|
||||||
|
Before you can create backups with borgmatic, you first need to initialize a
|
||||||
|
Borg repository so you have a destination for your backup archives. (But skip
|
||||||
|
this step if you already have a Borg repository.) To create a repository, run
|
||||||
|
a command like the following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --init --encryption repokey
|
||||||
|
```
|
||||||
|
|
||||||
|
This uses the borgmatic configuration file you created above to determine
|
||||||
|
which local or remote repository to create, and encrypts it with the
|
||||||
|
encryption passphrase specified there if one is provided. Read about [Borg
|
||||||
|
encryption
|
||||||
|
modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
|
||||||
|
for the menu of available encryption modes.
|
||||||
|
|
||||||
|
Also, optionally check out the [Borg Quick
|
||||||
|
Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
|
||||||
|
background about repository initialization.
|
||||||
|
|
||||||
|
Note that borgmatic skips repository initialization if the repository already
|
||||||
|
exists. This supports use cases like ensuring a repository exists prior to
|
||||||
|
performing a backup.
|
||||||
|
|
||||||
|
If the repository is on a remote host, make sure that your local user has
|
||||||
|
key-based SSH access to the desired user account on the remote host.
|
||||||
|
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
Now that you've configured borgmatic and initialized a repository, it's a
|
||||||
|
good idea to test that borgmatic is working. So to run borgmatic and start a
|
||||||
|
backup, you can invoke it like this:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borgmatic --verbosity 1
|
||||||
|
```
|
||||||
|
|
||||||
|
By default, this will also prune any old backups as per the configured
|
||||||
|
retention policy, and check backups for consistency problems due to things
|
||||||
|
like file damage.
|
||||||
|
|
||||||
|
The verbosity flag makes borgmatic list the files that it's archiving, which
|
||||||
|
are those that are new or changed since the last backup. Eyeball the list and
|
||||||
|
see if it matches your expectations based on the configuration.
|
||||||
|
|
||||||
|
|
||||||
|
## Autopilot
|
||||||
|
|
||||||
|
Running backups manually is good for validating your configuration, but I'm
|
||||||
|
guessing that you want to run borgmatic automatically, say once a day. To do
|
||||||
|
that, you can configure a separate job runner to invoke it periodically.
|
||||||
|
|
||||||
|
### cron
|
||||||
|
|
||||||
|
If you're using cron, download the [sample cron
|
||||||
|
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
|
||||||
|
Then, from the directory where you downloaded it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo mv borgmatic /etc/cron.d/borgmatic
|
||||||
|
sudo chmod +x /etc/cron.d/borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
You can modify the cron file if you'd like to run borgmatic more or less frequently.
|
||||||
|
|
||||||
|
### systemd
|
||||||
|
|
||||||
|
If you're using systemd instead of cron to run jobs, download the [sample
|
||||||
|
systemd service
|
||||||
|
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
|
||||||
|
and the [sample systemd timer
|
||||||
|
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
|
||||||
|
Then, from the directory where you downloaded them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
|
||||||
|
sudo systemctl enable borgmatic.timer
|
||||||
|
sudo systemctl start borgmatic.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Feel free to modify the timer file based on how frequently you'd like
|
||||||
|
borgmatic to run.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [How to make per-application backups](../../docs/how-to/make-per-application-backups.md)
|
||||||
|
* [How to deal with very large backups](../../docs/how-to/deal-with-very-large-backups.md)
|
||||||
|
* [How to inspect your backups](../../docs/how-to/inspect-your-backups.md)
|
||||||
|
* [borgmatic README](../../)
|
80
docs/how-to/upgrade.md
Normal file
80
docs/how-to/upgrade.md
Normal file
|
@ -0,0 +1,80 @@
|
||||||
|
---
|
||||||
|
title: How to upgrade borgmatic
|
||||||
|
---
|
||||||
|
## Upgrading
|
||||||
|
|
||||||
|
In general, all you should need to do to upgrade borgmatic is run the
|
||||||
|
following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip3 install --upgrade borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
See below about special cases.
|
||||||
|
|
||||||
|
|
||||||
|
### Upgrading from borgmatic 1.0.x
|
||||||
|
|
||||||
|
borgmatic changed its configuration file format in version 1.1.0 from
|
||||||
|
INI-style to YAML. This better supports validation, and has a more natural way
|
||||||
|
to express lists of values. To upgrade your existing configuration, first
|
||||||
|
upgrade to the new version of borgmatic.
|
||||||
|
|
||||||
|
As of version 1.1.0, borgmatic no longer supports Python 2. If you were
|
||||||
|
already running borgmatic with Python 3, then you can upgrade borgmatic
|
||||||
|
in-place:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip3 install --upgrade borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
But if you were running borgmatic with Python 2, uninstall and reinstall instead:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip uninstall borgmatic
|
||||||
|
sudo pip3 install borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
The pip binary names for different versions of Python can differ, so the above
|
||||||
|
commands may need some tweaking to work on your machine.
|
||||||
|
|
||||||
|
|
||||||
|
Once borgmatic is upgraded, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo upgrade-borgmatic-config
|
||||||
|
```
|
||||||
|
|
||||||
|
That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
|
||||||
|
(by default) using the values from both your existing configuration and
|
||||||
|
excludes files. The new version of borgmatic will consume the YAML
|
||||||
|
configuration file instead of the old one.
|
||||||
|
|
||||||
|
|
||||||
|
### Upgrading from atticmatic
|
||||||
|
|
||||||
|
You can ignore this section if you're not an atticmatic user (the former name
|
||||||
|
of borgmatic).
|
||||||
|
|
||||||
|
borgmatic only supports Borg now and no longer supports Attic. So if you're
|
||||||
|
an Attic user, consider switching to Borg. See the [Borg upgrade
|
||||||
|
command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
|
||||||
|
for more information. Then, follow the instructions above about setting up
|
||||||
|
your borgmatic configuration files.
|
||||||
|
|
||||||
|
If you were already using Borg with atticmatic, then you can upgrade
|
||||||
|
from atticmatic to borgmatic by running the following commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo pip3 uninstall atticmatic
|
||||||
|
sudo pip3 install borgmatic
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it! borgmatic will continue using your /etc/borgmatic configuration
|
||||||
|
files.
|
||||||
|
|
||||||
|
|
||||||
|
## Related documentation
|
||||||
|
|
||||||
|
* [Develop on borgmatic](../../docs/how-to/develop-on-borgmatic.md)
|
||||||
|
* [borgmatic README](../../)
|
2
setup.py
2
setup.py
|
@ -1,7 +1,7 @@
|
||||||
from setuptools import setup, find_packages
|
from setuptools import setup, find_packages
|
||||||
|
|
||||||
|
|
||||||
VERSION = '1.2.15'
|
VERSION = '1.2.16.dev0'
|
||||||
|
|
||||||
|
|
||||||
setup(
|
setup(
|
||||||
|
|
Loading…
Reference in a new issue