borgmatic/docs/how-to/deal-with-very-large-backups.md

title	eleventyNavigation
How to deal with very large backups	key parent order 📏 Deal with very large backups How-to guides 4

Biggish data

Borg itself is great for efficiently de-duplicating data across successive backup archives, even when dealing with very large repositories. But you may find that while borgmatic's default actions of create, prune, compact, and check works well on small repositories, it's not so great on larger ones. That's because running the default pruning, compact, and consistency checks take a long time on large repositories.

Prior to version 1.7.9 The default action ordering was prune, compact, create, and check.

A la carte actions

If you find yourself wanting to customize the actions, you have some options. First, you can run borgmatic's prune, compact, create, or check actions separately. For instance, the following optional actions are available (among others):

borgmatic create
borgmatic prune
borgmatic compact
borgmatic check

You can run borgmatic with only one of these actions provided, or you can mix and match any number of them in a single borgmatic run. This supports approaches like skipping certain actions while running others. For instance, this skips prune and compact and only runs create and check:

borgmatic create check

New in version 1.7.9 borgmatic now respects your specified command-line action order, running actions in the order you specify. In previous versions, borgmatic ran your specified actions in a fixed ordering regardless of the order they appeared on the command-line.

But instead of running actions together, another option is to run backups with create on a frequent schedule (e.g. with borgmatic create called from one cron job), while only running expensive consistency checks with check on a much less frequent basis (e.g. with borgmatic check called from a separate cron job).

New in version 1.8.5 Instead of (or in addition to) specifying actions on the command-line, you can configure borgmatic to skip particular actions.

Consistency check configuration

Another option is to customize your consistency checks. By default, if you omit consistency checks from configuration, borgmatic runs full-repository checks (repository) and per-archive checks (archives) within each repository. (Although see below about check frequency.) This is equivalent to what borg check does if run without options.

But if you find that archive checks are too slow, for example, you can configure borgmatic to run repository checks only. Configure this in the consistency section of borgmatic configuration:

checks:
    - name: repository

Prior to version 1.8.0 Put this option in the consistency: section of your configuration.

Prior to version 1.6.2 The checks option was a plain list of strings without the name: part, and borgmatic ran each configured check every time checks were run. For example:

checks:
    - repository

Here are the available checks from fastest to slowest:

• repository: Checks the consistency of the repository itself.
• archives: Checks all of the archives in the repository.
• extract: Performs an extraction dry-run of the latest archive.
• data: Verifies the data integrity of all archives contents, decrypting and decompressing all data.
• spot: Compares file counts and contents between your source files and the latest archive.

Note that the data check is a more thorough version of the archives check, so enabling the data check implicitly enables the archives check as well.

See Borg's check documentation for more information.

Spot check

The various consistency checks all have trade-offs around speed and thoroughness, but most of them don't even look at your original source files—arguably one important way to ensure your backups contain the files you'll want to restore in the case of catastrophe (or just an accidentally deleted file). Because if something goes wrong with your source files, most consistency checks will still pass with flying colors and you won't discover there's a problem until you go to restore.

New in version 1.8.10 Beta feature That's where the spot check comes in. This check actually compares your source file counts and data against those in the latest archive, potentially catching problems like incorrect excludes, inadvertent deletes, files changed by malware, etc.

However, because an exhaustive comparison of all source files against the latest archive might be too slow, the spot check supports sampling a percentage of your source files for the comparison, ensuring they fall within configured tolerances.

Here's how it works. Start by installing the xxhash OS package if you don't already have it, so the spot check can run the xxh64sum command and efficiently hash files for comparison. Then add something like the following to your borgmatic configuration:

checks:
    - name: spot
      count_tolerance_percentage: 10
      data_sample_percentage: 1
      data_tolerance_percentage: 0.5

The count_tolerance_percentage is the percentage delta between the source directories file count and the latest backup archive file count that is allowed before the entire consistency check fails. For instance, if the spot check runs and finds 100 source files on disk and 105 files in the latest archive, that would be within the configured 10% count tolerance and the check would succeed. But if there were 100 source files and 200 archive files, the check would fail. (100 source files and only 50 archive files would also fail.)

The data_sample_percentage is the percentage of total files in the source directories to randomly sample and compare to their corresponding files in the latest backup archive. A higher value allows a more accurate check—and a slower one. The comparison is performed by hashing the selected source files and counting hashes that don't match the latest archive. For instance, if you have 1,000 source files and your sample percentage is 1%, then only 10 source files will be compared against the latest archive. These sampled files are selected randomly each time, so in effect the spot check is probabilistic.

The data_tolerance_percentage is the percentage of total files in the source directories that can fail a spot check data comparison without failing the entire consistency check. The value must be lower than or equal to the contents_sample_percentage.

All three options are required when using the spot check. And because the check relies on these configured tolerances, it may not be a set-it-and-forget-it type of consistency check, at least until you get the tolerances dialed in so there are minimal false positives or negatives. It is recommended you run borgmatic check several times after configuring the spot check, tweaking your tolerances as needed. For certain workloads where your source files experience wild swings of file contents or counts, the spot check may not suitable at all.

What if you add, delete, or change a bunch of your source files and you don't want the spot check to fail the next time it's run? Run borgmatic create to create a new backup, thereby allowing the next spot check to run against an archive that contains your recent changes.

Because the spot check only looks at the most recent archive, you may not want to run it immediately after a create action (borgmatic's default behavior). Instead, it may make more sense to run the spot check on a separate schedule from create.

As long as the spot check feature is in beta, it may be subject to breaking changes. But feel free to use it in production if you're okay with that caveat, and please provide any feedback you have on this feature.

Check frequency

New in version 1.6.2 You can optionally configure checks to run on a periodic basis rather than every time borgmatic runs checks. For instance:

checks:
    - name: repository
      frequency: 2 weeks
    - name: archives
      frequency: 1 month

Prior to version 1.8.0 Put this option in the consistency: section of your configuration.

This tells borgmatic to run the repository consistency check at most once every two weeks for a given repository and the archives check at most once a month. The frequency value is a number followed by a unit of time, e.g. 3 days, 1 week, 2 months, etc. The set of possible time units is as follows (singular or plural):

• second
• minute
• hour
• day
• week (7 days)
• month (30 days)
• year (365 days)

The frequency defaults to always for a check configured without a frequency, which means run this check every time checks run. But if you omit consistency checks from configuration entirely, borgmatic runs full-repository checks (repository) and per-archive checks (archives) within each repository, at most once a month.

Unlike a real scheduler like cron, borgmatic only makes a best effort to run checks on the configured frequency. It compares that frequency with how long it's been since the last check for a given repository (as recorded in a file within ~/.borgmatic/checks). If it hasn't been long enough, the check is skipped. And you still have to run borgmatic check (or borgmatic without actions) in order for checks to run, even when a frequency is configured!

This also applies across configuration files that have the same repository configured. Make sure you have the same check frequency configured in each though—or the most frequently configured check will apply.

If you want to temporarily ignore your configured frequencies, you can invoke borgmatic check --force to run checks unconditionally.

New in version 1.8.6 borgmatic check --force runs check even if it's specified in the skip_actions option.

Running only checks

New in version 1.7.1 If you would like to only run consistency checks without creating backups (for instance with the check action on the command-line), you can omit the source_directories option entirely.

Prior to version 1.7.1 In older versions of borgmatic, instead specify an empty source_directories value, as it is a mandatory option there:

location:
    source_directories: []

Disabling checks

If that's still too slow, you can disable consistency checks entirely, either for a single repository or for all repositories.

New in version 1.8.5 Disabling all consistency checks looks like this:

skip_actions:
    - check

Prior to version 1.8.5 Use this configuration instead:

checks:
    - name: disabled

Prior to version 1.8.0 Put checks: in the consistency: section of your configuration.

Prior to version 1.6.2 checks: was a plain list of strings without the name: part. For instance:

checks:
    - disabled

If you have multiple repositories in your borgmatic configuration file, you can keep running consistency checks, but only against a subset of the repositories:

check_repositories:
    - path/of/repository_to_check.borg

Finally, you can override your configuration file's consistency checks and run particular checks via the command-line. For instance:

borgmatic check --only data --only extract

This is useful for running slow consistency checks on an infrequent basis, separate from your regular checks. It is still subject to any configured check frequencies unless the --force flag is used.

Troubleshooting

Broken pipe with remote repository

When running borgmatic on a large remote repository, you may receive errors like the following, particularly while "borg check" is validating backups for consistency:

    Write failed: Broken pipe
    borg: Error: Connection closed by remote host

This error can be caused by an ssh timeout, which you can rectify by adding the following to the ~/.ssh/config file on the client:

    Host *
        ServerAliveInterval 120

This should make the client keep the connection alive while validating backups.