Repairing catastrophic repository damage

[sshaikh]

Let's hypothetically I ran an errant dd command on the disk holding my repo, in such a way that after salvaging what was possible, a borg check --repair can't even complete:

Data integrity error: Invalid segment entry size 0 - too small [segment 56971, offset 110112635]

Deleting the index file and running --repair --repository-only gives the following:

Data integrity error: Invalid segment magic [segment 51735, offset 0]

What options do I have?

If the repo has to be written off, is it possible to repair it using a second repo that has largely been back up to with the same settings (I use borgmatic pointing to two repos). Is the most straightforward way just to copy the good repo over?

[ThomasWaldmann]

dd is rather bad, because it could have damaged the filesystem.

What you can do is:

• transfer all repository files (repo_dir/) that are still readable somehow to a new disk (with a non-corrupt filesystem). repo_dir/data/... is the most important part (these are the segment files that have your data). this is also that the situation does not get worse than it already is. Only work on the copy, do not modify the original.
• also transfer the other files from repo_dir/* if that still works. if not, you maybe can also take them from a freshly initialized repository.
• if you used encryption, you will also need key and passphrase
• borg check --repair should rescue everything that is still valid. If a segment has an invalid magic, it should skip that segment and continue with the remaining ones. If it doesn't, look into the file with that number and check what's inside. If it does not start with "BORG...", it might be a completely different content from a damaged fs - in that case you can just delete that file.

There is no tooling to repair one borg 1.x repo using the data from another borg 1.x repo. Even if you used the same settings, the crypto key, the id key and the chunker secret will be different, so the chunks are not "compatible" (this has crypto security reasons). borg2 (and also borg 1.4.5+) offers "related repositories" that use the same id key and same chunker secret to enable that in future).

But if you have another backup repo, then you didn't lose data just because you lost the first one, right?

If you just want to have a working backup repo on the first disk:

• format the disk (erases all data, creates a valid filesystem)
• initialize a new borg repo on the first disk
• start from scratch on the first disk
• DO NOT "copy over" the disk2 repo to disk1 and update them both independently, you would break crypto security and confuse borg's caching if you do so.

[sshaikh]

Useful info! This repo uses borg 1.4.0, so the shared repo magic is out of scope.

I was wondering if there was a way to salvage some of the archives, so I guess I'll have to keep deleting segments until --repair succeeds.

Ultimately yes, the actual data that is being backed up is fine (and is still being backed up to the good repo), so starting again is always available as a fallback option.

[sshaikh]

borg check --repair should rescue everything that is still valid. If a segment has an invalid magic, it should skip that segment and continue with the remaining ones. If it doesn't, look into the file with that number and check what's inside. If it does not start with "BORG...", it might be a completely different content from a damaged fs - in that case you can just delete that file.

When I delete such files I see the following:

FileNotFoundError: [Errno 2] No such file or directory: '/xxx/data/53/53463'

creating an (empty) file takes me back to the previous error.

[ThomasWaldmann]

can you try the repair with borg 1.4.5?

1.4.0 is rather old meanwhile and maybe we have fixed a related bug.

[ThomasWaldmann]

BTW, did you look at / into these corrupt files? size? what's inside?

[sshaikh]

can you try the repair with borg 1.4.5?

Done - no change in behavior

BTW, did you look at / into these corrupt files? size? what's inside?

I did, but I didn't take note other than a repeating bunch of chars. I suspect it was zero'd out? If I can get past this particular file then I'll find the next problematic segment.

[sshaikh]

I managed to make some progress by removing the index/hints/integrity files.

Found a bad file. I see these characters repeated:

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^

[sshaikh]

So each repair command takes a while (while replaying?) before erroring out on a bad file. I then delete that file and restart. I suspect this will take a while. Any tips on making the process more efficient? I suspect I need to scan and delete the respective files outside of borg?

EDIT: The following (AI) was used to search for files starting with NUL:

find data -type f -exec sh -c '
  [ "$(head -c 2 "$1" | od -An -tx1 | tr -d " ")" = "0000" ] && echo "$1"
' _ {} \;

The files listed were deleted.

[sshaikh]

New failure!

borg.helpers.errors.IntegrityError: Data integrity error: Segment entry checksum mismatch [segment 55263, offset 132018493]

Unlike before, this file is not full of NULs. It starts with BORG_SEGX. Of course the file could be corrupt, but I'd have hoped the repair would handle that situation? I'll hold off deleting the film in case you'd like a closer look.

[ThomasWaldmann]

A segment entry is just one part of the segment file (one file stores multiple objects inside it).

borg check --repair should try to recover such a segment, copying all valid entries to a new segment file, skipping the defect entry.

[sshaikh]

And if it doesn't as in this case? Would that be a bug?

[ThomasWaldmann]

Maybe.

A reproducer script would be good.