Unable to mount an XFS filesystem from Linux RAID6 array (“Log inconsistent”)

by

I want to extend suggestions above.

It is extremly worth setting up overlay block device, so any changes to the file system that you’ll do in attempt to recover it will not change anything on the RAID and this will allow you to reset everything and start from the beginning. Therefore, you’ll be given infinite number of attempts, thus releasing the psychological pressure.

I did that with Qemu’s qemu-nbd, Linux nbd.ko (Network Block Device driver) and the qcow2 overlay file.

  1. Connect additional disk where the overlay will be stored. Load NBD driver. Mount your scratch disk somewhere:
modprobe nbd
mount /dev/sdXXN /tmp/overlay
  1. Create a qcow2 overlay file:
qemu-img create -f qcow2 -b /dev/md125 -F raw /tmp/overlay/attempt1.qcow2
  1. Create a block device out of overlay file using qemu-nbd:
qemu-nbd -c /dev/nbd0 /tmp/overlay/attempt1.qcow2

Now you have a /dev/nbd0 which is a “writeable clone” of your array. You can safely write to this device, any changes will be written to /tmp/overlay/attempt1.qcow2. So, for example, when you attempt @shodanshok’s advice, apply it to /dev/nbd0.

  1. If you stuck, disconnect the overlay and remove the overlay file
qemu-nbd -d /dev/nbd0
rm /tmp/overlay/attempt1.qcow2

Then repeat everything from step (2). Alternatively, you can create as many overlays as space and /dev/nbdX devices permit (I have 16 of them, for instance) and work in parallel. All of them should use different overlay images, of course. This is useful if you happen to recover only partial data in some attempt and the other part of data in some other attempt.

When working with clones of XFS filesystem remember that each of them should have distinct UUID.

When (if) the correct recovery path is found, it can be reapplied to the raw device, “irreversibly recovering the filesystem”, or you can rent some space, dump recovered data there from overlay NBD’s, recreate RAID and file system and download it back.

I know, this is hard and cumbersome. This is why data recovery organizations charge a lot when they work with RAIDs. When you try it yourself, you’ll agree that these bills are’t as inflated as it could appear at first sight.

And I repeat that again, RAID6 of 30 devices is a pain. Better have e.g. 3 RAID6 arrays of 10 drives each, then stripe them together using layered MD RAID 0 or LVM. This will make things more manageable, and your reshape/check operations will not take weeks to complete. Yes, you do RAID consistency checks (scrubbing) regularly, at least every other month, don’t you?

Update: There is valuable information in comments, which is worth adding here.

  • I doubt qemu stuff will be available in the Synology DSM. But you can connect disks to ordinary PC with Linux and proceed. Or try booting Synology from network or LiveUSB — the NAS which can connect 30 disks is basically an ordinary amd64 rack-mountable computer. –

  • @Mark suggests another way to create an overlay:

@Bob, there are other options for creating an overlay — I’ve used a USB thumb drive and the steps at https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID

Nice way, which uses Device Mapper framework, likely to be present in the DSM! Also it is probably faster than my approach. It is dmsetup command who creates the virtual device with sparse overlay file. However, since the RAID array itself appears clean in your case and all we talk about is fixing a file system, I suggest to create overlay of an assembled array (/dev/md125) rather than of individual array components.

Leave a Comment