HELP! Made A Huge Mistake - Formatted from Reiserfs to XFS and Lost Data

[SuperNotSupper]

First off... I feel like an Idiot.

 

I saw the sticky about switching from Reiserfs to XFS and decided to go ahead and do it. All I saw was the instructions on how to do it and it was as simple as taking the array offline and switching them. What I didn't read was the part about losing all your data.  So I've formatted three of my drives and subsequently lost everything on it. So dumb on my part.

 

Can anyone help me recover the data? Is it possible? I would have to think the information is still on those drives and I can perform some sort of recovery to get it back. I will be devastated if I cannot get it back. Years of work on there that will be unrecoverable.

 

 

[CHBMB]

First thing I'd recommend is wait for someone like @johnnie.black to come along to offer some help.

Edited September 10, 2017 by CHBMB

[SuperNotSupper]

Thanks for the quick reply. I stopped the array immediately after realizing the grievous error I had made.

[CHBMB]

4 minutes ago, SuperNotSupper said:

Thanks for the quick reply. I stopped the array immediately after realizing the grievous error I had made.

 

I might be wrong, but I'm pretty sure others have recovered data successfully after similar events.  But now would be a good time to tell you before someone else does that Unraid isn't a substitute for good backups. 

[JorgeB]

Since they are reiser there's a good chance of recovering most/all data, first thing you need to do is click on each of those disks and change the filesystem back to reiser, so they will mount if they're fixed, then make sure you're running unRAID 6.3.3 or later as there were issues with reiserfsck on some of the previous release, finally start the array in maintenance mode and run:

 

reiserfsck --rebuild-tree --scan-whole-partition /dev/mdX

 

Replace X with the disks to recover, there's a small chance the superblock needs to be rebuilt, post here the reiserfsck output if more help needed.

[FreeMan]

I'm just going to chime in and say that if ReiserFS can recover from a disk formatting, it isn't all bad! I know the basic backstory on why everyone's moving away from it, but dang! that reiserfsck seems to be quite the handy little tool!

[SuperNotSupper]

27 minutes ago, johnnie.black said:

Since they are reiser there's a good chance of recovering most/all data, first thing you need to do is click on each of those disks and change the filesystem back to reiser, so they will mount if they're fixed, then make sure you're running unRAID 6.3.3 or later as there were issues with reiserfsck on some of the previous release, finally start the array in maintenance mode and run:

 

reiserfsck --rebuild-tree --scan-whole-partition /dev/mdX

 

Replace X with the disks to recover, there's a small chance the superblock needs to be rebuilt, post here the reiserfsck output if more help needed.

 

Johnnie -- many thanks for the prompt response.

 

When I ran the rebuild this is the message I got:

 

*************************************************************
** Do not  run  the  program  with  --rebuild-tree  unless **
** something is broken and MAKE A BACKUP  before using it. **
** If you have bad sectors on a drive  it is usually a bad **
** idea to continue using it. Then you probably should get **
** a working hard drive, copy the file system from the bad **
** drive  to the good one -- dd_rescue is  a good tool for **
** that -- and only then run this program.                 **
*************************************************************

Will rebuild the filesystem (/dev/md2) tree
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

reiserfs_open: the reiserfs superblock cannot be found on /dev/md2.
Failed to open the filesystem.

If the partition table has not been changed, and the partition is
valid  and  it really  contains  a reiserfs  partition,  then the
superblock  is corrupted and you need to run this utility with
--rebuild-sb.

 

 

 

 

Edited September 10, 2017 by SuperNotSupper

[JorgeB]

You need to rebuild the superblock first, follow these instructions carefully:

 

https://wiki.lime-technology.com/Check_Disk_Filesystems#Rebuilding_the_superblock

 

After it's done, rerun the previous command.

 

 

[SuperNotSupper]

24 minutes ago, johnnie.black said:

You need to rebuild the superblock first, follow these instructions carefully:

 

https://wiki.lime-technology.com/Check_Disk_Filesystems#Rebuilding_the_superblock

 

After it's done, rerun the previous command.

 

 

 

 

I followed the steps and reran for one of the drives. Here is my output:

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
Replaying journal: Trans replayed: mountid 95, transid 39935, desc 2822, len 1, commit 2824, next trans offset 2807
Trans replayed: mountid 95, transid 39936, desc 2825, len 1, commit 2827, next trans offset 2810
Trans replayed: mountid 95, transid 39937, desc 2828, len 1, commit 2830, next trans offset 2813
Trans replayed: mountid 95, transid 39938, desc 2831, len 1, commit 2833, next trans offset 2816
Trans replayed: mountid 95, transid 39939, desc 2834, len 1, commit 2836, next trans offset 2819
Trans replayed: mountid 95, transid 39940, desc 2837, len 1, commit 2839, next trans offset 2822
Replaying journal: Done.
Reiserfs journal '/dev/md1' in blocks [18..8211]: 6 transactions replayed

 

 

 

 

[JorgeB]

Rebuild tree is running, it will take a long time, 2 to 3 hours per TB.

[SuperNotSupper]

Awesome. Can I run more than one disc at a time? Is there a way to check status or to know when it is completed?

 

 

[CHBMB]

I don't know the answer to the question, but I'd run one at a time if it were me

[JorgeB]

7 minutes ago, SuperNotSupper said:

Awesome. Can I run more than one disc at a time?

 

Yes, open another SSH session or use screen.

 

7 minutes ago, SuperNotSupper said:

Is there a way to check status or to know when it is completed?

 

It will display some info, this is what a typical run looks like:

 

Will rebuild the filesystem (/dev/md6) tree
Will put log info to 'stdout'
Replaying journal: 
Replaying journal: Done.
Reiserfs journal '/dev/md6' in blocks [18..8211]: 0 transactions replayed
###########
reiserfsck --rebuild-tree started at Sat Sep  2 12:18:27 2017
###########

Pass 0:
Loading on-disk bitmap .. ok, 862357032 blocks marked used
Skipping 52923 blocks (super block, journal, bitmaps) 862304109 blocks will be read
0%....20%....40%....60%....80%....100%
	"r5" hash is selected
Flushing..finished
	Read blocks (but not data blocks) 862304109
		Leaves among those 851507
			- leaves all contents of which could not be saved and deleted 2
		Objectids found 3242

Pass 1 (will try to insert 851505 leaves):
Looking for allocable blocks .. finished
0%....20%....40%....60%....80%....100%
Flushing..finished
	851505 leaves read
		851501 inserted
		4 not inserted

Pass 2:
0%....20%....40%....60%....80%....100%
Flushing..finished
	Leaves inserted item by item 4
Pass 3 (semantic):
Flushing..finished
	Files found: 3100
	Directories found: 141
	Names pointing to nowhere (removed): 1
Pass 3a (looking for lost dir/files):
Looking for lost directories:
Flushing..finished
Pass 4 - finished
	Deleted unreachable items 6687
Flushing..finished
Syncing..finished
###########
reiserfsck finished at Sat Sep  2 22:45:17 2017

###########
####### Pass 0 #######
block 180906004: The number of items (65416) is incorrect, should be (1) - corrected
block 180906004: The free space (270) is incorrect, should be (3279) - corrected
pass0: vpf-10110: block 180906004, item (0): Unknown item type found [67117057 604052743 0xff85005a ??? (15)] - deleted
block 183197996: The number of items (65416) is incorrect, should be (1) - corrected
block 183197996: The free space (268) is incorrect, should be (1489) - corrected
pass0: vpf-10110: block 183197996, item (0): Unknown item type found [167841791 16909312 0x1a014d ??? (5)] - deleted
3240 directory entries were hashed with "r5" hash.
####### Pass 1 #######
####### Pass 2 #######
####### Pass 3 #########
rebuild_semantic_pass: The entry [112 113] ("Edge Of Tomorrow (Norge Warner 5000188798).iso") in directory [69 112] points to nowhere - is removed
vpf-10650: The directory [69 112] has the wrong size in the StatData (112) - corrected to (48)
vpf-10680: The file [106 107] has the wrong block count in the StatData (17056) - corrected to (10376)
####### Pass 3a (lost+found pass) #########

Edited September 10, 2017 by johnnie.black

[SuperNotSupper]

Forgive my ignorance on this, but is there a command to pull the journal log up? Nothing has happened passed the text output

 

"Reiserfs journal '/dev/md1' in blocks [18..8211]: 6 transactions replayed"

 

So at this point I do not see any of the display information you show above.

 

(I'm trying to be patient but I'm extremely anxious about seeing if this is resolved)

 

[JorgeB]

The info is shown in order, as long as there's no error and you didn't get the cursor back it's still running, there should also be disk activity.

[SuperNotSupper]

Alright, so I gave it some time to see if anything has changed. The reads/writes haven't changed on the monitor and nothing changed on the Putty script. I believe I might have hit enter or something when I was trying to copy the text into the forum.

 

I ran a second drive in another session and it is showing the results you posted above. (read/writes are changing in the monitor as well)

 

Should I rerun the rebuild for the first drive?

[JorgeB]

Make sure it's stopped (or hit ctrl +c) and run it again.

[garycase]

Reiserfsck can do some amazing recoveries -- hopefully it will help here.    It should as long as you didn't do ANYTHING on the disks except format them ... but as Johnnie noted it can take a LONG time.   Just have patience and hopefully you'll recover a significant amount of your data.

 

However, as CHBMB noted above ...

13 hours ago, CHBMB said:

now would be a good time to tell you before someone else does that Unraid isn't a substitute for good backups. 

 

Once you recover your data, you should keep this in mind and be sure you have backup copies of anything you would be upset about losing.    I always tell folks they should always assume their system is going to suffer a catastrophic crash at midnight; and if there's anything they're going to be upset about losing be sure it's backed up before then

[JorgeB]

12 hours ago, SuperNotSupper said:

Can I run more than one disc at a time?

 

One thing I forgot to mention about this, --rebuild-tree requires quite a bit of RAM, it's good to have 1GB per TB when running it on a single disk, so for running multiple instances and depending on the disks size, it may require more RAM than you have available, but there won't be any damage, if it runs out of RAM reiserfsck will abort and you'll need to run it again.

[BobPhoenix]

Just as an FYI on the recovery potential of reiserfsck.

 

I had a 2TB ReiserFS cache drive and I mistakenly started my array with it in the parity position and it started to build parity on my FULL cache drive (unRAID 4.7 time frame).  I caught it quickly and ran reiserfsck on it and got back all but 200GB of data.  I also got back most of my files with names intact as well.  Only had ~4 files that had lost+found names that I had to rename back to the correct file names again.

 

That is why I hated going to XFS but with all the troubles with an obsolete file system and newer versions of the kernels in unRAID made me change.

[SuperNotSupper]

The last drive is running tonight. Any steps I should take after I get the completed message shown below? Or is it a simple matter of bringing the array back online.

Flushing..finished8, 162 /sec
        Objects without names 16
        Dirs linked to /lost+found: 4
                Dirs without stat data found 1
        Files linked to /lost+found 12
        Objects having used objectids: 2
                files fixed 2
Pass 4 - finisheddone 733586, 158 /sec
        Deleted unreachable items 30949
Flushing..finished
Syncing..finished
###########
reiserfsck finished at Mon Sep 11 11:25:36 2017
###########

 

[SuperNotSupper]

17 hours ago, garycase said:

Reiserfsck can do some amazing recoveries -- hopefully it will help here.    It should as long as you didn't do ANYTHING on the disks except format them ... but as Johnnie noted it can take a LONG time.   Just have patience and hopefully you'll recover a significant amount of your data.

 

However, as CHBMB noted above ...

 

Once you recover your data, you should keep this in mind and be sure you have backup copies of anything you would be upset about losing.    I always tell folks they should always assume their system is going to suffer a catastrophic crash at midnight; and if there's anything they're going to be upset about losing be sure it's backed up before then

 

Thanks for the words of encouragement. I can't believe I formatted them without stopping to think that it would erase all the data. It seemed so easy to click from reiserfs to XFS that I didn't even think about it. Also the fact that it formatted in a matter of less than a minute didn't tip me off either. It was when I saw that I had 12 tb of space suddenly open up is when I panicked.

 

I suppose I would almost need to have another unRaid server of equal storage capacity to backup everything. I have over 20 TB of data.

 

Edited September 12, 2017 by SuperNotSupper

[garycase]

20TB sounds like a lot to back up, but not when you consider the size of modern drives AND the impact of losing that much data.   There are a variety of ways to do it -- a 2nd UnRAID server (a good idea);  a bunch of offline drives that you update periodically; or even a cloud backup if you have high enough internet bandwidth.

 

I have both a backup server (that my other 2 servers back up to) AND a set of offline disks (most of which are simply smaller disks (1-3TB) that I've repurposed to backups as I upgraded to larger disks) that I keep in a waterproof, fireproof, data-rated safe.

 

[JorgeB]

5 hours ago, SuperNotSupper said:

Or is it a simple matter of bringing the array back online.

 

Just start the array, hopefully most data will be on its place but you'll need to check lost+found folder on each disk for any lost/unmaned files.

[SuperNotSupper]

6 hours ago, johnnie.black said:

 

Just start the array, hopefully most data will be on its place but you'll need to check lost+found folder on each disk for any lost/unmaned files.

 

 

Success!!!! Thank you so much for your assistance. Let this be a lesson to other people in the future.

 

Any tips or tricks for items I'm seeing in the lost in found folder?