SuperNotSupper Posted September 10, 2017 Share Posted September 10, 2017 First off... I feel like an Idiot. I saw the sticky about switching from Reiserfs to XFS and decided to go ahead and do it. All I saw was the instructions on how to do it and it was as simple as taking the array offline and switching them. What I didn't read was the part about losing all your data. So I've formatted three of my drives and subsequently lost everything on it. So dumb on my part. Can anyone help me recover the data? Is it possible? I would have to think the information is still on those drives and I can perform some sort of recovery to get it back. I will be devastated if I cannot get it back. Years of work on there that will be unrecoverable. Quote Link to comment
CHBMB Posted September 10, 2017 Share Posted September 10, 2017 (edited) First thing I'd recommend is wait for someone like @johnnie.black to come along to offer some help. Edited September 10, 2017 by CHBMB Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 Thanks for the quick reply. I stopped the array immediately after realizing the grievous error I had made. Quote Link to comment
CHBMB Posted September 10, 2017 Share Posted September 10, 2017 4 minutes ago, SuperNotSupper said: Thanks for the quick reply. I stopped the array immediately after realizing the grievous error I had made. I might be wrong, but I'm pretty sure others have recovered data successfully after similar events. But now would be a good time to tell you before someone else does that Unraid isn't a substitute for good backups. Quote Link to comment
JorgeB Posted September 10, 2017 Share Posted September 10, 2017 Since they are reiser there's a good chance of recovering most/all data, first thing you need to do is click on each of those disks and change the filesystem back to reiser, so they will mount if they're fixed, then make sure you're running unRAID 6.3.3 or later as there were issues with reiserfsck on some of the previous release, finally start the array in maintenance mode and run: reiserfsck --rebuild-tree --scan-whole-partition /dev/mdX Replace X with the disks to recover, there's a small chance the superblock needs to be rebuilt, post here the reiserfsck output if more help needed. 2 Quote Link to comment
FreeMan Posted September 10, 2017 Share Posted September 10, 2017 I'm just going to chime in and say that if ReiserFS can recover from a disk formatting, it isn't all bad! I know the basic backstory on why everyone's moving away from it, but dang! that reiserfsck seems to be quite the handy little tool! Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 (edited) 27 minutes ago, johnnie.black said: Since they are reiser there's a good chance of recovering most/all data, first thing you need to do is click on each of those disks and change the filesystem back to reiser, so they will mount if they're fixed, then make sure you're running unRAID 6.3.3 or later as there were issues with reiserfsck on some of the previous release, finally start the array in maintenance mode and run: reiserfsck --rebuild-tree --scan-whole-partition /dev/mdX Replace X with the disks to recover, there's a small chance the superblock needs to be rebuilt, post here the reiserfsck output if more help needed. Johnnie -- many thanks for the prompt response. When I ran the rebuild this is the message I got: ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ************************************************************* Will rebuild the filesystem (/dev/md2) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md2. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. Edited September 10, 2017 by SuperNotSupper Quote Link to comment
JorgeB Posted September 10, 2017 Share Posted September 10, 2017 You need to rebuild the superblock first, follow these instructions carefully: https://wiki.lime-technology.com/Check_Disk_Filesystems#Rebuilding_the_superblock After it's done, rerun the previous command. 1 Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 24 minutes ago, johnnie.black said: You need to rebuild the superblock first, follow these instructions carefully: https://wiki.lime-technology.com/Check_Disk_Filesystems#Rebuilding_the_superblock After it's done, rerun the previous command. I followed the steps and reran for one of the drives. Here is my output: Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal: Trans replayed: mountid 95, transid 39935, desc 2822, len 1, commit 2824, next trans offset 2807 Trans replayed: mountid 95, transid 39936, desc 2825, len 1, commit 2827, next trans offset 2810 Trans replayed: mountid 95, transid 39937, desc 2828, len 1, commit 2830, next trans offset 2813 Trans replayed: mountid 95, transid 39938, desc 2831, len 1, commit 2833, next trans offset 2816 Trans replayed: mountid 95, transid 39939, desc 2834, len 1, commit 2836, next trans offset 2819 Trans replayed: mountid 95, transid 39940, desc 2837, len 1, commit 2839, next trans offset 2822 Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 6 transactions replayed Quote Link to comment
JorgeB Posted September 10, 2017 Share Posted September 10, 2017 Rebuild tree is running, it will take a long time, 2 to 3 hours per TB. 1 Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 Awesome. Can I run more than one disc at a time? Is there a way to check status or to know when it is completed? Quote Link to comment
CHBMB Posted September 10, 2017 Share Posted September 10, 2017 I don't know the answer to the question, but I'd run one at a time if it were me 1 Quote Link to comment
JorgeB Posted September 10, 2017 Share Posted September 10, 2017 (edited) 7 minutes ago, SuperNotSupper said: Awesome. Can I run more than one disc at a time? Yes, open another SSH session or use screen. 7 minutes ago, SuperNotSupper said: Is there a way to check status or to know when it is completed? It will display some info, this is what a typical run looks like: Will rebuild the filesystem (/dev/md6) tree Will put log info to 'stdout' Replaying journal: Replaying journal: Done. Reiserfs journal '/dev/md6' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sat Sep 2 12:18:27 2017 ########### Pass 0: Loading on-disk bitmap .. ok, 862357032 blocks marked used Skipping 52923 blocks (super block, journal, bitmaps) 862304109 blocks will be read 0%....20%....40%....60%....80%....100% "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 862304109 Leaves among those 851507 - leaves all contents of which could not be saved and deleted 2 Objectids found 3242 Pass 1 (will try to insert 851505 leaves): Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% Flushing..finished 851505 leaves read 851501 inserted 4 not inserted Pass 2: 0%....20%....40%....60%....80%....100% Flushing..finished Leaves inserted item by item 4 Pass 3 (semantic): Flushing..finished Files found: 3100 Directories found: 141 Names pointing to nowhere (removed): 1 Pass 3a (looking for lost dir/files): Looking for lost directories: Flushing..finished Pass 4 - finished Deleted unreachable items 6687 Flushing..finished Syncing..finished ########### reiserfsck finished at Sat Sep 2 22:45:17 2017 ########### ####### Pass 0 ####### block 180906004: The number of items (65416) is incorrect, should be (1) - corrected block 180906004: The free space (270) is incorrect, should be (3279) - corrected pass0: vpf-10110: block 180906004, item (0): Unknown item type found [67117057 604052743 0xff85005a ??? (15)] - deleted block 183197996: The number of items (65416) is incorrect, should be (1) - corrected block 183197996: The free space (268) is incorrect, should be (1489) - corrected pass0: vpf-10110: block 183197996, item (0): Unknown item type found [167841791 16909312 0x1a014d ??? (5)] - deleted 3240 directory entries were hashed with "r5" hash. ####### Pass 1 ####### ####### Pass 2 ####### ####### Pass 3 ######### rebuild_semantic_pass: The entry [112 113] ("Edge Of Tomorrow (Norge Warner 5000188798).iso") in directory [69 112] points to nowhere - is removed vpf-10650: The directory [69 112] has the wrong size in the StatData (112) - corrected to (48) vpf-10680: The file [106 107] has the wrong block count in the StatData (17056) - corrected to (10376) ####### Pass 3a (lost+found pass) ######### Edited September 10, 2017 by johnnie.black 1 Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 Forgive my ignorance on this, but is there a command to pull the journal log up? Nothing has happened passed the text output "Reiserfs journal '/dev/md1' in blocks [18..8211]: 6 transactions replayed" So at this point I do not see any of the display information you show above. (I'm trying to be patient but I'm extremely anxious about seeing if this is resolved) Quote Link to comment
JorgeB Posted September 10, 2017 Share Posted September 10, 2017 The info is shown in order, as long as there's no error and you didn't get the cursor back it's still running, there should also be disk activity. Quote Link to comment
SuperNotSupper Posted September 10, 2017 Author Share Posted September 10, 2017 Alright, so I gave it some time to see if anything has changed. The reads/writes haven't changed on the monitor and nothing changed on the Putty script. I believe I might have hit enter or something when I was trying to copy the text into the forum. I ran a second drive in another session and it is showing the results you posted above. (read/writes are changing in the monitor as well) Should I rerun the rebuild for the first drive? Quote Link to comment
JorgeB Posted September 11, 2017 Share Posted September 11, 2017 Make sure it's stopped (or hit ctrl +c) and run it again. Quote Link to comment
garycase Posted September 11, 2017 Share Posted September 11, 2017 Reiserfsck can do some amazing recoveries -- hopefully it will help here. It should as long as you didn't do ANYTHING on the disks except format them ... but as Johnnie noted it can take a LONG time. Just have patience and hopefully you'll recover a significant amount of your data. However, as CHBMB noted above ... 13 hours ago, CHBMB said: now would be a good time to tell you before someone else does that Unraid isn't a substitute for good backups. Once you recover your data, you should keep this in mind and be sure you have backup copies of anything you would be upset about losing. I always tell folks they should always assume their system is going to suffer a catastrophic crash at midnight; and if there's anything they're going to be upset about losing be sure it's backed up before then Quote Link to comment
JorgeB Posted September 11, 2017 Share Posted September 11, 2017 12 hours ago, SuperNotSupper said: Can I run more than one disc at a time? One thing I forgot to mention about this, --rebuild-tree requires quite a bit of RAM, it's good to have 1GB per TB when running it on a single disk, so for running multiple instances and depending on the disks size, it may require more RAM than you have available, but there won't be any damage, if it runs out of RAM reiserfsck will abort and you'll need to run it again. Quote Link to comment
BobPhoenix Posted September 11, 2017 Share Posted September 11, 2017 Just as an FYI on the recovery potential of reiserfsck. I had a 2TB ReiserFS cache drive and I mistakenly started my array with it in the parity position and it started to build parity on my FULL cache drive (unRAID 4.7 time frame). I caught it quickly and ran reiserfsck on it and got back all but 200GB of data. I also got back most of my files with names intact as well. Only had ~4 files that had lost+found names that I had to rename back to the correct file names again. That is why I hated going to XFS but with all the troubles with an obsolete file system and newer versions of the kernels in unRAID made me change. Quote Link to comment
SuperNotSupper Posted September 12, 2017 Author Share Posted September 12, 2017 The last drive is running tonight. Any steps I should take after I get the completed message shown below? Or is it a simple matter of bringing the array back online. Flushing..finished8, 162 /sec Objects without names 16 Dirs linked to /lost+found: 4 Dirs without stat data found 1 Files linked to /lost+found 12 Objects having used objectids: 2 files fixed 2 Pass 4 - finisheddone 733586, 158 /sec Deleted unreachable items 30949 Flushing..finished Syncing..finished ########### reiserfsck finished at Mon Sep 11 11:25:36 2017 ########### Quote Link to comment
SuperNotSupper Posted September 12, 2017 Author Share Posted September 12, 2017 (edited) 17 hours ago, garycase said: Reiserfsck can do some amazing recoveries -- hopefully it will help here. It should as long as you didn't do ANYTHING on the disks except format them ... but as Johnnie noted it can take a LONG time. Just have patience and hopefully you'll recover a significant amount of your data. However, as CHBMB noted above ... Once you recover your data, you should keep this in mind and be sure you have backup copies of anything you would be upset about losing. I always tell folks they should always assume their system is going to suffer a catastrophic crash at midnight; and if there's anything they're going to be upset about losing be sure it's backed up before then Thanks for the words of encouragement. I can't believe I formatted them without stopping to think that it would erase all the data. It seemed so easy to click from reiserfs to XFS that I didn't even think about it. Also the fact that it formatted in a matter of less than a minute didn't tip me off either. It was when I saw that I had 12 tb of space suddenly open up is when I panicked. I suppose I would almost need to have another unRaid server of equal storage capacity to backup everything. I have over 20 TB of data. Edited September 12, 2017 by SuperNotSupper Quote Link to comment
garycase Posted September 12, 2017 Share Posted September 12, 2017 20TB sounds like a lot to back up, but not when you consider the size of modern drives AND the impact of losing that much data. There are a variety of ways to do it -- a 2nd UnRAID server (a good idea); a bunch of offline drives that you update periodically; or even a cloud backup if you have high enough internet bandwidth. I have both a backup server (that my other 2 servers back up to) AND a set of offline disks (most of which are simply smaller disks (1-3TB) that I've repurposed to backups as I upgraded to larger disks) that I keep in a waterproof, fireproof, data-rated safe. Quote Link to comment
JorgeB Posted September 12, 2017 Share Posted September 12, 2017 5 hours ago, SuperNotSupper said: Or is it a simple matter of bringing the array back online. Just start the array, hopefully most data will be on its place but you'll need to check lost+found folder on each disk for any lost/unmaned files. Quote Link to comment
SuperNotSupper Posted September 12, 2017 Author Share Posted September 12, 2017 6 hours ago, johnnie.black said: Just start the array, hopefully most data will be on its place but you'll need to check lost+found folder on each disk for any lost/unmaned files. Success!!!! Thank you so much for your assistance. Let this be a lesson to other people in the future. Any tips or tricks for items I'm seeing in the lost in found folder? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.