Help needed with writing data back to drive now showing not installed

[thephoneguy]

I would love to be able to NOT loose about 2.7TB of data on a drive. Here is the scenario:

I had a drive that was failing/failed and had not yet replaced it. I went away for weekend and came back to see server stopped and a drive missing. I did a drive health check and saw that it was fine, so I inquired about power and it went out over weekend and my UPS was dead. So it was shut down and mobo dropped the drive. I shut down and did the initconfig cmd line in putty to put the drive back, but when I went to add the one drive that was failing, it was missing altogether. The array started and it was going to do a parity check, so i stopped, did initconfig again, and added all the drives and said trust parity, knowing that the info for the data is in parity. I screwed up as I dont have the super.dat file that had my failed drive (didn't realize that it only backs up last copy) and I have even done data recovery on the usb stick hoping to pull old super.dat file, but no luck.

 

What I need to know is, how can I "tell" unraid that my drive #4 that shows not installed is actually missing, so I can add a new drive and allow a rebuild with parity, which is valid. Any help would be GREATLY appreciated.

 

Three things;

I don't need to know what I did wrong, I know that already. I was missing a drive and my chache drive (which I had all forgotten about) messed with my count. I counted 10 drives before i did the init. Just forgot about the cache drive.

Why am I on 5.0.6? Because I am using Resilio Sync ( I have 3 unraid servers identically configured) and it runs flawlessly. When I tried moving to 6, it was a plugin nightmare. Worked only while version was valid. Update meant starting over again. With the Phaze plugin for 5, I can update and loose nothing. It has updated MANY times, all without issue. 6 did not work as intended. So no, i wont be going to 6. I have simple features, monitors, UPS etc. Love it as is.

The drive that is gone, is really gone. Its not valuable enough for me to pay for data restoration, but it is going to be a pain if i loose it.

 

[Helmonder]

I would love to be able to NOT loose about 2.7TB of data on a drive. Here is the scenario:

I had a drive that was failing/failed and had not yet replaced it. I went away for weekend and came back to see server stopped and a drive missing. I did a drive health check and saw that it was fine, so I inquired about power and it went out over weekend and my UPS was dead. So it was shut down and mobo dropped the drive. I shut down and did the initconfig cmd line in putty to put the drive back, but when I went to add the one drive that was failing, it was missing altogether. The array started and it was going to do a parity check, so i stopped, did initconfig again, and added all the drives and said trust parity, knowing that the info for the data is in parity. I screwed up as I dont have the super.dat file that had my failed drive (didn't realize that it only backs up last copy) and I have even done data recovery on the usb stick hoping to pull old super.dat file, but no luck.

 

What I need to know is, how can I "tell" unraid that my drive #4 that shows not installed is actually missing, so I can add a new drive and allow a rebuild with parity, which is valid. Any help would be GREATLY appreciated.

 

Three things;

I don't need to know what I did wrong, I know that already. I was missing a drive and my chache drive (which I had all forgotten about) messed with my count. I counted 10 drives before i did the init. Just forgot about the cache drive.

Why am I on 5.0.6? Because I am using Resilio Sync ( I have 3 unraid servers identically configured) and it runs flawlessly. When I tried moving to 6, it was a plugin nightmare. Worked only while version was valid. Update meant starting over again. With the Phaze plugin for 5, I can update and loose nothing. It has updated MANY times, all without issue. 6 did not work as intended. So no, i wont be going to 6. I have simple features, monitors, UPS etc. Love it as is.

The drive that is gone, is really gone. Its not valuable enough for me to pay for data restoration, but it is going to be a pain if i loose it.

 

Since you mention a parity check running I conclude you have a parity drive, that should mean that your drive is beiing emulated at the moment and your data is still there, right ?  That should mean that you can just assign a correct drive to the slot that got red balled and it will reconstruct..

 

I do not full grasp what you have been trying to do with initconfig and super.dat ..

 

Base question: Is your drive emulated right now or did something got messed up and it is not ?

[JorgeB]

...parity, which is valid.

 

The above is not true, because of this:

 

The array started and it was going to do a parity check, so i stopped, did initconfig again

 

After initconfig (and since you didn't check "parity is already valid") unRAID started a parity sync, so from sector 0 to whenever you stopped it parity is not valid anymore.

 

Depending on long it was and since its reiser some/most data may be still recoverable, I can post a procedure do do what you want but for best results the new disk should be the same size of the failed one.

[thephoneguy]

it did not have a chance to start the parity sync. The drive that i lost had a single share pointed to it, and the data and the share are gone. Parity was not touched, it is still valid, but drive gone.

[JorgeB]

The array started and it was going to do a parity check, so i stopped, did initconfig again

 

The only automatic parity operation that could start after doing a initconfig is a parity sync, so yes, your parity is corrupt, how much depends on how long it ran before you canceled.

[thephoneguy]

The parity sync never started, I stopped the array and initconfig LONG before it had a chance to do so. Is there any hope?

[JorgeB]

I understood by what you wrote that you started the array after the first initconfig, if you did (without checking "parity is already valid") a parity sync would start immediately, although if promptly canceled there's still a good chance of most/all data being recoverable using the reiser tools on the rebuilt disk.

[thephoneguy]

The time that transpired between accepting the config and stopping was literally less than a second, like when you mouse click something as you realize what you did was wrong. It was truly instant, as my brain registered the mistake. So parity sync does take a sec to start, it didnt have the time to.

So what do i do now to tell it to rebuild disk 4 even though it does not say "missing"

The actual drive is F$%*ed, so no recovery there. only option is to have parity rebuild the drive.

 

thanks for everyone's input

[RobJ]

If you could give us a little more info about your setup right now, we could probably help you better.  As it is, we don't know what is assigned, what is not, and what is emulated.  A picture of the Main screen would help, and/or a zip file with your syslog, super.dat, and disk.cfg.  You are at a point where no mistakes can be made, so it's best we know the exact situation, before giving you the right steps to take.

 

I assume you have a new drive ready to go...

[JorgeB]

I still think your parity will have some corruption but it should be fixable, you can try this:

 

-Utils -> New Config

-assign all disks, double check parity is the parity slot, assign a new disk of the same size of the failed disk you want to rebuild

-check both "parity is already valid" and "maintenance mode" before starting the array

-start the array

-stop array, unassign disk to rebuild

-start array, check emulated disk, if it mounts parity is good, if doesn't there's some corruption but proceed

-stop array, reassign new disk

-start array to begin rebuild

 

If the disk is showing unformatted finish the rebuild then run reiserfsck on it.

[thephoneguy]

Here is a pic of the server running. Basically I now have a drive missing, and its data. If the drive showed "missing" then data would be emulated. Its not. its as if the drive was never part of the server. Will your suggestion allow me to force the server to re aquire the lost drive and rebuiild, or will it simply add the blank storage space? Yes, I have a drive ready to roll.

[Helmonder]

Here is a pic of the server running. Basically I now have a drive missing, and its data. If the drive showed "missing" then data would be emulated. Its not. its as if the drive was never part of the server. Will your suggestion allow me to force the server to re aquire the lost drive and rebuiild, or will it simply add the blank storage space? Yes, I have a drive ready to roll.

 

(For the benefit of other readers: the fact that the drive is not beiing emulated is because of the initconfig that was done)

[JorgeB]

Here is a pic of the server running. Basically I now have a drive missing, and its data. If the drive showed "missing" then data would be emulated. Its not. its as if the drive was never part of the server. Will your suggestion allow me to force the server to re aquire the lost drive and rebuiild, or will it simply add the blank storage space? Yes, I have a drive ready to roll.

It will allow you to recover the missing disk data but success will depend on how valid current parity is.

[RobJ]

I still think your parity will have some corruption but it should be fixable, you can try this:

 

-Utils -> New Config

-assign all disks, double check parity is the parity slot, assign a new disk of the same size of the failed disk you want to rebuild

-check both "parity is already valid" and "maintenance mode" before starting the array

-start the array

-stop array, unassign disk to rebuild

-start array, check emulated disk, if it mounts parity is good, if doesn't there's some corruption but proceed

-stop array, reassign new disk

-start array to begin rebuild

 

If the disk is showing unformatted finish the rebuild then run reiserfsck on it.

 

I just want to concur with johnnie.black about the method above, since it sounds like you still are in doubt it could work.  What he's doing is first tricking the system into thinking there is a Disk 4, basically recovering your lost super.dat, then removing it so that once re-added, it will trigger a rebuild.  Just make sure you do NOT do any writing to the temporary Disk 4, such as formatting it (that's why he specifies Maintenance Mode).  And absolutely do not do any parity checking or building!  You'll have a chance on the second array start to check whether you can see the original files on Disk 4.  If you can see the files, you are now in position to prepare for rebuilding Disk 4!  But as he says, whether you can see them or you get file system errors, keep going!  Once the drive is rebuilt, reiserfsck can do its magic, and probably recover all or almost all of your Disk 4 files and folders.

 

In v4, there used to be a way to manually force a rebuild of a drive, using for example mdcmd set invalidslot 4 to force rebuilding Disk 4.  But as far as we know, it no longer works in v5.

[JorgeB]

Here is a pic of the server running. Basically I now have a drive missing, and its data. If the drive showed "missing" then data would be emulated. Its not. its as if the drive was never part of the server. Will your suggestion allow me to force the server to re aquire the lost drive and rebuiild, or will it simply add the blank storage space? Yes, I have a drive ready to roll.

 

Re-reading your post I'm hoping that screenshot was from before you did the initconfig because of all the writes and all.

[thephoneguy]

Screen shot is present day. Now. After this all went down. I posted screenshots of what it looked like before in the earlier post. Will try this now.

[JorgeB]

Yes, I was not thinking straight, as long as you didn't ran a correcting parity check, and since v5 doesn't support reconstruct write it doesn't matter if you wrote to the other disks, still believe that there will be some parity corruption due to the canceled parity sync but it should still work.

[thephoneguy]

So went through steps, logically it makes sense. It did not emulate, and it is showing unformatted, but is doing the re build so will wait until it is done, some time Wednesday from what it says, then run the  reiserfsck on disk 4. When I did the "slight of hand" it showed disk 4 as "not installed" but red rather than "missing". Like you said, we will see what comes of it, as its showing constant write activity to disk 4 and read from parity and all other drives as well. Its all going to come down to the parity, which I believe was good.

 

Thank you very much for your input and I will remain cautiously optimistic.

Edited February 21, 2017 by thephoneguy

[trurl]

33 minutes ago, thephoneguy said:

... doing the re build so will wait until it is done, some time Wednesday from what it says...

Shouldn't take that long

[thephoneguy]

Yes, speeding up. its 100MB per second now, started out at 34MB. Its decent hardware; supermicro aoc-sas2lp-mv8 and decent cpu/ram. Says 9 hours now.

[thephoneguy]

This is after the check and rebuild

 

root@media-svr:~# reiserfsck --check /dev/md4
reiserfsck 3.6.24

Will read-only check consistency of the filesystem on /dev/md4
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Tue Feb 21 09:42:44 2017
###########
Replaying journal: No transactions found
Zero bit found in on-disk bitmap after the last valid bit.
Checking internal tree..

Bad root block 0. (--rebuild-tree did not complete)

Aborted (core dumped)
 

[JorgeB]

Use rebuild tree:

 

reiserfsck --rebuild-tree /dev/md4

 

[thephoneguy]

I had another drive pop while doing the rebuild so will wait until its done. I have learned the lesson to make sure you replace a drive once it goes red, and quickly. SO not going to chance this one. Will run the rebuild tree after my drive 8 rebuilds. Stay tuned and thank you again.

[JorgeB]

2 minutes ago, thephoneguy said:

I have learned the lesson to make sure you replace a drive once it goes red, and quickly.

Yes, you should deal with that first.

[thephoneguy]

Im seriously stressing now, as I cannot see any files on my server, just the folder shares. WTF?!!

 

I can click the folder icon beside each drive and see contents, but the actual SMB paths are empty.

Edited February 21, 2017 by thephoneguy