2 disks in error state (disk dsbl)

[T800]

My movie server goes to sleep when not in use. Last parity check was 70 days ago because it was most likely asleep when the 1st of the month came around. Every now and then (a couple of months) I have a look at the GUI. I write data to it probably every other day. I tried to add a large file yesterday and finder in OSX came up with "disk is full". I entered the GUI via my phone just before I went to sleep to look at the disk space and found two errors:

Disk 5 in error state (disk dsbl)

and 

Disk 7 in error state (disk dsbl )

 

When I saw them on my phone for some reason I never noticed the dsbl to most likely mean disabled.

 

This morning I woke the server, both drives wouldn't do any smart tests and I could get anything from them. I stopped and put in maintenance mode and the disks were missing. I restarted the server and now the disks are missing but show up in "Unassigned Devices". I didn't scroll down enough during maintenance mode to see if they were in unassigned.

Both have SMART reports that say they have completed without error. I tried a short SMART report they both passed.

 

So right now I've got two 4 TB drives missing from my array with the content being "emulated" apparently and two 4TB drives as unassigned devices waiting to be added. 

 

What should I do?

 

 

 

 

[T800]

I realized I should have taken the syslog before I rebooted. Is there a reason why it doesn't save the last few?

 

Heres the smart logs:

 

 

tower-smart-20171207-0911.zip

tower-smart-20171207-0911 (1).zip

Edited December 7, 2017 by T800

[JorgeB]

SMART looks fine for both but I'd like to see the full diagnostics.

[T800]

As in an extended smart test?

[JorgeB]

That would also be a good idea to make sure the disks are really OK, but what I meant was going to Tools -> Diagnostics and upload the resulting zip.

[T800]

Here you go:

 

 

tower-diagnostics-20171207-1021.zip

[JorgeB]

I mostly wanted to check the hardware used since two simultaneous disable disks is unusual, but nothing jumps out, and they are on different controllers.

 

Make sure you enable system notifications to be warned the moment there's a problem and try rebuild both disks using the old disks, if you want run an extended SMART test on both before the rebuild.

[T800]

They weren't simultaneous, I forgot to mention in the notifications last night they were a week apart. 

 

Do I need to preclear them or just add to the array? 

[JorgeB]

Just stop the array, assign both, start the array to begin the rebuild.

[JorgeB]

Any more issues don't forget to grab the diagnostics before rebooting.

[T800]

Okay thanks, rebuilding now. See what happens in 30 hours when it finishes.

[SSD]

Before a rebuild, I always recommend looking at the simulated disk(s) to make sure everything looks ok.

 

Although the two disks dropped offline, their data is likely intact (except for recently added files since they dropped offline). And if rebuilding onto different disks, these original disks would act as a decent backup to protect against a bad rebuild.

 

Your chances of a problem are pretty low, but thought I'd mention for other readers and for the future. Rebuilding 2 disks leaves you unprotected during a whole dual operation, so you are dropping your pants completely when you didn't have to.

 

Btw, perfectly good drives that drop offline are often caused by bad or lose cabling. 

[T800]

 

Why so unprotected when rebuilding two disks? When you say didn't have to is that meaning I should have added 2 new disks instead?

[SSD]

2 hours ago, T800 said:

 

Why so unprotected when rebuilding two disks? When you say didn't have to is that meaning I should have added 2 new disks instead?

 

First should have checked emulated disks looked good. No extra disks needed. We had a rebuild complete about a month ago, but content was garbage. Rebuilt over the disk that was kicked (that would have been virtually perfect).

 

I know it's expeditious, but I really dislike the rebuild over the kicked disk concept. Could have done with 2 extra disks. Or could have rebuilt one at a time and done with one extra disk. Always good to have at least one disk in the wings.

 

If you have backups, none of this would matter as your data is protected.

[T800]

I see, would it have been easy to spot the garbage, 4 TB x 2 is a lot of content?

I used to have a hot spare ready to go but with the house move it became lower in priority, granted I may regret that tomorrow.

 

I've got two parity disks if that a makes any difference.

Edited December 7, 2017 by T800

[JorgeB]

I checked before and the emulated disks were mounting correctly so as long as parity is in sync data should be OK.

[SSD]

9 minutes ago, T800 said:

I see, would it have been easy to spot the garbage, 4 TB x 2 is a lot of content?

I used to have a hot spare ready to go but with the house move it became lower in priority, granted I may regret that tomorrow.

 

I've got two parity disks if that a makes any difference.

 

Without the two parity disks, you'd have had to use the two disk 'as is'. Any recently added content added you either would have been lost. Since neither disk actually failed, the loss would have been only recently added.

 

The key concept about parity is that it can be completely fubar and there would be no visible signs. Only a parity check would tell. Now unRaid does an excellent job of maintaining parity, but there are several ways it can get screwed up. And if one gets screwed up, so does the other. And it's normally not subtle when it happens. Compare with raid which stripes data. It has options in terms of how it reads the data (whichever it can do faster). It does integrity checks. Once any sort of out of sync condition is detected, the array is invalidated. So unlike unRaid, no way it can be corrupt if it's working.

 

I'd check the disk to make sure the directory shows up. And I'd check a few files, including older and newer content. Checking md5s on then if I had them, or playing the media (Quickly skipping through). I'd also copy off any content that I was particularly concerned about - like the 2000 pictures you recently copied of your wife's cell phone that you'd be shot if got lost! 

[T800]

How would I build one disk at a time?

Let's say I add my hot spare (arriving today) in the same scenario as yesterday would the array start if I only added the one disk and left the other unassigned?

What would happen to the content of the 2nd disk not added? Would it just stay in the parity disk until I add the 2nd?

 

How do I get md5s?

Edited December 8, 2017 by T800

[JorgeB]

8 minutes ago, T800 said:

would the array start if I only added the one disk and left the other unassigned?

Yes

 

8 minutes ago, T800 said:

What would happen to the content of the 2nd disk not added? Would it just stay in the parity disk until I add the 2nd?

Would remain emulated

 

8 minutes ago, T800 said:

How do I get md5s?

Use btrfs or manually create them with the file integrity plugin, corz, etc.

[T800]

22 minutes ago, johnnie.black said:

Yes

 

Would remain emulated

 

Use btrfs or manually create them with the file integrity plugin, corz, etc.

Clever stuff.

I noticed that the plugin may cause issues on reiserfs.

[JorgeB]

1 minute ago, T800 said:

I noticed that the plugin may cause issues on reiserfs.

Yes, but you shoudn't be using reiser anyway, reiser is a dead filesystem with terrible performance in some cases, and other issues.

[T800]

What are my options with a 50TB server with all 14 disks in it all set to reiser?

[JorgeB]

Convert them to xfs (or btrfs), or if you don't want to you can use corz to create and check checksums.

[T800]

Can it be done one disc at a time?

[JorgeB]

Yes, that's how it's usually done:

 

https://wiki.lime-technology.com/File_System_Conversion