Replacing 3 drives

[TODDLT]

I posted an earlier thread about replacing 3 aging drives in my array and looking for advice on 3 vs 4 TB, overall capacity, ended up talking through with the group who replied and decided to not worrying about upgrading capacity now.

 

The nail in the coffin was finding 3 TB Seagate NAS drives (ST3000VN000) on sale for $80.  I couldn't pass that up and bought 3 of them to replace 3 aging 8-9 yr old 2 TB drives.  A few questions for the group because I've never replaced 3 drives at once.

 

1. Would you attempt to replace 2 at once having dual parity.   Speedy but to me seems an unnecessary risk of something else going wrong.

2. How many pre-clears would you run in this scenario.  At the moment I'm doing 3 runs.  However, changing all 3 at once means if it turns out to be a bad batch, I'm hosed on one or the other.  

3. I'm thinking of doing the replacements one every couple weeks, or maybe even one a month.  Think this is necessary?  The sole purpose is to give the new drives a chance to fail if they might before trusting all 3 at once.  Having said that.. there is no telling how much use one drive may or may not get during a month except a parity check.  

 

The overall plan is to swap the 3 older drives out, adding 3 TB of space at the same time.  Then I will likely copy over the data from my last 2 TB (oh be it not quite so old) drive and just drop it from the array.  No needed timeline for this, but it gets rid of my last older slower drive, leaves me 2 open bays for future growth and I'ms till 1 TB ahead of my current capacity which already still has room to grow.

 

Any thoughts on a prudent way to go about this?

 

Thanks.

[garycase]

I'd do a single pre-clear pass to check for DOA or infant-mortality; then replace one drive at a time, with a parity check after each replacement.    The simple fact of doing a replacement actually exercises the drive well -- since every sector on the drive is written (with effectively random data); and a parity check afterwards reads all of that data and (assuming a good check) confirms it's good.

 

As you correctly noted, doing one drive at a time means you also have protection against a failure during the replacement process, which seems a prudent approach.

 

I would generally want to make a larger jump in capacity with a replacement; but I can understand the temptation to forego that based on the excellent pricing you got for the drives.

 

[HellDiverUK]

I never bother preclearing.  I just run a single full format of a drive in a Windows box, and that does it.  Any early failures of drives I've had have crapped out almost immediately.

[JorgeB]

4 hours ago, TODDLT said:

1. Would you attempt to replace 2 at once having dual parity.   Speedy but to me seems an unnecessary risk of something else going wrong.

 

Depends if the array is static or not during the replacement, if it's not replace one at a time, if it is I would replace two at once, you could use the old disks if anything goes wrong.

[TODDLT]

4 hours ago, johnnie.black said:

 

Depends if the array is static or not during the replacement, if it's not replace one at a time, if it is I would replace two at once, you could use the old disks if anything goes wrong.

 

I generally try to leave the array alone during parity checks and drive replacements.  If nothing else it's painfully slow during that time to access anything, though if urgent, it always works.

 

5 hours ago, garycase said:

I'd do a single pre-clear pass to check for DOA or infant-mortality; then replace one drive at a time, with a parity check after each replacement.    The simple fact of doing a replacement actually exercises the drive well -- since every sector on the drive is written (with effectively random data); and a parity check afterwards reads all of that data and (assuming a good check) confirms it's good.

 

As you correctly noted, doing one drive at a time means you also have protection against a failure during the replacement process, which seems a prudent approach.

 

I would generally want to make a larger jump in capacity with a replacement; but I can understand the temptation to forego that based on the excellent pricing you got for the drives.

 

 

I agree with making the larger jump... generally.  However, here was my analysis and thinking (copied from the other post:

 

Quote

I have 2 parity drives and 8 spinning data drives.  Leaves me 2 empty bays (one of which currently holds a spare).  

I have 13.3 TB used of 20.5 TB total space.. 65% full, so not in any danger of "needing" space having 7.2 TB free

There are 4 - 2 TB drives in the mix.  3 of them are 8-9 years old.

 

Scenario 1:

7.2 TB currently free expands to 11.2 TB if i just replace the remaining 2 TB drives with 3's.  

11.2 TB goes to 17.2 TB by filling in those last two spaces with more 3 - TB drives.

(this is at LEAST 5 years of storage growth)

That scenario buys me 10 TB of free space over what I currently have and would mean buying 6 - 3 TB drives at a cost of roughly $100 / 3 TB drive...  $600 for roughly 10 TB of added space.  $60 / TB

 

Scenario 2:

I swap the two parities out now with 4 TB drives, and used those 3 TB drives plus my warm spare to replace the 3 older 2 TB drives.  Then buy 4 TB to replace the last one plus filling in the two empty bays.  That nets me 13 TB of added space above my current capacity and costs me 5 drives at $135 each.  Then for the sake of keeping these scenarios equal i have to replace the spare...  another $135.  $810 or $62 per TB

 

I guess my thinking is trying to predict what the cost effective storage technology is going to be in 5 years is a little gray and hazy :), so not sure over investing in anything makes sense.  If Scenario 1 will last me 5 years or more (conservatively)...   wouldn't it make sense to ride it out and see what changes?  

 

The only difference is I am going to drop the 3rd 2 TB and add more drives when needed.  I'll have 8 TB free and can expand that to 14 without every upgrading my parity drives.  I've only used 13 so far.   5 years of growth and who knows what will be happening storage wise 5 years from now.

[JorgeB]

45 minutes ago, TODDLT said:

I generally try to leave the array alone during parity checks and drive replacements.  If nothing else it's painfully slow during that time to access anything, though if urgent, it always works.

 

By static I meant no changes to the array, including for example if you have docker using the array, any changes during the upgrade and parity won't be valid with the old disks, just recently replaced 6 disks on one of my servers, and did it 2 at a time, so it was done in 3 rebuilds instead of 6, it's another advantage of dual parity, and if the array is static there's no increased risk since you can always use the old disks if anything goes wrong.

[garycase]

Agree with your analysis from the earlier post => the "missing element" in my original comment was that I didn't know that you (a) only had 3TB parity drives; (b) still had 2 spare bays; and (c) had over 7TB of free space already.    Given those factors, I absolutely agree with your choice to just bump up to 3TB drives.    [If you already had 4TB (or larger) parity drives, then I'd have leaned towards buying larger drives.]

 

I also agree with Johnnie -- if the array is completely static (NO use at all) during the rebuilds; then there's not really any significant risk doing 2 rebuilds at once, since you're not replacing failed drives.    In the event of another failure, you could simply (a) replace the two original drives; (b) do a New Config with the "parity is already valid" box checked; and then (c) replace the failed drive.   Nevertheless, unless the extra rebuild time is an issue; I'd be inclined to just do 3 rebuild cycles instead of 2 and have the extra layer of protection.

 

[TODDLT]

8 minutes ago, garycase said:

Agree with your analysis from the earlier post => the "missing element" in my original comment was that I didn't know that you (a) only had 3TB parity drives; (b) still had 2 spare bays; and (c) had over 7TB of free space already.    Given those factors, I absolutely agree with your choice to just bump up to 3TB drives.    [If you already had 4TB (or larger) parity drives, then I'd have leaned towards buying larger drives.]

 

I also agree with Johnnie -- if the array is completely static (NO use at all) during the rebuilds; then there's not really any significant risk doing 2 rebuilds at once, since you're not replacing failed drives.    In the event of another failure, you could simply (a) replace the two original drives; (b) do a New Config with the "parity is already valid" box checked; and then (c) replace the failed drive.   Nevertheless, unless the extra rebuild time is an issue; I'd be inclined to just do 3 rebuild cycles instead of 2 and have the extra layer of protection.

 

 

4 hours ago, johnnie.black said:

 

By static I meant no changes to the array, including for example if you have docker using the array, any changes during the upgrade and parity won't be valid with the old disks, just recently replaced 6 disks on one of my servers, and did it 2 at a time, so it was done in 3 rebuilds instead of 6, it's another advantage of dual parity, and if the array is static there's no increased risk since you can always use the old disks if anything goes wrong.

 

I've learned to turn Plex off during parity checks (and would for rebuilds).  In fact, one of my "to do's" is to find out if there is some way to automate that process. IE i dont have to remember to go manually turn off Plex during a parity check.  The check speed is cut in half or worse when Plex is running.  I'm not totally sure what Plex is doing because it's not keeping drives spinning.  However, when Plex is up, the parity check speed is greatly reduced.

 

Anyway, thanks for the advice.  I have the 3rd new drive going through it's 3 pre-clears which should finish sometime in the morning or early afternoon hours tomorrow.  They seem to be taking 12-15 hours each.

 

[SSD]

I have Plex running an an SSD cache disk, and don't notice any speed degradation on parity checks, although Plex does have some maintenance activiites that I suppose could be having a negative effect that I am not noticing:

 

 

[Squid]

37 minutes ago, TODDLT said:

 In fact, one of my "to do's" is to find out if there is some way to automate that process. IE i dont have to remember to go manually turn off Plex during a parity check.  

Your scripts will be (changing Plex for whatever the container is actually called)

#!/bin/bash
docker stop Plex

and

#!/bin/bash
docker start Plex

 

[TODDLT]

2 hours ago, Squid said:

Your scripts will be (changing Plex for whatever the container is actually called)

#!/bin/bash
docker stop Plex

and

#!/bin/bash
docker start Plex

 

OK, so this is taking me outside my area a little.  

 

What does "#!/bin/bash" mean?  where do you put this so that it is executed when the parity check starts and stops?

[Squid]

10 minutes ago, TODDLT said:

OK, so this is taking me outside my area a little.  

 

What does "#!/bin/bash" mean?  where do you put this so that it is executed when the parity check starts and stops?

#!/bin/bash means that what follows below (docker start/stop) is executed by the bash interpreter....  Its the standard lead-in for any script unless the script is written in a different language (such as the linked script which uses #!/usr/bin/php -> it's a PHP script)

 

Create the two files and place them somewhere on the flash drive.  Install the user scripts plugin, and add that script linked above to the plugin (see the OP of the user scripts thread), set it to run at array start.  Change the 2 lines in the script as noted.  You're going to be changing it to something like /boot/plex_start  and  /boot/plex_stop  

 

One big note though is that those two scripts (plex_start and plex_stop) have to be created using Linux Style line endings.  Couple of ways to do that:

 

- Use nano from the command line to create the files (google "nano" for how to use it)

- Use Notepad++ on a windows box (not notepad) and set the EOL conversion to be Unix (LF) - its under the edit menu

- Create the files using Notepad, then install CA Config File Editor plugin, edit the files and switch it from DOS to Linux Style Line endings

- Many many more ways.

 

 

Edited May 16, 2017 by Squid

[TODDLT]

23 hours ago, Squid said:

#!/bin/bash means that what follows below (docker start/stop) is executed by the bash interpreter....  Its the standard lead-in for any script unless the script is written in a different language (such as the linked script which uses #!/usr/bin/php -> it's a PHP script)

 

Create the two files and place them somewhere on the flash drive.  Install the user scripts plugin, and add that script linked above to the plugin (see the OP of the user scripts thread), set it to run at array start.  Change the 2 lines in the script as noted.  You're going to be changing it to something like /boot/plex_start  and  /boot/plex_stop  

 

One big note though is that those two scripts (plex_start and plex_stop) have to be created using Linux Style line endings.  Couple of ways to do that:

 

- Use nano from the command line to create the files (google "nano" for how to use it)

- Use Notepad++ on a windows box (not notepad) and set the EOL conversion to be Unix (LF) - its under the edit menu

- Create the files using Notepad, then install CA Config File Editor plugin, edit the files and switch it from DOS to Linux Style Line endings

- Many many more ways.

 

 

Thanks for the help with this.  Some new territory for me here, but i'll do some digging on my own as to what Linux Style line endings are and I'm sure the steps you have mapped out will then make sense.

 

One piece I seem to be missing here though, is what connects the two (plex_start and plex_stop) to the automated process of running a parity check?  i'm trying to see Plex automatically turn off when a parity check starts, and then restart when the parity check is complete.

 

Also do these lines wind up as command lines in the "go" script? or do they land elsewhere in the unRAID command system?

Edited May 17, 2017 by TODDLT

[garycase]

15 minutes ago, TODDLT said:

...  as to what Linux Style line endings are and ..

 

Just means that lines end with a LF (line-feed) character instead of a CR-LF (carriage return followed by a line feed)

 

The easiest way to do that if you're using a Windows box to create the script files is to use the free Notepad++

[ https://notepad-plus-plus.org/ ]

 

 

15 minutes ago, TODDLT said:

One piece I seem to be missing here though, is what connects the two (plex_start and plex_stop) to the automated process of running a parity check?  i'm trying to see Plex automatically turn off when a parity check starts, and then restart when the parity check is complete.

 

You simply have to create your "start script" and "stop script"  (what you want to happen when a parity check starts and stops); and then put pointers to those in the script file Squid wrote:

Squid can provide the details, but if I understand it correctly, you need to have his script run at array start (i.e. in the Go file) -- and then when it detects a parity check starting or stopping it will invoke the script you've provided to do whatever actions you want (e.g. start or stop Plex).