dual parity and dual cache......why?

[French-Guy]

Hello,

 

I built almost 10 years ago an Unraid server (4.7). Very basic actually (Sempron LE 140 and cheap mATX board)

I've since then upgraded to V5 (not too long ago actually...!!!)

And I'm now thinking to upgrade to V6

But I saw that Unraid was able to use 2 parity drives....Why is it for?

Same question about multiple cache disks: Is having more than one just to get larger capacity?

 

Thanks

 

[trurl]

Hello,

 

I built almost 10 years ago an Unraid server (4.7). Very basic actually (Sempron LE 140 and cheap mATX board)

I've since then upgraded to V5 (not too long ago actually...!!!)

And I'm now thinking to upgrade to V6

But I saw that Unraid was able to use 2 parity drives....Why is it for?

Same question about multiple cache disks: Is having more than one just to get larger capacity?

 

Thanks

More data drives benefit from more parity drives. If you have 2 parity drives you can recover from 2 failed drives. For example, if you are rebuilding a failed drive and another drive fails, you still haven't lost anything.

 

While it is possible to configure multiple cache disks for more capacity, the main use is to have redundancy in the cache, so if one cache fails you still haven't lost anything from cache.

 

If you are still using your "very basic" hardware you may not have enough performance for some V6 features, including dual parity.

[Frank1940]

If you look at my setups, you will see that it has very similar specs to your setup.  The semipron is definitely marginal for dual parity.  My parity check time went for about 8 hours for single parity to 11 hours for dual parity.  The write to array speeds would be impacted in a similar fashion.  (Although, there is a lot of other factors involved that also impact the writing of files to the array besides the actual write speeds.  For example, if you are writing a lot of small files, the file allocation process will make a major impact and dynamically increase the time required to complete the transfer.)

 

When you decide to move from ver 5 to ver 6, I suggest that you read these:

 

      http://lime-technology.com/forum/index.php?topic=39032.0

and

      http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6

 

The first one is the quick guide and the second is very comprehensive and covers virtually every situation. 

 

By the way, from a practical standpoint, I personally consider single parity satisfactory for arrays with less than 8 data drives and a necessity for arrays with more than 12 data drives.  Between those two points, the user's comfort level can be the deciding factor.  But that assumes you run monthly parity checks and have the notification setup activated so you are notified of any problems that come up!  See here for a discusion about single vs dual parity.

 

        http://lime-technology.com/forum/index.php?topic=52229.msg501203#msg501203

[French-Guy]

If you look at my setups, you will see that it has very similar specs to your setup.  The semipron is definitely marginal for dual parity.  My parity check time went for about 8 hours for single parity to 11 hours for dual parity.  The write to array speeds would be impacted in a similar fashion.  (Although, there is a lot of other factors involved that also impact the writing of files to the array besides the actual write speeds.  For example, if you are writing a lot of small files, the file allocation process will make a major impact and dynamically increase the time required to complete the transfer.)

 

When you decide to move from ver 5 to ver 6, I suggest that you read these:

 

      http://lime-technology.com/forum/index.php?topic=39032.0

and

      http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6

 

The first one is the quick guide and the second is very comprehensive and covers virtually every situation. 

 

By the way, from a practical standpoint, I personally consider single parity satisfactory for arrays with less than 8 data drives and a necessity for arrays with more than 12 data drives.  Between those two points, the user's comfort level can be the deciding factor.  But that assumes you run monthly parity checks and have the notification setup activated so you are notified of any problems that come up!  See here for a discusion about single vs dual parity.

 

        http://lime-technology.com/forum/index.php?topic=52229.msg501203#msg501203

Hi Frank

I've already read multiple times your guide to upgrade, and even printed it out...Very useful, thanks

I've just purchased a WD My Book 4TB ($99 at Sam's Club) - Will have to open the case, see what drive is inside and if everything goes well, I will use it to upgrade my parity drive. I might get another one (price is good I think), but at the end I don't think I will go with more than 6 drives.....So looks like I won't probably need the dual parity

I'm looking at a new build, using the iStarUSA D-214 2U rack. Would like to get a mini ITX board, and a CPU with more power for MAYBE doing some plex transcoding.

Ideally, I would like to find the best compromise CPU power / power consumption/price

 

[garycase]

Agree with Frank that the risk of only using single parity goes up as the # of drives increases.

 

However, the benefit of dual parity is the same regardless of how many data drives you have => it still provides a 2nd layer of fault tolerance that will allow a disk rebuild to successfully complete even if a 2nd drive fails during the rebuild.  This is true whether you have 2 data drives or 20 data drives.  Clearly the likelihood of a 2nd failure increases as the number of drives increases, but it CAN happen with any number of drives.

 

Another factor I'd consider is whether or not you're adequately backed up.  Remember that RAID -- no matter how fault-tolerant -- is NOT a substituted for backups.  If you have current backups, then if a drive fails, and you can't successfully rebuild it due to a 2nd drive failure during the process; then you can always just copy all of the data that was on that drive back to your array from your backups.    If you don't have backups, that 2nd failure is clearly a FAR more consequential event -- a good reason to have a 2nd parity drive to protect against it, regardless of the number of data drives in the array.

 

 

 

[trurl]

I've just purchased a WD My Book 4TB ($99 at Sam's Club) - Will have to open the case, see what drive is inside and if everything goes well, I will use it to upgrade my parity drive.

There have been some reports of drives harvested from external enclosures having slightly less capacity than a normal internal drive of the same advertised capacity. So if you intend to use this as parity make sure it has the standard 4TB capacity or you may not be able to use standard 4TB data drives with it.

[French-Guy]

I've just purchased a WD My Book 4TB ($99 at Sam's Club) - Will have to open the case, see what drive is inside and if everything goes well, I will use it to upgrade my parity drive.

There have been some reports of drives harvested from external enclosures having slightly less capacity than a normal internal drive of the same advertised capacity. So if you intend to use this as parity make sure it has the standard 4TB capacity or you may not be able to use standard 4TB data drives with it.

Is there a way to check the available size without taking it out of the case?

[John_M]

Plug it into a USB port and see if you can get a SMART report from it via the Dashboard page of the webGUI. If you can't you might try via a console or telnet/ssh session:

 

smartctl -a -d sat /dev/sdX

 

where X is the actual letter allocated to it. See here: https://www.smartmontools.org/wiki/USB

 

[French-Guy]

OK, with the WD My Book plugged into a USB port, I can see it in the "Non protected array"

But I see 2 lines actualy:

Drive Partitions - Not In Protected Array

Device Model/Serial Mounted File System Temp Size Used %Used Free

/dev/sdg usb-WD_My_Book_1230_574343344532584431445344-0:0 * 4.00T

/dev/sdg1 usb-WD_My_Book_1230_574343344532584431445344-0:0 *

 

When I try to get a smart report from MyMain, I get this:

smartctl -a -d ata /dev/sdg (--)

smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build)

Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

 

Read Device Identity failed: Invalid argument

 

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

 

And when I try with PuttyTel, I type the command but then the PuttylTel screen disappear and then nothing!

[French-Guy]

In Disk Management, I get this (see attached)

[John_M]

That will be your capacity, then: 3,906,983,936 blocks, so multiply by 1024 to get the total number of bytes: 4,000,751,550,464 bytes. Now compare that with a normal internal 4 TB disk. I don't have any, I'm afraid.

 

Whether or not smartctl works across a USB to SATA bridge depends on the chip used, but I think you might have more chance of success with the

-d sat

option than the

-d ata

option, as suggested by the smarttools wiki. Note that WD uses encrypting bridge chips so if you pre-clear a MyBook disk in its case, while being a good test of its overall health, it won't result in a properly pre-cleared disk once you shuck it and connect it to a SATA port. If you do pre-clear it in its case try not to let it get too hot, especially if you can't monitor the temperature with SMART. The ventilation is poor and not designed for hours of continuous reading and writing. An external fan might be useful.

 

EDIT: The capacity of a typical 4 TB disk (Seagate ST4000DM000) seems to be

 

User Capacity:    4,000,787,030,016 bytes [4.00 TB]

 

extracted from a SMART report from someone's diagnostics. So yours looks a little bit smaller and wouldn't be suitable for use as a parity disk.

 

 

[French-Guy]

OK, thanks...

So might be a good option as a data drive though?

[garycase]

The standard byte count for a 4TB drive is 4,000,787,030,016

 

[garycase]

Seems John edited his post to show the actual number while I was posting.

 

But yes, you can certainly use the drive as a data drive.

 

[garycase]

You might also be able to restore the max capacity by removing what is likely an HPA using HDAT2 ( http://www.hdat2.com/ ), unless the extra space is hidden via some other mechanism.

 

There's also a "Linux way" to remove the HPA, but I don't recall the specifics -- someone else may post the details.

 

[John_M]

There's one discussion here: http://lime-technology.com/forum/index.php?topic=7042.0

 

[garycase]

There's one discussion here: http://lime-technology.com/forum/index.php?topic=7042.0

 

That shows both the "Linux way" and how to do it with HDAT2.  Personally, I find the simple "SetMax" command in HDAT2 the easiest method to use, but you can do whichever you find more comfortable.

 

[bombz]

Hey everyone, I just recently upgraded to the latest 'official' UnRAID and I saw it had the new added GUI from the local server (nice touch team) and with that DUAL PARITY! I didn't even know about it but it seems really awesome.

I read over what garycase had said about this a bit. My server is my 'baby' I don't have a backup of it, as that would get expensive to backup or mirror a duplicate 25ish TB of data. This dual parity is intriguing to me as it seems to have EXTRA protection if more then one drive fails.

 

As if now I have a 6TB parity disk, so with that, if you decide to run dual parity, do the parity disks HAVE to MATCH in size 2x6TB in this case, as general rule is parity has the be the biggest size in the array.

I think I want to use this feature for extra protection... any pointers? Should I be doing the same process? (Pre-clear the disk + add to array + run parity check) and I am done. I want to make sure I am understanding this correctly.

 

Any pointers , suggestions, recommendation are welcome!

 

Thanks team

[JorgeB]

As if now I have a 6TB parity disk, so with that, if you decide to run dual parity, do the parity disks HAVE to MATCH in size 2x6TB in this case, as general rule is parity has the be the biggest size in the array.

 

Rules for parity2 are the same as parity, it needs to be as large or larger than your data disks, current parity size doesn't matter.

[itimpi]

Hey everyone, I just recently upgraded to the latest 'official' UnRAID and I saw it had the new added GUI from the local server (nice touch team) and with that DUAL PARITY! I didn't even know about it but it seems really awesome.

I read over what garycase had said about this a bit. My server is my 'baby' I don't have a backup of it, as that would get expensive to backup or mirror a duplicate 25ish TB of data. This dual parity is intriguing to me as it seems to have EXTRA protection if more then one drive fails.

As has been repeated many times, unRAID is not a backup - it just provides tolerance against disk failure.  Irreplaceable files should still be backed up elsewhere - preferably at several locate/ns if they are really important.

 

As if now I have a 6TB parity disk, so with that, if you decide to run dual parity, do the parity disks HAVE to MATCH in size 2x6TB in this case, as general rule is parity has the be the biggest size in the array.

I think I want to use this feature for extra protection... any pointers? Should I be doing the same process? (Pre-clear the disk + add to array + run parity check) and I am done. I want to make sure I am understanding this correctly.

When using dual parity it is not a requirement that both parity drives are the same size - just that they equal or exceed the size of the largest data drive.

[bombz]

Hey everyone, I just recently upgraded to the latest 'official' UnRAID and I saw it had the new added GUI from the local server (nice touch team) and with that DUAL PARITY! I didn't even know about it but it seems really awesome.

I read over what garycase had said about this a bit. My server is my 'baby' I don't have a backup of it, as that would get expensive to backup or mirror a duplicate 25ish TB of data. This dual parity is intriguing to me as it seems to have EXTRA protection if more then one drive fails.

As has been repeated many times, unRAID is not a backup - it just provides tolerance against disk failure.  Irreplaceable files should still be backed up elsewhere - preferably at several locate/ns if they are really important.

 

As if now I have a 6TB parity disk, so with that, if you decide to run dual parity, do the parity disks HAVE to MATCH in size 2x6TB in this case, as general rule is parity has the be the biggest size in the array.

I think I want to use this feature for extra protection... any pointers? Should I be doing the same process? (Pre-clear the disk + add to array + run parity check) and I am done. I want to make sure I am understanding this correctly.

When using dual parity it is not a requirement that both parity drives are the same size - just that they equal or exceed the size of the largest data drive.

 

Much appreciated. Sorry I am behind the times with lack of information on some things. I always like to be sure, as unRAID is really important to me ! Thank you for the update

[bombz]

As if now I have a 6TB parity disk, so with that, if you decide to run dual parity, do the parity disks HAVE to MATCH in size 2x6TB in this case, as general rule is parity has the be the biggest size in the array.

 

Rules for parity2 are the same as parity, it needs to be as large or larger than your data disks, current parity size doesn't matter.

 

Thank you as well!

[dchamb]

For clarification, is the second parity drive a mirror of the first parity drive? 

 

I can see how it protects the array if the first parity drive fails, or if a data drive fails during a parity drive rebuild. But if 2 data drives fail at the same time, you lose data right?

[JorgeB]

3 minutes ago, dchamb said:

is the second parity drive a mirror of the first parity drive? 

No

 

3 minutes ago, dchamb said:

But if 2 data drives fail at the same time, you lose data right?

Dual parity protects against two simultaneous data disk failures.

[Frank1940]

14 minutes ago, dchamb said:

For clarification, is the second parity drive a mirror of the first parity drive? 

 

No.  Parity 1 is a very simple binary operation.  Parity 2 is a complex matrix operation and is very CPU intensive UNLESS you have a newer CPU which has an new operator which reduces the number of CPU cycles to do the calculations.

 

14 minutes ago, dchamb said:

But if 2 data drives fail at the same time, you lose data right?

 

You can ANY two drives fail and not lose data.  That includes both parity and data drives.  So you can have two data drives fail, a data and parity drive, or a two parity drives fail without data loss.  But if a third drive fails with any of these combinations, you will be losing data! 

Edited December 5, 2017 by Frank1940
left out a word