Methods to improve write operations to UnRAID

[Lev]

This idea requires more than just me, so I ask please to brainstorm with me in this thread as the sum of our knowledge may find a viable method to make this work with what little bits I know now to seed this idea.

 

UnRAID by design, is write I/O bound to a single or dual parity disk array. In this thread we'll explore ways to go beyond that using some creative approaches. The addition of a cache drive was a creative approach added years ago. Are there more? Let's collaborate and see what we find.

 

The hope is that even if we don't find a 100% solution, perhaps what's shared in this thread over the months and years ahead may give a spark to those much more intimate with the code, and maybe they'll see inspiration in something we mention here.

 

 

[Lev]

Here's a seed idea to brainstorm from, maybe it'll spawn something better, maybe it'll lead to something entirely different. 

 

The simplest and most common focus of improvements in write performance always focuses on single direct array writes, in which there's two major options:

1. Defer cache writes through use of a fast cache disk
2. Turbo Write mode, which is bound by the slowest write speed of either the parity or disk being written too.
    

Parallel discrete write operations: Allows two or more writes to occur on unique physical disks.

 

Most common use case:

    Source A  -> UnRAID UserShare A (parity protected)    This has the options and limits of those options above (cache disk and turbo write). It's also complicated because of the parity requirement involved.

 

This next example is easier to explain if we remove the parity requirement to start with, we will layer it back in later. Imagine an array that has no parity protection, it's an unprotected array, such that any disk can be written to at full speed.

 

    Source A  -> UnRAID UserShare A  (not parity protected, no disks in ShareB "included")

    Source B  -> UnRAID UserShare B  (not parity protected, no disks in ShareA "included")

 

That unfortunately is not a very common use case to have two sources to two shares. Let's take look at one source, same target in two parallel transfers.

 

    Source A  -> UnRAID UserShare A  (not parity protected, )

    Source A  -> UnRAID UserShare A  (not parity protected, )

 

This takes a big performance hit as both writes will go to the same physical disk as whichever physical disk that's selected by the user share according to 'Allocation Method' usually the lowest number disk first that qualifies for selection. If an enhancement to File Allocation is added, that checks to see if a write activity is underway, then rather than taking the lowest number disk it should us, it will increment the disk number by one to the second most viable disk according to its allocation check. That would look like this...

 

    Source A  -> UnRAID UserShare A  (not parity protected, High Water[diskX])

    Source A  -> UnRAID UserShare A  (not parity protected, High Water[diskX+1])

 

I've tested that above example doing it manually, its rather easy to prove. Depending how many disks you have in the UserShare, [diskX+N] is really the parallel limit.

 

Any questions or flaws in what I wrote out so far ? I'll attempt to solve adding parity back into this scenario, but there's not point doing that if what's written so far doesn't achieve Parallel Writes.

Edited October 14, 2017 by Lev
Added screenshot

[Lev]

reserving for future use. 

[BRiT]

Turbo Write Mode.

[BRiT]

For those unawares:

 

 

[Lev]

@BRiT I'm aware. But hang in there with me, lets go one or two levels deeper.

 

I should of pre-wrote the seed idea before posting to show that this thread is only for those on this forum who are experienced.

[1812]

Raid0 parity drives & done. 

 

Or raid10 if you want to double bag it "for safety."

[Lev]

5 minutes ago, 1812 said:

Raid0 parity drives & done. 

 

Or raid10 if you want to double bag it "for safety."

 

Is it working for you well? I've often considered that approach!

 

 

Edited August 26, 2017 by Lev

[Lev]

5 minutes ago, 1812 said:

Raid0 parity drives & done. 

 

Or raid10 if you want to double bag it "for safety."

 

Let me know cause I'll pull the trigger at BestBuy this weekend and run Raid10 across 4x8TB WD Reds

[1812]

1 minute ago, Lev said:

 

Is it working for you well? I've often considered that approach!

 

 

 

I don't do that but on my main server but would if I wanted faster writes to the array. I don't have a problem on the storage server with typical wrote speeds in unRaid.

 

However, I do have one unraid setup with an unprotected setup in which there is a single disk image in the array using 20 disks in a raid10 and a cache disk using 4 drives in a raid10 as well. This is for mainly virtualization use.

 

there are a couple folks on here who do use raided parity though. I think @johnnie.black does?

[tr0910]

I used raided parity on old 2tb drives on a 4tb disk array. Main reason was to use the old drives for something and rather than have 4tb parity with 2tb disks, I created an 4tb parity using 2x2tb raid0 attached to my Areca 1260 / 1280 controllers.

Speed was very good, but that wasn't my main reason for doing it. I could see a 2x8tb parity raid0 along with 2x8tb raid0 disk1, disk2, disk3 etc. Might need a newer generation Areca to be able to handle the throughput.

What is you use case where you are needing better write speed?

Sent from my Nexus 6 using Tapatalk

[Lev]

55 minutes ago, tr0910 said:

What is you use case where you are needing better write speed?

 

It's a weak case perhaps but I love to tinker. I now have two UnRAID Pro Servers, 14+ disks each, one primary, one for backup.

 

On the backup server, since it's new for me, never had a backup before! I tried to use split level and am in the progress of migrating a bunch of data from individual disks through the UnRAID user shares to force split level. Even though I'm not using parity a disk, its an unprotected array, it's time consuming because I can only write to one disk at a time.

 

If I could migrate for example two source disks (1st disk Movies, 2nd disk TV) at the same time each to a user share that is not overlapping on physical disks with each other using, then parallel disk writes would be achieved. I think I can manually force this to occur through modifying the excluded/included disks of the share, so it's led me to some more theories on how to achieve this even if I had parity enabled.

 

Perhaps in this there is a method for the future, when I want to push a backup from my primary server to my backup server, of both Movies and TV in parallel and achieve dual writes in the process.

 

I have an idea of how this would work achieving parallel write speeds even with there being a parity drive (thus overcoming the limitation imposed now by a parity disk) I just need more time to write it and build diagrams.

Edited August 26, 2017 by Lev

[JorgeB]

3 hours ago, 1812 said:

I think @johnnie.black does?

 

Nope, I believe @bjp999 uses one.

[SSD]

4 hours ago, johnnie.black said:

 

Nope, I believe @bjp999 uses one.

 

Yes, I have a RAID0 parity made up of 2 4T 7200 RPM HGST drives. If it's quite fast across the 8T spectrum, maxing at 330 MB/sec and averaging over 260. It is never the bottleneck on single writes, and with 8T 7200 RPM data drives, I'll often get 75+MB/sec writes without turbo write mode.

 

It's also quite nice to be able to upsize parity by using 2 smaller disks. If tomorrow there were a great deal on 10TB drives, I could take 2 existing 5T 7200 RPM drives, form a 10TB volume, use it for parity, and add the new 10TB drive as a data disk. I am able to take full advantage of the 10TB disk from day one without having to buy two. I make sure two buy 2 identical fast drives of each size to ultimately use in a RAID pair. I have a pair of 5T, 6T, and 8T in my array waiting for their turns to convert to parity as drives sizes continue to grow.

 

I use an Areca ARC-1280ML controller. It is somewhat overkill in terms of port count. The ARC-1231ML or ARC-1261ML would work fine. (The only difference between the cards is the number of SAS connectors.) Only problem is these cards are pretty wide, and the SAS plugs are on the far edge, making plugging in the SAS cables a challenge depending on the case depth. The 1280 has 6 SAS connectors including 2 on the top of the card. These 2 are very convenient, and they are the only ones I use.

 

These are all PCIe 1.x cards, so bandwidth is constrained compared with 2.0'cards. But in an x8 slot you'd be able to get 7 or 8 full speed drives.

 

The cards have a JBOD mode, which you can't use if you have even one RAID array defined. But the card does allow individual drives to be passed through, and they look and act like JBOD disks. There is a bit of configuration to set this up on each disk. If you need to rebuild a failed passthrough disk, you'd have to configure the new drive to the controller before you could use it.

 

Some of these Areca cards have a LAN port and custom Web interface where you can do such configurations and monitor the controller. It also allows very easy firmware updates.

 

Couple other comments. You can pull smart reports, but is a little tricky. I still use a customized myMain version that makes it seamless, but I don't know how well the stock GUI handles them. There is a program to include in the go file, by @bubbaQ, so that the drive names are reflected properly. And the card does its own spindown monitoring, with 1 hour max. It ignores the stock GUI attempts to spin disks up and down.

 

More information is contained in this thread ...

 

 

[bonienl]

On 8/26/2017 at 1:12 PM, bjp999 said:

Couple other comments. You can pull smart reports, but is a little tricky. I still use a customized myMain version that makes it seamless, but I don't know how well the stock GUI handles them. There is a program to include in the go file, by @bubbaQ, so that the drive names are reflected properly.

 

With unRAID6 GUI it is possible to set individual controller and device settings per disk to allow proper reading of SMART reports. See SMART settings of each disk.

 

The 60-persistent-storage.rules file of bubbaQ is outdated and doesn't work with the latest unRAID versions, better use the Dynamix SCSI Devices plugin, which inserts the necessary rules and keeps the rules file up-to-date.

 

[SSD]

6 hours ago, bonienl said:

 

With unRAID6 GUI it is possible to set individual controller and device settings per disk to allow proper reading of SMART reports. See SMART settings of each disk.

 

The 60-persistent-storage.rules file of bubbaQ is outdated and doesn't work with the latest unRAID versions, better use the Dynamix SCSI Devices plugin, which inserts the necessary rules and keeps the rules file up-to-date.

 

 

Thanks for the tip! I have been using the bubbaQ script continuously including the unRAID latest version (well, up through RC6 anyway - haven't moved to RC7a) and not had any problems. But will have to look to migrate to the plugin to see what changed.

[Lev]

I'm going to try the RAID parity disk approach. Many questions though on what the best configuration is.

 

Let's assume I've got 4 disks possible to use. Which RAID type is the best?

- Two UnRAID Parity Disks: Two RAID0 Arrays each using 2 disks

- Single UnRAID Parity Disk: One RAID10 Array using 4 disks

 

If there were 6 or 8 disks possible to use, would that change the answer to the previous question?

 

What should the strip size of the array be?

 

Will there be any significant advantage to using a RAID adapter (LSI 2008 in IR mode) vs using the motherboard bios? I always assumed that dedicated PCIe RAID Adapters are usually only helpful to off-load parity calculations when using RAID5 or others that have parity. Since the choices are between striping (RAID0) or striping and mirroring (RAID10), does the choice matter?

[SSD]

On 9/2/2017 at 6:47 AM, bonienl said:

 

With unRAID6 GUI it is possible to set individual controller and device settings per disk to allow proper reading of SMART reports. See SMART settings of each disk.

 

The 60-persistent-storage.rules file of bubbaQ is outdated and doesn't work with the latest unRAID versions, better use the Dynamix SCSI Devices plugin, which inserts the necessary rules and keeps the rules file up-to-date.

 

 

Can't really advise. I did the Areca because @WeeboTech was doing it and I knew it worked.

 

I am not sure the LSI RAID cards are supported.

 

I ran a ARC-1200 for a long time. PCIe 1, x1. Hit bandwidth constriction at 208 MB/sec, well slower than capable. Going to ARC-1280ML is a PCIe 1, x4 slot (it is an x8 card but I only had an x4 slot for it) has made a huge difference. It peaks in the mid 300s with a pair of 7200 RPM HGST in RAID0. TBH, not sure going much beyond that helps very much as the other disk is the bottleneck. With parallel writes the faster parity would show more differentiation.

[Lev]

Thanks @bjp999, I'll look into those.

 

I just tried my both my SuperMicro's on board RAID options, Intel and Adaptec, defining an array in either was not recognized by unRAID.

 

I'm also planning on using the same 7200 RPM HGST's so glad to hear the performance you got, it's in line with what I was expecting it'd be.

[SSD]

I had forgotten but there was another RAID card some users were experimenting with that DID work, but it did this truncation and there was no way to disable it. The Areca has a SETTING that you need to make sure is set to no truncation.

 

RAID cards can truncate the drive sizes to the nearest 1G, or 10G (presumably from a time when all manufacturers drives of the same advertised size were not exactly the same size). While adding the full capacities of 2 4T drives together results in a drive slightly larger than an 8T drive, with truncation that slight surplus turns into a slight deficit. And we all know that parity has to be the equal size or larger than others in the array. Even a tiny bit small is too small.

 

Look here:

 

 

[Lev]

Thanks for the warning on size. I'll be using two 6TB 7200 HGST drives with the next largest 8TB data drives. I'll try it on my backup server first. If successfully I'll consider doing it on my primary with 4 drives, two arrays for two parity disks.

 

Other than searching the forums or trying to ask someone more in the know (many of those is think are in the know already commented on the thread) perhaps there is a way to query the kernel and see what drivers may be built in for compatible raid cards?

[Lev]

Looks like the Areca ARC-1883i is the modern equivalent of the card you used. unRAID support unknown.

 

Pricey if purchased new, so I may have to take time to watch for a deal.

[SSD]

Here is an ARC-1231ML on eBay for $136. Not too bad.

 

http://www.ebay.com/itm/Areca-ARC-1231ML-2G-PCI-Express-SATA-II-3-0Gb-s-Controller-Card-w-Cooling-Fan-/322696649933?hash=item4b22381ccd:g:Sw0AAOSwnONZBNri

[Lev]

12 minutes ago, bjp999 said:

Here is an ARC-1231ML on eBay for $136. Not too bad.

 

http://www.ebay.com/itm/Areca-ARC-1231ML-2G-PCI-Express-SATA-II-3-0Gb-s-Controller-Card-w-Cooling-Fan-/322696649933?hash=item4b22381ccd:g:Sw0AAOSwnONZBNri

 

Wow thanks for that. Appreciate you looking as I hadn't seen anything in that price range.

 

Unfortunately I'm limited by being a 36bay 4U SM chasis, and that looks like a full height card. Only half height I can get.

 

Funny thing when I decided to go with 36 bays, I thought I'd be constrained first by hard drives than the need for add-on PCIe cards, guess I was wrong! Still only 20 of 36 drives. Still the desire for higher write speeds, is simply a nice to have, capacity I'll is much more of a need.

 

Thanks and let me know if you spot anything else!

Edited September 7, 2017 by Lev

[SSD]

The ARC-1200 is a half-height card - but is only a 2 port PCIe1.1 x1 card. I used one for years and worked great, but the max bandwidth I got was 208 MB/sec with a single RAID0 parity but it provides that same speed for well over half of the capacity. On the inner (slower) cylinders it is vastly faster than other drives. But on the outer (faster) cylinders, this can be slightly slower than a fast 8T drive.