[Solved] Speed Up Cache To Disk Write

[brando56894]

I'm going to assume the answer is "you're SOL" but I'll ask anyway...I had been using ZFS for about 3 years and the massive drop in performance is killing me. I used FreeNAS on and off during those 3 years but their GUI for 9.x/11 is really lacking in my opinion, Corral's GUI was awesome but the OS itself was a buggy mess. They're in the process of building the new GUI and adding back in Docker support but progress is slow. I decided to ditch ZFS (and my striped mirrored pool) and give unRAID a try since I came to the conclusion that I didn't need the extreme performance of RAID 10 and the data integrity it provided and would rather have the raw capacity. I've been using it for a month and I've been happy with everything...except for the sluggish performance I'm getting out of my array. 

 

I checked out unRAID a while ago after I saw one of Linus Sebastian's unRAID project videos and thought i'd give it a try but was completely put off by the single drive performance. Fast forward to now and I have everything setup, but my array can't keep with the rate at which I'm downloading movies and shows since I have a gigabit connection. Since I'm currently down 2 SATA ports (my HBA bit the dust) I have 5x 4 TB HDDs and 1x 512 GB SATA SSD and to my amazement my cache drive will consistently fill up before "The Mover" can push the data out to one of my drives ( I have it set to run every hour) . I'm using unRAID v6.4-RC7 which is supposed to have a "significantly faster mover" but that's not the case in my experience. About 24 hours ago I told Radarr to search for 80 movies in either 1080P or 4K and it found about 50 of them, ranging in size from 10 GB to 60 GB. I have cache=yes set on my downloads share, as well as movies and tv shows, along with cache-only set on my webserver VM, Windows VM, and system share. Within about an hour or two the cache drive is 100% full, all my containers and VMs are freaking out since there is no space, I can't stop anything since there is 0 space left on the drive, attempting to shut Docker down completely just leads to a GUI freeze, which eventually leads to me killing Radarr/Sonarr/Deluge/NZBget via htop and wait for them to die, then manually move files off of the cache via rsync, and wait for everything to become responsive again, which sometimes it doesn't which forces me to do a hard shutdown. At a few points there, IDK if the mover got confused or what, but it looked like it was moving data back into the cache almost as quickly as I was moving stuff off of it...and it wasn't stuff that I was currently downloading or post-processing (for example it had the whole series of South Park on there for some reason)  I don't know if this is a result of cache=yes, so I changed it to cache=prefer and cleared out all everything in downloads, movies and shows (the majority of the data, the other shares equal maybe 50 GB total) and let it continue to download things for a few hours. I checked on it hours later and once again the cache drive was 100% full....

 

Right now my only solution is to disable caching on the downloads, movies and shows shares, which sucks because those are the most active shares and would definitely benefit from the write cache. I just had Radarr post-process 11 HD/UHD movies and it took 2 freaking hours to do so!

 

I bought the Pro edition two days ago because my trial was up and I'm going to assume this won't be a huge problem that much longer because this was mostly caused by backfilling my server, but if it can't handle downloading and quickly post-processing/moving about 3-5 UHD movies which I'll be downloading less frequently than I am now I may have to go back to ZFS once the new UI is feature complete. I really like the looks and functionality of unRAID, it's just that the actual performance of it sucks in my opinion, or at least compared to what I was used to for years. I'm half tempted to give the ZFS plugin a try since i'm comfortable managing it via the CLI but I would rather have a GUI for it, and don't have any large drives to spare at the moment and I'm definitely not erasing 13 TB worth of data again.

Edited August 18, 2017 by brando56894

[JorgeB]

Since unRAID doesn't stripe disks it will never have the same performance as solutions that do it, it's also the reason for most unRAID advantages over them, but did you try Turbo write? For your current config it should keep up with gigabit writes without any issues.

 

 

[brando56894]

Yep, I turned that on a week or two ago :-/ These are all NAS quality drives (3 are HGST NAS drives and one is a WD Red) so it's not like they're not up to the task. I also enabled NCQ, which should theoretically help a bit, right?

[JorgeB]

I have no problem getting over gigabit speed with turbo write enable even on arrays with more disks, when writing to the beginning of a disk I can get close to 200MB/s sustained using 10GbE, e.g., this is from an 8 disk server with mostly Seagate 8TB archive drives

[SSD]

Confirm that your workflow is configured properly with Radarr. I don't use it, but you want your DL to go to the cache disk, your repair to happen in the cache disk, and then the unrar to go to the array.

 

Make sure your SSD is properly trimmed.

 

It would also be interesting to know where in the process the bottleneck is happening. Par check/repair performance could be bogging you down. There are multi-threaded and non-multithreaded versions, and you want the multithreaded. Some DLers include the single threaded one.

 

The array I/0 is going to be about 50-60 MB/sec with normal unRaid writes while you are DLing at 100. With turbo write, and assuming you are using 7200 drives (which you are not), the unRaid performance should be faster than that. But overall system performance may not be able to keep up.

 

The other thought, which you've probably already thought about, and you'll hate, is throttling back the download speed to find the point that your system can keep up.

 

In the positive side, what a good problem to have!

 

One other comment - continuous DLs at full gigabit = 6GB a minute, so a TB would take something under 3 hours. You'd fill 2 4 TB drives in a day. That is not sustainable. Slowing things down may not be such a problem. And would leave some bandwidth for other uses! 

 

Did ZFS handle this at full speed?

[brando56894]

6 minutes ago, johnnie.black said:

I have no problem getting over gigabit speed with turbo write enable even on arrays with more disks, when writing to the beginning of a disk I can get close to 200MB/s sustained using 10GbE, e.g., this is from an 8 disk server with mostly Seagate 8TB archive drives

 

I would post a similar one, but I'm at work and even though I do have my laptop with my I'm being connected through a VPN would skew the transfer rate. I'll post one up when I get home in like two hours. Isn't that hitting your cache drive and not the disks themselves? I have no problem with SMB transfers that hit the cache first, my issue is from the cache to disks.

 

Confirm that your workflow is configured properly with Radarr. I don't use it, but you want your DL to go to the cache disk, your repair to happen in the cache disk, and then the unrar to go to the array.

 

Yep, I have my folders setup like so: downloads -> two foldersL torrents and usenet -> incomplete and complete as subfolders of each of the parent folders -> category

 

This worked fine for me with ZFS, but I've been considering making torrents and usenet separate shares because the torrents don't need to be cached (and they're what is causing most of the problems since I have to seed for a while since it's a private tracker :-/ ) and go directly to the disk, whereas Usenet downloads would definitely benefit from being on the cache drive initially.

 

21 minutes ago, bjp999 said:

Make sure your SSD is properly trimmed. It would also be interesting to know where in the process the bottleneck is happening. Par check/repair performance could be bogging you down. There are multi-threaded and non-multithreaded versions, and you want the multithreaded. Some DLers include the single threaded one.

 

yep after it was 100% full I manually trimmed it, didn't really change much because the issue/bottleneck is moving the data to the mechanical drives not the SSD.

 

24 minutes ago, bjp999 said:

The array I/0 is going to be about 50-60 MB/sec with normal unRaid writes while you are DLing at 100. With turbo write, and assuming you are using 7200 drives (which you are not), the unRaid performance should be faster than that. But overall system performance may not be able to keep up.

 

I have turbowrite enabled (drives are spinning 100% of the time) and 3 out of the 4 drives (the HGST drives) are definitely 7200 RPM because I just looked it up. The only other bottleneck is my CPU when trying to extract archives that are tens of gigabytes (20-30 GB+), and I would assume this slows down the parity calculations a bit but not enough to make it detrimental. 

 

27 minutes ago, bjp999 said:

The other thought, which you've probably already thought about, and you'll hate, is throttling back the download speed to find the point that your system can keep up.

 

Definitely not doing that, because what's the point of paying for gigabit internet if I can't fully utilize it? I'd much rather ditch unRAID than drop back down to 150 Mbps.

 

29 minutes ago, bjp999 said:

One other comment - continuous DLs at full gigabit = 6GB a minute, so a TB would take something under 3 hours. You'd fill 2 4 TB drives in a day. That is not sustainable. Slowing things down may not be such a problem. And would leave some bandwidth for other uses!

 

It's "up to 1 Gbps" so I'm obviously not getting a full continuous gigabit per second rate, it's not even "true" gigabit since the download is capped at like 980 Mbps and the upload is capped at like 860 Mbps. It would peak at about 103 MB/sec, but sustained rate was around 60-70 MB/sec using two Usenet provides #FirstWorldProblems hahaha

 

31 minutes ago, bjp999 said:

Did ZFS handle this at full speed?

 

That I can't say because I've only had gigabit speeds for about 1.5-2 months and I just started using unRAID a month ago so I didn't really have the opportunity to download TBs worth of data since I already had pretty much everything I wanted. It had no problem with my 150 Mbps pipe though, but that's about 8-9x slower than what I currently have. I could write sequentially to my pool at about 1 GB/sec (yes gigabytes not gigabits) and randomly around 300-500 MB/sec, this is usually straight to the disks, but I did have an SSD in place as a SLOG (separate intent log, only useful for NFS and iSCSI since they do SYNC writes which can slow performance so the SLOG caches the SYNC writes and then batch pushes them ASYNC to the pool), so I have doubt it could handle it. Just for the hell of it I may test this out when my new motherboard comes, since I do have 4 1 TB drives that I can do striped mirrors of.

[SSD]

The WD Red is definitely not 7200 RPM. And 4TB drives are much slower than 8TB drives. Reconstruct writes are gated by the slowest disk. 

 

I understand not wanting to back your speed to 150 Mb, but I still think it would be useful to find the point at which it can keep up. Even if only as a baseline measure for comparison as you attempt performance improvements. It would be useful to know at what speed the throughout issues happen - 150, 250, 500, ... 

 

My first suggestion would be ditching the 4T drives and going to 8 or 10.TB 7200 RPM disks. With turbo write I think that would make considerable difference. And seems you are going to need the space!

[JorgeB]

22 minutes ago, brando56894 said:

Isn't that hitting your cache drive and not the disks themselves?

 

It's writing to the array, there's no cache on that server.

[trurl]

4 hours ago, brando56894 said:

I don't know if this is a result of cache=yes, so I changed it to cache=prefer and cleared out all everything in downloads, movies and shows (the majority of the data, the other shares equal maybe 50 GB total) and let it continue to download things for a few hours. I checked on it hours later and once again the cache drive was 100% full....

cache-prefer will result in files being moved from the array to cache. Please turn on Help when setting up your user shares. You probably also want to go to Global Share Settings and set Minimum Free for cache. Also, see here:

Moving this thread to General Support

[brando56894]

2 hours ago, bjp999 said:

The WD Red is definitely not 7200 RPM. And 4TB drives are much slower than 8TB drives. Reconstruct writes are gated by the slowest disk.

 

Yea, I know it's 5400 RPM, as seen in my attached speed test, it's 13 MB/sec slower. So there is a difference but it's not a huge difference.

 

 

2 hours ago, bjp999 said:

I understand not wanting to back your speed to 150 Mb, but I still think it would be useful to find the point at which it can keep up. Even if only as a baseline measure for comparison as you attempt performance improvements. It would be useful to know at what speed the throughout issues happen - 150, 250, 500, ... 

 

The problem isn't downloading and it keeping up, because I can always pause while par checking and extracting, I guess I could limit my torrent downloads from 3 to 1. 

 

2 hours ago, bjp999 said:

My first suggestion would be ditching the 4T drives and going to 8 or 10.TB 7200 RPM disks. With turbo write I think that would make considerable difference. And seems you are going to need the space!

 

Do you have an extra $1000 laying around that I could have? hahah

 

2 hours ago, johnnie.black said:

 

It's writing to the array, there's no cache on that server.

 

Wow, that's impressive!

 

1 hour ago, trurl said:

You probably also want to go to Global Share Settings and set Minimum Free for cache.

 

What's odd is I set it to 10 GB, and I just confirmed again that it does say 10 GB, but it seems like it didn't take hold.

 

1 hour ago, trurl said:

cache-prefer will result in files being moved from the array to cache.

 

Ah, that explains a lot.

Edited August 15, 2017 by brando56894

[trurl]

You must set Minimum Free to be larger than the largest file you expect to write.

 

When unRAID begins to write a file, it doesn't know how large the file will become. If you have more than minimum free, it can choose the drive and then if the file is larger than the available space, the write will fail. If the drive has less than minimum free, unRAID will choose another drive (depending on the other settings explained in the FAQ I linked).

[brando56894]

22 minutes ago, trurl said:

You must set Minimum Free to be larger than the largest file you expect to write.

 

When unRAID begins to write a file, it doesn't know how large the file will become. If you have more than minimum free, it can choose the drive and then if the file is larger than the available space, the write will fail. If the drive has less than minimum free, unRAID will choose another drive (depending on the other settings explained in the FAQ I linked).

 

Makes sense now, I didn't think I had to set it to something like 70 GB.

 

After setting the cache to "yes" on my shares, the mover is now moving stuff to the array at around 100 MB/sec with no VMs or containers running. I don't have much left to download but I'll let it run for a bit and see if it locks up when I have all the containers and both VMs up. Thanks everyone.

Edited August 15, 2017 by brando56894

[SSD]

55 minutes ago, brando56894 said:

 

Yea, I know it's 5400 RPM, as seen in my attached speed test, it's 13 MB/sec slower. So there is a difference but it's not a huge difference.

 

 

 

The problem isn't downloading and it keeping up, because I can always pause while par checking and extracting, I guess I could limit my torrent downloads from 3 to 1. 

 

13 is 10%. Not insignificant. My 8T 7200 RPM drives average about 175. So the difference between your 5400 RPM is about 30%. Very significant.

 

Even the 5400 RPM 8T drives are 145.

 

Quote

 

Do you have an extra $1000 laying around that I could have? hahah

 

No. But maybe not as much as you think. 8T drive sales have taken pricing under $20/T.

 

42 minutes ago, brando56894 said:

 

Makes sense now, I didn't think I had to set it to something like 70 GB.

 

70G is pretry high for cache. 70G sounds like a worst case fully reconstructed media file. Such files should be on your array not cache. Are your reseeded files staying on the cache? That might explain a lot.

 

An alternative is to install a larger unassigned device, like a 7200 RPM 8T drive for download and repair duty instead of the measly 0.5T SSD cache. It will be slower doing repairs, but you'd have 16x the space. Will run a lot longer before disk fills up! And you won't be using up the SSD. If you intend to do these downloads in bursts, this might be a decent solution.

 

If you're adventuresome, using BTRFS you could create an even larger volume (similar to RAID0) of unassigned devices and get an even bigger and faster "disk" for DLs, repairs and reseeding.

[brando56894]

I set it to 60 GB because that's around a common size for 4K UHD BluRays, no downloads, movies or shows should stay on the cache. I think my confusion between "yes" and "prefer" caused a lot of the issues.

Edited August 15, 2017 by brando56894

[SSD]

Forgive my confusion. I don't do torrents. But my understanding is that they come down in pieces, and are only assembled into the whole once repairs complete. And that the reassembly happens on the array. So the max file size on the cache would be pretty small.

[brando56894]

I never really looked into the ins and outs of it, but there isn't a repair or unpack step like there is with NZBs. The torrent data does come down in pieces but it's put in place once it's downloaded, some torrent clients allow you to pre-allocate the space needed before the whole file downloads in order to reduce fragmentation.

[JonathanM]

3 hours ago, brando56894 said:

I never really looked into the ins and outs of it, but there isn't a repair or unpack step like there is with NZBs. The torrent data does come down in pieces but it's put in place once it's downloaded, some torrent clients allow you to pre-allocate the space needed before the whole file downloads in order to reduce fragmentation.

Unless the torrent contents are actually archive files.

 

Fairly common in some circles.

[SSD]

It would seem that if you are DLing whole media files, the cache is something of a waste.

[brando56894]

5 hours ago, jonathanm said:

Unless the torrent contents are actually archive files.

 

Fairly common in some circles.

 

Yea, but those are usually extracted manually. I just started using Radarr (always used CouchPotato before with nzbToMedia for NZBs and just manually moved and post-processed torrents via CP) so I'm not sure if it is able to extract archives in torrents.

 

5 hours ago, bjp999 said:

It would seem that if you are DLing whole media files, the cache is something of a waste.

 

How so? They still need to be copied from one directory to another, so SSD to HDD is a lot quicker than HDD to HDD, especially if it's the same HDD.

[JonathanM]

2 minutes ago, brando56894 said:

Yea, but those are usually extracted manually.

Deluge has an extractor plugin, I imagine many torrent clients have the capability.

[SSD]

7 minutes ago, brando56894 said:

 

Yea, but those are usually extracted manually. I just started using Radarr (always used CouchPotato before with nzbToMedia for NZBs and just manually moved and post-processed torrents via CP) so I'm not sure if it is able to extract archives in torrents.

 

 

How so? They still need to be copied from one directory to another, so SSD to HDD is a lot quicker than HDD to HDD, especially if it's the same HDD.

 

If moved, not copied, it'd be instant. You'd want it to DL to the right user share.

[brando56894]

1 hour ago, jonathanm said:

Deluge has an extractor plugin, I imagine many torrent clients have the capability.

 

I just started using Deluge, and I must say I like it a lot better than Transmission, which I had been using for years. I haven't run across many zipped torrents yet since most of the ones I grab aren't archived.

 

58 minutes ago, bjp999 said:

 

If moved, not copied, it'd be instant. You'd want it to DL to the right user share.

 

I have to copy the file either way because I want it in my library, but I also want to keep seeding it. The only other option is to use hard links, which may be possible if I force the download and movies shares to one disk, but that's not optimal considering my disks are only 4 TB each and I have about 7 TB worth of movies, so I would have to copy those movies to the same disk that the seeds are on :-/

 

Copying them does take up a bunch of space since it's an exact duplicate, but I don't keep them for that long (24 hours to a few weeks, depending on size and where I grab it from). My ratio took quite a hit since I pulled like 50+ torrents from my tracker, so I turned off torrent searches in both Sonarr and Radarr until it gets back up to acceptable levels. I currently have 1.7 TB of seeding torrents so it shouldn't take that long.

[SSD]

@brando56894

 

You could link the file from the array into a seeding directory. One copy, two places at the same time. 

[brando56894]

Hard links will work across disks in the array?  I thought it had to be in the same physical disk or does that only apply to non-RAID setups (single disks)? Or are you talking about symlinks? I think Sonarr/Radarr only has the option to move, copy or do hardlinks.

[SSD]

Soft links?