Double or Triple "Cache" pools


1812

Recommended Posts

Ok, this may be dumb, but I have a use case that this would be really effective on.

 

 

Currently I pass trough 2 unassigned 10k drives to a vm as scratch disks for audio/video editing. In the vm, they are then setup as Raid 0. Super fast. The problem becomes that the drives are then bound to that VM. I can't use the disks for any other vm nor span a disk image (work areas) for separate vm's on that pool. I think it would be better to have the host (unRaid) manage the raid, and then mount the "network" disks and use them that way. Since the vm uses paravirtualized 10GBE adapters, performance should be no issue. And multiple vm's could access them as well.

 

Why don't I just add more drives to my current cache pool? Separation. I don't want the dockers that are running, or the mover, or anything else to interfere with performance. Plus, i'm not sure how mixing SSD's and spinners would work out. Maybe ok? I'm sure someone has done that.

 

TLDR: Essentially I'm suggesting that we be able to have more than one pool of drives in a specifiable raid setup (0 and 10! please!)

  • Like 1
  • Upvote 7
Link to comment

I was thinking about something similar the other day.  Except I was thinking about the unassigned devices plugin.  I honestly don't now how difficult it would be, but it seems that it should be possible without too much trouble to add support for creating a software raid array in unassigned devices. 

 

Unless I'm misunderstanding your post, this would provide a similar outcome wouldn't it? 

 

Link to comment

I was thinking about something similar the other day.  Except I was thinking about the unassigned devices plugin.  I honestly don't now how difficult it would be, but it seems that it should be possible without too much trouble to add support for creating a software raid array in unassigned devices. 

 

Unless I'm misunderstanding your post, this would provide a similar outcome wouldn't it?

 

I've asked this before. UnRaid is built on top of (or maybe more accurately hijacks) the software RAID feature of Linux. Decoupling is not going to happen.

 

It IS possible to have a RAID set in your server, it just has to be a hardware RAID. I have a RAID0 of 2 4T drives as my parity.

Link to comment

It's possible to use a btrfs pool using the unassigned devices plugin, it won't look pretty since it's not officially supported (ie, only one device shows as mounted)  but it works with any raid profile:

 

btrfs fi show /mnt/disks/Test
Label: none  uuid: 75c7d7f5-74e4-4662-b465-c400b7384a6c
        Total devices 2 FS bytes used 1.75GiB
        devid    1 size 232.89GiB used 2.03GiB path /dev/sdf1
        devid    3 size 232.89GiB used 2.03GiB path /dev/sde1

 

btrfs fi df /mnt/disks/Test
Data, RAID0: total=2.00GiB, used=1.75GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=2.02MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

ETA: You'll need to manually create the pool first (or use the cache slots).

Link to comment

It's possible to use a btrfs pool using the unassigned devices plugin, it won't look pretty since it's not officially supported (ie, only one device shows as mounted)  but it works with any raid profile:

 

btrfs fi show /mnt/disks/Test
Label: none  uuid: 75c7d7f5-74e4-4662-b465-c400b7384a6c
        Total devices 2 FS bytes used 1.75GiB
        devid    1 size 232.89GiB used 2.03GiB path /dev/sdf1
        devid    3 size 232.89GiB used 2.03GiB path /dev/sde1

 

btrfs fi df /mnt/disks/Test
Data, RAID0: total=2.00GiB, used=1.75GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=2.02MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

ETA: You'll need to manually create the pool first (or use the cache slots).

 

Would this allow the btrfs RAID set to appear as a drive and be used as parity? I think not but you've got me curious.

Link to comment

Currently I pass trough 2 unassigned 10k drives to a vm as scratch disks for audio/video editing...

How large are these? Might be simpler just to replace with SSDs and add them to your regular cache pool. Sometimes it's better in the long run to just spend money. ;D

 

I used a 10K drive as cache (because I already had it) when I built my first unRAID but decided it was too noisy and quickly replaced it with SSD. That was back on unRAID 4.7

Link to comment

Currently I pass trough 2 unassigned 10k drives to a vm as scratch disks for audio/video editing...

How large are these? Might be simpler just to replace with SSDs and add them to your regular cache pool. Sometimes it's better in the long run to just spend money. ;D

 

I used a 10K drive as cache (because I already had it) when I built my first unRAID but decided it was too noisy and quickly replaced it with SSD. That was back on unRAID 4.7

 

300GB each. Had them already. No sense in upgrading to ssd's for them because my server backplane only supports 3gbps sata (6 on sas). To add ssd's to the cache pool, I'd have to convert the existing 2 disk  pool to raid 10 for better performance while maintaining redundancy, which means not just adding 2 disks for capacity but instead buying 4 new larger SSD's.... and then still sharing i/o with apps/dockers running.

 

All my servers are in my basement, so noise is not a factor. Plus you never hear the 10k drives over the fan noise. I think I use unRaid a little different than some.

 

 

 

Maybe unassigned devices could have a raid option that would allow it to be more easily setup.... ;D

Link to comment

 

2 hours ago, hermy65 said:

 

Yep, im more so looking to split it out though so i can have better quality ssd's running my vms/docker containers and regular ssds handling tasks like downloads, etc

 

This post ^^^^^^

 

I really don't want to put 2-4 10k 300GB scratch drives, 2 Samsung Evo SSD's, 2 Kingston entry level SSD's, and 1 patriot blast  all in the same pool.... that just feels dirty.... And means that the 10k drives are spun up every time anything accesses the pool.

Link to comment

+1 to this. Multiple cache pools is something that would be really useful. My use case is I want to have separate SSD based cache for VMs/Dockers vs a cache pool for front ending network shares which would be based on spinning disk with more capacity. Even better would be able to choose which cache pool for certain mounts in a docker. Example: the Plex docker image resides on the spinning disk cache which is mirrored and /transcode filesystem being located on a SSD based pool that is set to raid 0. 

Link to comment

The fastest 10k drives, even in RAID0 are slower than even the crappiest SSD.  Throw those two things out and put in a single SSD like the 480GB Sandisk UltraII.  

 

unRAID now also supports NVMe, so you could go with something like an Intel 600p or ADATA SX3000 and stick it in a 4x PCI-E slot on an adaptor board.  This will give you lightning quick storage, like 2GB/s or more in the case of the ADATA.  

Link to comment
2 hours ago, HellDiverUK said:

The fastest 10k drives, even in RAID0 are slower than even the crappiest SSD.  Throw those two things out and put in a single SSD like the 480GB Sandisk UltraII.  

 

unRAID now also supports NVMe, so you could go with something like an Intel 600p or ADATA SX3000 and stick it in a 4x PCI-E slot on an adaptor board.  This will give you lightning quick storage, like 2GB/s or more in the case of the ADATA.  

 

 

your assumption is that i (and everyone else) have/has more pci-e slots available, which is not the case.

 

Also, your assertion that the fastest 10k drives in raid0 are slower than the crappiest ssd are incorrect. My kington SSD drives in this very same server read at 250+MB and write starting at 180MB and then drop off as the file write progresses. that's what 20 bucks get you for a cheap ssd. The 10k drives are 300+MB each way in raid 0. 

 

Regardless of those 2 facts, your comments do not diminish the initial request, which is to take 2 or more drives and make an independent cache pool for usage. I would love in the future to replace the 2 10k drives with quality ssd's and hit 1GBps or more depending on configuration. 

 

the premise of unRaid is working with what you've got. various drives, etc... and this request is in the spirit of that.

 

 

but building further on comments above, here is another scenario that would benefit from a separate cache pool:

 

I've just finished working on a project on the vm desktop I have setup. I copy the 200GB file to the array which goes to the cache drive first to be later moved at night. Using the para-virutlaized 10gbe adapter, it attempts to max out the 500MBps write speed of the quality ssd drive but gets choked up because that drive is also being used as a downloader ingest and plex transcode directory. So now my write speeds take a hit. 

 

 

Additionally, if unRaid wants small business attention, the ability to setup a separate cache pools could be beneficial. Imagine if 2 video editors were both trying to write to that single ssd... you're wasting company time and company money as they fight for i/o. You could give each their own and have no cross over issues. It's the same situation as multiple vm's and dockers sharing disk i/o in moderate to higher usage environments. 

Edited by 1812
  • Thanks 1
Link to comment

Seek time is more important for a scratch drive, and no matter what you think, RAID0 of 10k drives will be significantly slower than a SSD.  Been there, done that, got the t-shirt, the mug and the ticket stubs.

 

I don't know why you're talking about $20 drives, they're obviously something from years ago, probably Sandforce era, which are nothing like what a budget SSD of the last year or so can do.  

Link to comment

ssd drives were purchased 8 months ago.. so, not that old. I'm talking about them because you said 

 

3 hours ago, HellDiverUK said:

The fastest 10k drives, even in RAID0 are slower than even the crappiest SSD.

 

and that is not true.  source: actual usage numbers presented previously of the crappiest SSD which I happen own and use (in addition to some nicer ones.) You then try to change your assertion by adding an excluding argument via the Sandforce comment, but your "crappiest SSD" claim remains disproven.

 

I've even provided the read time on the raid0 10k drives vs the cheap ssd. If you skipped that part, the summary was that it was better. Your argument based on cheap ssd's is invalid and incorrect. This isn't an opinion, it isn't what i "think." These are actual numbers provided from real case usage scenarios and benchmarks I've done on the hardware.

 

You also fail to address the i/o limitation based on actual use scenarios as outlined above. Probably because either it doesn't effect your usage scenario, or you realized cost becomes a factor quickly to try and mitigate that bottleneck.

 

But to your assertion, let's take it to the logical conclusion which is using more and larger SSD's in the current pool configuration. For my needs and wants, and those of several others, I could go purchase four 2 tb quality ssd's for $2400 USD, put them in raid 10, and they'd give me 4 tb redundant with over 1GBPS read/write, and a decent amount of bandwidth since it's my only option because I have no more pci-e slots to use. That sounds like a great thing to do, you know, spend more on 4 drives than my entire server cluster cost to assemble (including the rack and double APC battery backup system.) Why 2 tb drives? I have a few vm's that run at the same time and I need a few persistent scratch disks to go with them. As also described, my file sizes are quite large and require more space on the cache drive than what would typically be available when also hosting vm's and docker downloads. So a couple 120gb drives won't cut it.

 

From a cost factor, I could buy 8 wd red 2 tb spinners for the price of 1 good 2 tb ssd. In raid 10 it they would reach or exceed the single ssd drive read/write speeds AND provide data redundancy not found on the single ssd. not to mention that i'd have 4x the storage over the single drive, and 2x the storage over the 2400 dollar raid 10 ssd pool while saving 3/4 the total price.

 

To use only cheap drives, with an average cost of 50-75 dollars, I would need a pool of at last thirty-three 120gb drives to get near 2 tb with no redundancy. That's 1,650-2,475 usd. I currently only have space for 23 drives on my main server so that's not doable. And also, that's ridiculous. I'm not Linus, and I don't get storinators or boxes of SSD's for free.

 

¯\_(ツ)_/¯

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • Upvote 1
Link to comment
4 hours ago, 1812 said:

I copy the 200GB file to the array which goes to the cache drive first to be later moved at night. Using the para-virutlaized 10gbe adapter, it attempts to max out the 500MBps write speed of the quality ssd drive but gets choked up because that drive is also being used as a downloader ingest and plex transcode directory. So now my write speeds take a hit. 

 

+1

 

I ran into this exact issue a few weeks ago. VMs were in the cache pool, and I was finding sustained disk transfers were seriously impacting performance of the VMs themselves. I ended up pulling them out of the cache pool, and installing them on separate SSD in Unassigned Devices. Now sustained I/Os have no impact on my VMs.

 

It's a solution, and works well, but multiple cache pools would make it a cleaner, more streamlined setup.

Link to comment
15 hours ago, DoeBoye said:

I ran into this exact issue a few weeks ago. VMs were in the cache pool, and I was finding sustained disk transfers were seriously impacting performance of the VMs themselves. I ended up pulling them out of the cache pool, and installing them on separate SSD in Unassigned Devices. Now sustained I/Os have no impact on my VMs.

 

I have also moved my VM's off the SSD cache to an SSD in Unassigned Devices.

 

I hope it's something unRaid are working on, given VM's are a promoted functionality of the software.

Link to comment
  • 3 weeks later...
24 minutes ago, 1812 said:

When your cache drive fills up from moving moving files to the server or downloading, it pauses your VM's.

Respectfully, that's what the min free space setting is for. As long as you use a share that is set to cache:prefer for stuff that you want to live on the cache drive and cache:yes for stuff that will be moved to the array, and have a proper setting for min free space, you will never run into this specific issue.

  • Upvote 2
Link to comment
12 minutes ago, jonathanm said:

Respectfully, that's what the min free space setting is for. As long as you use a share that is set to cache:prefer for stuff that you want to live on the cache drive and cache:yes for stuff that will be moved to the array, and have a proper setting for min free space, you will never run into this specific issue.

 

Such a civil response! I'd like to reply but I can't find my monocle.

 

 

 

 

Wait, there it is.

 

 

 

  ಠ_ರೃ

 

 

 

 

You are correct, but I'm lazy and don't want to adjust each of my 40+ shares to guesstimate the max file size ever each will see. And I'm not willing to clone the 40GB max size for mkv files to each one because then I have a 40GB hole in my SSD that I never get to use while waiting for the scheduled move. And that is the point of having the ssd cache, to keep from spinning up the disks unnecessarily and/or getting faster network transfer speeds. 

 

 

( I did make a joke about the "civil response," but i'm messing around, and it's been a looooong day. I'm not mocking you. I just can't bring myself to erase the ascii monocle guy at the point in the evening. )

 

 

 

Link to comment
  • 1 month later...

+1 for multiple cache pools. For my specific use case, I have my VMs currently on my nvme drive which is loaded up via the go file at the moment. The cache pool is being utilized by my btrfs raid0 ssd pool. I like that UD "sees" my nvme, but I don't like not having temps on it, and TBH (while I already have it setup) having to add things in a round about way feels.... "dirty" I believe is the word someone else used, lol. Multiple cache pools please! 

Edited by ryoko227
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.