20Gb Multichannel SMB transfers, XFS/BTRFS, HBA/onboard speed, cached speed and more?

[andynuke]

Hi All.

I watched with interest Linustechtips various videos using UnRAID for VM's and storage servers. I'm looking at migrating/amalgamating from various old NAS boxes into a new UnRAID server

I Would LOVE to do this with a new UnRAID box with dual 10Gbit card and a super-fast NVMe cache drive. BTRFS RAID-6 (I've no idea why only Synology have this working atm) and preferably multi-Channel SMB 3 (again I've been using this on Synology DSM6 for a while now and it's working flawlessley)


1. Terastation Pro II 8TB - Marvell Orion Arm (500Mhz RISC chip with Hardware RAID-5 XOR engine) using 4x Samsung 204i 2TB drives
2. Terastation Pro II 8TB - Marvell Orion Arm (500Mhz RISC chip with Hardware RAID-5 XOR engine) using 4x Hitachi 7200rpm Drives
3. Terastation Live 8TB - Marvell Orion Arm (500Mhz RISC chip with Hardware RAID-5 XOR engine) using 4x Seagate 7200rpm Drives
4. Synology 414J 20TB - MindSpeed Comcerto 2000 Dual Core 1.2GHz (Hardware RAID XOR) using 5x Seagate 5TB 5400 rpm Drives
5. Synology 1812+ 48TB - Intel Atom D2700 2c/4t CPU with 3GB RAM. using 6x Seagate 'Compute' 8TiB drives
6 Asus P8C-WS i7-2600 48TB w/32GB Ram and LSI MEGARAID 9261-8i (8ooMhz power PC XOR RAID chip). using 6x Seagate 'Compute' 8TB drives

 I have held onto the TeraStations as 'backups' as they have been bomb-proof and browsing the XFS RAID-5 shares on them has been much more predictable and faster than the Synology DiscStations that I got to replace them even though the transfer rate over is slower (200Mb i.e. 25MiB constant with 4x2TB drives in the TeraStations). The Synology DiscStations have fancier software, but I really don't use them for anything but file-servers. Now using ext4 RAID-5 on both the 1813+ and the 414J has proved to give unpredictable transfer speeds on consistent large BluRay rips, it can range from 20MB/sec to 107MB/Sec and they really slow down when full. Deleting files is a slow process on them. I'm writing this down to a filesystem thing on ext-4 as the much slower 800Mhz CPU's on the TeraStations positivley FLY with XFS. Plus the whole 'SYNOLOCKER' randsomware thing really got me thinking that linux based boxes were really no less of a security target than an average windows PC. I've recently upgraded the drives in my 1812+ to 6 off Seagate Barracuda 'Compute' Drives and done the same in my P8C-WS workstation to compare. The P8C-WS Hardware RAID card is hands down the winner, reaching 1GB/sec, 10x faster than the Synology DiscStations and 40x faster than the TeraStations. I realise that these NAS boxes are basically consumer friendly cheap hardware (atoms and phone SoC's) linked with a nice custom case.

Now the question is.
I've got £1000 to spend and would really like to upgrade to dual 10Gbit network, so I can move large cache files back and forth at 2GB/s to an NVMe cache using multi-channel SMB. I've been using the latest DiscStation Manager O/S with BTRFS RAID-6 and Multi-Channel SMB 3 for the last couple of months with both Gigabit ports of the 1812+ and it's reaching 2Gbit/s (i.e. 237MB/s) at peak and I've suffered Zero Problems on 30-40 TB's of test data transfer.
Here's the basics of my new 'Ultra Performance' UnRAID server idea

- ASUS Intel 1151 Socket Z370 Chipset Prime A D4 ATX Motherboard
- Intel i3-8100 Processor (3.6Ghz 4C/4T) (I noticed Linus had issues with his 4-channel SMB transfers on a 2 core machine, that went away when 4 cores were used)
- Corsair Vengeance LPX 8GB Dual Channel (2x4GB) DDR4 2400MHz
- Samsung 250GB NVME EVO SSD Cache Drive, (3.2GB/s Read / 1.5GB/s Write)
- 6 off Seagate 'compute' 8TB drives plugged into the onboard SATA 6G ports (or a HBA if they perform better with UnRAID than the onboard?)
- Intel X540-T2 Dual 10GBit Network Adapter.
- Whatever 16 tray rack case takes my fancy.

I'd really like to know:

1) When these 'Tweaks' Linus did to reach 1GB/sec transfer will be rolled into UnRAID?
2) Are there any plans to support Multi-Channel SMB 3.0? to hopefully get that up to 2GB/s dual link? (i've had it working great for 3-4 months using plain Gigabit and it 'just works' on windows 10 and Synology DSM6, no fancy link aggregation bonding configuration or switch required)
3) I'm happy with XFS as it has always 'seemed' more robust on the old TeraStations. Do you support (Patrol Read/Scheduled parity check).

Theoretically, I can just install Win 10 with the 9261-8i in this new box, use the NVME as a boot drive and reach the 1GB/s transfer limit of the 6xSeagate raid array. But it WOULD be awesome to use UnRAID and use the NVME as an ultra fast cache and dual channel SMB to hit 2GB/s to and from the box.

Sorry for such a long winded and technical first post by the way! Hello Everyone!


 

Edited October 21, 2017 by andynuke

[JorgeB]

3 hours ago, andynuke said:

1) When these 'Tweaks' Linus did to reach 1GB/sec transfer will be rolled into UnRAID?

 

They were included some time ago, since unRAID v6.2

 

4 hours ago, andynuke said:

2) Are there any plans to support Multi-Channel SMB 3.0? to hopefully get that up to 2GB/s dual link? (i've had it working great for 3-4 months using plain Gigabit and it 'just works' on windows 10 and Synology DSM6, no fancy link aggregation bonding configuration or switch required)

 

Samba multichannel support is experimental and should be used for testing only, but you can enable it manually on unRAID if you want.

 

4 hours ago, andynuke said:

3) I'm happy with XFS as it has always 'seemed' more robust on the old TeraStations. Do you support (Patrol Read/Scheduled parity check).

 

You can schedule parity checks, once a month is recommended.

 

[Perforator]

The limiting factor is going to be the disk drives. I have 10GB Ethernet between my unRaid server and workstation and a Raid 0 BTRFS cache pool of 2 Samsung 850EVO's and the highest transfer rate I can get is a little over 3GB/second. The problem is I'm copying from a single 850EVO on my workstation. I had a Samsung Ultra-M2 drive as my cache disk but I replaced it with the BTRFS pool because it kept over heating and throttling the transfer speed and that was with Gigabit Ethernet. Just get 2 10GB Ethernet cards and call it a day.

 

Edited October 21, 2017 by Perforator

[andynuke]

A. They were included some time ago, since unRAID v6.2
- Amazing, thanks Johnnie!

A. Samba multichannel support is experimental and should be used for testing only, but you can enable it manually on unRAID if you want.
- Yep, yep, I'm fully aware when I secure shelled into the Synology OS and modified the config files. I've been testing now for a couple of months before I dump the TeraStations (my go to XFS stalwarts) I've filled the array on the Synology multiple times, both in 'standard' single channel mode and multi-channel mode, in both RAID-5 and RAID-6 BTRFS and ext-4, broken the array and then repaired it, file integrity done with full hash has been perfect. I've run across no issues with multi-channel. I was a little worried about data fragmentation and speed as it is streaming in via 2 separate channels and i wasn't sure how the re-sync and write is done, but I've had 100% success so far. currently in 'testing' mode with 4 different paths.

1) UnRAID on a new Coffee Lake system (NVMe Cache) - 3GB/s Cache pool ?MB/s to drive pool
2) UnRAID on the old workstation (no cache) - ?MB/s to drive pool
3) The Synology 1812+ with BTRFS & Multi-Channel - 160-237MB/s to drive pool (I may try the SSD cache feature to get that to a solid 237MB/s)
4) LSI 9261 & Win on new Coffee Lake system. 1GB/s straight from the drive pool.

A. You can schedule parity checks, once a month is recommended.
Thanks, I've not started messing with UnRAID yet, the new components are on their way. I kinda thought it would be in there somewhere!

Edited October 21, 2017 by andynuke

[andynuke]

4 hours ago, Perforator said:

The limiting factor is going to be the disk drives. I have 10GB Ethernet between my unRaid server and workstation and a Raid 0 BTRFS cache pool of 2 Samsung 850EVO's and the highest transfer rate I can get is a little over 3GB/second. The problem is I'm copying from a single 850EVO on my workstation. I had a Samsung Ultra-M2 drive as my cache disk but I replaced it with the BTRFS pool because it kept over heating and throttling the transfer speed and that was with Gigabit Ethernet. Just get 2 10GB Ethernet cards and call it a day.

 

Thanks Perforator.

Yep, I'm aware of the heat/throttling issue on M.2. Good to know the Linux driver in unraid can transfer to the RAID-0 cache pool at the full 3GB/s thats perfect for me, as I never usually write more than 100GB/day in large files, so the 'mover' can move them to the drive pool as & when. I'm going to be using  6x Seagate 8TB PMR drives (ST8000DM004) on the Intel 370 Chipset motherboard controller, that are capable of 200MB/s each. Do you know what kind of speed I can expect to the drive pool? I'm a bit of a noob with UnRAID  Best I've had to/from a drivepool direct has been the 9261-8i, which on larger transfers (128k+) has been pretty much 1GB/s straight from the spinning rust. (3-400MB/s RAID-6 write though)

Edited October 21, 2017 by andynuke

[Perforator]

The mover runs in the background usually on a schedule that's configurable. The amount of time it takes to transfer from the cache pool to the array is not that critical unless you run out of space on the cache pool. The transfer rate is going to be limited by the write speed of a single drive.