Strange behavior with SAS card in the mix

[Fleat]

TLDR: Having a SAS card in the loop with only the necessary drive spun up causes slow transfer speeds and some other weirdness. Could use any advice people have.

 

Problem:

Everything was working great before I moved to a SAS card (for more drives). With a SAS card in place, I can only seem to achieve a max of around 40MB/s in transfers to external networked computers as well as internally networked VM's unless all drives in the array are spun up. Accessing a drive that causes a "spin up" in Unraid will also freeze access on any other drives that are currently being utilized.

 

For example, watching a movie stored on Disk 1 and everything is going fine until someone tries to stream a TV show stored on Disk 4 at the same time. This will essentially "freeze" the movie until Disk 4 is spun up and the information is found. With the current speeds, I cannot stream high quality FLAC music or my ripped Blu-Ray movies without getting "Reads too slow from server" in Kodi many times.

 

Troubleshooting Steps

Test #1: Tests performed with all drives on SAS2 backplane plugged into M1015 port 0
a. Transfer file directly from cache drive (SSD) to networked PC: 110MB/s
b. Transfer file directly from array drive (HD) to networked PC: 40MB/s
    Note: This is with only the necessary disk spun up
c. Transfer file directly from array drive (HD) to VM on cache drives: 40MB/s
    Note: This is with only the necessary disk spun up
d. Transfer file directly from array drive (HD) to networked PC: 110MB/s
    Note: All drives are spun up during this test
e. Transfer file directly from array drive (HD) to VM on cache drives: 150-200MB/s
    Note: All drives are spun up during this test

 

Test #2: Tests performed with a single drive from the array on port 1 of the M1015 using a 8087 to sata breakout cable (bypassing the SATA backplane)
a. Transfer file directly from single drive (HD) to networked PC: 40MB/s
    Note: This is with only the necessary disk spun up
b. Transfer file directly from single drive (HD) to VM on cache drives: 40MB/s
    Note: This is with only the necessary disk spun up
c. Transfer file directly from single drive (HD) to networked PC: 110MB/s
    Note: All drives are spun up during this test

 

Test #3: Tests performed with a single drive from the array on motherboard SATA
a. Transfer file directly from single drive (HD) to networked PC: 110MB/s
    Note: This is with only the necessary disk spun up
b. Transfer file directly from single drive (HD) to VM on cache drives: 150-200MB/s
    Note: This is with only the necessary disk spun up

Test #4: Bought a new SAS card HP220 (SAS2308 vs SAS2008 chipset) and flashed it to LSI IT firmware
All tests from Test #2 were performed with identical results aside from a few unexpected parity drive read errors.

 

Test #5: M1015 in an Xpenology setup with WD 8TB Reds in reverse sata breakout set up in SHR
All tests resulted in full gigabit transfer speeds to networked PCs
    Note: This test doesn't really validate much except that the SAS card is working

 

Other configuration tests
1. Swapped to a new SFF8087 cable
2. Swapped from the onboard Intel NIC to a dual port dedicated Intel card
3. Performed SMB tweaks per this post (https://forums.lime-technology.com/topic/46802-faq-for-unraid-v6/?page=2&tab=comments#comment-526285)
4. Removed all other cards and swapped pci-e slots with identical results
5. Ran the diskspeed script from these forums and performance seems adequate

6. Two full passes of Memtest on Unraid boot without any errors

 

Hardware Specifications: PCPartPicker

Software: Unraid 6.3.5 with dual parity (2 8TB WD Reds)

Diagnostics: The anonymized version contained more info than I felt it should - I can provide this upon request to Unraid folks

Diskspeed Script Results:

 

Edited October 16, 2017 by Fleat

[Squid]

5 hours ago, Fleat said:

Accessing a drive that causes a "spin up" in Unraid will also freeze access on any other drives that are currently being utilized.

Admittedly I didn't read you entire post (TLDR), but the above is quite normal on many add-on controllers (not just SAS).  The alternative is to utilize spinup groups where disk A, B, and C will all spin up simultaneously when any of them are accessed, or to always keep the drives spinning.

[Fleat]

18 minutes ago, Squid said:

Admittedly I didn't read you entire post (TLDR), but the above is quite normal on many add-on controllers (not just SAS).  The alternative is to utilize spinup groups where disk A, B, and C will all spin up simultaneously when any of them are accessed, or to always keep the drives spinning.

Thanks for the heads up about that. I will test out the extra SAS controller in that kind of scenario outside of Unraid.

[Fleat]

5 hours ago, Squid said:

Admittedly I didn't read you entire post (TLDR), but the above is quite normal on many add-on controllers (not just SAS).  The alternative is to utilize spinup groups where disk A, B, and C will all spin up simultaneously when any of them are accessed, or to always keep the drives spinning.

 

I do not seem to notice the same type of behavior when testing with the HP220 (SAS9205-8i) outside of Unraid.

 

My interim solution is to have a spin up group that spins up the entire array as that seems to retain the performance I would expect along with no weirdness of accessing various content across drives. This is not a great solution as I will soon have 20 drives that will have to spin up to access the media on one drive just to get the performance I need. Lots of wasted energy and generated heat that shouldn't be required.

[JorgeB]

The delay on spin up is normal, the low performance when only one disk is spun up it's not, at least I've never noticed that nor do I remember seeing anyone else with a similar issue.