Update

[clevoir]

I am running 6.3.2. using a ASUS M5A99X EVO motherboard, AMD 4300 Quad-Core CPU, 16GB ECC RAM and Corsait TX650 PSU

 

I have 18 disks served by 3 Supermicro AOC-SASLP-MV8 HBA cards

 

I am getting frustrated with slow parity check speeds, and sometimes have drive errors, which I am not sure are caused by the Marvel bug (I have tried rteplacing cables)

 

I did purchase 2 DELL H310 cards, but found them incompatitble with my motherboard.

 

I am looking for a motherboard with 3 x8 PCI Express slots, so that I can use DELL H310 or IBM 1015 cards, I would be grateful for any recommendations

[Dissones4U]

1 hour ago, clevoir said:

I am running 6.3.2. using a ASUS M5A99X EVO motherboard, AMD 4300 Quad-Core CPU, 16GB ECC RAM and Corsait TX650 PSU

 

I have 18 disks served by 3 Supermicro AOC-SASLP-MV8 HBA cards

 

I am getting frustrated with slow parity check speeds, and sometimes have drive errors, which I am not sure are caused by the Marvel bug (I have tried rteplacing cables)

 

I did purchase 2 DELL H310 cards, but found them incompatitble with my motherboard.

 

I am looking for a motherboard with 3 x8 PCI Express slots, so that I can use DELL H310 or IBM 1015 cards, I would be grateful for any recommendations

 

I was advised to look into the H310 just yesterday and I'm curious if what ashman70 told me might apply to you too? I'm assuming that when you bought the H310's you made sure you had the physical slots to support them and any incompatibility is in the firmware. ashman70's advice to me was "flashing the H310 firmware to Dell IT mode", might that make the card work for you?  Honestly, I have no idea, I'm new to much of this but I thought I'd throw it out there since I was looking at that card just yesterday. Good luck....

Edited March 5, 2017 by Dissones4U
typo

[tdallen]

Are you trying to retain your CPU, or willing to upgrade that as well?

[clevoir]

I am open to suggestions that mean that I will have to change the CPU too.

 

The DELL H310 cards were flashed to both DELL and LSI firmware but when plugged into my motherboard, the server used to crash after a number of minutes, this was diagnosed as an imcompatibility issue by other members.

 

I then sold to cards on eBay, with the winning bidder giving me feedback that they were working fine.

[tdallen]

I don't know AMD motherboards very well, so these comments are about Intel motherboards.  Unless you go to a server class motherboard you aren't going to get 3 cards operating in full x8 mode because even the most modern Skylake and Kaby lake boards with DMI 3.0 only support 20 PCIex lanes.  You'd have to go to a Socket 2011 board and its 40 PCIex lanes to get full support.  Then again, it really shouldn't be necessary to do that.  A PCIex 2.0 or 3.0 x8 SATA Controller operating in x4 mode should be sufficient for 8 spinning hard drives.  So a motherboard with 3 PCIex x16 slots capable of operating in x8/x8/x4 mode should be sufficient.  

 

I tend to go with Asus, ASRock, or Supermicro boards.  Do you have any brand preferences, your current board is an Asus.  Also, your current board is AMD - do you want to stay with AMD or are you willing to go Intel?

[clevoir]

I am willing to change I am looking for a motherboard which is proven to work with 3 Dell H310 / IBM 1015  cards that uses ECC RAM. x8 / x8 / x8 would be a bonus

Edited March 6, 2017 by clevoir

[Vr2Io]

How slow about the parity check ?

 

2 years ago, when I first try unRAID, I use Asus M5A88 with (Athlon™ II), I have 12 disks by on-board 6 SATA + 2 Marvell 9215 add-on card (4 SATA port, x1 PCIe card)

Parity check (single) could reach average ~100MB/s, that's great as those cards only x1 PCIe and cheap. But later I found SATA I/F error in Marvell port and make my disk SMART UDMA error, change cable would not help.

 

So I buy a China make LSI SAS2008 card and flash IT mode (H310 also SAS2008 chip), the parity check speed was same but no more I/F error found.

After a year, I upgrade CPU/MB to Intel platform (also low end) and migrate those 12 drive by 6 bigger and faster HDD, then my parity check speed raise to average ~160MB/s.

 

As data size grow, I add 2 more disk but slower then those 6 disk, the parity drop to ~140MB/s.

 

Due to those China make card was cheap, I also buy another SAS2308 ( PCIe 3.0 x8 ) card, as expect, speed would't increase. Anyway, this have confirm my MB haven't problem in running 2 LSI card and a 10Gbe LAN card.

 

For your case, I think you should check which HDD was slower in system, LSI card won't help if bottleneck not there. Running 2 LSI card in x8+x8 mode is enough (direct connect to CPU), the remain 2 disks use on-board SATA won't have different with 3 SAS card.

 

For future plan, I also want to got a MB have 3 full-speed PCIe, but this will happen in high-end platform only i.e. x99. If PCIe not come from CPU, suggest should be avoid.

Edited March 6, 2017 by Benson

[clevoir]

I think my main problem is that my existing cards are only running at x4 speed, 60MB/s is as fast as they will go.

 

Parity checks are taking 12+ hours to complete!

[Vr2Io]

So your largest disk was 3TB.

I notice your MB haven't display output, so how you arrange for the display + 3 HBA, which one in x1 slot ?

If 2 HBA run into extended PCIe slot , this may got slow issue.

 

For me, the AMD nor Intel platform, both use buit-in display, so I can free the x16 slot for HBA. ( both MB haven't x8+x8 support )

Pls note, although the parity speed ~140MB/s, due to largest disk size was 6TB, I aslo need ~13hrs to complete parity check.

 

Edited March 7, 2017 by Benson

[clevoir]

I am using 3 PCIe slots (1 of which is running at x4) for the HBA cards, and a PCI video card for the display.

 

My latest disk is 3TB, but all disks are only running at 3MB,s. Parity speeds run at 60MB's at fastest.

 

I'm looking for a solution to allow disks to run at SATA III speeds with parity speeds over 100MB's

[DoeBoye]

19 hours ago, clevoir said:

I think my main problem is that my existing cards are only running at x4 speed, 60MB/s is as fast as they will go.

 

Are you sure 60MB/s is a limitation of the card? I used to have those cards, and I recently just re-installed one when when one of my 1015s started getting wonky, and I would get parity check top speeds over 100MB/s on a regular basis. Admittedly, my overall parity checks run slower than that because I have a couple of older 2TB drives, but drives on the  AOC-SASLP-MV8 aren't holding up the show. 

 

According to the Supermicro data page, these cards can do 300 MB/s per channel in a 4x PCI-e slot

 

Quote

• 8-channel SAS/SATA adapter with 300MB/s per channel
• PCI-E x4 interface

 

I say this simply because you might not be looking at the right part as reason for the slowdown.

[garycase]

1 hour ago, clevoir said:

My latest disk is 3TB, but all disks are only running at 3MB,s. Parity speeds run at 60MB's at fastest.

 

3GB/s is not really a limiting factor -- that's still faster than any rotating platter disk can sustain.

 

As for the speed limitation of your SASLP-MV8 cards => these are 4-lane PCIe v1 cards ... so they're capable of 1GB/s total bandwidth (250MB/s/lane).   With 8 attached disks that's a theoretical 125MB/s/disk ... well above the 60MB/s you're seeing.    Real-world performance rarely hits the theoretical max, but you should still be able to get more than 100MB/s with these cards.

 

Remember that the parity check speed is always limited by the slowest disk currently involved in the check.    Look at your 18 disks and see which ones might have lower density platters that could be the reason for the slow parity check speeds.    Note what happens to the check speed as it crosses the "boundaries" of the smaller disks -- where only the largest disks (which are likely higher density) are involved.  e.g. your 3TB disk most likely has 1TB platters; and some of your 2TB disk MAY also have that density; but smaller disks likely have lower density platters, which will slow down the parity check to the speed of those disks.

 

[garycase]

53 minutes ago, DoeBoye said:

According to the Supermicro data page, these cards can do 300 MB/s per channel in a 4x PCI-e slot

 

NO -- that's a deceptive statistic.    They CAN do 300MB/s on each channel, since they use a SATA-II interface; but the card can't sustain that for ALL channels at once due to the bandwidth restriction of the PCIe bus.

[JorgeB]

Max speed during parity check of fully loaded SASLP is 80MB/s.

[garycase]

Thanks for the details Johnnie.   Could you post a link to the thread where you posted the results for a variety of cards?  [I should have bookmarked it -- and definitely will this time  ]

 

Clevoir => Johnnie has done extensive testing with a variety of cards using an all-SSD array, so any speed limitations are due solely to the cards, so his results are very trustworthy.   As I noted above, the theoretical max speeds aren't often reached -- and in this case real world performance is well below that.  

 

[JorgeB]

 

[clevoir]

Just double checked my last parity check and it was faster than I remember @ 9 hours, 49 minutes, 58 seconds. Average speed: 84.8 MB/sec

 

With H310 / 1015 x 8 cards, what would I be looking at?

[garycase]

2 hours ago, clevoir said:

With H310 / 1015 x 8 cards, what would I be looking at?

 

You'd only be limited by the speed your slowest disk can sustain.    As you can see in Johnnie's post, the H310's can do 320MB/s on a gen 3 PCIe port, down to 185MB/s on a gen 1 port -- it's unlikely you have any disks that can sustain speeds greater than the slowest of these speeds (Modern high density disks can sustain a bit more than that on the outer cylinders, but not by much -- and the inner cylinder speeds won't be that high).

 

[DoeBoye]

2 hours ago, garycase said:

NO -- that's a deceptive statistic.    They CAN do 300MB/s on each channel, since they use a SATA-II interface; but the card can't sustain that for ALL channels at once due to the bandwidth restriction of the PCIe bus

 

Yeah. I didn't think they could *actually* do that :). I was just sharing the specs Supermicro was claiming.

 

Though, in the OP's original post, he also mentions 18 drives on 3 cards, so that should bump the *theoretical* speed of his current system to 166 MB/s per drive.

 

Assuming the performance was fairly linear, if we apply johnnie.black's real-world 80MB/s for a fully loaded card to a card with only 6 drives attached, that should have given him 106MB/s max speeds...

 

All that to say, the 60MB"s he's was seeing was not a shortcoming of the card - a point we all agree upon!

[clevoir]

I'd be gratefull if someone could advise me which are my slowest drives, see attached syslog.

 

I have noticed different SATA versions between disks on UnRAID, and that some disks don't report SATA versions at all, only ATA versions instead?

tower-diagnostics-20170308-1903.zip

[garycase]

2 minutes ago, clevoir said:

I'd be gratefull if someone could advise me which are my slowest drives, see attached syslog.

Easier to do with a screenshot of the Main display tab showing all the drives.

 

[JorgeB]

4 minutes ago, clevoir said:

I'd be gratefull if someone could advise me which are my slowest drives

 

Use this:

 

Nice graphical display of speed of all disks.

 

 

[clevoir]

See attached screenshot

[ashman70]

I'd start by replacing that 1.5TB Samsung drive and then all the 1TB drives. Also check your MB manual because I know that one of the members here caught a problem I was having with my H310 and slow parity checks, it was that the 8x slot I had it in was actually running at 4x speed, so make sure the 8x slot you have yours in is actually running at that speed, or if you have another free 8x slot or 16x slot, try it in those slots to see if that fixes the problem.

Edited March 8, 2017 by ashman70

[clevoir]

Looks like I have a few slow disks!