Slow transfer speeds over 10Gb network to an NVME drive


Recommended Posts

I recently upgraded my cache drive to a Samsung 960 evo NVME drive to take advantage of the 10 gigabit connection to my server.  Oddly enough I cannot saturate the connection with a drive that is capable of writing at around 2 GB/s.  I am only getting 400-500 MB/s on write to the server, but if I read that file back to my windows PC i get the full network speed (1.07 GB/s). Anyone know how i can boost this speed?

Link to comment
30 minutes ago, FreeMan said:

I'm just curious what you do on your network that you need 10GbE?  Of course, I'm assuming (with all inherent dangers) that this is a home server, but I can't fathom why you'd need that speed internally at home.

 

Not a criticism, just trying to learn.

 

 

I recently upgraded my server and a single system with a dac to 10gbe. I love the new speed. 10gbe for like $70, can't beat it. I do lots of file transfers back and forth. I watch movies and do lots of work directly from the array/shared folders. If I can do something that will save me 10 seconds, I will do that. You would really be surprised at how much it begins to add up.

 

I work in User Experience and when I worked at Microsoft we did several studies around efficiency and productivity. There were very simple things you could do that would increase your efficiency greatly. The most stark results were from using dual monitors, but also things like sitting/arms/desk height, size of the keyboard/mouse, disk speed (mainly access speed), network speed.

 

Tuning these very small details will change your experience and help you to accomplish more.

 

I think the most interesting thing about the studies we did were around people's attitudes before and after. Subtle changes to their environment created a more positive attitude, they "felt" like they were doing more, but couldn't explain why.

Link to comment
1 hour ago, RonUSMC said:

I work in User Experience and when I worked at Microsoft

If I find you you had anything to do with the "touchscreen interface for 24-inch desktop monitor" decision, I'll reach right through this monitor and strangle you!!! Ugh... not my favorite decision... :(

 

On a serious note, thanks for the feedback, that's very interesting. I can't agree more on the monitor side - I'm looking at 1 of my 4 on this machine right now. ;) Once I've upgraded my disk controller and taken care of some car upgrades and other items, I may give 10gig networking a thought.

Link to comment
Just now, FreeMan said:

If I find you you had anything to do with the "touchscreen interface for 24-inch desktop monitor" decision, I'll reach right through this monitor and strangle you!!! Ugh... not my favorite decision... :(

 

On a serious note, thanks for the feedback, that's very interesting. I can't agree more on the monitor side - I'm looking at 1 of my 4 on this machine right now. ;) Once I've upgraded my disk controller and taken care of some car upgrades and other items, I may give 10gig networking a thought.

 

I actually worked on the Surface platform for over a year helping push the touchscreens at Microsoft.

http://www.google.com.pg/patents/US20100241955

 

I didn't really get into the ordeal when they were porting all of our hard work directly into the touchscreen laptops, but I am very opinionated and wouldn't have allowed many of the things they produced in the years after I left. I think there were some seriously lazy decisions made about those interactions and a complete lack of regard for environment size. I like to think it would have been better if I was there during that time. I did end up going back and helping with the release of Microsoft Surface Studio, but left immediately after.

  • Like 1
Link to comment

To answer FreeMan's question, it is a home network.  I have a 48 port switch in my media cabinet because I have hardwired almost everything in my house and i also have POE security cameras that run to it.  It had built in SFP+ ports on it so I decided to put them to use.  I had the transfer speeds working properly (1.15 GB/s), but now its back down to 700MB/s and my plex and plexpy dockers won't start.  Here is the log file from Plex. It will then just repeat starting plex.

 

Error.txt

Edited by mattcoughlin
Link to comment

might be an issue, i don' use this 10gbe card, but it says you are getting a 20% loss, though should be way more available bandwidth....:

 

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.4.0-k
Sep 30 22:07:53 Unraid kernel: ixgbe: Copyright (c) 1999-2016 Intel Corporation.
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: PCI Express bandwidth of 32GT/s available
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: MAC: 2, PHY: 17, SFP+: 5, PBA No: G30771-004
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: 90:e2:ba:47:d8:26
Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
Sep 30 22:07:53 Unraid kernel: e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
Sep 30 22:07:53 Unraid kernel: e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
Sep 30 22:07:53 Unraid kernel: e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode

 

 

your docker problem is most likely related to your call trace errors

 

Quote

------------[ cut here ]------------
Sep 30 23:51:22 Unraid kernel: WARNING: CPU: 3 PID: 21906 at fs/btrfs/extent-tree.c:134 btrfs_put_block_group+0x42/0x59
Sep 30 23:51:22 Unraid kernel: Modules linked in: md_mod xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat e1000e ixgbe ptp pps_core mdio mxm_wmi x86_pkg_temp_thermal coretemp i2c_i801 i2c_smbus i2c_core ahci nvme libahci nvme_core wmi [last unloaded: md_mod]
Sep 30 23:51:22 Unraid kernel: CPU: 3 PID: 21906 Comm: umount Not tainted 4.9.30-unRAID #1
Sep 30 23:51:22 Unraid kernel: Hardware name: MSI MS-7885/X99A RAIDER (MS-7885), BIOS P.50 07/19/2016
Sep 30 23:51:22 Unraid kernel: ffffc90002ea3d20 ffffffff813a4a1b 0000000000000000 ffffffff819692b0
Sep 30 23:51:22 Unraid kernel: ffffc90002ea3d60 ffffffff8104d0d9 0000008602ea3d00 ffff880453348c00
Sep 30 23:51:22 Unraid kernel: ffff880453348ce8 ffff88044c34c090 ffff880453348c00 ffff88044c34c0a0
Sep 30 23:51:22 Unraid kernel: Call Trace:
Sep 30 23:51:22 Unraid kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Sep 30 23:51:22 Unraid kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Sep 30 23:51:22 Unraid kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Sep 30 23:51:22 Unraid kernel: [<ffffffff812d6c15>] btrfs_put_block_group+0x42/0x59
Sep 30 23:51:22 Unraid kernel: [<ffffffff812dcf26>] btrfs_free_block_groups+0x19d/0x39b
Sep 30 23:51:22 Unraid kernel: [<ffffffff812ec0f4>] close_ctree+0x1d4/0x2e8
Sep 30 23:51:22 Unraid kernel: [<ffffffff81137ff1>] ? evict_inodes+0x128/0x137
Sep 30 23:51:22 Unraid kernel: [<ffffffff812c6f4b>] btrfs_put_super+0x14/0x16
Sep 30 23:51:22 Unraid kernel: [<ffffffff811232fb>] generic_shutdown_super+0x6a/0xeb
Sep 30 23:51:22 Unraid kernel: [<ffffffff8112355e>] kill_anon_super+0xd/0x17
Sep 30 23:51:22 Unraid kernel: [<ffffffff812c6e92>] btrfs_kill_super+0x11/0x94
Sep 30 23:51:22 Unraid kernel: [<ffffffff81123750>] deactivate_locked_super+0x3b/0x68
Sep 30 23:51:22 Unraid kernel: [<ffffffff811240ec>] deactivate_super+0x39/0x3c
Sep 30 23:51:22 Unraid kernel: [<ffffffff8113b02e>] cleanup_mnt+0x53/0x71
Sep 30 23:51:22 Unraid kernel: [<ffffffff8113b080>] __cleanup_mnt+0xd/0xf
Sep 30 23:51:22 Unraid kernel: [<ffffffff81062609>] task_work_run+0x6a/0x7d
Sep 30 23:51:22 Unraid kernel: [<ffffffff81002b24>] exit_to_usermode_loop+0x55/0x81
Sep 30 23:51:22 Unraid kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47
Sep 30 23:51:22 Unraid kernel: [<ffffffff8167f5c4>] entry_SYSCALL_64_fastpath+0xa7/0xa9
Sep 30 23:51:22 Unraid kernel: ---[ end trace 196aa00775cb4f80 ]---
Sep 30 23:51:22 Unraid kernel: ------------[ cut here ]------------
Sep 30 23:51:22 Unraid kernel: WARNING: CPU: 3 PID: 21906 at fs/btrfs/extent-tree.c:10045 btrfs_free_block_groups+0x37a/0x39b
Sep 30 23:51:22 Unraid kernel: Modules linked in: md_mod xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat e1000e ixgbe ptp pps_core mdio mxm_wmi x86_pkg_temp_thermal coretemp i2c_i801 i2c_smbus i2c_core ahci nvme libahci nvme_core wmi [last unloaded: md_mod]
Sep 30 23:51:22 Unraid kernel: CPU: 3 PID: 21906 Comm: umount Tainted: G        W       4.9.30-unRAID #1
Sep 30 23:51:22 Unraid kernel: Hardware name: MSI MS-7885/X99A RAIDER (MS-7885), BIOS P.50 07/19/2016
Sep 30 23:51:22 Unraid kernel: ffffc90002ea3d38 ffffffff813a4a1b 0000000000000000 ffffffff819692b0
Sep 30 23:51:22 Unraid kernel: ffffc90002ea3d78 ffffffff8104d0d9 0000273d02ea3d40 ffff88044c34c000
Sep 30 23:51:22 Unraid kernel: ffff880453348888 ffff880453348800 ffff88044c34cb30 0000000000000000
Sep 30 23:51:22 Unraid kernel: Call Trace:
Sep 30 23:51:22 Unraid kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Sep 30 23:51:22 Unraid kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Sep 30 23:51:22 Unraid kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Sep 30 23:51:22 Unraid kernel: [<ffffffff812dd103>] btrfs_free_block_groups+0x37a/0x39b
Sep 30 23:51:22 Unraid kernel: [<ffffffff812ec0f4>] close_ctree+0x1d4/0x2e8
Sep 30 23:51:22 Unraid kernel: [<ffffffff81137ff1>] ? evict_inodes+0x128/0x137
Sep 30 23:51:22 Unraid kernel: [<ffffffff812c6f4b>] btrfs_put_super+0x14/0x16
Sep 30 23:51:22 Unraid kernel: [<ffffffff811232fb>] generic_shutdown_super+0x6a/0xeb
Sep 30 23:51:22 Unraid kernel: [<ffffffff8112355e>] kill_anon_super+0xd/0x17
Sep 30 23:51:22 Unraid kernel: [<ffffffff812c6e92>] btrfs_kill_super+0x11/0x94
Sep 30 23:51:22 Unraid kernel: [<ffffffff81123750>] deactivate_locked_super+0x3b/0x68
Sep 30 23:51:22 Unraid kernel: [<ffffffff811240ec>] deactivate_super+0x39/0x3c
Sep 30 23:51:22 Unraid kernel: [<ffffffff8113b02e>] cleanup_mnt+0x53/0x71
Sep 30 23:51:22 Unraid kernel: [<ffffffff8113b080>] __cleanup_mnt+0xd/0xf
Sep 30 23:51:22 Unraid kernel: [<ffffffff81062609>] task_work_run+0x6a/0x7d
Sep 30 23:51:22 Unraid kernel: [<ffffffff81002b24>] exit_to_usermode_loop+0x55/0x81
Sep 30 23:51:22 Unraid kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47
Sep 30 23:51:22 Unraid kernel: [<ffffffff8167f5c4>] entry_SYSCALL_64_fastpath+0xa7/0xa9
Sep 30 23:51:22 Unraid kernel: ---[ end trace 196aa00775cb4f81 ]---

 

and some other fun disk errors

 

Oct  1 00:00:12 Unraid kernel: ata10.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen
Oct  1 00:00:12 Unraid kernel: ata10.00: irq_stat 0x08000000, interface fatal error
Oct  1 00:00:12 Unraid kernel: ata10: SError: { UnrecovData 10B8B BadCRC }
Oct  1 00:00:12 Unraid kernel: ata10.00: failed command: READ DMA EXT
Oct  1 00:00:12 Unraid kernel: ata10.00: cmd 25/00:40:a8:45:02/00:05:00:00:00/e0 tag 1 dma 688128 in
Oct  1 00:00:12 Unraid kernel:         res 50/00:00:a8:45:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)
Oct  1 00:00:12 Unraid kernel: ata10.00: status: { DRDY }
Oct  1 00:00:12 Unraid kernel: ata10: hard resetting link
Oct  1 00:00:13 Unraid kernel: ata10: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct  1 00:00:13 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:13 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:13 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:13 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:13 Unraid kernel: ata10.00: configured for UDMA/133
Oct  1 00:00:13 Unraid kernel: ata10: EH complete
Oct  1 00:00:13 Unraid kernel: ata10.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen
Oct  1 00:00:13 Unraid kernel: ata10.00: irq_stat 0x08000000, interface fatal error
Oct  1 00:00:13 Unraid kernel: ata10: SError: { UnrecovData 10B8B BadCRC }
Oct  1 00:00:13 Unraid kernel: ata10.00: failed command: READ DMA EXT
Oct  1 00:00:13 Unraid kernel: ata10.00: cmd 25/00:70:b8:54:02/00:04:00:00:00/e0 tag 7 dma 581632 in
Oct  1 00:00:13 Unraid kernel:         res 50/00:00:b8:54:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)
Oct  1 00:00:13 Unraid kernel: ata10.00: status: { DRDY }
Oct  1 00:00:13 Unraid kernel: ata10: hard resetting link
Oct  1 00:00:13 Unraid kernel: ata10: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct  1 00:00:13 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:13 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:13 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:13 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:13 Unraid kernel: ata10.00: configured for UDMA/133
Oct  1 00:00:13 Unraid kernel: ata10: EH complete
Oct  1 00:00:13 Unraid kernel: ata10.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen
Oct  1 00:00:13 Unraid kernel: ata10.00: irq_stat 0x08000000, interface fatal error
Oct  1 00:00:13 Unraid kernel: ata10: SError: { UnrecovData 10B8B BadCRC }
Oct  1 00:00:13 Unraid kernel: ata10.00: failed command: READ DMA EXT
Oct  1 00:00:13 Unraid kernel: ata10.00: cmd 25/00:40:f0:f4:02/00:05:00:00:00/e0 tag 11 dma 688128 in
Oct  1 00:00:13 Unraid kernel:         res 50/00:00:f0:f4:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)
Oct  1 00:00:13 Unraid kernel: ata10.00: status: { DRDY }
Oct  1 00:00:13 Unraid kernel: ata10: hard resetting link
Oct  1 00:00:14 Unraid kernel: ata10: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct  1 00:00:14 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:14 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:14 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:14 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:14 Unraid kernel: ata10.00: configured for UDMA/133
Oct  1 00:00:14 Unraid kernel: ata10: EH complete
Oct  1 00:00:15 Unraid kernel: ata10: limiting SATA link speed to 3.0 Gbps
Oct  1 00:00:15 Unraid kernel: ata10.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen
Oct  1 00:00:15 Unraid kernel: ata10.00: irq_stat 0x08000000, interface fatal error
Oct  1 00:00:15 Unraid kernel: ata10: SError: { UnrecovData 10B8B BadCRC }
Oct  1 00:00:15 Unraid kernel: ata10.00: failed command: READ DMA EXT
Oct  1 00:00:15 Unraid kernel: ata10.00: cmd 25/00:40:d0:f8:04/00:05:00:00:00/e0 tag 21 dma 688128 in
Oct  1 00:00:15 Unraid kernel:         res 50/00:00:d0:f8:04/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)
Oct  1 00:00:15 Unraid kernel: ata10.00: status: { DRDY }
Oct  1 00:00:15 Unraid kernel: ata10: hard resetting link
Oct  1 00:00:15 Unraid kernel: ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Oct  1 00:00:15 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:15 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:15 Unraid kernel: ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20160831/psargs-359)
Oct  1 00:00:15 Unraid kernel: ACPI Error: Method parse/execution failed [\_SB.PCI0.SAT1.SPT5._GTF] (Node ffff8804af1672d0), AE_NOT_FOUND (20160831/psparse-543)
Oct  1 00:00:15 Unraid kernel: ata10.00: configured for UDMA/133

 

 

 

Link to comment
5 hours ago, 1812 said:

might be an issue, i don' use this 10gbe card, but it says you are getting a 20% loss, though should be way more available bandwidth....:

 


Sep 30 22:07:53 Unraid kernel: ixgbe 0000:03:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

 

 

This is normal and it affects all PCIe 1.0 and 2.0 links due to 8b/10b encoding, PCIe 3.0 upgrades the encoding to 128b/130b.

Link to comment

Ok.  

 

Has anyone ever tried reformatting the unraid USB Boot drive and readding the data drives? I have had a few random issues with things like windows 10 vms not booting or crashplan (when i used to have it) getting into a reboot cycle every 15 minutes.  My thought is to just start the drive fresh if i can reassign the drives without loosing data.

 

Link to comment

I am having the same issues here. Can't saturate 10gbe at about 350mb/s. Direct I/O is giving me call traces and all docker apps had to be changed from mnt/usr/appdata to mnt/cache/appdata for them to be able to read/write. To top it off i am still only getting 350mb over 10gbe! I am out of ideas.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.