[SOLVED] Write speed dropped from 100 Mbps to 1 Mbps - UGH


Recommended Posts

Strange results like this can be very difficult to isolate ... but I do recall a somewhat similar issue I saw 2-3 years ago, when a single bad drive was apparently loading the SATA bus due to a weird failure mode of its controller board.    This was on a system with about 10 disks; and I isolated the specific issue by disconnecting ALL of the drives and then, one-at-a-time, connecting a single drive; doing a New Config with ONLY that drive (no parity); and then confirming I could read the drive okay and had good transfer speeds.  [i didn't do any writes so none of the disks would actually be changed]    I found one disk that, while readable, had exceptionally low transfer speeds => and then redid the array with all disks EXCEPT that one and everything was fine.    I also found that if that disk was simply connected to a SATA port (even if not part of the array), the problem re-occurred.

 

Link to comment
  • 4 months later...

Update:

 

First, thanks, @garycase for the suggestions!

 

1. Today I ran a New Config and brought a drive back one at a time, copying a handful of 3-5GB .mkv files to each. I ended up failing with the same Windows error on my 9th (of 11) and only that drive. I'm currently running a Parity-Sync and will try to replace the drive and see what results it yields, but so far a VERY positive lead.

 

2. A couple less important, but still interesting notes:

 

2.a. My copies, to whichever drive/drive combos/shares, would typically fail and produce the previously-reported Windows error (or a Linux fail, via mc) when copying 3+ GB .mkv or .iso files. I didn't have quite the same failure rate with .jpgs, though it was still present. Additionally, yesterday I copied a 100 GB file and - while transfer speeds ranged from 10-100 MB/sec (hovering mostly closer to 10 MB/sec) - the file did not fail once. I thought that was slightly interesting and perhaps may coincide with my next bullet.

 

2.b. When copying .mkv files, for instance, I noticed I zip through around 100 MB/sec for most of the file, but my file transfer stalls and bottoms down to 0 MB/sec at the beginning and end of pretty much each file. Thoughts?

 

Regardless, thanks @garycase/all...more to follow...

Link to comment
  • 2 weeks later...

@limetech @garycase @johnnie.black @Rajahal @jonp anyone...Help Please!

 

I replaced the 3 TB that I thought would resolve this, per the above and I'm producing the same copying error. For giggles I tried connecting from a Win 7 instead of 10 (nope), directly from a PC to the server (nope), and switched out the SATA port of the controller card from that disk (nope). While unRAID is an AMAZING solution and serving my real desire for it (reliable backup solution), my unRAID server is essentially unusable now.

 

 

Link to comment

Try googling    Windows error  0x8007003b   and look at some of the results.  (I have had mixed results when looking up windows error codes as there are a few of them that are simply fired out to the unsuspecting user when Windows does not have a clue about what the problem truly is...)  But you may get lucky and find a solution that will work for you.  Network transfer problems are always a bear to find a solution to as there are so many different components, software and OS's involved in the process.

Link to comment
1 hour ago, jcarmi04 said:

Thanks for the crack @Frank1940 however I've definitely made those rounds...and, unfortunately, I've timed out before when copying from disk to disk via mc (so thinking something solely server related).

This puts a different light on things.  I would suggest that you post a complete list of EVERYTHING that you have installed on and in your server --- Both hardware and software.  You might also post up a new diagnostic file after you have had a episode of slow disk-to-disk operation. 

Link to comment

Well, I have no idea what's going on...

 

I just created a couple single disk shares (d1 is to the M/B (Disk 1), d9 and d10 are to the SuperMicro card (Disks 9 and 10)). 

 

1. Win10 multi-disk share (Disks 1, 5, 8 (M/B), 10 and 11 (SM)) to "d1": SUCCESS
2. Win10 "d1" to "d9": FAIL
3. Win10 "d1" to "d10": SUCCESS
4. MC "d1" to "d9": SUCCESS
5. Win10 "d1" to "d9: SUCCESS
6. Win10 "d1" to the original multi-disk share: FAIL

 

The speeds aren't fast, but right now I'm going for operational. Log attached and server details below.

 

2x 5T (Toshiba Parity and disk)
2x 2T (Hitachi)
2x 3T (Seagate)
4x 2T (WD)
1x 3T (Toshiba)
1x 3T (WD)
1x 250G (Crucial Cache)
1x 4G (JD FireFly Flash)
1x ASRock Z87 Extreme M/B
1x Intel I5-4570 @ 3.2GHz CPU
4x 8G Kingston RAM
1x Cosair 650w PSU
2x AMD Radeon HD Video Cards (VMs)
1x SuperMicro AOC-USAS2-L8i (8 port)


unRAID Server Pro v6.1.9
HVM: Enabled
IOMMU: Enabled

Docker: Installed
Docker Containers:
-BTSync (gfjardim/btsync:latest) (Config using Cache and data stored to a Share)
-Plex (limetech/plex:latest) (Config using Cache and data accessed via Shares)

VMs:
-1x Win7 (Using Cache only)
-1x Win10 (Using Cache only)

tower118-diagnostics-20170617-0643.zip

Link to comment

You should NEVER make a copy from a Disk share to a User share (unless you know about this problem and the disk share is not in the user share).  See here:

EDIT: ... unless you know about this problem and the disk share is not in the user share and the disk has never been in the user share ... (The reason being that is if a disk is removed from a user share the files in that share that have been stored on that disk are NOT moved. However, no new files that are written to the share will not be stored on that disk. )

Edited by Frank1940
Link to comment

What version of unRaid are you on? The web guI lists the filesystem. The method of preclear is unrelated to the file system selection. Let's see if this is the problem before thinking about converting anything.

 

But speaking of preclear, when was the last time you precleared a disk? Did you have any performance problem doing so?

Link to comment

If all your performance issues are with writing, and not with reading, this is very likely the cause. Write issues with RFS get worse as the disk fills. So if you want to test on your least full disk, and see the problems don't happen on it, it might help convince that this is indeed the issue. But if all of your disks are well over half full, this won't be very helpful.

 

But I'm pretty convinced, and it needs to be done regardless. Very early in the process we'll know for sure.

 

To begin you need a new disk, big enough to hold the data on one of your largest disks. Often this means buying one new disk. The disk needs to be precleared and then added to the array. If you have no room to add a new disk or would prefer to avoid buying a new one, but you have sufficient space somewhere (on workstations, laptops, external USB drives, or some combination), to copy all the data off of any of the largest sized disks in your array, you can use that technique. (Don't need to delete the files! A delete is a write and very slow on RFS.)  The key is you need one big disk completely backed up elsewhere, in your array. I don't care what is on it. Whether newly precleared or existing, you'll then reformat it to xfs, and then copy data from your biggest fullest disk to it. You can start with just a few files. If the performance is good, we're on the right track.

Link to comment

@bjp999 I do have some slow access times (seems like the server is getting choked...but no reason it should), but I think my main faults result in writes. Since all of my disks are, in fact, over half full should I proceed as follows for testing:

 

1. Add new RFS-formatted drive to the array (I think I have a 1, 2, and 3T available I could use for testing)

2. Copy files to the drive and watch performance

3. Add new XFS-formatted drive to the array (will have to purchase a 5T)

4. Copy files to the drive and watch performance

 

I'm trying to understand the relevance of a RFS over half full and whether to include steps 1 and 2 or to exclude. Also, I can format a 1, 2, or 3T as XFS and replace any steps above or include (new steps 3 and 4, bumping the others to 5 and 6). *My largest drive is a 5T Parity and Disk 4 is also 5T.

Link to comment

I have lots of ideas of what I would do, but would take a while to explain and I'm heading to church and then father's day festivities. 

 

With your 2T and 3T drives, it is possible to do this without purchasing a new drive. Preclearing a disk does not make it RFS. The formatting happens in unRaid. You can change the global setting for filesystem from rfs to xfs, or change it on a specific disk.

 

Will post back later.

Link to comment

To convert from the  RFS to  XFS, there is a an excellent WIKI and support thread.  See here:

     and

 

https://wiki.lime-technology.com/File_System_Conversion#Mirroring_procedure_to_convert_drives

 

I used this when I convert both of my servers and all went very smoothly.  It will take several days to make the conversions but the actual data copying will run completely unattended.  If you have never used the command line interface, you need to play with the up-down-left-right arrow keys on your keyboard.  You can easily edit any previously command entered as the delete and backspace keys also work as you would expect them to.     I would suggest using these two commands as a start:

  

ls -al /boot      and    ls -al /boot/config   

 

You will find that the shell program for the command line used with unRAID has some very powerful editing tools that are built in.  (I am providing these suggestions as I don't know what you current knowledge and level of comfort is with the command line.  I apologize if you are  already familiar with them.) 

 

Edited by Frank1940
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.