Server responsiveness decreased dramatically during file transfers

[-Daedalus]

Hi all,

 

Not really sure where to put this, as I haven't been able to narrow down the root cause yet.

 

I've noticed that when a download completes (Deluge, Transmission docker instance, etc.) if the file is big enough (usually a couple of gigs or more will do it) the whole server hangs.

 

The webUI mostly works, but slows to a crawl.

Plex streams will break.

Locally streaming video will become a buffering mess.

Other VMs and docker instances will have their performance impacted.

 

While this is happening, this is what the stats look like:

 

From the image above, it looks very much like somewhere along the line files are being written to the array, despite all shares being "Cache: Yes".

 

The shares:

Downloads go to: user/downloads/incomplete

Once completed, Sonarr/CP move to: user/downloads/tv or user/downloads/movies

Then these are moved to: user/tv or user/movies, for Plex to pick up.

 

It's hard to tell where in the process the bottleneck is happening, but as stated, all are using cache, so AFAIK, this shouldn't happen. Anyone have any ideas? It's irritating when users report problems on a Minecraft server just because a download is finishing up.

 

Diagnostics attached. Thanks in advance!

 

server-diagnostics-20170820-1114.zip

Edited August 20, 2017 by -Daedalus

[itimpi]

Setting a share to Cache=Yes means that files are intended to end up on the array disks and mover is meant to move such files from the cache to the array!   Also with such a setting the array disks may need spinning up to check if any new file already exists which can cause short hiccups on accessing array disks.   If you want the shares to only ever be on the cache then they should be set to Cache=Only or Cache=Prefer).   Turn on the help to get a good description of what the various cache settings actually do.

[-Daedalus]

I'm aware of the mover, however this didn't happen during a time when the mover was active, and invoking the mover manually doesn't cause this issue.  Also, the issue lasts for several minutes to an hour or so, depending on the size of the files in question.

[itimpi]

If a file already exists on an array disk then that copy will be overwritten so that can also cause slowdowns.

 

you probably want to ensure that any shares you want to exist only on the cache are first set to Cache=Prefer and then run mover to make sure any files that exist for the share on the array drives are moved to the cache drive.    When that completes set them to Cache=only to make sure new files cannot be created on the array drives.

 

if you still get slowdowns after that I do not know what to suggest.

[-Daedalus]

I don't want the shares only on the cache. They're just downloaded junk, so I want it ultimately on the array, but I'd rather that happen at night, hence the standard cache=yes behaviour is desired. This happens daily, so manual intervention isn't feasible.

 

The diagnostics showing anyone anything useful?