unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables


Recommended Posts

Sorry for breaking in, not my intention to hijack this thread, but I may have something of interest to you guys while you are running your performance tests.

 

I have made a update to the GUI which allows real-time monitoring of disk read and write performance. For the moment it is just a test and available as a plugin, so it can be uninstalled easily.  ;)

 

 

Awesome, I love it, thanks for sharing!

 

At first glance, once of the things I noticed is that the individual disk speeds are lower than the parity check speed, and more closely match what I've been reporting in my tool.

 

For example, the parity check speed is reporting 148.6 MB/s, but the individual drives are fluctuating from low 130's to a high of 147 MB/s.

 

Hah! For a second there I saw disk speeds of 285 MB/s!!!  Must have caught a momentary burst at just the right moment.

 

bonienl, any concern that monitoring the speeds will impact performance?

 

What is the timeframe over which the speed is calculated?  Can it be configured to average over a longer time span?

 

Thanks!

-Paul

Link to comment

I have made a update to the GUI which allows real-time monitoring of disk read and write performance. For the moment it is just a test and available as a plugin, so it can be uninstalled easily.  ;)

 

Thanks bonienl, looks cool, and certainly more useful than the reads/writes numbers that don't mean much.

 

I also see some heavy fluctuating speeds, from ~170MB/s to over 400MB/s on my test server, maybe a higher sample time, if possible, will provide more accurate readings.

 

Any way would could add another toggle to show the total bytes read/written (reset when pressing clear statistics)? This could also be useful.

 

P.S.: it doesn't work on my cache disk, is this expected?

 

 

Link to comment

Last night I ran a parity check using the same settings from last time, but instead of sync_thresh=sync_window-1, I used sync_window-64, which my recent testing showed was a few MB/s faster.

 

Before:  2016-08-21, 17:16:08 7 hr, 40 min, 37 sec 108.6 MB/s OK

 

After:    2016-09-01, 09:35:19 7 hr, 35 min, 18 sec 109.8 MB/s OK

 

This change resulted in another 5 minutes saved.  The server is now within 15 minutes of the fastest parity check I ever accomplished on 5.0, back when I had single parity. 

 

I don't know if the dual parity is adding 15 minutes, or if there is still performance left to be discovered.  Not willing to go back to single parity to find out.

 

I'm working on a new version of the tunables tester with the recently discussed logic changes.

 

-Paul

Link to comment

If you are interested I have made another update of the Dynamix Disk IO plugin with improvements and corrections. It is available thru the plugin manager.

 

@bonienl

 

Just a FYI: While trying to update this plugin I get the following:

 

plugin: updating: dynamix.disk.io.plg
plugin: downloading: https://raw.githubusercontent.com/bergware/dynamix/master/archive/dynamix.disk.io.txz ... done
plugin: bad file MD5: /boot/config/plugins/dynamix.disk.io/dynamix.disk.io.txz

Link to comment

If you are interested I have made another update of the Dynamix Disk IO plugin with improvements and corrections. It is available thru the plugin manager.

 

@bonienl

 

Just a FYI: While trying to update this plugin I get the following:

 

plugin: updating: dynamix.disk.io.plg
plugin: downloading: https://raw.githubusercontent.com/bergware/dynamix/master/archive/dynamix.disk.io.txz ... done
plugin: bad file MD5: /boot/config/plugins/dynamix.disk.io/dynamix.disk.io.txz

 

I pushed a new version, you need to do a manual "check for updates" and then install. Should work!

 

Link to comment

I pushed a new version, you need to do a manual "check for updates" and then install. Should work!

 

Working now as of version 2016.09.03.

 

Thanks a million.

 

Great, please let me know if you observe anything strange/unusual/unexplainable. I am currently putting this through testing (so far all is looking good at my side).

 

Link to comment

I have made a update to the GUI which allows real-time monitoring of disk read and write performance. For the moment it is just a test and available as a plugin, so it can be uninstalled easily.  ;)

 

I too have been enjoying this tool, has helped with some testing I've been doing.  Thanks!

 

We're getting off-topic though, perhaps it deserves its own thread?

Link to comment

I have made a update to the GUI which allows real-time monitoring of disk read and write performance. For the moment it is just a test and available as a plugin, so it can be uninstalled easily.  ;)

 

I too have been enjoying this tool, has helped with some testing I've been doing.  Thanks!

 

We're getting off-topic though, perhaps it deserves its own thread?

 

Running the latest version - the Total column looks more like an average to me.  I would have expected a sum?

Link to comment

It looks like  I got a little but of a speed bump.

 

2/2/2016 13:05	45340	88.2 MB/s	0
3/1/2016 12:44	43864	91.2 MB/s	0
4/5/2016 12:47	44275	90.4 MB/s	0
6/7/2016 13:28	46698	85.7 MB/s	0
7/5/2016 13:33	46998	85.1 MB/s	0
2016 Sep  6 14:30:40	39639	100.9 MB/s	0

 

Looks like it shaved a couple hours off.  Nice.

 

I'm curious why your elapsed time is in seconds and not hours/minutes/seconds.

 

I'm still working on new testing logic.

Link to comment

It looks like  I got a little but of a speed bump.

 

2/2/2016 13:05	45340	88.2 MB/s	0
3/1/2016 12:44	43864	91.2 MB/s	0
4/5/2016 12:47	44275	90.4 MB/s	0
6/7/2016 13:28	46698	85.7 MB/s	0
7/5/2016 13:33	46998	85.1 MB/s	0
2016 Sep  6 14:30:40	39639	100.9 MB/s	0

 

Looks like it shaved a couple hours off.  Nice.

 

I'm curious why your elapsed time is in seconds and not hours/minutes/seconds.

 

I'm still working on new testing logic.

 

I pulled that straight from the log file in the config dir.

Link to comment
  • 2 weeks later...

Pauven, I've posted in the 6.2 Final announcement thread (here) about the dramatic performance boost I got with a change of a single tunable!  (By the way, thank you Johnnie!)  What's particularly important about it is that users upgrading to 6.2 will have the default md_sync_thresh value of 192.  If they have previously raised their tunables, perhaps by using the unraid-tunables-tester, they will probably have terrible performance, as I did.  I suspect you are about to see quite a bit more 'clientele' around here.

 

Another thing, I think finding the best relations between the tunables will be just as important as finding the best numbers.  They can be part of the guidelines posted for future users.

Link to comment

After a QUICK scan it appears that Tuneables applies mostly to optimizing Parity Check speeds... It would be nice to tune my 6.2 dual parity check below the 10 hour mark again (now at 15.5 hours)....

 

it would be even better if I can get writes to the SSD 850 Pro 250 GB Cache back up to the v5 ~100 MB/s range (currently at 57 MB/s). 

1 - Are the tunables applied to the Cache to increase write performance as well?

 

2 - May I run the current tunables on v6.2 or should I await a release announcement on an updated script?

 

Thank you!

Link to comment

Since I always found this script to be of immense use, here is the modified script that will work under 6.2 and under 6.1.9.

 

Only difference for 6.2 compatibility is that md_write_limit has been removed from 6.2, so the adjustments that the script makes as it runs have been removed.  (Note that it will still display the md_write_limit values in the summary)  (it was setting the md_write_limit that would crash the script on 6.2 even with changing the location of mdcmd within the original script)

 

And you should really read Pauven's comments a couple posts up (http://lime-technology.com/forum/index.php?topic=29009.msg424206#msg424206)  This script is not the end-all-be-all, but can help.  It did for me back in the day, but now my system works fast enough using LT's default values)

 

Oh crud.... I installed the OP .sh file and I am running that now in Screen on v6.2

Did I screw myself... what should I do???

 

edit:  stopped array, restarted the server.

 

So... do I install the .sh from the OP or Squids .sh?  I am running v 6.2

 

Thanks

Link to comment

Until pauven releases his 6.2 revamp you would install my hack to make the original script work under 6.2.  Ymmv however with using the hack as it does not take into consideration the new tunable that 6.2 offers.  (But it won't mess up your system)

 

Sent from my SM-T560NU using Tapatalk

 

Link to comment

Rather impressive results with the tunables so far.

 

With 6.2 Dual parity I can again read access files during the parity check with minimal disruption. 

The time for the parity check has been shaved down a bit.

 

I use the Supermicro SAS2LP for my Data drives... but the motherboard for my Cache and Parity Drives.... Not certain if I need to play with the following with this setup:

This tunable was added to improve the parity check performance some users were having with the SAS2LP, IIRC, you controller uses the same chipset, basically the SASLP and SAS2LP work faster if this value is approximately half the md_sync_window value, other controllers like most LSI based ones, work faster with a md_sync_threash just a little lower than md_sync_window.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.