Drive performance testing (version 2.6.5) for UNRAID 5 thru 6.4

[jbartlett]

3 hours ago, bjp999 said:

What tool did you use to product this output?

 

Same as the existing diskspeed bash script, using the dd command. It'll be used to read the drives, piping the output to the bit bucket, while logging it's ongoing process to a file. Other background processes will be scanning these log files and recording the progress since the dd output is done in such a way that each progress update overwrites the previous update and only the last one is preserved.

 

The dd command will be optimized to read the data in blocks of twice the physical sector size to maximize speed while minimizing the memory footprint.

 

 

[SSD]

1 minute ago, jbartlett said:

 

Same as the existing diskspeed bash script, using the dd command. It'll be used to read the drives, piping the output to the bit bucket, while logging it's ongoing process to a file. Other background processes will be scanning these log files and recording the progress since the dd output is done in such a way that each progress update overwrites the previous update and only the last one is preserved.

 

The dd command will be optimized to read the data in blocks of twice the physical sector size to maximize speed while minimizing the memory footprint.

 

Does that include those attractive graphics for each drive? I don't remember seeing it in the diskspeed output.

[jbartlett]

Just now, bjp999 said:

 

Does that include those attractive graphics for each drive? I don't remember seeing it in the diskspeed output.

 

This is a totally new version, complete re-write, and adding all the things I'd like to see. As mentioned, the logic driving it all during development is ColdFusion app code - like PHP but a different language. The ColdFusion code will be spawning jobs to execute the unix commands to get the system information, reading system files for controller/drive information, etc. to gather the information needed to display the web site as seen in the screen shots. I don't know PHP well enough to make this happen from scratch, but I know PHP enough to port a final product in ColdFusion to PHP.

 

I attached a dump of the variable created after scanning my backup UNRAID server for controllers, drives, and linking everything together including which drive is plugged into which physical controller port. This is the data driving everything the program does.

StorageInfo.html

[jbartlett]

I'll be creating a new thread for it once it's ready for wide-testing.

[jbartlett]

1 hour ago, bjp999 said:

 

Does that include those attractive graphics for each drive? I don't remember seeing it in the diskspeed output.

 

Not in the html dump file. I use the information gathered on the hard drive to programmatically figure out which drive image to display and if I should overlay the drive size over the image (as viewed on the WD & Sandisk drives. If I could find a specific image for a drive, I used that, otherwise a universal image.

 

For example, I have 11 images to represent 93 WD models in my database as they can be broken down into black, blue, gold, green, purple, brown, & red models. However, Seagate has media available for not only drive models but capacity as well so I have 88 images to cover 211 models.

 

The files will be stored in the plugin distribution but the app will be able to download drive images added later but not yet included in the package. So far, all drive images contribute only 5MB of space.

[jbartlett]

I've learned something new about hard drives which I'm incorporating into the rewrite/new version. Different drives have a different optimal block size to maximize speeds. The old script based version uses a block size of 1GB or or 1MB depending if you're running the full test or the fast version. Based on the physical drive geometry, this could under-report your drive speeds - though the 1GB size seems to maximize in the small sample set I tried but costly in RAM consumption as it required 1GB to be free.

 

The new version will test the drives to identify the optimal block size per drive for sustained reads by testing block sizes of 512B, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1M, 2M, 4M, 8M, 16M, 32M, and 64M. Each block size is allowed to read full bore for 5 seconds starting at the 2% mark of the drive and the block size that has the highest average read speed for each of those five seconds will be used for the benchmarking tests.

 

This does mean that in order to properly benchmark the drives, it will require a one-time check lasting 90 seconds per drive but the process can be stopped & resumed.

[SSD]

Maybe you could remember the benchmarking to avoid having to do over and over on successive runs?

 

I.wonder if the optimal block size might be in increments of the drive cache size. If so, you might want to include 128k and even 256k.

 

Might want a command line option to pick a single block size.

[jbartlett]

Sorry, I thought I had in there that it would save the results on the flash drive, probably removed it while proofing and forgot to add it back. 128 & 256k are in the test range, it starts at 512 bytes and doubles each test until 64 megs.

[SSD]

Sorry - meant 128M and 256M.

[jbartlett]

Interesting idea but I use the dd command with "iflag=direct" to bypass the drive cache which can skew the results.

[jbartlett]

Logic Change.

I'm going to change it so it tests each block size for two seconds - and the top three speeds on second #2 will be tested for 5 seconds. The block size that pushed the most data will be used.

 

This will take the time needed to test a drive from 90 seconds to 51. A system with ten drives will take 8 minutes and 30 seconds instead of 15 minutes.

[Videodr0me]

On 18.5.2017 at 5:52 AM, jbartlett said:

That's in the plan for the next version.

 

You said this in reply to concurrent read speed test allowing to see whether the sata controller or pcie bus throttles when reading in parallel from a number of drives. Please, please, please confirm that this is still planned....

[jbartlett]

11 hours ago, Videodr0me said:

You said this in reply to concurrent read speed test allowing to see whether the sata controller or pcie bus throttles when reading in parallel from a number of drives. Please, please, please confirm that this is still planned....

 

It is being added. Benchmarking a controller by reading 1 drive and then adding additional drives in sequence is mostly done. It was at this point that I discovered the bit about optimizing drive reads by block size and I've been focused on that which will enhance the controller/bus tests.

[Videodr0me]

4 hours ago, jbartlett said:

 

It is being added. Benchmarking a controller by reading 1 drive and then adding additional drives in sequence is mostly done. It was at this point that I discovered the bit about optimizing drive reads by block size and I've been focused on that which will enhance the controller/bus tests.

 

Fantastic. Ideally one should be able to chose the drives and the sequence in which they are added to the load. Also one should maybe allow to choose between reading large chunks (exceeding any internal drive cache) from a mid/low/high-speed section of the disks, as well as reading the same ultra short chunk over and over again which would most likely fit in any disk cache, mainly in order to test sata/pcie saturation.

 

Edited July 19, 2017 by Videodr0me

[jbartlett]

I may add cache testing in the future but the initial release is testing physical drive reads to read as fast as possible and bypassing the drive cache.

 

ETA: The current purpose of the app is not only to test how fast the drives are but to look for & identify slow spots in reading the surface media which could indicate a failing platter.

Edited July 19, 2017 by jbartlett

[jbartlett]

After pondering this, I see the merits of what you're asking but after some testing with a 1M direct read in a loop, I didn't see any remarkable speeds trying to utilize the drive cache without the system RAM cache - that is, it read slower.

[Videodr0me]

On 20.7.2017 at 5:09 AM, jbartlett said:

After pondering this, I see the merits of what you're asking but after some testing with a 1M direct read in a loop, I didn't see any remarkable speeds trying to utilize the drive cache without the system RAM cache - that is, it read slower.

This was to be expected. Data cached in RAM is read faster than data cached in the HDs internal Cache, which is in turn faster than data on the physical platters of the HD. The purpose was not to reach remarkable speeds (not faster than RAM) but see maximum speed of the SATA/PCIe subsystem, especially since with many drives some systems saturate the pci lanes and/or some sata links are not operating at maximum speed. This could help move controllers to different pci slots (with more lanes) or identify other system bottlenecks. On some  systems with dual parity (and slow cpus) it is sometimes hard to differentiale between bottlenecks because of parity calculation and pcie saturation. Thats at least my rational behind this feature request - maybe there is even a more elegant way to test max-sata speed, but it seems simple to offer as an option with the disclaimer "interpret results at your own peril". Anyhow: Any feature to test concurrent drive speed performance is welcomed - Keep up the fantastic work!

[jbartlett]

I think the switch to bypass the RAM cache also bypasses the disk cache and I couldn't find any way to still allow the disk cache but not the RAM cache. Shame. It would have allowed the bus to be saturated with fewer drives.

 

I'll have to go back to the original plan, simultaneously read from an increasing number of drives attached to a controller until it detects saturation. Or as the thought just occurred to me, simultaneously read from all drives on a controller at once. The stepping method would allow an eyeball to see incremental increases while all drive method going balls to the wall would identify saturation or max bandwidth utilization quickly.

[FreeMan]

On 7/9/2017 at 7:25 AM, jbartlett said:

If you'd like to help and to ensure your run is more pleasant, please run the attached "diskmodels.sh" script. What it does is to send the drive models of the drives you have connected to a web site app and it'll return either it was added, previously added, or contains unexpected characters.

 

If I understand this properly, this script is populating a database at "this" web site and that's what you're using to display all the cool graphics?

 

I've added 4 new drive types, and I'm really looking forward to testing this new version. Keep up the great work!

Edited July 23, 2017 by FreeMan

[Videodr0me]

4 hours ago, jbartlett said:

I'll have to go back to the original plan, simultaneously read from an increasing number of drives attached to a controller until it detects saturation. Or as the thought just occurred to me, simultaneously read from all drives on a controller at once. The stepping method would allow an eyeball to see incremental increases while all drive method going balls to the wall would identify saturation or max bandwidth utilization quickly.

I like the stepping method - especially if you can choose the sequence of drives (and which drives to include in the first place). The last step should then give the same result as a simulataneous read from all drives. Maybe one could just add a parameter on how many drives to add per step (according to the sequence chosen) ranging from 1, 2 ... to all. If all was chosen then the first step is the just a simultaneous read from all drives.

[Videodr0me]

6 hours ago, jbartlett said:

I think the switch to bypass the RAM cache also bypasses the disk cache and I couldn't find any way to still allow the disk cache but not the RAM cache. Shame. It would have allowed the bus to be saturated with fewer drives.

Don't know if it helps but hdparm seems to be able to do something related e.g. "hdparm -Tt /dev/sdx" - but I am unsure about the buffer/cache handling. Alternatively maybe dropping the linux caches yields the desired results:

sync

echo 3 > /proc/sys/vm/drop_caches

If this even works on slackware/unraid. Trying direct i/o might also be worth a try.

 

Edited July 23, 2017 by Videodr0me

[jbartlett]

6 hours ago, FreeMan said:

If I understand this properly, this script is populating a database at "this" web site and that's what you're using to display all the cool graphics?

 

"this" website will be the Lucee app server running via a plugin for the time being and you'll be able to start & stop it - such as to free up mem. Uninstalling the plugin will free up the memory the files on resource constrained systems and the plugin will save the drive configuration data on the flash drive.

 

I plan on having a repository of user-submitted drive images & information on my own server for people to select. I'm also pondering including all drive images or have the plugin import the images at scan time. Importing would cut down on the plugin size but if I happen to die & the repository goes offline, then people are stuck with the default image. At least at first install, people can update their own drive images.

[FreeMan]

1 hour ago, jbartlett said:

but if I happen to die

Please don't do that any time soon. I'm sure we're not the only ones who would miss you.

 

1 hour ago, jbartlett said:

the repository goes offline

Maybe Lime-Tech would be willing to host the images for you since we're all counting on them being around for a while. Alternatively, maybe you could host them in a Github project.

 

1 hour ago, jbartlett said:

the Lucee app server running via a plugin for the time being and you'll be able to start & stop it - such as to free up mem

Just how much memory will the server take? Will it need to run all the time or just when running a disk speed test? Would it be a reasonable possibility to fire up the server when a test is requested then shut it down when the testing is done, or, possibly 30 minutes after the last test has run/the server's been accessed? (i.e. If I run it for one drive, then test again for a 2nd drive, it wouldn't make sense to fire up the server & shut it down twice, but it would make sense to fire it up, run the first test, then the timer starts. A couple of minutes later, I run a 2nd test, the timer resets, etc.)

 

Sorry if none of that made sense to you, I'm not completely sure of the purpose of the server other than to collect the drive types that are being used in the wild.

[jbartlett]

The Lucee ColdFusion server will need to be running when doing the disk tests. I'll look into porting it to PHP once the code is stable. I just haven't done much PHP coding to develop an app of this scale from scratch. ColdFusion is a language I've been actively coding in since 1999.

 

I'll be trying to minimize the resource impact. So far, the app has been running fine with the 256MB of RAM allocated to the java server (Lucee is based in Java, compiles ColdFusion code into Java) but I haven't done any type of monitoring yet to see how much ram is utilized for trimming. I also will be pruning the files to remove any that aren't needed so they're not taking up memory by existing.

 

Lucee also runs under Docker and I may go that way to try to tie them together at some point as an alternate method of installation but I didn't want to over complicate things just now. KISS method.

[FreeMan]

On 7/24/2017 at 3:13 AM, jbartlett said:

the app has been running fine with the 256MB of RAM allocated to the java server

I think that if anyone has an issue with that memory footprint, they should drag their server into the 21st century.

 

On 7/24/2017 at 3:13 AM, jbartlett said:

I didn't want to over complicate things just now. KISS method.

Sounds like a good plan!

 

Looking forward to the updates, and now that I discovered there's a plugin (and installed it!) I'll be sure to get them!

 

You may want to consider updating the OP to indicate (up front and in big red letters) that there's a plugin now. I scanned the OP a day or two ago (i.e. looked at the bottom where the attachments are) looking to see if there was an update to the version I'd DL'd a while back. It wasn't until I started browsing back through previous posts that I started seeing mentions of a plugin.