AdamT

New unMenu Package: MRTG... any interest?

Recommended Posts

Folks, I just discovered unRAID recently and am setting up a box.  I was fiddling around with unRAID packages while waiting for my files to copy and noticed that there didn't seem to be any traffic plots.  So as a learning experience I got MRTG and SNMP working on my unRAID 5 beta 14.  Everything seems to work, but it's all very much a work in progress. (it's a beta package on a beta unRAID release!) If you haven't heard of MRTG, it's a traffic grapher (http://oss.oetiker.ch/mrtg/).  They way I've gotten it working is it:

 

1. requires unMenu

2. requires perl (via the unMenu package), doesn't require gcc (I made a binary package)

3. defaults to setting up a ro snmp community public (can be overridden)

4. creates a base mrtg.cfg file in /boot/config if there isn't one there (so you can customize it and not have it disappear)

5. creates a /tmp/mrtg directory where the log/png files are stored (these will disappear upon reboot).  The directory is user definable so, in theory, you could stick it on a cache drive or even on the array and have a full history saved.  MRTG's log files don't grow past a certain size, so no worries about running out of ram.

6. adds a 5 minute recurring cron job to update the mrtg charts

7. piggybacks on emhttp (the unRAID GUI) to create a simply/crappy website at http://tower/mrtg/index.html'>http://tower/mrtg/index.html (for unRAID 5.0b14) or http://tower/log/mrtg/index.html (for unRAID 4.7) which shows all the interfaces' daily charts and allows for drilling into the week/month/year data (make sure you include the index.html!)

 

Things to keep in mind:

- MRTG will give warnings the first 2 intervals as it has to create log files/image files (so if you have root email forwarded, you will see it) no longer an issue

- not sure how to get the index.php index.html to show up as the default page under http://tower/mrtg using emhttp

- probably some bugs in the unMenu conf file (it's my first one, so bear with me)

- all the dependencies other than perl take up 7.7MB of tgz files (and all but mrtg are Slackware packages)

 

the conf file is here

http://ruinedkingdoms.com/unRAID/mrtg-unmenu-package.conf

 

I (hopefully) attached a sample chart (a daily one)

 

There's no reason why you couldn't set up a bunch of other machines' interfaces (routers, other machines, etc) manually in the mrtg.cfg file and have them all tracked.  Also, SNMP allows for disk space/load average tracking as well as eth interface tracking so you could theoretically do disk space/load average charts with MRTG as well.

 

Anyway, if anyone ends up using it, let me know if you notice any issues and if you have any suggestions for the conf file.

Cheers,

Adam

 

Update 2012-01-09:

I've setup a 4.7 unRAID USB and this now works there as well.  The URL is unfortunately different between 4.7 and 5.0b14

for 4.7, it's http://tower/log/mrtg/index.html

for 5.0 beta 14, it's http://tower/mrtg/index.html'>http://tower/mrtg/index.html (the same as before)

make sure to include the "index.html" in your url. You wouldn't think it was required, but it seems emhttp requires it

Also, here is a sample screen shot of a bunch of different charts

http://ruinedkingdoms.com/unRAID/mrtg_sample_index.png

and the mrtg.cfg file that creates it

http://ruinedkingdoms.com/unRAID/mrtg.cfg

127.0.0.1_2-day.png.d8ba9fa1bcf55885fbad0476ba8eef7b.png

Share this post


Link to post
Share on other sites

your .conf file does not look too bad...  only major issue I see is the line adding the cron entry.

 

If you install several times, changing a variable each time, it will add a line for the cron task each time.

 

I like how you did it all on one line, but perhaps change it slightly to use "grep -v" to eliminate any existing cron task prior to adding the new one.

Something like this perhaps  (I added the grep command in "blue"):

(crontab -l | grep -v "mrtg.cfg"; echo "*/5 * * * * env LANG=C /usr/bin/mrtg $v_data/mrtg.cfg") | crontab -

 

A minor issue will be if a person enters a data path that contains a space or special characters the shell interprets, but it is hard to protect against bad data entered in the configuration fields.  You've done well in giving them examples of what is expected.

 

For a first time unMENU package, you figured out the format pretty well.  There is possibly one more dependency if this package is installed on unRAID 4.X.  Those versions do not have "php" installed.  They will probably not work as your index.php will not be interpreted.

 

Joe L.

Share this post


Link to post
Share on other sites

Thanks for the feedback!  I changed the cron line to your suggestion (I hadn't considered that).  I also removed the PHP dependency as I was only using it for the index.php.  I'm now just using a one-line perl script to generate an index.html file (so the mrtg page is now http://tower/mrtg/index.html).  I also ended up running mrtg twice as part of the install so no cron warnings are sent out.

 

I haven't tested it on 4.7 yet as I have only one USB memory stick (which is blacklisted) and I'm waiting for a better one to show up :-)  One thing I'm concerned about is I checked the version of Slackware for 5.0 and it was 13.1, so that is the version of tgz files I used in my conf file and where I compiled the mrtg tgz. I'm guessing 4.7 is using Slackware 12.x, would that cause a problem?

 

Also, although it installs just fine manually, it doesn't seem to want to auto install.  If I run /boot/packages/mrtg-2.17.3-i486-0.tgz.auto_install after a boot, it works fine.  I suspect it's getting auto installed before perl, but I can't see the messages on the console because it scrolls by too fast (and I can't find a message log file).  Is there any way to have the mrtg package auto install get scheduled for after the perl package install (besides adding a line to the 'go' script)?

 

Sorry to bug you with these questions.

Thanks,

Adam

Share this post


Link to post
Share on other sites

I suspect it's getting auto installed before perl, but I can't see the messages on the console because it scrolls by too fast (and I can't find a message log file).  Is there any way to have the mrtg package auto install get scheduled for after the perl package install (besides adding a line to the 'go' script)?

 

Sorry to bug you with these questions.

Thanks,

Adam

The .auto_install scripts are invoked in lexical order, by the one line added to the bottom of the "go" script.

To change the order, you'll need to modify that line, or... rename the .auto_install file  (which is named after your package file) to start with a letter > "p"

or... since you are pretty good with scripts, modify the package manager awk script to respect a new tag for PACKAGE_DEPENDENCY.

or, add a line like this towards the top of your installation lines in your package:

test -f "${PACKAGE_DIRECTORY}"/perl-5.10.0-i486-1.tgz.auto_install  && "${PACKAGE_DIRECTORY}"/perl-5.10.0-i486-1.tgz.auto_install

 

That may work if "make" exists, as perl seems to depend on it.    (the problem of dependent package gets pretty big after a while... I don't think there is a perfect solution)

 

 

Share this post


Link to post
Share on other sites

Make your base package install the perl package.

 

I do this a lot in my unMenu packages.  Take a look at my SABnzb, SickBeard, Couchpotato, Transmission packages for ideas.  On first install of the package it checkes to see if the required components are installed, if not it directs them to the correct URL to install that package.  Rinse and repeat until all dependencies have been satisfied.

Share this post


Link to post
Share on other sites

Thanks guys, will have a look.  My system's doing a parity check so probably won't be for a while. :-)

I got together a good sample page of a standardized set of charts, which I think I'll set up options for including in the future.

Each of the charts is a link that goes to more historical data (and hard numbers).

 

I outsmarted myself with the index.html generation though, turns out mrtg comes with a script that generates a landing page based upon your mrtg.cfg file :-)

 

Here is what it looks like:

http://ruinedkingdoms.com/unRAID/mrtg_sample_index.png

Share this post


Link to post
Share on other sites

I fiddled with it some more and used prostuff1's suggestion, now it works and stays working upon reboot.  Cleaned up some and added comments to the conf file and posted my sample mrtg.cfg (that does all the extra things), but for now the base package install just sets up ethernet interface monitoring.

 

So, for my use, it is fine. I'll see if anyone else is interested before I put more effort into this package.  I've got another one I'm looking into trying to get working.

 

Just to have all this info in the same place, here is

 

the link to the conf file (always points to the newest version)

http://ruinedkingdoms.com/unRAID/mrtg-unmenu-package.conf

 

the link to the sample screenshot (with all sorts of charts)

http://ruinedkingdoms.com/unRAID/mrtg_sample_index.png

 

the link to the mrtg.cfg file that shows how to do everything in the sample screenshot (the package just sets up the ethernet interface(s) by default)

http://ruinedkingdoms.com/unRAID/mrtg.cfg

 

Thanks for everyone's help.

Share this post


Link to post
Share on other sites

This looks good AdamT.

 

I was looking for something that give an overall picture of what's going on with my server, especially CPU load..  I'm running a HP N40L microserver and it would be good to evaluate how it's coping with running PS3MS during encoding tasks. :)

 

I'm running 4.7 at the moment, and installed Perl, and it appeared to install fine, but can't access tower/mrtg/index.html

 

I did get an error, that's all that appeared out of place.

ln: creating symbolic link `/usr/local/emhttp/mrtg': No such file or directory

 

I didn't edit the Configuration Variables, and haven't performed a reboot.

Share this post


Link to post
Share on other sites

Thanks for posting this. Very useful!

 

I'm running 4.7 and have it working. I also couldn't get the web interface running, but I have problems with other apps using emhttp, so I don't know if I have the same problem as Lacehim. I changed the cfg so it writes to a folder inside my lighttpd destination and that works for me. I prefer using lighttpd, anyhoo.

 

A couple of questions:

 

1. After I change the config, how do I restart mrtg or get it to reload the cfg so it writes a new index.html?

 

2. How do I add /mnt/cache as a disk target (or whatever it's called) -- I need to monitor that, rather than disk1/disk2

 

Thanks again!

Share this post


Link to post
Share on other sites

 

I'm running 4.7 at the moment, and installed Perl, and it appeared to install fine, but can't access tower/mrtg/index.html

 

I did get an error, that's all that appeared out of place.

ln: creating symbolic link `/usr/local/emhttp/mrtg': No such file or directory

 

 

Thanks! Ah, it looks like the emhttp directory is different in 4.7 (I only have 5.0 installed right now), I've got to find another USB stick to put 4.7 on so I can see what the differences on.  Was hoping my "real" stick would have come in by now from newegg.  I'll see if I can dig up one from somewhere and let you know what I find.  It looks like other than the different web directory, it should work fine (neilt0 was able to get it working using lighttpd).

Share this post


Link to post
Share on other sites

Thanks for posting this. Very useful!

 

I'm running 4.7 and have it working. I also couldn't get the web interface running, but I have problems with other apps using emhttp, so I don't know if I have the same problem as Lacehim. I changed the cfg so it writes to a folder inside my lighttpd destination and that works for me. I prefer using lighttpd, anyhoo.

 

A couple of questions:

 

1. After I change the config, how do I restart mrtg or get it to reload the cfg so it writes a new index.html?

 

2. How do I add /mnt/cache as a disk target (or whatever it's called) -- I need to monitor that, rather than disk1/disk2

 

Thanks again!

 

Yup, you've got the same issue as Lacehim, just a different path to the main web code in 4.7  I was piggybacking on emhttp only because it was there.  As long as it works for you under lighttpd, that's all that matters :-)

 

To answer your questions:

 

1. when you make changes to mrtg.cfg, make sure you update both files (fiddle with the one in /tmp/mrtg or wherever, but save it to the one in /boot/config when you're happy). I was hoping to just use the one in /boot/config all of the time, but mrtg has some weird problem (it thinks there are stale lock files in the /boot/config directory when there aren't). mrtg will see the changes the next time it runs via cron, although if you add a new chart, it will whine the first two times (missing log files, etc). I find that running it manually twice works better (instead of getting cron mails).

env LANG=C mrtg /path/to/mrtg.cfg

to update the mrtg landing page, run this

indexmaker --output=/path/to/index.html /path/to/mrtg.cfg

 

2. Hmm, maybe I should break out the snmpd.conf file as well, that's where you need to make the changes to do other disks.  As you've already made changes to the mrtg-unmenu-package.conf file, have another look in there and look for this bit:

PACKAGE_INSTALLATION echo "disk /mnt/disk1" >> /etc/snmp/snmpd.conf
PACKAGE_INSTALLATION echo "disk /mnt/disk2" >> /etc/snmp/snmpd.conf
PACKAGE_INSTALLATION echo "disk /mnt/disk3" >> /etc/snmp/snmpd.conf
PACKAGE_INSTALLATION echo "disk /mnt/disk4" >> /etc/snmp/snmpd.conf
PACKAGE_INSTALLATION echo "disk /mnt/disk5" >> /etc/snmp/snmpd.conf
PACKAGE_INSTALLATION echo "disk /mnt/disk6" >> /etc/snmp/snmpd.conf

 

add a line that looks like

PACKAGE_INSTALLATION echo "disk /mnt/cache" >> /etc/snmp/snmpd.conf

and note the order that it's in relative to the other disk statements

 

then just re-install the package.  After that, change your mrtg.cfg to have the bit that looks for disk usage

 

Target[server.disk]: dskPercent.1&dskPercent.2:public@localhost

Title[server.disk]: Disk Partition Usage

PageTop[server.disk]: <H1>Disk Partition Usage /mnt/disk1 and /mnt/disk2</H1>

MaxBytes[server.disk]: 100

ShortLegend[server.disk]: %

YLegend[server.disk]: Utilization

LegendI[server.disk]: /mnt/disk1

LegendO[server.disk]: /mnt/disk2

Options[server.disk]: gauge,growright,nopercent

Unscaled[server.disk]: ymwd

Change the red number(s) to be the one which signifies the offset you put your /mnt/cache line in earlier (if it was the first disk line, that's 1... second would be 2, etc).  Change the bolds to match for the legends and titles.  If you only want the one disk, set both numbers to the same thing.

 

Edit: also should mention, not sure how the cache drive works, but for regular disks in the array, mrtg throws an error when the array isn't running.

 

Hope that helps, and let me know if you have any more questions.

Share this post


Link to post
Share on other sites

I'll give this a go - I know of the program but have never put in the effort to find out how it works. Am I correct in assuming I need to install the mrtg slackware package (wherever it is) and then use the .conf file provided to enable it within unMenu?

Share this post


Link to post
Share on other sites

I'll give this a go - I know of the program but have never put in the effort to find out how it works. Am I correct in assuming I need to install the mrtg slackware package (wherever it is) and then use the .conf file provided to enable it within unMenu?

 

As far as I know, there is no Slackware mrtg package. My conf file installs one I compiled myself.

Share this post


Link to post
Share on other sites

plowna, just put the .conf in the unmenu folder, and install from packages in unmenu.

 

If your missing perl it will let you know to install it.

 

AdamT, I ran "which emhttp" on my tower, and it says emhttp is in /usr/local/sbin/emhttp so I've modified the .conf file with the additional /sbin folder.  I'll have to reboot and reinstall it I guess but it's the only reference to emhttp I can see in the .conf file.

Share this post


Link to post
Share on other sites

Got it installed & running ok. Ran a quick cfgmaker public@localhost > mrtg.cfg, I wasn't sure if the .cfg you put in was the same. I added a link to the 'Useful Links' page for easy access.

Share this post


Link to post
Share on other sites

Always interested on now toys. 

 

Not sure why it is not working for me.  I installed php before I installed the config file.  It appears to install correctly and I get a cfg file in my config folder.  When I go to http://tower2/mrtg I am asked for my password which would be normal but then i get a blank page.

 

Any ideas where i should look?

 

thanks

 

running beta14

Share this post


Link to post
Share on other sites

AdamT, I ran "which emhttp" on my tower, and it says emhttp is in /usr/local/sbin/emhttp so I've modified the .conf file with the additional /sbin folder.  I'll have to reboot and reinstall it I guess but it's the only reference to emhttp I can see in the .conf file.

 

Actually, you don't want to do that... the /usr/local/sbin/emhttp is the binary for emhttp.  I've got this fixed in my latest release, I'll post a longer explanation separately.

Share this post


Link to post
Share on other sites

Got it installed & running ok. Ran a quick cfgmaker public@localhost > mrtg.cfg, I wasn't sure if the .cfg you put in was the same. I added a link to the 'Useful Links' page for easy access.

 

Yup, that's basically what I did in the package install.  I also set the WorkDir and the base options to whatever was in the User Configuration settings.

Share this post


Link to post
Share on other sites

 

Any ideas where i should look?

 

thanks

 

running beta14

 

It looks like emhttp is very, very picky.  I have to actually include the index.html in the path, so for you it would be http://tower2/mrtg/index.html.  Let me know if that works, in the latest version I put a comment in the description about the need to have the index.html in there.

Share this post


Link to post
Share on other sites

Folks, I've just set up a 4.7 unRAID and have gotten it working there now as well.  I have a new version of the conf file at

http://ruinedkingdoms.com/unRAID/mrtg-unmenu-package.conf

 

Changes:

  • I found a place in 4.7 emhttp to sneak in the mrtg page, it is different from 5.0, unfortunately :-(
    for 4.7, it's http://tower/log/mrtg/index.html
    for 5.0 beta 14, it's http://tower/mrtg/index.html (the same as before)
    make sure to include index.html in your url, emhttp is picky!
  • added default support for /mnt/disk1 - /mnt/disk20 & /mnt/cache.  the cache drive should be 1 drive 'number' higher than your last disk.  If you want to double check, run this command from the command line after installing this new package:

    snmpwalk -v 1 -c public localhost dskPath


    this shows all the disks SNMP knows about (the blue depends on if you overrode the default values in the package configuration).

  • I broke out the snmpd.conf file, so if anyone wants to make changes to the SNMP service, the file to change is /boot/config/snmpd.conf.  make sure you do a

    /etc/rc.d/rc.snmpd restart


    after you make changes to snmpd.conf

  • there is only one copy needed of the mrtg.cfg file now.  turns out mrtg can't put a lockfile in a FAT32 partition (i.e. /boot), so I added a flag to have it put somewhere else.  only thing this means is all you have to do is update /boot/config/mrtg.cfg with any changes.  for advanced users, the flag I added was

    env LANG=C /usr/bin/mrtg --lock-file /tmp/mrtg_lockfile /boot/config/mrtg.cfg


     

 

I think that's it, let me know if anyone notices any other problems.  Hope you can all get MRTG working!

 

Thanks,

Adam

 

Edit: I cant spel

Share this post


Link to post
Share on other sites

it works nice  ;D

i have been playing with it for a while now and no issues :o

 

i was wondering is there a way to measure traffic to a share ?  ???

i am less interested at the disk partition usage as that is clear enough in unmenu/mymain but i would like to know what usershare is used the most

/mnt/user/movies or mnt/user/tv or whatever .....

 

Share this post


Link to post
Share on other sites

Thanks AdamT, it's working perfectly on 4.7 for me. :)

 

I loaded the updated .cfg file and it's displaying all that info perfectly!

 

Nice job!

 

If you can modify your inital post, you might want to update the information so that new users know about the difference between 4.7 & 5b14 index.html pages!

Share this post


Link to post
Share on other sites

i was wondering is there a way to measure traffic to a share ?  ???

 

Hmm, I did some digging and it's possible to show bytes read/written to each disk (here's a page on how to do it http://oss.oetiker.ch/mrtg-trac/wiki/Net-SNMP).  I'm not sure that shares will have this tracking because it talks about devices.  I guess it comes down to how unRAID treats shares under the covers. Unfortunately, the Slackware SNMP package doesn't seem to have support for disk i/o installed, so I can't see if it would work for shares as well.  It looks like SNMP needs to be compiled with '--with-mib-modules=ucd-snmp/diskio' but Slackware's isn't:

 

# snmpget -v 1 -c public localhost UCD-SNMP-MIB::versionConfigureOptions.0

UCD-SNMP-MIB::versionConfigureOptions.0 = STRING:  '--prefix=/usr' '--libdir=/usr/lib' '--sysconfdir=/etc/snmp' '--mandir=/usr/man' '--enable-ipv6' '--disable-debugging' '--enable-static=no' '--with-libwrap' '--with-perl-modules' '--with-default-snmp-version=3' '--with-sys-contact=root@example.org' '--with-sys-location=unknown' '--with-logfile=/var/log/snmpd.log' '--with-persistent-directory=/var/lib/net-snmp' '--without-rpm' '--program-suffix=' '--program-prefix=' '--build=i486-slackware-linux' 'build_alias=i486-slackware-linux' 'CFLAGS=-O2 -march=i486 -mtune=i686'

 

Let me see if I can re-compile Slackware's SNMP.  If there are too many dependancies, I won't be able to do it (I don't actually have a Slackware dev box setup, I'm compiling stuff on unRAID which doesn't have a lot of packages installed).

Share this post


Link to post
Share on other sites

Thanks AdamT, it's working perfectly on 4.7 for me. :)

 

I loaded the updated .cfg file and it's displaying all that info perfectly!

 

Nice job!

Thanks, glad it's working for you!

 

If you can modify your inital post, you might want to update the information so that new users know about the difference between 4.7 & 5b14 index.html pages!

Good idea, done.

Share this post


Link to post
Share on other sites

i was wondering is there a way to measure traffic to a share ?  ???

Let me see if I can re-compile Slackware's SNMP.  If there are too many dependancies, I won't be able to do it (I don't actually have a Slackware dev box setup, I'm compiling stuff on unRAID which doesn't have a lot of packages installed).

 

Ok, I was able to re-compile net-snmp with the required option and I can get stats by both disk (physical drive) and array (drive + parity).  That means with this you could track how much was read/written to each disk drive individually or to each "disk share".  Unfortunately, it doesn't see user shares at all.  I suspect that is because they are really done in software "above" the level that SNMP is monitoring.  However, if there is some file somewhere in unRAID that has this information, we can add it to the charts that way.

 

I haven't included this new net-snmp in the mrtg package yet.  is anyone interested in by disk read/writes? 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.