[Plugin] IPMI for unRAID 6.1+

[dmacias]

I am having some issues when powering down one of my servers, it simply hangs (see the pic). I had just updated the BMC firmware so I thought that could be the problem. I downgraded the firmware but no luck, still same problem. 

I decided to power down my two other servers and got into same problem.

 

Decided to uninstall the IPMI plugin on all three servers and they all power down as they should. Reinstalling the plugin will bring back the problems again.

 

Can I get some guidance for troubleshooting this pls…. (my signature is updated with current config)

 

 

So it just stops right there and won't shut down even after that? Do you also have fan control running? Only thing you could check is if you log back in remotely and run "ps aux | grep ipmi" and see if ipmiseld, ipmifan or 2 instances of ipmitail are still running. I have been moving everything from my C2750 to my X10SLL-F and have shutdown a dozen times without fail. I'll double check though and record the console as it shuts down. 

 

Edit. You could also test starting and stopping the services manually

/etc/rc.d/rc.ipmiseld start/stop

/etc/rc.d/rc.ipmitail start/stop

ipmifan --daemon

ipmifan --quit

 

Edit 2. I think it's ipmifan that's not shutting down. I changed some command line variables and the shutdown script uses -q instead of --quit.

 

[Leifgg]

11 hours ago, dmacias said:

So it just stops right there and won't shut down even after that? Do you also have fan control running? Only thing you could check is if you log back in remotely and run "ps aux | grep ipmi" and see if ipmiseld, ipmifan or 2 instances of ipmitail are still running. I have been moving everything from my C2750 to my X10SLL-F and have shutdown a dozen times without fail. I'll double check though and record the console as it shuts down. 

 

Edit. You could also test starting and stopping the services manually

/etc/rc.d/rc.ipmiseld start/stop

/etc/rc.d/rc.ipmitail start/stop

ipmifan --daemon

ipmifan --quit

 

Edit 2. I think it's ipmifan that's not shutting down. I changed some command line variables and the shutdown script uses -q instead of --quit.

 

Many thanks for the update. I have installed it on my Test server and enabled Event notifications, Footer setting as well as Fan control and everything looks fine including Power down.

I will test my Main and Backup servers later as well but I don’t expect any problems.

Once again thanks!

/Leif

 

Edited November 30, 2017 by Leifgg

[Leifgg]

3 hours ago, Leifgg said:

 

I have now tested my Main and Backup servers and problems are solved....

[ljm42]

I had the same issue on my main system and my VM, both on 6.4.0_rc14. They both hung when trying to shutdown after installing the 2017.11.29 version of the IPMI plugin (admittedly, there were likely other changes as well)

 

The VM was nice enough to drop some hints on the console, it complained about an undefined value in line 322 of ipmifan:
  https://github.com/dmacias72/IPMI-unRAID/blob/master/source/ipmi/usr/local/emhttp/plugins/ipmi/scripts/ipmifan#L322 
It was stuck in a loop, displaying that error over and over.

 

My main system did not leave any helpful hints on the console, it just hung.

 

Neither system uses the ipmi fan control, they just display temps and fan speeds.

 

After the first reboot, both systems lost their IPMI plugin settings and had to be reconfigured.  Subsequent reboots worked without a problem.

 

Note... the reason for the shutdown was to install the latest BMC/Bios (yay, the ASRock Java IPMI console no longer throws Java security errors!), so it is possible that is the reason I had to reconfigure the plugin.

[dmacias]

They both hung when trying to shutdown after installing the 2017.11.29 version of the IPMI plugin (admittedly, there were likely other changes as well)
 
The VM was nice enough to drop some hints on the console, it complained about an undefined value in line 322 of ipmifan:
  https://github.com/dmacias72/IPMI-unRAID/blob/master/source/ipmi/usr/local/emhttp/plugins/ipmi/scripts/ipmifan#L322 
It was stuck in a loop, displaying that error over and over.
 
My main system did not leave any helpful hints on the console, it just hung.
 
Neither system uses the ipmi fan control, they just display temps and fan speeds.
 
After the first reboot, both systems lost their IPMI plugin settings and had to be reconfigured.  Subsequent reboots worked without a problem.

Thanks I still wasn't sure why my system never hung but that explains it. The ipmi plugin Slackware package's do install script adds 3 lines to rc.local_shutdown to shut down the services. The ipmifan line had a -q instead of --quit. So that error was the fan script trying to run but not configured. Hence the line 322 error. Since I use fan control I wouldn't see that error.

[ljm42]

24 minutes ago, dmacias said:

Thanks I still wasn't sure why my system never hung but that explains it. The ipmi plugin Slackware package's do install script adds 3 lines to rc.local_shutdown to shut down the services. The ipmifan line had a -q instead of --quit. So that error was the fan script trying to run but not configured. Hence the line 322 error. Since I use fan control I wouldn't see that error.

 

I still have one more server to update. Any chance this can be fixed, or should I just delete the ipmifan script before the first reboot?

[dmacias]

 
I still have one more server to update. Any chance this can be fixed, or should I just delete the ipmifan script before the first reboot?

It's fixed in the latest. If you have already rebooted with the latest, it should work. The shutdown only gets written on plugin install but I didn't add anything to change the -q to --quit, maybe I should have. Just check /etc/rc.d/rc.local_shutdown for the ipmifan line. It should be at the button. Make sure it has --quit or remove it completely. As far as settings disappearing, it may be that the cfg files were open during the hang and were zero'd out when shutdown was forced.

[ljm42]

OK, I understand now. I can modify /etc/rc.d/rc.local_shutdown after upgrading the plugin on that last system.  But it makes me a little nervous knowing there is a script called during shutdown that can potentially block the shutdown.

 

I can reproduce the problem by executing /usr/sbin/ipmifan without any parameters, it goes into a loop throwing this error multiple times and never exiting: 

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/ipmi/scripts/ipmifan on line 322

Ideally, all of the scripts would check to confirm the appropriate ipmi variables have been set before they try doing anything.  This would ensure they don't have any unintended side effects down the road.

 

Is that reasonable? It isn't super urgent, since there is a workaround for the immediate problem.

[dmacias]

OK, I understand now. I can modify /etc/rc.d/rc.local_shutdown after upgrading the plugin on that last system.  But it makes me a little nervous knowing there is a script called during shutdown that can potentially block the shutdown.
 
I can reproduce the problem by executing /usr/sbin/ipmifan without any parameters, it goes into a loop throwing this error multiple times and never exiting: 

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/ipmi/scripts/ipmifan on line 322

Ideally, all of the scripts would check to confirm the appropriate ipmi variables have been set before they try doing anything.  This would ensure they don't have any unintended side effects down the road.
 
Is that reasonable? It isn't super urgent, since there is a workaround for the immediate problem.

I wouldn't call it a workaround. Most of the rc.scripts would do this if you called the wrong option. It would be the same as calling rc.libvirtd start instead of stop. That's what the -q did. It tried to start it in quiet mode. The main purpose of ipmifan is to start a loop. The errors don't add to the problem as it would still be in a loop. As long as ipmifan --quit is called it's benign and will exit after checking for pid or trying to kill any ipmifan processes. I call all three services in shutdown to stop ipmiseld, ipmitail and ipmifan.

I will double check that you couldn't somehow start the script from the webgui without the proper variables set. I believe that is already the case.

[ljm42]

5 minutes ago, dmacias said:

I wouldn't call it a workaround. Most of the rc.scripts would do this if you called the wrong option.

 

Oh ok, that is probably fine then.  Thanks again for all of your work on this.

[dmacias]

 
Oh ok, that is probably fine then.  Thanks again for all of your work on this.

Just wanted to put your mind at ease. It was my mistake. Thanks for all the feedback as always.

[lordbob75]

Hey I'm loving this plugin so far, but I'm having trouble with the temp reading.  There appear to be 2 items with the exact same name (115 and 116), and I'm using a script to pull the IPMI data to influxdb to grafana, where it seems to be pulling the wrong temp (22c vs 42c reported in unraid).  The 2nd temp item was showing at that temp earlier (though not now) so I think the script is pulling the wrong temp item.  I can't get it to pull the right one since I can't seem to edit the config file, or the changes just have no effect.

 

Tried renaming the 2nd temp, tried removing it, tried using 115_Temp instead, but it just seems to ignore every change to the config file, even when the config editor shows the correct config edits.

 

Can you provide any assistance?

[dmacias]

I'm not sure what you're trying to do. Could you elaborate a bit more. On what values you're trying to get and for what. The numbers are the sensor id's and the names are all from your system. This all comes from running ipmi-sensors.

[lordbob75]

17 minutes ago, dmacias said:

I'm not sure what you're trying to do. Could you elaborate a bit more. On what values you're trying to get and for what. The numbers are the sensor id's and the names are all from your system. This all comes from running ipmi-sensors.

No problem.

 

I installed your plugin and get readings right off the bat.  When I go to the Readings page, sort by name, there are 2 sensors named Temp, with nothing to distinguish the two.  Turning on advanced view will show the IDs, and they are 115 and 116.  115 appears to be my cpu temp (that the dynamix temp plugin grabs), whereas 116 does not appear to have any reading.

 

I have a script running on UnRAID (through the user scripts plugin) called IPMI that ports that data to my influxdb container.  I then use Grafana to display the data stored in the database.

 

When I set grafana to display the Temp value (from the unraid ipmi data), it displays as 23C.  When I look on the Readings page, one Temp is 42 and the other is N/A.

[dmacias]

No problem.
 
I installed your plugin and get readings right off the bat.  When I go to the Readings page, sort by name, there are 2 sensors named Temp, with nothing to distinguish the two.  Turning on advanced view will show the IDs, and they are 115 and 116.  115 appears to be my cpu temp (that the dynamix temp plugin grabs), whereas 116 does not appear to have any reading.
 
I have a script running on UnRAID (through the user scripts plugin) called IPMI that ports that data to my influxdb container.  I then use Grafana to display the data stored in the database.
 
When I set grafana to display the Temp value (from the unraid ipmi data), it displays as 23C.  When I look on the Readings page, one Temp is 42 and the other is N/A.

I would just use ipmi-sensors in your script to get the data you need. What board do you have? Maybe someone else can shed some light on your temp sensors.

[lordbob75]

1 hour ago, dmacias said:

I would just use ipmi-sensors in your script to get the data you need. What board do you have? Maybe someone else can shed some light on your temp sensors.

Well the script doesn't poll the data, it uses your plugin to do so, then just sends it to infulxdb.  It looks like the Temp is pulling the Inlet Temp instead, but not sure why since I can pull inlet temp with inlet temp too...

[simon021]

Love the plugin, very useful!

 

Would it be possible to allow the "System Level," which is the current draw, to be displayed in the bottom right?  I am able to read the power draw in the sensors, but it is not one of the options to have displayed in the bottom.

 

Thanks!

Edited December 12, 2017 by simon021

[dmacias]

Could you give me a readout of ipmi-sensors?

[lordbob75]

@dmacias I discovered that your plugin was causing massive lag on my webgui.

I've removed it and the lag went away instantly.  Not sure what's going on here but would be happy to help diagnose a fix.

 

Edit: posted accidentally before I wrote anything...

Edited December 13, 2017 by lordbob75

[dmacias]

[mention=11874]dmacias[/mention] I discovered that your plugin was causing massive lag on my webgui.
I've removed it and the lag went away instantly.  Not sure what's going on here but would be happy to help diagnose a fix.
 
Edit: posted accidentally before I wrote anything...

You mentioned earlier you were manually editing the config file. If you have a bad config or the settings are wrong it could lag the webgui. There would be a timeout for every ipmi command run. I would delete the ipmi plugin directory on the flash drive and start from scratch. You could try running ipmisensors from the command line and see if there's an error.

[lordbob75]

14 minutes ago, dmacias said:

1 hour ago, lordbob75 said:

[mention=11874]dmacias[/mention] I discovered that your plugin was causing massive lag on my webgui.
I've removed it and the lag went away instantly.  Not sure what's going on here but would be happy to help diagnose a fix.
 
Edit: posted accidentally before I wrote anything...

You mentioned earlier you were manually editing the config file. If you have a bad config or the settings are wrong it could lag the webgui. There would be a timeout for every ipmi command run. I would delete the ipmi plugin directory on the flash drive and start from scratch. You could try running ipmisensors from the command line and see if there's an error.

Huh, interesting.  I will attempt this.  Still wish I could get an accurate temp reading from it though.

 

Also, is there a way to turn off the sensor display on the dashboard homepage?

 

Edit:  Also the footer readings do not appear to stay on the gui, they display for a few seconds after the page loads, then disappears.

Edit2: Oh, removing all dashboard display sensors makes it go away from the dashboard.  Thought that left it up but blank.

Edited December 13, 2017 by lordbob75

[lordbob75]

18 hours ago, dmacias said:

You mentioned earlier you were manually editing the config file. If you have a bad config or the settings are wrong it could lag the webgui. There would be a timeout for every ipmi command run. I would delete the ipmi plugin directory on the flash drive and start from scratch. You could try running ipmisensors from the command line and see if there's an error.

This definitely seemed to be the issue.  Removed and reinstalled it and it works fine aside from the footers not staying visible.

[TUMS]

So I installed the ipmi plugin the other day. Everything seemed to be working good.  Set fan speed minimum to 31.  My fans were all humming away at 3400-3500rpm.  I come back today and my fans are all back down to 2400rpm and my HDD temps are pretty high. Somethings not right. Any ideas? thanks.

My board is X10SLH-F.

[dmacias]

So I installed the ipmi plugin the other day. Everything seemed to be working good.  Set fan speed minimum to 31.  My fans were all humming away at 3400-3500rpm.  I come back today and my fans are all back down to 2400rpm and my HDD temps are pretty high. Somethings not right. Any ideas? thanks.
My board is X10SLH-F.

Minimum of 31 is 31%. So 2400 rpms seems reasonable. What are the other values set to? They are what is going to determine fan speed above that.

[dmacias]

This definitely seemed to be the issue.  Removed and reinstalled it and it works fine aside from the footers not staying visible.

What version of unRAID? What sensors do you have set to show? Do you have the dynamix temp plugin installed?