APCUPSd: Custom commands at shutdown

[mattlach]

Hey all,

So I have been doing a bunch of reading on this and come up a little short.

Many guides suggest editing the /etc/apcupsd/apcontrol script to insert custom commands at shutdown time, but the comments in the file itself warns against doing this, as this file is replaced during updates.

There are many references to a "proper" way of doing this, but I have googled a lot and not come up with what this proper way is.

In my setup, I would like to use plink to send an ssh command to shutdown another host when battery power gets low.

I tried this by entering commands into apccontrol (like is recommended against) and my commands that work fine in shutting down the remote system when entered directly in console, are not having any effect once called from apccontrol. I am currently assuming this is because the server with apcupsd installed is running on an ssd, and thus shuts down VERY fast, possibly before the completion of my command.

Does anyone have any suggestions?

I was thinking of maybe creating a cronjob that runs on shutdown, but I have found cron - while great in time based commands - to be very unreliable when it comes to running things at startup and shutdown.

I'd appreciate any suggestions!

--Matt

[mattlach]

Actually, never mind, finally found this:

http://linux.die.net/man/8/apccontrol

If anyone has a good example of of a doshutdown script, showing how to correctly exit with status 0 or status 99, I'd really appreciate it!

--Matt

[tgalati4]

apcupsd is an old framework and there may be better ways to do this. Newer versions of Ubuntu have some apc shutdown capability already built in, so you want to research that first.

Using the existing apcupsd framework, you can set up a master and slave arrangement such that the UPS is connected to the master and any other computer on the LAN can be a slave (running apcupsd in slave mode) and will get a shutdown signal through ethernet. This assumes that your router has some UPS protection and any running PC's on your network are either laptops (with working batteries) or other desktops with dumb (not-connected) UPS's.

Many smartups have firmware that can be configured (through dip switches) to perform graceful shutdowns when the battery limit is reached, so you don't really do the doshutdown script. I would just set the shutdown delay to 60 seconds, send a network message out (this happens automatically), and have the machine shutdown after this 60 seconds. If your power is out for a minute, there is a good chance it will stay down for a while anyway.

If the power just flickers (because of maintanence on the lines, or trees brushing on the power lines) your system will stay up.

You will have to describe your Use Case in more detail. I would make a flowchart to show how and when you want your system (and other devices) to shut down and under what conditions.

Some other ups links:

http://blog.shadypixel.com/monitorin...-ubuntu-linux/

http://ubuntuforums.org/showthread.p...hlight=apcupsd

[mattlach]

First off, thank you for taking the time to help, I really appreciate it!

You will have to describe your Use Case in more detail. I would make a flowchart to show how and when you want your system (and other devices) to shut down and under what conditions.

I will start with this, as it may help in the following questions. Unfortunately I can't draw a flowchart right now, so my text description of my setup will have to do.

The Ubuntu server in this case, is a dedicated headless mini-install used ONLY for UPS monitoring.

It resides as a guest OS on my VMWare ESXi server, along with numerous other guests.

I have two separate UPS:s, one APC Smart-UPS 1500. (Only the ESXi host is plugged in to this one) and one APC Smart-UPS 750 (Switch, Wireless AP, FIOS ONT, and a handful of other small devices plugged into this one)

My router is actually pfSense running as a guest on the same ESXi host above.

Nothing on the smaller 750 unit requires a tidy shutdown, so I figured I'll just let it run dumb until battery is drained and it shuts itself down.

The ESXi host, however needs a tidy shutdown, and the Ubuntu server guest with apcupsd is intended to accomplish this.

It all looks like this (UPS:es not pictured) to put it into context.

apcupsd is an old framework and there may be better ways to do this. Newer versions of Ubuntu have some apc shutdown capability already built in, so you want to research that first.

I have done some reading here. APC does have a version of their PowerChute software available as a virtual appliance that can be installed in ESXi, but unfortunately this requires a fully licensed version of ESXi with vMA. Since this is a basement server, I can't afford a license like that, and am relying on the free standalone license.

I tried NUT at first, as NUT actually has a package that can be installed in ESXi as a slave for tidy shutdown purposes, but unfortunately I have found that NUT has rather poor compatibility with APC's Smart UPS models. Over the serial connection the NUT server just identifies it as an "unknown device" and refuses to do anything with it. Using the USBHID driver it detects it, but then immediately the NUT server segfaults. Same thing happens when I try to do it over SNMP over ethernet. Detects the unit, followed by an immediate segfault.

Based on this I decided to go with APCUPSd.

Using the existing apcupsd framework, you can set up a master and slave arrangement such that the UPS is connected to the master and any other computer on the LAN can be a slave (running apcupsd in slave mode) and will get a shutdown signal through ethernet. This assumes that your router has some UPS protection and any running PC's on your network are either laptops (with working batteries) or other desktops with dumb (not-connected) UPS's.

Unfortunately I am not aware of any version of the apcupsd client that is able to run natively in ESXi.


So, based on these limitations the setup I have in mind is as follows.

The mini dedicated Ubuntu server monitors the UPS via APCUPSd. This works beautifully. It detects it over the serial interface, and I can get all the information from it using the apcaccess command.

If power is lost, and the battery is drained down to my preset timeleft setting (currently I have it set to 10 minutes) I want it to use plink to ssh in to the ESXi server, and issue the command for orderly shutdown of all guests, followed by system shutdown.

(this happens to be ""/sbin/shutdown.sh && /sbin/poweroff")

Some people (like this guy) have had luck with this in editing the /etc/apcupsd/apccontrol script, but this doesn't work for me.

While using plink manually directly from the command line has the desired effect (shuts down all guests in an orderly fashion, followed by ESXi host shutdown) once embedded in the apccontrol script it does not work at all. I'm guessing this is because my server is very small and lightweight, and runs on a very fast SSD, so it shuts down very quickly once the poweroff command is issued, and thus the plink command to the ESXi host is never issued completely.

Based on this, I think what I want is to ignore the normal shutdown behavior of apcupsd, (since the guest will shutdown orderly once ESXi gets the command to do so anyway) and just have it issue the shutdown command to the ESXi host.


Based on my reading thus far, I think a doshutdown script that looks like this should accomplish that (or at least so I hope)

#!/bin/bash
plink -ssh -2 -pw password root@xxx.xxx.xxx.xxx "/sbin/shutdown.sh && /sbin/poweroff"
exit 99

Is that correct? (this is really simple and barely worth calling a script, but bash scripting is not my expertise by any stretch of the imagination) Then if I give it the "exit 99" it should ignore the normal shutdown behavior in apcontrol, and ONLY issue this shutdown line to ESXi.

After this, ESXi should then shut down each guest in orderly fashion, including the dedicated UPS Ubuntu server (as defined in ESXi configs) and shut itself down.

Not quite sure what will happen to the UPS shutdown itself after this. I read somewhere that with SmartUPS models you can define a delay after which the UPS shuts down, but I am not sure where this is (maybe in the web cofigurator for the UPS itself, or somewhere in the apcupsd config file?)

I have done numerous timed shutdowns of my host, and shutting down all the guests, followed by a host shutdown takes repeatedly 2min 35s (+/- a couple of seconds) so I figure I should be safe if I set the UPS delay to 5 minutes. Maybe even 10 (I currently have everything set to trigger a shutdown when TIMELEFT reaches 10 minutes, just to be on the safe side.

I'd appreciate any thoughts or insights you might have into this setup.

Thank you again,
Matt

[tgalati4]

Well, running an EXSi hypervisor does add some complexity. If you were running Xen, then you would have a linux hypervisor with apcupsd running in host process table and then could shut down your vm's in a controlled fashion.

So search the web, I found a few of these links:

https://communities.vmware.com/docs/DOC-11482

http://serverfault.com/questions/462...nected-via-usb

The second link discusses how you can use a NUT slave in an EXSi vm to trigger a shutdown. Understand that NUT is a generic UPS framework and apcupsd is a specific UPS framework for APC Smartups and similar devices that use the smartups commands.

Because of the length of time to shut down EXSi vm's, timing is important so that you have enough battery power to keep the server running while you dump vm clients as quickly as possible.

Otherwise, use the approved, corporate solution--buy the license and use powerchute.

[mattlach]

Well, running an EXSi hypervisor does add some complexity. If you were running Xen, then you would have a linux hypervisor with apcupsd running in host process table and then could shut down your vm's in a controlled fashion.

So search the web, I found a few of these links:

https://communities.vmware.com/docs/DOC-11482

http://serverfault.com/questions/462...nected-via-usb

The second link discusses how you can use a NUT slave in an EXSi vm to trigger a shutdown. Understand that NUT is a generic UPS framework and apcupsd is a specific UPS framework for APC Smartups and similar devices that use the smartups commands.

Because of the length of time to shut down EXSi vm's, timing is important so that you have enough battery power to keep the server running while you dump vm clients as quickly as possible.

Otherwise, use the approved, corporate solution--buy the license and use powerchute.

Thank you for that. Yes, it does add a little bit of complexity.

I vaguely recall playing with Xen back when I originally set this up in 2010, but I wasn't as happy with it as I was with ESXi Free. Took 4 years for me to finally cross "get a UPS for the server" off my to do list

As mentioned I did try NUT, but the NUT server seems incompatible with my APC SmartUPS 1500 (SUA1500) model, which is unfortunate.

The corporate approved method would be great, and probably worth it for a corporation, but for a home tinkerer like myself, the ESXi licenses are just a little bit too pricey, so I am stuck with the free license.

It's too bad there isn't a cheaper home use license for ESXi, but then again, there probably aren't that many of us playing with ESXi servers at home

I had a thought. What if the NUT client were compatible with the apcupsd server, so I could run apcupsd on the server, and set up the NUT client to get information from it and handle the shutdown, but that probably isn't the case.

I am going to try creating the doshutdown script with my plink command in it and see if that does it. (fingers crossed)

[volkswagner]

Is your pfsense machine capable of connecting to the smart UPS?

Perhaps pfsense can be the "leader of the pack"?

[mattlach]

Is your pfsense machine capable of connecting to the smart UPS?

Perhaps pfsense can be the "leader of the pack"?

Possibly. What would be the benefit of this? Does pfSense have good built in UPS support?

I generally try to avoid doing any custom stuff in pfSense and FreeNAS, partially because they are based on limited BSD images to save space, and partially because I find the default BSD console clunky and annoying compared to Ubuntu and other Linux distributions.

Appreciate your suggestion!

[tgalati4]

You can get the same functionality under freenas, you just have to add the correct packages and set up the configuration just as you would in Linux. Xen has improved a lot in 4 years. For the power consumption of a 96GB, Xeon server, you can buy new UPS each year for each device.

Installing apcupsd on freenas: https://forums.freenas.org/index.php...ps-lost.23348/

Is your UPS similar to this? http://forums.apc.com/message/49943

An interesting discussion on why APC does not support newer UPS's in linux: http://forums.apc.com/thread/5374

[mattlach]

You can get the same functionality under freenas, you just have to add the correct packages and set up the configuration just as you would in Linux.

Oh, I know I can, I guess I was wondering if he was suggesting there was any benefit in doing so, over the approach I have gone down with Ubuntu, which I much prefer due to the ease of the package manager, better documentation and my familiarity with all the command line syntax of commn commands, which is very subtly different in BSD, but enough to be an annoyance. (that, and I never understood why my delete key never works in BSD, instead inserting an annoying character )

I find it much nicer to manage my Ubuntu server, so to me, it will well worth the extra shared 128MB of RAM and one shared core that mostly idles, it cost me to install a basic dedicated Ubuntu server for the task.

Xen has improved a lot in 4 years.

I'm sure it has. I would love to try it again, as I much prefer open source initiatives, but I have everything set up the way I like it, and would need to find lots of time to start over with Xen. It's not a production server, but down time does matter, as I run home stuff off of it (including all the house TV through a MythBuntu backend running as a guest)

[QUOTE=tgalati4;13148538]
For the power consumption of a 96GB, Xeon server, you can buy new UPS each year for each device.

Well, I built it with low power L5640's, a free flow 4U case, so fan speeds can be lower, and used an 80+ platinum power supply, so power use is actually very reasonable.

Besides, what would be the use of having a bunch of UPS's and nothing to plug into them?

Installing apcupsd on freenas: https://forums.freenas.org/index.php...ps-lost.23348/

Yeah, I've seen that, thank you.

Unless there is a really good reason to do so, I prefer to keep my appliance installs (pfSense and FreeNAS) as appliance as possible, and not mess with installing packages and jails if I can avoid it.

Is your UPS similar to this? http://forums.apc.com/message/49943

No, the X series are newer post-Schneider acquisition models. My SUA1500 (and SUA750) are older models. They still use some stupid proprietary stuff (like non-standard serial cables) but nowhere near as bad as the newer models released after Schneider acquired the company.

An interesting discussion on why APC does not support newer UPS's in linux: http://forums.apc.com/thread/5374

This is interesting, considering their official PowerChute VM appliance is linux based (RHEL I believe)

Ever since the Schneider acquisition they have been acting very stupidly in regards to making everything opaque and closed and not interracting with the open source community. it is too bad.

There are few UPS:es out there as effective and bulletproof as their old Smart-UPS models, with their true sine wave designs, and dependable builds, which is why so many of them are still in use by hobbyists and companies 10 years after they were originally built.