This tutorial will show you how to configure a VMWare ESXi install to shutdown using an APC SmartUPS. This proven practice came about because our company was sending a Dell PowerEdge T300 server with ESXi and a APC UPS to one of our sales offices. Because it would just be this server and the UPS there, we needed a way for ESXi to shutdown the guest operating systems and then itself gracefully.
This document assumes that you have a VMWare ESXi install with VIMA running on it and an APC SmartUPS with a network management card installed.
This document includes scripts from the VMWare Communities user “lamw.” His scripts are available at http://communities.vmware.com/docs/DOC-9531 Lamw's scripts do not eed the enabling of the unsupported SSH option within ESXi.
Configure the SmartUPS
Install apcupsd on our VIMA virtual machine
Put lamw's scripts onto the VIMA virtual machine and configure
Test
How to configure ESXi to shutdown using an APC SmartUPS
|
|---|
Now we need to add the VIMA virtual machine's IP address into the allowed hosts for the PowerChute Network Shutdown on the UPS.
Open the UPS's Network Management Card's web interface. Login with the username and password of “apc.” Click on the UPS name in the top left.
Click on the PowerChute link in the left panel now and add the VIMA server's IP address to the list of allowed hosts. Click "Add."
Now we will install the latest version of apcupsd from www.apcupsd.com. The latest version as of writing is 3.14.5. Download the x86_64 version made for el5 (link).
Logon to your VIMA VM and run the following to install apcupsd.
sudo rpm -ivh apcupsd-3.14.5-1.el5.x86_64.rpm
Now we need to edit the apcupsd.conf file. Make sure to change to the IP address to the
IP of the UPS.
sudo rm /etc/apcupsd/apcupsd.conf
sudo nano /etc/apcupsd/apcupsd.conf
## apcupsd.conf v1.1 ##
UPSCABLE ether
UPSTYPE pcnet
LOCKFILE /var/lock
DEVICE UPS_IPADDRESS:apc:admin user phrase
UPSCLASS standalone
UPSMODE disable
Save and close.
Now we will have to disable the iptables firewall running on VIMA.
sudo chkconfig iptables off
sudo service iptables save
sudo service iptables stop
Now start the apcupsd daemon
sudo /etc/init.d/apcupsd start
Check is it running
ps ax|grep apc
5332 ? Ssl 0:00 /sbin/apcupsd -f /etc/apcupsd/apcupsd.conf
Wait a few minutes and you should see:
Broadcast message from root (Thu Mar 12 03:40:28 2009):
+
Communications restored with UPS s-vima+
Run "apcaccess" to confirm connection to the UPS.
apcaccess
This command should give you back something like this if it connected successfully.
+APC : 001,051,1224
DATE : Thu Jan 08 11:32:21 GMT 2009
HOSTNAME : s-vima
RELEASE : 3.14.4
VERSION : 3.14.4 (18 May 2008) red hat
UPSNAME : S-VIMA
CABLE : Ethernet Link
MODEL : PCNET UPS Driver
UPSMODE : Stand Alone
STARTTIME: Thu Jan 08 11:32:14 GMT 2009
STATUS :
LINEV : 231.8 Volts
LOADPCT : 5.2 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 62.0 Minutes
MBATTCHG : 10 Percent
MINTIMEL : 5 Minutes
MAXTIME : 0 Seconds
MAXLINEV : 234.7 Volts
MINLINEV : 233.2 Volts
OUTPUTV : 000.0 Volts
SENSE : High
DWAKE : 000 Seconds
DSHUTD : 090 Seconds
DLOWBATT : 05 Minutes
LOTRANS : 208.0 Volts
HITRANS : 253.0 Volts
RETPCT : 015.0 Percent
ITEMP : 29.7 C Internal
ALARMDEL : No alarm
BATTV : 27.5 Volts
LINEFREQ : 50.0 Hz
LASTXFER : Automatic or explicit self test
NUMXFERS : 0
TONBATT : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
SELFTEST : NO
STESTI : 336
STATFLAG : 0x07000000 Status Flag
REG1 : 0x40 Register 1
REG2 : 0x00 Register 2
REG3 : 0x00 Register 3
MANDATE : 12/13/03
SERIALNO : XXXXXXXXXX
BATTDATE : 12/13/03
NOMOUTV : 230 Volts
NOMBATTV : 24.0 Volts
EXTBATTS : 0
FIRMWARE : 600.3.I
APCMODEL : Smart-UPS 1000
END APC : Thu Jan 08 11:33:55 GMT 2009
These steps are taken from http://communities.vmware.com/docs/DOC-9531 by lamw.
Download ghettoShutdown.pl and upsVIShutdown.pl from lamw's page. Put them into “/home/vi-admin”
Set executable.
chmod 755 /home/vi-admin/ghettoShutdown.pl
chmod 755 /home/vi-admin/upsVIShutdown.pl
Edit the following in “upsVIShutdown.pl”
Insert your ESX/ESXi host(s), use the hostname that you used to add to VIMA managment interface, if you're unsure run:
sudo vifp listserver
Remember to make sure the host that is running your VIMA VM is listed as the last entry.
my @hosts = ("esxi.host1","esxi.host2”);
Modify the logoutput of the shutdown process if you like, default will go to “/tmp/upsShutdown.log”
my $log_output = "/tmp/upsShutdown.log";
Insert the displayName of VIMA or VM that is monitoring your UPS device (case sensitive). Very important step else you could end up shutting down the VM that is executing this script, we wouldn't want that.
my $ups_vm_name = "S-VIMA";
Insert the number of seconds to delay after a guestOS has initiated to be shutdown, this will vary depending on the type of applications the VM(s) may be running, tweak this value as needed. Note, the shutdownVM() is a non-blocking function, if you set the delay to be too short the host could be powered down before the VM(s) have completed powering off (default 15secs).
my $sec_to_sleep = 15;
Now we are going to tweak the apccontrol file to change what is done when the UPS issues a shutdown command.
nano /etc/apcupsd/apccontrol
Edit the doshutdown section like so, change the IP address to the ESXi host:
doshutdown)
echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
/home/vi-admin/upsVIShutdown.pl
${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
;;
Save and close the file.
If you go to the "Control" option on the left menu of the APC SmartUPS Network Management Card interface you can run a test to verify everything works. Change the
"Action" to "Turn UPS Off." Make sure that the ESXi server IS NOT being powered off of the UPS (just for testing).
Click "Next" and then "Intiate Control Action."
You should get this message broadcast on the VIMA server you installed apcupsd on.
Broadcast Message from root@s-vima (somewhere) at 11:50 ...
Remote Shutdown. Beginning Shutdown Sequence.
Broadcast Message from root@s-esximon (somewhere) at 11:50 ...UPS s-vima initiated Shutdown Sequence
That's it, done.
Resources
Author
Joseph Holland, Kepak Group.
Reviewers
William Lam (lamw), Innuendo_, llyubenov, Duncan Epping and Steve Chambers.
Disclaimer
You use this proven practice at your discretion. VMware and the author do not guarantee any results from the use of this proven practice. This proven practice is provided on an as-is basis and is for demonstration purposes only. |
Thanks for putting this detailed document together, great work.
I am always leery of disabling a firewall. Instead of disabling the firewall, I added two rules to the top of the INPUT chain in iptables. I didn't add it into the redhat chain that was already thee in case it gets auto generated. This seems to work so far. I have not yet tested a UPS-initiated shutdown. I did not add rules for SNMP or SNMPTRAPS because they seemed to already be there.
Here are the commands that I used. I hope this helps.
sudo iptables -I INPUT 1 -p TCP --dport apcupsd -m state --state NEW -j ACCEPT
sudo iptables -I INPUT 2 -p UDP --dport apc-3052 -m state --state NEW -j ACCEPT
sudo /etc/init.d/iptables save
Thanks Brian. I tried adding rules to VIMA's firewall to allow the UPS to talk with apcupsd, but I could not get it to work properly. If I get a chance to test this (may take a while as I do not have my spare UPS for testing anymore) or you can confirm that the UPS can communicate properly with the apcupsd daemon running on the VIMA VM I will add this into the document.
There are details on testing this here:
http://www.apcupsd.com/manual/Testing_Apcupsd.html#SECTION000145000000000000000
Joseph.
We performed a simulated power outage test of our entire infrastructure as hurricane season is imminent. The test consisted of making sure that our backups were completed and then unplugging both of our UPS from building power.
The VMs running on ESXi as well as the ESXi hosts themselves shut down properly.
It looks like the firewall rules work, but I would encourage someone else to test as well just to double-check.
Hi, I'm using ESXi U4 and version 4 of this document (shutdown via ssh with a script running on ESXi host).
I'm testing the script connecting via ssh to the ESXi U4 host and manually typing in the relevant commands.
Unfortunately when I issue the final command to physically shutdown the host (/sbin/poweroff) it refuses to do it (Poweroff: No.)
To make it work I have to force the shutdown with /sbin/poweroff -f (which causes a non graceful shutdown)
is there a workaround to this? Or am I doing something wrong?
These are the commands I've issued and related messages:
~ # /sbin/shutdown.sh
Running sfcbd-watchdog stop
Stopping sfcbd
Running wsmand stop
Stopping openwsmand
watchdog-openwsmand: Terminating watchdog with PID 1557
Running sfcbd stop
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog start
Running slpd stop
Stopping slpd
Running hostd stop
watchdog-hostd: Terminating watchdog with PID 1475
Running ntpd stop
Stopping ntpd
config implicitly loaded
Saving current state in /bootbank
Clock updated.
Time: 09:22:18 Date: 05/20/2009 UTC
~ #/sbin/poweroff
poweroff: No.
Solved!
It was the "sleep 30", too short to allow my VM to complete a graceful shutdown
I used a longer sleep and all is OK now
Thanks for the very usefull scripts and example!
One question: How can I add a script for remote shutting down my storage? I think it must be done before shutting down the vMA server. Where can i integrate it?
Thanks in advance.
Thanks for this helpful howto.
What version of apcupsd is right for ESX 3.5 hosts? I thought ESX (non-i) was based on something older than RHEL5 (and is i386 and not x86_64) but just checking...
Thanks again.
The apcupsd is not installed on the ESXi host, it's installed on VIMA/VMA which is based on RHEL5 and x86_64.
Thanks, Joseph! I did make one exception from your howto: Because I already have another machine connected to the monitored UPS, I installed and configued the VMA box as a client to this monitor box. Here's the relevant stuff from apcupsd.conf:
UPSCABLE ether
UPSTYPE net
DEVICE myapcupsdmonitor.examplecom:3551
I've tested this setup on other (physical) Linux boxes and it works fine.
I also added '/home/vi-admin/upsVIShutdown.pl' to apccontrol as per your howto.
So here's my question: With my VMA box as a client, will that perl script still shutdown each guest VM or will I need apcupsd instances running on each VM?
I should mention that the guest VMs are a mix of Windows, FreeBSD and OpenBSD. The Windows guest machines will gracefully shut down when hitting stop from the ESX VI client; the BSD boxes don't, so I need to run shutdown on them before powering off from ESX VI.
Thanks again!
How can I add a script for remote shutting down my storage? I think it must be done before shutting down the vMA server. Where can i integrate it?
Thanks in advance.
@FixR - I have sent you a PM about this.
Anyway what OS does your storage server run, is it as easy as running a shutdown command via SSH? If so I think the best place for you to put this script/command is after the "/home/vi-admin/upsVIShutdown.pl" line in the "apcupsd.conf" file. See below:
doshutdown)
echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
/home/vi-admin/upsVIShutdown.pl
/path/to/script/shutdown_storage.sh
${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
;;
The above "should" work, I'm not 100% though. Give it a shot and let me know how you get on.
Joseph.
Thanks to William Lam (lamw), Innuendo_, llyubenov, Duncan Epping and Steve Chambers.