ESXi host shutdown with APC UPS connected via USB

By Brian Farrugia 2nd April 2016

We had a situation were a customer has a single VMware ESXi 5.5 host connected to an APC UPS and he required that the VMs and host shut down cleanly when a power failure occurred and the server was running on the UPS batteries.
When the UPS has a network module, like the APC AP9630, we deploy the PowerChute Network Shutdown for Virtualization Virtual Appliance and configure it to handle the virtual machine and host shutdown. When power is restored it also handles the start-up of the virtual machines.

In the past we used to deploy the VMware vMA appliance and install the PowerChute Network Shutdown for ESX/ESXi application on it to script the shut down process.

In this case the UPS does not have the network module but only USB connectivity.

We decided to install the PowerChute Business Edition(PCBE), which was freely available with the UPS, on a virtual machine on the host. Connect the UPS to the virtual machine and with the use of VMware vSphere PowerCLI, create a script that would shut down the VMs and host.

Configuring the Virtual Machine

Instead of adding a new VM just for this purpose, we decide to us a VM which was used as an multi-purpose Application Server. This VM was hosting some of the applications used to deploy,monitor and manage some services like antivirus and WiFi access points.

These are the steps taken to configure the virtual machine.

Connect the UPS to the ESXi host via USB.
Right click on the virtual machine and click on edit settings.
In the Hardware tab, click on Add and select USB Controller. Click Next and Finish. You can skip this step if controller is already available.
In the Hardware tab, click on Add and select USB Device. Click Next and a list of devices connected to the Host’s USB should be available.
Highlight the APC UPS device as shown below. Your device may be shown differently depending on the model of the UPS.

Click Next and Finish.
Click OK to close the Edit Settings window and apply the changes.
Confirm that the UPS has been identified and installed in the virtual machine.

Configuring PowerChute Business Edition

Download the latest version of PCBE.
Install the Server and Agent components.
In the Device List Configuration wizard add the virtual machine to the Current Device List

The Shutdown Script

The below script has been compiled from postings by Alan Renouf and Patrick Tirlsten.

Alan’s script was the best one I found for the purpose I needed. As someone mentioned in the comments section of his site, the timer keeps resetting when the countdown reaches the first 60 seconds. I modified it slightly to fix this issue.

VMware vSphere PowerCLI needs to be installed on the VM where PCBE is installed for the script to function.

For the shut down to function properly, you need to use an account which has administrator rights on the ESXi host. I have created a new user on our host, named apcups.

The password is stored in an encrypted file, upssecurestring.txt, located in the script folder. Since script will be executed via the PowerChute service, which runs as the SYSTEM user, we need to generate the encrypted password file under that account. This can be done by following the procedure found here by using PSExec. You will also need to set-up the Powerchute Agent Service to Allow Service to interact with desktop.

The script below has been saved as shutdownvms.ps1 in the APC_Scripts folder with the variables changed to work on our environment.

# Script was compiled from contributions by Alan Renouf and Patrick Terlisten
# To create an encrypted password file, execute the following command
# Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File upssecurestring.txt
# Be sure to set $ScriptFolder to the folder where the script is located.
 
#Variable
$ScriptFolder = "C:\APC_Scripts"
$ESXUsername="apcups"
$ESXPassword=Get-Content $ScriptFolder\upssecurestring.txt | ConvertTo-SecureString
$ESXCredentials=New-Object -Typename System.Management.Automation.PSCredential -Argumentlist $ESXUsername, $ESXPassword
$ESXSRVIP = "<ESXi Hostname or IP Address>" 
#Name of the VM where PCBE is installed
$Powerchute_Agent_VM="<Name of the VM where PCBE is installed>"
 
#Add the VMware PowerCLI Snapin to the Powershell Environment
Add-PSSnapin VMware.VimAutomation.Core
#Connect to the ESXi host
Connect-VIServer -Server $ESXSRVIP -Credential $ESXCredentials
 
#Get the ESXi Host object from the virtual infrastructure
$ESXVMHOST=Get-VMHost
 
# Set the amount of time to wait before assuming the remaining powered on guests are stuck or 
# cannot be shutdown because VMware tools are not installed.
$waittime = 120 #Seconds
 
# For each of the VMs on the ESXi host
Write-Output "VM Shut down sequence Started"
Foreach ($VM in ($ESXVMHOST | Get-VM))
{
 if ($VM.Name -eq $Powerchute_Agent_VM) 
 {
 Write-Output "Skipping $VM because it is the PowerChute Agent"
 }
 
 else
 {
 if ($VM.PowerState -eq "PoweredOn") 
 {
 $VM | Shutdown-VMGuest -Confirm:$false
 Write-Output "***** $VM is ON and will be shut down *****"
 }
 Else {Write-Output "!!!!!!!!! $VM is already switched OFF!!!!!!!!!!!"}
 }
 }
$Time = [math]::Round((Get-Date).TimeofDay.TotalSeconds)
do {
 # Wait for the VMs to be Shut down cleanly
 $timeleft = $waittime - $Newtime
# Get number of VMs still powered ON. Deduct the PCBE VM as it will not be shut down
 $numvms = ($ESXVMHOST | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count - 1
 Write-Output "Waiting for shut down of $numvms VMs or until $timeleft seconds"
 sleep 10.0 
 $Newtime = [math]::Round((Get-Date).TimeofDay.TotalSeconds) - $Time
 } until (($numvms) -eq 0 -or $Newtime -ge $waittime)
 
# Shut down the ESXi Hosts
Write-Output "Initiating shut down of the ESXi Hosts"
$ESXVMHOST | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}
$finishtime = Get-Date -format "dd/MM/yyyy HH:mm:ss"
Write-Output "Shutdown Completed on $finishtime "
Disconnect-VIServer -Server $ESXSRVIP -Confirm:$false

# Script was compiled from contributions by Alan Renouf and Patrick Terlisten # To create an encrypted password file, execute the following command # Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File upssecurestring.txt # Be sure to set $ScriptFolder to the folder where the script is located. #Variable $ScriptFolder = "C:\APC_Scripts" $ESXUsername="apcups" $ESXPassword=Get-Content $ScriptFolder\upssecurestring.txt | ConvertTo-SecureString $ESXCredentials=New-Object -Typename System.Management.Automation.PSCredential -Argumentlist $ESXUsername, $ESXPassword $ESXSRVIP = "<ESXi Hostname or IP Address>" #Name of the VM where PCBE is installed $Powerchute_Agent_VM="<Name of the VM where PCBE is installed>" #Add the VMware PowerCLI Snapin to the Powershell Environment Add-PSSnapin VMware.VimAutomation.Core #Connect to the ESXi host Connect-VIServer -Server $ESXSRVIP -Credential $ESXCredentials #Get the ESXi Host object from the virtual infrastructure $ESXVMHOST=Get-VMHost # Set the amount of time to wait before assuming the remaining powered on guests are stuck or # cannot be shutdown because VMware tools are not installed. $waittime = 120 #Seconds # For each of the VMs on the ESXi host Write-Output "VM Shut down sequence Started" Foreach ($VM in ($ESXVMHOST | Get-VM)) { if ($VM.Name -eq $Powerchute_Agent_VM) { Write-Output "Skipping $VM because it is the PowerChute Agent" } else { if ($VM.PowerState -eq "PoweredOn") { $VM | Shutdown-VMGuest -Confirm:$false Write-Output "***** $VM is ON and will be shut down *****" } Else {Write-Output "!!!!!!!!! $VM is already switched OFF!!!!!!!!!!!"} } } $Time = [math]::Round((Get-Date).TimeofDay.TotalSeconds) do { # Wait for the VMs to be Shut down cleanly $timeleft = $waittime - $Newtime # Get number of VMs still powered ON. Deduct the PCBE VM as it will not be shut down $numvms = ($ESXVMHOST | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count - 1 Write-Output "Waiting for shut down of $numvms VMs or until $timeleft seconds" sleep 10.0 $Newtime = [math]::Round((Get-Date).TimeofDay.TotalSeconds) - $Time } until (($numvms) -eq 0 -or $Newtime -ge $waittime) # Shut down the ESXi Hosts Write-Output "Initiating shut down of the ESXi Hosts" $ESXVMHOST | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)} $finishtime = Get-Date -format "dd/MM/yyyy HH:mm:ss" Write-Output "Shutdown Completed on $finishtime " Disconnect-VIServer -Server $ESXSRVIP -Confirm:$false

The script will add the required PowerCLI snapin to the session. It will connect to the ESXi host specified in $ESXSRVIP and obtain the object of the host. It will then proceed to shut down all the VMs except the VM where the PCBE is installed. Every 10 seconds, it will check how many VMs are still ON. If the $waittime is exceeded, the shutdown of the Host will start. During the process, Write-Output is used to log the time the shut down started and the time it shut down the host. Additional information is also provided for VM power state and VM shut down progress.

The Log file will look something like this

Shutdown of VMs and ESXi Host started at 02/04/2016 17:54:13.56 
 
Name                           Port  User                          
----                           ----  ----                          
<ESXSRVIP>                      443   apcups                        
VM Shutdown sequence Started
!!!!!!!!! <VMNAME> is already switched OFF!!!!!!!!!!!
***** <VMNAME> is ON and will be shutdown *****
Skipping  <VMNAME>  because it is the Powerchute Agent
***** <VMNAME> is ON and will be shutdown *****
***** <VMNAME> is ON and will be shutdown *****
!!!!!!!!! <VMNAME> is already switched OFF!!!!!!!!!!!
***** <VMNAME> is ON and will be shutdown *****
***** <VMNAME> is ON and will be shutdown *****
!!!!!!!!! <VMNAME> is already switched OFF!!!!!!!!!!!
***** <VMNAME> is ON and will be shutdown *****
Waiting for shutdown of 6 VMs or until 120 seconds
Waiting for shutdown of 6 VMs or until 110 seconds
Waiting for shutdown of 5 VMs or until 99 seconds
Waiting for shutdown of 4 VMs or until 89 seconds
Waiting for shutdown of 3 VMs or until 79 seconds
Waiting for shutdown of 3 VMs or until 68 seconds
Waiting for shutdown of 2 VMs or until 58 seconds
Waiting for shutdown of 2 VMs or until 48 seconds
Waiting for shutdown of 2 VMs or until 37 seconds
Waiting for shutdown of 2 VMs or until 27 seconds
Waiting for shutdown of 2 VMs or until 17 seconds
Waiting for shutdown of 2 VMs or until 6 seconds
Initiating shutdown of the ESXi Hosts

Two of the VMs on the host do not have the VMware tools installed and so the script had to wait for all the the time specified in $waittime before shutting down the host.

It is suggested to configure the Virtual Machine Startup/Shutdown options to shut down the PCBE VM when the host is shut down and to power OFF VMs that do not have VMware tools installed as per below. You may also want to configure automatic start-up for when the power is restored if the host’s BIOS settings are configured to Power ON when power is restored.

Configuring the PowerChute Agent

Now that the applications have been installed and the script modified for our needs, we need to setup the Powerchute Agent to execute the shutdownvms.ps1 script.

On the virtual machine, navigate to the location “C:\Program Files (x86)\APC\PowerChute Business Edition\agent\cmdfiles”. There you should find a file named default.cmd. Edit the contents of the file and add the below. If you changed the location of the Script Folder or the name of the shutdownvms.ps1 file, make sure to amend the below line accordingly.

The out-file -Encoding ascii -filepath C:\APC_Scripts\shutdownlog.txt -Append will allow us to log the progress of the shut down.

1	@START "" "%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe" "C:\APC_Scripts\shutdownvms.ps1 \| out-file -Encoding ascii -filepath C:\APC_Scripts\shutdownlog.txt -Append"

Now open the Agent Web Interface and login.

The first time you login, you will need to complete the configuration wizard. In the Shutdown Settings section make sure that the command file, you added the above line to, has been selected. It is important to allow enough time for the virtual machines to shut down in the time required to run the command file. In the script, the $waittime was set to 120 seconds. That is the time the script will wait before shutting down the host. When shutting down the host, the Powerchute virtual machine will be shut down. The Time for operating system to shut down should be set to allow enough time for the VM and the host to shutdown properly.

Once all options are set, I’d suggest to schedule a time to test the script to ensure all is working properly for when the time comes when it is really need.

Please leave a comment should you have any suggestions or questions about the above procedure.

Thanks for visiting

By Brian Farrugia

I am the author of Phy2Vir.com. More info can be found on the about page.

Linux Microsoft Powershell

27 thought on “ESXi host shutdown with APC UPS connected via USB”

Shahid says:

31st August 2016 at 20:11

Hi,

Thanks for the great article.

Do you know if it can send alerts via e-mail too?

Thanks,

Reply
1. Brian Farrugia says:
  
  31st August 2016 at 20:34
  
  Hi,
  Thanks for the feedback. Really appreciate it.
  You should be able to setup e-mail notifications when installing Powerchute Business Edition. This should allow you receive e-mail notifications related to the ups power and health states.
  If I can access the system again I will try to confirm the above.
  
  Reply
Keaton says:

2nd March 2017 at 11:59

Hi

I have a similar situation and I will be following your steps. Im using ESXI 6 though so I hope it will still work.

Reply
1. Brian Farrugia says:
  
  2nd March 2017 at 12:17
  
  Hi Keaton,
  It should work.
  Please let us know how it goes.
  
  Brain
  
  Reply
  1. Keaton says:
    
    16th November 2017 at 13:52
    
    Sorry I didnt reply sooner but this solution did work for me thanks!
    
    Reply
    1. Brian Farrugia says:
      
      16th November 2017 at 14:14
      
      Great! Thanks for updating us.
      
      Reply
Fabian Fritsch says:

16th November 2017 at 10:50

Just implemented this solution under ESXi / vCenter 6.0, works like a charm.

Thank you for the great guide, this was really a life safer for me!

Reply
1. Brian Farrugia says:
  
  16th November 2017 at 12:31
  
  Hi!
  Very happy you found it useful.I really appreciate you leaving a comment and letting me know
  Thank you
  
  Reply
Jorden says:

3rd January 2018 at 14:32

How can this work if you have a mixture of windows servers and linux?

Reply
1. Brian Farrugia says:
  
  4th January 2018 at 09:02
  
  Hi,
  the windows and linux servers you refer to are they VMs or Physical Machines.
  If they are VMs, you need to make sure that VMware tools are installed on them so that the process can perform a clean shutdown.
  If they are physical machines, you can install the PowerChute Business edition agent on the Physical machines and point it to the the VM where the PowerChute Business Server is installed.
  
  Hope the above is useful.
  
  Reply
JP Bunt says:

10th March 2018 at 13:25

Nice post! Thanks for sharing. The saves me from buying a management card for the UPS.

I followed the steps and it works as expected.

One remark: With ESXi 6.0 the command to shut down the VMs is changed to: $VM | Stop-VMGuest -Confirm:$false

Reply
1. Brian Farrugia says:
  
  12th March 2018 at 10:10
  
  Hi,
  thanks for letting us know. I am pleased this post is still helping people out.
  Also, thank you for pointing out the new commandlet.
  
  Reply
Toledo says:

4th May 2018 at 22:39

Very good post. Thank you for taking the time to share.

Reply
Luis Cruz says:

17th September 2018 at 21:11

Hi Brian, and thanks for sharing your knowledge.
I have a esxi 5.5 home lab with about 6 vms and tried your solution.
I do seem to have a problem the apc ups SMT750 is shutting down before the esxi host, the windows 7 workstation running the script does shutdown properly. here is the script’s log saved.
VM Shut down sequence Started
Waiting for shut down of VMs or until 120 seconds
Waiting for shut down of VMs or until 110 seconds
Waiting for shut down of VMs or until 100 seconds
Waiting for shut down of VMs or until 90 seconds
Waiting for shut down of VMs or until 80 seconds
Waiting for shut down of VMs or until 70 seconds
Waiting for shut down of VMs or until 60 seconds
Waiting for shut down of VMs or until 50 seconds
Waiting for shut down of VMs or until 40 seconds
Waiting for shut down of VMs or until 30 seconds
Waiting for shut down of VMs or until 20 seconds
Waiting for shut down of VMs or until 10 seconds
Initiating shut down of the ESXi Hosts
Shutdown Completed on 17/09/2018 12:44:17

It claims that the esxi host has been properly shut down but I know that when the ups cuts power the host still up and running.
also in the esxi host logs there is no entry about shutting down only entry about boot.
Any ideas

Reply
1. Brian Farrugia says:
  
  17th September 2018 at 22:07
  
  Hi,
  my understanding from the log you attached is that the script is not managing to open a connection to the ESXi host
  The output you have seems to have the number of VMs on the host that are powered ON missing.
  Write-Output “Waiting for shut down of $numvms VMs or until $timeleft seconds”
  $numvms< should contain the number of VMs that are powered ON and need to be shutdown but in your output this value is missing. Make sure that the secure string text file is generated by executing the command Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File upssecurestring.txt
  in a powershell window which is running under the SYSTEM user by using PSEXEC as mentioned in the post.
  Also make sure that you have PowerCLI installed.
  If all is in place, you should be able to use PSexec to start a powershell session and run the script manually without needing to initiate it from the UPS by unplugging the power.
  Hope this helps.
  
  Reply
wassim says:

9th July 2019 at 19:39

thanks to shating your knowledge ,
i need to use in production envirenometn with esxi 6.7 , do you have an odea if it’s work or not
thanks

Reply
1. Brian Farrugia says:
  
  9th July 2019 at 21:06
  
  Hi Wassim, thank you for your comment. I have not tested it with ESXi 6.7 yet. At this moment I do not see why it shouldn’t unless some commands changed. Let us know if you try it 🙂
  
  Reply
Igor says:

11th July 2019 at 13:04

Hi! Help me, please)

ESXi 6.7 and PCBE 9.5, SmartUPS SRT 2200.

New-Object: Cannot find overload for “PSCredential” and the number of arguments: “2”.
C:\APC_scripts\shutdownvms.ps1: 11 sign: 17
+ … Credentials = New-Object -Typename System.Management.Automation.PSCrede …

Connect-VIServer: Unable to check the argument for the “Credential” parameter. The argument is empty or null. Specify a non-empty argument, and not
the value is NULL, and then repeat the command.
C:\APC_scripts\shutdownvms.ps1: 19 sign: 48
+ Connect-VIServer -Server $ ESXSRVIP -Credential $ ESXCredentials

I wrote the password as encrypted and not encrypted.
What could be the problem?

Reply
1. Brian Farrugia says:
  
  12th July 2019 at 06:42
  
  Hi,
  did you use the powershell command Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File upssecurestring.txt to generate your upssecurestring.txt file with the SYSTEM account?
  Did you place the txt file in the c:\APC_Scripts folder, or the scripts folder you setup?
  I am only receiving your error when I do not have that txt file in place.
  Hope this is helpful
  
  Reply
Oleg says:

2nd August 2019 at 09:40

My virtual machine $Powerchute_Agent_VM does not turn off on their own. It turns off by ESXi Hosts. Because of this, UPS does not turn off. Why is this happening?

log:
Name Port User
—- —- —-
192.168.100.100 443 apcups
VM Shut down sequence Started
Skipping WIN2016_1C because it is the PowerChute Agent
Waiting for shut down of 0 VMs or until 120 seconds
Initiating shut down of the ESXi Hosts

Type : Task
Value : haTask-ha-host-vim.HostSystem.shutdown-2143835752

Shutdown Completed on 01.08.2019 18:51:16

Reply
1. Brian Farrugia says:
  
  2nd August 2019 at 09:29
  
  Hi,
  it seems the script is working as it should. MAke sure you added the PowerChute VM, Win2016_C, in the Virtual Machine Startup/Shutdown settings on the ESXi host as mentioned in the blog post.
  Switching off the UPS after the script has run may not be supported via USB and Powerchute Business edition. I am not sure about this. You may need to reach out to APC through the fora.
  I know we can do it when using APC PowerChute Network Shutdown and the APC Network card on supported UPSs.
  
  Reply
Radu Stefan says:

11th April 2020 at 21:02

I have a small issue, when I run the script manually it works perfectly but not when run by the PowerChute Agent.

Reply
1. Brian Farrugia says:
  
  13th April 2020 at 14:41
  
  Hi Stefan,
  thanks for visiting. Most probably the issue here is that you didn’t store the encrypted password into the file using the SYSTEM account. If you do that you would need to run the script with the SYSTEM user for it to work.
  
  You need to use PSexec as explained here https://www.phy2vir.com/control-datacore-ssv-write-cache-with-apc-pcns/ to get it to work properly.
  You also need to set the Service “Powerchute Agent” to “Allow Service to interact with desktop” as mentioned in the link above.
  
  Let us know how it goes.
  Stay Safe
  Brian
  
  Reply
  1. Radu Stefan says:
    
    13th April 2020 at 23:13
    
    I have encrypted the password using PSexec but I still get the same result, the script does not work. What else can I do to test it ?
    
    Reply
    1. Radu Stefan says:
      
      14th April 2020 at 00:02
      
      I have fix the problem by replacing “$ESXPassword=Get-Content $ScriptFolder\upssecurestring.txt | ConvertTo-SecureString” with “$ESXPassword=ConvertTo-SecureString ‘insertpasswordhere’ -AsPlainText -Force” and it looks like the script now works.
      
      Reply
      1. Brian Farrugia says:
        
        1st May 2020 at 14:41
        
        Hi,
        yes that will work too but you are exposing the password in your script. If that is not a concern to you, then I am happy you managed to fix the issue and are using the script provided.
        
        Stay safe
        Brian
Max says:

26th May 2020 at 23:58

For proper work you need to replace string
Add-PSSnapin VMware.VimAutomation.Core
with
Import-Module VMware.VimAutomation.Core

Reply