We recently needed to installed and configure an ESXi 6.0 cluster on a set of PowerEdge M710HD Blade Servers. At first we wanted to setup a vSphere 6.5 cluster but after checking with the VMware HCL Guide we noted that this version is not supported and thus we had to proceed with 6.0.
We decided to use the DELL Customized ESXi ISO. At the time the latest version available here was VMware-VMvisor-Installer-6.0.0.update02-4192238.x86_64-Dell_Customized-A04.iso.
During the installation of the hypervisor, the PSOD below appeared.This happened on all the hosts in the chassis.
At the same time, the System Event Log logged the below errors
As you can see above, the errors point to the issue being caused due to a hardware failure.
In order to identify which component was failing, we booted the host with a Linux live DVD, in our case the DELL Support Live Image DVD, and in a terminal session we executed the command lspci
This command lists all the devices attached to the PCI bus. From the output it was noted that Bus 4 device 0 function 0 was referring to the Brocade CNA device, BR1741M-k,
whilst the device on bus 0 device 7 function 0 was the PCI Express controller port 7 where the Brocade CNA was connected:
At this stage we went again on the VMware HCL Guide page and looked for the compatibility between the BR1741M-k card and ESXi 6.0 U2
The card was indeed compatible with the version of ESXi we were installing and the latest firmware supported firmware was 126.96.36.199
We rebooted the server and upon boot we checked the FW version of our card and found it to be an older version; 188.8.131.52
At this stage we decided to update the firmware by downloading it form the QLogic OEM partners site, in this case DELL, but the latest version we could find on that site was 184.108.40.206. After a quick search we came across this whitepaper from DELL that the latest firmware could be downloaded from the BR-1020 download site. The site contained the latest firmware version, 220.127.116.11, but since it was not in the HCL we click on previous version.From there we proceeded with downloading the Multi-Boot Code Live DVD version 18.104.22.168 and booted the server off it. In our case the BIOS boot mode had to be changed from UEFI to BIOS mode for the boot DVD to work properly.
Once booting completed we were presented with a command line prompt where we typed the command :
# bcu boot --update brocade_adapter_boot_fw_v3-2-4-0 -a
The firmware/bootcode update process started
We then rebooted the server by typing the command reboot. During the boot cycle we confirmed that the FW was updated and changed back the boot mode to UEFI.
We then proceeded to install ESXi and this time we were not presented with the PSOD.
The issue we encountered seemed to be resolved by applying the firmware update on the card.
At this stage we wanted to make sure that the cards were detected properly and that the proper drivers were installed.
To our surprise, when we ran the command esxcli network nic list, the Brocade cards were not listed. Only the embedded Broadcom adapters were list.
Using esxcli software vib list or esxcli software vib get we didn’t find any vibs installed relating to the Brocade cards.
We didn’t expect this as we had used the Customised DELL ESXi DVD and expected these drivers to be injected. We downloaded the driver file which was linked in the HCL guide from the VMware website. Once downloaded we extracted the Offline bundle, BCD-bna-22.214.171.124-00000-offline_bundle-1634921, and uploaded it via WinSCP to /tmp on the ESXi host.
Once uploaded we placed the host in maintenance mode by executing
esxcli system maintenanceMode set --enable true
Then we installed the Offline Bundle by running the command:
esxcli software vib install -d /tmp/BCD-bna-126.96.36.199-00000-offline_bundle-1634921.zip
Below is the output of the command
Once done we rebooted the host and upon reboot we checked again if the network adapters were detected and if the corresponding VIB was installed
Once that was confirmed, we took the host out of maintenance mode and proceeded with the configuration.
Hope the above is helpful to someone. If it is helpful or you have something to add, please leave a comment