Troubleshooting poor storage controller performance with ESXTOP

Firstly consult the following logs to check for any controller issues:

/var/log/messages

and

/var/log/vmkernel

We should proceed by SSH'ing into the ESXI host and running the esxtop command:

esxtop

Press d (to go to the device statistics screen, then f (to go to the field selector menu) and finally press j (to add error stats).

From this view we should then look for the DAVG/cmd column (Device Average Latency) - this will give you an idea (latency in milliseconds) of how long the ESXI host is waiting for SCSI commands submitted to the SAN to come back with a reponse. According to VMware - anything greater than 50ms for the local HBA indicates device contention is occuring.

CMDS/s: This is the total amount of commands per second and includes IOPS (Input/Output Operations Per Second) and other SCSI commands such as SCSI reservations, locks, vendor string requests, unit attention commands etc. being sent to or coming from the device or virtual machine being monitored.

DAVG/cmd: This is the average response time in milliseconds per command being sent to the device.

KAVG/cmd: This is the amount of time the command spends in the VMkernel. 

GAVG/cmd: This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG 

** In order to determine which storage controller we should be looking at we can go to the vSphere Client >> ESXI Host >> Configuration >> Storage Adapters and identify the connected devices. **

HP also provide a live CD to perform a thorough diagnostics run on a large series of storage controllers - to download the ISO you should go to:

http://h20564.www2.hp.com/hpsc/swd/public/detail?swItemId=MTX_3c888073127c4c65b7bd8559eb

The current supported list is as follows (as of the 1st of June 2015):

     Smart Array 5312 Controller
     Smart Array 5302 Controller
     Smart Array 5304 Controller
     Smart Array 532 Controller
     Smart Array 5i Controller 
     Smart Array 641 Controller
     Smart Array 642 Controller
     Smart Array 6400 Controller
     Smart Array 6400 EM Controller
     Smart Array 6i Controller
     Smart Array P600 Controller
     Smart Array P400 Controller
     Smart Array P400i Controller
     Smart Array E200 Controller
     Smart Array E200i Controller
     Smart Array P800 Controller
     Smart Array E500 Controller
     Smart Array P700m Controller
     Smart Array P410i Controller
     Smart Array P411 Controller
     Smart Array P212 Controller
     Smart Array P712m Controller
     Smart Array B110i SATA RAID
     Smart Array P812 Controller
     Smart Array P220i Controller
     Smart Array P222 Controller
     Smart Array P420 Controller
     Smart Array P420i Controller
     Smart Array P421 Controller
     Dynamic Smart Array B320i RAID