Disk shrink does not work on ESXi guests
Actually, there is a way to shrink a VM on ESXi, although you need to shut down the VM for it. Here's how:
Zero all unused space inside the VM:
dd if=/dev/zero bs=1048576 of=/zero ; sync ; rm /zero
Do the same with other mount points, swap partitions, etc.
Shut down the VM.
SSH to ESXi, and issue this command:
vmkfstools -K /vmfs/volumes/volumename/vmname/vmname.vmdk
This will take some time... vmkfstools "punches holes" inside your VMDK, e.g. deallocates all blocks that are filled with zeros, effectively shrinking your VMDK.
http://www.vmware.com/pdf/vmware-tools-cli.pdf Page 13 :
Shrinking disks is not allowed under the following circumstances:
The virtual machine is hosted on an ESX/ESXi server. ESX/ESXi Server can shrink the size of a virtual disk only when a virtual machine is exported. The space occupied by the virtual disk on the ESX/ESXi server, however, does not change.