Here are the default devices for the Windows XP Professional VM:

Since I had built the 5 Windows XP VMs from a template I had created, I decided to find a way to convert the IDE disk to a SCSI disk and update the template incase I needed to resize the OS disk in the future. The easiest way I found to do this was as follows:-

Firstly I made the following assumptions:

• I would need to add a SCSI Controller to the VM and install the drivers before booting from the VMDK (I have seen cases before where moving a HD to new hardware caused a BSOD)
• If possible I would need to change the default Controller used for new VMDKs from IDE to SCSI
• I wanted to use the LSI Logic SCSI Controller
• I wanted to make the VMDK the first SCSI device (0:0) attached to the Controller

So, with the VM powered OFF I edited the settings for it through the VI Client and added a new SCSI Device (by default it doesn't seem possible to simply add a SCSI Controller to a VM without also attaching a SCSI Device to it):

I left the SCSI Device as being a CDROM and (optionally) changed the Virtual Device Node to SCSI (0:1) so that when I converted the existing disk and added it to the SCSI Controller it would be device SCSI (0:0). I clicked Next and Finish but I didn't click OK to commit the changes:

Before clicking OK, I selected the SCSI Controller and changed it's type as appropriate. Then I clicked OK to commit the changes to the VM:

I powered on the VM and noticed that the Windows Found New Hardware Wizard prompted me to install the driver for the SCSI controller. I knew that the controller used is a LSI20320-R so I downloaded it from here:

LSI20320-R - Driver Download (Click the Support and Downloads tab)

I downloaded the appropriate driver and extracted it (for Windows XP 32bit I used LSI20320-R_xp_50700_01034132IT_1201800_1005239.zip and within that symmpi_wXP_1201800.zip.) Then I used the Found New Hardware Wizard to browse for and install the driver. Once the hardware installation completed I shut down the VM. The VM now had the SCSI controller installed and I was confident I would be able to boot from it.

I then located the datastore where the VM resides. For example:

/vmfs/volumes/<datastore>/<vm>/

You can use the Service Console or an SSH\SCP application to open the <vm>.vmdk (not the <vm>-flat.vmdk). I edited the following line:

ddb.adapterType = "ide"

For the LSI SCSI Controller I changed it to:

ddb.adapterType = "lsilogic"

After saving the file I used the VI Client to edit the VM settings. I selected the IDE disk and removed it. I then clicked OK to commit the changes.

NOTE: Remember to only 'Remove from virtual machine' and NOT 'Remove from virtual machine and delete files from disk':

Then I used the VI Client to edit the VM settings again and add the disk back. This time, because I had edited the .vmdk file to specify that the disk uses the LSI Logic Controller, it attaches the disk to this instead of the IDE Controller used previously:

The disk is added as device SCSI (0:0):

I clicked Next and Finish to complete adding the disk. I then selected the CDROM (SCSI device 1) and removed it:

I clicked OK to commit the changes to the VM. I now had a VM using the LSI Logic SCSI Controller with the System OS disk connected to the Controller as device SCSI (0:0).

Since the VM had already booted with the LSI Logic Controller installed, when I powered on the VM Windows booted normally and was immediately accessible. Subsequently, any new virtual disks created for the VM are automatically created as SCSI disks and attached using the next available Virtual Device Node.

After some investigation I realised that when VCB performed a dump of the VMDK it was correctly dumping the used space. It then dawned on me that when Windows was deleting files from the disk it was actually only deleting the 'pointer' to the data and not the data itself. When VCB was dumping the VMDK it was correctly dumping all blocks of the disk that contained data. I then started looking for a way to clean blocks that were no longer being used but still contained remnants from deleted files. I found a tool called SDelete that writes zero's where the disk contains free space as seen by the Operating System.

I incorporated SDelete into a basic script that also uses Defrag (the Windows utility for defragmenting disks) to tidy up volumes and potentially reduce the size of backups when using VCB.

The script first enumerates all local drives on the VM. For each drive that Windows deems as being a local disk the script then defragments the volume and uses SDelete to zero any free space that the disk may have. The script is written in KiXtart.

Here is the script:

;=====================================================
;=====================================================
;
; ENUMERATE LOCAL DRIVES - DEFRAG & ZERO EMPTY BLOCKS
;
;=====================================================
;=====================================================

;=====================Accept EULA=====================
WriteValue("HKCU\Software\Sysinternals\SDelete", "EulaAccepted", "1", "REG_DWORD")

$Drives = GetObject("winmgmts:").ExecQuery("select Name,DriveType from Win32_LogicalDisk")

For Each $Drive in $Drives

If $Drive.DriveType = 3

SHELL "%comspec% /c " + chr(34) + "defrag.exe " + $Drive.name + " -f" + chr(34)
SHELL "%comspec% /c " + chr(34) + "sdelete.exe -c " + $Drive.name + chr(34)

EndIf

Next

;=====================================================

I use this by placing the folder CleanDisk (download link below) in C:\Program Files. I create a Windows scheduled task to run either KIX32.exe or WKIX32.exe on say a monthly basis.

NOTE: If you have multiple VMs with VMDKs residing on the same physical storage you will probably want to consider staggering the schedules to reduce I/O on the physical disks.

Download the script and necessary files - HERE

When the host had finished booting it naturally tried to reconfigure HA as necessary, but within two minutes it was appearing as being disconnected from vCenter. Initially I was a little stumped because I instinctively tried to SSH onto the host and was able to do so. This immediately ruled out any form of network or connectivity problem. For the sake of speed I decided to restart the host just incase a service had failed when it was booting up. After the host restarted it appeared in vCenter as usual. I then noticed an error on the host and checked the Alarms tab. The alarm triggered was:

Host connection and power state

The alarm was weird since I had only upgraded the RAM on the host. I couldn't correlate the significance of the alarm and the host disconnecting from the vCenter server. I assumed I had disturbed one of the power cables on the host but a quick check revealed both PSUs in the host had a live power feed and this alarm doesn't actually monitor the power anyway.

I decided to use the vSphere client to connect directly to the host. As soon as it connected I received an information message explaining the host was being managed by the vCenter server, but the IP address listed (for the vCenter server) was 127.0.0.1. This was clearly incorrect.

When a host is added to vCenter it uses the IP address stored within vCenter located in:

Administration --> vCenter Server Settings --> Runtime Settings --> vCenter Server Managed IP

A quick look at this setting on the vCenter server revealed that the Managed IP was blank.

I assumed that as the Managed IP was blank hosts were trying to connect to themselves instead of the vCenter server.

I used nano to open /etc/opt/vmware/vpxa/vpxa.conf on the host and it contained the following:

<vpxa>
...
<serverIp>127.0.0.1</serverIp>
...
</vpxa>

After changing the vCenter Server Managed IP address using the vSphere client I disconnected from both the vCenter server and the host. I then connected back onto the host using the vSphere client and it was now being managed by the vCenter server with the correct IP address. A quick check on the vCenter server itself revealed that the Host connection and power state alarm had ceased. I also verified that the vpxa.conf on the host was reporting the correct IP address.

I assume that this issue may have been caused during the upgrade from ESX 3.5 to vSphere 4. I had never experienced this problem before and it has not reoccured since.

Here is the error:

Host CPU is incompatible with the Virtual Machine's requirements at CPUID level 0x1 register 'ecx'.
Host bits: 0000:0000:0000:0000:0010:0010:0000:0001
Required: 1000:0000:0000:000x:xxx0:0x1x:xxx0:x001
Mismatch detected for these features:
*General incompatibilities; refer to KB article 1993 for possible solutions.

The only recent change was the upgrade to vSphere 4. The error detailed an incompatibility issue between the host and Virtual Machine CPU. The host hardware hadn't changed so I decided to investigate the Virtual Machine. After a quick look at the affected Virtual Machine's .vmx configuration file I noticed what appeared to be a CPU ID Mask. I checked the .vmx configuration against a couple of other Virtual Machines that were still able to VMotion and sure enough the CPU ID Mask was not present in these Virtual Machines.

The 5 lines in the non VMotion'ing Virtual Machine's .vmx file were:

cpuid.1.ecx = "R----R----R--R-0-----------H-R--"

cpuid.1.ecx.amd = "R---------------------------R---"
cpuid.80000001.ecx.amd = "------------------RR-RR---------"
cpuid.80000001.edx = "----R---------------------------"
cpuid.80000001.edx.amd = "--------------------------------"

I shut the Virtual Machine in question down and removed the 5 lines above from the .vmx file. I powered on the Virtual Machine and tried to VMotion it. I still received the same error as above. I shut down the Virtual Machine again, verified that the CPU ID Mask was not present in the .vmx and then powered it back on. After the second full shutdown I could successfully VMotion the Virtual Machine again.

NOTE: I have noticed that if you edit a .vmx file before shutting a Virtual Machine down it appears that the changes are overwritten because the running configuration is saved over the top of the existing configuration in the .vmx file itself. The easiest way to make a change to a .vmx is to shut the Virtual Machine down before editing it.

The CPU ID Mask can also be modified from the vSphere client by:

1) Shutdown the Virtual Machine
2) Right-click the Virtual Machine and select "Edit Settings..."
3) Select the "Options" tab
4) Click on "CPUID Mask"
5) Click on the "Advanced..." button

This issue affected 3 of 8 Virtual Machines in my 3 node vSphere cluster and I'm unsure why it only affected certain Virtual Machines. There seems to be nothing in common (that I could think of anyway) between the 3 affected Virtual Machines that I believe could have attributed to this problem.

EDIT (29/10/09): It seems VMware are now aware of the problem and have issued a workaround that requires you to use the vSphere client to remove the CPU ID mask. KB1011294

One of the new features of vSphere is the ability to resize disks without having to shut down the Virtual Machine. This was previously impossible in VI3. This greatly speeds up the resizing process which can be executed in a couple of stages:

1) Use the vSphere Client to edit the settings of the Virtual Machine in question. Select the hard disk and modify it's provisioned size as appropriate. Click OK to apply these changes - resizing the .vmdk file.

2) Verify that the .vmdk has been resized by opening the Management Console -> Disk Management to find the unallocated space on the disk that resides in the .vmdk (distinguished by the black colour in the legend at the bottom.) In this case you can see I have increased the size by 5GB.

Right click on the disk (in this case 'Disk 0') and select properties. On the Volumes tab make a note of the unallocated space, in my case it is 5122MB.

Download Dell's EXTPART and extract it on the server that contains the disk you want to resize. Navigate to c:\dell\ExtPart (the default extracted location) and run extpart.exe. When prompted enter the the Windows drive letter of the disk on the Virtual Machine e.g. c:. When prompted for the size to extend the partition by enter the number noted down earlier (I used 5122 in this example.) After doing so the disk should be resized. You can check this by opening the Management Console -> Disk Management and verifying the size of the partition.

NB - If you receive the following error:

"Unable to connect to c: or it does not exist"

There are a couple of workarounds that you could try.

1) Close the Management Console (if it is open) and try extpart.exe again.

2) Try restarting the VM in safe mode and then run extpart.exe. This is not ideal but it is still easier than other methods I have tried to resize .vmdk files.

I powered on the Virtual Machine after removing the old NIC and adding the new vmxnet3 NIC and logged into Windows. I could see Windows installing the relevant drivers for the new vmxnet3 adaptor that it had detected and Windows was prompting for a reboot. Before doing so I tried to set the IP address on the new NIC so that when the Virtual Machine rebooted it was immediately available on the network. When I tried to do so I was prompted with an error:

"The IP address you have entered for this network adapter is already assigned to another adapter"

Since I had 'physically' removed the original adaptor I checked device manager to try and uninstall it but it was not listed. When a device is removed from a physical server but was not uninstalled it becomes hidden in device manager. To view hidden devices you need to open a command prompt:

Start -> run -> cmd

Now type the following:

SET DEVMGR_SHOW_NONPRESENT_DEVICES=1
START DEVMGMT.MSC

This will start device manager with the option to 'Show hidden devices'. To enable this option go to:

View -> Show hidden devices

Now when you browse Network Adaptors you will see the hidden NIC (it will be ghosted out) and can right click and uninstall it.

After uninstalling the old NIC I was able to set the IP address of the server on the new NIC with no issues. After a reboot the server was operational again.

After a quick look in the application log on the server I could see that there was an error reported by the SQL instance (I use the SQL Server Express edition default installation for VCenter installations) just before the vpxd service had also logged an error. The first error appeared to be related to the vpxd service starting and establishing a connection to the SQL database. During the connection the authentication had failed and so the vpxd service had failed to start as per the second error.

The immediate reason for this happening seemed to be because during boot up the vpxd service was attempting to start before the SQL service, except the vpxd service was almost relying on the SQL service having started before so that it could establish a connection to the SQL database. Without the SQL service starting prior to the vpxd service there was no way that the connection could be established.

I found that the workaround to this issue is to make the vpxd service dependent on the SQL Server service so that it waits for the SQL Server service to start before it attempts to start itself and a connection to the SQL database is established.

The easiest way to do this is:

1) Establish the service name of the SQL instance for VCenter. If you have installed VCenter using SQL Server Express edition then by default the name is 'MSSQL$SQLEXP_VIM'. If you have used a different version of SQL Server you will need to open the computer management console services snapin (start -> run -> services.msc) and find the service called SQL Server. Right click on the service and select properties to view the service name and make a note of it.

2) Note: You can download the following regkey changes for SQL Server Express - HERE

Open regedit (start -> run -> regedit) and navigate to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vpxd

Open the Multi String value for DependOnService by double clicking on the value. On a new line add the service name of your SQL instance. I added the following on a new line because I use SQL Server Express edition:

3) Close the registry editor, open services.msc again and check that the VMware VirtualCenter Server service is now dependent on the relevant the relevant SQL Server service.

Since making the vpxd service dependent on the SQL Server service I have not had a problem rebooting a server running VCenter where the VMware VirtualCenter Server service fails to start.

Post 12th August I could not VMotion and was concerned (as people on the VMware forums were reporting they could not power on Virtual Machines that were off) that in the event of an ESX host failing High Availability (HA) would try to unsuccessfully boot the Virtual Machines on a different host.

I always try to distribute my Virtual Machines equally throughout a HA cluster to ensure minimum disruption to organisation services in the event of a ESX host failure, and to compensate for Virtual Machines that use more of the ESX host(s) resources. It soon became apparent though that it would have been a nightmare to shut down all Virtual Machines on any one ESX host in order to patch it with the update VMware had released (since you cannot patch until all Virtual Machines are VMotioned off of the host or shut down). On the VMware forums it was mentioned that if the date on an ESX host was changed to one prior to the bug taking effect you could successfully VMotion Virtual Machines and power them on.

I inevitably had to test this as it would cause the least impact to the organisation I work for. One concern I initially had was that Virtual Machines have an option of synchronizing their date with the ESX host on which they reside. If a domain controller Virtual Machine were to synchronize its date with an ESX host the time skew between the domain controller and its clients on the network would cause obvious problems. I verified that the default setting for time synchronization in VMware tools on each Virtual Machine is disabled by default:

I changed the date back to 10th August 2008 and disabled NTP on a couple of my ESX hosts using the VIClient through Virtual Center:

I was then able to VMotion the Virtual Machines from one ESX host onto the other. I corrected the date(s) and now had one ESX host (with no running Virtual Machines) I could patch using VMware Update Manager. I patched the ESX host and after it rebooted I again changed the date on another upatched ESX host to 10th August 2008 and moved all of its Virtual Machines onto the newly patched ESX host. A combination of changing dates on unpatched ESX hosts and then VMotioning Virtual Machines from them onto patched ESX hosts allowed me to patch all ESX hosts in the cluster. One thing I did notice, after changing the date to 10th August 2008 on an unpatched ESX host I would receive the error message "HA agent on (servername) in cluster (clustername) has an error" after a few minutes. When this error appears you obviously cannot initiate a VMotion but a VMotion that is already in progress remains unaffected.

It seems after some testing that when VMotioning a Virtual Machine from one ESX host to another the Virtual Machine can sometimes synchronise its time with the host on which it is migrating onto. My advice would be to patch ESX hosts without domain controller Virtual Machines running on them and when you do VMotion your domain controller Virtual Machines off of an unpatched ESX host onto an already patched ESX host make sure the ESX host that you are migrating the Virtual Machines onto has the correct date!

As a final test, a colleague took a single ESX host running a single Virtual Machine both with correct dates. He shutdown the Virtual Machine and then changed the date on the host (as though he was going to patch it). After changing the date on the host and rebooting it he then powered on the single Virtual Machine. Even though it had time synchronization disabled in the VMware tools the Virtual Machine automatically changed its date to match the host on which it resided during BIOS post. So if you do have any Virtual Machines on an ESX host that are powered off make sure you change the ESX host date back to the correct date before powering any Virtual Machines on!

All of my ESX hosts are now: 3.5.0 Build 110268.