Here is the error:

Host CPU is incompatible with the Virtual Machine's requirements at CPUID level 0x1 register 'ecx'.
Host bits: 0000:0000:0000:0000:0010:0010:0000:0001
Required: 1000:0000:0000:000x:xxx0:0x1x:xxx0:x001
Mismatch detected for these features:
*General incompatibilities; refer to KB article 1993 for possible solutions.

The only recent change was the upgrade to vSphere 4. The error detailed an incompatibility issue between the host and Virtual Machine CPU. The host hardware hadn't changed so I decided to investigate the Virtual Machine. After a quick look at the affected Virtual Machine's .vmx configuration file I noticed what appeared to be a CPU ID Mask. I checked the .vmx configuration against a couple of other Virtual Machines that were still able to VMotion and sure enough the CPU ID Mask was not present in these Virtual Machines.

The 5 lines in the non VMotion'ing Virtual Machine's .vmx file were:

cpuid.1.ecx = "R----R----R--R-0-----------H-R--"

cpuid.1.ecx.amd = "R---------------------------R---"
cpuid.80000001.ecx.amd = "------------------RR-RR---------"
cpuid.80000001.edx = "----R---------------------------"
cpuid.80000001.edx.amd = "--------------------------------"

I shut the Virtual Machine in question down and removed the 5 lines above from the .vmx file. I powered on the Virtual Machine and tried to VMotion it. I still received the same error as above. I shut down the Virtual Machine again, verified that the CPU ID Mask was not present in the .vmx and then powered it back on. After the second full shutdown I could successfully VMotion the Virtual Machine again.

NOTE: I have noticed that if you edit a .vmx file before shutting a Virtual Machine down it appears that the changes are overwritten because the running configuration is saved over the top of the existing configuration in the .vmx file itself. The easiest way to make a change to a .vmx is to shut the Virtual Machine down before editing it.

The CPU ID Mask can also be modified from the vSphere client by:

1) Shutdown the Virtual Machine
2) Right-click the Virtual Machine and select "Edit Settings..."
3) Select the "Options" tab
4) Click on "CPUID Mask"
5) Click on the "Advanced..." button

This issue affected 3 of 8 Virtual Machines in my 3 node vSphere cluster and I'm unsure why it only affected certain Virtual Machines. There seems to be nothing in common (that I could think of anyway) between the 3 affected Virtual Machines that I believe could have attributed to this problem.

EDIT (29/10/09): It seems VMware are now aware of the problem and have issued a workaround that requires you to use the vSphere client to remove the CPU ID mask. KB1011294