Exchange 2010 SP1 issues
With its recent release, I decided to apply Service Pack 1 to my Exchange 2010 installation. I had read about many of the new features it offered and I was more curious about the Outlook Web App (OWA) improvements than anything else. I set about extracting and running running the SP1 installer, installing any prerequisites that the installer prompted me for before the upgrade. The upgrade lasted around 45 minutes and I was surprised to see that I was not prompted to restart the server post installation (even though I did out of habit!) Shortly afterwards I ran a few checks to make sure everything was OK and I began to notice a few peculiar issues.
Issue 1. Module exppw.dll could not be loaded
The first thing that struck me when I went to test OWA was that it wasn't working. I host a few websites on the same server so I was quite shocked to see that none of them were actually functioning. For what it was worth - I had already restarted the server so an IISRESET probably wouldn't have done much good. I decided to check the Application Event Log and saw this:
Event ID: 2282
Error: The Module DLL 'C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll' could not be loaded due to a configuration problem. The current configuration only supports loading images built for a x86 processor architecture. The data field contains the error number. To learn more about this issue, including how to troubleshooting this kind of processor architecture mismatch error, see http://go.microsoft.com/fwlink/?LinkId=29349.
I also noticed the following warning in the System Event Log:
Event ID: 5139
Error: A listener channel for protocol 'http' in worker process '5436' serving application pool 'jjclements.co.uk' reported a listener channel failure. The data field contains the error number.
IIS was running, I could successfully Telnet into port 80 on the server. It appeared from the first (Application Event Log) error that IIS was trying to load a new module as part of the SP1 upgrade but couldn't. This was causing IIS to stop loading all of the sites that I had configured within it. I knew that since the release of SP1, Microsoft had released Update Rollup 1 to fix a few of the issues that SP1 brought with it. Out of best practice (and the hope that it may help my problem) I installed Update Rollup 1. Unfortunately it didn't resolve the issue.
Since the error was reporting an issue with IIS loading an x64 compiled module on x86 architecture I decided to investigate which website(s) was attempting to load the module. My reasoning for this was that by default, OWA runs in it's own application pool from a virtual directory located within the Default Website on an IIS installation. Although I had configured other application pools (for the other websites I am hosting) to run with "Enable 32-Bit Applications" set to True, the default setting of False still applied for the MSExchangeOWAAppPool application pool (so it seemed logical to assume that more than one website was probably trying to load the same module):
I used the IIS Manager to locate and then checked the web.config for the OWA virtual directory located:
C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\web.config
Sure enough there was an entry in the web.config for the site to load the exppw module, and because the MSExchangeOWAAppPool was correctly configured with "Enable 32-Bit Applications" set to False this was not the cause of my problem. Since it seemed that IIS was attempting to load the module for other application pools configured to run as 32bit (and I knowingly had other websites configured to run as 32bit) the next logical step was to check the main 'root' configuration file for IIS - applicationHost.config located:
The applicationHost.config file is used to store IIS configuration including sites, virtual directories, general settings, logging, caching etc. A quick search through this file revealed the exppw module was also being loaded globally (from within the <globalModules> tag):
<add name="exppw" image="C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll" />
I decided that I had a couple of options to fix this issue. The first was to stop IIS from loading the module globally by removing the entry from the applicationHost.config. I figured that this probably would have worked since the web.config for the MSExchangeOWAAppPool was loading it anyway. The second option I had was to make sure that the module was only ever loaded for an application pool when it's worker process architecture matched the architecture of the module that was being loaded. I chose the second option as I was concerned that modifying the applicationHost.config to prevent IIS from globally loading the exppw module may have caused issues if any of the other Exchange 2010 application pools were depending on the exppw module being loaded globally instead of from a local web.config.
IIS7 introduced the bitness32 and bitness64 preconditions to make sure that it will only load DLLs with the correct bitness in an application pool worker processes. Where specified, the bitness64 precondition is used by IIS to identify and then load modules only when the worker process is an x64 process - not if the worker process happens to be an x86 process.
I modified the applicationHost.config to reflect the bitness64 precondition for the exppw module:
<add name="exppw" image="C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll" preCondition="bitness64" />
Without even restarting IIS all of the websites that I host (including the Default Website and OWA application) started functioning again. The exppw module was now only being loaded where an application pool worker process was an x64 process.
Issue 2. Default Website additional bindings
While this didn't actually stop anything from working, the second problem that I happened to stumble across (when investigating Issue 1) was that the default website now had some additional, unconventional bindings:
It seemed that as well as the normal bindings allowing IIS to listen on ports 80 (http) and 443 (https) for all IP addresses on the server, the SP1 upgrade had added 2 additional bindings. Surprisingly, even though no host headers had been set, the additional bindings were also listening on ports 80 and 443 for the loopback address of 127.0.0.1. For consistency and just incase this caused any issues in the future, I removed the bindings. Seeing as no specific host headers were set for the new additional bindings I concluded that they would have been encompassed by the original 2 bindings anyway. A quick check revealed that none of my other websites had the additional bindings.
Issue 3. Incorrect HomeMTA attribute
Again, while this issue didn't actually appear to stop anything from working, the last problem I noticed was a consistent warning appearing in the Application Event Log. The warning was being generated at frequent intervals for each mailbox that I had configured within Exchange 2010:
Source: MSExchange ADAccess
Event ID: 2937
Error: Process w3wp.exe () (PID=7276). Object [CN=James,CN=Users,DC=jjclements,DC=co,DC=uk]. Property [HomeMTA] is set to value [jjclements.co.uk/Configuration/Deleted Objects/Microsoft MTA
DEL:75830fde-6ed2-4fe2-b91c-005d7f6b1e6d], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.
A quick search revealed that the HomeMTA user account attribute was now obsolete for Exchange 2010. Yet, for some bizarre reason Exchange 2010 was obviously still checking that the value was valid (and unfortunately logging a warning in the Application Event Log if it wasn't.)
I used ADSI Edit to check the attribute for one my user accounts and confirmed it was set to the Deleted Objects container. Even though I don't have lots of Exchange users, rather than use ADSI Edit to update the HomeMTA attribute manually for each and every mailbox user I wanted to update them collectively. I had read that the Update-Recipient cmdlet could be used to update user account attributes, including HomeMTA. So, from the Exchange Management Shell, I used the Get-Mailbox cmdlet to retrieve all the user accounts with a mailbox in my single Exchange 2010 database and then piped the results into the Update-Recipient cmdlet. The following command successfully updated the HomeMTA attribute for all user accounts that had a mailbox in my Exchange 2010 database and stopped the warning from appearing in the Application Event Log:
get-mailbox -database "<databasename>" | update-recipient