Issue 1. Module exppw.dll could not be loaded

The first thing that struck me when I went to test OWA was that it wasn't working. I host a few websites on the same server so I was quite shocked to see that none of them were actually functioning. For what it was worth - I had already restarted the server so an IISRESET probably wouldn't have done much good. I decided to check the Application Event Log and saw this:

Source: IIS-W3SVC-WP
Event ID: 2282
Level: Error

Error: The Module DLL 'C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll' could not be loaded due to a configuration problem. The current configuration only supports loading images built for a x86 processor architecture. The data field contains the error number. To learn more about this issue, including how to troubleshooting this kind of processor architecture mismatch error, see http://go.microsoft.com/fwlink/?LinkId=29349.

I also noticed the following warning in the System Event Log:

Source: WAS
Event ID: 5139
Level: Warning

Error: A listener channel for protocol 'http' in worker process '5436' serving application pool 'jjclements.co.uk' reported a listener channel failure. The data field contains the error number.

IIS was running, I could successfully Telnet into port 80 on the server. It appeared from the first (Application Event Log) error that IIS was trying to load a new module as part of the SP1 upgrade but couldn't. This was causing IIS to stop loading all of the sites that I had configured within it. I knew that since the release of SP1, Microsoft had released Update Rollup 1 to fix a few of the issues that SP1 brought with it. Out of best practice (and the hope that it may help my problem) I installed Update Rollup 1. Unfortunately it didn't resolve the issue.

Since the error was reporting an issue with IIS loading an x64 compiled module on x86 architecture I decided to investigate which website(s) was attempting to load the module. My reasoning for this was that by default, OWA runs in it's own application pool from a virtual directory located within the Default Website on an IIS installation. Although I had configured other application pools (for the other websites I am hosting) to run with "Enable 32-Bit Applications" set to True, the default setting of False still applied for the MSExchangeOWAAppPool application pool (so it seemed logical to assume that more than one website was probably trying to load the same module):

I used the IIS Manager to locate and then checked the web.config for the OWA virtual directory located:

C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\web.config

Sure enough there was an entry in the web.config for the site to load the exppw module, and because the MSExchangeOWAAppPool was correctly configured with "Enable 32-Bit Applications" set to False this was not the cause of my problem. Since it seemed that IIS was attempting to load the module for other application pools configured to run as 32bit (and I knowingly had other websites configured to run as 32bit) the next logical step was to check the main 'root' configuration file for IIS - applicationHost.config located:

C:\Windows\System32\inetsrv\config\applicationHost.config

The applicationHost.config file is used to store IIS configuration including sites, virtual directories, general settings, logging, caching etc. A quick search through this file revealed the exppw module was also being loaded globally (from within the <globalModules> tag):

<add name="exppw" image="C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll" />

I decided that I had a couple of options to fix this issue. The first was to stop IIS from loading the module globally by removing the entry from the applicationHost.config. I figured that this probably would have worked since the web.config for the MSExchangeOWAAppPool was loading it anyway. The second option I had was to make sure that the module was only ever loaded for an application pool when it's worker process architecture matched the architecture of the module that was being loaded. I chose the second option as I was concerned that modifying the applicationHost.config to prevent IIS from globally loading the exppw module may have caused issues if any of the other Exchange 2010 application pools were depending on the exppw module being loaded globally instead of from a local web.config.

IIS7 introduced the bitness32 and bitness64 preconditions to make sure that it will only load DLLs with the correct bitness in an application pool worker processes. Where specified, the bitness64 precondition is used by IIS to identify and then load modules only when the worker process is an x64 process - not if the worker process happens to be an x86 process.

I modified the applicationHost.config to reflect the bitness64 precondition for the exppw module:

<add name="exppw" image="C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll" preCondition="bitness64" />

Without even restarting IIS all of the websites that I host (including the Default Website and OWA application) started functioning again. The exppw module was now only being loaded where an application pool worker process was an x64 process.

Issue 2. Default Website additional bindings

While this didn't actually stop anything from working, the second problem that I happened to stumble across (when investigating Issue 1) was that the default website now had some additional, unconventional bindings:

It seemed that as well as the normal bindings allowing IIS to listen on ports 80 (http) and 443 (https) for all IP addresses on the server, the SP1 upgrade had added 2 additional bindings. Surprisingly, even though no host headers had been set, the additional bindings were also listening on ports 80 and 443 for the loopback address of 127.0.0.1. For consistency and just incase this caused any issues in the future, I removed the bindings. Seeing as no specific host headers were set for the new additional bindings I concluded that they would have been encompassed by the original 2 bindings anyway. A quick check revealed that none of my other websites had the additional bindings.

Issue 3. Incorrect HomeMTA attribute

Again, while this issue didn't actually appear to stop anything from working, the last problem I noticed was a consistent warning appearing in the Application Event Log. The warning was being generated at frequent intervals for each mailbox that I had configured within Exchange 2010:

Source: MSExchange ADAccess
Event ID: 2937
Level: Warning

Error: Process w3wp.exe () (PID=7276). Object [CN=James,CN=Users,DC=jjclements,DC=co,DC=uk]. Property [HomeMTA] is set to value [jjclements.co.uk/Configuration/Deleted Objects/Microsoft MTA
DEL:75830fde-6ed2-4fe2-b91c-005d7f6b1e6d], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.

A quick search revealed that the HomeMTA user account attribute was now obsolete for Exchange 2010. Yet, for some bizarre reason Exchange 2010 was obviously still checking that the value was valid (and unfortunately logging a warning in the Application Event Log if it wasn't.)

I used ADSI Edit to check the attribute for one my user accounts and confirmed it was set to the Deleted Objects container. Even though I don't have lots of Exchange users, rather than use ADSI Edit to update the HomeMTA attribute manually for each and every mailbox user I wanted to update them collectively. I had read that the Update-Recipient cmdlet could be used to update user account attributes, including HomeMTA. So, from the Exchange Management Shell, I used the Get-Mailbox cmdlet to retrieve all the user accounts with a mailbox in my single Exchange 2010 database and then piped the results into the Update-Recipient cmdlet. The following command successfully updated the HomeMTA attribute for all user accounts that had a mailbox in my Exchange 2010 database and stopped the warning from appearing in the Application Event Log:

get-mailbox -database "<databasename>" | update-recipient

Before hand, I decided to try a deployment on a test server just to see if I had any issues. The server is a single box that is also a Domain Controller. For testing purposes I also installed Exchange 2010 onto the same server. Initial installation and configuration was a breeze and I was soon sending and receiving email internally and externally. Upon investigation of some of the more granular Exchange configuration I soon had an issue when I tried to enable Recipient Filtering, there was simply no option to do so.

Recipient Filtering can be used for a couple of reasons. The first is to filter/prevent emails from being sent to specific email addresses that do exist within your organisation. The second is to prevent your email server from processing emails sent into your organisation to email addresses that do not exist. This typically happens in a couple of different situations when:

Scenario 1. A spammer targets 1000's of email addresses

• Spammer sends 1000's of emails with different FROM addresses (victim's addresses)
• Spammer sets the TO address as xxxx@yourdomain.com (non existent address in your organisation)
• Your email server receives and processes the emails but cannot deliver them to a mailbox
• Your email server replies to the 1000's of FROM addresses sending each 1 piece of NDR spam

Scenario 2. A spammer targets 1 email address 1000's of times

• Spammer sends 1000's of emails with a common FROM addresses (victim's address)
• Spammer sets the TO address as xxxx@yourdomain.com (non existent address in your organisation)
• Your email server receives and processes the emails but cannot deliver them to a mailbox
• Your email server replies to the FROM address sending it 1000's of pieces of NDR spam

Scenario 3. A spammer targets your email server

• Spammer sends 1000's of emails with different TO addresses (which won't exist in some organisations)
• Spammer sets the FROM address as xxxx@yourdomain.com (non existent address in your organisation)
• Recipient email servers send an NDR to the FROM address (non existent address in your organisation)
• Your server receives 1000's of pieces of NDR spam it then attempts to process and deliver

By enabling Recipient Filtering you make sure that if your email server does receive an email for an address that does not exist within your organisation it is blocked during initial SMTP communication. Recipient Filtering is by far faster and more efficient than evaluating the mail after it is received and already in the mail queue. This means that:

• Your server can no longer be used to send NDR spam to other email servers (Scenario 1 & 2)
• Your server will experience a reduced load if it comes under an NDR spam attack (Scenario 3)

The problem I had with Exchange 2010 was that I could not find where to configure Recipient Filtering! In Exchange 2003 it was configured under Global Settings --> Message Delivery. You then had to configure Advanced options on the SMTP Virtual Server and ensure the option for Apply Recipient Filter was applied. My Exchange 2010 installation did not have any such option(s) out of the box. It turns out the reason for this is that features of this nature are usually best suited to Exchange 2010 servers with the Edge Transport role installed. I attempted to modify the Exchange 2010 installation through Control Panel --> Programs and Features but the Edge Transport role was greyed out. An Exchange 2010 server with the Edge Transport role configured cannot have any other Exchange 2010 components installed, and since everything else was installed on my test server there was no way to enable the Edge Transport role.

After some searching around I found out that Recipient Filtering and a few other options like Sender ID are labelled as being Anti-spam features within Exchange 2010. I discovered that it is possible to install the Anti-spam feature (through the use of a PowerShell script) on an Exchange 2010 server configured with the Hub Transport role. This can be accomplished by:

• Open the Exchange Management Shell on the Exchange 2010 server
• Change directory into:
  

  C:\Program Files\Microsoft\Exchange Server\V14\Scripts

• Run the following to install the Anti-spam option:
  

  .\install-AntispamAgents.ps1

• Run the following to restart the Microsoft Exchange Transport service:
  

  restart-service msexchangetransport

• Open the Exchange Management Console and check that the Anti-Spam tab is available under Organization Configuration --> Hub Transport

Now that the Anti-spam features are available and enabled for use, I needed to make sure that the Recipient Filtering was configured. By opening the properties on the Recipient Filtering feature I could see that the option to 'Block messages set to recipients that do not exist in the directory' was not enabled. I checked this option to enable it as required.

A quick test using Telnet and the server was no longer receiving emails for addresses that do not exist in my organisation. Of course, doing this has a drawback. There would be nothing to stop a malicious user from using an automated tool harvest legitimate addresses within my organisation by sending email to random email addresses until successful. Fortunately, Exchange 2010 has Tar Pitting functionality enabled out of the box. Tar Pitting is the practice of artificially delaying server responses for specific SMTP communication patterns that indicate high volumes of spam or other unwelcome messages. By default Exchange 2010 delays the response sent to an email server (where an attempt is made to send to an address that does not exist in your organisation) by 5 seconds. Tar Pitting therefore makes directory harvest attacks too costly to automate efficiently. I personally think a 5 second delay is sufficient to thwart most directory harvest attacks, although the delay can be adjusted as necessary through the Exchange Management Shell on the Exchange 2010 server.

Run the following to retrieve the current Tar Pit interval in seconds:
Get-ReceiveConnector "Your Conector Name" | select tarpitinterval

Run the following to change the current Tar Pit interval (to 10 seconds in this case):
Set-ReceiveConnector "Your Conector Name" –TarpitInterval 00:00:10

Run the following to disable Tar Pitting:
Set-ReceiveConnector "Your Conector Name" –TarpitInterval 00:00:00

Not being able to logon I was unable to diagnose the problem locally so I used the computer management console on another client to view the event log on the Exchange server. A quick look revealed a frequent error in the application log:

Event ID: 9157
Description: Microsoft Exchange System Attendant does not have sufficient rights to read Exchange configuration objects in Active Directory

The problem occurred because there are 3 security groups in Active Directory that Exchange queries in the Users Organisational Unit (OU) during startup:

Exchange Enterprise Servers
Exchange Services
Exchange Domain Servers

As it turns out, during a recent restructure of Active Directory someone had moved the above groups into a different OU and this was the cause of Exchange hanging and recording the above entry to the event log. Moving the 3 security groups back into the Users OU almost immediately allowed Exchange to display the winlogon screen without requiring a restart.

More information is available here:

KB910413

There is a known problem with Vista clients that causes IE7 to crash when using OWA. I have found this is usually when users have gone through the effort of composing an email and hit the 'Send' button. IE7 then crashes with the standard issue 'Program not responding' response at any attempts to recover from it.

I have only ever noticed this problem on Exchange 2003 SP2 and cannot confirm if it happens pre SP2 but it does only seems to affect Vista clients running IE7 and not XP running IE7.

The cure to this problem is a Hotfix released from Microsoft. Although the Hotfix has been out for a while, unless you have tried it by chance there is likelihood that you wouldn't as the description is somewhat vague. The knowledge base article can be found here:

KB911829

And you can download the Hotfix from here:

Download Hotfix for KB911829

Install the Hotfix on your Exchange OWA facing server. You will have to reboot your Exchange server and key services like 'Microsoft Exchange Information Store' are stopped during the installation. This will obviously cause all outlook clients to disconnect from the Exchange server. You may also then need to instruct Vista users to log into OWA and navigate through:

Options -> E-mail Security -> Install S/MIME

to reinstall S/MIME.