Loadbalancing, ActiveSync and Affinity


Recently, a client was experiencing load issues on the Exchange 2010 Client Access Servers. The client also had installed a hardware load balancer to balance client traffic.

While investigating the PAL results, the ActiveSync connections chart showed a significantly unbalanced number of ActiveSync connections between the CAS servers.

It turned out the client had load balanced all client traffic using Source IP affinity for all protocols. This means each client gets assigned the same CAS server, based on the client’s IP address. While this may sound reasonable, for ActiveSync this may not be optimal. Reason is that most mobile telephony providers use some form of NAT translation for their clients, resulting in these devices to appear having the same IP address.

When organizations standardize on a NAT utilizing mobile telephony provider, the problem might emerge sooner as all of their mobile clients will be assigned to the same Client Access Server.

In the picture above you’ll see the top two mobile devices are being NAT’ed. When the top device connects to the Exchange environment, it gets assigned the 1st CAS server based on its IP address. When the 2nd mobile device connects, the load balancer sees the same IP address after which it will direct that traffic to 1st CAS server as well.

While affinity is not required for ActiveSync, it is recommended since for each newly appointed CAS server, the notification subscription to the mailbox to be informed of updates would have to be recreated. Of course, this would result in a performance penalty and increased latency. Another option would be Session ID, but some EAS clients unnecessarily create a new SSL session ID.

After switching affinity from Client IP to Authorization HTTP Header the ActiveSync clients spread out more evenly. When using Authorization HTTP Header affinity, the load balancer uses the base64 encoded credentials as part of the http client request, e.g.

POST http://mail.eightwone.com/Microsoft-Server-ActiveSync/default.eas?Cmd=Sync&..
..
Authorization: Basic YW55IGNhcm5hbCBwbGVhc3VyZS4=

After switching affinity for ECP as well (should be Cookie or Session ID), the load issues were gone.

Where in the past mobile clients were insignificant to Outlook clients when compared in numbers, the ongoing consumerization of IT movement results in an increasing mobile client population. The number of ActiveSync users may easily outweigh the number of Outlook clients, as many users use a phone or tablet (or both) in addition to Outlook, if they use Outlook at all.

Exchange ActiveSync and Inheritable Permissions issue


The issue and solution described here is by design, but not known by every customer so here’s my short write-up on this subject.

Recently, I was at a customer reporting issues with several users not being able to synchronize their mobile devices using ActiveSync. The customer was running Exchange 2010 SP1 and used various mobile devices, e.g. iPhones as well as Android phones and tablets. A quick look in the IIS logs revealed that devices were connecting properly, but they received HTTP return code 403 (forbidden):

2011-08-30 10:09:31 172.16.10.12 OPTIONS /Microsoft-Server-ActiveSync/default.eas User=XXXXX&DeviceId=d849cec9be024c828b9af73da93bb59b&DeviceType=htcbravo&Log=LdapC2_Error:UserPrincipalCouldNotBeFound_Dc:dc.domain.com_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f0%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5Fe205201e-d418-409a-a15b-4b51baef9bf4%2cNorm%5bResources%3a(DC)dc.domain.com(Health%3a-1%25%2cHistLoad%3a0)%2c%5d_ 443 domain\XXXXX 62.140.137.149 Android-EAS/0.1 403 0 0 124

Another clue was provided by the eventlog, which revealed MSExchange ActiveSync was reporting error 1053:

ss

The remainder of the message reads: “Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type “msExchangeActiveSyncDevices” and doesn’t have any deny permissions blocking such operations”. What happens when setting up ActiveSync is that Exchange tries to create a container named ExchangeActiveSyncDevices below the user object in Active Directory and will store in that container an MsExchActiveSync object for each ActiveSync device. Apparently Exchange doesn’t have sufficient permissions to create these objects.

To fix this, open up Active Directory Users and Computers. Now, to be able to inspect the security settings, we first need to activate Advanced Features if not already set. To do this, from the View menu option, select Advanced Features.

Next, navigate to the user object experiencing the issue. Open up Properties, select the Security tab and click Advanced.

image

Notice the Include inheritable permissions from this object’s parent is not set, the reason for Exchange not having any permissions on the object.

To fix the issue, simply check Include inheritable permissions from this object’s parent and click OK. You’ll return to the previous window where you’ll notice the Exchange Server account is now granted permissions on the object:

image

At this point, ActiveSync will work and Exchange will be able to create MsExchActiveSync objects in the ExchangeActiveSyncDevices container:

image

Note that Include inheritable permissions from this object’s parent by default is not enabled for members of the protected groups, e.g. Domain Admins. In fact, every hour the DACL on members of protected groups will be reset and inheritable permissions will be removed. This process is called AdminSDHolder which is to prevent inappropriate changes from being made to protected groups, accidently or otherwise.  Michael B. Smith did a nice write-up on this subject here. This is also the reason why bypassing the AdminSDHolder limitation by manually granting Exchange permissions would be inappropriate.

To prevent this issue, it is recommend to follow an old, yet far from rusty administrator best practice, which is to use one account for day-to-day operations, e.g. work and e-mail, and another account for administrative purposes.

Exchange & Windows Phone 7


This TechNet article on Windows Phone 7 got my attention. It appears you cannot fully utilize Exchange ActiveSync mailbox policies, unless you set AllowNonProvisionableDevices to True. If you don’t do that, you can only use the following properties, otherwise synchronization issues might be experienced:

  • PasswordRequired
  • MinPasswordLength
  • IdleTimeoutFrequencyValue
  • DeviceWipeThreshold
  • AllowSimplePassword
  • PasswordExpiration
  • PasswordHistory
  • DisableRemovableStorage
  • DisableIrDA
  • DisableDesktopSync
  • BlockRemoteDesktop
  • BlockInternetSharing

Another option is to create a seperate policy for Windows Phone 7 users.

Another thing worth mentioning is that when using multiple Exchange accounts on your Windows Phone 7, policies will be merged into a most restrictive set (credit to Dave Stork who got the information at TEE10).

Exchange ActiveSync and Hotmail


As of Monday, it is possible to synchronise your Hotmail account, i.e. e-mail, calendar and contacts, with your mobile using Exchange ActiveSync (EAS).

To synchronise your mobile with Hotmail, use the following settings:

Server m.hotmail.com
Username E-mail address, e.g. jandvries@hotmail.com
Password *****
Domain Leave blank
SSL Enabled

When asked, choose to accept the SSL certificate.

Synchronisation currently works for Windows Mobile 6.x, Windows Phone 7, iPhone, iPod Touch, iPad and Nokia E/S/N-Series with Mail for Exchange.

Windows Phone 7 to support multiple Exchange accounts


I seem to have missed the news from a few days back that during a broadcast on the @ch9live development network , Joe Belfiore, Microsoft Corporate Vice President for Windows Mobile, confirmed multiple Exchange account support in Windows Phone 7 Series. Goodbye to unsupported hacks or POP/IMAP’ing those additional Exchange boxes. I assume includes all the additional benefits like Direct Push (hello battery life!).

In addition, Belfiore stated that on Windows Phone 7 various calendars will be displayed in a single view using coler coding to differentiate between the calendars (i.e. accounts).

Thanks to fellow Exchange guy Magnus Björk for spotting this one.