Load balancing Exchange 2010 using a KEMP Virtual LoadMaster


In an earlier blog, I mentioned the requirement for an external load balancer when co-locating Exchange server roles, because Failover-Clustering and Network Load Balancing (NLB) are mutually exclusive. However, there are also situations when a load balancer is a better solution over Windows built-in NLB, mainly because there are some things NLB can’t do or doesn’t do well, like:

  • Service awareness: NLB distributes clients over member nodes, even over nodes of which required services, like IIS or RPC Client Access Service, are not responding;
  • Experience: Clients need to reconnect after adding or removing nodes;
  • Scalability : it’s not recommended to scale NLB beyond 8 nodes;
  • Affinity (also known as persistence or sticky sessions): NLB can only do Source IP affinity, i.e. distribute clients based on their IP address, while load balancers can utilize cookies or SSL session IDs.

Note: Why affinity is important and why Source IP can be bad sometimes, you can read in one of my earlier blogs on load balancing Exchange ActiveSync here.

To show you setting up a load balancer doesn’t have to be rocket science, I’ll demonstrate how to implement a load balancer for Exchange 2010 using a KEMP Virtual Loadmaster (VLM); setting up other load balancers should be similar, hardware appliances included, but keep in mind implementations by vendors vary, so check the product documentation as well. However, the basics are same, you only need to understand what you’re trying to achieve.

Note: The KEMP’s VLM used for this article runs on Hyper-V, but there are virtual load balancers for different hypervisors as well.

The setup we’re going to work with is roughly as follows:

Kemp-HA-Setup-v1

In the sample environment, I’ve installed two Exchange 2010 servers, L12EX1 and L12EX2; both hold the Client Access, Hub Transport and Mailbox server roles. The domain name used is litware.com, and we have no site nor subnet definitions, so everything is located in the default Active Directory site, Default-First-Site-Name. Clients will access Exchange services (HTTPS, MAPI) using a single FQDN, outlook.litware.com.

The Exchange servers are located in a dedicated subnet, so we’ll use a so called two-armed setup (2 NICs); one NIC will connect the VLM to the subnet where the Exchange servers are located; the other one will be used for client access. In order to have the VLM work transparently, we configure the VLM as default gateway on the CAS servers. The result is that the CAS servers will see the original client IP addresses instead of the VLM’s address, which is not only helpful in log files, but is also needed for throttling or when limiting SMTP connections to Receive Connectors based on IP addresses for example.

Note: This article doesn’t describe implementing SSL offloading; for more information on SSL offloading and how to configure it, check this Technet article. Also, this article doesn’t go into any built-in ability of load balancers to mirror or create standby copies, meant to prevent the load balancer from becoming a Single Point Of Failure (SPOF) or improve Availability level.

We’ll start off by downloading the KEMP Virtual Loadmaster here. After downloading, extract the contents and import the VM in Hyper-V. After firing it up, it will use DHCP or 192.168.0.1 if DHCP is unavailable. You can check the console to see what IP address is used:

image

Now, before we can configure the VLM, we need to perform the initial setup:

  • Use the console to log in using the administrator account or connect with a browser to the VLM’s IP address;
  • If you haven’t got an activation key, you can apply for a trial key;
  • Complete licensing of the VLM;
  • Configure VLM network interfaces;
  • Import Configure certificate

Note: Make sure you set the MAC addresses of your NICs to static. When going through the licensing process, the access code is based on MAC address. If you don’t, the license will be invalidated if you migrate to a different host.

Note: We’re going to load balance services over port 443 and the administrative web interface uses that port as well, so configure the GUI on a different IP address or port.

Next, we need to create a Client Access Server Array. Note that creating a CAS Array before creating or moving mailboxes is best practice, as it prevents having to reconfigure Outlook MAPI profiles when clients have already connected (unless you want to perform mailbox move tricks to force MAPI reconfiguration). Basically, the steps to perform are:

  • Create a DNS record with FQDN which is going to be used for clients to connect. In our example, the FQDN used is outlook.litware.com using IP number 172.16.10.100;
  • Create a CAS Array object using New-ClientAccessArray, i.e.New-ClientAccessArray -Name outlook-default -Fqdn outlook.litware.com -Site Default-First-Site-Name

image

  • As per best practice, we’re fixing the RPC (59531) and Addressbook (59532) ports by setting the following registry keys on each CAS server and restarting the related MSExchangeRpc and MSExchangeAB services:

HKLM\System\CurrentControlSet\Services\MSExchangeRPC\ParametersSystem\TCP/IP Port = 0xe88b (59531)  REG_DWORD

HKLM\System\CurrentControlSet\Services\MSExchangeAB\Parameters\RpcTcpPort = ”59532” (REG_SZ)

You can verify Exchange is listening on these ports using netstat –an | find “5953”.

image

  • Finally, we need to configure the mailbox databases with the new RPC endpoint using Set-MailboxDatabase in conjunction with the RpcClientAccessServer parameter:Get-MailboxDatabase | Set-MailboxDatabase -RpcClientAccessServer outlook.litware.com

Note: More information on creating CAS Arrays, check here.

After creating the CAS array, fixing the ports on Exchange and reconfiguring the RPC endpoint configuration on mailbox databases, configure the Exchange URLs to match the new client endpoint FQDN, outlook.litware.com. To so so, use cmdlets like Set-OWAVirtualDirectory –InternalURL https://outlook.litware.com/owa or Set-WebServicesVirtualDirectory –InternalURL https://outlook.litware.com/EWS/Exchange.asmx. In addition to InternalURL, set the ExternalURL as well depending on your setup, i.e. HTTPS services may be load balanced at the reverse proxy.

Now we’re ready to configure the VLM. We start off by creating Virtual Services, which are a combination of IP address and ports. Each Virtual Service has it’s own characteristics, like persistence, scheduling (distribution), can have its own certificate, distribution mechanism and appointed set of real (backend) servers and related service monitors.

We decided to use a single IP address for the various Exchange services, so we only need to configure a single Virtual Service for each port, via Virtual Services > Add New:

image

In the next screen you need to configure the Virtual Service settings like persistence and scheduling, as well as configure the real servers, i.e. the backend servers actually providing the service. You can also configure how the service health on the real server is monitored, i.e. is the service up or down. If a service on a real server is considered down, the load balancer won’t send clients to that server for that particular Virtual Service.

Note: The overview below is taken from a non-SSL offloading (SSL acceleration) configuration; when enabled, it will show additional options on the certificate to use.

image

Note: When using “Least Connection” persistence as recommended in the KEMP documentation, be advised a client traffic storm can occur after the Real Server comes online. Reason is it starts without connections, so all new clients will be directed to this server. Other products have mechanisms in place to prevent this by throttling traffic, gradually increasing the connections; F5 calls this feature Slow Ramp Timeout in their F5 BIG-IP Local Traffic Manager products.

When configuring the Virtual Service, click Add New to add a Real Server to the Virtual Service.

image

A suggestion on how to configure the Virtual Services:

Virtual Address Port Service Name Persistence Scheduling
172.16.10.101 443 Exchange-HTTPS Super HTTP Round Robin
172.16.10.101 59531 Exchange-RPC Source IP Round Robin
172.16.10.101 59532 Exchange-AB Source IP Round Robin
172.16.10.101 135 Exchange-EPM Source IP Round Robin

Note: When required, you can also load balance inbound SMTP traffic using ports 25/587, IMAP4 (ports 143/993) and POP (110/995) using no persistence.

Note: Using Source IP can result in an unbalanced distribution of client load, when SNAT devices come into play. For an example scenario, see my earlier article on Load balancing, ActiveSync and Affinity.

And that’s basically it. When you want to channel specific HTTP services (Outlook WebApp, Exchange ActiveSync, Autodiscover etc.) you can appoint different FQDNs for each service and configure different FQDN/IP addresses per service in DNS, after which you can configure separate Virtual Services with more specific options. For example, you can not only configure specific persistence or scheduling settings for per Virtual Service, but also Real Servers checks (depending on the protocol). Instead of checking if a Real Server responds on port 443, you can check if the server responds on a different URL, e.g. https://<server>/owa.

image

Another bonus of using a load balancer, depending on functionality of the product used of course, is that you can (temporarily) disable a real server from the VLM. After doing this, clients won’t be directed to the corresponding Exchange server, which is very useful when you want to perform maintenance.

image

In this article we quickly went through setting up a KEMP VLM to load balance Exchange 2010 services. However, the article is based on certain decisions regarding the configuration, which can differ from organizational to organization. For more information on deploying KEMP VLM and its possibilities, check out the KEMP Virtual LoadMaster Deployment Guide here.

Most vendors, like KEMP, provide template functionality, which enables you to quickly set up the load balancer using preconfigured settings; make sure you inspect those settings afterwards (i.e. know what you’re doing). You can download KEMP templates here. Unfortunately, these files are in binary format so you can’t edit them nor can you export Virtual Services, otherwise I could have provided you with the template for the above settings.

Be advised that I am in no way connected to KEMP and this article hasn’t been sponsored  or commissioned by KEMP technologies, apart from providing an NFR license for writing and testing purposes.

2011, a short Retrospective


Happy new year to all my dear readers and followers. It’s been an interesting year, both from a personal (2nd kid) as well as a professional perspective (job change). It’s also a year with less blogging and community participation than originally planned. Therefor I’ve recycled my New Years’ resolutions of 2011 for 2012. I continue hoping that what you find here may help you in some way.

I’d also like to share with you some blog statistics of 2011, it’s 2nd year running:

Next to the Main, Versions, Builds and Dates and Toolkit pages, these were the Top 5 posts of 2011:

Top 5 posts of all time:

Top 5 referrers of 2011:

Again, thanks for visiting and keep coming back! Don’t forget, you can also follow me on Twitter.

The case of the missing Free/Busy public folder


A customer who recently migrated to Exchange 2010 and was still in the co-existence setup with Exchange 2003, reported lots of users experiencing issues with regards to Free/Busy information. Symptoms were inaccurate or missing Free/Busy information, which especially gets annoying when scheduling appointments.

It turned out these users were using Outlook 2003; users on Outlook 2007 or later experienced no issues. As you probably know, Outlook 2003 still utilizes public folders to publish users’ Free/Busy information. This information is consulted by Outlook 2003 when scheduling appointments; Outlook 2007 or later uses Exchange Web Services for this purpose.

A quick look in the Eventlog revealed lots of 14031 errors were generated by the FreeBusy Assistant:

Err14031

This told us Exchange was unable to store Free/Busy information in a public folder, in this case /o=EUROPE/ou=First Administrative Group. A quick look at the Public Folder Management Console in Exchange 2010 showed that the folder didn’t exist. Since the Free/Busy public folder to be used by an Outlook 2003 user is determined by the legacyExchangeDN attribute, this was the cause of the issue.

The reason for the folder’s absence was unknown so one can only speculate. My best guess is improper decommissioning of the organization and administrative group originally hosting those users, identified by that “orphaned” legacyExchangeDN.

With the Free/Busy public folder missing and the original Exchange infrastructure gone, there are two alternatives to resolving this issue, apart from upgrading clients to a recent version of Outlook of course:

  1. Edit the legacyExchangeDN attribute of the users affected;
  2. Recreate the Free/Busy public folder.

The 1st option has consequences for the users, like being able to reply to earlier e-mail by co-workers. This can be resolved by adding the current legacyExchangeDN as an X500 address to the current set of e-mail addresses, but that also makes things a bit messy.

The 2nd option is to recreate the Free/Busy public folder; Here’s how to proceed:

  1. First, using the Exchange System Manager (luckily, Exchange 2003 was still present), create an Administrative Group, e.g. First Administrative Group
  2. Then, using ADSIedit.msc, navigate to CN=<Administrative Group>,CN=Administrative Groups,CN=<Organization>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<domain>
  3. Right-click the Administrative Group, e.g. First Administrative Group, and click Properties. There, edit the legacyExhangeDN attribute. Set it to match the missing Free/Busy public folder, e.g. /o=EUROPE/ou=First Administrative Group
  4. Next, edit the siteFolderServer attribute. Set it to match the distinguishedName of a a public folder database. Note that you can pick the Exchange 2003 as well as Exchange 2010 Public Folder database here. In this example, I picked the Exhange 2003 public folder database, hence the storage group (SG1):

siteFolderServer

Now we need to wait for the store to recreate the Free/Busy public folder during its maintenance cycle, which may take up to 24h. If you’re in a hurry, and the situation allows you because of the service interruption, you can also restart the Information Store. When the Information Store has created the Free/Busy public folder, event 3047 is logged by the MSExchangeIS Public Store:

Recreate_FB_PF

To verify this, startup the Public Folder Management Console or any other Public Folder management tool, and you’ll see the newly created folder:

PFMC_Appear

After a while you’ll notice Outlook 2003 users are now storing their Free/Busy information in this public folder and Free/Busy will work again for these users. You can verify clients are storing Free/Busy information using EMS, ExFolders or MFCMAPI, e.g.:


Finally, don’t forget to create replicas of this new Free/Busy public folder when appropriate.

Exchange 2010 SP2 features, MCM:Exchange 2010 exam-only


The first day of TechEd NA 2011 brought us much exciting and some less exciting news on the Exchange frontier.

First, the announcement of changes in Exchange 2010 Service Pack 2. Besides some 500 bug fixes, SP2 contains the following new features:

  • Address Book Policies (also known as GAL segmentation). ABPs are meant to segmentize the address book, giving users a certain view of the address book like Address List Segregation did for Exchange 2003/2007. ABPs were already announced back in January. I wonder how this affects for instance MailTips, as MailTips might report on organization-wide figures (sending mail to X users) while the end user may only see a small fragment of the population. Also, be advised that clients bypassing the CAS server for directory lookups, e.g. LDAP queries, don’t benefit from ABPs. Think Outlook for Mac but also multifunctionals, fax solutions etc.;
  • OWA mini. This will be a lightweight browser like OMA in the past, meant for simple browsers;
  • Hybrid Configuration. This wizard is to make the configuration of an on-premises Exchange and Office 365/Exchange Online more simple, reducing the steps required from 49 to 6;
  • OWA Cross-Site redirection. This will allow clients to be silently redirected to the proper site if they log on to a CAS server located in a site different than the site where their mailbox is hosted and externalURL has been specified there. This greatly increases the single sign-on experience.

Be advised that Exchange Server 2010 Service Pack 2 will require schema changes to support the new features. SP2 is scheduled for the 2nd half of 2011.

Second, starting July 2011, Microsoft announced the exam-only Microsoft Certified Master: Exchange Server 2010 certification. This is for IT Professionals with 5 years of experience who think they can do the exams without the intensive 3 weeks training. Microsoft already did the same thing to the MCM: SQL Server 2008 program last year. The Exchange MCM exam is two-fold:

  1. MCM: Exchange Server 2010, Knowledge Exam. This exam will be offered by Prometric at select testing centers worldwide;
  2. MCM: Exchange Server 2010, Lab. This exam will be offered by Microsoft via direct remote proctoring at select Microsoft facilities worldwide.

I think while its great to have the option to take the exam in a facility in the region, but the absence of 3 week intensive training including meeting and being tutored by some people of the Exchange team and meeting Exchange fellows from all over the world seems a big miss. Also, how will the market respond to MCM’s who did the 3-week training versus MCM’s who didn’t; would the latter be considered inferior or less knowledgeable? If I had the choice, I’d go through the additional 3 weeks of training, extending your network and having a chance to ask your questions at the source.

Thanks to people like Dave Stork and Jeff Guillet for live reports through Twitter (#msteched).

You can watch a recording of Greg Taylor’s session on SP2 features here. The official related Exchange team blog is here. More information on the new Microsoft Certified Master: Exchange Server 2010 program here; the original announcement is here.

Besides all this, a recording of Scott Schnoll’s session on Exchange 2010 Tips & Tricks can be viewed here.

(Updated on May 17th with session links)