Review: Exchange Data Center Switchover Tool (Updated)

Last week, the Exchange team released what they called the “Exchange 2010 datacenter switchover tool” (note that the title mentions troubleshooter). The tool could prove helpful to some and can be insightful to others.

While I applaud any effort put in to minimize risks and the possibility of human error, especially in stressful situations like data center switchovers, I do have some suggestions for improvement.

First, the name. A “tool” might imply it’s something to aid in the switchover process, while in fact it’s more of an interactive decision maker or guide walking you through the process and can be utilized to practice dry runs or test formalized procedures.

That brings me to my second point, which is the format. A process like a data center switchover with all its decision moments is perhaps better translated to a flow chart rather than an interactive PowerPoint slide deck, which looks good on screen but can’t be printed. Also, a PDF or XPS might be more convenient; not everyone has PowerPoint at hand all the time, especially when working remotely on servers.

Finally, the contents is almost taken directly from the original Technet data center switchover article here, with the same questions and steps. It could perhaps be turned in a more valuable tool if it could read the environment and tailor questions based on what it discovers.

You can check out the “troubleshooter” yourself by downloading it here. Of course, this is only the first version; I suggest you leave feedback and suggestions on how to improve the tool in the accompanying article on the Exchange Team blog here.

Update October, 24th:UC Architects fellow Serkan Varoglu created a Exchange Data Center Switchover workflow diagram; you can download it here.

Load balancing Exchange 2010 using a KEMP Virtual LoadMaster

In an earlier blog, I mentioned the requirement for an external load balancer when co-locating Exchange server roles, because Failover-Clustering and Network Load Balancing (NLB) are mutually exclusive. However, there are also situations when a load balancer is a better solution over Windows built-in NLB, mainly because there are some things NLB can’t do or doesn’t do well, like:

  • Service awareness: NLB distributes clients over member nodes, even over nodes of which required services, like IIS or RPC Client Access Service, are not responding;
  • Experience: Clients need to reconnect after adding or removing nodes;
  • Scalability : it’s not recommended to scale NLB beyond 8 nodes;
  • Affinity (also known as persistence or sticky sessions): NLB can only do Source IP affinity, i.e. distribute clients based on their IP address, while load balancers can utilize cookies or SSL session IDs.

Note: Why affinity is important and why Source IP can be bad sometimes, you can read in one of my earlier blogs on load balancing Exchange ActiveSync here.

To show you setting up a load balancer doesn’t have to be rocket science, I’ll demonstrate how to implement a load balancer for Exchange 2010 using a KEMP Virtual Loadmaster (VLM); setting up other load balancers should be similar, hardware appliances included, but keep in mind implementations by vendors vary, so check the product documentation as well. However, the basics are same, you only need to understand what you’re trying to achieve.

Note: The KEMP’s VLM used for this article runs on Hyper-V, but there are virtual load balancers for different hypervisors as well.

The setup we’re going to work with is roughly as follows:

Kemp-HA-Setup-v1

In the sample environment, I’ve installed two Exchange 2010 servers, L12EX1 and L12EX2; both hold the Client Access, Hub Transport and Mailbox server roles. The domain name used is litware.com, and we have no site nor subnet definitions, so everything is located in the default Active Directory site, Default-First-Site-Name. Clients will access Exchange services (HTTPS, MAPI) using a single FQDN, outlook.litware.com.

The Exchange servers are located in a dedicated subnet, so we’ll use a so called two-armed setup (2 NICs); one NIC will connect the VLM to the subnet where the Exchange servers are located; the other one will be used for client access. In order to have the VLM work transparently, we configure the VLM as default gateway on the CAS servers. The result is that the CAS servers will see the original client IP addresses instead of the VLM’s address, which is not only helpful in log files, but is also needed for throttling or when limiting SMTP connections to Receive Connectors based on IP addresses for example.

Note: This article doesn’t describe implementing SSL offloading; for more information on SSL offloading and how to configure it, check this Technet article. Also, this article doesn’t go into any built-in ability of load balancers to mirror or create standby copies, meant to prevent the load balancer from becoming a Single Point Of Failure (SPOF) or improve Availability level.

We’ll start off by downloading the KEMP Virtual Loadmaster here. After downloading, extract the contents and import the VM in Hyper-V. After firing it up, it will use DHCP or 192.168.0.1 if DHCP is unavailable. You can check the console to see what IP address is used:

image

Now, before we can configure the VLM, we need to perform the initial setup:

  • Use the console to log in using the administrator account or connect with a browser to the VLM’s IP address;
  • If you haven’t got an activation key, you can apply for a trial key;
  • Complete licensing of the VLM;
  • Configure VLM network interfaces;
  • Import Configure certificate

Note: Make sure you set the MAC addresses of your NICs to static. When going through the licensing process, the access code is based on MAC address. If you don’t, the license will be invalidated if you migrate to a different host.

Note: We’re going to load balance services over port 443 and the administrative web interface uses that port as well, so configure the GUI on a different IP address or port.

Next, we need to create a Client Access Server Array. Note that creating a CAS Array before creating or moving mailboxes is best practice, as it prevents having to reconfigure Outlook MAPI profiles when clients have already connected (unless you want to perform mailbox move tricks to force MAPI reconfiguration). Basically, the steps to perform are:

  • Create a DNS record with FQDN which is going to be used for clients to connect. In our example, the FQDN used is outlook.litware.com using IP number 172.16.10.100;
  • Create a CAS Array object using New-ClientAccessArray, i.e.New-ClientAccessArray -Name outlook-default -Fqdn outlook.litware.com -Site Default-First-Site-Name

image

  • As per best practice, we’re fixing the RPC (59531) and Addressbook (59532) ports by setting the following registry keys on each CAS server and restarting the related MSExchangeRpc and MSExchangeAB services:

HKLM\System\CurrentControlSet\Services\MSExchangeRPC\ParametersSystem\TCP/IP Port = 0xe88b (59531)  REG_DWORD

HKLM\System\CurrentControlSet\Services\MSExchangeAB\Parameters\RpcTcpPort = ”59532” (REG_SZ)

You can verify Exchange is listening on these ports using netstat –an | find “5953”.

image

  • Finally, we need to configure the mailbox databases with the new RPC endpoint using Set-MailboxDatabase in conjunction with the RpcClientAccessServer parameter:Get-MailboxDatabase | Set-MailboxDatabase -RpcClientAccessServer outlook.litware.com

Note: More information on creating CAS Arrays, check here.

After creating the CAS array, fixing the ports on Exchange and reconfiguring the RPC endpoint configuration on mailbox databases, configure the Exchange URLs to match the new client endpoint FQDN, outlook.litware.com. To so so, use cmdlets like Set-OWAVirtualDirectory –InternalURL https://outlook.litware.com/owa or Set-WebServicesVirtualDirectory –InternalURL https://outlook.litware.com/EWS/Exchange.asmx. In addition to InternalURL, set the ExternalURL as well depending on your setup, i.e. HTTPS services may be load balanced at the reverse proxy.

Now we’re ready to configure the VLM. We start off by creating Virtual Services, which are a combination of IP address and ports. Each Virtual Service has it’s own characteristics, like persistence, scheduling (distribution), can have its own certificate, distribution mechanism and appointed set of real (backend) servers and related service monitors.

We decided to use a single IP address for the various Exchange services, so we only need to configure a single Virtual Service for each port, via Virtual Services > Add New:

image

In the next screen you need to configure the Virtual Service settings like persistence and scheduling, as well as configure the real servers, i.e. the backend servers actually providing the service. You can also configure how the service health on the real server is monitored, i.e. is the service up or down. If a service on a real server is considered down, the load balancer won’t send clients to that server for that particular Virtual Service.

Note: The overview below is taken from a non-SSL offloading (SSL acceleration) configuration; when enabled, it will show additional options on the certificate to use.

image

Note: When using “Least Connection” persistence as recommended in the KEMP documentation, be advised a client traffic storm can occur after the Real Server comes online. Reason is it starts without connections, so all new clients will be directed to this server. Other products have mechanisms in place to prevent this by throttling traffic, gradually increasing the connections; F5 calls this feature Slow Ramp Timeout in their F5 BIG-IP Local Traffic Manager products.

When configuring the Virtual Service, click Add New to add a Real Server to the Virtual Service.

image

A suggestion on how to configure the Virtual Services:

Virtual Address Port Service Name Persistence Scheduling
172.16.10.101 443 Exchange-HTTPS Super HTTP Round Robin
172.16.10.101 59531 Exchange-RPC Source IP Round Robin
172.16.10.101 59532 Exchange-AB Source IP Round Robin
172.16.10.101 135 Exchange-EPM Source IP Round Robin

Note: When required, you can also load balance inbound SMTP traffic using ports 25/587, IMAP4 (ports 143/993) and POP (110/995) using no persistence.

Note: Using Source IP can result in an unbalanced distribution of client load, when SNAT devices come into play. For an example scenario, see my earlier article on Load balancing, ActiveSync and Affinity.

And that’s basically it. When you want to channel specific HTTP services (Outlook WebApp, Exchange ActiveSync, Autodiscover etc.) you can appoint different FQDNs for each service and configure different FQDN/IP addresses per service in DNS, after which you can configure separate Virtual Services with more specific options. For example, you can not only configure specific persistence or scheduling settings for per Virtual Service, but also Real Servers checks (depending on the protocol). Instead of checking if a Real Server responds on port 443, you can check if the server responds on a different URL, e.g. https://<server>/owa.

image

Another bonus of using a load balancer, depending on functionality of the product used of course, is that you can (temporarily) disable a real server from the VLM. After doing this, clients won’t be directed to the corresponding Exchange server, which is very useful when you want to perform maintenance.

image

In this article we quickly went through setting up a KEMP VLM to load balance Exchange 2010 services. However, the article is based on certain decisions regarding the configuration, which can differ from organizational to organization. For more information on deploying KEMP VLM and its possibilities, check out the KEMP Virtual LoadMaster Deployment Guide here.

Most vendors, like KEMP, provide template functionality, which enables you to quickly set up the load balancer using preconfigured settings; make sure you inspect those settings afterwards (i.e. know what you’re doing). You can download KEMP templates here. Unfortunately, these files are in binary format so you can’t edit them nor can you export Virtual Services, otherwise I could have provided you with the template for the above settings.

Be advised that I am in no way connected to KEMP and this article hasn’t been sponsored  or commissioned by KEMP technologies, apart from providing an NFR license for writing and testing purposes.

Exchange can’t start due to misconfigured AD sites

Recently, a customer had issues with their Exchange server which didn’t start properly after rebooting. After checking out the Eventlog, I noticed the it was full of messages, generated by all services. The most interesting events were the ones generated by MSExchange ADAccess:

MSExchange ADAccess, EventID 2141
Process STORE.EXE (PID=2996). Topology discovery failed, error 0x8007077f

MSExchange ADAccess, EventID 2142

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1760). Topology discovery failed, error 0x8007077f

Also, the results of the active directory discovery process generated every 15 minutes, which are normally logging in event 2080, “Exchange Active Directory Provider has discovered the following servers with the following characteristics”, was missing.

Note that because the system could start the Microsoft Exchange Active Directory Topology service (until it failed and is restarted by dependent services), Exchange’s other services were also triggered, leading to almost indefinitely restarting services as configured in their corresponding service recovery actions sections.

Now, since I had connected to a domain controller using an RDP session from my client, and I was able to connect to port 389 (Global Catalog) from Exchange using LDP, so communications looked ok. Then, I switched to Active Directory Sites and Services:

image

As you can see from the shot, here was a potential cause of the problem. First, there was a site without domain controllers. Second, there were no subnets defined. So, in this situation, it is undetermined in which site Exchange is located.

When a system can’t be determined to which site a computer belongs, the function DSGetSiteName, used to retrieve the current site, returns an error 1919 0x77f (ERROR_NO_SITENAME). Consequently, the Exchange Active Directory discovery process fails and eventually Exchange fails. You can inspect the current discovered site using nltest /dsgetsite or by having a peek in the registry at HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\DynamicSiteName.

Now, to solve the situation we have three options:

  1. Making the site association static using a registry key, which isn’t a best practice.If you must, set registry key HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SiteName (REG_SZ) to the desired site name;
  2. Adding proper subnet definitions;
  3. Remove the empty site definition.

It turned out the empty site was a place holder for a future site, so we went with the option of adding proper subnet definitions. After adding subnet definitions, like you normally should when working with multiple sites, including the scopes where the Exchange servers and domain controllers were located, and associating it with the main site, things started working again.

Note that the NetLogon service determines site association membership at startup and every 15 minutes. The Microsoft Exchange Discovery Topology service maintains this information by caching the information in the msExchServerSite attribute of the Exchange server object, in order to reduce load on active directory and DNS. Therefor, you might need to wait or restart Microsoft Exchange Discovery Topology  if you want to renew site association membership.

TechEd North America 2012 sessions

With the TechEd North America 2012 event still running, recordings and slide decks of finished sessions are becoming available online. Here’s an overview of the Exchange-related sessions:


Setting up Disclaimers in Exchange 2010 SP1

Disclaimers are a controversial subject when added to e-mail messages; some lawyers think they should be mandatory for business communications, some think they’re just jibber-jabber. Fact of the matter is that your e-mail message may end up anywhere and in some countries you can’t legally bind recipients after receiving something he or she may not even wished to receive in the first place. If you’re interested, Exchange fellow Tony Redmond blogged shortly on some legal aspects of disclaimers in June here.

Despite the disputed legal status of these disclaimers, I am however consulted on how to configure them once in a while, so here’s how to implement them properly in Exchange 2010 SP1.

Disclaimers are set up using transport rules. You can inspect the current set of transport rules in the Exchange Management Console (EMC) under Organization Configuration > Hub Transport > Transport Rules or by using the Get-TransportRule cmdlet using the Exchange Management Shell (EMS). By default there will be no transport rules defined.

Before we begin you need to be aware of the following:

  • Transport rules will be processed in the order of priority. If a message meets the conditions configured in a transport rule, the actions defined in that rule will be processed. When done, the Exchange will continue evaluating transport rules with a higher priority number (so rule with priority will be evaluated first);
  • Adding disclaimers will invalidate signed and may corrupt encrypted messages.

This introduces the following complications:

  • When using multiple disclaimers, you need to mark the message after processing to prevent processing by one of the other transport rules. If you don’t, disclaimers will pile up at the end of your message.
  • Strangely enough, you can only make an exception for signed or encrypted messages, not both. This means that when you create a rule it is either for unsigned or unencrypted messages; what we want here is rules for unsigned, unencrypted messages.

To work around these limitations, we’ll use message classification for tagging encrypted or signed messages, so we can use that tag in the exception clause. In additional we’ll use custom message headers for flagging processed messages. Note that we could use custom headers for signed, encrypted messages as well but we don’t want to add yet another unnecessary named property.

We’ll start off by create the rules which are going to tag encrypted or signed e-mail messages. We choose to use message classifications for this purpose, so we need to add a new message classification first. This can only be performed using the EMS using the New-MessageClassification cmdlet:

New-MessageClassification “SignedOrEncrypted” –DisplayName “Signed or Encrypted Message” –SenderDescription “Signed or Encrypted Message” –PermissionMenuVisible:$false

image

The DisplayName and SenderDescription parameters are mandatory and their purpose is to state the purpose of the classification when displaying the message in Outlook. By setting PermissionMenuVisible to $false, users won’t be able to assign this classification to messages Outlook themselves.

Next, we’ll set up the transport rules to tag messages using this classification. Since these will be the first transport rules we’re going to create we don’t need to worry about the priority numbers yet. Since there isn’t much editing required with these rules, we can use EMS to define these rules:

New-TransportRule “Tag Encrypted Messages” –Enabled $true –MessageTypeMatches “Encrypted” –ApplyClassification “SignedOrEncrypted”

New-TransportRule “Tag Signed Messages” –Enabled $true –MessageTypeMatches “Signed” –ApplyClassification “SignedOrEncrypted”

image

Now we can start setting up the default (English) disclaimer using transport rules. This default disclaimer will be the default disclaimer attached to all outgoing e-mail messages. Using EMC, open up the Transport Rules tab and select New Transport Rule. Fill in a proper description and click Next.

Now we can select the conditions to meet in order for the rule to be processed. In this example, select From users that are inside or outside the organization and Sent to users that are inside or outside the organization or partners. In the bottom pane you can now fine tune these conditions by clicking on the underlined elements. Click Inside the organization, part of sent to users condition. Set this parameter to Outside the organization. When finished, click Next.

image

Next we’ll define the actions for this rule. First, select Append disclaimer text and fallback to action if unable to apply. In the bottom pane, leave append unchanged so the disclaimer will be added to the bottom of the message. Note that Exchange can’t insert the disclaimer below replies, only at the end of the complete message. Click disclaimer text and insert your disclaimer text in the editing window. Note that you can use HTML here as well as certain Active Directory attributes to make the disclaimer dynamic. For information on which Active Directory attributes are available, consult this TechNet article. When make use of HTML options, I suggest editing it in an HTML editor or something similar, so you can preview the result. Also, when using styles include those as well. Finally, leave wrap unchanged so Exchange will create a new message using the disclaimer text attach the original message attached if the disclaimer can’t be inserted in the original message.

Second, check Set header with value. Set header to X-Disclaimer and value to Yes. Note that this value is case-sensitive when we’re matching it later on. When finished, click Next.

image

Now we can enter exceptions for this rule. Select Except when the message is marked as classification and set classification to Signed or Encrypted Message. This will make sure that signed or encrypted messages tagged earlier will be skipped. Also check Except when the message header contains specific words; set Message header to X-Disclaimer and specific words to Yes. This will make sure the message is skipped when we’ve already added a disclaimer. When done, click Next.

image

Check the generated cmdlet on the configuration summary screen and click New to create the transport rule. Click Finish when done. Note that this rule will be added with Priority 0, so you need to alter the order setting Priority of this rule to the highest value possible, e.g. 2. As stated before, rules will be evaluated in ascending priority order.

Now we’re set up for a default disclaimer in English, but suppose you want to add a different disclaimer in a different language for users working in a different country. In order to establish this repeat the above steps for creating a disclaimer using transport rule, but at the conditions page add an additional condition by selecting When the sender’s properties contain specific words, setting the Property CountryOrRegion to the country code required, e.g. NL. To see which country code you need to use, consult the ISO 3166 table or check the C attribute in Active Directory (CountryOrRegion in OPATH maps to the C attribute in LDAP).  Needless to say, this filtering requires that the sender’s Country has been configured properly in Active Directory.

image

Then on the actions page, configure the alternative disclaimer text. After saving, don’t forget to put this rule before the default disclaimer rule by changing it’s Priority to n-1 , i.e. when the default disclaimer has priority 3 set the priority for this country specific disclaimer to 2.

image

The way this works is as follows:

  1. The first two rules will set the classification to SignedOrEncrypted for signed or encrypted messages;
  2. The Disclaimer NL rule checks the sender’s CountryOrRegion attribute; when it’s NL and the message isn’t classified as SignedOrEncrypted and the message doesn’t contain an “X-Disclaimer” header field with a value of “Yes”, it will add the configured disclaimer and set the header field “X-Disclaimer” to “Yes”;
  3. The Disclaimer (Default) rule checks if the message isn’t classified as SignedOrEncrypted and the message doesn’t contain an “X-Disclaimer” header field with a value of “Yes”. When both are not true, it will add the configured disclaimer and set the header field “X-Disclaimer” to “Yes”. This rule will be skipped for messages by sender with CountryOrRegion NL because the Disclaimer NL rule will be processed earlier and will add X-Disclaimer=Yes to the header.

Managing Remote IP Ranges of Receive Connectors

When managing receive connectors in Exchange, you probably had to configure IP addresses or IP ranges on those receive connectors. This may be required when limiting access to a certain receive connector for applications to drop their mail using SMTP. Of course this can be done using the Exchange Management Console, but this may become tedious when lots of addresses are involved. Also, when multiple Hub transport servers are involved you may need to keep those IP ranges in sync on those Hub Transport servers in which case mismatches are likely.

As you’ve probably guessed, a little PowerShell makes life more easier. To configure the allowed IP ranges we need to use Set-ReceiveConnector and configure the RemoteIPRanges attribute. We’ll use a text file to maintain the list of allowed IP ranges and a PowerShell one-liner to set RemoteIPRanges.

The file should contain IP ranges in a RemoteIPRanges acceptable format, e.g.:

  • 192.168.1.10
  • 192.168.1.20-192.168.1.29
  • 192.168.2.0/24

When we have prepared the file, we can use the following cmdlet to set RemoteIPRanges:

Get-ReceiveConnector *\APPRELAY | Set-ReceiveConnector -RemoteIPRanges (Get-Content RemoteIPRanges.txt)

This will configure all receive connectors named APPRELAY on all Hub Transport servers in the organization using IP ranges defined in the file RemoteIPRanges.txt. Be advised that this cmdlet overwrites the current configuration of RemoteIPRanges; if you need to add it to the current configured set of IP ranges on each receive connector, use the following cmdlet:

Get-ReceiveConnector *\Appl-Relay | ForEach { Set-ReceiveConnector -RemoteIPRanges ($_.RemoteIPRanges+ (Get-Content ipranges.txt) | Sort -Unique) }

By adding the Sort -Unique filter, we make sure each range is only specified once. This prevents errors caused by setting a range using the RemoteIPRanges.txt file when that range has already been configured in the current value of RemoteIPRanges.

Note that when inspecting the results you can set $FormatEnumerationLimit to a value higher than the default (16) to have Get-ReceiveConnector * | fl RemoteIPRanges display all its values. Also, keep in mind when configuring connectors that the connector with the most specific matching IP address wins.

TechEd North America 2011 sessions

With the end of TechEd NA 2011, so ends a week of interesting sessions. Here’s a quick overview of recorded Exchange-related sessions for your enjoyment:

More Exchange 2010 Tested Solutions

Microsoft published additional technical solution papers:

  • Two Sites, 20,000 mailboxes
    Virtualized solution running Hyper-V on Dell R910 Servers, EMC CLARiiON Storage, and Brocade Network Solutions
  • Two sites, 15,000 mailboxes
    Virtualized solution running on Unisys ES7000 Servers and Hitachi Adaptable Modular Storage 2000 Family

Note that I’ve added a seperate Tested Solutions page to the References section where I’ll try to keep track of these tested solutions.

SSL client compatibility

Exchange fellow Jetze Mellema blogged (in Dutch) about a useful online check, which will allow you to check your current client – computer or smartphone – against a set of certificates from different vendors. The short – and more memorable and mobile friendly – URL for this test is as follows: http://m.ssltest.net.

The creator, SSL reseller FairSSL, also keep a total overview, which is located at http://www.ssltest.net/compare/sar.php. Note that the table’s titles are hard to read, but when hovering above the cells the corresponding product will be displayed.

MS publishes Exchange 2010 Tested Solutions

Microsoft published several technical solution papers on how to design, test and verify Exchange 2010 solutions for a certain number of mailboxes for a few vendors. The papers are as follows:

The documents walk you through the steps regarding design decisions, challenges while meeting customer requirements.