powershellLast update: Version 1.01, July 14th, 2014.

Those leveraging quota settings to manage their Exchange environments, you are probably periodically running some sort of script or set of cmdlets to retrieve information on mailbox sizes, quota settings and if any mailbox is above any of the quota thresholds. For a quick indication of the current size in relation to the quota settings, StorageLimitStatus may contain one of the following indicators depending on the quota settings on the mailbox or mailbox database hosting the mailbox:

  • BelowLimit – Speaks for itself
  • IssueWarning – Mailbox size above Issue Warning limit
  • ProhibitSend – Mailbox size above Prohibit Send limit
  • NoChecking – No quota checking
  • MailboxDisabled – Mailbox size above Prohibit Send and Receive quota limit

So, to get a list of all mailboxes with any over-quota status, you can use:

Get-MailboxDatabase | Get-MailboxStatistics | Where {$_.StorageLimitStatus -match 'IssueWarning|ProhibitSend|MailboxDisabled'} | Select DisplayName, ItemCount, TotalItemSize, StorageLimitStatus, LastLogonTime

Unfortunately, in Exchange 2013 the StorageLimitStatus gets no longer populated:


As KB2819389 explains, this is by design. In Exchange 2013, mailbox quotas are no longer cached. By not being cached, retrieving quota information may result in poor performance as it queries Active Directory for quota related attributes. The argument is a bit puzzling, considering there is a NoADLookup switch which directs the cmdlet to retrieve information from the mailbox database (cache) instead of Active Directory. Perhaps a better workaround would have been to make NoADLookup a parameter, make it $true by default and leave StorageLimitStatus unpopulated when NoADLookup is $true.

Of course, that does not help customers who want a quick quota report. For this purpose I have created two things in 1 script:

  1. A helper function Get-StorageLimitStatus() which will take a mailbox statistics object and return a StorageLimitStatus object.
  2. A script Get-MyMailboxStatistics.ps1, a proxy function for Get-MailboxStatistics which will use the Get-StorageLimitStatus helper function to populate the StorageLimitStatus.

When you want to use the helper function, extract it and include it in your quota reporting script or PowerShell profile (making it available when firing up a shell). To use the helper function in the cmdlet shown earlier, use:

Get-MailboxDatabase | Get-MailboxStatistics | Where {$_.StorageLimitStatus -match 'IssueWarning|ProhibitSend|MailboxDisabled'} | Select -ExcludeProperty StorageLimitStatus DisplayName, ItemCount, TotalItemSize, @{n="StorageLimitStatus"; e={ Get-StorageLimitStatus $_}}, LastLogonTime

This will remove StorageLimitStatus from the output and add a calculated field bearing the same the name, calling the Get-StorageLimitStatus helper function with the current mailbox statistics object to set its value.

This is a proxy function for the Exchange Management Shell cmdlet Get-MailboxStatistics. This means that the current, original cmdlet was used to create a wrapper which will call the original cmdlet. Having a wrapper allows you to restrict or enhance the original cmdlet and tailor it to your needs.

A quick tip on how to create a proxy script in the clipboard (more information on creating proxy commands here):

$data= New-Object System.Management.Automation.CommandMetaData (Get-Command Get-MailboxStatistics) 
[System.Management.Automation.ProxyCommand]::create($data) | clip.exe

Downside is that future changes to the Get-MailboxStatistics cmdlet will not be automatically incorporated in the wrapper. Feeding it objects also doesn’t work, but you can work around that by temporary storing the objects in a variable and passing that to the script (see examples below).

To populate the StorageLimitStatus, we will post-process each object in the output of Get-MailboxStatistics, using Add-Member to overwrite (-Force) its current value and –PassThru to pass it along in the pipeline. Being a proxy command, the parameter options are identical to the original Get-MailboxStatistics. Some examples:

.\Get-MailboxStatistics.ps1 -Database MDB2
$m= Get-Mailbox –Database MDB2 
$m | .\Get-MailboxStatistics.ps1 | ft –AutoSize DisplayName,TotalItemSize,StorageLimitStatus


Do be aware that this will incur Active Directory queries and thus performance of the script may not seem fast. However, in previous versions of Exchange you got immediate results as all the quota information was readily available from the cache. On the plus side, the status you see will be non-cached, current information.

On a final note and maybe needless to say that in order to use this you need to run it from the Exchange Management Shell and since it’s an unsigned script you need to set ExecutionPolicy to Unrestricted.

Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

You can download the script from the TechNet Gallery here.

Revision History
See Technet Gallery page.

Can’t Create Mailboxes in Remote Sites

Ex2013 LogoRecently I got an e-mail from someone who had problems creating mailboxes in a new environment. When trying to create a mailbox, he received a following message stating, “Load balancing failed to find a valid mailbox database.” Apparently, the Mailbox Resources Management Agent (a Cmdlet Extension Agent) could not find an eligible mailbox database candidate.


The MRMA uses the following selection process when picking a candidate for mailbox creation or moving:

  1. Create a list of all mailbox databases;
  2. Remove databases marked for exclusion;
  3. Remove databases out of the management scope;
  4. Remove databases from remote (AD) sites;
  5. Pick a random online, healthy database from the list.

This person had a DAG, two mailbox databases (MDB1, MDB2) and two sites (AMS and LON).

We first checked the more or less obvious, which is to see if databases are not excluded from the provisioning process, so we entered Get-MailboxDatabase | fl *FromProvisioning:


Databases seemed enabled for provisioning. We then checked the status of the active database copies:


The copies looked healthy, but we noticed all databases were mounted in a remote site (derived from the server name starting with LON; we’re working from AMS). Looking back at the database selection process, it explained why it probably didn’t work and since the active copies should be moved back to the preferred site AMS anyway we moved the active copies back:


After moving the active database copies back to the location where we were performing our cmdlets from solved things.

Note that we could have discovered the issue using the Verbose parameter with the cmdlet. For example, New-Mailbox in conjunction with Verbose will show the selection process. The following screenshot shows an unsuccessful selection process considering available databases:


This screenshot shows a successful selection process.


More information on automatic mailbox distribution and controlling its behavior here.

NGN Exchange Event, Tips & Tricks Presentation

On October 31st, the NGN – a Dutch society for IT professionals – held its 3rd Exchange themed event, this time at The Reehorst in Ede (NL). Because of the recently released Exchange 2013 and all the news and related questions, we planned for a whole day of sessions and it was nice to see the turn up was nearly 100 IT professionals.

Since all people would still be on pre-2013 versions of Exchange, I figured a presentation using real-world Exchange 2010 Tips and Tricks might be more appropriate. I was glad a quick poll amongst the attendees showed a significant increase in Exchange 2010 deployments (around 80%) when compared to last year’s event, but as expected there’s still some Exchange 2007 and few Exchange 2003 out there.

I decided to stick with two deep-dive topics, which were Message Trackings Logs and Cmdlet Extension Agents. On those topics I went from basics to more advanced examples, hoping it would ignite people with no experience and people with experience could still pick up a thing or two.I’m still waiting for evaluation results, the only way to get feedback from these sessions apart from the occasional e-mail or tweet.

(picture by Dave Stork)

You can find my presentation here (partially Dutch) and the accompanying sample script on Message Tracking Logs here and the one on Cmdlet Extension Agents here (script); the ScriptingAgent.xml file can be downloaded here.

As always, these events are also a time to catch up with fellow Exchange people and discuss topics with attendees during the breaks. There were even Exchange fellows present who didn’t have a session, like Johan Veldhuis (MVP) and Maarten Piederiet (MCM); they did join in on the Q&A Panel.

The sessions and speakers were:

  • Introduction (Jaap Wesselius, MVP)
  • Building with Exchange 2013: Architecture (Dave Stork)
  • Exchange and Virtualisation (Jetze Mellema)
  • Exchange 2010 Tips & Tricks (Ashley Flentge, MCM & Michel de Rooij)
  • Exchange 2013 Coexistence and Migrations (Kay Sellenrode, MCM and MCA)
  • Exchange and Load Balancing (Jetze Mellema)
  • Q&A Panel

The NGN published all presentations in a single ZIP file which can be downloaded here. Unfortunately, NGN didn’t record the sessions so I can’t share those with you. They did record the Q&A Panel session; you can view it here (in Dutch):

PS: When you see references to “exchangedag”, like in the Twitter hashtag, you need to know “dag” means day in Dutch; it’s no form of professional deformation.

Cmdlet Extension Agents Part 2: Postconfiguring Mailboxes

Cmdlet Extension Agents Part 1: Automatic archive creation

Almost a year ago, I posted an article in which I tried to show the power of Cmdlet Extension Agents in Exchange 2010, or more specifically, the Scripting Agent. Unfortunately, the Cmdlet Extension Agents are often overlooked or ignored, despite customers having requirements to customize things immediately after creating a mailbox. Therefor, I decided to write another article on this topic, hoping people take up using Scripting Agents.

Now while you can also put all sorts of post-configuration tasks in provisioning scripts, using the Scripting Agent when possible has a big bonus, because those additional actions not only run when you run the cmdlet directly from the Exchange Management Shell but also when you run them indirectly by using the Exchange Management Console.

So, as this follow up of the previous article, in which I explained what the CmdLet Extension Agents are and how to utilize the Scripting Agent to automate tasks, I’ll show you another example of a Scripting Agent and quickly walk you through it, so you can experiment with it (first in a lab of course) and tune it to your own requirements.

In this example, we’ll disable ActiveSync and configure SingleItemRecovery when creating a new user with a mailbox or mailbox-enabling an existing user. Therefor, the cmdlets we’re going to customize are New-Mailbox and Enable-Mailbox.

Open up Notepad and create a file \bin\CmdletExtensionAgents\ScriptingAgentConfig.xml located in Env:ExchangeInstallPath, e.g. C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents, using the following contents:

Note: If you’ve already got a ScriptingAgentConfig.xml file, you need to integrate the following content.

<?xml version="1.0" encoding="utf-8" ?>
 <Configuration version="1.0">
 <Feature Name="Mailboxes" Cmdlets="New-Mailbox,Enable-Mailbox">
 <ApiCall Name="OnComplete">
   if($succeeded) {
     $Name= $provisioningHandler.UserSpecifiedParameters["Name"]
     Set-Mailbox $Name -SingleItemRecoveryEnabled $true
     Set-CASMailbox $Name -ActiveSyncEnabled $false

As you can see, you’re not limited to 1 action or related cmdlets (*-Mailbox). A small explanation:

  • The Cmdlets specified in this feature extension dictates which cmdlets will be extended, in this case New-Mailbox and Enable-Mailbox;
  • OnComplete dictates that our script will fire when the cmdlet has finished;
  • We check for OnComplete parameter $succeeded, only configuring the mailbox when the preceding events were successful;
  • $provisioningHandler.UserSpecifiedParameters contains user provided parameters passed to the cmdlet. So, $provisioningHandler.UserSpecifiedParameters[“Name"] will return the value of –Name;
  • We set SingleItemRecovery to $true for the mailbox specified by $Name;
  • We disable ActiveSync client access for this mailbox as well.

As mentioned in part 1, distribute this XML file to all your Exchange servers in the local CmdletExtensionAgents folder. When you haven’t already enabled the Scripting Agent, do so by running the following cmdlet:

Enable-CmdletExtensionAgent “Scripting Agent”

Now, when we create a new mailbox or mailbox-enable an existing user:


.. you’ll see the SingleItemRecovery has been enabled and ActiveSync has been disabled for this mailbox by the scripting agent:


I recommend you start checking out the Scripting Agent if you haven’t already done so. You can use these examples as a starting point and work from there. More information on the Scripting Agent, alternative APIs etc. can be found here.

Thoughts on “Automatic E-mail Server Notifications in Exchange 2010″

In an article on, Markus Klein elaborates on the reasons behind the changed message delivery notification (MDN) behavior in Exchange 2010. Examples of MDNs are read or delivery receipts or out of office messages. Issues may arise with MDNs because Exchange 2010 (and Exchange 2007) will use a blank sender address and not all e-mail systems can cope with that, making Exchange compliant with the related RFC. The article ends with workarounds to mitigate the issue. Here are my thoughts on that article.

The article refers to RFC2298, dated March 1998. However, MDNs are defined by RFC3798 of May 2004, which obsoletes RFC2298. Nevertheless, like Klein indicated, both RFCs dictate the following:

The envelope sender address (i.e., SMTP MAIL FROM) of the MDN MUST be null (<>), specifying that no Delivery Status Notification messages or other messages indicating successful or unsuccessful delivery are to be sent in response to an MDN.

The idea behind using a blank sender address is that e-mail systems will not return DSN messages, e.g. mailbox unavailable or disk quota exceeded, as a reply to an MDN, preventing potential message loops. However, there are some side-effects as not all e-mail systems or messaging hygiene products are RFC compliant. For example, the default setting of ForeFront Protection 2010 for Exchange is to block messages with an empty sender address. These products may simply block those messages, since blank senders could potentially be an indicator for spoofed messages. When you suspect such product to be causing the issue, check and reconfigure when appropriate.

The author continues the article by describing how to configure and troubleshoot routing of MDNs to the internet. The author shows how to enable and inspect the receive connector logs. Instead, I suggest monitoring the send connector logs when troubleshooting MDN delivery. Inspecting the send connector log files, you can get a clue on why MDN delivery fails and will see if Exchange is trying to deliver the MDN at all, and if so, the reason why. To enable send connector logging use the following cmdlet:

Set-SendConnector <ConnectorID> -ProtocolLoggingLevel verbose

The log files are generated in the “V14\TransportRoles\Logs\ProtocolLog\SmtpSend” folder below the location where you installed Exchange.

Finally, the author suggests the following workarounds:

  1. Use Outlook “out of office”
  2. Switch Relay Provider
  3. Implement Exchange Server Edge Roles

The first workaround is a less preferable option, as it’s configured per-user as a rule and rules, stored in the user’s mailbox, can’t easily be managed. When using the OOF option, administrators can, using the Get-MailboxAutoReplyConfiguration and Set-MailboxAutoReplyConfiguration cmdlets. Also, it makes the end user responsible for working around the issue. Meanwhile, despite this instruction, you can still expect lots of users to keep using the OOF function.

The second and third suggestions are non-options, since they don’t eliminate the issue and will only add a product and an extra hop to the e-mail route. Yes, you can switch to using a different SMTP relay or implement an Exchange Edge server which will accept MDN messages with an empty sender address. However, that may not be the final destination of the e-mail message, so the (unpredictable) MDN delivery issue remains. Nobody can guarantee that the e-mail system or message hygiene appliance at the recipient blocks blocks your OOF message with an empty sender address. You can read that between the lines of the PSS statement the author quotes as well:

The Exchange edge server will not reject the OOF message as the edge server will be incorporated into the Exchange organization. The HUB server will transfer the OOF messages in the address of OOF mailbox to the edge server and the edge server will then send the messages with empty return path e.g. blank sender, MAIL FROM: <> “null” to Internet.

Now, when the issue lies outside of your Exchange organization, e.g. the hosted message hygiene service or destination mail system, you might be left with no other option than to violate RFC3798 by adding a sender address. In Exchange this isn’t possible, but other e-mail gateways could help you with that. Note that when using a hosted message hygiene service or appliance for outbound messages, using a non-blank sender might be less of an issue since you’re offloading the delivery, compared to trying to deliver the message to the destination mail system yourself.

However, when opting to resort to these measures, I’d strongly suggest reconsidering sending out of office messages (or MDNs in general) outside of your Exchange organization, regardless of the sender. Spammers love confirmed e-mail addresses, so treasure your business e-mail addresses like you probably treat your own personal address.

Note that this blog isn’t to condemn the author of the discussed article, but to clarify things up since many people moving from Exchange 2003 to Exchange 2007 or Exchange 2010 may run into these behavioral differences. You’re invited to comment or share your opinions in the comments below.

Exchange Management Console & IE9 issue fixed

Finally, today the Exchange team made available a fix to solve the issues when using the Management Console of Exchange 2007 or 2010 in conjunction with Internet Explorer 9.

As you probably know, when using Internet Explorer 9 you can’t close the Exchange Management Console properly as it gives you the error “You must close all dialog boxes before you can close Exchange Management Console” having no dialogs open.

To solve this issue, you had to do the resort to measures like killing the EMC process using Task Manager.

To properly install the hotfix:

  1. Request hotfix ID 2624899 from support here. For a direct download link click here.
  2. Download and install MS11-081 (2586448). You can retrieve this update here.
  3. Install the hotfix ID 2624899.

Microsoft states it expects to incorporate this fix in a future update of Internet Explorer 9.

While releasing a fix for the IE9 issue is great after all these month, I can’t help but wonder why the fix has not been made public.

Exchange 2010 RTM EOL’s on October 11th

After returning from holiday, between all the Build Windows (Windows 8 ) news, a quick heads-up for those with lagging upgrade schemes or any other valid reason to be still running Exchange 2010 RTM. On October 11th, 2011, support for Exchange 2010 RTM will end.

This should be of no surprise when you practice proper lifecycle management or track Microsoft’s KB bulletins as this information was published on the lifecycle page as well as knowledge base article KB2615653.

For those doing fresh installs and still wondering if this affects their process of installing SP1 versions starting by using the RTM files; since Exchange 2007, Service Packs for Exchange contain all binaries enabling you to perform a fresh installation as well as an upgrade using the same set of files.

You can download Exchange 2010 Service Pack 1 here.

Cmdlet Extension Agents Part 1: Automatic archive creation

Cmdlet Extension Agents Part 2: Postconfiguring Mailboxes

An Exchange fellow inquired about the possibility to automatically enable personal archives when creating mailboxes with the added requirement to create those personal archives in a specific mailbox database, depending on the location of the mailbox. Dedicated mailbox databases were used for personal archives. Simply said, the idea was that mailboxes located in database MDB1 should get a personal archive in mailbox database ADB1, MDB2 in ADB2, etc.

Your first thought could be creating a script to automatically provision those personal archives in the proper database depending on the mailbox database. But alas, when using Exchange 2010’s automatic mailbox provisioning system you never know upfront what mailbox database will be appointed.

That leads us to Exchange 2010’s Cmdlet Extension Agents, more specific the Scripting Agent. I won’t go into much detail now on those Agents, but look at them as a way to extend cmdlets by adding pre- and post-jobs, additional constraints, reporting or override parameters.

Now, when you haven’t already done so, first exclude the mailbox databases containing personal archives from automatic provisioning. If you have a dedicated server for hosting personal archives, use the IsExcludedFromProvisioning with the Set-MailboxServer cmdlet; to exclude a mailbox database use IsExcludedFromProvisioning with the Set-MailboxDatabase, e.g.

Set-MailboxDatabase <Archive Database ID> –IsExcludedFromProvisioning $true


I’ll first show you how the scripted version could work. We’ll start by creating some mailboxes. We don’t require anything fancy, so this will do:

$pwd= ConvertTo-SecureString -AsPlainText “Welcome1 -Force
1..10 | ForEach { New-Mailbox “User$_ -Password $pwd -UserPrincipalName user$_@<domain> }

A quick overview of the result shows the mailboxes are created in a round robin fashion:


What you could do now is enabling the archive on ADB1 for MDB1 and ADB2 for MDB2 mailboxes, e.g.

Get-Mailbox –Database MDB1 | Enable-Mailbox –Archive -ArchiveDatabase ADB1
Get-Mailbox –Database MDB2 | Enable-Mailbox –Archive -ArchiveDatabase ADB2


This is what we wanted. As you probably understand, the main disadvantage now is that this only works for the current mailbox population. Administrators should appoint the proper mailbox database for personal archives when creating new mailboxes. Can the Scripting Agent overcome this problem?

Let’s have a look on how to configure the Scripting Agent. Open up Notepad and create a file \bin\CmdletExtensionAgents\ScriptingAgentConfig.xml located in Env:ExchangeInstallPath, e.g. C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents, using the following contents:

<?xml version="1.0" encoding="utf-8" ?>
  <Configuration version="1.0">
  <Feature Name="MailboxProvisioning" Cmdlets="New-Mailbox">
  <ApiCall Name="OnComplete">
  If($succeeded) {
    $Name= $provisioningHandler.UserSpecifiedParameters["Name"]
    If ((Get-Mailbox $Name).ArchiveDatabase -eq $null) {
      $MailboxDatabase= (Get-Mailbox $Name).Database
      $ArchiveDatabase= "A"+ ( $MailboxDatabase.Name).Substring( 1)
      Enable-Mailbox $Name -Archive -ArchiveDatabase $ArchiveDatabase

A small explanation might be appropriate:

  • The Cmdlets specified in this feature extension dictates which cmdlets will be extended;
  • OnComplete dictates that our script will fire when the cmdlet has finished;
  • We check for OnComplete parameter $succeeded, only enabling archives when the preceding cmdlet was successful;
  • $provisioningHandler.UserSpecifiedParameters contains user provided parameters passed to the cmdlet. So, $provisioningHandler.UserSpecifiedParameters["Name"] will return the value of -Name;
  • We’ll check if the mailbox already has a personal archive configured; if not, we can proceed;
  • Next, we’ll get the current MailboxDatabase. Then we’ll map that to our personal archive naming scheme by stripping the first character and prefix it with “A”;
  • Finally, we can execute the cmdlet to enable the personal archive of the mailbox on the database specified.

Now, before we test our scripting agent, we need to distribute the XML file on all of our Exchange servers. The reason for this is that you don’t know which Exchange server an administrator will connect to or which server will execute the cmdlet. The location to copy the XML file to is the local CmdletExtensionAgents folder.

Now there’s one more thing we need to do, which is enabling the Scripting Agent. The Scripting Agent is disabled by default. Use the Enable-CmdletExtensionAgent cmdlet to enable it, e.g.:

Enable-CmdletExtensionAgent “Scripting Agent”

Now, when we use the same cmdlet we used before to create those mailboxes, we get the following result:


As you can see, archive databases are now nicely aligned with the automatically assigned mailbox databases.

A small note for those wishing to experiment with the Scripting Agent. Alternatively to OnComplete, you can also try defining the personal archive parameters using the ApiCall ProvisionDefaultProperties. This ApiCall can be used to define default attributes when creating a mailbox. However, this leads to a catch 22 situation and has to do with the Mailbox Resources Management Agent.

By default the Mailbox Resources Management Agent has higher priority (2) than the Scripting Agent (6). This means it will override any settings made in our Scripting Agent.


The Mailbox Resources Management Agent is responsible for the automatic mailbox distribution when you don’t specify a mailbox database when creating a mailbox. But it is also responsible for assigning a mailbox database for the personal archive when you don’t specify the ArchiveDatabase parameter.

So, unless we want to add all the automatic mailbox distribution logic to our script, we can’t use the ProvisionDefaultProperties ApiCall properly, because if we want to use that, we need to assign the Scripting Agent a higher priority than the Mailbox Resources Management Agent, but at that point we have no database value so we can’t determine the proper archive database.

If you’re interested in playing with this, check out the ScriptingAgentConfig.xml.sample file which is located in the CmdletExtensionAgents as well. If you’re looking for more information on Cmdlet Extension Agents, check here; information on the Scripting Agent can be found here. More information on the automatic mailbox distribution process here.

Updating Exchange 2010 DAG Members

With all the (re-)releases of rollups, the question might rise on how to perform a proper up or downgrade of all DAG configuration members.

Basically, the procedure is straightforward and should be followed per DAG member:

  1. Appoint (next) DAG member;
  2. Move away all active copies on that DAG member;
  3. Prevent copies from activating on that DAG member;
  4. Perform maintenance, e.g. down or upgrade DAG member;
  5. Enable possible activation on that DAG member again;
  6. Optionally redistribute database copies.

Note that in a DAG configuration with 2 members, you need to be aware that during maintenance you have a temporary situation with no fail-over options. If that’s undesirable, consider implementing a 3rd DAG member.

To make the above procedure  easier and automated regarding moves and activation (un)blocking, additional scripts are available since SP1 for Exchange 2010. These scripts are located in the Scripts folder, below the Exchange installation folder. By default the location of the scripts will be C:\Program Files\Microsoft\Exchange Server\v14\Scripts.

Utilizing them, the procedure is quite easy as you can see below. Note that the example uses a DAG named DAG1 with nodes ex2010a and ex2010b as members. They both host 2 databases, ex2010mdb1 and ex2010mdb2; both host 1 active copy and a passive copy of the other database.

  1. Appoint (next) DAG member, e.g. ex2010a;
  2. Run StartDagServerMaintenance.ps1 targeting that DAG member, e.g.:
    .\StartDagServerMaintenance.ps1 –server ex2010a

  3. Perform maintenance;
  4. Run StopDagServerMaintenance.ps1 targeting that DAG member, e.g.:
    .\StopDagServerMaintenance.ps1 –server ex2010a
  5. Repeat steps 2-3 for the other DAG member(s):image
  6. Optionally run RedistributeActiveDatabases.ps1 for the DAG, e.g.:
    .\RedistributeActiveDatabases.ps1 –DagName DAG1 –BalanceDBsByActivationPreference –Confirm:$false


Be advised that when upgrading on major levels (RTM to SP1 or SP1 to SP2), you can’t move a database to a lower level host. This means that when upgrading a node from SP1 to SP2 and moving a database to that SP2 node in the process, you can’t move that database to any SP1 nodes in the DAG. Keep this in mind when planning your upgrade, because it will impact the availability level by limiting your fallback options, albeit temporarily.

A Decade in High Availability

A recent post from Elden Christensen, Sr. Program Manager Lead for Clustering & High Availability, reminded me of one of my former employers. When I joined that company back in 2000 for starting up a professional services based on Windows Server 2000 Data Center Edition, the company was already an established professional services provider in the business critical computing niche market, e.g. Tandem/Compaq/HP NonStop systems, mostly used in the financial markets, e.g. banks or stock exchanges. The Windows platform was regarded as inferior at that time by the NonStop folks and they had good arguments back then.

Remember, those were also the early days where no one was surprised to see an occasional blue screen (people were also using Windows 9x) and what we now know as virtualization was already happening on mainframes in the form of partitioning. At that time, Microsoft with their Windows Server platform had ambitions to enter the data center environment, where the NonStop platform was an established platform for ages and professionals had developed best practices for those environments.

Another part of the discussion was the Fault-Tolerance  versus High Availability topic, where NonStop was already an established Fault Tolerant solution for business critical environments, Windows still had only ambitions to move towards that market with the Data Center product. A logical move, looking at the status of (web) applications, SQL and last but not least, Exchange and where it was going and what customers expected of those products regarding availability and reliability. To repeat an infamous quote of a NonStop colleague back then, “E-mail is not business critical”. But that was almost 10 years ago, things have changed .. or haven’t they?

Single Point of Failure
First I’ll start by introducing the availability concept, which revolves around eliminating the single point of failure. This is an element in the whole system of hardware, software and organization that can cause downtime for a system, i.e. disruption of services. After identifying a single point of failure, we want to eliminate it to prevent downtime which is, after all, the ultimate goal for a business critical system. We can approach this task using two different strategies, Fault Tolerant (FT) or High Availability (HA). The task of identifying and eliminating single points of failure is an ongoing process, as most IT environments are subject to change over time.

To understand the Fault Tolerant and High Availability strategies we need to define the term “Availability”. In the dictionary, availability is defined as the quality or state of being available or an available person or thing, where in both cases available means present or ready for immediate use. The availability is mostly expressed as a percentage, for example when used in a service level agreement, but what does that percentage mean? To explain this take a look at the following diagram:


I assume this lifecycle speaks for itself. Using this diagram, the availability is calculated as follows: MTBF / (MTBF + MTTR). The related expected downtime is calculated as ( 1 – Availability% ) * 1 year. Note that the time between failure and recovery isn’t used in the calculation.

I’ll use a simple example, a 500 GB Seagate Barracuda 7200.12 (ST3500412AS) with a MTBF specification of 750,000 hours. You have a 24 hours replacement contract and need about 4 hours to restore the backup. The availability would then be 750,000 / ( 750,000 + 28 ) = 0.9999626680% resulting in a yearly downtime of ( 1 – 0.9999626680) * ( 365 days  * 24 hours * 60 minutes ) = 19,6 hours.

Of course with hardware these numbers are theoretical and to some extent a marketing thing; how else can Seagate specify an MTBF of 750,000 hours ( 85 years ). I tend to look at it as an indication of the reliability you can expect. For example, compare the MTBF of 7200.12 drive with an enterprise class drive, Seagate’s ES product line. The ST3500320NS has an MTBF of 1,200,000 hours.

That’s the reason you should use enterprise class drives in your storage solution instead of desktop drives, which aren’t supposed to run in 24×7 environments. To add to that, the MTBF decreases when used in series (RAID 0 = 1 / (1/MTBF1 + .. 1/MTBFn)) or increases when used in parallel (RAID 1 = MTBF * ( 1 + 1/2 + 1/n) ) configurations. When trying to do calculations for the whole supply chain, with all the elements and their individual specifications and support contracts, this can get very complex.

The 9’s
imageWhen talking about availability this is often shown using a series of 9’s, e.g. 99.9%. The more 9’s it has, the better (less downtime). Note that for each increased level of availability, the required effort increases significantly. By effort, don’t think of technical solutions only. It also means organizational measures like having skilled personnel and proper procedures.

A fact is that only a small percentage of the causes for outage is technical, the majority of incidents is due to human error. And yes, that includes that bad driver which is programmed by humans. This is why changes in properly managed infrastructure should always go through test and acceptance procedures in environments representative or identical to the production environment. Unfortunately, this doesn’t always happen as not all IT departments have this luxury, mostly because of financial reasons.

Availability% Downtime / Year Downtime / Month
99.0% 3.65 days 7.3 hrs
99.9% 8.76 hrs 43.8 min
99.99% 52 min 4.3 min
99.999% 5.2 min 26 sec
99.9999% 31 sec 2.6 sec

Fault Tolerant
imageThe goal of a Fault Tolerant solution is to maximize the Mean Time Between Failure (MTBF). This is achieved by mirroring or replicating systems. These monolithic systems run software in parallel on identical hardware. This is called Lockstep (which, for your information, refers to synchronized marching).

Because Fault Tolerant systems run in parallel, the results of an operation can be compared. When the results don’t match, a fault occurs. Since the faulty system can’t be identified using 2 parallel systems, there’s also a variation to this architecture where one server functions as master and one as slave, the slave functioning as a hot-standby. To solve the ambiguity, you could use three systems where the majority of the systems determine the right output.

When faults are detected in a Fault Tolerant system, the failing component (or system) is disabled and the mirror takes over. This makes the experience transparent for the end-user. There is one caveat: since Fault Tolerant systems run software in parallel, software faults are also mirrored.

Examples of Fault Tolerant components are ECC RAM, multiple NICs in Fault Tolerant configuration, multipath network software, RAID 1+ disk systems or storage with replication technology. Examples of Fault Tolerant systems are HP NonStop (propriety), Stratus ftServer or Unisys ES7000. There are also software-based solutions like Marathon EverRun or VMWare’s FT offering.

High Availability
High Availability aims to maximize minimize the MTTR. This can be achieved by redundant or standby (cold, hot) systems or non-technical measures like on-site support contracts. Systems take over the functionality of the failing system after the failure occurred. Therefor, High Availability solutions aren’t always completely transparent for the user. The effects of a failing system and the consequences for end end-user depend on the software, e.g. a seamless reconnect or requirement to login again. Another point of attention is the potential loss of information caused by pending transactions being lost because of the failure. To make the experience more transparent for the user, application need to be resilient, e.g. detecting failure and retrying the transaction.

Examples of High Availability technologies are load balancing – software or hardware-based – and replication, where load balancing is used for static data and replication for dynamic data.

The Present
After a decade, technology has evolved but is still founded on old concepts. Network load balancing is still here and clustering (anyone remember Wolfpack?), although we moved from shared nothing to to replication technology, remains largely unchanged. This means either there hasn’t been much innovation or the technologies do a decent job; After all, it’s still a matter of demand and supply. Yes, we moved from certified configurations-only shared storage solutions to flexible Database Availability Groups (hey, this is still and Exchange blog), but most changes are in the added functionality category or to take away constraints, e.g. cluster modes (majority node set, etc.), multiple replicas and configurable replication.

Windows Server
Data Center Edition
x86 x64
2000 Max. 32 GB
32 CPUs
4 nodes
2003 SP2 Max. 128 GB
32 CPUs
8 nodes
Max. 512 GB
64 CPUs
8 nodes
2003 R2 SP2 Max. 64 GB
32 CPUs
8 nodes
Max. 2 TB
64 CPUs
8 nodes
2008 Max.  64 GB
32 CPUs
16 nodes
Max. 2 TB
64 CPUs
16 nodes
2008 R2 N/A Max. 2 TB
64 CPUs (256 logical)
16 nodes

What about Fault Tolerance and Windows’ Data Center Edition as the panacee for all your customers requiring “maximum uptime”? The issue with Fault Tolerant was that it came with a hefty price tag, especially in those days. Costs were an x-fold of the costs involved with High Availability solutions on decent (read: stable) hardware. So, for those extra 9’s you needed deep pockets. For example, around 2001 an Compaq ES7000 with Windows Server 2000 Data Center Edition, the joint-support queue (e.g. Microsoft and OEM) and services came with a $2m price tag for which you got the promise of 99,9% availability.

Compare that to buying a few Proliant’s with Windows Server 2000 Advanced Server, some Fault Tolerant components (FT NICs, RAID), off the shelf High Available technology and dedicated personnel (justifiable with that DCE price tag) for .. say, $250,000. With skilled personnel and operated in a controlled environment you could easily reach 99% availability. Is that price difference worth 3 days of downtime? Also, the simplicity to implement those technologies made High Availability in Windows accessible for the masses and now – certainly in the Exchange world – seldom see load balancing or forms of clustering not being utilized.

Note that in the past decade, I’ve never encountered Data Center for hosting Exchange. In fact, as of Exchange 2003, support for on Data Center was dropped. Nowadays, Data Center is regarded as an attractive option for large-scale virtualizations based on Hyper-V, not only because Data Center costs less than back then (about $3000 per CPU – hurray for multi core, but with a 2 CPU minimum) and runs certified on more hardware, but also because it comes with unlimited virtualization rights, meaning you may run Windows Server 2008 R2 (or previous version) Standard, Enterprise, and Datacenter in the virtual instances without the need to purchase additional licenses for those.

With all the large-scale virtualization and consolidation projects going on, virtualizing Exchange or other parts of your IT infrastructure, it’s good to know that there are other options when required by the business.