Exchange 2010 RTM EOL’s on October 11th

After returning from holiday, between all the Build Windows (Windows 8 ) news, a quick heads-up for those with lagging upgrade schemes or any other valid reason to be still running Exchange 2010 RTM. On October 11th, 2011, support for Exchange 2010 RTM will end.

This should be of no surprise when you practice proper lifecycle management or track Microsoft’s KB bulletins as this information was published on the lifecycle page as well as knowledge base article KB2615653.

For those doing fresh installs and still wondering if this affects their process of installing SP1 versions starting by using the RTM files; since Exchange 2007, Service Packs for Exchange contain all binaries enabling you to perform a fresh installation as well as an upgrade using the same set of files.

You can download Exchange 2010 Service Pack 1 here.

Cmdlet Extension Agents Part 1: Automatic archive creation

Cmdlet Extension Agents Part 2: Postconfiguring Mailboxes

An Exchange fellow inquired about the possibility to automatically enable personal archives when creating mailboxes with the added requirement to create those personal archives in a specific mailbox database, depending on the location of the mailbox. Dedicated mailbox databases were used for personal archives. Simply said, the idea was that mailboxes located in database MDB1 should get a personal archive in mailbox database ADB1, MDB2 in ADB2, etc.

Your first thought could be creating a script to automatically provision those personal archives in the proper database depending on the mailbox database. But alas, when using Exchange 2010’s automatic mailbox provisioning system you never know upfront what mailbox database will be appointed.

That leads us to Exchange 2010’s Cmdlet Extension Agents, more specific the Scripting Agent. I won’t go into much detail now on those Agents, but look at them as a way to extend cmdlets by adding pre- and post-jobs, additional constraints, reporting or override parameters.

Now, when you haven’t already done so, first exclude the mailbox databases containing personal archives from automatic provisioning. If you have a dedicated server for hosting personal archives, use the IsExcludedFromProvisioning with the Set-MailboxServer cmdlet; to exclude a mailbox database use IsExcludedFromProvisioning with the Set-MailboxDatabase, e.g.

Set-MailboxDatabase <Archive Database ID> –IsExcludedFromProvisioning $true

image

I’ll first show you how the scripted version could work. We’ll start by creating some mailboxes. We don’t require anything fancy, so this will do:

$pwd= ConvertTo-SecureString -AsPlainText “Welcome1 -Force
1..10 | ForEach { New-Mailbox “User$_ -Password $pwd -UserPrincipalName user$_@<domain> }

A quick overview of the result shows the mailboxes are created in a round robin fashion:

image

What you could do now is enabling the archive on ADB1 for MDB1 and ADB2 for MDB2 mailboxes, e.g.

Get-Mailbox –Database MDB1 | Enable-Mailbox –Archive -ArchiveDatabase ADB1
Get-Mailbox –Database MDB2 | Enable-Mailbox –Archive -ArchiveDatabase ADB2

image

This is what we wanted. As you probably understand, the main disadvantage now is that this only works for the current mailbox population. Administrators should appoint the proper mailbox database for personal archives when creating new mailboxes. Can the Scripting Agent overcome this problem?

Let’s have a look on how to configure the Scripting Agent. Open up Notepad and create a file \bin\CmdletExtensionAgents\ScriptingAgentConfig.xml located in Env:ExchangeInstallPath, e.g. C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents, using the following contents:

<?xml version="1.0" encoding="utf-8" ?>
  <Configuration version="1.0">
  <Feature Name="MailboxProvisioning" Cmdlets="New-Mailbox">
  <ApiCall Name="OnComplete">
  If($succeeded) {
    $Name= $provisioningHandler.UserSpecifiedParameters["Name"]
    If ((Get-Mailbox $Name).ArchiveDatabase -eq $null) {
      $MailboxDatabase= (Get-Mailbox $Name).Database
      $ArchiveDatabase= "A"+ ( $MailboxDatabase.Name).Substring( 1)
      Enable-Mailbox $Name -Archive -ArchiveDatabase $ArchiveDatabase
    }
  }
  </ApiCall>
  </Feature>
  </Configuration>

A small explanation might be appropriate:

  • The Cmdlets specified in this feature extension dictates which cmdlets will be extended;
  • OnComplete dictates that our script will fire when the cmdlet has finished;
  • We check for OnComplete parameter $succeeded, only enabling archives when the preceding cmdlet was successful;
  • $provisioningHandler.UserSpecifiedParameters contains user provided parameters passed to the cmdlet. So, $provisioningHandler.UserSpecifiedParameters[“Name”] will return the value of -Name;
  • We’ll check if the mailbox already has a personal archive configured; if not, we can proceed;
  • Next, we’ll get the current MailboxDatabase. Then we’ll map that to our personal archive naming scheme by stripping the first character and prefix it with “A”;
  • Finally, we can execute the cmdlet to enable the personal archive of the mailbox on the database specified.

Now, before we test our scripting agent, we need to distribute the XML file on all of our Exchange servers. The reason for this is that you don’t know which Exchange server an administrator will connect to or which server will execute the cmdlet. The location to copy the XML file to is the local CmdletExtensionAgents folder.

Now there’s one more thing we need to do, which is enabling the Scripting Agent. The Scripting Agent is disabled by default. Use the Enable-CmdletExtensionAgent cmdlet to enable it, e.g.:

Enable-CmdletExtensionAgent “Scripting Agent”

Now, when we use the same cmdlet we used before to create those mailboxes, we get the following result:

image

As you can see, archive databases are now nicely aligned with the automatically assigned mailbox databases.

A small note for those wishing to experiment with the Scripting Agent. Alternatively to OnComplete, you can also try defining the personal archive parameters using the ApiCall ProvisionDefaultProperties. This ApiCall can be used to define default attributes when creating a mailbox. However, this leads to a catch 22 situation and has to do with the Mailbox Resources Management Agent.

By default the Mailbox Resources Management Agent has higher priority (2) than the Scripting Agent (6). This means it will override any settings made in our Scripting Agent.

image

The Mailbox Resources Management Agent is responsible for the automatic mailbox distribution when you don’t specify a mailbox database when creating a mailbox. But it is also responsible for assigning a mailbox database for the personal archive when you don’t specify the ArchiveDatabase parameter.

So, unless we want to add all the automatic mailbox distribution logic to our script, we can’t use the ProvisionDefaultProperties ApiCall properly, because if we want to use that, we need to assign the Scripting Agent a higher priority than the Mailbox Resources Management Agent, but at that point we have no database value so we can’t determine the proper archive database.

If you’re interested in playing with this, check out the ScriptingAgentConfig.xml.sample file which is located in the CmdletExtensionAgents as well. If you’re looking for more information on Cmdlet Extension Agents, check here; information on the Scripting Agent can be found here. More information on the automatic mailbox distribution process here.

Updating Exchange 2010 DAG Members

With all the (re-)releases of rollups, the question might rise on how to perform a proper up or downgrade of all DAG configuration members.

Basically, the procedure is straightforward and should be followed per DAG member:

  1. Appoint (next) DAG member;
  2. Move away all active copies on that DAG member;
  3. Prevent copies from activating on that DAG member;
  4. Perform maintenance, e.g. down or upgrade DAG member;
  5. Enable possible activation on that DAG member again;
  6. Optionally redistribute database copies.

Note that in a DAG configuration with 2 members, you need to be aware that during maintenance you have a temporary situation with no fail-over options. If that’s undesirable, consider implementing a 3rd DAG member.

To make the above procedure  easier and automated regarding moves and activation (un)blocking, additional scripts are available since SP1 for Exchange 2010. These scripts are located in the Scripts folder, below the Exchange installation folder. By default the location of the scripts will be C:\Program Files\Microsoft\Exchange Server\v14\Scripts.

Utilizing them, the procedure is quite easy as you can see below. Note that the example uses a DAG named DAG1 with nodes ex2010a and ex2010b as members. They both host 2 databases, ex2010mdb1 and ex2010mdb2; both host 1 active copy and a passive copy of the other database.

  1. Appoint (next) DAG member, e.g. ex2010a;
  2. Run StartDagServerMaintenance.ps1 targeting that DAG member, e.g.:
    .\StartDagServerMaintenance.ps1 –server ex2010a

    image
  3. Perform maintenance;
  4. Run StopDagServerMaintenance.ps1 targeting that DAG member, e.g.:
    .\StopDagServerMaintenance.ps1 –server ex2010a
  5. Repeat steps 2-3 for the other DAG member(s):image
  6. Optionally run RedistributeActiveDatabases.ps1 for the DAG, e.g.:
    .\RedistributeActiveDatabases.ps1 –DagName DAG1 –BalanceDBsByActivationPreference –Confirm:$false

    image

Be advised that when upgrading on major levels (RTM to SP1 or SP1 to SP2), you can’t move a database to a lower level host. This means that when upgrading a node from SP1 to SP2 and moving a database to that SP2 node in the process, you can’t move that database to any SP1 nodes in the DAG. Keep this in mind when planning your upgrade, because it will impact the availability level by limiting your fallback options, albeit temporarily.

A Decade in High Availability

A recent post from Elden Christensen, Sr. Program Manager Lead for Clustering & High Availability, reminded me of one of my former employers. When I joined that company back in 2000 for starting up a professional services based on Windows Server 2000 Data Center Edition, the company was already an established professional services provider in the business critical computing niche market, e.g. Tandem/Compaq/HP NonStop systems, mostly used in the financial markets, e.g. banks or stock exchanges. The Windows platform was regarded as inferior at that time by the NonStop folks and they had good arguments back then.

Remember, those were also the early days where no one was surprised to see an occasional blue screen (people were also using Windows 9x) and what we now know as virtualization was already happening on mainframes in the form of partitioning. At that time, Microsoft with their Windows Server platform had ambitions to enter the data center environment, where the NonStop platform was an established platform for ages and professionals had developed best practices for those environments.

Another part of the discussion was the Fault-Tolerance  versus High Availability topic, where NonStop was already an established Fault Tolerant solution for business critical environments, Windows still had only ambitions to move towards that market with the Data Center product. A logical move, looking at the status of (web) applications, SQL and last but not least, Exchange and where it was going and what customers expected of those products regarding availability and reliability. To repeat an infamous quote of a NonStop colleague back then, “E-mail is not business critical”. But that was almost 10 years ago, things have changed .. or haven’t they?

Single Point of Failure
First I’ll start by introducing the availability concept, which revolves around eliminating the single point of failure. This is an element in the whole system of hardware, software and organization that can cause downtime for a system, i.e. disruption of services. After identifying a single point of failure, we want to eliminate it to prevent downtime which is, after all, the ultimate goal for a business critical system. We can approach this task using two different strategies, Fault Tolerant (FT) or High Availability (HA). The task of identifying and eliminating single points of failure is an ongoing process, as most IT environments are subject to change over time.

Availability
To understand the Fault Tolerant and High Availability strategies we need to define the term “Availability”. In the dictionary, availability is defined as the quality or state of being available or an available person or thing, where in both cases available means present or ready for immediate use. The availability is mostly expressed as a percentage, for example when used in a service level agreement, but what does that percentage mean? To explain this take a look at the following diagram:

Lifecycle

I assume this lifecycle speaks for itself. Using this diagram, the availability is calculated as follows: MTBF / (MTBF + MTTR). The related expected downtime is calculated as ( 1 – Availability% ) * 1 year. Note that the time between failure and recovery isn’t used in the calculation.

I’ll use a simple example, a 500 GB Seagate Barracuda 7200.12 (ST3500412AS) with a MTBF specification of 750,000 hours. You have a 24 hours replacement contract and need about 4 hours to restore the backup. The availability would then be 750,000 / ( 750,000 + 28 ) = 0.9999626680% resulting in a yearly downtime of ( 1 – 0.9999626680) * ( 365 days  * 24 hours * 60 minutes ) = 19,6 min.

Of course with hardware these numbers are theoretical and to some extent a marketing thing; how else can Seagate specify an MTBF of 750,000 hours ( 85 years ). I tend to look at it as an indication of the reliability you can expect. For example, compare the MTBF of 7200.12 drive with an enterprise class drive, Seagate’s ES product line. The ST3500320NS has an MTBF of 1,200,000 hours.

That’s the reason you should use enterprise class drives in your storage solution instead of desktop drives, which aren’t supposed to run in 24×7 environments. To add to that, the MTBF decreases when used in series (RAID 0 = 1 / (1/MTBF1 + .. 1/MTBFn)) or increases when used in parallel (RAID 1 = MTBF * ( 1 + 1/2 + 1/n) ) configurations. When trying to do calculations for the whole supply chain, with all the elements and their individual specifications and support contracts, this can get very complex.

The 9’s
imageWhen talking about availability this is often shown using a series of 9’s, e.g. 99.9%. The more 9’s it has, the better (less downtime). Note that for each increased level of availability, the required effort increases significantly. By effort, don’t think of technical solutions only. It also means organizational measures like having skilled personnel and proper procedures.

A fact is that only a small percentage of the causes for outage is technical, the majority of incidents is due to human error. And yes, that includes that bad driver which is programmed by humans. This is why changes in properly managed infrastructure should always go through test and acceptance procedures in environments representative or identical to the production environment. Unfortunately, this doesn’t always happen as not all IT departments have this luxury, mostly because of financial reasons.

Availability% Downtime / Year Downtime / Month
99.0% 3.65 days 7.3 hrs
99.9% 8.76 hrs 43.8 min
99.99% 52 min 4.3 min
99.999% 5.2 min 26 sec
99.9999% 31 sec 2.6 sec

Fault Tolerant
imageThe goal of a Fault Tolerant solution is to maximize the Mean Time Between Failure (MTBF). This is achieved by mirroring or replicating systems. These monolithic systems run software in parallel on identical hardware. This is called Lockstep (which, for your information, refers to synchronized marching).

Because Fault Tolerant systems run in parallel, the results of an operation can be compared. When the results don’t match, a fault occurs. Since the faulty system can’t be identified using 2 parallel systems, there’s also a variation to this architecture where one server functions as master and one as slave, the slave functioning as a hot-standby. To solve the ambiguity, you could use three systems where the majority of the systems determine the right output.

When faults are detected in a Fault Tolerant system, the failing component (or system) is disabled and the mirror takes over. This makes the experience transparent for the end-user. There is one caveat: since Fault Tolerant systems run software in parallel, software faults are also mirrored.

Examples of Fault Tolerant components are ECC RAM, multiple NICs in Fault Tolerant configuration, multipath network software, RAID 1+ disk systems or storage with replication technology. Examples of Fault Tolerant systems are HP NonStop (propriety), Stratus ftServer or Unisys ES7000. There are also software-based solutions like Marathon EverRun or VMWare’s FT offering.

High Availability
High Availability aims to maximize minimize the MTTR. This can be achieved by redundant or standby (cold, hot) systems or non-technical measures like on-site support contracts. Systems take over the functionality of the failing system after the failure occurred. Therefor, High Availability solutions aren’t always completely transparent for the user. The effects of a failing system and the consequences for end end-user depend on the software, e.g. a seamless reconnect or requirement to login again. Another point of attention is the potential loss of information caused by pending transactions being lost because of the failure. To make the experience more transparent for the user, application need to be resilient, e.g. detecting failure and retrying the transaction.

Examples of High Availability technologies are load balancing – software or hardware-based – and replication, where load balancing is used for static data and replication for dynamic data.

The Present
After a decade, technology has evolved but is still founded on old concepts. Network load balancing is still here and clustering (anyone remember Wolfpack?), although we moved from shared nothing to to replication technology, remains largely unchanged. This means either there hasn’t been much innovation or the technologies do a decent job; After all, it’s still a matter of demand and supply. Yes, we moved from certified configurations-only shared storage solutions to flexible Database Availability Groups (hey, this is still and Exchange blog), but most changes are in the added functionality category or to take away constraints, e.g. cluster modes (majority node set, etc.), multiple replicas and configurable replication.

Windows Server
Data Center Edition
x86 x64
2000 Max. 32 GB
32 CPUs
4 nodes
N/A
2003 SP2 Max. 128 GB
32 CPUs
8 nodes
Max. 512 GB
64 CPUs
8 nodes
2003 R2 SP2 Max. 64 GB
32 CPUs
8 nodes
Max. 2 TB
64 CPUs
8 nodes
2008 Max.  64 GB
32 CPUs
16 nodes
Max. 2 TB
64 CPUs
16 nodes
2008 R2 N/A Max. 2 TB
64 CPUs (256 logical)
16 nodes

What about Fault Tolerance and Windows’ Data Center Edition as the panacee for all your customers requiring “maximum uptime”? The issue with Fault Tolerant was that it came with a hefty price tag, especially in those days. Costs were an x-fold of the costs involved with High Availability solutions on decent (read: stable) hardware. So, for those extra 9’s you needed deep pockets. For example, around 2001 an Compaq ES7000 with Windows Server 2000 Data Center Edition, the joint-support queue (e.g. Microsoft and OEM) and services came with a $2m price tag for which you got the promise of 99,9% availability.

Compare that to buying a few Proliant’s with Windows Server 2000 Advanced Server, some Fault Tolerant components (FT NICs, RAID), off the shelf High Available technology and dedicated personnel (justifiable with that DCE price tag) for .. say, $250,000. With skilled personnel and operated in a controlled environment you could easily reach 99% availability. Is that price difference worth 3 days of downtime? Also, the simplicity to implement those technologies made High Availability in Windows accessible for the masses and now – certainly in the Exchange world – seldom see load balancing or forms of clustering not being utilized.

Note that in the past decade, I’ve never encountered Data Center for hosting Exchange. In fact, as of Exchange 2003, support for on Data Center was dropped. Nowadays, Data Center is regarded as an attractive option for large-scale virtualizations based on Hyper-V, not only because Data Center costs less than back then (about $3000 per CPU – hurray for multi core, but with a 2 CPU minimum) and runs certified on more hardware, but also because it comes with unlimited virtualization rights, meaning you may run Windows Server 2008 R2 (or previous version) Standard, Enterprise, and Datacenter in the virtual instances without the need to purchase additional licenses for those.

With all the large-scale virtualization and consolidation projects going on, virtualizing Exchange or other parts of your IT infrastructure, it’s good to know that there are other options when required by the business.

Connecting the Disconnected in Exchange 2010

My article for Simple-Talk on mailbox and personal archive deletion, recovery and purging in Exchange 2010 has been published on Simple Talk.

The article is about:

  • The relationships between Active Directory and Mailboxes and Archives in the store;
  • Mailbox Retention;
  • Deleting and Recovering a Mailbox
  • Deleting and Recovering a Personal Archive
  • Linked Mailboxes
  • Cleanup

You can check out the article, titled Connecting the Disconnected in Exchange 2010, at:
http://www.simple-talk.com/sysadmin/exchange/connecting-the-disconnected-in-exchange-2010/

Note: I submitted the following additional SP1 info to the publisher so this information might be included in the article at some point.

Since writing the article SP1 was released which contains a new cmdlet to purge mailboxes, Remove-StoreMailbox. This is also great for situations when you’ve moved a lot of mailboxes and you need to make that space available in the source database (when moved, mailboxes are soft-deleted). Usage is as follows:

Remove-StoreMailbox -Database <DatabaseID> -Identity <MailboxID> -MailboxState [Disabled|SoftDeleted]

For example, to remove pmortimer’s mailbox from database DB1 after moving it elsewhere, use:

Remove-StoreMailbox –database DB1 –identity pmortimer –MailboxState SoftDeleted

The command to remove all SoftDeleted mailboxes from database DB1 would be:

Get-MailboxStatistics -Database DB1 | where {$_.DisconnectReason -eq “SoftDeleted”} | foreach {Remove-StoreMailbox -Database $_.database -Identity $_.mailboxguid -MailboxState SoftDeleted}

The new Remove-StoreMailbox cmdlet also enables us to use single command for removing all disconnected mailboxes:

Get-MailboxStatistics –Database DB1 | Where-Object {$_.DisconnectReason –eq “Disabled”} | ForEach {Remove-StoreMailbox –Database $_.database –identity $_.mailboxguid –MailboxState Disabled

Remote Desktop Connection Manager

A few days ago, Microsoft released an installable Remote Desktop Connection Manager, or RDCMan for short. RDCMan (v2.2) looks like the MMC version of Remote Desktops, but contains additional functionality like grouping and export/import. Great for people working in lab or demo environments where you have to switch back and forth  – and manage – a lot of remote sessions.

You can download Remote Desktop Connection Manager here.