Exchange 2013 Schema Version

For planning and validation purposes, Exchange 2013 preparation of the forest and domain results in the following:

  • rangeUpper property of CN=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,<Forest DN> is set to 15137;
  • objectVersion property of cn=<ExOrg>,cn=Microsoft Exchange,cn=Services,cn=Configuration,<Forest DN> is set to 15449;
  • objectVersion property in the Microsoft Exchange System Objects container of <Domain NC> is set to 13236.

The Exchange Schema Versions page has been updated with this information.

Exchange can’t start due to misconfigured AD sites

Recently, a customer had issues with their Exchange server which didn’t start properly after rebooting. After checking out the Eventlog, I noticed the it was full of messages, generated by all services. The most interesting events were the ones generated by MSExchange ADAccess:

MSExchange ADAccess, EventID 2141
Process STORE.EXE (PID=2996). Topology discovery failed, error 0x8007077f

MSExchange ADAccess, EventID 2142

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1760). Topology discovery failed, error 0x8007077f

Also, the results of the active directory discovery process generated every 15 minutes, which are normally logging in event 2080, “Exchange Active Directory Provider has discovered the following servers with the following characteristics”, was missing.

Note that because the system could start the Microsoft Exchange Active Directory Topology service (until it failed and is restarted by dependent services), Exchange’s other services were also triggered, leading to almost indefinitely restarting services as configured in their corresponding service recovery actions sections.

Now, since I had connected to a domain controller using an RDP session from my client, and I was able to connect to port 389 (Global Catalog) from Exchange using LDP, so communications looked ok. Then, I switched to Active Directory Sites and Services:

image

As you can see from the shot, here was a potential cause of the problem. First, there was a site without domain controllers. Second, there were no subnets defined. So, in this situation, it is undetermined in which site Exchange is located.

When a system can’t be determined to which site a computer belongs, the function DSGetSiteName, used to retrieve the current site, returns an error 1919 0x77f (ERROR_NO_SITENAME). Consequently, the Exchange Active Directory discovery process fails and eventually Exchange fails. You can inspect the current discovered site using nltest /dsgetsite or by having a peek in the registry at HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\DynamicSiteName.

Now, to solve the situation we have three options:

  1. Making the site association static using a registry key, which isn’t a best practice.If you must, set registry key HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SiteName (REG_SZ) to the desired site name;
  2. Adding proper subnet definitions;
  3. Remove the empty site definition.

It turned out the empty site was a place holder for a future site, so we went with the option of adding proper subnet definitions. After adding subnet definitions, like you normally should when working with multiple sites, including the scopes where the Exchange servers and domain controllers were located, and associating it with the main site, things started working again.

Note that the NetLogon service determines site association membership at startup and every 15 minutes. The Microsoft Exchange Discovery Topology service maintains this information by caching the information in the msExchServerSite attribute of the Exchange server object, in order to reduce load on active directory and DNS. Therefor, you might need to wait or restart Microsoft Exchange Discovery Topology  if you want to renew site association membership.

Thoughts on "VMware Zimbra vs Microsoft Exchange"

Note: This blog was written together with Dave Stork after reading a Zimbra and Exchange product comparison. You can find the article on Dave’s blog here, including a personal note by Dave.

In a blog post by Christopher Wells, alias vSamurai, the author positions VMware Zimbra Collaboration Server (ZCS 7.x) as an enterprise-ready drop-in replacement for Microsoft Exchange Server 2010 environments of all sizes. He also suggests Zimbra is a better multi-tenant solution for ISPs. The author does this by comparing both products in a feature comparison.

These reviews are helpful in order for companies to make an informed decision. After all, there’s nothing wrong with a bit of competition. However, Dave Stork and I wanted to create a response, because some statements are flawed or just plain wrong. In the process, we will be following the structure of the referenced blog:

Backup and Restore
The author starts off by claiming that “the ease with which backup and restore can be performed in Zimbra outweighs the capabilities of Exchange”. While it’s interesting to note the author implicitly admits Exchange is more capable, he misses the point. The product should follow a well-designed backup and recovery strategy, based on customer demands and compliance regulations. Where Exchange has server, database, mailbox and single item recovery options, Zimbra is built on top of MySQL, meaning recovery requires brick level restore or (partially) restoring information from MySQL dumps. Also, in Zimbra the databases only contains meta information; the actual messages and attachments are stored on the file system. While this makes sense for Zimbra, as many SQL people consider storing binary data in databases a bad practice, it increases the complexity of backup and restore, because meta information and file system needs to be in sync. Note that Exchange’s Extensible Storage Engine (ESE) is purpose-built for storing mailbox information, including attachments.

Scalability
Then, the author claims that Zimbra has better scaling capabilities than Exchange. First, let’s start by looking at the definition of scaling. A system is said to scale well if:

  • it can handle increased load without (serious) performance penalties, or
  • the system is able to accommodate growth by adding resources (scale up) or additional systems (scale out).

Ideally, scaling up should show a linear pattern, meaning two systems equal can handle twice the load. Scaling out most of the time doesn’t, which makes sense when looking at how computers are designed using shared resources like buses for example.

Now, scaling isn’t solely a matter of hardware; a system also requires software built to scale. The role-based model of Exchange, with its specific roles for serving mailboxes and handling replication, routing e-mail and servicing clients, is a good example of a thought-out scalability supporting concept. Of course, you can install all roles on a single server, which is currently the recommended practice by Microsoft, but you’re still able to design fit-for-purpose farms and clusters.

Thus, the ability to scale is determined by the whole set of components playing well together, hardware and software. With this in mind we’d like to include an interesting table which is part of the VMware (acquired Zimbra early 2010) study “Zimbra Collaboration, Server Performance on VMware vSphere 5.0”:

In their analysis, VMware primarily focuses on the CPU utilization figure. That figure implies that Zimbra has more headroom than Exchange using the same configuration. However, Exchange also has several background processes which perform tasks in the background, like optimizing the database to reduce the number of IOPS. Yes this takes up a certain % of CPU cycles, but optimizing storage for sequential access could explain the significant 240% decrease in IOPS for Exchange. Lower IOPS reduces storage requirements – and costs – for Exchange. The over 60% lower latency figure for Exchange is also an indication overall processing of messages is faster in Exchange.

Costs
As often in these Open Source Software (OSS) discussions, the cost card is played. The author claims that on average, Zimbra is 50% cheaper than Exchange. However, this claim is made without any supporting references or figures, making it difficult to verify this statement. However, from our experiences, those claims are often primarily based on retail prices and licensing costs. What is often overlooked (or ignored) in comparisons with OSS, are training costs or hidden costs like support or maintenance.

Functionality is also a potential cost saver, as companies can work more efficiently due to added or enhanced functionality. These savings depend on customer needs, although some are widely used and immediately contribute to lower costs, like for example AutoDiscover (automatic configuration of Outlook 2007 and later clients or ActiveSync devices).

Exchange natively supports Outlook, common browsers and mobile devices; Zimbra requires an Outlook plug-In, Zimbra Connector for Microsoft Outlook, increasing support and maintenance costs. Note that this connector is only available for Zimbra Collaboration Server Network Edition Professional users.

Regarding maintenance, Exchange requires Exchange, Active Directory and (optionally, but a big bonus) PowerShell skills. Zimbra consists of a set of 3rd party products, requiring knowledge of each product, like Postfix, mbox e-mail storage, MySQL, Apache. OpenLDAP, SpamAssassin, ClamAV and shell scripting. Of course, more components mean more products to configure and maintain, increasing maintenance costs.

Storage Benefits
A full paragraph is dedicated to the benefits of using Zimbra with NetApp storage. However, the NetApp products and technologies mentioned are not Zimbra specific, and therefor in our opinion do not add anything to the discussion.

Feature Comparison
The author then continues with a “direct” feature comparison between Zimbra and Exchange. Let’s have a look:

1. Platform Architecture
First, author claims ESE is over 20 years old, the .EDB file is non-modular and the ESE engine is non-tunable. Yes, ESE exists for over 20 years, but that’s also 20 years of experience in building a fit-for-purpose database engine. With each new Exchange version, ESE was redesigned to meet evolving requirements and expectations in a changing world. When looking at the VMware IOPS comparison in the Scalability section, it’s Zimbra that should worry about storage.

Second, author claims Database Availability Groups (DAGs), based on Fail-over Clustering, isn’t a proven technology for large deployments. Exchange 2010 is on the market since October 2009. Like many Exchange fellows, we have designed or seen large Exchange deployments (i.e. thousands of mailboxes). Also, if millions of Office 365 users aren’t proof of a successful large scale multi-tenant ISP-like deployment based using multiple data center DAGs, what is?

To be honest, is it really that important which exact technology is used and how old it is? In the end functionality and performance are more important, as they are relevant in any business case for Exchange. What would a decision maker most likely ask, “Does it use Microsoft SQL Server?” or “What can we do with it and how much will it cost?”. We think and know out of experience it will probably be the latter.

2. Reliability & Robustness
The author claims Microsoft is considering (moving Exchange storage to) SQL and needs to prove robustness of the new architecture. While Microsoft has considered the SQL storage engine several times, it decided to stick with the optimized ESE engine. This was also true for Exchange 2010 back in 2009, like you can read in this blog. Main reason for deciding to stick with ESE is performance.

When pleading for ZCS, the author states “Linux has better uptime”. While this may have been true in the Windows 98 era, from experience, managed Exchange systems can reach similar uptime figures. On the contrary, I’ve seen Linux systems crashing every few days. The only conclusion you can draw here is that reliability not only depends on hardware and software components and their quality, it also depends a lot on if and how systems are managed. Also, don’t confuse uptime with availability, as planned downtime will reset my uptime statistic, but that’s all it is: a statistic.

3. Tiered Storage (was Platform Scalability)
Tiered storage, or Hierarchical Storage Management, is about classifying data in terms of things like security, performance or pricing. Exchange itself partly supports this concept, using elements like DAGs, databases, mailboxes, personal archives and retention policies. For example, you can home your mailbox on multiple lean and mean servers using fast SAS storage while personal archives, used to automatically store e-mail older than 1 year using retention policies, are served by a fat server using inexpensive SATA disks on JBOD storage.

ZCS utilizes a built-in HSM solution which automatically moves items from the (fast) primary volume to the (cheaper) secondary volume. The database holds information on the actual location where the item resides. Conceptually, this matches the Exchange concept of primary mailbox and personal archive using retention policies. However, retention policies are more powerful and – when permitted – give users control over what to archive and when. When Exchange customers want to use a deeper level of storage tiering, they can opt for 3rd party solutions like Symantec Enterprise Vault (item-level stubbing) or storage solutions.

Note however, there are some important factors to take into consideration with stubbing:

  • Data stored on a different tier, e.g. tape, isn’t always available online;
  • Tiered storage adds complexity, introducing the need to compare reduced costs for storage against additional costs due to increased complexity;
  • Stubbing may impact future migration or transition options, e.g. vendor support, or recovery options.

4. High Availability
Author claims DAGs do not provide Exchange infrastructure protection and have a learning curve. The first part of that claim is absolutely true: DAGs are designed to increase the availability of Exchange databases served by Exchange servers holding the Mailbox role, while providing a fail‑over mechanism. Covering for the other tasks are the other Exchange roles. Mail flow within an Exchange Environment is automatically redundant when you have multiple Hub Transport servers, as they monitor connectivity and possible routes for delivery. For client access, multiple Client Access servers can be made redundant using load balancing technology. Exchange has these built-in features that work independent of where Exchange is running, i.e. they also work in a non-virtualized system and no additional high priced product is required to make the underlying services highly available.

Regarding the learning curve claim, every new technology has a learning curve. DAG is built on top of fail-over clustering (nothing new) and easier to manage than its predecessors, CCR and SCR. Then again, we’d prefer Exchange admins who know what they’re doing, rather than somebody who learned an SRM trick.

Speaking of which, the whole argument that “ZCS with VMware’s Site Recovery Manager (SRM) is proven, scalable and effective” is apparently nothing more than a plug for VMware’s SRM product in conjunction with VMware licenses (vSphere required), as we see no credible arguments.

5. Platform Extensibility
The author states that Microsoft recommends using its proprietary shell. We assume he means PowerShell, which is here to stay. Other vendors, like Cisco or Quest, are adopting it and offer modules to manage their products using PowerShell. Heck, even Zimbra offers PowerShell scripts to manage Zimbra through encapsulated SOAP requests. For the record, we both don’t know of any Exchange admin complaining about some Linux product requiring bash (Bourne-Again shell) or perl for scripting, turning this in a non-argument.

The author continues by apparently mixing a few things up. The argument given for ZCS is that “SOAP API allows server access using web services framework for client access and Zimlets for integration with 3rd-party services” while Exchange offers “limited SOAP access” and “Outlook add-ins require developer effort”. This is apples versus oranges; Outlook is a fat client and Zimlets are like web parts. If you want to make a nice dashboard, we’d suggest you use something like Sharepoint instead of bloating your e-mail web client.

Finally, SOAP and Exchange Web Services (EWS) are targeted at developers, PowerShell at automation. If you’re curious about the power of EWS, we’d suggest you check out the excellent blog by Glen Scales.

6. Platform Openness
While Exchange is mostly closed source, a lot has changed since the 90’s. Exchange has a developer center nowadays, where SDK and APIs are published on how to interact with certain parts of the Exchange ecosystem, e.g.:

7. Open Standard Protocols Support
It’s true that the current Outlook version doesn’t support all available standards for exchanging calendaring or contact information. However, for most companies that isn’t an issue. When required, solutions and workarounds are available.
Also see “Mobile Support”.

8. Rebranding
The author claims Outlook Web Access (OWA) has a single theme. That might have been the case with the RTM version, but since SP1 we have over 28 themes to choose from. If that’s not enough, there’s even an Exchange Server 2010 SP1 Outlook Web App Customization SDK to take customization into your own hands. Note that the SDK also documents integrating IM (e.g. Lync).

9. Web Client Support
Regarding Web Client support, the author states “limited browser support for OWA” (Outlook Web App). Since SP1, OWA has full support for IE7+, Firefox 3.01+ (Windows, MacOS, Linux), Chrome 3.0.195.27+ (Windows), Safari 3.1+ (MacOS). In addition, OWA Mini, targeted at simple mobile browsers, reincarnated in Exchange 2010 SP2.

Yes, there are browsers out there that don’t have the full featured Premium OWA (like Opera), but “limited browser support for OWA” is a bit over-simplified, especially if you take into consideration the combined market shares of the fully supported browsers (without Safari, between 81-91% since December 2011).

10. Mac Support
Outlook team and Mac Outlook are produced by two different teams, which might be one of the reasons for the feature disparity between Outlook 2010 and Outlook for Mac 2011. Apart from differences caused by the underlying operating system, we agree features should be as on par as possible for all available platforms.

Note that the mentioned Zimbra desktop client doesn’t support Exchange’s native MAPI protocol, adding the requirement to enable the IMAP or POP protocol on the Exchange server.

11. Linux
The author proceeds by arguing there’s no Outlook client or Exchange Server for Linux. That is a moot point; there’s also no Zimbra server for Windows. Also, when somebody’s trying to convince you using arguments like, “ZCS server components love the Linux platforms”, that’s not very convincing now, and is often seen with discussions when emotions prevail over rational thinking.

12. Mobile Support
More and more (mobile) clients are adopting the Exchange ActiveSync (EAS) protocol for exchanging e-mail, calendar, contact and task information with Exchange. In fact, even Blackberry announced they will adopt EAS in their upcoming Blackberry 10 OS product. This is probably driven by Microsoft releasing EAS protocol as part of their Open Specifications Promise, turning EAS more or less into the de‑facto standard for (corporate) e-mail synchronization for mobile clients.

Zimbra partially supports EAS for e-mail, calendar and contacts, but requires the Zimbra Mobile add-on. It is a bit unclear if tasks are synced, here it seems so for Pro users but here it is advised against while here the screenshots tell yet another story. Confusing.

13. Multi-tenancy
The author doesn’t show how Zimbra is a better multi-tenancy solution for ISPs when compared to Exchange 2010. But since Exchange 2010 Service Pack 2, there is no need for third party hosting software as it is now fully incorporated in Exchange without extra costs.
However; the intent was possibly to prove this implicitly via the costs argument of on-premises deployments. One other way is to look at actual hosted Zimbra and Exchange solutions available commercially.

Let’s compare costs from random Zimbra providers (picked from Zimbra’s Partners list), Exchange hosting providers and Office 365 subscriptions. It is not an extensive comparison, but it should give us an indication. Some (not all) are shown here:

Product MrMail Professional Zimbra Mailbox CVM Zimbra Professional Suite PayPerCloud Hosted Exchange Professional Office 365 Exchange Online Office 365 Plan E1
Storage 8GB 1GB 25GB 25GB 25GB
(mailbox, sharepoint is separate and additional)
Own mail domain Yes Yes Yes Yes Yes
Attachment size 20MB ? ? 25MB 25MB
Web Access yes yes yes yes yes
POP / IMAP yes/yes yes/yes yes/yes yes/yes yes/yes
ActiveSync yes yes yes yes yes
Antimalware yes yes yes yes yes
SharePoint or similar yes yes no no yes
Lync IM/Presence no no no no yes
Price per user per month $8.61* $7* $7.95** $4** $8**

*) discounts possible with more mailboxes
**) Note that prices are per month, but only apply with an annual subscription.

This table shows that the Exchange subscriptions are comparable or provide more functionality for lower costs. We do not see the 50% cost benefit argument at all and in our opinion shows that Exchange 2010 is a very viable multi-tenancy solution for ISPs.

One very important difference we want to point out is the available storage per mailbox. This tended to be a lot (several factors) more with Exchange than with Zimbra, without heavily impacting the price. This fact alone suggests that Exchange can be a very viable groupware solution to ISPs.

Final words
This concludes the authors’ feature comparison, but there are still some important elements missing, like product support, directory integration, IPv6 readiness, traffic management (e.g. ethical walls) or IRM. Also, what about integration or support of Unified Communications technologies, like single inbox – including voicemail – or voice access to mailbox?

Now don’t get the impression we want to condemn Christopher for trying to compare both products, even though by reading just the header and counting the numerous VMware-related logos on the site we were a bit hesitant regarding what the “conclusion” would be (we have a saying here, We from WC Eend recommend WC Eend).

We do appreciate good comparisons, because it can shake up our opinions of what is and what should be with Exchange and start interesting discussions. It‘s also an opportunity to learn about similar products. We believe competition is healthy and comparisons can be educational; It can help companies make a better fit for their needs and budget, or at least provide a starting point.

It is however crucial for a fair comparison that the facts, conclusions and opinions stated are correct and sound. Unfortunately, this is not the case with this article. There are numorous factual errors and most opinions stated are poorly argumented. To add to that, the author uses a feature list which can be found on the internet in several places, like here. This may be an indication authors are copying content, without knowledge or cross-checking facts.

Therefore, with the information provided in Christophers blogpost, one can’t conclude that Zimbra is an adequate replacement for all environments, Enterprise or SMB. Also, we do not see any indication that Zimbra is better suited for multi-tenancy by ISPs. If anything, we think we have shown that Exchange is a more than capable, competitive and well-though product.

You’re invited to comment or share your opinions in the comments below.

Update (April 10th): Apparently, on March 21st Wells posted a follow up on his Zimbra versus Exchange viewpoint. Looking at it, Wells seems to enjoy the attention.  Despite saying discussing viewpoints keeps vendors’ focus sharp, he doesn’t come up with arguments on why our post was – in Wells’ words – flawed. While I believe Zimbra serves a purpose – and it certainly isn’t on my radar as Wells says – I feel Zimbra or other non-Exchange evangelists should be able to take feedback like a pro. When you ignore other viewpoints or remain silent when asked for arguments, it’s more like a monologue rather than the interaction Wells claimed he’s in favour of.

Finally, our post didn’t go unnoticed, as Tony Redmond referred to it an article on Windows IT Pro. In the article, called Dispelling myths and other half truths, Redmond addresses some of Wells’ flawed claims as well.

Retrieving DCs functional capabilities

While constructing a page for the forest and domain functional levels, and the maximum functional level for domain controllers, I wanted to show an example of how to retrieve this Active Directory attribute for all domain controllers. You could for example incorporate this in your automated procedures to check for any Windows 2003 servers and take further actions when needed.

The script is below. It outputs object so you use the pipe for further processing. For information on the possible values for msDS-Behavior-Version check out the new AD Functional Levels page here.

#--------------------------------------------------------------------------------
# Name         : Get-DCMSDSBehaviorVersion.ps1
# Created By   : Michel de Rooij
# E-mail       : michel@eightwone.com
# Date         : 20120307
# Source       : http://eightwone.com
# Version      : 1.0
#
# THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK
# OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE USER.
#--------------------------------------------------------------------------------

$RootDSE= [ADSI]'LDAP://RootDSE'
$objSearchRoot= New-Object DirectoryServices.DirectoryEntry( "LDAP://"+ $RootDSE.configurationNamingContext)
$objSearch= New-Object DirectoryServices.DirectorySearcher
$objSearch.SearchRoot= $objSearchRoot
$objSearch.Filter= "(objectClass=nTDSDSA)"
$objSearch.SearchScope = "SubTree"
$Results= @()
$objSearch.FindAll() | ForEach {
    $objItem = $_.getDirectoryEntry()
    $obj= New-Object PSObject -Property @{
        Servername= $objParent = $objItem.psbase.parent.DNSHostName.ToString()
        MSDSBehaviorVersion= $objItem.Get("MSDS-Behavior-Version")
    }
    $Results+= $obj
}
$Results

Note that if you have over 1000 domain controllers, you need to set the $objSearch.Pagesize, e.g. 1000. It may also encounter problems in forests with multiple roots.

Thoughts on “Five things that annoy me about Exchange 2010″

An article on SearchExchange by Greg Shields, MVP Remote Desktop Services, VMWare vExpert and a well-known writer and speaker, covered Greg’s annoyances with Exchange Server 2010. I doubt these points will be in the top 5 annoyances of the common Exchange 2010 administrator, apart from some arguments being flawed. Here’s why.

Role Bases Access Control Management
First, the article complains about the complexity of Exchange 2010’s Role Based Access Control system, or RBAC for short. It’s clear what RBACs purpose is, managing the security of your Exchange environment using elements like roles, groups, scopes and memberships (for more details, check out one of my earlier posts on RBAC here). RBAC was introduced with Exchange 2010 to provide organizations more granular control when compared to earlier versions of Exchange, where you had manage security using a (limited) set of groups and Active Directory. In smaller organizations the default setup may – with a little modification here and there – suffice.

For other, larger organizations this will enable them to fine-tune the security model to their business demands. And yes, this may get very complex.

Is this annoying? Is it bad that it can’t be managed from the GUI in all its facets? I think not for two reasons. First is that Exchange administrators should familiarize themselves with PowerShell anyway; it is here to stay and the scripting language of choice for recent Microsoft product releases. Second is that, in most cases, setting RBAC up will be a one-time exercise. Thinking it through and setting it up properly is just one of the aspects of configuring the Exchange 2010 environment. Also, I’ve seen pre-Exchange 2010 organizations with delegated permission models that also took a significant amount of time to fully comprehend, beating the authors “20 minute test” easily.

DAGs three server minimum
The article talks about the requirement to put the 3rd copy in a Database Availability Group (DAG) on a “witness server” in a separate site to get the best level of high availability. It looks like the author mixed things up as there’s no such thing as a “witness server” nor a requirement to host anything as such in a separate site. In DAG, there’s a witness share (File Share Witness or FSW for short) which is used to determine majority for DAG configurations with an even number of members. This share is hosted on a member server, preferably as part of the e-mail infrastructure, e.g. a Hub Transport server, located in the same site as the (largest part of) population resides.

Also, there’s no requirement for a 3rd DAG member when high availability is required. The 3rd DAG member is a Microsoft recommendation when using JBOD storage. For disaster recovery a remote 3rd DAG member could make sense, but then you wouldn’t require a “witness server” given the odd number of DAG members.

Note is that there’s a difference between high availability and disaster recovery. Having multiple copies in the same site is to offer high availability; having remote copies is to provide resilience.  More information on DAGs and the role of the File Share Witness in my Datacenter Activation Coordination article here.

Server Virtualization and DAGs
Next, the article continues that Exchange 2010 DAGs don’t support high availability options provided by the virtualization platform, which is spot on. Microsoft and VMWare have been squabbling for some time over DAGs in combination of with VMWare’s HA/DRS options, leading to the mentioned support statement from Microsoft.  VMWare did their part by putting statements like “VMware does not currently support VMware VMotion or VMware DRS for Microsoft Cluster nodes“; what doesn’t help is putting this in the best practices guide as a side note on page 64. More recently, VMWare published a support table for VMWare HA/DRS and Exchange 2010 indicating a “YES” for VMWare HA/DRS in combination with Exchange 2010 DAG. I hope that was a mistake.

In the end, I doubt if DAGs being non-supported in conjunction with VMWare HA/DRS (or similar products from other vendors) will be a potential deal breaker, like the author states. That might be true for organizations already utilizing those options as part of their strategy. In that case it would come down to evaluating running Exchange DAGs without those options (which it happily will). Not only will that offer organizations Exchange’s availability and resilience options with a much greater flexibility and function set than a non-application aware virtualization platform would, it also saves you some bucks in the process as well. For example, where VMWare can recover from data center or server failures, DAGs can also recover from database failures and several forms of corruption.

Exchange 2010 routing
The article then continues with Exchange 2010 following Active Directory sites for routing. While this is true, this isn’t something new. With the arrival Exchange 2007, routing groups and routing group connectors were traded for AD sites to manage routing of messages.

The writers annoyance here is that Exchange must be organized to follow AD site structure. Is that bad? I think not. Of course, with Exchange 2003 organizations could skip defining AD sites so they should (re)think their site structure anyway since more and more products use AD site information. I also think organizations that haven’t designed an appropriate AD site structure following recommendations may have issues bigger than Exchange. In addition, other products like System Center also rely on a prope AD site design.

Also, when required organizations can control message flow in Exchange using hub sites or connector scoping for instance. It is also possible to override site link costs for Exchange. While not all organizations will utilize these settings, they will address most needs for organizations. Also, by being site-aware, Exchange 2010 can offer functionality not found in Exchange 2003, e.g. autodiscover or CAS server/CAS Arrays having site-affinity.

Ultimately, it is possible to set up a separate Exchange forest. By using a separate forest with a different site structure, organizations can isolate directory and Exchange traffic to route it through different channels.

CAS High Availability complicated?
The last annoyance mentioned in the article is about the lack of wizards to configure CAS HA features, e.g. to configure a CAS array with network load balancing like the DAG wizard installs and configures fail-over clustering for you. While true, I don’t see this as an issue. While setting up NLB was not too complex and fit for small businesses, nowadays Microsoft recommends using a hardware load balancer, making NLB of less importance. And while wizards are nice, most steps should be performed as part of a (semi)automated procedure, e.g. reconfiguring after a fail/switch-over. This procedure or script can be tested properly, making it less prone to error.

The article also finds network load balancing and Windows fail-over clustering being mutually exclusive an annoyance. Given that hardware load balancing is recommended and cost effective, supported appliances became available this restriction is becoming a non- issue.

More information on configuring CAS arrays here and details on NLB with clustering here.

Final words
Now don’t get the impression I want to condemn Greg for sharing his annoyances with us. But when reading the article I couldn’t resist responding on some inaccuracies, sharing my views in the process. Most important is that we learn from each other while discussing our perspectives and views on the matter. Having said that, you’re invited to comment or share your opinions in the comments below.

Comparing Active Directory Permissions

Every now and then you might be required to compare Active Directory account permissions. When it concerns one or few accounts, you could do the manual side-by-side comparison using Active Directory and Computers. However, when you need to check multiple accounts this task becomes tedious.

Now you could follow the practice laid out by Exchange fellow Andy Grogan here,  generating permissions output using Quest Active Roles and comparing the textual output with a comparison utility like WinMerge or WinDiff. But you can also perform this comparison using PowerShell’s Compare-Object cmdlet, which I’ll show you here.

For this task we’re going to use the Quest AD extensions (Active Roles), which you can download here. Install these extensions on a domain-joined system where PowerShell is already installed. After installation, start the ActiveRoles Management Shell and enter the following, where IdA and IdB are the Identities of the objects you want to compare:

$a= Get-QadPermission <IdA> -Inherited -SchemaDefault
$b= Get-QadPermission <IdB> -Inherited –SchemaDefault

Now $a and $b contain the permission sets of both objects. Next, we’re going to utilize compare-object to compare these two sets. When we use Compare-Object $a $b you get the following output:

image

Not quite helpful this output but it isn’t unexpected. Since we’re comparing two object sets compare-object generates a result with objects. We can make this more readable by specifying the PassThru parameter so we can post-process these objects, like displaying certain fields using the Format-Table cmdlet, e.g.

Compare-Object $a $b -PassThru | ft SideIndicator,AccountName,Rights,Source,ApplyTo

image

Presto! The SideIndicator  is included to see in which set the attribute is contained, e.g. “<=” means the element is contained in the 1st specified (reference) object and “=>” means its is contained in the 2nd (difference) object.

If you want to include equal objects in the output as well, add the IncludeEqual parameter to the Compare-Object cmdlet.

Active Directory Migration Tool 3.2

At last, the Active Directory Migration Tool (ADMT) 3.2 was released to public. ADMT can be used to migrate and reorganize objects across in inter-forest (cross-forest) or intra-forest scenarios.

The previous version, ADMT 3.1, dates back to July, 2008 and is supported only on Windows Server 2008, but not R2. With the release of R2 this lead to added  complexity because projects needed to introduce a different OS in their environment.

The new 3.2 version of ADMT only runs on Windows Server 2008 R2, but not on earlier versions. As you can also see from the table below, the new ADMT version requires Windows Server 2003 domain functional level in both the source and the target domain:

ADMT
Version
OS
Support
Source Domain
Target Domain
2000 2003 2008 2008
R2
2000 2003 2008 2008
R2
3.1 2008 YES YES YES YES YES YES
3.2 2008 R2 YES YES YES YES YES YES

You can download ADMT 3.2 here. The updated ADMT Guide: Migrating and Restructuring Active Directory Domains can be found here.

Note that ADMT uses Password Export Server (PES) which is currently still at version 3.1. Be advised that this version of PES isn’t supported on Windows Server 2008 R2, so I expect an update for PES as well. PES 3.1 can be downloaded here (x64) or here (x86).

Kerberos Max Token Size

Ok, not directly Exchange related but an issue I’d like to share. In one of my earlier articles you can read I’m working on on a project where we’ll be performing a cross-forest migration of accounts and Exchange mailboxes. Migrating the Active Directory user accounts is done using ADMT v3.1 with SIDHistory. No problem so far, until we noticed some migrated users weren’t receiving Group Policy Objects and experienced authorization errors from time to time. After identifying several users experiencing similar issues, we noticed the following common eventlog entries:

System eventlog (the number 3888 varied):

Event ID : 6
Source : Kerberos
The kerberos SSPI package generated an output token of size 3888 bytes, which was too large to fit in the 2e00 buffer buffer provided by process id 0. If the condition persists, please contact your system administrator.

The Application eventlog contained the following event:

Event ID : 1053
Source : UserEnv
Windows cannot determine the user or computer name. (). Group Policy processing aborted.

Turns out, Kerberos is the culprit. GPO processing aborted because their Kerberos information exceeded the maximum Kerberos token size. This problem may occur when users belong to (too) many groups (.. don’t ask). In addition, memberships coming from SIDhistory are also added to the token, roughly doubling numbers.

As MS KB articles 263693 and 327825 suggest, we raised the MaxTokenSize limited to 65535 (0xFFFF) in the following registry location (if the value is not present, create it as REG_DWORD):

HKLM\SYSTEM\CurrentControlSet\Control\LSA\Kerberos\Parameters\MaxTokenSize

After a restart, all problems were gone. This isn’t a standard GPO setting; when required, you need to create an .adm GPO template yourself which is described in KB article 938118. Hope you’ll find this information useful to keep in mind when performing your ADMT scenarios at clients with excessive group usage.

ForeFront Identity Manager 2010 RTM

By now you’ve probably already heared ForeFront Identity Manager 2010 went RTM on March 2nd. FIM 2010 is the successor to ILM, the Identity Lifecycle Manager. FIM is an solution to manage identities and credentials in heterogeneous environments. It contains functionality for user (de)provisioning, password synchronization, group management, self-service and workflow-like applications. So for instance, FIM can enable organizations to automatically create an Active Directory user with an Exchange mailbox with all the proper settings when a new employee has been entered into the HRM system (or disabled or removed when the employee leaves the organization, depending on requirements).

You can download the trial here. More information on the FIM portal here.

Cross-Forest Mailbox Move (2)

Note: This is part 2; part 1 can be found here.

After the post on experiences regarding Cross-Forest Mailbox Move, the problems with the “sample” Powershell script and the script created in good ol’ VB, I got lots of requests to publish the script. After thinking this over, I made it ready for publishing. That means stripping excessive code and changing domain names etc. to a more descriptive labels.

The script does require some explanation:

  • The script uses 1 input file (users.txt) and produces 1 output file (output.log) (included below);
  • The reason for sending output to screen (optional) as well as file is to be able to check it properly (e.g. using notepad) when running it for a set of users;
  • users.txt contains a single line with the source and target account names. This is the same file we used for ADMT imput. Reason for having a source as well as a new account name is that in the migration process account are renamed. ADMT can do this for you, but the script will need both the old and the new name name to connect to the objects and copy/set several attributes;
  • It’s VB, had been kept simple and didn’t went through a code beautifier. That means no full function headers, input/output descriptions or extensive error handling;
  • Modify the constants using information from your environment, e.g. source/target domain, servers, LegacyExchangeDN etc.
  • The script uses fixed servers. This is to make sure we’re talking to the same server(s) as ADMT and to prevent replication issues because of lag;
  • Use the script at your own risk. I cannot accept any responsibility for consequences when using this in your production environment;
  • Use it in a lab environment first; test, test, test!

Users.txt

SourceName,TargetSAM
jtest,jtest

CrossForestMovePrep.vbs

'*--------------------------------------------------------------------------------
'* Name     : PrepareForestMove
'* Created By      : Michel de Rooij
'* E-mail    : michel@eightwone.com
'* Date            : 20100217
'* Version    : 0.22
'*--------------------------------------------------------------------------------
'* Changes:
'* 0.21 Initial version
'* 0.22 Made changes to address single-value proxyAddress attributes
'*--------------------------------------------------------------------------------

Option Explicit

Const strUserfile            = "users.txt"
Const strOutputFileName            = "output.log"
Const DEBUGOUTPUT            = 1

Const conSourceServer               = "dc.olddomain.nl"
Const conSourceDomain                = "olddomain.nl"
Const contargetServer                = "dc.newdomain.com"
Const conTargetDomain                = "newdomain.com"

Const conLegacyExchangeDN            = "/o=NEWEXORG/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn="
Const conTargetEmailDomain            = "target.com"

' AD putex cmds
Const ADS_PROPERTY_CLEAR = 1
Const ADS_PROPERTY_UPDATE = 2
Const ADS_PROPERTY_APPEND = 3
Const ADS_PROPERTY_DELETE = 4

' FileSystem
Const ForWriting =2
Const ForReading =1

'*********************************************************
' MAIN
'*********************************************************

Dim oFSO, strFile, objFile, hOutputFileHandle, bProcessLine, strLine
Dim arrUsers, strUser, strCmd, strNewUser

Set oFSO = CreateObject("Scripting.FileSystemObject")
Set hOutputFileHandle= oFSO.OpenTextFile( strOutputFileName, ForWriting, True)

debug("Start")

strFile= strUserFile

if NOT oFSO.fileExists( strFile) then
 die( "Input file "& strFile& " does not exist")
end if

debug("Reading names from "& strFile)
set objFile= oFSO.OpenTextFile( strFile, ForReading, True)

while not objFile.atEndOfStream

 bProcessLine= True
 strLine= objFile.readLine

 if isEmpty(strLine) then
 bProcessLine= False
 Else
 If left(strLine, 1)= ";" Then
 bProcessLine= False
 Else
 If inStr( strLine, ",") > 0 Then
 ' Line OK
 arrUsers= split( strLine, ",")
 strUser= arrUsers(0)
 strNewUser= arrUsers(1)

 If strUser= "SourceName" Then
 ' Input file header, skip        
 bProcessLine= False
 End If
 Else
 bProcessLine= False
 debug("** INFO: Skipping line "& strLine)
 End If    
 End If
 End If

 If bProcessLine Then

 debug(strNewUser& ": Syncing Exchange Attributes from "& struser)
 syncAttributes strUser, conSourceServer, conSourceDomain, strNewUser, conTargetServer, conTargetDomain

 End If
Wend

debug("Finished")

objFile.Close
hOutputFileHandle.Close  

set hOutputFileHandle= Nothing
set objFile= Nothing
set oFSO= Nothing

wscript.quit(0)

'*******************************************************************
' Purpose: output to screen when DEBUGOUTPUT is 1, always to file
'*******************************************************************
Function debug(strMsg)
 hOutputFileHandle.write ("["& FormatDateTime(now(),4) & "] "& strMsg & chr(13)& chr(10))
 if DEBUGOUTPUT=1 then
 wscript.echo strMsg
 end if
End Function

'*********************************************************
' Purpose: terminate with message
'*********************************************************
Function die(strMsg)
 wscript.echo strMsg
 wscript.quit (1)
End Function

'*********************************************************
' displayString
' Returns string from varType item/elements
'*********************************************************
Function displayString( varObj)
 Dim tmp, item
 tmp= ""
 select case VarType( varObj)

 Case vbEmpty
 tmp= "(Empty)"
 Case vbNull
 tmp= "(Null)"
 Case vbInteger, vbLong, vbSingle, vbDouble, vbByte, vbDecimal, vbCurrency, vbDate, vbBoolean
 tmp= cStr( varObj)
 Case vbString
 tmp= varObj
 Case vbObject
 tmp= "(Object)"
 Case vbvariant
 tmp= "(Variant)"
 Case 8209
 tmp= "("& OctetToHexStr( varObj)& ")"
 Case vbArray, 8204
 For each item in varObj
 If tmp="" Then
 tmp= tmp+ item
 Else
 tmp= tmp+ ", "+item
 End If
 Next
 tmp= "["& tmp& "]"
 Case Else

 End Select

 displaystring= tmp& " #"& varType( varObj)

End Function

'*********************************************************
' OctetToHexStr
' Convert OctetString (byte array) to Hex string.
'*********************************************************
Function OctetToHexStr (arrbytOctet)
 Dim k
 OctetToHexStr = ""
 For k = 1 To Lenb (arrbytOctet)
 OctetToHexStr = OctetToHexStr & Right("0" & Hex(Ascb(Midb(arrbytOctet, k, 1))), 2)
 Next
End Function

'*********************************************************
' syncAttributes
' migrates attributes from source to target
'*********************************************************
Function syncAttributes (strUser, SourceServer, SourceDomain, strNewUser, TargetServer, TargetDomain)

 dim strDNSource, strDNTarget, objSource, objTarget, n, strMail
 strDNSource= getDN( struser, SourceServer, SourceDomain, "")
 strDNTarget= getDN( strNewuser, TargetServer, TargetDomain, "")
 If strDNSource<>"" AND strDNTarget <> "" Then
 set objSource= getObject( "LDAP://"& SourceServer& "/"& strDNSource)
 set objTarget= getObject( "LDAP://"& TargetServer& "/"& strDNTarget)

 copyAttribute "mail", objSource, objTarget, False
 copyAttribute "mailNickname", objSource, objTarget, False
 copyAttribute "msExchMailboxGuid", objSource, objTarget, False
 setAttribute "targetaddress", objSource.get( "mail"), objTarget
 copyAttribute "proxyAddresses", objSource, objTarget, True
 addAttribute "proxyAddresses", "X500:"& objSource.get( "LegacyExchangeDN"), objTarget

 strMail= objSource.get( "mail")
 n= instr( strMail, "@")
 debug( strMail)
 addAttribute "proxyAddresses", "smtp:"& left( strMail, n-1)& "@"& conTargetEMailDomain, objTarget

 setAttribute "msExchRecipientDisplayType", -2147483642, objTarget
 setAttribute "msExchRecipientTypeDetails", 128, objTarget
 setAttribute "legacyExchangeDN", conLegacyExchangeDN& objSource.get("cn"), objTarget

 objTarget.setInfo

 Else
 debug("*** ERR: Cannot retrieve DNs for Source or Target")
 syncAttributes= False
 End If
End Function

'*********************************************************
' getDN
' Retrieves the DN for a user object
'*********************************************************
Function getDN( struser, strServer, strDomain, strOU)
 dim objConn, objCmd, strQuery, objRS, strAttr, strRDNLDAP, strDNSLDAP
 strRDNLDAP= RDN2LDAPPATH( strOU)
 strDNSLDAP= DNSDomain2LDAPPath( strDomain)
 strAttr= "distinguishedName"
 set objConn= createObject( "ADODB.Connection")
 set objCmd= createObject( "ADODB.Command")
 objConn.Provider= "ADsDSOObject"
 objConn.Open "ADs provider"
 objCmd.ActiveConnection= objConn
 strQuery= "<LDAP://"& strServer
 If strServer <> "" Then
 strQuery= strQuery& "/"
 End If
 strQuery= strQuery& strRDNLDAP
 If strOU <> "" Then
 strQuery= strQuery& ","
 End If
 strQuery= strQuery& strDNSLDAP& ">"
 strQuery= strQuery+ ";(&(objectCategory=person)(objectClass=user)(SAMAccountName="& strUser&"));"& strAttr& ";subtree"
 objCmd.CommandText = strQuery
 on error resume next
 set objRS= objCmd.execute
 if err.number <> 0 Then
 debug( "*** ERR: Error "& err.number& " executing ["& strQuery& "]")
 getDN= ""
 Else
 on error goto 0
 Select Case objRS.recordCount
 Case 0
 debug( "*** ERR: User object "& struser& " not found")
 getDN= ""
 Case 1
 getDN= objRS.Fields( strAttr)
 'debug( getDN)
 Case Else
 debug("*** ERR: Ambigious user object "& struser)
 getDN= ""
 End Select
 End If
 set objRS= Nothing
 set objCmd= Nothing
 set objConn= Nothing
End Function

'*********************************************************
' DNSDomain2LDAPPath( str)
' Makes an LDAP notation for a DNS domain name
' e.g. corp.local => DC=corp,DC=local
'*********************************************************
Function DNSDomain2LDAPPath( str)
 Dim tmp1, tmp2, tmp3
 tmp1= split( str, ".")
 tmp2= ""
 For each tmp3 in tmp1
 If tmp2<>""  then
 tmp2= tmp2& ","
 End If
 tmp2= tmp2& "dc="& tmp3
 Next
 DNSDomain2LDAPPath= tmp2
End Function

'*********************************************************
' RDN2LDAPPath( str)
' Makes an LDAP notation for a Relative Distinguished Name
' e.g. Domain Accounts/3rd party => OU=3rd party,OU=Domain Accounts
'*********************************************************
Function RDN2LDAPPath( str)
 Dim tmp1, tmp2, tmp3
 tmp1= split( str, "/")
 tmp2= ""
 For each tmp3 in tmp1
 If tmp2<>"" then
 tmp2= ","& tmp2
 End If
 tmp2= "ou="& tmp3& tmp2
 Next
 RDN2LDAPPath= tmp2
End Function

'*********************************************************
' copyAttribute
' Copies attribute(s) to target
'*********************************************************
Function copyAttribute( strAttribute, objSource, objTarget, boolMulti)
dim boolUpdate, varItem
If isEmpty( objSource.get( strAttribute)) Then
  debug( strAttribute& " not set, clearing")
  objTarget.PutEx ADS_PROPERTY_CLEAR, strAttribute, 0
Else
  varItem= objSource.get( strAttribute)
  If boolMulti Then
    if isArray( varItem) Then
      debug( "Setting "& strAttribute& " to multi-value "& displayString( varItem))
      objTarget.PutEx ADS_PROPERTY_UPDATE, strAttribute, varItem
    Else
      debug( "Setting "& strAttribute& " to single-value "& displayString( varItem))
      objTarget.PutEx ADS_PROPERTY_UPDATE, strAttribute, array( varItem)
    End If
  Else
    debug( "Setting "& strAttribute& " to "& displayString( varItem))
    objTarget.Put strAttribute, varItem
  End If
End If
on error goto 0
objTarget.SetInfo
End Function

'*********************************************************
' setAttribute
' Sets attribute to target
'*********************************************************
Function setAttribute( strAttribute, strValue, objTarget)
 debug("Setting "& strAttribute& " to "& displayString(strValue))
 objTarget.Put strAttribute, strValue
 objTarget.SetInfo
End Function

'*********************************************************
' addAttribute
' Adds attribute to target
'*********************************************************
Function addAttribute( strAttribute, varAttribute, objTarget)
 dim boolUpdate, tmp
 boolUpdate= True
 If isEmpty( varAttribute) Then
 ' not set, skipping
 Else
 'on error resume next
 If isEmpty( objTarget.get( strAttribute)) Then
 boolUpdate= True
 Else
 If isArray( objTarget.get( strAttribute)) Then
 For each tmp in objTarget.get( strAttribute)
 If tmp = varAttribute Then
 boolUpdate= False
 End If
 Next
 Else
 boolUpdate= varAttribute= objTarget.get( strAttribute)
 End If
 End If
 on error goto 0
 If boolUpdate Then
 debug("Adding "& varAttribute& " to "& strAttribute)
 objTarget.PutEx ADS_PROPERTY_APPEND, strAttribute, array( varAttribute)
 Else
 debug( varAttribute& " already in "& strAttribute)
 End If
 End If
 objTarget.SetInfo
End Function

Output.log (sample)

[16:24] Start
[16:24] Reading names from users.txt
[16:24] jtest: Syncing Exchange Attributes from jtest
[16:24] Setting mail to jtest@source.nl #8
[16:24] Setting mailNickname to jtest #8
[16:24] Setting msExchMailboxGuid to (70C2360FB0330346A925172CA0473B9F) #8209
[16:24] Setting targetaddress to jtest@target.com #8
[16:24] Setting proxyAddresses to [SMTP:jtest@source.nl, X400:c=US;a= ;p=DemoOrg;o=Exchange;s=Old;g=Mr;] #8204
[16:24] Adding X500:/o=DemoOrg/ou=First Administrative Group/cn=Recipients/cn=jtest to proxyAddresses
[16:24] jtest@source.nl.nl
[16:24] Adding smtp:jtest@target.com to proxyAddresses
[16:24] Setting msExchRecipientDisplayType to -2147483642 #3
[16:24] Setting msExchRecipientTypeDetails to 128 #2
[16:24] Setting legacyExchangeDN to /o=NEWORG/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Mr Old #8
[16:24] Finished