Cmdlet Extension Agents Part 2: Postconfiguring Mailboxes


Cmdlet Extension Agents Part 1: Automatic archive creation

Almost a year ago, I posted an article in which I tried to show the power of Cmdlet Extension Agents in Exchange 2010, or more specifically, the Scripting Agent. Unfortunately, the Cmdlet Extension Agents are often overlooked or ignored, despite customers having requirements to customize things immediately after creating a mailbox. Therefor, I decided to write another article on this topic, hoping people take up using Scripting Agents.

Now while you can also put all sorts of post-configuration tasks in provisioning scripts, using the Scripting Agent when possible has a big bonus, because those additional actions not only run when you run the cmdlet directly from the Exchange Management Shell but also when you run them indirectly by using the Exchange Management Console.

So, as this follow up of the previous article, in which I explained what the CmdLet Extension Agents are and how to utilize the Scripting Agent to automate tasks, I’ll show you another example of a Scripting Agent and quickly walk you through it, so you can experiment with it (first in a lab of course) and tune it to your own requirements.

In this example, we’ll disable ActiveSync and configure SingleItemRecovery when creating a new user with a mailbox or mailbox-enabling an existing user. Therefor, the cmdlets we’re going to customize are New-Mailbox and Enable-Mailbox.

Open up Notepad and create a file \bin\CmdletExtensionAgents\ScriptingAgentConfig.xml located in Env:ExchangeInstallPath, e.g. C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents, using the following contents:

Note: If you’ve already got a ScriptingAgentConfig.xml file, you need to integrate the following content.

<?xml version="1.0" encoding="utf-8" ?>
 <Configuration version="1.0">
 <Feature Name="Mailboxes" Cmdlets="New-Mailbox,Enable-Mailbox">
 <ApiCall Name="OnComplete">
   if($succeeded) {
     $Name= $provisioningHandler.UserSpecifiedParameters["Name"]
     Set-Mailbox $Name -SingleItemRecoveryEnabled $true
     Set-CASMailbox $Name -ActiveSyncEnabled $false
   }
 </ApiCall>
 </Feature>
 </Configuration>

As you can see, you’re not limited to 1 action or related cmdlets (*-Mailbox). A small explanation:

  • The Cmdlets specified in this feature extension dictates which cmdlets will be extended, in this case New-Mailbox and Enable-Mailbox;
  • OnComplete dictates that our script will fire when the cmdlet has finished;
  • We check for OnComplete parameter $succeeded, only configuring the mailbox when the preceding events were successful;
  • $provisioningHandler.UserSpecifiedParameters contains user provided parameters passed to the cmdlet. So, $provisioningHandler.UserSpecifiedParameters[“Name”] will return the value of –Name;
  • We set SingleItemRecovery to $true for the mailbox specified by $Name;
  • We disable ActiveSync client access for this mailbox as well.

As mentioned in part 1, distribute this XML file to all your Exchange servers in the local CmdletExtensionAgents folder. When you haven’t already enabled the Scripting Agent, do so by running the following cmdlet:

Enable-CmdletExtensionAgent “Scripting Agent”

Now, when we create a new mailbox or mailbox-enable an existing user:

image

.. you’ll see the SingleItemRecovery has been enabled and ActiveSync has been disabled for this mailbox by the scripting agent:

image

I recommend you start checking out the Scripting Agent if you haven’t already done so. You can use these examples as a starting point and work from there. More information on the Scripting Agent, alternative APIs etc. can be found here.

Exchange can’t start due to misconfigured AD sites


Recently, a customer had issues with their Exchange server which didn’t start properly after rebooting. After checking out the Eventlog, I noticed the it was full of messages, generated by all services. The most interesting events were the ones generated by MSExchange ADAccess:

MSExchange ADAccess, EventID 2141
Process STORE.EXE (PID=2996). Topology discovery failed, error 0x8007077f

MSExchange ADAccess, EventID 2142

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1760). Topology discovery failed, error 0x8007077f

Also, the results of the active directory discovery process generated every 15 minutes, which are normally logging in event 2080, “Exchange Active Directory Provider has discovered the following servers with the following characteristics”, was missing.

Note that because the system could start the Microsoft Exchange Active Directory Topology service (until it failed and is restarted by dependent services), Exchange’s other services were also triggered, leading to almost indefinitely restarting services as configured in their corresponding service recovery actions sections.

Now, since I had connected to a domain controller using an RDP session from my client, and I was able to connect to port 389 (Global Catalog) from Exchange using LDP, so communications looked ok. Then, I switched to Active Directory Sites and Services:

image

As you can see from the shot, here was a potential cause of the problem. First, there was a site without domain controllers. Second, there were no subnets defined. So, in this situation, it is undetermined in which site Exchange is located.

When a system can’t be determined to which site a computer belongs, the function DSGetSiteName, used to retrieve the current site, returns an error 1919 0x77f (ERROR_NO_SITENAME). Consequently, the Exchange Active Directory discovery process fails and eventually Exchange fails. You can inspect the current discovered site using nltest /dsgetsite or by having a peek in the registry at HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\DynamicSiteName.

Now, to solve the situation we have three options:

  1. Making the site association static using a registry key, which isn’t a best practice.If you must, set registry key HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SiteName (REG_SZ) to the desired site name;
  2. Adding proper subnet definitions;
  3. Remove the empty site definition.

It turned out the empty site was a place holder for a future site, so we went with the option of adding proper subnet definitions. After adding subnet definitions, like you normally should when working with multiple sites, including the scopes where the Exchange servers and domain controllers were located, and associating it with the main site, things started working again.

Note that the NetLogon service determines site association membership at startup and every 15 minutes. The Microsoft Exchange Discovery Topology service maintains this information by caching the information in the msExchServerSite attribute of the Exchange server object, in order to reduce load on active directory and DNS. Therefor, you might need to wait or restart Microsoft Exchange Discovery Topology  if you want to renew site association membership.

TechEd North America 2012 sessions


With the TechEd North America 2012 event still running, recordings and slide decks of finished sessions are becoming available online. Here’s an overview of the Exchange-related sessions:


Thoughts on "VMware Zimbra vs Microsoft Exchange"


Note: This blog was written together with Dave Stork after reading a Zimbra and Exchange product comparison. You can find the article on Dave’s blog here, including a personal note by Dave.

In a blog post by Christopher Wells, alias vSamurai, the author positions VMware Zimbra Collaboration Server (ZCS 7.x) as an enterprise-ready drop-in replacement for Microsoft Exchange Server 2010 environments of all sizes. He also suggests Zimbra is a better multi-tenant solution for ISPs. The author does this by comparing both products in a feature comparison.

These reviews are helpful in order for companies to make an informed decision. After all, there’s nothing wrong with a bit of competition. However, Dave Stork and I wanted to create a response, because some statements are flawed or just plain wrong. In the process, we will be following the structure of the referenced blog:

Backup and Restore
The author starts off by claiming that “the ease with which backup and restore can be performed in Zimbra outweighs the capabilities of Exchange”. While it’s interesting to note the author implicitly admits Exchange is more capable, he misses the point. The product should follow a well-designed backup and recovery strategy, based on customer demands and compliance regulations. Where Exchange has server, database, mailbox and single item recovery options, Zimbra is built on top of MySQL, meaning recovery requires brick level restore or (partially) restoring information from MySQL dumps. Also, in Zimbra the databases only contains meta information; the actual messages and attachments are stored on the file system. While this makes sense for Zimbra, as many SQL people consider storing binary data in databases a bad practice, it increases the complexity of backup and restore, because meta information and file system needs to be in sync. Note that Exchange’s Extensible Storage Engine (ESE) is purpose-built for storing mailbox information, including attachments.

Scalability
Then, the author claims that Zimbra has better scaling capabilities than Exchange. First, let’s start by looking at the definition of scaling. A system is said to scale well if:

  • it can handle increased load without (serious) performance penalties, or
  • the system is able to accommodate growth by adding resources (scale up) or additional systems (scale out).

Ideally, scaling up should show a linear pattern, meaning two systems equal can handle twice the load. Scaling out most of the time doesn’t, which makes sense when looking at how computers are designed using shared resources like buses for example.

Now, scaling isn’t solely a matter of hardware; a system also requires software built to scale. The role-based model of Exchange, with its specific roles for serving mailboxes and handling replication, routing e-mail and servicing clients, is a good example of a thought-out scalability supporting concept. Of course, you can install all roles on a single server, which is currently the recommended practice by Microsoft, but you’re still able to design fit-for-purpose farms and clusters.

Thus, the ability to scale is determined by the whole set of components playing well together, hardware and software. With this in mind we’d like to include an interesting table which is part of the VMware (acquired Zimbra early 2010) study “Zimbra Collaboration, Server Performance on VMware vSphere 5.0”:

In their analysis, VMware primarily focuses on the CPU utilization figure. That figure implies that Zimbra has more headroom than Exchange using the same configuration. However, Exchange also has several background processes which perform tasks in the background, like optimizing the database to reduce the number of IOPS. Yes this takes up a certain % of CPU cycles, but optimizing storage for sequential access could explain the significant 240% decrease in IOPS for Exchange. Lower IOPS reduces storage requirements – and costs – for Exchange. The over 60% lower latency figure for Exchange is also an indication overall processing of messages is faster in Exchange.

Costs
As often in these Open Source Software (OSS) discussions, the cost card is played. The author claims that on average, Zimbra is 50% cheaper than Exchange. However, this claim is made without any supporting references or figures, making it difficult to verify this statement. However, from our experiences, those claims are often primarily based on retail prices and licensing costs. What is often overlooked (or ignored) in comparisons with OSS, are training costs or hidden costs like support or maintenance.

Functionality is also a potential cost saver, as companies can work more efficiently due to added or enhanced functionality. These savings depend on customer needs, although some are widely used and immediately contribute to lower costs, like for example AutoDiscover (automatic configuration of Outlook 2007 and later clients or ActiveSync devices).

Exchange natively supports Outlook, common browsers and mobile devices; Zimbra requires an Outlook plug-In, Zimbra Connector for Microsoft Outlook, increasing support and maintenance costs. Note that this connector is only available for Zimbra Collaboration Server Network Edition Professional users.

Regarding maintenance, Exchange requires Exchange, Active Directory and (optionally, but a big bonus) PowerShell skills. Zimbra consists of a set of 3rd party products, requiring knowledge of each product, like Postfix, mbox e-mail storage, MySQL, Apache. OpenLDAP, SpamAssassin, ClamAV and shell scripting. Of course, more components mean more products to configure and maintain, increasing maintenance costs.

Storage Benefits
A full paragraph is dedicated to the benefits of using Zimbra with NetApp storage. However, the NetApp products and technologies mentioned are not Zimbra specific, and therefor in our opinion do not add anything to the discussion.

Feature Comparison
The author then continues with a “direct” feature comparison between Zimbra and Exchange. Let’s have a look:

1. Platform Architecture
First, author claims ESE is over 20 years old, the .EDB file is non-modular and the ESE engine is non-tunable. Yes, ESE exists for over 20 years, but that’s also 20 years of experience in building a fit-for-purpose database engine. With each new Exchange version, ESE was redesigned to meet evolving requirements and expectations in a changing world. When looking at the VMware IOPS comparison in the Scalability section, it’s Zimbra that should worry about storage.

Second, author claims Database Availability Groups (DAGs), based on Fail-over Clustering, isn’t a proven technology for large deployments. Exchange 2010 is on the market since October 2009. Like many Exchange fellows, we have designed or seen large Exchange deployments (i.e. thousands of mailboxes). Also, if millions of Office 365 users aren’t proof of a successful large scale multi-tenant ISP-like deployment based using multiple data center DAGs, what is?

To be honest, is it really that important which exact technology is used and how old it is? In the end functionality and performance are more important, as they are relevant in any business case for Exchange. What would a decision maker most likely ask, “Does it use Microsoft SQL Server?” or “What can we do with it and how much will it cost?”. We think and know out of experience it will probably be the latter.

2. Reliability & Robustness
The author claims Microsoft is considering (moving Exchange storage to) SQL and needs to prove robustness of the new architecture. While Microsoft has considered the SQL storage engine several times, it decided to stick with the optimized ESE engine. This was also true for Exchange 2010 back in 2009, like you can read in this blog. Main reason for deciding to stick with ESE is performance.

When pleading for ZCS, the author states “Linux has better uptime”. While this may have been true in the Windows 98 era, from experience, managed Exchange systems can reach similar uptime figures. On the contrary, I’ve seen Linux systems crashing every few days. The only conclusion you can draw here is that reliability not only depends on hardware and software components and their quality, it also depends a lot on if and how systems are managed. Also, don’t confuse uptime with availability, as planned downtime will reset my uptime statistic, but that’s all it is: a statistic.

3. Tiered Storage (was Platform Scalability)
Tiered storage, or Hierarchical Storage Management, is about classifying data in terms of things like security, performance or pricing. Exchange itself partly supports this concept, using elements like DAGs, databases, mailboxes, personal archives and retention policies. For example, you can home your mailbox on multiple lean and mean servers using fast SAS storage while personal archives, used to automatically store e-mail older than 1 year using retention policies, are served by a fat server using inexpensive SATA disks on JBOD storage.

ZCS utilizes a built-in HSM solution which automatically moves items from the (fast) primary volume to the (cheaper) secondary volume. The database holds information on the actual location where the item resides. Conceptually, this matches the Exchange concept of primary mailbox and personal archive using retention policies. However, retention policies are more powerful and – when permitted – give users control over what to archive and when. When Exchange customers want to use a deeper level of storage tiering, they can opt for 3rd party solutions like Symantec Enterprise Vault (item-level stubbing) or storage solutions.

Note however, there are some important factors to take into consideration with stubbing:

  • Data stored on a different tier, e.g. tape, isn’t always available online;
  • Tiered storage adds complexity, introducing the need to compare reduced costs for storage against additional costs due to increased complexity;
  • Stubbing may impact future migration or transition options, e.g. vendor support, or recovery options.

4. High Availability
Author claims DAGs do not provide Exchange infrastructure protection and have a learning curve. The first part of that claim is absolutely true: DAGs are designed to increase the availability of Exchange databases served by Exchange servers holding the Mailbox role, while providing a fail‑over mechanism. Covering for the other tasks are the other Exchange roles. Mail flow within an Exchange Environment is automatically redundant when you have multiple Hub Transport servers, as they monitor connectivity and possible routes for delivery. For client access, multiple Client Access servers can be made redundant using load balancing technology. Exchange has these built-in features that work independent of where Exchange is running, i.e. they also work in a non-virtualized system and no additional high priced product is required to make the underlying services highly available.

Regarding the learning curve claim, every new technology has a learning curve. DAG is built on top of fail-over clustering (nothing new) and easier to manage than its predecessors, CCR and SCR. Then again, we’d prefer Exchange admins who know what they’re doing, rather than somebody who learned an SRM trick.

Speaking of which, the whole argument that “ZCS with VMware’s Site Recovery Manager (SRM) is proven, scalable and effective” is apparently nothing more than a plug for VMware’s SRM product in conjunction with VMware licenses (vSphere required), as we see no credible arguments.

5. Platform Extensibility
The author states that Microsoft recommends using its proprietary shell. We assume he means PowerShell, which is here to stay. Other vendors, like Cisco or Quest, are adopting it and offer modules to manage their products using PowerShell. Heck, even Zimbra offers PowerShell scripts to manage Zimbra through encapsulated SOAP requests. For the record, we both don’t know of any Exchange admin complaining about some Linux product requiring bash (Bourne-Again shell) or perl for scripting, turning this in a non-argument.

The author continues by apparently mixing a few things up. The argument given for ZCS is that “SOAP API allows server access using web services framework for client access and Zimlets for integration with 3rd-party services” while Exchange offers “limited SOAP access” and “Outlook add-ins require developer effort”. This is apples versus oranges; Outlook is a fat client and Zimlets are like web parts. If you want to make a nice dashboard, we’d suggest you use something like Sharepoint instead of bloating your e-mail web client.

Finally, SOAP and Exchange Web Services (EWS) are targeted at developers, PowerShell at automation. If you’re curious about the power of EWS, we’d suggest you check out the excellent blog by Glen Scales.

6. Platform Openness
While Exchange is mostly closed source, a lot has changed since the 90’s. Exchange has a developer center nowadays, where SDK and APIs are published on how to interact with certain parts of the Exchange ecosystem, e.g.:

7. Open Standard Protocols Support
It’s true that the current Outlook version doesn’t support all available standards for exchanging calendaring or contact information. However, for most companies that isn’t an issue. When required, solutions and workarounds are available.
Also see “Mobile Support”.

8. Rebranding
The author claims Outlook Web Access (OWA) has a single theme. That might have been the case with the RTM version, but since SP1 we have over 28 themes to choose from. If that’s not enough, there’s even an Exchange Server 2010 SP1 Outlook Web App Customization SDK to take customization into your own hands. Note that the SDK also documents integrating IM (e.g. Lync).

9. Web Client Support
Regarding Web Client support, the author states “limited browser support for OWA” (Outlook Web App). Since SP1, OWA has full support for IE7+, Firefox 3.01+ (Windows, MacOS, Linux), Chrome 3.0.195.27+ (Windows), Safari 3.1+ (MacOS). In addition, OWA Mini, targeted at simple mobile browsers, reincarnated in Exchange 2010 SP2.

Yes, there are browsers out there that don’t have the full featured Premium OWA (like Opera), but “limited browser support for OWA” is a bit over-simplified, especially if you take into consideration the combined market shares of the fully supported browsers (without Safari, between 81-91% since December 2011).

10. Mac Support
Outlook team and Mac Outlook are produced by two different teams, which might be one of the reasons for the feature disparity between Outlook 2010 and Outlook for Mac 2011. Apart from differences caused by the underlying operating system, we agree features should be as on par as possible for all available platforms.

Note that the mentioned Zimbra desktop client doesn’t support Exchange’s native MAPI protocol, adding the requirement to enable the IMAP or POP protocol on the Exchange server.

11. Linux
The author proceeds by arguing there’s no Outlook client or Exchange Server for Linux. That is a moot point; there’s also no Zimbra server for Windows. Also, when somebody’s trying to convince you using arguments like, “ZCS server components love the Linux platforms”, that’s not very convincing now, and is often seen with discussions when emotions prevail over rational thinking.

12. Mobile Support
More and more (mobile) clients are adopting the Exchange ActiveSync (EAS) protocol for exchanging e-mail, calendar, contact and task information with Exchange. In fact, even Blackberry announced they will adopt EAS in their upcoming Blackberry 10 OS product. This is probably driven by Microsoft releasing EAS protocol as part of their Open Specifications Promise, turning EAS more or less into the de‑facto standard for (corporate) e-mail synchronization for mobile clients.

Zimbra partially supports EAS for e-mail, calendar and contacts, but requires the Zimbra Mobile add-on. It is a bit unclear if tasks are synced, here it seems so for Pro users but here it is advised against while here the screenshots tell yet another story. Confusing.

13. Multi-tenancy
The author doesn’t show how Zimbra is a better multi-tenancy solution for ISPs when compared to Exchange 2010. But since Exchange 2010 Service Pack 2, there is no need for third party hosting software as it is now fully incorporated in Exchange without extra costs.
However; the intent was possibly to prove this implicitly via the costs argument of on-premises deployments. One other way is to look at actual hosted Zimbra and Exchange solutions available commercially.

Let’s compare costs from random Zimbra providers (picked from Zimbra’s Partners list), Exchange hosting providers and Office 365 subscriptions. It is not an extensive comparison, but it should give us an indication. Some (not all) are shown here:

Product MrMail Professional Zimbra Mailbox CVM Zimbra Professional Suite PayPerCloud Hosted Exchange Professional Office 365 Exchange Online Office 365 Plan E1
Storage 8GB 1GB 25GB 25GB 25GB
(mailbox, sharepoint is separate and additional)
Own mail domain Yes Yes Yes Yes Yes
Attachment size 20MB ? ? 25MB 25MB
Web Access yes yes yes yes yes
POP / IMAP yes/yes yes/yes yes/yes yes/yes yes/yes
ActiveSync yes yes yes yes yes
Antimalware yes yes yes yes yes
SharePoint or similar yes yes no no yes
Lync IM/Presence no no no no yes
Price per user per month $8.61* $7* $7.95** $4** $8**

*) discounts possible with more mailboxes
**) Note that prices are per month, but only apply with an annual subscription.

This table shows that the Exchange subscriptions are comparable or provide more functionality for lower costs. We do not see the 50% cost benefit argument at all and in our opinion shows that Exchange 2010 is a very viable multi-tenancy solution for ISPs.

One very important difference we want to point out is the available storage per mailbox. This tended to be a lot (several factors) more with Exchange than with Zimbra, without heavily impacting the price. This fact alone suggests that Exchange can be a very viable groupware solution to ISPs.

Final words
This concludes the authors’ feature comparison, but there are still some important elements missing, like product support, directory integration, IPv6 readiness, traffic management (e.g. ethical walls) or IRM. Also, what about integration or support of Unified Communications technologies, like single inbox – including voicemail – or voice access to mailbox?

Now don’t get the impression we want to condemn Christopher for trying to compare both products, even though by reading just the header and counting the numerous VMware-related logos on the site we were a bit hesitant regarding what the “conclusion” would be (we have a saying here, We from WC Eend recommend WC Eend).

We do appreciate good comparisons, because it can shake up our opinions of what is and what should be with Exchange and start interesting discussions. It‘s also an opportunity to learn about similar products. We believe competition is healthy and comparisons can be educational; It can help companies make a better fit for their needs and budget, or at least provide a starting point.

It is however crucial for a fair comparison that the facts, conclusions and opinions stated are correct and sound. Unfortunately, this is not the case with this article. There are numorous factual errors and most opinions stated are poorly argumented. To add to that, the author uses a feature list which can be found on the internet in several places, like here. This may be an indication authors are copying content, without knowledge or cross-checking facts.

Therefore, with the information provided in Christophers blogpost, one can’t conclude that Zimbra is an adequate replacement for all environments, Enterprise or SMB. Also, we do not see any indication that Zimbra is better suited for multi-tenancy by ISPs. If anything, we think we have shown that Exchange is a more than capable, competitive and well-though product.

You’re invited to comment or share your opinions in the comments below.

Update (April 10th): Apparently, on March 21st Wells posted a follow up on his Zimbra versus Exchange viewpoint. Looking at it, Wells seems to enjoy the attention.  Despite saying discussing viewpoints keeps vendors’ focus sharp, he doesn’t come up with arguments on why our post was – in Wells’ words – flawed. While I believe Zimbra serves a purpose – and it certainly isn’t on my radar as Wells says – I feel Zimbra or other non-Exchange evangelists should be able to take feedback like a pro. When you ignore other viewpoints or remain silent when asked for arguments, it’s more like a monologue rather than the interaction Wells claimed he’s in favour of.

Finally, our post didn’t go unnoticed, as Tony Redmond referred to it an article on Windows IT Pro. In the article, called Dispelling myths and other half truths, Redmond addresses some of Wells’ flawed claims as well.

Exchange PST Capture Tool released


It took a while, but today the Exchange Team released the long awaited Microsoft Exchange PST Capture Tool (initial version 14.3.16.4). The tool can be used to discover and inject PST files in an Exchange 2010 Exchange Online mailbox or archive.

The tool was originally from Red Gate and known as PST Importer. It’s architecture consists of three components: the central service, (optional) agents for PST discovery, registration and collecting PST files and an administrative console (image by Red Gate):

The online documentation can be found here.

Note that although it’s only supported for Exchange 2010 and Exchange Online, you can use it with Exchange 2007; it’s only untested (and probably unsupported) with that product.

You can read the official announcement here; you can download the tool and the agents here.