Unarchiving Mailbox Items

With the introduction of Exchange 2010 at the end of 2009, a native feature was added to Exchange Server for which organizations required 3rd party products before that. The feature which I am talking about is Exchange’s Personal Archives, Online Archives, or In-Place Archiving as it is called nowadays.

Background
Archives were introduced at a time when Office 365 was in its early days, many organizations were running Exchange on-premises with mailbox quotas as bandwidth and storage were limited or relatively expensive. It was up to end users to make sure their mailbox remained within its limits, either by removing either old items, large items or just move them out of their mailbox to those pesky .PST files.

Archives introduced benefits such as lowering disk footprint by taking infrequently used items out of the primary mailbox (which then could only synchronize in full) to the archive, which is basically an additional mailbox for long-term storage. Exchange’s built-in Messaging Records Management (MRM) through retention policies and tags can be used for automatic moving of older items to the archive.

Archives also come with few downsides, especially in the early days. Most notably are perhaps clients not supporting archives at all, or searches not spanning both mailbox and archive. Also, and this is not to be underestimated, end users do not always grasp the concept of archives and the impact on the tasks and tools they use. It’s not uncommon to see people panicking about “missing data” in service tickets, only to discover their “missing data” was moved to their archive by the company retention policy after some digging.

In recent years, I have seen archives becoming less relevant, and organizations adopting the large mailbox concept in favor of lean and mean mailboxes with archives. There are still exceptions of course, usually in the form of substantial – usually shared – mailboxes. For those, staying with Exchange Online archives – and when needed auto-expanding archives – is usually still an option due to the different type of mailbox interaction, or to circumvent Exchange’s storage limitations or Outlook for Desktop’s synchronizing of offline cache files before issues might be seen. The maximum number of items per folder is such a limit, however these have been raised or done away with in recent years. Non-stubbing 3rd party archive solutions taking data out of Exchange can also be a option.

The Problem
Switching to the large mailbox concept creates a problem for those organizations that have already enabled in-place archives for their end users: How to get that data back from those archives to the primary mailbox. While retention policies can move data in opposite direction, there is no such thing as a reverse-retention policy. Also, not every organization would like to instruct end users to unarchive this contents themselves, as it is prone to failure, blocks Outlook for Desktop from doing anything else and might result in abandoned operations which limits future actions as moves are still happening in the background.

When investigating a possible solution I found that there is no other way to accomplish this, than to programmatically move contents from the in-place archive to the primary mailbox. While there is a ‘archive’ operation for mailbox items (which moves it to the assigned Archive folder, not the in-place archive) there is no other single API call to perform this task. Also, the solution would have to use Exchange Web Services, as a limitation in Microsoft Graph makes it incapable of moving messages between multiple mailboxes.

Note: If I overlooked something in this area, please let me know.

Solution
To help organizations accomplish this task, I wrote a PowerShell script which requires the following:

  • Exchange Server 2013 SP1 or later, or Exchange Online.
  • Exchange Web Services (EWS) Managed API 2.21 or later (how to, NuGet package exchange.webservices.managed.api).
  • When using OAuth, the MSAL library is required (NuGet package Microsoft.Identity.Client). Also, you need to have registered an App in Azure Active Directory; the Tenant ID, Application ID and certificate or secret is what you need to provide the script with to operate successfully.
  • In addition to installing the NuGet packages, you can also store the DLLs in the same folder as the script.

Note: Untested with Primary mailboxes on-premises and Exchange Online Archives.

The script Invoke-Unarchive will perform the following tasks:

  • Invoke-Unarchive will move contents from the in-place archive back to the primary mailbox.
  • The most optimal operation will be chosen:
    • Folders present in archive but not in primary mailbox will be moved in one operation.
    • Folders present in archive and primary mailbox are merged. Items in those folders are moved in batches.
    • The same steps are repeated recursively per folder for the whole archive.
  • If, after moving, a folder in the archive is empty, and it is not a non-removable well-known folder, it will be removed.
  • Optionally, Invoke-Unarchive can also move contents stored in the Recoverable Items from the archive to the primary mailbox.
  • Invoke-Unarchive will handle throttling, either by honoring the returned back-off period or by adding delays between operations.
  • Moving items is asynchronous, and Invoke-Unarchive needs to wait for Exchange to complete the previous move to folder X before it can move the next set of items to folder X.

Do not forget to reassign retention policies causing archival, or you might have the run the script again at later moment.

Syntax
The parameters to call Invoke-Unarchive.ps1 are:

  • Identity to specify one or more mailboxes to unarchive items for.
  • Server to specify the FQDN of the Client Access Server to use. When omitted, Autodiscover will be used.
  • IncludeRecoverableItems to instruct the script to process deletions stored in the Recoverable Items as well.
  • Impersonation to use impersonation when accessing the mailbox. When using modern authentication (OAuth), impersonation is mandatory.
  • Force to force moving of items without prompting.
  • NoProgressBar to prevent progress status.
  • TrustAll to accept all certificates including self-signed certificates.
  • TenantId specifies the ID of the Tenant when using a mailbox hosted in Exchange Online.
  • ClientId to specify the Application ID of the registered application in Azure Active Directory.
  • Credentials to specify the Basic Authentication credentials for on-premises usage or against Exchange Online when OAuth is not an option.
  • CertificateThumbprint is the thumbprint of the certificate to use for OAuth. The certificate with the public key needs to stored with the registered application for authentication. The certificate with the private key should be present in the local certificate store.
  • CertificateFile and CertificatePassword to specify the file of the certificate to use. The file shoud contain the private key, and the password to unlock the file can be specified using CertificatePassword.
  • Secret can be used to specify the secret to authenticate using the registered application.

Note that Credentials, CertificateThumbprint, CertificateFile + CertificatePassword and Secret are mutually exclusive.

Example
Below shows an example run against a test-mailbox using modern authentication (OAuth). The common parameter Verbose is used to display additional output.

.\Invoke-Unarchive.ps1 -Identity michel@myexchangelabs.com -Server outlook.office365.com -Impersonation -Secret <Secret> -TenantId <Tenant> -ClientId <AppId> -Verbose
image

You can find the script on GitHub here.

Final Notes
The EWS operation – especially moving items – is not necessarily slow, but against Exchange Online processing large archives can take considerable amount of time due to throttling. When moving a significant number of items using Outlook for Desktop, you will likely run into Outlook abandoning the operation after which you need to wait for Exchange to finish pending moves before you can continue with this task. Using the script, you can take away this unarchiving task from end users by running the operation in the background in one or multiple runs.

Outlook versions supporting Personal Archives and Retention Policies

Outlook 2013 IconIn an earlier article here, I mentioned which Outlook 2010 versions were eligible for Personal Archives.

Now, with the release of Office 2013, and Outlook 2013 in particular, it’s good to get an update on the Outlook versions and their support of Personal Archives and Retention Policies in Exchange Server 2010 or Exchange Server 2013.

The following Outlook versions support Personal Archives and Retention Policies:

Retail Licenses Volume Licenses
  • Outlook 2013 stand-alone
  • Outlook 2010 stand-alone
  • Outlook 2007 stand-alone
  • Outlook 2007 included with Microsoft Office Ultimate 2007
  • Outlook 2013 stand-alone
  • Outlook 2013 included with Microsoft Office Professional Plus 2013
  • Outlook 2010 stand-alone
  • Outlook 2010 included with Microsoft Office Professional Plus 2010
  • Outlook 2007 stand-alone
  • Outlook 2007 included with Microsoft Office Professional Plus 2007
  • Outlook 2007 included with Microsoft Office Enterprise 2007

Few notes:

  • As mentioned here, Outlook 2007 doesn’t support adjusting retention policies;
  • Personal Archives and Retention Policies require an Microsoft Exchange Server Enterprise Client Access License.

Fix for Outlook 2007 Personal Archive update

As announced early February, the Office team has released a fix for the Outlook 2007 Personal Archive issues. This fix is also integrated in the Office 2007 Cumulative Update for February 2011 (kb2493732).

The description of the Outlook fix included in this CU can be found at kb2475891.Unfortunately, it doesn’t mention any specific Personal Archive issues.

You can request the individual hotfix here.

Outlook 2007 personal archives issue, fix in Feb’11 CU

The Exchange team put a notice up today on their website on an important update for Outlook 2007 regarding personal archive support on Exchange 2010. Unfortunately, the post doesn’t contain any information regarding the issue itself, only that it may result in inaccessible archives. The fix will be included in the February 2011 Cumulative Update for Office 2007, which is to be released later this February.

This Outlook 2007 personal archive support is becoming some story. After the the initial update enabling this functionality, Outlook 2007 Cumulative Update of December (kb2412171), the update was re-released in January to include 3 fixes (see kb2485531). Now we can expect another fix for an undisclosed issue.

I suggest keeping an eye on the Outlook team blog for updates regarding this issue.

Outlook 2007 archive KB published, omits archive function

After releasing the hotfix for enabling Exchange 2010 personal archives for Outlook 2007 users yesterday, the related knowledgebase article was published today (KB2458611).

What you might notice when reading the list of fixed issued is that the article doesn’t mention the included support for archives. Interesting. One might wonder what that implies from a support perspective.

Update: There are reports of issues with Outlook 2007 after installing update KB2412171, like Loss of Archive/Auto-Archive functions (not to be confused with personal archives), performance issues and Outlook Anywhere/NTLM issues.  Since KB2458611 is based on that update, so you might expect similar issues. More information here by Outlook MVP Ben Schorr.

Update: Official information confirms three issues have been identified with this fix:

  1. Secure Password Authentication issues when configured but server doesn’t support it (e.g. GMail);
  2. If you haven’t configured an Exchange account,
    1. Performance issues;
    2. Inability to configure AutoArchive for IMAP/POP3 or Outlook Live accounts.

More information on the Outlook Product Team blog here.

Update: The Microsoft Exchange team blogged about the December 2010 update for Office 2007 here. They state that “This update extends archive support to Outlook 2007”. In the same blog, the archiving functionality I described earlier here is hereby also officially confirmed.

Update: The Outlook team re-released the hotfix which addresses the 3 issues mentioned above (and on which the Outlook team blogged here). You can download the updated KB2412171 hotfix here.

Outlook 2007 hotfix for Exchange 2010 Personal Archive support

Update: After fixing some issues (see here) the Outlook team re-released hotfix KB2412171 and made it available for direct download here or you can get it through Windows Update.

It seems the Office 2007 Cumulative Update for December 2010 (KB2479671) will contain support for the long awaited support for Exchange 2010 Personal Archives in Outlook 2007. Note that support is limited as not all functionality found in Outlook 2010 is present, but it is usable from a customer perspective.

After applying the Outlook 2007 hotfix (SP2 is a prerequisite), archive-enabled users can do the following:

  • Access and move messages to or from their personal archive;
  • Mailboxes for which you have full access permissions will be automatically added to your Outlook view (known as auto mapping of shared mailboxes) inspecting the msExchDelegateListLink attribute (more info on this at Steve Goodman’s blog here);
  • If you have full access permissions to an archive-enabled mailbox, that archive will be accessible and automatically displayed as well.
    In the screenshot below you’ll the the user’s mailbox (Francis Blake), his Personal Archive and he has full access permissions to Philip Mortimer’s mailbox, who is also archive-enabled:

image

Unfortunately, there are also a few things not implemented, e.g.

  • No support for archiving policies, meaning users won’t be able to override centrally configured retention and archiving policy settings using personal tags in Outlook. More info on archiving policies and personal tags here;
  • Not integral searches, meaning you need to search the mailbox or the personal archive seperately. Searching “all mail items” will not retrieve results from the personal archive, as shown in the screenshot below:

image

It is unknown if these omissions are going to be introduced in a later hotfix, as the initial hotfix for personal archives in Outlook 2007 was set for Q1/Q2 2011.

Be advised that this hotfix will enable personal archive functionality for the following product versions only:

  • Office Ultimate (Retail)
  • Office ProPlus (Volume License)
  • Office Enterprise (Volume License)
  • Outlook Standalone (Retail or Volume License)

Note that the Outlook 2007 update for archive support is still not mentioned on the cumulative update page as the cumulative update package has yet to be released. However, the KB article has been published (KB2458611). You can request and retrieve the individual hotfix here.

Thanks for Ankur Kothari (@ankbomb) from product management for the early update.

Geek Out with Perry series

As EHLO posted today, there is a new video of the Perry Clark, General Manager, Exchange from the Mailbox team. Its the latest addition to the Geek Out with Perry series, short (around) 10 minute videos hosted by Ann Vu (Product Manager), where various in-depth topics like Archiving, Storage and Data Protection are discussed. Perhaps not all of you are aware of the series, so for your convenience I’ve embedded the videos:

Answers to Questions on Archiving with Tiered Storage and Stubbing

Archiving in Exchange

Answering Data Protection Questions: Replication and Backups

Managing Storage Efficiently

Data Protection Evolution

For those interested, Perry blogs and videos can be found on Ask Perry.