Unarchiving Mailbox Items

Posted on by

With the introduction of Exchange 2010 at the end of 2009, a native feature was added to Exchange Server for which organizations required 3rd party products before that. The feature which I am talking about is Exchange’s Personal Archives, Online Archives, or In-Place Archiving as it is called nowadays.

Background
Archives were introduced at a time when Office 365 was in its early days, many organizations were running Exchange on-premises with mailbox quotas as bandwidth and storage were limited or relatively expensive. It was up to end users to make sure their mailbox remained within its limits, either by removing either old items, large items or just move them out of their mailbox to those pesky .PST files.

Archives introduced benefits such as lowering disk footprint by taking infrequently used items out of the primary mailbox (which then could only synchronize in full) to the archive, which is basically an additional mailbox for long-term storage. Exchange’s built-in Messaging Records Management (MRM) through retention policies and tags can be used for automatic moving of older items to the archive.

Archives also come with few downsides, especially in the early days. Most notably are perhaps clients not supporting archives at all, or searches not spanning both mailbox and archive. Also, and this is not to be underestimated, end users do not always grasp the concept of archives and the impact on the tasks and tools they use. It’s not uncommon to see people panicking about “missing data” in service tickets, only to discover their “missing data” was moved to their archive by the company retention policy after some digging.

In recent years, I have seen archives becoming less relevant, and organizations adopting the large mailbox concept in favor of lean and mean mailboxes with archives. There are still exceptions of course, usually in the form of substantial – usually shared – mailboxes. For those, staying with Exchange Online archives – and when needed auto-expanding archives – is usually still an option due to the different type of mailbox interaction, or to circumvent Exchange’s storage limitations or Outlook for Desktop’s synchronizing of offline cache files before issues might be seen. The maximum number of items per folder is such a limit, however these have been raised or done away with in recent years. Non-stubbing 3rd party archive solutions taking data out of Exchange can also be a option.

The Problem
Switching to the large mailbox concept creates a problem for those organizations that have already enabled in-place archives for their end users: How to get that data back from those archives to the primary mailbox. While retention policies can move data in opposite direction, there is no such thing as a reverse-retention policy. Also, not every organization would like to instruct end users to unarchive this contents themselves, as it is prone to failure, blocks Outlook for Desktop from doing anything else and might result in abandoned operations which limits future actions as moves are still happening in the background.

When investigating a possible solution I found that there is no other way to accomplish this, than to programmatically move contents from the in-place archive to the primary mailbox. While there is a ‘archive’ operation for mailbox items (which moves it to the assigned Archive folder, not the in-place archive) there is no other single API call to perform this task. Also, the solution would have to use Exchange Web Services, as a limitation in Microsoft Graph makes it incapable of moving messages between multiple mailboxes.

Note: If I overlooked something in this area, please let me know.

Solution
To help organizations accomplish this task, I wrote a PowerShell script which requires the following:

  • Exchange Server 2013 SP1 or later, or Exchange Online.
  • Exchange Web Services (EWS) Managed API 2.21 or later (how to, NuGet package exchange.webservices.managed.api).
  • When using OAuth, the MSAL library is required (NuGet package Microsoft.Identity.Client). Also, you need to have registered an App in Azure Active Directory; the Tenant ID, Application ID and certificate or secret is what you need to provide the script with to operate successfully.
  • In addition to installing the NuGet packages, you can also store the DLLs in the same folder as the script.

Note: Untested with Primary mailboxes on-premises and Exchange Online Archives.

The script Invoke-Unarchive will perform the following tasks:

  • Invoke-Unarchive will move contents from the in-place archive back to the primary mailbox.
  • The most optimal operation will be chosen:
    • Folders present in archive but not in primary mailbox will be moved in one operation.
    • Folders present in archive and primary mailbox are merged. Items in those folders are moved in batches.
    • The same steps are repeated recursively per folder for the whole archive.
  • If, after moving, a folder in the archive is empty, and it is not a non-removable well-known folder, it will be removed.
  • Optionally, Invoke-Unarchive can also move contents stored in the Recoverable Items from the archive to the primary mailbox.
  • Invoke-Unarchive will handle throttling, either by honoring the returned back-off period or by adding delays between operations.
  • Moving items is asynchronous, and Invoke-Unarchive needs to wait for Exchange to complete the previous move to folder X before it can move the next set of items to folder X.

Do not forget to reassign retention policies causing archival, or you might have the run the script again at later moment.

Syntax
The parameters to call Invoke-Unarchive.ps1 are:

  • Identity to specify one or more mailboxes to unarchive items for.
  • Server to specify the FQDN of the Client Access Server to use. When omitted, Autodiscover will be used.
  • IncludeRecoverableItems to instruct the script to process deletions stored in the Recoverable Items as well.
  • Impersonation to use impersonation when accessing the mailbox. When using modern authentication (OAuth), impersonation is mandatory.
  • Force to force moving of items without prompting.
  • NoProgressBar to prevent progress status.
  • TrustAll to accept all certificates including self-signed certificates.
  • TenantId specifies the ID of the Tenant when using a mailbox hosted in Exchange Online.
  • ClientId to specify the Application ID of the registered application in Azure Active Directory.
  • Credentials to specify the Basic Authentication credentials for on-premises usage or against Exchange Online when OAuth is not an option.
  • CertificateThumbprint is the thumbprint of the certificate to use for OAuth. The certificate with the public key needs to stored with the registered application for authentication. The certificate with the private key should be present in the local certificate store.
  • CertificateFile and CertificatePassword to specify the file of the certificate to use. The file shoud contain the private key, and the password to unlock the file can be specified using CertificatePassword.
  • Secret can be used to specify the secret to authenticate using the registered application.

Note that Credentials, CertificateThumbprint, CertificateFile + CertificatePassword and Secret are mutually exclusive.

Example
Below shows an example run against a test-mailbox using modern authentication (OAuth). The common parameter Verbose is used to display additional output.

.\Invoke-Unarchive.ps1 -Identity michel@myexchangelabs.com -Server outlook.office365.com -Impersonation -Secret <Secret> -TenantId <Tenant> -ClientId <AppId> -Verbose
image

You can find the script on GitHub here.

Final Notes
The EWS operation – especially moving items – is not necessarily slow, but against Exchange Online processing large archives can take considerable amount of time due to throttling. When moving a significant number of items using Outlook for Desktop, you will likely run into Outlook abandoning the operation after which you need to wait for Exchange to finish pending moves before you can continue with this task. Using the script, you can take away this unarchiving task from end users by running the operation in the background in one or multiple runs.

42 thoughts on “Unarchiving Mailbox Items”

  1. Hi Michel,

    Thank you for providing this tool, it looks like it will truly save my bacon.

    However, I’m experiencing issues when moving some mailboxes, it appears to throttle every time. I’m initiating larger mailboxes.

    Have you experienced this and is there a possible workaround?

    Here’s the exact error I’m receiving:

    Folder \Inbox exists, merging contents to unarchive 2871 item(s) and 10 folder(s)
    VERBOSE: Collecting folders to unarchive in \Inbox
    VERBOSE: Retrieving items to unarchive from \Inbox ..
    VERBOSE: Discovered 2871 items in \Inbox
    VERBOSE: Unarchiving 50/2871 item(s) from \Inbox
    WARNING: EWS operation failed (), will retry later
    WARNING: Previous EWS operation failed, waiting for 1500ms
    WARNING: EWS operation failed (), will retry later
    WARNING: Previous EWS operation failed, waiting for 2250ms
    WARNING: EWS operation failed (), will retry later
    WARNING: Previous EWS operation failed, waiting for 3375ms
    WARNING: EWS operation failed (ErrorServerBusy), will retry later
    WARNING: Throttling detected; server requested us to backoff for 316848ms

    Like

    Reply ↓

    • Unfortunately, cannot do a lot about getting throttled. I have on the to do list to add unarchiving to a (sub)folder, because merging takes substantially longer and hits throttling quickly (multiple calls to process things at item-level) than moving back a folder (single call operation).

      Liked by 1 person

      Reply ↓

  2. Not used this yet but this looks brilliant and is exactly what I was looking for. Will be running some tests with this soon and can leave some better feedback. Will also keep an eye out for the addition of the “unarchiving to a subfolder rather than merging” as that will help – especially with the amount of mailboxes I’m going to run this against.

    Like

    Reply ↓

  3. Hi,
    I installed Microsoft.Identity.Client silently to skip the dependency because i was constantly getting error of dependency loop. Now you can see in following transcript i have Module installed,

    PS C:\Users\userx\OneDrive\Common Need To Know\PowerShell\Invoke-Unarchive-main> Install-Package Microsoft.Identity.Client -ProviderName NuGet -Verbose
    VERBOSE: Using the provider ‘NuGet’ for searching packages.
    VERBOSE: Searching repository ‘https://www.nuget.org/api/v2/FindPackagesById()?id=’Microsoft.Identity.Client” for ”.
    VERBOSE: Total package yield:’1’ for the specified package ‘Microsoft.Identity.Client’.
    VERBOSE: Skipping installed package Microsoft.Identity.Client 4.42.1.

    but when i try to run the command it gives me following error of Dll.

    PS C:\Users\userx\OneDrive\Common Need To Know\PowerShell\Invoke-Unarchive-main> .\Invoke-Unarchive.ps1 -Identity TestArchive@noveltypharma.onmicrosoft.com -Credentials $Credentials -Impersonation -Server outlook.office365.com -Verbose
    VERBOSE: Loading module C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loading module from path ‘C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll’.
    VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 loaded
    VERBOSE: Required module Microsoft.Identity.Client.dll could not be located
    PS C:\Users\userx\OneDrive\Common Need To Know\PowerShell\Invoke-Unarchive-main>

    Please help me in this case

    Like

    Reply ↓

  4. Hi Michel, With your help, i could successfully Unarchive mailboxes of Office 365 and move all data to primary mailboxes. its really a great must have tool..

    Like

    Reply ↓

  5. Hi Michel! Thanks for your post!
    When I try to install Microsoft.Identity.Client always I was constantly getting error of dependency loop: “Install-Package : Dependency loop detected for package ‘Microsoft.Identity.Client'”. Could you help me?

    Like

    Reply ↓

    • I had that one time with other module, which I needed to forcibly remove (from its folder), reinstall and the necessary restarts. There’s always the option of getting the DLL and putting it in the same folder as the script.

      Like

      Reply ↓

      • Hello,

        I copied the dll into the same folder where the script is located, but I am stll getting an error:
        VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
        VERBOSE: Loading module D:\Master\scripts\EXO\Archive\Microsoft.Identity.Client.dll
        VERBOSE: Loading module from path ‘D:\Master\scripts\EXO\Archive\Microsoft.Identity.Client.dll’.
        Import-ModuleDLL : Problem loading module Microsoft.Identity.Client: Could not load file or assembly ‘Mono.Android,
        Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065’ or one of its dependencies. The system cannot find
        the file specified.
        At D:\Master\scripts\EXO\Archive\Invoke-Unarchive.ps1:794 char:5
        + Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

        Can you help me here? Which DLL copied you to the script folder?

        Many Thanks
        Michael

        Like

        Reply ↓

          • Hello,

            the Microsoft.Exchange.WebServices v2.2.1.0 is installed, only the Microsoft.Identity.Client reported that error with “Dependency loop detected”. I could instalI it with the command “Install-Package Microsoft.Identity.Client -ProviderName NuGet -skipdependencies” and found in the folder of the package several subfolders with dll files in it. I tried all of them, but I am still getting the error ” Could not load file or assembly ‘Mono.Android”.

            Michael

            Like

            Reply ↓

            • Welcome to packaging – SkipDepencencies just ignores any issues, it does not solve them. Which is why I have the option to look for related DLLs in the script folder. Downloading, unblocking and running the script from elevated prompt (in order to be able to load the modules) usually does the trick.

              Like

              Reply ↓

              • I copied all dll files to the script folder, but I am still getting the same error. Which version of Microsoft.Identity.Client did you use? I have installed 4.45.

                Many Thanks!

                Like

                Reply ↓

  6. Hey Michael, will be really great if I understand how to get the “ClientId” value. Call me newbie but I don´t know from where I get that information. My setup is simple: all mailboxes in exchange online and I need to move the archive content back to mailbox to migrate the data to google mailbox and then purge the mailbox content.

    Like

    Reply ↓

  7. Hi Michel, really great tool here, I was wondering if you knew how I might find out what’s causing this issue: https://i.imgur.com/FW9qy0x.png
    This is on-premises Exchange 2013 SP1 and it seems like mailboxes with smaller archives (around 3000 items?) will show no error, or show a few errors and then continue successfully, but larger archives like the one shown just loop the same error forever. I’ve tried adjusting the throttling policy on the server but it doesn’t seem to help so far.
    Do you have any suggestions for figuring out what’s going on, or any throttling policy settings that may help? Thanks.

    Like

    Reply ↓

    • Oh and I forgot to mention, despite what the output is showing, it is actually moving messages from the archive to the primary mailbox, slowly, at about 3-4 messages per second.

      Like

      Reply ↓

    • The call to move items is asynchronous. When instructing Exchange to move items to folder X, folder X gets sort of locked and you cannot move more items to folder X until the previous move operation has finished. If you took care of throttling, it could be that the operation is resource constraint. The script tries to ease off when it gets told the ServerBusyException (message indicates it is), by adding delays and lowering the batch size (varies from 10 to 50 items per move request). After the request, Exchange can still be busy moving items from archive to mailbox, so then it’s a matter of waiting. Haven’t found a way to speed up this process, if only Managed Folder Assistant could move stuff back as well 🙂

      Like

      Reply ↓

  8. Hi Michel

    I have client who has 1 year retention but now wants emails moved back and have 2 year retention. What’s the best way of doing this. Current mailbox size is 29GB and archive is 104GB auto expanding. So end goal is main mailbox should have 2 years worth of emails and rest stay in in place archive

    Like

    Reply ↓

    • Change retention policy, depending on if you made new tags or updated the existing one wait for Managed Folder Assistant to process mailbox, and finally unarchive items younger than X days. That last part is currently not an option of this script, but I put it on the to list since it sounds like a good idea when customers increase the retention time before archiving items.

      Like

      Reply ↓

  9. Hello,

    I worked for a long time to get all pre requisites completed for your script and get it working, I got stuck and can’t proceed go ahead of here. Let me know please if I can get any help.

    VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
    VERBOSE: Module Microsoft.Identity.Client v4.24.0.0 already loaded
    VERBOSE: Will use provided secret to authenticate
    VERBOSE: Authentication token acquired
    Processing mailbox ***@******.com (***@******.com)
    VERBOSE: Using ***@******.com for impersonation
    VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
    WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (403)
    Forbidden.”
    C:\Users\*****\Desktop\Invoke-Unarchive.ps1 : Cannot access primary mailbox of ***@******.com:
    At line:1 char:1
    + .\Invoke-Unarchive.ps1 -Identity ***@******.com -Server outlook …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Unarchive.ps1

    Liked by 1 person

    Reply ↓

  10. Hi there,

    I have set up everything as explained in the guide but I am getting the following PS before the script will run. Both DLL’s load succesfully.

    Import-ModuleDLL problem initializing test-object from module microsoft.identitity.client.

    Any idea what might be causing this issue?

    Like

    Reply ↓

  11. Keep getting this:
    WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “Credentials are required to make a service request.”

    Not very clear on how to use a secret with this script – any chance you could clarify how I should form the arguments correctly?

    Like

    Reply ↓

    • Or I get this one: WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (403) Forbidden.”

      My command:
      .\Invoke-Unarchive.ps1 -Identity “me@myupn.com” -Server outlook.office365.com -Impersonation -Secret $(ConvertTo-SecureString “client secret value” -AsPlainText -Force) -TenantId “directory-tenant-id” -ClientId “application-client-id” -Verbose

      Like

      Reply ↓

  12. Hi,
    I managed to connect to V3 EXO PowerShell with the -CertificateThumbprint but I am still getting the below error:-
    WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote
    server returned an error: (403) Forbidden.”

    Screenshot of the connection here –> https://onedrive.live.com/?authkey=%21ACL0V%2Du2SQwf2lg&cid=A9FBF65766E40807&id=A9FBF65766E40807%211705&parId=root&o=OneUp

    Like

    Reply ↓

  14. Great Script!
    It would be awesome if you can update this script to work on On-Premise Exchange 2016, and not just Online Exchange only.

    Like

    Reply ↓

      • Sure no problems, I just assumed it was online only as I was having trouble running this script on my on-premise Exchange 2016 server. I had used your example, but just omitted the “online” parameters since this was a local exchange.

        .\Invoke-Unarchive.ps1 -Identity -Verbose

        But I got this error
        “Parameter set cannot be resolved using the specified named parameters”

        So I’m assuming it was expecting those 365 parameters (e.g. -Secret, -TenantID, etc..)

        Is there anyway to replace them with local exchange parameters?

        Like

        Reply ↓

  15. If I just have one mailbox that I’m playing with (testing), can’t I just disable the retention policy, manually move everything from Archive back to Mail Store, disable the archive for that user, then enable the retention policy again? This is a sincere question as I’ve never disabled the retention policy (maybe it’s not even possible…maybe I have to change it from 2 years to 30 years as the default?)

    Like

    Reply ↓

    • If you disable a retention policy tag (part of policies), not policies themselves. Items flagged with that tag are suspended from processing.
      If you set the retention policy for a mailbox to $null, mailbox records management will skip it (naturally, not that Default Retention Policy is by default assigned).
      If you assign a different retention policy, that policy will become effective and items will get re-evaluated (personal tags are retained).
      If you remove a tag from a policy, items in a mailbox with that policy will still be subject to that tag (expire etc).
      If you delete a tag, items that were flagged with that tag (any policy) re-evaluated.

      But – you can overall suspend retention also for a mailbox, without reconfiguring any of this, using Set-Mailbox with the RetentionHoldEnabled parameter, which allows you to suspend MRM processing.

      Like

      Reply ↓

  16. Thanks Michel de Rooij for sharing such a lovely script. Would really appreciate if you please let us know the command to be run for Exchange 2016 with basic authentication

    Like

    Reply ↓

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.