For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:
- Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
- Importing existing data. Accidental import from – for example – a PST file to a mailbox can lead to duplicate entries.
When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.
When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).
When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).
Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.
Using the Remove-DuplicateItems.p1 script requires Exchange 2007 or later and Exchange Web Services (EWS) Managed API 1.2 (or later) which you can download here. Alternatively, when you already got the EWS Managed API Microsoft.Exchange.WebServices.DLL somewhere, you can copy it to the same folder where the script is located and it will be picked up. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.
Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post.
The script Remove-DuplicateItems.ps1 uses the following syntax:
Remove-DuplicateItems.ps1 [[-Identity] <String>] [[-Type] <String>] [-Retain <String>] [-Server <String>] [-Impersonation] [-DeleteMode <String>] [-Credentials <PSCredential>] [-Mode <String>] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String>] [-ExcludeFolders <String>] [-MailboxWide] [-Force] [-WhatIf] [-Confirm] [<CommonParameters>]
A quick walk-through on the parameters and switches:
- Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
- Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
- Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
- Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
- When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
- DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
- Credentials specifies credentials to use (provide credentials using Get-Credential).
- Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:
|Contacts||File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size|
|Distribution List||FileAs, Number of Members, Size|
|Calendar||Subject, Location, Start & End Date, Size|
|Task||Subject, Start Date, Due Date, Status, Size|
|Note||Contents, Color, Size|
|Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size|
- MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter when running against mailboxes on Exchange Server 2007.
- ArchiveOnly specifies you only want to process personal archives of specified users.
- IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around of at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. Matching is always case-insensitive. You can also well-known folders, by using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are Calendar, Contacts, Inbox, Notes, SentItems, and Tasks.
- ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolder.
- MailboxWide specifies to perform duplicate cleanup against whole mailbox, instead of per folder. A WORD OF CAUTION: The first unique item encountered will be retained. Since there is no way to order mailbox folders, when an item is found in Folder A and in Folder B, it is undetermined which item will be kept.
- When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
- When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
- When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
- For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
- You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.
So, suppose you want to remove duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:
Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
Alternative, you can use an e-mail address and specify credentials. This allows the script to run against mailboxes in Office 365, for example:
Remove-DuplicateItems.ps1 -Identity email@example.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest
An example using IncludeFolder and ExcludeFolder:
$cred= Get-Credential Remove-DuplicateItems.ps1 -Identity firstname.lastname@example.org -Credentials $cred -IncludeFolders 'Archive','Project*','#Inbox#' -ExcludeFolders 'ProjectX'
In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Mailbox field. An example of how the CSV could look:
Identity francis philip
The cmdlet could then be something like:
Import-CSV users.csv1 | Remove-DuplicateItems.ps1 -DeleteMode HardDelete -Impersonation
You can download the script from the TechNet Gallery here.
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.
See TechNet Gallery page.