Removing Duplicate Items from a Mailbox

powershellLast version: 1.4, June 24th, 2014.

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

  • Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
  • Importing existing data. Accidental import from – for example – a PST file to a mailbox  can lead to duplicate entries.

image

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange 2007 or later and Exchange Web Services (EWS) Managed API 1.2 (or later) which you can download here. Alternatively, when you already got the EWS Managed API Microsoft.Exchange.WebServices.DLL somewhere, you can copy it to the same folder where the script is located and it will be picked up. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation for Exchange 2010 using RBAC, see this article or check here for details on how to configure impersonation for Exchange 2007.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [-Mailbox] <String> [[-Type] <String>] [-Retain <String>] [-Server <String>] [-Impersonation] [-DeleteMode <String>] [-Credentials <PSCredential>] [-Mode <String>] [-MailboxOnly] [-ArchiveOnly] [-WhatIf] [-Confirm] [<CommonParameters>]

A quick walk-through on the parameters and switches:

  • Mailbox is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
  • Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
  • Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
  • When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
  • DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
  • Credentials specifies credentials to use (provide credentials using Get-Credential).
  • Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:
ItemClass Criteria
Contacts File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List FileAs, Number of Members, Size
Calendar Subject, Location, Start & End Date, Size
Task Subject, Start Date, Due Date, Status, Size
Note Contents, Color, Size
Mail Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other Subject, DateTimeReceived
  • MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter  when running against mailboxes on Exchange Server 2007.
  • ArchiveOnly specifies you only want to process personal archives of specified users.

Few notes:

  • When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
  • When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
  • When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
  • For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
  • You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

So, suppose you want to remove  duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Mailbox migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

image

Alternative, you can use an e-mail address and specify credentials.  This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.onmicrosoft.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Mailbox field. An example of how the CSV could look:

Mailbox
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 -DeleteMode HardDelete -Impersonation

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision History
See TechNet Gallery page.

56 thoughts on “Removing Duplicate Items from a Mailbox

  1. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 20, 2013 | NeWay

  2. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 21, 2013 | NeWay

  3. First, this is a VERY cool script, so thank you. But I can’t get it to work, and am hoping you can help. I get an error stating “Cant access mailbox information store”
    i’m running this from a management server (not an Exchange box). The account I’m using is an Exchange org admin. Thanks!!

      • Thank you, I had to grant impersonation rights. At first I granted Full Access, but I guess it didn’t work. I’m running Exchange 2010, in case anyone else has this issue. Thanks again for a VERY useful script!

  4. Thank you for your awesome script!
    It has been very useful.
    Any chance to run it in Online Archive mailboxes?
    Thank you!
    JP

  5. It looks like its running fine, but don’t find the dubs that i made by copying mails, why is this? and why does it not run on custom folders?

    • Depending on the way you copied, it could be the “duplicates” have a different PidTagSearchKey. Try running it in Full mode instead of the default Quick Mode (-Mode), in which case it’ll use a predefined set of attributes depending on the type of items to find matching items (see article for which attributes are checked). By default it will scan all folder types (-Type), looking for duplicates on a per folder basis.

      • Thanks, I found the solution. it does not work fully before i configure the impersonation rights. Im running it from a workstation with web services 2.0 against a exchange 2010 installation. Wonderful script you have made!

  6. I receive an error with the script when installed on Exchange 2012. I do have the dll in the folder with the ps1, and have the EWS installed. I downloaded again and verified the download.

    …..Remove-DuplicateItems.ps1:349 char:156
    + … ationsMode and $AffectedTaskOccurences:$AffectedTaskOccurences”
    + ~~~~~~~~~~~~~~~~~~~~~~~~
    Variable reference is not valid. ‘:’ was not followed by a valid variable name character. Consider using ${} to delimit the name.
    + CategoryInfo : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : InvalidVariableReferenceWithDrive

  7. I get the following error :
    You cannot call a method on a null-valued expression.
    At C:\psscripts\Remove-DuplicateItems.ps1:271 char:82
    + $key= $Item.DateTimeReceived.ToString <<<< ()
    + CategoryInfo : InvalidOperation: (ToString:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

  8. Hello,

    When I run the script, everything looks fine, except for the fact that is removes 0 items. It’s for shure that the mailbox where I try it on is full of duplicate items. This is the best way I found on the net to remove duplicate items, but doesn’t work for me. Please Help!

  9. Hi
    Wonderful script!
    I´m doing it on exchange 2013 and has change some of the rows to get it working.
    1. First probably a typo:
    … and $AffectedTaskOccurences:$AffectedTaskOccurences”
    … and AffectedTaskOccurences:$AffectedTaskOccurences”
    2. Change all text: Exchange2007_sp1 to Exchange2013
    3. On 2 occurences replace:
    $ItemIds= [activator]::createinstance(([type]‘System.Collections.Generic.List`1′).makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
    with:
    $type = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
    $type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
    $Itemids = [Activator]::CreateInstance($type)

    Now the script works! But still getting an error at the end:
    C:\root\Remove-DuplicateItems.ps1 : Problem processing mailbox it (it@google.com)
    At line:1 char:1
    + .\Remove-DuplicateItems.ps1 -mailbox it -type all -impersonation -deletemode har …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    I Think it fails the error check at the end but cannot solve it.
    $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
    If(! ( Process-Mailbox $RootFolder)) {…

    Any ideas?

    /MrTbone

    • 1) Thanks. Display only, won’t affect script;
      2) No. Defines minimum level of EWS support required.
      3) PowerShell v3 thing – fixed in Fix-MailboxFolder script, still need to propagate to this script.

      Regarding the message, make sure you have permissions (impersonation – preferred – or otherwise).

      • Hi

        The script is doing its job with previous changes. duplicates are deleted.
        I have the impersonation rights.
        But I get another error message in the middle of the process to:

        Process-Mailbox : Detected problem removing items (loop)
        At C:\root\Remove-DuplicateItems.ps1:458 char:16
        + If(! ( Process-Mailbox $RootFolder)) {
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

        This leads me to the lines:
        $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
        If(! ( Process-Mailbox $RootFolder)) {…

        Any surgestions?
        /MrTbone

        • The script checks if the number of items to be deleted are in actually deleted. However, with larger batches Exchange may still be busy deleting items (we hand off a list of items to delete) so numbers don’t add up. In such cases I’d recommend using smaller batches ($MaxItemBatchSize= 100). I may parametrize this in future versions.

          • Hi!
            Thanks for the Quick reply.
            This script is so cool!!!!

            had to change $type to $typ cause it was used earlier in script:
            $typ = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
            $typ = $typ.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
            $Itemids = [Activator]::CreateInstance($typ)

            After this and raising the max items all seems fine.

            Thanks!
            /MrT

  10. Is there anyway to use this with office365/exchange online, or will it only work with an in house exchange

    As we have just moved to office365 from another hosted exchange and using the built in Microsoft tools to move the mail have managed to end up with every mailbox with duplicate items

    Thanks
    Jamie

    • Hi I have tried to use this against office365 but get the following error

      PS C:\Users\cnl_admin\Desktop> .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type
      Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose

      Security warning
      Run only scripts that you trust. While scripts from the internet can be useful
      this script can potentially harm your computer. Do you want to run
      C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1?
      [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): r
      VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web
      Services\2.0\\Microsoft.Exchange.WebServices.dll
      Processing mailbox jamiec
      VERBOSE: Set to trust all certificates
      VERBOSE: Looking up EWS URL using Autodiscover for jamie.cox@domain.com
      C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1 : A positional parameter
      cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl”
      with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
      At line:1 char:1
      + .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type Mail -DeleteMode
      MoveToDeleted …
      + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      ~~~
      + CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1]
      , ParameterBindingException
      + FullyQualifiedErrorId : PositionalParameterNotFound,Remove-DuplicateItem
      s.ps1

  11. Hi Michel

    I have now run this with great success in Exchange 2013. But the largest mailboxes seems to fail with the largest folders.
    inbox and sent items in the largest mailboxes get:

    Exception calling “FindItems” with “2” argument(s): “Unexpected end of file while parsing Name has occurred. Line 1, po
    sition 1244584.”
    At C:\root\Remove-DuplicateItems.ps1:253 char:25
    + $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : XmlException

    Have tried the Maxfolder and Maxitem but no success.
    Could it be some limit i EWS config?

    I also get an error deleteing in a few folders:
    Process-Mailbox : Detected problem removing items (loop)
    At C:\root\Remove-DuplicateItems.ps1:459 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
    any ideas?

    /MrT

    • 1) Do you have a number so I can try reproduce the problem?
      2) I check if the number of items returned have decreased by the number of items removed. However, it’s asynchronous and Exchange – especially with large counts – may still be removing things in in the background. Suggest using low MaxItems.

      • Hi
        Not so happy today!
        10% of the mailboxes has not deleted their duplicates. Still there
        Instead they have deleted all mail from all folders between a week back and a year back
        Cannot se a pattern. but it´s kaos! :-( MrT

  12. Would REALLY be great if this worked against Archive mailboxes. I am about to undertake a project where I have a huge migration with import from .pst, we had hoped to go to archive mailbox, but cant, it will take an extra step to do this. But testing on regular Exchange 2013 mailbox works like a champ. Thanks

  13. Hi Micheal,

    I testing your script but this send me a this message….
    [PS] C:\Documents and Settings\acweinzierll\Desktop>Remove-DuplicateItems.ps1 -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
    The term ‘Remove-DuplicateItems.ps1′ is not recognized as a cmdlet, function, operable program, or script file. Verify
    the term and try again.
    At line:1 char:26
    + Remove-DuplicateItems.ps1 <<<< -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mo
    de Full -Verbose

    I have Exchange 2007

    Can YOu help Me… Thanks…

  14. VERIFIED TO WORK WELL ON EXCHANGE 2010 SP3! – 3/10/2014

    Very powerful script! Once I installed Exchange Web Services (EWS) Managed API 1.2 and assigned the Hygiene Management role to the account I was running the scripts under I had no problems!

    Be sure to follow the formatting provided in the examples as I noticed weird issues when I specified a different order of the arguments than that of the examples.

    Below is what worked best for my environment!!!

    ./Remove-DuplicateItems.ps1 -Mailbox -johnsmith -Type Mail -Impersonation -DeleteMode HardDelete -Mode Full -Verbose

    **Used to remove duplicates on imported mail that orginated from Google Apps via Outlook 2007 and GASMO**

  15. Hello! Congrats for your script. It’s really fantastic!! I’ve added impersonation rights, everything runs OK on most of mailboxes. But I have a little problem in one of them. The scripts crashes when starts analyzin the folder “Sent Items”. I supose there is a problem on this folder, otherwise the script wouldn’t run in the other mailboxes. I’m wondering if there is any possibility to define which folders analyze or better, add a folder exception… Is it possible? If not, have you ever been in the same situation, same problem? Many thanks!!

  16. Hi Michel,
    Thank you for your wonderful script.

    I used the script to remove duplicate contacts from an Exchange 2013 SP1 folder where most of them were duplicates (800 unique contacts, 138,000 duplicates)

    I got this error:
    Process-Mailbox : Detected problem removing items (loop)
    At .\Remove-DuplicateItems.ps1:470 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

    The lines that cause this error start at line 369:
    $OldOffSet= $ItemView.Offset
    $ItemView.Offset+= $ItemSearchResults.Items.Count – $ItemIds.Count
    If( $OldOffset -eq $ItemView.Offset) {
    Write-Error “Detected problem removing items (loop)”
    $ProcessingOK= $False
    }
    For me, most of the time the number of items deleted ($ItemIds.Count) was the same as the number of items in the batch ($ItemSearchResults.Items.Count), so this test condition was usually matching, and halting the script.
    I removed the test, and Remove-DuplicateItems.ps1 ran ok.

    Are there any side effects that might occur without this test?

    PS: I also tested increasing the $MaxItemBatchSize to 1000 and it seemed to work ok.

  17. Hi Michel
    I would like to add Appointment RequiredAttendees to the mode=full duplicate-detection $key.
    Does FindItems populate RequiredAttendees with the [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties view?
    If not, what changes are needed to retrieve the RequiredAttendees property?
    Thanks for your help,
    John

  18. Hello,

    We are using this script to try to remove some duplicate contacts. We have a sync process that pushes out all of our contacts from a public folder to everyone’s contacts. The tool we use to sync isn’t great, so we have dupes. The script works great, but it is removing the most recently added contact and I’d like to remove the old one. Any ideas on modifiying the script to make this happen?

  19. Pingback: Script Updates | EighTwOne (821)

  20. Hi Michel,

    Great script and great that it now also works with EO / O365.
    Only issue I have now is that it doesnt search through folders created by the user.

    Any suggestions?

    Cheers,
    Pierre.

  21. Hi,

    I’m trying to run this script against an Office365 tenant that I’ve just migrated our customer to as the migration process created quite a lot of duplicate items (which we expected it might).

    I’ve added the Impersonation rights and full access mailbox permissions to the user i’m running the script as but I just can get it to execute?

    autodiscovering the EWS url seems to fail (even though autodiscover works fine when setting up outlook profiles etc.) and i’m not sure what format to enter the -server parameter as? I’ve tried the generic outlook.office365.com, I’ve tried the mailbox users exchange GUID@domain.com (as per the outlook profile) but nothing seems to work.

    Any suggestions???

    Many thanks in advance :)
    James.

  22. I keep getting a “Can’t access mailbox information store At Line:1 char:28 + .\remove-duplicateitems.ps1 <<<< -Mailbox

    I've got application impersonation set on the account

      • I’ve tried both, same results

        This is the script that I’m running….

        .\Remove-DuplicateItems.ps1 -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Quick -Verbose

        This is the output that I get…..

        VERBOSE: Loading C:\temp\duplicates\Microsoft.Exchange.WebServices.dll
        VERBOSE: Loading module from path ‘C:\temp\duplicates\Microsoft.Exchange.WebServices.dll’.
        Processing mailbox aliasname (aliasname@email.com)
        VERBOSE: Set to trust all certificates
        VERBOSE: Using aliasname@email.com for impersonation
        VERBOSE: Looking up EWS URL using Autodiscover for aliasname@email.com
        VERBOSE: Using EWS on CAS https://servername
        VERBOSE: DeleteMode is MoveToDeletedItems
        VERBOSE: Processing Mail items
        C:\temp\duplicates\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
        At line:1 char:28
        + .\Remove-DuplicateItems.ps1 <<<< -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeleted -Mode
        Quick -MailboxOnly -Verbose
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

  23. Hello Michel,

    Situation long version: I have a mailbox of 14gb and with a ton of folders/e-mails, something went wrong with the migration and in some folders there are six duplicate items.

    I ran this script with the following command: .\Remove-DuplicateItems.ps1 -Mailbox email@email.nl -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest -Verbose

    – I then get a popup asking me for credentials and I enter the credentials of the specific mailbox and it started moving over 13k duplicated to the deleted items folder.

    After it was done doing this, i checked a couple of folders and it showed some still have duplicate items. Checking the log it shows it never told there are duplicates in there.

    Processing folder FOLDER NAME
    Procession folder FOLDER NAME

    when it did remove duplicates it shows:

    Processing folder FOLDER NAME
    Removing # items from FOLDER NAME

    I tried running the script again cause maybe that might help, but It removed 0 items the second time around.

    Any tips on getting the rest of the duplicates removed?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s