Removing Duplicate Items from a Mailbox

powershellLast version: 1.41, November 26th, 2014.

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

  • Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
  • Importing existing data. Accidental import from – for example – a PST file to a mailbox  can lead to duplicate entries.


When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Using the Remove-DuplicateItems.p1 script requires Exchange 2007 or later and Exchange Web Services (EWS) Managed API 1.2 (or later) which you can download here. Alternatively, when you already got the EWS Managed API Microsoft.Exchange.WebServices.DLL somewhere, you can copy it to the same folder where the script is located and it will be picked up. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation for Exchange 2010 using RBAC, see this article or check here for details on how to configure impersonation for Exchange 2007.

The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [-Mailbox] <String> [[-Type] <String>] [-Retain <String>] [-Server <String>] [-Impersonation] [-DeleteMode <String>] [-Credentials <PSCredential>] [-Mode <String>] [-MailboxOnly] [-ArchiveOnly] [-WhatIf] [-Confirm] [<CommonParameters>]

A quick walk-through on the parameters and switches:

  • Mailbox is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
  • Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
  • Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
  • When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
  • DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
  • Credentials specifies credentials to use (provide credentials using Get-Credential).
  • Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:
ItemClass Criteria
Contacts File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List FileAs, Number of Members, Size
Calendar Subject, Location, Start & End Date, Size
Task Subject, Start Date, Due Date, Status, Size
Note Contents, Color, Size
Mail Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other Subject, DateTimeReceived
  • MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter  when running against mailboxes on Exchange Server 2007.
  • ArchiveOnly specifies you only want to process personal archives of specified users.

Few notes:

  • When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
  • When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
  • When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
  • For more info on PidTagSearchKey, see Note that PidTagSearchKey will have duplicate values for copied objects.
  • You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

So, suppose you want to remove  duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Mailbox migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose


Alternative, you can use an e-mail address and specify credentials.  This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Mailbox -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Mailbox field. An example of how the CSV could look:


The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 -DeleteMode HardDelete -Impersonation

You can download the script from the TechNet Gallery here.

Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision History
See TechNet Gallery page.

92 thoughts on “Removing Duplicate Items from a Mailbox

  1. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 20, 2013 | NeWay

  2. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 21, 2013 | NeWay

  3. First, this is a VERY cool script, so thank you. But I can’t get it to work, and am hoping you can help. I get an error stating “Cant access mailbox information store”
    i’m running this from a management server (not an Exchange box). The account I’m using is an Exchange org admin. Thanks!!

      • Thank you, I had to grant impersonation rights. At first I granted Full Access, but I guess it didn’t work. I’m running Exchange 2010, in case anyone else has this issue. Thanks again for a VERY useful script!

  4. Thank you for your awesome script!
    It has been very useful.
    Any chance to run it in Online Archive mailboxes?
    Thank you!

  5. It looks like its running fine, but don’t find the dubs that i made by copying mails, why is this? and why does it not run on custom folders?

    • Depending on the way you copied, it could be the “duplicates” have a different PidTagSearchKey. Try running it in Full mode instead of the default Quick Mode (-Mode), in which case it’ll use a predefined set of attributes depending on the type of items to find matching items (see article for which attributes are checked). By default it will scan all folder types (-Type), looking for duplicates on a per folder basis.

      • Thanks, I found the solution. it does not work fully before i configure the impersonation rights. Im running it from a workstation with web services 2.0 against a exchange 2010 installation. Wonderful script you have made!

  6. I receive an error with the script when installed on Exchange 2012. I do have the dll in the folder with the ps1, and have the EWS installed. I downloaded again and verified the download.

    …..Remove-DuplicateItems.ps1:349 char:156
    + … ationsMode and $AffectedTaskOccurences:$AffectedTaskOccurences”
    + ~~~~~~~~~~~~~~~~~~~~~~~~
    Variable reference is not valid. ‘:’ was not followed by a valid variable name character. Consider using ${} to delimit the name.
    + CategoryInfo : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : InvalidVariableReferenceWithDrive

  7. I get the following error :
    You cannot call a method on a null-valued expression.
    At C:\psscripts\Remove-DuplicateItems.ps1:271 char:82
    + $key= $Item.DateTimeReceived.ToString <<<< ()
    + CategoryInfo : InvalidOperation: (ToString:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

  8. Hello,

    When I run the script, everything looks fine, except for the fact that is removes 0 items. It’s for shure that the mailbox where I try it on is full of duplicate items. This is the best way I found on the net to remove duplicate items, but doesn’t work for me. Please Help!

  9. Hi
    Wonderful script!
    I´m doing it on exchange 2013 and has change some of the rows to get it working.
    1. First probably a typo:
    … and $AffectedTaskOccurences:$AffectedTaskOccurences”
    … and AffectedTaskOccurences:$AffectedTaskOccurences”
    2. Change all text: Exchange2007_sp1 to Exchange2013
    3. On 2 occurences replace:
    $ItemIds= [activator]::createinstance(([type]’System.Collections.Generic.List`1′).makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
    $type = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
    $type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
    $Itemids = [Activator]::CreateInstance($type)

    Now the script works! But still getting an error at the end:
    C:\root\Remove-DuplicateItems.ps1 : Problem processing mailbox it (
    At line:1 char:1
    + .\Remove-DuplicateItems.ps1 -mailbox it -type all -impersonation -deletemode har …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    I Think it fails the error check at the end but cannot solve it.
    $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
    If(! ( Process-Mailbox $RootFolder)) {…

    Any ideas?


    • 1) Thanks. Display only, won’t affect script;
      2) No. Defines minimum level of EWS support required.
      3) PowerShell v3 thing – fixed in Fix-MailboxFolder script, still need to propagate to this script.

      Regarding the message, make sure you have permissions (impersonation – preferred – or otherwise).

      • Hi

        The script is doing its job with previous changes. duplicates are deleted.
        I have the impersonation rights.
        But I get another error message in the middle of the process to:

        Process-Mailbox : Detected problem removing items (loop)
        At C:\root\Remove-DuplicateItems.ps1:458 char:16
        + If(! ( Process-Mailbox $RootFolder)) {
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

        This leads me to the lines:
        $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
        If(! ( Process-Mailbox $RootFolder)) {…

        Any surgestions?

        • The script checks if the number of items to be deleted are in actually deleted. However, with larger batches Exchange may still be busy deleting items (we hand off a list of items to delete) so numbers don’t add up. In such cases I’d recommend using smaller batches ($MaxItemBatchSize= 100). I may parametrize this in future versions.

          • Hi!
            Thanks for the Quick reply.
            This script is so cool!!!!

            had to change $type to $typ cause it was used earlier in script:
            $typ = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
            $typ = $typ.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
            $Itemids = [Activator]::CreateInstance($typ)

            After this and raising the max items all seems fine.


  10. Is there anyway to use this with office365/exchange online, or will it only work with an in house exchange

    As we have just moved to office365 from another hosted exchange and using the built in Microsoft tools to move the mail have managed to end up with every mailbox with duplicate items


    • Hi I have tried to use this against office365 but get the following error

      PS C:\Users\cnl_admin\Desktop> .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type
      Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose

      Security warning
      Run only scripts that you trust. While scripts from the internet can be useful
      this script can potentially harm your computer. Do you want to run
      [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): r
      VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web
      Processing mailbox jamiec
      VERBOSE: Set to trust all certificates
      VERBOSE: Looking up EWS URL using Autodiscover for
      C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1 : A positional parameter
      cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl”
      with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
      At line:1 char:1
      + .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type Mail -DeleteMode
      MoveToDeleted …
      + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      + CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1]
      , ParameterBindingException
      + FullyQualifiedErrorId : PositionalParameterNotFound,Remove-DuplicateItem

        • Hi, I get exactly the same error:

          A positional parameter cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.

          and have seen your reply which refers to version 1.3. Where can I find the latest version of your PS script?

          Kind regards,
          Sven Hoelbling

          • Sorry, I noticed that I already had version 1.4, but I still get this error.
            Can you let me please know what I’m missing?
            All mailboxes are located in Office 365, but it is a hybrid deployment with on-premise Exchange 2013 SP 1 servers. When I point the script to these servers using the -server switch the script returns:
            Can’t access mailbox information store
            Shouldn’t it use autodiscover as Outlook does (in a hybrid depoyment the DNS entry for autodiscover points to the same onpremise CAS server)

            Kind regards,
            Sven Hoelbling

            • 1) You will find the latest version of the script here.
              2) Hybrid shouldn’t matter. If your internal DNS is set up correctly for Autodiscover (or using domain-joined client?). Can’t access mailbox store when overriding the EWS web service URL might possibly indicate a permission issue (or other blockade, e.g. firewall).

  11. Hi Michel

    I have now run this with great success in Exchange 2013. But the largest mailboxes seems to fail with the largest folders.
    inbox and sent items in the largest mailboxes get:

    Exception calling “FindItems” with “2” argument(s): “Unexpected end of file while parsing Name has occurred. Line 1, po
    sition 1244584.”
    At C:\root\Remove-DuplicateItems.ps1:253 char:25
    + $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : XmlException

    Have tried the Maxfolder and Maxitem but no success.
    Could it be some limit i EWS config?

    I also get an error deleteing in a few folders:
    Process-Mailbox : Detected problem removing items (loop)
    At C:\root\Remove-DuplicateItems.ps1:459 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
    any ideas?


    • 1) Do you have a number so I can try reproduce the problem?
      2) I check if the number of items returned have decreased by the number of items removed. However, it’s asynchronous and Exchange – especially with large counts – may still be removing things in in the background. Suggest using low MaxItems.

      • Hi
        Not so happy today!
        10% of the mailboxes has not deleted their duplicates. Still there
        Instead they have deleted all mail from all folders between a week back and a year back
        Cannot se a pattern. but it´s kaos! :-( MrT

  12. Would REALLY be great if this worked against Archive mailboxes. I am about to undertake a project where I have a huge migration with import from .pst, we had hoped to go to archive mailbox, but cant, it will take an extra step to do this. But testing on regular Exchange 2013 mailbox works like a champ. Thanks

  13. Hi Micheal,

    I testing your script but this send me a this message….
    [PS] C:\Documents and Settings\acweinzierll\Desktop>Remove-DuplicateItems.ps1 -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
    The term ‘Remove-DuplicateItems.ps1′ is not recognized as a cmdlet, function, operable program, or script file. Verify
    the term and try again.
    At line:1 char:26
    + Remove-DuplicateItems.ps1 <<<< -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mo
    de Full -Verbose

    I have Exchange 2007

    Can YOu help Me… Thanks…

  14. VERIFIED TO WORK WELL ON EXCHANGE 2010 SP3! – 3/10/2014

    Very powerful script! Once I installed Exchange Web Services (EWS) Managed API 1.2 and assigned the Hygiene Management role to the account I was running the scripts under I had no problems!

    Be sure to follow the formatting provided in the examples as I noticed weird issues when I specified a different order of the arguments than that of the examples.

    Below is what worked best for my environment!!!

    ./Remove-DuplicateItems.ps1 -Mailbox -johnsmith -Type Mail -Impersonation -DeleteMode HardDelete -Mode Full -Verbose

    **Used to remove duplicates on imported mail that orginated from Google Apps via Outlook 2007 and GASMO**

  15. Hello! Congrats for your script. It’s really fantastic!! I’ve added impersonation rights, everything runs OK on most of mailboxes. But I have a little problem in one of them. The scripts crashes when starts analyzin the folder “Sent Items”. I supose there is a problem on this folder, otherwise the script wouldn’t run in the other mailboxes. I’m wondering if there is any possibility to define which folders analyze or better, add a folder exception… Is it possible? If not, have you ever been in the same situation, same problem? Many thanks!!

  16. Hi Michel,
    Thank you for your wonderful script.

    I used the script to remove duplicate contacts from an Exchange 2013 SP1 folder where most of them were duplicates (800 unique contacts, 138,000 duplicates)

    I got this error:
    Process-Mailbox : Detected problem removing items (loop)
    At .\Remove-DuplicateItems.ps1:470 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

    The lines that cause this error start at line 369:
    $OldOffSet= $ItemView.Offset
    $ItemView.Offset+= $ItemSearchResults.Items.Count – $ItemIds.Count
    If( $OldOffset -eq $ItemView.Offset) {
    Write-Error “Detected problem removing items (loop)”
    $ProcessingOK= $False
    For me, most of the time the number of items deleted ($ItemIds.Count) was the same as the number of items in the batch ($ItemSearchResults.Items.Count), so this test condition was usually matching, and halting the script.
    I removed the test, and Remove-DuplicateItems.ps1 ran ok.

    Are there any side effects that might occur without this test?

    PS: I also tested increasing the $MaxItemBatchSize to 1000 and it seemed to work ok.

  17. Hi Michel
    I would like to add Appointment RequiredAttendees to the mode=full duplicate-detection $key.
    Does FindItems populate RequiredAttendees with the [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties view?
    If not, what changes are needed to retrieve the RequiredAttendees property?
    Thanks for your help,

  18. Hello,

    We are using this script to try to remove some duplicate contacts. We have a sync process that pushes out all of our contacts from a public folder to everyone’s contacts. The tool we use to sync isn’t great, so we have dupes. The script works great, but it is removing the most recently added contact and I’d like to remove the old one. Any ideas on modifiying the script to make this happen?

  19. Pingback: Script Updates | EighTwOne (821)

  20. Hi Michel,

    Great script and great that it now also works with EO / O365.
    Only issue I have now is that it doesnt search through folders created by the user.

    Any suggestions?


  21. Hi,

    I’m trying to run this script against an Office365 tenant that I’ve just migrated our customer to as the migration process created quite a lot of duplicate items (which we expected it might).

    I’ve added the Impersonation rights and full access mailbox permissions to the user i’m running the script as but I just can get it to execute?

    autodiscovering the EWS url seems to fail (even though autodiscover works fine when setting up outlook profiles etc.) and i’m not sure what format to enter the -server parameter as? I’ve tried the generic, I’ve tried the mailbox users exchange (as per the outlook profile) but nothing seems to work.

    Any suggestions???

    Many thanks in advance :)

  22. I keep getting a “Can’t access mailbox information store At Line:1 char:28 + .\remove-duplicateitems.ps1 <<<< -Mailbox

    I've got application impersonation set on the account

      • I’ve tried both, same results

        This is the script that I’m running….

        .\Remove-DuplicateItems.ps1 -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Quick -Verbose

        This is the output that I get…..

        VERBOSE: Loading C:\temp\duplicates\Microsoft.Exchange.WebServices.dll
        VERBOSE: Loading module from path ‘C:\temp\duplicates\Microsoft.Exchange.WebServices.dll’.
        Processing mailbox aliasname (
        VERBOSE: Set to trust all certificates
        VERBOSE: Using for impersonation
        VERBOSE: Looking up EWS URL using Autodiscover for
        VERBOSE: Using EWS on CAS https://servername
        VERBOSE: DeleteMode is MoveToDeletedItems
        VERBOSE: Processing Mail items
        C:\temp\duplicates\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
        At line:1 char:28
        + .\Remove-DuplicateItems.ps1 <<<< -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeleted -Mode
        Quick -MailboxOnly -Verbose
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

              • Hi, this is a fantastic script that has saved some heartache for me thank you.

                It is working fine if you use -credentials. But I can’t do that for all users so I need impersonation.

                Is there a way to get more verbose debug about why the mailbox information store cannot be opened?

                I have all the rights set up and am using powershell.
                -server consistently produces ‘couldn’t open mailbox information store’ ?


  23. Hello Michel,

    Situation long version: I have a mailbox of 14gb and with a ton of folders/e-mails, something went wrong with the migration and in some folders there are six duplicate items.

    I ran this script with the following command: .\Remove-DuplicateItems.ps1 -Mailbox -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest -Verbose

    – I then get a popup asking me for credentials and I enter the credentials of the specific mailbox and it started moving over 13k duplicated to the deleted items folder.

    After it was done doing this, i checked a couple of folders and it showed some still have duplicate items. Checking the log it shows it never told there are duplicates in there.

    Processing folder FOLDER NAME
    Procession folder FOLDER NAME

    when it did remove duplicates it shows:

    Processing folder FOLDER NAME
    Removing # items from FOLDER NAME

    I tried running the script again cause maybe that might help, but It removed 0 items the second time around.

    Any tips on getting the rest of the duplicates removed?

  24. Have you ever thought about writing an ebook or guest authoring on other blogs?
    I have a blog centered on the same ideas you discuss and would
    really like to have you share some stories/information. I know
    my viewers would value your work. If you are even remotely interested, feel free to send me an email.

  25. Hi Michel:

    we have a problem w/CRM synching – it is creating dup contacts in Outlook and we cannot figure out why. Is there a way to have your script be triggered when the dup comes in and delete it right then and there? thanks

    • No and yes. No, the script is not created with that in mind. Yes, you can use EWS to subscribe to a folder and trigger actions when changes occur. I have something similar in the pipeline, now I only need to find some time or clone myself :)

      • Let me ask this – I am hoping to use your script to remove dups from our Outlook clients, but is there a way your script can be modified to remove dups from the CRM database master from which the dups originated?
        thank you.

        • That sounds like a CRM/SQL operation – you may want to check with the vendor. Removing duplicate tuples in SQL is not a problem, maintaining referential integrity and consistency might me.

  26. Hi, I also seem to be having some permission errors, but can’t seem to find the problem…..
    Working on an exchange 2007 server.
    Installed the EWS API and gave the user full control and impersonation rights:
    Get-Mailbox mailboxname | Add-ADPermission –User ADM_user –ExtendedRights ms-Exch-EPI-May-Impersonate
    Then ran the script:
    Remove-DuplicateItems.ps1 -Mailbox mailboxname -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server EWSServer

    Error is:
    .\Remove-DuplicateItems.ps1 : Can’t access mailbox
    information store
    At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<<< -Mailbox mailboxname -Type Calendar -Imperson
    ation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio

    Could you please help?

  27. This script is a thing of beauty. I work in a shop heavy with Mac users. At some point in the past many of them were hit but the Outlook for Mac/iOS/sync/duplicate bug. I had one user with over 1.5 million items. I used ODIR and it worked well but it was a very tedious process to migrate messages into folders in batches <10,000 messages so that ODIR could work without crashing. This script just deleted 580146 duplicates from one mailbox while I slept. Thank you!

    We do have a 3rd party archiving solution that changes the appends the IPM.NOTE class with an extension. In this case there are 2 of each message as the sizes and classes are different. Is there an easy add to the script to compare 2 separate message classes? IPM.Note and IPM.Note.ExShortcut.owa and only keep the shortcut version?

  28. Pingback: Removing Duplicate Items from a Mailbox - League Team

  29. Hi Michel,
    The script sounds great, but I just can’t get past the “Can’t access mailbox information store” error.

    I’m on an Exchange 2007 environment here and I need to remove duplicate contacts (50k+) from a single mailbox (i.e. “contoso\user”)

    What I’ve done:

    1. created new account “contoso\me-tst” with no mailbox and no administrative privileges

    2. granted ms-exch-epi-impersonation permission on all CAS servers to “contoso\me-tst”

    Get-ExchangeServer | where {$_.IsClientAccessServer -eq $TRUE} | ForEach-Object {Add-ADPermission -Identity $_.distinguishedname -User (Get-User -Identity me-tst | select-object).identity -extendedRight ms-Exch-EPI-Impersonation}

    3. granted ms-exxch-epi-may-impersonate an fullaccess permission on “contoso\user” mailbox to “contoso\me-tst”

    Add-ADPermission -Identity user -User me-tst -extendedRight ms-Exch-EPI-May-Impersonate

    4. “$a = get-credential” -> “contoso\me-tst” & password

    5. executed script -> scrip fails with “Can’t access mailbox information store” / “(401)Unauthorized”

    [PS] D:\Scripting>.\Remove-DuplicateItems.ps1 -mailbox user -type contacts -retain Oldest -server CAS001.contoso.local -deletemode harddelete -credentials $a -mailboxonly -mode full -impersonation -verbose
    VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
    Processing mailbox user (user@contoso.local)
    VERBOSE: Set to trust all certificates
    VERBOSE: Using credentials contoso\me-tst
    VERBOSE: Using user@contoso.local for impersonation
    VERBOSE: Using Exchange Web Services URL https://CAS001.contoso.local/EWS/Exchange.asmx
    VERBOSE: DeleteMode is harddelete
    VERBOSE: Processing contacts items
    D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<> Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

    – Permissions are there
    https://CAS001.contoso.local/EWS/Exchange.asmx is accessible

    The script fails when first getting the root folder ($RootFolder= … line530 and on).

    Is it not weird that is says “VERBOSE: Using user@contoso.local for impersonation”? should this not be something like “Impersonating user@contoso.local” or “using me-tst@contoso.local“?

    Do you have any idea what could be wrong?

    Thank you!


    • 99% permissions / firewall / EWS / CAS config. Take into account waiting period for new permissions to become effective.
      ps: Wording thing. When using impersonation, you are effectively using the account specified (see line above mentioning account used for impersonation).

      • The waiting period is not it. I have wated considerably longer than it takes to replicate the permissions.

        … oh wait … as we speak …

        it was a certificate issue. I typed the external URL as the Server and now it works! AWESOME!

      • And now I get this:
        VERBOSE: Searching for folder class
        VERBOSE: Processing folder Contactpersonen
        D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
        At line:1 char:28
        + .\Remove-DuplicateItems.ps1 <<<< -mailbox user -type contacts -retain Oldest -server email.sdr.rot
        $a -mailboxonly -mode full -impersonation -verbose
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.p

        Exception calling "FindItems" with "2" argument(s): "The request failed. The operation has timed out"

        since it just worked on another mailbox i'm thinking that it is simply taking too long to collect the data (55k+ contacts). Is there a way around this?

        • Good one. For change EWS timeout, just below:
          $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
          $EwsService.Timeout= [System.Int32]::MaxValue

  30. Hi, I have this working against mailboxes in O365 but am finding that after it has completed, there are still duplicates, not as many but still lots.
    When it runs, it says it has deleted loads of duplicates but then when I check using OWA, there are still duplicates.
    How does it determine if the messages are duplicates or not? Is there a different detection method?

    • Default it uses a MAPI property named PidTagSearchKey to identify duplicates. However, depending on how those duplicates got in the mailbox, that property might not match. In those situations, you can use mode ‘Full’ which uses a predefined set of properties to identify duplicates. Be advised that some 3rd party products (i.e. archiving) may modify those attributes or contents – changing item size – in which case the properties will not match up. In those cases, only option is to alter the set of attributes in the script.

      • Hi, thanks for the reply.
        I did run it with mode Full and checked a couple of messages and see they have the same Message-ID, so I thought I’d try removing the other conditions from the criteria but thought I’d check running mode quick first to see what happens, and voila! It worked like a charm; no more duplicates :) Thanks for your help.

        • Hi, also, I’m trying to run this against 80ish O365 mailboxes but can’t get the csv to work. I setup impersonation in O365 but when running the script it fails on autodiscovery. Is there a way to test if impersonation is working correctly? I followed your impersonation guide to set it up.
          I tried without impersonation by adding the admin account to have Full Access permissions to all the mailboxes but that also didn’t work, same autodiscover/mailbox access error.
          I also tried adding the -Credentials (Get-Credential) switch, then when prompted entered the credentials for the first mailbox in the csv but then those credentials were used for all subsequent mailboxes also, and although it appeared to be running through all the mailboxes, I noticed it was actually just doing the first mailbox over and over. Is there a way to make it ask for the credentials for each of the mailboxes in the csv? It’ll be time consuming to enter them all, but still quicker than running the Remove-Duplicates command on every mailbox.

          • If autodiscover is not resolving properly from your internal network (maybe running split DNS & Hybrid ?), you can use Server to specify the external Office 365 URL, e.g. But, to be honest, I never have to use that when Autodiscover and DNS is set up correctly, and the request is not blocked. That nice thing with impersonation is that you only need to arrange for 1 account which can impersonate the mailbox owners when accessing those mailboxes in that CSV.

            • Thanks for the help but I’ve resorted to creating a script that runs the delete-duplicates.ps1 for each individual mailbox which it prompts for the credentials for each. It’s a long-winded process but seems to be working.
              I did spend a few hours trying to get it working using impersonation and delegated access but couldn’t make it work.
              Autodiscover is working fine and I’m working from home, so no split DNS in the mix or hybrid setup. I tried using -server but still same issue. I have no outbound firewall so don’t think the request could be blocked, It works fine when running on a single mailbox when I put in -Credentials (get-credentials) but doesn’t work if I try and use impersonation or delegated permissions. If I try using the -credentials switch and put in the admin account, which has full access, with inheritance, it still can’t connect to the mailbox with error about accessing the information store.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s