Removing Duplicate Items from a Mailbox

powershellFor those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

  • Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
  • Importing existing data. Accidental import from – for example – a PST file to a mailbox  can lead to duplicate entries.

image

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange 2007 or later and Exchange Web Services (EWS) Managed API 1.2 (or later) which you can download here. Alternatively, when you already got the EWS Managed API Microsoft.Exchange.WebServices.DLL somewhere, you can copy it to the same folder where the script is located and it will be picked up. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation for Exchange 2010 using RBAC, see this article or check here for details on how to configure impersonation for Exchange 2007.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [-Mailbox] <String> [[-Type] <String>] [-Server <String>] [-Impersonation] [-DeleteMode <String>] [-Mode <String>][-WhatIf] [-Confirm] [<CommonParameters>]

A quick walk-through on the parameters and switches:

  • Mailbox is the name of the mailbox to process;
  • Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default);
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover;
  • When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used;
  • DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
  • Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:
ItemClass Criteria
Contacts File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone
Distribution List FileAs, Number of Members
Calendar Subject, Location, Start & End Date
Task Subject, Start Date, Due Date, Status
Note Contents, Color
Mail Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender
Other Subject, DateTimeReceived

Few notes:

  • When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
  • When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
  • When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
  • For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.

So, suppose you want to remove  duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Mailbox <MailboxID> -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

image

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Mailbox field. An example of how the CSV could look:

Mailbox
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 -DeleteMode HardDelete -Impersonation

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision History
See TechNet Gallery page.

32 thoughts on “Removing Duplicate Items from a Mailbox

  1. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 20, 2013 | NeWay

  2. Pingback: NeWay Technologies – Weekly Newsletter #48 – June 21, 2013 | NeWay

  3. First, this is a VERY cool script, so thank you. But I can’t get it to work, and am hoping you can help. I get an error stating “Cant access mailbox information store”
    i’m running this from a management server (not an Exchange box). The account I’m using is an Exchange org admin. Thanks!!

      • Thank you, I had to grant impersonation rights. At first I granted Full Access, but I guess it didn’t work. I’m running Exchange 2010, in case anyone else has this issue. Thanks again for a VERY useful script!

  4. Thank you for your awesome script!
    It has been very useful.
    Any chance to run it in Online Archive mailboxes?
    Thank you!
    JP

  5. It looks like its running fine, but don’t find the dubs that i made by copying mails, why is this? and why does it not run on custom folders?

    • Depending on the way you copied, it could be the “duplicates” have a different PidTagSearchKey. Try running it in Full mode instead of the default Quick Mode (-Mode), in which case it’ll use a predefined set of attributes depending on the type of items to find matching items (see article for which attributes are checked). By default it will scan all folder types (-Type), looking for duplicates on a per folder basis.

      • Thanks, I found the solution. it does not work fully before i configure the impersonation rights. Im running it from a workstation with web services 2.0 against a exchange 2010 installation. Wonderful script you have made!

  6. I receive an error with the script when installed on Exchange 2012. I do have the dll in the folder with the ps1, and have the EWS installed. I downloaded again and verified the download.

    …..Remove-DuplicateItems.ps1:349 char:156
    + … ationsMode and $AffectedTaskOccurences:$AffectedTaskOccurences”
    + ~~~~~~~~~~~~~~~~~~~~~~~~
    Variable reference is not valid. ‘:’ was not followed by a valid variable name character. Consider using ${} to delimit the name.
    + CategoryInfo : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : InvalidVariableReferenceWithDrive

  7. I get the following error :
    You cannot call a method on a null-valued expression.
    At C:\psscripts\Remove-DuplicateItems.ps1:271 char:82
    + $key= $Item.DateTimeReceived.ToString <<<< ()
    + CategoryInfo : InvalidOperation: (ToString:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

  8. Hello,

    When I run the script, everything looks fine, except for the fact that is removes 0 items. It’s for shure that the mailbox where I try it on is full of duplicate items. This is the best way I found on the net to remove duplicate items, but doesn’t work for me. Please Help!

  9. Hi
    Wonderful script!
    I´m doing it on exchange 2013 and has change some of the rows to get it working.
    1. First probably a typo:
    … and $AffectedTaskOccurences:$AffectedTaskOccurences”
    … and AffectedTaskOccurences:$AffectedTaskOccurences”
    2. Change all text: Exchange2007_sp1 to Exchange2013
    3. On 2 occurences replace:
    $ItemIds= [activator]::createinstance(([type]‘System.Collections.Generic.List`1′).makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
    with:
    $type = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
    $type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
    $Itemids = [Activator]::CreateInstance($type)

    Now the script works! But still getting an error at the end:
    C:\root\Remove-DuplicateItems.ps1 : Problem processing mailbox it (it@google.com)
    At line:1 char:1
    + .\Remove-DuplicateItems.ps1 -mailbox it -type all -impersonation -deletemode har …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    I Think it fails the error check at the end but cannot solve it.
    $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
    If(! ( Process-Mailbox $RootFolder)) {…

    Any ideas?

    /MrTbone

    • 1) Thanks. Display only, won’t affect script;
      2) No. Defines minimum level of EWS support required.
      3) PowerShell v3 thing – fixed in Fix-MailboxFolder script, still need to propagate to this script.

      Regarding the message, make sure you have permissions (impersonation – preferred – or otherwise).

      • Hi

        The script is doing its job with previous changes. duplicates are deleted.
        I have the impersonation rights.
        But I get another error message in the middle of the process to:

        Process-Mailbox : Detected problem removing items (loop)
        At C:\root\Remove-DuplicateItems.ps1:458 char:16
        + If(! ( Process-Mailbox $RootFolder)) {
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

        This leads me to the lines:
        $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
        If(! ( Process-Mailbox $RootFolder)) {…

        Any surgestions?
        /MrTbone

        • The script checks if the number of items to be deleted are in actually deleted. However, with larger batches Exchange may still be busy deleting items (we hand off a list of items to delete) so numbers don’t add up. In such cases I’d recommend using smaller batches ($MaxItemBatchSize= 100). I may parametrize this in future versions.

          • Hi!
            Thanks for the Quick reply.
            This script is so cool!!!!

            had to change $type to $typ cause it was used earlier in script:
            $typ = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
            $typ = $typ.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
            $Itemids = [Activator]::CreateInstance($typ)

            After this and raising the max items all seems fine.

            Thanks!
            /MrT

  10. Is there anyway to use this with office365/exchange online, or will it only work with an in house exchange

    As we have just moved to office365 from another hosted exchange and using the built in Microsoft tools to move the mail have managed to end up with every mailbox with duplicate items

    Thanks
    Jamie

    • Hi I have tried to use this against office365 but get the following error

      PS C:\Users\cnl_admin\Desktop> .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type
      Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose

      Security warning
      Run only scripts that you trust. While scripts from the internet can be useful
      this script can potentially harm your computer. Do you want to run
      C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1?
      [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): r
      VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web
      Services\2.0\\Microsoft.Exchange.WebServices.dll
      Processing mailbox jamiec
      VERBOSE: Set to trust all certificates
      VERBOSE: Looking up EWS URL using Autodiscover for jamie.cox@domain.com
      C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1 : A positional parameter
      cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl”
      with “2″ argument(s): “The Autodiscover service couldn’t be located.”‘.
      At line:1 char:1
      + .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type Mail -DeleteMode
      MoveToDeleted …
      + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      ~~~
      + CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1]
      , ParameterBindingException
      + FullyQualifiedErrorId : PositionalParameterNotFound,Remove-DuplicateItem
      s.ps1

  11. Hi Michel

    I have now run this with great success in Exchange 2013. But the largest mailboxes seems to fail with the largest folders.
    inbox and sent items in the largest mailboxes get:

    Exception calling “FindItems” with “2″ argument(s): “Unexpected end of file while parsing Name has occurred. Line 1, po
    sition 1244584.”
    At C:\root\Remove-DuplicateItems.ps1:253 char:25
    + $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : XmlException

    Have tried the Maxfolder and Maxitem but no success.
    Could it be some limit i EWS config?

    I also get an error deleteing in a few folders:
    Process-Mailbox : Detected problem removing items (loop)
    At C:\root\Remove-DuplicateItems.ps1:459 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
    any ideas?

    /MrT

    • 1) Do you have a number so I can try reproduce the problem?
      2) I check if the number of items returned have decreased by the number of items removed. However, it’s asynchronous and Exchange – especially with large counts – may still be removing things in in the background. Suggest using low MaxItems.

  12. Would REALLY be great if this worked against Archive mailboxes. I am about to undertake a project where I have a huge migration with import from .pst, we had hoped to go to archive mailbox, but cant, it will take an extra step to do this. But testing on regular Exchange 2013 mailbox works like a champ. Thanks

  13. Hi Micheal,

    I testing your script but this send me a this message….
    [PS] C:\Documents and Settings\acweinzierll\Desktop>Remove-DuplicateItems.ps1 -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
    The term ‘Remove-DuplicateItems.ps1′ is not recognized as a cmdlet, function, operable program, or script file. Verify
    the term and try again.
    At line:1 char:26
    + Remove-DuplicateItems.ps1 <<<< -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mo
    de Full -Verbose

    I have Exchange 2007

    Can YOu help Me… Thanks…

  14. VERIFIED TO WORK WELL ON EXCHANGE 2010 SP3! – 3/10/2014

    Very powerful script! Once I installed Exchange Web Services (EWS) Managed API 1.2 and assigned the Hygiene Management role to the account I was running the scripts under I had no problems!

    Be sure to follow the formatting provided in the examples as I noticed weird issues when I specified a different order of the arguments than that of the examples.

    Below is what worked best for my environment!!!

    ./Remove-DuplicateItems.ps1 -Mailbox -johnsmith -Type Mail -Impersonation -DeleteMode HardDelete -Mode Full -Verbose

    **Used to remove duplicates on imported mail that orginated from Google Apps via Outlook 2007 and GASMO**

  15. Hello! Congrats for your script. It’s really fantastic!! I’ve added impersonation rights, everything runs OK on most of mailboxes. But I have a little problem in one of them. The scripts crashes when starts analyzin the folder “Sent Items”. I supose there is a problem on this folder, otherwise the script wouldn’t run in the other mailboxes. I’m wondering if there is any possibility to define which folders analyze or better, add a folder exception… Is it possible? If not, have you ever been in the same situation, same problem? Many thanks!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s