Removing Duplicate Items from a Mailbox

Posted on June 21, 2013 by Michel de Rooij

Latest version: 2.41, April 18th, 2023

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
Importing existing data. Accidental import from – for example – a PST file to a mailbox can lead to duplicate entries.

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange Web Services (EWS) Managed API and for OAuth authentication the Microsoft Authentication Library (MSAL) libraries. You can install these packages from NuGet, or place their DLL’s in the same folder as the script. For an example of how to install EWS.Managed.Api from NuGet, see this article; for MSAL follow the same process but with the package titled ‘Microsoft.Identity.Client’.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions when using Basic Authentication; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post. Using a registered app with OAuth is always through Impersonation.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [[-Identity] ] [[-Type] ] [-Retain ] [-Server ] [-Impersonation] [-DeleteMode ] [-Credentials ] [-Mode ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-PriorityFolders <String[]>] [-NoSize] [-CleanupMode] [-NoProgressBar] [-Force] [-WhatIf] [-Confirm] [-Secret] [-CertificateThumbprint] [-CertificateFile] [-CertificatePassword] [-TenantId] [-ClientId] [-TrustAll] [-ExchangeSchema <String>] [-NoSCP]

A quick walk-through on the parameters and switches:

Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:

ItemClass	Criteria
Contacts	File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List	FileAs, Number of Members, Size
Calendar	Subject, Location, Start & End Date, Size
Task	Subject, Start Date, Due Date, Status, Size
Note	Contents, Color, Size
Mail	Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other	Subject, DateTimeReceived

MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter when running against mailboxes on Exchange Server 2007.
ArchiveOnly specifies you only want to process personal archives of specified users.
IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
CleanupMode specifies to cleanup duplicates per folder (Folder, default), the whole mailbox (Mailbox), or multiple mailboxes (MultiMailbox, identities specified using Identity). The first unique item encountered will be retained. For Mailbox-level cleanup, PriorityFolders can be used to give priority to retaining items in specified folders before those found in other folders.
PriorityFolders specifies which folders have priority over other folders, identifying items in these folders first when using MailboxWide mode. Usage of wildcards and well-known folders is identical to IncludeFolders.
NoSize tells script to not use size to match items in Full mode.
NoProgressBar prevents displaying a progress bar as folders and items are being processed.
Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.
TrustAll can be used to accept all certificates, e.g. self-signed certificates or when accessing Exchange using endpoint with a different certificate.
ExchangeSchema can be used to specify the Exchange schema to use when connecting to Exchange server or Exchange Online. Defaults to Exchange2013_SP1 or Exchange2016 when -Server is specified and is ‘outlook.office365.com’ (Exchange Online endpoint).
NoSCP to skip SCP lookups in Active Directory for Autodiscover.

For authentication, the following parameters are available:

Credentials specifies credentials to use for Basic Authentication.
TenantId specifies the identity of the Tenant (OAuth)
ClientId specifies the Id of the registered application (OAuth).
CertificateThumbprint specifies the thumbprint of the certificate from personal store to use for authentication (OAuth).
CertificateFile specifies the external certificate file (pfx) to use for authentication (OAuth). This certificate needs to contain a private key; the registered application needs to contain the certificate’s public key.
CertificatePassword optionally specifies the password to use with the certificate file (OAuth).
Secret specifies the secret to use with the application (OAuth).

Few notes:

When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter	Matches
FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA
Folder*	\TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\Folder	\TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\FolderA\	\TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So, suppose you want to remove duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

Alternative, you can use an e-mail address and specify credentials. This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Identity olrik@office365tenant.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

A more complex example using IncludeFolders, ExcludeFolders and PriorityFolders:

$Credentials= Get-Credential
 .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '#Inbox#\*','\Projects\*' -ExcludeFolders 'Keep Out' -PriorityFolders '*Important*' -CleanupMode Mailbox

This will remove duplicate items from the specified mailbox in Office365, using the following options:

Fixed Server FQDN – bypassing AutoDiscover.
Limits operation against the Well-Known Inbox folder, top Projects folder, and all of their subfolders.
Excluding any folder named Keep Out.
Duplicates are checked over the whole mailbox.
Priority is given to folders containing the word Important, causing items in
those folders to be kept over items in other folders when duplicates are found.

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 ..

Download
The script is available on GitHub here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

484 thoughts on “Removing Duplicate Items from a Mailbox”

Pingback: NeWay Technologies – Weekly Newsletter #48 – June 20, 2013 | NeWay
Pingback: NeWay Technologies – Weekly Newsletter #48 – June 21, 2013 | NeWay
CL on July 24, 2013 at 9:50 PM said:

First, this is a VERY cool script, so thank you. But I can’t get it to work, and am hoping you can help. I get an error stating “Cant access mailbox information store”
i’m running this from a management server (not an Exchange box). The account I’m using is an Exchange org admin. Thanks!!

LikeLike

Reply ↓
- Michel de Rooij on July 24, 2013 at 10:47 PM said:
  
  Sounds like a permission issue; check permissions or configure impersonation (see links in article for instructions)
  
  LikeLike
  
  Reply ↓
  - CL on July 25, 2013 at 2:06 PM said:
    
    Thank you, I had to grant impersonation rights. At first I granted Full Access, but I guess it didn’t work. I’m running Exchange 2010, in case anyone else has this issue. Thanks again for a VERY useful script!
    
    LikeLike
    
    Reply ↓
Joao Pedro on August 24, 2013 at 7:25 PM said:

Thank you for your awesome script!
It has been very useful.
Any chance to run it in Online Archive mailboxes?
Thank you!
JP

LikeLike

Reply ↓
- Michel de Rooij on August 24, 2013 at 9:46 PM said:
  
  Option to run against archives has been requested before. It’s on the ‘to do’ list, unfortunately no ETA yet.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on April 1, 2015 at 12:22 PM said:
    
    (Implemented)
    
    LikeLike
    
    Reply ↓
Danni on August 29, 2013 at 7:21 AM said:

It looks like its running fine, but don’t find the dubs that i made by copying mails, why is this? and why does it not run on custom folders?

LikeLike

Reply ↓
- Michel de Rooij on August 29, 2013 at 8:42 AM said:
  
  Depending on the way you copied, it could be the “duplicates” have a different PidTagSearchKey. Try running it in Full mode instead of the default Quick Mode (-Mode), in which case it’ll use a predefined set of attributes depending on the type of items to find matching items (see article for which attributes are checked). By default it will scan all folder types (-Type), looking for duplicates on a per folder basis.
  
  LikeLike
  
  Reply ↓
  - Danni on August 29, 2013 at 8:47 AM said:
    
    Thanks, I found the solution. it does not work fully before i configure the impersonation rights. Im running it from a workstation with web services 2.0 against a exchange 2010 installation. Wonderful script you have made!
    
    LikeLike
    
    Reply ↓
Matt Shooshtari on September 25, 2013 at 1:05 AM said:

I receive an error with the script when installed on Exchange 2012. I do have the dll in the folder with the ps1, and have the EWS installed. I downloaded again and verified the download.

…..Remove-DuplicateItems.ps1:349 char:156
+ … ationsMode and $AffectedTaskOccurences:$AffectedTaskOccurences”
+ ~~~~~~~~~~~~~~~~~~~~~~~~
Variable reference is not valid. ‘:’ was not followed by a valid variable name character. Consider using ${} to delimit the name.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : InvalidVariableReferenceWithDrive

LikeLike

Reply ↓
Bis on September 26, 2013 at 5:49 PM said:

I get the following error :
You cannot call a method on a null-valued expression.
At C:\psscripts\Remove-DuplicateItems.ps1:271 char:82
+ $key= $Item.DateTimeReceived.ToString <<<< ()
+ CategoryInfo : InvalidOperation: (ToString:String) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull

LikeLike

Reply ↓
Maarten on September 30, 2013 at 1:54 PM said:

Hello,

When I run the script, everything looks fine, except for the fact that is removes 0 items. It’s for shure that the mailbox where I try it on is full of duplicate items. This is the best way I found on the net to remove duplicate items, but doesn’t work for me. Please Help!

LikeLike

Reply ↓
- Michel de Rooij on September 30, 2013 at 2:15 PM said:
  
  First of all, it only scans per folder. Second, it’s likely to be permission related.
  
  LikeLike
  
  Reply ↓
Torbjörn Granheden on November 28, 2013 at 1:57 PM said:

Hi
Wonderful script!
I´m doing it on exchange 2013 and has change some of the rows to get it working.
1. First probably a typo:
… and $AffectedTaskOccurences:$AffectedTaskOccurences”
… and AffectedTaskOccurences:$AffectedTaskOccurences”
2. Change all text: Exchange2007_sp1 to Exchange2013
3. On 2 occurences replace:
$ItemIds= [activator]::createinstance(([type]’System.Collections.Generic.List`1′).makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
with:
$type = (“System.Collections.Generic.List”+’`’+”1”) -as “Type”
$type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
$Itemids = [Activator]::CreateInstance($type)

Now the script works! But still getting an error at the end:
C:\root\Remove-DuplicateItems.ps1 : Problem processing mailbox it (it@google.com)
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -mailbox it -type all -impersonation -deletemode har …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
I Think it fails the error check at the end but cannot solve it.
$RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
If(! ( Process-Mailbox $RootFolder)) {…

Any ideas?

/MrTbone

LikeLike

Reply ↓
- Michel de Rooij on November 28, 2013 at 2:08 PM said:
  
  1) Thanks. Display only, won’t affect script;
  2) No. Defines minimum level of EWS support required.
  3) PowerShell v3 thing – fixed in Fix-MailboxFolder script, still need to propagate to this script.
  
  Regarding the message, make sure you have permissions (impersonation – preferred – or otherwise).
  
  LikeLike
  
  Reply ↓
  - Torbjörn Granheden on November 28, 2013 at 7:13 PM said:
    
    Hi
    
    The script is doing its job with previous changes. duplicates are deleted.
    I have the impersonation rights.
    But I get another error message in the middle of the process to:
    
    Process-Mailbox : Detected problem removing items (loop)
    At C:\root\Remove-DuplicateItems.ps1:458 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
    
    This leads me to the lines:
    $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
    If(! ( Process-Mailbox $RootFolder)) {…
    
    Any surgestions?
    /MrTbone
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 28, 2013 at 8:58 PM said:
      
      The script checks if the number of items to be deleted are in actually deleted. However, with larger batches Exchange may still be busy deleting items (we hand off a list of items to delete) so numbers don’t add up. In such cases I’d recommend using smaller batches ($MaxItemBatchSize= 100). I may parametrize this in future versions.
      
      LikeLike
      
      Reply ↓
      - Torbjörn Granheden on November 29, 2013 at 8:39 AM said:
        
        Hi!
        Thanks for the Quick reply.
        This script is so cool!!!!
        
        had to change $type to $typ cause it was used earlier in script:
        $typ = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
        $typ = $typ.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
        $Itemids = [Activator]::CreateInstance($typ)
        
        After this and raising the max items all seems fine.
        
        Thanks!
        /MrT
        
        LikeLike
Jamie Cox on December 3, 2013 at 9:32 PM said:

Is there anyway to use this with office365/exchange online, or will it only work with an in house exchange

As we have just moved to office365 from another hosted exchange and using the built in Microsoft tools to move the mail have managed to end up with every mailbox with duplicate items

Thanks
Jamie

LikeLike

Reply ↓
- Michel de Rooij on December 3, 2013 at 10:07 PM said:
  
  Currently only on-prem; making the scripts suited for and test them with o365 is on the to do list.
  
  LikeLike
  
  Reply ↓
  - Dipin Kannan on March 23, 2016 at 7:36 AM said:
    
    I am also having issue in Office 365. It will be very helpful if the script works for Office 365 also. Thanks
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on March 23, 2016 at 3:29 PM said:
      
      It works on Office 365 – use the e-mail address and provide credentials.
      
      LikeLike
      
      Reply ↓
- Jamie Cox on December 3, 2013 at 10:21 PM said:
  
  Hi I have tried to use this against office365 but get the following error
  
  PS C:\Users\cnl_admin\Desktop> .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type
  Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose
  
  Security warning
  Run only scripts that you trust. While scripts from the internet can be useful
  this script can potentially harm your computer. Do you want to run
  C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1?
  [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): r
  VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web
  Services\2.0\\Microsoft.Exchange.WebServices.dll
  Processing mailbox jamiec
  VERBOSE: Set to trust all certificates
  VERBOSE: Looking up EWS URL using Autodiscover for jamie.cox@domain.com
  C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1 : A positional parameter
  cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl”
  with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
  At line:1 char:1
  + .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type Mail -DeleteMode
  MoveToDeleted …
  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ~~~
  + CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1]
  , ParameterBindingException
  + FullyQualifiedErrorId : PositionalParameterNotFound,Remove-DuplicateItem
  s.ps1
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 9, 2014 at 2:28 PM said:
    
    Version 1.3 runs against Office 365!
    
    LikeLike
    
    Reply ↓
    - Sven Hoelbling on August 26, 2014 at 2:14 AM said:
      
      Hi, I get exactly the same error:
      
      A positional parameter cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
      
      and have seen your reply which refers to version 1.3. Where can I find the latest version of your PS script?
      
      Kind regards,
      Sven Hoelbling
      
      LikeLike
      
      Reply ↓
      - Sven Hoelbling on August 26, 2014 at 3:28 AM said:
        
        Sorry, I noticed that I already had version 1.4, but I still get this error.
        Can you let me please know what I’m missing?
        All mailboxes are located in Office 365, but it is a hybrid deployment with on-premise Exchange 2013 SP 1 servers. When I point the script to these servers using the -server switch the script returns:
        Can’t access mailbox information store
        Shouldn’t it use autodiscover as Outlook does (in a hybrid depoyment the DNS entry for autodiscover points to the same onpremise CAS server)
        
        Kind regards,
        Sven Hoelbling
        
        LikeLike
      - Michel de Rooij on August 26, 2014 at 7:58 AM said:
        
        1) You will find the latest version of the script here.
        2) Hybrid shouldn’t matter. If your internal DNS is set up correctly for Autodiscover (or using domain-joined client?). Can’t access mailbox store when overriding the EWS web service URL might possibly indicate a permission issue (or other blockade, e.g. firewall).
        
        LikeLike
      - Rob Head on April 1, 2015 at 11:49 AM said:
        
        Hi,
        
        I have version 1.4 but it still exhibits the same problem. I’ve confirmed my client has full connectivity to O365 and that auto discovery is working fine for the mailbox. Has any progress been made resolving this problem?
        
        Thanks.
        
        LikeLike
      - Michel de Rooij on April 1, 2015 at 12:36 PM said:
        
        Just ran it succesfully against a hybrid deployment with Autodiscover and Impersonation. To exclude (Autodiscover) redirection from the equation, use “-Server outlook.office365.com” to address cloud-based mailboxes.
        
        LikeLike
Torbjörn Granheden on December 4, 2013 at 7:23 AM said:

Hi Michel

I have now run this with great success in Exchange 2013. But the largest mailboxes seems to fail with the largest folders.
inbox and sent items in the largest mailboxes get:

Exception calling “FindItems” with “2” argument(s): “Unexpected end of file while parsing Name has occurred. Line 1, po
sition 1244584.”
At C:\root\Remove-DuplicateItems.ps1:253 char:25
+ $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : XmlException

Have tried the Maxfolder and Maxitem but no success.
Could it be some limit i EWS config?

I also get an error deleteing in a few folders:
Process-Mailbox : Detected problem removing items (loop)
At C:\root\Remove-DuplicateItems.ps1:459 char:16
+ If(! ( Process-Mailbox $RootFolder)) {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
any ideas?

/MrT

LikeLike

Reply ↓
- Michel de Rooij on December 4, 2013 at 11:24 AM said:
  
  1) Do you have a number so I can try reproduce the problem?
  2) I check if the number of items returned have decreased by the number of items removed. However, it’s asynchronous and Exchange – especially with large counts – may still be removing things in in the background. Suggest using low MaxItems.
  
  LikeLike
  
  Reply ↓
  - Torbjörn Granheden on December 4, 2013 at 11:31 AM said:
    
    Hi
    Not so happy today!
    10% of the mailboxes has not deleted their duplicates. Still there
    Instead they have deleted all mail from all folders between a week back and a year back
    Cannot se a pattern. but it´s kaos!
    😦 MrT
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on December 4, 2013 at 11:41 AM said:
      
      Sorry to hear. Don’t know what happened, but you can – as a safety measure – utilize the MoveToDeletedItems option and run Full mode (where specific attributes are matched instead of relying on the Exchange generated PidTagSearchKey key which will be identical for copied objects, see http://msdn.microsoft.com/en-us/library/office/cc815908.aspx)
      
      I don’t do anything regarding date filtering ATM, so I wonder what happened since you mention a specific period.
      
      LikeLike
      
      Reply ↓
Robert 68 on December 8, 2013 at 11:28 AM said:

Hi Michel,

Does it also work in Public Folders?

Thanks,
Robert

LikeLike

Reply ↓
terry on January 7, 2014 at 7:45 PM said:

Would REALLY be great if this worked against Archive mailboxes. I am about to undertake a project where I have a huge migration with import from .pst, we had hoped to go to archive mailbox, but cant, it will take an extra step to do this. But testing on regular Exchange 2013 mailbox works like a champ. Thanks

LikeLike

Reply ↓
- Michel de Rooij on January 7, 2014 at 8:16 PM said:
  
  Thanks, I’ll put it on the to do list (also for the script to remove class X items, eg stubs)
  
  LikeLike
  
  Reply ↓
Cristian Weinzierl on February 20, 2014 at 9:08 PM said:

Hi Micheal,

I testing your script but this send me a this message….
[PS] C:\Documents and Settings\acweinzierll\Desktop>Remove-DuplicateItems.ps1 -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
The term ‘Remove-DuplicateItems.ps1’ is not recognized as a cmdlet, function, operable program, or script file. Verify
the term and try again.
At line:1 char:26
+ Remove-DuplicateItems.ps1 <<<< -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mo
de Full -Verbose

I have Exchange 2007

Can YOu help Me… Thanks…

LikeLike

Reply ↓
Dustin Day on March 10, 2014 at 7:14 PM said:

VERIFIED TO WORK WELL ON EXCHANGE 2010 SP3! – 3/10/2014

Very powerful script! Once I installed Exchange Web Services (EWS) Managed API 1.2 and assigned the Hygiene Management role to the account I was running the scripts under I had no problems!

Be sure to follow the formatting provided in the examples as I noticed weird issues when I specified a different order of the arguments than that of the examples.

Below is what worked best for my environment!!!

./Remove-DuplicateItems.ps1 -Mailbox -johnsmith -Type Mail -Impersonation -DeleteMode HardDelete -Mode Full -Verbose

**Used to remove duplicates on imported mail that orginated from Google Apps via Outlook 2007 and GASMO**

LikeLike

Reply ↓
Gerard IT on April 1, 2014 at 1:14 PM said:

Hello! Congrats for your script. It’s really fantastic!! I’ve added impersonation rights, everything runs OK on most of mailboxes. But I have a little problem in one of them. The scripts crashes when starts analyzin the folder “Sent Items”. I supose there is a problem on this folder, otherwise the script wouldn’t run in the other mailboxes. I’m wondering if there is any possibility to define which folders analyze or better, add a folder exception… Is it possible? If not, have you ever been in the same situation, same problem? Many thanks!!

LikeLike

Reply ↓
- Michel de Rooij on June 9, 2014 at 2:23 PM said:
  
  Not encountered it before, check permissions. Folder inclusion/exclusion is on the to-do list, no ETA.
  
  LikeLike
  
  Reply ↓
- Michel de Rooij on January 25, 2017 at 5:18 PM said:
  
  Version 1.6 introduces IncludeFolder and ExcludeFolder options
  
  LikeLike
  
  Reply ↓
  - Thomas juhl Olesen (@spinnetho) on February 9, 2017 at 7:03 PM said:
    
    How do i explude more than one folder , I tried with -excludefolder “Folder 1”, “Folder 2” but it only works when putting one single foldername
    
    LikeLike
    
    Reply ↓
    - Thomas juhl Olesen (@spinnetho) on February 9, 2017 at 7:04 PM said:
      
      OH and GREAT script by the way !! very helpfull
      
      LikeLike
      
      Reply ↓
John Anson on May 25, 2014 at 12:12 AM said:

Hi Michel,
Thank you for your wonderful script.

I used the script to remove duplicate contacts from an Exchange 2013 SP1 folder where most of them were duplicates (800 unique contacts, 138,000 duplicates)

I got this error:
Process-Mailbox : Detected problem removing items (loop)
At .\Remove-DuplicateItems.ps1:470 char:16
+ If(! ( Process-Mailbox $RootFolder)) {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

The lines that cause this error start at line 369:
$OldOffSet= $ItemView.Offset
$ItemView.Offset+= $ItemSearchResults.Items.Count – $ItemIds.Count
If( $OldOffset -eq $ItemView.Offset) {
Write-Error “Detected problem removing items (loop)”
$ProcessingOK= $False
}
For me, most of the time the number of items deleted ($ItemIds.Count) was the same as the number of items in the batch ($ItemSearchResults.Items.Count), so this test condition was usually matching, and halting the script.
I removed the test, and Remove-DuplicateItems.ps1 ran ok.

Are there any side effects that might occur without this test?

PS: I also tested increasing the $MaxItemBatchSize to 1000 and it seemed to work ok.

LikeLike

Reply ↓
- Michel de Rooij on May 30, 2014 at 10:49 PM said:
  
  The removal is an asynchronous process, which may be an issue with systems that can’t keep up (fiddling with the batch size may work or re-running which isn’t ideal).
  A better, more robust method is on the to-do list.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 9, 2014 at 2:22 PM said:
    
    Version 1.3 does not have the asynchronous deletion issue 🙂
    
    LikeLike
    
    Reply ↓
John Anson on May 25, 2014 at 1:34 AM said:

Hi Michel
I would like to add Appointment RequiredAttendees to the mode=full duplicate-detection $key.
Does FindItems populate RequiredAttendees with the [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties view?
If not, what changes are needed to retrieve the RequiredAttendees property?
Thanks for your help,
John

LikeLike

Reply ↓
- Michel de Rooij on May 30, 2014 at 11:01 PM said:
  
  What’s in ‘FirstClassProperties’ depends on the type of item being queried, but I believe it is for calendar items.
  
  LikeLike
  
  Reply ↓
Tiernest on June 2, 2014 at 8:33 PM said:

Hello,

We are using this script to try to remove some duplicate contacts. We have a sync process that pushes out all of our contacts from a public folder to everyone’s contacts. The tool we use to sync isn’t great, so we have dupes. The script works great, but it is removing the most recently added contact and I’d like to remove the old one. Any ideas on modifiying the script to make this happen?

LikeLike

Reply ↓
- Michel de Rooij on June 5, 2014 at 12:53 PM said:
  
  Updated version 1.2 allows you to retain oldest or newest received item (previously was undetermined)
  
  LikeLike
  
  Reply ↓
Pingback: Script Updates | EighTwOne (821)
Pierre on June 11, 2014 at 10:00 AM said:

Hi Michel,

Great script and great that it now also works with EO / O365.
Only issue I have now is that it doesnt search through folders created by the user.

Any suggestions?

Cheers,
Pierre.

LikeLike

Reply ↓
- Michel de Rooij on June 11, 2014 at 10:05 AM said:
  
  Likely permissions issue. I have run it against several mailboxes, including my own in Office 365 and believe me, it has lots of user-defined folders.
  
  LikeLike
  
  Reply ↓
  - Pierre on June 11, 2014 at 3:14 PM said:
    
    You were of course correct!
    
    Thanks again.
    
    LikeLike
    
    Reply ↓
JB on June 17, 2014 at 11:50 AM said:

Hi,

I’m trying to run this script against an Office365 tenant that I’ve just migrated our customer to as the migration process created quite a lot of duplicate items (which we expected it might).

I’ve added the Impersonation rights and full access mailbox permissions to the user i’m running the script as but I just can get it to execute?

autodiscovering the EWS url seems to fail (even though autodiscover works fine when setting up outlook profiles etc.) and i’m not sure what format to enter the -server parameter as? I’ve tried the generic outlook.office365.com, I’ve tried the mailbox users exchange GUID@domain.com (as per the outlook profile) but nothing seems to work.

Any suggestions???

Many thanks in advance 🙂
James.

LikeLike

Reply ↓
CP on July 1, 2014 at 12:16 AM said:

I keep getting a “Can’t access mailbox information store At Line:1 char:28 + .\remove-duplicateitems.ps1 <<<< -Mailbox

I've got application impersonation set on the account

LikeLike

Reply ↓
- Michel de Rooij on July 1, 2014 at 12:49 AM said:
  
  Running it from PowerShell prompt (not EMS)?
  Can you please post the whole output or mail it (contact form)?
  
  LikeLike
  
  Reply ↓
  - CP on July 1, 2014 at 3:13 AM said:
    
    I’ve tried both, same results
    
    This is the script that I’m running….
    
    .\Remove-DuplicateItems.ps1 -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Quick -Verbose
    
    This is the output that I get…..
    
    VERBOSE: Loading C:\temp\duplicates\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loading module from path ‘C:\temp\duplicates\Microsoft.Exchange.WebServices.dll’.
    Processing mailbox aliasname (aliasname@email.com)
    VERBOSE: Set to trust all certificates
    VERBOSE: Using aliasname@email.com for impersonation
    VERBOSE: Looking up EWS URL using Autodiscover for aliasname@email.com
    VERBOSE: Using EWS on CAS https://servername
    VERBOSE: DeleteMode is MoveToDeletedItems
    VERBOSE: Processing Mail items
    C:\temp\duplicates\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
    At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<<< -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeleted -Mode
    Quick -MailboxOnly -Verbose
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 1, 2014 at 9:47 AM said:
      
      Check your EWS URL configuration (Get-WebServicesVirtualDirectory) in your environment.
      The EWS URL should be https:///EWS/Exchange.asmx.
      You can override it by pointing to a specific CAS box using -Server (the script will postfix the /EWS/Exchange.asmx)
      
      LikeLike
      
      Reply ↓
      - CP on July 1, 2014 at 6:25 PM said:
        
        Pointing to the correct server and EWS URL…still getting same error. Any other ideas?
        
        LikeLike
      - Michel de Rooij on July 2, 2014 at 1:51 PM said:
        
        Permissions, misconfiguration, software blocking access, mailbox/db inaccessible, proxy configuration ..
        
        LikeLike
      - kcitesting on August 16, 2014 at 12:24 PM said:
        
        Any update to this at all please? I too have this exact same issue. I’ve added the rights (full and impersonate) thanks
        
        LikeLike
      - kcitesting on August 16, 2014 at 12:34 PM said:
        
        Hi, this is a fantastic script that has saved some heartache for me thank you.
        
        It is working fine if you use -credentials. But I can’t do that for all users so I need impersonation.
        
        Is there a way to get more verbose debug about why the mailbox information store cannot be opened?
        
        I have all the rights set up and am using powershell.
        -server outlook.office365.com consistently produces ‘couldn’t open mailbox information store’ ?
        
        Thanks
        
        LikeLike
      - Michel de Rooij on August 20, 2014 at 6:56 PM said:
        
        Echoing the contents of $_.Exception should give you a reason/clue.
        
        LikeLike
Danny on July 2, 2014 at 3:35 PM said:

Hello Michel,

Situation long version: I have a mailbox of 14gb and with a ton of folders/e-mails, something went wrong with the migration and in some folders there are six duplicate items.

I ran this script with the following command: .\Remove-DuplicateItems.ps1 -Mailbox email@email.nl -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest -Verbose

– I then get a popup asking me for credentials and I enter the credentials of the specific mailbox and it started moving over 13k duplicated to the deleted items folder.

After it was done doing this, i checked a couple of folders and it showed some still have duplicate items. Checking the log it shows it never told there are duplicates in there.

Processing folder FOLDER NAME
Procession folder FOLDER NAME

when it did remove duplicates it shows:

Processing folder FOLDER NAME
Removing # items from FOLDER NAME

I tried running the script again cause maybe that might help, but It removed 0 items the second time around.

Any tips on getting the rest of the duplicates removed?

LikeLike

Reply ↓
- Michel de Rooij on July 2, 2014 at 8:03 PM said:
  
  It can not delete what it can not find / access. Add -Debug to see more of what it’s doing.
  
  LikeLike
  
  Reply ↓
  - Maicon Motta on August 12, 2015 at 4:30 PM said:
    
    I’am having the same problem, the script It’s working fine, but he cant detect duplicated items, I did the debug and i saw two items that suppost to be equals they have a the field “modified” diferentes.
    
    I’am trying to cleam those duplicated items because when i migrated e-mails from Gmail to O365 he just duplicated e-mails to diferente folders, the same email are inbox, stared, important, all e-mails etc…
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 12, 2015 at 8:28 PM said:
      
      Feel free to modify the matching mechanism if the Search Key or predefined set of attributes to match don’t meet your requirements. However, make sure to test and backup, as hard deletes (if chosen) are hard deletes, and there is no backup in Office 365 🙂
      
      LikeLike
      
      Reply ↓
Patrick on July 8, 2014 at 10:43 AM said:

When running against Exchange 2013 CU5 you’ll need to use the latest version of the EWS API (2.2 at the moment) link: http://www.microsoft.com/en-us/download/details.aspx?id=42951

LikeLike

Reply ↓
- Michel de Rooij on July 8, 2014 at 10:53 AM said:
  
  No, 2.2 adds some client-side functionality, i.e. attachment handling. My scripts are not using functionality requiring EWS 2.2.
  
  LikeLike
  
  Reply ↓
Nabil Salameh on September 19, 2014 at 11:31 AM said:

I am trying your solution but it says (Can’t Acces Mailbox Information Store) !!
Waiting your Reply,

LikeLike

Reply ↓
search results on September 22, 2014 at 7:10 AM said:

Have you ever thought about writing an ebook or guest authoring on other blogs?
I have a blog centered on the same ideas you discuss and would
really like to have you share some stories/information. I know
my viewers would value your work. If you are even remotely interested, feel free to send me an email.

LikeLike

Reply ↓
Brian Hatchell on October 8, 2014 at 7:03 PM said:

Michel:

I have it working well with O365 users (not Hybrid) so long as you specify the server properly. You are obviously an Exchange rockstar? Thanks for the script!

LikeLike

Reply ↓
- Michel de Rooij on October 8, 2014 at 7:10 PM said:
  
  Thank you. Odd, I’ve used it on my Small Business account without problems, no need to specify the server name (relying on Autodiscover)
  
  LikeLike
  
  Reply ↓
Harvey on October 28, 2014 at 5:04 PM said:

Hi Michel:

we have a problem w/CRM synching – it is creating dup contacts in Outlook and we cannot figure out why. Is there a way to have your script be triggered when the dup comes in and delete it right then and there? thanks

LikeLike

Reply ↓
- Michel de Rooij on October 28, 2014 at 5:07 PM said:
  
  No and yes. No, the script is not created with that in mind. Yes, you can use EWS to subscribe to a folder and trigger actions when changes occur. I have something similar in the pipeline, now I only need to find some time or clone myself 🙂
  
  LikeLike
  
  Reply ↓
  - harvey on December 9, 2014 at 8:47 PM said:
    
    Let me ask this – I am hoping to use your script to remove dups from our Outlook clients, but is there a way your script can be modified to remove dups from the CRM database master from which the dups originated?
    thank you.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on January 12, 2015 at 3:24 PM said:
      
      That sounds like a CRM/SQL operation – you may want to check with the vendor. Removing duplicate tuples in SQL is not a problem, maintaining referential integrity and consistency might me.
      
      LikeLike
      
      Reply ↓
Richard Hamilton on November 12, 2014 at 7:39 PM said:

I have same issue saying it cannot access information store. User Powershell on Exchange 2013 CU 6.

LikeLike

Reply ↓
- Michel de Rooij on November 12, 2014 at 8:30 PM said:
  
  Please check this article for configuring proper permissions, apart from configuring Autodiscover properly or using Server parameter:
  
  Impersonation: To be, or pretend to be
  
  LikeLike
  
  Reply ↓
Walter on November 18, 2014 at 10:00 AM said:

Hi, I also seem to be having some permission errors, but can’t seem to find the problem…..
Working on an exchange 2007 server.
Installed the EWS API and gave the user full control and impersonation rights:
Get-Mailbox mailboxname | Add-ADPermission –User ADM_user –ExtendedRights ms-Exch-EPI-May-Impersonate
Then ran the script:
Remove-DuplicateItems.ps1 -Mailbox mailboxname -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server EWSServer

Error is:
.\Remove-DuplicateItems.ps1 : Can’t access mailbox
information store
At line:1 char:28
+ .\Remove-DuplicateItems.ps1 <<<< -Mailbox mailboxname -Type Calendar -Imperson
ation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server
EWSServer
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
tion
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
n,Remove-DuplicateItems.ps1

Could you please help?

LikeLike

Reply ↓
Brad on December 7, 2014 at 12:42 AM said:

This script is a thing of beauty. I work in a shop heavy with Mac users. At some point in the past many of them were hit but the Outlook for Mac/iOS/sync/duplicate bug. I had one user with over 1.5 million items. I used ODIR and it worked well but it was a very tedious process to migrate messages into folders in batches <10,000 messages so that ODIR could work without crashing. This script just deleted 580146 duplicates from one mailbox while I slept. Thank you!

We do have a 3rd party archiving solution that changes the appends the IPM.NOTE class with an extension. In this case there are 2 of each message as the sizes and classes are different. Is there an easy add to the script to compare 2 separate message classes? IPM.Note and IPM.Note.ExShortcut.owa and only keep the shortcut version?

LikeLike

Reply ↓
- Michel de Rooij on December 7, 2014 at 6:11 PM said:
  
  Sounds like you want a mix of this script with my other script to remove certain message class items at https://eightwone.com/2013/05/16/removing-messages-by-message-class-from-mailbox/
  So, the IPM.Note and the IPM.Note.ExShortcut.owa are in the same folder and – based on certain props – you want duplicates detected and of the duplicates, only remove the IPM.Note when there is an IPM.Note.ExShortcut.owa present?
  
  LikeLike
  
  Reply ↓
  - Brad on December 8, 2014 at 9:42 PM said:
    
    Exactly.
    
    LikeLike
    
    Reply ↓
    - Brad on December 9, 2014 at 4:22 AM said:
      
      Actually, as I get deeper in, there are 2 scenarios. The one mentioned above and a second where the stubs are duplicated and the script doesn’t look at the stubs for duplicates at all. Or so it seems.
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on December 9, 2014 at 9:16 AM said:
        
        The script checks for duplicates per folder level. Also, depending on how the archiving product treats the message you may need to use attribute-level comparison instead of relying on PidTagSearchKey
        
        LikeLike
    - Michel de Rooij on December 9, 2014 at 9:18 AM said:
      
      Ok, could do that but may be later this week as I’m a bit busy atm
      
      LikeLike
      
      Reply ↓
Pingback: Removing Duplicate Items from a Mailbox - League Team
milangajiceu on February 9, 2015 at 10:28 AM said:

Hi Michel,
The script sounds great, but I just can’t get past the “Can’t access mailbox information store” error.

I’m on an Exchange 2007 environment here and I need to remove duplicate contacts (50k+) from a single mailbox (i.e. “contoso\user”)

What I’ve done:

1. created new account “contoso\me-tst” with no mailbox and no administrative privileges

2. granted ms-exch-epi-impersonation permission on all CAS servers to “contoso\me-tst”

Get-ExchangeServer | where {$_.IsClientAccessServer -eq $TRUE} | ForEach-Object {Add-ADPermission -Identity $_.distinguishedname -User (Get-User -Identity me-tst | select-object).identity -extendedRight ms-Exch-EPI-Impersonation}

3. granted ms-exxch-epi-may-impersonate an fullaccess permission on “contoso\user” mailbox to “contoso\me-tst”

Add-ADPermission -Identity user -User me-tst -extendedRight ms-Exch-EPI-May-Impersonate

4. “$a = get-credential” -> “contoso\me-tst” & password

5. executed script -> scrip fails with “Can’t access mailbox information store” / “(401)Unauthorized”

[PS] D:\Scripting>.\Remove-DuplicateItems.ps1 -mailbox user -type contacts -retain Oldest -server CAS001.contoso.local -deletemode harddelete -credentials $a -mailboxonly -mode full -impersonation -verbose
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox user (user@contoso.local)
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials contoso\me-tst
VERBOSE: Using user@contoso.local for impersonation
VERBOSE: Using Exchange Web Services URL https://CAS001.contoso.local/EWS/Exchange.asmx
VERBOSE: DeleteMode is harddelete
VERBOSE: Processing contacts items
D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store At line:1 char:28
+ .\Remove-DuplicateItems.ps1 <<> Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

Checked:
– Permissions are there
– https://CAS001.contoso.local/EWS/Exchange.asmx is accessible

The script fails when first getting the root folder ($RootFolder= … line530 and on).

Is it not weird that is says “VERBOSE: Using user@contoso.local for impersonation”? should this not be something like “Impersonating user@contoso.local” or “using me-tst@contoso.local“?

Do you have any idea what could be wrong?

Thank you!

Milan

LikeLike

Reply ↓
- Michel de Rooij on February 9, 2015 at 2:11 PM said:
  
  99% permissions / firewall / EWS / CAS config. Take into account waiting period for new permissions to become effective.
  ps: Wording thing. When using impersonation, you are effectively using the account specified (see line above mentioning account used for impersonation).
  
  LikeLike
  
  Reply ↓
  - milangajiceu on February 9, 2015 at 3:23 PM said:
    
    The waiting period is not it. I have wated considerably longer than it takes to replicate the permissions.
    
    … oh wait … as we speak …
    
    it was a certificate issue. I typed the external URL as the Server and now it works! AWESOME!
    
    LikeLike
    
    Reply ↓
  - milangajiceu on February 9, 2015 at 3:33 PM said:
    
    And now I get this:
    VERBOSE: Searching for folder class
    VERBOSE: Processing folder Contactpersonen
    D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
    At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<<< -mailbox user -type contacts -retain Oldest -server email.sdr.rot
    $a -mailboxonly -mode full -impersonation -verbose
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.p
    
    Exception calling "FindItems" with "2" argument(s): "The request failed. The operation has timed out"
    
    since it just worked on another mailbox i'm thinking that it is simply taking too long to collect the data (55k+ contacts). Is there a way around this?
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on February 9, 2015 at 3:52 PM said:
      
      Good one. For change EWS timeout, just below:
      $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
      Add:
      $EwsService.Timeout= [System.Int32]::MaxValue
      
      LikeLike
      
      Reply ↓
milangajiceu on February 9, 2015 at 3:53 PM said:

I just tried it once more and it worked. AWESOME!

LikeLike

Reply ↓
Miles Thompson on March 6, 2015 at 11:03 PM said:

Hi, I have this working against mailboxes in O365 but am finding that after it has completed, there are still duplicates, not as many but still lots.
When it runs, it says it has deleted loads of duplicates but then when I check using OWA, there are still duplicates.
How does it determine if the messages are duplicates or not? Is there a different detection method?

LikeLike

Reply ↓
- Michel de Rooij on March 7, 2015 at 3:21 PM said:
  
  Default it uses a MAPI property named PidTagSearchKey to identify duplicates. However, depending on how those duplicates got in the mailbox, that property might not match. In those situations, you can use mode ‘Full’ which uses a predefined set of properties to identify duplicates. Be advised that some 3rd party products (i.e. archiving) may modify those attributes or contents – changing item size – in which case the properties will not match up. In those cases, only option is to alter the set of attributes in the script.
  
  LikeLike
  
  Reply ↓
  - Miles Thompson on March 7, 2015 at 8:58 PM said:
    
    Hi, thanks for the reply.
    I did run it with mode Full and checked a couple of messages and see they have the same Message-ID, so I thought I’d try removing the other conditions from the criteria but thought I’d check running mode quick first to see what happens, and voila! It worked like a charm; no more duplicates 🙂 Thanks for your help.
    
    LikeLike
    
    Reply ↓
    - Miles Thompson on March 8, 2015 at 12:33 AM said:
      
      Hi, also, I’m trying to run this against 80ish O365 mailboxes but can’t get the csv to work. I setup impersonation in O365 but when running the script it fails on autodiscovery. Is there a way to test if impersonation is working correctly? I followed your impersonation guide to set it up.
      I tried without impersonation by adding the admin account to have Full Access permissions to all the mailboxes but that also didn’t work, same autodiscover/mailbox access error.
      I also tried adding the -Credentials (Get-Credential) switch, then when prompted entered the credentials for the first mailbox in the csv but then those credentials were used for all subsequent mailboxes also, and although it appeared to be running through all the mailboxes, I noticed it was actually just doing the first mailbox over and over. Is there a way to make it ask for the credentials for each of the mailboxes in the csv? It’ll be time consuming to enter them all, but still quicker than running the Remove-Duplicates command on every mailbox.
      Thanks
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on March 8, 2015 at 10:08 AM said:
        
        If autodiscover is not resolving properly from your internal network (maybe running split DNS & Hybrid ?), you can use Server to specify the external Office 365 URL, e.g. https://outlook.office365.com/ews/exchange.amsx. But, to be honest, I never have to use that when Autodiscover and DNS is set up correctly, and the request is not blocked. That nice thing with impersonation is that you only need to arrange for 1 account which can impersonate the mailbox owners when accessing those mailboxes in that CSV.
        
        LikeLike
      - Miles Thompson on March 9, 2015 at 9:28 AM said:
        
        Thanks for the help but I’ve resorted to creating a script that runs the delete-duplicates.ps1 for each individual mailbox which it prompts for the credentials for each. It’s a long-winded process but seems to be working.
        I did spend a few hours trying to get it working using impersonation and delegated access but couldn’t make it work.
        Autodiscover is working fine and I’m working from home, so no split DNS in the mix or hybrid setup. I tried using -server outlook.office365.com but still same issue. I have no outbound firewall so don’t think the request could be blocked, It works fine when running on a single mailbox when I put in -Credentials (get-credentials) but doesn’t work if I try and use impersonation or delegated permissions. If I try using the -credentials switch and put in the admin account, which has full access, with inheritance, it still can’t connect to the mailbox with error about accessing the information store.
        
        LikeLike
Pingback: Exportar detalle de Shortcuts de Enterprise Vault desde Exchange | chorbo.es
Tom on April 24, 2015 at 1:42 PM said:

Script looks fantastic. Is there a way to specify a certain date range of duplicates to check for? For example, only compare the last 30 days worth of items?

LikeLike

Reply ↓
- Michel de Rooij on April 24, 2015 at 1:48 PM said:
  
  Currently no. But it’s a script, so feel free to introduce enhancements for your situation as required. I could add it to the ‘wish list’ (but no ETA).
  
  LikeLike
  
  Reply ↓
ltheil on June 20, 2015 at 4:25 AM said:

I’ve needed a script like this for a long time, but my powershell and exchange dev skills were too limited. This script works perfectly with impersonation rights. It did not work without that role assigned though. It acted like it was working but ended up finding no duplicates when i could see the duplicates in the mailboxes i was searching.

LikeLike

Reply ↓
- ltheil on June 25, 2015 at 11:36 PM said:
  
  So after running it a few times, I’ve run into issues which i believe may be related to limits in the number of items per batch or possibly an ews throttling limit. It errors and says that EWS is not available because none of the client access servers in the destination site could process the request.
  
  I have some mailboxes that have over 10,000 duplicates.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 30, 2015 at 9:34 AM said:
    
    If you’re running it against Office 365, there is no way to customize throttling. For local configuration, I’ll put up a blog shortly as I get this asked quite a few times.
    
    LikeLike
    
    Reply ↓
    - Nick C on March 9, 2022 at 9:20 PM said:
      
      Hey Michel,
      
      Did you happen to throw up a blog for this? We’re currently dealing with a calendar that’s got 1.7m items we’re trying to deduplicate and your script is working phenomenally well except now we’re getting hit with EWS operation failed, server busy. Hoping you’ve got some info on helping with that throttling customization.
      
      LikeLike
      
      Reply ↓
- ltheil on June 28, 2015 at 1:49 PM said:
  
  I was able to tweak the script to get it work for what i needed. I spoke to Microsoft about getting a tool from them to remove the duplicates and they were not able to provide me with anything so this script is EXTREMELY useful! Thank you for sharing it.
  
  LikeLike
  
  Reply ↓
Hans on June 26, 2015 at 8:48 AM said:

Hello.

I’m searching for a script which could process public folders on Exchange Server 2013. There are always many duplicates, because mails, which had been sent to more than one user, are moved to the public folder by each user after he recognized or processed the mail. Is it currently possible to process these public folder with your script? If yes: what is the syntax? If no: is there any hope that you implement such function?

Many thanks for providing this script and kind regards
Hans

LikeLike

Reply ↓
- Michel de Rooij on July 17, 2015 at 8:45 PM said:
  
  Sorry for the delayed response. Public Folders should be accessed differently compared to mailboxes. However, processing is similar – I have jotted it down on the ‘requested features’.
  
  LikeLike
  
  Reply ↓
  - Ben on October 21, 2015 at 7:53 AM said:
    
    Any luck with the requested feature of removing duplicates from the public folder?
    
    LikeLike
    
    Reply ↓
Guy Thompson on July 7, 2015 at 3:06 PM said:

Hello,

I have been using this on Office 365 for a while now.

to make sure this works with power-shell you have to specify the account you setup impersonation with in the command.

For example :-

.\Remove-DuplicateItems.ps1 -Mailbox Person@email.com -Type all -DeleteMode MoveToDeletedItems -Mode Quick -Verbose -Server outlook.office365.com -Credentials Admin@Email.com

This will then bring up the input box & then will it cache the credentials…just running it without that string will present you with the infamous :-

C:\Scripts\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Mailbox correctgroup@lilyskitchen.co.uk …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Matt on July 16, 2015 at 5:02 AM said:
  
  For some reason the -Credentials property isn’t working for me, I get this error:
  
  Remove-DuplicateItems.ps1 : Cannot process argument
  transformation on parameter ‘Credentials’. Cannot convert the “admin@domain.com” value of type “System.String” to
  type “System.Management.Automation.PSCredential”.
  At line:1 char:144
  + … m -Credentials admin@domain.com
  + ~~~~~~~~~~~~~~~~~~~~
  + CategoryInfo : InvalidData: (:) [Remove-DuplicateItems.ps1], ParameterBindingArgumentTransformationExce
  ption
  + FullyQualifiedErrorId : ParameterArgumentTransformationError,Remove-DuplicateItems.ps1
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on July 16, 2015 at 8:38 AM said:
    
    You need to provide Credentials of type PSCredential, e.g.
    $Cred= Get-Credential
    … -Credential $Cred
    
    or .. -Credentials (Get-Credential)
    
    LikeLike
    
    Reply ↓
    - Matt on July 16, 2015 at 5:48 PM said:
      
      Thank you sir that got it working.
      
      LikeLike
      
      Reply ↓
Rahul Singh on August 6, 2015 at 7:59 AM said:

We have our mailboxes hosted on Office 365, due to some reasons few mailboxes have duplicate mails. These mailboxes have large size – average 25 GB.

I found your script very useful and tried it. Thanks a lot for sharing it online.

Now I m facing a problem, whenever I run this script against large mailboxes, the script breaks the execution and throws a exception. I had tried to get the exception error message, mentioned below:
————————————————————————————————————————–
Exception calling “FindItems” with “2” argument(s): “The request failed. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.”
————————————————————————————————————————–

Any help will be greatly appreciable.

Regards

LikeLike

Reply ↓
- Michel de Rooij on August 6, 2015 at 9:06 AM said:
  
  Is your network/internet connection reliable?
  Also make sure you aren’t running the script (or other scripts, tools, applications) multiple times using the same credentials to prevent running into Office 365 throttling your requests, see https://msdn.microsoft.com/en-us/library/office/jj945066%28v=exchg.150%29.aspx
  
  LikeLike
  
  Reply ↓
  - Rahul Singh on August 6, 2015 at 4:28 PM said:
    
    Our Internet Connection is fine. We have dedicated 1 admin id for the same.
    
    Moreover I found that the script is detecting the duplicates from folders other than Inbox & Sent Items.
    
    To simulate the same I had duplicated (Copy & Paste) some messages in Inbox and Test folder. Duplicates were deleted from the Test folder but not from the Inbox.
    
    Can you please advise what to do???
    
    LikeLike
    
    Reply ↓
Ramakrishnan on August 18, 2015 at 8:32 PM said:

Will this help me to delete the duplicate calendar folder named – Calendar which is present in the user mailbox.

The affected user account has two calendar folder

English calendar
German Kalender

The user doesn’t have a mobile device associated , not sure how this happened all of a sudden as the migration happened years ago.

Any information is highly appreciated !!.

Thanks.

Regards,
Ramakrishnan

LikeLike

Reply ↓
- Michel de Rooij on August 18, 2015 at 8:57 PM said:
  
  No, this script is to detect and remove duplicate items, not folders. If the folders should be merged, you can check out one of my other scripts, Fix-MailboxFolders (https://eightwone.com/2013/01/19/fixing-well-known-folders-troubles/). However, that script will use default localized names for Well Known folders like Calendar. See related blog on how to customize or define a custom ‘locale’ to cover specific scenarios. Note however, only ‘source’ can be customized, the destination folder names are determined by the configured mailbox locale.
  
  LikeLike
  
  Reply ↓
Alex Pavlenko on September 2, 2015 at 1:08 PM said:

Michel, how does your script compare to this one below which relies on iCALUID method to look for duplicates? I used both (yours and that one below) and it seemed that below one produces some false positives while your script reports 0 duplicates in such mailboxes. I just wanted to ask for your technical opinion which method is more reliable?

http://blogs.msdn.com/b/emeamsgdev/archive/2015/02/12/powershell-remove-duplicate-calendar-appointments.aspx

LikeLike

Reply ↓
- Michel de Rooij on September 2, 2015 at 1:48 PM said:
  
  I won’t recommend relying on iCALUID, as this property is set by the client. Mine uses PidTagSearchKey by default, but for calendar items in your case it may be better to resort to mode Full to use property-based matching. However, mileage may vary items if duplicate items are processed in some way, seemingly staying identical but altering their size for example. The script from the MSFT guy has something similar, but in that regards it does not work differently from mine.
  
  LikeLike
  
  Reply ↓
  - Alex Pavlenko on September 2, 2015 at 2:00 PM said:
    
    Michel – yes this is exactly what I am observing with iCALUUD vs. PidTagSearchKEy thanks! Will try Full Mode now.
    I have another question on using -WhatIf. I did not seem to find any -WhatIf logic in your code. Running with or without WhatIf produces the same output. Am I mistaking perhaps and logic is built into the core PS -WhatIf module which is not exposed in your code? I wanted to analyze appointments that script finds before letting it to delete anything.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on September 2, 2015 at 3:22 PM said:
      
      The ShouldProcess() should take care of that logic
      
      LikeLike
      
      Reply ↓
Michael on September 22, 2015 at 2:39 AM said:

I nearly gave up on this script after running into the “Can’t access mailbox information store” error everytime. Impersonation, Get-Credential, setting the servername, writing the mailbox/usernames in all kind of formats…- nothing worked.
The key to get the script running: using the “-MailboxOnly” option, although the server is running Exchange 2010.
AUSFÜHRLICH: Removing 1908 items from Kalender
AUSFÜHRLICH: Total number of items removed: 1908
AUSFÜHRLICH: Processing finished

Even with all the time lost tinkering- this is SO MUCH faster than every other “remove duplicates”-tool out there.
Thank you very much!

LikeLike

Reply ↓
Mark on October 29, 2015 at 11:13 PM said:

So i tried this script and it says it working but its not processing the duplicates.

Exchange 2010 SP3 Any help is appreciated

VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Ser
Processing mailbox User (mark.st.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using Exchange Web Services URL https://NLKVEXCMBX0
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing All items
VERBOSE: Processing primary mailbox User
VERBOSE: Searching for all folder classes
VERBOSE: Processing folder Calendar
VERBOSE: Processing folder Contacts
VERBOSE: Processing folder Lync Contacts
VERBOSE: Processing folder Conversation Action Settings
VERBOSE: Processing folder Conversation History
VERBOSE: Processing folder Drafts
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Journal
VERBOSE: Processing folder Junk E-Mail
VERBOSE: Processing folder News Feed
VERBOSE: Processing folder Notes
VERBOSE: Processing folder Outbox
VERBOSE: Processing folder Quick Step Settings
VERBOSE: Processing folder RSS Feeds
VERBOSE: Processing folder Sent Items
VERBOSE: Processing folder Suggested Contacts
VERBOSE: Processing folder Sync Issues
VERBOSE: Processing folder Conflicts
VERBOSE: Processing folder Local Failures
VERBOSE: Processing folder Server Failures
VERBOSE: Processing folder Tasks
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
- Michel de Rooij on October 30, 2015 at 11:05 AM said:
  
  Could be that there are no duplicates matched using on PidTagSearchKey (Exchange provisioned key, default mode). Try -Mode Full to use attribute matching.
  
  LikeLike
  
  Reply ↓
  - Mark on November 10, 2015 at 4:01 PM said:
    
    I got it work, with the Impersonation flag.
    
    Thanks this script is going to me a lot of time.
    
    LikeLike
    
    Reply ↓
Anthony Godwin on November 15, 2015 at 1:56 PM said:

Greetings.. We have very large mail boxes. We were able to get the script to work on our smaller mail boxes… but it failed on the larger mail boxes with the message “can’t access mailbox information store at line 1 char 1.” Do you have a solution for larger mailboxes. The duplicates items have resulted in some of our mailboxes exceeding 35 gigabytes.

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2015 at 7:32 PM said:
  
  You can adjust the timeout, add $EwsService.ServiceTimeout and set it to a high(er) value (default is 100000 msec). You can also tweak the batch sizes (folders, items and deletions) at the top of the script to make the result sets to work with smaller. It’s on the list to make parameters out of these and introduce keep-alives for Exchange 2013 and up (including Office 365).
  
  LikeLike
  
  Reply ↓
  - Anthony Godwin on November 16, 2015 at 4:05 AM said:
    
    Thanks very much. I shall try that.
    
    LikeLike
    
    Reply ↓
  - Anthony Godwin on November 17, 2015 at 3:50 PM said:
    
    Can you please let me know the proper syntax and where to place this in the script. Thanks!
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 17, 2015 at 4:17 PM said:
      
      $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
      $EwsService.ServiceTimeout= 300000
      
      LikeLike
      
      Reply ↓
Tony Kelly on December 23, 2015 at 12:20 AM said:

Hi, thanks for this great script. I’m giving it a go now for our public folders (in ‘WhatIf’ mode), but it seems to get stuck on a certain folder, even after leaving it overnight. Is there any way to get it to skip past when it gets stuck?

LikeLike

Reply ↓
Oliver on January 6, 2016 at 8:23 AM said:

Hey Michel, greate article & script.

We struggle with duplicate contacts created by the MS Dynamics CRM “Server side sync” feature.
CRM related Outlook contacts are flagged with an user-defined attribute “crmLinkstate” which can have the following values:
2->contact is “tracked” in CRM
1->track is pending
0->untracked

Would it be possible to alter your script to query on this crmLinkstate Attribute to get rid of “untracked” Contact duplicates?

Thanks for your Feedback,
Oliver

LikeLike

Reply ↓
- Michel de Rooij on January 6, 2016 at 2:28 PM said:
  
  Would that be only untracked duplicate contacts or all untracked contacts?
  
  LikeLike
  
  Reply ↓
  - Oliver on January 6, 2016 at 2:59 PM said:
    
    It should only query on untracked duplicates as we first need to clean up related duplicates in CRM System.
    
    BR, Oliver
    
    LikeLike
    
    Reply ↓
Deniz on March 3, 2016 at 4:55 PM said:

Write-Error : A positional parameter cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl” with “2
” argument(s): “The Autodiscover service couldn’t be located.”‘.
At C:\Windows\system32\Remove-MessageClassItems.ps1:434 char:28
+ Write-Error <<<< "Autodiscover failed: " $error[0]
+ CategoryInfo : InvalidArgument: (:) [Write-Error], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.WriteErrorCommand

I can't remove the stubs..
Can you help me pls?

LikeLike

Reply ↓
- Michel de Rooij on March 23, 2016 at 3:05 PM said:
  
  You need to set up Autodiscover – or make it accessible (ie no firewall blockades). You can override the discovery process using the Server parameter.
  
  LikeLike
  
  Reply ↓