Removing Duplicate Items from a Mailbox

Posted on June 21, 2013 by Michel de Rooij

Latest version: 2.41, April 18th, 2023

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
Importing existing data. Accidental import from – for example – a PST file to a mailbox can lead to duplicate entries.

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange Web Services (EWS) Managed API and for OAuth authentication the Microsoft Authentication Library (MSAL) libraries. You can install these packages from NuGet, or place their DLL’s in the same folder as the script. For an example of how to install EWS.Managed.Api from NuGet, see this article; for MSAL follow the same process but with the package titled ‘Microsoft.Identity.Client’.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions when using Basic Authentication; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post. Using a registered app with OAuth is always through Impersonation.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [[-Identity] ] [[-Type] ] [-Retain ] [-Server ] [-Impersonation] [-DeleteMode ] [-Credentials ] [-Mode ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-PriorityFolders <String[]>] [-NoSize] [-CleanupMode] [-NoProgressBar] [-Force] [-WhatIf] [-Confirm] [-Secret] [-CertificateThumbprint] [-CertificateFile] [-CertificatePassword] [-TenantId] [-ClientId] [-TrustAll] [-ExchangeSchema <String>] [-NoSCP]

A quick walk-through on the parameters and switches:

Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:

ItemClass	Criteria
Contacts	File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List	FileAs, Number of Members, Size
Calendar	Subject, Location, Start & End Date, Size
Task	Subject, Start Date, Due Date, Status, Size
Note	Contents, Color, Size
Mail	Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other	Subject, DateTimeReceived

MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter when running against mailboxes on Exchange Server 2007.
ArchiveOnly specifies you only want to process personal archives of specified users.
IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
CleanupMode specifies to cleanup duplicates per folder (Folder, default), the whole mailbox (Mailbox), or multiple mailboxes (MultiMailbox, identities specified using Identity). The first unique item encountered will be retained. For Mailbox-level cleanup, PriorityFolders can be used to give priority to retaining items in specified folders before those found in other folders.
PriorityFolders specifies which folders have priority over other folders, identifying items in these folders first when using MailboxWide mode. Usage of wildcards and well-known folders is identical to IncludeFolders.
NoSize tells script to not use size to match items in Full mode.
NoProgressBar prevents displaying a progress bar as folders and items are being processed.
Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.
TrustAll can be used to accept all certificates, e.g. self-signed certificates or when accessing Exchange using endpoint with a different certificate.
ExchangeSchema can be used to specify the Exchange schema to use when connecting to Exchange server or Exchange Online. Defaults to Exchange2013_SP1 or Exchange2016 when -Server is specified and is ‘outlook.office365.com’ (Exchange Online endpoint).
NoSCP to skip SCP lookups in Active Directory for Autodiscover.

For authentication, the following parameters are available:

Credentials specifies credentials to use for Basic Authentication.
TenantId specifies the identity of the Tenant (OAuth)
ClientId specifies the Id of the registered application (OAuth).
CertificateThumbprint specifies the thumbprint of the certificate from personal store to use for authentication (OAuth).
CertificateFile specifies the external certificate file (pfx) to use for authentication (OAuth). This certificate needs to contain a private key; the registered application needs to contain the certificate’s public key.
CertificatePassword optionally specifies the password to use with the certificate file (OAuth).
Secret specifies the secret to use with the application (OAuth).

Few notes:

When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter	Matches
FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA
Folder*	\TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\Folder	\TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\FolderA\	\TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So, suppose you want to remove duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

Alternative, you can use an e-mail address and specify credentials. This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Identity olrik@office365tenant.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

A more complex example using IncludeFolders, ExcludeFolders and PriorityFolders:

$Credentials= Get-Credential
 .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '#Inbox#\*','\Projects\*' -ExcludeFolders 'Keep Out' -PriorityFolders '*Important*' -CleanupMode Mailbox

This will remove duplicate items from the specified mailbox in Office365, using the following options:

Fixed Server FQDN – bypassing AutoDiscover.
Limits operation against the Well-Known Inbox folder, top Projects folder, and all of their subfolders.
Excluding any folder named Keep Out.
Duplicates are checked over the whole mailbox.
Priority is given to folders containing the word Important, causing items in
those folders to be kept over items in other folders when duplicates are found.

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 ..

Download
The script is available on GitHub here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

484 thoughts on “Removing Duplicate Items from a Mailbox”

Damir Reić on May 25, 2016 at 9:05 PM said:

Hello,

thank you for the script. Found a little bug, it doesn’t properly detect number of duplicated items. I used it on O365.

So here is what i did:
1) created random folder inside outlook
2) take some mail from inbox and copy it
3) paste it 11 times inside random folder

Running script only detected 8 of them in random folder. Took new mail from inbox, copied it to random folde 2 times and run the script again. Script deleted previous copies that remain in random folder from 1st copy action. It didn’t remove 2 new copied mails.

I tried running the script again in 1st and 2nd scenario after 1st run finished and it didn’t detect or delete remaining copies.

I have a problem from Google migration to O365 so I just wanted to test before I execute it customer account.

Thanks!

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2016 at 10:03 PM said:
  
  So, in Outlook, were all those changes already synced back to Office 365 (assuming you’re running cached mode)?
  
  LikeLike
  
  Reply ↓
  - Damir Reić on May 25, 2016 at 10:12 PM said:
    
    Well that is kinda hard to tell since O365 OWA is not displaying duplicated mails, only Outlook client is.
    
    LikeLike
    
    Reply ↓
    - Damir Reić on May 25, 2016 at 10:37 PM said:
      
      I just tested again and waited few minutes before deleting. Still all duplicated mails were not deleted.
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on May 25, 2016 at 11:07 PM said:
        
        What cmdline are you using? Note that default it utilizes PR_SEARCH_KEY to match duplicates; that key should be a unique and when copying items, that attribute should be preserved and identical on those items. Can you inspect with for example MFCMAPI if that is the case? Alternatively, you can use -Mode Full to match on a predetermined set of attributes (see help)
        
        LikeLike
      - Damir Reić on May 26, 2016 at 7:57 AM said:
        
        .\Remove-DuplicateItems.ps1 -Mailbox test@domain.com -type mail -Credentials $Credentials -Mode Full -DeleteMode MoveToDeletedItems -Verbose
        
        LikeLike
      - Anthony on May 26, 2016 at 12:03 PM said:
        
        Do you have a utility to remove duplications from the online version of Office365 for contacts, calendars and mailboxes, tasks, notes? Thanks in advance for your response.
        
        LikeLike
      - Michel de Rooij on May 26, 2016 at 12:45 PM said:
        
        That’s where the script is for – not only mail-items. Have you checked with MFCMAPI to verify the PR_SEARCH_KEY attributes are indeed identical? The cmdline you use should work. On another note, those items are of type mail (IPF.Note); not some sort of stub or something else? (you can use MFCMAPI for this as well or add the Message Class field to the view)
        
        LikeLike
      - Damir Reić on May 26, 2016 at 1:19 PM said:
        
        I did that just now, they all have same PR_SEARCH_KEY
        
        LikeLike
      - Michel de Rooij on May 26, 2016 at 3:51 PM said:
        
        Then the script should work
        
        LikeLike
      - Damir Reić on May 31, 2016 at 9:48 PM said:
        
        Found out why it didn’t want to work. Mails don’t have same PR_SEARCH_KEY, so for example same (content) mail is inside “all mails” and “important” folder but PR_SEARCH_KEY does not match even though all data of the mail that is displayed inside outlook client is the same. Would you be interested to modify the script to fix our problem (we can discuss about the payment)? I don’t know what I was comparing few days ago…
        
        LikeLike
derek macleod on July 25, 2016 at 12:27 PM said:

Do you know how I could get this to work when the PID searches do not match, and the Received timestamp, size, and last modified properties are all a tiny bit different? (so, essentially relying on sender, subject, time sent which remain unique)

LikeLike

Reply ↓
- Michel de Rooij on July 25, 2016 at 1:37 PM said:
  
  Below ‘# Use predefined criteria for matching duplicates depending on ItemClass’ is the logic to match items, depending on item type (different items provide different attributes). I can’t determine “a tiny bit different” for you. There is also the danger of removing valid items, for example it happens quite often people forget to state something or add an attachment, making things like subject, sender and recipients match and only the timestamp ‘a tiny bit’ different, yet both messages should be kept.
  
  LikeLike
  
  Reply ↓
  - derek macleod on July 25, 2016 at 2:54 PM said:
    
    Thank you sir! I was able to comment out a couple of the predefined criteria to get it working. Lovely script that.
    
    LikeLike
    
    Reply ↓
Enrique on July 28, 2016 at 3:12 PM said:

Hello Michel, the script is GenieL.
We migrated from Lotus to Exchange 2013 but this generate many duplicates process.
I read carefully your blog and I changed the script and the parameters to work. But, now does not fail but gives 0 items to remove

PS C:\Windows\system32> Remove-DuplicateItems.ps1 -Mailbox email@domain.com -Type Mail -DeleteMode MoveToDeletedItems -ArchiveOnly -Verbose -Server Exchange.local -Mode Full
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox email@domain.com (email@domain.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using Exchange Web Services URL https://Exchange.local/EWS/Exchange.asmx
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing Mail items
VERBOSE: Processing archive mailbox email@domain.com
VERBOSE: Searching for folder class
VERBOSE: Processing folder (Custom Expiration\Manage Folders)
VERBOSE: Processing folder Elementos eliminados
VERBOSE: Processing folder FolderHiddenPublic
VERBOSE: Processing folder Folders
VERBOSE: Processing folder Data
VERBOSE: Processing folder Personal
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Sent Items
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
- Eric on August 9, 2016 at 5:48 PM said:
  
  I am having this same issue only on Wave 15.1 of Office 365.
  
  Previous comments lead me to think that it may be a permissions issue. We have duplicates that came in from an IMAP sync and PST import but the script output looks very similar to what you posted – 0 items removed. I have granted the admin account full mailbox access as well as ApplicationImpersonationRights via the Discovery Management admin role but no joy.
  
  The above links regarding permissions appear to be dead so maybe there is something else I am missing with permissions here.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on August 10, 2016 at 10:01 PM said:
    
    15.1 is a bit too generic – what is you specific version, i.e. what does Outlook’s Connection Status / Version report? Meanwhile I’ll check against an 15.1.249.20 tenant.
    
    I’ve update the link to information on how to configure impersonation.
    
    LikeLike
    
    Reply ↓
    - Eric on August 12, 2016 at 12:27 AM said:
      
      Well, my admin display version is 15.1.549.26. I’m not sure if that correlates to the specific Exchange Online version. In any case, I have verified the impersonation as well as given full mailbox access, so I don’t think permissions should be an issue at this point.
      
      However, I have tried both the Full and Quick mode with no items being found but no errors showing either.
      
      Thanks very much for the script and for your assistance!
      
      Eric
      
      LikeLike
      
      Reply ↓
Sandra on October 7, 2016 at 10:27 AM said:

Hi,

Thanks for sharing such a wonderful post with us.
I had this issue & your blog as well as one other blog, helped me to overcome this scenario of duplicate emails. Another link is as follows:
[link removed]

LikeLike

Reply ↓
lbit on November 3, 2016 at 8:25 PM said:

I’m using v1.41 The script runs fine but finds no duplicates. I even tried selected an item in my inbox, ctrl-c, ctrl-v to duplicate it. Still the script says:
PS C:\Users\xxxx\Downloads> .\Remove-DuplicateItems.ps1 -Mailbox xxxx@yyy.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose -MailboxOnly -Server outlook.office365.com -Credential (Get-Credential)

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\1.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox xxxx@yyy.com (xxxx@yyy.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials xxxx@yyy.com
VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing Mail items
VERBOSE: Processing primary mailbox xxxx@yyy.com
VERBOSE: Searching for folder class
VERBOSE: Processing folder Clutter
VERBOSE: Processing folder Drafts
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Junk Email
VERBOSE: Processing folder Outbox
VERBOSE: Processing folder VoiceOutbox
VERBOSE: Processing folder Quick Step Settings
VERBOSE: Processing folder Sent
VERBOSE: Processing folder Sent Items
VERBOSE: Processing folder Sync Issues1
VERBOSE: Processing folder Conflicts
VERBOSE: Processing folder Local Failures
VERBOSE: Processing folder Server Failures
VERBOSE: Processing folder Trash
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
lbit on November 4, 2016 at 3:28 PM said:

I got it working for me. It wasn’t for for 4 different reasons:
1) autodiscover isn’t working for my office365 account, I just bypassed the issue adding parameter -Server outlook.office365.com
2) -Mode Quick doesn’t seem to be able to retrieve any useful identifier for emails, so I used -Mode Full
3) Even in -Mode Full, most of my duplicates are not recognized because they have identical time, sender, messageid, etc, but different size. That is, even if I manually duplicate a message copying it to a different mailbox, and then back, it gets a different (larger) size. Since this script compare also size (in Full Mode) it finds no duplicates. Solved by commenting out this line:
# if ($Item.Size) { $key+= “,”+$Item.Size.ToString()}
4) The search for a duplicate is done using an arrayList which (is inefficient and) always fails for me. Maybe because even if the two compared strings are identical they are still different objects.
Solved by replacing line
$UniqueList= [System.Collections.ArrayList]@()
with
$UniqueList = New-Object ‘System.Collections.Generic.HashSet[string]’

I also added two more options to Retain: Largest and Smallest, in order to be able to retain the smallest.

LikeLike

Reply ↓
Oliver on November 10, 2016 at 7:10 AM said:

Hello Michel,

sorry, I Need to come back to this post from early this year: https://eightwone.com/2013/06/21/removing-duplicate-items-from-a-mailbox/#comment-143576

Would it be possible to use your script to:
1) Lookup only contacts within a Mailbox that have the flag “crmLinkState=2”? (This would secure to work only on contacts that have been synced from Dynamics CRM)
2) Do a merge/cleanup with data from 1) and define a master record that has a certain value (e.g. “merged master”) in the field “Nickname”

Background:
We found a way to merge all duplicate contact records within our CRM System. To have a filtering Option, we´d update all remaining records with the above mentioned string within the field “Nickname” which would also be synced to Exchange mailboxes.

Thanks in advance for any Reply.

Regards,
Oliver

LikeLike

Reply ↓
Frédéric Lemieux on August 3, 2017 at 9:23 PM said:

I would like to be able to run the script from my computer without having the username password
To use the office administrator 365

LikeLike

Reply ↓
- Michel de Rooij on August 3, 2017 at 9:27 PM said:
  
  Then you need to store and reuse the credentials.
  
  LikeLike
  
  Reply ↓
  - Frédéric Lemieux on August 4, 2017 at 1:11 PM said:
    
    it’s possible run script for other user without password use ? how ?
    
    LikeLike
    
    Reply ↓
grahampriley on August 9, 2017 at 11:21 AM said:

Thanks for this script. Do you know if it is possible to run it in scan / report mode and not delete anything, just report on duplicates found?

Regards,
Graham

LikeLike

Reply ↓
- Michel de Rooij on August 9, 2017 at 11:22 AM said:
  
  Not the individual items, but -WhatIf:$true should show what it would do and -Verbose would show what folders it would process (and how many duplicates are found). I have a Reporting option on the ‘to do’ list.
  
  LikeLike
  
  Reply ↓
  - grahampriley on August 9, 2017 at 11:55 AM said:
    
    Doh, I never thought of using WhatIf. Thanks, that should do it. 🙂
    
    LikeLike
    
    Reply ↓
Derron on August 16, 2017 at 5:11 PM said:

Script seems to run without errors, the include folder is processed, but it doesn’t identify or remove any obvious duplicates. I have commented out the item.size lines for -Full, but obvious duplicates are still not removed. Tenant version is: 15.1.1341.23. Impersonation is enabled. Duplicates were copied with a third party migrator that we are trying to clean up but the same product did both runs so they are identical messages. Wonderful script if it will do what we need.

LikeLike

Reply ↓
- Michel de Rooij on August 16, 2017 at 5:55 PM said:
  
  It does provided permissions are in order and the set of attributes related to the item type matches. Use Verbose to see at least if it can access the folder with those duplicates.
  
  LikeLike
  
  Reply ↓
David on September 6, 2017 at 3:20 AM said:

Just completed a GSuite migration to Office365 and I’m using your script to clean up the duplicate mess.

Small folders process without any issues, but larger folders fail after deleting a few thousand duplicates…

VERBOSE: Removing 15596 items from Archive
Process-Mailbox : Problem removing items: Exception calling “DeleteItems” with “4” argument(s): “The server cannot
service this request right now. Try again later.”
At D:\Users\XX\Downloads\Remove-DuplicateItems.ps1:675 char:16
+ … If(! ( Process-Mailbox $RootFolder $IncludeSearchCollection $Exc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

D:\Users\XX\Downloads\Remove-DuplicateItems.ps1 : Problem processing primary mailbox of
user@domain.com (user@domain.com)
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Identity user@domain.com -Cr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

Usually, a new powershell is required to resume the operation.

Do you have any suggestions on how to complete the operation in a single pass?

LikeLike

Reply ↓
- Michel de Rooij on September 18, 2017 at 10:04 AM said:
  
  Robust version coming up that can deal with throttling, will update when I’m back from honeymoon next week
  
  LikeLike
  
  Reply ↓
Omar on September 12, 2017 at 11:52 PM said:

Should this be able to run against a shared mailbox? I’ve a shared mailbox (ex-user mailbox that has been converted to shared, with assigned email address) on O365 that i specify in identity, using my own credentials. The script runs but from the verbose output it’s obvious it’s running on my own mailbox and archive.

LikeLike

Reply ↓
- Michel de Rooij on September 18, 2017 at 10:07 AM said:
  
  It should, using identy to specify e-mail address of shared mailbox, and credentials with proper access to that mailbox or impersonation permissions (w/ impersonation switch)
  
  LikeLike
  
  Reply ↓
  - Omar on September 18, 2017 at 8:41 PM said:
    
    Ahh, forgot the impersonation switch. That did it!
    
    LikeLike
    
    Reply ↓
Denis on October 3, 2017 at 7:38 AM said:

Hello Michel, Thanks for this script.

But i have some problems…

D:\Remove-DuplicateItems.ps1 : Can’t access mailbox information store: Exception calling “FindItems” with “2” argument(
s): “The server cannot service this request right now. Try again later.”
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Mailbox xxxxxxxxx@xxxx.xx -Server out …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Michel de Rooij on October 3, 2017 at 9:13 AM said:
  
  That’s throttling on the server-end. Updated script resilient to throttling is on its way, stay tuned.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on October 21, 2017 at 1:03 PM said:
    
    Handling of throttling conditions included per v1.8
    
    LikeLike
    
    Reply ↓
Akvile on October 19, 2017 at 11:51 AM said:

Hi, I am getting an error: Problem removing items. Maybe you have any ideas what the problem is? Thank you!

Are you sure you want to perform this action?
Performing the operation “Remove-DuplicateItems.ps1” on target “Remove 152 items from \Inbox”.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “Y”): Y
VERBOSE: Removing 152 items from \Inbox
Process-Mailbox : Problem removing items: Attempted to divide by zero.
At C:\Program Files\Microsoft\Exchange\Web Services\2.2\Remove-DuplicateItems.ps1:1040 char:29
+ … If (! ( Process-Mailbox -Folder $RootFolder -IncludeFilter $Inclu …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailboxting error

LikeLike

Reply ↓
- Michel de Rooij on October 21, 2017 at 1:02 PM said:
  
  Found it – ~ line 920 it mentions $RemoveList where it should use $DuplicateList. Fixed in v1.81.
  
  LikeLike
  
  Reply ↓
Kevin Bouwman on November 13, 2017 at 3:44 PM said:

I am getting a repeated warning. I getting so many that the log text leading up to it has scrolled out of reach for a cut and paste. Several lines after the message telling me the script has started processing the mailbox I get:
VERBOSE:Adding folder \ (priority 0)
WARNING: Error performing operation FindFolders with Search options in . Error: Exception calling “FindFolders” with “3” argument(s): “Value cannot be null. Parameter name: parentFolderId”
VERBOSE:Adding folder \\ (priority 0)
WARNING: Error performing operation FindFolders with Search options in . Error: Exception calling “FindFolders” with “3” argument(s): “Value cannot be null. Parameter name: parentFolderId”
This repeats with another back slash added until there are well over 800 hundred slashes.
The script then ends with the message: Cannot access mailbox information store, error: The script failed due to call depth overflow.
At line:1 char:1
+ C:\Users\Me\Remove-DuplicateItems.ps1 -Identity me@company.biz -Server …
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId: Microsoft.PowerShell.Commands.WriteErrorException,Remove-Duplicates.ps1

The command line (v1.81) I am using looks like this:
C:\Users\Me\Remove-DuplicateItems.ps1 -Identity me@company.biz -Server outlook.office365.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose -Credentials (Get-Credential) -Retain Oldest

LikeLike

Reply ↓
- Michel de Rooij on November 13, 2017 at 3:54 PM said:
  
  From what PowerShell version are you running this? ($PSVersionTable)
  
  LikeLike
  
  Reply ↓
  - Kevin Bouwman on November 17, 2017 at 10:46 PM said:
    
    3.0
    
    LikeLike
    
    Reply ↓
    - Kevin Bouwman on November 17, 2017 at 10:55 PM said:
      
      Name Value
      —- —–
      PSVersion 3.0
      WSManStackVersion 3.0
      SerializationVersion 1.1.0.1
      CLRVersion 4.0.30319.42000
      BuildVersion 6.2.9200.22198
      PSCompatibleVersions {1.0, 2.0, 3.0}
      PSRemotingProtocolVersion 2.2
      
      I realize now that I am probably using version 1.82 of the script as well.
      
      LikeLike
      
      Reply ↓
    - GSinnema on November 18, 2017 at 2:48 PM said:
      
      Same problem here, Using script version 1.82, Exchange 2010 SP3, Windows Server 2012, Powershell version 3.0
      
      LikeLike
      
      Reply ↓
      - GSinnema on November 18, 2017 at 2:50 PM said:
        
        WARNING: Error performing operation FindFolders without Search options in . Error: Exception calling “FindFolders” with “2” argument(s): “Value cannot be null.
        Parameter name: parentFolderId”
        VERBOSE: Adding folder
        \\\\\\\\\\\( Lots of \ here )\\\\\\\\\\\\\
        (priority 0)
        E:\Mailimport\Remove-DuplicateItems.ps1 : Cannot access mailbox information store, error: The script failed due to call
        depth overflow.
        At line:1 char:1
        
        LikeLike
      - Michel de Rooij on November 18, 2017 at 2:51 PM said:
        
        I’ll try to come up with a workaround for older PS versions- seems like a thing with the regexp.
        For the time being, it’s tested and working on PS5 (e.g. w10).
        
        LikeLike
      - GSinnema on November 18, 2017 at 3:19 PM said:
        
        Tnx, I’ve updated WMF to version 5.0 on Server 2012 using: https://www.microsoft.com/en-us/download/details.aspx?id=50395
        The script is fully working now
        
        LikeLike
Kevin Bouwman on November 22, 2017 at 4:48 PM said:

I updated to 5.0 and the scripted worked for me too. This is great. I was surprised I didn’t have more duplicates. I found about 100 in 150,000 emails. It is really nice to have this cleaned up. Thanks Michel

LikeLike

Reply ↓
Ravi on January 24, 2018 at 5:42 PM said:

Awsm script !!! but can i run this for Sent items folder only ???

LikeLike

Reply ↓
- Michel de Rooij on January 24, 2018 at 6:16 PM said:
  
  Yes – use IncludeFolders and the tag #SentItems#, e.g. -IncludeFolders ‘#SentItems#’. But it seems you found a bug; if you only specify one element, it processes all folders. Bug filed; workaround is to specify the same folder twice, e.g. -IncludeFolders ‘#SentItems#’, ‘#SentItems#’
  
  LikeLike
  
  Reply ↓
srecko jovancevic on March 6, 2018 at 9:31 AM said:

sorry your scri[t does not work on o365 I have scaned an mailbox qith cript 60000 mails and more an returned 0 duplicates and atleast 40 % are duplicates

LikeLike

Reply ↓
- Michel de Rooij on March 6, 2018 at 6:31 PM said:
  
  It works. Not finding any duplicaties is either a problem with duplicate detection, but more often related to permissions.
  
  LikeLike
  
  Reply ↓
Stefan Inhofer on April 13, 2018 at 12:50 PM said:

I have a Problem with Exchange 2016. The Skript detects duplicate contacts and removes them. But it does not detect duplicate Mails.
It lists all the Folders of the Mailbox that exist. It Shows the correct number of items that are in the Mailbox but it Shows 0 items removed. I spedify “-Mode Full -NoSize” but the duplicates are not recognized. The items are created by multiple runs of a restore, so they should be the same.

Using the user credentials or impersonation makes no difference.

LikeLike

Reply ↓
- Michel de Rooij on April 18, 2018 at 9:11 PM said:
  
  For e-mail with ‘Full’ there is a set of attributes which should match for it to be considered a duplicate. With NoSize, you can exempt size from this comparison, but if those others are not *all* identical, it’s not considered a duplicate. You can use tool like MFCMapi (or Outlook itself) to inspect the following attributes: DateTimeReceived, Subject, InternetMessageId, DateTimeSent, Sender and optional Size.
  
  LikeLike
  
  Reply ↓
Oleksiy Piantak on April 18, 2018 at 11:22 AM said:

Hello! Can you please explain How I can use -Mode Full with parameter NoSize

Thanks!

LikeLike

Reply ↓
- Michel de Rooij on April 18, 2018 at 9:13 PM said:
  
  Use Mode Full to use attribute-based matching of duplicates, not PidTagSearchKey. When using Mode Full, you can use NoSize to exempt the size of the items from the comparison.
  
  LikeLike
  
  Reply ↓
Anthony Garrett on April 24, 2018 at 9:49 AM said:

Hi Michel. I have been running this script against a fairly large mailbox with a fair amount of duplicates and am getting following errors:

VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
VERBOSE: Processing primary mailbox tony@billingford.com
VERBOSE: Collecting folders to process, type All
WARNING: Previous EWS operation failed, adjusted sleep timer to 300ms
C:\Users\Redblade\Downloads\Remove-DuplicateItems (1).ps1 : Cannot access mailbox information store, error: Unable to find type [Microsoft.Exchange.WebServices.Data.ServerBusyException].
At line:1 char:2
+ & ‘.\Remove-DuplicateItems (1).ps1’ -Mailbox tony@billingford.com -S …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems (1).ps1

Have checked impersonation which is setup correctly for the $usercredential account. Thing is if I run this against smaller mailboxes this seems to run fine, potentially ruling out impersonation rights or connectivity. There is another mailbox in the tenancy which is >10gb which gives the same error, all other mailboxes run fine. Any ideas would be much appreciated?

Thanks

LikeLike

Reply ↓
Jörg Dewitz on April 28, 2018 at 10:18 AM said:

Michel,

many thanks for this script. I run it on an Exchange 2016 server.

.\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Type Tasks -Impersonation -DeleteMode SoftDelete -Mode Full -Verbose –WhatIf

The script finds many duplicate mails and calendar entries but no duplicate tasks. However, there are many duplicate tasks in the mailbox.

Can you give me a clue?

Thanks a lot
J

LikeLike

Reply ↓
- Michel de Rooij on April 28, 2018 at 11:57 AM said:
  
  With Full mode, if one or more of the Subject, Start Date, Due Date, Status, Size attributes are different, it’s not considered a duplicate.
  
  LikeLike
  
  Reply ↓
  - T. Nieresel on May 2, 2018 at 12:51 PM said:
    
    Hello Michel,
    
    I have a similar problem with the processing of tasks in user mailboxes. I’ve used the following Parameters:
    .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Type All -Impersonation -DeleteMode HardDelete -Verbose
    
    The Skript was able to remove evry duplicate item from the mailbox except for tasks. Even with specifing tasks as the inteded search type and playing aroud with the -DeleteMode parameter, I was unable to remove them.
    
    Does it have anything to do with the german translation for the task folder? Or is there anything else I can try?
    
    Thank you in advance!
    
    Best regards,
    T. Nieresel
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 2, 2018 at 11:13 PM said:
      
      Not related to localization; I’ll have look
      
      LikeLike
      
      Reply ↓
Oleksiy Piantak on May 1, 2018 at 7:09 PM said:

Hello!
Can you please explain I have larger mailbox 90 GB and into folders about 80k items.
Script didn’t work properly and didn’t all cleaned up msg. Used Full mode
Why? And How are using pig tiles or maybe more parameters?
Thanks!

LikeLike

Reply ↓
- Michel de Rooij on May 1, 2018 at 7:27 PM said:
  
  1) What you mean by pig tiles?
  2) Duplicate checks are performed on multiple attributes which all need to be equal depending on the class to be considered duplicate. That is not always the case as 3rd party software may touch items or dehydrate stubs which then look the same, but are not. Read the help for info on attributes to be considered.
  
  LikeLike
  
  Reply ↓
Rob Gray on May 5, 2018 at 11:04 AM said:

Michel
I have managed to modify your script in a fairly minor way to work on Public Folders.
Would you like me to submit my version (1.83PF) to you as I dont feel publishing it myself is valid as 95% ( & probably more) of the work is yours !

LikeLike

Reply ↓
John on June 5, 2018 at 9:08 PM said:

hi when Using mailboxwide switch, how to change the priority of the folders. As I want to keep the emails in all the subfolders instead of Inbox and Important Folders after migrating IMAP mailbox to Office 365.

LikeLike

Reply ↓
- Michel de Rooij on June 9, 2018 at 12:57 PM said:
  
  Use PriorityFolders to give folders priority over others; items founds in these folders are retained over duplicates in other folders. The order of folders specified with PriorityFolders also impacts which items to retain; PriorityFolders A,B will have items in folder A retained over duplicates in B. PriorityFolders also accepts well known folders, wildcards etc.
  
  LikeLike
  
  Reply ↓
dan on June 14, 2018 at 3:07 PM said:

Somehow a user’s contacts got duplicated and then some. She’s got over 500,000. I ran this once and it picked up and removed ~105,000. It completely ignored most of them though. One for example, there appears to be hundreds of, and it only has a name (which is a phone number) and company name. That’s not being picked up in quick or full mode. Any suggestions?

LikeLike

Reply ↓
- Michel de Rooij on June 14, 2018 at 3:51 PM said:
  
  In Full, all these attributes need to be an exact match to be considered a duplicate:
  FileAs, GivenName, Surname, CompanyName, BusinessPhone, HomePhone, MobilePhone and Size
  
  LikeLike
  
  Reply ↓
  - amanfromthere on June 14, 2018 at 5:06 PM said:
    
    Even in quick mode, it’s not finding any of them.
    
    End of the log…
    –
    VERBOSE: Collecting folders to process, type Contacts
    VERBOSE: Adding folder \Contacts (priority 0)
    VERBOSE: Adding folder \ExternalContacts (priority 0)
    VERBOSE: Adding folder \PersonMetadata (priority 0)
    VERBOSE: Found 3 folders that match search criteria
    Processing folder \Contacts
    Processing folder \ExternalContacts
    Processing folder \PersonMetadata
    1752592 items processed and 0 removed in 02:36:45 – average 11181 items/min
    WARNING: Cannot bind to ArchiveMsgFolderRoot – skipping. Error: Exception calling “Bind” with “2” argument(s): “The specified folder could not be found in
    the store.”
    VERBOSE: Processing xx@yy.net finished
    
    Still have ~390,000 that need to be removed, grateful for any advice you can provide.
    
    Thank you
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on June 15, 2018 at 10:05 AM said:
      
      Depending on how duplicates were created, it might find attributes are bit different causing small variations in size. Give -NoSize a shot (with -Report -WhatIf:$true)
      
      LikeLike
      
      Reply ↓
amanfromthere on June 14, 2018 at 3:08 PM said:

Somehow a user’s contacts got duplicated and then some. She’s got over 500,000. I ran this once and it picked up and removed ~105,000. It seemed to ignored most of them though. One contact for example there appears to be hundreds of, and it only has a name (which is a phone number) and company name. That’s not being picked up in quick or full mode. Any suggestions?

LikeLike

Reply ↓
Bora Buyuksoy on September 14, 2018 at 3:16 PM said:

Hi,
nice work but when I run the command, only 1786 items get processed in inbox and no more. Is there a limit set somewhere? There are more than 150000 mails in inbox only.

Thanks

LikeLike

Reply ↓
- Michel de Rooij on September 14, 2018 at 3:40 PM said:
  
  There is no programmatic limit
  
  LikeLike
  
  Reply ↓
  - Bora Buyuksoy on September 14, 2018 at 3:42 PM said:
    
    What do you think might be the problem?
    
    LikeLike
    
    Reply ↓
- Bora Buyuksoy on September 14, 2018 at 5:57 PM said:
  
  Found the issue: I was using admin account to access the client mailbox. When I used -credentials switch with users own credentials it started to process all mails.
  
  Thanks
  
  LikeLike
  
  Reply ↓
Jonas on October 29, 2018 at 2:17 PM said:

Hi Michel.

First thanks, thanks a lot for this script and for all the information about Exchange in your blog.

We have a problem with your script. The script run without problems but it detects a lot of emails duplicated, says at the end that they have been removed but the emails duplicated (and triplicated) are still in the mailbox.

[PS] >.\Remove-DuplicateItems.ps1 -Mailbox xxx@xxxx.com -Type mail -Impersonation -DeleteMode harddelete -Mode full -IncludeFolders “#sentitems#” -mailboxonly -confirm:$false

Processing mailbox xxx@xxxx.com (xxx@xxxx.com)
93158 items processed and 20903 removed in 00:10:50 – average 8597 items/min

We tried with impersonation and with the user’s credentials but the results were the same.

We tried to check only the folder or all the mailbox but again the results were the same.

At the first execution the script removed some of the detected duplicated emails but didn’t remove all of them.

After that it doesn’t remove more duplicated emails even though it detects them.

I can’t imagine why the script detects but doesn’t remove them. Do you have any idea about it?

Thanks in advance for your response.

LikeLike

Reply ↓
- Michel de Rooij on October 29, 2018 at 3:26 PM said:
  
  And you are validating where? Outlook in cached mode can take some time to reflect changes, as the deletion is asynchronous.
  
  LikeLike
  
  Reply ↓
  - Jonas on October 29, 2018 at 3:56 PM said:
    
    Thanks for the quick response.
    
    I’m checking the mailbox folder statistics in server side and I’m trying to remove the items since some days ago. The duplicated emails are still there.
    
    I don’t know why it doesn’t remove the duplicated emails because there is no errors in the execution. Maybe I have to put some flags and modify the script to debug my problem and see where is the strange thing which produce this behavior.
    
    Thanks.
    
    LikeLike
    
    Reply ↓
  - Jonas on October 31, 2018 at 6:00 PM said:
    
    At the end I distribute the duplicated email in 4 folders with 10.000 duplicated emails each one and it works as desired. The emails are analyzed, duplicates founded and removed (each folder contains 7.000 email duplicated more ore less)
    
    The problem seems to be when there a too much emails to remove.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 1, 2018 at 8:19 AM said:
      
      Glad you got it working. Had people running it without issues for >50k items in folders, but that was a while ago. I’ll have another look at operations at scale. Thanks for your feedback.
      
      LikeLike
      
      Reply ↓
Michael Donovan on November 13, 2018 at 9:06 PM said:

Great thanks go out to you Michel as you patiently navigate these questions and provide excellent detailed answer. I just wanted to pass along a tip on an issue I was having.
All of my attempts to use this script were failing with authorization errors. I finally ran it against my own mailbox with the same results. Double and triple checked my credentials…still no luck.
Knowing I had MFA enabled, I used my app password and it all started working.

Really very nice of you to share and continue to develop a script like this. We as a community can not thank you enough.

LikeLike

Reply ↓
- Michel de Rooij on November 13, 2018 at 11:14 PM said:
  
  Yeah, MFA could be blocking, EWS protocol could be disabled at the mailbox level, basic auth could be blocked in Exchange Online for EWS .. lots of conditions.
  
  LikeLike
  
  Reply ↓
Michael Donovan on November 13, 2018 at 11:41 PM said:

So I have a quick question. I have some shared mailboxes that I have full access to. I am identifying the shared mailbox with the identity parameter. I am using my credentials for authentication. In verbose mode, the script says its processing the shared mailbox. But the scan appears to be scanning my own mailbox. I do not have impersonation enabled. And if I try to use the switch it fails. Must I have impersonation even if I have full access?
On the bright said, I have thoroughly cleaned dupes out of my own mailbox. Command line below is what I used (notwithstanding some anonymity.)
$creds = get-credentials
.\Remove-DuplicateItems.ps1 -Identity sharedmailbox@company.com -type Mail -Server outlook.office365.com -Credentials $creds -mode Quick -DeleteMode SoftDelete -MailboxWide -MailboxOnly -Verbose

Some of the verbosity was…
VERBOSE: Processing folder \Inbox\FTC
VERBOSE: Processing folder \Inbox\Incident emails
VERBOSE: Processing folder \Inbox\Helpdesk
32310 items processed and 37 removed in 00:03:41 – average 8750 items/min
VERBOSE: Processing sharedmailbox@company.com finished

The folders mentioned above were in my mailbox not the shared mailbox.

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2018 at 12:11 PM said:
  
  Let me check..
  
  LikeLike
  
  Reply ↓
- Michel de Rooij on November 17, 2018 at 12:59 PM said:
  
  Please get update 1.86 which should solve this.
  
  LikeLike
  
  Reply ↓
  - Michael Donovan on December 13, 2018 at 8:31 PM said:
    
    I have to follow up on my use of this great script. I finally got application impersonation permissions to run this. (This is a difficult role to explain to someone even if they know PowerShell really well.)
    In short, this script works great for me now. I discovered that using the mode body yields the most dupes. I encourage anyone using this to try many combinations of the switches to get what you need form this.
    Michel-Thanks again! I can’t help notice what a great over all site you have. I am starting to really dig into in. You are indeed an MVP!
    
    LikeLike
    
    Reply ↓
Mario Mesa on November 15, 2018 at 6:27 PM said:

Good afternoon; is it possible to delete duplicates only based on Subject Line and date?

My used got the same email twice multiple times from the same user but from different alias so the only thing that it matches besides the body content is the subject line and date which are the same.

Thanks;
Mario

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2018 at 6:32 PM said:
  
  Not yet, but working on version where you can customize the fields used for matching. Goes without saying that matching on less fields means greater chance of matching on less identical items (matching on only subject and date sounds bit risky)
  
  LikeLike
  
  Reply ↓
  - Mario Mesa on November 15, 2018 at 6:35 PM said:
    
    Agree; unfortunately thats my case 😦
    Let me know if this new version is coming out soon.
    
    LikeLike
    
    Reply ↓
Michael Donovan on November 15, 2018 at 7:35 PM said:

So I have a significant update on my issue. I had been running this program on my workstation with my Outlook open. In retrospect, that was quite foolish. I am trying it on an exchange server (which also feels foolish.) I believe my issues are one of environment. I will report my findings as they develop.

LikeLike

Reply ↓
lt14 on November 29, 2018 at 9:11 AM said:

After my Remove-DuplicateItems.ps1 was executed and completed, some mailboxes got an automatic email about “Unread email has been deleted”. Some users did have unread duplicated emails and it looks like the script or Exchange-server send an email to the sender of all unread emails, so both internal and external recipients in other organisations got a notification about this. Do you know how to turn of this notification emails?

LikeLike

Reply ↓
- Michel de Rooij on May 2, 2019 at 7:03 PM said:
  
  What unread mail deletion notification is that? Is it from on-prem Exchange? I do suppress read notifications (supported for Ex2013+) and meeting cancellations.
  
  LikeLike
  
  Reply ↓
  - lt14 on May 8, 2019 at 2:54 PM said:
    
    It was Exchange 2016 CU11. What I remember, it was NDRs back to the senders of those old emails saying “You got an unread email deleted by the recipient…”. We did not test to manually delete some of these old unread emails to see if that also gerenated NDRs. We did just close that ticket. Let’s see if it happens more times. And can reply here with more details in that case. Thanks for a good script btw!
    
    LikeLike
    
    Reply ↓
Kevin on February 21, 2019 at 4:36 AM said:

You are a life saver!! After my parent company switched from exchange to Gmail (losing a few years worth of emails in the process), I’m now switching back to Office 365 for my team and the labeling in gmail created so many issues. Thanks to your script I can merge the gmail inbox AND the original PST file from the exchange server and not have to panic about the duplicates. Incredibly well explained and detailed and reading through the responses above saved me a ton of time. (I had never used a script before in Powershell and with your direction was able to get this up and running in less than an hour) Thanks again!

LikeLike

Reply ↓
- Michel de Rooij on February 21, 2019 at 7:53 AM said:
  
  Glad is was helpful
  
  LikeLike
  
  Reply ↓
Bartel Onzea on April 30, 2019 at 1:23 AM said:

Great script. I am running into problems with mailboxes in a different language. I cannot do -DeleteMode MoveToDeletedItems because the DeletedItems for a German mailbox is named ‘Gelöschte Elemente’ (Error:Exception calling “DeleteItems”…

LikeLike

Reply ↓
- Michel de Rooij on May 2, 2019 at 7:16 PM said:
  
  That shouldn’t be the problem, because we’re referring to the DeletedItems folder through something called the WellKnownFolders, which is language-independent. Care to elaborate a bit more on the issue you’re encountering?
  
  LikeLike
  
  Reply ↓
BDP on May 14, 2019 at 11:32 PM said:

I’m still running into the EWS throttling issue on large mailboxes (error below; example mailbox is ~200,000 items). I’ve read some comments that this could be an issue with interrupted internet connectivity. So I re-ran the command from an Azure server to alleviate any local issues and ran into the same error.

Cannot access mailbox information store, error:
Exception calling “LoadPropertiesForItems” with “2” argument(s): “The request failed. The read operation failed, see inner exception.”

I’ve tried multiple combinations of parameters and it seems to be only affecting running in Full mode (full script below).
.\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Credentials $creds -Mode Full -NoSize -Impersonation -Server outlook.office365.com -Type mail -verbose

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2019 at 9:40 PM said:
  
  1) What version of EWS Managed API are you using?
  2) Mailbox parameter has been renamed to Identity for a while now.
  First thing the script does is collect all folders and select the ones which pass the Include/Exclude filter. That should not be limited due to no. of items in a folder. That ‘Exception calling “LoadPropertiesForItems” with “2” argument(s): ‘ is the only thing reported?
  
  LikeLike
  
  Reply ↓
  - Bardia Mehrabian on May 26, 2019 at 6:51 AM said:
    
    I can’t speak for BDP, but I am facing the same exact issue across multiple different mailboxes. I am trying this against O365. The EWS Managed API I’m using is v15.00.0913.015. The exact error that I get is:
    
    TerminatingError(): “Exception calling “LoadPropertiesForItems” with “2” argument(s): “The server cannot service this request right now. Try again later.”
    
    It definitely seems to be struggling with large mailbox sizes with a couple folders that are 10GB+ (50k+ items). Unsure how to handle this situation. Is it best to try and split these across multiple folders in some way? Guess I’ll have to find a way to do that programmatically…
    
    LikeLike
    
    Reply ↓
Jonathan TAYAR on September 20, 2019 at 6:50 PM said:

Hi,

First of all, thanks a lot for your help to the community.I’ve discovered your thread a few weeks ago. I can’t believe you’ve helped so many people. I’m also having issues on a migration from G Suite to O365.
I’ve downloaded your script. I’ve been trying to use it for several days. I went through most of the issues people also had on this topic, and got them solved one after the other. (credentials, impersonation, etc).

I’ve run the script in Full Mode and it did not find a lot of duplicates (but I know I have at least 17Gb of them :-). I’ve just launched a new run with the -noSize option, I’ll see if it works.

I have one question : I have found manually a few examples of emails that are duplicated. Following example is :
– the original email is folder “Important”
– the first copy I can find is in folder “PARTNER A” (G suite label). On that email I can spot one difference : Office has considered that my forwarding this email was to be in this folder, not in important. So the mailchain is a bit different here in this folder
– the second copy I can find is in folder “Archive”, and it looks pretty much like the one in folder “Important”.
Do you know a way for me to check the matching criterias in the script on these 3 particular emails to understand why they were not picked up ?

My current command line :
.\Remove-DuplicateItems.ps1 -Mailbox myuser@mydomain.onmicrosoft.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Quick -Server outlook.office365.com -Impersonation -Credentials (Get-Credential) -MailboxWide -NoSize -Report -WhatIf:$true

Thanks in advance for your answer !

LikeLike

Reply ↓
- Jonathan TAYAR on September 20, 2019 at 6:52 PM said:
  
  Just realized : better with Full Mode to use NoSize
  .\Remove-DuplicateItems.ps1 -Mailbox myuser@mydomain.onmicrosoft.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Quick -Server outlook.office365.com -Impersonation -Credentials (Get-Credential) -MailboxWide -NoSize -Report -WhatIf:$true
  
  My question is still on 🙂
  
  LikeLike
  
  Reply ↓
Gui Lima (@guilhermelima) on February 23, 2020 at 1:30 AM said:

Hi dear,
It is an amazing script, really liked it. What if we could use an parameter to say what should be check as a duplicated, like “messageID”. In my environment, mails were duplicated due an replication during migration and with an forward message redirection. It means that mails were received with seconds in difference between the migration process and the forward process. If I had the parameter to say that the messageID is the same, the duplicated mails could be removed.

Thank you for sharing it! 🙂

LikeLike

Reply ↓
Jamie H on May 5, 2020 at 12:41 PM said:

Hi,

Thanks for this handy script, I’m having an issue in full mode however where it says Cannot access mailbox information store, error: Exception calling “LoadPropertiesForItems” with “2” argument(s): “Requested value ‘GroupMailbox’ was not found.

I am using exchange api 2.2 with the following command .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Server outlook.office365.com -Credentials $Credentials -Mode Body -PriorityFolders ‘Holidays’,’Relief’,’IT Related Stuff’,'[Gmail]’,’Starred’ -MailboxWide -Report -Verbose

Any ideas?

Thanks
Jamie

LikeLike

Reply ↓
- Michel de Rooij on May 5, 2020 at 1:28 PM said:
  
  2.2 doesn’t understand Group Mailboxes – the updated 2.2.1.2 can process them. Get the package from here, https://www.nuget.org/packages/Exchange.WebServices.Managed.Api/, or when its too complex with nuget, use ‘Download Package’, open the nupkg archive with 7zip or something else, and extract the Microsoft.Exchange.WebServices.dll & xml files to the folder where the script resides. When using verbose, script should report ‘Loaded EWS Managed API v2.2.1.0’.
  
  LikeLike
  
  Reply ↓
  - Roger on November 19, 2020 at 6:14 PM said:
    
    Hello Michel,
    I am getting exactly the same error as Jamie.
    I performed the fix you mentioned, and turned on verbose output, and it reports it’s “Loaded EWS Managed API v2.2.1.0” but still gets the same error.
    Any further adice?
    Thanks,
    Roger.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 26, 2020 at 12:01 AM said:
      
      Could you elaborate bit more on the scenario?
      
      LikeLike
      
      Reply ↓
      - Roger on December 9, 2020 at 6:50 PM said:
        
        Hello Michel,
        Sorry for not replying sooner – I got it working after creating a new VM, and setting all back up – thanks!
        One question – I copied an email to my Inbox and it wasn’t detected as a duplicate – is it because a modified time maybe different even though the received time is the same?
        Thanks,
        Roger.
        
        LikeLike
      - Michel de Rooij on February 22, 2021 at 5:42 PM said:
        
        Copy is new instance- it might only be considered duplicate when performing attribute-level comparison.
        
        LikeLike
Sven Schaffranneck on June 12, 2020 at 10:34 PM said:

Hi, any news on removing duplicate items for public folder? Or may anyone else has a fork which works with PF? Thank you very much for your work so far!

LikeLike

Reply ↓
- Rob Gray on June 14, 2020 at 11:37 PM said:
  
  I have modified version that works on our public folder configuration….
  note it is based 99% on v1.86 of Michel’s script & requires a mailbox with access to the entire tree to be reviewed.
  https://www.dropbox.com/s/mjfu94ie2nf6rlv/RemDup186rg2.ps1?dl=0
  
  I run it on a portion of the tree at a time.
  Write-Output “Start– $(get-date -Format “ddd dd MMM yyyy HHmmss”) —–”
  if(!$Credentials)
  {
  $Credentials= Get-Credential;
  }
  
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2004” -Credentials $Credentials -Impersonation -Report -Force
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2005” -Credentials $Credentials -Impersonation -Report -Force
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2006” -Credentials $Credentials -Impersonation -Report -Force
  
  I will have to diff it with current v1.88
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 15, 2020 at 8:10 AM said:
    
    Sorry Sven, did not have time yet to put some decent effort in myself. But power of community and GitHub mechanics allow anyone to improve or enhance functionality. It’s also not off my list.
    
    LikeLike
    
    Reply ↓
    - Rob Gray on June 15, 2020 at 11:44 PM said:
      
      Michel. I have uploaded my code as an ‘issue’ to Git as dont know how to create a branch on your git.
      
      LikeLike
      
      Reply ↓
- Rob Gray on June 17, 2020 at 12:01 PM said:
  
  fork created off 1.88; refer github
  
  LikeLike
  
  Reply ↓
  - Rob Gray on May 14, 2021 at 5:01 AM said:
    
    fork updated to 2.03; works for OnPrem. not tested for cloud based.
    https://github.com/RobGray-au/Remove-DuplicateItems
    
    LikeLike
    
    Reply ↓
skifwhite on July 14, 2020 at 3:46 PM said:

is it possible to use this script with MFA? Or to disable the MFA is a necessary step?

LikeLike

Reply ↓
- Michel de Rooij on July 14, 2020 at 5:40 PM said:
  
  Implementing modern authentication support is on the shortlist.
  
  LikeLike
  
  Reply ↓
  - skifwhite on July 14, 2020 at 9:36 PM said:
    
    Can we expect this feature on the nearest update?
    
    LikeLike
    
    Reply ↓
  - Mars on July 16, 2020 at 8:23 AM said:
    
    Hello Michel, would be great if you could implement this in nearest release, this feature is expected very much on our side, thanks a lot for your magic script!
    
    LikeLike
    
    Reply ↓
  - skifwhite on July 16, 2020 at 2:28 PM said:
    
    Could you please also clarify, what I do wrong?
    PS C:\Users\User\Downloads\Remove-DuplicateItems-master> .\Remove-DuplicateItems.ps1 -Identity ‘user@contoso.com’ -Server outlook.office365.com -Credentials $Credentials -IncludeFolders ‘#Inbox#/03 COOL/0 BPM’ -DeleteMode SoftDelete -MailboxOnly -verbose
    VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loaded EWS Managed API v15.00.0913.015
    VERBOSE: Set to trust all certificates
    VERBOSE: Using credentials admin@contoso.com
    VERBOSE: Processing items of type All, delete mode is SoftDelete
    Processing mailbox user@contoso.com (user@contoso.com)
    VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
    VERBOSE: Constructing folder matching rules
    VERBOSE: Processing primary mailbox user@contoso.com
    VERBOSE: Collecting folders to process, type All
    
    (I have changed user email intentionally) It stops on the latest message and nothing continue.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 16, 2020 at 2:39 PM said:
      
      When granting FullAccess, did you specify InheritanceType All
      
      LikeLike
      
      Reply ↓
Moula Muhaidyn on August 16, 2020 at 1:10 PM said:

How to delete the duplicate in In-Place Archiving in Exchange Server

LikeLike

Reply ↓
- Michel de Rooij on August 16, 2020 at 9:53 PM said:
  
  By default, it will process in-place archives as well if the mailbox has an associated in-place archive.
  If you want to process in-place archives only, there’s a switch ArchiveOnly for that.
  
  LikeLike
  
  Reply ↓
Josh Gardner on September 29, 2020 at 10:22 PM said:

Michel,
I really look forward to getting this to work, however I seem to run across a similar issue as many others:

Cannot access mailbox information store, error: Exception calling “.ctor” with “0” argument(s): “This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.”

I have given myself impersonation rights, as well as full permissions to the mailbox in question (a different user). I’ve tried both -Impersonation and the -Credential method.

I see a lot of issues for people regarding “Cannot access mailbox information store” but that seems to be the permission issue that is supposed to be fixed by giving impersonation/full access (with inheritance). However, I don’t see anything regarding the follow up part of the error mentioning FIPS. Unfortunately I’m in an environment where I cannot simply “disable FIPS”, it is a security requirement. Does that mean I’m out of luck? Is there some sort of switch that bypasses what FIPS may be complaining about? Some sort of alteration in the script that will allow it to be FIPS “compliant”. It seems to scope the mailbox just fine in the debug, but fails right after it starts to process the folders. I can even see the first item of the folder that it hits. If you have any idea how to address this, I’d super appreciate it.

.\Remove-DuplicateItems.ps1 -Mailbox user.samaccountname -Type all -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -Debug

I’m on-prem with Exchange 2016 CU15
Using Remove-DuplicateItems.ps1 Version 1.88
$PSVersionTable

Name Value
—- —–
PSVersion 5.1.14393.3866
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14393.3866}
BuildVersion 10.0.14393.3866
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

If, in the case I’m out of luck because of FIPS, I still appreciate your time, not only in responding, but creating what looks to be a superb script.

LikeLike

Reply ↓
- Michel de Rooij on October 1, 2020 at 7:14 PM said:
  
  Sorry, scripts are untested with FIPS locked down environments. That said, it’s not that the script shouldn’t work, I assume it’s a matter of crypto settings and defaults. Not totally into this, but could you set this (I assume no it environment is locked down) and run the script?
  HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy\Enabled=0
  
  LikeLike
  
  Reply ↓
  - Josh Gardner on October 13, 2020 at 3:20 PM said:
    
    Sorry for the delay. Unfortunately, manually setting the registry key (as you mentioned above) did not work. Nor did temporarily removing the GPO that enabled FIPS cryptography. Very unfortunate. I was going through your script to find what could potentially be causing this. I noticed this section:
    
    Function Get-Hash {
    param(
    [string]$string
    )
    $md5 = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
    $data = ([system.Text.Encoding]::UTF8).GetBytes( $string)
    return ([System.BitConverter]::ToString( $md5.ComputeHash( $data)) -replace ‘ – ‘, ”)
    }
    
    When I run that function all by its lonesome in a PowerShell session, I get presented with the same error! I have changed the encoding from MD5 to SHA256 in that function and now it works (or so it seems, I’m still running a test). OOOH man, so excited lol. Thanks. Crossing my fingers 🙂
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on October 13, 2020 at 3:24 PM said:
      
      That actually makes sense. Let me know, might need to update it with hashing method using more accepted method nowadays 🙂
      
      LikeLike
      
      Reply ↓
      - Josh Gardner on October 14, 2020 at 12:54 PM said:
        
        The script works fantastically now. Love it! It has relieved a huge weight off my chest. Thanks again!
        
        LikeLike
Jonathan K on October 1, 2020 at 9:52 PM said:

This script looks perfect, especially since I was advised post-migration that users had been forwarding all their email from an old domain to their new one, creating duplicates of their entire mailboxes. I’m trying to run this from the admin context against Office365, but can’t get it to work. Do you need to use the credentials of the account that you’re accessing, or can a Global Admin’s account be used? That seemed to be the function of the impersonate, but I haven’t been able to get it to execute properly. I’ve managed to get as far as getting an auth error with the user mailboxes. Appreciate your help and your awesome script!

LikeLike

Reply ↓
- Jonathan K on October 1, 2020 at 10:00 PM said:
  
  And it turns out I’m an idiot. Got it working, it looks great. Thank you!
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on October 1, 2020 at 10:27 PM said:
    
    Glad you got it working
    
    LikeLike
    
    Reply ↓
Pedro Sgobin on October 16, 2020 at 8:30 PM said:

Using your script to clean the GSuite to Office 365 duplication mess.
Thank you.

LikeLike

Reply ↓
Patrick on November 26, 2020 at 8:13 PM said:

Hello,
first of all Thanks for the Script and the Script to remove Items from a Mailbox. Is there a Way to only prioritize Subfolders of a Folder and not the Folder itself?

LikeLike

Reply ↓
- Michel de Rooij on November 27, 2020 at 12:38 AM said:
  
  Yeah, it’s a bit ugly, but try -PriorityFolders ‘\Folder\**’. Use -Verbose to validate processing and prioritizing of folders.
  
  LikeLike
  
  Reply ↓
Pingback: Annual Report 2020 | EighTwOne (821)
Kevin on February 1, 2021 at 6:51 PM said:

Can this script (or maybe this one with modifications) be used to remove duplicates between two separate non-archive mailboxes? I’m hoping to locate something that can identify e-mails with identical message IDs distributed between two mailboxes and allow the deletion of one set of messages. The ability to dump the IDs and message properties to a CSV for review would also be useful. Any thoughts or pointers to get me going in the right direction? Have strong Powershell skills but the EWS component has always featured a layer of complexity beyond what I’m familiar with. Thanks in advance!

LikeLike

Reply ↓
- Michel de Rooij on February 3, 2021 at 4:50 PM said:
  
  This script, no. Can it be made, yes. However, that would basically mean going through both, collecting things uniquely identifying mail in one (e.g. message ID and path for reference), and looking for presence in the other (reporting ID and path in mailbox A and mailbox B). That is the business case, if I may ask?
  
  LikeLike
  
  Reply ↓
Kevin on February 3, 2021 at 10:24 PM said:

Thanks Michel. Business case is a one-off unique situation – a user’s mailbox was split into two due to high folder counts and at Microsoft support’s recommendation. It was discovered later that for a period of time, emails were delivered to both mailboxes. We want to dedupe these message before we archive his mailbox as the dupes will be a source of annoyance / confusion for the end user.

I’ve opened a case with Microsoft about this as well, but thus far they’ve been less than helpful. Your suggestions are helpful though, and I’ve made some progress just binding to a folder and producing FirstClassProperties of items.

LikeLike

Reply ↓
- Michel de Rooij on February 28, 2021 at 5:35 PM said:
  
  Hi Kevin, to circle back on this: Could be more simple on second thought: The entry which initializes the variable to keep track of entries ‘seen’ before is initialized at the start when processing the mailbox. It is re-initialized after processing the folder, depending MailboxWide switch. You could of course move initialization to start of script, and comment all reinitializations out. Then, pass the mailboxes you want to process (you can pass more identities) so it will ‘know’ items seen in the first mailbox, and identical items from second (and third etc). identity will be considered duplicate. Now, to save you work, in 2.03 I removed MailboxWide and introduced -CleanupMode where you can specify MultiMailbox (and fixed identity parameter accepting multiple entries). So, you can do this (added whatif & report to make example non-destructive):
  .\Remove-DuplicateItems.ps1 -Identity peter@contoso.com,frank@contoso.com .. -CleanupMode MultiMailbox -Verbose -WhatIf:$true -Report
  
  LikeLike
  
  Reply ↓