Removing Duplicate Items from a Mailbox

Posted on June 21, 2013 by Michel de Rooij

Latest version: 2.41, April 18th, 2023

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
Importing existing data. Accidental import from – for example – a PST file to a mailbox can lead to duplicate entries.

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange Web Services (EWS) Managed API and for OAuth authentication the Microsoft Authentication Library (MSAL) libraries. You can install these packages from NuGet, or place their DLL’s in the same folder as the script. For an example of how to install EWS.Managed.Api from NuGet, see this article; for MSAL follow the same process but with the package titled ‘Microsoft.Identity.Client’.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions when using Basic Authentication; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post. Using a registered app with OAuth is always through Impersonation.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [[-Identity] ] [[-Type] ] [-Retain ] [-Server ] [-Impersonation] [-DeleteMode ] [-Credentials ] [-Mode ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-PriorityFolders <String[]>] [-NoSize] [-CleanupMode] [-NoProgressBar] [-Force] [-WhatIf] [-Confirm] [-Secret] [-CertificateThumbprint] [-CertificateFile] [-CertificatePassword] [-TenantId] [-ClientId] [-TrustAll] [-ExchangeSchema <String>] [-NoSCP]

A quick walk-through on the parameters and switches:

Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:

ItemClass	Criteria
Contacts	File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List	FileAs, Number of Members, Size
Calendar	Subject, Location, Start & End Date, Size
Task	Subject, Start Date, Due Date, Status, Size
Note	Contents, Color, Size
Mail	Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other	Subject, DateTimeReceived

MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter when running against mailboxes on Exchange Server 2007.
ArchiveOnly specifies you only want to process personal archives of specified users.
IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
CleanupMode specifies to cleanup duplicates per folder (Folder, default), the whole mailbox (Mailbox), or multiple mailboxes (MultiMailbox, identities specified using Identity). The first unique item encountered will be retained. For Mailbox-level cleanup, PriorityFolders can be used to give priority to retaining items in specified folders before those found in other folders.
PriorityFolders specifies which folders have priority over other folders, identifying items in these folders first when using MailboxWide mode. Usage of wildcards and well-known folders is identical to IncludeFolders.
NoSize tells script to not use size to match items in Full mode.
NoProgressBar prevents displaying a progress bar as folders and items are being processed.
Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.
TrustAll can be used to accept all certificates, e.g. self-signed certificates or when accessing Exchange using endpoint with a different certificate.
ExchangeSchema can be used to specify the Exchange schema to use when connecting to Exchange server or Exchange Online. Defaults to Exchange2013_SP1 or Exchange2016 when -Server is specified and is ‘outlook.office365.com’ (Exchange Online endpoint).
NoSCP to skip SCP lookups in Active Directory for Autodiscover.

For authentication, the following parameters are available:

Credentials specifies credentials to use for Basic Authentication.
TenantId specifies the identity of the Tenant (OAuth)
ClientId specifies the Id of the registered application (OAuth).
CertificateThumbprint specifies the thumbprint of the certificate from personal store to use for authentication (OAuth).
CertificateFile specifies the external certificate file (pfx) to use for authentication (OAuth). This certificate needs to contain a private key; the registered application needs to contain the certificate’s public key.
CertificatePassword optionally specifies the password to use with the certificate file (OAuth).
Secret specifies the secret to use with the application (OAuth).

Few notes:

When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter	Matches
FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA
Folder*	\TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\Folder	\TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\FolderA\	\TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So, suppose you want to remove duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

Alternative, you can use an e-mail address and specify credentials. This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Identity olrik@office365tenant.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

A more complex example using IncludeFolders, ExcludeFolders and PriorityFolders:

$Credentials= Get-Credential
 .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '#Inbox#\*','\Projects\*' -ExcludeFolders 'Keep Out' -PriorityFolders '*Important*' -CleanupMode Mailbox

This will remove duplicate items from the specified mailbox in Office365, using the following options:

Fixed Server FQDN – bypassing AutoDiscover.
Limits operation against the Well-Known Inbox folder, top Projects folder, and all of their subfolders.
Excluding any folder named Keep Out.
Duplicates are checked over the whole mailbox.
Priority is given to folders containing the word Important, causing items in
those folders to be kept over items in other folders when duplicates are found.

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 ..

Download
The script is available on GitHub here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

395 thoughts on “Removing Duplicate Items from a Mailbox”

Pingback: NeWay Technologies – Weekly Newsletter #48 – June 20, 2013 | NeWay
Pingback: NeWay Technologies – Weekly Newsletter #48 – June 21, 2013 | NeWay
CL on July 24, 2013 at 9:50 PM said:

First, this is a VERY cool script, so thank you. But I can’t get it to work, and am hoping you can help. I get an error stating “Cant access mailbox information store”
i’m running this from a management server (not an Exchange box). The account I’m using is an Exchange org admin. Thanks!!

LikeLike

Reply ↓
- Michel de Rooij on July 24, 2013 at 10:47 PM said:
  
  Sounds like a permission issue; check permissions or configure impersonation (see links in article for instructions)
  
  LikeLike
  
  Reply ↓
  - CL on July 25, 2013 at 2:06 PM said:
    
    Thank you, I had to grant impersonation rights. At first I granted Full Access, but I guess it didn’t work. I’m running Exchange 2010, in case anyone else has this issue. Thanks again for a VERY useful script!
    
    LikeLike
    
    Reply ↓
Joao Pedro on August 24, 2013 at 7:25 PM said:

Thank you for your awesome script!
It has been very useful.
Any chance to run it in Online Archive mailboxes?
Thank you!
JP

LikeLike

Reply ↓
- Michel de Rooij on August 24, 2013 at 9:46 PM said:
  
  Option to run against archives has been requested before. It’s on the ‘to do’ list, unfortunately no ETA yet.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on April 1, 2015 at 12:22 PM said:
    
    (Implemented)
    
    LikeLike
    
    Reply ↓
Danni on August 29, 2013 at 7:21 AM said:

It looks like its running fine, but don’t find the dubs that i made by copying mails, why is this? and why does it not run on custom folders?

LikeLike

Reply ↓
- Michel de Rooij on August 29, 2013 at 8:42 AM said:
  
  Depending on the way you copied, it could be the “duplicates” have a different PidTagSearchKey. Try running it in Full mode instead of the default Quick Mode (-Mode), in which case it’ll use a predefined set of attributes depending on the type of items to find matching items (see article for which attributes are checked). By default it will scan all folder types (-Type), looking for duplicates on a per folder basis.
  
  LikeLike
  
  Reply ↓
  - Danni on August 29, 2013 at 8:47 AM said:
    
    Thanks, I found the solution. it does not work fully before i configure the impersonation rights. Im running it from a workstation with web services 2.0 against a exchange 2010 installation. Wonderful script you have made!
    
    LikeLike
    
    Reply ↓
Matt Shooshtari on September 25, 2013 at 1:05 AM said:

I receive an error with the script when installed on Exchange 2012. I do have the dll in the folder with the ps1, and have the EWS installed. I downloaded again and verified the download.

…..Remove-DuplicateItems.ps1:349 char:156
+ … ationsMode and $AffectedTaskOccurences:$AffectedTaskOccurences”
+ ~~~~~~~~~~~~~~~~~~~~~~~~
Variable reference is not valid. ‘:’ was not followed by a valid variable name character. Consider using ${} to delimit the name.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : InvalidVariableReferenceWithDrive

LikeLike

Reply ↓
Bis on September 26, 2013 at 5:49 PM said:

I get the following error :
You cannot call a method on a null-valued expression.
At C:\psscripts\Remove-DuplicateItems.ps1:271 char:82
+ $key= $Item.DateTimeReceived.ToString <<<< ()
+ CategoryInfo : InvalidOperation: (ToString:String) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull

LikeLike

Reply ↓
Maarten on September 30, 2013 at 1:54 PM said:

Hello,

When I run the script, everything looks fine, except for the fact that is removes 0 items. It’s for shure that the mailbox where I try it on is full of duplicate items. This is the best way I found on the net to remove duplicate items, but doesn’t work for me. Please Help!

LikeLike

Reply ↓
- Michel de Rooij on September 30, 2013 at 2:15 PM said:
  
  First of all, it only scans per folder. Second, it’s likely to be permission related.
  
  LikeLike
  
  Reply ↓
Torbjörn Granheden on November 28, 2013 at 1:57 PM said:

Hi
Wonderful script!
I´m doing it on exchange 2013 and has change some of the rows to get it working.
1. First probably a typo:
… and $AffectedTaskOccurences:$AffectedTaskOccurences”
… and AffectedTaskOccurences:$AffectedTaskOccurences”
2. Change all text: Exchange2007_sp1 to Exchange2013
3. On 2 occurences replace:
$ItemIds= [activator]::createinstance(([type]’System.Collections.Generic.List`1′).makegenerictype([Microsoft.Exchange.WebServices.Data.ItemId]))
with:
$type = (“System.Collections.Generic.List”+’`’+”1”) -as “Type”
$type = $type.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
$Itemids = [Activator]::CreateInstance($type)

Now the script works! But still getting an error at the end:
C:\root\Remove-DuplicateItems.ps1 : Problem processing mailbox it (it@google.com)
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -mailbox it -type all -impersonation -deletemode har …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
I Think it fails the error check at the end but cannot solve it.
$RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
If(! ( Process-Mailbox $RootFolder)) {…

Any ideas?

/MrTbone

LikeLike

Reply ↓
- Michel de Rooij on November 28, 2013 at 2:08 PM said:
  
  1) Thanks. Display only, won’t affect script;
  2) No. Defines minimum level of EWS support required.
  3) PowerShell v3 thing – fixed in Fix-MailboxFolder script, still need to propagate to this script.
  
  Regarding the message, make sure you have permissions (impersonation – preferred – or otherwise).
  
  LikeLike
  
  Reply ↓
  - Torbjörn Granheden on November 28, 2013 at 7:13 PM said:
    
    Hi
    
    The script is doing its job with previous changes. duplicates are deleted.
    I have the impersonation rights.
    But I get another error message in the middle of the process to:
    
    Process-Mailbox : Detected problem removing items (loop)
    At C:\root\Remove-DuplicateItems.ps1:458 char:16
    + If(! ( Process-Mailbox $RootFolder)) {
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
    
    This leads me to the lines:
    $RootFolder= [Microsoft.Exchange.WebServices.Data.Folder]::Bind( $EwsService, [Microsoft.Exchange.WebServices.Data.WellknownFolderName]::MsgFolderRoot)
    If(! ( Process-Mailbox $RootFolder)) {…
    
    Any surgestions?
    /MrTbone
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 28, 2013 at 8:58 PM said:
      
      The script checks if the number of items to be deleted are in actually deleted. However, with larger batches Exchange may still be busy deleting items (we hand off a list of items to delete) so numbers don’t add up. In such cases I’d recommend using smaller batches ($MaxItemBatchSize= 100). I may parametrize this in future versions.
      
      LikeLike
      
      Reply ↓
      - Torbjörn Granheden on November 29, 2013 at 8:39 AM said:
        
        Hi!
        Thanks for the Quick reply.
        This script is so cool!!!!
        
        had to change $type to $typ cause it was used earlier in script:
        $typ = (“System.Collections.Generic.List”+’`’+”1″) -as “Type”
        $typ = $typ.MakeGenericType(“Microsoft.Exchange.WebServices.Data.ItemId” -as “Type”)
        $Itemids = [Activator]::CreateInstance($typ)
        
        After this and raising the max items all seems fine.
        
        Thanks!
        /MrT
        
        LikeLike
        
        Reply ↓
Jamie Cox on December 3, 2013 at 9:32 PM said:

Is there anyway to use this with office365/exchange online, or will it only work with an in house exchange

As we have just moved to office365 from another hosted exchange and using the built in Microsoft tools to move the mail have managed to end up with every mailbox with duplicate items

Thanks
Jamie

LikeLike

Reply ↓
- Michel de Rooij on December 3, 2013 at 10:07 PM said:
  
  Currently only on-prem; making the scripts suited for and test them with o365 is on the to do list.
  
  LikeLike
  
  Reply ↓
  - Dipin Kannan on March 23, 2016 at 7:36 AM said:
    
    I am also having issue in Office 365. It will be very helpful if the script works for Office 365 also. Thanks
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on March 23, 2016 at 3:29 PM said:
      
      It works on Office 365 – use the e-mail address and provide credentials.
      
      LikeLike
      
      Reply ↓
- Jamie Cox on December 3, 2013 at 10:21 PM said:
  
  Hi I have tried to use this against office365 but get the following error
  
  PS C:\Users\cnl_admin\Desktop> .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type
  Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose
  
  Security warning
  Run only scripts that you trust. While scripts from the internet can be useful
  this script can potentially harm your computer. Do you want to run
  C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1?
  [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): r
  VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web
  Services\2.0\\Microsoft.Exchange.WebServices.dll
  Processing mailbox jamiec
  VERBOSE: Set to trust all certificates
  VERBOSE: Looking up EWS URL using Autodiscover for jamie.cox@domain.com
  C:\Users\cnl_admin\Desktop\Remove-DuplicateItems.ps1 : A positional parameter
  cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl”
  with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
  At line:1 char:1
  + .\Remove-DuplicateItems.ps1 -Mailbox jamiec -Type Mail -DeleteMode
  MoveToDeleted …
  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ~~~
  + CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1]
  , ParameterBindingException
  + FullyQualifiedErrorId : PositionalParameterNotFound,Remove-DuplicateItem
  s.ps1
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 9, 2014 at 2:28 PM said:
    
    Version 1.3 runs against Office 365!
    
    LikeLike
    
    Reply ↓
    - Sven Hoelbling on August 26, 2014 at 2:14 AM said:
      
      Hi, I get exactly the same error:
      
      A positional parameter cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover service couldn’t be located.”‘.
      
      and have seen your reply which refers to version 1.3. Where can I find the latest version of your PS script?
      
      Kind regards,
      Sven Hoelbling
      
      LikeLike
      
      Reply ↓
      - Sven Hoelbling on August 26, 2014 at 3:28 AM said:
        
        Sorry, I noticed that I already had version 1.4, but I still get this error.
        Can you let me please know what I’m missing?
        All mailboxes are located in Office 365, but it is a hybrid deployment with on-premise Exchange 2013 SP 1 servers. When I point the script to these servers using the -server switch the script returns:
        Can’t access mailbox information store
        Shouldn’t it use autodiscover as Outlook does (in a hybrid depoyment the DNS entry for autodiscover points to the same onpremise CAS server)
        
        Kind regards,
        Sven Hoelbling
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on August 26, 2014 at 7:58 AM said:
        
        1) You will find the latest version of the script here.
        2) Hybrid shouldn’t matter. If your internal DNS is set up correctly for Autodiscover (or using domain-joined client?). Can’t access mailbox store when overriding the EWS web service URL might possibly indicate a permission issue (or other blockade, e.g. firewall).
        
        LikeLike
        
        Reply ↓
        
        Rob Head on April 1, 2015 at 11:49 AM said:
        
        Hi,
        
        I have version 1.4 but it still exhibits the same problem. I’ve confirmed my client has full connectivity to O365 and that auto discovery is working fine for the mailbox. Has any progress been made resolving this problem?
        
        Thanks.
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on April 1, 2015 at 12:36 PM said:
        
        Just ran it succesfully against a hybrid deployment with Autodiscover and Impersonation. To exclude (Autodiscover) redirection from the equation, use “-Server outlook.office365.com” to address cloud-based mailboxes.
        
        LikeLike
        
        Reply ↓
Torbjörn Granheden on December 4, 2013 at 7:23 AM said:

Hi Michel

I have now run this with great success in Exchange 2013. But the largest mailboxes seems to fail with the largest folders.
inbox and sent items in the largest mailboxes get:

Exception calling “FindItems” with “2” argument(s): “Unexpected end of file while parsing Name has occurred. Line 1, po
sition 1244584.”
At C:\root\Remove-DuplicateItems.ps1:253 char:25
+ $ItemSearchResults= $EwsService.FindItems($SubFolder.Id, …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : XmlException

Have tried the Maxfolder and Maxitem but no success.
Could it be some limit i EWS config?

I also get an error deleteing in a few folders:
Process-Mailbox : Detected problem removing items (loop)
At C:\root\Remove-DuplicateItems.ps1:459 char:16
+ If(! ( Process-Mailbox $RootFolder)) {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox
any ideas?

/MrT

LikeLike

Reply ↓
- Michel de Rooij on December 4, 2013 at 11:24 AM said:
  
  1) Do you have a number so I can try reproduce the problem?
  2) I check if the number of items returned have decreased by the number of items removed. However, it’s asynchronous and Exchange – especially with large counts – may still be removing things in in the background. Suggest using low MaxItems.
  
  LikeLike
  
  Reply ↓
  - Torbjörn Granheden on December 4, 2013 at 11:31 AM said:
    
    Hi
    Not so happy today!
    10% of the mailboxes has not deleted their duplicates. Still there
    Instead they have deleted all mail from all folders between a week back and a year back
    Cannot se a pattern. but it´s kaos!
    😦 MrT
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on December 4, 2013 at 11:41 AM said:
      
      Sorry to hear. Don’t know what happened, but you can – as a safety measure – utilize the MoveToDeletedItems option and run Full mode (where specific attributes are matched instead of relying on the Exchange generated PidTagSearchKey key which will be identical for copied objects, see http://msdn.microsoft.com/en-us/library/office/cc815908.aspx)
      
      I don’t do anything regarding date filtering ATM, so I wonder what happened since you mention a specific period.
      
      LikeLike
      
      Reply ↓
Robert 68 on December 8, 2013 at 11:28 AM said:

Hi Michel,

Does it also work in Public Folders?

Thanks,
Robert

LikeLike

Reply ↓
terry on January 7, 2014 at 7:45 PM said:

Would REALLY be great if this worked against Archive mailboxes. I am about to undertake a project where I have a huge migration with import from .pst, we had hoped to go to archive mailbox, but cant, it will take an extra step to do this. But testing on regular Exchange 2013 mailbox works like a champ. Thanks

LikeLike

Reply ↓
- Michel de Rooij on January 7, 2014 at 8:16 PM said:
  
  Thanks, I’ll put it on the to do list (also for the script to remove class X items, eg stubs)
  
  LikeLike
  
  Reply ↓
Cristian Weinzierl on February 20, 2014 at 9:08 PM said:

Hi Micheal,

I testing your script but this send me a this message….
[PS] C:\Documents and Settings\acweinzierll\Desktop>Remove-DuplicateItems.ps1 -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose
The term ‘Remove-DuplicateItems.ps1’ is not recognized as a cmdlet, function, operable program, or script file. Verify
the term and try again.
At line:1 char:26
+ Remove-DuplicateItems.ps1 <<<< -Mailbox nrgonzalezd -Type contacts -Impersonation -DeleteMode MoveToDeletedItems -Mo
de Full -Verbose

I have Exchange 2007

Can YOu help Me… Thanks…

LikeLike

Reply ↓
Dustin Day on March 10, 2014 at 7:14 PM said:

VERIFIED TO WORK WELL ON EXCHANGE 2010 SP3! – 3/10/2014

Very powerful script! Once I installed Exchange Web Services (EWS) Managed API 1.2 and assigned the Hygiene Management role to the account I was running the scripts under I had no problems!

Be sure to follow the formatting provided in the examples as I noticed weird issues when I specified a different order of the arguments than that of the examples.

Below is what worked best for my environment!!!

./Remove-DuplicateItems.ps1 -Mailbox -johnsmith -Type Mail -Impersonation -DeleteMode HardDelete -Mode Full -Verbose

**Used to remove duplicates on imported mail that orginated from Google Apps via Outlook 2007 and GASMO**

LikeLike

Reply ↓
Gerard IT on April 1, 2014 at 1:14 PM said:

Hello! Congrats for your script. It’s really fantastic!! I’ve added impersonation rights, everything runs OK on most of mailboxes. But I have a little problem in one of them. The scripts crashes when starts analyzin the folder “Sent Items”. I supose there is a problem on this folder, otherwise the script wouldn’t run in the other mailboxes. I’m wondering if there is any possibility to define which folders analyze or better, add a folder exception… Is it possible? If not, have you ever been in the same situation, same problem? Many thanks!!

LikeLike

Reply ↓
- Michel de Rooij on June 9, 2014 at 2:23 PM said:
  
  Not encountered it before, check permissions. Folder inclusion/exclusion is on the to-do list, no ETA.
  
  LikeLike
  
  Reply ↓
- Michel de Rooij on January 25, 2017 at 5:18 PM said:
  
  Version 1.6 introduces IncludeFolder and ExcludeFolder options
  
  LikeLike
  
  Reply ↓
  - Thomas juhl Olesen (@spinnetho) on February 9, 2017 at 7:03 PM said:
    
    How do i explude more than one folder , I tried with -excludefolder “Folder 1”, “Folder 2” but it only works when putting one single foldername
    
    LikeLike
    
    Reply ↓
    - Thomas juhl Olesen (@spinnetho) on February 9, 2017 at 7:04 PM said:
      
      OH and GREAT script by the way !! very helpfull
      
      LikeLike
      
      Reply ↓
John Anson on May 25, 2014 at 12:12 AM said:

Hi Michel,
Thank you for your wonderful script.

I used the script to remove duplicate contacts from an Exchange 2013 SP1 folder where most of them were duplicates (800 unique contacts, 138,000 duplicates)

I got this error:
Process-Mailbox : Detected problem removing items (loop)
At .\Remove-DuplicateItems.ps1:470 char:16
+ If(! ( Process-Mailbox $RootFolder)) {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

The lines that cause this error start at line 369:
$OldOffSet= $ItemView.Offset
$ItemView.Offset+= $ItemSearchResults.Items.Count – $ItemIds.Count
If( $OldOffset -eq $ItemView.Offset) {
Write-Error “Detected problem removing items (loop)”
$ProcessingOK= $False
}
For me, most of the time the number of items deleted ($ItemIds.Count) was the same as the number of items in the batch ($ItemSearchResults.Items.Count), so this test condition was usually matching, and halting the script.
I removed the test, and Remove-DuplicateItems.ps1 ran ok.

Are there any side effects that might occur without this test?

PS: I also tested increasing the $MaxItemBatchSize to 1000 and it seemed to work ok.

LikeLike

Reply ↓
- Michel de Rooij on May 30, 2014 at 10:49 PM said:
  
  The removal is an asynchronous process, which may be an issue with systems that can’t keep up (fiddling with the batch size may work or re-running which isn’t ideal).
  A better, more robust method is on the to-do list.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 9, 2014 at 2:22 PM said:
    
    Version 1.3 does not have the asynchronous deletion issue 🙂
    
    LikeLike
    
    Reply ↓
John Anson on May 25, 2014 at 1:34 AM said:

Hi Michel
I would like to add Appointment RequiredAttendees to the mode=full duplicate-detection $key.
Does FindItems populate RequiredAttendees with the [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties view?
If not, what changes are needed to retrieve the RequiredAttendees property?
Thanks for your help,
John

LikeLike

Reply ↓
- Michel de Rooij on May 30, 2014 at 11:01 PM said:
  
  What’s in ‘FirstClassProperties’ depends on the type of item being queried, but I believe it is for calendar items.
  
  LikeLike
  
  Reply ↓
Tiernest on June 2, 2014 at 8:33 PM said:

Hello,

We are using this script to try to remove some duplicate contacts. We have a sync process that pushes out all of our contacts from a public folder to everyone’s contacts. The tool we use to sync isn’t great, so we have dupes. The script works great, but it is removing the most recently added contact and I’d like to remove the old one. Any ideas on modifiying the script to make this happen?

LikeLike

Reply ↓
- Michel de Rooij on June 5, 2014 at 12:53 PM said:
  
  Updated version 1.2 allows you to retain oldest or newest received item (previously was undetermined)
  
  LikeLike
  
  Reply ↓
Pingback: Script Updates | EighTwOne (821)
Pierre on June 11, 2014 at 10:00 AM said:

Hi Michel,

Great script and great that it now also works with EO / O365.
Only issue I have now is that it doesnt search through folders created by the user.

Any suggestions?

Cheers,
Pierre.

LikeLike

Reply ↓
- Michel de Rooij on June 11, 2014 at 10:05 AM said:
  
  Likely permissions issue. I have run it against several mailboxes, including my own in Office 365 and believe me, it has lots of user-defined folders.
  
  LikeLike
  
  Reply ↓
  - Pierre on June 11, 2014 at 3:14 PM said:
    
    You were of course correct!
    
    Thanks again.
    
    LikeLike
    
    Reply ↓
JB on June 17, 2014 at 11:50 AM said:

Hi,

I’m trying to run this script against an Office365 tenant that I’ve just migrated our customer to as the migration process created quite a lot of duplicate items (which we expected it might).

I’ve added the Impersonation rights and full access mailbox permissions to the user i’m running the script as but I just can get it to execute?

autodiscovering the EWS url seems to fail (even though autodiscover works fine when setting up outlook profiles etc.) and i’m not sure what format to enter the -server parameter as? I’ve tried the generic outlook.office365.com, I’ve tried the mailbox users exchange GUID@domain.com (as per the outlook profile) but nothing seems to work.

Any suggestions???

Many thanks in advance 🙂
James.

LikeLike

Reply ↓
CP on July 1, 2014 at 12:16 AM said:

I keep getting a “Can’t access mailbox information store At Line:1 char:28 + .\remove-duplicateitems.ps1 <<<< -Mailbox

I've got application impersonation set on the account

LikeLike

Reply ↓
- Michel de Rooij on July 1, 2014 at 12:49 AM said:
  
  Running it from PowerShell prompt (not EMS)?
  Can you please post the whole output or mail it (contact form)?
  
  LikeLike
  
  Reply ↓
  - CP on July 1, 2014 at 3:13 AM said:
    
    I’ve tried both, same results
    
    This is the script that I’m running….
    
    .\Remove-DuplicateItems.ps1 -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Quick -Verbose
    
    This is the output that I get…..
    
    VERBOSE: Loading C:\temp\duplicates\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loading module from path ‘C:\temp\duplicates\Microsoft.Exchange.WebServices.dll’.
    Processing mailbox aliasname (aliasname@email.com)
    VERBOSE: Set to trust all certificates
    VERBOSE: Using aliasname@email.com for impersonation
    VERBOSE: Looking up EWS URL using Autodiscover for aliasname@email.com
    VERBOSE: Using EWS on CAS https://servername
    VERBOSE: DeleteMode is MoveToDeletedItems
    VERBOSE: Processing Mail items
    C:\temp\duplicates\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
    At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<<< -Mailbox aliasname -Type Mail -Impersonation -DeleteMode MoveToDeleted -Mode
    Quick -MailboxOnly -Verbose
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 1, 2014 at 9:47 AM said:
      
      Check your EWS URL configuration (Get-WebServicesVirtualDirectory) in your environment.
      The EWS URL should be https:///EWS/Exchange.asmx.
      You can override it by pointing to a specific CAS box using -Server (the script will postfix the /EWS/Exchange.asmx)
      
      LikeLike
      
      Reply ↓
      - CP on July 1, 2014 at 6:25 PM said:
        
        Pointing to the correct server and EWS URL…still getting same error. Any other ideas?
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on July 2, 2014 at 1:51 PM said:
        
        Permissions, misconfiguration, software blocking access, mailbox/db inaccessible, proxy configuration ..
        
        LikeLike
        
        Reply ↓
        
        kcitesting on August 16, 2014 at 12:24 PM said:
        
        Any update to this at all please? I too have this exact same issue. I’ve added the rights (full and impersonate) thanks
        
        LikeLike
        
        Reply ↓
        
        kcitesting on August 16, 2014 at 12:34 PM said:
        
        Hi, this is a fantastic script that has saved some heartache for me thank you.
        
        It is working fine if you use -credentials. But I can’t do that for all users so I need impersonation.
        
        Is there a way to get more verbose debug about why the mailbox information store cannot be opened?
        
        I have all the rights set up and am using powershell.
        -server outlook.office365.com consistently produces ‘couldn’t open mailbox information store’ ?
        
        Thanks
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on August 20, 2014 at 6:56 PM said:
        
        Echoing the contents of $_.Exception should give you a reason/clue.
        
        LikeLike
        
        Reply ↓
Danny on July 2, 2014 at 3:35 PM said:

Hello Michel,

Situation long version: I have a mailbox of 14gb and with a ton of folders/e-mails, something went wrong with the migration and in some folders there are six duplicate items.

I ran this script with the following command: .\Remove-DuplicateItems.ps1 -Mailbox email@email.nl -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest -Verbose

– I then get a popup asking me for credentials and I enter the credentials of the specific mailbox and it started moving over 13k duplicated to the deleted items folder.

After it was done doing this, i checked a couple of folders and it showed some still have duplicate items. Checking the log it shows it never told there are duplicates in there.

Processing folder FOLDER NAME
Procession folder FOLDER NAME

when it did remove duplicates it shows:

Processing folder FOLDER NAME
Removing # items from FOLDER NAME

I tried running the script again cause maybe that might help, but It removed 0 items the second time around.

Any tips on getting the rest of the duplicates removed?

LikeLike

Reply ↓
- Michel de Rooij on July 2, 2014 at 8:03 PM said:
  
  It can not delete what it can not find / access. Add -Debug to see more of what it’s doing.
  
  LikeLike
  
  Reply ↓
  - Maicon Motta on August 12, 2015 at 4:30 PM said:
    
    I’am having the same problem, the script It’s working fine, but he cant detect duplicated items, I did the debug and i saw two items that suppost to be equals they have a the field “modified” diferentes.
    
    I’am trying to cleam those duplicated items because when i migrated e-mails from Gmail to O365 he just duplicated e-mails to diferente folders, the same email are inbox, stared, important, all e-mails etc…
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 12, 2015 at 8:28 PM said:
      
      Feel free to modify the matching mechanism if the Search Key or predefined set of attributes to match don’t meet your requirements. However, make sure to test and backup, as hard deletes (if chosen) are hard deletes, and there is no backup in Office 365 🙂
      
      LikeLike
      
      Reply ↓
Patrick on July 8, 2014 at 10:43 AM said:

When running against Exchange 2013 CU5 you’ll need to use the latest version of the EWS API (2.2 at the moment) link: http://www.microsoft.com/en-us/download/details.aspx?id=42951

LikeLike

Reply ↓
- Michel de Rooij on July 8, 2014 at 10:53 AM said:
  
  No, 2.2 adds some client-side functionality, i.e. attachment handling. My scripts are not using functionality requiring EWS 2.2.
  
  LikeLike
  
  Reply ↓
Nabil Salameh on September 19, 2014 at 11:31 AM said:

I am trying your solution but it says (Can’t Acces Mailbox Information Store) !!
Waiting your Reply,

LikeLike

Reply ↓
search results on September 22, 2014 at 7:10 AM said:

Have you ever thought about writing an ebook or guest authoring on other blogs?
I have a blog centered on the same ideas you discuss and would
really like to have you share some stories/information. I know
my viewers would value your work. If you are even remotely interested, feel free to send me an email.

LikeLike

Reply ↓
Brian Hatchell on October 8, 2014 at 7:03 PM said:

Michel:

I have it working well with O365 users (not Hybrid) so long as you specify the server properly. You are obviously an Exchange rockstar? Thanks for the script!

LikeLike

Reply ↓
- Michel de Rooij on October 8, 2014 at 7:10 PM said:
  
  Thank you. Odd, I’ve used it on my Small Business account without problems, no need to specify the server name (relying on Autodiscover)
  
  LikeLike
  
  Reply ↓
Harvey on October 28, 2014 at 5:04 PM said:

Hi Michel:

we have a problem w/CRM synching – it is creating dup contacts in Outlook and we cannot figure out why. Is there a way to have your script be triggered when the dup comes in and delete it right then and there? thanks

LikeLike

Reply ↓
- Michel de Rooij on October 28, 2014 at 5:07 PM said:
  
  No and yes. No, the script is not created with that in mind. Yes, you can use EWS to subscribe to a folder and trigger actions when changes occur. I have something similar in the pipeline, now I only need to find some time or clone myself 🙂
  
  LikeLike
  
  Reply ↓
  - harvey on December 9, 2014 at 8:47 PM said:
    
    Let me ask this – I am hoping to use your script to remove dups from our Outlook clients, but is there a way your script can be modified to remove dups from the CRM database master from which the dups originated?
    thank you.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on January 12, 2015 at 3:24 PM said:
      
      That sounds like a CRM/SQL operation – you may want to check with the vendor. Removing duplicate tuples in SQL is not a problem, maintaining referential integrity and consistency might me.
      
      LikeLike
      
      Reply ↓
Richard Hamilton on November 12, 2014 at 7:39 PM said:

I have same issue saying it cannot access information store. User Powershell on Exchange 2013 CU 6.

LikeLike

Reply ↓
- Michel de Rooij on November 12, 2014 at 8:30 PM said:
  
  Please check this article for configuring proper permissions, apart from configuring Autodiscover properly or using Server parameter:
  https://eightwone.com/2014/08/13/application-impersonation-to-be-or-pretend-to-be/
  
  LikeLike
  
  Reply ↓
Walter on November 18, 2014 at 10:00 AM said:

Hi, I also seem to be having some permission errors, but can’t seem to find the problem…..
Working on an exchange 2007 server.
Installed the EWS API and gave the user full control and impersonation rights:
Get-Mailbox mailboxname | Add-ADPermission –User ADM_user –ExtendedRights ms-Exch-EPI-May-Impersonate
Then ran the script:
Remove-DuplicateItems.ps1 -Mailbox mailboxname -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server EWSServer

Error is:
.\Remove-DuplicateItems.ps1 : Can’t access mailbox
information store
At line:1 char:28
+ .\Remove-DuplicateItems.ps1 <<<< -Mailbox mailboxname -Type Calendar -Imperson
ation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -mailboxonly -server
EWSServer
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
tion
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
n,Remove-DuplicateItems.ps1

Could you please help?

LikeLike

Reply ↓
Brad on December 7, 2014 at 12:42 AM said:

This script is a thing of beauty. I work in a shop heavy with Mac users. At some point in the past many of them were hit but the Outlook for Mac/iOS/sync/duplicate bug. I had one user with over 1.5 million items. I used ODIR and it worked well but it was a very tedious process to migrate messages into folders in batches <10,000 messages so that ODIR could work without crashing. This script just deleted 580146 duplicates from one mailbox while I slept. Thank you!

We do have a 3rd party archiving solution that changes the appends the IPM.NOTE class with an extension. In this case there are 2 of each message as the sizes and classes are different. Is there an easy add to the script to compare 2 separate message classes? IPM.Note and IPM.Note.ExShortcut.owa and only keep the shortcut version?

LikeLike

Reply ↓
- Michel de Rooij on December 7, 2014 at 6:11 PM said:
  
  Sounds like you want a mix of this script with my other script to remove certain message class items at https://eightwone.com/2013/05/16/removing-messages-by-message-class-from-mailbox/
  So, the IPM.Note and the IPM.Note.ExShortcut.owa are in the same folder and – based on certain props – you want duplicates detected and of the duplicates, only remove the IPM.Note when there is an IPM.Note.ExShortcut.owa present?
  
  LikeLike
  
  Reply ↓
  - Brad on December 8, 2014 at 9:42 PM said:
    
    Exactly.
    
    LikeLike
    
    Reply ↓
    - Brad on December 9, 2014 at 4:22 AM said:
      
      Actually, as I get deeper in, there are 2 scenarios. The one mentioned above and a second where the stubs are duplicated and the script doesn’t look at the stubs for duplicates at all. Or so it seems.
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on December 9, 2014 at 9:16 AM said:
        
        The script checks for duplicates per folder level. Also, depending on how the archiving product treats the message you may need to use attribute-level comparison instead of relying on PidTagSearchKey
        
        LikeLike
        
        Reply ↓
    - Michel de Rooij on December 9, 2014 at 9:18 AM said:
      
      Ok, could do that but may be later this week as I’m a bit busy atm
      
      LikeLike
      
      Reply ↓
Pingback: Removing Duplicate Items from a Mailbox - League Team
milangajiceu on February 9, 2015 at 10:28 AM said:

Hi Michel,
The script sounds great, but I just can’t get past the “Can’t access mailbox information store” error.

I’m on an Exchange 2007 environment here and I need to remove duplicate contacts (50k+) from a single mailbox (i.e. “contoso\user”)

What I’ve done:

1. created new account “contoso\me-tst” with no mailbox and no administrative privileges

2. granted ms-exch-epi-impersonation permission on all CAS servers to “contoso\me-tst”

Get-ExchangeServer | where {$_.IsClientAccessServer -eq $TRUE} | ForEach-Object {Add-ADPermission -Identity $_.distinguishedname -User (Get-User -Identity me-tst | select-object).identity -extendedRight ms-Exch-EPI-Impersonation}

3. granted ms-exxch-epi-may-impersonate an fullaccess permission on “contoso\user” mailbox to “contoso\me-tst”

Add-ADPermission -Identity user -User me-tst -extendedRight ms-Exch-EPI-May-Impersonate

4. “$a = get-credential” -> “contoso\me-tst” & password

5. executed script -> scrip fails with “Can’t access mailbox information store” / “(401)Unauthorized”

[PS] D:\Scripting>.\Remove-DuplicateItems.ps1 -mailbox user -type contacts -retain Oldest -server CAS001.contoso.local -deletemode harddelete -credentials $a -mailboxonly -mode full -impersonation -verbose
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox user (user@contoso.local)
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials contoso\me-tst
VERBOSE: Using user@contoso.local for impersonation
VERBOSE: Using Exchange Web Services URL https://CAS001.contoso.local/EWS/Exchange.asmx
VERBOSE: DeleteMode is harddelete
VERBOSE: Processing contacts items
D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store At line:1 char:28
+ .\Remove-DuplicateItems.ps1 <<> Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

Checked:
– Permissions are there
– https://CAS001.contoso.local/EWS/Exchange.asmx is accessible

The script fails when first getting the root folder ($RootFolder= … line530 and on).

Is it not weird that is says “VERBOSE: Using user@contoso.local for impersonation”? should this not be something like “Impersonating user@contoso.local” or “using me-tst@contoso.local“?

Do you have any idea what could be wrong?

Thank you!

Milan

LikeLike

Reply ↓
- Michel de Rooij on February 9, 2015 at 2:11 PM said:
  
  99% permissions / firewall / EWS / CAS config. Take into account waiting period for new permissions to become effective.
  ps: Wording thing. When using impersonation, you are effectively using the account specified (see line above mentioning account used for impersonation).
  
  LikeLike
  
  Reply ↓
  - milangajiceu on February 9, 2015 at 3:23 PM said:
    
    The waiting period is not it. I have wated considerably longer than it takes to replicate the permissions.
    
    … oh wait … as we speak …
    
    it was a certificate issue. I typed the external URL as the Server and now it works! AWESOME!
    
    LikeLike
    
    Reply ↓
  - milangajiceu on February 9, 2015 at 3:33 PM said:
    
    And now I get this:
    VERBOSE: Searching for folder class
    VERBOSE: Processing folder Contactpersonen
    D:\Scripting\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
    At line:1 char:28
    + .\Remove-DuplicateItems.ps1 <<<< -mailbox user -type contacts -retain Oldest -server email.sdr.rot
    $a -mailboxonly -mode full -impersonation -verbose
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.p
    
    Exception calling "FindItems" with "2" argument(s): "The request failed. The operation has timed out"
    
    since it just worked on another mailbox i'm thinking that it is simply taking too long to collect the data (55k+ contacts). Is there a way around this?
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on February 9, 2015 at 3:52 PM said:
      
      Good one. For change EWS timeout, just below:
      $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
      Add:
      $EwsService.Timeout= [System.Int32]::MaxValue
      
      LikeLike
      
      Reply ↓
milangajiceu on February 9, 2015 at 3:53 PM said:

I just tried it once more and it worked. AWESOME!

LikeLike

Reply ↓
Miles Thompson on March 6, 2015 at 11:03 PM said:

Hi, I have this working against mailboxes in O365 but am finding that after it has completed, there are still duplicates, not as many but still lots.
When it runs, it says it has deleted loads of duplicates but then when I check using OWA, there are still duplicates.
How does it determine if the messages are duplicates or not? Is there a different detection method?

LikeLike

Reply ↓
- Michel de Rooij on March 7, 2015 at 3:21 PM said:
  
  Default it uses a MAPI property named PidTagSearchKey to identify duplicates. However, depending on how those duplicates got in the mailbox, that property might not match. In those situations, you can use mode ‘Full’ which uses a predefined set of properties to identify duplicates. Be advised that some 3rd party products (i.e. archiving) may modify those attributes or contents – changing item size – in which case the properties will not match up. In those cases, only option is to alter the set of attributes in the script.
  
  LikeLike
  
  Reply ↓
  - Miles Thompson on March 7, 2015 at 8:58 PM said:
    
    Hi, thanks for the reply.
    I did run it with mode Full and checked a couple of messages and see they have the same Message-ID, so I thought I’d try removing the other conditions from the criteria but thought I’d check running mode quick first to see what happens, and voila! It worked like a charm; no more duplicates 🙂 Thanks for your help.
    
    LikeLike
    
    Reply ↓
    - Miles Thompson on March 8, 2015 at 12:33 AM said:
      
      Hi, also, I’m trying to run this against 80ish O365 mailboxes but can’t get the csv to work. I setup impersonation in O365 but when running the script it fails on autodiscovery. Is there a way to test if impersonation is working correctly? I followed your impersonation guide to set it up.
      I tried without impersonation by adding the admin account to have Full Access permissions to all the mailboxes but that also didn’t work, same autodiscover/mailbox access error.
      I also tried adding the -Credentials (Get-Credential) switch, then when prompted entered the credentials for the first mailbox in the csv but then those credentials were used for all subsequent mailboxes also, and although it appeared to be running through all the mailboxes, I noticed it was actually just doing the first mailbox over and over. Is there a way to make it ask for the credentials for each of the mailboxes in the csv? It’ll be time consuming to enter them all, but still quicker than running the Remove-Duplicates command on every mailbox.
      Thanks
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on March 8, 2015 at 10:08 AM said:
        
        If autodiscover is not resolving properly from your internal network (maybe running split DNS & Hybrid ?), you can use Server to specify the external Office 365 URL, e.g. https://outlook.office365.com/ews/exchange.amsx. But, to be honest, I never have to use that when Autodiscover and DNS is set up correctly, and the request is not blocked. That nice thing with impersonation is that you only need to arrange for 1 account which can impersonate the mailbox owners when accessing those mailboxes in that CSV.
        
        LikeLike
        
        Reply ↓
        
        Miles Thompson on March 9, 2015 at 9:28 AM said:
        
        Thanks for the help but I’ve resorted to creating a script that runs the delete-duplicates.ps1 for each individual mailbox which it prompts for the credentials for each. It’s a long-winded process but seems to be working.
        I did spend a few hours trying to get it working using impersonation and delegated access but couldn’t make it work.
        Autodiscover is working fine and I’m working from home, so no split DNS in the mix or hybrid setup. I tried using -server outlook.office365.com but still same issue. I have no outbound firewall so don’t think the request could be blocked, It works fine when running on a single mailbox when I put in -Credentials (get-credentials) but doesn’t work if I try and use impersonation or delegated permissions. If I try using the -credentials switch and put in the admin account, which has full access, with inheritance, it still can’t connect to the mailbox with error about accessing the information store.
        
        LikeLike
        
        Reply ↓
Pingback: Exportar detalle de Shortcuts de Enterprise Vault desde Exchange | chorbo.es
Tom on April 24, 2015 at 1:42 PM said:

Script looks fantastic. Is there a way to specify a certain date range of duplicates to check for? For example, only compare the last 30 days worth of items?

LikeLike

Reply ↓
- Michel de Rooij on April 24, 2015 at 1:48 PM said:
  
  Currently no. But it’s a script, so feel free to introduce enhancements for your situation as required. I could add it to the ‘wish list’ (but no ETA).
  
  LikeLike
  
  Reply ↓
ltheil on June 20, 2015 at 4:25 AM said:

I’ve needed a script like this for a long time, but my powershell and exchange dev skills were too limited. This script works perfectly with impersonation rights. It did not work without that role assigned though. It acted like it was working but ended up finding no duplicates when i could see the duplicates in the mailboxes i was searching.

LikeLike

Reply ↓
- ltheil on June 25, 2015 at 11:36 PM said:
  
  So after running it a few times, I’ve run into issues which i believe may be related to limits in the number of items per batch or possibly an ews throttling limit. It errors and says that EWS is not available because none of the client access servers in the destination site could process the request.
  
  I have some mailboxes that have over 10,000 duplicates.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 30, 2015 at 9:34 AM said:
    
    If you’re running it against Office 365, there is no way to customize throttling. For local configuration, I’ll put up a blog shortly as I get this asked quite a few times.
    
    LikeLike
    
    Reply ↓
    - Nick C on March 9, 2022 at 9:20 PM said:
      
      Hey Michel,
      
      Did you happen to throw up a blog for this? We’re currently dealing with a calendar that’s got 1.7m items we’re trying to deduplicate and your script is working phenomenally well except now we’re getting hit with EWS operation failed, server busy. Hoping you’ve got some info on helping with that throttling customization.
      
      LikeLike
      
      Reply ↓
- ltheil on June 28, 2015 at 1:49 PM said:
  
  I was able to tweak the script to get it work for what i needed. I spoke to Microsoft about getting a tool from them to remove the duplicates and they were not able to provide me with anything so this script is EXTREMELY useful! Thank you for sharing it.
  
  LikeLike
  
  Reply ↓
Hans on June 26, 2015 at 8:48 AM said:

Hello.

I’m searching for a script which could process public folders on Exchange Server 2013. There are always many duplicates, because mails, which had been sent to more than one user, are moved to the public folder by each user after he recognized or processed the mail. Is it currently possible to process these public folder with your script? If yes: what is the syntax? If no: is there any hope that you implement such function?

Many thanks for providing this script and kind regards
Hans

LikeLike

Reply ↓
- Michel de Rooij on July 17, 2015 at 8:45 PM said:
  
  Sorry for the delayed response. Public Folders should be accessed differently compared to mailboxes. However, processing is similar – I have jotted it down on the ‘requested features’.
  
  LikeLike
  
  Reply ↓
  - Ben on October 21, 2015 at 7:53 AM said:
    
    Any luck with the requested feature of removing duplicates from the public folder?
    
    LikeLike
    
    Reply ↓
Guy Thompson on July 7, 2015 at 3:06 PM said:

Hello,

I have been using this on Office 365 for a while now.

to make sure this works with power-shell you have to specify the account you setup impersonation with in the command.

For example :-

.\Remove-DuplicateItems.ps1 -Mailbox Person@email.com -Type all -DeleteMode MoveToDeletedItems -Mode Quick -Verbose -Server outlook.office365.com -Credentials Admin@Email.com

This will then bring up the input box & then will it cache the credentials…just running it without that string will present you with the infamous :-

C:\Scripts\Remove-DuplicateItems.ps1 : Can’t access mailbox information store
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Mailbox correctgroup@lilyskitchen.co.uk …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Matt on July 16, 2015 at 5:02 AM said:
  
  For some reason the -Credentials property isn’t working for me, I get this error:
  
  Remove-DuplicateItems.ps1 : Cannot process argument
  transformation on parameter ‘Credentials’. Cannot convert the “admin@domain.com” value of type “System.String” to
  type “System.Management.Automation.PSCredential”.
  At line:1 char:144
  + … m -Credentials admin@domain.com
  + ~~~~~~~~~~~~~~~~~~~~
  + CategoryInfo : InvalidData: (:) [Remove-DuplicateItems.ps1], ParameterBindingArgumentTransformationExce
  ption
  + FullyQualifiedErrorId : ParameterArgumentTransformationError,Remove-DuplicateItems.ps1
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on July 16, 2015 at 8:38 AM said:
    
    You need to provide Credentials of type PSCredential, e.g.
    $Cred= Get-Credential
    … -Credential $Cred
    
    or .. -Credentials (Get-Credential)
    
    LikeLike
    
    Reply ↓
    - Matt on July 16, 2015 at 5:48 PM said:
      
      Thank you sir that got it working.
      
      LikeLike
      
      Reply ↓
Rahul Singh on August 6, 2015 at 7:59 AM said:

We have our mailboxes hosted on Office 365, due to some reasons few mailboxes have duplicate mails. These mailboxes have large size – average 25 GB.

I found your script very useful and tried it. Thanks a lot for sharing it online.

Now I m facing a problem, whenever I run this script against large mailboxes, the script breaks the execution and throws a exception. I had tried to get the exception error message, mentioned below:
————————————————————————————————————————–
Exception calling “FindItems” with “2” argument(s): “The request failed. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.”
————————————————————————————————————————–

Any help will be greatly appreciable.

Regards

LikeLike

Reply ↓
- Michel de Rooij on August 6, 2015 at 9:06 AM said:
  
  Is your network/internet connection reliable?
  Also make sure you aren’t running the script (or other scripts, tools, applications) multiple times using the same credentials to prevent running into Office 365 throttling your requests, see https://msdn.microsoft.com/en-us/library/office/jj945066%28v=exchg.150%29.aspx
  
  LikeLike
  
  Reply ↓
  - Rahul Singh on August 6, 2015 at 4:28 PM said:
    
    Our Internet Connection is fine. We have dedicated 1 admin id for the same.
    
    Moreover I found that the script is detecting the duplicates from folders other than Inbox & Sent Items.
    
    To simulate the same I had duplicated (Copy & Paste) some messages in Inbox and Test folder. Duplicates were deleted from the Test folder but not from the Inbox.
    
    Can you please advise what to do???
    
    LikeLike
    
    Reply ↓
Ramakrishnan on August 18, 2015 at 8:32 PM said:

Will this help me to delete the duplicate calendar folder named – Calendar which is present in the user mailbox.

The affected user account has two calendar folder

English calendar
German Kalender

The user doesn’t have a mobile device associated , not sure how this happened all of a sudden as the migration happened years ago.

Any information is highly appreciated !!.

Thanks.

Regards,
Ramakrishnan

LikeLike

Reply ↓
- Michel de Rooij on August 18, 2015 at 8:57 PM said:
  
  No, this script is to detect and remove duplicate items, not folders. If the folders should be merged, you can check out one of my other scripts, Fix-MailboxFolders (https://eightwone.com/2013/01/19/fixing-well-known-folders-troubles/). However, that script will use default localized names for Well Known folders like Calendar. See related blog on how to customize or define a custom ‘locale’ to cover specific scenarios. Note however, only ‘source’ can be customized, the destination folder names are determined by the configured mailbox locale.
  
  LikeLike
  
  Reply ↓
Alex Pavlenko on September 2, 2015 at 1:08 PM said:

Michel, how does your script compare to this one below which relies on iCALUID method to look for duplicates? I used both (yours and that one below) and it seemed that below one produces some false positives while your script reports 0 duplicates in such mailboxes. I just wanted to ask for your technical opinion which method is more reliable?

http://blogs.msdn.com/b/emeamsgdev/archive/2015/02/12/powershell-remove-duplicate-calendar-appointments.aspx

LikeLike

Reply ↓
- Michel de Rooij on September 2, 2015 at 1:48 PM said:
  
  I won’t recommend relying on iCALUID, as this property is set by the client. Mine uses PidTagSearchKey by default, but for calendar items in your case it may be better to resort to mode Full to use property-based matching. However, mileage may vary items if duplicate items are processed in some way, seemingly staying identical but altering their size for example. The script from the MSFT guy has something similar, but in that regards it does not work differently from mine.
  
  LikeLike
  
  Reply ↓
  - Alex Pavlenko on September 2, 2015 at 2:00 PM said:
    
    Michel – yes this is exactly what I am observing with iCALUUD vs. PidTagSearchKEy thanks! Will try Full Mode now.
    I have another question on using -WhatIf. I did not seem to find any -WhatIf logic in your code. Running with or without WhatIf produces the same output. Am I mistaking perhaps and logic is built into the core PS -WhatIf module which is not exposed in your code? I wanted to analyze appointments that script finds before letting it to delete anything.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on September 2, 2015 at 3:22 PM said:
      
      The ShouldProcess() should take care of that logic
      
      LikeLike
      
      Reply ↓
Michael on September 22, 2015 at 2:39 AM said:

I nearly gave up on this script after running into the “Can’t access mailbox information store” error everytime. Impersonation, Get-Credential, setting the servername, writing the mailbox/usernames in all kind of formats…- nothing worked.
The key to get the script running: using the “-MailboxOnly” option, although the server is running Exchange 2010.
AUSFÜHRLICH: Removing 1908 items from Kalender
AUSFÜHRLICH: Total number of items removed: 1908
AUSFÜHRLICH: Processing finished

Even with all the time lost tinkering- this is SO MUCH faster than every other “remove duplicates”-tool out there.
Thank you very much!

LikeLike

Reply ↓
Mark on October 29, 2015 at 11:13 PM said:

So i tried this script and it says it working but its not processing the duplicates.

Exchange 2010 SP3 Any help is appreciated

VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Ser
Processing mailbox User (mark.st.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using Exchange Web Services URL https://NLKVEXCMBX0
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing All items
VERBOSE: Processing primary mailbox User
VERBOSE: Searching for all folder classes
VERBOSE: Processing folder Calendar
VERBOSE: Processing folder Contacts
VERBOSE: Processing folder Lync Contacts
VERBOSE: Processing folder Conversation Action Settings
VERBOSE: Processing folder Conversation History
VERBOSE: Processing folder Drafts
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Journal
VERBOSE: Processing folder Junk E-Mail
VERBOSE: Processing folder News Feed
VERBOSE: Processing folder Notes
VERBOSE: Processing folder Outbox
VERBOSE: Processing folder Quick Step Settings
VERBOSE: Processing folder RSS Feeds
VERBOSE: Processing folder Sent Items
VERBOSE: Processing folder Suggested Contacts
VERBOSE: Processing folder Sync Issues
VERBOSE: Processing folder Conflicts
VERBOSE: Processing folder Local Failures
VERBOSE: Processing folder Server Failures
VERBOSE: Processing folder Tasks
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
- Michel de Rooij on October 30, 2015 at 11:05 AM said:
  
  Could be that there are no duplicates matched using on PidTagSearchKey (Exchange provisioned key, default mode). Try -Mode Full to use attribute matching.
  
  LikeLike
  
  Reply ↓
  - Mark on November 10, 2015 at 4:01 PM said:
    
    I got it work, with the Impersonation flag.
    
    Thanks this script is going to me a lot of time.
    
    LikeLike
    
    Reply ↓
Anthony Godwin on November 15, 2015 at 1:56 PM said:

Greetings.. We have very large mail boxes. We were able to get the script to work on our smaller mail boxes… but it failed on the larger mail boxes with the message “can’t access mailbox information store at line 1 char 1.” Do you have a solution for larger mailboxes. The duplicates items have resulted in some of our mailboxes exceeding 35 gigabytes.

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2015 at 7:32 PM said:
  
  You can adjust the timeout, add $EwsService.ServiceTimeout and set it to a high(er) value (default is 100000 msec). You can also tweak the batch sizes (folders, items and deletions) at the top of the script to make the result sets to work with smaller. It’s on the list to make parameters out of these and introduce keep-alives for Exchange 2013 and up (including Office 365).
  
  LikeLike
  
  Reply ↓
  - Anthony Godwin on November 16, 2015 at 4:05 AM said:
    
    Thanks very much. I shall try that.
    
    LikeLike
    
    Reply ↓
  - Anthony Godwin on November 17, 2015 at 3:50 PM said:
    
    Can you please let me know the proper syntax and where to place this in the script. Thanks!
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 17, 2015 at 4:17 PM said:
      
      $EwsService= New-Object Microsoft.Exchange.WebServices.Data.ExchangeService( $ExchangeVersion)
      $EwsService.ServiceTimeout= 300000
      
      LikeLike
      
      Reply ↓
Tony Kelly on December 23, 2015 at 12:20 AM said:

Hi, thanks for this great script. I’m giving it a go now for our public folders (in ‘WhatIf’ mode), but it seems to get stuck on a certain folder, even after leaving it overnight. Is there any way to get it to skip past when it gets stuck?

LikeLike

Reply ↓
Oliver on January 6, 2016 at 8:23 AM said:

Hey Michel, greate article & script.

We struggle with duplicate contacts created by the MS Dynamics CRM “Server side sync” feature.
CRM related Outlook contacts are flagged with an user-defined attribute “crmLinkstate” which can have the following values:
2->contact is “tracked” in CRM
1->track is pending
0->untracked

Would it be possible to alter your script to query on this crmLinkstate Attribute to get rid of “untracked” Contact duplicates?

Thanks for your Feedback,
Oliver

LikeLike

Reply ↓
- Michel de Rooij on January 6, 2016 at 2:28 PM said:
  
  Would that be only untracked duplicate contacts or all untracked contacts?
  
  LikeLike
  
  Reply ↓
  - Oliver on January 6, 2016 at 2:59 PM said:
    
    It should only query on untracked duplicates as we first need to clean up related duplicates in CRM System.
    
    BR, Oliver
    
    LikeLike
    
    Reply ↓
Deniz on March 3, 2016 at 4:55 PM said:

Write-Error : A positional parameter cannot be found that accepts argument ‘Exception calling “AutodiscoverUrl” with “2
” argument(s): “The Autodiscover service couldn’t be located.”‘.
At C:\Windows\system32\Remove-MessageClassItems.ps1:434 char:28
+ Write-Error <<<< "Autodiscover failed: " $error[0]
+ CategoryInfo : InvalidArgument: (:) [Write-Error], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.WriteErrorCommand

I can't remove the stubs..
Can you help me pls?

LikeLike

Reply ↓
- Michel de Rooij on March 23, 2016 at 3:05 PM said:
  
  You need to set up Autodiscover – or make it accessible (ie no firewall blockades). You can override the discovery process using the Server parameter.
  
  LikeLike
  
  Reply ↓
Damir Reić on May 25, 2016 at 9:05 PM said:

Hello,

thank you for the script. Found a little bug, it doesn’t properly detect number of duplicated items. I used it on O365.

So here is what i did:
1) created random folder inside outlook
2) take some mail from inbox and copy it
3) paste it 11 times inside random folder

Running script only detected 8 of them in random folder. Took new mail from inbox, copied it to random folde 2 times and run the script again. Script deleted previous copies that remain in random folder from 1st copy action. It didn’t remove 2 new copied mails.

I tried running the script again in 1st and 2nd scenario after 1st run finished and it didn’t detect or delete remaining copies.

I have a problem from Google migration to O365 so I just wanted to test before I execute it customer account.

Thanks!

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2016 at 10:03 PM said:
  
  So, in Outlook, were all those changes already synced back to Office 365 (assuming you’re running cached mode)?
  
  LikeLike
  
  Reply ↓
  - Damir Reić on May 25, 2016 at 10:12 PM said:
    
    Well that is kinda hard to tell since O365 OWA is not displaying duplicated mails, only Outlook client is.
    
    LikeLike
    
    Reply ↓
    - Damir Reić on May 25, 2016 at 10:37 PM said:
      
      I just tested again and waited few minutes before deleting. Still all duplicated mails were not deleted.
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on May 25, 2016 at 11:07 PM said:
        
        What cmdline are you using? Note that default it utilizes PR_SEARCH_KEY to match duplicates; that key should be a unique and when copying items, that attribute should be preserved and identical on those items. Can you inspect with for example MFCMAPI if that is the case? Alternatively, you can use -Mode Full to match on a predetermined set of attributes (see help)
        
        LikeLike
        
        Reply ↓
        
        Damir Reić on May 26, 2016 at 7:57 AM said:
        
        .\Remove-DuplicateItems.ps1 -Mailbox test@domain.com -type mail -Credentials $Credentials -Mode Full -DeleteMode MoveToDeletedItems -Verbose
        
        LikeLike
        
        Reply ↓
        
        Anthony on May 26, 2016 at 12:03 PM said:
        
        Do you have a utility to remove duplications from the online version of Office365 for contacts, calendars and mailboxes, tasks, notes? Thanks in advance for your response.
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on May 26, 2016 at 12:45 PM said:
        
        That’s where the script is for – not only mail-items. Have you checked with MFCMAPI to verify the PR_SEARCH_KEY attributes are indeed identical? The cmdline you use should work. On another note, those items are of type mail (IPF.Note); not some sort of stub or something else? (you can use MFCMAPI for this as well or add the Message Class field to the view)
        
        LikeLike
        
        Reply ↓
        
        Damir Reić on May 26, 2016 at 1:19 PM said:
        
        I did that just now, they all have same PR_SEARCH_KEY
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on May 26, 2016 at 3:51 PM said:
        
        Then the script should work
        
        LikeLike
        
        Damir Reić on May 31, 2016 at 9:48 PM said:
        
        Found out why it didn’t want to work. Mails don’t have same PR_SEARCH_KEY, so for example same (content) mail is inside “all mails” and “important” folder but PR_SEARCH_KEY does not match even though all data of the mail that is displayed inside outlook client is the same. Would you be interested to modify the script to fix our problem (we can discuss about the payment)? I don’t know what I was comparing few days ago…
        
        LikeLike
        
        Reply ↓
derek macleod on July 25, 2016 at 12:27 PM said:

Do you know how I could get this to work when the PID searches do not match, and the Received timestamp, size, and last modified properties are all a tiny bit different? (so, essentially relying on sender, subject, time sent which remain unique)

LikeLike

Reply ↓
- Michel de Rooij on July 25, 2016 at 1:37 PM said:
  
  Below ‘# Use predefined criteria for matching duplicates depending on ItemClass’ is the logic to match items, depending on item type (different items provide different attributes). I can’t determine “a tiny bit different” for you. There is also the danger of removing valid items, for example it happens quite often people forget to state something or add an attachment, making things like subject, sender and recipients match and only the timestamp ‘a tiny bit’ different, yet both messages should be kept.
  
  LikeLike
  
  Reply ↓
  - derek macleod on July 25, 2016 at 2:54 PM said:
    
    Thank you sir! I was able to comment out a couple of the predefined criteria to get it working. Lovely script that.
    
    LikeLike
    
    Reply ↓
Enrique on July 28, 2016 at 3:12 PM said:

Hello Michel, the script is GenieL.
We migrated from Lotus to Exchange 2013 but this generate many duplicates process.
I read carefully your blog and I changed the script and the parameters to work. But, now does not fail but gives 0 items to remove

PS C:\Windows\system32> Remove-DuplicateItems.ps1 -Mailbox email@domain.com -Type Mail -DeleteMode MoveToDeletedItems -ArchiveOnly -Verbose -Server Exchange.local -Mode Full
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox email@domain.com (email@domain.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using Exchange Web Services URL https://Exchange.local/EWS/Exchange.asmx
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing Mail items
VERBOSE: Processing archive mailbox email@domain.com
VERBOSE: Searching for folder class
VERBOSE: Processing folder (Custom Expiration\Manage Folders)
VERBOSE: Processing folder Elementos eliminados
VERBOSE: Processing folder FolderHiddenPublic
VERBOSE: Processing folder Folders
VERBOSE: Processing folder Data
VERBOSE: Processing folder Personal
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Sent Items
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
- Eric on August 9, 2016 at 5:48 PM said:
  
  I am having this same issue only on Wave 15.1 of Office 365.
  
  Previous comments lead me to think that it may be a permissions issue. We have duplicates that came in from an IMAP sync and PST import but the script output looks very similar to what you posted – 0 items removed. I have granted the admin account full mailbox access as well as ApplicationImpersonationRights via the Discovery Management admin role but no joy.
  
  The above links regarding permissions appear to be dead so maybe there is something else I am missing with permissions here.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on August 10, 2016 at 10:01 PM said:
    
    15.1 is a bit too generic – what is you specific version, i.e. what does Outlook’s Connection Status / Version report? Meanwhile I’ll check against an 15.1.249.20 tenant.
    
    I’ve update the link to information on how to configure impersonation.
    
    LikeLike
    
    Reply ↓
    - Eric on August 12, 2016 at 12:27 AM said:
      
      Well, my admin display version is 15.1.549.26. I’m not sure if that correlates to the specific Exchange Online version. In any case, I have verified the impersonation as well as given full mailbox access, so I don’t think permissions should be an issue at this point.
      
      However, I have tried both the Full and Quick mode with no items being found but no errors showing either.
      
      Thanks very much for the script and for your assistance!
      
      Eric
      
      LikeLike
      
      Reply ↓
Sandra on October 7, 2016 at 10:27 AM said:

Hi,

Thanks for sharing such a wonderful post with us.
I had this issue & your blog as well as one other blog, helped me to overcome this scenario of duplicate emails. Another link is as follows:
[link removed]

LikeLike

Reply ↓
lbit on November 3, 2016 at 8:25 PM said:

I’m using v1.41 The script runs fine but finds no duplicates. I even tried selected an item in my inbox, ctrl-c, ctrl-v to duplicate it. Still the script says:
PS C:\Users\xxxx\Downloads> .\Remove-DuplicateItems.ps1 -Mailbox xxxx@yyy.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose -MailboxOnly -Server outlook.office365.com -Credential (Get-Credential)

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\1.2\\Microsoft.Exchange.WebServices.dll
Processing mailbox xxxx@yyy.com (xxxx@yyy.com)
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials xxxx@yyy.com
VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: DeleteMode is MoveToDeletedItems
VERBOSE: Processing Mail items
VERBOSE: Processing primary mailbox xxxx@yyy.com
VERBOSE: Searching for folder class
VERBOSE: Processing folder Clutter
VERBOSE: Processing folder Drafts
VERBOSE: Processing folder Inbox
VERBOSE: Processing folder Junk Email
VERBOSE: Processing folder Outbox
VERBOSE: Processing folder VoiceOutbox
VERBOSE: Processing folder Quick Step Settings
VERBOSE: Processing folder Sent
VERBOSE: Processing folder Sent Items
VERBOSE: Processing folder Sync Issues1
VERBOSE: Processing folder Conflicts
VERBOSE: Processing folder Local Failures
VERBOSE: Processing folder Server Failures
VERBOSE: Processing folder Trash
VERBOSE: Total number of items removed: 0
VERBOSE: Processing finished

LikeLike

Reply ↓
lbit on November 4, 2016 at 3:28 PM said:

I got it working for me. It wasn’t for for 4 different reasons:
1) autodiscover isn’t working for my office365 account, I just bypassed the issue adding parameter -Server outlook.office365.com
2) -Mode Quick doesn’t seem to be able to retrieve any useful identifier for emails, so I used -Mode Full
3) Even in -Mode Full, most of my duplicates are not recognized because they have identical time, sender, messageid, etc, but different size. That is, even if I manually duplicate a message copying it to a different mailbox, and then back, it gets a different (larger) size. Since this script compare also size (in Full Mode) it finds no duplicates. Solved by commenting out this line:
# if ($Item.Size) { $key+= “,”+$Item.Size.ToString()}
4) The search for a duplicate is done using an arrayList which (is inefficient and) always fails for me. Maybe because even if the two compared strings are identical they are still different objects.
Solved by replacing line
$UniqueList= [System.Collections.ArrayList]@()
with
$UniqueList = New-Object ‘System.Collections.Generic.HashSet[string]’

I also added two more options to Retain: Largest and Smallest, in order to be able to retain the smallest.

LikeLike

Reply ↓
Oliver on November 10, 2016 at 7:10 AM said:

Hello Michel,

sorry, I Need to come back to this post from early this year: https://eightwone.com/2013/06/21/removing-duplicate-items-from-a-mailbox/#comment-143576

Would it be possible to use your script to:
1) Lookup only contacts within a Mailbox that have the flag “crmLinkState=2”? (This would secure to work only on contacts that have been synced from Dynamics CRM)
2) Do a merge/cleanup with data from 1) and define a master record that has a certain value (e.g. “merged master”) in the field “Nickname”

Background:
We found a way to merge all duplicate contact records within our CRM System. To have a filtering Option, we´d update all remaining records with the above mentioned string within the field “Nickname” which would also be synced to Exchange mailboxes.

Thanks in advance for any Reply.

Regards,
Oliver

LikeLike

Reply ↓
Frédéric Lemieux on August 3, 2017 at 9:23 PM said:

I would like to be able to run the script from my computer without having the username password
To use the office administrator 365

LikeLike

Reply ↓
- Michel de Rooij on August 3, 2017 at 9:27 PM said:
  
  Then you need to store and reuse the credentials.
  
  LikeLike
  
  Reply ↓
  - Frédéric Lemieux on August 4, 2017 at 1:11 PM said:
    
    it’s possible run script for other user without password use ? how ?
    
    LikeLike
    
    Reply ↓
grahampriley on August 9, 2017 at 11:21 AM said:

Thanks for this script. Do you know if it is possible to run it in scan / report mode and not delete anything, just report on duplicates found?

Regards,
Graham

LikeLike

Reply ↓
- Michel de Rooij on August 9, 2017 at 11:22 AM said:
  
  Not the individual items, but -WhatIf:$true should show what it would do and -Verbose would show what folders it would process (and how many duplicates are found). I have a Reporting option on the ‘to do’ list.
  
  LikeLike
  
  Reply ↓
  - grahampriley on August 9, 2017 at 11:55 AM said:
    
    Doh, I never thought of using WhatIf. Thanks, that should do it. 🙂
    
    LikeLike
    
    Reply ↓
Derron on August 16, 2017 at 5:11 PM said:

Script seems to run without errors, the include folder is processed, but it doesn’t identify or remove any obvious duplicates. I have commented out the item.size lines for -Full, but obvious duplicates are still not removed. Tenant version is: 15.1.1341.23. Impersonation is enabled. Duplicates were copied with a third party migrator that we are trying to clean up but the same product did both runs so they are identical messages. Wonderful script if it will do what we need.

LikeLike

Reply ↓
- Michel de Rooij on August 16, 2017 at 5:55 PM said:
  
  It does provided permissions are in order and the set of attributes related to the item type matches. Use Verbose to see at least if it can access the folder with those duplicates.
  
  LikeLike
  
  Reply ↓
David on September 6, 2017 at 3:20 AM said:

Just completed a GSuite migration to Office365 and I’m using your script to clean up the duplicate mess.

Small folders process without any issues, but larger folders fail after deleting a few thousand duplicates…

VERBOSE: Removing 15596 items from Archive
Process-Mailbox : Problem removing items: Exception calling “DeleteItems” with “4” argument(s): “The server cannot
service this request right now. Try again later.”
At D:\Users\XX\Downloads\Remove-DuplicateItems.ps1:675 char:16
+ … If(! ( Process-Mailbox $RootFolder $IncludeSearchCollection $Exc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailbox

D:\Users\XX\Downloads\Remove-DuplicateItems.ps1 : Problem processing primary mailbox of
user@domain.com (user@domain.com)
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Identity user@domain.com -Cr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

Usually, a new powershell is required to resume the operation.

Do you have any suggestions on how to complete the operation in a single pass?

LikeLike

Reply ↓
- Michel de Rooij on September 18, 2017 at 10:04 AM said:
  
  Robust version coming up that can deal with throttling, will update when I’m back from honeymoon next week
  
  LikeLike
  
  Reply ↓
Omar on September 12, 2017 at 11:52 PM said:

Should this be able to run against a shared mailbox? I’ve a shared mailbox (ex-user mailbox that has been converted to shared, with assigned email address) on O365 that i specify in identity, using my own credentials. The script runs but from the verbose output it’s obvious it’s running on my own mailbox and archive.

LikeLike

Reply ↓
- Michel de Rooij on September 18, 2017 at 10:07 AM said:
  
  It should, using identy to specify e-mail address of shared mailbox, and credentials with proper access to that mailbox or impersonation permissions (w/ impersonation switch)
  
  LikeLike
  
  Reply ↓
  - Omar on September 18, 2017 at 8:41 PM said:
    
    Ahh, forgot the impersonation switch. That did it!
    
    LikeLike
    
    Reply ↓
Denis on October 3, 2017 at 7:38 AM said:

Hello Michel, Thanks for this script.

But i have some problems…

D:\Remove-DuplicateItems.ps1 : Can’t access mailbox information store: Exception calling “FindItems” with “2” argument(
s): “The server cannot service this request right now. Try again later.”
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Mailbox xxxxxxxxx@xxxx.xx -Server out …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Michel de Rooij on October 3, 2017 at 9:13 AM said:
  
  That’s throttling on the server-end. Updated script resilient to throttling is on its way, stay tuned.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on October 21, 2017 at 1:03 PM said:
    
    Handling of throttling conditions included per v1.8
    
    LikeLike
    
    Reply ↓
Akvile on October 19, 2017 at 11:51 AM said:

Hi, I am getting an error: Problem removing items. Maybe you have any ideas what the problem is? Thank you!

Are you sure you want to perform this action?
Performing the operation “Remove-DuplicateItems.ps1” on target “Remove 152 items from \Inbox”.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “Y”): Y
VERBOSE: Removing 152 items from \Inbox
Process-Mailbox : Problem removing items: Attempted to divide by zero.
At C:\Program Files\Microsoft\Exchange\Web Services\2.2\Remove-DuplicateItems.ps1:1040 char:29
+ … If (! ( Process-Mailbox -Folder $RootFolder -IncludeFilter $Inclu …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Process-Mailboxting error

LikeLike

Reply ↓
- Michel de Rooij on October 21, 2017 at 1:02 PM said:
  
  Found it – ~ line 920 it mentions $RemoveList where it should use $DuplicateList. Fixed in v1.81.
  
  LikeLike
  
  Reply ↓
Kevin Bouwman on November 13, 2017 at 3:44 PM said:

I am getting a repeated warning. I getting so many that the log text leading up to it has scrolled out of reach for a cut and paste. Several lines after the message telling me the script has started processing the mailbox I get:
VERBOSE:Adding folder \ (priority 0)
WARNING: Error performing operation FindFolders with Search options in . Error: Exception calling “FindFolders” with “3” argument(s): “Value cannot be null. Parameter name: parentFolderId”
VERBOSE:Adding folder \\ (priority 0)
WARNING: Error performing operation FindFolders with Search options in . Error: Exception calling “FindFolders” with “3” argument(s): “Value cannot be null. Parameter name: parentFolderId”
This repeats with another back slash added until there are well over 800 hundred slashes.
The script then ends with the message: Cannot access mailbox information store, error: The script failed due to call depth overflow.
At line:1 char:1
+ C:\Users\Me\Remove-DuplicateItems.ps1 -Identity me@company.biz -Server …
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId: Microsoft.PowerShell.Commands.WriteErrorException,Remove-Duplicates.ps1

The command line (v1.81) I am using looks like this:
C:\Users\Me\Remove-DuplicateItems.ps1 -Identity me@company.biz -Server outlook.office365.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Verbose -Credentials (Get-Credential) -Retain Oldest

LikeLike

Reply ↓
- Michel de Rooij on November 13, 2017 at 3:54 PM said:
  
  From what PowerShell version are you running this? ($PSVersionTable)
  
  LikeLike
  
  Reply ↓
  - Kevin Bouwman on November 17, 2017 at 10:46 PM said:
    
    3.0
    
    LikeLike
    
    Reply ↓
    - Kevin Bouwman on November 17, 2017 at 10:55 PM said:
      
      Name Value
      —- —–
      PSVersion 3.0
      WSManStackVersion 3.0
      SerializationVersion 1.1.0.1
      CLRVersion 4.0.30319.42000
      BuildVersion 6.2.9200.22198
      PSCompatibleVersions {1.0, 2.0, 3.0}
      PSRemotingProtocolVersion 2.2
      
      I realize now that I am probably using version 1.82 of the script as well.
      
      LikeLike
      
      Reply ↓
    - GSinnema on November 18, 2017 at 2:48 PM said:
      
      Same problem here, Using script version 1.82, Exchange 2010 SP3, Windows Server 2012, Powershell version 3.0
      
      LikeLike
      
      Reply ↓
      - GSinnema on November 18, 2017 at 2:50 PM said:
        
        WARNING: Error performing operation FindFolders without Search options in . Error: Exception calling “FindFolders” with “2” argument(s): “Value cannot be null.
        Parameter name: parentFolderId”
        VERBOSE: Adding folder
        \\\\\\\\\\\( Lots of \ here )\\\\\\\\\\\\\
        (priority 0)
        E:\Mailimport\Remove-DuplicateItems.ps1 : Cannot access mailbox information store, error: The script failed due to call
        depth overflow.
        At line:1 char:1
        
        LikeLike
        
        Reply ↓
      - Michel de Rooij on November 18, 2017 at 2:51 PM said:
        
        I’ll try to come up with a workaround for older PS versions- seems like a thing with the regexp.
        For the time being, it’s tested and working on PS5 (e.g. w10).
        
        LikeLike
        
        Reply ↓
        
        GSinnema on November 18, 2017 at 3:19 PM said:
        
        Tnx, I’ve updated WMF to version 5.0 on Server 2012 using: https://www.microsoft.com/en-us/download/details.aspx?id=50395
        The script is fully working now
        
        LikeLike
        
        Reply ↓
Kevin Bouwman on November 22, 2017 at 4:48 PM said:

I updated to 5.0 and the scripted worked for me too. This is great. I was surprised I didn’t have more duplicates. I found about 100 in 150,000 emails. It is really nice to have this cleaned up. Thanks Michel

LikeLike

Reply ↓
Ravi on January 24, 2018 at 5:42 PM said:

Awsm script !!! but can i run this for Sent items folder only ???

LikeLike

Reply ↓
- Michel de Rooij on January 24, 2018 at 6:16 PM said:
  
  Yes – use IncludeFolders and the tag #SentItems#, e.g. -IncludeFolders ‘#SentItems#’. But it seems you found a bug; if you only specify one element, it processes all folders. Bug filed; workaround is to specify the same folder twice, e.g. -IncludeFolders ‘#SentItems#’, ‘#SentItems#’
  
  LikeLike
  
  Reply ↓
srecko jovancevic on March 6, 2018 at 9:31 AM said:

sorry your scri[t does not work on o365 I have scaned an mailbox qith cript 60000 mails and more an returned 0 duplicates and atleast 40 % are duplicates

LikeLike

Reply ↓
- Michel de Rooij on March 6, 2018 at 6:31 PM said:
  
  It works. Not finding any duplicaties is either a problem with duplicate detection, but more often related to permissions.
  
  LikeLike
  
  Reply ↓
Stefan Inhofer on April 13, 2018 at 12:50 PM said:

I have a Problem with Exchange 2016. The Skript detects duplicate contacts and removes them. But it does not detect duplicate Mails.
It lists all the Folders of the Mailbox that exist. It Shows the correct number of items that are in the Mailbox but it Shows 0 items removed. I spedify “-Mode Full -NoSize” but the duplicates are not recognized. The items are created by multiple runs of a restore, so they should be the same.

Using the user credentials or impersonation makes no difference.

LikeLike

Reply ↓
- Michel de Rooij on April 18, 2018 at 9:11 PM said:
  
  For e-mail with ‘Full’ there is a set of attributes which should match for it to be considered a duplicate. With NoSize, you can exempt size from this comparison, but if those others are not *all* identical, it’s not considered a duplicate. You can use tool like MFCMapi (or Outlook itself) to inspect the following attributes: DateTimeReceived, Subject, InternetMessageId, DateTimeSent, Sender and optional Size.
  
  LikeLike
  
  Reply ↓
Oleksiy Piantak on April 18, 2018 at 11:22 AM said:

Hello! Can you please explain How I can use -Mode Full with parameter NoSize

Thanks!

LikeLike

Reply ↓
- Michel de Rooij on April 18, 2018 at 9:13 PM said:
  
  Use Mode Full to use attribute-based matching of duplicates, not PidTagSearchKey. When using Mode Full, you can use NoSize to exempt the size of the items from the comparison.
  
  LikeLike
  
  Reply ↓
Anthony Garrett on April 24, 2018 at 9:49 AM said:

Hi Michel. I have been running this script against a fairly large mailbox with a fair amount of duplicates and am getting following errors:

VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
VERBOSE: Processing primary mailbox tony@billingford.com
VERBOSE: Collecting folders to process, type All
WARNING: Previous EWS operation failed, adjusted sleep timer to 300ms
C:\Users\Redblade\Downloads\Remove-DuplicateItems (1).ps1 : Cannot access mailbox information store, error: Unable to find type [Microsoft.Exchange.WebServices.Data.ServerBusyException].
At line:1 char:2
+ & ‘.\Remove-DuplicateItems (1).ps1’ -Mailbox tony@billingford.com -S …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems (1).ps1

Have checked impersonation which is setup correctly for the $usercredential account. Thing is if I run this against smaller mailboxes this seems to run fine, potentially ruling out impersonation rights or connectivity. There is another mailbox in the tenancy which is >10gb which gives the same error, all other mailboxes run fine. Any ideas would be much appreciated?

Thanks

LikeLike

Reply ↓
Jörg Dewitz on April 28, 2018 at 10:18 AM said:

Michel,

many thanks for this script. I run it on an Exchange 2016 server.

.\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Type Tasks -Impersonation -DeleteMode SoftDelete -Mode Full -Verbose –WhatIf

The script finds many duplicate mails and calendar entries but no duplicate tasks. However, there are many duplicate tasks in the mailbox.

Can you give me a clue?

Thanks a lot
J

LikeLike

Reply ↓
- Michel de Rooij on April 28, 2018 at 11:57 AM said:
  
  With Full mode, if one or more of the Subject, Start Date, Due Date, Status, Size attributes are different, it’s not considered a duplicate.
  
  LikeLike
  
  Reply ↓
  - T. Nieresel on May 2, 2018 at 12:51 PM said:
    
    Hello Michel,
    
    I have a similar problem with the processing of tasks in user mailboxes. I’ve used the following Parameters:
    .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Type All -Impersonation -DeleteMode HardDelete -Verbose
    
    The Skript was able to remove evry duplicate item from the mailbox except for tasks. Even with specifing tasks as the inteded search type and playing aroud with the -DeleteMode parameter, I was unable to remove them.
    
    Does it have anything to do with the german translation for the task folder? Or is there anything else I can try?
    
    Thank you in advance!
    
    Best regards,
    T. Nieresel
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 2, 2018 at 11:13 PM said:
      
      Not related to localization; I’ll have look
      
      LikeLike
      
      Reply ↓
Oleksiy Piantak on May 1, 2018 at 7:09 PM said:

Hello!
Can you please explain I have larger mailbox 90 GB and into folders about 80k items.
Script didn’t work properly and didn’t all cleaned up msg. Used Full mode
Why? And How are using pig tiles or maybe more parameters?
Thanks!

LikeLike

Reply ↓
- Michel de Rooij on May 1, 2018 at 7:27 PM said:
  
  1) What you mean by pig tiles?
  2) Duplicate checks are performed on multiple attributes which all need to be equal depending on the class to be considered duplicate. That is not always the case as 3rd party software may touch items or dehydrate stubs which then look the same, but are not. Read the help for info on attributes to be considered.
  
  LikeLike
  
  Reply ↓
Rob Gray on May 5, 2018 at 11:04 AM said:

Michel
I have managed to modify your script in a fairly minor way to work on Public Folders.
Would you like me to submit my version (1.83PF) to you as I dont feel publishing it myself is valid as 95% ( & probably more) of the work is yours !

LikeLike

Reply ↓
John on June 5, 2018 at 9:08 PM said:

hi when Using mailboxwide switch, how to change the priority of the folders. As I want to keep the emails in all the subfolders instead of Inbox and Important Folders after migrating IMAP mailbox to Office 365.

LikeLike

Reply ↓
- Michel de Rooij on June 9, 2018 at 12:57 PM said:
  
  Use PriorityFolders to give folders priority over others; items founds in these folders are retained over duplicates in other folders. The order of folders specified with PriorityFolders also impacts which items to retain; PriorityFolders A,B will have items in folder A retained over duplicates in B. PriorityFolders also accepts well known folders, wildcards etc.
  
  LikeLike
  
  Reply ↓
dan on June 14, 2018 at 3:07 PM said:

Somehow a user’s contacts got duplicated and then some. She’s got over 500,000. I ran this once and it picked up and removed ~105,000. It completely ignored most of them though. One for example, there appears to be hundreds of, and it only has a name (which is a phone number) and company name. That’s not being picked up in quick or full mode. Any suggestions?

LikeLike

Reply ↓
- Michel de Rooij on June 14, 2018 at 3:51 PM said:
  
  In Full, all these attributes need to be an exact match to be considered a duplicate:
  FileAs, GivenName, Surname, CompanyName, BusinessPhone, HomePhone, MobilePhone and Size
  
  LikeLike
  
  Reply ↓
  - amanfromthere on June 14, 2018 at 5:06 PM said:
    
    Even in quick mode, it’s not finding any of them.
    
    End of the log…
    –
    VERBOSE: Collecting folders to process, type Contacts
    VERBOSE: Adding folder \Contacts (priority 0)
    VERBOSE: Adding folder \ExternalContacts (priority 0)
    VERBOSE: Adding folder \PersonMetadata (priority 0)
    VERBOSE: Found 3 folders that match search criteria
    Processing folder \Contacts
    Processing folder \ExternalContacts
    Processing folder \PersonMetadata
    1752592 items processed and 0 removed in 02:36:45 – average 11181 items/min
    WARNING: Cannot bind to ArchiveMsgFolderRoot – skipping. Error: Exception calling “Bind” with “2” argument(s): “The specified folder could not be found in
    the store.”
    VERBOSE: Processing xx@yy.net finished
    
    Still have ~390,000 that need to be removed, grateful for any advice you can provide.
    
    Thank you
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on June 15, 2018 at 10:05 AM said:
      
      Depending on how duplicates were created, it might find attributes are bit different causing small variations in size. Give -NoSize a shot (with -Report -WhatIf:$true)
      
      LikeLike
      
      Reply ↓
amanfromthere on June 14, 2018 at 3:08 PM said:

Somehow a user’s contacts got duplicated and then some. She’s got over 500,000. I ran this once and it picked up and removed ~105,000. It seemed to ignored most of them though. One contact for example there appears to be hundreds of, and it only has a name (which is a phone number) and company name. That’s not being picked up in quick or full mode. Any suggestions?

LikeLike

Reply ↓
Bora Buyuksoy on September 14, 2018 at 3:16 PM said:

Hi,
nice work but when I run the command, only 1786 items get processed in inbox and no more. Is there a limit set somewhere? There are more than 150000 mails in inbox only.

Thanks

LikeLike

Reply ↓
- Michel de Rooij on September 14, 2018 at 3:40 PM said:
  
  There is no programmatic limit
  
  LikeLike
  
  Reply ↓
  - Bora Buyuksoy on September 14, 2018 at 3:42 PM said:
    
    What do you think might be the problem?
    
    LikeLike
    
    Reply ↓
- Bora Buyuksoy on September 14, 2018 at 5:57 PM said:
  
  Found the issue: I was using admin account to access the client mailbox. When I used -credentials switch with users own credentials it started to process all mails.
  
  Thanks
  
  LikeLike
  
  Reply ↓
Jonas on October 29, 2018 at 2:17 PM said:

Hi Michel.

First thanks, thanks a lot for this script and for all the information about Exchange in your blog.

We have a problem with your script. The script run without problems but it detects a lot of emails duplicated, says at the end that they have been removed but the emails duplicated (and triplicated) are still in the mailbox.

[PS] >.\Remove-DuplicateItems.ps1 -Mailbox xxx@xxxx.com -Type mail -Impersonation -DeleteMode harddelete -Mode full -IncludeFolders “#sentitems#” -mailboxonly -confirm:$false

Processing mailbox xxx@xxxx.com (xxx@xxxx.com)
93158 items processed and 20903 removed in 00:10:50 – average 8597 items/min

We tried with impersonation and with the user’s credentials but the results were the same.

We tried to check only the folder or all the mailbox but again the results were the same.

At the first execution the script removed some of the detected duplicated emails but didn’t remove all of them.

After that it doesn’t remove more duplicated emails even though it detects them.

I can’t imagine why the script detects but doesn’t remove them. Do you have any idea about it?

Thanks in advance for your response.

LikeLike

Reply ↓
- Michel de Rooij on October 29, 2018 at 3:26 PM said:
  
  And you are validating where? Outlook in cached mode can take some time to reflect changes, as the deletion is asynchronous.
  
  LikeLike
  
  Reply ↓
  - Jonas on October 29, 2018 at 3:56 PM said:
    
    Thanks for the quick response.
    
    I’m checking the mailbox folder statistics in server side and I’m trying to remove the items since some days ago. The duplicated emails are still there.
    
    I don’t know why it doesn’t remove the duplicated emails because there is no errors in the execution. Maybe I have to put some flags and modify the script to debug my problem and see where is the strange thing which produce this behavior.
    
    Thanks.
    
    LikeLike
    
    Reply ↓
  - Jonas on October 31, 2018 at 6:00 PM said:
    
    At the end I distribute the duplicated email in 4 folders with 10.000 duplicated emails each one and it works as desired. The emails are analyzed, duplicates founded and removed (each folder contains 7.000 email duplicated more ore less)
    
    The problem seems to be when there a too much emails to remove.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 1, 2018 at 8:19 AM said:
      
      Glad you got it working. Had people running it without issues for >50k items in folders, but that was a while ago. I’ll have another look at operations at scale. Thanks for your feedback.
      
      LikeLike
      
      Reply ↓
Michael Donovan on November 13, 2018 at 9:06 PM said:

Great thanks go out to you Michel as you patiently navigate these questions and provide excellent detailed answer. I just wanted to pass along a tip on an issue I was having.
All of my attempts to use this script were failing with authorization errors. I finally ran it against my own mailbox with the same results. Double and triple checked my credentials…still no luck.
Knowing I had MFA enabled, I used my app password and it all started working.

Really very nice of you to share and continue to develop a script like this. We as a community can not thank you enough.

LikeLike

Reply ↓
- Michel de Rooij on November 13, 2018 at 11:14 PM said:
  
  Yeah, MFA could be blocking, EWS protocol could be disabled at the mailbox level, basic auth could be blocked in Exchange Online for EWS .. lots of conditions.
  
  LikeLike
  
  Reply ↓
Michael Donovan on November 13, 2018 at 11:41 PM said:

So I have a quick question. I have some shared mailboxes that I have full access to. I am identifying the shared mailbox with the identity parameter. I am using my credentials for authentication. In verbose mode, the script says its processing the shared mailbox. But the scan appears to be scanning my own mailbox. I do not have impersonation enabled. And if I try to use the switch it fails. Must I have impersonation even if I have full access?
On the bright said, I have thoroughly cleaned dupes out of my own mailbox. Command line below is what I used (notwithstanding some anonymity.)
$creds = get-credentials
.\Remove-DuplicateItems.ps1 -Identity sharedmailbox@company.com -type Mail -Server outlook.office365.com -Credentials $creds -mode Quick -DeleteMode SoftDelete -MailboxWide -MailboxOnly -Verbose

Some of the verbosity was…
VERBOSE: Processing folder \Inbox\FTC
VERBOSE: Processing folder \Inbox\Incident emails
VERBOSE: Processing folder \Inbox\Helpdesk
32310 items processed and 37 removed in 00:03:41 – average 8750 items/min
VERBOSE: Processing sharedmailbox@company.com finished

The folders mentioned above were in my mailbox not the shared mailbox.

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2018 at 12:11 PM said:
  
  Let me check..
  
  LikeLike
  
  Reply ↓
- Michel de Rooij on November 17, 2018 at 12:59 PM said:
  
  Please get update 1.86 which should solve this.
  
  LikeLike
  
  Reply ↓
  - Michael Donovan on December 13, 2018 at 8:31 PM said:
    
    I have to follow up on my use of this great script. I finally got application impersonation permissions to run this. (This is a difficult role to explain to someone even if they know PowerShell really well.)
    In short, this script works great for me now. I discovered that using the mode body yields the most dupes. I encourage anyone using this to try many combinations of the switches to get what you need form this.
    Michel-Thanks again! I can’t help notice what a great over all site you have. I am starting to really dig into in. You are indeed an MVP!
    
    LikeLike
    
    Reply ↓
Mario Mesa on November 15, 2018 at 6:27 PM said:

Good afternoon; is it possible to delete duplicates only based on Subject Line and date?

My used got the same email twice multiple times from the same user but from different alias so the only thing that it matches besides the body content is the subject line and date which are the same.

Thanks;
Mario

LikeLike

Reply ↓
- Michel de Rooij on November 15, 2018 at 6:32 PM said:
  
  Not yet, but working on version where you can customize the fields used for matching. Goes without saying that matching on less fields means greater chance of matching on less identical items (matching on only subject and date sounds bit risky)
  
  LikeLike
  
  Reply ↓
  - Mario Mesa on November 15, 2018 at 6:35 PM said:
    
    Agree; unfortunately thats my case 😦
    Let me know if this new version is coming out soon.
    
    LikeLike
    
    Reply ↓
Michael Donovan on November 15, 2018 at 7:35 PM said:

So I have a significant update on my issue. I had been running this program on my workstation with my Outlook open. In retrospect, that was quite foolish. I am trying it on an exchange server (which also feels foolish.) I believe my issues are one of environment. I will report my findings as they develop.

LikeLike

Reply ↓
lt14 on November 29, 2018 at 9:11 AM said:

After my Remove-DuplicateItems.ps1 was executed and completed, some mailboxes got an automatic email about “Unread email has been deleted”. Some users did have unread duplicated emails and it looks like the script or Exchange-server send an email to the sender of all unread emails, so both internal and external recipients in other organisations got a notification about this. Do you know how to turn of this notification emails?

LikeLike

Reply ↓
- Michel de Rooij on May 2, 2019 at 7:03 PM said:
  
  What unread mail deletion notification is that? Is it from on-prem Exchange? I do suppress read notifications (supported for Ex2013+) and meeting cancellations.
  
  LikeLike
  
  Reply ↓
  - lt14 on May 8, 2019 at 2:54 PM said:
    
    It was Exchange 2016 CU11. What I remember, it was NDRs back to the senders of those old emails saying “You got an unread email deleted by the recipient…”. We did not test to manually delete some of these old unread emails to see if that also gerenated NDRs. We did just close that ticket. Let’s see if it happens more times. And can reply here with more details in that case. Thanks for a good script btw!
    
    LikeLike
    
    Reply ↓
Kevin on February 21, 2019 at 4:36 AM said:

You are a life saver!! After my parent company switched from exchange to Gmail (losing a few years worth of emails in the process), I’m now switching back to Office 365 for my team and the labeling in gmail created so many issues. Thanks to your script I can merge the gmail inbox AND the original PST file from the exchange server and not have to panic about the duplicates. Incredibly well explained and detailed and reading through the responses above saved me a ton of time. (I had never used a script before in Powershell and with your direction was able to get this up and running in less than an hour) Thanks again!

LikeLike

Reply ↓
- Michel de Rooij on February 21, 2019 at 7:53 AM said:
  
  Glad is was helpful
  
  LikeLike
  
  Reply ↓
Bartel Onzea on April 30, 2019 at 1:23 AM said:

Great script. I am running into problems with mailboxes in a different language. I cannot do -DeleteMode MoveToDeletedItems because the DeletedItems for a German mailbox is named ‘Gelöschte Elemente’ (Error:Exception calling “DeleteItems”…

LikeLike

Reply ↓
- Michel de Rooij on May 2, 2019 at 7:16 PM said:
  
  That shouldn’t be the problem, because we’re referring to the DeletedItems folder through something called the WellKnownFolders, which is language-independent. Care to elaborate a bit more on the issue you’re encountering?
  
  LikeLike
  
  Reply ↓
BDP on May 14, 2019 at 11:32 PM said:

I’m still running into the EWS throttling issue on large mailboxes (error below; example mailbox is ~200,000 items). I’ve read some comments that this could be an issue with interrupted internet connectivity. So I re-ran the command from an Azure server to alleviate any local issues and ran into the same error.

Cannot access mailbox information store, error:
Exception calling “LoadPropertiesForItems” with “2” argument(s): “The request failed. The read operation failed, see inner exception.”

I’ve tried multiple combinations of parameters and it seems to be only affecting running in Full mode (full script below).
.\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Credentials $creds -Mode Full -NoSize -Impersonation -Server outlook.office365.com -Type mail -verbose

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2019 at 9:40 PM said:
  
  1) What version of EWS Managed API are you using?
  2) Mailbox parameter has been renamed to Identity for a while now.
  First thing the script does is collect all folders and select the ones which pass the Include/Exclude filter. That should not be limited due to no. of items in a folder. That ‘Exception calling “LoadPropertiesForItems” with “2” argument(s): ‘ is the only thing reported?
  
  LikeLike
  
  Reply ↓
  - Bardia Mehrabian on May 26, 2019 at 6:51 AM said:
    
    I can’t speak for BDP, but I am facing the same exact issue across multiple different mailboxes. I am trying this against O365. The EWS Managed API I’m using is v15.00.0913.015. The exact error that I get is:
    
    TerminatingError(): “Exception calling “LoadPropertiesForItems” with “2” argument(s): “The server cannot service this request right now. Try again later.”
    
    It definitely seems to be struggling with large mailbox sizes with a couple folders that are 10GB+ (50k+ items). Unsure how to handle this situation. Is it best to try and split these across multiple folders in some way? Guess I’ll have to find a way to do that programmatically…
    
    LikeLike
    
    Reply ↓
Jonathan TAYAR on September 20, 2019 at 6:50 PM said:

Hi,

First of all, thanks a lot for your help to the community.I’ve discovered your thread a few weeks ago. I can’t believe you’ve helped so many people. I’m also having issues on a migration from G Suite to O365.
I’ve downloaded your script. I’ve been trying to use it for several days. I went through most of the issues people also had on this topic, and got them solved one after the other. (credentials, impersonation, etc).

I’ve run the script in Full Mode and it did not find a lot of duplicates (but I know I have at least 17Gb of them :-). I’ve just launched a new run with the -noSize option, I’ll see if it works.

I have one question : I have found manually a few examples of emails that are duplicated. Following example is :
– the original email is folder “Important”
– the first copy I can find is in folder “PARTNER A” (G suite label). On that email I can spot one difference : Office has considered that my forwarding this email was to be in this folder, not in important. So the mailchain is a bit different here in this folder
– the second copy I can find is in folder “Archive”, and it looks pretty much like the one in folder “Important”.
Do you know a way for me to check the matching criterias in the script on these 3 particular emails to understand why they were not picked up ?

My current command line :
.\Remove-DuplicateItems.ps1 -Mailbox myuser@mydomain.onmicrosoft.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Quick -Server outlook.office365.com -Impersonation -Credentials (Get-Credential) -MailboxWide -NoSize -Report -WhatIf:$true

Thanks in advance for your answer !

LikeLike

Reply ↓
- Jonathan TAYAR on September 20, 2019 at 6:52 PM said:
  
  Just realized : better with Full Mode to use NoSize
  .\Remove-DuplicateItems.ps1 -Mailbox myuser@mydomain.onmicrosoft.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Quick -Server outlook.office365.com -Impersonation -Credentials (Get-Credential) -MailboxWide -NoSize -Report -WhatIf:$true
  
  My question is still on 🙂
  
  LikeLike
  
  Reply ↓
Gui Lima (@guilhermelima) on February 23, 2020 at 1:30 AM said:

Hi dear,
It is an amazing script, really liked it. What if we could use an parameter to say what should be check as a duplicated, like “messageID”. In my environment, mails were duplicated due an replication during migration and with an forward message redirection. It means that mails were received with seconds in difference between the migration process and the forward process. If I had the parameter to say that the messageID is the same, the duplicated mails could be removed.

Thank you for sharing it! 🙂

LikeLike

Reply ↓
Jamie H on May 5, 2020 at 12:41 PM said:

Hi,

Thanks for this handy script, I’m having an issue in full mode however where it says Cannot access mailbox information store, error: Exception calling “LoadPropertiesForItems” with “2” argument(s): “Requested value ‘GroupMailbox’ was not found.

I am using exchange api 2.2 with the following command .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Server outlook.office365.com -Credentials $Credentials -Mode Body -PriorityFolders ‘Holidays’,’Relief’,’IT Related Stuff’,'[Gmail]’,’Starred’ -MailboxWide -Report -Verbose

Any ideas?

Thanks
Jamie

LikeLike

Reply ↓
- Michel de Rooij on May 5, 2020 at 1:28 PM said:
  
  2.2 doesn’t understand Group Mailboxes – the updated 2.2.1.2 can process them. Get the package from here, https://www.nuget.org/packages/Exchange.WebServices.Managed.Api/, or when its too complex with nuget, use ‘Download Package’, open the nupkg archive with 7zip or something else, and extract the Microsoft.Exchange.WebServices.dll & xml files to the folder where the script resides. When using verbose, script should report ‘Loaded EWS Managed API v2.2.1.0’.
  
  LikeLike
  
  Reply ↓
  - Roger on November 19, 2020 at 6:14 PM said:
    
    Hello Michel,
    I am getting exactly the same error as Jamie.
    I performed the fix you mentioned, and turned on verbose output, and it reports it’s “Loaded EWS Managed API v2.2.1.0” but still gets the same error.
    Any further adice?
    Thanks,
    Roger.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 26, 2020 at 12:01 AM said:
      
      Could you elaborate bit more on the scenario?
      
      LikeLike
      
      Reply ↓
      - Roger on December 9, 2020 at 6:50 PM said:
        
        Hello Michel,
        Sorry for not replying sooner – I got it working after creating a new VM, and setting all back up – thanks!
        One question – I copied an email to my Inbox and it wasn’t detected as a duplicate – is it because a modified time maybe different even though the received time is the same?
        Thanks,
        Roger.
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on February 22, 2021 at 5:42 PM said:
        
        Copy is new instance- it might only be considered duplicate when performing attribute-level comparison.
        
        LikeLike
        
        Reply ↓
Sven Schaffranneck on June 12, 2020 at 10:34 PM said:

Hi, any news on removing duplicate items for public folder? Or may anyone else has a fork which works with PF? Thank you very much for your work so far!

LikeLike

Reply ↓
- Rob Gray on June 14, 2020 at 11:37 PM said:
  
  I have modified version that works on our public folder configuration….
  note it is based 99% on v1.86 of Michel’s script & requires a mailbox with access to the entire tree to be reviewed.
  https://www.dropbox.com/s/mjfu94ie2nf6rlv/RemDup186rg2.ps1?dl=0
  
  I run it on a portion of the tree at a time.
  Write-Output “Start– $(get-date -Format “ddd dd MMM yyyy HHmmss”) —–”
  if(!$Credentials)
  {
  $Credentials= Get-Credential;
  }
  
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2004” -Credentials $Credentials -Impersonation -Report -Force
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2005” -Credentials $Credentials -Impersonation -Report -Force
  & $RDup -Mailbox FileMailer -PublicFolders -PFStart “_Projects\2006” -Credentials $Credentials -Impersonation -Report -Force
  
  I will have to diff it with current v1.88
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 15, 2020 at 8:10 AM said:
    
    Sorry Sven, did not have time yet to put some decent effort in myself. But power of community and GitHub mechanics allow anyone to improve or enhance functionality. It’s also not off my list.
    
    LikeLike
    
    Reply ↓
    - Rob Gray on June 15, 2020 at 11:44 PM said:
      
      Michel. I have uploaded my code as an ‘issue’ to Git as dont know how to create a branch on your git.
      
      LikeLike
      
      Reply ↓
- Rob Gray on June 17, 2020 at 12:01 PM said:
  
  fork created off 1.88; refer github
  
  LikeLike
  
  Reply ↓
  - Rob Gray on May 14, 2021 at 5:01 AM said:
    
    fork updated to 2.03; works for OnPrem. not tested for cloud based.
    https://github.com/RobGray-au/Remove-DuplicateItems
    
    LikeLike
    
    Reply ↓
skifwhite on July 14, 2020 at 3:46 PM said:

is it possible to use this script with MFA? Or to disable the MFA is a necessary step?

LikeLike

Reply ↓
- Michel de Rooij on July 14, 2020 at 5:40 PM said:
  
  Implementing modern authentication support is on the shortlist.
  
  LikeLike
  
  Reply ↓
  - skifwhite on July 14, 2020 at 9:36 PM said:
    
    Can we expect this feature on the nearest update?
    
    LikeLike
    
    Reply ↓
  - Mars on July 16, 2020 at 8:23 AM said:
    
    Hello Michel, would be great if you could implement this in nearest release, this feature is expected very much on our side, thanks a lot for your magic script!
    
    LikeLike
    
    Reply ↓
  - skifwhite on July 16, 2020 at 2:28 PM said:
    
    Could you please also clarify, what I do wrong?
    PS C:\Users\User\Downloads\Remove-DuplicateItems-master> .\Remove-DuplicateItems.ps1 -Identity ‘user@contoso.com’ -Server outlook.office365.com -Credentials $Credentials -IncludeFolders ‘#Inbox#/03 COOL/0 BPM’ -DeleteMode SoftDelete -MailboxOnly -verbose
    VERBOSE: Loading C:\Program Files\Microsoft\Exchange\Web Services\2.2\\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loaded EWS Managed API v15.00.0913.015
    VERBOSE: Set to trust all certificates
    VERBOSE: Using credentials admin@contoso.com
    VERBOSE: Processing items of type All, delete mode is SoftDelete
    Processing mailbox user@contoso.com (user@contoso.com)
    VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
    VERBOSE: Constructing folder matching rules
    VERBOSE: Processing primary mailbox user@contoso.com
    VERBOSE: Collecting folders to process, type All
    
    (I have changed user email intentionally) It stops on the latest message and nothing continue.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 16, 2020 at 2:39 PM said:
      
      When granting FullAccess, did you specify InheritanceType All
      
      LikeLike
      
      Reply ↓
Moula Muhaidyn on August 16, 2020 at 1:10 PM said:

How to delete the duplicate in In-Place Archiving in Exchange Server

LikeLike

Reply ↓
- Michel de Rooij on August 16, 2020 at 9:53 PM said:
  
  By default, it will process in-place archives as well if the mailbox has an associated in-place archive.
  If you want to process in-place archives only, there’s a switch ArchiveOnly for that.
  
  LikeLike
  
  Reply ↓
Josh Gardner on September 29, 2020 at 10:22 PM said:

Michel,
I really look forward to getting this to work, however I seem to run across a similar issue as many others:

Cannot access mailbox information store, error: Exception calling “.ctor” with “0” argument(s): “This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.”

I have given myself impersonation rights, as well as full permissions to the mailbox in question (a different user). I’ve tried both -Impersonation and the -Credential method.

I see a lot of issues for people regarding “Cannot access mailbox information store” but that seems to be the permission issue that is supposed to be fixed by giving impersonation/full access (with inheritance). However, I don’t see anything regarding the follow up part of the error mentioning FIPS. Unfortunately I’m in an environment where I cannot simply “disable FIPS”, it is a security requirement. Does that mean I’m out of luck? Is there some sort of switch that bypasses what FIPS may be complaining about? Some sort of alteration in the script that will allow it to be FIPS “compliant”. It seems to scope the mailbox just fine in the debug, but fails right after it starts to process the folders. I can even see the first item of the folder that it hits. If you have any idea how to address this, I’d super appreciate it.

.\Remove-DuplicateItems.ps1 -Mailbox user.samaccountname -Type all -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose -Debug

I’m on-prem with Exchange 2016 CU15
Using Remove-DuplicateItems.ps1 Version 1.88
$PSVersionTable

Name Value
—- —–
PSVersion 5.1.14393.3866
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14393.3866}
BuildVersion 10.0.14393.3866
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

If, in the case I’m out of luck because of FIPS, I still appreciate your time, not only in responding, but creating what looks to be a superb script.

LikeLike

Reply ↓
- Michel de Rooij on October 1, 2020 at 7:14 PM said:
  
  Sorry, scripts are untested with FIPS locked down environments. That said, it’s not that the script shouldn’t work, I assume it’s a matter of crypto settings and defaults. Not totally into this, but could you set this (I assume no it environment is locked down) and run the script?
  HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy\Enabled=0
  
  LikeLike
  
  Reply ↓
  - Josh Gardner on October 13, 2020 at 3:20 PM said:
    
    Sorry for the delay. Unfortunately, manually setting the registry key (as you mentioned above) did not work. Nor did temporarily removing the GPO that enabled FIPS cryptography. Very unfortunate. I was going through your script to find what could potentially be causing this. I noticed this section:
    
    Function Get-Hash {
    param(
    [string]$string
    )
    $md5 = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
    $data = ([system.Text.Encoding]::UTF8).GetBytes( $string)
    return ([System.BitConverter]::ToString( $md5.ComputeHash( $data)) -replace ‘ – ‘, ”)
    }
    
    When I run that function all by its lonesome in a PowerShell session, I get presented with the same error! I have changed the encoding from MD5 to SHA256 in that function and now it works (or so it seems, I’m still running a test). OOOH man, so excited lol. Thanks. Crossing my fingers 🙂
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on October 13, 2020 at 3:24 PM said:
      
      That actually makes sense. Let me know, might need to update it with hashing method using more accepted method nowadays 🙂
      
      LikeLike
      
      Reply ↓
      - Josh Gardner on October 14, 2020 at 12:54 PM said:
        
        The script works fantastically now. Love it! It has relieved a huge weight off my chest. Thanks again!
        
        LikeLike
        
        Reply ↓
Jonathan K on October 1, 2020 at 9:52 PM said:

This script looks perfect, especially since I was advised post-migration that users had been forwarding all their email from an old domain to their new one, creating duplicates of their entire mailboxes. I’m trying to run this from the admin context against Office365, but can’t get it to work. Do you need to use the credentials of the account that you’re accessing, or can a Global Admin’s account be used? That seemed to be the function of the impersonate, but I haven’t been able to get it to execute properly. I’ve managed to get as far as getting an auth error with the user mailboxes. Appreciate your help and your awesome script!

LikeLike

Reply ↓
- Jonathan K on October 1, 2020 at 10:00 PM said:
  
  And it turns out I’m an idiot. Got it working, it looks great. Thank you!
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on October 1, 2020 at 10:27 PM said:
    
    Glad you got it working
    
    LikeLike
    
    Reply ↓
Pedro Sgobin on October 16, 2020 at 8:30 PM said:

Using your script to clean the GSuite to Office 365 duplication mess.
Thank you.

LikeLike

Reply ↓
Patrick on November 26, 2020 at 8:13 PM said:

Hello,
first of all Thanks for the Script and the Script to remove Items from a Mailbox. Is there a Way to only prioritize Subfolders of a Folder and not the Folder itself?

LikeLike

Reply ↓
- Michel de Rooij on November 27, 2020 at 12:38 AM said:
  
  Yeah, it’s a bit ugly, but try -PriorityFolders ‘\Folder\**’. Use -Verbose to validate processing and prioritizing of folders.
  
  LikeLike
  
  Reply ↓
Pingback: Annual Report 2020 | EighTwOne (821)
Kevin on February 1, 2021 at 6:51 PM said:

Can this script (or maybe this one with modifications) be used to remove duplicates between two separate non-archive mailboxes? I’m hoping to locate something that can identify e-mails with identical message IDs distributed between two mailboxes and allow the deletion of one set of messages. The ability to dump the IDs and message properties to a CSV for review would also be useful. Any thoughts or pointers to get me going in the right direction? Have strong Powershell skills but the EWS component has always featured a layer of complexity beyond what I’m familiar with. Thanks in advance!

LikeLike

Reply ↓
- Michel de Rooij on February 3, 2021 at 4:50 PM said:
  
  This script, no. Can it be made, yes. However, that would basically mean going through both, collecting things uniquely identifying mail in one (e.g. message ID and path for reference), and looking for presence in the other (reporting ID and path in mailbox A and mailbox B). That is the business case, if I may ask?
  
  LikeLike
  
  Reply ↓
Kevin on February 3, 2021 at 10:24 PM said:

Thanks Michel. Business case is a one-off unique situation – a user’s mailbox was split into two due to high folder counts and at Microsoft support’s recommendation. It was discovered later that for a period of time, emails were delivered to both mailboxes. We want to dedupe these message before we archive his mailbox as the dupes will be a source of annoyance / confusion for the end user.

I’ve opened a case with Microsoft about this as well, but thus far they’ve been less than helpful. Your suggestions are helpful though, and I’ve made some progress just binding to a folder and producing FirstClassProperties of items.

LikeLike

Reply ↓
- Michel de Rooij on February 28, 2021 at 5:35 PM said:
  
  Hi Kevin, to circle back on this: Could be more simple on second thought: The entry which initializes the variable to keep track of entries ‘seen’ before is initialized at the start when processing the mailbox. It is re-initialized after processing the folder, depending MailboxWide switch. You could of course move initialization to start of script, and comment all reinitializations out. Then, pass the mailboxes you want to process (you can pass more identities) so it will ‘know’ items seen in the first mailbox, and identical items from second (and third etc). identity will be considered duplicate. Now, to save you work, in 2.03 I removed MailboxWide and introduced -CleanupMode where you can specify MultiMailbox (and fixed identity parameter accepting multiple entries). So, you can do this (added whatif & report to make example non-destructive):
  .\Remove-DuplicateItems.ps1 -Identity peter@contoso.com,frank@contoso.com .. -CleanupMode MultiMailbox -Verbose -WhatIf:$true -Report
  
  LikeLike
  
  Reply ↓
Vikas on February 21, 2021 at 4:31 PM said:

Tried executing this script. It doesn’t give error but is not deleting duplicate items from Office 365.
I am trying to delete all duplicate mails from server in multiple folders. Have changed email for security purpose
Am I missing any thing
Script used is :
$Credentials= Get-Credential
.\Remove-DuplicateItems.ps1 -Mailbox abc@mycompany.com -Server outlook.office365.com -Credentials $Credentials -DeleteMode HardDelete -ExcludeFolders ‘Keep Out’ -PriorityFolders ‘*Important*’ -MailboxWide

LikeLike

Reply ↓
- Michel de Rooij on February 21, 2021 at 11:20 PM said:
  
  Not missing anything. Is it processing the folders and items you expect (-Verbose); use -WhatIf:$True and -Report to see what it would do. Could be the items are not considered duplicate for several reasons.
  
  LikeLike
  
  Reply ↓
Vikas on February 28, 2021 at 2:59 PM said:

Hi Micheal ,

issue was created after migrating from Gmail to Office 365 of many duplicate emails.
After running this script. Script is working and deleting duplicate emails when duplicates are in sub-folders. How ever it is not deleting emails if duplicate emails are present in folders as same level
eg
+Inbox
++Subfolder
+TestFolder
+Archive

So if mails are duplicates between Inbox,Achive and Testfolder it is not working. Please help. Script I am running is below :

$Credentials= Get-Credential
.\Remove-DuplicateItems.ps1 -Mailbox myemail@testdomain.com -Server outlook.office365.com -Credentials $Credentials -Type Mail -DeleteMode HardDelete -Verbose -MailboxWide -Mode Full

LikeLike

Reply ↓
- Michel de Rooij on February 28, 2021 at 6:14 PM said:
  
  Fixed in 2.03 – note that I replaced MailboxWide switch, and you need to use CleanupMode Mailbox, eg
  .\Remove-DuplicateItems.ps1 -Identity frank@contoso.com .. -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
  
  LikeLike
  
  Reply ↓
Joel Cottrell on March 9, 2021 at 8:38 PM said:

Hi there. This looks like a great script. Unfortunately it appears that i’m running into an issue just running this script. This may be a simple fix, but could you point me in the right direction as to how I can get this running?

PS D:\Install\Scripts\Remove-DuplicateItems> .\Remove-DuplicateItems.ps1 -Identity test@test.com -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
D:\Install\Scripts\Remove-DuplicateItems\Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the
specified named parameters.
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Identity test@test.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Michel de Rooij on March 9, 2021 at 10:35 PM said:
  
  It’s missing a paramater to dermine what mode it needs run in (and other parameters). In this case, Credentials (for Basic Auth) or one of the other params for OAuth are required. Credentials is also required now for running against local environment. Let me know if that poses problem.
  
  LikeLike
  
  Reply ↓
  - Joel Cottrell on March 10, 2021 at 2:40 PM said:
    
    Thanks. That got me closer:
    
    PS D:\Install\Scripts\Remove-DuplicateItems> $Credentials= Get-Credential
    
    cmdlet Get-Credential at command pipeline position 1
    Supply values for the following parameters:
    Credential
    PS D:\Install\Scripts\Remove-DuplicateItems> .\Remove-DuplicateItems.ps1 -Identity test@test.com -CleanupMode Mailbox -Verbose -WhatIf:$true -Report -Credentials $Credentials
    VERBOSE: Loading module D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loading module from path ‘D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll’.
    Import-ModuleDLL : Problem loading module Microsoft.Exchange.WebServices: Could not load file or assembly
    ‘file:///D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll’ or one of its dependencies. The
    module was expected to contain an assembly manifest.
    At D:\Install\Scripts\Remove-DuplicateItems\Remove-DuplicateItems.ps1:1316 char:5
    + Import-ModuleDLL -Name ‘Microsoft.Exchange.WebServices’ -FileName …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL
    
    I have not installed the Exchange Web Services (EWS) Managed API tool, but I have copied the *.dll & *.xml files from your Github page to the same directory as the script. I also unblocked the files to no avail.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on March 13, 2021 at 1:07 PM said:
      
      Since you already unblocked the file, “D:” is a local drive? (network mapped drives are likely not trusted)
      Likely a .NET Framework dependency – the DLLs I posted on GitHub for convenience depend on .NET Framework 4.0
      
      LikeLike
      
      Reply ↓
Ryan Shelvik on March 10, 2021 at 6:21 PM said:

I’m trying out this stellar looking script for the first time. Got EWS installed, etc etc and tried to run it and got the following message:

Import-ModuleDLL : Problem loading module Microsoft.Exchange.WebServices: Could not load file or assembly
‘file:///C:\RBS\Support\Remove-DuplicateItems-master\Microsoft.Exchange.WebServices.dll’ or one of its dependencies.
Operation is not supported. (Exception from HRESULT: 0x80131515)
At C:\RBS\Support\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1:1316 char:5
+ Import-ModuleDLL -Name ‘Microsoft.Exchange.WebServices’ -FileName …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

Here’s the command I was using:

.\Remove-DuplicateItems.ps1 -Identity username@domain.com -Credentials $Credentials

Any help is appreciated.

LikeLike

Reply ↓
- Michel de Rooij on March 13, 2021 at 1:24 PM said:
  
  Likely a .NET Framework dependency missing. The DLLs from nuget package depend on .NET Framework 4.0
  
  LikeLike
  
  Reply ↓
  - Lee Howard on April 7, 2021 at 1:27 AM said:
    
    Hi Michel,
    
    We are in the process of moving to Office 365 and on our on-prem Exchange we use Import-CalendarCSV.ps1 and then Remove-DuplicateItems.ps1 for for creating and cleaning up Calendar items.
    
    Could you expand on your comment of ‘Likely a .NET Framework dependency missing. The DLLs from nuget package depend on .NET Framework 4.0’
    I can’t find out what i need to install to get this working, i have the same problem.
    
    I also can’t get impersonation working properly…it looks pretty straight forward from the guides I have found. You mention licensing in your article about set up of impersonation I can’t see anything about this anywhere else.
    
    Thanks in advance 🙂
    
    Lee
    
    LikeLike
    
    Reply ↓
  - Lee Howard on April 8, 2021 at 4:55 AM said:
    
    Hi Michel,
    Could you expand on what dependency are missing?
    Thanks,
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on April 8, 2021 at 7:43 AM said:
      
      The EWS Managed API is built for certain .NET Framework versions. Depending on your OS, this one might be installed or not. NuGet package 2.2.1.0 depends on .NET Framework 4. Also make sure that if you manually copy files over from internet locations, those might be blocked – this includes loading of DLLs as well (Explorer > Properties and see if you need to Unblock)
      
      LikeLike
      
      Reply ↓
      - Lee howard on April 8, 2021 at 8:18 AM said:
        
        Hey Michel,
        
        Thanks for that, I’ll check the DLLs, it’s when I try and install using powershell. In either Windows 10 or server 2012r2
        
        Cheers,
        
        LikeLike
        
        Reply ↓
Lee Howard on April 13, 2021 at 1:04 AM said:

Hi Michel,

Thanks for your help, I managed to get it working by copying the DLL’s into the script folder.
I have a problem with the script not removing duplicates, how does it check for a duplicate?
It is on the title and time?

What’s really odd is that sometimes it will remove items from the calendar if I leave it for a day….but not all items.

For testing i have a CSV file that I import twice, then I run the remove duplicates.

Thanks for any help 🙂

LikeLike

Reply ↓
Benjamin Gaulke on May 28, 2021 at 11:03 AM said:

Hey Michel,

I have a problem getting the script to work leveraging OAuth as a registered App.

After migrating to O365 most mailboxes are full of duplicates and we would like to run your script against the tenant to clean up all mailboxes, without having to have all our users type in / give up their credentials. As I understood it, we should be able to achieve this, using OAuth / registered App and running the script against a tenant-id, providing a csv-file with all users and using a client-id and the corresponding secret.

Individually, all these components seem to work fine, but when put together, as per your last example in the script, (altered according to our tenant / client-id / secret, of course), we still get the message:
Cannot bind to MsgFolderRoot: Exception when calling “Bind” with 2 Arguments: “Credentials are required to make a service request.”

What am I overlooking?

Thank you in advance for your time and response.

Benjamin

LikeLike

Reply ↓
Chad H on May 31, 2021 at 12:25 AM said:

Hi Michel

Was hoping for some assistance here, getting a generic error running the script against a single mailbox in Exchange 2013 CU 23 environment:

“Parameter set cannot be resolved using the specified named parameters.”

I am executing the following:

.\Remove-DuplicateItems.ps1 -Identity %MailBox_Name% -type All -Impersonation -DeleteMode HardDelete -Mode Full -Retain oldest -Force -Server %ServerName%

Am I missing any mandatory parameters here? Thanks in advance!

LikeLike

Reply ↓
- Michel de Rooij on May 31, 2021 at 12:42 AM said:
  
  Missing one of the sets of authentication parameters (for local operation, Credentials)
  
  LikeLike
  
  Reply ↓
cdholmes31 on June 1, 2021 at 12:08 PM said:

Thanks so much for the prompt response, I did use the credentials parameter by declaring it in a variable then executing the script, so the following are now used:

$Credentials = Get-Credential

Password has been converted to securestring also

.\Remove-DuplicateItems.ps1 -Identity %MailBox_Name% -Credentials $Credentials -type All -Impersonation -DeleteMode HardDelete -Mode Full -Retain oldest -Force -Server %ServerName%

But when using the creds parameter I look to run into an issue with converting it back to a string value it appears:

Invalid credentials provided: Cannot convert value “%cleartextPassword” to type “System.Int32”. Error: “Input string was not in a correct format.”

Thanks again!

LikeLike

Reply ↓
- Michel de Rooij on June 3, 2021 at 2:46 PM said:
  
  In PowerShell, variable names are prefixed with $, not prefixed/encapsulated in %
  
  LikeLike
  
  Reply ↓
  - cdholmes31 on June 3, 2021 at 9:53 PM said:
    
    Understood, I actually didn’t encapsulate the values in % here, that was to distinguish that actual values were used in that place, so these would be actual names that were not displayed, hope this clarifies it.
    
    LikeLike
    
    Reply ↓
Djanis on June 7, 2021 at 5:56 PM said:

Hello, I’m from France, nice script, I’m trying to get this work but I’m getting errors after launching the script. Thanks for your help

.\Remove-DuplicateItems.ps1 -Identity test@XXXXXX.fr-Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double quotation marks (“&”) to pass it as part of a string.
At C:\Users\OKIALY\Downloads\Remove-DuplicateItems.ps1:220 char:189
+ … ata-ga-click=”(Logged out) Header, go to Features”>Features <span cla …
+ ~
The '→Mobile <span cla …
+ ~
The '→</s …
+ ~

LikeLike

Reply ↓
- Djanis on June 7, 2021 at 5:58 PM said:
  
  The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double quotation marks (“&”) to pass it as part of a string.
  At C:\Users\OKIALY\Downloads\Remove-DuplicateItems.ps1:222 char:165
  + … ck Link–secondary no-underline f5 Bump-link–hover”>Mobile <span cla …
  + ~
  The '→Actions <span cla …
  + ~
  The '→</s …
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 8, 2021 at 6:32 AM said:
    
    Don’t know what you did, but make sure you fetched the script – not GitHub’s page, i.e.
    https://github.com/michelderooij/Remove-DuplicateItems/raw/master/Remove-DuplicateItems.ps1
    
    LikeLike
    
    Reply ↓
    - BOLUKA LUKOKI on June 8, 2021 at 11:59 AM said:
      
      Thanks for the help, it works now, with the new script
      
      LikeLike
      
      Reply ↓
Alex Rocha on July 8, 2021 at 1:28 AM said:

Hi! thank you for script, but I execute on office365 account and I have this error
“Cannot access mailbox information store for acount@xxx…. The
property ‘DisplayName’ cannot be found on this object. Verify that the property exists.
Whats is it?

LikeLike

Reply ↓
- Michel de Rooij on July 8, 2021 at 8:56 AM said:
  
  Could you run with -Verbose and see where the problem arises
  
  LikeLike
  
  Reply ↓
  - Alex Rocha on July 8, 2021 at 3:45 PM said:
    
    Thank you for feedback, i solved problem change user language to English.
    
    But my problem not was solved i did migration from Google, users duplicate mail in two or more folders, don’t same folder, because on google if message remove inbox, and check with one or more labels, this message migrate to Office365 for multiple folders
    
    Example
    Message1 archived on Google without label, migrate to Archive on Office365
    Massage2 archived on Google and check label “A” migrate to Office365 folder ‘Archive’ and Folder “A”
    
    On Google massage2 appears only one time
    
    Sorry my english!
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 8, 2021 at 7:45 PM said:
      
      No worries, I understood you question. We can do per-mailbox removal of duplicate entries for which the GMail scenario you described would be an excellent example. You may want to look into the CleanupMode parameter, setting it to Mailbox. To control if folders get preference (items retained there before other folders), use the PriorityFolders parameter.
      
      LikeLike
      
      Reply ↓
      - alex rocha on July 8, 2021 at 9:35 PM said:
        
        Hi! I execute cmd but don’t clean mailbox, i have this message, thank you for support
        
        PS C:\Tools> .\Remove-DuplicateItems.ps1 -Identity vallery.vier@….. -Type Mail -CleanupMode Mailbox -DeleteMode HardDelete -Mode Full -Server outlook.office365.com -Credentials (Get-Credential) -verbose
        
        cmdlet Get-Credential at command pipeline position 1
        Supply values for the following parameters:
        Credential
        VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
        VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
        VERBOSE: Using credentials ti@….
        VERBOSE: Cleanup Mode: Mailbox
        Processing mailbox vallery.vier@…. (vallery….@)
        VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
        VERBOSE: Constructing folder matching rules
        VERBOSE: Processing primary mailbox vallery….@
        VERBOSE: Collecting folders to process..
        VERBOSE: Adding folder \Archive (priority 0)
        VERBOSE: Adding folder \ARQUIVO DGS (priority 0)
        VERBOSE: Adding folder \Deleted Items (priority 0)
        VERBOSE: Adding folder \Drafts (priority 0)
        VERBOSE: Adding folder \Important (priority 0)
        VERBOSE: Adding folder \Inbox (priority 0)
        VERBOSE: Adding folder \Junk Email (priority 0)
        VERBOSE: Adding folder \Outbox (priority 0)
        VERBOSE: Adding folder \Sent Items (priority 0)
        VERBOSE: Adding folder \Starred (priority 0)
        VERBOSE: Found 10 folders that match search criteria
        VERBOSE: Processing folder \Junk Email
        VERBOSE: Processing folder \Inbox
        VERBOSE: Processing folder \Outbox
        VERBOSE: Processing folder \Starred
        VERBOSE: Processing folder \Sent Items
        VERBOSE: Processing folder \ARQUIVO DGS
        VERBOSE: Processing folder \Archive
        VERBOSE: Removing 1 items from \Archive
        VERBOSE: Processing folder \Deleted Items
        VERBOSE: Processing folder \Important
        VERBOSE: Processing folder \Drafts
        VERBOSE: Cleaning unique list (finished mailbox)
        17536 items processed and 1 removed in 00:05:04 – average 3455 items/min
        WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified folder
        could not be found in the store.”
        VERBOSE: Processing vallery….. finished
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on July 8, 2021 at 9:58 PM said:
        
        The default mode uses an attribute which Exchange uniquely assigns to every item. Since you’re coming from GMail, that might not work, and you may need to use attribute-level comparison (-Mode Full). Note that before you perform a run, you can report on duplicate items using -Report -WhatIf:$True
        
        LikeLike
        
        Reply ↓
        
        alex rocha on July 8, 2021 at 11:10 PM said:
        
        Hello! I run PS cmd with mode full, Do i need to do other conf at my line?
        
        .\Remove-DuplicateItems.ps1 -Identity vallery.vier@contoso.com -Type Mail -CleanupMode Mailbox -DeleteMode HardDelete -Mode Full -Server outlook.office365.com -Credentials (Get-Credential) -verbose
        
        Thanks!
        
        LikeLike
        
        Reply ↓
        
        Michel de Rooij on July 9, 2021 at 7:49 AM said:
        
        Looks good to go. If duplicates are still not detected, one of the attributes may differ between folderA and folderB. There is another option “Body” comparing only body messages, but there might be legal implications. Last resort is to tweak the comparison.
        
        LikeLike
        
        Reply ↓
        
        Alex Rocha on July 10, 2021 at 3:23 AM said:
        
        Thank you for you script, i run with option Nosize and detect duplicates!!
        
        LikeLike
        
        Reply ↓
Karim Aly on July 15, 2021 at 12:49 PM said:

What if the duplicated messages are in the deleted items folder? can you tell me how to make him scan only the deleted items folder?

LikeLike

Reply ↓
- Michel de Rooij on July 15, 2021 at 7:57 PM said:
  
  Specify IncludeFolders parameter to limit operation to certain folders or folder structures matching patterns. As mentioned in the help for IncludeFolders, you can specify well-known folders such as Deleted Items using ‘##’, e.g. ‘#DeletedItems#’. Deleting items from deleting items doesn’t make sense, but DeleteMode by default moves them to the dumpster (Recoverable Items), so you don’t need to change that.
  Command would then be something like
  .\Remove-DuplicateItems.ps1 -Identity bob@contoso.com -Impersonation -IncludeFolders ‘#DeletedItems#’
  You can always specify -WhatIf:$true to see what it would do, or add -Verbose to see additional information.
  
  LikeLike
  
  Reply ↓
  - Karim Aly on August 3, 2021 at 8:56 AM said:
    
    I find this absolutely useful, but I am still unable to imagine what is the whole command line would look like.
    And another thing, how i can install this module “Remove-DuplicateItems.ps1”?
    I am lost 😦
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 3, 2021 at 8:59 AM said:
      
      1) It’s not a module, it’s a script. Download and store it on the machine you want to run it on.
      2) It depends on some modules – install these modules; consult https://eightwone.com/2020/10/05/ews-webservices-managed-api/
      3) Check Get-Help .\Remove-Duplicates -Examples to see some examples on usage.
      
      LikeLike
      
      Reply ↓
Vit Vet on September 22, 2021 at 7:28 PM said:

I am having issues with a 401 Unauthorized error.

PS C:\Users\users\Downloads\Remove-DuplicateItems-1.84\Remove-DuplicateItems-1.84> .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders Inbox -MailboxOnly -Verbose
VERBOSE: Loaded EWS Managed API v2.2.1.0
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials user@domain.com
VERBOSE: Processing items of type All, delete mode is SoftDelete
Processing mailbox user@domain.com (user@domain.com)
VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
WARNING: Cannot bind to MsgFolderRoot – skipping. Error: Exception calling “Bind” with “2” argument(s): “The request
failed. The remote server returned an error: (401) Unauthorized.”
VERBOSE: Processing user@domain.com finished

LikeLike

Reply ↓
- Michel de Rooij on September 23, 2021 at 8:29 PM said:
  
  – Are credentials valid and do credentials provide access to this mailbox – full access (when granted properly, eg using InheritanceType on top of information store) or impersonation (yet impersonation not specified, so guess not)
  – Is Basic Authentication allowed?
  – Is Basic Authentication allowed for the condition, e.g. location and user (e.g. no Conditional Access blocking things)
  
  LikeLike
  
  Reply ↓
bobebukk on September 23, 2021 at 6:05 PM said:

failing at the first hurdle, trying both ISE and exchange online PowerShell.
Have tried different parameters and my mailbox has full access, although reading this error – this isn’t permissions.

Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
bobebuk on September 23, 2021 at 6:12 PM said:

receiving the error, using both ise and powershell for exchange online.

s.ps1 -Identity fred@contoso.com -Type contacts -Server outlook.office365.com -Verbose
D:\Scripts\RBetts\Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
- Michel de Rooij on September 23, 2021 at 8:35 PM said:
  
  You need to provide one of the authentication methods:
  * Basic using -Credential
  * Modern Auth using TenantId, ClientId and one of the following: Secret, CertificateThumbprint, CertificateFile(+CertificatePassword as secure string)
  
  LikeLike
  
  Reply ↓
Rob on September 24, 2021 at 3:46 PM said:

still getting the same error 😦

.\Remove-DuplicateItems.ps1 -Identity fred.blogs@contosso.com -Tena
ntId xyzabc-abcd-abcde -WhatIf -contacts

Parameter set cannot be resolved using the
specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
- Michel de Rooij on September 24, 2021 at 11:03 PM said:
  
  Seem to be missing one or more parameters for one of the authentication methods, eg
  * Basic using -Credential
  * Modern Auth using TenantId, ClientId and one of the following: Secret, CertificateThumbprint, CertificateFile(+CertificatePassword as secure string)
  
  LikeLike
  
  Reply ↓
Kristaps on October 4, 2021 at 1:17 PM said:

Hi!

I get this error when running against o365

Write-Verbose : A parameter cannot be found that matches parameter name ‘f’.
At C:\Users\xxx\Desktop\Remove-DuplicateItems-master\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1:1438 char:5
2
+ Write-Verbose ‘Using EWS endpoint {0}’ -f $EwsService.Url
+ ~~
+ CategoryInfo : InvalidArgument: (:) [Write-Verbose], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.WriteVerboseCommand

LikeLike

Reply ↓
- Michel de Rooij on October 4, 2021 at 1:21 PM said:
  
  Thanks for catching that – somehow parenthesis got dropped. Get 2.06 or replace line with
  Write-Verbose (‘Using EWS endpoint {0}’ -f $EwsService.Url)
  
  LikeLike
  
  Reply ↓
  - Kristaps on October 4, 2021 at 2:06 PM said:
    
    Thanks for the quick reply!
    
    FYI – if anyone gets this error :
    Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified object was not
    found in the store., The process failed to get the correct properties.”
    WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified folder
    could not be found in the store.”
    
    this means that user which credentials you used to connect doesn’t have permissions to read the mailbox. in my case I used this script against a shared mailbox and used global admin account (no MFA) to connect and it did not work until I gave the global admin read ann manage rights for the shared mailbox.
    
    LikeLike
    
    Reply ↓
Daniel on October 22, 2021 at 9:54 AM said:

Hi,

I am getting following error. Any Ideas?
[PS] C:\temp\Remove-DuplicateItems-master\Remove-DuplicateItems-master>.\Remove-DuplicateItems.ps1 -Identity -Credentials $Credent
ials -Type Calendar -Mode Quick -MailboxOnly -Server outlook.office365.com -Verbose
AUSFÜHRLICH: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
AUSFÜHRLICH: Module Microsoft.Identity.Client v4.25.0.0 already loaded
AUSFÜHRLICH: Using credentials
AUSFÜHRLICH: Cleanup Mode: Folder
Processing mailbox ()
AUSFÜHRLICH: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
AUSFÜHRLICH: Constructing folder matching rules
AUSFÜHRLICH: Processing primary mailbox
AUSFÜHRLICH: Collecting folders to process..
C:\temp\Remove-DuplicateItems-master\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for
: Ausnahme beim Festlegen von “PropertySet”: “Der Wert “Microsoft.Exchange.WebServices.Data.PropertySet” vom Typ
“Microsoft.Exchange.WebServices.Data.PropertySet” kann nicht in den Typ “Microsoft.Exchange.WebServices.Data.PropertySet” konvertiert werden.”
In Zeile:1 Zeichen:1
+ .\Remove-DuplicateItems.ps1 -Identity -Credent …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
Phillip on January 10, 2022 at 7:41 PM said:

Hello,

Looking over your script to see if this is something we can implement. During our testing with it, we are finding that it does identify duplicate items for the mailbox, but is not moving forward with deleting the items? We are noting seeing much in Verbos output that would lead us to any conclusions so thought I would ping you and pick your brain.

Hybrid M365

Thank you again!

LikeLike

Reply ↓
- Michel de Rooij on January 11, 2022 at 3:13 PM said:
  
  By default is identifies duplicates based on PR_SEARCH_KEY. Depending on how you got the duplicates in, that may not work and you may need attribute-level comparison (with or without size using -NoSize). Also make sure proper permissions have been set up, impersonation is preferred.
  
  LikeLike
  
  Reply ↓
Naushad on January 11, 2022 at 2:58 PM said:

Parameter set cannot be resolved using the specified named parameters.
+ CategoryInfo : InvalidArgument: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : AmbiguousParameterSet

LikeLike

Reply ↓
- Michel de Rooij on January 11, 2022 at 3:12 PM said:
  
  You need to provide the script the right parameters so it knows which authentication method to use (and what parameters to expect, as it’s now ambigious):
  1) BasicAuth : -Credentials
  2) Modern Auth:
  2a) -CertificateThumbprint -TenantId -ClientId
  2b) -Secret -TenantId -ClientId
  2c) -CertificateFile [-CertificatePassword ] -TenantId -ClientId
  
  LikeLike
  
  Reply ↓
jedivader20 on January 20, 2022 at 7:55 AM said:

I was having issues getting this to authenticate for the longest time, always getting 401 Unauthorised. I even setup a policy to allow EWS basic auth for the account that I was trying to use for impersonation. As it turned out, it was disabled at a tenant level, and per-user policies weren’t going to cut it.

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-june-2021-update/ba-p/2454827

This fixed the issue permanently for me.

LikeLike

Reply ↓
Anthony on February 16, 2022 at 5:36 AM said:

Hi Not sure if i have missed something but i followed the details of this post and the one for the EWS.Websrvices.Managed.Api post but i am getting the following error when attempting to run the script:
Exception calling “LoadFile” with “1” argument(s): “An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions
of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please
enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.”
At C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1:343 char:13
+ [void][Reflection.Assembly]::LoadFile( “$EWSDLLPath\$EWSD …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : NotSupportedException

Load-EWSManagedAPIDLL : Problem loading Microsoft.Exchange.WebServices.dll
At C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1:994 char:5
+ Load-EWSManagedAPIDLL
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Load-EWSManagedAPIDLL

Is there something im missing here?

LikeLike

Reply ↓
- Anthony on February 16, 2022 at 5:46 AM said:
  
  Ok so I have found what i missed.
  There is the need to Unblock not just the DLL and XML files but also the PS1 file as well.
  This has allowed the script to run.
  
  I am now getting another error:
  C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1 : Autodiscover failed, error: Exception calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover
  service couldn’t be located.”
  At line:1 char:1
  + .\Remove-DuplicateItems.ps1 -Identity user@domain.com -Typ …
  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
  + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
  
  Ill be looking furhter into this myself and if i find the answer to this also i will add this to my reply.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on February 16, 2022 at 10:01 AM said:
    
    Be sure there is an autodiscover record published and its location accessible for @domain.com. To manually override, use -Server, e.g. -Server outlook.office365.com for Office 365.
    
    LikeLike
    
    Reply ↓
David Roberts on March 1, 2022 at 10:44 AM said:

This script looks incredible, so first off, thank you for the dedicated work on it over the years.

Unfortunately I can’t get this to authenticate with Exchange Online. I’m not 100% sure how the App Registration in AAD interacts with EXOL. I’m using an existing App Registration created for BitTitan to migrate emails into our tenant, so it should already have impersonation permissions as BitTitan can write into any mailbox in our tenant and we use the impersonation option within BitTitan. The app has Office 365 Exchange Online – EWS.AccessAsUser.All delegated permission, as well as Microsoft Graph – User.Read. I’ve added a secret to the app to use with this script. I’m running the following:

$secret = ConvertTo-SecureString “XXXXXXXXXXX” -AsPlainText -Force

.\Remove-DuplicateItems.ps1 -Identity user@domain.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Retain Oldest -IncludeFolders ‘#Inbox#\*’ -Server outlook.office365.com -Secret $secret -TenantId -ClientId

I get the following result:

WARNING: Cannot bind to Inbox: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”
WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”
WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

We have Modern Authentication enabled in our tenant and Basic Authentication turned off for all options. I can see the app authentication in the AAD sign-in logs as being successful, so it feels like that part of the authentication process is working, but not something at the EXOL level maybe? I don’t know.

Any pointers would be greatly appreciated! Thanks in advance.

LikeLike

Reply ↓
- Michel de Rooij on March 6, 2022 at 8:58 AM said:
  
  I assume you also specified the proper ‘TenantId’ and ‘ClientId’ IDs with those parameters (not specified)?
  Since the script is not from a verified publisher, we need app permissions, not delegated permissions. Also, the permission I usually assign is Office 365 Exchange Online > full_access_as_app; if orgs worry about this blanket permission, they can restrict it to certain mailboxes using application access policies.
  
  LikeLike
  
  Reply ↓
  - David Roberts on March 7, 2022 at 12:50 PM said:
    
    Thanks for the reply Michel.
    
    After a bit of research I came across that exact permission, which I applied to the Azure AD App Registration and then it all started working.
    
    Yes I did indeed have the Tenant & Client IDs specified, I just removed them from the command.
    
    Thanks again for the script, it was just what I needed!
    
    LikeLike
    
    Reply ↓
    - djansen1987 on September 8, 2022 at 7:50 PM said:
      
      Would you like to share that ?
      
      LikeLike
      
      Reply ↓
Glenn on March 28, 2022 at 9:46 PM said:

Great script. Question: I’m testing it now and it seems to remove the first of 2 emails matched (or, the first one displayed in Outlook, at any rate). What would need to be changed to make it remove the 2nd one instead? I realize this is a fringe case, but it is useful if you have to import emails exported from IMAP, but you already imported some of them via ActiveSync, and some of the headers are different, apparently, so internal duplicate detection in Outlook doesn’t work… *sigh*. Seems the first displayed email (from the ActiveSync export) retains some of the group name expansion and short names, whereas the 2nd one (from the IMAP export) does not, so it would make more sense to keep the first one in my case just for the user experience.

LikeLike

Reply ↓
- Michel de Rooij on March 31, 2022 at 8:49 AM said:
  
  Defining a criterium based on headers is a challenge, apart from the potential performance as you have to inspect each item.
  Only thing you can use for for determining what gets deleted is Retain, where you can specify Oldest or Newest. Depending on the way on you those duplicates got in, you may want to use Mode Full to have attribute-level comparison instead of PidTagSearchKey, which is an attribute set by Exchange when the item gets stored, and is preserved when copying, exporting/importing etc.
  
  LikeLike
  
  Reply ↓
Andrea Gallina on April 15, 2022 at 1:14 PM said:

I receive this error when i launch the script…i don’t know how to solve.. Can you help me?

PS C:\script> .\Remove-DuplicateItems.ps1 -Mailbox user@contoso.com -Server outlook.office365.com -Credentials $Credentials -Type Mail -DeleteMode HardDelete -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
DETTAGLIATO: Loading module C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll
DETTAGLIATO: Caricamento del modulo dal percorso ‘C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll’.
DETTAGLIATO: Module Microsoft.Exchange.WebServices v2.2.1.0 loaded
DETTAGLIATO: Loading module C:\Program Files\PackageManagement\NuGet\Packages\Microsoft.Identity.Client.4.43.0\lib\monoandroid10.0\Microsoft.Identity.Client.dll
DETTAGLIATO: Caricamento del modulo dal percorso ‘C:\Program Files\PackageManagement\NuGet\Packages\Microsoft.Identity.Client.4.43.0\lib\monoandroid10.0\Microsoft.Identity.Client.dll’.
Import-ModuleDLL : Problem loading module Microsoft.Identity.Client: Non è stato possibile caricare il file o l’assembly ‘Mono.Android, Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065’ o
una delle relative dipendenze. Impossibile trovare il file specificato.
In C:\script\Remove-DuplicateItems.ps1:1379 car:5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

LikeLike

Reply ↓
- Michel de Rooij on April 15, 2022 at 4:13 PM said:
  
  Depending on your security settings, you may need to run PowerShell with Elevated Permissions in order to be able to load the DLL modules.
  
  LikeLike
  
  Reply ↓
SteveJ on April 30, 2022 at 1:48 PM said:

Great script 🙂 However I’m having an issue getting it to process Inbox subfolders.

I use the following parameter -IncludeFolders ‘#INBOX#\*’ and the script process the Inbox OK but ignores the dozens of subfolders. The mailbox is hosted in Exchange Online BTW.

Am I missing something?

LikeLike

Reply ↓
- SteveJ on April 30, 2022 at 1:52 PM said:
  
  I just found that if I create a new subfolder, the script finds it. But it isn’t finding the dozens of existing folders (that were imported from a PST file if that makes any difference).
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on April 30, 2022 at 7:04 PM said:
    
    Are you using Impersonation? If not, does the account provided have sufficient permissions on those imported subfolders.
    Also, have you specified -Type Mail by any chance? Could be the Folder type of those imported folders are not Mail (IPF.Note)
    
    LikeLike
    
    Reply ↓
Bernard Koops on May 2, 2022 at 12:31 PM said:

Great script, doing what’s advertised, love it. Ran the script for the shitload of extra data that came to be due to the old registry setting to copy a sent email in the shared mailbox sent items, which as well was done by EAC rule… just love (not) what was done by my predecessors without documenting their actions and hiding it in obscure gpo’s…

I made 1 (cosmetic) adjustment – line 1155 had ‘Processed folder {0} or {1}’ which should of course be ‘Processed folder {0} of {1}’ 😉

LikeLike

Reply ↓
- Michel de Rooij on May 3, 2022 at 10:19 AM said:
  
  Thanks for catching that.
  
  LikeLike
  
  Reply ↓
Ricardo Marinho | Webideia on May 18, 2022 at 12:51 AM said:

Hi,
Script runs but no duplicates are found/deleted.

Anyone experienced same behaviour?

LikeLike

Reply ↓
- Michel de Rooij on May 19, 2022 at 4:09 PM said:
  
  Perhaps it is not finding what you expected it to find. There could be many reasons, usually because properties not directly visible that could identify duplicates are not identical. Also, make sure you have proper permissions on mailbox and folder, preferably using Impersonation so you do not have that issue.
  
  LikeLike
  
  Reply ↓
sathish on May 24, 2022 at 2:42 AM said:

Some reason the script is throwing below error in Ms365 online mailbox

PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Mailbox agnoc -Type All -Impersonation -DeleteMode HardDelete -Mode Full -Verbose
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:246 char:1
+ [cmdletbinding(
+ ~~~~~~~~~~~~~~~
Unexpected attribute ‘cmdletbinding’.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:251 char:1
+ param(
+ ~~~~~
Unexpected token ‘param’ in expression or statement.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:285 char:20
+ [string]$Type= ‘All’,
+ ~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:302 char:22
+ [string]$Retain= ‘Newest’,
+ ~~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:351 char:26
+ [string]$DeleteMode= ‘SoftDelete’,
+ ~~~~~~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:368 char:20
+ [string]$Mode= ‘Quick’,
+ ~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : UnexpectedAttribute

LikeLike

Reply ↓
- Michel de Rooij on May 24, 2022 at 6:55 AM said:
  
  Did you download the script, or copy/paste between GitHub and your environment? If the latter, likely issue with that (turning quotes into smart quotes etc.)
  
  LikeLike
  
  Reply ↓
  - sathish on May 25, 2022 at 3:27 AM said:
    
    I have downloaded from git and getting below error. please suggest
    
    PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Identity agnoc -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials $agccred -Retain Oldest
    
    Processing mailbox agnoc (agnoc)
    
    *********************************************************************************************
    
    C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1 : Autodiscover failed: Exception
    calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover service couldn’t be located.”
    At line:1 char:1
    + .\Remove-DuplicateItems.ps1 -Identity -Type Mail – …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    
    *********************************************************************************************
    
    PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Identity agnoc -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials $agccred -Retain Oldest -Server outlook.office365.com
    
    Processing mailbox agnoc (agnoc)
    
    *********************************************************************************************
    WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote
    server returned an error: (401) Unauthorized.”
    WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The
    remote server returned an error: (401) Unauthorized.”
    
    *********************************************************************************************
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 25, 2022 at 8:32 AM said:
      
      I see two messages:
      – “The Autodiscover service couldn’t be located”: Either configure Autodiscover properly, or use Server to point to the EWS endpoint.
      – 401 Unauthorized is a permissions issue – either you have none, or BasicAuth is blocked (you specified Credentials, so using BasicAuth)
      
      LikeLike
      
      Reply ↓
D.andre on May 25, 2022 at 12:36 PM said:

Hello.
I have 2 issues with the script :
The first one when i execute the script for the first time. I have this.

COMMENTAIRES : Module Microsoft.Identity.Client v4.29.0.0 loaded
Import-ModuleDLL : Problem initializing test-object from module Microsoft.Identity.Client: Constructeur introuvable.
Impossible de trouver un constructeur approprié pour le type Microsoft.Identity.Client.MsalClientException.
Au caractère D:\Temp\Remove-DuplicateItems.ps1:1381 : 5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

And after a retry, i have this :

D:\Temp\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for XXXX@XXXXX: Exception lors de la définition de «PropertySet»: «Impossible de convertir la valeur «
Microsoft.Exchange.WebServices.Data.PropertySet» du type «Microsoft.Exchange.WebServices.Data.PropertySet» en type «Microsoft.Exchange.WebServices.Data.PropertySet».»
Au caractère Ligne:1 : 1
+ .\Remove-DuplicateItems.ps1 -Identity u007259 -Retain Newest -DeleteM …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

I execute the script with an account who have fullaccess on the mailbox and I give the role “ApplicatioImpersonation” on my user.

Have you any clues ?

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2022 at 3:21 PM said:
  
  FullAccess is not Impersonation – if you have Full Access, don’t specify Impersonation and make sure you have proper permissions on Top of Information store and folder-level as well (InheritanceType). When specifying Impersonation, be sure to grant these permissions properly and give it some minutes. Impersonation is preferred due to it less potential for issues or blockage by owner.
  
  LikeLike
  
  Reply ↓
  - d.andre on May 31, 2022 at 9:32 AM said:
    
    I have configure the impersonation for the user who launch the script and i have the same error.
    Impersonisation ApplicationImpersonation admExchange User Direct admExchange
    
    I try without the Impersonation argument after i give to my user full accesright with inheritancetype at all with this command and i have the same error.
    
    Add-MailboxPermission -Identity u007259 -User admexchange -AccessRights FullAccess -InheritanceType all
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 31, 2022 at 10:50 AM said:
      
      To be sure, is this Exchange on-premises and if so what version?
      
      LikeLike
      
      Reply ↓
      - D.andre on May 31, 2022 at 10:53 AM said:
        
        Yes it’s an exhange 2016 on-premise
        
        LikeLike
        
        Reply ↓
D.andre on May 25, 2022 at 12:59 PM said:

Hello.

I have 2 error when i use the script with this command
.\Remove-DuplicateItems.ps1 -Identity Username -Retain Newest -DeleteMode SoftDelete -Type Calendar -UseDefaultCredentials -server FQDN name server -Mode Full -Verbose

The first one when i lauch the script for the first time

COMMENTAIRES : Module Microsoft.Identity.Client v4.29.0.0 loaded
Import-ModuleDLL : Problem initializing test-object from module Microsoft.Identity.Client: Constructeur introuvable.
Impossible de trouver un constructeur approprié pour le type Microsoft.Identity.Client.MsalClientException.
Au caractère D:\Temp\Remove-DuplicateItems.ps1:1381 : 5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

The second one is this. I have full access right on the mailbox and the user have the ” ApplicationImpersonation” role.

COMMENTAIRES : Collecting folders to process..
D:\Temp\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for D.ANDRE@chru-nancy.fr: Exception lors
de la définition de «PropertySet»: «Impossible de convertir la valeur «Microsoft.Exchange.WebServices.Data.PropertySet
» du type «Microsoft.Exchange.WebServices.Data.PropertySet» en type «Microsoft.Exchange.WebServices.Data.PropertySet».»
Au caractère Ligne:1 : 1
+ .\Remove-DuplicateItems.ps1 -Identity u007259 -Retain Newest -DeleteM …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

Have you an idea of what happen.

Best regards

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2022 at 3:25 PM said:
  
  They sometimes change classes in the module, which I pick one to see if loading and initializing goes OK. Made a note to check/revise.
  
  LikeLike
  
  Reply ↓
  - Eric Hester on August 22, 2022 at 7:36 PM said:
    
    I am also getting this error. Not sure I understand the solution?
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 22, 2022 at 10:17 PM said:
      
      The solution was mine to make- has been implemented. If you still experience this issue, let me know.
      
      LikeLike
      
      Reply ↓
Jonathan on June 21, 2022 at 10:39 PM said:

I cannot get the Microsoft.Identity.Client.dll module DLL to load. I know there are .Net 4 dependencies and I’m running Windows 10 and have .Net 4.8 installed – so I should be covered on dependencies right? Also, I downloaded and moved the Microsoft.Identity.Client.dll and the Microsoft.Identity.Client.xml into the script directory but still, same error. Any ideas? Thank you

VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
VERBOSE: Loading module C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll
VERBOSE: Loading module from path ‘C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll’.
Import-ModuleDLL : Problem loading module Microsoft.Identity.Client: Could not load file or assembly
‘file:///C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll’ or one of its dependencies. The module
was expected to contain an assembly manifest.
At C:\Users\jharris\Downloads\removedups\Remove-DuplicateItems.ps1:1481 char:5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

LikeLike

Reply ↓
- Michel de Rooij on June 22, 2022 at 6:26 AM said:
  
  Odd, I have no issues with the DLLs that I stored with the script on GitHub on w10.
  After downloading the DLL and manifest XML, you did unblock those files?
  
  LikeLike
  
  Reply ↓
Jonathan on June 22, 2022 at 4:42 AM said:

Amazing script Michel. The script is running on a mailbox in my Office 365 tenant, but it seems to be skipping the Inbox and a subfolder of the inbox with the following errors:

VERBOSE: Processing folder \Inbox\ECI
WARNING: Error performing operation FindItems without Search options in ECI. Error Exception calling “FindItems” with
“1” argument(s): “The property Hashtags is valid only for Exchange Exchange2015 or later versions.”
VERBOSE: Cleaning unique list (finished folder)
VERBOSE: Processing folder \Inbox
WARNING: Error performing operation FindItems without Search options in Inbox. Error Exception calling “FindItems” with
“1” argument(s): “The property Hashtags is valid only for Exchange Exchange2015 or later versions.”
VERBOSE: Cleaning unique list (finished folder)

I did some research and found this article talking about restricting the FirstClassProperties call to only the required properties (and consequently excluding hashtags). This is a bit over my head but maybe this line in your script is returning an incompatible property in my situation?

$ItemView.PropertySet= New-Object Microsoft.Exchange.WebServices.Data.PropertySet( [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)

https://stackoverflow.com/questions/52573779/the-property-hashtags-is-valid-only-for-exchange-exchange2015-or-later-versions

LikeLike

Reply ↓
- Michel de Rooij on June 22, 2022 at 8:25 AM said:
  
  First time I see this one and cannot repro. I do target a specific product level [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1. Could you replace that entry with [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2015 in your copy of the script and let me know if this solves it for you? Might have to make a modification.
  
  LikeLike
  
  Reply ↓
  - Harris, Jonathan on June 22, 2022 at 2:41 PM said:
    
    Replacing the 2 entries with 2015 fixed it. It processed both Inbox folders this time – no errors. I really appreciate your quick response and your help!
    
    Also, I wrote a comment before this about not being able to get the script to run. Obviously, I got it working finally. I had to re-download the Identity DLL from GitHub. No idea why the first one didn’t work even after unblocking it.
    
    Really appreciate your help,
    
    JonathanHarris Assistant Vice President, Information Technology Extell Financial Services Inc 9911 Shelbyville Road, Louisville, KY40223 t: +1646-790-4047|f: +1212-712-6100| c: +1347-450-3179 jharris@extell.com|www.extell.com This email is confidential and may contain information that is privileged, attorney work product and/or exempt from disclosure under applicable law and should be read or retained only by the intended recipient. Any review, reliance or distribution by others or forwarding without express permission of the sender is strictly prohibited. If you received this transmission in error, please immediately contact the sender and delete all copies including all attachments.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on June 23, 2022 at 12:29 PM said:
      
      Thanks for the update, much appreciated.
      
      LikeLike
      
      Reply ↓
Michael Kenny on July 18, 2022 at 4:41 PM said:

I am clearly missing something but when I run the script in PS it thinks for a second and then goes back to the PS prompt without doing anything. I am running Office 365 and able to login to exchange online, etc. No errors, but no success either.

LikeLike

Reply ↓
- Michel de Rooij on July 18, 2022 at 7:05 PM said:
  
  Look at the usage examples, and use -Verbose switch to let is present more output on what’s going on.
  
  LikeLike
  
  Reply ↓
  - Michael on July 18, 2022 at 8:49 PM said:
    
    It was an issue with Powershell 7. Moved to another PC with PS 5 on it and everything worked well. Totally an issue between my chair and my keyboard….
    
    LikeLike
    
    Reply ↓
Roger Vaughan on July 26, 2022 at 1:25 PM said:

Hi Michel,
Trying to change from BasicAuth (which I have used for years) to modern. I am running this script : .\Remove-DuplicateItems.ps1 -Mailbox xxx@xxx.com -TenantId xxx -ClientId xxx -Secret (ConvertTo-SecureString “xxx” -AsPlainText -Force) -Server outlook.office365.com -Mode Body -DeleteMode SoftDelete -MailboxOnly -Verbose.
I am getting this error:
WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “Credentials are required to make a service request.”
I believe I have set up the APP ok with redirection URI and given full_access_as_app permissions to Office 365 Exchange Online.
Any assistance would be great.

LikeLike

Reply ↓
- djansen1987 on September 8, 2022 at 7:51 PM said:
  
  Tried the same as you but with no success either. I also tried the application permission “EWS.AccessAsUser.All”. I do not have a redirect URI (kept it empty) and wondering what you gave up.
  
  My command:
  
  $secret = read-host -AsSecureString -Prompt “secure value”
  
  .\Remove-DuplicateItems.ps1 -report -whatif -Mailbox “user@domaininO365.ext” -Server outlook.office365.com -Verbose -type mail -TenantId “1234-5678-9123456-12435-13245jh32145fg324” -ClientId “dfg45ewrdsff-sdfghgf657-234rfd-gr234-345gfd435dfdfge” -Secret $secret -MailboxOnly
  
  VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
  VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
  VERBOSE: Will use provided secret to authenticate
  VERBOSE: Authentication token acquired
  VERBOSE: Cleanup Mode: Folder
  Processing mailbox user@domaininO365.ext (“user@domaininO365.ext”)
  VERBOSE: Using user@domaininO365.ext for impersonation
  VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
  VERBOSE: Constructing folder matching rules
  WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “Credentials are required to make a service request.”
  
  LikeLike
  
  Reply ↓
  - Roger on November 1, 2022 at 6:42 PM said:
    
    Ok, I finally got it working.
    It needed the Application ID to be the “ClientID”, and then I created a new Secret. It’s not the Secret ID in Azure you need to then use, it’s the Value (which makes sense if you think of it).
    As per your script, i read user input (using Read-Host) to convert the secret value to be a secure string (used $string as the variable), then used -Secret $secret in the command.
    
    LikeLike
    
    Reply ↓
Tim on October 30, 2022 at 11:45 AM said:

Hi Michel, basic question for you – if we export a PST with purview there is a deduplication option, which removes dupes. Any drawbacks of this method vs your script?

LikeLike

Reply ↓
- Michel de Rooij on October 31, 2022 at 10:34 AM said:
  
  Purview export dedupe has a fixed method to identify duplicates – the script is bit more flexible (and customizable when needed). See the following article on the recipe, but also make sure you read the notes on downside (“Limitations of the de-duplication algorithm”):
  https://learn.microsoft.com/en-us/microsoft-365/compliance/de-duplication-in-ediscovery-search-results?view=o365-worldwide#how-duplicate-messages-are-identified
  
  LikeLike
  
  Reply ↓
Michael Bayless on November 21, 2022 at 11:45 PM said:

Really LOVE this script. I appreciate all the work and detail you put into this it’s a great utility.

I had things working for awhile and all of a sudden it stopped working. I get either a 401 Unauthorized or a 403 Forbidden depending on if I use Get-Credentials (401) or Certificate based authentication(403). I have verified that the user connecting has appropriate permissions: AccessRights(FullAccess) InheritanceType(All). User is also global admin. I do have MFA turned on and it never prompts for this, not sure why. It is connecting to M365.

Here is the command with User Auth: .\Remove-DuplicateItems.ps1 -Identity -Type Mail -Mode Full -Credentials (Get-Credential) -Verbose -Report >> c:\Util\USERDuplicate.txt

Here is the command with Certificate Auth: .\Remove-DuplicateItems.ps1 -Identity -Type Mail -Mode Full -Verbose -Report >> c:\Util\DanyelleDuplicate.txt –TenantId ‘<Directory (tenant) ID from AAD' -ClientId 'Application (client) ID from AAZ' -CertificateThumbprint 'Thumbprint from certificate installed in personal store for user (and computer)'

As mentioned I had it working before with just logging in with the administrator. Not sure what has changed but it's driving me nuts.

Any help would be greatly appreciated!

LikeLike

Reply ↓
- Michel de Rooij on November 22, 2022 at 12:36 AM said:
  
  Likely, modern authentication is required to connect. Therefor, passing credentials doesn’t work any longer. The app which you refer to using ClientId, is it properly registered, is the certificate still valid to authenticate and not expire, and does the app have the necessary permissions (Exchange.ManageAsApp)? Also, are there no conditional access rules effective which potentially could block your authentication attempt? (look in Azure Sign-Ins and filter on your ClientId (Application ID).
  
  LikeLike
  
  Reply ↓
  - Michael Bayless on November 23, 2022 at 6:03 PM said:
    
    First, I can’t tell you how much I appreciate your help!
    – Modern Authentication — this is correct with Microsoft’s move to requiring modern authentication I’m thinking this will be come much more of an issue.
    
    -App — Everything was registered properly. I created the app / certificate login stuff specifically for this so all certs were good. I followed this and it worked almost perfectly: https://learn.microsoft.com/en-us/exchange/client-developer/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth
    
    -Permissions — So this is where my issue was. I verified that the Exchange.ManageAsApp was there and Admin Consent was granted. However, I had to go just a little further. I followed the following article and added additional permissions (pretty much all of them so I can manage all aspects of the exchange with the app logon). https://www.michev.info/Blog/Post/3180/exchange-api-permissions-missing
    
    -WOHOO it’s now working again.
    
    LikeLike
    
    Reply ↓
David McFarland on December 14, 2022 at 11:39 PM said:

First off, I wanted to say that this script is amazing. I am really hoping it will fix a duplication issue we are having. When I run the script against a mailbox, it looks like its working fine, successfully removes a the first few thousand emails and then starts giving me this error. WARNING: Error performing operation RemoveItems with . Error: Exception calling “DeleteItems” with “5” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.” I am guessing this is a timeout from the server, but I was hoping you might be able to clarify what I am seeing. Using TenantID and Client with Secret to Auth.

LikeLike

Reply ↓
- Michel de Rooij on December 15, 2022 at 9:13 AM said:
  
  No idea, it handles throttling but not disconnects etc. Do know that it processes/removes items in batches, so when you run the command again, it might find/delete items which are still in the process of being deleted from the previous run, which might also lead to interesting messages.
  
  LikeLike
  
  Reply ↓
KerG on December 16, 2022 at 11:53 PM said:

Thanks for the work put into this. Very helpful for sure. So I am running into an issue where it deleted a lot (over 100,000) duplicates in multiple folders. However, there are a few folders left that have multiple dupes. I have run it again, just on a single small folder to test and the script does see them and says that it has removed them, but they remain in the mailbox. I have started a new session, reconnected and ran it multiple times with the same results over a 5-6 hour period. Any thoughts?

LikeLike

Reply ↓
- Michel de Rooij on December 17, 2022 at 4:31 PM said:
  
  Is OWA showing the same thing? Could be your Outlook/OST is catching up on the removals.
  Script is operating against a non-cached view, so if it finds duplicates, they are (still) there, but if Exchange is still processing it might be locking the folder, blocking any other (EWS-based) operations against it. And unfortunately nothing much you can do once it has been initiated.
  
  LikeLike
  
  Reply ↓
misterkgg on December 17, 2022 at 5:34 PM said:

OWA showing the same thing. Not sure how Exchange still processing after 24 hours. May just need to recreate the mailbox at this point

LikeLike

Reply ↓
- Michel de Rooij on December 17, 2022 at 5:49 PM said:
  
  Can take a while, but not *that* long (also, the removal is per batch of max. 100 items).
  
  LikeLike
  
  Reply ↓
  - misterkgg on December 17, 2022 at 5:55 PM said:
    
    Yeah, not sure what happened. Started off great, then just stopped. I will recreate the mailbox and save myself the effort. Curious to know what happened though as I’m sure will use this in the future.
    
    LikeLike
    
    Reply ↓
Roger on January 19, 2023 at 4:59 PM said:

Hi Michel,
Does the duplication take into account message headers?
I am noticing that a fair amount of duplicates that a client has are not getting removed as the messages have a few differences. To give you a rough idea…
1.) User 1 and user 2 receive the same email from an external sender.
2.) They both move this email to a shared mailbox. (as to “file” it).
3.) I see the duplicates in the shared mailbox so run the duplicate removal job.
4.) The messages both remain.
5.) I assess the message headers for both – and the final received by server in the headers differs for both (Both 365).

With this in mind, is there a way to omit the actual headers from being assessed? I have also seen issues with duplicate messages where the message size is different and the “created date” different too.
None of the switches (Body, Full (with NoSize) or Quick) seem to pick them up.
Thanks in advance.
Roger.

LikeLike

Reply ↓
Jordan on March 8, 2023 at 8:23 PM said:

Hi Michel,

Trying to remove duplicate contacts from a user, its removing all contacts from the user completely not just duplicates.

.\Remove-DuplicateItems.ps1 -Identity user@domain.etc -Type Contacts -DeleteMode MoveToDeletedItems -Mode Full -NoSize -Retain Newest -TenantId ‘123456-1234-1234-1234-123456’ -ClientId ‘1234567-1234-1234-1245-87654323456’ -Secret $Secret -Report -WhatIf:$true

This is my What If output –

What if: Performing the operation “Process-Mailbox” on target “Remove 1264 items from \Contacts”.

The user only has 1265 contacts total listed again their account, of which I can visibly see there are hundreds of duplicates.

Any ideas on what’s happening here?

Many Thanks,

LikeLike

Reply ↓
- Michel de Rooij on March 9, 2023 at 8:58 PM said:
  
  Yeah, it didn’t load the properties for non-mail items used for matching. Just pushed version 2.3 which should work.
  Thanks for catching this.
  
  LikeLike
  
  Reply ↓
Joshua Huffman on May 11, 2023 at 4:59 AM said:

This works great. But i am having an issue. I am connected using Azure app for authentication. The process starts failing at exactly 1 hour and 5 minutes. The inboxes i am scanning and cleaning are not completed by the time this happens. I then have to start the process over and get a little more done. The users in question have 3k folders. Is there any way to extend the time out.

WARNING: Error performing operation FindItems with Search options in Dunvale Sale. Error: Exception calling “FindItems” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

Above is the error i start getting at exactly 1 hour 5 minutes.

LikeLike

Reply ↓
- Michel de Rooij on May 11, 2023 at 7:18 AM said:
  
  It’s not a timeout, most likley your token expired. Reauthentication is not handled at the moment, but point noted.
  Note that you can just call it again and it will (though iteratively) do its task.
  
  LikeLike
  
  Reply ↓