Removing Duplicate Items from a Mailbox

Posted on June 21, 2013 by Michel de Rooij

Latest version: 2.41, April 18th, 2023

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
Importing existing data. Accidental import from – for example – a PST file to a mailbox can lead to duplicate entries.

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange Web Services (EWS) Managed API and for OAuth authentication the Microsoft Authentication Library (MSAL) libraries. You can install these packages from NuGet, or place their DLL’s in the same folder as the script. For an example of how to install EWS.Managed.Api from NuGet, see this article; for MSAL follow the same process but with the package titled ‘Microsoft.Identity.Client’.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions when using Basic Authentication; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post. Using a registered app with OAuth is always through Impersonation.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [[-Identity] ] [[-Type] ] [-Retain ] [-Server ] [-Impersonation] [-DeleteMode ] [-Credentials ] [-Mode ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-PriorityFolders <String[]>] [-NoSize] [-CleanupMode] [-NoProgressBar] [-Force] [-WhatIf] [-Confirm] [-Secret] [-CertificateThumbprint] [-CertificateFile] [-CertificatePassword] [-TenantId] [-ClientId] [-TrustAll] [-ExchangeSchema <String>] [-NoSCP]

A quick walk-through on the parameters and switches:

Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:

ItemClass	Criteria
Contacts	File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List	FileAs, Number of Members, Size
Calendar	Subject, Location, Start & End Date, Size
Task	Subject, Start Date, Due Date, Status, Size
Note	Contents, Color, Size
Mail	Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other	Subject, DateTimeReceived

MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter when running against mailboxes on Exchange Server 2007.
ArchiveOnly specifies you only want to process personal archives of specified users.
IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
CleanupMode specifies to cleanup duplicates per folder (Folder, default), the whole mailbox (Mailbox), or multiple mailboxes (MultiMailbox, identities specified using Identity). The first unique item encountered will be retained. For Mailbox-level cleanup, PriorityFolders can be used to give priority to retaining items in specified folders before those found in other folders.
PriorityFolders specifies which folders have priority over other folders, identifying items in these folders first when using MailboxWide mode. Usage of wildcards and well-known folders is identical to IncludeFolders.
NoSize tells script to not use size to match items in Full mode.
NoProgressBar prevents displaying a progress bar as folders and items are being processed.
Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.
TrustAll can be used to accept all certificates, e.g. self-signed certificates or when accessing Exchange using endpoint with a different certificate.
ExchangeSchema can be used to specify the Exchange schema to use when connecting to Exchange server or Exchange Online. Defaults to Exchange2013_SP1 or Exchange2016 when -Server is specified and is ‘outlook.office365.com’ (Exchange Online endpoint).
NoSCP to skip SCP lookups in Active Directory for Autodiscover.

For authentication, the following parameters are available:

Credentials specifies credentials to use for Basic Authentication.
TenantId specifies the identity of the Tenant (OAuth)
ClientId specifies the Id of the registered application (OAuth).
CertificateThumbprint specifies the thumbprint of the certificate from personal store to use for authentication (OAuth).
CertificateFile specifies the external certificate file (pfx) to use for authentication (OAuth). This certificate needs to contain a private key; the registered application needs to contain the certificate’s public key.
CertificatePassword optionally specifies the password to use with the certificate file (OAuth).
Secret specifies the secret to use with the application (OAuth).

Few notes:

When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter	Matches
FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA
Folder*	\TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\Folder	\TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\FolderA\	\TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA	\TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So, suppose you want to remove duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

Alternative, you can use an e-mail address and specify credentials. This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Identity olrik@office365tenant.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

A more complex example using IncludeFolders, ExcludeFolders and PriorityFolders:

$Credentials= Get-Credential
 .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '#Inbox#\*','\Projects\*' -ExcludeFolders 'Keep Out' -PriorityFolders '*Important*' -CleanupMode Mailbox

This will remove duplicate items from the specified mailbox in Office365, using the following options:

Fixed Server FQDN – bypassing AutoDiscover.
Limits operation against the Well-Known Inbox folder, top Projects folder, and all of their subfolders.
Excluding any folder named Keep Out.
Duplicates are checked over the whole mailbox.
Priority is given to folders containing the word Important, causing items in
those folders to be kept over items in other folders when duplicates are found.

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 ..

Download
The script is available on GitHub here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

484 thoughts on “Removing Duplicate Items from a Mailbox”

Vikas on February 21, 2021 at 4:31 PM said:

Tried executing this script. It doesn’t give error but is not deleting duplicate items from Office 365.
I am trying to delete all duplicate mails from server in multiple folders. Have changed email for security purpose
Am I missing any thing
Script used is :
$Credentials= Get-Credential
.\Remove-DuplicateItems.ps1 -Mailbox abc@mycompany.com -Server outlook.office365.com -Credentials $Credentials -DeleteMode HardDelete -ExcludeFolders ‘Keep Out’ -PriorityFolders ‘*Important*’ -MailboxWide

LikeLike

Reply ↓
- Michel de Rooij on February 21, 2021 at 11:20 PM said:
  
  Not missing anything. Is it processing the folders and items you expect (-Verbose); use -WhatIf:$True and -Report to see what it would do. Could be the items are not considered duplicate for several reasons.
  
  LikeLike
  
  Reply ↓
Vikas on February 28, 2021 at 2:59 PM said:

Hi Micheal ,

issue was created after migrating from Gmail to Office 365 of many duplicate emails.
After running this script. Script is working and deleting duplicate emails when duplicates are in sub-folders. How ever it is not deleting emails if duplicate emails are present in folders as same level
eg
+Inbox
++Subfolder
+TestFolder
+Archive

So if mails are duplicates between Inbox,Achive and Testfolder it is not working. Please help. Script I am running is below :

$Credentials= Get-Credential
.\Remove-DuplicateItems.ps1 -Mailbox myemail@testdomain.com -Server outlook.office365.com -Credentials $Credentials -Type Mail -DeleteMode HardDelete -Verbose -MailboxWide -Mode Full

LikeLike

Reply ↓
- Michel de Rooij on February 28, 2021 at 6:14 PM said:
  
  Fixed in 2.03 – note that I replaced MailboxWide switch, and you need to use CleanupMode Mailbox, eg
  .\Remove-DuplicateItems.ps1 -Identity frank@contoso.com .. -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
  
  LikeLike
  
  Reply ↓
Joel Cottrell on March 9, 2021 at 8:38 PM said:

Hi there. This looks like a great script. Unfortunately it appears that i’m running into an issue just running this script. This may be a simple fix, but could you point me in the right direction as to how I can get this running?

PS D:\Install\Scripts\Remove-DuplicateItems> .\Remove-DuplicateItems.ps1 -Identity test@test.com -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
D:\Install\Scripts\Remove-DuplicateItems\Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the
specified named parameters.
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Identity test@test.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Remove-DuplicateItems.ps1], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
- Michel de Rooij on March 9, 2021 at 10:35 PM said:
  
  It’s missing a paramater to dermine what mode it needs run in (and other parameters). In this case, Credentials (for Basic Auth) or one of the other params for OAuth are required. Credentials is also required now for running against local environment. Let me know if that poses problem.
  
  LikeLike
  
  Reply ↓
  - Joel Cottrell on March 10, 2021 at 2:40 PM said:
    
    Thanks. That got me closer:
    
    PS D:\Install\Scripts\Remove-DuplicateItems> $Credentials= Get-Credential
    
    cmdlet Get-Credential at command pipeline position 1
    Supply values for the following parameters:
    Credential
    PS D:\Install\Scripts\Remove-DuplicateItems> .\Remove-DuplicateItems.ps1 -Identity test@test.com -CleanupMode Mailbox -Verbose -WhatIf:$true -Report -Credentials $Credentials
    VERBOSE: Loading module D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll
    VERBOSE: Loading module from path ‘D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll’.
    Import-ModuleDLL : Problem loading module Microsoft.Exchange.WebServices: Could not load file or assembly
    ‘file:///D:\Install\Scripts\Remove-DuplicateItems\Microsoft.Exchange.WebServices.dll’ or one of its dependencies. The
    module was expected to contain an assembly manifest.
    At D:\Install\Scripts\Remove-DuplicateItems\Remove-DuplicateItems.ps1:1316 char:5
    + Import-ModuleDLL -Name ‘Microsoft.Exchange.WebServices’ -FileName …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL
    
    I have not installed the Exchange Web Services (EWS) Managed API tool, but I have copied the *.dll & *.xml files from your Github page to the same directory as the script. I also unblocked the files to no avail.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on March 13, 2021 at 1:07 PM said:
      
      Since you already unblocked the file, “D:” is a local drive? (network mapped drives are likely not trusted)
      Likely a .NET Framework dependency – the DLLs I posted on GitHub for convenience depend on .NET Framework 4.0
      
      LikeLike
      
      Reply ↓
Ryan Shelvik on March 10, 2021 at 6:21 PM said:

I’m trying out this stellar looking script for the first time. Got EWS installed, etc etc and tried to run it and got the following message:

Import-ModuleDLL : Problem loading module Microsoft.Exchange.WebServices: Could not load file or assembly
‘file:///C:\RBS\Support\Remove-DuplicateItems-master\Microsoft.Exchange.WebServices.dll’ or one of its dependencies.
Operation is not supported. (Exception from HRESULT: 0x80131515)
At C:\RBS\Support\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1:1316 char:5
+ Import-ModuleDLL -Name ‘Microsoft.Exchange.WebServices’ -FileName …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

Here’s the command I was using:

.\Remove-DuplicateItems.ps1 -Identity username@domain.com -Credentials $Credentials

Any help is appreciated.

LikeLike

Reply ↓
- Michel de Rooij on March 13, 2021 at 1:24 PM said:
  
  Likely a .NET Framework dependency missing. The DLLs from nuget package depend on .NET Framework 4.0
  
  LikeLike
  
  Reply ↓
  - Lee Howard on April 7, 2021 at 1:27 AM said:
    
    Hi Michel,
    
    We are in the process of moving to Office 365 and on our on-prem Exchange we use Import-CalendarCSV.ps1 and then Remove-DuplicateItems.ps1 for for creating and cleaning up Calendar items.
    
    Could you expand on your comment of ‘Likely a .NET Framework dependency missing. The DLLs from nuget package depend on .NET Framework 4.0’
    I can’t find out what i need to install to get this working, i have the same problem.
    
    I also can’t get impersonation working properly…it looks pretty straight forward from the guides I have found. You mention licensing in your article about set up of impersonation I can’t see anything about this anywhere else.
    
    Thanks in advance 🙂
    
    Lee
    
    LikeLike
    
    Reply ↓
  - Lee Howard on April 8, 2021 at 4:55 AM said:
    
    Hi Michel,
    Could you expand on what dependency are missing?
    Thanks,
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on April 8, 2021 at 7:43 AM said:
      
      The EWS Managed API is built for certain .NET Framework versions. Depending on your OS, this one might be installed or not. NuGet package 2.2.1.0 depends on .NET Framework 4. Also make sure that if you manually copy files over from internet locations, those might be blocked – this includes loading of DLLs as well (Explorer > Properties and see if you need to Unblock)
      
      LikeLike
      
      Reply ↓
      - Lee howard on April 8, 2021 at 8:18 AM said:
        
        Hey Michel,
        
        Thanks for that, I’ll check the DLLs, it’s when I try and install using powershell. In either Windows 10 or server 2012r2
        
        Cheers,
        
        LikeLike
Lee Howard on April 13, 2021 at 1:04 AM said:

Hi Michel,

Thanks for your help, I managed to get it working by copying the DLL’s into the script folder.
I have a problem with the script not removing duplicates, how does it check for a duplicate?
It is on the title and time?

What’s really odd is that sometimes it will remove items from the calendar if I leave it for a day….but not all items.

For testing i have a CSV file that I import twice, then I run the remove duplicates.

Thanks for any help 🙂

LikeLike

Reply ↓
Benjamin Gaulke on May 28, 2021 at 11:03 AM said:

Hey Michel,

I have a problem getting the script to work leveraging OAuth as a registered App.

After migrating to O365 most mailboxes are full of duplicates and we would like to run your script against the tenant to clean up all mailboxes, without having to have all our users type in / give up their credentials. As I understood it, we should be able to achieve this, using OAuth / registered App and running the script against a tenant-id, providing a csv-file with all users and using a client-id and the corresponding secret.

Individually, all these components seem to work fine, but when put together, as per your last example in the script, (altered according to our tenant / client-id / secret, of course), we still get the message:
Cannot bind to MsgFolderRoot: Exception when calling “Bind” with 2 Arguments: “Credentials are required to make a service request.”

What am I overlooking?

Thank you in advance for your time and response.

Benjamin

LikeLike

Reply ↓
Chad H on May 31, 2021 at 12:25 AM said:

Hi Michel

Was hoping for some assistance here, getting a generic error running the script against a single mailbox in Exchange 2013 CU 23 environment:

“Parameter set cannot be resolved using the specified named parameters.”

I am executing the following:

.\Remove-DuplicateItems.ps1 -Identity %MailBox_Name% -type All -Impersonation -DeleteMode HardDelete -Mode Full -Retain oldest -Force -Server %ServerName%

Am I missing any mandatory parameters here? Thanks in advance!

LikeLike

Reply ↓
- Michel de Rooij on May 31, 2021 at 12:42 AM said:
  
  Missing one of the sets of authentication parameters (for local operation, Credentials)
  
  LikeLike
  
  Reply ↓
cdholmes31 on June 1, 2021 at 12:08 PM said:

Thanks so much for the prompt response, I did use the credentials parameter by declaring it in a variable then executing the script, so the following are now used:

$Credentials = Get-Credential

Password has been converted to securestring also

.\Remove-DuplicateItems.ps1 -Identity %MailBox_Name% -Credentials $Credentials -type All -Impersonation -DeleteMode HardDelete -Mode Full -Retain oldest -Force -Server %ServerName%

But when using the creds parameter I look to run into an issue with converting it back to a string value it appears:

Invalid credentials provided: Cannot convert value “%cleartextPassword” to type “System.Int32”. Error: “Input string was not in a correct format.”

Thanks again!

LikeLike

Reply ↓
- Michel de Rooij on June 3, 2021 at 2:46 PM said:
  
  In PowerShell, variable names are prefixed with $, not prefixed/encapsulated in %
  
  LikeLike
  
  Reply ↓
  - cdholmes31 on June 3, 2021 at 9:53 PM said:
    
    Understood, I actually didn’t encapsulate the values in % here, that was to distinguish that actual values were used in that place, so these would be actual names that were not displayed, hope this clarifies it.
    
    LikeLike
    
    Reply ↓
Djanis on June 7, 2021 at 5:56 PM said:

Hello, I’m from France, nice script, I’m trying to get this work but I’m getting errors after launching the script. Thanks for your help

.\Remove-DuplicateItems.ps1 -Identity test@XXXXXX.fr-Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double quotation marks (“&”) to pass it as part of a string.
At C:\Users\OKIALY\Downloads\Remove-DuplicateItems.ps1:220 char:189
+ … ata-ga-click=”(Logged out) Header, go to Features”>Features <span cla …
+ ~
The '→Mobile <span cla …
+ ~
The '→</s …
+ ~

LikeLike

Reply ↓
- Djanis on June 7, 2021 at 5:58 PM said:
  
  The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double quotation marks (“&”) to pass it as part of a string.
  At C:\Users\OKIALY\Downloads\Remove-DuplicateItems.ps1:222 char:165
  + … ck Link–secondary no-underline f5 Bump-link–hover”>Mobile <span cla …
  + ~
  The '→Actions <span cla …
  + ~
  The '→</s …
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on June 8, 2021 at 6:32 AM said:
    
    Don’t know what you did, but make sure you fetched the script – not GitHub’s page, i.e.
    https://github.com/michelderooij/Remove-DuplicateItems/raw/master/Remove-DuplicateItems.ps1
    
    LikeLike
    
    Reply ↓
    - BOLUKA LUKOKI on June 8, 2021 at 11:59 AM said:
      
      Thanks for the help, it works now, with the new script
      
      LikeLike
      
      Reply ↓
Alex Rocha on July 8, 2021 at 1:28 AM said:

Hi! thank you for script, but I execute on office365 account and I have this error
“Cannot access mailbox information store for acount@xxx…. The
property ‘DisplayName’ cannot be found on this object. Verify that the property exists.
Whats is it?

LikeLike

Reply ↓
- Michel de Rooij on July 8, 2021 at 8:56 AM said:
  
  Could you run with -Verbose and see where the problem arises
  
  LikeLike
  
  Reply ↓
  - Alex Rocha on July 8, 2021 at 3:45 PM said:
    
    Thank you for feedback, i solved problem change user language to English.
    
    But my problem not was solved i did migration from Google, users duplicate mail in two or more folders, don’t same folder, because on google if message remove inbox, and check with one or more labels, this message migrate to Office365 for multiple folders
    
    Example
    Message1 archived on Google without label, migrate to Archive on Office365
    Massage2 archived on Google and check label “A” migrate to Office365 folder ‘Archive’ and Folder “A”
    
    On Google massage2 appears only one time
    
    Sorry my english!
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on July 8, 2021 at 7:45 PM said:
      
      No worries, I understood you question. We can do per-mailbox removal of duplicate entries for which the GMail scenario you described would be an excellent example. You may want to look into the CleanupMode parameter, setting it to Mailbox. To control if folders get preference (items retained there before other folders), use the PriorityFolders parameter.
      
      LikeLike
      
      Reply ↓
      - alex rocha on July 8, 2021 at 9:35 PM said:
        
        Hi! I execute cmd but don’t clean mailbox, i have this message, thank you for support
        
        PS C:\Tools> .\Remove-DuplicateItems.ps1 -Identity vallery.vier@….. -Type Mail -CleanupMode Mailbox -DeleteMode HardDelete -Mode Full -Server outlook.office365.com -Credentials (Get-Credential) -verbose
        
        cmdlet Get-Credential at command pipeline position 1
        Supply values for the following parameters:
        Credential
        VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
        VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
        VERBOSE: Using credentials ti@….
        VERBOSE: Cleanup Mode: Mailbox
        Processing mailbox vallery.vier@…. (vallery….@)
        VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
        VERBOSE: Constructing folder matching rules
        VERBOSE: Processing primary mailbox vallery….@
        VERBOSE: Collecting folders to process..
        VERBOSE: Adding folder \Archive (priority 0)
        VERBOSE: Adding folder \ARQUIVO DGS (priority 0)
        VERBOSE: Adding folder \Deleted Items (priority 0)
        VERBOSE: Adding folder \Drafts (priority 0)
        VERBOSE: Adding folder \Important (priority 0)
        VERBOSE: Adding folder \Inbox (priority 0)
        VERBOSE: Adding folder \Junk Email (priority 0)
        VERBOSE: Adding folder \Outbox (priority 0)
        VERBOSE: Adding folder \Sent Items (priority 0)
        VERBOSE: Adding folder \Starred (priority 0)
        VERBOSE: Found 10 folders that match search criteria
        VERBOSE: Processing folder \Junk Email
        VERBOSE: Processing folder \Inbox
        VERBOSE: Processing folder \Outbox
        VERBOSE: Processing folder \Starred
        VERBOSE: Processing folder \Sent Items
        VERBOSE: Processing folder \ARQUIVO DGS
        VERBOSE: Processing folder \Archive
        VERBOSE: Removing 1 items from \Archive
        VERBOSE: Processing folder \Deleted Items
        VERBOSE: Processing folder \Important
        VERBOSE: Processing folder \Drafts
        VERBOSE: Cleaning unique list (finished mailbox)
        17536 items processed and 1 removed in 00:05:04 – average 3455 items/min
        WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified folder
        could not be found in the store.”
        VERBOSE: Processing vallery….. finished
        
        LikeLike
      - Michel de Rooij on July 8, 2021 at 9:58 PM said:
        
        The default mode uses an attribute which Exchange uniquely assigns to every item. Since you’re coming from GMail, that might not work, and you may need to use attribute-level comparison (-Mode Full). Note that before you perform a run, you can report on duplicate items using -Report -WhatIf:$True
        
        LikeLike
      - alex rocha on July 8, 2021 at 11:10 PM said:
        
        Hello! I run PS cmd with mode full, Do i need to do other conf at my line?
        
        .\Remove-DuplicateItems.ps1 -Identity vallery.vier@contoso.com -Type Mail -CleanupMode Mailbox -DeleteMode HardDelete -Mode Full -Server outlook.office365.com -Credentials (Get-Credential) -verbose
        
        Thanks!
        
        LikeLike
      - Michel de Rooij on July 9, 2021 at 7:49 AM said:
        
        Looks good to go. If duplicates are still not detected, one of the attributes may differ between folderA and folderB. There is another option “Body” comparing only body messages, but there might be legal implications. Last resort is to tweak the comparison.
        
        LikeLike
      - Alex Rocha on July 10, 2021 at 3:23 AM said:
        
        Thank you for you script, i run with option Nosize and detect duplicates!!
        
        LikeLike
Karim Aly on July 15, 2021 at 12:49 PM said:

What if the duplicated messages are in the deleted items folder? can you tell me how to make him scan only the deleted items folder?

LikeLike

Reply ↓
- Michel de Rooij on July 15, 2021 at 7:57 PM said:
  
  Specify IncludeFolders parameter to limit operation to certain folders or folder structures matching patterns. As mentioned in the help for IncludeFolders, you can specify well-known folders such as Deleted Items using ‘##’, e.g. ‘#DeletedItems#’. Deleting items from deleting items doesn’t make sense, but DeleteMode by default moves them to the dumpster (Recoverable Items), so you don’t need to change that.
  Command would then be something like
  .\Remove-DuplicateItems.ps1 -Identity bob@contoso.com -Impersonation -IncludeFolders ‘#DeletedItems#’
  You can always specify -WhatIf:$true to see what it would do, or add -Verbose to see additional information.
  
  LikeLike
  
  Reply ↓
  - Karim Aly on August 3, 2021 at 8:56 AM said:
    
    I find this absolutely useful, but I am still unable to imagine what is the whole command line would look like.
    And another thing, how i can install this module “Remove-DuplicateItems.ps1”?
    I am lost 😦
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 3, 2021 at 8:59 AM said:
      
      1) It’s not a module, it’s a script. Download and store it on the machine you want to run it on.
      2) It depends on some modules – install these modules; consult https://eightwone.com/2020/10/05/ews-webservices-managed-api/
      3) Check Get-Help .\Remove-Duplicates -Examples to see some examples on usage.
      
      LikeLike
      
      Reply ↓
Vit Vet on September 22, 2021 at 7:28 PM said:

I am having issues with a 401 Unauthorized error.

PS C:\Users\users\Downloads\Remove-DuplicateItems-1.84\Remove-DuplicateItems-1.84> .\Remove-DuplicateItems.ps1 -Mailbox user@domain.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders Inbox -MailboxOnly -Verbose
VERBOSE: Loaded EWS Managed API v2.2.1.0
VERBOSE: Set to trust all certificates
VERBOSE: Using credentials user@domain.com
VERBOSE: Processing items of type All, delete mode is SoftDelete
Processing mailbox user@domain.com (user@domain.com)
VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
WARNING: Cannot bind to MsgFolderRoot – skipping. Error: Exception calling “Bind” with “2” argument(s): “The request
failed. The remote server returned an error: (401) Unauthorized.”
VERBOSE: Processing user@domain.com finished

LikeLike

Reply ↓
- Michel de Rooij on September 23, 2021 at 8:29 PM said:
  
  – Are credentials valid and do credentials provide access to this mailbox – full access (when granted properly, eg using InheritanceType on top of information store) or impersonation (yet impersonation not specified, so guess not)
  – Is Basic Authentication allowed?
  – Is Basic Authentication allowed for the condition, e.g. location and user (e.g. no Conditional Access blocking things)
  
  LikeLike
  
  Reply ↓
bobebukk on September 23, 2021 at 6:05 PM said:

failing at the first hurdle, trying both ISE and exchange online PowerShell.
Have tried different parameters and my mailbox has full access, although reading this error – this isn’t permissions.

Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
bobebuk on September 23, 2021 at 6:12 PM said:

receiving the error, using both ise and powershell for exchange online.

s.ps1 -Identity fred@contoso.com -Type contacts -Server outlook.office365.com -Verbose
D:\Scripts\RBetts\Remove-DuplicateItems.ps1 : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
- Michel de Rooij on September 23, 2021 at 8:35 PM said:
  
  You need to provide one of the authentication methods:
  * Basic using -Credential
  * Modern Auth using TenantId, ClientId and one of the following: Secret, CertificateThumbprint, CertificateFile(+CertificatePassword as secure string)
  
  LikeLike
  
  Reply ↓
Rob on September 24, 2021 at 3:46 PM said:

still getting the same error 😦

.\Remove-DuplicateItems.ps1 -Identity fred.blogs@contosso.com -Tena
ntId xyzabc-abcd-abcde -WhatIf -contacts

Parameter set cannot be resolved using the
specified named parameters.
At line:1 char:1

LikeLike

Reply ↓
- Michel de Rooij on September 24, 2021 at 11:03 PM said:
  
  Seem to be missing one or more parameters for one of the authentication methods, eg
  * Basic using -Credential
  * Modern Auth using TenantId, ClientId and one of the following: Secret, CertificateThumbprint, CertificateFile(+CertificatePassword as secure string)
  
  LikeLike
  
  Reply ↓
Kristaps on October 4, 2021 at 1:17 PM said:

Hi!

I get this error when running against o365

Write-Verbose : A parameter cannot be found that matches parameter name ‘f’.
At C:\Users\xxx\Desktop\Remove-DuplicateItems-master\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1:1438 char:5
2
+ Write-Verbose ‘Using EWS endpoint {0}’ -f $EwsService.Url
+ ~~
+ CategoryInfo : InvalidArgument: (:) [Write-Verbose], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.WriteVerboseCommand

LikeLike

Reply ↓
- Michel de Rooij on October 4, 2021 at 1:21 PM said:
  
  Thanks for catching that – somehow parenthesis got dropped. Get 2.06 or replace line with
  Write-Verbose (‘Using EWS endpoint {0}’ -f $EwsService.Url)
  
  LikeLike
  
  Reply ↓
  - Kristaps on October 4, 2021 at 2:06 PM said:
    
    Thanks for the quick reply!
    
    FYI – if anyone gets this error :
    Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified object was not
    found in the store., The process failed to get the correct properties.”
    WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The specified folder
    could not be found in the store.”
    
    this means that user which credentials you used to connect doesn’t have permissions to read the mailbox. in my case I used this script against a shared mailbox and used global admin account (no MFA) to connect and it did not work until I gave the global admin read ann manage rights for the shared mailbox.
    
    LikeLike
    
    Reply ↓
Daniel on October 22, 2021 at 9:54 AM said:

Hi,

I am getting following error. Any Ideas?
[PS] C:\temp\Remove-DuplicateItems-master\Remove-DuplicateItems-master>.\Remove-DuplicateItems.ps1 -Identity -Credentials $Credent
ials -Type Calendar -Mode Quick -MailboxOnly -Server outlook.office365.com -Verbose
AUSFÜHRLICH: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
AUSFÜHRLICH: Module Microsoft.Identity.Client v4.25.0.0 already loaded
AUSFÜHRLICH: Using credentials
AUSFÜHRLICH: Cleanup Mode: Folder
Processing mailbox ()
AUSFÜHRLICH: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
AUSFÜHRLICH: Constructing folder matching rules
AUSFÜHRLICH: Processing primary mailbox
AUSFÜHRLICH: Collecting folders to process..
C:\temp\Remove-DuplicateItems-master\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for
: Ausnahme beim Festlegen von “PropertySet”: “Der Wert “Microsoft.Exchange.WebServices.Data.PropertySet” vom Typ
“Microsoft.Exchange.WebServices.Data.PropertySet” kann nicht in den Typ “Microsoft.Exchange.WebServices.Data.PropertySet” konvertiert werden.”
In Zeile:1 Zeichen:1
+ .\Remove-DuplicateItems.ps1 -Identity -Credent …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

LikeLike

Reply ↓
Phillip on January 10, 2022 at 7:41 PM said:

Hello,

Looking over your script to see if this is something we can implement. During our testing with it, we are finding that it does identify duplicate items for the mailbox, but is not moving forward with deleting the items? We are noting seeing much in Verbos output that would lead us to any conclusions so thought I would ping you and pick your brain.

Hybrid M365

Thank you again!

LikeLike

Reply ↓
- Michel de Rooij on January 11, 2022 at 3:13 PM said:
  
  By default is identifies duplicates based on PR_SEARCH_KEY. Depending on how you got the duplicates in, that may not work and you may need attribute-level comparison (with or without size using -NoSize). Also make sure proper permissions have been set up, impersonation is preferred.
  
  LikeLike
  
  Reply ↓
Naushad on January 11, 2022 at 2:58 PM said:

Parameter set cannot be resolved using the specified named parameters.
+ CategoryInfo : InvalidArgument: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : AmbiguousParameterSet

LikeLike

Reply ↓
- Michel de Rooij on January 11, 2022 at 3:12 PM said:
  
  You need to provide the script the right parameters so it knows which authentication method to use (and what parameters to expect, as it’s now ambigious):
  1) BasicAuth : -Credentials
  2) Modern Auth:
  2a) -CertificateThumbprint -TenantId -ClientId
  2b) -Secret -TenantId -ClientId
  2c) -CertificateFile [-CertificatePassword ] -TenantId -ClientId
  
  LikeLike
  
  Reply ↓
jedivader20 on January 20, 2022 at 7:55 AM said:

I was having issues getting this to authenticate for the longest time, always getting 401 Unauthorised. I even setup a policy to allow EWS basic auth for the account that I was trying to use for impersonation. As it turned out, it was disabled at a tenant level, and per-user policies weren’t going to cut it.

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-june-2021-update/ba-p/2454827

This fixed the issue permanently for me.

LikeLike

Reply ↓
Anthony on February 16, 2022 at 5:36 AM said:

Hi Not sure if i have missed something but i followed the details of this post and the one for the EWS.Websrvices.Managed.Api post but i am getting the following error when attempting to run the script:
Exception calling “LoadFile” with “1” argument(s): “An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions
of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please
enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.”
At C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1:343 char:13
+ [void][Reflection.Assembly]::LoadFile( “$EWSDLLPath\$EWSD …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : NotSupportedException

Load-EWSManagedAPIDLL : Problem loading Microsoft.Exchange.WebServices.dll
At C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1:994 char:5
+ Load-EWSManagedAPIDLL
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Load-EWSManagedAPIDLL

Is there something im missing here?

LikeLike

Reply ↓
- Anthony on February 16, 2022 at 5:46 AM said:
  
  Ok so I have found what i missed.
  There is the need to Unblock not just the DLL and XML files but also the PS1 file as well.
  This has allowed the script to run.
  
  I am now getting another error:
  C:\users\\Documents\PS\Remove-DuplicateItems-1.84\Remove-DuplicateItems.ps1 : Autodiscover failed, error: Exception calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover
  service couldn’t be located.”
  At line:1 char:1
  + .\Remove-DuplicateItems.ps1 -Identity user@domain.com -Typ …
  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
  + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
  
  Ill be looking furhter into this myself and if i find the answer to this also i will add this to my reply.
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on February 16, 2022 at 10:01 AM said:
    
    Be sure there is an autodiscover record published and its location accessible for @domain.com. To manually override, use -Server, e.g. -Server outlook.office365.com for Office 365.
    
    LikeLike
    
    Reply ↓
David Roberts on March 1, 2022 at 10:44 AM said:

This script looks incredible, so first off, thank you for the dedicated work on it over the years.

Unfortunately I can’t get this to authenticate with Exchange Online. I’m not 100% sure how the App Registration in AAD interacts with EXOL. I’m using an existing App Registration created for BitTitan to migrate emails into our tenant, so it should already have impersonation permissions as BitTitan can write into any mailbox in our tenant and we use the impersonation option within BitTitan. The app has Office 365 Exchange Online – EWS.AccessAsUser.All delegated permission, as well as Microsoft Graph – User.Read. I’ve added a secret to the app to use with this script. I’m running the following:

$secret = ConvertTo-SecureString “XXXXXXXXXXX” -AsPlainText -Force

.\Remove-DuplicateItems.ps1 -Identity user@domain.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Retain Oldest -IncludeFolders ‘#Inbox#\*’ -Server outlook.office365.com -Secret $secret -TenantId -ClientId

I get the following result:

WARNING: Cannot bind to Inbox: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”
WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”
WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

We have Modern Authentication enabled in our tenant and Basic Authentication turned off for all options. I can see the app authentication in the AAD sign-in logs as being successful, so it feels like that part of the authentication process is working, but not something at the EXOL level maybe? I don’t know.

Any pointers would be greatly appreciated! Thanks in advance.

LikeLike

Reply ↓
- Michel de Rooij on March 6, 2022 at 8:58 AM said:
  
  I assume you also specified the proper ‘TenantId’ and ‘ClientId’ IDs with those parameters (not specified)?
  Since the script is not from a verified publisher, we need app permissions, not delegated permissions. Also, the permission I usually assign is Office 365 Exchange Online > full_access_as_app; if orgs worry about this blanket permission, they can restrict it to certain mailboxes using application access policies.
  
  LikeLike
  
  Reply ↓
  - David Roberts on March 7, 2022 at 12:50 PM said:
    
    Thanks for the reply Michel.
    
    After a bit of research I came across that exact permission, which I applied to the Azure AD App Registration and then it all started working.
    
    Yes I did indeed have the Tenant & Client IDs specified, I just removed them from the command.
    
    Thanks again for the script, it was just what I needed!
    
    LikeLike
    
    Reply ↓
    - djansen1987 on September 8, 2022 at 7:50 PM said:
      
      Would you like to share that ?
      
      LikeLike
      
      Reply ↓
Glenn on March 28, 2022 at 9:46 PM said:

Great script. Question: I’m testing it now and it seems to remove the first of 2 emails matched (or, the first one displayed in Outlook, at any rate). What would need to be changed to make it remove the 2nd one instead? I realize this is a fringe case, but it is useful if you have to import emails exported from IMAP, but you already imported some of them via ActiveSync, and some of the headers are different, apparently, so internal duplicate detection in Outlook doesn’t work… *sigh*. Seems the first displayed email (from the ActiveSync export) retains some of the group name expansion and short names, whereas the 2nd one (from the IMAP export) does not, so it would make more sense to keep the first one in my case just for the user experience.

LikeLike

Reply ↓
- Michel de Rooij on March 31, 2022 at 8:49 AM said:
  
  Defining a criterium based on headers is a challenge, apart from the potential performance as you have to inspect each item.
  Only thing you can use for for determining what gets deleted is Retain, where you can specify Oldest or Newest. Depending on the way on you those duplicates got in, you may want to use Mode Full to have attribute-level comparison instead of PidTagSearchKey, which is an attribute set by Exchange when the item gets stored, and is preserved when copying, exporting/importing etc.
  
  LikeLike
  
  Reply ↓
Andrea Gallina on April 15, 2022 at 1:14 PM said:

I receive this error when i launch the script…i don’t know how to solve.. Can you help me?

PS C:\script> .\Remove-DuplicateItems.ps1 -Mailbox user@contoso.com -Server outlook.office365.com -Credentials $Credentials -Type Mail -DeleteMode HardDelete -CleanupMode Mailbox -Verbose -WhatIf:$true -Report
DETTAGLIATO: Loading module C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll
DETTAGLIATO: Caricamento del modulo dal percorso ‘C:\Program Files\PackageManagement\NuGet\Packages\Exchange.WebServices.Managed.Api.2.2.1.2\lib\net35\Microsoft.Exchange.WebServices.dll’.
DETTAGLIATO: Module Microsoft.Exchange.WebServices v2.2.1.0 loaded
DETTAGLIATO: Loading module C:\Program Files\PackageManagement\NuGet\Packages\Microsoft.Identity.Client.4.43.0\lib\monoandroid10.0\Microsoft.Identity.Client.dll
DETTAGLIATO: Caricamento del modulo dal percorso ‘C:\Program Files\PackageManagement\NuGet\Packages\Microsoft.Identity.Client.4.43.0\lib\monoandroid10.0\Microsoft.Identity.Client.dll’.
Import-ModuleDLL : Problem loading module Microsoft.Identity.Client: Non è stato possibile caricare il file o l’assembly ‘Mono.Android, Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065’ o
una delle relative dipendenze. Impossibile trovare il file specificato.
In C:\script\Remove-DuplicateItems.ps1:1379 car:5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

LikeLike

Reply ↓
- Michel de Rooij on April 15, 2022 at 4:13 PM said:
  
  Depending on your security settings, you may need to run PowerShell with Elevated Permissions in order to be able to load the DLL modules.
  
  LikeLike
  
  Reply ↓
SteveJ on April 30, 2022 at 1:48 PM said:

Great script 🙂 However I’m having an issue getting it to process Inbox subfolders.

I use the following parameter -IncludeFolders ‘#INBOX#\*’ and the script process the Inbox OK but ignores the dozens of subfolders. The mailbox is hosted in Exchange Online BTW.

Am I missing something?

LikeLike

Reply ↓
- SteveJ on April 30, 2022 at 1:52 PM said:
  
  I just found that if I create a new subfolder, the script finds it. But it isn’t finding the dozens of existing folders (that were imported from a PST file if that makes any difference).
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on April 30, 2022 at 7:04 PM said:
    
    Are you using Impersonation? If not, does the account provided have sufficient permissions on those imported subfolders.
    Also, have you specified -Type Mail by any chance? Could be the Folder type of those imported folders are not Mail (IPF.Note)
    
    LikeLike
    
    Reply ↓
Bernard Koops on May 2, 2022 at 12:31 PM said:

Great script, doing what’s advertised, love it. Ran the script for the shitload of extra data that came to be due to the old registry setting to copy a sent email in the shared mailbox sent items, which as well was done by EAC rule… just love (not) what was done by my predecessors without documenting their actions and hiding it in obscure gpo’s…

I made 1 (cosmetic) adjustment – line 1155 had ‘Processed folder {0} or {1}’ which should of course be ‘Processed folder {0} of {1}’ 😉

LikeLike

Reply ↓
- Michel de Rooij on May 3, 2022 at 10:19 AM said:
  
  Thanks for catching that.
  
  LikeLike
  
  Reply ↓
Ricardo Marinho | Webideia on May 18, 2022 at 12:51 AM said:

Hi,
Script runs but no duplicates are found/deleted.

Anyone experienced same behaviour?

LikeLike

Reply ↓
- Michel de Rooij on May 19, 2022 at 4:09 PM said:
  
  Perhaps it is not finding what you expected it to find. There could be many reasons, usually because properties not directly visible that could identify duplicates are not identical. Also, make sure you have proper permissions on mailbox and folder, preferably using Impersonation so you do not have that issue.
  
  LikeLike
  
  Reply ↓
sathish on May 24, 2022 at 2:42 AM said:

Some reason the script is throwing below error in Ms365 online mailbox

PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Mailbox agnoc -Type All -Impersonation -DeleteMode HardDelete -Mode Full -Verbose
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:246 char:1
+ [cmdletbinding(
+ ~~~~~~~~~~~~~~~
Unexpected attribute ‘cmdletbinding’.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:251 char:1
+ param(
+ ~~~~~
Unexpected token ‘param’ in expression or statement.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:285 char:20
+ [string]$Type= ‘All’,
+ ~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:302 char:22
+ [string]$Retain= ‘Newest’,
+ ~~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:351 char:26
+ [string]$DeleteMode= ‘SoftDelete’,
+ ~~~~~~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
At C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1:368 char:20
+ [string]$Mode= ‘Quick’,
+ ~~~~~~~
The assignment expression is not valid. The input to an assignment operator must be an object that is able to accept assignments, such as a variable or a property.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : UnexpectedAttribute

LikeLike

Reply ↓
- Michel de Rooij on May 24, 2022 at 6:55 AM said:
  
  Did you download the script, or copy/paste between GitHub and your environment? If the latter, likely issue with that (turning quotes into smart quotes etc.)
  
  LikeLike
  
  Reply ↓
  - sathish on May 25, 2022 at 3:27 AM said:
    
    I have downloaded from git and getting below error. please suggest
    
    PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Identity agnoc -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials $agccred -Retain Oldest
    
    Processing mailbox agnoc (agnoc)
    
    *********************************************************************************************
    
    C:\Users\support\Desktop\RemoveDuplicateiteminmailbox\Remove-DuplicateItems.ps1 : Autodiscover failed: Exception
    calling “AutodiscoverUrl” with “2” argument(s): “The Autodiscover service couldn’t be located.”
    At line:1 char:1
    + .\Remove-DuplicateItems.ps1 -Identity -Type Mail – …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1
    
    *********************************************************************************************
    
    PS C:\Users\support\Desktop\RemoveDuplicateiteminmailbox> .\Remove-DuplicateItems.ps1 -Identity agnoc -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials $agccred -Retain Oldest -Server outlook.office365.com
    
    Processing mailbox agnoc (agnoc)
    
    *********************************************************************************************
    WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The remote
    server returned an error: (401) Unauthorized.”
    WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. The
    remote server returned an error: (401) Unauthorized.”
    
    *********************************************************************************************
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 25, 2022 at 8:32 AM said:
      
      I see two messages:
      – “The Autodiscover service couldn’t be located”: Either configure Autodiscover properly, or use Server to point to the EWS endpoint.
      – 401 Unauthorized is a permissions issue – either you have none, or BasicAuth is blocked (you specified Credentials, so using BasicAuth)
      
      LikeLike
      
      Reply ↓
D.andre on May 25, 2022 at 12:36 PM said:

Hello.
I have 2 issues with the script :
The first one when i execute the script for the first time. I have this.

COMMENTAIRES : Module Microsoft.Identity.Client v4.29.0.0 loaded
Import-ModuleDLL : Problem initializing test-object from module Microsoft.Identity.Client: Constructeur introuvable.
Impossible de trouver un constructeur approprié pour le type Microsoft.Identity.Client.MsalClientException.
Au caractère D:\Temp\Remove-DuplicateItems.ps1:1381 : 5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

And after a retry, i have this :

D:\Temp\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for XXXX@XXXXX: Exception lors de la définition de «PropertySet»: «Impossible de convertir la valeur «
Microsoft.Exchange.WebServices.Data.PropertySet» du type «Microsoft.Exchange.WebServices.Data.PropertySet» en type «Microsoft.Exchange.WebServices.Data.PropertySet».»
Au caractère Ligne:1 : 1
+ .\Remove-DuplicateItems.ps1 -Identity u007259 -Retain Newest -DeleteM …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

I execute the script with an account who have fullaccess on the mailbox and I give the role “ApplicatioImpersonation” on my user.

Have you any clues ?

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2022 at 3:21 PM said:
  
  FullAccess is not Impersonation – if you have Full Access, don’t specify Impersonation and make sure you have proper permissions on Top of Information store and folder-level as well (InheritanceType). When specifying Impersonation, be sure to grant these permissions properly and give it some minutes. Impersonation is preferred due to it less potential for issues or blockage by owner.
  
  LikeLike
  
  Reply ↓
  - d.andre on May 31, 2022 at 9:32 AM said:
    
    I have configure the impersonation for the user who launch the script and i have the same error.
    Impersonisation ApplicationImpersonation admExchange User Direct admExchange
    
    I try without the Impersonation argument after i give to my user full accesright with inheritancetype at all with this command and i have the same error.
    
    Add-MailboxPermission -Identity u007259 -User admexchange -AccessRights FullAccess -InheritanceType all
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on May 31, 2022 at 10:50 AM said:
      
      To be sure, is this Exchange on-premises and if so what version?
      
      LikeLike
      
      Reply ↓
      - D.andre on May 31, 2022 at 10:53 AM said:
        
        Yes it’s an exhange 2016 on-premise
        
        LikeLike
D.andre on May 25, 2022 at 12:59 PM said:

Hello.

I have 2 error when i use the script with this command
.\Remove-DuplicateItems.ps1 -Identity Username -Retain Newest -DeleteMode SoftDelete -Type Calendar -UseDefaultCredentials -server FQDN name server -Mode Full -Verbose

The first one when i lauch the script for the first time

COMMENTAIRES : Module Microsoft.Identity.Client v4.29.0.0 loaded
Import-ModuleDLL : Problem initializing test-object from module Microsoft.Identity.Client: Constructeur introuvable.
Impossible de trouver un constructeur approprié pour le type Microsoft.Identity.Client.MsalClientException.
Au caractère D:\Temp\Remove-DuplicateItems.ps1:1381 : 5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

The second one is this. I have full access right on the mailbox and the user have the ” ApplicationImpersonation” role.

COMMENTAIRES : Collecting folders to process..
D:\Temp\Remove-DuplicateItems.ps1 : Cannot access mailbox information store for D.ANDRE@chru-nancy.fr: Exception lors
de la définition de «PropertySet»: «Impossible de convertir la valeur «Microsoft.Exchange.WebServices.Data.PropertySet
» du type «Microsoft.Exchange.WebServices.Data.PropertySet» en type «Microsoft.Exchange.WebServices.Data.PropertySet».»
Au caractère Ligne:1 : 1
+ .\Remove-DuplicateItems.ps1 -Identity u007259 -Retain Newest -DeleteM …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Remove-DuplicateItems.ps1

Have you an idea of what happen.

Best regards

LikeLike

Reply ↓
- Michel de Rooij on May 25, 2022 at 3:25 PM said:
  
  They sometimes change classes in the module, which I pick one to see if loading and initializing goes OK. Made a note to check/revise.
  
  LikeLike
  
  Reply ↓
  - Eric Hester on August 22, 2022 at 7:36 PM said:
    
    I am also getting this error. Not sure I understand the solution?
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on August 22, 2022 at 10:17 PM said:
      
      The solution was mine to make- has been implemented. If you still experience this issue, let me know.
      
      LikeLike
      
      Reply ↓
Jonathan on June 21, 2022 at 10:39 PM said:

I cannot get the Microsoft.Identity.Client.dll module DLL to load. I know there are .Net 4 dependencies and I’m running Windows 10 and have .Net 4.8 installed – so I should be covered on dependencies right? Also, I downloaded and moved the Microsoft.Identity.Client.dll and the Microsoft.Identity.Client.xml into the script directory but still, same error. Any ideas? Thank you

VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
VERBOSE: Loading module C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll
VERBOSE: Loading module from path ‘C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll’.
Import-ModuleDLL : Problem loading module Microsoft.Identity.Client: Could not load file or assembly
‘file:///C:\Users\jharris\Downloads\removedups\Microsoft.Identity.Client.dll’ or one of its dependencies. The module
was expected to contain an assembly manifest.
At C:\Users\jharris\Downloads\removedups\Remove-DuplicateItems.ps1:1481 char:5
+ Import-ModuleDLL -Name ‘Microsoft.Identity.Client’ -FileName ‘Mic …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-ModuleDLL

LikeLike

Reply ↓
- Michel de Rooij on June 22, 2022 at 6:26 AM said:
  
  Odd, I have no issues with the DLLs that I stored with the script on GitHub on w10.
  After downloading the DLL and manifest XML, you did unblock those files?
  
  LikeLike
  
  Reply ↓
Jonathan on June 22, 2022 at 4:42 AM said:

Amazing script Michel. The script is running on a mailbox in my Office 365 tenant, but it seems to be skipping the Inbox and a subfolder of the inbox with the following errors:

VERBOSE: Processing folder \Inbox\ECI
WARNING: Error performing operation FindItems without Search options in ECI. Error Exception calling “FindItems” with
“1” argument(s): “The property Hashtags is valid only for Exchange Exchange2015 or later versions.”
VERBOSE: Cleaning unique list (finished folder)
VERBOSE: Processing folder \Inbox
WARNING: Error performing operation FindItems without Search options in Inbox. Error Exception calling “FindItems” with
“1” argument(s): “The property Hashtags is valid only for Exchange Exchange2015 or later versions.”
VERBOSE: Cleaning unique list (finished folder)

I did some research and found this article talking about restricting the FirstClassProperties call to only the required properties (and consequently excluding hashtags). This is a bit over my head but maybe this line in your script is returning an incompatible property in my situation?

$ItemView.PropertySet= New-Object Microsoft.Exchange.WebServices.Data.PropertySet( [Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)

https://stackoverflow.com/questions/52573779/the-property-hashtags-is-valid-only-for-exchange-exchange2015-or-later-versions

LikeLike

Reply ↓
- Michel de Rooij on June 22, 2022 at 8:25 AM said:
  
  First time I see this one and cannot repro. I do target a specific product level [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1. Could you replace that entry with [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2015 in your copy of the script and let me know if this solves it for you? Might have to make a modification.
  
  LikeLike
  
  Reply ↓
  - Harris, Jonathan on June 22, 2022 at 2:41 PM said:
    
    Replacing the 2 entries with 2015 fixed it. It processed both Inbox folders this time – no errors. I really appreciate your quick response and your help!
    
    Also, I wrote a comment before this about not being able to get the script to run. Obviously, I got it working finally. I had to re-download the Identity DLL from GitHub. No idea why the first one didn’t work even after unblocking it.
    
    Really appreciate your help,
    
    JonathanHarris Assistant Vice President, Information Technology Extell Financial Services Inc 9911 Shelbyville Road, Louisville, KY40223 t: +1646-790-4047|f: +1212-712-6100| c: +1347-450-3179 jharris@extell.com|www.extell.com This email is confidential and may contain information that is privileged, attorney work product and/or exempt from disclosure under applicable law and should be read or retained only by the intended recipient. Any review, reliance or distribution by others or forwarding without express permission of the sender is strictly prohibited. If you received this transmission in error, please immediately contact the sender and delete all copies including all attachments.
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on June 23, 2022 at 12:29 PM said:
      
      Thanks for the update, much appreciated.
      
      LikeLike
      
      Reply ↓
Michael Kenny on July 18, 2022 at 4:41 PM said:

I am clearly missing something but when I run the script in PS it thinks for a second and then goes back to the PS prompt without doing anything. I am running Office 365 and able to login to exchange online, etc. No errors, but no success either.

LikeLike

Reply ↓
- Michel de Rooij on July 18, 2022 at 7:05 PM said:
  
  Look at the usage examples, and use -Verbose switch to let is present more output on what’s going on.
  
  LikeLike
  
  Reply ↓
  - Michael on July 18, 2022 at 8:49 PM said:
    
    It was an issue with Powershell 7. Moved to another PC with PS 5 on it and everything worked well. Totally an issue between my chair and my keyboard….
    
    LikeLike
    
    Reply ↓
Roger Vaughan on July 26, 2022 at 1:25 PM said:

Hi Michel,
Trying to change from BasicAuth (which I have used for years) to modern. I am running this script : .\Remove-DuplicateItems.ps1 -Mailbox xxx@xxx.com -TenantId xxx -ClientId xxx -Secret (ConvertTo-SecureString “xxx” -AsPlainText -Force) -Server outlook.office365.com -Mode Body -DeleteMode SoftDelete -MailboxOnly -Verbose.
I am getting this error:
WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “Credentials are required to make a service request.”
I believe I have set up the APP ok with redirection URI and given full_access_as_app permissions to Office 365 Exchange Online.
Any assistance would be great.

LikeLike

Reply ↓
- djansen1987 on September 8, 2022 at 7:51 PM said:
  
  Tried the same as you but with no success either. I also tried the application permission “EWS.AccessAsUser.All”. I do not have a redirect URI (kept it empty) and wondering what you gave up.
  
  My command:
  
  $secret = read-host -AsSecureString -Prompt “secure value”
  
  .\Remove-DuplicateItems.ps1 -report -whatif -Mailbox “user@domaininO365.ext” -Server outlook.office365.com -Verbose -type mail -TenantId “1234-5678-9123456-12435-13245jh32145fg324” -ClientId “dfg45ewrdsff-sdfghgf657-234rfd-gr234-345gfd435dfdfge” -Secret $secret -MailboxOnly
  
  VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
  VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
  VERBOSE: Will use provided secret to authenticate
  VERBOSE: Authentication token acquired
  VERBOSE: Cleanup Mode: Folder
  Processing mailbox user@domaininO365.ext (“user@domaininO365.ext”)
  VERBOSE: Using user@domaininO365.ext for impersonation
  VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
  VERBOSE: Constructing folder matching rules
  WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “Credentials are required to make a service request.”
  
  LikeLike
  
  Reply ↓
  - Roger on November 1, 2022 at 6:42 PM said:
    
    Ok, I finally got it working.
    It needed the Application ID to be the “ClientID”, and then I created a new Secret. It’s not the Secret ID in Azure you need to then use, it’s the Value (which makes sense if you think of it).
    As per your script, i read user input (using Read-Host) to convert the secret value to be a secure string (used $string as the variable), then used -Secret $secret in the command.
    
    LikeLike
    
    Reply ↓
Tim on October 30, 2022 at 11:45 AM said:

Hi Michel, basic question for you – if we export a PST with purview there is a deduplication option, which removes dupes. Any drawbacks of this method vs your script?

LikeLike

Reply ↓
- Michel de Rooij on October 31, 2022 at 10:34 AM said:
  
  Purview export dedupe has a fixed method to identify duplicates – the script is bit more flexible (and customizable when needed). See the following article on the recipe, but also make sure you read the notes on downside (“Limitations of the de-duplication algorithm”):
  https://learn.microsoft.com/en-us/microsoft-365/compliance/de-duplication-in-ediscovery-search-results?view=o365-worldwide#how-duplicate-messages-are-identified
  
  LikeLike
  
  Reply ↓
Michael Bayless on November 21, 2022 at 11:45 PM said:

Really LOVE this script. I appreciate all the work and detail you put into this it’s a great utility.

I had things working for awhile and all of a sudden it stopped working. I get either a 401 Unauthorized or a 403 Forbidden depending on if I use Get-Credentials (401) or Certificate based authentication(403). I have verified that the user connecting has appropriate permissions: AccessRights(FullAccess) InheritanceType(All). User is also global admin. I do have MFA turned on and it never prompts for this, not sure why. It is connecting to M365.

Here is the command with User Auth: .\Remove-DuplicateItems.ps1 -Identity -Type Mail -Mode Full -Credentials (Get-Credential) -Verbose -Report >> c:\Util\USERDuplicate.txt

Here is the command with Certificate Auth: .\Remove-DuplicateItems.ps1 -Identity -Type Mail -Mode Full -Verbose -Report >> c:\Util\DanyelleDuplicate.txt –TenantId ‘<Directory (tenant) ID from AAD' -ClientId 'Application (client) ID from AAZ' -CertificateThumbprint 'Thumbprint from certificate installed in personal store for user (and computer)'

As mentioned I had it working before with just logging in with the administrator. Not sure what has changed but it's driving me nuts.

Any help would be greatly appreciated!

LikeLike

Reply ↓
- Michel de Rooij on November 22, 2022 at 12:36 AM said:
  
  Likely, modern authentication is required to connect. Therefor, passing credentials doesn’t work any longer. The app which you refer to using ClientId, is it properly registered, is the certificate still valid to authenticate and not expire, and does the app have the necessary permissions (Exchange.ManageAsApp)? Also, are there no conditional access rules effective which potentially could block your authentication attempt? (look in Azure Sign-Ins and filter on your ClientId (Application ID).
  
  LikeLike
  
  Reply ↓
  - Michael Bayless on November 23, 2022 at 6:03 PM said:
    
    First, I can’t tell you how much I appreciate your help!
    – Modern Authentication — this is correct with Microsoft’s move to requiring modern authentication I’m thinking this will be come much more of an issue.
    
    -App — Everything was registered properly. I created the app / certificate login stuff specifically for this so all certs were good. I followed this and it worked almost perfectly: https://learn.microsoft.com/en-us/exchange/client-developer/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth
    
    -Permissions — So this is where my issue was. I verified that the Exchange.ManageAsApp was there and Admin Consent was granted. However, I had to go just a little further. I followed the following article and added additional permissions (pretty much all of them so I can manage all aspects of the exchange with the app logon). https://www.michev.info/Blog/Post/3180/exchange-api-permissions-missing
    
    -WOHOO it’s now working again.
    
    LikeLike
    
    Reply ↓
David McFarland on December 14, 2022 at 11:39 PM said:

First off, I wanted to say that this script is amazing. I am really hoping it will fix a duplication issue we are having. When I run the script against a mailbox, it looks like its working fine, successfully removes a the first few thousand emails and then starts giving me this error. WARNING: Error performing operation RemoveItems with . Error: Exception calling “DeleteItems” with “5” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.” I am guessing this is a timeout from the server, but I was hoping you might be able to clarify what I am seeing. Using TenantID and Client with Secret to Auth.

LikeLike

Reply ↓
- Michel de Rooij on December 15, 2022 at 9:13 AM said:
  
  No idea, it handles throttling but not disconnects etc. Do know that it processes/removes items in batches, so when you run the command again, it might find/delete items which are still in the process of being deleted from the previous run, which might also lead to interesting messages.
  
  LikeLike
  
  Reply ↓
KerG on December 16, 2022 at 11:53 PM said:

Thanks for the work put into this. Very helpful for sure. So I am running into an issue where it deleted a lot (over 100,000) duplicates in multiple folders. However, there are a few folders left that have multiple dupes. I have run it again, just on a single small folder to test and the script does see them and says that it has removed them, but they remain in the mailbox. I have started a new session, reconnected and ran it multiple times with the same results over a 5-6 hour period. Any thoughts?

LikeLike

Reply ↓
- Michel de Rooij on December 17, 2022 at 4:31 PM said:
  
  Is OWA showing the same thing? Could be your Outlook/OST is catching up on the removals.
  Script is operating against a non-cached view, so if it finds duplicates, they are (still) there, but if Exchange is still processing it might be locking the folder, blocking any other (EWS-based) operations against it. And unfortunately nothing much you can do once it has been initiated.
  
  LikeLike
  
  Reply ↓
misterkgg on December 17, 2022 at 5:34 PM said:

OWA showing the same thing. Not sure how Exchange still processing after 24 hours. May just need to recreate the mailbox at this point

LikeLike

Reply ↓
- Michel de Rooij on December 17, 2022 at 5:49 PM said:
  
  Can take a while, but not *that* long (also, the removal is per batch of max. 100 items).
  
  LikeLike
  
  Reply ↓
  - misterkgg on December 17, 2022 at 5:55 PM said:
    
    Yeah, not sure what happened. Started off great, then just stopped. I will recreate the mailbox and save myself the effort. Curious to know what happened though as I’m sure will use this in the future.
    
    LikeLike
    
    Reply ↓
Roger on January 19, 2023 at 4:59 PM said:

Hi Michel,
Does the duplication take into account message headers?
I am noticing that a fair amount of duplicates that a client has are not getting removed as the messages have a few differences. To give you a rough idea…
1.) User 1 and user 2 receive the same email from an external sender.
2.) They both move this email to a shared mailbox. (as to “file” it).
3.) I see the duplicates in the shared mailbox so run the duplicate removal job.
4.) The messages both remain.
5.) I assess the message headers for both – and the final received by server in the headers differs for both (Both 365).

With this in mind, is there a way to omit the actual headers from being assessed? I have also seen issues with duplicate messages where the message size is different and the “created date” different too.
None of the switches (Body, Full (with NoSize) or Quick) seem to pick them up.
Thanks in advance.
Roger.

LikeLike

Reply ↓
Jordan on March 8, 2023 at 8:23 PM said:

Hi Michel,

Trying to remove duplicate contacts from a user, its removing all contacts from the user completely not just duplicates.

.\Remove-DuplicateItems.ps1 -Identity user@domain.etc -Type Contacts -DeleteMode MoveToDeletedItems -Mode Full -NoSize -Retain Newest -TenantId ‘123456-1234-1234-1234-123456’ -ClientId ‘1234567-1234-1234-1245-87654323456’ -Secret $Secret -Report -WhatIf:$true

This is my What If output –

What if: Performing the operation “Process-Mailbox” on target “Remove 1264 items from \Contacts”.

The user only has 1265 contacts total listed again their account, of which I can visibly see there are hundreds of duplicates.

Any ideas on what’s happening here?

Many Thanks,

LikeLike

Reply ↓
- Michel de Rooij on March 9, 2023 at 8:58 PM said:
  
  Yeah, it didn’t load the properties for non-mail items used for matching. Just pushed version 2.3 which should work.
  Thanks for catching this.
  
  LikeLike
  
  Reply ↓
Joshua Huffman on May 11, 2023 at 4:59 AM said:

This works great. But i am having an issue. I am connected using Azure app for authentication. The process starts failing at exactly 1 hour and 5 minutes. The inboxes i am scanning and cleaning are not completed by the time this happens. I then have to start the process over and get a little more done. The users in question have 3k folders. Is there any way to extend the time out.

WARNING: Error performing operation FindItems with Search options in Dunvale Sale. Error: Exception calling “FindItems” with “2” argument(s): “The request failed. The remote server returned an error: (401) Unauthorized.”

Above is the error i start getting at exactly 1 hour 5 minutes.

LikeLike

Reply ↓
- Michel de Rooij on May 11, 2023 at 7:18 AM said:
  
  It’s not a timeout, most likley your token expired. Reauthentication is not handled at the moment, but point noted.
  Note that you can just call it again and it will (though iteratively) do its task.
  
  LikeLike
  
  Reply ↓
Tim on June 18, 2023 at 10:47 AM said:

Hey Michel,

Just wondering if you have seen this when trying to run the script against an o365 mailbox.

At C:\scripts\Remove-DuplicateItems.ps1:213 char:17
+ Sign up
+ ~
The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double
quotation marks (“&”) to pass it as part of a string.
At C:\scripts\Remove-DuplicateItems.ps1:218 char:209
+ … k Button–medium Button d-lg-none color-fg-inherit p-1″> <span cla …
+ ~
The '<' operator is reserved for future use.
At C:\scripts\Remove-DuplicateItems.ps1:241 char:45
+
+ ~
Missing file specification after redirection operator.
At C:\scripts\Remove-DuplicateItems.ps1:439 char:13
+ CI/CD & Automation

LikeLike

Reply ↓
- Michel de Rooij on June 18, 2023 at 11:09 AM said:
  
  Looks.like you are trying to execute HTML instead of PS 🙂
  
  LikeLike
  
  Reply ↓
  - Tim on June 18, 2023 at 2:20 PM said:
    
    Thanks for that, I downloaded the file incorrectly from github and you put me on the right path. I may have some other questions to come though.
    
    LikeLike
    
    Reply ↓
Tim on June 19, 2023 at 5:26 AM said:

Hey Michel,
wondering if you can assist with the below?

PS C:\scripts> .\Remove-DuplicateItems.ps1 -Mailbox staffmember@email.com -Server outlook.office365.com -Credentials (Get-Credential) -type ALL -Impersonation -Mode Full -DeleteMode MoveToDeletedItems -Verbose

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
VERBOSE: Loading module C:\scripts\Microsoft.Exchange.WebServices.dll
VERBOSE: Loading module from path ‘C:\scripts\Microsoft.Exchange.WebServices.dll’.
VERBOSE: Module Microsoft.Exchange.WebServices v15.0.0.0 loaded
VERBOSE: Loading module C:\Program
Files\WindowsPowerShell\Modules\Microsoft.Identity.Client\4.53.0\Microsoft.Identity.Client.dll
VERBOSE: Loading module from path ‘C:\Program
Files\WindowsPowerShell\Modules\Microsoft.Identity.Client\4.53.0\Microsoft.Identity.Client.dll’.
VERBOSE: Module Microsoft.Identity.Client v4.53.0.0 loaded
Problem initializing Exchange Web Services using schema Exchange2016 and TimeZone AUS Eastern Standard Time
At C:\scripts\Remove-DuplicateItems.ps1:1571 char:9
+ Throw( ‘Problem initializing Exchange Web Services using sche …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (Problem initial…n Standard Time:String) [], RuntimeException
+ FullyQualifiedErrorId : Problem initializing Exchange Web Services using schema Exchange2016 and TimeZone AUS Ea
stern Standard Time

LikeLike

Reply ↓
- Michel de Rooij on June 19, 2023 at 9:27 AM said:
  
  The EWS module you’re using looks bit dated (v15 are the ones published by MS but very dated)
  Get most recent one from NuGet, https://www.nuget.org/packages/Exchange.WebServices.Managed.Api/
  
  LikeLike
  
  Reply ↓
  - Tim on June 20, 2023 at 10:48 AM said:
    
    I had another issue where I still was using basic authentication by using the credential script. Are you able to provide an example of using the script using the app Auth switches please? I know I am so close but not yet a cigar..
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on June 26, 2023 at 2:46 PM said:
      
      1) Create self-signed cert
      2) Register app, and upload public cert part (.cer) as authentication method (or use secret if you must).
      3) You now have tenantId, clientId (AppId) and certificate. Call script with:
      
      If you loaded the cert in your personal store (with private part, eg .pfx w/password)
      .\Remove-DuplicateItems.ps1 -TenantId -ClientId -CertificateThumbprint
      
      If certificate is on file:
      .\Remove-DuplicateItems.ps1 -TenantId -ClientId -CertificateFile [-CertificatePassword ]
      
      Note: CertificatePassword or Secret are secure strings, so read or convert them if needed, eg.
      $Secret= Read-Host ‘Secret’ -AsSecureString
      $Password= ConvertTo-SecureString ‘H3ll0’ -AsPlainText -Force
      
      LikeLike
      
      Reply ↓
Richard on September 10, 2023 at 7:57 PM said:

Hi, I keep getting:

PS C:\temp> .\Remove-DuplicateItems.ps1 -Identity mail@box.com -DeleteMode MoveToDeletedItems -Impersonation -Credentials $cred -Retain Oldest -Server outlook.office365.com -Verbose -WhatIf # -Mode Full
VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
VERBOSE: Using credentials mail@box.com
VERBOSE: Cleanup Mode: Folder
Processing mailbox mail@box.com (mail@box.com)
VERBOSE: Using richard@contoso.com for impersonation
VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
WARNING: Cannot bind to MsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. De externe server heeft een fout geretourneerd: (401) Niet gemachtigd.”
WARNING: Cannot bind to ArchiveMsgFolderRoot: Exception calling “Bind” with “2” argument(s): “The request failed. De externe server heeft een fout geretourneerd: (401) Niet gemachtigd.”
VERBOSE: Processing mail@box.com finished

Impersonation has been set on the same user I want to clean. But that user does have MFA, perhaps that’s the problem. I don’t know how to bypass that. Please advise 🙂

LikeLike

Reply ↓
- Michel de Rooij on September 10, 2023 at 10:55 PM said:
  
  You use Credentials; be advised OAUth is required for EXO. Create an app with proper permissions (Manage as App) and secret/certificate and use its application ID together with the tenant ID and secret/cert to authenticate.
  
  LikeLike
  
  Reply ↓
  - Richard on September 11, 2023 at 5:44 AM said:
    
    I am using credentials, but I guess that’s not what you mean.
    Can you give me a little bit more info on how to do this?
    
    LikeLike
    
    Reply ↓
    - Richard on September 12, 2023 at 9:13 PM said:
      
      Fixed it already, tnx!
      
      LikeLike
      
      Reply ↓
      - Michel de Rooij on September 12, 2023 at 9:39 PM said:
        
        Glad you got it sorted.
        
        LikeLike
Richard on September 13, 2023 at 6:05 AM said:

Is it not finding duplicates:

Running this:

.\Remove-DuplicateItems.ps1 -Identity *** -DeleteMode MoveToDeletedItems -Retain Oldest -Server outlook.office365.com -Secret $secret -TenantId $tenantId -ClientId $clientId -Verbose -Mode Full -MailboxOnly -WhatIf

…
VERBOSE: Processing folder \Logboek
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Notities
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Ongewenste e-mail
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Postvak IN
What if: Performing the operation “Process-Mailbox” on target “Remove 4 items from \Postvak IN”.
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Postvak IN\Belangrijk!
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Contactpersonen\Recipient Cache
VERBOSE: Cleaning unique list (Finished Folder)
VERBOSE: Processing folder \Yammer-hoofdmap\Uitgaand
VERBOSE: Cleaning unique list (Finished Folder)
759494 items processed and 0 removed in 00:24:02 – average 31585 items/min

I have got many duplicates, but it isn’t finding them.
Any advice?

LikeLike

Reply ↓
- Richard on September 15, 2023 at 11:07 AM said:
  
  -NoSize did the job…
  
  LikeLike
  
  Reply ↓
  - Michel de Rooij on September 15, 2023 at 11:50 AM said:
    
    That switch was added for a reason. Cause is usually 3rd party software modifying properties, causing items to look identical, but they aren’t.
    
    LikeLike
    
    Reply ↓
Dimitris Mezinis on September 14, 2023 at 8:07 AM said:

Hello, I am getting this error:

VERBOSE: Will use provided secret to authenticate
Exception calling “Create” with “1” argument(s): “Could not load file or assembly ‘Microsoft.IdentityModel.Abstractions, Version=6.22.0.0, Culture=
neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified.”
At XXXXX\Remove-DuplicateItems.ps1:1636 char:13
+ $App= [Microsoft.Identity.Client.ConfidentialClientApplic …

I have already installed
Install-Package Microsoft.IdentityModel.Abstractions -RequiredVersion 6.22.0

Can you help me sort it out?

LikeLike

Reply ↓
- Michel de Rooij on September 15, 2023 at 11:53 AM said:
  
  The following 2 PS modules are needed: Microsoft.Exchange.WebServices and Microsoft.Identity.Client
  In order to be able to load these, running the script elevated is needed.
  
  LikeLike
  
  Reply ↓
  - Dimitris Mezinis on September 15, 2023 at 1:39 PM said:
    
    [SOLVED ]I was using an elevated prompt and I had installed the modules required. The problem was that I had installed Microsoft.Identity.Client v4.53 from PowerShellGet and I guess when it was looking for Microsoft.IdentityModel.Abstractions couldn’t locate it although I had C:\Program Files\PackageManagement\NuGet\Packages in Env PATH and PS Path.
    
    The solution was to remove package Microsoft.Identity.Client (v4.53) and to install v4.56 from Nuget with -SkipDependencies. Then I copied from NuGet\Packages Microsoft.Exchange.WebServices, Microsoft.Identity.Client AND Microsoft.IdentityModel.Abstractions the net45 .dlls to Remove-DuplicateItems.ps1 folder and now script runs flawlessly.
    
    Thank you for your time and a really life-saving script!
    
    LikeLike
    
    Reply ↓
    - Thibaut on December 22, 2023 at 12:20 AM said:
      
      Could you please explain me what you exactly copied to Remove-DuplicateItems.ps1 folder ?
      
      I have same issue, but persist after getting Microsoft.Exchange.WebServices, Microsoft.Identity.Client and Microsoft.IdentityModel.Abstractions from NuGet.
      
      LikeLike
      
      Reply ↓
Grésau on September 27, 2023 at 2:38 PM said:

I run this script to remove duplicates:

.\Remove-DuplicateItems.ps1 -Identity user.name@something.se -Type Mail -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

cmdlet Remove-DuplicateItems.ps1 at command pipeline position 1
Supply values for the following parameters:
UseDefaultCredentials:

i have no idea what kind of parameter it wants….. i tried 0, 1 true, false, $true, $false.
but it wont proceed, im sure i do something wrong but thisd scriopt would save my day if it worked.

I get an error message saying;
.\Remove-DuplicateItems.ps1 : Cannot process argument transformation on parameter ‘UseDefaultCredentials’. Cannot convert value “System.String” to type “System.Manageme
nt.Automation.SwitchParameter”. Boolean parameters accept only Boolean values and numbers, such as $True, $False, 1 or 0.
At line:1 char:1
+ .\Remove-DuplicateItems.ps1 -Identity user.name@something.se …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Remove-DuplicateItems.ps1], ParameterBindingArgumentTransformationException
+ FullyQualifiedErrorId : ParameterArgumentTransformationError,Remove-DuplicateItems.ps1

this is directly after this prompt:

LikeLike

Reply ↓
- Michel de Rooij on September 28, 2023 at 12:40 AM said:
  
  It’s a switch, no value needed (or if you must, use -UseDefaultCredentials:$true or $false)
  
  LikeLike
  
  Reply ↓
Marcel on October 3, 2023 at 2:12 PM said:

Hi Michel. Thanks for your script it looks very good and seems to save a lot of work.

We tried the script and have created duplicates manually using Outlook.
If we run the script we get the following warnings and no duplicate has been removed.

——————–

[PS] C:\Windows\system32>C:\Users\adm_server\Downloads\Remove-DuplicateItems-master\Remove-DuplicateItems.ps1 -Identity *OurMailbox* -Type All -DeleteMode MoveToDeletedItems -Mode Full -Impersonation -Verbose -UseDefaultCredentials:$true -NoSize

VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
VERBOSE: Module Microsoft.Identity.Client v4.25.0.0 already loaded
VERBOSE: Using Default Credentials
VERBOSE: Cleanup Mode: Folder
Processing mailbox *OurMailbox*
VERBOSE: Using *OurMailbox* for impersonation
VERBOSE: Looking up EWS URL using Autodiscover for *OurMailbox*
VERBOSE: Using EWS endpoint https://MAILSERVER/EWS/Exchange.asmx
VERBOSE: Constructing folder matching rules
WARNING: Cannot bind to MsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.
WARNING: Cannot bind to ArchiveMsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.
VERBOSE: Processing *OurMailbox* finished

——————–

Any Ideas?

Thanks a lot

LikeLike

Reply ↓
- Michel de Rooij on October 4, 2023 at 11:23 PM said:
  
  I cannot reproduce. Can you tell me bit more on the environment, what Exchange server version is used, did you install the exchange.webservices.managed.api and microsoft.identity.client modules from NuGet and if so, what PowerShell version was used.
  
  LikeLike
  
  Reply ↓
  - Marcel Neubauer on October 13, 2023 at 10:45 AM said:
    
    Sorry for the delay. Lot to do at the moment.
    We use Exchnage 2019 fully patched till Oct23.
    
    We`ve installed both modules and tried to put the files inside the script folder as well -> Same “Error”
    
    Exchange Health Checker shows a healthy sytem. We have no idea were the errors are coming from.
    
    Thanks a lot for your time
    
    LikeLike
    
    Reply ↓
    - Marcel Neubauer on November 7, 2023 at 8:09 AM said:
      
      Any ideas?
      
      LikeLike
      
      Reply ↓
  - Marc Snijders on November 16, 2023 at 11:31 AM said:
    
    [PS] C:\temp> .\Remove-DuplicateItems.ps1 -mailbox mailboxname -type calendar -deletemode movetodeleteditems -mode full -Impersonation -UseDefaultCredentials:$true -verbose -server outlook.office365.com
    VERBOSE: Module Microsoft.Exchange.WebServices v2.2.1.0 already loaded
    VERBOSE: Module Microsoft.Identity.Client v4.8.1.0 already loaded
    VERBOSE: Using Default Credentials
    VERBOSE: Cleanup Mode: Folder
    Processing mailbox mailboxname (mailboxname@mailboxname.com)
    VERBOSE: Using mailboxname for impersonation
    VERBOSE: Using Exchange Web Services URL https://outlook.office365.com/EWS/Exchange.asmx
    VERBOSE: Constructing folder matching rules
    WARNING: Cannot bind to MsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.
    WARNING: Cannot bind to ArchiveMsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.
    VERBOSE: Processing mailboxname finished
    
    LikeLike
    
    Reply ↓
    - Michel de Rooij on November 21, 2023 at 5:12 PM said:
      
      UseDefaultCredentials is meant for Exchange on-premises, using the current security context (account) to access Exchange on-premises.
      To access Exchange Online – as indicated by specifying the Exchange Online endpoint outlook.office365.com – you need to use modern authentication (OAuth). To use OAuth, you first need to register an enterprise application in Azure, see https://learn.microsoft.com/exchange/client-developer/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth?WT.mc_id=M365-MVP-5000284. Then, when calling the script, you need to use a certificate or secret when authenticating, apart from specifying the tenant id, client id (generated by registering the application). For example, the authentication parameters look like:
      .\remove-DuplicateItems.ps1 -Identity “id” -Server outlook.office365.com -TenantId ‘tenant-guid’ -ClientId ‘app-id’ -Secret ‘secure-string-with-secret’
      
      LikeLike
      
      Reply ↓
Gad on November 22, 2023 at 12:00 AM said:

hi,

how can i reslove :
WARNING: Cannot bind to MsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.
WARNING: Cannot bind to ArchiveMsgFolderRoot: Cannot find an overload for “Bind” and the argument count: “2”.

i have on-prem exchnage 2016 ?

LikeLike

Reply ↓