Exchange 2010-2013 Migration and OAB

Ex2013 LogoLast year, Exchange fellows Andrew Higginbotham, Paul Cunningham as well as the Exchange Team reported on checking, and when necessary configuring, your Offline Address Book (OAB) in your current Exchange Server 2010 environment, prior to installing Exchange Server 2013. Not doing so could result in a complete download of the Offline Address Book created by Exchange Server 2013, titled ‘Default Offline Address List (Ex2013)’.

Today I received a report that there is a different symptom of configuration absence. In this case, the customer reported on the inability to download the offline address book, and upon further inspection the Autodiscover server did not report back on the offline address book URL to use. In other words, OAB information was absent from the Autodiscover response, and Outlook gets confused. Note that this issue was reported in Outlook 2010 after installing Exchange Server 2013 Cumulative Update 10. I’m not sure if this change in behavior was introduced in these later builds of Exchange 2013 or Outlook, but it’s still a good thing to know.

The remedy here of course is to configure any (Exchange 2010) mailbox database with unconfigured Offline Address Book setting, and point them to the default offline address book using:

Get-MailboxDatabase | Where-Object {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where-Object {$_.IsDefault -eq $True})

Client Message Size Limits

powershellLast Update: Version 1.2, February 15th, 2017

Exchange Server 2013 and Exchange Server 2016 enforces certain message size limits when it comes to client messages. These limits are in-place so clients can’t generate excessive load on your Exchange environment. These limits are determined for various access methods in multiple web.config files on Exchange Client Access Servers as well as Mailbox Servers.

Sometimes you may have good reasons to increase those limits. For example, when migrating to Office 365 using a product like MigrationWiz, you may want to increase the limit for Exchange Web Service (EWS) requests to allow for migration of larger items. Another example is when you want to allow for bigger attachments in Outlook WebApp (OWA). On TechNet, there’s an article on how to reconfigure these limits. However, the process consists of editing multiple web.config files, replacing multiple values in the same file, and following this process on each Exchange 2013/2016 server in your environment. This is not only labor intensive and prone to error, but becomes tedious when you consider that each Cumulative Update will overwrite your web.config files.

But do not despair. To execute these changes for OWA and EWS, I have created a PowerShell script which will perform these tasks for you.

Requirements
Using the script requires Exchange 2013. You need to provide the server name (default is local server) or AllServers to apply to all Exchange 2013/2016 servers in your environment. The script will modify the web.config remotely using the system share (e.g. C$), using the location of the Exchange installation, and uses IISRESET tool to restart IIS. It will create a backup of the web.config before modifying it.

Notes:

  1. The script checks for running in elevated administrator mode when running against the local machine.
  2. Current version of the script requires Exchange Management Shell, to run Exchange cmdlets for checking installed roles a.o., as the web.config files which require editing depend on the installed roles.
  3. For OWA, add ~33% to the value you want to specify to compensate for encoding overhead.
  4. When connected to an Exchange server, the script processes the server hosting the EMS session last to prevent abortion caused by IIS reset.
  5. Script currently runs against Exchange 2013 or Exchange 2016.

Usage
The script Configure-ClientSizeLimits.ps1 uses the following syntax:

.\Configure-ClientSizeLimits.ps1 [-Server |-AllServers] [-OWA <size>] [-EWS <size>] [-EAS <size>] [-Reset] [-NoBackup] 

A quick walk-through on the parameters and switches:

  • Server specifies the server to configure. When omitted, it will configure the local server. This parameter is mutually exclusive with AllServers.
  • AllServers switch specifies to configure all Exchange 2013 servers. This switch is mutually exclusive with Server.
  • OWA configures the message size limit for Outlook Web Access. Value is in bytes.
  • EWS configures the message size limit for Exchange Web Services. Value is in bytes.
  • EAS configures the message size limit for Exchange ActiveSync. Value is in bytes.
  • NoBackup tells the script not to make backup copies of modified web.config files.
  • Reset switch specifies to perform an IISRESET against servers after reconfiguration of client-specific message size limits.

So, suppose you want to configure OWA, EWS and EAS client message size limits on the local Exchange server, you can use:

.\Configure-ClientSizeLimits.ps1 -OWA 15MB -EWS 15MB EAS 15MB

cs

If you want to configure EWS limits for all servers, resetting IIS, you could use:

.\Configure-ClientSizeLimits.ps1 -AllServers -EWS 10240 -Reset

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision
See TechNet Gallery page.

To Do
Compatibility with Exchange 2010 and removal of dependency on Exchange Management Shell.

Clearing AutoComplete and other Recipient Caches

Exchange 2010 Logo

Last version: 1.1, March 12th, 2016.

Anyone who has participated in migrations or transitions to Exchange has most likely encountered or has had to work around potential issues caused by the nickname cache. A “cache,” also known by its file extension, NK2 in older Outlook clients, is a convenience feature in Outlook and Outlook WebApp (OWA) which lets users pick recipients from a list of frequently-used recipients. This list is displayed when the end user types in the first few letters.

The potential issue revolves around end users using those lists to send messages, as the list contains cached recipient information. Because this information is static, it may become invalid at some point. Thus, when users pick recipients when sending messages, they may be sending messages to non-existent recipients or invalid e-mail addresses, which create issues like non-delivery of e-mail.

Read the full article over on ENow Solutions Engine blog.

Clean-AutoComplete

Using the script mentioned in the article, which can be used to clear cached recipient information, is straightforward. It requires Exchange 2010 or later and Exchange Web Services Managed API 1.2 (or later) which you can download here. Alternatively, you can copy the Microsoft.Exchange.WebServices.DLL with the script as it will also look for it in the current folder.

The script Clean-AutoComplete.ps1 has the following syntax:

Clear-AutoComplete.ps1 [-Mailbox] <String> [-Server <String>] [-Impersonation] [-Credentials <PSCredential>] [-Type <Array>]

Where:

  • Mailbox is the name or e-mail address of the mailbox.
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will use AutoDiscover.
  • Switch Impersonation specifies if impersonation will be used for mailbox access, otherwise the current user context will be used.
  • Credentials specifies the user credentials to use.
  • Type specifies what cached recipient information to clear. Options are Outlook  (Outlook AutoComplete stream), OWA (OWA Autocomplete stream), SuggestedContacts, RecipientCache or All. Default is Outlook,OWA.

So for example, suppose you want to clear the Autocomplete stream used by Outlook on a mailbox, you can use:

Clear-AutoComplete.ps1 -Mailbox Olrik -Type Outlook -Verbose

ScreenCapTo remove the Autocomplete stream used by OWA on your Office 365 account, you can use:

Clear-AutoComplete.ps1 -Mailbox olrik@office365tenant.com –Credentials (Get-Credential) –Type OWA

Be advised that clearing the Outlook AutoComplete stream will only have effect for Outlook running in Online mode. Outlook caches this information as well in the OST file, leaving the options of running Outlook with the /CleanAutocompleteCache switch, or remove and let Outlook recreate the OST file. The temporary Stream_AutoComplete files*.dat files created under %USERPROFILE%\AppData\Local\Microsoft\Outlook\RoamCache are used by Outlook to speed things up.

Disabling Auto-Complete and Suggested Contacts
Alternatively, you can disable Auto-Complete, the equivalent of unchecking the Outlook option ‘Use Auto-Complete List to suggest names when typing in the To, Cc and Bcc line‘, by setting the following registry key:

Note: In the examples below, you need to modify the version number in the examples corresponding to the Outlook version you wish to apply these settings against. Use 16.0 as indicated for Outlook 2016, but change it to 15.0 for Outlook 2013, or 14.0 for Outlook 2010.

HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Preferences\
ShowAutoSug=0 (REG_DWORD)

To configure this setting using a Group Policy, use the following registry setting:

HKEY_CURRENT_USER\Software\Policies\Microsoft\office\16.0\Outlook\Preferences\ShowAutoSug=0 (REG_DWORD)

You can also disable Suggested Contacts folder, the equivalent of unchecking the Outlook option ‘Automatically create Outlook contacts for recipients that do not belong to an Outlook Address Book’, with the following registry key:

HKEY_CURRENT_USER\Software\Microsoft\office\16.0\Outlook\Contact\CreateContactsForOneOffs= 0 (REG_DWORD)

The related Group Policy setting is:

HKEY_CURRENT_USER\Software\Policies\Microsoft\office\16.0\Outlook\Contact\CreateContactsForOneOffs= 0 (REG_DWORD)

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Download
You can download the script from my Technet Gallery here.

Revision History
See Technet Gallery page.

 

 

Slow Mailbox (Migration) Throughput and HP NIC Drivers

Hewlett-Packard_logo-web[1]A small post on an issue I recently encountered when doing preparations for a mailbox migration. The system was an HP Proliant DL370 G6 system, prepared and configured with the OS and Exchange by the customer’s IT department.

Things looked OK and we were going to perform some test migrations to get an throughput estimation for this configuration  to help us organize migration batches. To our dismay, speeds were way lower than we were used to see with similar configurations; mailboxes were migrated with an average speed of 7-8 MB/min, where we used to see something in the 50-120 range.

A quick look on the performance monitor didn’t show anything out of the ordinary, except for very low downstream network throughput. With the servers (the original Exchange server as well as the new one) being on the same subnet and physically next to each other, networking components were also not deemed suspect (other servers on same switch were not experiencing this issue).

I then tried something simple: copying near 100 MB of files from the source Exchange server to the new one. It went at an ridiculous slow speed of 60-80 KB/s. Copying those same files from the new server to the source server was instantly. I verified this against a vacant server on the same switch; copying from and to the source Exchange server on that server was instant, both up- and downstream.

So, if SMB was having trouble getting packets across, that could explain the slow mailbox migration speeds. Attention shifted to the networking configuration on the new Exchange server, which was equipped with a HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter. I checked the driver version of one of the NC375i’s instances through Network Connections > Properties (of instance) > Configure > Driver (tab). It reported QLogic Corp. driver 4.7.17.926 (qlxgnd64.sys) was used.

After some searching on HP’s support site I discovered an advisory which could apply to my situation as it applies to the same qlxgnd64.sys driver version 4.7.17.926: c03734205, “Advisory: HP NC Network Adapters – Certain HP NC-Series Network Adapters May Experience Very Slow Bandwidth During Large File Transfers on Windows Server 2008 and Windows Server 2008 R2”.

The advisory gives the option to either keep the driver and disabling Large Receive Offload (LRO) or to upgrade to driver version 4.7.18.131. We choose the latter:

image

After upgrading the driver, we moved a mailbox and and throughput speeds were within the expected range again as we found out when producing a quick stats report using the cmdlet (Exchange 2010):

Get-MoveRequest | Where { $_.Status -eq "Completed" } | Get-MoveRequestStatistics | Select DisplayName,TotalMailboxSize,TotalMailboxItemCount,@{n="Speed MB/min"; e={ [int]($_.BytesTransferred.ToMB() / $_.TotalInProgressDuration.TotalMinutes) }}

image

In my opinion, it’s another fine example of the value of testing and validating your configuration and any amendments you make before putting them in production and be cautious with what I call “blindly updating” of system components such as drivers or driver packs (e.g. HP’s SPP or Service Pack for ProLiant).

If you don’t have the luxury of a test- and acceptance environment, just as with Service Packs, Rollups and Cumulative Updates, have a waiting period and check the vendor’s support site for any reported issues before implementing updates yourself; according to this discussion on the HP support forum, the issue with the 4.7.17.926 QLogic driver existed for quite some time.

The Attribute, the Myth, the legacyExchangeDN

Exchange 2010 LogoAfter some recent Exchange troubleshooting I decided to do a small write-up on an attribute most people working with Exchange know about, the infamous exchangeLegacyDN.

History
In the early days of Exchange, the NT world was flat. Exchange utilized its own hierarchical X.400 directory service and to uniquely identify objects it used an attribute called obj-Dist-Name. It contained a constructed value using elements like organization, containers and the canonical name to construct the entry, e.g. /o=Contoso/ou=EMEA/cn=Recipients/cn=User.

Then came the introduction of Active Directory with the release of Windows Server 2000. In AD, while you could refer to object using obj-Dist-Name’s counterpart distinguishedName, objects are primarily identified using their static Global Unique Identifier (GUID). The distinguishedName is constructed using relative names like the OU and CN, e.g. CN=User,OU=EMEA,DC=contoso,DC=com. Because it’s dynamic, the distinguishedName will change when you move or rename the object.

For backward compatibility with older versions of Exchange, clients or 3rd party tooling the legacyExchangeDN attribute was introduced which was to provide a unique key for the Exchange object. A process called Recipient Update Service (RUS) was responsible for populating the legacyExchangeDN value using the current location in conjunction with the given name (CN). Since Exchange 2007, legacyExchangeDN is set at creation time or can be updated manually using the Update-Recipient cmdlet. When a conflicts are detected, a unique value was constructed by adding a GUID-like series of hex characters.

When public folders are used for publishing Free/Busy information, the legacyExchangeDN was used to determine in which public folder the information was published. More information on this in an earlier blog here.

Typical values for the legacyExchangeDN attribute are:

  • Pre-Exchange 2007:
    • /o=Contoso/ou=First Administrative Group/cn=Recipients/cn=UserA
  • Exchange 2007 and later:
    • /o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=UserA

Until Exchange 2010 SP1 Rollup 6 the legacyExchangeDN value was predictable, but as of this Rollup (and Exchange 2013), 3 random hex characters are added for uniqueness.

Exchange Internal Addressing
While SMTP addressing is the de facto e-mail addressing standard, Exchange internally still uses an X.500 addressing scheme. Using X.500 implies that an X.500 is required, which is why mail objects in an Exchange organization such as mailboxes, require a properly populated legacyExchangeDN.

The most encountered symptom of not having properly populated or unpopulated legacyExchangeDN attributes is failure of e-mail delivery or transport, which I’ll elaborate on a bit below. I’ll mention it again: legacyExchangeDN is required, unlike some Exchange admins I overheard a while ago when they were looking for the cause of several NDRs discussing SMTP was used so it should be ‘no problem’.

IMCEA and IMCEAEX
When sending a message, the sender and recipient are checked against the Global Address List (GAL). When it can’t find a match in the GAL, IMCEA encapsulation is used which stands for Internet Mail Connector Encapsulated Addressing. An IMCEA encapsulated address looks like:

IMCEAEX-_O=CONTOSO_OU=First+20Administrative+20Group_cn=Recipients_cn=philip@contoso.com

A trailing “EX” at the end of IMCEAEX indicates that a non-SMTP address was encapsulated. The used SMTP domain “contoso.local”  is taken from the GAL object unless the lookup failed; in that case the forest DN is used.

Migrations and Co-existence Issues
Clients like Outlook cache information like the legacyExchangeDN for name lookups (the infamous name cache or .nk2 files). It will also store the legacyExchangeDN (PR_EMAIL_ADDRESS) with e-mail item. This is used when replying to old e-mail and which is why you’ll see X.500 addresses when you open the e-mail outside of the organization.

While using the legacyExchangeDN makes your client insensitive to e-mail address changes, name changes etc. it will provide a (small) challenge when for example a mailbox is moved to a different forest (or to or from Office 365 for that matter) as it will get a new legacyExchangeDN value.

To attach those old legacyExchangeDN addresses to an object in the new environment, add an X.500 entry to the proxy addresses (X.500 refers to the protocols built on the X.400 standard), e.g. X500:/o=Contoso/ou=First Administrative Group/cn=Recipients/cn=UserA.

When the target environment already contains a mail-enabled contact, ideally you want to preserve the current legacyExchangeDN as x500 address on the mailbox object (depending on if the solution transforms the contact of creates a new object). This way, users in the target environment won’t be getting NDRs because their legacyExchangeDN became invalid by the move, e.g.

Delivery has failed to these recipients or groups:
Philip Mortimer
The e-mail address you entered couldn’t be found. Please check the recipient’s e-mail address and try to resend the message. If the problem continues, please contact your helpdesk
Diagnostic Information for administrators:
Generating Server: L13L14EX1.fabrikam.local
IMCEAEX-_O=CONTOSO_OU=First+20Administrative+20Group_cn=Recipients_cn=philip@contoso.com

In this case, the user tried to reply using the X.500 address /O=Contoso/OU=First Administrative Group/cn=Recipients/cn=philip. contoso.local (later more on address conversion).

If preserving the contact’s legacyExchangeDN isn’t possible, you could collect existing legacyExchangeDN values and stamp them as X500 address on the mailbox before or after moving it.

Unpopulated legacyExchangeDN
As explained earlier, the legacyExchangeDN is required for mail-enabled Exchange objects, including mail-enabled contacts. I’ve seen contacts with unpopulated legacyExchangeDN attributes in situations where the GAL Sync tool was misconfigured.

Sending an e-mail to such a contact will also result in an NDR:

Delivery has failed to these recipients or groups:
Francis Blake
The e-mail address you entered couldn’t be found. Please check the recipient’s e-mail address and try to resend the message. If the problem continues, please contact your helpdesk
Diagnostic Information for administrators:
Generating Server: L13L14EX1.fabrikam.local
IMCEAEX-_O=NT5_ou=00000000000000000000000000000000_cn=4D182F9E133B564F88CA6A86830D4314@contoso.local
#550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found ##

Solving this situation depends on the solution used. Basically, GAL Sync or Identity Management products directly manipulate AD objects, bypassing the regular process like provisioning Exchange attributes. Therefor, they should call “Update-Recipient” to provision Exchange attributes like the legacyExchangeDN or any e-mail addresses to be set via E-Mail Address Policies.

Microsoft’s Identity Management product, ForeFront Identity Manager’s, contains a GAL agent which has a setting which will trigger the provisioning of Exchange attributes. Make sure this setting is enabled and you provided a valid URI:

image

Reporting
Instead of waiting for users to report on (internal) NDR messages, you can consult the transport logs for failed delivery messages. For example, to report on all NDR messages of the last 7 days on all transport servers, you can run the following cmdlet from the Exchange Management Shell:

Get-TransportServer | Get-MessageTrackinglog -EventID FAIL -Start (Get-Date).AddDays(-7) -ResultSize Unlimited | Where {$_.Recipients -match "^IMCEAEX*"}

NDRReportReconstructing legacyExchangeDN
If for some reason you need to reconstruct the X.500 address using the information from the NDR message, support article KB2807779 contains a short instruction or utilize the small interactive script below to convert the reported IMCEAEX address to an X500 entry:

$Addr= Read-Host "Enter full IMCEAEX address:"
$Repl= @(@("_","/"), @("\+20"," "), @("\+28","("), @("\+29",")"), @("\+2C",","), @("\+3F","?"), @("\+5F", "_" ), @("\+40", "@" ), @("\+2E", "." )) 
$Repl | ForEach { $Addr= $Addr -replace $_[0], $_[1] } 
$Addr= "X500:$Addr" -replace "IMCEAEX-","" -replace "@.*$", "" 
Write-Host $Addr

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Removing Duplicate Items from a Mailbox

powershellLast version: 1.84, December 6th, 2017.

For those involved with Exchange migration projects or managing Exchange environments, at some point you probably have experienced the situation where people ended up with duplicate items in their mailbox. Duplicate items can be caused by many things, but most common are:

  • Synchronization tools or plug-in. Entries from the mailbox are treated as new entries and as a consequence are added to the mailbox when synchronizing information back to the mailbox, creating duplicates. In the past, I’ve seen this happening with Nokia PC Suite and Google Apps Sync for example;
  • Importing existing data. Accidental import from – for example – a PST file to a mailbox  can lead to duplicate entries.

image

When looking for a solution, you’ll probably encounter MSKB299349, “How to remove duplicate imported items in Outlook”. This article describes a manual procedure to remove duplicates entries from your calendar, contacts, inbox or other folders. Not a very helpful and labor intensive.

When continuing your search, you’ll find lots (I mean lots!) of tools and Outlook add-ins, like Vaita’s DIR or MAPILab’s Duplicate Remover. Not all this software is free (some even require payment per duplicate removal of appointments, contacts or e-mail) and some might not even work (MAPI-based tools may not work against Exchange 2013).

When you finally have selected a tool, in most cases they require installation of a piece of software and someone to perform the removal process using the tool or Outlook with add-in. When you’re an Apple shop you’ll require different tools, unless you’re running a Windows desktop somewhere (I’ll just pretend I didn’t hear you saying ‘Why don’t you install the tool on the Exchange server’).

Wouldn’t it be nice if you’d have a PowerShell script you can conveniently run from any workstation (or server) with PowerShell installed, removing those duplicate items from a user’s mailbox remotely? If the answer is yes, the Remove-DuplicateItems.ps1 script may be something for you.

Requirements
Using the Remove-DuplicateItems.p1 script requires Exchange 2007 or later and Exchange Web Services (EWS) Managed API 2.0 (or later) which you can download here. Alternatively, when you already got the EWS Managed API Microsoft.Exchange.WebServices.DLL somewhere, you can copy it to the same folder where the script is located and it will be picked up. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation for Exchange On-Premises or Office 365, see this blog post.

Usage
The script Remove-DuplicateItems.ps1 uses the following syntax:

Remove-DuplicateItems.ps1 [[-Identity] ] [[-Type] ] [-Retain ] [-Server ] [-Impersonation] [-DeleteMode ] [-Credentials ] [-Mode ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-PriorityFolders <String[]>] [-NoSize] [-MailboxWide] [-NoProgressBar] [-Force] [-WhatIf] [-Confirm] []

A quick walk-through on the parameters and switches:

  • Identity is the e-mail address or name of the mailbox to process. If name is used, it is matched against cn/SAMAccountname/email address of local AD.
  • Type determines what folders are checked for duplicates. Valid options are Mail, Calendar, Contacts, Tasks, Notes or All (Default).
  • Retain determines which item to retain by comparing last modification times. Valid options are Newest (default) or Oldest.
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
  • When the Impersonation switch is specified, impersonation will be used for mailbox access, otherwise the current user context will be used.
  • DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder).
  • Credentials specifies credentials to use (provide credentials using Get-Credential).
  • Mode determines how items are matched. Options are Quick, which uses PidTagSearchKey and is the default mode, or Full which uses a predefined set of attributes to match items, depending on the item class:
ItemClass Criteria
Contacts File As, First Name, Last Name, Company Name, Business Phone, Mobile Phone, Home Phone, Size
Distribution List FileAs, Number of Members, Size
Calendar Subject, Location, Start & End Date, Size
Task Subject, Start Date, Due Date, Status, Size
Note Contents, Color, Size
Mail Subject, Internet Message ID, DateTimeSent, DateTimeReceived, Sender, Size
Other Subject, DateTimeReceived
  • MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter  when running against mailboxes on Exchange Server 2007.
  • ArchiveOnly specifies you only want to process personal archives of specified users.
  • IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
  • ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
    Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
  • MailboxWide performs duplicate cleanup against whole mailbox, instead of per folder. By default, the first unique item encountered will be retained. When an item is found in Folder A and in Folder B, it is undetermined which item will be kept, unless PriorityFolders is used.
  • PriorityFolders specifies which folders have priority over other folders, identifying items in these folders first when using MailboxWide mode. Usage of wildcards and well-known folders is identical to IncludeFolders.
  • NoSize tells script to not use size to match items in Full mode.
  • NoProgressBar prevents displaying a progress bar as folders and items are being processed.
  • Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.

Few notes:

  • When MoveToDeletedItems is specified, the Deleted Items folder will be skipped;
  • When Type is omitted or set to All, all folders are scanned, including folders like Conversation History, RSS Feeds, etc.;
  • When Quick mode is used and PidTagSearchKey is missing or inaccessible, search will fall back to Full mode;
  • For more info on PidTagSearchKey, see http://msdn.microsoft.com/en-us/library/cc815908.aspx. Note that PidTagSearchKey will have duplicate values for copied objects.
  • You need to specify MailboxOnly when running against mailboxes on Exchange Server 2007 as the Exchange 2010 personal archive options in EWSare not support in Exchange 2007 mode.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter Matches
FolderA \TopFolderA\FolderA, \TopFolderB\FolderA
Folder* \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\*Folder* \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\*FolderA\* \TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA \TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So, suppose you want to remove  duplicate Appointments from the calendar of mailbox migtester1 using attribute matching, moving duplicate items to the DeletedItems, using Impersonation and you want to generate extra output using Verbose. In such case, you could use the following cmdlet:

Remove-DuplicateItems.ps1 -Identity migtester1 -Type Calendar -Impersonation -DeleteMode MoveToDeletedItems -Mode Full -Verbose

image

Alternative, you can use an e-mail address and specify credentials.  This allows the script to run against mailboxes in Office 365, for example:

Remove-DuplicateItems.ps1 -Identity olrik@office365tenant.com -Type Mail -DeleteMode MoveToDeletedItems -Mode Full -Credentials (Get-Credential) -Retain Oldest

A more complex example using IncludeFolders, ExcludeFolders and PriorityFolders:

$Credentials= Get-Credential
 .\Remove-DuplicateItems.ps1 -Mailbox olrik@office365tenant.com -Server outlook.office365.com -Credentials $Credentials -IncludeFolders '#Inbox#\*','\Projects\*' -ExcludeFolders 'Keep Out' -PriorityFolders '*Important*' -MailboxWide

This will remove duplicate items from the specified mailbox in Office365, using the following options:

  • Fixed Server FQDN – bypassing AutoDiscover.
  • Limits operation against the Well-Known Inbox folder, top Projects folder, and all of their subfolders.
  • Excluding any folder named Keep Out.
  • Duplicates are checked over all folders.
  • Priority is given to folders containing the word Important, causing items in
    those folders to be kept over items in other folders when duplicates are found.

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-DuplicateItems.ps1 -DeleteMode HardDelete -Impersonation

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision History
See TechNet Gallery page.

Removing Messages by Message Class

powershellLast version: 1.81, December 6th, 2017.

Recently, I was asked if it is possible to remove stub items. The reason was they were going to transition to a newer version of Exchange and they wouldn’t be using the archiving solution in the new environment. When required, vendor tooling would be used to search through the existing archives.

In such cases it makes sense to remove the stubs from the mailbox, which are shortcut messages that points to a copy of the original message in the archive solution. The new environment won’t contain the required Outlook plugins or extensions to retrieve the original message from the archive using the stub, making the stub lead to a partial or empty message.

To identify stubs, one can filter on an attribute of each item, MessageClass. This attribute defines which kind of item it is (in fact, determines what form Outlook should use in order to present or process the information). Examples of MessageClass definitions are IPM.Note (regular e-mail messages), IPM.Note.EnterpriseVault.Shortcut (message archived by Enterprise Vault) or IPM.ixos-archive (message archived by Opentext/IXOS LiveLink E-Mail Archive).

To identify stubs from Outlook, add the Message Class field to your Outlook view, e.g.:

StubsOutlook

When you want to remove the stubs using Outlook, you can utilize the Advanced Find function of Outlook, but that is a very labor intensive, tedious and non-centralized per-mailbox procedure:

SearchFromOutlook

Now I wouldn’t have started this article if the same thing wasn’t possible with a little bit of scripting against Exchange Web Services and so the script Remove-MessagesClassItems.ps1 was born. Using this script requires Exchange 2007 or later and Exchange Web Services Managed API 2.0 (or later) which you can download here or you can copy the Microsoft.Exchange.WebServices.DLL locally and adjust the DLL path mentioned in the script when necessary. The script has been developed and tested against Exchange 2007, meaning it’s a PowerShell 1.0 script which should be compatible with later versions of PowerShell or Exchange.

Also take notice that since you’ll be processing user mailboxes, you’ll need to have full mailbox access or impersonation permissions; the latter is preferred. For details on how to configure impersonation, I have written this article.

The script Remove-MessagesClassItems.ps1 uses the following syntax:

Remove-MessageClassItems.ps1 [-Identity]  [-MessageClass]  [-Server ] [-Impersonation] [-Credentials ] [-DeleteMode ] [-ScanAllFolders] [-Before ] [-MailboxOnly] [-ArchiveOnly] [-IncludeFolders <String[] [-IncludeFolders <String[]>] [-ExcludeFolders <String[]>] [-NoProgressBar] [-WhatIf] [-Confirm]

A quick walk-through on the parameters and switches:

  • Identity is the name or e-mail address of the mailbox.
  • MessageClass specifies the Message Class to remove, for example IPM.Note.EnterpriseVault.Shortcut (EnterpriseVault). You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘IPM.ixos*’ or ‘*EnterpriseVault*’. Matching is always case-insensitive.
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover.
  • Switch Impersonation specifies if impersonation will be used for mailbox access, otherwise the current user context will be used.
  • DeleteMode specifies how to remove messages. Possible values are HardDelete (permanently deleted), SoftDelete (use dumpster, default) or MoveToDeletedItems (move to Deleted Items folder). Note that the Deleted Items folder will be processed, unless MoveToDeletedItems is used.
  • Credentials specifies the user credentials to use.
  • Before can be used to only remove items received before specified date.
  • MailboxOnly specifies you only want to process the primary mailbox of specified users. You als need to use this parameter  when running against mailboxes on Exchange Server 2007.
  • ArchiveOnly specifies you only want to process personal archives of specified users.
  • IncludeFolders specifies one or more names of folder(s) to include, e.g. ‘Projects’. You can use wildcards around or at the end to include folders containing or starting with this string, e.g. ‘Projects*’ or ‘*Project*’. To match folders and subfolders, add a trailing \*, e.g. Projects\*. This will include folders named Projects and all subfolders. To match from the top of the structure, prepend using ‘\’. Matching is case-insensitive.
  • ExcludeFolders specifies one or more folder(s) to exclude. Usage of wildcards and well-known folders identical to IncludeFolders.
    Note that ExcludeFolders criteria overrule IncludeFolders when matching folders.
  • NoProgressBar prevents displaying a progress bar as folders and items are being processed.
  • ReplaceClass specifies that instead of removing the item, its PR_MESSAGE_CLASS class property will be modified to this value. For example, can be used in conjunction with MessageClass to modify any IPM.Note items pending Evault archival back to regular items, using: -MessageClass IPM.Note.EnterpriseVault.PendingArchive -ReplaceClass IPM.Note
  • Report reports individual items detected as duplicate. Can be used together with WhatIf to perform pre-analysis.

Well-Known Folders
For IncludeFolders, ExcludeFolders and PriorityFolders, you can also use well-known folders using this format: #WellKnownFolderName#, e.g. #Inbox#. Supported are #Calendar#, #Contacts#, #Inbox#, #Notes#, #SentItems#, #Tasks#, #JunkEmail# and #DeletedItems#. The script uses the currently configured Well-Known Folder of the mailbox to be processed.

Patterns
Here are some examples of using pattern matching in IncludeFolders, ExcludeFolders or PriorityFolders, based on the following tree structure:

+ TopFolderA
  + FolderA
    + SubFolderA
    + SubFolderB
  + FolderB
+ TopFolderB

The following filters will match folders from the above structure:

Filter Matches
FolderA \TopFolderA\FolderA, \TopFolderB\FolderA
Folder* \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
FolderA\*Folder* \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB
\*FolderA\* \TopFolderA, \TopFolderA\FolderA, \TopFolderA\FolderB, \TopFolderA\FolderA\SubFolderA, \TopFolderA\FolderA\SubFolderB, \TopFolderB\FolderA
\*\FolderA \TopFolderA\FolderA, \TopFolderB\FolderA

Usage
So for example, suppose you want to remove  IPM.Note.EnterpriseVault.Shortcut items from the mailbox of user1 and personal archive when enabled, moving the items to the DeletedItems by Impersonation. In such case, you could use the following cmdlet:

Remove-MessageClassItems.ps1 -Identity user1 -MessageClass IPM.Note.EnterpriseVault.Shortcut -DeleteMode MoveToDeletedItems -Impersonation –Verbose 

Note: Screenshot shows Mailbox parameter, which is per 1.52 renamed to Identity

SampleOutput

Note: By default, Remove-MessageClassItems will only search IPF.Note class folders (i.e. containing mail items), so you’ll only see those being processed. If you want all folders scanned (also classless), use the ScanAllFolders switch.

The script also supports Office 365. For example, to remove all items with ‘Enterprise’ in their message class text, received before 1/1/2014, only from the primary mailbox, excluding the folder Personal, you can use:

$Credentials= Get-Credential
Remove-MessageClassItems.ps1 -Identity olrik@office365tenant.com -DeleteMode MoveToDeletedItems -MessageClass *EnterpriseVault* -Before 1/1/2014 -MailboxOnly -ExcludeFolder Personal -Credentials $Credentials

In case you want to process multiple mailboxes, you can use a CSV file which needs to contain the Identity field. An example of how the CSV could look:

Identity
francis
philip

The cmdlet could then be something like:

Import-CSV users.csv1 | Remove-MessageClassItems.ps1 -MessageClass IPM.Note.EnterpriseVault.Shortcut -DeleteMode HardDelete -Impersonation

You’re feedback is welcomed through the comments; if you got scripting suggestions, please use the contact form.

You can download the script from the Technet Gallery here.

Revision History
See TechNet Gallery page.

New-MoveRequest fails with “Database .. doesn’t exist”

Exchange 2010 LogoA quick write up on something which you might encounter when trying to move mailboxes across forests. Often, operators don’t always have full access permissions in both environments and rely on provided accounts or delegated permissions to execute the move.

Suppose you’re in the target environment and are asked to migrate mailboxes cross-forest from a previous version of Exchange to Exchange 2010. One of the first steps would be to run the Prepare-MoveRequest.ps1 script to prepare the target object and populate it with Exchange-related attributes from the source environment.

You then come at a point in such a migration, after running Active Directory Migration Tool – ADMT and perhaps some additional tweaks, when you need to actually move the user’s mailbox. So, you enter for example:

$cred= Get-Credentials
New-MoveRequest –Identity UserX –RemoteLegacy –TargetDatabase DB1 -RemoteGlobalCatalog SourceDC –RemoteCredential $cred –TargetDeliveryDomain maildomain.com

Mailbox move request are being queued but for some mailboxes you may encounter the following error message: “Database xxxxx-xxxx-xxxx-xxxx-xxxxx doesn’t exist”. That error message may get you worrying, but let’s investigate.

image

We’re in the target environment and the source environment is running a legacy version of Exchange so setting up a remote PowerShell session and using Get-MailboxDatabase <GUID> is not an option. All roads lead to Rome when querying Active Directory, but in this case let’s load up our trusty LDP to look up that GUID; it should map to a database in the source environment.

In LDP, use Connection > Connect and enter the name of a remote domain controller, like the one specified as RemoteGlobalCatalog, in our example RemoteDC. Then, use Bind providing the credentials you used with RemoteCredential.

Next, select View > Tree and use <GUID=xxxxx-xxxx-xxxx-xxxx-xxxxx> as BaseDN (replace those x’s with the GUID reported by the failing New-MoveRequest and make sure you include those less and greater than symbols).

CantMoveLookup

Now, normally the database object with that GUID and its properties will be returned. If not, something is probably wrong with the permissions on the mailbox, directly or indirectly. Depending on whether you’re able to migrate other users from the same database, server up to the organization, you can rule out if it’s a per-mailbox issue, database permission issue etc.

This proves that as always, preparation is everything. Therefor, prior to migration be sure to check or correct effective permissions on mailboxes to prevent any surprises in that area when you’re actually migrating.

Note that the permission ACLs on a mailbox needs to be in a so-called non-canonical order. To fix this, you may find the FixMailboxSD utility helpful. More information on the importance of canonical ordering of mailbox permissions in Exchange here.

Fixing Well-Known Folders Troubles (Update)

powershellLast update: Version 1.47, April 17th, 2017

Depending on your migration scenario, you could be exporting and importing PST files when migrating mailbox contents from one Exchange environment to another. Now folders in PST files are just folders, i.e. Inbox is a folder just like any other folder and well-known folders lose their identification. Because of this, you may end up with unexpected additional folders in your mailbox after importing PST files.

For example, when importing a non-US mailbox PST in a mailbox set to en-US you may end up with well-known folders and their regional counterparts, e.g. “Inbox” and “Postvak In” or “Calendar” and “Agenda”.

image

It can also happen that you import the PST file before setting the regional settings of the mailbox. In that case the folders with the local names are already present and Exchange will generate sequence number folders, e.g. Inbox1 or Calendar1.

image

Note that the above behavior goes for all well-known folders, such as Inbox, Calendar, Contacts and Sent Items. Users logging in early, i.e. logging on to their mailbox while they shouldn’t, can also create these situations.

One way to fix this situation is to manually move contents to the proper location using an e-mail client, which is tedious and something you don’t want to trouble end users or admins with. Another way is to set the mailbox’ regional configuration before importing the PST file.

A better way is of course to have an Exchange Management Shell script which talks to Exchange Web Services, essentially doing the same but in an automated kind of fashion. Here’s where the Fix-MailboxFolder.ps1 script comes into play. Using Fix-MailboxFolders.ps1 requires Exchange 2010 SP1 and Exchange Web Services 1.2 (or later), which you can download here. The script hasn’t been tested yet against Exchange 2013.

Since the script will process user mailboxes, it needs to be run by a user with full mailbox permissions or impersonation permissions. Granting full mailbox access can be done on several levels, e.g. mailbox or database. For example, to grant ExAdmin Full Access on a mailbox:

Add-MailboxPermission –Identity User –User ExAdmin –AccessRight FullAccess

However, a more elegant and preferred method is to utilize impersonation through Role-Based Access Control (RBAC). Information on configuring impersonation in your Exchange environment using RBAC is found here. As an example, to grant ExAdmin permissions to impersonate all(!) users in an organization, use the following cmdlets:

New-ManagementRoleAssignment –Name:impersonationAssignmentName –Role:ApplicationImpersonation –User:ExAdmin

After executing this cmdlet, the account ExAdmin can succesfully impersonate mail-enabled users. To limit impersonation to a certain subset of the user population, you could use a write scope; for more information on RBAC and write scopes, check out my past blog on this topic here.

Fix-MailboxFolders.ps1 uses the following syntax:

Fix-MailboxFolders.ps1 [-Mailbox]  [[-Language] ] [[-FromLanguage] ] [[-Server] ] [-ScanNumericals] [-Impersonation] []

A quick walk-through on the parameters and switches:

  • Mailbox is the name or e-mail address of the mailbox of which to fix the folder structure. If you specify a mailbox, the current Active Directory domain is consulted. When specifying an e-mail address, Autodiscover is used.
  • Language is the language to configure. If this isn’t specified, the script will use a default value of en-US;
  • FromLanguage is the language in which to scan for folders. When omitted, the script will use the currently configured mailbox language;
  • Server is the name of the Client Access Server to access for Exchange Web Services. When omitted, the script will attempt to use Autodiscover;
  • The switch ScanNumericals will tell the script to look for folders in the language specified by FromLanguage with sequence numbers, e.g. Inbox1 or Calendar1;
  • When the Impersonation switch is specified, impersonation will be used for mailbox access. When this switch isn’t used, the script will run in the context of the current user.
  • Credentials can be used to specify the credentials to access the mailbox or impersonator when Impersonation is specified as well.
  • Of the common parameters, only the Verbose switch is currently supported and will tell you exactly what the script is doing

So, for example assume you have a mailbox with en-US folders but also Dutch folders and the proper regional setting is en-US, use the following cmdlet:

.\Fix-MailboxFolders.ps1 –Mailbox Francis -Language nl-NL -FromLanguage us-EN –Impersonation

image

To fix the folders of an Office 365 mailbox, connect remotely to an Exchange Online session, and run the script using the e-mail address of the mailbox, specifying credentials optionally in combination with impersonation with sufficient permissions.

If you want to fix the folders of multiple mailboxes you can use a CSV file. The CSV needs to contain at least the Mailbox, but can optionally settings for Language and FromLanguage since the script will accept both through the pipe.

A sample of how the csv could look:

Mailbox,Language,FromLanguage
francis,en-US,nl-NL
philip,en-US,nl-NL

The cmdlet should then be something like:

Import-Csv .\Users.csv | .\Fix-MailboxFolders.ps1 -Language en-US -ScanNumericals -Impersonation –Verbose

Should you be running this in a resource-forest deployment, the script will by default try to retrieve the e-mail address of the specified mailbox – assuming you didn’t enter an alias instead of an e-mail address – in the current domain. You can override that by modifying the line which reads:

$ADSearch= New-Object DirectoryServices.DirectorySearcher( [ADSI]””)

to

$ADSearch= New-Object DirectoryServices.DirectorySearcher( [ADSI]”dc=user,dc=contoso,dc=com”)

where user.contoso.com is the distinguished name of your account forest.

Be advised that the script currently only contains en-US and nl-NL settings; you need to add additional language settings to the $LanguageInfo structure. The $LanguageInfo structure contains localized names for each of the well-known folders and has the following format.

"en-US"= @{ 
     "Inbox"="Inbox"; 
     "SentItems"="Sent Items"; 
     "Notes"="Notes"; 
     "Drafts"="Drafts"; 
     "DeletedItems"="Deleted Items"; 
     "Outbox"="Outbox"; 
     "Contacts"="Contacts"; 
     "Calendar"="Calendar"; 
     "Tasks"="Tasks"; 
     "JunkEmail"="Junk E-mail"; 
     "Journal"="Journal"; 
     "DateFormat"="M/d/yyyy"; 
     "TimeFormat"="h:mm tt" 
};

To increase readability I’ve created one setting per line in the script; everything could be stored on a single line of course. The values DateFormat and TimeFormat should be provided in a valid format for the related regional setting. Depending on feedback and demand, I can add additional languages to future versions of the script.

I will gladly take feedback on the script in the comments or through the contact form. If you want to start programming against Exchange Web Services yourself, I would highly recommend Glen Scales blog to look for PowerShell sample code.

You can download the script from the Technet Gallery here.

Retrieving DCs functional capabilities

While constructing a page for the forest and domain functional levels, and the maximum functional level for domain controllers, I wanted to show an example of how to retrieve this Active Directory attribute for all domain controllers. You could for example incorporate this in your automated procedures to check for any Windows 2003 servers and take further actions when needed.

The script is below. It outputs object so you use the pipe for further processing. For information on the possible values for msDS-Behavior-Version check out the new AD Functional Levels page here.

#--------------------------------------------------------------------------------
# Name         : Get-DCMSDSBehaviorVersion.ps1
# Created By   : Michel de Rooij
# E-mail       : michel@eightwone.com
# Date         : 20120307
# Source       : http://eightwone.com
# Version      : 1.0
#
# THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK
# OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE USER.
#--------------------------------------------------------------------------------

$RootDSE= [ADSI]'LDAP://RootDSE'
$objSearchRoot= New-Object DirectoryServices.DirectoryEntry( "LDAP://"+ $RootDSE.configurationNamingContext)
$objSearch= New-Object DirectoryServices.DirectorySearcher
$objSearch.SearchRoot= $objSearchRoot
$objSearch.Filter= "(objectClass=nTDSDSA)"
$objSearch.SearchScope = "SubTree"
$Results= @()
$objSearch.FindAll() | ForEach {
    $objItem = $_.getDirectoryEntry()
    $obj= New-Object PSObject -Property @{
        Servername= $objParent = $objItem.psbase.parent.DNSHostName.ToString()
        MSDSBehaviorVersion= $objItem.Get("MSDS-Behavior-Version")
    }
    $Results+= $obj
}
$Results

Note that if you have over 1000 domain controllers, you need to set the $objSearch.Pagesize, e.g. 1000. It may also encounter problems in forests with multiple roots.