Category Archives: Exchange Server

Exchange Updates – September 2021

Posted on by
5

The Exchange Team released the quarterly Cumulative Updates for Exchange Server 2019 as well as Exchange 2016.

Be advised that these CUs will introduce something which is called the Exchange Emergency Mitigation Service. This service is designed to distribute and implement mitigations addressing potential threats. For this, the URL Rewrite Module needs to be installed on the Exchange server. When you have Exchange running on Windows Server 2012 R2, you will also need an update for the Universal C Runtime (KB2999226). Periodically, the EEM service will reach out to the Office Config Service (OCS) through endpoint https://officeclient.microsoft.com, and update its set of configured mitigations. More on EEM and managing its configuration here.

Links to the updates as well as a description of changes and fixes are described below. The column Schema and AD indicate if the CU contains Schema (/PrepareSchema) and Active Directory (PrepareAD) changes compared to the previous CU. Refer to the Exchange Schema page for schema and related versioning information.

VersionBuildKBDownloadUMLPSchemaAD
Exchange 2019 CU1115.2.986.5KB5005334Download NY
Exchange 2016 CU2215.1.2375.7KB5005333DownloadUMLPNY

Exchange 2019 CU11 fixes:

  • 5006980 Bad signature error using PerfView in Exchange Server 2019 and 2016 (KB5006980)
  • 5006982 On-premises Exchange queues back up because of incorrect default value (KB5006982)
  • 5006983 Exchange Server 2019 and 2016 certificates created during setup use SHA-1 hash (KB5006983)
  • 5006984 PrepareAD fails if Computers container or RODCs are renamed or moved in Exchange Server 2019 and 2016 (KB5006984)
  • 5006986 Opening an Outlook message from the desktop removes line spacing (KB5006986)
  • 5006988 Export of .pst file is unexpectedly triggered again in Exchange Server 2019 and 2016 (KB5006988)
  • 5006989 Accepted domains with wildcards for subdomains are not honored when Edge server maps AddressSpaces (KB5006989)
  • 5006990 Exchange CU installation fails after you configure fallback to use default character set (5006990)
  • 5006991 Mail quota warning messages no longer sent daily in Exchange Server 2019 (KB5006991)
  • 5006992 No room lists found when trying to add a room in OWA in Exchange Server 2019 or 2016 (KB5006992)
  • 5006993 Can’t log on to OWA in Chrome if SSL is offloaded in Exchange Server 2019 and 2016 (KB5006993)
  • 5006994 BCC values not retained in Sent Items in a shared mailbox in Exchange Server 2019 and 2016 (5006994)
  • 5006995 Korean email messages display some recipients incorrectly in Exchange Server 2019 and 2016 (KB5006995)
  • 5006996 Export-AutoDiscoverConfig exposes admin password and does not work against domain controllers that require signing (KB5006997)
  • 5006997 Korean messages in OWA display “From” as “Start date” after you filter the list in Exchange Server 2019 and 2016
  • 5006999 “401” error and Outlook repeatedly prompts for credentials in Exchange Server 2019 (KB5006999)
  • 5007042 Error window appears when you view features in OWA Virtual Directory (KB5007042)
  • 5007043 Exchange Server SU updates Add/Remove Programs incorrectly (KB5007043)
  • 5007044 Start-MailboxAssistant not available in EMS in Exchange Server 2019 (KB5007044)

Exchange 2016 CU22 fixes:

  • 5006980 Bad signature error using PerfView in Exchange Server 2019 and 2016 (KB5006980)
  • 5006982 On-premises Exchange queues back up because of incorrect default value (KB5006982)
  • 5006983 Exchange Server 2019 and 2016 certificates created during setup use SHA-1 hash (KB5006983)
  • 5006984 PrepareAD fails if Computers container or RODCs are renamed or moved in Exchange Server 2019 and 2016 (KB5006984)
  • 5006986 Opening an Outlook message from the desktop removes line spacing (KB5006986)
  • 5006988 Export of .pst file is unexpectedly triggered again in Exchange Server 2019 and 2016 (KB5006988)
  • 5006989 Accepted domains with wildcards for subdomains are not honored when Edge server maps AddressSpaces (KB5006989)
  • 5006992 No room lists found when trying to add a room in OWA in Exchange Server 2019 or 2016 (KB5006992)
  • 5006993 Can’t log on to OWA in Chrome if SSL is offloaded in Exchange Server 2019 and 2016 (KB5006993)
  • 5006994 BCC values not retained in Sent Items in a shared mailbox in Exchange Server 2019 and 2016 (5006994)
  • 5006995 Korean email messages display some recipients incorrectly in Exchange Server 2019 and 2016 (KB5006995)
  • 5006996 Export-AutoDiscoverConfig exposes admin password and does not work against domain controllers that require signing (KB5006997)
  • 5006997 Korean messages in OWA display “From” as “Start date” after you filter the list in Exchange Server 2019 and 2016
  • 5007042 Error window appears when you view features in OWA Virtual Directory (KB5007042)
  • 5007043 Exchange Server SU updates Add/Remove Programs incorrectly (KB5007043)

Notes:

  • If these Cumulative Updates contain schema changes compared to the Cumulative Update you have deployed, you need to run Setup with /PrepareSchema. If they contain Active Directory changes, you need to run /PrepareAD. Alternatively, permissions permitting, you can let Setup perform this step. Consult the Exchange schema versions page for schema and related versioning information.
  • When upgrading from an n-2 or earlier version of Exchange, or an early version of the .NET Framework, consult Upgrade Paths for CU’s & .NET.
  • Don’t forget to put the Exchange server in maintenance mode prior to updating. Regardless, setup will put the server in server-wide offline mode post-analysis, before making actual changes.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), support requires you to trail at most one version (n-1).
  • If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
  • Cumulative Updates can be installed directly; no need to install RTM prior to installing Cumulative Updates.
  • Once upgraded, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
  • The recommended upgrade order is internet-facing, non-internet-facing servers first, followed by Edge Transports.

Caution:

As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or forums for any issues.

Unarchiving Mailbox Items

Posted on by
70

With the introduction of Exchange 2010 at the end of 2009, a native feature was added to Exchange Server for which organizations required 3rd party products before that. The feature which I am talking about is Exchange’s Personal Archives, Online Archives, or In-Place Archiving as it is called nowadays.

Background
Archives were introduced at a time when Office 365 was in its early days, many organizations were running Exchange on-premises with mailbox quotas as bandwidth and storage were limited or relatively expensive. It was up to end users to make sure their mailbox remained within its limits, either by removing either old items, large items or just move them out of their mailbox to those pesky .PST files.

Archives introduced benefits such as lowering disk footprint by taking infrequently used items out of the primary mailbox (which then could only synchronize in full) to the archive, which is basically an additional mailbox for long-term storage. Exchange’s built-in Messaging Records Management (MRM) through retention policies and tags can be used for automatic moving of older items to the archive.

Archives also come with few downsides, especially in the early days. Most notably are perhaps clients not supporting archives at all, or searches not spanning both mailbox and archive. Also, and this is not to be underestimated, end users do not always grasp the concept of archives and the impact on the tasks and tools they use. It’s not uncommon to see people panicking about “missing data” in service tickets, only to discover their “missing data” was moved to their archive by the company retention policy after some digging.

In recent years, I have seen archives becoming less relevant, and organizations adopting the large mailbox concept in favor of lean and mean mailboxes with archives. There are still exceptions of course, usually in the form of substantial – usually shared – mailboxes. For those, staying with Exchange Online archives – and when needed auto-expanding archives – is usually still an option due to the different type of mailbox interaction, or to circumvent Exchange’s storage limitations or Outlook for Desktop’s synchronizing of offline cache files before issues might be seen. The maximum number of items per folder is such a limit, however these have been raised or done away with in recent years. Non-stubbing 3rd party archive solutions taking data out of Exchange can also be a option.

The Problem
Switching to the large mailbox concept creates a problem for those organizations that have already enabled in-place archives for their end users: How to get that data back from those archives to the primary mailbox. While retention policies can move data in opposite direction, there is no such thing as a reverse-retention policy. Also, not every organization would like to instruct end users to unarchive this contents themselves, as it is prone to failure, blocks Outlook for Desktop from doing anything else and might result in abandoned operations which limits future actions as moves are still happening in the background.

When investigating a possible solution I found that there is no other way to accomplish this, than to programmatically move contents from the in-place archive to the primary mailbox. While there is a ‘archive’ operation for mailbox items (which moves it to the assigned Archive folder, not the in-place archive) there is no other single API call to perform this task. Also, the solution would have to use Exchange Web Services, as a limitation in Microsoft Graph makes it incapable of moving messages between multiple mailboxes.

Note: If I overlooked something in this area, please let me know.

Solution
To help organizations accomplish this task, I wrote a PowerShell script which requires the following:

  • Exchange Server 2013 SP1 or later, or Exchange Online.
  • Exchange Web Services (EWS) Managed API 2.21 or later (how to, NuGet package exchange.webservices.managed.api).
  • When using OAuth, the MSAL library is required (NuGet package Microsoft.Identity.Client). Also, you need to have registered an App in Azure Active Directory; the Tenant ID, Application ID and certificate or secret is what you need to provide the script with to operate successfully.
  • In addition to installing the NuGet packages, you can also store the DLLs in the same folder as the script.

Note: Untested with Primary mailboxes on-premises and Exchange Online Archives.

The script Invoke-Unarchive will perform the following tasks:

  • Invoke-Unarchive will move contents from the in-place archive back to the primary mailbox.
  • The most optimal operation will be chosen:
    • Folders present in archive but not in primary mailbox will be moved in one operation.
    • Folders present in archive and primary mailbox are merged. Items in those folders are moved in batches.
    • The same steps are repeated recursively per folder for the whole archive.
  • If, after moving, a folder in the archive is empty, and it is not a non-removable well-known folder, it will be removed.
  • Optionally, Invoke-Unarchive can also move contents stored in the Recoverable Items from the archive to the primary mailbox.
  • Invoke-Unarchive will handle throttling, either by honoring the returned back-off period or by adding delays between operations.
  • Moving items is asynchronous, and Invoke-Unarchive needs to wait for Exchange to complete the previous move to folder X before it can move the next set of items to folder X.

Do not forget to reassign retention policies causing archival, or you might have the run the script again at later moment.

Syntax
The parameters to call Invoke-Unarchive.ps1 are:

  • Identity to specify one or more mailboxes to unarchive items for.
  • Server to specify the FQDN of the Client Access Server to use. When omitted, Autodiscover will be used.
  • IncludeRecoverableItems to instruct the script to process deletions stored in the Recoverable Items as well.
  • Impersonation to use impersonation when accessing the mailbox. When using modern authentication (OAuth), impersonation is mandatory.
  • Force to force moving of items without prompting.
  • NoProgressBar to prevent progress status.
  • TrustAll to accept all certificates including self-signed certificates.
  • TenantId specifies the ID of the Tenant when using a mailbox hosted in Exchange Online.
  • ClientId to specify the Application ID of the registered application in Azure Active Directory.
  • Credentials to specify the Basic Authentication credentials for on-premises usage or against Exchange Online when OAuth is not an option.
  • CertificateThumbprint is the thumbprint of the certificate to use for OAuth. The certificate with the public key needs to stored with the registered application for authentication. The certificate with the private key should be present in the local certificate store.
  • CertificateFile and CertificatePassword to specify the file of the certificate to use. The file shoud contain the private key, and the password to unlock the file can be specified using CertificatePassword.
  • Secret can be used to specify the secret to authenticate using the registered application.

Note that Credentials, CertificateThumbprint, CertificateFile + CertificatePassword and Secret are mutually exclusive.

Example
Below shows an example run against a test-mailbox using modern authentication (OAuth). The common parameter Verbose is used to display additional output.

.\Invoke-Unarchive.ps1 -Identity michel@myexchangelabs.com -Server outlook.office365.com -Impersonation -Secret <Secret> -TenantId <Tenant> -ClientId <AppId> -Verbose
image

You can find the script on GitHub here.

Final Notes
The EWS operation – especially moving items – is not necessarily slow, but against Exchange Online processing large archives can take considerable amount of time due to throttling. When moving a significant number of items using Outlook for Desktop, you will likely run into Outlook abandoning the operation after which you need to wait for Exchange to finish pending moves before you can continue with this task. Using the script, you can take away this unarchiving task from end users by running the operation in the background in one or multiple runs.

Security Updates Exchange 2013-2019 (Jul2021)

Posted on by
6

Update July 20th: Added VC++2012 requirement to tip on running MT to prepare Exchange 2013 schema separately.

Another month, another Patch Tuesday! A quick blog on the July’s security updates for Exchange Server 2013 up to 2019.

The vulnerabilities addressed in these security updates are:

VulnerabilityCategorySeverityRating
CVE-2021-31196Remote Code Execution ImportantCVSS:3.0 7.2 / 6.3
CVE-2021-34470Elevation of PrivilegeImportantCVSS:3.0 8.0 / 7.0
CVE-2021-33768Elevation of PrivilegeImportantCVSS:3.0 8.0 / 7.0
CVE-2021-31206Remote Code ExecutionImportantCVSS:3.0 7.6 / 7.1

Note:

  • When looking at the MSRC information, you will notice 3 additional CVE issues addressed for July 13th. However, as far as I can see CVE-2021-34473, CVE-2021-34523 and CVE-2021-33766 were addressed in the April 2021 and eventually the May 2021 Security Updates, which also would explain MSRC’s mention of earlier CUs, such as Exchange 2019 CU8.
  • CVE-2021-31206 was the vulnerability discovered at the Pwn2Own 2021 contest.

Vulnerabilities mentioned in the table above are addressed in the following security updates:

ExchangeDownloadBuildKBSupersedes
Exchange 2019 CU10Download15.2.922.13KB5004780
Exchange 2019 CU9Download15.2.858.15KB5004780
Exchange 2016 CU21Download15.1.2308.14KB5004779
Exchange 2016 CU20Download15.1.2242.12KB5004779
Exchange 2013 CU23Download15.0.1497.23KB5004778

Notes:

  • CVE-2021-33768 does not seem applicable to Exchange 2019 CU9 or Exchange 2016 CU20.
  • CVE-2021-34470 is only addressed in the security update for Exchange 2013 CU23.

More detailed information can be found at the original blog post here, which mentions some specific post-deployment instructions:

  • When running n-1 CU of Exchange 2019 (CU9) or Exchange 2016 (CU20), and you do not plan to upgrade to the latest CU yet but do wish to install this Security Update, you must also update the AD Schema using the CU10 or CU21 installation files.
  • When you are running Exchange 2013 CU23 in your organization, and no later Exchange builds are present, you need to deploy a schema update immediately after deploying the Security Update. After deploying the SU, from an elevated CMD prompt, run Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms from Exchange’s bin folder. You you need to separate the update from deploying the update, see end of article for a tip.

The blog also mentions some issues, which are identical to the ones mentioned with the May 2021 Security Updates:

  • Accounts ending in ‘$’ cannot use EMS or access the ECP.
  • Cross-forest Free/Busy might stop working resulting in 400 Bad Request (solution).
  • Running cmdlets against EMC using invoked runspace might result in no-language mode error (info).

Be advised that these security updates are Cumulative Update level specific. You cannot apply the update for Exchange 2019 CU9 to Exchange 2019 CU8. Also, the security update download has the same name for different Cumulative Updates, and I would suggest tagging the file name with the CU level, e.g. Exchange2019-CU9-KBXXXXXX-x64-en.msp.

On another note, after deploying the security updates Exchange will start reporting its version number in the HTTP response header.

As a reminder, run the Security Update from an elevated command prompt to prevent issues during installation. In other words: Do not just double-click on the .MSP file. And on a final note, as with any patch or update, I’d recommend to apply this in a acceptance environment first, prior to implementing it in production. However, it is not recommended to wait for regular maintenance cycles when it concerns security updates, and follow a more agile approach. The rating implies a form of urgency.

OWA/ECP and HMAC errors
There are reports of the Security Update breaking OWA/ECP. Symptoms are browsers displaying an HMAC error:

Server Error in '/owa' Application.

ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
    
Exception Details: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

It is likely related to “Microsoft Exchange Server Auth Certificate”, which can be expired, invalid or for other reasons not being picked up. The reported solution is renewing the “Microsoft Exchange Server Auth Certificate”. This procedure can be found here. Do note that it may take an hour for the certificate to become effective. Meanwhile, you can check the comments in the original Exchange Team post, which is lively with feedback and responses.

Exchange 2013 CU23 SU & Schema Updating
Because with Exchange 2013 CU23 schema preparation needs to occur immediately after deploying the SU on (the first) Exchange 2013 CU23 server, a tip might be that you could deploy Exchange 2013 CU23 Management Tools on a workstation, install the SU on that workstation, then run the PrepareSchema from there before deploying the SU on any Exchange 2013 CU23 server.

This might also be helpful in multi-domain organizations, or organizations where AD and Exchange are managed by different teams or require separate changes. Note that performing the schema update this way requires Visual C++ 2012 Runtime, otherwise you will run into a “Exchange Server setup didn’t complete the operation” and the ExchangeSetup.log will contain “Could not load file or assembly ‘Microsoft.Exchange.CabUtility.dll”.

Exchange Updates – June 2021

Posted on by
Reply

The Exchange Team released the quarterly Cumulative Updates for Exchange Server 2019 and Exchange 2016. Biggest change for both editions is support for Anti-Malware Scan Interface (AMSI) integration, available on Windows Server 2016 and Windows Server 2019. It allows real-time scanning of HTTP payloads, blocking known malicious content before it reaches Exchange.

Links to the updates as well as a description of changes and fixes are described below. The column Schema and AD indicate if the CU contains Schema (/PrepareSchema) and Active Directory (PrepareAD) changes compared to the previous CU.

VersionBuildKBDownloadUMLPSchemaAD
Exchange 2019 CU1015.2.922.7KB5003612 Download YY
Exchange 2016 CU21 15.1.2308.8 KB5003611 DownloadUMLPYY

Exchange 2019 CU10 fixes:

  • 5004612 Message body not displayed in OWA if the message was added in Outlook to a new mailbox
  • 5004613 OutOfMemory exception when moving a public folder that has a large ICS sync state
  • 5004614 Korean text is garbled in calendar invitation to a user with a Chinese display name
  • 5004615 “InvalidOperationException” and Store Worker process crashes during mailbox move
  • 5004616 Changing the email address in EAC doesn’t work in modern browsers
  • 5004617 TLS 1.2 is not set as default after you install Exchange 2019 with Edge Transport role
  • 5004618 MSExchangeMailboxAssistants 4999 Crash in ELCAssistant.InvokeInternalAssistant with System.NullReferenceException
  • 5004619 Mailbox creation through ECP fails after installing Exchange Server 2019 or 2016 April update
  • 5004622 “Cannot Send Mail – Your mailbox is full” error when you use iPhone mail to send very large attachments
  • 5004623 PrepareADSchema required because of Active Directory schema change

Exchange 2016 CU21 fixes:

  • 5004612 Message body not displayed in OWA if the message was added in Outlook to a new mailbox
  • 5004613 OutOfMemory exception when moving a public folder that has a large ICS sync state
  • 5004614 Korean text is garbled in calendar invitation to a user with a Chinese display name
  • 5004615 “InvalidOperationException” and Store Worker process crashes during mailbox move
  • 5004616 Changing the email address in EAC doesn’t work in modern browsers
  • 5004618 MSExchangeMailboxAssistants 4999 Crash in ELCAssistant.InvokeInternalAssistant with System.NullReferenceException
  • 5004619 Mailbox creation through ECP fails after installing Exchange Server 2019 or 2016 April update
  • 5004622 “Cannot Send Mail – Your mailbox is full” error when you use iPhone mail to send very large attachments
  • 5004623 PrepareADSchema required because of Active Directory schema change
  • 5004629 No version updating after you install Exchange Server 2016

Notes:

  • If these Cumulative Updates contain schema changes compared to the Cumulative Update you have deployed, you need to run Setup with /PrepareSchema. If they contain Active Directory changes, you need to run /PrepareAD. Alternatively, permissions permitting, you can let Setup perform this step. Consult the Exchange schema versions page for version number comparison.
  • When upgrading from an n-2 or earlier version of Exchange, or an early version of the .NET Framework, consult Upgrade Paths for CU’s & .NET.
  • Don’t forget to put the Exchange server in maintenance mode prior to updating. Regardless, setup will put the server in server-wide offline mode post-analysis, before making actual changes.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), support requires you to trail at most one version (n-1).
  • If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
  • Cumulative Updates can be installed directly; no need to install RTM prior to installing Cumulative Updates.
  • Once upgraded, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
  • The recommended upgrade order is internet-facing, non-internet-facing servers first, followed by Edge Transports.

Caution:

As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or forums for any issues.

Security Updates Exchange 2013-2019 (May2021)

Posted on by
Reply

Another month, another Patch Tuesday! A quick blog on May’s security updates for Exchange Server 2013 up to 2019.

These fixes address the following vulnerabilities:

VulnerabilityCategorySeverityRating
CVE-2021-31209SpoofingImportantCVSS:3.0 6.5 / 5.7
CVE-2021-31207Security Feature BypassModerateCVSS:3.0 6.6 / 5.8
CVE-2021-31198Remote Code ExecutionImportantCVSS:3.0 7.8 / 6.8
CVE-2021-31195Remote Code Execution ImportantCVSS:3.0 6.5 / 5.7

These vulnerabilities can be fixed by single security update for Exchange, which you can find below:

ExchangeDownloadBuildKBSupersedes
Exchange 2019 CU9Download15.2.858.12KB5003435KB5001779
Exchange 2019 CU8Download15.2.792.15KB5003435KB5001779
Exchange 2016 CU20Download15.1.2242.10KB5003435KB5001779
Exchange 2016 CU19Download15.1.2176.14KB5003435KB5001779
Exchange 2013 CU23Download15.0.1497.18KB5003435KB5001779

More detailed information can be found at the original blog post here, which also mentions some known issues and workarounds which you might encounter after deploying these updates.

Be advised that these security updates are Cumulative Update level specific. You cannot apply the update for Exchange 2019 CU9 to Exchange 2019 CU8. Also, the security update download has the same name for different Cumulative Updates, and I would suggest tagging the file name with the CU level, e.g. Exchange2019-CU9-KB5003435-x64-en.msp.

Also, run the Security Update from an elevated command prompt, to prevent issues during installation (other words: Do not just double-click on the .MSP file). And on a final note, as with any patch or update, I’d recommend to apply this in a acceptance environment first, prior to implementing it in production. However, it is not recommended to wait for regular maintenance cycles when it concerns security updates, and follow a more agile approach. The rating implies a form of urgency.