Few days ago, the Exchange Product made several announcements related to Exchange Server and its future. The overall message throughout these announcements can be interpreted as that Microsoft is publicly declaring to be committed to developing and supporting the Exchange Server product. This is especially of interest to those customers running it as part of their on-premises infrastructure. It is also assuring those that believe the road ahead was a dead end, eventually forcing them to move to Exchange Online, or look for alternatives.
The announcements made were in the area of:
Lifecycle policies remain intact for current versions of Exchange Server.
The next version of Exchange Server, also known as Exchange vNext, will move to a continuous support model, but comes with requirements.
Upgrade path for Exchange vNext.
Modern Authentication support for non-hybrid Exchange 2019 deployments.
Exchange 2019 support for TLS 1.3.
Possibility to receive pre-release builds of Exchange server through Microsoft’s TAP program.
Exchange Admin Center will receive overview section for Exchange servers update status in Exchange hybrid deployments.
HCW will allow admins to skip configuration steps.
Script to remove obsolete mitigations from EEMS.
Microsoft Exchange Conference Community Virtual Airlift (MEC) for September 13-14! (register)
Updated: 1.2 adds default ExchangeOnlineManagement cmdlets scanning and authentication options.
Since the original announcement on deprecation of Basic Authentication, organizations had time to analyze their environment which may include Exchange-related procedures and tools. These usually also contain scripts or commands, which depend on the Exchange Online Management module. A previous blog on its history and how version 2 of this module lends itself for unattended operation with certificate-based modern authentication support can be found here.
The initial release of the Exchange Online Management v2 – or EXOv2 – module offered a an additional small set of cmdlets which utilized REST-based services. Apart from the functional discrepancies, such as having to specify a property set to indicate which properties to return, the big advantage of these added commands was that they did not depend on the Windows Remote Management (WinRM) client using Basic Authentication for token exchange. Disabling Basic Authentication on WinRM client lead to messages such as:
Connecting to remote server outlook.office365.com failed with the following error message : The WinRM client cannot process the request. Basic authentication is currently disabled in the client configuration.
This dependency makes it challenging for organizations to turn off Basic Authentication altogether, or lead to problems when they did. Fast forward to the present, where the Exchange Online Management module in its current release is offering nearly all Exchange cmdlets in REST-based form, with full functional parity.
While I expect Microsoft to reach full command parity before they flick the Basic Authentication switch to off, there are also other use cases for which analyzing scripts might be helpful:
Ths initial purpose was identifying commands which require RPS (Remote PowerShell), and thus thus require WinRM Basic Authentication enabled. Because the Exchange Team did an amazing job in catching up in the recent months, only few Exchange Online cmdlets are still lacking REST support in my tenant at this moment, e.g. New-ApplicationAccessPolicy. But then again, your mileage may vary, as the recent Preview 7 module removed few UnifiedGroup related cmdlets which had issues.
New Exchange Online commands may not receive immediate REST support.
Organizations might want to cross-reference commands with scripts.
Identifying Exchange Online commands and parameters in scripts helps in determining the minimum set of permissions required to run the script.
To analyze and report on Exchange Online scripts, I created a simple script Analyze-ExoScript.ps1. This script, which is available on GitHub here, does the following:
Connect to Exchange Online using RPS and inventory the commands available. Note that this requires the UseRPSSession switch when connecting, which is only available per 2.0.6-Preview3 of the module. If your organization only runs GA versions of the module, this script cannot be used.
Connect to Exchange Online using REST and inventory the commands available. It will re-use the account used for authenticating the RPS session, which should prevent receiving another authentication dialog or MFA challenge.
Cache cmdlet information in an external file to prevent having to connect to Exchange Online for every run. The file is named EXO-CmdletInfo.xml and will be stored in the same folder as the script.
Process the script and report on the Exchange-related commands used.
Usage Calling Analyze-ExoScript is straightforward:
File is the name of one or more files which you want to analyze. Note that the script accepted pipeline output, so you can also feed it filenames using Get-ChildItem for example.
The ShowAll switch tells the script to output all found commands, not only the Exchange ones.
The switch Refresh tells the script to ignore saved command information, trigger reconnecting to Exchange Online in order to refresh the command sets.
Credential specifies the (Basic Authentication) credential to pass to Connect-ExchangeOnline.
Organization and AppId can be used to specify the tenant ID (x.onmicrosoft.com) and registered application ID to use with Connect-ExchangeOnline using Modern Authentication. This also requires one of the following:
CertificateThumbprint of the certificate to use for authentication.
CertificateFile of the file containing the certificate to use, together with CertificatePassword to specify its password.
When asked to authenticate, make sure your role has the necessary Exchange-related permissions as that will determine the Exchange Online cmdlets available to you, and consequently also the commands which Analyze-ExoScript will recognize in scripts to process.
For example, to process a script Fix-MailboxFolders.ps1, use:
The output consists of objects, which allow for further filtering:
The returned properties are:
Command is the Exchange Online command identified
Type will tell you if the command supports REST or requires RPS.
Parameters are the parameters used together with the command. This includes common parameters, which might be less usable for role assignment purposes.
Alt contains alternative REST-based cmdlet you could consider using for performance reasons, e.g. Get-EXOMailbox instead of Get-Mailbox.
File and Line are the file containing the command and on which line it is located.
AST To analyze code, I leveraged PowerShell feature called Abstract Syntax Tree, which was an interesting exploration in itself. PowerShell AST can be used to decompose PowerShell code into tokens. This is way better than simply looking for strings, and does away with having to interpret code yourself to see if something is a command, comment or just some string. AST allows for analysis of these tokens, in this case filtering on commands which are related to Exchange Online. If you want to get started on AST, check out this article, or plunge in the PowerShell SDK straightaway.
Final Words When every Exchange Online command discovered is found to be offering REST support, you can turn off Basic Authentication on the client, for example through GPO or by reconfiguring WinRM:
winrm set winrm/config/client/auth @{Basic="false"}
Only thing you might need to refactor is if and how the script connects to Exchange Online, as Basic Authentication allowed for connecting to Exchange Online using (stored) credentials for example. Examples on how to use more secure Modern Authentication-based methods to connect can be found in an earlier article here.
The Exchange PG released May updates for Exchange Server 2013, 2016 and 2019.
Note that per this cycle, Security Updates will be packaged in an executable wrapper. This should trigger the running elevated prompt, thus preventing any potential issues from simply double-clicking the .MSP file. More about the new package format, options for logging and command-line switches are mentioned in an article dedicated to the change of distribution method here.
The vulnerability addressed in the Security Updates for May is:
KB5013118 Exchange Service Host service fails after installing March 2022 security update
Important: As mentioned in the announcement, you must run /PrepareAllDomains after deploying the SU because of hardening measures. Exception is when you have multiple domains and some of them are never prepped; in that case prepare the individual domains required. Using your currently deployed binaries, run the following command, where the /IAccept switch you need to use depends on the Exchange version deployed and whether you provide diagnostics information:
Be advised that these security updates are Cumulative Update level specific. You cannot apply the update for Exchange 2019 CU12 to Exchange 2019 CU11. Also, the security update download has the same name for different Cumulative Updates, and I would suggest tagging the file name with the CU level, e.g. Exchange2019-CU10-KBXXXXXX-x64-en.msp.
Exchange servers running as part of hybrid deployment are running services, and thus need to be included in the patch cycle. If you are running Exchange 2019 CU12 Management Tools-only (for recipient management), you do not need to deploy this SU.
On a final note, as with any patch or update, I’d recommend to apply this in a test environment first, prior to implementing it in production. However, it is not recommended to wait for regular maintenance cycles when it concerns security updates, and follow a more agile approach; the ratings are an indication of the urgency.
In the announcement of the most recent set of Cumulative Updates for Exchange Server 2019 and 2016, Microsoft introduced some changes – features if you will – as well, which were received with enthusiasm. An overview of these Cumulative Updates and the features introduced was given in an earlier article. In this article however, I would like to zoom in on one of those features, which also happens to be a popular topic among customers running Exchange Hybrid deployments, “The Last Exchange Server”.
Up to Exchange 2019 CU12 (2022 H1), customers that migrated to Exchange Online were still required to leave Exchange-related components running on-premises. Even today, with all the information published around this topic, I am surprised this still surprised customers. This Exchange server running on-premises is to be used for managing recipients which have their source of authority in Active Directory, leveraging Active Directory Connect to propagate objects to Azure Active Directory and thus Exchange Online. Also, when there is a need to relay messages from applications or multi-functional devices, customers often need to have an Exchange server on-premises to accept these messages, as Exchange is the only supported mail relay product for hybrid deployments.
The Exchange Team released the quarterly half-yearly Cumulative Updates for Exchange Server 2019 and Exchange 2016. You read that right, half-yearly updates are replacing the cadence of quarterly update servicing model for Exchange Server. Effectively, this will be Exchange 2019 only, as Exchange 2016 will be out of mainstream support in H2 of 2022, and will therefor only receive Security Updates after this round. Note that this change also alters the effective ‘current’ state (n-1 or later) of your Exchange Server environment from half year to one year.
And that’s not the only good news that comes with these sets of updates. In short:
If you run Exchange 2019 in Hybrid only for the purpose of managing recipients, you can now use Exchange 2019 CU12’s Exchange Management Tools to accomplish this; no more need to have an Exchange server running just for this. More details here.
Exchange 2019 CU12 will reintroduce the Hybrid Key option. Its Hybrid Configuration Wizard supports this licensing method.
Exchange 2019 CU12 support managing the Hybrid Agent with MFA-enabled accounts.
Exchange 2019 CU12 adds support for Windows Server 2022, both for its underlying operating system, as well as deployment in environments running Windows Server 2022 Domain Controllers.
Note that while Windows Server 2022 supports TLS 1.3, Exchange 2019 CU12 on WS2022 does not yet support it. Adding support is scheduled for somewhere next year.
The supportability matrix has been updated for the supported Windows Server 2022 scenarios.
Exchange Server is now also part of Microsoft’s Bounty Program, which is an indication of continued focus for customers still running Exchange Servers on-premises.
Links to the updates as well as a description of changes and fixes are described below. The column Schema and AD indicate if the CU contains Schema (/PrepareSchema) and Active Directory (PrepareAD) changes compared to the previous CU. Refer to the Exchange Schema page for schema and related versioning information.
Apart from DST changes and the fixes mentioned below, these Cumulative Updates also contain a change which will not allow using UNC paths with several cmdlets. More information about this change and cmdlets affected can be found here: KB5014278.
Exchange 2019 CU12 fixes:
5012757 “Migration user… can’t be found” error when using Start-MigrationUser after batch migration fails
5012758 Start-MailboxAssistant is not available in Exchange Server 2019
5012760 You can’t access OWA or ECP after installing the July 2021 security update
5012761 External attendees see “Send the Response Now” although no response was requested in Exchange Server
5012762 PST creation is unexpectedly triggered again during multiple mailbox export
5012765 Email stuck in queue starting from “2022/1/1 00:01:00 UTC+0” on all Exchange on-premises servers
5012766 Transport Services fail repeatedly because of * Accepted Domain
5012768 Start-MigrationUser and Stop-MigrationUser are unavailable for on-premises Exchange Server 2019 and 2016
5012769 Invalid New Auth Certificate for servers that are not on UTC time zone
5012770 No response from public folder for users migrating to Microsoft Exchange 2019
5012772 Items are skipped at the start of a new search page request
5012773 OWAMailboxPolicy is bypassed and high resolution profile images can be uploaded
5012774 Can’t change default path for Trace log data in Exchange Server 2019 and 2016
5012775 No additional global catalog column in the address book service logs
5012776 Exchange Server 2019 help link in OWA redirects users to online help for Exchange Server 2016
5012777 Can’t find forwarded messages that contain attachments in Exchange Server 2019
5012778 Exchange Server stops responding when processing PDF files with set transport rule
5012779 Invalid new auth certificate for servers that are not on UTC time zone
5012780 Disable-Mailbox does not remove LegacyExchangeDN attribute from on-premises Exchange 2019
5012781 Exchange Server 2019 and 2016 DLP doesn’t detect Chinese resident ID card numbers
5012782 MS ExchangeDiagnostic Service causes errors during service startup and initialization in Microsoft Exchange 2019
5012783 Can’t restore data of a mailbox when LegacyDN is empty in the database
5012784 Exchange 2016 CU21 and Exchange 2019 CU10 cannot save “Custom Attributes” changes in EAC
5012785 Read Only Domain Controllers (RODCs) in other domains do not get desired permissions
5012786 Forwarded meeting appointments are blocked or considered spam
5012787 Download domains created per CVE-2021-1730 don’t support ADFS authentication in OWA
5012789 Can’t use Copy Search Results after eDiscovery & Hold search
5012790 OWA doesn’t remove the “loading” image when a message is opened in Chrome and Edge browsers
5012791 MailboxAuditLog doesn’t work in localized (non-English) environments
Exchange 2016 CU23 fixes:
5012757 “Migration user… can’t be found” error when using Start-MigrationUser after batch migration fails
5012760 You can’t access OWA or ECP after installing the July 2021 security update
5012761 External attendees see “Send the Response Now” although no response was requested in Exchange Server
5012765 Email stuck in queue starting from “2022/1/1 00:01:00 UTC+0” on all Exchange on-premises servers
5012768 Start-MigrationUser and Stop-MigrationUser are unavailable for on-premises Exchange Server 2019 and 2016
5012769 Invalid New Auth Certificate for servers that are not on UTC time zone
5012774 Can’t change default path for Trace log data in Exchange Server 2019 and 2016
5012779 Invalid new auth certificate for servers that are not on UTC time zone
5012780 Disable-Mailbox does not remove LegacyExchangeDN attribute from on-premises Exchange 2019
5012781 Exchange Server 2019 and 2016 DLP doesn’t detect Chinese resident ID card numbers
5012782 MS ExchangeDiagnostic Service causes errors during service startup and initialization in Microsoft Exchange 2019
5012783 Can’t restore data of a mailbox when LegacyDN is empty in the database
5012784 Exchange 2016 CU21 and Exchange 2019 CU10 cannot save “Custom Attributes” changes in EAC
5012786 Forwarded meeting appointments are blocked or considered spam
5012787 Download domains created per CVE-2021-1730 don’t support ADFS authentication in OWA
5012789 Can’t use Copy Search Results after eDiscovery & Hold search
5012791 MailboxAuditLog doesn’t work in localized (non-English) environments
5012829 Group metrics generation fails in multidomain environment
Notes:
If these Cumulative Updates contain schema changes compared to the Cumulative Update you currently have deployed, you need to run Setup with /PrepareSchema. If they contain Active Directory changes, you need to run /PrepareAD. Alternatively, permissions permitting, you can let Setup perform this step. Consult the Exchange schema versions page for schema and related versioning information.
When upgrading from an n-2 or earlier version of Exchange, or an early version of the .NET Framework, consult Upgrade Paths for CU’s & .NET.
Don’t forget to put the Exchange server in maintenance mode prior to updating. Regardless, setup will put the server in server-wide offline mode post-analysis, before making actual changes.
When using Exchange hybrid deployments or Exchange Online Archiving (EOA), support requires you to trail at most one version (n-1).
If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
Cumulative Updates can be installed directly; no need to install RTM prior to installing Cumulative Updates.
Once upgraded, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
The recommended upgrade order is internet-facing, non-internet-facing servers first, followed by Edge Transports.
Caution:
As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or forums for any issues.
Back in September 2019, Microsoft announced it would start to turn off Basic Authentication for non-SMTP protocols in Exchange Online on tenants where the authentication protocol was detected as inactive. This is part of an overall movement to deprecate the less secure Basic Authentication, which is unfit to face the security challenges of the modern world, being subject to things like password spray attacks. It’s modern successor, modern authentication or OAuth2, uses a token and claim based mechanism contrary to sending accounts and passwords, and is the preferred authentication method. When combined with Azure AD for authentication, Modern Authentication also supports features such as Multi-Factor Authentication or Conditional Access.
The original date for disabling of Basic Authentication was October 13th, 2020. Then the world had other matters to deal with, and Microsoft extended the timelines. After initially postponing turning Basic Authentication off to second half of 2021, the ‘start date’ for permanently turning the lights off for Basic Authentication was set to October 1st, 2022, as per this article on Docs and MC286990 in the Message Center. Mind the ‘start’ in start date, as flicking the switch for millions of tenants takes time before it becomes effective on your tenant. Organizations do need to anticipate on this change for the first of October.
On September 1st, Microsoft published an update to these timelines as there were still some cases were organizations could not make the deadline of October 1st. To meet these customers “one last time”, organizations can now use the self-service diagnostics to extend disabling of Basic Authentication to January 2023. This needs to be done per protocol, also if organizations requested opt-out or re-enabled Basic Authentication earlier. Details as well as instructions and revised timelines on Basic Authentication switching off are laid out in a new article.
Until then, organizations can (re-)enable Basic Authentication for the protocols they need, using the self-help system in the Microsoft 365 admin center. After entering “Diag: Enable Basic Auth in EXO” in the problem search query, the request will be checked, and Basic Authentication will get enabled. But with the end of support for Basic Authentication, so will this temporary workaround. On a side note, per end of 2020, newly created tenants already have basic authentication disabled by means of security defaults – if those organizations require Basic Authentication for some reason, they will also need to reconfigure security defaults which by default is an all or nothing option for all protocols.
So, with the doomsday counter ticking away for Basic Authentication, what are the consequences for Exchange related workloads organizations might wonder. In this article, I will address some of these concerns.
Update: Microsoft meanwhile has disclosed much awaited details on changes in the native Mail app on iOS. This update is effective per iOS 15.6, and adds support for migrating configured accounts from using Basic Authentication to using Modern Authentication (OAuth). For this work work seamlessly and without user interruption, some configuration is needed on the back-end. Details can be found in a separate article here on this specific topic.
A short blog on something which I find still surprises admins and consultants working with Exchange Online Management module for Powershell. The Exchange Online Management v2 module has been offering support for REST for a while now. One of the benefits of using these REST-based cmdlets, apart from performance and resilience, is that it uses Modern Authentication to connect to Exchange Online, which is the way forward, as Basic Authentication gets directed to the exit.
Now the initial versions of the module supported a limited set of 9 cmdlets. The REST cmdlets used the EXO prefix, such as Get-EXOMailbox as counterpart of Get-Mailbox. I wrote an earlier blog about using EXOv2, configuring the app in Azure and alternative ways to authenticate here.
Per version 2.0.6, which is still in preview, around 250 additional cmdlets got REST support as well, but using their original name and parameter set. You can check the number of cmdlets available after connecting, e.g.
As you can see above, after connecting this version supports 397 cmdlets for my role in addition to the 31 available pre-connecting. Your exact number might vary, based on the roles assigned to your account.
The confusion usually starts when people enter commands or run scripts, and find cmdlets are “missing”. Often they find that the Set/New cmdlet is unavailable, while the Get is available, e.g.
Before, this could be an indication those commands were removed from the role assigned to you, such as the New-MailboxImportRequest cmdlet which only is available if you have mailbox import/export assigned. But in this situation, it could be that the cmdlet does not have a REST call (yet). In those cases, you need to connect using a regular Remote PowerShell session, by specifying -UseRPSSession:
When connecting this way, I have 739 cmdlets at my disposal, including the ones which do not support REST. Note that cmdlets which support REST still will use REST; the commands that require Remote PowerShell will use the imported cmdlet. As a reminder, Remote Powershell requires Basic Authentication, and therefor must be enabled on the system you are connecting from.
Tip: Did you know you can view the release notes of the installed Exchange Online Management module by inspecting the ReleaseNotes property, e.g.
The Exchange PG released March updates for Exchange Server 2013, 2016 and 2019. More detailed information on patching and how to get current when running an earlier CU of Exchange, can be found at the original blog post here.
The vulnerabilities addressed in these security updates are:
These vulnerabilities are addressed in the following security updates below. The exception is KB5010324 which does not fix CVE-2022-24463 for Exchange 2013. If this is because of the severity classification or the problem being non-existent for Exchange 2013, has not been not disclosed.
Finally, KB5010324 also contains the following additional fix for Exchange 2013:
5012925 RFC certificate timestamp validation in Exchange Server 2013
Be advised that these security updates are Cumulative Update level specific. You cannot apply the update for Exchange 2019 CU11 to Exchange 2019 CU10. Also, the security update download has the same name for different Cumulative Updates, and I would suggest tagging the file name with the CU level, e.g. Exchange2019-CU10-KBXXXXXX-x64-en.msp.
As a reminder, run the Security Update from an elevated command prompt to prevent issues during installation. In other words: Do not just double-click on the .MSP file. And on a final note, as with any patch or update, I’d recommend to apply this in a test environment first, prior to implementing it in production. However, it is not recommended to wait for regular maintenance cycles when it concerns security updates, and follow a more agile approach; the ratings are an indication of the urgency.
More detailed information can be found at the original blog post here. The security update also fixes the OWA redirection problem for Exchange hybrid deployments introduced with the November security updates.
Be advised that these security updates are Cumulative Update level specific. You cannot apply the update for Exchange 2019 CU11 to Exchange 2019 CU10. Also, the security update download has the same name for different Cumulative Updates, and I would suggest tagging the file name with the CU level, e.g. Exchange2019-CU10-KBXXXXXX-x64-en.msp.
As a reminder, run the Security Update from an elevated command prompt to prevent issues during installation. In other words: Do not just double-click on the .MSP file. And on a final note, as with any patch or update, I’d recommend to apply this in a test environment first, prior to implementing it in production. However, it is not recommended to wait for regular maintenance cycles when it concerns security updates, and follow a more agile approach; the ratings are an indication of the urgency.
Happy new year to all my dear readers and followers. While some thought after 2020 things would improve soon, 2021 proved to be another year of dealing with measures. That said, 2021 was not very different compared to 2020 from a work perspective, with many organizations now professionalizing their quickly adopted digital transformation, further normalization for work from home scenarios. The year again proved to be a challenge for those busy working on customer projects while catering to the community as well. In the end, it is all about finding a balance, especially if you are working from home and the rest of the household is as well.
Now, without further ado, here are some of this blog’s and GitHub’s top entries for 2021.
General Stats
Number of views: 271,751 (3,086,610 all-time)
Unique visitors: 168,011 (1,627,860 all-time)
Number of posts: 17 (647 total)
Followers: 508
Busiest day: March 4, 2021 (5,616)
Popular time of day: Wednesday, 3pm
Popular blogs
Not surprisingly, posts related to the Hafnium security updates lead the way in 2021:
While I published some additional scripts on my GitHub repository, some scripts like the script to download Ignite sessions or remove duplicate items from Exchange mailboxes remain popular. I performed a total of 144 updates on the scripts, and have 102 people following my GitHub profile.
EighTwOne (821) 