Last Update: November 4th
About one month ago, Exchange Server 2016 Cumulative Update 3 was released which supported deployment on Windows Server 2016. However, recently issues are being reported on various communities as well in related blog comments, where Exchange 2016 became unstable, symptoms being randomly crashing IIS application pools (which says nothing about the root cause).
Microsoft acknowledged there is an issue with Exchange Server 2016 CU3 on Windows Server 2016:
If you attempt to run Microsoft Exchange 2016 CU3 on Windows Server 2016, you will experience errors in the IIS host process W3WP.exe. There is no workaround at this time. You should postpone deployment of Exchange 2016 CU3 on Windows Server 2016 until a supported fix is available.
So, be advised to hold off to deploying Exchange 2016 on Windows Server 2016 until further notice.
Update: The Exchange Team has also posted a notice that an update is in the works, and to delay further Exchange 2016 deployments on Windows Server 2016 until this delay has been made available. No ETA on the update yet.
Update: Made changes to reflect that IIS Request Filtering will not work.
This week, Microsoft released a security fix MS15-034 (KB3042553) for IIS which potentially allows for remote code execution on IIS, denial of service attacks (DOS) or bugchecking of servers. Since Exchange leverages IIS, Exchange servers are affected.
The vulnerability is easy to exploit, using an HTTP or HTTPS request and specifying a Range header with a value of 18446744073709551615 (maximum 64-bit unsigned integer). The Range header, introduced in the HTTP/1.1 specification, can be used by the requester to receive only a portion of data, for example the first few bytes of a JPG to determine its dimensions.The issue occurs when you specify out of bounds value. for example, when using cURL you can specify:
curl -v https://exchangeserver.contoso.com/iisstart.htm -H "Host: contoso.com" -H "Range: bytes = 0-8192" -k
Exchange-fellow Dave Stork did a nice write-up
on the issue and how to prevent it from happening, i.e.
- The most recommended solution is of course to install the KB3042553 security fix on servers running IIS, starting with servers that are internet-facing.
- Filter requests on your reverse proxy, load balancer or IPS solution:
- KEMP has provided instructions how to accomplish this on their Loadmasters here.
- F5 has provided instructions here.
- ISC SANS institute provided instructions for SNORT here.
- Disable IIS kernel caching, but this is not recommended due to negative impact on performance.
Unfortunately, Request Filtering is not an option so you can not prevent the exploit using IIS’ built-in Request Filtering feature. The Request Filtering will occur after parsing of the Range header, and it is in this parsing causing the issue.
A short notice on an issue when you have deployed Exchange 2013 Cumulative Update 6 in coexistence in an Exchange 2007 environment. Exchange fellow Tony Redmond did a write-up on the issue here.
The issue prevents ActiveSync users whose mailbox reside on Exchange 2007 to authenticate properly when their requests are being proxied from Exchange 2013 CU6 to Exchange 2007. It has been identified in KB2997847. Alternatively, you direct Exchange 2007 EAS traffic directly to Exchange 2007 CAS servers when they are internet-facing and published.
Be advised that a previous known issue in this deployment scenario with delegates and dismounting stores has been identified in KB2997209.
Both articles provide links to request these hotfixes.
Another Exchange fellow, Jason Sherry, is keeping track of resolved and open Exchange 2013 CU6 issues here.
As mentioned earlier, when you have deployed Exchange Server 2013 Cumulative Update 6 in a Hybrid deployment, several Office 365-related mailbox functions will not show up in the Exchange Admin Center (EAC). The issue was identified by Microsoft in KB2997355 and a fix was published.
However, the script to fix the issue looks for the XAML file in the default Program Files folder, using the default Exchange installation folder. Better is to check the actual Exchange installation folder, which can easily be accomplished in Exchange Management Shell using the $exinstall environment variable, or by reading the folder from the registry.
To help those installing Exchange in a non-default installation folder, and I know there are quite a few of you out there, who are hesitant to correcting the installation path in the provided FixIt script, I have create an alternative version of the Exchange2013-KB2997355-FixIt script. This version will read the installation path from the registry. Not disturbing but changed as well is correcting the XAML file in one go, unlike the official script which performs 3 consecutive read/modify/write actions on the same file.
You can download the Exchange2013-KB2997355-FixIt-v2.ps1 script here.
After installing Exchange 2013 Service Pack 1, people reported issues with Transport Agents. Symptoms are that the Transport service doesn’t start or stops shortly after starting the service or you can’t install the 3rd party product.
Products experiencing the issue are TrendMicro ScanMail, McAfee Email Security (GroupShield), Symantec Mail Security for Exchange, AVG for Servers, ESET Mail Security for Exchange and CodeTwo Exchange Rules. Products from other vendors may be affected as well.
Microsoft is aware of this issue and has published KB2938053 which has a small Exchange2013-KB2938053-FixIt.zip script to fix the issue.
The cause of the issue lies in XML files containing invalid XML markup in the form of “comments” which prevents .NET from loading the XML files, e.g.
<!-- 15.0.847.30 -------------------------------->
The two files containing the invalid XML markup are:
Be advised that the script supplied in the KB article tries to locate and fix various alternate versions of those files. Something you might want to consider as well when fixing it manually, should you be unable to locate the specific files mentioned above.
After running the script you should be able to start the Transport service or install 3rd party containing transport agents..
Update (3/5): Updated blog after official KB article got published. The issue was also blogged on by fellows Jason Sherry, Paul Cunningham while Tony Redmond has additionanal background details here.
Today the Exchange Team released security fixes for the issue described in bulletin MS13-105. Fixes have been released for the following product levels:
- Exchange 2007 SP3
Exchange 2007 SP3 Rollup 12, KB2903911, v8.3.342.4
- Exchange 2010 SP2
Exchange 2010 SP2 Rollup 8, KB2903903, v14.2.390.3
- Exchange 2010 SP3
Exchange 2010 SP3 Rollup 4, KB2905616, v184.108.40.206
- Exchange 2013 CU2
Security Update For Exchange 2013 CU2, KB2880833, v15.0.712.31
- Exchange 2013 CU3
Security Update For Exchange 2013 CU3, KB2880833, v15.0.775.41
Note that depending on the release scheme fixes are either made available through a Rollup or as security fix; the Rollups only address the vulnerabilities mentioned in security bulletin.
Note that this Rollup or security fix replaces MS13-061 – you can install MS13-105 over installations containing MS13-061 (no need to uninstall it first).
Microsoft published an important hotfix for .NET 4.5 earlier this year. It wasn’t picked up on by many, therefor a quick write up on the matter.
Since Exchange 2013 is built on top of .NET 4.5, it is recommended to install the hotfix on all Exchange 2013 Mailbox and Multi-Role servers. The hotfix will reduce the memory consumption of the store worker processes.
If you’re using Windows Server 2008 R2, the hotfix is KB2803754 and can be requested here; when using Windows Server 2012 the hotfix is KB2803755 which can be requested here.
After installing the hotfix, you need to do one of the following things:
- Set the following registry key:
- Set the COMPLUS_DisableRetStructPinning environment variable to 1
I’d prefer the first option. Note that you need to restart the server for the change to become effective.
Thanks to Tony Redmond for the heads up.
Today the rereleases of MS13-061 Security Fix for Exchange 2013 CU1 and Exchange 2013 CU2 saw daylight. This security update KB2874216 fixes the issue described in Microsoft Security Bulletin MS13-061 and supposedly fixes the issues found with the original release. After installing the v2 patch, the version will be upped 2 notches compared to the original patch.
As mentioned in an earlier article, security fixes are Cumulative Update level specific. In practice, this means there are two different versions of the security update patch file: one for CU1 and one for CU2.
Be advised both files carry the same file name, Exchange2013-KB2874216-v2-x64-en.msp. I suggest adding some form of Cumulative Update identification to the file name when you archive it, e.g. Exchange2013-KB2874216-v2-x64-en-CU2.msp.
As with any patch or update, I’d recommend to thoroughly test this in a test and acceptance environment first, prior to implementing it in production. If you don’t have the resources and risk management can agree, you might want to consider postponing implementation for a short period while monitoring for issues in the online.
You can download the security updates here:
UPDATE: The MS13-061 security update for Exchange 2013 CU1 & CU2 has been pulled until further notice. Microsoft recommends not installing MSI13-061 at the moment and disable Data Loss Prevention and WebReady as described in the Oracle Outside In Contains Multiple Exploitable Vulnerabilities section in the MS13-061 bulletin.
After some people reported issues after installing the MS13-061 (KB2874216) security update on Exchange 2013, it turns out MS13-061 breaks your installation of Exchange 2013 and you can experience the following symptoms:
- The Microsoft Exchange Search Host Controller service is missing;
- You see a new service named “Host Controller service for Exchange”;
- Content index (CI) for mailbox databases shows Failed on affected server.
This is described in KB2879739 including the ‘workaround’, which is consists of three steps:
- Set HKLM\SOFTWARE\Microsoft\Search Foundation for Exchange\Data Directory to $exinstall\Bin\Search\Ceres\HostController\Data (REG_SZ), where $exinstall is the installation folder of your Exchange 2013 installation folder, e.g. C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data;
- Set HKLM\SYSTEM\CurrentControlSet\Services\HostControllerService\DisplayName=”Microsoft Exchange Search Host Controller” (REG_SZ);
- Set HKLM\SYSTEM\CurrentControlSet\Services\HostControllerService\DependOnService=”http” (REG_MULTI_SZ);
- (Re)start the “Microsoft Exchange Search Host Controller” service.
For your convenience, I’ve create a small quick & dirty script as a potential time saver (as far as you can call a three-liner a script and don’t expect extensive error handling as well). This script Workaround-KB2879739.ps1 performs the steps described in the KB2879739 so you can run it right after deploying MS13-061 / KB2874216 on your Exchange 2013 server.
You can download the script here.
UPDATE: The MS13-061 security update for Exchange 2013 CU1 & CU2 has been pulled until further notice.If you have installed it, there are issues with it which can be fixed (link). Microsoft recommends not installing MSI13-061 at the moment and disable Data Loss Prevention and WebReady as described in the Oracle Outside In Contains Multiple Exploitable Vulnerabilities section in the MS13-061 bulletin.
Today the Exchange Team released the first Security Update for Exchange 2013. This security update KB2874216 fixes the issue described in Microsoft Security Bulletin MS13-061.
As mentioned in an earlier article, security fixes are Cumulative Update level specific. How that would turn out in practice remained to be seen at the time of writing that article, but at the moment it means there are two different versions of the security update, one patch file for CU1 and one for CU2 (or the re-release of CU2 actually, version 15.0.712.24 – more information on that here). I assume the .MSP format limits the ability to merge the two and let it make an intelligent decision on what to install.
Be warned that both files carry the same file name, I suggest adding some form of Cumulative Update identification to the file name when archiving it, e.g. Exchange2013-KB2874216-x64-en-CU2.msp.
As with any patch or update, I’d recommend to thoroughly test this in a test and acceptance environment first, prior to implementing it in production.
You can download the security updates here: