Exchange and .NET Framework 4.7

Ex2013 Logo A quick heads-up on that .NET Framework 4.7 has recently been released and will be made available through Windows Update channels. The current versions of Exchange Server are not supported with this version of the .NET Framework, and you should not install or update to this version.

Similar to the situation with .NET Framework 4.61 around a year ago, you can prevent  (accidental) upgrades of the .NET Framework by creating the following registry key on your Exchange servers:

HKLM:\Software\Microsoft\NET Framework Setup\NDP\WU\BlockNetFramework47 = 1 (REG_DWORD)

To report on the currently installed .NET Framework version on one or more computers, you can use this PowerShell script, Get-DotNetVersion.ps1. It will not only report the .NET Framework version information, but also if those registry entries to block .NET Framework 4.6.1 or .NET Framework 4.7 upgrades are present.

[PS] C:\> .\get-DotNetVersion.ps1 -ComputerName ex1,ex2 | ft -a

Computer Release NetFramework Net461Block Net47Block
-------- ------- ------------ ----------- ----------
ex1      461268  4.7          False       True
ex2      461268  4.7          False       False

The related article by the Exchange Team on this topic contains steps on how to recover the situation, in case you did upgrade. Of course, with all the dependencies on the .NET Framework by Exchange Server, you may prefer migrating contents to a new Exchange servers with a supported .NET Framework, and decommission servers where you had to remove the unsupported .NET Framework from.

More information can be found in KB4024204.

PS: The updated Unattended Exchange 2013 & 2016 Installation script will now also set the .NET Framework 4.7 blockade registry key.

Exchange Updates – March 2017

Ex2013 LogoToday, the Exchange Team released the March updates for Exchange Server 2013 and 2016, as well as Exchange Server 2010 and 2007. The latter will receive its last update, as Exchange 2007 will reach end-of-life April 11, 2017.

As announced in December updates, Exchange 2013 CU16 and Exchange 2016 CU5 require .NET 4.6.2. The recommended upgrade paths:

  • If you are still on .NET 4.6.1, you can upgrade to .NET 4.6.2 prior of after installing the latest Cumulative Update.
  • If you are on .NET 4.52, upgrade to Exchange 2016 CU4 or Exchange 2013 CU15 if you are not already on that level, then upgrade to .NET 4.6.2, and finally upgrade to the the latest Cumulative Update.

The Cumulative Updates also include DST changes, which is also contained in the latest Rollups published for Exchange 2010 and 2007.

For a list of fixes in these updates, see below.

Exchange 2016 CU5 15.1.845.34 KB4012106 Download UMLP
Exchange 2013 CU16 15.0.1293.2 KB4012112 Download UMLP
Exchange 2010 SP3 Rollup 17 14.3.352.0 KB4011326 Download
Exchange 2007 SP3 Rollup 23 8.3.517.0 KB4011325 Download
  • KB4015665 SyncDelivery logging folders and files are created in wrong location in Exchange Server 2016
  • KB4015664 A category name that has different case-sensitivity than an existing name is not created in Exchange Server 2016
  • KB4015663 “The message content has become corrupted” exception when email contains a UUE-encoded attachment in Exchange Server 2016
  • KB4015662 Deleted inline picture is displayed as attachment after you switch the message to plain text in Exchange Server 2016
  • KB4015213 Email is still sent to Inbox when the sender is deleted from the Trusted Contacts list in Exchange Server 2016
  • KB4013606 Search fails on Exchange Server 2016 or Exchange Server 2013
  • KB4012994 PostalAddressIndex element isn’t returning the correct value in Exchange Server 2016

Exchange 2013 CU16 fixes:

  • KB4013606 Search fails on Exchange Server 2016 or Exchange Server 2013

Exchange 2010 SP3 RU17 fixes:

  • KB4014076 Migration ends and errors reported when you on-board or off-board a mailbox through Exchange Online in an Exchange Server 2010 hybrid environment
  • KB4014075 UNC path does not open in OWA when the path contains non-ASCII characters in an Exchange Server 2010 environment
  • KB4013917 You cannot search in a shared mailbox through OWA in an Exchange Server 2010 Service Pack 3 (Update Rollup 15 or 16) environment
  • KB4012911 Culture element is added in the wrong order when you use the ResolveNames operation in EWS in Exchange Server 2010

Notes:

  • Exchange 2016 CU5 doesn’t include schema changes, however, Exchange 2016 CU5 as well as Exchange 2013 CU16 may introduce RBAC changes in your environment. Where applicable, use setup /PrepareSchema to update the schema or /PrepareAD to apply RBAC changes, before deploying or updating Exchange servers. To verify this step has been performed, consult the Exchange schema overview.
  • When upgrading your Exchange 2013 or 2016 installation, don’t forget to put the server in maintenance mode when required. Do note that upgrading, before installing the Exchange binaries, setup will put the server in server-wide offline-mode.
  • Using Windows Management Framework (WMF)/PowerShell version 5 on anything earlier than Windows Server 2016 is not supported. Don’t install WMF5 on your Exchange servers running on Windows Server 2012 R2 or earlier.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), you are allowed to stay at least one version behind (n-1).
  • If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
  • Cumulative Updates can be installed directly, i.e. no need to install RTM prior to installing Cumulative Updates.
  • Once installed, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
  • The order of upgrading servers with Cumulative Updates is irrelevant.

Caution: As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or forums for any issues.

MS17-015: Security Fix for Exchange 2013 SP1+CU14 & 2016 CU3

Ex2013 LogoMicrosoft published security fixes for the issue described in bulletin MS17-105. Fixes have been released for the following product levels:

You are reading it correctly: the later Cumulative Updates are not affected. Earlier builds will not receive a security fix, as support is provided up to N-2 generation builds. Reason for Exchange 2013 SP1 being in there is that Service Packs are on a different support scheme.

Note that this Rollup or security fix replaces MS16-108 (kb3184736) – you can install MS13-105 over installations containing this security fix (no need to uninstall it first).

Exchange admins & PowerShell

imageMany people I encounter in the field of Office 365 or Exchange have an infrastructure background. That is, they know a lot about their product(s), how to make it work (or don’t), how to manage, deploy or troubleshoot, etcetera.

Then there is, the let us call it, the reality check of the cloud era, with a roller coaster of cloud-originating developments. This requires a different management focus for these products, resulting in products architected for scale, and introducing configuration and management instruments primarily designed to be ready for automation and operate on scale as well. PowerShell support in Microsoft products is such an instrument.

The introduction of PowerShell required folks with an infrastructure background to develop a new skill: instead of clicking buttons in an interface, they should also become a PowerShell practitioner. Not necessarily wizard level, but at least they need to know their way around when managing their environment using PowerShell, reading and interpreting scripts provided by Microsoft or other vendors prior to usage, or even make changes to make those scripts fit for their own environment.

Writing scripts is another matter. This requires a tad different mindset, where you make repeatable tasks repeatable (time-saving), less prone to error (job-saving), and reusable by your coworkers or even the community who may need to perform the same task. Of course, everybody also expects your scripts to be generic (no hard-coded elements), robust and resilient, adding 90% more code (a bit exaggerated, but you get the idea).

What most of administrators struggle with, is making the connection between managing the product using PowerShell, and how to start using PowerShell to develop their own set of scripts or tools to automate tasks their environment. Administrators wanting to learn such skills will usually find is great books about the product, and great books on learning (generic) PowerShell. Of course, existing scripts found using their favorite search engine can also be a great starting point, provided somebody already developed it for the task you are trying to accomplish.

With the Exchange Server 2016 administrator in mind, Exchange fellows Dave Stork and Damian Scoles tried to bridge that gap with their book, Practical PowerShell: Exchange Server 2016. It uses some practical Exchange-themed examples, how to approach the problem, and how to go from running a few cmdlets in sequence to developing small scripts which operate against one or multiple servers. Also, while this book aims at the on-premises Exchange administrators, the skills learned are not lost when the organization moves to Exchange Online as these scripting skills are compatible.

Knowing how difficult it can be to transfer knowledge to paper from my own experience, I think Dave & Damian did a respectable job. The timing of the book release is also interesting, as the product which introduced PowerShell to so many of us, Exchange Server 2007, is going End of Life soon, on April 2011, 2017 to be exact. Realizing PowerShell has been around now for so many years, there is no excuse to get your PowerShell skills going, unless you want to share the faith of dinosaurs.

More information on the book, including a sample chapter, is available at https://www.practicalpowershell.com. You can also order the book from Amazon here.

Results Install-Exchange15 survey

stats chartA short blog on a small survey I’ve been running for some time now on the usage of Install-Exchange15, the PowerShell script for fully automated deployment of Exchange 2013 or Exchange 2016.

I started the survey because I was curious on a few things:

  • How the script is used; do folks use it for deploying in lab environments, or also actual production environments.
  • What Exchange versions are deployed; only current ones (n-2 at most, i.e. lagging 2 Cumulative Update generations at most), or also older versions.
  • What operating systems are used to deploy Exchange using this script.

The second and last items are of most interest, as keeping backward compatibility in the script, for example like deploying Exchange Server 2013 SP1 on Windows Servers 2008, requires keeping a lot of ‘legacy code’ in there.

Fortunately, the survey shows many of you use the script to deploy recent Exchange builds on current operating systems. So, in time, you will see support for older builds and operating systems being removed, making the script more lean and mean as well.

Now, on to the results:

In what environments do you use the script to deploy Exchange?

Lab

Production

Yes

86%

72%

No

14%

28%

Do you use Install-Exchange15.ps1 for previous (N-2 or older) Exchange 2013/2016 builds?

Yes 28%
No 72%

On which Operating Systems do you deploy Exchange 2013/2016? (multiple options possible)

Windows Server 2008 0%
Windows Server 2008 R2 18%
Windows Server 2012 18%
Windows Server 2012 R2 100%
Windows Server 2016 8%

Finally, a summary of the feedback and requests send in by respondents through the open comments section:

  • Installation on Windows Server 2016. The survey was created before Windows Server 2016 was supported, so we used the feedback given on people deploying on WS2016 in the above results.
  • In general, positive feedback on having this script for automated deployment, as well as the SCP feature.
  • Request for having a GUI to create the answer file.
  • Request to having the option to configure the virtual directories after installation. However, the script allows for inserting custom (Exchange) cmdlets in its post-configure phase.
  • Request to output cause of failed Exchange setup to the screen. That however, is something I wouldn’t recommend; the Exchange setup log files contain the details.
  • Request to have some sort of visible clue if the installation was successful or not.

Exchange Updates – December 2016

Ex2013 LogoToday, the Exchange Team released the December updates for Exchange Server 2013 and 2016, as well as Exchange Server 2010 and 2007.

Changes introduced in Exchange Server 2016 CU4 are an updated OWA composition window, and on Windows Server 2016, KB3206632 needs to be present before CU4 can be deployed. This KB fixes – among other things – the problem with DAG members experiencing crashing IIS application pools. Be advised that the KB3206632 update for Windows Server 2016 is a whopping 1 GB. You can download this update here.

The biggest change for Exchange Server 2013 is support for .NET 4.6.2. Be advised organizations running Exchange 2016 or 20103 on .NET 4.6.1 should start testing with and planning to deploy .NET 4.6.2, as the March updates will require .NET 4.6.2, which is currently still optional.

Both Cumulative Updates introduce an important fix for indexing of Public Folder when they were in the process of being migrated. To ensure proper indexing, it is recommended to move public folder mailboxes to a different database so they will get indexed properly.

The Cumulative Updates also include DST changes, which is also contained in the latest Rollups published for Exchange 2010 and 2007.

For a list of fixes in these updates, see below.

Exchange 2016 CU4 15.1.669.32 KB3177106 Download UMLP
Exchange 2013 CU15 15.0.1263.5 KB3197044 Download UMLP
Exchange 2010 SP3 Rollup 16 14.3.339.0 KB3184730 Download
Exchange 2007 SP3 Rollup 22 8.3.502.0 KB3184712 Download
  • KB 3202691 Public folders indexing doesn’t work correctly after you apply latest cumulative updates for Exchange Server
  • KB 3201358 Set-Mailbox and New-Mailbox cmdlets prevent the use of the -Office parameter in Exchange Server 2016
  • KB 3202998 FIX: MSExchangeOWAAppPool application pool consumes memory when it recycles and is marked as unresponsive by Health Manager
  • KB 3208840 Messages for the health mailboxes are stuck in queue on Exchange Server 2016
  • KB 3186620 Mailbox search from Exchange Management Shell fails with invalid sort value in Exchange Server 2016
  • KB 3199270 You can’t restore items to original folders from Recoverable Items folder in Exchange Server 2016
  • KB 3199353 You can’t select the receive connector role when you create a new receive connector in Exchange Server 2016
  • KB 3193138 Update to apply MessageCopyForSentAsEnabled to any type of mailbox in Exchange Server 2016
  • KB 3201350 IMAP “unread” read notifications aren’t suppressed in Exchange Server 2016
  • KB 3209036 FIX: “Logs will not be generated until the problem is corrected” is logged in an Exchange Server 2016 environment
  • KB 3212580 Editing virtual directory URLs by using EAC clears all forms of authentication in Exchange Server 2016

Exchange 2013 CU15 fixes:

  • KB 3198092 Update optimizes how HTML tags in an email message are evaluated in an Exchange Server 2013 environment
  • KB 3189547 FIX: Event ID 4999 for a System.FormatException error is logged in an Exchange Server 2013 environment
  • KB 3198046 FIX: The UM mailbox policy is not honored when UM call answering rules setting is set to False in an Exchange Server 2013 environment
  • KB 3195995 FIX: Event ID 4999 with MSExchangerepl.exe crash in Exchange Server 2013
  • KB 3208886 Send As permission of a linked mailbox is granted to a user in the wrong forest in Exchange Server 2013
  • KB 3188315 Messages are stuck in outbox when additional mailboxes are configured in Outlook on Exchange Server 2013
  • KB 3202691 Public folders indexing doesn’t work correctly after you apply latest cumulative updates for Exchange Server
  • KB 3203039 PostalAddressIndex property isn’t returning the correct value in Exchange Server 2013
  • KB 3198043 Shadow OAB downloads don’t complete in Exchange Server 2013
  • KB 3208885 Microsoft.Exchange.Store.Worker.exe process crashes after mailbox is disabled in Exchange Server 2013
  • KB 3201966 MSExchangeOWAAppPool application pool consumes excessive memory on restart in Exchange Server 2013
  • KB 3209041 FIX: An ActiveSync device becomes unresponsive until the sync process is complete in an Exchange Server environment
  • KB 3209013 FIX: The LegacyDN of a user is displayed incorrectly in the From field in the Sent Items folder on an ActiveSync device
  • KB 3209018 FIX: Results exported through the eDiscovery PST Export Tool never finishes in an Exchange Server environment
  • KB 3199519 Inconsistent search results when email body contains both English and non-English characters in Exchange Server 2013
  • KB 3211326 “ConversionFailedException” error when emails are sent on the hour in Exchange Server 2013

Notes:

  • Exchange 2016 CU4 doesn’t incldue schema changes compared to CU4, however, Exchange 2016 CU4 as well as Exchange 2013 CU15 may introduce RBAC changes in your environment. Where applicable, use setup /PrepareSchema to update the schema or /PrepareAD to apply RBAC changes, before deploying or updating Exchange servers. To verify this step has been performed, consult the Exchange schema overview.
  • When upgrading your Exchange 2013 or 2016 installation, don’t forget to put the server in maintenance mode when required. Do note that upgrading, before installing the Exchange binaries, setup will put the server in server-wide offline-mode.
  • Using Windows Management Framework (WMF)/PowerShell version 5 on anything earlier than Windows Server 2016 is not supported. Don’t install WMF5 on your Exchange servers running on Windows Server 2012 R2 or earlier.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), you are allowed to stay at least one version behind (n-1).
  • If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
  • Cumulative Updates can be installed directly, i.e. no need to install RTM prior to installing Cumulative Updates.
  • Once installed, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
  • The order of upgrading servers with Cumulative Updates is irrelevant.

Caution: As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or TechNet forum for any issues.

Advisory: Hold off deploying Exchange 2016 CU3 on WS2016 for now

Ex2013 LogoLast Update: December 13th, 2016: The Windows team published an update for Windows Server 2016, which should fix the issue with DAG members crashing when restarted. The related article is KB3206632, and you can download it here. Be advised, the Windows Server 2016 update – which also fixes other issues – is nearly 1 GB!

About one month ago, Exchange Server 2016 Cumulative Update 3 was released which supported deployment on Windows Server 2016. However, recently issues are being reported on various communities as well in related blog comments, where Exchange 2016 became unstable, symptoms being randomly crashing IIS application pools (which says nothing about the root cause).

Microsoft acknowledged there is an issue with Exchange Server 2016 CU3 on Windows Server 2016:

If you attempt to run Microsoft Exchange 2016 CU3 on Windows Server 2016, you will experience errors in the IIS host process W3WP.exe. There is no workaround at this time. You should postpone deployment of Exchange 2016 CU3 on Windows Server 2016 until a supported fix is available.

So, be advised to hold off to deploying Exchange 2016 on Windows Server 2016 until further notice.

Update: The Exchange Team has also posted a notice that an update is in the works, and to delay further Exchange 2016 deployments on Windows Server 2016 until this delay has been made available. No ETA on the update yet.

Exchange Server Role Requirements Calculator 8.3

Exchange 2010 Mailbox Role Sizing Calculator 16.4The Exchange team published an update for the Exchange Server Role Requirements Calculator, the tool to aid you in properly sizing your Exchange Server 2013 or Exchange Server 2016 deployment.

The new version number is 8.3, and it contains two major enhancements compared to version 7.9:

  • Added ability for the calculator to automatically determine the number of Mailbox servers and DAGs that need to be deployed to meet the chosen input requirements
  • Added Read from Passive support for Exchange 2016 deployments which results in decreased bandwidth utilization for HA copies

You can download the calculator here. For more information, please consult the list of changes here or Read Me here.

Exchange Updates – September 2016

Ex2013 LogoNote: There are issues with Exchange 2013 CU14 and Exchange 2016 CU3 with regards to failing Content Indexing. Fellow MVP Jaap Wesselius blogged about this here. For now, recommendation is to not upgrade to CU14, until further notice. Also, there are acknowledged issues when running Exchange 2016 CU3 on Windows Server 2016. Don’t deploy Exchange 2016 CU3 on that OS until further notice.

Today, the Exchange Team released the september updates for Exchange Server 2013 and Exchange Server 2016.

The biggest changes are for Exchange Server 2016:

  • Exchange Server 2016 CU3 or later support on Windows Server 2016, which is expected to be released at Ignite next week. Windows Server 2016 Domain Controllers are supported; requirement is just Forest Functional Level at Windows Server 2008 R2 or later. Note that it is also announced Exchange Server 2013 will not be supported (as in: now, and in the future). Performance-wise, it is recommended to exclude Exchange setup and log folders, as well as the noderunner processes in Windows Defender.
  • Finally, Exchange Server 2016 CU3 introduces the long-awaited Read from Passive feature. This means, indexes will be generated using (local) passive databases copies, and no longer require coordination with the server holding the active database copy. The result is lower bandwidth requirements, and – compared to Exchange Server 2013 – faster fail-overs. Be advised this feature does not apply to lagged copies.
  • An update for the Mailbox Server Role Calculator(s) for Exchange 2016 is available now (v8.3), incorporating Read from Passive changes. This allows organizations to not only size their deployment, but also predict the positive effect on bandwidth usage for current environments as well by using numbers. You can download the calculator here.

For a list of fixes in these updates, see below.

Exchange 2016 Cumulative Update 3 15.1.544.27 KB3152589 Download UMLP
Exchange 2013 Cumulative Update 14 15.0.1236.3 KB3177670 Download UMLP

  • KB 3154387 The DFS health set is listed as “Unhealthy” in an Exchange Server 2016 environment
  • KB 3175080 Cannot log on to OWA when FIPS is enabled in an Exchange Server 2016 environment
  • KB 3176377 Links to access Exchange items in SharePoint eDiscovery search result fail with an HTTP error 500 in Exchange Server
  • KB 3161916 Data loss may occur during public folder migration to Exchange 2013, Exchange 2016, or Exchange Online
  • KB 3176540 OWA error reporting responds with a HTTP error 500 in OwaSerializationException
  • KB 3190887 Upgrading Exchange Server causes the server to go offline unexpectedly
  • KB 3191075 You can’t install Cumulative Update 2 for Exchange Server 2016 on a Russian version operating system

  • KB 3132513 “The Delegates settings were not saved correctly” when you try to add a user to Exchange Server 2013 from Microsoft Outlook
  • KB 3172017 “NotFound Export failed with error type: ‘NotFound'” error occurs when you perform an eDiscovery search in Exchange Server 2013
  • KB 3176377 Links to access Exchange items in SharePoint eDiscovery search result fail with an HTTP error 500 in Exchange Server
  • KB 3176540 OWA error reporting responds with a HTTP error 500 in OwaSerializationException
  • KB 3176873 Can’t create a new profile or connect to Exchange Server 2013 when an organization contains many address lists
  • KB 3061079 RPC Client Access service crashes and Event 4999 is logged in Exchange Server 2013
  • KB 3134918 An IRM-protected message sent to an external contact isn’t returned in a search or discovery results when journaling is implemented in an Exchange Server 2013 environment
  • KB 3190887 Upgrading Exchange Server causes the server to go offline unexpectedly

These Cumulative Updates for Exchange Server 2016 and 2013 include the security update released last week, MS16-108. The Cumulative Updates for Exchange Server 2016 and 2013 also include DST changes.

Notes:

  • Exchange 2016 CU3 includes schema changes (version 15326, reference), and Exchange 2016 CU3 as well as Exchange 2013 CU14 may introduce RBAC changes in your environment. Where applicable, make sure you run /PrepareSchema to update the schema or /PrepareAD to apply RBAC changes, before deploying or updating Exchange servers. To verify this step has been performed, consult the Exchange schema overview.
  • When upgrading your Exchange 2013 or 2016 installation, don’t forget to put the server in maintenance mode when required. Do note that upgrading, before installing the Exchange binaries, setup will put the server in server-wide offline-mode.
  • Using Windows Management Framework (WMF)/PowerShell version 5 on anything earlier than Windows Server 2016 is not supported. Don’t install WMF5 on your Exchange servers running on Windows Server 2012 R2 or earlier.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), you are required to stay current.
  • If you want to speed up the update process for systems without internet access, you can follow the procedure described here to disable publisher’s certificate revocation checking.
  • Cumulative Updates can be installed directly, i.e. no need to install RTM prior to installing Cumulative Updates.
  • Once installed, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles.
  • The order of upgrading servers with Cumulative Updates is irrelevant.

Caution: As for any update, I recommend to thoroughly test updates in a test environment prior to implementing them in production. When you lack such facilities, hold out a few days and monitor the comments on the original publication or TechNet forum for any issues.

Public Folder Hierarchy and Client Access

Ex2013 LogoWhen investigating performance issues of a multi-node, multi-role Exchange 2013 server deployment, I found the CPU utilization of a single Exchange 2013 server constantly above the load of the rest.

When checking the Processor Utilization % for all Exchange servers using Performance Monitor, the daily trend image looked like this:

clip_image002

As you can clearly see, one single server is constantly experiencing more load than the other servers. It is also above the 80% mark, causing all sorts of potential side-effects if Managed Availability would kick in.

When checking the processes on that server, the major CPU load was generated by the Microsoft.Exchange.RPCClientAccess.service as well as the related w3svc# process. The load balancer performed a near even distribution of client connections over these servers. You can use the Exchange Performance Health Checker script with the LoadBalancingReport switch to verify this.

Next, we checked if there was an overactive mailbox on that particular server. For that purpose, we ran the following cmdlet in the Exchange Management Shell, which showed us the Public Folder mailbox was very active:

Get-StoreUsageStatistics –Server <ExchangeServer> | ? {$_.DigestCategory –eq ‘timeInServer’} | Sort TimeOnServer –Descending

image

Note: More on tracking overactive mailboxes using Get-StoreUsageStatistics in this excellent write-up by Andrew HigginBotham.

Another clue was provided through the PublicFolders Healthset, which was picked up by System Center Operations Manager as well:

The PublicFolders Health Set has detected a problem with PublicFolderMailbox.ConnectionCount at 10-7-2016 06:12:22. 0 failures were found. The Health Manager is reporting that The total number of hierarchy connections for public folder mailbox PFMailbox1 has reached 2001. Consider creating a new public folder mailbox for load balancing hierarchy accesses.

Apparently, there were more than 2,000 connections being made to the PFMailbox1 Public Folder mailbox. This was odd, as there were multiple Public Folder mailboxes created with hierarchy. Users are expected to be automatically distributed over these mailboxes, falling within the 2,000 concurrent logons limit as mentioned here. Note that this limit applies to public folder mailboxes serving hierarchy as well; even if clients don’t access Public Folders, they still will connect to these Public Folder mailboxes in order to obtain hierarchy information.

Next thing we checked was to which default Public Folder mailbox mailboxes were configured to connect. To accomplish this we can inspect the mailbox property DefaultPublicFolderMailbox:

Get-Mailbox –ResultSize Unlimited | Group-Object DefaultPublicFolderMailbox –NoElement

Count Name
----- ----
10139 contoso.com/Accounts/Users/PFMailbox1

Apparently all mailboxes were automatically set to connect to a single Public Folder mailbox. Then maybe something was preventing the other Public Folders from serving hierarchy:

Get-Mailbox –PublicFolder | Select Name,*Hierarchy*

Name       IsExcludedFromServingHierarchy IsHierarchyReady
----       ------------------------------ ----------------
PFMailbox1 False                          True
PFMailbox2 False                          False
PFMailbox3 False                          False
PFMailbox4 False                          False

IsExcludedFromServingHierarchy was False for all 4 servers, which indicates they are not blocked from serving hierarchy. However, the hierarchy was not ‘ready’ for 3 of them. This could be due to the hierarchy being out of date or not being created at all.

The output of (Get-PublicFolderMailboxDiagnostics PFMailbox2 -IncludeHierarchyInfo).SyncInfo indeed indicated there were problems synchronizing contents from the PFMailbox1 mailbox. We then ran the following cmdlet to trigger updating synchronizing the hierarchy again:

Update-PublicFolderMailbox –InvokeSynchronizer –Identity PFMailbox2

image

The Get-Mailbox –Identity PFMailbox2 –PublicFolder | Select Name,*Hierarchy* now showed IsHierarchyReady was True. We ran the same cmdlet for the other two Public Folder mailboxes as well.

After a while, we verified the effect on the assignment of DefaultPublicFolderMailbox on the mailboxes:

Get-Mailbox –ResultSize Unlimited | Group DefaultPublicFolderMailbox –NoElement

Count Name
----- ----
2601  contoso.com/Accounts/Users/PFMBPFMailbox2
2309  contoso.com/Accounts/Users/PFMBPFMailbox4
2632  contoso.com/Accounts/Users/PFMBPFMailbox1
2597  contoso.com/Accounts/Users/PFMBPFMailbox3

Public folder assignments were now (more or less) equally distributed over the 4 Public Folder mailboxes, and life was good.

We also verified Public Folder access distribution by querying the Exchange RpcClientAccess log files. An excellent tool to aid in this task is LogParser with LogParser Studio. We configured LogParser Studio to query log files at ‘<Installation folder>\Logging\RPC Client Access’ on the Exchange servers. The query used, grouped all entries per date, operation (in this case we are only interested in PublicLogon), and part of the field ‘operation-specific’; more exactly, the legacyDN part which tells which (Public Folder) mailbox was accessed:

SELECT EXTRACT_PREFIX([#Fields: date-time], 0, ‘T’) As Date, Count (*) as Total, [Operation],
EXTRACT_PREFIX(EXTRACT_SUFFIX([operation-specific], 0, ‘cn=’), 0, ‘ in database ‘) as PFMailbox
FROM ‘[LOGFILEPATH]’
WHERE [operation]=’PublicLogon’
AND [failures] IS NULL
GROUP BY Date, [Operation], PFMailbox
ORDER BY Date ASC

The output showed all Public Folder mailboxes were now accessed by clients, and logons to the Public Folder mailboxes were now (more or less) equally distributed:

image