When investigating performance issues of a multi-node, multi-role Exchange 2013 server deployment, I found the CPU utilization of a single Exchange 2013 server constantly above the load of the rest.
When checking the Processor Utilization % for all Exchange servers using Performance Monitor, the daily trend image looked like this:
As you can clearly see, one single server is constantly experiencing more load than the other servers. It is also above the 80% mark, causing all sorts of potential side-effects if Managed Availability would kick in.
When checking the processes on that server, the major CPU load was generated by the Microsoft.Exchange.RPCClientAccess.service as well as the related w3svc# process. The load balancer performed a near even distribution of client connections over these servers. You can use the Exchange Performance Health Checker script with the LoadBalancingReport switch to verify this.
Next, we checked if there was an overactive mailbox on that particular server. For that purpose, we ran the following cmdlet in the Exchange Management Shell, which showed us the Public Folder mailbox was very active:
Get-StoreUsageStatistics –Server <ExchangeServer> | ? {$_.DigestCategory –eq ‘timeInServer’} | Sort TimeOnServer –Descending
Note: More on tracking overactive mailboxes using Get-StoreUsageStatistics in this excellent write-up by Andrew HigginBotham.
Another clue was provided through the PublicFolders Healthset, which was picked up by System Center Operations Manager as well:
The PublicFolders Health Set has detected a problem with PublicFolderMailbox.ConnectionCount at 10-7-2016 06:12:22. 0 failures were found. The Health Manager is reporting that The total number of hierarchy connections for public folder mailbox PFMailbox1 has reached 2001. Consider creating a new public folder mailbox for load balancing hierarchy accesses.
Apparently, there were more than 2,000 connections being made to the PFMailbox1 Public Folder mailbox. This was odd, as there were multiple Public Folder mailboxes created with hierarchy. Users are expected to be automatically distributed over these mailboxes, falling within the 2,000 concurrent logons limit as mentioned here. Note that this limit applies to public folder mailboxes serving hierarchy as well; even if clients don’t access Public Folders, they still will connect to these Public Folder mailboxes in order to obtain hierarchy information.
Next thing we checked was to which default Public Folder mailbox mailboxes were configured to connect. To accomplish this we can inspect the mailbox property DefaultPublicFolderMailbox:
Get-Mailbox –ResultSize Unlimited | Group-Object DefaultPublicFolderMailbox –NoElement
Count Name
----- ----
10139 contoso.com/Accounts/Users/PFMailbox1
Apparently all mailboxes were automatically set to connect to a single Public Folder mailbox. Then maybe something was preventing the other Public Folders from serving hierarchy:
Get-Mailbox –PublicFolder | Select Name,*Hierarchy*
Name IsExcludedFromServingHierarchy IsHierarchyReady
---- ------------------------------ ----------------
PFMailbox1 False True
PFMailbox2 False False
PFMailbox3 False False
PFMailbox4 False False
IsExcludedFromServingHierarchy was False for all 4 servers, which indicates they are not blocked from serving hierarchy. However, the hierarchy was not ‘ready’ for 3 of them. This could be due to the hierarchy being out of date or not being created at all.
The output of (Get-PublicFolderMailboxDiagnostics PFMailbox2 -IncludeHierarchyInfo).SyncInfo indeed indicated there were problems synchronizing contents from the PFMailbox1 mailbox. We then ran the following cmdlet to trigger updating synchronizing the hierarchy again:
Update-PublicFolderMailbox –InvokeSynchronizer –Identity PFMailbox2
The Get-Mailbox –Identity PFMailbox2 –PublicFolder | Select Name,*Hierarchy* now showed IsHierarchyReady was True. We ran the same cmdlet for the other two Public Folder mailboxes as well.
After a while, we verified the effect on the assignment of DefaultPublicFolderMailbox on the mailboxes:
Get-Mailbox –ResultSize Unlimited | Group DefaultPublicFolderMailbox –NoElement
Count Name
----- ----
2601 contoso.com/Accounts/Users/PFMBPFMailbox2
2309 contoso.com/Accounts/Users/PFMBPFMailbox4
2632 contoso.com/Accounts/Users/PFMBPFMailbox1
2597 contoso.com/Accounts/Users/PFMBPFMailbox3
Public folder assignments were now (more or less) equally distributed over the 4 Public Folder mailboxes, and life was good.
We also verified Public Folder access distribution by querying the Exchange RpcClientAccess log files. An excellent tool to aid in this task is LogParser with LogParser Studio. We configured LogParser Studio to query log files at ‘<Installation folder>\Logging\RPC Client Access’ on the Exchange servers. The query used, grouped all entries per date, operation (in this case we are only interested in PublicLogon), and part of the field ‘operation-specific’; more exactly, the legacyDN part which tells which (Public Folder) mailbox was accessed:
SELECT EXTRACT_PREFIX([#Fields: date-time], 0, ‘T’) As Date, Count (*) as Total, [Operation],
EXTRACT_PREFIX(EXTRACT_SUFFIX([operation-specific], 0, ‘cn=’), 0, ‘ in database ‘) as PFMailbox
FROM ‘[LOGFILEPATH]’
WHERE [operation]=’PublicLogon’
AND [failures] IS NULL
GROUP BY Date, [Operation], PFMailbox
ORDER BY Date ASC
The output showed all Public Folder mailboxes were now accessed by clients, and logons to the Public Folder mailboxes were now (more or less) equally distributed:
Like this:
Like Loading...
A little notice on a potential helpful feature which was introduced to Outlook at some point, but I wasn’t aware of before (or it’s just new). At least the option is available in Outlook v1905 build 11629.20008 C2R; it might also be available in standalone.
Remediation
Note that unfortunately, recent versions of VSphere running Exchange virtualized workloads also have this issue. On the plus side, they allow for separate (de)installation of the file system driver (NSX File Introspection Driver) and the network driver (NSX Network Introspection Driver). I am pretty sure removing the network driver would suffice, which might be a viable solution for some folks as well.
![rss[1]](https://eightwone.files.wordpress.com/2013/05/rss1.png)
![exchange2007logo2[1]](https://eightwone.files.wordpress.com/2009/12/exchange2007logo21.jpg)
![technical_support_outage_advisory[1]](https://eightwone.files.wordpress.com/2012/12/technical_support_outage_advisory1-e1354800107909.jpg?w=584)
