Exchange 2013 KB articles RSS feed

rss[1]Like most people I still use RSS feeds to keep track of news and updates from various sources. But not everyone is aware you can keep track of new or updated Microsoft’s knowledgebase articles using RSS feeds, sometimes categorized per product. I already blogged about the availability these feeds about 2,5 years ago.

Now with all the releases since then, it’s time to update this information with current products, especially with the feed for Exchange 2013 related articles becoming available recently:

For a complete list of the knowledgebase articles RSS feeds check here.

The case of the not updating Outlook for Mac 2011

exchange2007logo2[1]I had contact with a Twitter user on an issue with Outlook for Mac 2011 talking against Exchange Server 2007 on Small Business Server 2008.

When configuring a new account, Outlook for Mac reported “Account cannot be added.  Note that Outlook 2011 requires Exchange Server 2007 SP1 Update Rollup 4 or later.”

image

However, that couldn’t be right because that user claimed to be running a higher version of Exchange 2007. After manually entering the server name, a connection could be established and an initial download of folders and contents took place. However, items weren’t updated and contacts and calendar remained empty.

After trying and checking some things, I asked to turn on Outlook for Mac’s logging hoping to find something in the Exchange Web Services log (Outlook for Mac 2011 is EWS based). You can enable logging by checking Window > Error Log > Errors > Settings > Turn on logging for troubleshooting. After a while I was sent the log file Microsoft Outlook_Troubleshooting_0.log which contained the following excerpt:

2013-01-24 08:55:34.392,0xFFFFFFFF,Outlook Exchange Web Services,Info,"EWS: Response data received on thread=0x7d27bdb4, XML data=
<?xml version=""1.0"" encoding=""utf-8""?><soap:Envelope xmlns:soap=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xmlns:xsd=""http://www.w3.org/2001/XMLSchema""><soap:Header><t:ServerVersionInfo MajorVersion=""8"" MinorVersion=""3"" MajorBuildNumber=""297"" MinorBuildNumber=""0"" Version=""Exchange2007_SP1"" xmlns:t=""http://schemas.microsoft.com/exchange/services/2006/types"" /></soap:Header>

First, Exchange reports version 8.3.297.0 which corresponds with Exchange 2007 SP3 RU9 (EWS can report slightly different version than actual version), so something else was wrong while that’s well above Exchange 2007 SP1 RU4.

2013-01-24 08:55:39.355,0xFFFFFFFF,Outlook Exchange Web Services,Info,"EWS: Sending request on connection=0x7dc89be8, URL=/EWS/Exchange.asmx, SoapAction=""http://schemas.microsoft.com/exchange/services/2006/messages/SyncFolderItems"""
2013-01-24 08:55:39.358,0xFFFFFFFF,Outlook Exchange Web Services,Info,EWS: Received response on connection=0x7d31dae8; status=500
..
2013-01-24 08:55:49.861,0xFFFFFFFF,Outlook Exchange Web Services,Info,"EWS: Sending request on connection=0x7d71a648, URL=/EWS/Exchange.asmx, SoapAction=""http://schemas.microsoft.com/exchange/services/2006/messages/GetItem"""	
2013-01-24 08:55:49.863,0xFFFFFFFF,Outlook Exchange Web Services,Info,EWS: Received response on connection=0x7dc26638; status=500
..
2013-01-24 08:55:39.359,0xFFFFFFFF,Outlook Exchange Web Services,Info,"EWS: Sending request on connection=0x7d31dae8, URL=/EWS/Exchange.asmx, SoapAction=""http://schemas.microsoft.com/exchange/services/2006/messages/GetItem"""	
2013-01-24 08:55:39.477,0xFFFFFFFF,Outlook Exchange Web Services,Info,EWS: Received response on connection=0x7d7005c8; status=200

I then noticed various EWS requests returned http status code 200 (means OK) but also 500′s, which correspond to “Internal Server Error”. It happened after various requests (e.g. SyncFolderItems, GetFolder, GetItem) but not for all requests.

Now, code 500 isn’t very helpful (general terminal failure) and a quick restart of IIS with iisreset /restart /noforce didn’t solve things.

After some digging it turned out the seemingly unrelated KB2264110 pointed in the right direction. I say unrelated, because it’s on messages not being updated on Blackberry Internet Service (BIS) after installing Exchange Server 2007 SP2. Turned out the performance counters on the Exchange 2007 server were corrupt and rebuilding them solved the issue.

To rebuild the performance libraries, perform the following steps from an elevated command prompt:

  1. CD %SystemRoot%\System32
  2. Run lodctr /R (/R is case-sensitive) which will rebuild all known counters
  3. Run wmiadap /f which will update the WMI performance classes
  4. Restart the Exchange 2007 server

After these steps, Outlook for Mac 2011 could sync again with Exchange Server 2007 SP3.

Exchange and potential Packet Loss on VMWare

technical_support_outage_advisory[1]Yesterday, I noticed a VMware knowledgebase article, updated on November 14th, which could be worth taking notice of when you’re running Exchange – or any other application – in a virtualized environment based on VMware technology.

VMware’s KB article 2039495 mentions that in VMware ESXi 4.x and 5.x, very high traffic bursts may cause the VMXnet3 driver to start dropping packets in the Guest OS. This has been observed on Windows Server 2008 R2 running Exchange 2010 with – as VMware puts it – a high number of Exchange users. What the article fails to mention is the configuration used by customers experiencing the issue. It might for example be valuable to know if a DAG was used, if the traffic (MAPI, replication) was split over multiple NICs or if it occurred with iSCSI storage. I won’t be surprised if the issue occurs with other high traffic situations as well, e.g. seeding. Luckily, Exchange is capable of handling certain hiccups so customers might not be even aware of the issue.

After some more digging I found another article, KB 1010071, which mentions a packet drop issue with VMware Guests known since ESX 3. This article explains a bit more why the issue occurs in the first place, being the network driver running out of receive buffers, causing the packets to be dropped between the Virtual Switch and the Guest OS driver.

One could argue about the impact of a few lost packets. However, as traffic increases the (potential) number of lost packets increases. Each lost packet results in retransmission of unacknowledged packets, which impacts overall throughput causing increased latencies.

VMware’s temporary solution to this problem is:

  1. Open up the Windows guest;
  2. Open the properties of the VMXNET3 NIC;
  3. On the Advanced tab, increase the Small Rx Buffers or Rx Ring #1 Size;
  4. What KB1010071 mentions and KB2039495 doesn’t, is that when using jumbo frames – not seldom used, e.g. replication  – you might need to adjust the Rx Ring #2 size and Large Rx Buffers values.

Now I say temporary, because VMware’s solution of course isn’t  a real solution; it’s only meant to – in their own words – reduce packet drops. Also, the KB1010071 article states you should “determine an appropriate setting by experimenting with different buffer sizes”. That doesn’t sound like an permanent, assuring solution for a virtualization environment running business critical applications now, does it?

All things considered, I’d recommend configuring these parameters to their maximum setting, preferably at installation time, unless anyone knows of a reason not to. In addition, this is another case for the best practice to split MAPI and replication traffic on Exchange using multiple NICs.

Finally, I already learnt of two other applications experiencing the issue. Therefor I think the problem is not Exchange 2010 specific, as KB2039495 might imply. If you have similar experiences, experienced differences between GbE and 10Ge, please use the comments to share.

Review: Exchange Data Center Switchover Tool (Updated)

Last week, the Exchange team released what they called the “Exchange 2010 datacenter switchover tool” (note that the title mentions troubleshooter). The tool could prove helpful to some and can be insightful to others.

While I applaud any effort put in to minimize risks and the possibility of human error, especially in stressful situations like data center switchovers, I do have some suggestions for improvement.

First, the name. A “tool” might imply it’s something to aid in the switchover process, while in fact it’s more of an interactive decision maker or guide walking you through the process and can be utilized to practice dry runs or test formalized procedures.

That brings me to my second point, which is the format. A process like a data center switchover with all its decision moments is perhaps better translated to a flow chart rather than an interactive PowerPoint slide deck, which looks good on screen but can’t be printed. Also, a PDF or XPS might be more convenient; not everyone has PowerPoint at hand all the time, especially when working remotely on servers.

Finally, the contents is almost taken directly from the original Technet data center switchover article here, with the same questions and steps. It could perhaps be turned in a more valuable tool if it could read the environment and tailor questions based on what it discovers.

You can check out the “troubleshooter” yourself by downloading it here. Of course, this is only the first version; I suggest you leave feedback and suggestions on how to improve the tool in the accompanying article on the Exchange Team blog here.

Update October, 24th:UC Architects fellow Serkan Varoglu created a Exchange Data Center Switchover workflow diagram; you can download it here.