A short write-up after some recent articles which were published to clarify and emphasize Microsoft’s current position on virtualization and the support for storing Exchange information on NFS volumes. I will stick to the headlines, as the topic has already been touched several times by people from the Exchange community, after which I would mostly be repeating things that have already been said. Yet, many customers still have the perception that Exchange on NFS is supported or are actually running this configuration, often the result of a push from the storage or virtualization vendor. As it is not, I will repeat key information here to counter misleading information, hoping it might prevent customers from selecting unsupported configurations.
End of last year, a lively discussion was revived on some distribution lists and forums on why NFS was still not supported for storing Exchange information. However, it was all speculation as the creator of the product did not take part. The official support statement was (and is) that Exchange is not supported on NFS and only block-level storage is supported. Tony Redmond did a write-up on that here.
Then, in the preamble of the Microsoft Exchange Conference 2014, a ‘suggestion’ to support NFS was put on the community ideascale site, where people can propose suggestions for Exchange. This site is not an official channel but it does provide a way for the community to gather suggestions and check for demand. So, it allowed to verify if the current lack of NFS support was major thing or not, as people producing the most noise do not necessarily represent the majority. Response seemed limited, except for some hardware vendors who made lots of noise, possibly in an attempt to get traction in the Exchange community.
Then, Tony did a follow-up article after a discussion with Jeff Mealiffe, knowledgeable on Exchange, Sizing and Virtualization and nicknamed ‘The PerfGuy’ for obvious reasons. In the article, the problem areas of NFS are set out. Interestingly (but not surprising), Exchange is similar to SQL Server from a storage perspective, the latter having very specific documentation regarding storage requirements. Also mentioned is that successfully running JetStress by the vendor is no indication on the supportability of storage configurations. After all, that JetStress succesfully runs for a certain amount of hours is great, but it is a storage performance validation tool, not a storage supportability validation tool. At the Microsoft Exchange Conference 2014, using arguments presented earlier in the article, Jeff reaffirmed the non-support of NFS in his presentation.
The discussion seemed to die down until few weeks ago when Tony was in a Twitter conversation with one Josh Odgers, engineer at one of the storage vendors. In the discussion Odgers dropped the rationale and even went so far as to insult people. When searching online, you will find other rants as well, so I guess Josh’ employer does not have any form of social media guidelines for their employees. That does not help when you are trying to lobby for your cause (and potential markets for your storage appliances). Tony wrote an extensive response here, I recommend checking it out.
Now what storage vendors and their employees do or do not do is up to them. However, things like this may become an issue when vendors repeatingly and knowingly position their storage solution as a supported alternative to customers, like for example Odgers does for Nutanix (NDFS is Nutanix’ proprietary distributed NFS implementation). Yes, I’m sure it flies like a rocket and I am sure some customers will be persuaded by sales people to a game of chance by running Exchange on their appliances. As an Exchange consultant however, I prefer supported solutions and so should you. Or have a serious chat with the Risk Manager.
Update (Jul 9,2014): The UC Architects fellow Mahmoud Magdy posted a blog on his experiences and encountered limitations of storage appliances such as Nutanix here.
EighTwOne (821) 
Pingback: Weekly IT Newsletter – June 30–July 4, 2014 | Just a Lync Guy
Pingback: NeWay Technologies – Weekly Newsletter #102 – July 3, 2014 | NeWay
Pingback: NeWay Technologies – Weekly Newsletter #102 – July 4th, 2014 | NeWay
Pingback: Unified Boxes, The Sum of all fears | Be Busbared
Pingback: Intersting things I have seen on the internet, July 14th | 503 5.0.0 polite people say HELO
t’s pretty well known that Exchange on NFS Datastores from an enterprise-class NFS provider works just fine … as does Oracle, SQL, and pretty much any other workload that can be virtualized.
I would have recommended the same before I ever joined a storage vendor. It’s really simple. With VMware, the Microsoft OS see’s a completely compatible SCSI storage controller that implements everything that a physical SCSI controller with associated firmware does. Just in software. Even VMware has come out and said they see absolutely no technical issues with this deployment.
I wholly agree with Jeff’s statement that the following attributes are important for any storage solution used for Microsoft Exchange:
Forced Unit Access (FUA) and Write-Through, including statements such as “All components in a solution must honor the write-to-stable media intent. This includes, but is not limited to, caching components.”
Write Ordering. This is required to preserve the integrity of transactions going to the database (you clearly don’t want data written in the wrong order).
Torn I/O Protection. Storage must ensure that all of the data for a transaction is written and never reports success when partial writes occur. (This presentation is a quick read for those who want more information on data corruptions)
VMware as well as the Nutanix filesystem keeps these characteristic as a core part of their design. FUA and write-through are supported on VMware SCSI devices on top of NFS… and the Nutanix system will not acknowledge any writes until they are fully written to persistent underlying storage in *TWO* locations within our cluster. There is no cache’s or anything like that that are required to be flushed. Pull the plug on any Nutanix system and you will find we don’t lose or corrupt data. That is a key feature of any good storage array… a deep respect that a write acknowledgement is only performed after it has been persisted on persistent storage.
I believe that Microsoft (and Jeff) has been burnt in the past with shoddy NFS deployments, so they have rightly so retreated to what they are comfortable with. I do understand their stance and the fact they are just trying to minimize risk for the customer as well as reduce their role in certifying any storage equipment. But the risks outlined are not at the NFS or iSCSI layer, they could exist at any layer. All storage systems aren’t created equally, just like all hard drives and all firmware is not. Some storage systems have fundamental flaws. You aren’t safe on a JBOD either as we have found problems with the key concepts Jeff outlined on even device-level drivers for SSDs and HDDs. We have seen SSD and HDD caches that don’t respect FUA/Write-Through requirement. So your Microsoft Exchange deployment on Physical Servers and locally attached disks is at risk of data corruption and data loss due to the faulty firmware for the SSD/HDD.
This is something we do an incredible amount of data integrity testing for with our platform. We certify all layers of the stack including the SSD and HDD firmware. We pull powercords with active data runs and ensure our software and the firmware for the hardware can handle that while maintaining data integrity.
While Josh may have been a bit too passionate and cavalier his delivery, there is really no technical limitation specific to NFS. A good NFS implementation works as well as any other network-storage protocol for VMware to store VMs, and respect data integrity.
Customers should be educated that their real risk is using any platform that is not tested well for data integrity and the concepts Jeff outlined.
LikeLike
I’m sure your appliances might do a fine job, and maybe even your employer as well, supporting Exchange deployments, testing configurations and dedicating resources to the cause. But that is not the point.
First, corrupted databases due to storage issues will land on MS’ plate. Given the little interest in Exchange on NFS, I do not see this changing overnight.
Second, when asked to make recommendations, non-supported solution will not be one of them. NFS is not supported, end of story. If customers choose to ignore this for all the wrong reasons, being persuaded by sales people often one of them, and thereby implicitly accepting the potential consequences, which I tried to make explicit, that will be their call.
Would you care to elaborate on the ‘We have seen SSD and HDD caches that don’t respect FUA/Write-Through requirement’ statement?
LikeLike
Pingback: Unified Boxes, The Sum of all fears - SureSkillz