Exchange data: NTFS vs. ReFS

Posted on May 16, 2015 by Michel de Rooij

For Exchange, NTFS has been the file system of choice since time immemorial. In 2012, Windows Server 2012 introduced a new file system: Resilient File System or just ReFS. ReFS was designed to overcome some of the limitations of NTFS, in particular in the area of maintaining data integrity. More information on ReFS in comparison to NTFS can be found here.

At that time Windows Server 2012 went RTM, the latest version of Exchange, Exchange 2010, was not supported to run on ReFS. Present day, Exchange 2010 still doesn’t support ReFS. However, when Exchange 2013 entered the arena shortly after Windows Server 2012, it came with support for both NTFS and ReFS file systems. NTFS was still considered best practice, with ReFS being a supported option with the added recommendation to turn off ReFS’ integrity checking feature, and disabling it for Content Index-exclusive volume is optional. It may therefor come as no surprise that nearly all customers are deploying Exchange 2013 on NTFS volumes only.

That may change with Exchange 2016. As announced at Ignite 2015, for Exchange 2016 more emphasis will be put on following the Preferred Architecture design when deploying Exchange on-premises. The Exchange 2016 Preferred Architecture contains guidance to use ReFS formatted, BitLocker encrypted data volumes with Exchange 2016. The latter option is of course to protect organizations against theft of physical storage devices.

With some time to spare, I was interested to see what the impact would be on the storage performance when using NTFS or ReFS, and especially the performance penalty when enabling BitLocker on a volume. Similar to a comparison I did between Exchange 2010 and Exchange 2013 on different operating systems, I ran a JetStress 2013 test utilizing these 3 file systems to get a sense of what to expect.

The ESE engine files from Exchange 2013 CU8 were used for testing, along with the following parameters:

Mode	Test Disk Subsystem Throughput
Thread Count	12 (fixed)
Min/Max DB Cache	32 MB / 256 MB
Ins / Del / Repl / Read %	40/20/5/35
Lazy Commits	70%
Run Background DB Maintenance	True
Databases	1 x DB (186GB), 3 Copies
Running Time	2 Hours

Databases and logs were stored on a DAS SSD drive, and the volume was GPT partitioned with 64K allocation units. ReFS Integrity checking was disabled for the volume using:

Format-Volume –DriveLetter X -FileSystem ReFS -AllocationUnitSize 65536 -SetIntegrityStreams $false

The drive supported hardware encryption for BitLocker, which offloads encryption to the drive. You can verify that hardware encryption is used after enabling BitLocker on the volume by inspecting the BitLocker status using the manage-bde utility or Get-BitLockerVolume cmdlet:

As you can see from the EncryptionMethod property, this volume is protected using hardware-based BitLocker encryption. Perhaps needless to say, but the CPU performance penalty is substantial when using BitLocker with software encryption, and this mode is not to be used with I/O intensive applications like Exchange.

Note that if you deploy a Database Availability Group on ReFS formatted storage, and you want to use AutoReseed, you need to create or configure your DAG using the FileSystem parameter specifying ReFS, e.g.

New-DatabaseAvailabilityGroup -Name DAG1 -FileSystem ReFS

This makes sure that AutoReseed prepares volumes using the proper file system.

The results from the JetStress tests are show in the following table:

Test	NTFS	ReFS		ReFS+BitLocker
JetStress Version	15.0.658.4
ESE.DLL	15.0.1076.9
Operating System	6.2.9200.0
Overall Test Result	Passed	Passed		Passed
Achieved Transactional IOPS	1,613.13	1,407.55	-13%	1,379.98	-14%
Database Reads Average Latency (msec)	8.53	10.50	-23%	9.73	-14%
Database Writes Average Latency (msec)	12.80	20.80	-63%	19.98	-56%
Database Reads/sec	895.25	787.08	-12%	769.47	-14%
Database Writes/sec	726.48	628.55	-13%	618.65	-15%
Database Reads Average Bytes	35,220.22	35,375.26	0%	35,437.64	1%
Database Writes Average Bytes	34,389.82	34,510.95	0%	34,496.88	0%
Log Reads Average Latency (msec)	4.64	5.06	-9%	5.00	-8%
Log Writes Average Latency (msec)	5.16	7.22	-40%	6.73	-30%
Log Reads/sec	18.64	16.29	-13%	16.08	-14%
Log Writes/sec	87.25	72.81	-17%	73.82	-15%
Log Reads Average Bytes	232,562.72	232,562.01	0%	232,562.30	0%
Log Writes Average Bytes	25,005.97	26,210.03	5%	25,589.45	2%
Avg. % Processor Time	4.28	3.66	14%	3.60	16%

Some observations and notes:

ReFS caused a ~13-14% IOPS drop when compared to NTFS.
Using ReFS resulted in increased I/O latencies, especially write operations.
ReFS had a positive impact on the processor utilization, lowering average utilization by around 15%.
For some reason, average write latencies were lower using ReFS with BitLocker rather than without it (~10%).

Given the impact of file system choice on I/O performance and CPU utilization, I hope next versions of Exchange Server Role Calculator will feature an option to select which file system will be used to store Exchange data, as the difference in I/O performance and CPU utilization between NTFS and ReFS seems significant. Though this small test was performed with Exchange 2013 running on Windows Server 2012 R2, It could be that Exchange 2016 or the next version of Windows Server 2016 contain changes that will diminish the differences or perhaps even grant ReFS an advantage over NTFS. This is something we will only know after these products have shipped, something worth investigating later this year.

The JetStress reports can be found here.

I will finish with a short disclaimer: This test was only performed to get an indication of performance impact of using different file systems with Exchange 2013 utilizing identical hardware. The results are purely indicative, and not necessarily representative for other configurations nor meant to provide guidance or proof. Always test and validate your configuration using tools like JetStress before putting Exchange in production.

Geek Out with Perry series

Posted on August 21, 2010 by Michel de Rooij

As EHLO posted today, there is a new video of the Perry Clark, General Manager, Exchange from the Mailbox team. Its the latest addition to the Geek Out with Perry series, short (around) 10 minute videos hosted by Ann Vu (Product Manager), where various in-depth topics like Archiving, Storage and Data Protection are discussed. Perhaps not all of you are aware of the series, so for your convenience I’ve embedded the videos:

Answers to Questions on Archiving with Tiered Storage and Stubbing

Archiving in Exchange

Answering Data Protection Questions: Replication and Backups

Managing Storage Efficiently

Data Protection Evolution

For those interested, Perry blogs and videos can be found on Ask Perry.

Connecting StorCenter to ESXi using iSCSI

Posted on January 4, 2010 by Michel de Rooij

Recently I got myself an Iomega IX2-200 StorCenter. It’s a nice little device which will do nicely for my lab. When playing aroung with the device I wanted to connect it to my ESXi 4 servers using ISCSI. Yes, I’m running VMWare ESXi, main reason for that being one of my BSD guests and Hyper-V doesn’t do BSD.

Below are the steps I used to utilize the StorCenter as an ESXi datastore. I’ll be using the ESXi iSCSI Software Adapter, use CHAP (could’nt get Mutual CHAP to work, anyone?) and assume networking has been properly configured. Also, in this example we’ll be using VMFS volumes for VMDK storage, not Raw Device Mappings.

Note that during taking the screenshots I discovered a 1Gb test iSCSI target was too small (ESXi complained in the Add Storage / Select Block Size dialog), so I upped it to 16 GB using the StorCenter dashboard.

First enable iSCSI on the StorCenter. In the dashboard, select the Settings tab, click iSCSI and check Enable iSCSI. Leave iSNS discovery unchecked as ESXi doesn’t support it. Leave the option Enable two-way authentication (Mutual CHAP) unchecked.

Next, I’m going to add an iSCSI target. Select the Shared Storage tab and click Add. Change Shared Storage Type to iSCSI Drive and give it a name, e.g. esxtest. Then specify the initial size, e.g. 16Gb (you can increase this size when required). Leave Enable security checked. Click Next. Leave all User Access set to None; I’ll do that in the next step. Click Apply

Because I’m going to use CHAP I need to create an account on the StorCenter for the iSCSI initiator (i.e. ESXi) to authenticate itself. Select tab Users and click Add. Specify a Username and a password. This password MUST be between 12-16 characters. Uncheck Administrator and Add a secured folder for this user. Click Next; when asked about Group memberships click Next again. Now specify which users have access to which folders and iSCSI drives. Check the Read/Write option for the users created earlier and click Next.

In VI client, select the host’s Configuration tab and select Storage Adapters.

Select the iSCSI Software Adapter, e.g. vmhba34, and click Properties. Click Configure and make sure iSCSI is enabled (enabling may require restart). Click OK to close this dialog. Now, before connecting to the iSCSI target I’m going to specify the credentials first. I’ll use global settings so new connections will inherit these settings by default. To start configuring authentication, in the iSCSI Initiator Properties dialog on the General tab, click CHAP.
In the CHAP Credentials dialog, set CHAP to Use Chap and specify the Name and Secret (i.e. password) of the user created on the StorCenter. Since I’m not using Mutual CHAP, leave that setting to Do not use CHAP. Click OK.

Now I’m going to connect to the iSCSI target. Being lazy, select the Dynamic Discovery tab and click Add. Specify the address of the StorCenter. Click OK when done. The iSCSI server you just specified will now be added to the list of Send Targets. Click Close; when asked about rescanning the HBA select Yes.The iSCSI target will now be listed in the View section.
In the VI client, select Storage and click Add Storage. Select the Disk/LUN Storage Type and click Next. Select the added iSCSI target. Next. Next. Specify the name of the Datastore. Next. Specify the block size and required capacity. Next.