For Exchange, NTFS has been the file system of choice since time immemorial. In 2012, Windows Server 2012 introduced a new file system: Resilient File System or just ReFS. ReFS was designed to overcome some of the limitations of NTFS, in particular in the area of maintaining data integrity. More information on ReFS in comparison to NTFS can be found here.
At that time Windows Server 2012 went RTM, the latest version of Exchange, Exchange 2010, was not supported to run on ReFS. Present day, Exchange 2010 still doesn’t support ReFS. However, when Exchange 2013 entered the arena shortly after Windows Server 2012, it came with support for both NTFS and ReFS file systems. NTFS was still considered best practice, with ReFS being a supported option with the added recommendation to turn off ReFS’ integrity checking feature, and disabling it for Content Index-exclusive volume is optional. It may therefor come as no surprise that nearly all customers are deploying Exchange 2013 on NTFS volumes only.
That may change with Exchange 2016. As announced at Ignite 2015, for Exchange 2016 more emphasis will be put on following the Preferred Architecture design when deploying Exchange on-premises. The Exchange 2016 Preferred Architecture contains guidance to use ReFS formatted, BitLocker encrypted data volumes with Exchange 2016. The latter option is of course to protect organizations against theft of physical storage devices.
With some time to spare, I was interested to see what the impact would be on the storage performance when using NTFS or ReFS, and especially the performance penalty when enabling BitLocker on a volume. Similar to a comparison I did between Exchange 2010 and Exchange 2013 on different operating systems, I ran a JetStress 2013 test utilizing these 3 file systems to get a sense of what to expect.
The ESE engine files from Exchange 2013 CU8 were used for testing, along with the following parameters:
| Mode |
Test Disk Subsystem Throughput |
| Thread Count |
12 (fixed) |
| Min/Max DB Cache |
32 MB / 256 MB |
| Ins / Del / Repl / Read % |
40/20/5/35 |
| Lazy Commits |
70% |
| Run Background DB Maintenance |
True |
| Databases |
1 x DB (186GB), 3 Copies |
| Running Time |
2 Hours |
Databases and logs were stored on a DAS SSD drive, and the volume was GPT partitioned with 64K allocation units. ReFS Integrity checking was disabled for the volume using:
Format-Volume –DriveLetter X -FileSystem ReFS -AllocationUnitSize 65536 -SetIntegrityStreams $false
The drive supported hardware encryption for BitLocker, which offloads encryption to the drive. You can verify that hardware encryption is used after enabling BitLocker on the volume by inspecting the BitLocker status using the manage-bde utility or Get-BitLockerVolume cmdlet:
As you can see from the EncryptionMethod property, this volume is protected using hardware-based BitLocker encryption. Perhaps needless to say, but the CPU performance penalty is substantial when using BitLocker with software encryption, and this mode is not to be used with I/O intensive applications like Exchange.
Note that if you deploy a Database Availability Group on ReFS formatted storage, and you want to use AutoReseed, you need to create or configure your DAG using the FileSystem parameter specifying ReFS, e.g.
New-DatabaseAvailabilityGroup -Name DAG1 -FileSystem ReFS
This makes sure that AutoReseed prepares volumes using the proper file system.
The results from the JetStress tests are show in the following table:
| Test |
NTFS
|
ReFS
|
ReFS+BitLocker
|
| JetStress Version |
15.0.658.4
|
| ESE.DLL |
15.0.1076.9
|
| Operating System |
6.2.9200.0
|
| Overall Test Result |
Passed
|
Passed
|
|
Passed
|
|
| Achieved Transactional IOPS |
1,613.13
|
1,407.55
|
-13%
|
1,379.98
|
-14%
|
| Database Reads Average Latency (msec) |
8.53
|
10.50
|
-23%
|
9.73
|
-14%
|
| Database Writes Average Latency (msec) |
12.80
|
20.80
|
-63%
|
19.98
|
-56%
|
| Database Reads/sec |
895.25
|
787.08
|
-12%
|
769.47
|
-14%
|
| Database Writes/sec |
726.48
|
628.55
|
-13%
|
618.65
|
-15%
|
| Database Reads Average Bytes |
35,220.22
|
35,375.26
|
0%
|
35,437.64
|
1%
|
| Database Writes Average Bytes |
34,389.82
|
34,510.95
|
0%
|
34,496.88
|
0%
|
| Log Reads Average Latency (msec) |
4.64
|
5.06
|
-9%
|
5.00
|
-8%
|
| Log Writes Average Latency (msec) |
5.16
|
7.22
|
-40%
|
6.73
|
-30%
|
| Log Reads/sec |
18.64
|
16.29
|
-13%
|
16.08
|
-14%
|
| Log Writes/sec |
87.25
|
72.81
|
-17%
|
73.82
|
-15%
|
| Log Reads Average Bytes |
232,562.72
|
232,562.01
|
0%
|
232,562.30
|
0%
|
| Log Writes Average Bytes |
25,005.97
|
26,210.03
|
5%
|
25,589.45
|
2%
|
| Avg. % Processor Time |
4.28
|
3.66
|
14%
|
3.60
|
16%
|
Some observations and notes:
- ReFS caused a ~13-14% IOPS drop when compared to NTFS.
- Using ReFS resulted in increased I/O latencies, especially write operations.
- ReFS had a positive impact on the processor utilization, lowering average utilization by around 15%.
- For some reason, average write latencies were lower using ReFS with BitLocker rather than without it (~10%).
Given the impact of file system choice on I/O performance and CPU utilization, I hope next versions of Exchange Server Role Calculator will feature an option to select which file system will be used to store Exchange data, as the difference in I/O performance and CPU utilization between NTFS and ReFS seems significant. Though this small test was performed with Exchange 2013 running on Windows Server 2012 R2, It could be that Exchange 2016 or the next version of Windows Server 2016 contain changes that will diminish the differences or perhaps even grant ReFS an advantage over NTFS. This is something we will only know after these products have shipped, something worth investigating later this year.
The JetStress reports can be found here.
I will finish with a short disclaimer: This test was only performed to get an indication of performance impact of using different file systems with Exchange 2013 utilizing identical hardware. The results are purely indicative, and not necessarily representative for other configurations nor meant to provide guidance or proof. Always test and validate your configuration using tools like JetStress before putting Exchange in production.
The Exchange team published an update for the Exchange Server Role Requirements Calculator, the tool to aid you in properly sizing your Exchange Server 2013 or Exchange Server 2016 deployment.
EighTwOne (821) 