Exchange data: NTFS vs. ReFS

chartFor Exchange, NTFS has been the file system of choice since time immemorial. In 2012, Windows Server 2012 introduced a new file system: Resilient File System or just ReFS. ReFS was designed to overcome some of the limitations of NTFS, in particular in the area of maintaining data integrity. More information on ReFS in comparison to NTFS can be found here.

At that time Windows Server 2012 went RTM, the latest version of Exchange, Exchange 2010, was not supported to run on ReFS. Present day, Exchange 2010 still doesn’t support ReFS. However, when Exchange 2013 entered the arena shortly after Windows Server 2012, it came with support for both NTFS and ReFS file systems. NTFS was still considered best practice, with ReFS being a supported option with the added recommendation to turn off ReFS’ integrity checking feature, and disabling it for Content Index-exclusive volume is optional. It may therefor come as no surprise that nearly all customers are deploying Exchange 2013 on NTFS volumes only.

That may change with Exchange 2016. As announced at Ignite 2015, for Exchange 2016 more emphasis will be put on following the Preferred Architecture design when deploying Exchange on-premises. The Exchange 2016 Preferred Architecture contains guidance to use ReFS formatted, BitLocker encrypted data volumes with Exchange 2016. The latter option is of course to protect organizations against theft of physical storage devices.

With some time to spare, I was interested to see what the impact would be on the storage performance when using NTFS or ReFS, and especially the performance penalty when enabling BitLocker on a volume. Similar to a comparison I did between Exchange 2010 and Exchange 2013 on different operating systems, I ran a JetStress 2013 test utilizing these 3 file systems to get a sense of what to expect.

The ESE engine files from Exchange 2013 CU8 were used for testing, along with the following parameters:

Mode Test Disk Subsystem Throughput
Thread Count 12 (fixed)
Min/Max DB Cache 32 MB / 256 MB
Ins / Del / Repl / Read % 40/20/5/35
Lazy Commits 70%
Run Background DB Maintenance True
Databases 1 x DB (186GB), 3 Copies
Running Time 2 Hours

Databases and logs were stored on a DAS SSD drive, and the volume was GPT partitioned with 64K allocation units. ReFS Integrity checking was disabled for the volume using:

Format-Volume –DriveLetter X -FileSystem ReFS -AllocationUnitSize 65536 -SetIntegrityStreams $false

The drive supported hardware encryption for BitLocker, which offloads encryption to the drive. You can verify that hardware encryption is used after enabling BitLocker on the volume by inspecting the BitLocker status using the manage-bde utility or Get-BitLockerVolume cmdlet:

image

As you can see from the EncryptionMethod property, this volume is protected using hardware-based BitLocker encryption. Perhaps needless to say, but the CPU performance penalty is substantial when using BitLocker with software encryption, and this mode is not to be used with I/O intensive applications like Exchange.

Note that if you deploy a Database Availability Group on ReFS formatted storage, and you want to use AutoReseed, you need to create or configure your DAG using the FileSystem parameter specifying ReFS, e.g.

New-DatabaseAvailabilityGroup -Name DAG1 -FileSystem ReFS

This makes sure that AutoReseed prepares volumes using the proper file system.

The results from the JetStress tests are show in the following table:

Test

NTFS

ReFS

ReFS+BitLocker

JetStress Version

15.0.658.4

ESE.DLL

15.0.1076.9

Operating System

6.2.9200.0

Overall Test Result

Passed

Passed

 

Passed

Achieved Transactional IOPS

1,613.13

1,407.55

-13%

1,379.98

-14%

Database Reads Average Latency (msec)

8.53

10.50

-23%

9.73

-14%

Database Writes Average Latency (msec)

12.80

20.80

-63%

19.98

-56%

Database Reads/sec

895.25

787.08

-12%

769.47

-14%

Database Writes/sec

726.48

628.55

-13%

618.65

-15%

Database Reads Average Bytes

35,220.22

35,375.26

0%

35,437.64

1%

Database Writes Average Bytes

34,389.82

34,510.95

0%

34,496.88

0%

Log Reads Average Latency (msec)

4.64

5.06

-9%

5.00

-8%

Log Writes Average Latency (msec)

5.16

7.22

-40%

6.73

-30%

Log Reads/sec

18.64

16.29

-13%

16.08

-14%

Log Writes/sec

87.25

72.81

-17%

73.82

-15%

Log Reads Average Bytes

232,562.72

232,562.01

0%

232,562.30

0%

Log Writes Average Bytes

25,005.97

26,210.03

5%

25,589.45

2%

Avg. % Processor Time

4.28

3.66

14%

3.60

16%

Some observations and notes:

  • ReFS caused a ~13-14% IOPS drop when compared to NTFS.
  • Using ReFS resulted in increased I/O latencies, especially write operations.
  • ReFS had a positive impact on the processor utilization, lowering average utilization by around 15%.
  • For some reason, average write latencies were lower using ReFS with BitLocker rather than without it (~10%).

Given the impact of file system choice on I/O performance and CPU utilization, I hope next versions of Exchange Server Role Calculator will feature an option to select which file system will be used to store Exchange data, as the difference in I/O performance and CPU utilization between NTFS and ReFS seems significant. Though this small test was performed with Exchange 2013 running on Windows Server 2012 R2, It could be that Exchange 2016 or the next version of Windows Server 2016 contain changes that will diminish the differences or perhaps even grant ReFS an advantage over NTFS. This is something we will only know after these products have shipped, something worth investigating later this year.

The JetStress reports can be found here.

I will finish with a short disclaimer: This test was only performed to get an indication of performance impact of using different file systems with Exchange 2013 utilizing identical hardware. The results are purely indicative, and not necessarily representative for other configurations nor meant to provide guidance or proof. Always test and validate your configuration using tools like JetStress before putting Exchange in production.

8 thoughts on “Exchange data: NTFS vs. ReFS

  1. Pingback: erdekes | IT?

  2. Pingback: IT/DEV Connections 2015 Wrap-Up | EighTwOne (821)

  3. Pingback: Exchange Server 2016 – Genel Bakış: Tasarıma ve dizayna dair. | IT diaries by barisca & seldaa

  4. Pingback: Старые песни о главном: Microsoft Exchange Server 2016 | ITband.ru

  5. Is there any recent comparison chart for Exchange 2016 as above for NTFS and ReFS file system. If possible please suggest the best suitable file system for 2016 Exchnage server.

    • None yet, but given that ReFS or NTFS is an OS feature, I assume it will show similar outcome. That might change to, when Windows Server 2016 goes GTA and reaches supported state for deploying Exchange. Also, for Exchange 2016, the preferred file system for Exchange databases (and logs) storage is ReFS, where it was supported for Exchange 2013 deployments.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s