Exchange and VMWare Guest Introspection

Ex2013 LogoIn this long overdue article, I would like to share an experience, where a customer was upgrading from Exchange 2010 to Exchange 2013. Note that this could also apply to customers migrating from Exchange 2007 or migrating to Exchange 2016 as well. The Exchange 2013 servers were hosted on VMWare vSphere 5.5U2; the Exchange 2010 servers on a previous product level.

The customer saw a negative impact on the end user experience of Outlook 2010 users, especially those working in Online Mode. Other web-based services like Exchange Web Services (EWS) were affected as well. The OWA experience was good.

Symptoms
After migrating end user mailboxes from Exchange 2010 to Exchange 2013 (but as indicated, this applies to Exchange 2016 as well), end users reported delays in their Outlook client responses, where sometimes Outlook seemed to ‘hang’ when performing certain actions like accessing a Shared Mailbox. Also, when opening up the meeting planner in order to schedule a room using Scheduling Assistant, it could take a significant amount of time, (i.e. minutes) before the schedule of all the rooms was being displayed.

The end users’ primary mailbox was configured to use Cached Mode, except for VDI users who used their primary mailbox in Online Mode. Shared Mailboxes were used in Online Mode due to the size (Outlook 2010, so no slider).

Analysis
First, the overall health of the Exchange environment was checked to exclude it as a potential cause. Exchange performance metrics were monitored, as well as Managed Availability status and events, logs like the RCA logs, and VMWare CPU Ready % to check for potential vCPU allocation issues (read: oversubscription). None of these metrics caused any reason for concern.

After reconfiguring the HOSTS file, in order to bypass the load balancer and direct traffic to a single Exchange server to simplify troubleshooting, the symptoms remained. Then, we checked:

  • TCP/IP optimization settings, e.g. RSS, Chimney, etc.
  • VMWare VMXNet3 offloading, e.g. Large Send Offload, TCP Checksum Offloading
  • VMWare VMXNet3 buffer settings

All those settings were also found to be on their recommended values.

We started digging in from the client’s perspective, and used WireShark to see what was going on on the wire. After filtering on the Exchange host, we saw the following pattern:

image

Note that this customer used SSL Offloading, so mailbox access took place on port 80 instead of 443 (RPC/http).

As you might notice, there is a consistent 200ms delay after the client receives its response (e.g. packets 106 and 110). When searching around for ‘200ms’ and ‘delay’, you may end up with articles describing the effect of the Nagle algorithm (Delayed ACK). Nagle is meant to reduce chatter on the wire, but can have a negative effect on near real-time communications, especially with small packets. Also, while 200ms might seem small, looking at the number of packets exchanged between Outlook and Exchange, this can add up quite quickly. Most of these articles will also describe a fix, recommending to configure a registry key TcpAckFrequency, and set it to 1 (default is 2). For testing purposes, we configured this key and after the mandatory reboot, the end user Outlook experience was snappy. However, setting this key would impact all client communications (real as well as VDI clients); not a recommended long-term solution due to side effects on the network.

After removing the registry key, investigating was continued. Since there was no issue with the Exchange 2010, we started to suspect there was perhaps an issue with VMWare, or there was some form of network optimization or packet inspection going on. This, due to the fact there was no problem with the old Exchange environment, and the elements that changed when migrating were VMWare vSphere version, physical vSphere hosts, and last but not least, the protocol switched. This client didn’t use Outlook Anywhere, so RPC/http was not enabled for Exchange 2010 prior to migration, and clients connected using MAPI. After some more investigating, some potentially related articles on the VMWare knowledgebase were found, talking about latency issues in certain VMWare Tools versions, the VMWare guest driver set, and downgrading these to 5.1 would have the same effect as configuring TcpAckFrequency. Unfortunately, this wasn’t an option as the hardware level of the VMWare guests already was on a certain level.

introRemediation
When installing VMWare Tools, the package comes with some system-level drivers which handle communications between the guest and the host or other guests. One of these drivers is the VMWare Guest Introspection driver (or VMCI Drivers, and formerly VShield Drivers). This component can be identified in the guest in the presence of the system drivers vnetflt and vsepflt, and accommodates agentless antivirus solutions like McAfee MOVE. However, it seems to also interfere with certain workloads in their driver ecosystem, thus negatively impacting real-time communications. I wasn’t able to test if the change from MAPI to RPC/http (or later MAPIhttp) also contributed to this effect, as the Introspection driver may not scan MAPI RPC packets at all, in which case there is no overhead introduced.

Needless to say disabling the Guest Introspection component might be less desirable for some organizations, and in those cases, when you experience this issue, I suggest contacting your VMWare representative, after verifying your VMWare Tools are part of the list of recommended versions.

In the end, in this situation Guest Introspection was disabled and a file-level scanner was introduced (with the required exclusions, of course). Performance for Online Mode was optimal when accessing Online Mode mailboxes, and using Exchange web services like Scheduling Assistant showed room planning in seconds rather than minutes.

image.pngNote that unfortunately, recent versions of VSphere running Exchange virtualized workloads also have this issue. On the plus side, they allow for separate (de)installation of the file system driver (NSX File Introspection Driver) and the network driver (NSX Network Introspection Driver). I am pretty sure removing the network driver would suffice, which might be a viable solution for some folks as well.

If you have any insights to share, please leave them in the comments.

HTTP Proxy TargetBackEnd limits

powershellLast Update: February 4th, 2016

When deploying Exchange 2013 or Exchange 2016 in co-existence with a legacy version of Exchange, there comes a point where all traffic is routed through Exchange 2013/2016. Traffic for mailboxes hosted on legacy Exchange versions will be proxied by Exchange 2013/2016 to the back end.

This proxy process has some built-in limits for certain protocols, which you could encounter. Symptoms of these limits are Event 2022’s being logged in the Application log by the MSExchange Front End HTTP Proxy service:

image

Per Exchange 2013 CU7, this message should be considered a notice, despite the confusing event description. No connections are being blocked. However, the events create noise in your logs, which can be prevented by raising these limits. To accomplish this, you need to dive in to the web.config of the applicable HTTP Proxy protocols:

  • $ExInstall\FrontEnd\HttpProxy\sync\web.config (for ActiveSync, EAS)
  • $ExInstall\FrontEnd\HttpProxy\rpc\web.config (for OA, RPC/http)

In those files, create or adjust the entry in the <appsettings> configuration node, where <value> is the limit you want to configure (default is 150):

<add key=”HttpProxy.ConcurrencyGuards.TargetBackendLimit” value=”<value>” />

After adjusting these values, recycle the relevant application pools, e.g. MSExchangeSyncAppPool and MSExchangeRPCProxyAppPool.

The above steps need to be performed on all Exchange 2013/2016 Client Access Servers.

To automate this process of tedious editing in web.config files, I have created a small script which lets you alter these values for EAS and RPC against the local server or remotely. The script, Configure-HTTPProxyTargetBackEnd.ps1, has the following parameters:

  • Server to specify server to configure. When omitted, will configure local server.
  • AllServers to process all discoverable Exchange Client Access servers
  • TargetBackEnd specifies Target Backend limit (default 150).
  • NoRecycle to prevent recycling the MSExchangeSyncAppPool and MSExchangeRPCProxyAppPool

For example, to configure the local server with a limit of 2000 for Exchange Active-Sync and RPC access, use:

.\Configure-HTTPProxyTargetBackEnd.ps1 -TargetBackEnd 2000

image

Note that the script will create a backup copy of the web.config files before editing, using the current timestamp.

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision
See TechNet Gallery page.

Blocking Mixed Exchange 2013/2016 DAG

Ex2013 LogoIn the RTM version of Exchange 2016, there’s an issue in that it is allows you to add Exchange 2016 Mailbox servers to Exchange 2013 Database Availability Groups, and vice-versa. As stated in the Release Notes (you do read those?), creating such a mixed version DAG is not supported. In theory, you could even jeopardize your Exchange data, as database structures from both versions are different. This action is also not prevented from the Exchange Admin Center, requiring organizations to have very strict procedures and knowledgeable Exchange administrators.

If you are worried about this situation and you want to prevent accidently adding Mailbox servers to an existing DAG consisting of members of a different Exchange version, there is a way (until this is blocked by the product itself, of course). Cmdlet Extension Agents to the rescue!

The Scripting Agent not only allows you to add additional instructions to existing Exchange cmdlets, but also to provide additional validation before cmdlets are executed. I did two short articles on Cmdlet Extension Agents’ Scripting Agent here and here, so I will skip introductions.

First you need to download a file named ScriptingAgentConfig.xml from the location below. If you already have Scripting Agents, you need to integrate the code in your existing ScriptingAgentConfig.xml files. The code checks if the server you want to add using the Add-DatabaseAvailabilityGroup cmdlet is of a different major version than one of the current DAG members.

Next, you need to copy this ScriptingAgentConfig.xml file to $ENV:ExInstallPath on every Exchange 2013 and Exchange 2016 server in your organization, e.g. C:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml.  To help your with this process, Exchange fellow Paul Cunningham made a small script to push this XML from the current folder to every Exchange server in your organization, PushScriptingAgentConfig.ps1.

Last step is to enable the Scripting Agent using:

Enable-CmdletExtensionAgent ‘Scripting Agent’

After distributing the scripting agent file and enabling the scripting agent, when you try to add an Exchange 2016 (version 15.1) server to an Database Availability Group consisting of Exchange 2013 Mailbox servers, using Add-DatabaseAvailabilityGroupServer, you will receive an error message:

DAGCheck

This also works vice-versa, thus when you inadvertently try to add Exchange 2013 servers to an Exchange 2016 Database Availability Group, provided you distributed the XML on the Exchange 2013 servers as well. The error is also thrown when you try to perform this action using the Exchange Admin Console.

You can download the ScriptingAgentConfig.XML for blocking Mixed Exchange 2013/2016 DAGs from the TechNet here.

Exchange 2010-2013 Migration and OAB

Ex2013 LogoLast year, Exchange fellows Andrew Higginbotham, Paul Cunningham as well as the Exchange Team reported on checking, and when necessary configuring, your Offline Address Book (OAB) in your current Exchange Server 2010 environment, prior to installing Exchange Server 2013. Not doing so could result in a complete download of the Offline Address Book created by Exchange Server 2013, titled ‘Default Offline Address List (Ex2013)’.

Today I received a report that there is a different symptom of configuration absence. In this case, the customer reported on the inability to download the offline address book, and upon further inspection the Autodiscover server did not report back on the offline address book URL to use. In other words, OAB information was absent from the Autodiscover response, and Outlook gets confused. Note that this issue was reported in Outlook 2010 after installing Exchange Server 2013 Cumulative Update 10. I’m not sure if this change in behavior was introduced in these later builds of Exchange 2013 or Outlook, but it’s still a good thing to know.

The remedy here of course is to configure any (Exchange 2010) mailbox database with unconfigured Offline Address Book setting, and point them to the default offline address book using:

Get-MailboxDatabase | Where-Object {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where-Object {$_.IsDefault -eq $True})

KEMP LoadMaster & HA Virtual ID

imageA small heads-up on something which you need to configure when deploying a Highly Available setup of physical or virtual KEMP LoadMaster devices in environments with redundant network routing components, but this may apply to other components with similar functionality as well. While in typical environments the LoadMaster’s default setting will never be an issue, it can easily be overlooked or not immediately considered suspect when you do have issues, for example in hosted environments.

Note: If you are looking for more information on load balancing Exchange 2013 using KEMP LoadMaster devices, Exchange-fellow Jeff Guillet did an excellent multi-part write-up on this topic here.

When configuring multiple LoadMaster’s in a High Availability setup, one of the settings is the HA Virtual ID parameter, which is located System Configuration > Miscellaneous Options > HA Parameters. This setting configures the routing identifier used by the LoadMaster as part of the VRRP or Virtual Router Redundancy Protocol (see RFC5798).

The HA Virtual ID is used to construct a unique MAC address, so that all devices in the same VRRP group can communicate. The MAC address uses a format as defined by VRRP, and is 00:00:5E:00:01:<ID> for IPv4 and 00:00:5E:00:02:<ID> for IPv6.  One device, the Master being the Active LoadMaster, owns the VRRP group and manages its MAC address and shared IP address.

As you can imagine, using the same identifier for multiple non-related devices on the same segment may cause unexpected behavior, like LoadMasters being unable to communicate with eachother, both HA LoadMasters thinking they are the Active HA node, or other disruptive behavior. This is likely caused by a device other than LoadMasters managing the VRRP group.

Therefor, it is recommended to always change the default value of ‘1’, but always consult with the network or hosting people which value to use, as different vendors use their own default ID. For example, Cisco may use a different default value than FortiNet or CheckPoint for their redundant networking components. Of course, you also need to use different values when using multiple HA LoadMaster deployments on the same segment.

Client Message Size Limits

powershellLast Update: Version 1.2, February 15th, 2017

Exchange Server 2013 and Exchange Server 2016 enforces certain message size limits when it comes to client messages. These limits are in-place so clients can’t generate excessive load on your Exchange environment. These limits are determined for various access methods in multiple web.config files on Exchange Client Access Servers as well as Mailbox Servers.

Sometimes you may have good reasons to increase those limits. For example, when migrating to Office 365 using a product like MigrationWiz, you may want to increase the limit for Exchange Web Service (EWS) requests to allow for migration of larger items. Another example is when you want to allow for bigger attachments in Outlook WebApp (OWA). On TechNet, there’s an article on how to reconfigure these limits. However, the process consists of editing multiple web.config files, replacing multiple values in the same file, and following this process on each Exchange 2013/2016 server in your environment. This is not only labor intensive and prone to error, but becomes tedious when you consider that each Cumulative Update will overwrite your web.config files.

But do not despair. To execute these changes for OWA and EWS, I have created a PowerShell script which will perform these tasks for you.

Requirements
Using the script requires Exchange 2013. You need to provide the server name (default is local server) or AllServers to apply to all Exchange 2013/2016 servers in your environment. The script will modify the web.config remotely using the system share (e.g. C$), using the location of the Exchange installation, and uses IISRESET tool to restart IIS. It will create a backup of the web.config before modifying it.

Notes:

  1. The script checks for running in elevated administrator mode when running against the local machine.
  2. Current version of the script requires Exchange Management Shell, to run Exchange cmdlets for checking installed roles a.o., as the web.config files which require editing depend on the installed roles.
  3. For OWA, add ~33% to the value you want to specify to compensate for encoding overhead.
  4. When connected to an Exchange server, the script processes the server hosting the EMS session last to prevent abortion caused by IIS reset.
  5. Script currently runs against Exchange 2013 or Exchange 2016.

Usage
The script Configure-ClientSizeLimits.ps1 uses the following syntax:

.\Configure-ClientSizeLimits.ps1 [-Server |-AllServers] [-OWA <size>] [-EWS <size>] [-EAS <size>] [-Reset] [-NoBackup] 

A quick walk-through on the parameters and switches:

  • Server specifies the server to configure. When omitted, it will configure the local server. This parameter is mutually exclusive with AllServers.
  • AllServers switch specifies to configure all Exchange 2013 servers. This switch is mutually exclusive with Server.
  • OWA configures the message size limit for Outlook Web Access. Value is in bytes.
  • EWS configures the message size limit for Exchange Web Services. Value is in bytes.
  • EAS configures the message size limit for Exchange ActiveSync. Value is in bytes.
  • NoBackup tells the script not to make backup copies of modified web.config files.
  • Reset switch specifies to perform an IISRESET against servers after reconfiguration of client-specific message size limits.

So, suppose you want to configure OWA, EWS and EAS client message size limits on the local Exchange server, you can use:

.\Configure-ClientSizeLimits.ps1 -OWA 15MB -EWS 15MB EAS 15MB

cs

If you want to configure EWS limits for all servers, resetting IIS, you could use:

.\Configure-ClientSizeLimits.ps1 -AllServers -EWS 10240 -Reset

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision
See TechNet Gallery page.

To Do
Compatibility with Exchange 2010 and removal of dependency on Exchange Management Shell.

(Re)configuring IM Integration

Last Update: August 16th, 2016.

powershellNote: The procedure has changed for Exchange 2016, which can use overrides to make this setting persistent. For these instructions, consult this article.

Anyone who has configured Exchange 2013 IM integration with Lync Server at some point has to modify the web.config file on the Mailbox servers to configure OWA with the proper certificate for enabling IM. Another thing (read: nuisance) is that when you have configured IM integration and you apply a Cumulative Update to Exchange 2013, the web.config will be overwritten, in which case you need to reapply those changes to the web.config file.

This is where the script Configure-IMIntegration.ps1 might come in handy.

Requirements
Using the script requires Exchange 2013 and Lync Server. You need to provide the Lync pool and the Mailbox server you want to configure needs to have a valid certificate assigned to IIS (or UM) service. The script will modify the web.config remotely using the system share (e.g. C$), using the location of the Exchange installation, and uses WMI to recycle the OWA Application Pool in IIS. It will create a backup of the web.config before modifying it.

Note that the script does not perform the following steps:

  • It does not perform the Lync Server parts to configure IM integration, e.g. configure Exchange as a trusted application.
  • It does not configure Lync Server as an partner application for Exchange (Configure-EnterprisePartnerApplication.ps1).

Usage
The script Configure-IMIntegration.ps1 uses the following syntax:

.\Configure-IMIntegration.ps1 [-Server <String>[]] -PoolFQDN <String> [-AllCAS] [-AllMailbox] [-UM] [-Thumbprint <String>]

A quick walk-through on the parameters and switches:

  • Server specifies the server(s) to configure. When omitted, it will configure the local server. This parameter is mutually exclusive with AllMailbox.
  • AllMailbox switch specifies to configure all Mailbox servers. This switch is mutually exclusive with Server.
  • AllCAS switch specifies to enable IM integration on all Client Access servers.
  • PoolFQDN specifies the FQDN of the Lync Pool to use. This parameter is required.
  • UM specifies that the script should look for a certificate assigned to UM services instead of IIS (default).
  • Thumbprint to manually specify the thumbprint of the certificate to use.

So, suppose you want to quickly reconfigure IM integration on a Mailbox server after applying a Cumulative Update, you can use:

.\Configure-IMIntegration.ps1 -PoolFQDN lync.contoso.com –Server exchange01.contoso.com

image

Or, you can quickly configure Mailbox servers and CAS servers for IM integration after performing the required steps to configure the trusted application settings and installing and assigning the certificate for UM:

.\ Configure-IMIntegration.ps1 -PoolFQDN lync.contoso.com -AllMailbox –AllCAS

image

Also, in the example above, the CAS servers had already been enabled for IM.

Download
You can download the script from the TechNet Gallery here.

Feedback
Feedback is welcomed through the comments. If you got scripting suggestions or questions, do not hesitate using the contact form.

Revision History
See TechNet Gallery page.

Impersonation: To be, or pretend to be

imageAs frequent readers of this blog may know, I made several Exchange-related scripts available to the community. Some of these scripts make use of what is called Exchange Web Services (EWS). I receive lots of questions via e-mail and through the comments about configuring impersonation or permission-related issues when running those scripts, which support delegated access as well as impersonation, against mailboxes. This blog shows how can configure delegation, why you should use impersonation, and how to configure impersonation on Exchange 2007 up to Exchange 2013 and Exchange Online in Office 365.

Introduction

EWS provides functionality to allow client applications, such as Outlook or OWA apps, tools, or in my case scripts, to communicate with Exchange server. Even Exchange itself makes uses of EWS when performing Free/Busy lookups by the Availability services for example. EWS was introduced in Exchange Server 2007 back in December 2006, which now seems decades ago.

Some of these EWS scripts or tools access or even manipulate mailbox contents. In the MAPI era, in order for you to access a mailbox that’s not yours, you required delegated full access permissions. These permissions could be granted at the mailbox, mailbox database or mailbox server level. The latter would grant you access to all mailboxes hosted in that mailbox database. For example, to grant an account Archibald full access permission on the mailbox of Nestor, you would typically use something like:

Add-MailboxPermission –Identity Nestor –User Archibald –AccessRights FullAccess –InheritanceType All

Note: Specifying InheritanceType is sometimes overlooked. Not specifying it only configures an Access Control Entry (ACE) on the top level folder (InheritanceType None), resulting in symptoms like scripts not processing subfolders for example.

EWS enables you to use another access method besides delegation, which is impersonation. Impersonation, as the many online available dictionaries may tell to you, is ‘an act of pretending to be another person for the purpose of entertainment or fraud’ or something along those lines. In the Exchange world, this means you can have an account which has the permission to pretend to be the owner of the mailbox, including being subject to the same effective permissions. So, if for some reason the owner only has Read permission on a certain folder, so will the impersonator. Typical use cases for impersonation are for example applications for archiving, reporting or migration, but also scheduled scripts that need to process mailboxes could be one.

Before we dive into the configuration itself, first some of the reasons why you should should prefer Impersonation over delegated access:

  • No mailbox needed for the account requesting access.
  • Throttling benefits, since the operation is subject to the throttling policy settings configured on the mailbox accessed, not the throttling policy configured on the mailbox requesting access. To bypass these delegate limits, one had to configure and assign a separate throttling policy with no limits for the account. Of course, a bad behaving application could then run without boundaries from a resource perspective, something throttling policies try to limit.
  • In Exchange 2010 and up, impersonation leverages Role Based Access Control, which is better manageable than a collection of distributed  ACEs.
  • Actions performed by the impersonator are on behalf of the impersonated. This may complicate auditing, as logging will come up with actions performed by the impersonated user, not the impersonator.

Note that where ‘user’ is specified below with regards to granting permissions, one could also specify a security group as well unless mentioned otherwise.

Impersonation on Exchange 2007

On Exchange 2007, you configure impersonation by granting the following two permissions:

  • The ms-Exch-EPI-Impersonation permission grants the impersonator the right to submit impersonation calls. It is configured on Client Access Servers. This does not grant the impersonation right, just the right the make the call through a CAS server.
  • The ms-Exch-EPI-May-Impersonate when granted, allows the impersonator to impersonate selected accounts.

To configure these permissions in your Exchange 2007 environment, use:

Get-ClientAccessServer | Add-AdPermission –User svcExchangeScripts –ExtendedRights ms-Exch-EPI-Impersonation

Then, we can configure impersonation permission on the mailbox level:

Get-Mailbox Tintin| Add-ADPermission –User svcExchangeScripts –ExtendedRights ms-Exch-EPI-May-Impersonate

on the database level:

Get-MailboxDatabase MailboxDB1 | Add-ADPermission –User svcExchangeScripts –ExtendedRights ms-Exch-EPI-May-Impersonate

or mailbox server level:

Get-MailboxServer MailboxServer1 | Add-ADPermission –User svcExchangeScripts –ExtendedRights ms-Exch-EPI-May-Impersonate

Be advised that members of the various built-in Admin groups are by default explicitly denied impersonation permissions on the server and database level, and deny overrules allow. You will notice this when querying impersonation configuration settings, for example on the database level (in the screenshot example, olrik was granted impersonation permissions):

Get-MailboxDatabase | Get-AdPermission | Where { $_.ExtendedRights –like ‘ms-Exch-EPI-Impersonation’} | Format-Table Identity, User, Deny, IsInherited, ExtendedRights –AutoSize

image

Note that permissions assigned on the mailbox may not immediately be reflected as you are administering them in Active Directory. Changes in Active Directory are subject to AD replication, and the Exchange Information Store caches information for up to 2 hours, so worst case it may take up to 2 hours and 15 minutes for new permission settings to be re-read from Active Directory.

Impersonation on Exchange 2010 and 2013

Exchange 2010 introduced Role Based Access Control, better known by its acronym RBAC. For a quick introduction to RBAC, see one of my earlier blogs here. There is a management role associated with impersonation, which is ApplicationImpersonation.

To enable a user impersonation rights, create a new assignment for ApplicationImpersonation and assign it to the user:

New-ManagementRoleAssignment –Name 'AIsvcExchangeScripts' –Role ApplicationImpersonation –User svcExchangeScripts

Note that if we want to assign these permissions to a security group, we need to use the SecurityGroup parameter instead of User, specifying the group name.

Now be careful, when used like this you will have granted that user or group permission to impersonate all users in your Exchange organization. Here is where RBAC comes into play, or more specific the RBAC feature named management role scopes. With write scopes for example, you can limit the scope of where you can make changes in Active Directory. For more information on management role scopes, see here.

Let  us assume we want to limit the scope to a distribution group named ‘All Employees’, using New-ManagementScope in combination with RecipientRestrictionFilter. Note that when specifying MemberOfGroup in the filter, you need to use the distinguishedName of the group:

New-ManagementScope –Name 'Employee Mailboxes' –RecipientRestrictionFilter { MemberOfGroup –eq 'CN=All Employees,OU=Distribution Groups,OU=NL,DC=contoso,DC=com'} 

We can then apply this scope to the assignment created earlier:

Set-ManagementRoleAssignment –Identity 'AIsvcExchangeScripts' –CustomWriteScope 'Employee Mailboxes'

Be advised that in a multi-forest environment, impersonation doesn’t work when you assign permissions to cross-forest accounts. You either need to assign impersonation permissions to an account residing in the same forest as Exchange, or create a linked role group.

Impersonation on Exchange Online

Impersonation is available in most Office 365 plans, but currently not in the small business plans.  To configure Impersonation in Exchange Online we need to connect anyway, so we’ll first open a remote PowerShell session to Exchange Online:

$EXO= New-PsSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -AllowRedirection -Authentication Basic
Import-PsSession $EXO

Provide tenant administrator credentials when prompted. You can then see if you have the ApplicationImpersonation role at your disposal using:

Get-ManagementRole –Identity ApplicationImpersonation

If nothing is returned, you may need to resort to delegate access permissions.

Configuring impersonation is identical to configuring it in Exchange 2013. Nonetheless, some people may be more comfortable using the Exchange Admin Center. If so:

  1. Open up Exchange Admin Center.
  2. Navigate to Permissions > Admin Roles
  3. Now we can’t directly assign a management role through EAC, so assume we’ll create a role group for our application account by clicking New (+).
  4. Enter a name for your role group, e.g. ExchangeMaintenanceScripts.
  5. Add the role ApplicationImpersonation.
  6. Add the accounts which need Impersonation permissions, e.g. svcExchangeScript.
  7. Optionally, you can also select a Write Scope, which you need to create upfront through Exchange Management Shell.
  8. In Exchange on-premises, instead of a Write Scope you will have the option to select a a specific OU instead (scope filter RecipientRoot parameter) .
  9. When done, Save.

image

One word of caution: scopes are not automatically updated when objects referenced are relocated or change names. Now, for your own environment you may have this under control through some form of change management process. For Exchange Online however, your tenant might get relocated without notice. Therefor, should impersonation fail, verify any management scopes you may have defined for distinguishedName references, and check if they require updating, e.g.

Set-ManagementScope -Name 'All Employees' -RecipientRestrictionFilter { MemberOfGroup -eq 'CN=All Employees,OU=contoso.onmicrosoft.com,OU=Microsoft Exchange Hosted Organizations,DC=EURPR05A001,DC=prod,DC=outlook,DC=com'}

Final words

Note that many EWS-based scripts or tools do not natively support EWS but make use of the Exchange Web Services Managed API. This installable package consists of support files (e.g. DLL’s) which provide EWS functions to your PowerShell environment. You can download the current version of EWS Managed API here (2.2). You can read more on developing with EWS Managed API here, or you can have a peek at the source of code of one of my EWS scripts or the ones published by Exchange MVP-fellow Glen Scales’ here.

Internal Message Classifications visible in Outlook

Ex2013 LogoMessage classifications were introduced with Exchange 2007 which seems like ages ago now. They are a piece of metadata which you can assign to messages, for example the intended audience or sensitivity of messages. These message can then be treated accordingly by the recipient or you can leverage transport rules functionality and Rights Management Services to act on or protect these messages.

Let’s assume you have created a custom message classification using the following cmdlet:

New-MessageClassification –Name ‘InternalUseOnly’ –DisplayName ‘Internal Use Only’ –SenderDescription ‘This message is for internal use only.’

When you retrieve the list of message classifications using Get-MessageClassifications you will notice three additional classifications:

image

Exchange comes with these message classifications which are used by Exchange internally: ExAttachmentRemoved, ExOrarMail and ExPartnerMail. These should not be used by users, let alone be visible. To make them hidden, the PermissionMenuVisible attribute is set to $false for these classifications. This will make them not show up in Outlook WebApp:image

Now, using classifications in Outlook is less admin-friendly and requires exporting of classification information and configuring Outlook to read these classifications from a file. In short, the process described on TechNet TechNet to use message classifications from Outlook is as follows:

From the Exchange Management Shell, run the Export-OutlookClassification.ps1 script from Exchange scripts folder, e.g.

& ‘C:\Program Files\Microsoft\Exchange Server\v15\Scripts\Export-OutlookClassification.ps1’ | Set-Content ‘C:\OutlookClass.xml’

Next, copy the XML file to a location on the client or networked location which is readable by Users. On the client, make the following registry changes:

[HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Policy]
"AdminClassificationPath"="c:\\Classifications.xml"
"EnableClassifications"=dword:00000001
"TrustClassifications"=dword:00000001

Note: For the purpose of this example the XML is stored as C:\Temp\OutlookClass.xml . Note that “15.0” is for configuring Outlook 2013, replace with 14.0 for Outlook 2010 and 12.0 for Outlook 2007.

Restart Outlook so it will use these settings. When composing a message you will now see the message classification options appear under Options > Permission:

image

Apart from the message classification “Internal Use Only” we created, you will also see that Outlook shows the internal classifications by their display name. That should not be happening.

When you open up the Outlook classifications export file, you will spot that it contains all classifications, including the internal ones:image

So, what you can do now and what the documentation seems to fail to mention, is that after exporting message classifications you may want to remove the internal classifications “Attachment Removed” (ExAttachmentRemoved), “Originator Requested Alternate Recipient Mail” (ExOrarMail) and “Partner Mail” (ExPartnerMail) from the XML export file. Downside is that message with these internal classifications will not display the related description in Outlook, but that should not be an issue and a better option than users being able to select them.

When you have removed the three entries from the XML file and restarted Outlook, the built-in options will no longer be on the permission menu:

image

Can’t Create Mailboxes in Remote Sites

Ex2013 LogoRecently I got an e-mail from someone who had problems creating mailboxes in a new environment. When trying to create a mailbox, he received a following message stating, “Load balancing failed to find a valid mailbox database.” Apparently, the Mailbox Resources Management Agent (a Cmdlet Extension Agent) could not find an eligible mailbox database candidate.

image

The MRMA uses the following selection process when picking a candidate for mailbox creation or moving:

  1. Create a list of all mailbox databases;
  2. Remove databases marked for exclusion;
  3. Remove databases out of the management scope;
  4. Remove databases from remote (AD) sites;
  5. Pick a random online, healthy database from the list.

This person had a DAG, two mailbox databases (MDB1, MDB2) and two sites (AMS and LON).

We first checked the more or less obvious, which is to see if databases are not excluded from the provisioning process, so we entered Get-MailboxDatabase | fl *FromProvisioning:

image

Databases seemed enabled for provisioning. We then checked the status of the active database copies:

image

The copies looked healthy, but we noticed all databases were mounted in a remote site (derived from the server name starting with LON; we’re working from AMS). Looking back at the database selection process, it explained why it probably didn’t work and since the active copies should be moved back to the preferred site AMS anyway we moved the active copies back:

image

After moving the active database copies back to the location where we were performing our cmdlets from solved things.

Note that we could have discovered the issue using the Verbose parameter with the cmdlet. For example, New-Mailbox in conjunction with Verbose will show the selection process. The following screenshot shows an unsuccessful selection process considering available databases:

image

This screenshot shows a successful selection process.

image

More information on automatic mailbox distribution and controlling its behavior here.