Configuring Exchange Online with IMAP & OAuth2

Not too long ago, the Exchange product group enabled Modern Authentication (or OAuth2) support for IMAP and SMTP in Exchange Online, and shortly after for POP3 as well. This support was much needed with the imminent deactivation of Basic Authentication. With Modern Authentication available, vendors, developers as well as organizations running custom scripts are given time to adopt Modern Authentication where applicable.

By delaying the original end date of Basic Authentication from October 13, 2020 to Q3’ish 2021 due to the Corona situation, the adoption period is increased significantly. That does not mean however developers and organizations can sit back and relax: Act sooner rather than later, the end of Basic Authentication is nigh.

The benefits of Modern Authentication are of course that it is a more secure model (e.g. resistant to password spray attacks), as well that it can leverage Microsoft 365 functionality like Conditional Access to limit protocols to certain locations.

That said, in this article I will show you how to approve usage of a popular 3rd party e-mail application Thunderbird, using IMAP protocol in conjunction with the Modern Authentication scheme. The procedures below have been run against Thunderbird 78.0b4 on Windows as well as Ubuntu.

Third Party Applications
Before we move on to Thunderbird, we first make sure the organization settings allow for third party applications to access your mailbox Exchange Online. This process has been blogged about for common popular applications, such as the native iOS Mail app or the Gmail app on Android. So, how to go ahead if your organization restricts access to third party applications, and they only want to allow specific applications, which is of course good practice.

The easiest way to add Thunderbird to the allowed applications and grant consent to the organization, is by constructing an admin consent URL. To construct the consent URL, take the following URL:

https://login.microsoftonline.com/<TenantID>/oauth2/authorize?client_id=<AppID>&response_type=code&prompt=admin_consent

and,

  1. Replace <TenantID> with your Tenant ID. This piece of information can be found under the Azure Active Directory blade in the Azure portal.
  2. Replace <AppID> with the Application ID (sometimes also referred to as Client ID) of the application you want to provide consent for. As we can see in the table below, the ID of Thunderbird is 08162f7c-0fd2-4200-a84a-f25a4db0b584.
ApplicationID
Thunderbird08162f7c-0fd2-4200-a84a-f25a4db0b584
Gmail app2cee05de-2b8f-45a2-8289-2a06ca32c4c8
iOS Accounts (Apple Mail app)f8d98a96-0999-43f5-8af3-69971c7bb423

Open your browser, and visit this URL as an administrator. You will be greeted with a consent form, in which you will be asked to accept for your organization. Because the redirect_uri is empty here, you will likely be send to a non-existing location after giving consent, but that’s OK.

When you look at the Enterprise Applications blade in the Azure Portal, you will notice the Thunderbird app has been added. Here you can further customize it, like any enterprise application supporting Modern Authentication, e.g.

  • Restrict access to specific users or groups.
  • Use Conditional Access to restrict access to certain locations.
clip_image001

Another thing to note is that permissions for Thunderbird app will have been translated to the following Graph permissions:

APIPermissionType
Microsoft GraphRead and write access to mailboxes via IMAP.Delegated
Microsoft GraphRead and write access to mailboxes via POP.Delegated
Microsoft GraphRead and write access to mailboxes via SMTP AUTH.Delegated
Microsoft GraphSign in and read user profile.Delegated

We should now be ready on the back-end.

Thunderbird
Now as an end user, start Thunderbird. Do not start configuring the account yet, as we first need to modify a Thunderbird setting to allow for successful Modern Authentication through a browser popup. Click the ‘hamburger’ menu to open the Options window. Scroll all the way down, and open the Config Editor. Click ‘I Accept the risk’. In the settings overview, set General.UserAgent.CompatMode.Firefox setting to True:

Preference NameStatusTypeValue
general.useragent.compatMode.firefoxmodifiedbooleanTrue

Close the Config Editor and Preferences tab. We can now set up our account in Thunderbird.

Select Add Mail Account, and enter your name and e-mail address. You can leave the password empty, as we will be using an Oauth token which we will retrieve later on. Press Continue to have Thunderbird figure out where your mailbox is hosted. When it properly discovers the mailbox location, it will set the configuration as follows:

image

If Thunderbird can’t figure out your settings (for some reason the Windows build could, but the Ubuntu build couldn’t), configure them as indicated above. We can’t select OAuth2 for authentication here, so leave Authentication as is; we will correct this right after we click Done.

Note: Configure manually would be the place you expect to set authentication to OAuth2 straight away, but with the build we used, the OAuth2 option is not available from the manual account setup dialog. Therefore, we need to set up the account and correct settings afterwards.

  1. In the Server Settings window related to your account, select OAuth2 authentication:
    clip_image001[14]
  2. In the Outgoing Server (SMTP) settings, select Offic365 (Microsoft) – smtp.office365.com, click Edit and set authentication for outbound SMTP to OAuth2 as well.
    clip_image002
    Note: The Thunderbird build running on Ubuntu doesn’t provide the OAuth2 authentication option for SMTP.

When finished, click ‘Get Messages’. The familiar Microsoft 365 authentication browser dialog should show up. After signing in, the next question will be to grant consent to the Thunderbird application to it can access your mailbox data and send e-mail:

clip_image001[18]

Note that this dialog can not be suppressed, as currently only interactive applications are supported. If you are working on an app or script which needs unattended access, please use Graph API.

After the user provides consent, Thunderbird is ready and will start fetching your default folders and mail items. If you want to view additional folders, you need to subscribe to them by right-clicking the account and picking Subscribe. Only folders with mail-items are supported, despite you can select every folder in your mailbox including Calendar or Contacts.

Logging
If you have people in your organization requiring some form of proof that Modern Authentication is being used, you can use the Enterprise Applications / Sign-Ins view from the Azure Active Directory portal.

Alternatively, you can use Thunderbird’s built-in logging capabilities. To accomplish the latter, set the following environment variables before starting Thunderbird:

MOZ_LOG=IMAP:5,timestamp
MOZ_LOG_FILE=%APPDATA%\ThunderBird-imap.log

In the generated ThunderBird-imap.log file like shown below, you should be able to spot Modern Authentication (XOAuth2) being selected:

2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: I/IMAP 259C3800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP ReadNextLine [stream=1991DE80 nb=28 needmore=0]
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: I/IMAP 259C3800:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP Try to log in
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP IMAP auth: server caps 0x840087635, pref 0x800000000, failed 0x0, avail caps 0x800000000
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP (GSSAPI = 0x1000000, CRAM = 0x20000, NTLM = 0x100000, MSN = 0x200000, PLAIN = 0x1000, LOGIN = 0x2, old-style IMAP login = 0x4, auth external IMAP login = 0x20000000, OAUTH2 = 0x800000000)
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP Trying auth method 0x800000000
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP IMAP: trying auth method 0x800000000
2020-06-30 13:10:16.726000 UTC - [(null) 15696: IMAP]: D/IMAP XOAUTH2 auth
2020-06-30 13:10:16.775000 UTC - [(null) 15696: IMAP]: D/IMAP ReadNextLine [stream=280D87C0 nb=180 needmore=0] 2020-06-30 13:10:16.775000 UTC - [(null) 15696: IMAP]: I/IMAP 2089A000:outlook.office365.com:NA:CreateNewLineFromSocket: * OK The Microsoft Exchange IMAP4 service is ready. [QQBNADQAUABSADAAMQAwADEAQwBBADAAMAA3ADYALgBlAHUAcgBwAHIAZAAwADEALgBwAHIAbwBkAC4AZQB4AGMAaABhAG4AZwBlAGwAYQBiAHMALgBjAG8AbQA=]

Comparing Sets of Cmdlets

powershellWith the speed of development in Office 365, it is sometimes hard to track which changes have been made to your tenant. Of course, there is the roadmap and message board which you can use to keep up to date, but those are in general high level descriptions. Sometimes you may want to see what are the changes at the cmdlet level in your tenant, between tenants, or Azure Active Directory module. And there is also the occasional gem in the form of a yet undocumented cmdlet or parameter which could hint at upcoming features.

For this purpose I have created a simple script which has two purposes:

  1. Export information on the current cmdlets available through Exchange Online or Azure Active Directory.
  2. Compare two sets of exported information, and display changes in a readable way.

The script is in PowerShell (of course), and is called Compare-Cmdlets.ps1. To export information, you need to be already connected to either Exchange Online or Azure Active Directory (or both).

To export cmdlet information, use:

.\Compare-Cmdlets.ps1 –Export

For Exchange Online and Azure Active Directory, separate export files are created. The files are prefixed with a timestamp and postfixed with the Exchange Online build or Azure Active Directory module version, e.g. 201803121814-ExchangeOnline-15.20.548.21.xml or 201803121815-AzureAD-2.0.0.137.xml.

After a few days/week, or when connected to another tenant or using a new Azure Active Directory PowerShell module, run the export again. You will now have 2 sets of Exchange Online or Azure Active Directory cmdlets, which you can compare using the following sample syntax:

Compare-Cmdlets.ps1 -ReferenceCmds .\201801222108-ExchangeOnline-15.20.428.21.xml -DifferenceCmds .\201803120926-ExchangeOnline-15.20.548.21.xml

image

A progress bar is shown as comparison might take a minute. When the script has finished checking the two sets, you will see output indicating changes in cmdlets, parameters or switches, e.g.

image

Download
You can find the script on the TechNet Gallery or GitHub.

Office 365 and “There are no items to show in this folder”

Be advised that when accessing shared mailboxes on Office 365 using Outlook in online mode, you may experience an issue with Outlook not properly updating the mailbox view.

Instead, Outlook will return a “There are no items to show in this view” message. The folder in the folder navigation pane displayed the proper number of (unread) items in the folder.

This could be the symptom of an issue which was already solved in Exchange 2010 Service Pack 1 Rollup 5. It seems the Office 365 data centers are not running a current version of Exchange, as today I received the message the Office 365 environment is currently being upgraded with Exchange 2010 Service Pack 2. The message also mentions the upgrade is to be completed at the end of the month.

More information on the issue in knowledge base articles kb2500648, announcing the fix is included in Exchange 2010 SP1 RU5.

Until then, the suggested workaround is to click one of the columns twice after which Outlook will update the view properly. Of course, you could also enable cached mode, if your setup and company policy permits (e.g. not running Outlook on terminal server).

Exchange Hybrid deployment and SMTP inspection

When setting up secure SMTP connections, also known as SMTPS or SMTP over TLS (Transport Layer Security), you encounter issues with SMTP obfuscating appliances, like Cisco ASA or PIX.

These appliances contain a feature called fixup protocol smtp, SMTP fixup, (E)SMTP inspect(ion) or Mailguard (Cisco), which – when enabled – limit the number of allowed SMTP verbs or obfuscate parts of SMTP dialogs. This tampering will cause problems when you try to configure SMTPS connections between e-mail servers and will prevent you from configuring a working Exchange Hybrid deployment, where TLS secured communications between the on-premise environment and Office 365 is enforced.

Note: By enforced, I mean it requires TLS which is more strict than opportunistic TLS,  which will attempt to set up TLS before continuing with regular SMTP communications.

You’ll end up with Office 365 specific Receive and Send connectors after setting up your Hybrid configuration using the Hybrid Configuration Wizard (or manually). Then, when sending e-mail between on-premise and Office 365, you notice e-mail doesn’t arrive and will remain queued.

After verifying your ISA or TMG isn’t blocking things (ports are open, SMTP filter not blocking STARTTLS etc), you start troubleshooting the issue by enabling Verbose logging for the Office 365 send connector (of course, this can also be achieved using the Exchange Management Console):

Set-SendConnector “Outbound to Office 365” –ProtocolLoggingLevel Verbose

The logs will by default be generated in the $exinstall \V14\TransportRoles\Logs\ProtocolLog\SmtpSend ($exinstall will contain the Exchange installation path).

If you look at the current log file, you’ll see the following pattern for each possible destination – in the example below the TX2EHSMHS017.bigfish.com – as the local Exchange 2010 hybrid server will try to get the message(s) delivered:

Outbound to Office 365,08CEBBEB07152FBC,0,,65.55.88.22:25,*,,attempting to connect 
Outbound to Office 365,08CEBBEB07152FBC,1,192.168.1.11:51841,65.55.88.22:25,+,, 
Outbound to Office 365,08CEBBEB07152FBC,2,192.168.1.11:51841,65.55.88.22:25,<,220 **********************************************************************************************, 
Outbound to Office 365,08CEBBEB07152FBC,3,192.168.1.11:51841,65.55.88.22:25,>,EHLO mail.contoso.com, 
Outbound to Office 365,08CEBBEB07152FBC,4,192.168.1.11:51841,65.55.88.22:25,<,250-TX2EHSMHS017.bigfish.com Hello [92.70.102.115], 
Outbound to Office 365,08CEBBEB07152FBC,5,192.168.1.11:51841,65.55.88.22:25,<,250-SIZE 157286400, 
Outbound to Office 365,08CEBBEB07152FBC,6,192.168.1.11:51841,65.55.88.22:25,<,250-PIPELINING, 
Outbound to Office 365,08CEBBEB07152FBC,7,192.168.1.11:51841,65.55.88.22:25,<,250-ENHANCEDSTATUSCODES, 
Outbound to Office 365,08CEBBEB07152FBC,8,192.168.1.11:51841,65.55.88.22:25,<,250-XXXXXXXA, 
Outbound to Office 365,08CEBBEB07152FBC,9,192.168.1.11:51841,65.55.88.22:25,<,250-AUTH, 
Outbound to Office 365,08CEBBEB07152FBC,10,192.168.1.11:51841,65.55.88.22:25,<,250-8BITMIME, 
Outbound to Office 365,08CEBBEB07152FBC,11,192.168.1.11:51841,65.55.88.22:25,<,250-BINARYMIME, 
Outbound to Office 365,08CEBBEB07152FBC,12,192.168.1.11:51841,65.55.88.22:25,<,250 XXXXXXXB, 
Outbound to Office 365,08CEBBEB07152FBC,13,192.168.1.11:51841,65.55.88.22:25,*,,Connector is configured to send mail only over TLS connections and remote doesn't support TLS 
Outbound to Office 365,08CEBBEB07152FBC,14,192.168.1.11:51841,65.55.88.22:25,>,QUIT,

As you can notice in this example, the SMTP banner has been turned into stars and AUTH and STARTTLS have been changed by the appliance into XXXXXXXA and XXXXXXXB. This behavior is typical of Cisco’s Mailguard feature; other smtp obfuscating appliances might result in different output. Key indicator is, expected elements are missing or garbled.

In such cases, you need to make sure SMTP traffic between on-premise and Office 365 goes through unfiltered. Depending on the capabilities of the device, you could allow the FOPE addresses to go through unfiltered. If that’s not an option, or you don’t like managing that address set, you need to disable the feature completely.

Note that Cisco is not the only vendor offering SMTP obfuscating products; companies like Checkpoint, Barracuda, Sonicwall or Symantec offer products with the feature mentioned above.

Federation Information could not be received ..

When setting up federation between Exchange organization using the Hybrid Configuration Wizard, to connect your on-premise Exchange environment with Office 365, you may encounter an error message in the 2nd step of the Hybrid Configuration Wizard, Update-HybridConfiguration:

image

When inspecting the Update-HybridConfiguration results, it reads:

Updating hybrid configuration failed with error ‘Subtask Configure execution failed: Creating Organization Relationships.
Execution of the Get-FederationInformation cmdlet had thrown an exception. This may indicate invalid parameters in your Hybrid Configuration settings.
Federation information could not be received from the external organization.
..

The problem lies in the sentence “Federation Information could not be received from external organization”. To see where it goes wrong, you could run Set-HybridConfiguration and Update-HybridConfiguration manually, using additional parameters as shown in the result screen, providing proper credentials and additionally addint the Verbose parameter to increase output logged.

Most of the time this problem comes down to one of the following issues, which can be verified by running the following cmdlet, where example.com is to be replaced by the federated domain:

Get-FederationInformation example.com –Verbose

You’ll probably see the cmdlet going trough the discovery motions using autodiscover, ending up in the same “Federation information could not be received from the external organization” message. When analyzing this output, you’ll see it contains two hints on the potential issues:

Get-FederationInformation : The discovery process returned the following results:
Type=Failure;Url=https://autodiscover.example.com/autodiscover/autodiscover.svc;Exception=Discovery for domain example.com failed.; …

1. Autodiscover
The first issue often overlooked is internal autodiscover not working via DNS or not being set up properly, i.e. split DNS configurations. Some customers don’t configure internal DNS autodiscover records or don’t allow (internal) autodiscover to go through the proxy from inside.

Normally, when using regular clients like Outlook, this isn’t an issue because domain joined clients will be using SCP records from AD. However, federation will use DNS records so you need allow it or set it up in DNS; a CNAME for autodiscover.example.com as well as autodiscover.service.example.com pointing to your hybrid server will suffice.

Also make sure you enable WSSecurity authentication for autodiscover on your hybrid server using:

Set-AutodiscoverVirtualDirectory -Identity ‘autodiscover (Default Web Site)’ –WSSecurityAuthentication $true

2. Proxy rules
In the error message you’ll notice a autodiscover-related request going to /autodiscover/autodiscover.svc. These requests are normally caught by the autodiscover rule (path /autodiscover/*) in ISA or TMG. Issue is, these service requests require unauthenticated traffic. When you turn on logging in TMG, you’ll notice the requests for /autodiscover/autodiscover.svc will be denied.

To solve this issue, and to get federation working, you need to set up an additional ISA/TMG rule pointing to the hybrid server, using the proper public names (don’t forget autodiscover), bound to the OWA listener. You need to allow All Users (as opposed to Authenticated Users), set authentication to “No Authentication, but users can authenticate directly” and configure the following paths:

  • /EWS/Exchange.asmx/wssecurity
  • /Autodiscover/Autodiscover.svc
  • /Autodiscover/Autodiscover.svc/wssecurity

After configuring the rule, you need to put it above all the other Exchange rules, making it the first matching rule when federation traffic hits ISA/TMG.

After applying the rule changes, Get-FederationInfo example.com should work and you can continue with the Hybrid Configuration.

For more information on setting up co-existence, consult the steps provided by the Exchange Deployment Assistent.

Exchange PST Capture Tool released

It took a while, but today the Exchange Team released the long awaited Microsoft Exchange PST Capture Tool (initial version 14.3.16.4). The tool can be used to discover and inject PST files in an Exchange 2010 Exchange Online mailbox or archive.

The tool was originally from Red Gate and known as PST Importer. It’s architecture consists of three components: the central service, (optional) agents for PST discovery, registration and collecting PST files and an administrative console (image by Red Gate):

The online documentation can be found here.

Note that although it’s only supported for Exchange 2010 and Exchange Online, you can use it with Exchange 2007; it’s only untested (and probably unsupported) with that product.

You can read the official announcement here; you can download the tool and the agents here.

Remote PowerShell to Office 365

imageWhile trying Office 365 you might want to connect your to a remote Exchange Management Shell session instead of using the portal interface. Here’s how to proceed.

Start up a PowerShell session. The first thing we’re going to do next is store credentials in a variable for later usage:

$cred= Get-Credential

A popup will be displayed where you can enter your Office 365 admin credentials, e.g. myadminname@yourdomain.onmicrosoft.com.

Next, create a new remote PowerShell session using the following cmdlet:

$o365= New-PsSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -Credential $cred -AllowRedirection -Authentication Basic

image

Next, we can import the session. However, this might be confusing since you have no context; are you creating a mailbox local or in the Office 365 environment?

The cool thing is that with Import-Session you can specify a prefix. This prefix can be specified before the cmdlet noun so that PowerShell knows which session you want the cmdlet to run against. As you probably know, cmdlets are normally constructed using <verb>-<noun> syntax, but this should be <verb>-<session prefix><noun>. When the session prefix is omitted, PowerShell assumes the current session.

For example, let’s import our Office 365 session with a prefix of “o365”:

Import-PsSession $o365 –Prefix o365

image

Now, we can use that “o365” prefix before the noun. For example, to get a list of our Office 365 mailboxes, you’d use something like:

Get-o365Mailbox

image

Cool and simple, eh?

Don’t forget to close your online session afterwards using:

Remove-PsSession $o365

Have fun exploring Office 365 using PowerShell.