When setting up secure SMTP connections, also known as SMTPS or SMTP over TLS (Transport Layer Security), you encounter issues with SMTP obfuscating appliances, like Cisco ASA or PIX.

These appliances contain a feature called fixup protocol smtp, SMTP fixup, (E)SMTP inspect(ion) or Mailguard (Cisco), which – when enabled – limit the number of allowed SMTP verbs or obfuscate parts of SMTP dialogs. This tampering will cause problems when you try to configure SMTPS connections between e-mail servers and will prevent you from configuring a working Exchange Hybrid deployment, where TLS secured communications between the on-premise environment and Office 365 is enforced.

Note: By enforced, I mean it requires TLS which is more strict than opportunistic TLS,  which will attempt to set up TLS before continuing with regular SMTP communications.

You’ll end up with Office 365 specific Receive and Send connectors after setting up your Hybrid configuration using the Hybrid Configuration Wizard (or manually). Then, when sending e-mail between on-premise and Office 365, you notice e-mail doesn’t arrive and will remain queued.

After verifying your ISA or TMG isn’t blocking things (ports are open, SMTP filter not blocking STARTTLS etc), you start troubleshooting the issue by enabling Verbose logging for the Office 365 send connector (of course, this can also be achieved using the Exchange Management Console):

Set-SendConnector “Outbound to Office 365” –ProtocolLoggingLevel Verbose

The logs will by default be generated in the $exinstall \V14\TransportRoles\Logs\ProtocolLog\SmtpSend ($exinstall will contain the Exchange installation path).

If you look at the current log file, you’ll see the following pattern for each possible destination – in the example below the TX2EHSMHS017.bigfish.com – as the local Exchange 2010 hybrid server will try to get the message(s) delivered:

Outbound to Office 365,08CEBBEB07152FBC,0,,65.55.88.22:25,*,,attempting to connect 
Outbound to Office 365,08CEBBEB07152FBC,1,192.168.1.11:51841,65.55.88.22:25,+,, 
Outbound to Office 365,08CEBBEB07152FBC,2,192.168.1.11:51841,65.55.88.22:25,<,220 **********************************************************************************************, 
Outbound to Office 365,08CEBBEB07152FBC,3,192.168.1.11:51841,65.55.88.22:25,>,EHLO mail.contoso.com, 
Outbound to Office 365,08CEBBEB07152FBC,4,192.168.1.11:51841,65.55.88.22:25,<,250-TX2EHSMHS017.bigfish.com Hello [92.70.102.115], 
Outbound to Office 365,08CEBBEB07152FBC,5,192.168.1.11:51841,65.55.88.22:25,<,250-SIZE 157286400, 
Outbound to Office 365,08CEBBEB07152FBC,6,192.168.1.11:51841,65.55.88.22:25,<,250-PIPELINING, 
Outbound to Office 365,08CEBBEB07152FBC,7,192.168.1.11:51841,65.55.88.22:25,<,250-ENHANCEDSTATUSCODES, 
Outbound to Office 365,08CEBBEB07152FBC,8,192.168.1.11:51841,65.55.88.22:25,<,250-XXXXXXXA, 
Outbound to Office 365,08CEBBEB07152FBC,9,192.168.1.11:51841,65.55.88.22:25,<,250-AUTH, 
Outbound to Office 365,08CEBBEB07152FBC,10,192.168.1.11:51841,65.55.88.22:25,<,250-8BITMIME, 
Outbound to Office 365,08CEBBEB07152FBC,11,192.168.1.11:51841,65.55.88.22:25,<,250-BINARYMIME, 
Outbound to Office 365,08CEBBEB07152FBC,12,192.168.1.11:51841,65.55.88.22:25,<,250 XXXXXXXB, 
Outbound to Office 365,08CEBBEB07152FBC,13,192.168.1.11:51841,65.55.88.22:25,*,,Connector is configured to send mail only over TLS connections and remote doesn't support TLS 
Outbound to Office 365,08CEBBEB07152FBC,14,192.168.1.11:51841,65.55.88.22:25,>,QUIT,

As you can notice in this example, the SMTP banner has been turned into stars and AUTH and STARTTLS have been changed by the appliance into XXXXXXXA and XXXXXXXB. This behavior is typical of Cisco’s Mailguard feature; other smtp obfuscating appliances might result in different output. Key indicator is, expected elements are missing or garbled.

In such cases, you need to make sure SMTP traffic between on-premise and Office 365 goes through unfiltered. Depending on the capabilities of the device, you could allow the FOPE addresses to go through unfiltered. If that’s not an option, or you don’t like managing that address set, you need to disable the feature completely.

Note that Cisco is not the only vendor offering SMTP obfuscating products; companies like Checkpoint, Barracuda, Sonicwall or Symantec offer products with the feature mentioned above.