Troubleshooting Intra-Org SMTP Traffic Issues (and disabling Cisco ASA ESMTP Inspection)

Original post is circa 2014, and migrated from Wordpress

I had a particular issue that is not well documented on the Technet site so I decided I would blog about it and share my experience.

Today I was doing an Exchange 2003 to 2010 upgrade for a customer. Their AD setup was quite typical, two Active Directory sites under a single forest interconnected by a Cisco ASA in each site doing IPSEC. They wanted an Active / Active configuration (one site will be doing SMTP and CAS traffic, which is NY and the FL site will be doing just CAS traffic) which was fine. Here comes the problems.

We tested mail flow from the NY site both external (inbound/outbound) to a user in the NY site along with the Exchange 2003 setup. All worked without a problem. When we tried to send an internal or external message to a user within the FL site, it would get stuck in the messaging queue with the status of “Retry”.

The first thing when troubleshooting intra org SMTP traffic is to review the Receive connectors. By default, you would have two…

• Client <servername>
• Default <servername>

The default receive connector is used for SMTP traffic, and is listening over TCP 25. The client receive connector is used for ESMTP, as it is listening over TCP 587.

You should look at several things:

1) What are the subnets allowed within the Network’s tab on my Default <servername> connector? By default, you should be allowing all subnets to utilize this connector to send message to this server.

2) What is my authentication set to for this receive connector? I have seen this one a lot, and where people think it’s an awesome idea to remove all types of authentication. Its not, don’t do it. By default is what you see below, and should be kept this way. If you need to change this, I would recommend making a new SMTP connector with the values desired

3) Who is actually allowed to use this connector? The values for the defaults are below (Exchange 2010 SP3) but should be kept alone. If you want to select “Anonymous” I would recommend setting up a new Receive connector and setting up the network section to allow only the servers that would need to relay off of this server. This prevents your Exchange 2010 servers from being a generic SMTP relay out in the interwebs for anyone to use and abuse.

Since we are troubleshooting Exchange 2010 intra-org SMTP connectivity, we want to make sure that we keep the Exchange servers checked within the Authentication and Permissions group tabs. If this is unchecked, check it off and restart the “Microsoft Exchange Transport Service” (anything changed in here should have a restart of this service).

If this is fine, we should move onto actually checking if port 25 is available between the two sites. You can check this by running the following from the command line..

telnet <Name of the HUB server in the other site) 25

The response should look similar to below

If we are able to telnet to this without a problem, then check out the other ports listed here. You should have these also open and available to for intra org SMTP communications and also for the HUB Transport services to work properly.

The best troubleshooting in my opinion would be to turn up the IntraOrgProtocolLoggingLevel to Verbose. This should be done on all HUB transport servers when troubleshooting, and can be done like this:

Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel Verbose

Once this is done, attempt to send another test message. Give it about 5 minutes and then check the logs from the sending HUB transport server. These logs can be located (by default) at %PROGRAMFILES%\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLogs\SmtpSend

From there review the errors. There are usually typically a handful that can be there, but the one distinct one I saw today was “cannot achieve Exchange server authentication — failed”.

When reviewing further I noticed the following error within the queue viewer:

“451 5.7.3 — Cannot achieve Exchange Server authentication”

The Cisco ASA does protocol level filtering, the 250 STARTTLS reply is not established and dropped. To get IntraOrg SMTP Traffic to work, we would need to turn off the ESMTP inspection which can be done as shown below (thank you Lazy Network Admin for these steps below)

CiscoASA# config t
CiscoASA(config)# policy-map global_policy
CiscoASA(config-pmap)# class inspection_default
CiscoASA(config-pmap-c)# no inspect esmtp
CiscoASA(config-pmap-c)# exit
CiscoASA(config-pmap)# exit
CiscoASA(config)# exit
CiscoASA# wr me

Once this is done retest, and you should be receiving emails from the other site. If you did turn on the intra organization protocol logging, I would recommend turning it off.

Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel None

Now, these are not the only troubleshooting steps in the world. If you continue to have issues with intra-organization mail delivery I would also check within the Exchange Toolbox the routing configuration, and run through the Mail Flow Troubleshooting wizard.

Hope this is helpful to all! Any questions please let me know below.

Adam F