Exchange Server: Address Book Service, EMC SourceOne and E_MAPI_FAILURES

So I was recently tasked with a pretty interesting case escalation this week, which was in regards to an address book sync failure from a third party application (EMC SourceOne).

Environment

For the sake of this blog, the environment was pretty simple. A single Exchange 2010 SP3UR5 CAS / HUB combination with an Exchange 2010 SP3UR5 Mailbox role. No HA on the DAG and no CAS Array. The EMC SourceOne server is in the same AD site. Multiple global catalog servers and domain controllers utilizing Kerberos.

The Issue

This was a fun one, mainly due to the lack of information provided. What I understood is that the EMC SourceOne application would utilize a local MAPI profile (Control Panel > Mail (32)) to connect to the Exchange role over MAPI / TCP, then start the Address Book Sync through the CAS to the Address Book Service. This is similar to the way RIM BlackBerry Enterprise Server does it (well.. I can speak for BES 4 / 5), and also how both applications do further lookups.

When the EMC SourceOne server would reach out to the CAS’s Address Book Service it would fail out with the error E_MAPI_FAILURE. Not very helpful, since that would be a few hundred things.. lets dive into troubleshooting.

Troubleshooting

The first thing that ran into my head was “is this being throttled”? If you don’t know much about client access throttling, I would recommend reading up on it. It does change a bit in Exchange 2013, but I can write a blog about this later if there is interest. The best way to check throttling is to go directly to the Exchange server, open the Event Viewer: Application logs and filter them for “ADACCESS Event ID 2915”, which is client throttling. You can look at the SID~ within the log to see which account is being throttled..

The Event ID looks something like this:

ID: 2915
 Level: Error
 Source: MSExchange ADAccess
 Message: Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=5372). User 'Sid~DOMAIN\SERVICEACCOUNT~RCA~false' has gone over budget '263' times for component 'RCA' within a one minute period. Info: 'Policy:[Fallback], Parts:MaxConcurrency:262;'. Threshold value: '100'.

Ironically, NOBODY went other throttling as per the Event Viewer: Application logs this month. Cool, lets move on.

If it’s not client throttling, then what could it be.. it could be NTLM token bloat if NTLM was actually enabled, but they are using Kerberos so that’s not the case. The mailbox does exist for the service account, and the permissions are setup properly. The next logical step is to take a deeper look at the address book service logs (which are located by default at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Address Book Service\)

Looking at the logs using this wonderful tool (TextAnalysisTool.net — really nifty tool everyone should use if you want an easy to use log searching tool) and filtering for the service account (SourceOne-Service) I found the following..

10666 2014-08-20T17:18:04.826Z,165558,1221,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,DNToEph,80004005,0,,Throttled,,15000
10689 2014-08-20T17:18:20.201Z,165558,1222,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,ResolveNames,80004005,0,,Throttled,,15000
10743 2014-08-20T17:18:35.108Z,165558,1223,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,Unbind,,0,60395,,,14906

So what you see above is within the Address Book Service logs that during the address book sync the service account for SourceOne was throttled during an NSPI call to the GAL. Afterwards, that connection was unbound since it went over its budget and sub-sequentially dropped, which would have caused the E_MAPI_FAILURE error mid-sync.

Now that we actually have the error, and know the issue how do we fix this? (thank you for asking kind voice, let me explain that part)

The Fix

This one had me scratching my head at first until I looked at my notes. Originally in Exchange 2010 RTM the Address Book Service would throttle the amount of connections concurrent connections to 50 maximum, and any new connection (when it hit its maximum of 50) would be dropped. This setting use to exist within the exchange-addressbook.service.exe.config file, but was move in Exchange 2010 SP1+ to the registry along to the Client Access throttling policies (RPC Client Access, also known as RCA). What confused me originally is that the CAS does NOT show any ADACCESSS Event ID 2915 errors at all for this account, but I will digress.

To fix this, you should actually create a new throttling policy and either remove the RCA limits or up them to the limit you know you are hitting (I would speak to the vendor to figure out the desired limit, they are the best resource).

1. Create a new throttling policy

The creation of the throttling policy is pretty simple. All you have to do is open Exchange Management Shell (as an account with Org Admin or Org Management rights) and run the following:

New-ThrottlingPolicy "EMC SourceOne"

2. Once the throttling policy is created, you would need to set the RCA* values within it to something higher than the defaults or simply remove them.

In my scenario, I know the EMC SourceOne application is going to only run the sync over a few minutes in the AM, so I chose to remove the RCA* values completely (e.g: setting them to $NULL)

Set-ThrottlingPolicy "EMC SourceOne" -RCAPercentTimeInAD $null -RCAMaxConcurrency $null -RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null

You then can review the policy to make sure the RCA* values within the throttling policy are shown as blank (meaning there is nothing actually set)

Get-ThrottlingPolicy -Identity "EMC SourceOne" | Select RCA*

3. Apply the throttling policy to the service account

Set-Mailbox -Identity SourceOne-Service@company.com -ThrottlingPolicy "EMC SourceOne"

You can then view the mailbox settings itself to make sure the throttling policy change from the default (or whatever you had previously) to the EMC SourceOne policy..

Get-Mailbox -Identity SourceOne-Service@company.com | Select ThrottlingPolicy

4. Test and review the ABS (Address Book Service) logs. It should not work.

TL;DR: for some reason the address book service throttling does not show up within the actual Event Viewer: Application log under Event ID 2915 when the service account goes over budget. To fix this create a new throttling policy and then up (or remove) the limit for the RPC Client Access (RCA*) attributes. I do not recommend completely removing the limit unless you are sure it will NOT affect production in a negative way, so contact the vendor for there best practices (in this case, EMC Source One would be EMC).

Any other questions, comments or anything like that drop them below in the comments.

Cheers!

- Adam F