Troubleshooting an authentication issue in StoreFront and Netscaler Gateway

I love writing these kinds of posts – real world examples of troubleshoot live environments, and sharing the methodology I used to find the root cause. As with all of my posts, click on the screenshots to enlarge.

[updated on 11/10/13 to add an additional caveat when using the GC for multi-domain authentication]

I was asked to investigate a “Cannot complete your request” error which was preventing users from logging into their StoreFront site externally via a Netscaler Gateway.

 

As usual, the error message wasn’t particularly helpful or descriptive, so we’re going to have to do some elimination. This is how I found and fixed the error.

Direct Storefront login

The first step was to log into the Storefront site directly (bypassing Netscaler Gateway) to see if it was an issue affecting the entire server.

Direct Login to StoreFront was working OK.

Check StoreFront configuration

Let’s check that someone hasn’t messed with the StoreFront configuration.

 

Pass-through from Netscaler Gateway is present, and enabled. All looks good so far.

Check StoreFront event logs

Whilst on the Storefront server, looking at the event log gave the first clue as to why logins via Netscaler weren’t working. Storefront (finally) get’s their own event logs so you don’t have to trawl through the application or system log to find these now (thank you Citrix!). They are under “Event Viewer -> Windows Logs -> Applications and Services -> Citrix Delivery Services”

Here is the error generated when someone logs in via Netscaler Gateway:

The error “FailedMissingDomain”, and the username of the format “SAMAccountName” rather than “DOMAIN\SAMAccountName” indicated that the users domain name wasn’t being passed to StoreFront, which of course could not then authenticate the user to enumerate the applications.

Looks like it’s time to move over to the Netscaler to figure out why our domain name isn’t being sent to Storefront.

How Netscaler Gateway handles AD domains

There are two ways that a users domain can be specified when logging into Netscaler Gateway.

1. In the Session Profile

In the Published applications tab of the session profile, there is a “Single Sign-on Domain” field. If your users are only authenticating to a single domain, you can specify it here, and Netscaler Gateway will prefix this to the username when passing the credentials to StoreFront.

2. As part of the username (e.g. UPN format)

Active Directory has supported UPN-based logins since the Windows 2000 days where you login using username@domain.com

However, most companies don’t use this as it’s quite a bit more typing for users, and if your AD forest is different to your external email domain, which is often the case, users get confused between their AD UPN and email address resulting in account lock-outs and increased helpdesk calls.

How Netscaler Gateway supports multiple AD domains

In this instance, the client wanted to login to multiple AD domains (in the same AD forest) but not force the user to enter either their domain name or UPN – they just wanted users to enter their NT-style userid (SAMAccountName in AD).

There are two ways of supporting multiple domains on a Netscaler Gateway.

1. Create multiple LDAP Authentication profiles

You create an LDAP authentication server and profile for each NT domain that users need to authenticate against. These are all bound (in order of most popular first) to the Access Gateway vServer, so when a user logs in, an LDAP authentication is fired off to each one of them in turn. It only needs one to return positive (i.e. from the correct domain) to allow the user to login.

You could use this method for when your domains aren’t in the same forest, or users have identical accounts in multiple domains.

2. Authenticate against a Global Catalog

If your multiple AD domains are part of the same forest, you can authenticate against the AD Global Catalog server using SAMAccountName, and use the LDAP SSO Name attribute to retrieve the users UPN (stored in the UserPrincipalName attribute), which you can then pass to StoreFront so it knows which domain the user is from.

To set an LDAP authentication server to point at your global catalog, set the LDAP port to 3268, and check with your AD team that they’ve not adjusted any of the default attribute set that a GC server hosts.

Set the Base DN to a high-enough level that it covers all of the child domains present on that Global Catalog.

There are a couple of  important caveats to be aware of when using this method

1. Duplicate accounts

If you have users who have identical accounts in multiple domains in the forest (which is allowed), then querying the GC with your sAMAccountName will authenticate against the first domain returned by LDAP as it can’t distinguish which domain account you want. This may not actually be the account the user intends.

To work around this, users will either have to login with their UPN (which includes the domain) or stick to method #1 above. Either way may involve creating multiple LDAP services and binding them all against the Access Gateway vServer – remember only one has to return TRUE for a user to be authenticated.

2. Password changes

If configured, the Netscaler supports changing an expired password. Because the Global Catalog is a read only directory containing a partial attribute set you can’t change a users password using it. I note that Netscaler 10.1 supports LDAP referrals, but I’ve not yet tested whether a request to a GC to change a password would result in a referral issued pointing tothe regular LDAP service on a DC where this would be possible (I suspect not – asking far too much of a simple GC!)

Telling StoreFront the domain

So we’ve authenticated the user (on the Netscaler) to one or more possible AD domains. But now we need to let StoreFront know which domain the user is in.

For both of the above methods ensure you set the “SSO Name Attribute” in your session profile to UserPrincipalName so that the UPN is extracted via LDAP and passed to StoreFront.

If you do use this method, ensure that you don’t set a Single Sign-on domain in the session profile, and that you have set the correct CredentialIndex field in your session profile:

In this instance, I’m only using single-factor authentication, so the credential index was PRIMARY.

If you were using two-factor, and had RADIUS bound as the primary authentication policy in the vServer, you would need to change this to SECONDARY to ensure that the UPN attribute is extracted from the correct (LDAP) authentication profile. See my other post for a clever way to allow a different username to be used for each authentication factor.

OK, but why can’t users log in?

So, we’ve got this far, and everything is configured correctly. Our Global Catalog server is up and reachable from the Netscaler. We need to dive into the AAA logfile, which is accessed by opening a SSH session to the Netscaler, dropping out to the shell (just type shell) then type:

cat /tmp/aaad.debug

Then perform a test login to Netscaler Gateway, whilst watching the output in your SSH session. On our problematic Netscaler, this was the output:

The Netscaler is authenticating us OK, but is failing to retrieve the UserPrincipalName attribute, so of course can’t pass it as the username to StoreFront, resulting in the “MissingDomain” error we observed earlier.

Lets look at the LDAP server configuration again. Look very closely at the SSO Name Attribute field. Do you spot what’s causing the attribute look-up failure?

BINGO! There is a typo in the SSO Name Attribute field, where the UserPrincipalName attribute has become UserPrincipleName

Correcting this typo fixed the issue, as the attribute was then read correctly from LDAP: