Recently, I’ve been troubleshooting multiple WSS/MOSS implementations experiencing problems with the people picker.

I must confess that only documentation is poor and other blog posts are sometimes unclear to me. I decided to make a digest but comprehensive attempt to document the people picker behavior and configuration in and AD/Integrated Windows Authentication (IWA) configurations. Note: what applies to IWA also applies to Basic, Digest and client certificate mapping authentication: what matters is AD to be the repository for authentication information.

It is therefore critical to correctly understand how AD, domains, forests and trusts are working in order to get your people-picker working correctly (displays what you effectively want to see), rapidly (does not take ages to return a result) and reliably (its behavior is predictable and persistent over time). I am often surprise by the fact that very few SharePoint specialists really know about the windows and AD  internals, this often lead to improper people picker configuration.

How it works, with the default configuration

The people-picker is a SharePoint interface responsible for querying repositories for identities or groups in order to grant them permission in the SharePoint application. It is implemented as part of the WFE role, this means that when you’re using it, the WFE you’re connected to will attempt to contact AD in order to returns items matching your query’s criteria. Here is, step by step, how it exactly works from an AD/Windows point-of-view:

1. A users submit a query to the people-picker
2. The WFE performs a DNS query in order to locate a domain controller hosting the Global Catalog Service. There are actually two possible DNS queries:
3. The first query will include the server’s Active Directory site name, in order to locate a domain controller that reside on the same site or “covers” it (does not reside in the same site but is configured as a candidate to receive request from originating from that site).If that first query succeeds, there’ll be no second.
4. It it fails and the DNS reports that there is no such name, a second query will take place without any reference to the server’s site.
5. with an IP address of a DC in hand, SharePoint will initiate a connection from a local random port to the remote port 3368 (Global Catalog LDAP over TCP) against the select domain controller. This first connection, which is is anonymous, will report to the SharePoint server extra information over the DC it contacted. It will include various LDAP information, the its exact capabilities as well as the authentication mechanism it supports. This entry point is known as the “Root” or “RootDSE”
6. Once SharePoint know how to “talk” to AD, it will perform a query whose part of the parameters are based on the users' input. This query is actually programmatically powered by the System.DirectoryService Namespace. since that query is made against the Global Catalog Service, it can only use a subset of the AD attribute (known as the partial attribute set). the exact list of attributes depends on the Windows version of the DC and on the presence of schema extension (MS Exchange, OCS or custom…). If the AD requires authentication, which is by default the case, SharePoint will authenticate using the context of the IIS application pool the SharePoint web application runs under. It can be Local system or Network Service (then DOMAIN\SERVERNAME$ is used) or a specified user
7. SharePoint displays the results to the user

The query, as stored in the code:

• (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(|(name={0}*)(displayName={0}*)(cn={0}*)(mail={0}*)(sn={0}*)(SamAccountName={1}*)(proxyAddresses=SMTP:{0})(proxyAddresses=sip:{0}){2}))", "(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(|(name={0})(displayName={0})(cn={0})(mail={0})(samAccountName={0})(proxyAddresses=SMTP:{0})(proxyAddresses=sip:{0})))", "(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(|(mail={0})(proxyAddresses=SMTP:{0})))"), new SearchParameter("(&(objectCategory=group)(|(name={0}*)(displayname={0}*)(cn={0}*)(SamAccountName={1}*)(mail={0}*)(proxyAddresses=SMTP:{0}){2}))", "(&(objectCategory=group)(|(name={0})(displayname={0})(SamAccountName={0})(mail={0})(proxyAddresses=SMTP:{0})))", "(&(objectCategory=group)(|(mail={0})(cn={0})(proxyAddresses=SMTP:{0})))"), new SearchParameter("(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(|(name={0}*)(displayname={0}*)(cn={0}*)(SamAccountName={1}*){2}))", "(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(|(name={0})(displayName={0})(cn={0})(samAccountName={0})))

The query criteria in clear:

• A user or a group
• If it is user, at least of of the following attribute must begin with the input the user provided: name, displayName, cn, mail, sn, SamAccountName, proxyAddresses (with SMTP or sip)
• It it is a user it must not be disabled
• If it is a group, at least of of the following attribute must begin with the input the user provided: name, displayname, cn, SamAccountName
• if it is a group, it must be a security group (domain local, global or universal), not a distribution group

The following attributes are requested to be returned for each record found:

• objectSID
• mail
• displayName
• title
• department
• proxyAddresses
• cn
• samAccountName
• groupType
• userAccountControl
• distinguishedName

And finally, the following server controls are specified:

• LDAP_PAGED_RESULT_OID_STRING: Page the results and get 20 results per page
• LDAP_SERVER_DOMAIN_SCOPE_OID: Instructs the DC not to generate LDAP continuation references in response to a search operation.

Key points:

• By default, SharePoint only knows about the AD forest its server(s) belong(s) to
• SharePoint uses DC locator DNS records to locate a DC hosting the Global Catalog Service
• It issues a queries using the LDAP-like dialect against that Global Catalog Service using System.DirectoryService Namespace
• There is no “security trimming” per se. The queries returns the results based on what the IIS application pool identity is allowed to see, not the end-user’s identity

Still unclear to me:

• In the case (non default and not recommended), in the case the IIS Application pool runs under the context of a user from another domain, which domain controller of which domain will be used?

Querying Additional Forest or Domains

In this section, I will cover the most frequent configuration: the SharePoint server belongs to a forest and/or domain that has 2-way trusts established with other forests or domain. The accounts of the users accessible SharePoint therefore belong to those trusted domains and since the trust is in both direction, the identity of the IIS application pool is also capable of authenticating against those trusted domains.

In order to instruct SharePoint to query those trusted domain or forests, the command “STSADM.exe -o setproperty -pn peoplepicker –searchadforests” must be used. But it seems like many people are getting confused, for good reasons, with the exact syntax of the parameter to be passed.

first you keep to gather the following information:

• Do you want to query a forest (in this case, you’ll need a forest-trust) or a domain (in this case, an external trust is sufficient)
• What is the DNS name of the forest you wish to query (also the DNS name of its root domain) or what is the DNS name of the

Then, based on the information above, you can assemble the parameter correctly. The configuration of each forest or domain to be queried must be separated with a semi-colon and inside the configuration, the first word must be forest: or domain: and it must be followed by a valid DNS name. Example:

• STSADM.exe -o setproperty -pn peoplepicker–searchadforests –pv “forest:dune.local;domain:carthag.local;domain:tuono.local”

Key points:

• The DC Locator process used with the standard configuration is still applicable BUT is extended beyond the local forest boundaries. This means that the SharePoint servers must be able to resolve names of remote forests/domains domain controllers (SRV records and A records)
• Also, Active Directory Sites should be identically named between forests, otherwise, SharePoint may not target the “closest” domain controller
• If you use “Selective authentication” or “SID Filtering” in order to restrict authentication through trusts, you must make sure that the IIS application pool identity is allowed to authenticate against the remote forest/domain it queries
• Needless to say that if there are firewall between SharePoint and the remote forest/domains, they must be configured adequately
• As I said above,if the “forest” argument is specified, a forest-trust must be in place
• If you still wish to query the forest/domain SharePoint belongs to, you’ll have to add it as part of the parameter too
• If you configure a forest to be queried, it is not necessary to declare all or some child domains separately, they will be queried anyway
• If “forest” is specified, the Global Catalog Service will be used to perform the query, if “Domain” is specified, the LDAP service will be used instead. Global Catalog (GC in short) can query all objects inside a given forest BUT knows only, as stated above, about a limited set of attributes while LDAP knows about ALL attributes but its boundaries is the domain it is targets
• Do not forget to perform IISRESET on each SharePoint server where the configuration must be applied

Still unclear to me:

• I have a great level of certainty that the query against each forest/domain are not performed in parallel, at least not the DNS part

The One-way trust Configuration

This configuration is identical to the above except that since the IIS application pool identity is unable to authenticate against the remote forest/domain due to the limitation set by one-way trust, alternate credentials must be specified. Those credentials must be from the forest/domain to be queried or from a trusted domain, as long as it is allowed to authenticate and is not denied to logon remotely.

The tough part of the job in this case is to apply the configuration.

first, a key must be generated in order to encrypt/decrypt the alternate credentials that will be used, in order to do so, you have to run the command hereunder on every SharePoint server where the people picker will be used (shortcut: do it on all servers!):

• STSADM.exe -o setapppassword -password MYPASSWORD. where MYPASSWORD is the key –> it makes sense that the more complex, the better but is is not ruled by the Windows password policy

Secondly, you have to provide the list of forest/domains to be queried as well as the credentials to do so. Each block is separated by semi-colons and each element in a block is separated by comma. Example:

• STSADM.exe -o setproperty -pn peoplepicker–searchadforests –pv “forest:dune.local,DUNE\PAULA,PasswordOfPaulA;domain:carthag.local,CARTHAG\Gurney,PasswordOfGurney;domain:tuono.local,TUONO\SHANIA,PasswordOfShania”

Key points:

• Make sure you have valid credentials for each forest/domain
• Make sure each forest/domain element is correctly structured
• Do not forget to perform IISRESET on each SharePoint server where the configuration must be applied

Troubleshooting Tips

Useful tools

• NTLTEST command-line: (part of the windows Server 2008 Support tools or built-in Windows Server 2008)
• Wireshark network capture utility
• LDP.exe simple LDAP client (part of the windows Server 2008 Support tools or built-in Windows Server 2008)
• Active Directory User and computer (ADUC) Console
• ADInsight from MS Sysinsternals (not super reliable alas)

Global Approach

The most straightforward way to trace the behavior of the people picker is to take a network capture while performing the search from the Web GUI, taking care of flushing the DNS resolver cache (ipconfig /flushdns). Take this capture of the WFE your target with your tests and filter the results as follows:

Apply a display filter to show only DNS requests, you should see requests like the following:

If you attempt to query a domain: _ldap._tcp.Site-Name._sites.domain.local: type SRV, class IN (first attempts, in order to locate a DC in the same site) or _ldap._tcp.domain.local: type SRV, class IN (any DC in that domain, regardless of the site)

If you specified to query a forest, you should see _gc instead of _ldap

If you don’t see those DNS request or if you see them but they fail, make sure the SharePoint servers are able to resolve names from remote domains ad are configured with the correct DNS Servers and optionally with a list of suffixes

Once name resolution is working fine, go back to the capture and make sure you see LDAP (port 389) or GC (same as LDAP but on port 3268).

for each domain or forest, you should see a “bindRequest” with a successful response followed by a “seachRequest” followed by a successful response as well. drill-down into the search request for the details about the query submitted: the filter and conditions applied in particular

Retrieving the server’s AD site

Execute the command “NLTEST /dsgetsite”. It should return the AD site the SharePoint server belongs to. If it does not, there is a serious AD configuration problem ;)

Retrieving a DC for a given domain

Execute the command “NLTEST /dsgetdc:mydomain.local

If the list of flags it return includes the following, you’re ok:

• CLOSE_SITE= the DC is in the same AD site as the SharePoint server or is “covering” that site
• LDAP: the DC is LDAP server (all DCs are but must advertise it)
• GC: the DC is also global catalog (NOT all DCs are but if they are, they must advertise it too)

Note: Other information returned by the command might also be useful for troubleshooting: Name and IP address of the DC…

Simple LDAP connection test

1. On the SharePoint server, start ldp.exe

2. Go to the menu “Connection” and click “Connect”. Enter the IP or the host name of the remote DC. You should test with a FQ host name in order to test DNS too. Select port 389 for LDAP or 3268 for GC. If it works, it will return a list of server-related information

3. repeat this test for each DC SharePoint would potentially target

Testing the credentials to connect to a remote domain using LDAP (one-way trust scenario in particular)

If the test above works, proceed to this one:

1. Return to the menu “connection” then click “Bind” then enter the credential of the remote domain to be browsed (including the domain name in the 3^rd textbox

2. If it fails, you’ll see a message such as

res = ldap_bind_s(ld, NULL, &NtAuthIdentity, 1158); // v.3

               {NtAuthIdentity: User=’myuser’; Pwd= <unavailable>; domain = ‘mydomain’.}

Error <49>: ldap_bind_s() failed: Invalid Credentials.

3. If it succeeds, it will report

res = ldap_bind_s(ld, NULL, &NtAuthIdentity, 1158); // v.3

               {NtAuthIdentity: User=’myuser’; Pwd= <unavailable>; domain = ‘mydomain’.}

Authenticated as dn:’myuser’.

Top People-Picker Reliability/Performance Killers

• The more forest/domains there are two query, the slower it will be to get results
• Problematic Name Resolution Process: In order to resolve DC locator records, Windows will exhaust all possible name resolution methods, from DNS to broadcast… And this for each declared forest/domain
• Unresponsive DC brings major slow down
• Suboptimal/Inconsistence AD Site configuration: Site-awareness is a key factor, this makes sure SharePoint always query the nearest DC
• Network devices/Security devices breaking the TCP traffic: from broken NIC to firewall, anything generating TCP retransmit or “forcibly closing” connections
• Load on Active Directory/Domain controller or security settings on the domain controller (preventing DoS attacks for example)
• If custom filter is used: Improperly written filter: make sure the complexity of criteria remains reasonable, only indexed attributed are queried and of course if the GC is used, only attributed that are part of the partial attribute set

Additional information’s

• MS TechNet LDAP Query Basics
• MS TechNet Select users from multiple forest domains
• MS TechNet Peoplepicker-searchadforests: Stsadm property (Office SharePoint Server)
• MS TechNet Active Directory Trust Types
• MS KB How to query Active Directory by using a bitwise filter
• Joel Oleson Multi Forest/Cross Forest People Picker peoplepicker-searchadcustomquery
• Bill Baer Configuring SharePoint Products and Technologies for Cross-Forest Deployments

What’s next?

In the next posts, I will cover:

• Additional filtering capabilities of LDAP searches
• Detailed configuration for each scenario
• how people picker is related to authentication and profile import (MOSS only)
• Guidance to optimize people-picker in different scenario’s

And cut!