A customer experienced a tricky SharePoint issue in his high-security environment.
This problem was not really related to SharePoint directly but rather to IIS, Active Directory and windows authentication in general.
But before giving out the explanation, some troubleshooting steps!
Since SharePoint did not give any clue except the usual “Unspecified error” so-called friendly message and because the issue only occurred in production, not in test (as usually!), I could not afford to disable friendly error messages the now well-known way:
1. On each WFE, backup the web.config of each web site implemented by SharePoint (except central admin and SSP of course) then edit it
2. Look for “CallStack” and replace the value False with True
3. Then look for “CustomErrors”. And set the mode to “off”
4. Save File and provoke the problem again
Instead, I could put my hands on the IIS logs, which were hopefully configured to W3C Extended with all fields enabled and start analyzing them (LogParser is your friend!).
Very strangely enough, the user could manage to be authenticated but then an internal server error occurred (HTTP 500). This error was linked to the Win32 status 0xc0000413 translated in hex, which was actually totally unknown to me. Platform SDK could shine a light on this by reporting this error as STATUS_AUTHENTICATION_FIREWALL_FAILED. Needless to say that there is no firewall running on the production boxes so what was this application firewall then?
I starting thinking about what could, at Windows-security level, impact cross-forest authentication but AFTER real authentication occurred and bingo, I found the culprit: The AD feature brought by Windows Server 2003 and named “Selective authentication”, which allows administrator to authorize or deny in a fine-grained manner security principals from other forests trusted by the forest where the server is running.
Solution was therefore simple: edit the computer account of each WFE in AD, go to the “Security” tab then grant “Allow to authenticate” to the “Domain Users” groups from the remote forest's domain.
Conclusions: even if you’re not allowed to authenticate, you get authenticated first and then everything fails with a dirty Internal Server Error if you use this security feature to protect IIS…
Talking about forests, I suggest you watch an older movie directed by John Boorman called the Emerald forest. Really worth it, for those, like me, who can stand Boorman’s style
Talking about “Selective authentication”, feel free to read Technet for additional information: http://technet.microsoft.com/en-us/library/cc738653.aspx
And cut!
PS: Thanks to Dirk T for reminding me the name of that feature!