A customer experienced a tricky SharePoint issue in his high-security environment.
This problem was not really related to SharePoint directly but rather to IIS, Active Directory and windows authentication in general.

But before giving out the explanation, some troubleshooting steps!
Since SharePoint did not give any clue except the usual “Unspecified error” so-called friendly message and because the issue only occurred in production, not in test (as usually!), I could not afford to disable friendly error messages the now well-known way:
1. On each WFE, backup the web.config of each web site implemented by SharePoint (except central admin and SSP of course) then edit it
2. Look for “CallStack” and replace the value False with True
3. Then look for “CustomErrors”. And set the mode to “off”
4. Save File and provoke the problem again

Instead, I could put my hands on the IIS logs, which were hopefully configured to W3C Extended with all fields enabled and start analyzing them (LogParser is your friend!).
Very strangely enough, the user could manage to be authenticated but then an internal server error occurred (HTTP 500). This error was linked to the Win32 status 0xc0000413 translated in hex, which was actually totally unknown to me. Platform SDK could shine a light on this by reporting this error as STATUS_AUTHENTICATION_FIREWALL_FAILED. Needless to say that there is no firewall running on the production boxes so what was this application firewall then?
I starting thinking about what could, at Windows-security level, impact cross-forest authentication but AFTER real authentication occurred and bingo, I found the culprit: The AD feature brought by Windows Server 2003 and named “Selective authentication”, which allows administrator to authorize or deny in a fine-grained manner security principals from other forests trusted by the forest where the server is running.
Solution was therefore simple: edit the computer account of each WFE in AD, go to the “Security” tab then grant “Allow to authenticate” to the “Domain Users” groups from the remote forest's domain.

Conclusions: even if you’re not allowed to authenticate, you get authenticated first and then everything fails with a dirty Internal Server Error if you use this security feature to protect IIS…

Talking about forests, I suggest you watch an older movie directed by John Boorman called the Emerald forest. Really worth it, for those, like me, who can stand Boorman’s style

Talking about “Selective authentication”, feel free to read Technet for additional information: http://technet.microsoft.com/en-us/library/cc738653.aspx

And cut!

PS: Thanks to Dirk T for reminding me the name of that feature!

Since Windows 2000 reached the shelves there has been a lot said over the fact that applications is general and web applications in particular should be « Kerberized » (made Kerberos aware).
Unfortunately, making such a step is not that easy may introduce unexpected result that are not always well known by IT pro’s.

In a nutshell, Kerberos has the following pro’s ad con’s:

Pro’s:

• “Said to be more secure”. Usually true, because the way to encrypt information is better, because security-sensitive information less often travel over the wire and because it includes additional protection measures such as pre-authentication. Why “said to by”? Because it was demonstrated already long ago that under some circumstances, all those security measures can be cracked relatively easily using the appropriate tools and enough processing power. See http://www.securiteam.com/windowsntfocus/5BP0H0A6KM.html for more information.
• “Said to increase interoperability”. Because Kerberos is a standard, it is interoperable between platforms. And it works! Why “said to” then? Because Microsoft introduced some customization that are not part of the standard. Therefore they are not usable by other platforms unless those extensions are ported adequately (never seen).. See http://msdn.microsoft.com/en-us/library/aa302203.aspx for more information
• “Said to decrease the load on domain controllers and improve performances”. Well, it actually depends. If the client systems is a Windows system member of the same domain as the IIS and the domain controller or I there are Kerberos trusts between domains each party belong to, then it will work fine, if not, IIS will contact a DC like it does for NTLM or Basic, for example.
• Allows a server to pass the security context of a user to another system. This is known as “delegation”. This requires extra configuration, which may be complex to setup. But in some scenario’s, it’s really worth it because it opens doors to rich experience in secure distributed computing!

Con’s

• Depending on the environment, the Kerberization may involve a (very very) complex configuration: on servers, in AD and on clients as well in some cases
• Using a load balancer will involve this complex configuration anyway!
  Using Kerberos between a client and IIS while both are member of the same AD domain or members of domains trusted using a Kerberos-based trust, the size of an HTTP request will increased depending on the group membership of the user and optionally, the size of SID History.
• Since Kerberos is based on names (host names), it will never work if a web service is accessed by its IP
• There is no way, up to my knowledge, to enforce Kerberos and therefore make sure NTLM is never used. This is caused by the fact that Microsoft always plans for offering a fallback in case Kerberos would not work.
• Finally, Kerberos requires AD and Windows. Moreover, it requires IIS to work with Integrated Windows Security. For example, this means that it is not possible to Kerberize an application using form-based authentication.

In coming posts, I’ll discuss in details how to configure Kerberos with IIS-based web application, identify some gotcha’s but also when it can save the day, with sharePoint in particular!

And cut!

Like the Avatar, Patrick Tisseghem mastered all SharePoint elements. Sadly, he passed away this week. I will never thank him enough for being such a great teacher.

My thoughts are going to his family, friends and colleagues from U2U.
He will live forever through the incomparable contribution he brought to the SharePoint community.

And cut!

Some posts in NG as well as a customer case reminded me of that infrequent issue that may happen when a user visit a web site or attempts to use a web-based application hosted on IIS.

HTTP requests/responses contain two parts: the header and the body. The header contains most of time technical information exchanged between the client and the server, the body contains user-oriented information like the content of a webpage or a file to download, for example. The error message ere above is therefore generated by the server because the request sent by the client contains a header that is simply too large compared to what the server expects. But what can make a header section so large? It all depends on how you browser was configured (and the underlying OS as well in some case), but most of time, the culprits of larger header are cookies (header: Cookie) and authentication information (Header: Authorization).

When this error is experienced on an Internet site, there is not much you can do except cleaning up your cookies and hoping it will work afterwards while.
When this error shows up in an intranet environment when web servers are running IIS and possibly SharePoint, SQL Server Reporting Services or Exchange with OWA on top of it, this is caused by a combination of multiple factors which are the following:
- The web server (or web site) was configured to use Integrated Windows Authentication and Kerberos in particular
- The client is able to authenticate using Kerberos (the client system is member of an AD forest, the user too)
- The size of the user’s security token is large. This can be caused by: the user being member of many AD groups  (hundreds of groups), the user’s object in AD contains SID history information as consequence of a domain migration/consolidation, the groups the user is member of is also affected by SID history, just like the user.
IIS is configured by default
- The client sends Kerberos-based authentication AND authorization information. Unlike NTLM and Basic which only send authentication information, therefore smaller, Kerberos includes information such as group membership and SID history information in the request’s header.
-> Depending on the user’s group membership and SID history information, some users may be affected, others not…

Hopefully, there are many solution to work around this problem but all of them have their trade-offs:

Use IP address in the URL instead of host names
Since Kerberos solely works with host names, using UIP addresses will automatically force negotiation to NTLM instead. Of course, this does not only degrades security but it also involves hard coding IP’s, which is rarely practically and also means that this can work if you only host one and only one web site on your IIS, finally, if your application uses “delegation” of credentials, which is a Kerberos feature, it will not work anymore (I am thinking about SQL SRS in particular in this case, or even Exchange OWA when used in FE-BE scenario’s)

Configure IIS to use NTLM only
As part of the Integrated windows Authentication setup, you can simply configure IIS to use NTLM only, the following MS KB article will show you how to do so: http://support.microsoft.com/kb/215383. Although this configuration is less impacting that the first one, you’re still left with lower security as well an compatibility issue with Kerberos is required by your application.

Configure IIS to accept larger headers
You can do so by configuring IIS in registry. It is important to note that this configuration will apply to all web sites running on the system running IIS because this settings is used at kernel-component level (http.sys), it will therefore impact all your applications on that server. This MS KB article explains all those settings http://support.microsoft.com/kb/820129, the one that is to change being MaxFieldLength. Be very careful doing so because this will increase the memory used by the system (kernel memory) to handle requests. On a “busy” (read: getting a lot of requests, not fewer large requests) 32-bit system, this can exhaust kernel memory. If the boot.ini switch /3GB is used (possibility combined to /USERVA), the situation can get worse since less memory is available to the kernel. On a 64-bit system, this configuration is harmless, even if the application is running in 32-bit mode, since this is handled in kernel mode. Of course, take care of what your application is doing with those headers too…

Plan a crash-diet for your AD users
By reducing (read optimizing) their group membership and removing SID history information from both user’s and group’s AD object attribute. Though this solution will be profitable in all scenario and not only web authentication (faster logon, less memory usage on application servers, Exchange mailbox servers…), never implement without a careful impact assessment. This MS KB article explains how to automate this task: http://support.microsoft.com/kb/295758

Now for some background information:
- Since that error is generated by the HTTP.SYS driver, is will not be logged in the W3C of whatever files but in HTTPERR.LOG. To track who is affected, simply look at the client’ip in the log. Of course, since authentication information cannot be handled, user’s identity remains unknown.
- To estimate the token size for general purposes, no only for this issue in particular, you can use this tool provided by MS: http://www.microsoft.com/downloads/details.aspx?FamilyID=4A303FA5-CF20-43FB-9483-0F0B0DAE265C&displaylang=en
- The way IIS handles headers and therefore authorization information is one thing, the way Windows system, in general, handle large token is another. Make sure you keep you IIS and windows configuration consistent so that authentication is successful end-to-end. Read this MS paper for more information: http://www.microsoft.com/downloads/details.aspx?familyid=22DD9251-0781-42E6-9346-89D577A3E74A&displaylang=en
- A good network monitor (I like to call it http://www.wireshark.org/) and well as a troubleshooting tool like MS WFetch (http://www.microsoft.com/downloads/details.aspx?FamilyID=b134a806-d50e-4664-8348-da5c17129210&DisplayLang=en) will help the troubleshooting task. As usual, MS LogParser is your friend when it comes to analyzing IIS log files, look at this forum for help parsing HTTPERR.LOG http://forums.iis.net/1141.aspx or keep on reading this blog for more LogParser scripts.
- For simplicity’s sake, I use the work “Kerberos” when talking about authentication protocol between client and web server. The actual protocol is SPNEGO or “Negotiate”, which is a wrapper for multiple authentication protocols. Refer to Wikipedia for the details: http://en.wikipedia.org/wiki/SPNEGO

And cut!

While configuring client to web server Kerberos-based authentication can be relatively simple, doing the same for server to server communication (SQL, Crawl or SSP for example) can turn out to be something really tricky.
Hopefully Microsoft recently published an excellent paper on this subject on TechNet: http://technet.microsoft.com/en-us/library/cc263449.aspx

While talking about Kerberos client to server, I’d definitely recommend reading Ken Schaefer’s serie of blog posts over Kerberos and IIS, which is a must for all IIS admins. Please note that 90% is applicable to SharePoint too…: http://www.adopenstatic.com/cs/blogs/ken/archive/2006/10/19/512.aspx

And cut!