All product based on SharePoint technologies come with a built-in logging engine named Unified Logging System (ULS). It allows the applications and related component (Microsoft or third-parties) to log activity to the Windows application Event Log and/or to a log file on each server running SharePoint.

Log Location

The log files are, by default, located under C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS, the file names always start with a prefix consisting in the name of the server they were generated on: <servername>-<output-format>.log.

Depending upon their configuration, some event may also be logged in the Windows Application event log.

Configuration

To change the location of the log files, the following PowerShell script can be used:

[VOID][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
[VOID][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Administration")
$SPDiagnosticsService = [Microsoft.SharePoint.Administration.SPDiagnosticsService]::Local
$SPDiagnosticsService.LogLocation = "G:\Logs\"
$SPDiagnosticsService.Update()

To change the number of log files to be maintained, you can set the “LogsToKeep” property:

Beware: as soon as the Update() method is invoked, the log files above the value specified will be deleted!

Note: this correspond to the setting stored in the registry at the following location: HKLM:\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS. You might therefore therefore be tempted to edit it directly but MS discourage to do so and recommended to use the SharePoint API instead.

To list/set the verbosity level of each component, STSADM can be used:

To list all level (including hidden ones): stsadm.exe -o listlogginglevels [-showhidden]

To set the level for a given category: stsadm.exe -o setlogginglevel   [-category < [CategoryName | Manager:CategoryName [;...]] ] {-default | -tracelevel < trace level setting> [-windowslogginglevel] <Windows event log level setting>}

More information:

• http://technet.microsoft.com/en-us/library/cc262191.aspx
• http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.administration.spdiagnosticsservice.aspx

Log File Format

The log files expose the following fields:

• Timestamp: Equivalent to the “TimeGenerated” field in the “Application” event Log
• Process: the image name of the process logging its activity followed by its process ID (PID) between parentheses. Interestingly, IIS worker processes may also log their activity, they are therefore logged under w3wp.exe
• TID
• Area: This maps the “Source” field in the “Application” event Log
• Category: this maps the “Category” field in the “Application” event Log
• EventID: A unique internal Event ID
• Level
• Message
• Correlation: may contain a link to the the EventID of another logged event

Exploitation/Analysis

There are multiple ways to analyze ULS logs, such as:

• Using the good old notepad, which is a valid option when your servers are not too busy and if you know when, where and what to look for
• Using the Swiss army knife of log files: MS LogParser, see below for examples.
• Using some interesting CodePlex projects: An extension to the Central Administration web site: http://www.codeplex.com/wssmosslogfilereader or a WinForm-based application (which also makes use of MS LogParser): http://www.codeplex.com/ShrptNinjaToolkit
• [UPDATED] From the MS Office Project Web Log, how to read logs using Excel: http://blogs.msdn.com/brismith/archive/2009/01/04/project-server-2007-reading-sharepoint-uls-logs-with-excel.aspx and http://blogs.msdn.com/brismith/archive/2006/12/14/finding-your-way-around-the-sharepoint-logs.aspx

Some Log Parser Queries applicable to ULS Logs

• Troubleshooting Indexing with ULS Log by Joel Oleson
• ULS Log File Data Mining
• I will also upload my own queries in a near future, so check back this location regularly

More Information

• http://msdn.microsoft.com/en-us/library/aa979590.aspx

And cut!

Some posts in NG as well as a customer case reminded me of that infrequent issue that may happen when a user visit a web site or attempts to use a web-based application hosted on IIS.

HTTP requests/responses contain two parts: the header and the body. The header contains most of time technical information exchanged between the client and the server, the body contains user-oriented information like the content of a webpage or a file to download, for example. The error message ere above is therefore generated by the server because the request sent by the client contains a header that is simply too large compared to what the server expects. But what can make a header section so large? It all depends on how you browser was configured (and the underlying OS as well in some case), but most of time, the culprits of larger header are cookies (header: Cookie) and authentication information (Header: Authorization).

When this error is experienced on an Internet site, there is not much you can do except cleaning up your cookies and hoping it will work afterwards while.
When this error shows up in an intranet environment when web servers are running IIS and possibly SharePoint, SQL Server Reporting Services or Exchange with OWA on top of it, this is caused by a combination of multiple factors which are the following:
- The web server (or web site) was configured to use Integrated Windows Authentication and Kerberos in particular
- The client is able to authenticate using Kerberos (the client system is member of an AD forest, the user too)
- The size of the user’s security token is large. This can be caused by: the user being member of many AD groups  (hundreds of groups), the user’s object in AD contains SID history information as consequence of a domain migration/consolidation, the groups the user is member of is also affected by SID history, just like the user.
IIS is configured by default
- The client sends Kerberos-based authentication AND authorization information. Unlike NTLM and Basic which only send authentication information, therefore smaller, Kerberos includes information such as group membership and SID history information in the request’s header.
-> Depending on the user’s group membership and SID history information, some users may be affected, others not…

Hopefully, there are many solution to work around this problem but all of them have their trade-offs:

Use IP address in the URL instead of host names
Since Kerberos solely works with host names, using UIP addresses will automatically force negotiation to NTLM instead. Of course, this does not only degrades security but it also involves hard coding IP’s, which is rarely practically and also means that this can work if you only host one and only one web site on your IIS, finally, if your application uses “delegation” of credentials, which is a Kerberos feature, it will not work anymore (I am thinking about SQL SRS in particular in this case, or even Exchange OWA when used in FE-BE scenario’s)

Configure IIS to use NTLM only
As part of the Integrated windows Authentication setup, you can simply configure IIS to use NTLM only, the following MS KB article will show you how to do so: http://support.microsoft.com/kb/215383. Although this configuration is less impacting that the first one, you’re still left with lower security as well an compatibility issue with Kerberos is required by your application.

Configure IIS to accept larger headers
You can do so by configuring IIS in registry. It is important to note that this configuration will apply to all web sites running on the system running IIS because this settings is used at kernel-component level (http.sys), it will therefore impact all your applications on that server. This MS KB article explains all those settings http://support.microsoft.com/kb/820129, the one that is to change being MaxFieldLength. Be very careful doing so because this will increase the memory used by the system (kernel memory) to handle requests. On a “busy” (read: getting a lot of requests, not fewer large requests) 32-bit system, this can exhaust kernel memory. If the boot.ini switch /3GB is used (possibility combined to /USERVA), the situation can get worse since less memory is available to the kernel. On a 64-bit system, this configuration is harmless, even if the application is running in 32-bit mode, since this is handled in kernel mode. Of course, take care of what your application is doing with those headers too…

Plan a crash-diet for your AD users
By reducing (read optimizing) their group membership and removing SID history information from both user’s and group’s AD object attribute. Though this solution will be profitable in all scenario and not only web authentication (faster logon, less memory usage on application servers, Exchange mailbox servers…), never implement without a careful impact assessment. This MS KB article explains how to automate this task: http://support.microsoft.com/kb/295758

Now for some background information:
- Since that error is generated by the HTTP.SYS driver, is will not be logged in the W3C of whatever files but in HTTPERR.LOG. To track who is affected, simply look at the client’ip in the log. Of course, since authentication information cannot be handled, user’s identity remains unknown.
- To estimate the token size for general purposes, no only for this issue in particular, you can use this tool provided by MS: http://www.microsoft.com/downloads/details.aspx?FamilyID=4A303FA5-CF20-43FB-9483-0F0B0DAE265C&displaylang=en
- The way IIS handles headers and therefore authorization information is one thing, the way Windows system, in general, handle large token is another. Make sure you keep you IIS and windows configuration consistent so that authentication is successful end-to-end. Read this MS paper for more information: http://www.microsoft.com/downloads/details.aspx?familyid=22DD9251-0781-42E6-9346-89D577A3E74A&displaylang=en
- A good network monitor (I like to call it http://www.wireshark.org/) and well as a troubleshooting tool like MS WFetch (http://www.microsoft.com/downloads/details.aspx?FamilyID=b134a806-d50e-4664-8348-da5c17129210&DisplayLang=en) will help the troubleshooting task. As usual, MS LogParser is your friend when it comes to analyzing IIS log files, look at this forum for help parsing HTTPERR.LOG http://forums.iis.net/1141.aspx or keep on reading this blog for more LogParser scripts.
- For simplicity’s sake, I use the work “Kerberos” when talking about authentication protocol between client and web server. The actual protocol is SPNEGO or “Negotiate”, which is a wrapper for multiple authentication protocols. Refer to Wikipedia for the details: http://en.wikipedia.org/wiki/SPNEGO

And cut!

While M. Night Shyamalan (http://www.imdb.com/name/nm0796117/) is trying to sell some tickets for his new (below the average) movie The Happening (http://www.imdb.com/title/tt0949731/), I had the opportunity to help some AD admins reporting their questions in newsgroups.

A recurring one is “how can I know to which DC my users are going to authenticate”? Well, there are multiples solutions, it all depends on how you want to do and how much effort you want to invest…

For example, you can use the NLTEST command from the windows Resource Kit Tools: NLTEST /server:MYCLIENTCOMPUTER /dsgetdc:MYDOMAINDNSNAME will report to which DC a user belonging to the domain MYDOMAINDNSNAME if logged on the computer named MYCLIENTCOMPUTER

Another simple method, provided by the talented MVP Gilles Laurent (http://glsft.free.fr), is to create or modify your common logon script in batch with a line such as:echo %date:~-10%
%time:~-11,8%,%computername%,%logonserver:~2%>>\\server\share\LogonUsers.log and to create a share on a server where all user have permission to write.

Finally, there is the LogParser way. Assuming successful logon events are audited on all your DC’s, you can download and run the following scripts against all you DC’s (as long as you have the appropriate privileges): http://www.marc-antho-etc.net/scriptdetail.aspx?script=adSuccessNetLogons.cmd.txt and http://www.marc-antho-etc.net/scriptdetail.aspx?script=adSuccessNetLogons.lpq.txt. These will create a CSV file with the following fields: DC,Date-Time,Domain,User,Authentication,ClientName and ClientIP

It also takes care of not including anonymous logons. But be aware that this may turn to be a slow process across WAN’s…

Once you have the CSV in hand, you can look for the mapping between a DC and the client IP or the client Name or even run additional data extraction query such as:Returning the number of logons per authentication provider (NTLM, Kerberos…): http://www.marc-antho-etc.net/scriptdetail.aspx?script=adSuccessInteractLogonsByAuth.lpq.txt... And many more you’ll find in the AD section of the Scripting and Automation Area: http://www.marc-antho-etc.net/scripts.aspx?prodtech=AD 

Now let’s hope you’ll find better twists in your AD than in The Happening.

And Cut!