Converting NCSA log files to W3C format

30. November 2005 Bob IIS , Scripting Comments (0)

One of the great utilities that ships with IIS is the CONVLOG.EXE application, which converts W3C or MS Internet Standard log files to NCSA format, where they can be processed by any of the applications that only parse NCSA log file information. The trouble is, what happens when you already have NCSA log files and you want W3C log files? You can't use the CONVLOG.EXE application, it only works in the opposite direction.

With that in mind, I wrote the following Windows Script Host (WSH) script that will read the current directory and convert all NCSA-formatted log files to W3C format. To use this code, just copy the code into notepad, and save it with a ".vbs" file extension on your system. To run it, copy the script to a folder that contains NCSA log files, (named "nc*.log"), then double-click it.

Option Explicit

Dim objIISLog
Dim objFSO
Dim objFolder
Dim objFile
Dim objOutputFile
Dim strInputPath
Dim strOutputPath
Dim strLogRecord

Set objFSO = WScript.CreateObject("Scripting.FileSystemObject")

Set objFolder = objFSO.GetFolder(".")

For Each objFile In objFolder.Files

 strInputPath = LCase(objFile.Name)

 If Left(strInputPath,2) = "nc" And Right(strInputPath,4) = ".log" Then

  strOutputPath = objFolder.Path & "\" & "ex" & Mid(strInputPath,3)
  strInputPath = objFolder.Path & "\" & strInputPath

  Set objIISLog = CreateObject("MSWC.IISLog")
  objIISLog.OpenLogFile strInputPath, 1, "", 0, ""
  Set objOutputFile = objFSO.CreateTextFile(strOutputPath)

  objIISLog.ReadLogRecord

  objOutputFile.WriteLine "#Software: Microsoft Internet Information Services 5.0"
  objOutputFile.WriteLine "#Version: 1.0"
  objOutputFile.WriteLine "#Date: " & BuildDateTime(objIISLog.DateTime)
  objOutputFile.WriteLine "#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)"

  Do While Not objIISLog.AtEndOfLog

   strLogRecord = BuildDateTime(objIISLog.DateTime)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.ClientIP)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.UserName)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.ServerIP)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.ServerPort)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.Method)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.URIStem)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.URIQuery)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.ProtocolStatus)
   strLogRecord = strLogRecord & " " & FormatField(objIISLog.UserAgent)
   objOutputFile.WriteLine strLogRecord

   objIISLog.ReadLogRecord

  Loop

  objIISLog.CloseLogFiles 1
  objIISLog = Null
 
 End If

Next

Function FormatField(tmpField)
 On Error Resume Next
 FormatField = "-"
 If Len(tmpField) > 0 Then FormatField = Trim(tmpField)
End Function

Function BuildDateTime(tmpDateTime)
 On Error Resume Next
 tmpDateTime = CDate(tmpDateTime)
 BuildDateTime = Year(tmpDateTime) & "-" & _
  Right("0" & Month(tmpDateTime),2) & "-" & _
  Right("0" & Day(tmpDateTime),2) & " " & _
  Right("0" & Hour(tmpDateTime),2) & ":" & _
  Right("0" & Minute(tmpDateTime),2) & ":" & _
  Right("0" & Second(tmpDateTime),2)
End Function

I hope this helps!

How to Record Logon Activity in W3C Extended Log File Format using WSH

11. June 2002 Bob Windows Comments (1)

Many years ago I put together a bunch of information about logging system activity in W3C format by using Group Policy Objects and Windows Script Host. All of that information was supposed to become Microsoft KB article 324414, but I changed teams and I eventually lost track of its status. Recently I had a need for the information in that KB article and discovered that it was never published, so I had to look for my notes to reconstruct what was supposed to be in the KB article, and I thought that all that effort would make a good blog post.

(Note: This blog post has been updated a few times since it was first posted in order to keep it up-to-date.)

IN THIS POST

APPLIES TO

  • Windows Server 2008 R2
  • Windows 7
  • Windows Server 2008
  • Windows Vista
  • Windows Server 2003 R2
  • Windows Server 2003
  • Windows XP
  • Windows Server 2000

SUMMARY

The steps in this blog post will show you how to configure your network for additional logon/logoff information for all domain clients by using a sample Windows Script Host (WSH) script to create log files that conform to the W3C Extended Log File (ExLF) Format.

The W3C Extended Log File Format is currently used on Windows servers by the various web services that install with Internet Information Services. These log files are kept in your %SystemRoot%\System32\LogFiles or %SystemRoot%\Inetsrv\Logs\LogFiles folder. By configuring this sample logging script through a domain-level Group Policy, a new folder named Activity will be created under the %SystemRoot%\System32\LogFiles folder containing log entries formatted like the following example:

#Description: Log file for all LOGON/LOGOFF activity
#Date: 2002-01-01 21:28:50
#Fields: date time s-computername cs-username cs-method
2002-01-01 21:28:50 MYCOMPUTER LOCALHOST\SYSTEM STARTUP
2002-01-01 21:32:55 MYCOMPUTER MYDOMAIN\userone LOGON
2002-01-01 21:45:58 MYCOMPUTER MYDOMAIN\userone LOGOFF
2002-01-01 21:47:00 MYCOMPUTER MYDOMAIN\usertwo LOGON
2002-01-01 21:52:02 MYCOMPUTER MYDOMAIN\usertwo LOGOFF
2002-01-01 21:53:09 MYCOMPUTER LOCALHOST\SYSTEM SHUTDOWN

Since there are a wide variety of applications that can process log files in the W3C Extended Log File Format, recording logs in this format allows domain administrators to use tools they are already familiar with when analyzing network logon/logoff information.

NOTE: The W3C Extended Log File Format requires that all times must be kept in Greenwich Mean Time (GMT). As such, all logon/logoff activity recorded by the script in this article will be listed in GMT. This allows a uniform standard for large-scale networks that traverse multiple time zones.

MORE INFORMATION

Step 1 - Create the Sample Logging Script

  1. Log on to your Windows Domain Controller as a Domain Administrator.
  2. Open Windows Notepad by clicking Start, then All Programs, then Accessories, and then Notepad.
  3. Type or paste the following WSH code into notepad: 
    Option Explicit
    On Error Resume Next
     
    ' declare all variables
    Dim objFSO,objFile
    Dim objNet,objShell
    Dim objProcess,objArgs
    Dim strFolder,strFile
    Dim blnFileExists
    Dim objDateTime,lngTimeZoneOffset
    Dim strYear,strMonth,strDay
    Dim strLongDate,strShortDate
    Dim strShortTime,strMethod
    Dim strComputerName,strUserDomain,strUserName
     
    ' create all objects
    Set objNet = WScript.CreateObject("WScript.Network")
    Set objFSO = WScript.CreateObject("Scripting.FileSystemObject")
    Set objShell = WScript.CreateObject("WScript.Shell")
    Set objProcess = objShell.Environment("PROCESS")
    Set objArgs = WScript.Arguments
     
    ' process arguments
    If objArgs.Count <> 1 Then WScript.Quit
    strMethod = UCase(objArgs(0))
     
    ' perform date operations
    lngTimeZoneOffset = GetTimeZoneOffset()
    objDateTime = Now() - lngTimeZoneOffset
    strYear  = CStr(Year(objDateTime))
    strMonth = Right("00" & CStr(Month(objDateTime)),2)
    strDay   = Right("00" & CStr(Day(objDateTime)),2)
    strLongDate = strYear & "-" & strMonth & "-" & strDay
    strShortDate = Right(strYear,2) & strMonth & strDay
    strShortTime = FormatDateTime(objDateTime,4) & ":" & Right("00" & CStr(Second(objDateTime)),2)
     
    ' get network information
    strComputerName = objNet.ComputerName
    If Len(strComputerName) = 0 Then strComputerName = "LOCALHOST"
    strUserDomain = objNet.UserDomain
    If Len(strUserDomain) = 0 Then strUserDomain = "LOCALHOST"
    strUserName = objNet.UserName
    If Len(strUserName) = 0 Then strUserName = "()"
     
    ' get windows directory name
    strFolder = objProcess("WINDIR")
     
    ' check for and create "System32" folder
    strFolder = strFolder & "\System32"
    If objFSO.FolderExists(strFolder) = False Then
      objFSO.CreateFolder(strFolder)
    End If
     
    ' check for and create "LogFiles" folder
    strFolder = strFolder & "\LogFiles"
    If objFSO.FolderExists(strFolder) = False Then
      objFSO.CreateFolder(strFolder)
    End If
     
    ' check for and create "ACTIVITY" folder
    strFolder = strFolder & "\ACTIVITY"
    If objFSO.FolderExists(strFolder) = False Then
      objFSO.CreateFolder(strFolder)
    End If
     
    ' set up log file name
    strFile = "ex" & strShortDate & ".log"
     
    ' check if log file exists
    blnFileExists = objFSO.FileExists(strFolder & "\" & strFile)
     
    ' open or create the log file
    Set objFile = objFSO.OpenTextFile(strFolder & "\" & strFile,8,True)
     
    ' write headers if new file
    If blnFileExists = False Then
      objFile.WriteLine "#Description: Log file for all LOGON/LOGOFF activity"
      objFile.WriteLine "#Date: " & strLongDate & " " & strShortTime
      objFile.WriteLine "#Fields: date time s-computername cs-username cs-method"
    End If
     
    ' write the log data
    objFile.WriteLine strYear & "-" & strMonth & "-" & strDay & " " & _
      strShortTime & " " & _
      strComputerName & " " & _
      strUserDomain & "\"  & _
      strUserName & " " & _
      strMethod
     
    ' close the log file
    objFile.Close
     
    Function GetTimeZoneOffset()
      On Error Resume Next
      Dim tmpShell,tmpOffset
      Set tmpShell = WScript.CreateObject("WScript.Shell")
      tmpOffset = objShell.RegRead("HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation\ActiveTimeBias")
      If Len(tmpOffset) = 0 Then
        tmpOffset = objShell.RegRead("HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation\Bias")
      End If
      ' set a default offset if none can be determined
      If Len(tmpOffset) = 0 Then tmpOffset = "0"
      ' calculate offset in hours
      tmpOffset = (CLng(tmpOffset) * -1) / 60
      ' calculate offset in 1/24 of a day
      tmpOffset = tmpOffset / 24
      GetTimeZoneOffset = tmpOffset
    End Function
  4. Save the file:
    • Click the File menu, and then Save.
    • When the Save As dialog appears, choose your desktop as the destination.
    • Enter activity.vbs for the File name.
    • Click the Save button.
  5. Click the File menu, and then Exit to close Notepad.

Step 2 - Copy the Sample Logging Script to your Group Policy Folders

To use the sample script with the Default Domain Policy Group Policy Object (GPO), you first need to determine the Globally Unique Identifier (GUID) for the GPO. To do so, use the following steps:

  1. Start the Active Directory Users and Computers snap-in in the Microsoft Management Console (MMC). To do so, click Start, point to All Programs, point to Administrative Tools, and then click Active Directory Users and Computers.
  2. Right-click your domain, and then click Properties.
  3. Click the Group Policy tab.
  4. Highlight the Default Domain Policy, and then click the Properties button:
    • The GUID for the GPO will be listed as the Unique name property in the Summary section of the properties dialog.
    • The Default Domain Policy GUID will always be {31B2F340-016D-11D2-945F-00C04FB984F9}, if you choose enable logging in a different policy this will be a different GUID.
  5. Click the Cancel button to close the GPO properties dialog.
  6. Click the Cancel button to close the domain properties dialog.

To use the sample script with the GPO, you will need to copy the activity.vbs script on your desktop to each of the following paths:

%SystemRoot%\SYSVOL\sysvol\<DOMAIN>\Policies\<GUID>\USER\Scripts\Logon
%SystemRoot%\SYSVOL\sysvol\<DOMAIN>\Policies\<GUID>\USER\Scripts\Logoff
%SystemRoot%\SYSVOL\sysvol\<DOMAIN>\Policies\<GUID>\MACHINE\Scripts\Startup
%SystemRoot%\SYSVOL\sysvol\<DOMAIN>\Policies\<GUID>\MACHINE\Scripts\Shutdown

Where <DOMAIN> is the Fully Qualified Domain Name (FQDN) of your domain, (e.g. mydomain.local ), and <GUID> is the Globally Unique Identifier (GUID) for the Default Domain Policy GPO.

Step 3 - Configure the Script to Record LOGON/LOGOFF Activity

  1. Start the Active Directory Users and Computers snap-in in the Microsoft Management Console (MMC). To do this, click Start , point to Programs , point to Administrative Tools , and then click Active Directory Users and Computers .
  2. Right-click your domain, then click Properties .
  3. Click the Group Policy tab.
  4. Highlight the Default Domain Policy , then click the Edit button.
  5. In the console tree, click the plus sign (+) next to the Windows Settings under User Configuration , then highlight Scripts (Logon/Logoff) .
  6. Add the Logon script:
    1. In the right pane, double-click the Logon item.
    2. Click the Add button.
    3. Click the Browse button.
    4. Highlight activity.vbs , then click the Open button.
    5. Type LOGON in the Script Parameters box.
    6. Click OK to add the script.
    7. Click OK to close the Logon scripts dialog.
  7. Add the Logoff script:
    1. In the right pane, double-click the Logoff item.
    2. Click the Add button.
    3. Click the Browse button.
    4. Highlight activity.vbs , then click the Open button.
    5. Type LOGOFF in the Script Parameters box.
    6. Click OK to add the script.
    7. Click OK to close the Logoff scripts dialog.
  8. Close the Group Policy Editor.
  9. Click OK to close the domain properties dialog.

Step 4 - Configure the Script to Record STARTUP/SHUTDOWN Activity

  1. Start the Active Directory Users and Computers snap-in in the Microsoft Management Console (MMC). To do this, click Start , point to Programs , point to Administrative Tools , and then click Active Directory Users and Computers .
  2. Right-click your domain, then click Properties .
  3. Click the Group Policy tab.
  4. Highlight the Default Domain Policy , then click the Edit button.
  5. In the console tree, click the plus sign (+) next to the Windows Settings under Computer Configuration , then highlight Scripts (Startup/Shutdown) .
  6. Add the Startup script:
    1. In the right pane, double-click the Startup item.
    2. Click the Add button.
    3. Click the Browse button.
    4. Highlight activity.vbs , then click the Open button.
    5. Type STARTUP in the Script Parameters box.
    6. Click OK to add the script.
    7. Click OK to close the Startup scripts dialog.
  7. Add the Shutdown script:
    1. In the right pane, double-click the Shutdown item.
    2. Click the Add button.
    3. Click the Browse button.
    4. Highlight activity.vbs , then click the Open button.
    5. Type SHUTDOWN in the Script Parameters box.
    6. Click OK to add the script.
    7. Click OK to close the Shutdown scripts dialog.
  8. Close the Group Policy Editor.
  9. Click OK to close the domain properties dialog.

TROUBLESHOOTING

If the Logon Script does not run, you may need to check your network connection speed as the script may not run when you first log on to the network. For additional information on this issue, click the article numbers below to view the articles in the Microsoft Knowledge Base:

302104 The Logon Script Does Not Run During the Initial Logon Process

REFERENCES

For more information on the extended log file format, see the specification in the W3C Working Draft at the following URL:

http://www.w3.org/TR/WD-logfile

For additional information on assigning Logon/Logoff Scripts, click the article number below to view the article in the Microsoft Knowledge Base:

322241 HOW TO: Assign Scripts in Windows 2000

For additional information on the Extended Log File Format, click the article numbers below to view the articles in the Microsoft Knowledge Base:

194699 Extended Log File Format Always in GMT

271196 IIS Log File Entries Have the Incorrect Date and Time Stamp

242898 IIS Log File Naming Syntax