Windows Server 2003 Failover Clustering:
In Windows 2003 Failover Clustering, the cluster service on each node constantly writes to a live debug output file. These files are located in the %SystemRoot%\Cluster folder on each node in the cluster and the name of the file is Cluster.Log.
- Cluster log is local and specific to each node’s cluster service.
- Each node has a unique log file.
- Log file is in a text format (easily viewed with Word, Notepad)
- Clustering logs information about cluster activities - i.e. updates between nodes, errors and warnings related to problems. This log is very valuable when trying to troubleshoot just about any problem encountered with a cluster.
- The cluster log is a record of the Cluster service (Clussvc.exe and associated processes) activity in a server cluster members. A diagnostic log.
Windows Server 2008 Failover Clustering:
In Windows Server 2008, the Cluster.log file is not auto-generated. A change was made to make the cluster debug logging mechanism more in line with how the rest of Windows handles event logging. This functionality is with Windows Event Tracing (ETW) process. This is the same logging infrastructure that handles events for other aspects such as the System or Application Event logs you view in Event Viewer. Instead of writing to a Cluster.Log text file, it writes it to a Diagnostics log located in the C:\Windows\System32\winevt\logs folder.
- 3 diagnostics logs will exist:
Format in Windows Server 2008: ClusterLog.etl.001, ClusterLog.etl.002, ClusterLog.etl.003
Format in Windows Server 2008 R2:
Microsoft-Windows-FailoverClustering Diagnostic.etl.001
Microsoft-Windows-FailoverClustering Diagnostic.etl.002
Microsoft-Windows-FailoverClustering Diagnostic.etl.003
- By default, keep the most recent three ETL files. Only one of these files is the active or “live” log at any given time.
- Cluster event tracing is enabled by default when you first configure the cluster and start the cluster service. The log files are stored in: %WinDir%\System32\winevt\logs\
- The log files are stored in an *.etl format.
- Every time a server is rebooted, it takes the previous *.etl file and appends it with a 00X suffix a new ETL file is generated and logging is conducted on the new log file until the server is rebooted again.
- Each event will be one line at a time.
- The default size of these logs is 100MB each.
- You can see the FailoverClustering ETW trace session (Microsoft-Windows-FailoverClustering) in Reliability and Performance Monitor(or Performance Monitor_ under Data Collector Sets, Event Trace Sessions.
Going through event by event in these diagnostic event logs can be time consuming. Instead, you can create a Cluster.Log text file with commands that combine all three of these logs into one to make the review of it much easier. i.e. A readable version of the ETL files. The trace sessions are dumped into a text file that looks very similar to the legacy CLUSTER.LOG.
The new logging mechanism provides more options and flexibility. ex. /copy and /span options make targeting a specific time frame and getting the logs from all nodes much easier.
If you are running Windows Server 2008, you can use the cluster.exe command line . If you are running Windows Server 2008 R2, you can use either cluster.exe command line or the cluster PowerShell cmdlets.
*Note:
- PowerShell Get-ClusterLog command goes out to all nodes and generates a Cluster.Log on each node and stores it in the C:\Windows\Cluster\Reports folder. Get-ClusterLog has switches you might want to use (i.e. timespan, include only one node, store file on each node)
- Cluster.Log will be a snapshot in time. It will take what’s there right now and it will not be updated with anything after it’s generated/created.
To Generate/Create the Cluster.Log in Windows Server 2008
(not created by default in Windows Server 2008)
From one of the nodes of the cluster, open a Command Prompt with Administrator rights. To generate/create the log is to type “cluster log /g”. A cluster.log file will be created and stored in the %windir%\Cluster\Reports directory on each node of the cluster.
There are a number of options/commands/switches for the cluster.exe:
/Copy:
- Take the cluster.log that is generated on each node and copy it to a directory.
The directory that you specify should be a subdirectory under the path which the command prompt is showing
i.e To store the logs at c:\archive\logs, then set the command prompt to c:\archive and then execute the “cluster log /g /copy:logs” command.
/Span:
- Specifies the number of minutes to go back in time for the log collection. (note: Generating the cluster.log does not remove any data from the servers)
/Node:
- Specification of a specific node. All the other nodes will not have a log created/generated.
- Useful if not all the cluster nodes are up or some don’t have the cluster service started - which can cause a long delay with cluster log command.
/Level:<0-5>
- Change the logging level.
- For Windows Server 2008, this has a default level of 3, which is the equivalent of what is captured by cluster.log in previous versions of Windows Server
- If change this level to a higher number, more detailed information will be logged, the .etl file capturing the tracing will fill faster and there can be a small impact on system performance.
Note: This can also be accomplished in Powershell via command Get-ClusterLog.
ex. Get-ClusterLog -Destination - this command creates the cluster log files for each cluster nodes and copies all logs to a central location or destination specified in the command. This is useful when you want to view all cluster logs from different nodes from a single place.
Failover Cluster Manager:
- The new interface for managing a cluster. Tool for managing groups, resources, dependencies & more.
- "Cluster Events" - can view System event errors and warnings from within Failover Cluster Manager. "Cluster Events" - link provides a way to display all warnings and errors that have occurred with Failover Cluster as the source in the past 24 hours. It pulls events from all nodes. Use the Query option to filter/look for specific events.
- "Validate this Cluster" - before a Cluster can be “certified,” all it's components must be listed on the Windows Server Catalog and it must pass a Cluster Validate. Run Cluster Validate before the cluster is created or just after. If there is a problem after a Cluster is in Production, run Cluster Validate again as a troubleshooting tool. This can be ran while in Production fine. It will ask if you want to take the running groups offline or leave online(default). Cluster Validate will create a file in the C:\Windows\Cluster\Reports directory every time you run it and it will create the file on all nodes that Cluster Validate was ran against.
Event Channels:
Channels relevant to Failover Clustering:
- Informational & Diagnostic events
Tracerpt: (as an alterative to creating the Cluster.Log)
The .etl files themselves are not consumable by any viewer directly, but you can dump the contents into several different formats using tracerpt.exe
The tracerpt command can be used to parse Event Trace Logs, log files generated by Performance Monitor and real-time Event Trace providers. It generates dump files, report files, and report schemas.
Image 1: Event Viewer
Image 2: Performance Monitor - Event Trace Sessions (also visible in Server Manager)
Image 3:
Image 4:
Image 5: Create Cluster Log
Image 6: Increase Cluster Log Size
Image 7: Increase Cluster Log Logging Level
Default: Level 3 - Error, Warning, Information
Level 5 adds Verbose & Debug logging information
Level 0 would disable logging
Image 8: View Cluster Properties
Image 9: Cluster Command Options
T-SQL Cluster Queries:
SELECT SERVERPROPERTY('IsClustered') --If clustered the query will return “1”
SELECT SERVERPROPERTY('MachineName')
SELECT SERVERPROPERTY('InstanceName')
SELECT SERVERPROPERTY('ServerName')
-- Get all Cluster Shared Drive Letters
SELECT Drivename FROM sys.dm_io_cluster_shared_drives;
SELECT * FROM fn_servershareddrives()
-- Get all Cluster Node Machine Names
SELECT NodeName FROM sys.dm_os_cluster_nodes;
SELECT * FROM fn_virtualservernodes()
-- Determing what machine owns the cluster resources
SELECT SERVERPROPERTY('ComputerNamePhysicalNetBIOS')
Resources:
Cluster Diagnostics and Verification Tool (ClusDiag.exe)
|
|
|
|
|
|