VIOPS: Proven Practices for Deploying and Managing VMware : Document List - Logging

Setting up a Splunk Server to Monitor a VMware Environment

Tue, 30 Jun 2009 22:43:32 GMT

Introduction

Gathering and maintaining log files is an important part of a server administrator's duties. Using a centralized logging server, such as a sylog server offers several benefits. The log files become useful for troubleshooting purposes, if needed. Also, keeping an unaltered set of logs in a different location can aid in forensic activities after an attack.

This document explains how to set up Splunk for monitoring a VMware Environment. This includes monitoring the ESX/ESXi Server logs, the vCenter Server Logs and some of the add-on services to vCenter. It also includes generic event logging for Windows and Linux guest operating systems.

Intended Audience

VMware Certified Professionals, System Management / Sysadmin / Operations

Outline

Requirements
Preparing the Splunk Server
Installing Splunk Server
Setting up ESX Servers and Linux VMs for Monitoring
Setting up ESXi Servers for Monitoring
Setting up vCenter Servers and Windows VMs for Monitoring
Adding Miscellaneous Log Files to Splunk for Monitoring

Resources

Author

David Convery, VMware vExpert 2009

Also check out http://www.dailyhypervisor.com/ |

Disclaimer

You use this proven practice at your discretion. VMware, http://www.dailyhypervisor.com and the author do not guarantee any results from the use of this proven practice. This proven practice is provided on an as-is basis and is for demonstration purposes only.

ESX Server 3 vmkernel.log Lists

Sat, 11 Apr 2009 00:14:03 GMT

Introduction

This document provides the red (critical - immediate action), orange (warning - track this alert) and black (ignore but trend on this message) for the vmkernel.log of ESX Server 3.

Intended Audience

VMware Certified Professionals (VCPs) and systems management professionals when implementing log management for VI3.

Outline

Introduction to vmkernel.log
vmkernel.log Red List
vmkernel.log Orange List
vmkernel.log Black List

Introduction to vmkernel.log

All the hypervisor/vmkernel messages are posted in the Console OS log:

/var/log/vmkernel.log

Good ways to access this log:

Centralize it (doc TBD) via syslog, and us a tool like Splunk!
Use less|more to view it live
tail -f /var/log/vmkernel.log

vmkernel.log Red List

Messages on the Red List should be picked up by the log monitoring automation and alerted as a high priority outage for immediate investigation.

1. KB 1003615 - host attached to share storage failures

Message


Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)SCSI: 3753: AsyncIO timeout (5000); aborting cmd w/ sn 941463, handle 1472/0x40211a28
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)LinSCSI: 3616: Aborting cmds with world 1024, originHandle 0x40211a28, originSN 941463 from vmhba0:0:5
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)<6>qla24xx_abort_command(0): handle to abort=857
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)LinSCSI: 2604: Forcing host status from 2 to SCSI_HOST_OK
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)LinSCSI: 2606: Forcing device status from SDSTAT_GOOD to SDSTAT_BUSY
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)SCSI: 3753: AsyncIO timeout (5000); aborting cmd w/ sn 1073299, handle 2415/0x4020c038

Impact

Storage event that has caused an outage to every host accessing the shared storage.

Action

Review the storage controller or processor logs on the array for any events or messages occurring around the date the instance occurred. There are number of reasons why this kind of outage occurs like, controller problem, failing hard drive, SAN Copy operation being initiated. Also review the switch logs for the same time frame to see if the switches played a factor in this outage.

vmkernel.log Orange List

Messages on the Orange List should be picked up by the log monitoring automation and alerted as a warning for review.

vmkernel.log Black List

Messages on the Black List should be collected by the log monitoring automation and trended for future analysis.

Resources

This doc on the web @ ESX Server 3 vmkernel.log Lists

Authors

Steve Chambers

VirtualCenter 2 Logs

Fri, 10 Apr 2009 23:16:29 GMT

Introduction

This document describes all of the logs available on VirtualCenter 2.

As well as providing the location, purpose and management details for each log, there is also a link for each log to another analysis document that has a list of messages that form red (critical), orange (warning) and black (ignore) lists.

NOTE This doc is a work in progress - want to help complete it? PM/email schambers

Intended Audience

VMware Certified Professionals (VCPs) who are designing and implementing log management for VI3.

Outline

Introduction to VirtualCenter 2 Logs
VirtualCenter
Tomcat
Capacity Planner
Update Manager
Converter
License Manager

1. Introduction to VirtualCenter 2 Logs

There are logs for each of the major components of VI3 management. All of these components may run on the same virtual machine (or physical, if you are Old Skool), or may be distributed onto their own servers, virtual or otherwise

2. VirtualCenter

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs

Purpose

VirtualCenter process log files

Management

TBD

Analysis

Link TBD

3. Tomcat

Location

C:\Program Files\VMware\Infrastructure\VirtualCenter Server\tomcat\logs

Purpose

Tomcat log files, the process used for the web API.

Management

TBD

Analysis

TBD

4. Capacity Planner

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware Capacity Planner\Logs

Purpose

Capacity Planner log files

Management

TBD

Analysis

TBD

5. Update Manager

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Logs

Purpose

VMware Update Manager log files

Management

TBD

Analysis

TBD

6. Converter

Location

C:\Documents and Settings\All Users\Application Data\VMware\
VMware Converter Enterprise\Logs

Purpose

VMware Converter log files.

Manageement

TBD

Analysis

TBD

7. License Manager

Location

C:\Temp\imgrd.log

Purpose

License Manager log file.

Management

TBD

Analysis

TBD

Resources

This doc on the web @ VirtualCenter 2 Logs

Authors

Steve Chambers

ESX Server 3 Log Map

Fri, 10 Apr 2009 22:17:34 GMT

Introduction

This document describes all of the logs available on ESX Server 3.

NOTE This doc is a work in progress - want to help complete it? PM/email schambers

Intended Audience

VMware Certified Professionals (VCPs) who are designing and implementing log management for VI3.

Outline

Introduction to ESX Server 3 Logs
vmkernel
vmkwarning
vmksummary
hostd
vpxa
HA
COS
esxupdate
boot
webAccess
firewall
secure

1. Introduction to ESX Server 3 Logs

I've grouped the ESX logs into three parts:

Hypervisor
Management
Console OS (COS)

2. vmkernel

Location

/var/log/vmkernel

Purpose

VMkernel log, records activities related to the virtual machines and ESX hosts.

Management

Logs rotated by logrotate, see KB 3402740.

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension

Analysis

ESX Server 3 vmkernel.log Lists

3. vmkwarning

Location

/var/log/vmkwarning

Purpose

VMkernel Warnings log, records activities with the virtual machines, a subset of the vmkernel log.

Management

Logs rotated by logrotate, see KB 3402740.

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension

Analysis

4. vmksummary

Location

/var/log/vmksummary

/var/log/vmksummary.txt

Purpose

VMkernel Summary log, used to determine uptime and availability statistics for ESX hosts human-readable summary found in /var/log/vmksummary.txt.

Management

Analysis

5. hostd

Location

/var/log/vmware/hostd.log

Purpose

ESX Server host agent log, contains information on the agent that manages and configures the ESX host and its virtual machines.

Manageement

hostd.log is symbollically linked to the current real file. Do an 'ls -l hostd.log' to see the link, or just browse (with less|more|tail) hostd.log to open the real file.

Analysis

6. vpxa

Location

/var/log/vmware/vpxa.log

Purpose

Vpxa log, contains information on the agent that communicates with vCenter Server.

Management

vpxa.log is symbollically linked to the current real file. Do an 'ls -l vpxa.log' to see the link, or just browse (with less|more|tail) hostd.log to open the real file.

Analysis

7. HA

Location

/var/log/vmware/aam

Purpose

High Availability (HA) log, logs information related to the High Availability service. In ESX 3.0.x these logs were in /opt/LGTOaam512/ instead.

Management

TBD

Analysis

TBD

8. COS

Location

/var/log/messages

Purpose

Service console log, contains all general log messages used to troubleshoot virtual machines or an ESX host.

Management

TBD

Analysis

TBD

9. esxupdate

Location

/var/log/vmware/esxupdate.log

Purpose

ESX Update log, logs all updates done through the esxupdate tool.

Management

TBD

Analysis

TBD

10. boot

Location

/var/log/vmware/esxcfg-boot.log

Purpose

ESX Boot log, logs all ESX boot events.

Management

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.

Analysis

TBD

11. webAccess

Location

/var/log/vmware/webAccess

Purpose

Web Access log, records information on Web-based access to an ESX host.

Management

TBD

Analysis

TBD

12. firewall

Location

/var/log/vmware/esxcfg-firewall.log

Purpose

ESX Firewall log, logs all firewall rule events.

Management

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.

Analysis

TBD

13. secure

Location

/var/log/secure

Purpose

Authentication log, contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.

Management

Logs rotated by logrotate, see KB 3402740.

Analysis

TBD

Resources

This doc on the web @

Authors

Steve Chambers