VIOPS: Proven Practices for Deploying and Managing VMware : All Content - Logging

Setting up a Splunk Server to Monitor a VMware Environment

dconvery — Tue, 30 Jun 2009 22:43:32 GMT

Introduction

Gathering and maintaining log files is an important part of a server administrator's duties. Using a centralized logging server, such as a sylog server offers several benefits. The log files become useful for troubleshooting purposes, if needed. Also, keeping an unaltered set of logs in a different location can aid in forensic activities after an attack.

This document explains how to set up Splunk for monitoring a VMware Environment. This includes monitoring the ESX/ESXi Server logs, the vCenter Server Logs and some of the add-on services to vCenter. It also includes generic event logging for Windows and Linux guest operating systems.

Intended Audience

VMware Certified Professionals, System Management / Sysadmin / Operations

Outline

Requirements
Preparing the Splunk Server
Installing Splunk Server
Setting up ESX Servers and Linux VMs for Monitoring
Setting up ESXi Servers for Monitoring
Setting up vCenter Servers and Windows VMs for Monitoring
Adding Miscellaneous Log Files to Splunk for Monitoring

Resources

Author

David Convery, VMware vExpert 2009

Also check out http://www.dailyhypervisor.com/ |

Disclaimer

You use this proven practice at your discretion. VMware, http://www.dailyhypervisor.com and the author do not guarantee any results from the use of this proven practice. This proven practice is provided on an as-is basis and is for demonstration purposes only.

Community Help wanted - How to Find ESX host's CPU Usage and Memory Usage

virtualbox — Tue, 14 Apr 2009 12:03:19 GMT

Hi Steve,

Thanks for your co-operation. Now lemme draw a diagram so that you can understand my requirement.

When I am clicking on the ESX Host and on the summary page I can see at the Resource section that how much CPU Usage and Memory usages are at this moment. So now my requirement is I want to know which file in vm-support can tell me this.

I am quite sure that there must be some files to tell me this result. Now why I require this is I have written an application which can replay the whole vm-support and can give me the each and every result related to this particular Host. Basically I have replayed the VC in my own way using the VC. So using this application I want to visualize the Resource section of a ESX host (CPU Usage and Memory Usage).

Also I want to achieve the same thing for the VMs as well like the way it is today showing under VC -> select VM -> go to Sumarry Page -> See the Resource Section. I am uploading the screenshot.

How to arrange log analysis lists?

schambers — Sat, 11 Apr 2009 11:35:18 GMT

Goal: to group log messages (and monitoring alerts, like SNMP traps next) into groups so you can work out:

a) CRIT What do I really have to worry about - immediate action - like a full COS root filesystem, or something that how high impact/urgency

b) WARN What messages do I need to be aware of, like a path failure or disk capacity warnings, something with medium impact/urgency

c) WATCH What messages should I track for trending - like %RDY time for a guest?

d) IGNORE What messages don't I care about?

Is there a standard already for the colour scheme for these kind of message groups? I've chosen what makes sense to me... does anyone else have input for this grouping?

ESX Server 3 Log Map

schambers — Fri, 10 Apr 2009 22:17:34 GMT

Introduction

This document describes all of the logs available on ESX Server 3.

As well as providing the location, purpose and management details for each log, there is also a link for each log to another analysis document that has a list of messages that form red (critical), orange (warning) and black (trend) lists.

NOTE This doc is a work in progress - want to help complete it? PM/email schambers

Intended Audience

VMware Certified Professionals (VCPs) who are designing and implementing log management for VI3.

Outline

Introduction to ESX Server 3 Logs
vmkernel
vmkwarning
vmksummary
hostd
vpxa
HA
COS
esxupdate
boot
webAccess
firewall
secure

1. Introduction to ESX Server 3 Logs

I've grouped the ESX logs into three parts:

Hypervisor
Management
Console OS (COS)

2. vmkernel

Location

/var/log/vmkernel

Purpose

VMkernel log, records activities related to the virtual machines and ESX hosts.

Management

Logs rotated by logrotate, see KB 3402740.

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension

Analysis

ESX Server 3 vmkernel.log Lists

3. vmkwarning

Location

/var/log/vmkwarning

Purpose

VMkernel Warnings log, records activities with the virtual machines, a subset of the vmkernel log.

Management

Logs rotated by logrotate, see KB 3402740.

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension

Analysis

4. vmksummary

Location

/var/log/vmksummary

/var/log/vmksummary.txt

Purpose

VMkernel Summary log, used to determine uptime and availability statistics for ESX hosts human-readable summary found in /var/log/vmksummary.txt.

Management

Analysis

5. hostd

Location

/var/log/vmware/hostd.log

Purpose

ESX Server host agent log, contains information on the agent that manages and configures the ESX host and its virtual machines.

Manageement

hostd.log is symbollically linked to the current real file. Do an 'ls -l hostd.log' to see the link, or just browse (with less|more|tail) hostd.log to open the real file.

Analysis

6. vpxa

Location

/var/log/vmware/vpxa.log

Purpose

Vpxa log, contains information on the agent that communicates with vCenter Server.

Management

vpxa.log is symbollically linked to the current real file. Do an 'ls -l vpxa.log' to see the link, or just browse (with less|more|tail) hostd.log to open the real file.

Analysis

7. HA

Location

/var/log/vmware/aam

Purpose

High Availability (HA) log, logs information related to the High Availability service. In ESX 3.0.x these logs were in /opt/LGTOaam512/ instead.

Management

TBD

Analysis

TBD

8. COS

Location

/var/log/messages

Purpose

Service console log, contains all general log messages used to troubleshoot virtual machines or an ESX host.

Management

TBD

Analysis

TBD

9. esxupdate

Location

/var/log/vmware/esxupdate.log

Purpose

ESX Update log, logs all updates done through the esxupdate tool.

Management

TBD

Analysis

TBD

10. boot

Location

/var/log/vmware/esxcfg-boot.log

Purpose

ESX Boot log, logs all ESX boot events.

Management

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.

Analysis

TBD

11. webAccess

Location

/var/log/vmware/webAccess

Purpose

Web Access log, records information on Web-based access to an ESX host.

Management

TBD

Analysis

TBD

12. firewall

Location

/var/log/vmware/esxcfg-firewall.log

Purpose

ESX Firewall log, logs all firewall rule events.

Management

Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.

Analysis

TBD

13. secure

Location

/var/log/secure

Purpose

Authentication log, contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.

Management

Logs rotated by logrotate, see KB 3402740.

Analysis

TBD

Resources

This doc on the web @

Authors

Steve Chambers

ESX Server 3 vmkernel.log Lists

schambers — Sat, 11 Apr 2009 00:14:03 GMT

Introduction

This document provides the red (critical - immediate action), orange (warning - track this alert) and black (ignore but trend on this message) for the vmkernel.log of ESX Server 3.

Intended Audience

VMware Certified Professionals (VCPs) and systems management professionals when implementing log management for VI3.

Outline

Introduction to vmkernel.log
vmkernel.log Red List
vmkernel.log Orange List
vmkernel.log Black List

Introduction to vmkernel.log

All the hypervisor/vmkernel messages are posted in the Console OS log:

/var/log/vmkernel.log

Good ways to access this log:

Centralize it (doc TBD) via syslog, and us a tool like Splunk!
Use less|more to view it live
tail -f /var/log/vmkernel.log

vmkernel.log Red List

Messages on the Red List should be picked up by the log monitoring automation and alerted as a high priority outage for immediate investigation.

1. KB 1003615 - host attached to share storage failures

Message


Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)SCSI: 3753: AsyncIO timeout (5000); aborting cmd w/ sn 941463, handle 1472/0x40211a28
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)LinSCSI: 3616: Aborting cmds with world 1024, originHandle 0x40211a28, originSN 941463 from vmhba0:0:5
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.325 cpu3:1032)<6>qla24xx_abort_command(0): handle to abort=857
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)LinSCSI: 2604: Forcing host status from 2 to SCSI_HOST_OK
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)LinSCSI: 2606: Forcing device status from SDSTAT_GOOD to SDSTAT_BUSY
Dec  8 20:06:01 esx013 vmkernel: 29:16:44:10.326 cpu3:1032)SCSI: 3753: AsyncIO timeout (5000); aborting cmd w/ sn 1073299, handle 2415/0x4020c038

Impact

Storage event that has caused an outage to every host accessing the shared storage.

Action

Review the storage controller or processor logs on the array for any events or messages occurring around the date the instance occurred. There are number of reasons why this kind of outage occurs like, controller problem, failing hard drive, SAN Copy operation being initiated. Also review the switch logs for the same time frame to see if the switches played a factor in this outage.

vmkernel.log Orange List

Messages on the Orange List should be picked up by the log monitoring automation and alerted as a warning for review.

vmkernel.log Black List

Messages on the Black List should be collected by the log monitoring automation and trended for future analysis.

Resources

This doc on the web @ ESX Server 3 vmkernel.log Lists

Authors

Steve Chambers

Community help wanted - index and analyze VI3 logs

schambers — Fri, 10 Apr 2009 23:45:00 GMT

Hello all, Texiwill, Roger Lund and Simon Shelston, and a few others, have shown great interest in helping the community get a proper grip of log management for VI3 (and of course, re-using the framework for vSphere).

Important note: anyone can use VIOPS to write their own view of how logging should be done - there is only one simple rule on VIOPS and that is all content must provide actionable guidance. Anyone can write a doc on VIOPS, there are no committees. There is a lightweight approval process that means when you write a doc on here and click "Submit for approval", then someone like me will get in touch and help you finish the doc. It's a friendly place, get stuck in!

The idea for how to document log management on VIOPS is to blend experience of our vExperts and VMware staff to produce a definitive guide to log management, as quoted on the front page of the Logging community. We want to help everyone using VI3 to be able to :

1) Understand what logs exist, where they are, their purpose - consider this a mapping exercise

2) How to manage the logs - like rotation, centralization, things like that

3) How to analyze the logs - what to watch for, what's safe to ignore, tools to use.

This will be a high quality effort if the VIOPS community pitches in... here's how you can help:

If you've ever had an issue associated with a logfile entry, and that had a resolution, then post it here

An example might be a SCSI bad code in /var/log/vmkernel - what was the code, what was the problem, how was it resolved? Is there a kb article for it?

The goal is to start at the top with the two documents below, then create a document for each log and build a red list (critical, watch for an action this!), orange list (warning, certainly capture and trend this), and black list (ignore this, developer diarrhea).

So from the doc ESX Server 3 Log Map, there are links to documents that form the red/orange/black lists for each component/log.

I'm kicking off efforts inside VMware to collate internal information, but the community input based on their experience will make this effort special and unique.

Thanks for reading, I hope you will contribute to this effort and help build the only VI3 community help index out there

Cheers - Steve

PS. Complementary to this effort is how to centralize and analyze the logs with tools like Splunk. If you have your own methods - please share. We don't need comprehensive installation details, just a high level approach for us to say "Oh THAT'S how you do it!"...

VirtualCenter 2 Logs

schambers — Fri, 10 Apr 2009 23:16:29 GMT

Introduction

This document describes all of the logs available on VirtualCenter 2.

NOTE This doc is a work in progress - want to help complete it? PM/email schambers

Intended Audience

VMware Certified Professionals (VCPs) who are designing and implementing log management for VI3.

Outline

Introduction to VirtualCenter 2 Logs
VirtualCenter
Tomcat
Capacity Planner
Update Manager
Converter
License Manager

1. Introduction to VirtualCenter 2 Logs

There are logs for each of the major components of VI3 management. All of these components may run on the same virtual machine (or physical, if you are Old Skool), or may be distributed onto their own servers, virtual or otherwise

2. VirtualCenter

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs

Purpose

VirtualCenter process log files

Management

TBD

Analysis

Link TBD

3. Tomcat

Location

C:\Program Files\VMware\Infrastructure\VirtualCenter Server\tomcat\logs

Purpose

Tomcat log files, the process used for the web API.

Management

TBD

Analysis

TBD

4. Capacity Planner

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware Capacity Planner\Logs

Purpose

Capacity Planner log files

Management

TBD

Analysis

TBD

5. Update Manager

Location

C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Logs

Purpose

VMware Update Manager log files

Management

TBD

Analysis

TBD

6. Converter

Location

C:\Documents and Settings\All Users\Application Data\VMware\
VMware Converter Enterprise\Logs

Purpose

VMware Converter log files.

Manageement

TBD

Analysis

TBD

7. License Manager

Location

C:\Temp\imgrd.log

Purpose

License Manager log file.

Management

TBD

Analysis

TBD

Resources

This doc on the web @ VirtualCenter 2 Logs

Authors

Steve Chambers

@rogerlund Talkshoe podcast on logging

rlund — Wed, 01 Apr 2009 12:55:04 GMT

Thanks Steve.

I wanted to bring this up , as I do not read about it as much as some other topic's. I also believe that awareness is important, and the fact is not everyone monitor's the log files or any type of monitoring for their VMware environment.