Proven Practice: VI3 Security Risk Assessment

Metadata

How to use this proven practice

This proven practice is in the form of guidance to follow in a procedural manner.

A VMware Certified Professional should work with their IT Security professionals to walk through each section, analyse their environment, and produce a report highlighting compliance and gaps.

Proven Practice

1 Virtual Infrastructure Risk Assessment Overview

1.1 Introduction

VMware® ESX Server installs directly on server hardware, or “bare metal”, and inserts a virtualisation layer between the hardware and the operating system. ESX Server partitions a physical server into multiple secure and portable virtual machines that can run side by side on the same physical server. Each virtual machine represents a complete system—with processors, memory, networking, storage and BIOS—so that Windows, Linux, Solaris and NetWare operating systems and software applications run in virtualised environment without any modification. The bare metal architecture gives ESX Server complete control over the server resources allocated to each virtual machine and provides for near native virtual machine performance and enterprise class scalability. Virtual machines have built in high availability, resource management and security features that can provide better service levels to software applications than static physical environments.

VMware ESX Server architecture

Source: (VMware) Server Configuration Guide

From a security perspective, VMware Infrastructure consists of several major components:

• Virtualisation layer, consisting of the VMkernel and the virtual machine monitor

• Virtual machines

• ESX Server Service Console

• ESX Server virtual networking layer

• Virtual storage

• VirtualCenter

For each of the components, risks have been assessed independently to allow a methodical approach to discerning risk and countermeasure.

Note: To assess overall relative risk the countermeasures should be read in conjunction with each other.

Risk assessments are addressed in sub-sections as per the components list above. Each sub-section provides an overview, risk assessment and includes any additional best practice configurations.

1.1.1 Definitions

The following definitions were used as reference points within the document.

Potential Impact

1.2 General Security Features

1.2.1 Product Overview – VMware ESX

VMware ESX 3 Server presents a generic x86 platform by virtualising four key hardware components: processor, memory, disk, and network. An operating system is then installed into this virtualised platform. The virtualisation layer, or VMkernel, is a kernel designed by VMware specifically to run virtual machines. It controls the hardware utilised by ESX Server hosts and schedules the allocation of hardware resources among the virtual machines. A Service Console using a modified version of Red Hat Enterprise 3 is accessible and provides a local management interface and API to the ESX kernel. Because the VMkernel is fully dedicated to supporting virtual machines and is not used for other purposes, the interface to the VMkernel is strictly limited to the API required to manage virtual machines. There are no public interfaces to the VMkernel, and it cannot execute arbitrary code.

The VMkernel alternates among all the virtual machines on the host in running the virtual machine instructions on the processor. Every time a virtual machine’s execution is stopped, a context switch occurs. During the context switch the processor register values are saved and the new context is loaded. When a given virtual machine’s turn comes around again, the corresponding register state is restored.

Each virtual machine has an associated virtual machine monitor (VMM). The VMM uses binary translation to modify the guest operating system kernel code so it can run in a less-privileged processor ring. This is analogous to what a Java virtual machine does using just-in-time translation. Additionally, the VMM virtualises a chip set for the guest operating system to run on. The device drivers in the guest cooperate with the VMM to access the devices in the virtual chip set. The VMM passes request to the VMkernel to complete the device virtualisation and support the requested operation.

The following outlines key security features of the product:

• Compatibility with SAN security practices. VMware Infrastructure enforces security policies with LUN zoning and LUN masking.

• Implementation of secure networking features. VLAN tagging enhances network security by tagging and filtering network traffic on VLANs, and Layer network security policies enforce security for virtual machines at the Ethernet layer.

• Integration with Microsoft® Active Directory. VMware Infrastructure bases access controls on existing Microsoft Active Directory authentication mechanisms.

• Custom roles and permissions. VMware Infrastructure enhances security and flexibility with user-defined roles which can be managed in a granular way.

• Resource pool access control and delegation. VMware Infrastructure secures resource allocation at different levels and VM management can be delegated accordingly.

• Audit trails.VMware Infrastructure maintains a record of significant configuration changes and the administrator who initiated each one. Reports for event tracking can be exported.

• Session management. VMware Infrastructure enables discovery and, if necessary, terminate VirtualCenter user sessions.

• Vulnerability Response. VMware has implemented internal processes to ensure VMware products meet highest standards for security. The VMware Security Response Policy (www.vmware.com/vmtn/technology/security/security_response.html) documents VMware’s commitments for resolving possible vulnerabilities in VMware products.

• Security Certification. VMware ESX Server 2.5.0 and VirtualCenter 1.2.0 have been validated under the U.S. Common Criteria Evaluation and Validation Scheme (CCEVS) process, achieving EAL2 certification. VMware ESX Server 3.0 and VirtualCenter 2.0 are currently being tested for certification at EAL4+.

The following table details predetermined TCP and UDP ports used for management access to a VirtualCenter management server, ESX Server host(s) and other network components of a virtual infrastructure built on VMware ESX.

In addition to the above, to ensure the protection of data transmitted to and from external network connections, ESX server uses one of the strongest block ciphers available – 256 bit AES block encryption. ESX Server also uses 1024 bit RSA for key exchange. These encryption algorithms protect the following connections:

• VI client connections to the VC Server and to ESX hosts via the Service Console.

• VI Web Access connections to ESX hosts via the Service Console.

• Service Console connections to virtual machines through the VMkernel.

• SSH connections to the ESX hosts via the Service Console.

2 ESX Server Service Console

2.1 Overview

Whether using the VI management client or the command line, all configuration tasks for ESX Server are performed through the Service Console, including configuring storage, controlling aspects of virtual machine behaviour, and setting up virtual switches or virtual networks. A person logged in to the Service Console with privileged permissions has the ability to modify, shut down, or even destroy virtual machines on that host. While VMware ESX Server management clients use authentication and encryption to prevent unauthorised access to the Service Console, other services might not offer the same protection. If attackers gain access to the Service Console, they are free to reconfigure many attributes of the ESX Server host. The Service Console is the point of control for ESX Server and safeguarding it from misuse is crucial.

The ESX Server 3.0 Service Console provides an execution environment to monitor and administer the entire ESX Server host. The Service Console operating system is a reduced version of Red Hat Enterprise Linux, Update 6. Much of the functionality not necessary for interacting with the ESX Server virtualisation layer has been removed, therefore not all vulnerabilities of this distribution apply to the Service Console. VMware monitors and tracks all known security exploits that apply to this particular reduced version and issues custom updates as and when needed.

2.2 Risk Assessment

*1. VirtualCenter has the added benefit that authorisation and authentication are performed via the standard central Active Directory service, instead of using special local accounts in the Service Console. In addition, roles and users are stored in a database, providing an easy way to view the current permissions as well as take a snapshot of them. VirtualCenter also keeps track of every task invoked through it, providing an automatic audit trail.

2.3 Additional Best Practice Configuration

The following best practice configurations will be adopted within the design for this project.

Maintain Thorough Logging

In addition to identifying system issues, logging allows tracking of any unusual activity that might be a precursor to an attack and also allows a post-mortem to be carried out on any compromised systems.

Log files can be accessed by navigating to the /var/log/ directory and provide an important tool for diagnosing security breaches as well as other system issues. They also provide key sources of audit information. In addition to storing log information in files on the local file system, you can send this log information to a remote system. The syslog program is typically used for computer system management and security auditing, and it can serve these purposes well for ESX Server hosts. You can select individual Service Console components for which you want the logs sent to a remote system.

The following tips provide best practices for logging:

• Ensure accurate time-keeping.

• Control growth of log files.

• Use remote syslog logging.

• Remote logging to a central host provides a way to greatly increase administration capabilities. By gathering log files onto a central host, you can easily monitor all hosts with a single tool as well as do aggregate analysis and searching to look for things like coordinated attacks on multiple hosts.

• Use local and remote sudo logging.

• If you have configured sudo to enable controlled execution of privileged commands, you can benefit from using syslog to audit use of these commands. The following instructions show how to log all privileged command executions using syslog. You can then benefit from the other syslog features such as remote logging and log file rotation.

• Configure sudo to use syslog to record all occurrences of its use.

• Add an entry to /etc/syslog.conf to send the logging information to a file and, optionally, to a remote host.

• In ESX Server 3, only SNMPv1 is supported, and only for queries.

Use a Directory Service for Authentication for access to the ESX Server Service Console

Advanced configuration and troubleshooting of an ESX Server host may require local privileged access to the Service Console. For this circumstance, it is recommended to set up individual host-localised user accounts and groups for selected administrators with overall responsibility for your virtual infrastructure. These accounts would correspond to real individuals and not be accounts shared by multiple people. More preferable, and as proposed for the project will be to configure the server to authenticate users via Active Directory thereby centralising accounts and allowing continuity of existing policies for password complexity, aging and reuse.

Note: Although you can create host local accounts on the Service Console that correspond to each global account, this presents the problem of having to manage user names and passwords in multiple places. It is recommended to use a directory service, such as NIS or LDAP, to define and authenticate users on the Service Console, so local user accounts do not have to be created.

Because Service Console authentication is Unix-based, it cannot use Active Directory to define user accounts. However, it can use Active Directory to authenticate users. Individual user accounts can be defined on the host, then use the local Active Directory domain to manage the passwords and account status.

Root User Logon

The root user of the Service Console has almost unlimited capabilities, and securing this account is important to secure the ESX Server host. By default, remote access via ssh is enabled, but not for the root account. Files can be copied remotely to and from the Service Console using an scp (secure cp) client, such as WinSCP.

Enabling remote root access is not recommended, because it opens the system to network based attack should someone obtain the root password. The recommended approach is to log in remotely using a regular user account, then use sudo to perform privileged commands.

Note: The sudo command enhances security because it grants root privileges only for select activities, in contrast with the su command, which grants root privileges for all activities. Using sudo also provides superior accountability because all sudo activities are logged, whereas if you use su, ESX Server only logs the fact that the user switched to root by way of su. The sudo command also provides a way for you to grant or revoke execution rights to commands on an as-needed basis.

Root access can be disallowed on the console of the ESX Server host. This approach forces anyone who wants to access the system to first log in using a regular user account, then use sudo or su to perform tasks. If disallowing root login on the console, a non-privileged account on the host should be created to enable logins. This should be a local account so that in case the network connection to the directory service is lost, access to the host is still possible. Access can be assured by defining a local password for this account, which will then override authentication via directory services. The net effect is that administrators can still access the system, but they never have to log in as root. Instead, they use sudo to perform particular tasks or su to perform arbitrary commands.

Because su is a powerful command, access will be limited to it. By default, only users that are members of the wheel group in the Service Console have permission to run su. If a user attempts to run su - to gain root privileges and that user is not a member of the wheel group, the su - attempt fails and the event is logged.

The following list of recommendations for using sudo for the project will be :

• Configure local and remote sudo logging.

• Create a special admins group and allow only members of that group to use sudo.

• Use sudo aliases to determine the authorisation scheme, then add and remove users in the alias definitions instead of in the commands specification.

• Be careful to permit only the minimum necessary operations to each user and alias. Permit very few users to run the su command, because su opens a shell that has full root privileges but is not auditable.

• Require users to enter their own passwords when performing operations. This is the default setting. Do not require the root password, because this presents a security risk, and do not disable password checking.

In sudo the authentication only persists for a brief period of time before sudo asks for a password again.

3 ESX Server Kernel (Virtualisation layer)

3.1 Overview

The virtualisation layer, or VMkernel, is a kernel designed by VMware from the ground up to run virtual machines. It controls the hardware utilised by ESX Server hosts and schedules the allocation of hardware resources among the virtual machines. Because the VMkernel is fully dedicated to supporting virtual machines and is not used for other purposes, the interface to the VMkernel is strictly limited to the API required to manage virtual machines.

3.2 Risk Assessment

*1. The RAM allocated to a virtual machine by the VMM is defined by the virtual machine’s BIOS settings. The memory is allocated by the VMkernel when it defines the resources to be used by the virtual machine. A guest operating system uses physical memory allocated to it by the VMkernel and defined in the virtual machine’s configuration file.

The operating system that executes within a virtual machine expects a zero-based physical address space, as provided by real hardware. The VMM gives each virtual machine the illusion that it is using such an address space, virtualising physical memory by adding an extra level of address translation. A machine address refers to actual hardware memory, while a physical address is a software abstraction used to provide the illusion of hardware memory to a virtual machine.

4 ESX Server Virtual Networking Layer

4.1 Overview

The virtual networking layer consists of the virtual network devices through which virtual machines and the Service Console interface with the rest of the network. ESX Server relies on the virtual networking layer to support communications between virtual machines and their users. In addition, ESX Server hosts use the virtual networking layer to communicate with iSCSI SANs, NAS storage, and so forth. The virtual networking layer includes virtual network adapters and the virtual switches.

VMware ESX Server virtual networking layer

Source: (VMware) Server Configuration Guide

VMware Infrastructure 3 provides virtual network adapters to guest operating systems that have these characteristics:

• They have their own MAC addresses and unicast/multicast/broadcast filters.

• They are strictly layered Ethernet adapter devices.

• They interact with the low-level VMkernel layer stack via a common API.

The ESX Server 3 networking stack uses a modular design for flexibility. A virtual switch is “built to order” at run time from a collection of small functional units, such as:

• The core layer forwarding engine

• VLAN tagging, stripping, and filtering units

• Virtual port capabilities specific to a particular adapter or a specific port on a virtual switch

• Level security, checksum, and segmentation offload units

When the virtual switch is built at run time, ESX Server loads only those components it needs. It installs and runs only what is actually needed to support the specific physical and virtual Ethernet adapter types used in the configuration.

The following diagram shows how various networks can be segregated within an ESX Server host.

Network Segregation

Source: (VMware) Server Configuration Guide

4.2 Risk Assessment

4.3 Additional Best Practice configuration

The following best practice configurations will be adopted within the design for this project.

Do Not Create a Default Port Group

During ESX Server installation, there is an option to create a default virtual machine port. However, this option creates a virtual machine port group on the same network interface as the Service Console. If this setting is left unchanged, it could allow virtual machines to detect sensitive and often unencrypted information. Since the Service Console should always be on a separate, private network, this option should never be used except in a test environment. This option will be disabled in the standard VMware ESX Server image for this project.

Use a Dedicated, Isolated Network for VMotion and iSCSI

Because VMotion information is not encrypted, the entire state of a virtual machine could potentially be snooped on the network used for VMotion. Therefore, it is critical that this network be isolated from any other use. To encrypt VMotion traffic, there is the option of using hardware-based SSL encryption. Encryption is not available for iSCSI disk I/O, so this network should be strictly controlled, too. In the case of the project design the network for VMotion will be dedicated, isolated and non-routable.

Do Not Use Promiscuous Mode on Network Interfaces

ESX Server has the ability to run virtual network adapters in promiscuous mode. Promiscuous mode may be enabled on virtual switches that are bound to a physical network adapter (vmnic) and virtual switches that do not bind to a physical network adapter (vmnet). When promiscuous mode is enabled for a vmnic switch, all virtual machines connected to the virtual switch have the potential of reading all packets sent across that network, from other virtual machines as well as any physical machines or other network devices. When promiscuous mode is enabled for a vmnet switch, all virtual machines connected to the vmnet switch have the potential of reading all packets across that network — that is, traffic among the virtual machines connected to that vmnet switch.

While promiscuous mode can be useful for tracking network activity, it is an insecure mode of operation because any adapter in promiscuous mode has access to the packets regardless of whether some of the packets should be received only by a particular network adapter. This means that an administrator or root user within a virtual machine can potentially view traffic destined for other guest operating systems. Promiscuous mode should only be used for security monitoring, debugging, or troubleshooting.

By default, promiscuous mode is set to Reject and will remain so in the standard VMware ESX server image for this project.

5 Virtual Machines

5.1 Overview

Virtual machines are the containers in which guest operating systems and their applications run. By design, all VMware virtual machines are isolated from one another. Virtual machine isolation is imperceptible to the guest operating system. Even a user with system administrator privileges or kernel system level access on a virtual machine’s guest operating system cannot breach this layer of isolation to access another virtual machine without privileges explicitly granted by the ESX Server system administrator.

This isolation enables multiple virtual machines to run securely while sharing hardware and ensures both their ability to access hardware and their uninterrupted performance. For example, if a guest operating system running in a virtual machine crashes, other virtual machines on the same ESX Server host continue to run. The guest operating system crash has no effect on:

• The ability of users to access the other virtual machines

• The ability of the running virtual machines to access the resources they need

• The performance of the other virtual machines

Each virtual machine is isolated from other virtual machines running on the same hardware. While virtual machines share physical resources such as CPU, memory, and I/O devices, a guest operating system in an individual virtual machine cannot detect any device other than the virtual devices made available to it.

Virtual Machine Resources

Source: (VMware) Server Configuration Guide

5.2 Risk Assessment

5.3 Additional Best Practice configuration

The following best practice configurations will be adopted within the Low Level design for this component.

Create template images

Only approved pre-hardened and patched server images will be available for deployment.

VM Configuration Options

Unneeded devices such as cd, floppy, usb drives will be disconnected by default on virtual machines and only approved staff will have rights to modify based upon role based permissions.

Copy and paste operations between the guest operating system and remote console will be disabled so that sensitive information can not be inadvertently copied over.

6 Virtual Storage

6.1 Overview

Virtual disk files are stored on high performance shared storage such as Fibre Channel or iSCSI SAN. VMFS is a cluster file system which enables multiple installations of ESX Server to have concurrent fast access to the same virtual machine storage. Since virtual machines are hardware independent and portable across servers, VMFS ensures that individual servers are not single points of failure and enables resource balancing across multiple servers.

Fibre Channel HBA consolidation allows the sharing of storage network components across many virtual machines while maintaining hardware fault tolerance.

Virtual Storage

Source: (VMware) vmware.com

6.2 Risk Assessment

7 VirtualCenter

7.1 Overview

VirtualCenter is composed of five main components:

• VirtualCenter Management Server is the central control node for configuring, provisioning and managing virtualised IT environments.

• VirtualCenter Database is used to store persistent information about the physical servers, resource pools and virtual machines managed by the VirtualCenter Management Server. The database resides on standard versions of Oracle, Microsoft® SQL Server, or Microsoft® MSDE.

• Virtual Infrastructure Client allows administrators and users to connect remotely to the VirtualCenter Management Server or individual ESX Servers from any Windows PC.

• VirtualCenter Agent connects VMware ESX Servers with the VirtualCenter Management Server.

• Virtual Infrastructure Web access allows virtual machine management and access to virtual machine graphical consoles without installing a client.

VMware VirtualCenter Overview

Source: (VMware) vmware.com

7.2 Risk Assessment

7.3 Additional Best Practice Configuration

The following best practice configurations will be adopted within the Low Level design for this component.

Role Based Administration

VirtualCenter has a advanced system of roles and permissions, to allow granular determination of authorisation for administrative and user tasks, based on user or group and inventory item, such as clusters, resource pools, and hosts. This system ensures that only the minimum necessary privileges are assigned to people in order to prevent unauthorised access or modification. Custom task based roles can also be defined to specifically tailor user access and functionality. The design will detail role based administration access to VirtualCenter deployments.

Limit Network Connectivity to VirtualCenter

The only network connection VirtualCenter requires is to the ESX Server Service Console and to a network on which instances of VI Client are running. Avoid putting the VirtualCenter server on any other network, such as your production or storage networks. Limiting network connectivity reduces the possible avenues of attack.

It is recommended to further protect the VirtualCenter server using a firewall. This firewall may sit between the clients and the VirtualCenter server, or both the VirtualCenter Server and the clients may sit behind the firewall, depending on the deployment. The main consideration is ensuring that a firewall is present at what is considered to be an entry point for the system as a whole.

Note: Networks configured with a VirtualCenter server can receive communications from several types of clients: the VI Client, VI Web Access, or third-party network management clients that use the SDK to interact with the host. During normal operation, VirtualCenter listens on designated ports for data from the hosts it is managing and from clients. VirtualCenter also assumes that the hosts it is managing listen for data from VirtualCenter on designated ports. If a firewall is present between any of these components, it must be ensured that the appropriate ports are open to support data transfer through the firewall.

Ensure Proper Security Measures Are Used when Configuring the Database for VirtualCenter

It is recommended to install the VirtualCenter database on a separate server and subject it to the same security measures as any production database. Permissions used for access should be configured to the database to the minimum necessary.

Enable Full and Secure Use of Certificate-based Encryption

All versions of VMware products, including all releases of VirtualCenter Server use X.509 certificates to encrypt session information sent over SSL (secure sockets layer protocol) connections between server and client components.

During the installation of VMware products, default, self-signed certificates are automatically generated. However, the default certificates generated by VirtualCenter up to and including version 2.0.1 Patch 1 are defective and should not be used.

Note: By contrast, the default certificates generated by ESX Server hosts are valid and can be used as-is. This requires that any VI Client that wishes to connect to ESX Server directly (that is, without going through VirtualCenter), must pre-trust the default certificates.

For environments that require strong security, VMware recommends that administrators replace all default self-signed certificates generated at installation time with legitimate certificates signed by their local root certificate authority or public, third-party certificates available from multiple public certificate authorities. Server-certificate verification on all VI Client installations and the VirtualCenter host should be enabled. This involves a modification to the Windows registry on all client hosts.

Note: VirtualCenter asks for root credentials when it first connects to an ESX Server host. The root password for that host is cached only long enough to enable VirtualCenter management functionality, and the communication channel to the host is encrypted. VirtualCenter then creates a user called vpxuser with a pseudo-randomly generated password and uses the vpxuser account for subsequent connections and management operations. The vpxuser account for each ESX Server host has a unique, 32 -character (2 56-bit) password that is generated from a cryptographically random string of data that is mapped to a set of legal password characters. Once generated, the password is encrypted using 102 4-bit RSA key encryption. The password is also stored encrypted on the host, as any local account password would be.

The vpxuser account is created for VirtualCenter management when a host is added to VirtualCenter and is used only to authenticate the connection between VirtualCenter and the ESX Server host. Entries corresponding to the account are added to /etc/passwd and /etc/shadow, but no process actually runs as vpxuser on ESX Server.

The vpxuser password is reset every time a host is added to VirtualCenter. If VirtualCenter is disconnected from a host, it tries to reconnect with the vpxuser and password that is stored encrypted in the VirtualCenter database. If that fails, the user is prompted to reenter the root password so the system can reset (that is, automatically generate a new password for the vpxuser account).

In the VirtualCenter code, database specific variable protection mechanisms, such as parameterised queries in SQL Server are used extensively, thereby greatly reducing the risk of any SQL injection attack. The VIM API, which is the main SDK library, allows for a mechanism to specify privileges necessary to invoke the API as part of the API definition. This ensures that security implications are taken into consideration from the beginning of writing a new API.

This concludes the VMware® Virtual Infrastructure3 Risk Assessment.