Introduction
This document is intended as a mini guide for deploying Red Hat Cluster suite on CentOS 5 with active-standby configuration of one shared IP address and one shared filesystem mount point. This is 1st revision and please excuse me for grammar and typos since English language is not my primary language. In the past I have deployed RHCS on physical node using Qlogic iSCSI HBA's for connection with the storage (7 node cluster) with Heartbeat HA solution for loadbalancing between in one ISP for mail service hosting. Cluster's capacity was 3 million mails in a hour with heavy workload on working hours.
Outline
Create new virtual machine and do a fresh install of CentOS on ESX 3.5 hardware node
Clone virtual machine to get required number of nodes (2 nodes will be covered in this document at the beggining and I will update it from time to time to add more nodes)
Storage setup
iSCSI initiator setup
RHCS setup on nodes
Author
Written by Milan Karalic VI:OPS profile
System administrator
More topics can be found here
Disclamer
You use this proven practice at your discretion. VMware and the author do not guarantee any results from the use of this proven practice. This proven practice is provided on an as-is basis and is for demonstration purposes only.
How to use this Practice
This is more like general guidelines for deploying virtual-to-virtual cluster on your VMware infrastructure. I will write dedicated document with detailed setup of HA solution and 7 node cluster when I get resources I need for this projec
Deploying Red Hat Cluster Suite on VMware
h2. Creating VM
I choose to create VM with 10 GB hard drive, 2 vCPU's,2 GB and 2 NIC interfaces of memory to simulate exact hardware node configuration but I advise you to use 1vCPU as Steven supposed in the comment below this document. Power up your VMware Infrastructure Client and logon to ESX server using administrator priviledges.
File -> New -> Virtual Machine
Select Typical Virtual Machine Configuration and press Next
Name your VM (I will name first machine gfsnode01) and press Next
Select your Datastore (If you are using Virtual Center with HA or DRS select datastore visible to all ESX nodes in your setup in order to have VMotion enabled properly)
Check Guest Operating System: radial button to Linux and from dropdown menu select Red Hat Enterprise 5 (32 or 64-bit depending on your installation, I'v used 32-bit ISO image) and click on Next
Select Number of virtual processors: 1
Specify the ammount of memory (2048 MB)
Select 2 NIC cards and assign them to proper network (I have created dedicated vSwitch for iSCSI communication with storage) and make sure that Connect on Power On is checked
Assign vmdk disk filesize (10 GB)
Finish
After the setup is finished connect the CD/DVD drive to VM and power it on (or connect iso file, depending on your installation source I prefer iso files located on my datastore). During the OS installation process be sure to select Development tools, Cluster and Cluster Server packages to be installed (when you are prompted for package installation select Customize Now and locate options mentioned above).
When you are finished with installing Guest OS on your VM, clone it and do the necessary customization (hostname and ip address changes eg. gfsnode02 as hostname) and power it on.
h2. Storage
As storage I have EMC CX-320 which iscsi setup is covered in excellent document located here by Cormac Hogan.
In this document I'm using iqn.2007-08.admin.rs:<hostname> as initiator name under Guest OS.
h2. iSCSI Initiator setup
Download the latest copy of open-iscsi initiator from http://www.open-iscsi.org and compile it on your Guest OS ( execute make && make install after you extract tarball on your system) on both nodes. This step could be done prior cloning VM but I prefer this way as it is more fun (at least for me).
Edit /etc/iscsi/initiatorname.iscsi with your favorite editor and change the InitiatorName line to have IQN you want for initiator (in my case it was
InitiatorName=iqn.2007-08.admin.rs:gfsnode01 on gfsnode01 and InitiatorName=iqn.2007-08.admin.rs:gfsnode02 on gfsnode02) and start the initiator by issuing command
service iscsi start.
After the initiator is started issue command iscsiadm -m discovery -t st -p ClariionIP to get target name you are allowed to connect (this part is covered in Clariion setup)
and after it execute iscsiadm -m node -p ClariionIP -T ClariionTarget --login to establish iSCSI session with your storage. If You did everything as it should be done you
will find new disk inserted (/dev/sdX) by issuing dmesg or fdisk -l.
Note that I'm using dedicated vSwitch for iSCSI connection and I got one interface on each node in the same subnet as my storage.
h2. RHCS setup
So we are here now, we got connection to shared storage on both nodes now we will create cluster between nodes. Edit your /etc/hosts file on both nodes and add line for every node you have in your cluster (in this case gfsnode01 and gfsnode02). Create file /etc/cluster/cluster.conf and edit is to suite your needs. At the moment in my lab I have configured one test resource for active-standby cluster with shared IP and shared filesystem failover. You can find my cluster.conf file attached to this document.
h6. Create filesystem on iSCSI attached disk
create physical volume using pvcreate command on one node: pvcreate /dev/sdX (where X is the device letter on your system)
create volume group using vgcreate again only on one node: vgcreate volume_name device (vgcreate vggfs /dev/sdX)
create logical volume with desired size using lvcreate: lvcreate -L sizeG volume_group_name (lvcreate -L 250G vggfs)
activate volume group (vgchange -ay volume_group_name)
run the mkfs.gfs to create GFS filesystem or mkfs.gfs2 for GFS2 FS
After you configured your cluster run necessary startup scripts:
/etc/init.d/cman start
/etc/init.d/clvmd start
/etc/init.d/rgmanager start
You can see your cluster status by issuing command clustat and you should get output like
Cluster Status for gfs_cluster @ Fri Dec 5 23:39:11 2008
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
gfsnode01 1 Online, Local, rgmanager
gfsnode02 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:test gfsnode01 started
Issue mount or df command to verify if you'r resource is mounted and ip addr to verify if your shared ip resource is configured. If you did everything right you got your first virtual-to-virtual RHCS deployed.
Outro
I'm almost secure that many of you will say that this is common practice but I just wanted to point out on general guidelines for setup of virtual-to-virtual RHCS. Since this is my first document I ever wrote suggestions and critics are welcome. If you have any questions do not hesitate to contact me via PM.
Steve Chambers says:
I updated the blog - check out the Applications zone home page... I also cross posted: http://communities.vmware.com/message/1117259
Can you post links to this article on RedHat sites?
Cheers
Milan Karalic says:
in response to: Steve Chambers
I have updated part for vCPU's and iSCSI part. Will need to go trough installation process to attach some screenshoots. Expect them on Monday
Codefire says:
How did you configure fencing for this?
Oh, I see now. You're using manual fencing.
Are you familiar or have you used any type of automated fencing options on this? I've heard there's a VMWare fencing agent but no details on it and I heard it doesn't support HA or DRS.
Milan Karalic says:
in response to: Codefire
For fencing I use simple perl script that is based on VMWare perl api to connect to virtual centar with credentials of fence user and do the power-cycle of virtual machine.
I'll have to update this page abit
Codefire says:
Please, if you could. It'd help us out a ton.
Thanks,
Milan Karalic says:
in response to: Codefire
#!/usr/bin/perl -w
#server=myvirtcenterbox.mycorp.com
#vmname=mybadvm
#username=root
#password=mypasswd
#operation=reset
use strict;
use warnings;
use FindBin;
use lib "$FindBin::Bin/../";
use VMware::VIM2Runtime;
use VMware::VILib;
use AppUtil::VMUtil;
$Util::script_version = "0.1";
sub fail
{
my($msg)=@_;
print $msg."\n";
exit 1;
}
sub get_options_stdin
{
my $opt;
my $line = 0;
my $name;
my $val;
my $in;
while( defined($in = <>) )
{
$_ = $in;
chomp;
strip leading and trailing whitespace
s/^\s*//;
s/\s*$//;
skip comments
next if /^#/;
$line+=1;
$opt=$_;
next unless $opt;
($name,$val)=split /\s=\s/, $opt;
if ( $name eq "" )
{
print STDERR "parse error: illegal name in option $line\n";
exit 2;
}
#DO NOTHING -- this field is used by fenced
elsif ($name eq "agent" )
{
}
elsif ($name eq "server" )
{
#hack to put stdin directly into the
#options var that the vitoolkit expects.
$Opts::options{server} = $val; } elsif ($name eq "vmname" ) { $Opts::options{vmname} = $val;
}
elsif ($name eq "username" )
{
$Opts::options{username} = $val; } elsif ($name eq "password" ) { $Opts::options{password} = $val;
}
elsif ($name eq "operation" )
{
$Opts::options{operation} = $val; } elsif ($name eq "alert" ) { $Opts::options{alert} = $val;
}
#Excess name/vals will fail
else
{
fail "parse error: unknown option \"$opt\"";
}
}
}
get_options_stdin();
$Opts::options = Opts::get_url();
Util::connect();
my $vm_name = Opts::get_option ('vmname');
my $host = Opts::get_option ('host');
my $email_alert = Opts::get_option ('alert');
my $vm_views = VMUtils::get_vms ('VirtualMachine', $vm_name, $host);
sub reset_vm {
foreach (@$vm_views) {
my $mor_host = $_->runtime->host;
my $hostname = Vim::get_view(mo_ref => $mor_host)->name;
eval {
$_->ResetVM();
Util::trace(0, "\nvirtual machine '" . $_->name . "' under host".
" $hostname reset successfully ");
};
if ($@) {
if (ref($@) eq 'SoapFault') {
Util::trace (0, "\nError in '" . $_->name . "' under host $hostname: ");
if (ref($@->detail) eq 'InvalidState') {
Util::trace(0,"Host is in maintenance mode");
}
elsif (ref($@->detail) eq 'InvalidPowerState')
elsif (ref($@->detail) eq 'NotSupported')
else
}
else
}
}
}
sub poweroff_vm {
foreach (@$vm_views) {
my $mor_host = $_->runtime->host;
my $hostname = Vim::get_view(mo_ref => $mor_host)->name;
eval {
$_->PowerOffVM();
Util::trace (0, "\nvirtual machine '" . $_->name .
"' under host $hostname powered off ");
};
if ($@) {
if (ref($@) eq 'SoapFault') {
Util::trace (0, "\nError in '" . $_->name . "' under host $hostname: ");
if (ref($@->detail) eq 'InvalidPowerState') {
Util::trace(0, "The attempted operation".
" cannot be performed in the current state" );
}
elsif (ref($@->detail) eq 'InvalidState')
elsif(ref($@->detail) eq 'NotSupported')
else
}
else
}
}
}
if($vm_views) {
my $op = Opts::get_option('operation');
if( $op eq "reset" ) {
reset_vm();
}
elsif( $op eq "poweroff" ) {
poweroff_vm();
}
}
Util::disconnect();
if (defined $email_alert) {
my $action = Opts::get_option('operation');
open (MAIL, "|/usr/sbin/sendmail -t ");
print MAIL "From: root\n";
print MAIL "To: $email_alert\n";
print MAIL "Subject: Node $vm_name has been fenced!\n\n";
print MAIL "This is an automated email sent to alert you\n";
print MAIL "that cluster node $vm_name has been fenced.\n";
print MAIL "Action taken: $action \n";
close (MAIL);
}
1;
and line in cluster.conf under fencing should state:
fencedevice agent="fence_vi3" server="myvirtcenterbox.mycorp.com" vmname="mybadvm" username="root" password="mypasswd" operation="reset" alert="me@mycorp.com"
Notice I'v left out < and /> because line is not showing up when I try to post it like that.
Codefire says:
Very cool, thanks.
Great proven practice, Milan! Some suggested changes / additions:
In the Creating VM section, although you choose a 2vCPU VM because that's what it was on the hardware config, we always suggest starting with 1vCPU and only increasing if required. Also, someone might try this on a smaller physical host of 2 pCPUs which would give awful performance because of scheduling both available pCPUs. 2 vCPU should only be used (a) if really needed, and (b) if the underlying platform can support it well (4+ pCPUs).
In the storage section, it was actually Cormac Hogan who wrote the iSCSI guide
I made the last edit, but if you look a the Author section, it was all Cormac...
Do you have any screenshots? Even the simplest screenshot livens up these docs
Could you add a tag of "proven_practice"
Thanks, Milan, this is great work! Next question is, how do we promote it? I think we should create a blog article in this Applications zone - then we need to post links to it from other sites like VMTN...
Cheers
Steve