VIOPS: Proven Practices for Deploying and Managing VMware: Message List - Blades vs. traditional Rack servers

Re: Blades vs. traditional Rack servers

ryancox — Fri, 12 Sep 2008 15:48:01 GMT

Elsewhere on the web, James Hamilton has a great post up on this topic:

http://perspectives.mvdirona.com/2008/09/11/WhyBladeServersArentTheAnswerToAllQu estions.aspx

-ryan

Re: Blades vs. traditional Rack servers

deepchand — Fri, 12 Sep 2008 08:34:04 GMT

I would say that c3000 chassis has curtain points to remember:

- If you are not going to a addition mezzanine card for network in our blade servers than you will be not able to use a Network level redundancy in your C3000. This is by design; because we also faced a similar at one of our customer site.
- If you are not planning to have more than 3 to 4 servers than it would be good for you to go with a DL servers.
- Even a startup process of your SAN switch, Network Switch is very important if you are using it with SAN storage.

Re: Blades vs. traditional Rack servers

virgilwashere — Wed, 10 Sep 2008 23:58:18 GMT

What we are really discussing here is scale out vs scale up.

Here is an IBM redbook that covers this very well. I didn't look very hard, but I'm sure HP, Sun, Dell etc all have similar documents.

http://www.redbooks.ibm.com/abstracts/redp3953.html

jlmedinan wrote:

German (or Germán):

Blades are a great choice for many things.... but not for Virtualization. First: Blades are no cheap!!!. More money for CPU's, RAM and others. Blades requires a large inversion in infrastructure (addecuate racks, cooling and heat dissipation). Second: Virtualization, specially VMware ESX, requires flexibility, versatility and brute power.

Blade components should not be any more or less expensive than their equivalent rack counterparts.

Other aspect, and not less important, is heat: If you plan to massively populate your datacenter with blades, prepare to redesing it!!!: Better cooling, Heat extraction..... Blades are like Hell's kitchen!!!

One of the biggest problems with blade deployments is that people replace 42RU of rack servers with 42RU of blade servers. Given that you are probably doubling your effective computing resources with the blades is there any wonder they generate more heat?

The only fair comparison is 16 blades vs 16 1RU servers, in the case of HP, or 14 for IBM etc. If you use bigger rack servers, compare against bigger blades. HP BL685, IBM LS42 etc.

If you want to eliminate cables, think in infiniband or datacenter Ethernet, but not in blades

I have a fully redundant configuration for a blade with 2 HBAs, 6 NICs, which implies 2 FC switches, 6 Ethernet switches => 2 FC fabrics, 3 ethernet zones/layer 2 switch domains.

Uplinks - 2 switches for each of the following

2x4G for FC Fabrics
1x1G for Ethernet zone 1 (SC and VMK)
2x1G for Ethernet zone 2 (real traffic)
2x1G for Ethernet zone 3 (real traffic)

+1 iLO

Gives me 15 cables into a chassis for FC/ethernet. Compared to ... 8 (9 w/ iLO) per server if they were rack mount (assuming you didn't need the bandwidth aggregation).

And then there's the power cabling. 6 power cables for a HP enclosure. 32 power cables for 16 1RU servers.

I think cabling plant savings alone can be enormous.

The 10G Ethernet version saves a few more cables too, and they are all fibre cables, so you get standardisation.

Infiniband in a blade environment works wonders too. 3 cables (1 iLO). But make sure your FC+Ethernet gateway is close enough :)

I prefer Big nodes (currently I'm using SUN x4600 with un 8 AMD quad core in a 4U form factor) and I obtain more power in a rack than if it was blade Populated. I can install up to 10 x4600 in each rack (a total of 80 CPU/ 320 cores) per 48U rack... HP do not recommend to install more than 2 chasis per rack (16 blades per chassis, 2 CPU per blade = 64 CPU = 256 cores)

"HP do not recommend" is based on most data centres still only using ambient air cooling. In a properly configured data centre (hot/cold aisles etc), you can 3 or 4 HP c7000 or IBM BC-H enclosures.

4 chassis x 16 blades x 2 cpu x 4 cores = 512 cores. This is a 15kW rack. But as I'm sure you are aware, CPU is not the limiting resource in a VMware environment, memory is.

128G x 16 blades = a lot of VMs.

I'm not saying blades are the only answer. But they are very viable.

A question back to you - how long does it take to evacuate one of your x4600s when you need to do ESX/hardware maintenance ?

Virgil

Re: Blades vs. traditional Rack servers

jlmedinan — Wed, 10 Sep 2008 22:39:26 GMT

German (or Germán):

Blades are a great choice for many things.... but not for Virtualization. First: Blades are no cheap!!!. More money for CPU's, RAM and others. Blades requires a large inversion in infrastructure (addecuate racks, cooling and heat dissipation). Second: Virtualization, specially VMware ESX, requires flexibility, versatility and brute power. Blades are compact and small... not flexibles. Blades are no flexibles: You can not load into a blade alll I/O that you can want, for exmaple; expansion is expesive and propietary (look the prices of a blade switch!!), and normally, at the end, you look exatly the same numbers of cables than traditional servers. Power: For years, I'm trying someboy convince me that a server that works at 15 or 20 celcius degrees up than other offers the same performance!!!.

Other aspect, and not less important, is heat: If you plan to massively populate your datacenter with blades, prepare to redesing it!!!: Better cooling, Heat extraction..... Blades are like Hell's kitchen!!!

If you want to eliminate cables, think in infiniband or datacenter Ethernet, but not in blades. Take in mind that the first "Blade Taliban", HP, has presented a "big node" Proliant DL785 G5, with up 8 AMD quad core....

I prefer Big nodes (currently I'm using SUN x4600 with un 8 AMD quad core in a 4U form factor) and I obtain more power in a rack than if it was blade Populated. I can install up to 10 x4600 in each rack (a total of 80 CPU/ 320 cores) per 48U rack... HP do not recommend to install more than 2 chasis per rack (16 blades per chassis, 2 CPU per blade = 64 CPU = 256 cores)

I hope it was helpfull...

Re: Blades vs. traditional Rack servers

virgilwashere — Wed, 10 Sep 2008 13:28:16 GMT

Hi guys,

Here is something else to consider.

Harpstein wrote:

After a lot of research, meetings, and arguments, we went with rack servers for our primary cluster. The reason for doing so was Jason eluded to with single points of failure. We were having a hard time convincing administration that virtualization was the route to follow for our server infrastructure. So a single failure ANYWHERE within the system was not acceptable. Having rack servers also gave us flexibility with peripherals such as NIC's and HBA's. We went with HP DL380 G5's. They have dual quad core Xeon's, so you do catch a break with VMWare's "odd" per socket licensing. We had 32 GB of RAM installed along with 2 additional 2 port NIC's for a total of 6 network links. We also used 2 single port HBA's (FC-2142SR). All this was put into an APC rack with redundant

So with IBM and HP you get the same spec hardware, in a chassis that eliminates a large number of cables, aside from power savings and rack space.

Dual quad core, up to 64Gb of RAM, 6 NICs and 2 HBAs - but yes it's a dual port HBA if you want that many NICs.

However,

With all this in place, we have eliminated all single points of failure out to the desktops, or heaven forbid, a bomb taking out the data center.

You need to stop thinking about the underlying server so much, and start with the *service* that is being offered.

Addressing single points of failure in the ESX hosts is a good thing. I don't mean to imply otherwise.

But you cannot get past the fact that the server itself (and the mainboard, as Jason said above), is a single point of failure. This is why VMware has HA and DRS features.

These are implemented at the cluster level, so your SPoF is now the cluster/farm. Even if you spread your cluster out over multiple blade enclosures, the cluster is still the SPoF.

Then onto datacentres - and bring on Site Recovery Manager.

So, you can do wonderful things at the infrastructure layer, but until you address the service that the VM is providing, and add redundancy/resiliancy there, you are just moving the problem.

Personally, I'm a blade advocate, and much prefer working in the infrastructure layer. Doing the cultural change thing to get an organisation to recognise services and not servers, it hard work.

Virgil

Re: Blades vs. traditional Rack servers

schambers — Wed, 10 Sep 2008 12:38:47 GMT

It shouldn't take long - start with this Proven Practice: Document Template

Cheers!

Steve

Re: Blades vs. traditional Rack servers

jasonwilley — Tue, 09 Sep 2008 21:33:46 GMT

Sounds like a good plan Steve, I'm just getting up to speed up how to do that.

Tim - if you wanted to work together on it I would be open to that, as I am new to the format here.

Re: Blades vs. traditional Rack servers

schambers — Tue, 09 Sep 2008 18:49:10 GMT

DUDES - write a proven practice!

Re: Blades vs. traditional Rack servers

Harpstein — Tue, 09 Sep 2008 18:47:36 GMT

LOL...not at all, Jason. I thought it was a very good perspective from the blade side of things. I just thought I would go over what we did from the rack mount side. I thought it was very unbiased and informative. We're actually thinking about doing a blade solution for our BCDR site.

Re: Blades vs. traditional Rack servers

jasonwilley — Tue, 09 Sep 2008 18:44:44 GMT

I hope I didn't come off as anti-blade. The blades are great if you have the budget flexibility and the operational readiness.