This Question is Answered

1 "helpful" answer available (2 pts)
14 Replies Last post: Sep 12, 2008 8:48 AM by Ryan Cox  
Click to view German Lopez's profile   4 posts since
Sep 8, 2008

Sep 8, 2008 11:51 AM

Blades vs. traditional Rack servers

We are in the design stage of choosing hardware for our enviroment. I need some direction in regards to the footprint of server to use. We are evaluating the blade solution from HP (C3000) and the traditional rack server (DL380). Any input?

  4 posts since
Sep 8, 2008
1. Sep 8, 2008 6:01 PM in response to: German Lopez
Re: Blades vs. traditional Rack servers

When it comes to blades vs. traditional servers, it comes down to your internal readiness level and budget in my opinion

 

First off: blades require a larger upfront investment in general, as the chassis is generally the largest cost (depending on the modules you put in the chassis). They also usually require more buy-in and a higher level of participation from your SAN and network team than you typically would require in a typical physical server deployment.

 

One other thing that you should be aware of when looking at blades is that they can be made as fault tolerant as a traditional rack server. For example, in my environment we mandate that all servers must have more than one NIC (counting the onboards as one) and we must have two HBAs in every server. Well the HP blades typically only have 2-3 mezzanine slots per blade, depending on the form factor. So that means that if you have a need for additonal NICs and a SAN connection, you will need to rely on one dual HBA to provide this connectivity. This may or may not be an issue for your SAN team, but it's a good thing to know upfront. When I was discussing this with HP, they raised and excellent point - you will always have a single point of failure at some level, usually it ends up being the system board. The mean time between failure is extremely high on the mezzanine cards in use on the blades, so while you are extending out the single point of failure beyond the system board to the mezzanine cards, it is not significantly higher risk.

 

I also wouldn't deploy your whole environment in a single chassis...chassis failures are incredibly rare according to HP, but you would definitely hate to one of the few that have experienced one.

 

Now on the converse, once you have deployed them, and learned them - they are much more convenient and flexible that traditional rack servers. They save a great amount of power (50% in our environment), and they have some great advantages such as the Onboard Administrator that makes configuring your ILOs and network settings amazingly simple.

 

I think there is an increasing benefit to high density virtualization, so if you have the option you may want to consider a small number of larger servers as opposed to a large number of smaller servers. in addition to ease of management, there are now huge licensing benefits in the Windows environment. You can read more about this in my blog http://virtuallycrazy.blogspot.com/2008/07/high-density-virtualization.html

 

One other link : HP just announced a blade targeted around high density virtualization. http://h18004.www1.hp.com/products/servers/proliant-bl/c-class/495c-g5/compariso n.html

 

I hope this was helpful.

 

 

 

Click to view Steve Chambers's profile Admin's 213 posts since
May 31, 2008
2. Sep 9, 2008 6:33 AM in response to: Jason Willey
Re: Blades vs. traditional Rack servers

Very helpful!

Click to view Tim Harper's profile   26 posts since
Sep 3, 2008
3. Sep 9, 2008 10:33 AM in response to: German Lopez
Re: Blades vs. traditional Rack servers

I guess I'll represent the other side of the coin, lol.

 

After a lot of research, meetings, and arguments, we went with rack servers for our primary cluster. The reason for doing so was Jason eluded to with single points of failure. We were having a hard time convincing administration that virtualization was the route to follow for our server infrastructure. So a single failure ANYWHERE within the system was not acceptable. Having rack servers also gave us flexibility with peripherals such as NIC's and HBA's. We went with HP DL380 G5's. They have dual quad core Xeon's, so you do catch a break with VMWare's "odd" per socket licensing. We had 32 GB of RAM installed along with 2 additional 2 port NIC's for a total of 6 network links. We also used 2 single port HBA's (FC-2142SR). All this was put into an APC rack with redundant PDU's that are plugged into 3 phase / 5 wire curcuits that span 2 separate UPS's. We used a EMC CX3-40 SAN with redundant paths to the hosts, as well as redundant fabric switches and redundant processors. The servers, outfitted with full carepacks and licensed came in at roughly $12k per. We started with four in our primary cluster. The G5's are also HP's latest attempt at going green, so the processors are energy efficient and they have switching power supplies that have the ability to throttle themselves according to current demands in the server.

 

With all this in place, we have eliminated all single points of failure out to the desktops, or heaven forbid, a bomb taking out the data center.

 

Hope this gives you the other half of the scenerio!

 

Thanks,

Tim

Click to view Steve Chambers's profile Admin's 213 posts since
May 31, 2008
4. Sep 9, 2008 11:42 AM in response to: Tim Harper
Re: Blades vs. traditional Rack servers

Tim, can you capture your approach into a proven practice - it's your way of achieving a goal....

  4 posts since
Sep 8, 2008
5. Sep 9, 2008 11:44 AM in response to: Tim Harper
Re: Blades vs. traditional Rack servers

I hope I didn't come off as anti-blade. The blades are great if you have the budget flexibility and the operational readiness.

Click to view Tim Harper's profile   26 posts since
Sep 3, 2008
6. Sep 9, 2008 11:47 AM in response to: Jason Willey
Re: Blades vs. traditional Rack servers

LOL...not at all, Jason. I thought it was a very good perspective from the blade side of things. I just thought I would go over what we did from the rack mount side. I thought it was very unbiased and informative. We're actually thinking about doing a blade solution for our BCDR site.

Click to view Steve Chambers's profile Admin's 213 posts since
May 31, 2008
7. Sep 9, 2008 11:49 AM in response to: Tim Harper
Re: Blades vs. traditional Rack servers

DUDES - write a proven practice!

 

 

  4 posts since
Sep 8, 2008
8. Sep 9, 2008 2:33 PM in response to: Steve Chambers
Re: Blades vs. traditional Rack servers

Sounds like a good plan Steve, I'm just getting up to speed up how to do that.

 

Tim - if you wanted to work together on it I would be open to that, as I am new to the format here.

Click to view Steve Chambers's profile Admin's 213 posts since
May 31, 2008
9. Sep 10, 2008 5:38 AM in response to: Jason Willey
Re: Blades vs. traditional Rack servers

It shouldn't take long - start with this Proven Practice: Document Template

 

Cheers!

Steve

Click to view Virgil's profile   11 posts since
Sep 10, 2008
10. Sep 10, 2008 6:28 AM in response to: Tim Harper
Re: Blades vs. traditional Rack servers

Hi guys,

 

Here is something else to consider.

Harpstein wrote:

After a lot of research, meetings, and arguments, we went with rack servers for our primary cluster. The reason for doing so was Jason eluded to with single points of failure. We were having a hard time convincing administration that virtualization was the route to follow for our server infrastructure. So a single failure ANYWHERE within the system was not acceptable. Having rack servers also gave us flexibility with peripherals such as NIC's and HBA's. We went with HP DL380 G5's. They have dual quad core Xeon's, so you do catch a break with VMWare's "odd" per socket licensing. We had 32 GB of RAM installed along with 2 additional 2 port NIC's for a total of 6 network links. We also used 2 single port HBA's (FC-2142SR). All this was put into an APC rack with redundant

 

So with IBM and HP you get the same spec hardware, in a chassis that eliminates a large number of cables, aside from power savings and rack space.

Dual quad core, up to 64Gb of RAM, 6 NICs and 2 HBAs - but yes it's a dual port HBA if you want that many NICs.

 

However,

 

With all this in place, we have eliminated all single points of failure out to the desktops, or heaven forbid, a bomb taking out the data center.

You need to stop thinking about the underlying server so much, and start with the *service* that is being offered.

Addressing single points of failure in the ESX hosts is a good thing. I don't mean to imply otherwise.

But you cannot get past the fact that the server itself (and the mainboard, as Jason said above), is a single point of failure. This is why VMware has HA and DRS features.

 

These are implemented at the cluster level, so your SPoF is now the cluster/farm. Even if you spread your cluster out over multiple blade enclosures, the cluster is still the SPoF.

 

Then onto datacentres - and bring on Site Recovery Manager.

 

So, you can do wonderful things at the infrastructure layer, but until you address the service that the VM is providing, and add redundancy/resiliancy there, you are just moving the problem.

 

Personally, I'm a blade advocate, and much prefer working in the infrastructure layer. Doing the cultural change thing to get an organisation to recognise services and not servers, it hard work.

 

Virgil

Click to view José Luis Medina's profile   4 posts since
Sep 10, 2008
11. Sep 10, 2008 3:39 PM in response to: German Lopez
Re: Blades vs. traditional Rack servers

German (or Germán):

 

Blades are a great choice for many things.... but not for Virtualization. First: Blades are no cheap!!!. More money for CPU's, RAM and others. Blades requires a large inversion in infrastructure (addecuate racks, cooling and heat dissipation). Second: Virtualization, specially VMware ESX, requires flexibility, versatility and brute power. Blades are compact and small... not flexibles. Blades are no flexibles: You can not load into a blade alll I/O that you can want, for exmaple; expansion is expesive and propietary (look the prices of a blade switch!!), and normally, at the end, you look exatly the same numbers of cables than traditional servers. Power: For years, I'm trying someboy convince me that a server that works at 15 or 20 celcius degrees up than other offers the same performance!!!.

 

Other aspect, and not less important, is heat: If you plan to massively populate your datacenter with blades, prepare to redesing it!!!: Better cooling, Heat extraction..... Blades are like Hell's kitchen!!!

 

If you want to eliminate cables, think in infiniband or datacenter Ethernet, but not in blades. Take in mind that the first "Blade Taliban", HP, has presented a "big node" Proliant DL785 G5, with up 8 AMD quad core....

 

I prefer Big nodes (currently I'm using SUN x4600 with un 8 AMD quad core in a 4U form factor) and I obtain more power in a rack than if it was blade Populated. I can install up to 10 x4600 in each rack (a total of 80 CPU/ 320 cores) per 48U rack... HP do not recommend to install more than 2 chasis per rack (16 blades per chassis, 2 CPU per blade = 64 CPU = 256 cores)

 

I hope it was helpfull...

Click to view Virgil's profile   11 posts since
Sep 10, 2008
12. Sep 10, 2008 4:58 PM in response to: José Luis Medina
Re: Blades vs. traditional Rack servers

What we are really discussing here is scale out vs scale up.

 

 

 

Here is an IBM redbook that covers this very well. I didn't look very hard, but I'm sure HP, Sun, Dell etc all have similar documents.

 

http://www.redbooks.ibm.com/abstracts/redp3953.html

 

 

 

jlmedinan wrote:

 

German (or Germán):

 

Blades are a great choice for many things.... but not for Virtualization. First: Blades are no cheap!!!. More money for CPU's, RAM and others. Blades requires a large inversion in infrastructure (addecuate racks, cooling and heat dissipation). Second: Virtualization, specially VMware ESX, requires flexibility, versatility and brute power.

 

 

Blade components should not be any more or less expensive than their equivalent rack counterparts.

 

Other aspect, and not less important, is heat: If you plan to massively populate your datacenter with blades, prepare to redesing it!!!: Better cooling, Heat extraction..... Blades are like Hell's kitchen!!!

 

 

One of the biggest problems with blade deployments is that people replace 42RU of rack servers with 42RU of blade servers. Given that you are probably doubling your effective computing resources with the blades is there any wonder they generate more heat?

 

The only fair comparison is 16 blades vs 16 1RU servers, in the case of HP, or 14 for IBM etc. If you use bigger rack servers, compare against bigger blades. HP BL685, IBM LS42 etc.

 

If you want to eliminate cables, think in infiniband or datacenter Ethernet, but not in blades

 

I have a fully redundant configuration for a blade with 2 HBAs, 6 NICs, which implies 2 FC switches, 6 Ethernet switches => 2 FC fabrics, 3 ethernet zones/layer 2 switch domains.

Uplinks - 2 switches for each of the following

  • 2x4G for FC Fabrics
  • 1x1G for Ethernet zone 1 (SC and VMK)
  • 2x1G for Ethernet zone 2 (real traffic)
  • 2x1G for Ethernet zone 3 (real traffic)

+1 iLO

Gives me 15 cables into a chassis for FC/ethernet. Compared to ... 8 (9 w/ iLO) per server if they were rack mount (assuming you didn't need the bandwidth aggregation).

 

And then there's the power cabling. 6 power cables for a HP enclosure. 32 power cables for 16 1RU servers.

 

I think cabling plant savings alone can be enormous.

 

The 10G Ethernet version saves a few more cables too, and they are all fibre cables, so you get standardisation.

 

Infiniband in a blade environment works wonders too. 3 cables (1 iLO). But make sure your FC+Ethernet gateway is close enough :)

 

I prefer Big nodes (currently I'm using SUN x4600 with un 8 AMD quad core in a 4U form factor) and I obtain more power in a rack than if it was blade Populated. I can install up to 10 x4600 in each rack (a total of 80 CPU/ 320 cores) per 48U rack... HP do not recommend to install more than 2 chasis per rack (16 blades per chassis, 2 CPU per blade = 64 CPU = 256 cores)

"HP do not recommend" is based on most data centres still only using ambient air cooling. In a properly configured data centre (hot/cold aisles etc), you can 3 or 4 HP c7000 or IBM BC-H enclosures.

 

4 chassis x 16 blades x 2 cpu x 4 cores = 512 cores. This is a 15kW rack. But as I'm sure you are aware, CPU is not the limiting resource in a VMware environment, memory is.

128G x 16 blades = a lot of VMs.

 

I'm not saying blades are the only answer. But they are very viable.

 

A question back to you - how long does it take to evacuate one of your x4600s when you need to do ESX/hardware maintenance ?

 

 

Virgil

Click to view Deep Chand's profile   5 posts since
Sep 12, 2008
13. Sep 12, 2008 1:34 AM in response to: Virgil
Re: Blades vs. traditional Rack servers

I would say that c3000 chassis has curtain points to remember:

 

  • - If you are not going to a addition mezzanine card for network in our blade servers than you will be not able to use a Network level redundancy in your C3000. This is by design; because we also faced a similar at one of our customer site.
  • - If you are not planning to have more than 3 to 4 servers than it would be good for you to go with a DL servers.
  • - Even a startup process of your SAN switch, Network Switch is very important if you are using it with SAN storage.
Click to view Ryan Cox's profile   1 posts since
Sep 12, 2008
14. Sep 12, 2008 8:48 AM in response to: German Lopez
Re: Blades vs. traditional Rack servers

Elsewhere on the web, James Hamilton has a great post up on this topic:

 

http://perspectives.mvdirona.com/2008/09/11/WhyBladeServersArentTheAnswerToAllQu estions.aspx

 

-ryan

More Like This

  • Retrieving data ...