You have no budget. The higher-ups don’t seem to understand the need to go virtual. Can you really get there and is it worth the battle?

Back in May of 2012 I wrote on article about VMware Essentials Plus and my take on wether it was worth the money or not. A little over two years later I thought I would give an update on our setup as well as talk about if it is really worth it. I have seen a few people reference the previous article and saying it sounds great but they just don’t have budget – and of course other people saying if you don’t buy the really expensive stuff your dumb and don’t know a thing about virtualizing.

Where we are now

Back in 2012 we were looking at Essentials Plus for 4 major features. I’m going to do a quick recap of how each feature has worked out for us over the past 2 years.

Data Recovery

VMwares built-in VDR (renamed to VDP I believe) ended up just not working out for us. When it worked, it worked. But probably about once every 2-3 months something would go horribly wrong and corrupt the database. After that the only solution, and VMware’s own official answer, was to wipe the backup store and start over. After about a year of that I got fed up and switched to Veeam. They also have a what is roughly equivalent to an “Essentials Plus” package that is pretty cheap. I forget the up front cost but it is for up to three physical hosts again and is right around $350/yr SnS. I set it up once and haven’t touched it sense. And every time it runs I get really pretty reports e-mailed to me so I know it works.

vMotion

This has been the best bang for the buck. I can’t count the number of times I have been able to, in the middle of the day, shut down a host for maintenance (replace the host, upgrade RAM, etc) and nobody on staff even knew I did anything. Being able to migrate live running virtual machines from one physical host to another, I simply cannot describe how that has improved my ability to do work. No more coming in on holidays so I can shut down 10 virtual servers for 2 hours so I can get the maintenance done. If this was the only thing that Essentials Plus offered, it would still be worth it.

High Availability

This is nice. We have only used it once, or rather we have only needed it to be used once. And it worked perfectly. While doing some work on an unrelated server I accidentally unplugged one of the VM hosts. And worse, I didn’t realize it for a few minutes. After about 2 minutes I got an alert on my cell saying a bunch of servers were offline, “uh oh”. About a minute later, before I even had a chance to get logged into VMware console I got another alert saying they were back online. The HA service did what it was supposed to and fired the virtual machines back up before I could even do it myself. The users noticed about a 3 minute down time on the servers (one of them was the mail server, the rest they probably never noticed being rebooted) but never complained because it was back up before they thought to call me.

Web Client

I said in my previous article that Web Client was completely useless to me because it did not support console access on Mac. vSphere 5.5 added an HTML5 console feature. It works about 95% of the time. The 5% it doesn’t work is because of mouse issues. I don’t know if that is Firefox (doesn’t work in Safari last time I tried) issue or a vSphere issue, but randomly the mouse will vanish completely from the system (not just the web browser window). I have to switch applications out of Firefox and then back in to get it back. Other than that it works great. I can now do a complete install and maintenance of a virtual machine from Firefox on my Mac, never having to use the vSphere Windows Client, which is even deprecated now.

Our setup

Here is our current setup now:

• 3 Physical Hosts, each host has:
  • 32GB RAM (soon to be upgraded to 64GB)
  • 2 data NICs (i.e. normal traffic)
  • 2 iSCSI NICs (for shared storage)
  • Single 8-core HT CPU
  • Single drive for boot image (no RAID)
• 1 iSCSI storage with 4 NICs
• Two 24-port switches connected together in a stack configuration
  • Each host has one data NIC and one iSCSI NIC connected to each switch
  • The iSCSI storage has 2 iSCSI NICs connected to each switch.
  • This means we can lose a NIC or even an entire switch and still keep running.

We didn’t buy all this at once. We did it in stages. We started with one host and local storage. Then we went to two hosts and local storage. Then we added an iSCSI shared storage with manual load balancing (i.e. remove the VM from one host and add it to the other host).

Next we added a second switch for redundancy (as well as extra capacity). Finally we bought Essentials Plus and a third host to automate everything.

Let’s talk money

One of the big things people run into is the cost of getting virtualized. There is a fairly high cost associated with virtualizing, but it is an up front cost primarily, not an ongoing cost. Assuming you are a company that is not financially mission critical you can minimize these costs quite a bit. What I mean by this is if you are a web hosting company and you get paid to make sure people’s servers are running, you don’t want to cut any corners, you want to get the more expensive stuff that is going to work (redundant power, redundant network, etc.). So lets talk cost and stages.

Stage 1 – Single virtual host (entry level) – $1,700

Dell R320, Xeon-E5-2407, 16GB RAM, 2 * 500GB HDD, PERC H310 RAID controller – $1,700

This system will give you the ability to virtualize I would say probably 8-12 small VMs. By small I mean amount of RAM needed. If you are virtualizing a SQL database server, you are going to need to beef up the RAM. Most of my VMs do fine with 2GB of RAM, some at 1GB. CPU is really not going to be an issue, we run about 8 VMs per host currently and they average about 12% CPU usage. This is a slower chip than what we bought but should be fine for entry level. Guess what, this is all you need to get yourself virtualized.

This setup will give you 2 NICs (build-in) for load-balancing and failover of your VMs in terms of networking. The 2 500GB HDDs should be plenty of storage in a mirror configuration, so your data is protected from a hard drive failure (I recommend setting up a 20GB logical drive for ESXi and the rest for your data store). If your long term goal is to really go fully virtualized I recommend a few additions to your computer:

• Bump the CPU to the E5-2420 v2. It pushes you up to 6 cores. Add $150
• iDRAC Enterprise (remote console access). Add $290
• Dual Hot-Plug Redundant power supply (350W). Add $230
• ReadyRails with Cable Management. Add $70
• Extra NICs for iSCSI later (Broadcom 5720 DP 1Gb). Add $100

Buy what you think you actually need. The iDRAC provides remote console access via the web browser (java applet actually, but it works on Mac and Windows) as well as lots of monitoring information about your server. I bought it because my servers are in another room on the other side of the building, and soon will be in another building across campus. I don’t want to have to spend 5 minutes walking to get to the server console to hit CTRL-ALT-DEL. If your servers are next to your office or, for some of you, in your office, the iDRAC is probably not going to be helpful. The ReadyRails are nice because they let you slide the server out while it’s still running and wired, again really not necessary but it is convenient. Dual power supplies provide redundancy AND power statistics in VMware. Don’t ask me why, but if you don’t buy the hot-plug redundant power supplies then they don’t provide power usage information (i.e. how many watts the server is using). VMware will actually use this information to tell you how much power (in watts) each VM is using individually.

All four of the above items are optional but helpful. The iDRAC and ReadyRails you can buy and install later. The CPU and power supplies you get what you get. Technically you can replace the CPU, but it totally isn’t worth it financially. If you can swing it, I recommend the CPU and power supplies.

Stage 2 – Second virtual host – $1,700

Buy another of the above. If possible get the same configuration. If Dell (or whomever you buy from) has upgraded the system, try to match it as much as possible. It just makes things easier in the long wrong for management.

With your second host, you get expanded capacity and that is it. In most cases, you can manage to get stage 1 and 2 done at the same time. You already have servers running. Even if they are a bit older but still okay hardware for virtualization, go ahead and start there. Use one old server and buy one new server. Run the critical stuff on the new server and the less critical stuff, testing etc., on the older server.

Stage 3 – Shared storage – $1,600

Now that you have two hosts, you would like to be able to use shared storage so you can move VMs from one host to another. QNAP makes great stuff and lets you get in super cheap. If you are going 16-bay I would suggest looking at Promise Technologies instead. Their stuff at that size is not much more expensive than QNAP but gives you much better performance.

• QNAP TS-469U-SP – $1,200
• Hitachi UA722010CLA330 1TB * 4 – $95/ea

Put those drives in the QNAP, set them up RAID 5 and create an iSCSI share. That will give you 3TB of storage. Guess what, I got about 24 VMs running and they are only taking up 1.5TB of space. 3TB will last you a long time.

Now to keep costs down, we are not going to buy any extra NICs. We are going to drop the redundancy out of our hosts and make one NIC for VM traffic and one NIC for iSCSI. Or, if you were planning ahead and bought the Broadcom 5720 dual port NIC for your servers @ $100, you are totally set. Use the on-board for VM traffic and the two Broadcom ports for iSCSI.

What does this give you? You now have two VM hosts using shared storage. You can cold-migrate your VMs between the two hosts. To do that you simply shutdown the VM, remove it from Inventory (do not delete from disk) and then on the other host add it in and start it up. Boom your done. Not as fancy as a hot-migration, but if you need to do maintenance on a host your servers are now only down for a few minutes rather than a few hours.

Stage 4 – VMware Essentials Plus – $5,000

Yes, this is the expensive one. If you are a non-profit (e.g. church or other charity) you should be able to get it for about $3,000. Annual SnS is required for first year (included in above prices) and optional after that, but it is honestly worth getting. Regular price of SnS is about $1,000/yr or about $550 for non-profits.

Here is the catch. vSphere requires it be run on it’s own server, it can’t be virtualized on one of the servers you are managing. You know those old servers you have lying around because you virtualized them? Snag one and install vSphere on it. In my case I actually installed the stand-alone ESXi on it and installed the vSphere appliance on top of that. I did this because I use that host to virtualize one or two other “beta test” applications where I don’t want to even touch the cluster.

This will probably be the hardest sell for you. “Why do we need to spend $5,000? We are already virtualized.” Go back to the top of this article for some good reasons, as well as the previous article. But what it comes down to is it makes your life easier. You can do maintenance on servers without staying late or disrupting everybody’s work. The biggest sell I could make is that the $5,000 is a one-time fee and then it is only $1,000 (or less) each year after and that annual money gives you all the upgrades and phone support if you need it.

Stage 5 – Third virtual host – $1,700

Your copy of VMware Essentials Plus allows you to manage up to 3 physical hosts. You might as well get that third host at some point. This will give you full fault tolerance because let’s face it, you probably are running those 2 physical hosts at more than 40% capacity in terms of RAM. So if one fails (or is down for maintenance) you are suddenly over-committed on RAM and everything is going to slow down.

Summary

Let’s say you do all these stages one a year. At the end of 5 years you will have spent $12,700 (for those of you doing the math, I added $1,000 for the fifth year to cover the SnS fee). That is the equivalent of $2,540 per year. I would have a hard time selling $12,700 to my boss. Even now that I can point to all the things it has improved. That is a large chunk of money. I can sell $2,000 per year and one big year of $5,000. Oh, and here is a tip. If you are having trouble with that $5,000 number try a multi-year technique. Ask if you can budget $2,500 for the next two years and roll the money from one into the next. We are not allowed to do that normally, but for special cases we can. For example, say in 2014 and 2015 you budget $2,500 each year. You request and get permission to roll that $2,500 into January of 2015 and spend the $5,000 that month. Finance people respond far more positively to a very specific, very tight timeframe than “I want to be able to roll money from one year into the next”.

Savings

So far, we have just talked about costs. Now lets talk about savings. I can only cover a few savings costs because most of the savings will be in your time. How much time do you spend managing multiple physical servers? How much time do you spend coming in late so you can stay after hours to do maintenance?

Electrical savings

First, head on over to anandtech and take a look at their article on the cost of running a computer 24/7, it’s a pretty good read. Now I don’t have exact numbers, but I believe we pay somewhere around 15 cents per kilowatt hour on average (remember, these things run 24/7 so they are running off-peak and on-peak). My high-end, energy efficient servers run at 130 watts. If you are using older, non-energy efficient servers they probably use more. If you don’t have one, go buy a kill-o-watt and measure a few of your servers. The kill-o-watt should have a mode that tells you how many kWh (killowatt hours) the device has used. Run it for a 24-hour period. That will tell you how many kWh get used per day.

Here is a quick and dirty primer on these calculations. My servers tell me how many watts they are using, 130 watts. To calculate how many kWh per day, I take that number, divide by 1,000 (kilo) and then multiply by 24. This gives me 3.12 kWh. Now, we pay 15 cents per kWh, so I multiply that by 0.15 and get $0.47 (rounded up). So it costs me 50 cents per day to run one server. Not very much. But, I have 24 virtualized servers. I would say about 18 of them were at one point physical or would have been physical machines (we used to have a lot of servers). That can add up fast.

But lets be more practical for a small business. Lets say you have 8 servers or will have 8 servers in the next few years due to expansion. With only 8 servers you are fine with two hosts, so first we need to subtract out the 2 VM hosts and the 1 vSphere server. That gives us 5 eliminated physical servers. Let’s also assume your existing servers are older and not power efficient, so they use 200 watts (and it may very well be more) each. That means each server costs 72 cents per day to run, or $262 per year in electrical costs alone. You have eliminated 5 physical servers (most of which were immediate) over the past 5 years of this process. That is a savings of $1,314 per year in electrical costs. Even if you subtract from that savings the cost of the new iSCSI storage, probably also around 150 watts, you are still saving around $1,000 per year.

Hardware savings

So over a 5 year window you save the company $6,570. Subtract that from your overall cost leaves you with a net cost of $6,130. This savings was extremely useful when we needed to pull the trigger on Essentials Plus. What was also useful was this simple argument: On average, over the past few years we have added 2 (virtual) servers a year to handle various tasks. Time cards, facility management, website, etc. The cost to add these servers in a non-virtualized environment would be about $150 per year in electrical and the $1,200 for a super-cheap rack-mount server. All that, times two. And, these electrical cost are cumulative, but lets forget the electrical and just talk hardware. Let’s also say you only buy one new server per year. Over the next 5 years (from when you start your virtualization project) you would have spent $6,000, which means that is a savings towards your virtualization cost.

Now we subtract that from our remaining cost and we are left with $570 in real costs over a 5 year period. Think your CFO will go for you saying the total cost over 5 years is $114 per year? I think they would.

Time savings

Make sure to write up your estimates of how much time you spend managing X number of physical machines. We had around a dozen when we started. It takes a fair amount of time to manage and monitor that many physical boxes. To say nothing of how much time it takes to rewire everything when you run out of room and need to shift things around. Your time may not be much. It may only be 20 minutes a week. But remember that 48 cents a day? 20 minutes a week times 52 weeks is 17 hours. How much do you make an hour where you could be using that time for other things? What could you accomplish if you had an extra 2 days a year to work on a specific project?

Last Thoughts

The important thing is to talk to people with a specific plan, especially one that covers multiple years not just “we need to do everything all at once”. It is also very important to have a long term plan. You may only have 6 physical servers now. But where will you be in 5 years? How many servers are you going to be running by then? How critical is your data? How critical is it to be RAID 5, or RAID 6, or RAID 10? Is it worth getting the 4-bay unit for $1,200 or does it make more sense to go to a 16-bay unit for $2,200 and use it for both file storage and VM storage? Is your data really critical? Do you need to survive pretty much any failure? Go with a dual-controller unit from Promise for around $6,000, which by the way also gives you eight 1Gb iSCSI links. They also have a model that gives you two 10Gb iSCSI links, per controller.

Are your servers nearby or far away from you? If they are far away, get the iDRAC module. Do you expect to put a ton of virtuals on the server? Upgrade the ram to 32GB or 64GB (we are at 32GB per host now, getting ready to go to 64GB). So you expect to have a lot of network traffic? Make sure you have at-least 2 NICs dedicated to VM traffic per host instead of one.

Hope this helps, if you managed to read all the way to here, congratulations to you!