Thoughts From the Mind of a Previous VMware Admin (aka Game Master)

Do you ever feel like your Windows server is not performing well in the VMware environment, as if it were a sponsor-less contestant in the arena of the annual hunger games?  As a VMware admin and consultant I witnessed first hand the fine art of deception for presenting restricted resources to virtual machines.  An application owner or Windows admin would come to my cube or put in a help desk ticket asking for a new virtual machine.  They would often ask me for an insane amount of resources for this VM because the application vendor demanded it.  So just like in the Hunger Games, I would plop the new VM into the environment but at the same time I would unleash harsh conditions on the VM without the requestor ever knowing.  In my defense, I was kind of forced to be one Mean Game Master!

Memory Restrictions are like Surviving in the Arena with Heavy Wounds

To limit : something that bounds, restrains, or confines.

Windows administrators use Task Manager or other tools to gauge performance in virtual machines, and while this is great for looking into each process for CPU and Memory usage percentages, the data is highly inaccurate.  For instance, you can look at the resources tab and see that your virtual machine has 4GB of RAM.  Feeling confident in this number meeting the application requirements, you then fire up the application.  At first things seem to be humming along as if your chariot has entered the arena with trailing flames of speed, but then intermittently the application speed goes downhill with no apparent reason.  Chances are that your VM Admin(aka the Game Master) has implemented a obstacle in the arena known as a Memory Limit.

• Ask your VM Admin for a screenshot of the Resources tab of the Virtual Machine, specifically the Memory settings area.
• If you see that a value has been given other than the amount of RAM you expect, you are a victim of a Memory Limit.
• The only good values on this screen would be the exact allocated memory or having the checkbox set for “Unlimited.”

So what does this mean to you?  Memory Limits are designed to aid VMware Admins in restricting the amount of physical memory actually allocated to virtual machines.   For instance, you request a 4GB of RAM machine but the VMware admin implements a 2GB limitation.  Your virtual machine is actually using what is known as a balloon mechanism so as to mask the memory limit from the operating system.  So as you gaze at the resources tab, you would see 4GB of RAM available for use, however in the background, you are paging the remaining necessary memory to disk!  The only “sponsor care package” you can receive in this scenario would be in the form of having insanely fast storage underlying the virtual machine disk files.

• Plead with the Game Master to remove the constraints!
• Set the Memory Limit to Unlimited or the exact amount of Memory Allocated.

Now if you have survived this long, we need to investigate the VMware ESX Host memory utilization.  How over-subscribed is the host holding your VMs? If many virtual machines are fighting for memory, the ESX Host will implement what is known as Memory Swapping. There is no tool built into the operating system that you the OS admin can use to detect this is occurring against your VM’s fight for power. If Memory Swapping is in fact occurring you now have a major issue.  All VMs on the host are experiencing latency when utilizing memory because a big chunk of that memory is virtualized on the host disk drives or Storage Area Network.

• Petition for your workload to be moved to a less contentious host!

Continue reading in part 2 of this series here where I discuss more havoc in the arena surrounding processor restrictions!