In production environment, these numbers should be low. Some numbers such as shutdown should also match the change request and happens during the green zone. Some exceptions apply, such as your VDI design includes scheduled reboot on the weekend.

Imbalance

Imbalance is the root cause behind why contention happens despite overall utilization has not reached 100%.

There are 3 types of imbalances in a vSphere Cluster.

Inter-Host Imbalance

You want clusters to be well utilized but balanced across all hosts in the cluster.

• High Utilisation 🡪 Performance risk.

• Low Utilization 🡪 Wastage capacity.

The cluster is imbalanced when there are High ESXi hosts and Low ESXi hosts.

Focus is imbalance at high utilization. Imbalance can happen at low utilization. While that’s mathematically true, it’s not operationally important.

The formula is basically High Hosts / Low Hosts.

Let’s visualize using a cluster of 6 hosts as an example:

Quantification Strategy

The formula uses a relative number, so we can compare across clusters of different sizes.

Looking at the values, we can see that each color falls within some specific ranges, that is intentionally designed not to overlap.

We can now translate the color into numbers.

Limitation

Summarizing always results in the loss of details.

For example, we don’t know if it’s 1 High Host over 2 Low Host, or 4 hosts over 8 hosts as both returns 0.5.

Alternatively, we can concanetate the numbers. So we get 1.2 and 4.8.

• Advantage: we know 1.2 is different to 4.8

• Disadvantage: the sort across many numbers will be less meaningful as the values are no longer relative.

Example

Let’s use a 12-node cluster as an example.

The following shows the complete permutation of the 169 possible Cluster Imbalance values.

Intra-Host Imbalance

You want all cores used first, before threads are sharing a core.

Since HT is dual thread, the worst score is 2.0.

When there are idle cores, no core should be running 2 threads. Since real world is not perfect world, the diagram shows gradient. It changes from red to green as Core Utilization hits 100%.

Imbalance Score of 1.5 at means half the cores are running 2 threads.

For an ESXi with 64 cores, 128 threads, a score of 1.5 at 40 cores means:

• 20 cores are running 2 threads

• 20 cores are running 1 thread

• 24 cores are idle.

Ideally, it should be 60 cores running 1 thread each.

The imbalance should be used together with Core Utilization.

ESXi CPU Imbalance = Count of Running Threads / Count of Running Cores

vSphere Cluster ESXi CPU Imbalance = Average (ESXi CPU Imbalance)

Performance

Operationally, you manage at cluster level, not at Resource Pool, ESXi host, or data center level. It’s the sweet spot for starting your monitoring and troubleshooting. As usual, we start with the contention metric, followed by the utilization metric.

By definition, the metrics are average numbers. So be careful as there can be VM that has issue but obscured in the cluster wide average. Even the so-called total or summation is mathematically an average. For example, the Total CPU Wait counter is the sum of all ESXi CPU Wait metrics, which in turn is the sum of all the VMs. At the end you get a large number, which you need to normalize and convert into average. Since you divide it against the cluster total, you get average.

Contention vs Consumption

There is a common misconception that you cannot have performance issue when cluster has low utilization. We introduced that problem in Part 1 Chapter 1 of the book.

Is there corelation between cluster utilization and cluster contention?

I’ll show 2 opposite examples.

Example showing Correlation

If every VM is given the same treatment by the cluster, then yes.

Here is a cluster experiencing regular high utilization in the last 7 days. You can clearly see the peak. The cluster has 14 ESXi Hosts.

A logical question here would be what’s the impact on VM performance? Are they getting the CPU they asked? The cluster has 550 running VM.

This is where the contention metrics come in. One tracks the depth of the problem, the other the breadth of the problem.

The counter Percentage of VMs facing CPU Ready > 1% shows a nearly identical pattern. We can see that a big percentage of the VM population is affected.

The second counter tracks the depth, giving the absolute worst CPU Ready value experienced by any VM in the cluster.

Example showing no Correlation

Performance is unmet demand. VM 007 can face very high contention when all other VMs on the same cluster face no contention.

It is possible for VMs in the cluster to suffer from poor performance, while the cluster utilization is low. One main reason is cluster utilization looks at the provider layer (ESXi), while performance looks at individual consumer (VM).

The following cluster has 32 ESXi supporting 2357 VM. The average demand across the cluster is <40%. Since it has 32 ESXi and 2357 VM, we can retire 8 ESXi or add 1K VM.

And yet the VMs in the clusters are facing contention. Both VM CPU Ready and CPU Co-stop are high.

Let me take another example, where you can see the corelation between cluster utilization and VM contention in the cluster. My apology that the picture is not sharp. You can see the cluster has 774 running VM at the start. One month later it has dropped to 629, a drop of 145 VM or 19%. The second line chart reveals the number of running vCPU dropped from 3019 to 1980, a whopping 1039 vCPU or 34%. That indicates the big VMs were moved out.

This cluster was running mission critical VMs. What’s going on?! What caused the mass evacuation.

Notice the mass evacuation happened multiple times, so it’s not accidental.

Looking at the last chart. It has 2 line. Maroon showing utilization, blue showing contention. Can you figure out what happened?

The cluster utilization was hovering around 50%. In that entire month, it barely moved. This cluster was probably 16 nodes, so 50% utilization means you can easily take out a few ESXi hosts actually.

The Max VM CPU Contention told a different story. Notice it spiked well above 75%. That impacted at least 1 VM. There were multiple spikes, leading to multiple complaints, and eventually infrastructure was forced to evacuate the cluster to fix the performance problem. Notice the counter dropped gradually in November, despite utilization remains fairly stable.

Example for Memory

We covered 2 examples for CPU. What about memory, since it’s a form storage. It’s just a disk space basically, so can VM experience contention when Cluster consume metric is not high?

I’d zoom into ESXi, so it’s easier to see. What do you deduce from this ESXi? This chart shows 1 month worth of data.

It has 759 GB of usable memory. All the powered on VM has 444 GB configured, out of which only 413 GB is mapped to physical DIMM. So there is plenty of memory left.

To confirm that it has plenty of memory, let’s plot Balloon. What do you expect?

There is no ballooning. ESXi was under no memory pressure whatsoever.

So that’s the situation at provider level. How about consumer level?

VCF Operations has a metric that tracks the highest memory contention experienced by any VM in the host. This is a good leading indicator as all it takes is 1 VM, it matters not which VM.

As we can see here, there is a problem.

Can you explain why?

A VM experiences contention when the page is not in the DIMM. It was compressed or swapped out. Checking the compressed metric, it reveals that pages had to be brought it. Notice the swap metric lagged a bit, which makes sense.

I am not able to explain the earlier dropped, the one in red circle. If you can drop me a note.

Let’s complete by plotting Swapped. I’m plotting all the way to the beginning of tracking.

It’s all 0. What happened?

That means all the pages could be compressed, so ESXi decided to compress instead of putting them into swapped file.

Now that we know it’s due to compression, we know the contention on 5 September was caused by compression. When was that page compressed, no one knows. Plotting back, the compression started around 2 August.

The compression was only 342 MB. Not even 0.1% of consumed memory. But if you are unlucky, it was the active VM that got hit, as in the case here.

The past is harder to debug, as we lack the ability to travel back in time and see the environment as it was. My guess here is the VM had limit, be it indirectly via resource pool or directly.

Cluster Performance (%)

We’ve covered in the VM chapter how we quantify the KPI of a single VM. How would you represent all the VMs in the cluster? Do you simply average the VM KPI (%)?

The answer is no. A cluster has a different purpose to a VM, so we need to see it from cluster point of view. For examples:

• Contention inside a VM (this means Windows or Linux) is not that relevant to the cluster performance.

• ESXi physical network is relevant to the cluster performance, but not to the VM performance.

A cluster is more of a group of ESXi hosts serving a VM.

The metrics are grouped into 2:

• Breadth

• Depth.

We cover why we need both dimensions in Part 1 Chapter 2 Performance Management.

The metrics are using the 20-second peak so it does not miss short bursts. The limitation of this metric is outlier. A large cluster with thousands of VMs can easily have 1 VM having contention due to limit. One way to minimize the false positive is to set a higher threshold.

For the breadth, we use 90^th percentile instead of average. Average tends to be too late. On the other than, 95^th percentile suffers from cluster imbalance.

In a cluster with many hosts, there can be imbalance. It is not possible to aggregate at the host level first. This 2-level aggregation can result in suboptimal number. The drilling down to specific host can be facilitated with the host level metric.

Metrics

vMotion is included as it does impact the VM performance (although the end users may not notice in most cases) and it’s a leading indicator that the cluster is struggling to serve the load hence it has to shuffle the VMs around.

Take a look at this cluster. It has 488 running VMs on 16 ESXi host. Notice the percentage of VM being vMotion jumped to 5.3%, as 26 VMs were vMotion.

What do you think will happen to the VM CPU Ready and CPU Co-stop?

They rose. Since only 5% was impacted, the rise will be minimal.

The threshold should reflect reality. For examples:

• While the impact on VMs is the same with Ready, using the same range for CPU Co-stop, CPU Overlap and CPU Other Wait will elevate the KPI score, as practically these 3 have lower score.

• Dropped packet and error packets are very rare. Instead of summing them up, which will result in an average, I took the worst among ESXi host. Since many ESXi sports 25 Gb NIC, I set the threshold to be very low. On the other hand, I did not set green = 0, so the KPI does tolerate some issue.

• Ballooning does not actually impact performance. That’s why it’s given 4x the threshold of zipped + swapped. I may change to 6x, but will have to consider large cluster. A cluster with 20 TB of RAM means 4% ballooned translates into 800 GB.

• Ballooned, swapped, and zipped are given higher threshold as they could have happened in the past. These metrics are “sticky”. They also may not impact performance. But since many clusters sport >4 TB of RAM, I need to balance the absolute size.

Metrics Not Used

The following metrics are not included:

• “Percentage of VM population experience CPU Ready > 1%” and “Percentage of VM population experience Memory Contention > 0%” as they measure the same purpose with the average.