As usual, we check contention first. There are no network latency and packet retransmit metrics.

Consumption

As usual, check the non-unicast packets first and make sure they match the expectation at that time.

Non-Unicast Packets

All Packets

There is another metric ACTN/s, which is the number of actions per second. The actions here are the kernel actions. It is an internal counter, not relevant to day to day operations.

Configuration

This panel mixes physical and virtual. For virtual, it shows both the kernel network and VM network. I find it easier to use the information in vSphere Client.

The metric DTYP (Virtual network device type, where H means Hub and S means switch) does not seem to be available anymore.

vSphere Client separates the components. You can see the virtual switches, the kernel network and physical cards. The level of details is more comprehensive.

RDMA Device

Remote Direct Memory Access (RDMA) enable direct access to the physical network card, bypassing the OS overhead. The following screenshot, taken from here, shows 2 types of access from application (that lives inside a VM. The VMs are not shown).

Usage

Since it’s about network, you get both the TX (transmit or sent) and RX (received or incoming).

For contention, there is only packet dropped. There is no packet retransmit or latency. The metrics are:

For utilization, you get them in both amount of data, and number of packets. Both are important metrics. There is no breakdown on the type of packets (broadcast, multicast, unicast).

There is no packet size. This can be handy to determine if they are much smaller or larger than you expect. For example, if you expect jumbo frame but the reality is much smaller.

These metrics are not available in vSphere Client UI, so you need to use esxtop to get the visibility. Just in case you’re wondering where I got the following screenshot from, they are courtesy of Shoby Cherian and Aditya Kiran Pentyala.

You also get the queue usage information.

For more reading on RDMA, I found this academic paper, title “Understanding the concepts and mechanisms of RDMA” useful.

Configuration

vSphere Client provides the following information. You get the first 4 columns in esxtop.

The information you get in esxtop covers the first 4 columns in the preceding screenshot. They are:

Part 3 Chapter 8

Microsoft Windows and Linux have different techniques for resource management. They also have different metrics names and accounting strategy.

Introduction

There are 3 tools in Windows:

• Performance Monitor. This is the oldest one, and is superceded by the next two.

• Task Manager

• Resource Monitor

They use different names. It is also possible that they use different formula.

One major difference between Guest OS and VM is that Windows/Linux runs a lot of threads and processes. The problem is there is minimal observability on individual thread or process. For example, there is no CPU queue metric, memory page fault, network dropped packet, and disk latency at process level.

The following shows Windows Sysinternal¹, a great tool for Windows troubleshooting. As you can see, they are just utilization metrics. There is no contention metric.

CPU

Performance Monitor is still the main tool for Microsoft Windows, despite the fact it has not been enhanced for decades. Go to docs.microsoft.com and browse for Windows Server. It took me to this article, which cover PerfMon. Many explanations on metrics at https://learn.microsoft.com/ are still based on end-of-life Windows.

PerfMon groups the counters under Processor group. However, it places the Processor Queue Length and Context Switches metrics under the System group. The System group covers system wide metrics, not just CPU.

The following screenshot shows the counters under Processor group.

The counters can be grouped into 2:

• Time-based

• Non time-based.

PerfMon UI provides a description, which I use as a reference below.

The non time-based counters are:

System Calls/second:

• It is not a CPU metric but is commonly used next to the queue and context-switch counters.

• A call is a request to the kernel to do work on behalf. It occurs because the program cannot perform a task itself because it involves protected resources (like hardware or memory belonging to other processes). This instruction forces the CPU to switch from Ring 3 (User Mode) to Ring 0 (Kernel Mode).

• It measures the number of times a software application transitions from User Mode to Kernel Mode to request a service from the kernel.

• If System Calls/sec is high and Privileged Time (%) is also high (e.g., >30%), the CPU is spending too much time on OS management rather than running your actual application code.

• Values consistently exceeding 20,000 per processor core are generally considered high and warrant investigation. In most cases, it’s below 2000 per core.

• This metric is not collected by Tools.

Thread State

In Windows, an application consists of ≥ 1 process. A process consists of ≥ threads. Thread is what is scheduled in the CPU.

It’s easier to understand the CPU scheduling when you think in terms of 2 distinct objects:

• Processors

• Threads.

The following information is taken from the famous Windows Internal book, 7^th edition and from this Microsoft link. A thread is always in one of these 8 possible states.

You may see a higher or lower number than the number of processors as the observability tools counts the threads, not the processors. Windows PerfMon or WMI code cycles through each thread and get the state. It collects the state at that particular moment. In the 100 milliseconds or so it takes to finish counting thousands of threads, the "Standby" slots on the CPUs could have been emptied and refilled hundreds of times.

CPU Issue

This means there is not enough CPU, or there is the performance issue is with CPU.

Non-CPU Issue

CPU is much faster than memory, disk or network. These 2 states capture times where CPU is waiting for data.

Other States

Run Queue

Number of threads in the processor queue. Unlike Linux, Windows excludes the threads that are running (being executed).

Let’s take a VM configured with 8 vCPUs. The Guest OS sees 8 threads so it will schedule up to 8 parallel processes. If there is more demand, it will have to queue them. This means the queue needs to be accounted for in Guest OS sizing.

Because it reports the queue, this is the primary counter to measure Guest OS CPU performance. It tells you if the CPU is struggling to serve the demand or not.

What is a healthy value?

Here is my take:

Windows Performance Monitor UI description is not consistent with MSDN documentation (based on Windows Server 2016 documentation). The description shown in Windows UI is “Processor Queue Length is the number of threads in the processor queue. Unlike the disk counters, this counter shows ready threads only, not threads that are running. There is a single queue for processor time even on computers with multiple processors. Therefore, if a computer has multiple processors, you need to divide this value by the number of processors servicing the workload. A sustained processor queue of less than 10 threads per processor is normally acceptable, dependent of the workload.”

MSDN document states that a sustained processor queue of greater than 2 threads generally indicate processor congestion. SQL Server document states 3 as the threshold. Let me know if you have seen other recommendations from Microsoft or Linux.

Windows or Linux utilization may be 100%, but as long as the queue is low, the workload is running as fast as it can. Adding more vCPU will in fact slow down the performance as you have higher chance of context switching.

You should profile your environment, because the number can be high for some VMs. Just look at the numbers I got below. I profiled 3385 Windows VM in the last 1 month. I plotted the highest value.

Columns explanation:

• 95P means the value at 95^th percentile during the last 1 week.

  • I take 95^th percentile because there are 30240 data points.

  • While the collection is every 5 minutes, the value is a snapshot taken every 20 seconds. It is the highest among 15 data points. There are 15 data points * 7 days * 24 hours * 12 = 30240 data points.

• Max Queue means the highest in the last 1 week. That means it is the highest of 30240 data points.

• Net Run at Max means the value of CPU Net Run when the value of vCPU Queue is at the highest. Using the first VM as an example, Net Run showed 52% when CPU Queue was at 58%. It’s a correlated number.

• Context Switch as Max means the value of Microsoft Windows CPU Context Switch when the value of vCPU Queue is at the highest.

• vCPU is the present configured CPU. This number is generally stable, hence I took the present.

High CPU Queue

Let’s drill down to the first VM in the preceding table.

How can an 8 vCPU Windows Server VM have CPU Queue of 53 when only running at 52%?

We expect the number to be around 8 since only half the CPU are running. Basically it should be 0 Ready + 8 Standby + 0 Deferred Ready.

What do you notice from the following chart?

There seems to be no strong correlation between CPU Queue and CPU Context Switch.

The CPU Queue:

• Fluctuates from 14 to 59 constantly. The pattern is repeatable. When I zoomed in, each cycle has a sharp jump within 5 – 10 minutes, then slowly drops over 1 hour.

• Trend upwards over 7 days. It’s a gentle trend. This means something that slowly deteriorates caused it. Memory leak?

FYI:

• CPU Ready, CPU Co-Stop and CPU Other Wait are all low. This means NUMA is not an issue.

• Memory contention is flat 0.

• Virtual Disk latency is low most of the time, and did not correspond to the above 3 charts.

Here is CPU consumption metrics:

We can see massive imbalance. While the average CPU Run is 50%, the gap between the busiest vCPU and the least busy is >140%. Seems like the threads congregate in some of the CPU.

I don’t have the details of the invidual vCPU. It will be interesting to plot each to see if there is high thread migation.

Let’s pick a shorter time window so we can see the chart clearly.

The following chart shows the pattern over 6 hours.

The spike on CPU Context switch does not match the CPU Queue. There is a match to CPU Run, which makes sense. CPU Usage Disparity went down, because the overall CPU Run went up.

Other things to check:

• Is it running a multi-threaded application? Some applications spawn many threads. If yes, it’s possible that many threads are waiting for a Software Lock (like a Mutex or Semaphore). The threads in the queue cannot be distributed to those idle cores because they are all waiting for a specific resource held by a thread on a different core. They are marked as "Ready" because they aren't waiting on I/O, but they are effectively stuck in a bottlenecked pipeline.

• If Processor% Privileged Time counter is high but User Time counter is low, the kernel (drivers/interrupts) is hogging the CPU, not the application. We have a driver or kernel-locking.

Low CPU Run

How about when Net Run (%) is low?

The following Microsoft Windows server VM has 2 vCPU only.

FYI:

• CPU Ready is below 0.5%. CPU Co-Stop and CPU Other Wait are all low. This is expected as the VM is basically idle.

• Memory contention is flat 0.

• Virtual Disk latency is 1 ms, and did not correspond to the above 3 charts.

Queue and Idle

It is possible that Guest OS shows high CPU queue when it was idling. This is abnormal, indicating the application created a lot of threads. The following shows the application was idling while having high queue.

The CPU Run Queue spikes multiple times. It does not match the CPU Usage. It also did not match CPU Context Switch Rate pattern. The spike only last 20 seconds, as the 5-minute average shows identical pattern but much smaller number.

Overall Behaviour

What is the behaviour of CPU Queue?

I profiled 800 VM in the last 1 month. For each VM, I extracted the peak value and the 99^th percentile value. From the following scatter chart, you can see that the value at 99^th percentile is less than half. Using a spreadsheet, the average value of the 99th percentile is 34% for peak that is ≥ 3.

BTW, the value from Guest OS displays the last observed value only; it is not an average. Windows & Linux do not provide the highest and lowest variants either.

The counter name in Tools is guest.processor.queue. For Windows, it is based on Win32_PerfFormattedData_PerfOS_System = @#ProcessorQueueLength from WMI

Reference: Windows

I can’t find documentation that states if CPU Hyper Threading (HT) technology provides 2x the number of queue length. Logically it should as the threads are at the start of the CPU pipelines, and both threads are interspersed in the core pipeline.

CPU Priority

If a process is often in queue, one possility is it has lower relative priority. Priority is a concept of Windows that ESXi does not have. ESXi uses a fair-share scheduler instead, as it does not have foreground processes.

Windows or Linux provides priority for foreground activities, as that’s what the user experience. For Windows, there are 6 levels as shown below. Ensure all your agents are given lower priority and limited CPU resource.

Context Switch

CPU Context Switch costs performance “due to running the task scheduler, TLB flushes, and indirectly due to sharing the CPU cache between multiple tasks”. It’s important to track this counter and at least know what’s an acceptable behaviour for that specific application.

Context switches are considered “expensive” operations, as the CPU can complete many instructions within the time taken to switch context from one process to another. A sustained high number degrade performance noticeably through "cache pollution" — where the CPU has to constantly reload data into L1/L2 caches because the previous thread was swapped out.

For further reading, read this paper.

Based on Windows 10 Performance Monitor documentation, context switches/sec is the combined rate at which all processors on the computer are switched from one thread to another. All else being equal, the more the processors, the higher the context switch. Note that thread switches can occur either inside of a single multi-thread process or across processes. A thread switch can be caused either by one thread asking another for information, or by a thread being pre-empted by another, higher priority thread becoming ready to run.

Here is my take:

There are context switch metrics on the System and Thread objects. VCF Operations only report the total.

The rate of Windows or Linux switching CPU context per second ranges widely. The following is taken from a Windows 10 desktop with 8 physical threads, which runs around 10% CPU. I observe the value hovers from 10K to 50K.

Correlation

The value does not correlate well with CPU “utilization”, because not all CPU instructions require context switching. Overall, the higher the utilization the higher the chance of CPU context switch. I plotted 3328 VM on a scatter chart.

The above does not mean there is no correlation. The following chart shows a near perfect corelation. Every time CPU Usage went up, CPU Context Switch also. I should have plotted the disk IO or network IO as IO operations tend to require context switch.

CPU context switch can happen even in a single thread application. The following shows a VDI VM with 4 vCPU. I plotted the CPU Usage Disparity vs CPU Context Switch. You can see the usage disparity went up to 78%, meaning the gap between the busiest vCPU and the idlest vCPU is 78%. This was running a security agent, which is unlikely to be designed to occupy multiple vCPU.

Let’s plot the context switch at the same period. There is a spike at the same time, indicating that the agent was busy context switching. Note that it does not always have to be this way. The red dot shows there is no spike in context switch even though the vCPU Usage Disparity went up.

Range Analysis

The values of CPU Context Switch vary widely. For many VM, the values will be in low hundreds. In extreme situation, it can sustain well beyond 10 millions, as shown in the preceding chart. The above VM was not doing heavy IOPS nor high CPU usage. There was no correlation with these 2 metrics either.

Because of above, it’s important to profile and establish a normal base line for that specific application. What is healthy for 1 VM may not be healthy for another.

You can see from the table that some VM experience prolonged CPU context switch, while others do not. The VM #4 only has a short burst as the value at worst 5^th percentile dropped to 3796. Momentary peak of context switch may not cause performance problem so in general it’s wiser to take the value somewhere between 95^th and 99^th percentile.

Let’s drill down to see the first VM. This CentOS VM sporting only 4 vCPU constantly hit almost 1 million context switches. The pattern match CPU Usage.

Do you add more CPU?

You should not, as the queue remain manageable. Check what the queue like on a more granular reading.

The following distribution chart shows the values from 11372 VMs. I had to use log-10 scale as the values vary wildly.

Majority of Guest OS spends well below 10K. You can see that the values between 0 – 10000 accounts for 80%.

Now that you know the wide distribution, you can use buckets. Adjust the bucket size by grouping all the values above 10K as one bucket, and splitting 0 – 10K bucket into multiple buckets. You can see more than half has < 1000 CPU Context Switch Rate.

Thread Ping Pong

The following is a Windows Server 2019 DC edition VM with 10 vCPU. It’s basically idle, as you can see below.

Footnotes

1. Your anti-walware may classify it as a threat even though it is obviously not a malware. Malware uses psexec, a component of sysinternal, to execute commands on remote machines. Some of the executables also behave in such a way that anti-malware software classifies them as malicious. ↩