Advanced Troubleshooting.
The best troubleshooting is not having to. Walk the argument as a map.
The premise
01 · THE INCIDENT
02 · AVAILABILITY
03 · PERFORMANCE
04 · THE OPERATING MODEL
Each pass around the loop moves work out of the reactive track
VMware Cloud Foundation 9
↗Advanced Troubleshooting
How to architect a self-driving private cloud
The best practice for troubleshooting turns out to be not to troubleshoot in the first place.
Advanced Troubleshooting
If a slowness was avoidable, troubleshooting it later is a sign your architecture or operations already failed upstream.
Explicitly out of scope
Security
Forensic work after an attacker is already in — not troubleshooting.
Capacity Shortfall
Prevented by policy and automatic closure, not diagnosed.
Incident
1st occurrence · nothing in place to catch it
Issue!— time elapses —▶Troubleshoot + RCA▶Alert Set
2nd occurrence · the alert already exists
Alert Triggered— fast —▶Same problem · minutesSimilar problem · the hour
The whole cost of the first incident is the price of not yet having an alert. That RCA's one required output is an alert — so the next occurrence never pays it again.
Type of problem · input: logs
↗Availability
Up or down. Black and white, even when it flips intermittently.
Manual · 6Optimized · 4Ideal · 2
Availability
ManualIssue!▶Analysis▶Upload Logs┄▶Waiting▶Check the rest
OptimizedIssue!▶Joint Analysis▶RCA Report▶Alert Setup
IdealCall Home▶Preventive Action
There is no "Issue!" box in the ideal row at all. Every arrow you delete from the manual row is time your customers stop feeling.
Availability → Codifying
↗Codifying an Incident
A lesson only counts once it’s closed with an alert and a health check.
Alert set-upHealth check
Codifying an Incident
Incident!▶Vendor Recommendation▶Holistic Review
The vendor scopes their fix to the object that broke. If it happened on VM 007, nobody analysed your other thousand VMs.
Watch for alert storms
Review the whole alert set top-down, as one integrated system, whenever you add something new — and make sure remediation actions never conflict.
Type of problem · input: metric
↗Performance
Up, just slow. Random, may not reproduce — and within your control.
Manual · no RCAOptimized · 5Ideal · no incident
Performance
ManualBlame Storming▶Ping Pong▶No RCA
OptimizedSLA breached▶Big Picture▶Joint Analysis▶RCA▶Alert Setup
IdealKPI crossed▶Remediation▶Optimization
Nobody owns the "what" in the manual row, so every team argues from opinion. The ideal row fires on a leading KPI, before the SLA breaks — so it isn't an incident at all.
Deep dive
↗The Five Checks
Any slowness reduces to five checks. No more.
ContentionConsumptionConfigurationAvailabilityBugs
The Five Checks
Monitoring
1 check
1 check
Contention — the "What"
The primary metric. Defines the problem and quantifies the slowness. Start here.
Troubleshooting
4 checks
4 checks
Consumption
Configuration
Availability
Bugs
Observable doesn't mean debuggable. You can't step through a private cloud like code — you reason from these five.
Performance (%)
Worst peak, all VMs
Green
Yellow
Orange
Red
vCPU Ready
0–4%
4–8%
8–16%
16–32%
vCPU Co-Stop
0–2%
2–4%
4–8%
8–16%
Memory Latency
0–1%
1–2%
2–4%
4–8%
R/W Disk Latency
0–40 ms
40–80 ms
80–120 ms
120–160 ms
How deep is the single worst VM. How broad is the 90th percentile across all of them — halve every number above. One bad actor can't hide a widespread problem.
Check 02 · Consumption
↗Capacity
Consumption explains why contention happens — four shapes.
High consumptionImbalanceConcurrent peakMonster consumer
Capacity
High Consumption
Utilization, reservation, allocation — across CPU, memory, disk, network. Disk means IOPS and throughput; network means packets/sec and throughput, each direction.
Imbalance
One area saturates while the average looks fine. Inter-node across a cluster is usually a monster consumer; intra-node inside a host is usually NUMA.
Concurrent Peak
Workloads peak together and the combined spike saturates a resource that looked fine on average. Classic trigger: security scans across hundreds of guests at once.
Monster Consumer
One oversized VM or pod relative to its provider. Multi-chip-module CPUs compound it — NUMA effects now appear inside a single socket.
Check 03 · Configuration
↗Configuration
Three kinds — and not equally visible.
IncorrectOutdatedComplex
Configuration
Incorrect — violates best practice or your own standard. Security controls especially can cost performance.
Outdated — check the whole system before updating one part, and never assume a fix was back-ported.
Complex — may never cost latency. It quietly costs money instead.
Track suboptimal configuration even when nothing is slow today.
Capacity → Management
↗Capacity Management
Maximize consumption. Minimize contention. Stay under the SLA.
Capacity remainingTime remainingPerformance %Risk %
Capacity Management
Dimension 01 · Consumption
Capacity Remaining (%) — from predictive projection. Ideal pairs a low capacity remaining with a high time remaining.
Time Remaining (days) — the runway. Set the threshold to your procurement window.
Dimension 02 · Contention
Performance (%) — actual slowness. 20-second average, reported every 5 minutes.
Performance Risk (%) — potential slowness. A leading indicator that triggers remediation before it happens.
Stop Provisioning
NO
Either Capacity or Performance is red.
CHECK
Both orange → go to Performance Risk. Prod: orange or red → NO. Non-prod: red → NO.
YES
Every other permutation.
36
144 data points in 12 hours, so the 95th percentile lands on the 36th-worst minute. An average would smooth away exactly the spike you're trying to catch.
Play the machine
Scenario 1 of 3Capacity · OrangePerformance · OrangeRisk · 55%Prod cluster
Poll the room, then answer.
Leading indicator
↗Risk (%)
Reality vs. risk. Read early enough, you never troubleshoot at all.
UtilizationAllocationMonster VMCluster balance
Risk (%)
Red thresholds
ESXi CPU thread utilization 90–95
Net CPU overcommit ratio 3–4
ESXi memory ballooned 2–4×
Net memory overcommit 1.5–2×
ESXi CPU imbalance 95–100
CPU monster VM ÷ host 4–8×
ESXi disk IOPS 100–150K
Memory monster VM ÷ host 4–8×
Percentage of vMotion 4–8%
Inter-host CPU imbalance ≥ 100
That last row — inter-host CPU imbalance — is the least obvious number in the whole set. It gets its own card.
Risk → Cluster balance
↗Cluster Imbalance
High hosts vs. low hosts, collapsed to one score — any cluster size.
Cluster Imbalance
↑ Core utilization
High
Low
Thread utilization →
High — core > 90% and thread > 80%. 90, not 100: by the time a host is pegged for 300 seconds it's already too late, and NUMA means some cores struggle while others idle.
Low — core < 50% and thread < 40%. A gap wide enough to be operationally meaningful.
Everything between counts for neither, on purpose.
Red = high hosts with nowhere to migrate to. Green = balanced and well used. Grey = wasted capacity. Built from host counts, so a 4-host and a 64-host cluster compare directly.
The Solution
↗Advanced Operations
Two tracks, same endpoint — reactive shrinks as proactive grows.
Reactive · complaint-drivenProactive · scheduled
Advanced Operations
Reactive
Complaint or alert → walk the same four checks by hand: availability, configuration, capacity, known bugs.
Proactive
Daily automated health check · Insight · Performance Quantification · Monthly Baselining.
Human Expert
▶Finite Remediation Action
Reactive resolves straight to the action. Proactive findings pass through the expert first — they're important, not urgent. Both land on the same closed set, and the same last resort: move the workload out.
Operating model → Alerts
↗Alerts
The backstop, not the front line.
NOC: nowDaily check: 2×Business app: 4×Alerts: negative
Alerts
An alert firing means every scheduled check above it already missed the problem.
A negative guard rail — the last thing in your day that should ever be new information. Week to year, L2 does the deep analysis and long-term planning across capacity, performance and configuration: the same three checks, on a much longer horizon.
Premise
Advanced Troubleshooting
Ops maturity
Manual
00:00
01 / 43
Presenter note
Open cold. The audience came for a troubleshooting talk — tell them the goal is to need less of it. Pause after the headline.
Next · The premise
Premise
Incident
Availability
Performance
Operating Model
Click to engageSpace next← back0 mapC contentsN notes? this bar
The closed loop
The best practice for troubleshooting
is not to troubleshoot.
is not to troubleshoot.
isvara.io
43 beats: use → and ← to walk them, or tap any card to jump to it. Drag to pan. On a keyboard, click the map once to take control, then Space steps forward, 0 fits the whole thing and N shows the presenter notes. Present goes fullscreen; End or Escape comes back. The ↗ on a card opens that section of the guide.
The four regionsone argument, left to right
01
The incident
The unit of cost. Only the first outage is forgiven, and the RCA has exactly one required output: an alert.
02
Availability
Up or down, input is a log message. Three maturity levels, and the rails get shorter as you climb them.
03
Performance
Up but slow, input is a metric. Any slowness reduces to five checks, and two of them carry the whole branch.
04
The operating model
Reactive and proactive tracks landing on the same finite set of remediations. Each pass moves work out of the reactive track.
Read in the guide