VMware Advanced Memory Tiering Tips for Success

Your Practical Guide to Doubling Memory Without Doubling Your Budget

This document provides tips for successfully deploying VMware Advanced Memory Tiering with NVMe on VCF 9.0.

Let’s Talk About Memory Tiering

Resource Hub: Memory Tiering Resource Hub

Memory Tiering with NVMe is one of the standout features in VMware Cloud Foundation (VCF) 9.0, and honestly, it’s been the most talked-about topic since VMware Explore 2025 and in my daily customer conversations. Why all the buzz? Because it solves a real problem that hits every IT budget: memory costs are eating up a huge chunk of hardware spending.

We’ve all been there. You need more memory for your VMs, but when you price out DRAM modules, your CFO takes notice.Memory Tiering completely changes the conversation. By leveraging NVMe storage as a memory tier, you’re getting enterprise-grade memory expansion at NVMe prices. And trust me, that’s a game-changer for infrastructure planning.

What You Actually Get

With the default 1:1 ratio, you double your memory capacity right out of the box (a 4x improvement from tech preview). If you’ve got a host with 1TB of DRAM, you’re now looking at 2TB total. Same hardware, double the capacity. That’s ROI.

But it’s not just about capacity. You’re getting better VM consolidation ratios, which means more workloads per host. And here’s the flexibility part I love: you can start small and scale as needed. Enable it on one host to test, roll it out to a few more, or deploy cluster wide. 

How It Actually Works (the Simple Version)

Think of it as a two-tiered memory system. Tier 0 is your DRAM, the fast stuff that handles your active memory pages. Tier 1 is your NVMe storage, which takes care of the cold or dormant pages that aren’t active.

The beauty is in the intelligence. The system automatically manages which pages go where. Your VM’s active working set stays in DRAM for maximum performance, while cold pages move to NVMe. Critical detail: vmkernel memory never touches NVMe, as ESX runs on DRAM at all times.

Plus, this integrates seamlessly with all the features you know and love from VCF such as vMotion, DRS, HA, vSAN. No special workflows, no new operational procedures. It just works.

Before You Start: The Assessment You Can’t Skip

See also: Blog Series Part 1: Prerequisites & Hardware

The Golden 50% Rule

Here’s your first checkpoint: keep active memory at 50% or less of your DRAM capacity. We want that active working set (the hot data your VMs are constantly accessing) to fit comfortably in DRAM and have room to spare. That way, you get fast DRAM speeds for everything that matters performance-wise and have extra room for potential failure, vmkernel pages, etc.

Now, if you’re running at 70% active memory against DRAM, you may encounter a problem. Some of that active data will end up in NVMe tier, and you may see performance impacts. Don’t skip this assessment step. Check out this performance whitepaper to learn about acceptable active memory percentages. 

How to Check Active Memory (Three Ways)

You’ve got options here. In vCenter UI, navigate to VM > Monitor > Performance > Advanced, switch to memory view, and look for active memory. Heads up —it only shows in real-time mode because it’s a Level 1 stat. Need historical data? You can bump statistics from Level 1 to Level 2. Fair warning: my database jumped from 16GB to 43GB when I did this. Make sure you’ve got the storage space before you go down that road.

Prefer tools? VCF Operations gives you great visibility if you’re already using it. Or grab RVTools, which collects real-time memory stats. Pro tip: run your RVTools collection during peak hours. You want to see your workloads under actual load, not during that Sunday morning maintenance window when nothing’s happening. You could also leverage Live Optics for this purpose and run multiple times for a better overall picture. 

Oh, and I almost forgot—I built a tool specifically for this assessment: the Memory Tiering Assessment Tool (MTAT). It’ll check VM active memory against host DRAM and do all the math for you, telling you straight up whether your workloads qualify or not. It will save you a ton of time.

What Works (and What Doesn’t)

Memory Tiering is perfect for your bread-and-butter workloads: general virtualization, VDI deployments, dev/test environments, web and application servers, and databases with moderate memory activity. These are the workloads that have a natural hot/cold memory pattern where some pages are accessed frequently, and others sit idle for extended periods. That’s the sweet spot.

Now, let’s talk about what doesn’t work in VCF 9.0. High-performance or latency-sensitive VMs aren’t a good fit because they need predictable, ultra-low-latency memory access. Security VMs using SEV, SGX, or TDX have encryption requirements that don’t play nice with tiering yet. Fault Tolerance VMs need synchronous memory state, and that’s incompatible with our asynchronous tier management. “Monster” VMs (1TB+ memory or 128 vCPUs) can overwhelm the page management system. And nested virtualization? Too complex for the current implementation.

But here’s the good news: you’re not stuck if you have these workloads. Dedicate some hosts to them while enabling Memory Tiering on the rest of your cluster. Or disable it at the VM level for specific machines. The flexibility is there. And keep watching our roadmap as we’re actively working on expanding compatibility.

What You’ll Need

Requirements are straightforward: 

• VMware Cloud Foundation 9.0 (or later)
• Enable SSH on your ESX hosts because you’ll need command-line access for partition creation. 
• PowerCLI if you want to automate things. Trust me, for cluster deployments, scripting saves hours.

Hardware: Where You Cannot Compromise

Let me be completely direct here: hardware selection is everything. Since we’re using NVMe SSD devices as a memory tier, they need to be both incredibly reliable and fast under sustained heavy use. This isn’t a “good enough” situation. Remember, these drives are handling memory pages, not just storage I/O. The performance and endurance requirements are serious.

The Specs that Matter

• You need Class D or higher endurance (at least 7,300 TBW). 
• Or if your OEM doesn’t use the class system, look for Enterprise Mixed drives with at least 3 DWPD (Drive Writes Per Day). 
• For performance, you’re targeting Class F (100,000 to 350,000 writes per second) or Class G (350,000+ writes per second).

DWPD measures how many times you can write the drive’s full capacity daily over a 3-5 year warranty. A 1TB SSD with 3 DWPD handles 3TB of writes every day for years. That’s the endurance you need.

Why does this matter so much? Because we’re potentially doing massive read/write operations to NVMe SSDs as the system manages memory tiers. Use a cheap consumer drive? You’re risking premature failure, system instability, and performance issues. Memory Tiering reduces your TCO dramatically, but don’t undo those savings by cheaping out on the drives. This is not the place to cut corners; trust me on this.

Double-Check Everything

Before you buy anything, check the Broadcom Compatibility Guide and verify your drives meet the requirements. Here’s a neat trick: even if you’re not running vSAN, you can use the vSAN Compatibility Guide for Memory Tiering. Same requirements, same verification process.

OEM Translator Guide

Different vendors speak different languages when it comes to drive specs. Here’s your quick translation guide:

• DELL: Look for ‘Mixed-Use Enterprise NVMe’ in their configurator. You’ll see options like ‘1.6TB Enterprise NVMe Mixed Use AG Drive U.2 with carrier.’ Make sure it shows 3 DWPD or higher. 
• Heads up: you might need different NVMe pairs for Memory Tiering versus vSAN in a ReadyNode setup.
• HPE: Makes it easier with their SSD Recommendation Tool. Filter to ‘Mixed Use’ workload type, select both ‘NVMe Mainstream Performance’ and ‘NVMe High Performance‘, then check the datasheets to confirm 3+ DWPD.
• Lenovo: Go with Mixed Use NVMe drives that hit that 3 DWPD mark. When you’re configuring a purchase, select the NVMe backplane and choose Enterprise-class or Advanced Data-Center class drives. Good options include ThinkSystem CD8P, PM1745, P5620, 7450 MAX, and PS1030.

And don’t forget about form factors (you’ve got flexibility here). Standard 2.5-inch U.2 drives, E3.S pluggable devices, even M.2 if that’s all you have open slots for. I’ve seen customers get creative with M.2 slots when their 2.5-inch bays were maxed out with vSAN storage. The Compatibility Guide lets you filter by all these factors.

Configuration: Simpler Than You Think

See also: Blog Series Part 6 – Deployment Considerations

Once you’ve got the right hardware, configuration is surprisingly straightforward. Two steps: 

1. Create a partition.
2. Enable the feature. 

That’s it. But let’s walk through some nuances that’ll save you headaches later.

Creating the Partition

First, clean up any existing partitions on your NVMe device. The max partition size is 4TB, but if your device is bigger, that’s actually good news because it enables cell cycling (wear leveling) and potentially extends drive life. Here’s the catch: even though your partition might be 4TB, the system only uses what it needs based on your DRAM and ratio. So 1TB DRAM + 1:1 ratio = 1TB of actual NVMe usage, even if the partition is 4TB.

esxcli system tierdevice create -d /vmfs/devices/disks/<UID>

Critical point: you CANNOT use two non-RAID devices per host for redundancy. The system will only mount one drive, chosen randomly at boot. For redundancy, use hardware RAID 1. No shortcuts here. 

Enabling Memory Tiering

You’ve got three approaches. ESXCLI for individual hosts, vCenter UI for manual configuration, or my recommendation, VMware Configuration Profiles (Desired State Configuration) for cluster-wide deployment. Configuration Profiles automate everything: rolling reboots, VM migration, even maintaining vSAN availability during the process. Set it and forget it.

After reboot, verify you’re seeing 2x memory capacity in vCenter. Check Advanced System Settings, Monitor tab, and Hardware > Memory. If you’re not seeing double, something went wrong.

Sizing: Keep It Simple

See also: Blog Series Part 3 & Part 5

Default 1:1 ratio means if you have 1TB DRAM, get at least 1TB NVMe. But consider going bigger. As discussed earlier, larger devices enable cell cycling, give you headroom for future ratio changes, and handle unexpected growth. Given the price difference between NVMe and DRAM, this is a smart investment. Don’t change the ratio unless you’ve thoroughly inspected your active memory on your hosts. Remember we want to be under 50% for 1:1 ratios, and even lower for bigger ratios. 

Security and Operations

• Memory Tiering supports both VM and host-level encryption. It’s transparent with vMotion, fully integrated with HA, and DRS understands both tiers. 
• Maintain consistent configs using Configuration Profiles, take a phased deployment approach, and document any VM-level exceptions.

Performance: The 50% Rule in Action

Monitor active memory regularly. If you’re consistently over 50% DRAM, you may start to see some performance degradation. The system handles page management automatically as active pages stay in DRAM, and cold pages migrate to NVMe when there is memory pressure. No manual intervention required. And remember: vmkernel pages never touch NVMe, keeping ESX performance rock-solid.

Monitoring and Resources

• Watch these metrics: active memory percentage, page access patterns, cluster utilization trends, and per-VM consumption. 
• Monitor NVMe read/write latency.
• Establish baselines before deployment, compare after. 
• VCF Operations gives enhanced visibility if you’re using it. 

Final Thoughts

Memory Tiering isn’t just a feature; it’s a fundamental shift in how we think about memory economics. You’re doubling capacity without doubling your budget. You’re consolidating more workloads per host. You’re simplifying capacity planning. And you’re doing it all with seamless vSphere integration that just works.

Start with proper assessment, choose the right hardware, configure methodically, and monitor actively. That’s your path to successful deployment. And as we continue evolving Memory Tiering with broader workload support and enhanced capabilities, you’re already positioned to take advantage of those innovations while capturing immediate cost savings today.

——-

Based on VMware Cloud Foundation 9.0 | Author: Dave Morera, Staff Technical Marketing Architect