Powering Gen AI with flexible, fast, cost-efficient infrastructure
Artificial intelligence is no longer confined to R&D labs or tech giants; companies of all sizes are looking into how to use this tool to further their business goals. Whether it is detecting fraud in real time, accelerating drug discovery, improving product quality, or enabling smarter customer service, Gen AI is now critical to enterprise success, and it’s transforming industries across the board. But deploying AI workloads at scale can be costly and complex, requiring high-performance infrastructure, advanced telemetry, and seamless management tools.
That’s where the Intel Gaudi 3 running on VMware Cloud Foundation 9.0 (VCF 9.0) comes into play. By combining Intel’s innovative hardware and Broadcom’s suite of VMware solutions, it gives customers a flexible, efficient, and enterprise-ready platform to run cutting-edge AI workloads, reducing total cost of ownership (TCO) and accelerating time to value.
A better together solution: Intel Gaudi 3 + VMware Cloud Foundation 9.0
Support for Gaudi 3 on VCF 9.0
The latest release of VMware Cloud Foundation now supports Intel Gaudi 3 AI accelerators, enabling enterprises to run large-scale AI workloads within a fully virtualized environment. This means customers can deploy AI models faster, scale resources efficiently, and manage compute with the same tools they use across their virtualized datacenter on a daily basis.
Gaudi 3 is optimized for on-premises environments and tailored to support the growing complexity of large language models (LLMs) and multimodal AI systems. Compared to its predecessor, Gaudi 2, the third-generation accelerator delivers:
- 4× AI compute performance for BF16 operations Featuring 64 Tensor Processor Cores (TPCs) and 8 Matrix Multiplication Engines (MMEs), Gaudi 3 enables massive parallelism for deep learning workloads.
- 1.5× higher memory bandwidth with 128 GB of HBM2e memory and 3.7 TB/s bandwidth, it supports large batch sizes and expansive model architectures, ideal for LLMs and computer vision.
- 2× networking bandwidth for scale-out Integrated 800 GbE networking eliminates the need for proprietary interconnects, enabling standard Ethernet-based scaling across nodes
Accelerated performance, reduced costs
Intel Gaudi 3 is a purpose-built AI accelerator that delivers powerful, scalable and cost-efficient AI training and inference. With exceptional compute throughput, high memory capacity and bandwidth, as well as open-standard Ethernet Networking (RoCE), it’s an ideal accelerator for deep learning and inference at scale. And unlike many AI platforms that rely on proprietary infrastructure, Gaudi 3 offers performance that rivals bare metal, without the cost or complexity of managing isolated hardware silos. In addition, Gaudi 3 offers flexible deployment options between virtualized and containerized environments (Kubernetes, VMware VKS), by providing support via the Gaudi operator.
When paired with VCF 9.0, customers get an infrastructure that can quickly spins up AI-ready environments using pre-configured virtual machines, drastically shortening provisioning times-from weeks to hours; enabling administrators, data scientists, and AI/ML engineers to provision their own AI environments as needed in an accelerated fashion without jumping through a chain of IT tickets. Whether you’re training large language models or running inference at the edge, this solution enables speed, flexibility, and control.
Intel Gaudi 3 supports a wide range of AI applications including large language models (LLMs), multi-modal models, vision and retrieval-augmented generation (RAG) workflows. With software optimizations for PyTorch and popular inference frameworks like vLLM and Text Generation Inference (TGI), Gaudi 3 is designed for performance and to ease the adoption of AI workloads. Additionally, developers can access open-source recipes and tools through the Open Platform for Enterprise AI (OPEA), enabling easier adoption and optimization of AI workloads with Intel Gaudi 3 accelerators running on VMware Cloud Foundation 9.
Flexibility for hybrid workloads
One of the standout capabilities of VCF 9.0 includes its ability to support an open ecosystem of hardware vendors for CPUs, GPUs, and AI accelerators such a Gaudi 3 within a single, unified platform. This flexibility makes it easy to scale up or out as needed, while also enabling consolidation. Multiple workloads can share the same physical infrastructure, optimizing utilization, minimizing idle time, and taking full advantage of IT investments by organizations.
Gaining insight with the Intel Gaudi 3 vCenter plugin for telemetry
Intel® Device Manager for VMware vCenter Server is an extension that integrates directly into the VMware vSphere Client, offering a seamless and intuitive interface for monitoring telemetry and managing hardware configuration. Once installed, it provides administrators with a centralized view of hardware-level metrics right from vSphere Client interface, making it easier to monitor system health, performance, and resource utilization for supported Intel devices. To streamline AI infrastructure management with Gaudi, Intel offers enhanced passthrough capabilities via its Enhanced DirectPath IO driver. When installed, such driver enables direct access to Gaudi 3 telemetry through the vCenter plugin, without compromising the isolation and flexibility of virtual machines. It also enforces device grouping, allowing multiple Gaudi 3 accelerators to be logically combined and presented as a single virtual instance. This simplifies resource management and makes it easier to allocate compute at scale, particularly in environments where AI workloads require access to large, shared pools of accelerators.
The plugin offers powerful functionality when paired with the Intel® Gaudi 3 AI accelerators and the Enhanced DirectPath IO driver. The plugin surfaces a comprehensive set of metrics for each Gaudi 3 device, including: temperature, power, health state, reset uptime, memory utilization and firmware information. With better visibility, IT teams can fine-tune performance, proactively identify bottlenecks, and ensure optimal resource allocation, all within the vSphere interface they know and trust.
For additional information and setup instructions, refer to: https://www.intel.com/content/www/us/en/developer/articles/guide/device-manager-vc-user-guide.html
Conclusion
VMware Cloud Foundation and Intel Gaudi 3 are powering businesses with AI while transforming the enterprise landscape with VMs instead of bare metal servers. For businesses exploring or expanding their AI initiatives, Intel Gaudi 3 on VMware Cloud Foundations offers a smart and strategic solution. By delivering lower up-front costs, faster deployments, deployment flexibility and real-time observability into AI infrastructure performance; this solution gives businesses the tools to compete in the age of AI, without changing their entire IT environment.
Next steps
Learn more:
- VMware Private AI
- Broadcom Compatibility Guide
- Intel Gaudi 3 Accelerators
- Intel Gaudi 3 white paper
- Intel Device Manager for VMware vCenter — User Guide
- Intel Device Manager for VMware vCenter — Download Link
Get started:
- If you are attending VMware Explore 2025, make sure to attend session INVB2239LVS session for more technical details.
Authors:
Anirudh Krishna Lakshmanan (Intel)
