When it comes to infusing artificial intelligence (AI) into enterprise applications, developers, platform engineers and data scientists are facing a tremendous opportunity. However, in Tanzu, we are also hearing about their struggles to get to production, not to mention achieving positive return on investment (ROI). Even highly skilled and well-funded IT organizations are lamenting that they are not able to get their AI application experiments to market with a repeatability and predictability that includes model safety and cost control. Some of these organizations have decided to overcome these hurdles by bringing in outside services only to quickly realize that they cannot sustainably scale that approach. What they need is to adopt an application platform – or extend their current platform – to accommodate new frameworks and guardrails for AI applications.
VMware Tanzu AI Solutions, available in VMware Tanzu Platform, offers a groundbreaking set of tools designed to streamline intelligent application development and delivery, allowing teams to innovate faster while maintaining control and compliance. Tanzu AI Solutions includes a category defining capability we call AI middleware, that can help expedite ROI with your AI apps. AI middleware is more than a secure API gateway. It provides a set of tools to help simplify delivery, and future proof your application platform investment, while providing the right controls for governance, scalability, and data security.
Everything old is new again: With AI apps, “middleware” makes a comeback!
The concept of middleware has evolved significantly, transitioning from traditional platforms-as-a-service (PaaS) to today’s platform utility models. As AI technologies become more integrated into applications, the role of AI middleware becomes crucial in managing the complexities these systems introduce. AI middleware acts as a bridge that facilitates safer and more efficient delivery of AI-powered applications and establishes the guardrails needed for responsible AI application use.
AI applications present unique challenges because of their non-deterministic nature, making them difficult to test and predict. Organizations often struggle to support the dynamic and unpredictable behaviors that AI introduces with their current platform and teams. As such, there is a pressing need to bridge the capabilities gap in both platform and personnel, so that developers can get their AI apps to production more quickly.
AI middleware in Tanzu AI Solutions creates the bridge to enable development of AI apps with varying levels of agency–from predictive, to fully autonomous. AI middleware not only supports the integration of AI logic into applications but also provides the necessary safety features to reduce risk. These include mechanisms such as rate limiting, model access controls, and audit capabilities, all of which enable the responsible and cost effective use of AI in applications.
Increase data security and privacy for AI models
We have all heard the stories of private information leaked to models. Externally hosted models are especially vulnerable to leakage of sensitive company information that should not be shared with third-party providers. Establishing an AI middleware layer with proper safeguards and risk assessments is crucial to protecting data in such scenarios. Tanzu AI Solutions includes AI middleware that provides secure connections and model isolation with private model hosting from VMware Private AI Foundation.
- Secure Connections: Tanzu Platform offers a single “golden command” that seamlessly binds data securely into applications, ensuring a streamlined and reliable process. This approach eliminates the need for manual handling of service credentials, as Tanzu Platform manages secure binding automatically. By centralizing and securing credential management through the AI middleware in Tanzu AI Solutions – which is included with Tanzu Platform 10 and above – organizations can significantly reduce the risk of sensitive information leaks, providing both developers and organizations greater peace of mind when handling critical data.
- Private AI Hosting: For enterprises with stringent privacy and security requirements, VMware offers a robust solution for hosting AI models in secure or air-gapped environments. By leveraging VMware Private AI Foundation in combination with NVIDIA’s advanced technology, organizations can deploy and manage AI models while ensuring complete data protection. This setup guarantees that sensitive data remains confined within the organization’s premises, providing peace of mind and meeting compliance standards without compromising performance or innovation.
Maximize model utilization and control costs
AI application costs can quickly spiral out of control without proper governance over how models are accessed, deployed, and consumed. Without clear monitoring and management, organizations may face unexpected expenses due to overuse, inefficient resource allocation, or unchecked scaling of AI tools. Through its powerful AI middleware, Tanzu AI Solutions increases access control and optimizes model usage for improved cost control.
- Rate Limiting and Usage Controls: enables secure and efficient operations by implementing robust guardrails. These guardrails enforce strict rate limits on model queries, preventing unauthorized or excessive usage that could strain resources or compromise system performance. By setting these boundaries, Tanzu Platform not only protects the integrity of the AI models but also promotes controlled usage, enabling organizations to maintain optimal functionality and reliability.
- Token Usage Monitoring: Gain real-time insights into token consumption to help track and manage spending effectively. This allows better budget control while maintaining optimal application performance, allowing teams to identify patterns, reduce waste, and plan usage more efficiently.
Keep up with the latest model changes
LLMs are being released at an alarming rate, with new models constantly emerging offering improvements in size, efficiency and performance. Some models prioritize sheer scale and power, while others focus on being smaller and more cost-effective. In all likelihood, an application estate will need a mix of both large and small models to cover different use cases, or they might need multiple models in the case of agentic patterns.
Because of model proliferation and model optimization efforts in the market, organizations need to continuously evaluate model accuracy and costs to ensure they are meeting performance and ROI expectations. When utilizing multiple models across their estate, organizations will experience integration challenges with the proliferation of APIs. They will also run into adaptation challenges as foundation models get trained on new data and they need to swap models to improve performance. The AI middleware in Tanzu AI solutions enables organizations to code once, and not have to make changes to code to accommodate model swapping.
- Consistent API: by providing a standardized implementation of the OpenAI API across all models, enterprises can test new model versions or use completely different model-as-a-service vendors, without the need to make application level code changes. While the “OpenAI format” might seem universal already, in practice it isn’t. It is time intensive to repeatedly make these swaps over an entire application estate. Introducing an abstraction that creates consistent swapping, sensible error messages, and other developer support capabilities enables organizations to keep up with the rate of continuous model improvement, and reduces vendor lock-in. Check out our recent Cloud Foundry Weekly podcast that talks more about model flexibility.
Tanzu AI Solutions with AI middleware helps enterprises extend to integrate AI at scale
AI middleware is not just a technical necessity but a strategic enabler for building scalable and safer, AI-powered applications. As AI continues to evolve, so too must the platforms and infrastructure that support it.
Tanzu has introduced a fresh take on a trusted software category: middleware. Our AI middleware capability in Tanzu AI Solutions (included in Tanzu Platform) integrates seamlessly into enterprise ecosystems, bridging the gap between AI innovation and operational compliance while empowering platform engineers and developers with tools to deliver intelligent apps faster and more safely.
To recap, here’s how AI middleware in Tanzu AI Solutions seamlessly integrates with development patterns to keep AI application development and delivery running smoothly and securely:
- Model Brokering: Presents a standard OpenAI API, no matter what model is used, to allow developers to integrate once.
- Secure Authentication: Provides token-based authentication for API calls, delivering centralized access management.
- Governance: Data science teams curate models and platform teams can control how those models are accessed and define how much is consumed.
- Observability: Integration with metrics and evaluation collection to track app-level spend, token limits, content and output quality.
Start Your AI Journey Today
Tanzu Platform (and Tanzu Application Service) customers can use the GenAI service as part of their existing license so you can get started adding AI models to your applications today!
If you would like to discuss your AI apps use case with the Tanzu team, please reach out to us.
