Why is Continuous Validation important?
VMware Marketplace is a one stop-shop for VMware customers to discover, try and deploy various third-party and open-source solutions onto their VMware environments. All deployable assets on VMware Marketplace are pre-tested on their respective VMware environments. This empowers users to deploy these software offerings with confidence, knowing they have access to proven, validated solutions. However, with increased industry focus on ‘earlier, more often’ software release cycles, these marketplace assets run the risk of quickly becoming incompatible with the latest version of their respective deployment platforms.
VMware is excited to offer continuous and automated validation for all Tanzu ecosystem solutions on VMware Marketplace! Read more about the announcement “VMware Introduces Continuous and Automated Validation for Its ISV and Ecosystem Solutions”
What is Continuous Validation?
VMware Marketplace now offers automatic platform compatibility and functional checks on our published, cloud native, contained based solutions. This automatic process validates publisher solutions on pre-selected target platforms (i.e Tanzu Kubernetes Grid) both during the publishing process and later on, when newer versions of the underlying target platform are released.
For Marketplace users, this guarantees that the latest version of a chosen solution will work as expected on the latest version of their chosen deployment platform at any point in time. For Publishers, this means taking advantage of real time validation of the latest version of their apps on the latest version of Tanzu Kubernetes Grid, completely eliminating the need for manual compatibility testing against every new update! Plus, continuous validation also gives publishers the flexibility to define custom test scenarios on their apps for an extra boost of confidence that their app will work as expected – every time.
In this first phase, continuous validation as a feature is available for Tanzu Kubernetes Grid (TKG) platforms and Helm based Kubernetes solutions. Additional VMware-based deployment platforms will be enabled with this feature soon.
Figure 1: Continuous Validation Process Flow
There are two main validation types:
Platform Validation
Performs application-independent testing on predefined target platforms, focusing on generic, application lifecycle and platform tests. For example, VMware Marketplace will check that the application comes up successfully, it self-heals, persists data correctly and so on, during deployment.
Application Validation
These are application-specific tests provided by the publisher which range from probing the live web application to other lower-level checks that are executed inside the target container.
How does it work?
The Continuous Validation Engine input consists of a) The application to be tested along with any additional deployment options, b) The platform where the system should run the tests on and additionally for Application Tests c) The custom test suite in the form of a container image along with optional test run options. The engine will provision the desired target platform, deploy the application and run either a predefined common test-suite for platform tests, or in the case of application tests, the custom test suite provided by the publisher. The output of the process will contain enough information to troubleshoot and visualize the performed validation in the form of logs and XUnit reports.
Figure 2: Components of Continuous Validation
This engine is integrated with VMware Marketplace in two areas, during solution onboarding and the self-service validation section under the existing published solutions.
Solution onboarding
Whenever a publisher publishes a new solution or a new version of an existing solution on the Marketplace, the Continuous Validation Engine will automatically trigger a platform validation consisting of deploying the solution plus running a generic set of tests as described above. Publishers have the option to provide custom Helm Chart install options which will be used by the Engine to deploy the app before running the tests. Successfully passing the platform validation step is mandatory for a solution to be approved by the VMware Marketplace. Application or functional validation are optional and can be manually setup and triggered as shown next.
Figure 3: Solution Onboarding
Self-Service Validation Section
Additionally, publishers now have access to a new section on VMware Marketplace where they can define, run, visualize and troubleshoot new and existing platform and application validation for their solutions.
Figure 4: New Marketplace UI with platform and app test features
The user interface provides a way to set up both platform and application tests where the publisher can choose the desired target platform, the application run options and the test suite if applicable.
Figure 5: Platform Validation setup
Figure 6: Application Validation setup
Once set, the publisher has the option to run those tests manually but more importantly, the continuous validation engine will make sure to automatically re-run platform and application tests if there is a new release of the published solution or of the target deployment platform.
Conclusion & Next Steps
Continuous and automated validation is a win-win for both the publishers offering containerized solutions on Tanzu as well as Marketplace users! VMware Marketplace is working to extend this capability to other VMware deployment platforms as well as other form factors such as Virtual Machines and Operators – continue to watch this space for more exciting updates!
Learn More
Interested in learning more about VMware Marketplace? Read our introductory blog, follow us on twitter or visit the catalog of 2000+ solutions. If you have any questions, please reach out to us at [email protected].
