VMware Cloud Provider

API Guide for Container Service Extension 4.2

Introduction

The Container Service Extension(CSE) 4.2 has been released with VMware Cloud Director Extension for Tanzu Mission Control-Self-Managed(TMC-SM for VCD). You can find announcement blogs for CSE here and for TMC-SM for VCD here. We reviewed API guide for 4.0 release in the past. While the VCD API is supported, there’s still a need for a detailed guide on using the Cluster API with VMware Cloud Director (VCD) to create and manage Tanzu Kubernetes Grid (TKG) clusters. Generating the necessary payload for these operations can be involved process, as it requires manually adjusting the payload produced by the Cluster API. This blog post will simplify this process, providing a step-by-step guide to help our customers create the correct payload for their VCD infrastructure, ensuring they can effectively integrate and manage their TKG clusters.

Version Support

This API guide is applicable to clusters created by CSE 4.2 Tanzu Kubernetes Clusters.

The existing prerequisites for customers to create TKG clusters in their organizations also apply to the automation flow. These prerequisites are summarized here and can be found in the official documentation to onboard Provider and Tenant Admin users. The following sections provide an overview of the requirements for both cloud provider administrators and Tenant Admin users.

For Cloud Provider Admin

The Steps to onboard the customers is demonstrated in this video and documented here. Once customer organization and its users are onboarded, they can use next section to use APIs, or consume it to create automated Cluster operations.

Cloud Provider Admin Steps

As a quick summary following steps are expected to be performed by cloud provider to onboard and prepare the customer:

  1. CSE 4.2 Interoperability Matrix
  2. Allow necessary communication for CSE server
  3. Start CSE server (Refer Demo and Official Documentation)
  4. Onboard the customer ((Refer Demo and Official Documentation)

When the cloud provider has onboarded the customer onto the Container Service Extension, the organization administrator must create and assign users with the capability to create and manage TKG clusters for the customer organization. This documentation outlines the procedure for creating a user with the “Kubernetes cluster author” role within the tenant organization.

Customer Org Admin/Kubernetes Cluster Author Steps

When the cloud provider has onboarded the customer onto the Container Service Extension, the organization administrator must create and assign users with the capability to create and manage TKG clusters for the customer organization. This documentation outlines the procedure for creating a user with the “Kubernetes cluster author” role within the tenant organization.

In this section we assume that the user “c1kubadmin” has obtained the necessary resources and access within the customer organization to execute Kubernetes cluster operations.

Generate ‘capiyaml’ payload

1. Collect VCD Infrastructure and Kubernetes Cluster details

This Operation requires following information for VCD tenant portal. The following table describes what information is required to generate the payload. The right column describes example values used as reference in this blog post.

Infrastructure details:

InputExample Value for this blog
VCD_SITEhttps://vcd01.vcf.corp.local
VCD_ORGANIZATIONCustomer1
VCD_ORGANIZATION_VDCc1
VCD_ORGANIZATION_VDC_NETWORKnetwork1
VCD_CATALOGcse shared catalog
Table 1 – Infrastructure details

InputExample Value for this blog
VCD_TEMPLATE_NAMEUbuntu 20.04 and Kubernetes v1.27.5+vmware.1
VCD_CONTROL_PLANE_SIZING_POLICYTKG small
VCD_CONTROL_PLANE_STORAGE_PROFILElab-shared-storage
VCD_CONTROL_PLANE_PLACEMENT_POLICYcapacity
VCD_WORKER_STORAGE_PROFILElab-shared-storage
CONTROL_PLANE_MACHINE_COUNT1
WORKER_MACHINE_COUNT1
VCD_REFRESH_TOKEN_B64“MHB1d0tXSllVb2twU2tGRjExNllCNGZnVWZqTm5UZ2U=”
*Encode your API token to base_64
Ref VMware Doc to Generate token before transforming it to Base64
Table 2 – Kubernetes Cluster Properties

2. Install tools on local machine to generate the capiyaml

Once the tenant user has collected all the information, user will have to install following components such as  Clusterctl 1.4.0, Kind(0.17.0), and Docker (20.10.21) on end user’s machine. The following step requires above collected information, and not the access to VCD Infrastructure to generate capiyaml payload.

3. Copy TKG Custom Resource Files locally

The CSE 4.2 release supports and recommends CAPVCD version 1.2.0. Copy TKG CRS Files locally. Follow the instructions from the official capvcd documentation to download files and generate capi yaml here. For completeness, the steps are described below. 

In case the desired Kubernetes version from a particular TKG version is missing (TKG version) is missing from the folder, please use the script to collect product versions using the script here and create files based on the instructions. The script is located here. The following table provides supported list of etcd, coredns, tkg, tkr versions for CSE 4.2 release.

Kubernetes Versionetcd ImageTagCoreDNS ImageTagComplete Unique VersionOVATKG Product VersionTKr Version
v1.27.5+vmware.1v3.5.7_vmware.6v1.10.1_vmware.7v1.27.5+vmware.1-tkg.1ubuntu-2004-kube-v1.27.5+vmware.1-tkg.1-0eb96d2f9f4f705ac87c40633d4b69st.ova2.4v1.27.5—vmware.1-tkg.1
v1.26.8+vmware.1v3.5.6_vmware.20v1.10.1_vmware.7v1.26.8+vmware.1-tkg.1ubuntu-2004-kube-v1.26.8+vmware.1-tkg.1-b8c57a6c8c98d227f74e7b1a9eef27st.ova2.4v1.26.8—vmware.1-tkg.1
v1.25.13+vmware.1v3.5.6_vmware.20v1.10.1_vmware.7v1.25.13+vmware.1-tkg.1ubuntu-2004-kube-v1.25.13+vmware.1-tkg.1-0031669997707d1c644156b8fc31ebst.ova2.4v1.25.13—vmware.1-tkg.1
Table 3 Kubernetes, etcd, Coredns , and other component versions for Tanzu Kubernetes Versions for CSE 4.2

Create a folder structure for CAPVCD in your working directory. 

Copy the contents from templates directory to ~/infrastructure-vcd/v1.2.0/ . 

Copy metadata.yaml to ~/infrastructure-vcd/v1.2.0/

After copying all files the folder structure should look as follows:

4. Compose ‘clusterctl’ yaml to generate CAPI yaml

Copy the ~/infrastructure-vcd/v1.2.0/clusterctl.yaml to ~/.cluster-api/clusterctl.yaml. clusterctl command uses clusterctl.yaml from ~/.cluster-api/clusterctl.yaml to create the capiyaml payload. Update infrastructure details from step 1 in this ~/.cluster-api/clusterctl.yaml for coming steps. 

Update the providers.url in ~/.cluster-api/clusterctl.yaml to ~/infrastructure-vcd/v1.2.0/infrastructure-components.yaml. 

At this point your ~/cluster-api/clusterctl.yaml values should look as follows:

5. Create kind cluster and clusterctl to generate capi.yaml for CSE API payload

6. Initialize clusterctl and Generate capiyaml on the kind cluster

clusterctl init --core cluster-api:v1.4.0 -b kubeadm:v1.4.0 -c kubeadm:v1.4.0 -i vcd:v1.2.0

clusterctl generate cluster thursday31 --kubernetes-version v1.27.5-crs > thursday31.yaml

7. Update “Kind: Cluster” objects to reflect cluster type “TKG”

At this point, the capiyaml is ready to be consumed by VCD APIs to perform various operations. For verification, make sure cluster name, namespace values are consistent. Copy the content of capiyaml to generate jsonstring using similar tool as here. Or execute following command with ‘jq’.

jq -Rs '.' < capiyaml_filename.yaml

Please note that non-supported TKG versions can result in unexpected behavior when performing API calls.

Cluster Operations with VCD APIs

List Clusters

List all clusters in the customer organization. for CSE 4.2 release the CAPVCD version is 1.2, and for the API

Get Clusters

Filter Clusters by name

Get Cluster by ID

Get Kubeconfig of the cluster

In CSE 4.2, entity.status.capvcd.private.kubeconfig is marked as a secure field. Regular GET call will not have this property. Users need to invoke a behavior that decrypts the content.

Execute the Behavior invocation API

Fetch the property from the header “Location”, and associated task href

Perform GET call to return the full decrypted contents of the RDE as part of task Result.

Create a Cluster

With CSE 4.2 release, secret object in the capiYaml is an optional field. Alternatively the users can use “spec.vcdKe.secure.apiToken” in the API call. This approach gives advantage that the apiToken is marked as a secure field and will not be part of regular GET payload.

Resize a Cluster

  • Fetch the Cluster ID("id": "urn:vcloud:entity:vmware:capvcdCluster:<ID>) from the above API call’s output.
  • Copy the complete output of the API response.
  • Notedown eTag Value from API response header
  • Modify “capiyaml” with following values:
    • To resize Control Plane VMs Modify kubeadmcontrolplane.spec.replicas with desired number of control plane vms. Note only odd numbers of control plane are supported.
    • To resize Worker Plane VMS Modify MachineDeployment.spec.replicas with desired number of worker plane VMs
  • While performing the PUT API call, ensure to include fetched eTag value as If-Match

Upgrade a Cluster

To Upgrade a cluster, Provider admin needs to publish desired the Tanzu Kubernetes templates to the customer organization in catalog used by Container Service Extension.

collect the GET API response for the cluster to be upgraded as follows:

  • Fetch the Cluster ID("id": "urn:vcloud:entity:vmware:capvcdCluster:<ID>) from the above API call’s output.
  • Copy the complete output of the API response.
  • Notedown eTag Value from API response header
  • The customer user performing cluster upgrade will require access to Table 3 information. Modify Following values matching the target TKG version.

While performing the PUT API call, ensure to include fetched eTag value as If-Match

Delete a Cluster

Insert below key/value pair to mark cluster as delete or force delete. 

"isVCDKECluster": true, –Add or modify the this field to delete the cluster

 "markForDelete": true, — Add or modify the this field to force delete the cluster

Example snippet of the API call is as below: 

For actual operation, user must take entire output of GET API call with ETAG value as described above. 

Recommendation for API Usage during automation

DO NOT hardcode API urls with RDE versions. ALWAYS parameterize RDE versions. For example:

Ensure to declare 1.2.0 as a variable. This will ensure easy API client upgrades to future versions of CSE.

Ensure the API client code ignores any unknown/additional properties while unmarshaling the API response

Summary

To summarize, we looked at CRUD operations for a Tanzu Kubernetes clusters on VMware Cloud Director platform using VMware Cloud Director supported APIs. Please feel free to checkout other resources for Container Service Extension as follows:

  1. Generate API token using VMware Cloud Director
  2. CSE 4.0 Official Documentation
  3. Cluster API for VMware Cloud Director Platform official Documentation

Sachi Bhatt

Sachi is a Technical Product Manager at VMware in Cloud Services Business Unit.

Related Articles