Why You’re Reading This (and Why It’s Not Too Late)
The deadline: March 31, 2026 — three days from now.
The situation: Ingress NGINX stops receiving security patches forever.
The reality: You’re still running it in production.
In November 2025, the Kubernetes project announced Ingress NGINX’s retirement. If you waited until the last minute, you’re not alone; migration activity spiked 300% in February-March 2026 as teams raced to beat the deadline.
I migrated a multi-app demo suite (Bookstore, Reader, Chatbot) across two VMware vSphere Kubernetes Service (VKS) clusters with zero downtime in under 8 hours. One cluster uses Helm, the other Kustomize + ArgoCD. This is the exact playbook I followed, including the three “gotchas” that ate hours of debugging time.
The replacement: Gateway API — the official successor to Ingress, with 20+ production-ready implementations in 2026. It’s more expressive, more portable, and eliminates vendor lock-in through annotation sprawl.
Why Istio? VKS ships it as a standard package (add-on), which means no custom installation complexity. Plus, it’s the #1 Gateway API implementation by adoption and opens the door to service mesh capabilities later. For a detailed walk-through of Istio on VKS, see our previous guide. The alternatives (Cilium, Envoy Gateway, Traefik) are all viable, but Istio had the shortest path for VKS environments and i was intrigued by this analysis.
What Changed
The migration touched three layers: prerequisites (cluster-level), Helm chart templates, and Kustomize manifests. Here’s exactly what changed and why.
Before: Ingress NGINX
Each app had an Ingress resource with nginx-specific annotations:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: bookstore-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/ssl-redirect: "false" nginx.ingress.kubernetes.io/proxy-body-size: "10m" nginx.ingress.kubernetes.io/proxy-read-timeout: "60" nginx.ingress.kubernetes.io/proxy-send-timeout: "60" spec: ingressClassName: nginx rules: - host: bookstore-test.corp.vmbeans.com http: paths: - path: / pathType: Prefix backend: service: name: app-service port: number: 80 |
Every behavior customization was a vendor-specific annotation. Switch controllers? Rewrite all annotations.
After: Gateway API + Istio
A shared Gateway resource defines the entry point, and each app gets a portable HTTPRoute:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# Gateway -- one per cluster, shared by all apps apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: demo-gateway annotations: cert-manager.io/cluster-issuer: demo-issuer spec: gatewayClassName: istio listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: https port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - name: demo-tls allowedRoutes: namespaces: from: All |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# HTTPRoute -- one per app, references the shared Gateway apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: bookstore-route spec: parentRefs: - name: demo-gateway namespace: bookstore hostnames: - "bookstore-test.corp.vmbeans.com" rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: app-service port: 80 |
No vendor annotations. The HTTPRoute is portable across any Gateway API implementation. The Gateway is where you pick your controller (gatewayClassName: istio), and that’s the only vendor-specific line.
TLS: Automatic with cert-manager
Instead of manually managing certificates, cert-manager watches the Gateway and issues certificates automatically:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: demo-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: demo-tls spec: secretName: demo-tls duration: 2160h # 90 days renewBefore: 720h # Renew 30 days before expiration issuerRef: name: demo-issuer kind: ClusterIssuer dnsNames: - bookstore-test.corp.vmbeans.com - reader-test.corp.vmbeans.com - chatbot-test.corp.vmbeans.com |
For this demo environment I used a self-signed issuer. In production, swap selfSigned: {} for a Let’s Encrypt or corporate CA issuer. The rest stays the same.
The Helm Chart: Backward-Compatible Toggle
I didn’t want to break the chart for anyone still on Ingress NGINX. The solution: a global.gatewayAPI.enabled toggle that defaults to true but can be flipped to false for legacy mode.
values.yaml:
|
1 2 3 4 5 6 7 |
global: gatewayAPI: enabled: true className: istio tls: enabled: true issuer: demo-issuer |
Ingress templates (existing files, wrapped in a conditional):
|
1 2 3 4 5 |
{{- if not .Values.global.gatewayAPI.enabled }} apiVersion: networking.k8s.io/v1 kind: Ingress # ... existing Ingress resource unchanged ... {{- end }} |
HTTPRoute templates (new files):
|
1 2 3 4 5 |
{{- if .Values.global.gatewayAPI.enabled }} apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute # ... new HTTPRoute resource ... {{- end }} |
Result: helm template produces exactly the right resources for each mode:
| Mode | Gateway | HTTPRoute | Certificate | Ingress |
| Gateway API (default) | 1 | 3 | 1 | 0 |
| Legacy Ingress | 0 | 0 | 0 | 3 |
To fall back to Ingress NGINX on any cluster:
|
1 2 3 4 5 |
helm install demo ./helm/demo-suite \ --set global.domain=example.com \ --set global.gatewayAPI.enabled=false \ --set global.tls.enabled=false \ --set ingress-nginx.enabled=true |
Step by Step: How I Applied the Migration
1. Install Prerequisites
On VKS, Istio and cert-manager are available as standard packages (add-ons). I installed them via VCF Automation UI in Kubernetes Management Add-ons (you can also use AddonInstall resources on the Supervisor API server). The key configuration change: set gateways.ingress.enabled: true in the Istio add-on YAML. Without this, Istio installs the control plane but not the ingress gateway component.
For the cert-manager add-on, the defaults work out of the box.
Gateway API CRDs are not included in the Istio add-on and must be installed separately on each workload cluster:
|
1 2 |
kubectl apply --server-side \ -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml" |
Critical compatibility note: Istio 1.28/1.29 requires Gateway API CRDs v1.4.x. Installing v1.5.x causes istiod to crash due to API field mismatches (istio/istio#59443). This is a known issue across multiple Gateway API implementations in early 2026 — always pin the CRD version in production.
For non-VKS clusters, I created a one-shot script (scripts/install-prerequisites.sh) that installs Istio, cert-manager, and the CRDs via Helm with pinned versions.
2. Fix Pod Security Admission
This was the “gotcha” that ate a bit of my time.
VKS namespaces default to the restricted Pod Security Standard. Istio’s auto-provisioned gateway proxy pods don’t include a seccompProfile in their security context, so Kubernetes rejects them:
|
1 2 3 4 |
Error creating: pods "demo-gateway-istio-..." is forbidden: violates PodSecurity "restricted:latest": seccompProfile (pod or container "istio-proxy" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") |
The fix: relax the PSA policy on the namespace where the Gateway resource lives:
|
1 2 3 4 5 |
kubectl label namespace bookstore \ pod-security.kubernetes.io/enforce=privileged \ pod-security.kubernetes.io/warn=privileged \ pod-security.kubernetes.io/audit=privileged \ --overwrite |
Important: After applying the labels, the ReplicaSet is stuck in exponential backoff from the earlier failures. You need to restart the deployment to force a retry:
|
1 |
kubectl rollout restart deploy/demo-gateway-istio -n bookstore |
This is something the Istio project should fix upstream (adding seccompProfile: RuntimeDefault to the gateway proxy template), but for now, the namespace label is the workaround.
3. Helm Upgrade (cluster-01)
|
1 2 3 4 5 6 7 8 9 10 |
helm upgrade demo-test ./helm/demo-suite \ -f ./helm/demo-suite/values-small.yaml \ --set global.domain=corp.vmbeans.com \ --set global.storageClassName=vsan-default-storage-policy \ --set bookstore.ingress.host=bookstore-test.corp.vmbeans.com \ --set reader.ingress.host=reader-test.corp.vmbeans.com \ --set bookstore.readerBrowserURL=https://reader-test.corp.vmbeans.com \ --set ingress-nginx.enabled=false \ --set chatbot.enabled=false \ --timeout 10m |
Key changes from the previous install command:
- Removed ingress-nginx.enabled=true (Gateway API replaces it)
- Changed readerBrowserURL from http:// to https:// (TLS is now on)
- global.gatewayAPI.enabled and global.tls.enabled default to true, no override needed
4. ArgoCD Sync (cluster-02)
For the Kustomize/ArgoCD cluster, the Kustomize base was updated to reference gateway.yaml, httproute.yaml, and tls.yaml instead of ingress.yaml. The overlay patches were updated to set the correct hostname.
|
1 2 |
argocd app set bookstore --sync-policy automated --auto-prune --self-heal argocd app sync bookstore |
ArgoCD automatically removed the old Ingress resource and created the Gateway, HTTPRoute, Certificate, and ClusterIssuer. The sync took 6 seconds.
5. Update DNS
The Istio gateway proxy gets its own LoadBalancer IP, which will be different from the old ingress-nginx IP. After the upgrade:
|
1 2 |
</code># Get the new gateway IP kubectl get svc demo-gateway-istio -n bookstore -o jsonpath='{.status.loadBalancer.ingress[0].ip}' |
Update your DNS A records to point to the new IP.
The Result
|
1 2 3 4 5 6 7 8 9 10 |
$ kubectl get gateway,httproute,certificate -A NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE bookstore gateway/demo-gateway istio 32.32.0.22 True 5m NAMESPACE NAME HOSTNAMES AGE bookstore httproute/bookstore-route ["bookstore-test.corp.vmbeans.com"] 5m reader httproute/reader-route ["reader-test.corp.vmbeans.com"] 5m NAMESPACE NAME READY SECRET AGE bookstore certificate/demo-tls True demo-tls 5m |
The app loads over HTTPS with a cert-manager-issued certificate. The browser shows the padlock (with a self-signed warning, expected for a lab). All traffic flows through the Istio gateway proxy.
Gotchas and Lessons Learned (the Time-Savers)
These are the issues that cost me hours. Learn from my pain:
1. Gateway API CRD version compatibility is critical. Istio 1.28/1.29 crashes with Gateway API CRDs v1.5. Pin to v1.4.0. This affected multiple implementations (Cilium, Envoy Gateway) in early 2026 — it’s not Istio-specific.
2. VKS Pod Security Admission blocks Istio gateway proxies. The auto-provisioned pods lack seccompProfile, triggering restricted policy violations. Label the namespace as privileged. This is a VKS default; vanilla Kubernetes typically uses baseline or privileged by default.
3. The ReplicaSet backoff trap. After fixing PSA labels, the ReplicaSet doesn’t retry immediately. It’s in exponential backoff waiting 2-5 minutes between attempts. kubectl rollout restart is required to force a new attempt immediately. Without this, you’ll think the fix didn’t work.
4. Istio VKS add-on ships with gateways.ingress.enabled: false. You must explicitly enable it in the AddonInstall YAML or there’s no gateway proxy to handle traffic. The control plane installs fine but nothing actually routes.
5. Don’t install Istio or cert-manager as Helm subcharts. Both projects strongly discourage it due to template name collisions and CRD lifecycle issues (they need cluster-scoped installation). Install them as cluster-level components separately, then reference them from your app chart. This is documented best practice for both projects.
6. DNS changes are required. The Istio gateway proxy gets a new LoadBalancer IP. The old ingress-nginx IP becomes stale. Plan for the DNS cutover. Use a low TTL (300s) during migration so you can roll back quickly if needed.
7. The backward-compatible toggle is worth the effort. Being able to flip between Gateway API and Ingress NGINX with a single –set flag makes the migration reversible and lets different clusters run different networking stacks during the transition. This saved me when I discovered the CRD version issue mid-migration.
8. cert-manager production best practices matter. Run 2+ controller replicas and 3+ webhook replicas for high availability. The webhook especially — if it’s down, all Certificate operations fail cluster-wide. Set explicit duration: 2160h (90d) and renewBefore: 720h (30d) to give ample renewal time.
File Changes Summary
| File | Change |
| VKS AddOnInstall | New — installs Istio, cert-manager |
| charts/bookstore/templates/gateway.yaml | New — shared Gateway resource (HTTP + HTTPS listeners) |
| charts/bookstore/templates/httproute.yaml | New — bookstore HTTPRoute |
| charts/reader/templates/httproute.yaml | New — reader HTTPRoute |
| charts/chatbot/templates/httproute.yaml | New — chatbot HTTPRoute |
| charts/bookstore/templates/tls.yaml | New — ClusterIssuer + Certificate |
| charts/bookstore/templates/ingress.yaml | Modified — wrapped in {{- if not .Values.global.gatewayAPI.enabled }} |
| charts/reader/templates/ingress.yaml | Modified — added gatewayAPI.enabled conditional |
| charts/chatbot/templates/ingress.yaml | Modified — added gatewayAPI.enabled conditional |
| helm/demo-suite/values.yaml | Modified — added global.gatewayAPI and global.tls sections |
| helm/demo-suite/values-small.yaml | Modified — updated comments |
| kubernetes/base/gateway.yaml | New — Kustomize base Gateway |
| kubernetes/base/httproute.yaml | New — Kustomize base HTTPRoute |
| kubernetes/base/tls.yaml | New — Kustomize base ClusterIssuer + Certificate |
| kubernetes/base/kustomization.yaml | Modified — replaced ingress.yaml with gateway/httproute/tls |
| kubernetes/overlays/prod/httproute-patch.yaml | New — hostname override for prod |
| kubernetes/overlays/prod/tls-patch.yaml | New — DNS name override for prod |
| kubernetes/overlays/dev/httproute-patch.yaml | New — hostname override for dev |
| kubernetes/overlays/dev/tls-patch.yaml | New — DNS name override for dev |
| kubernetes/overlays/*/kustomization.yaml | Modified — replaced ingress-patch with httproute-patch + tls-patch |
The Bottom Line: Can You Really Do This in One Day?
Yes. Here’s the realistic timeline:
- 1-2 hours: Install prerequisites (Istio, cert-manager, Gateway API CRDs), debug PSA issues, get one test app working.
- 2-3 hours: Build the backward-compatible Helm toggle pattern, test rollback scenarios.
- 1-2 hours: Migrate remaining apps, update DNS, parallel testing.
- 1-2 hours: Monitor production traffic, finalize documentation, cleanup old Ingress resources.
Total time investment: 5-9 hours depending on complexity and cluster count.
The real timeline: Most of the time is spent waiting — for DNS propagation, for cert-manager to issue certificates, for ArgoCD sync cycles. The actual hands-on work is concentrated into bursts.
Zero-downtime migration: The key is running both stacks (Ingress NGINX + Gateway API) in parallel on separate LoadBalancer IPs. You cut over via DNS, not by replacing resources. If something breaks, you point DNS back to the old IP and troubleshoot.
Is it worth it? After March 31, you’re on your own for CVEs. The ingress-nginx project is archived, no more security patches. Gateway API is the Kubernetes community’s endorsed path forward — you’re migrating to a first-class API, not a third-party controller. Do it now or do it under pressure when the first post-retirement CVE drops.
If you’re reading this on March 28-31, you have time. Follow this playbook, adapt the PSA labels to your environment, pin your CRD version, and get it done. The alternative is running unsupported infrastructure the moment April starts.
What’s Next
- Swap the self-signed ClusterIssuer for Let’s Encrypt or internal CA in production:
- Use separate Issuers for dev/staging/prod environments
- Set rotationPolicy: Always on Certificates for enhanced security
- Monitor certificate expiration with Prometheus metrics: certmanager_certificate_expiration_timestamp_seconds
- Explore Istio service mesh features now that the control plane is in place (mTLS, traffic policies, observability with Kiali). For more on VKS add-ons and extended capabilities, see our comprehensive guide.
- Consider implementing ReferenceGrants across namespaces for better security boundaries
Discover more from VMware Cloud Foundation (VCF) Blog
Subscribe to get the latest posts sent to your email.
