I recently got a first-hand look at how vulnerable interconnected computer systems around the world can be—even systems like industrial PCs in manufacturing that were once thought to be secured in an air-gapped network. It was just under two months ago when a bug in widely used security software and a bad update brought companies around the world to their knees. For example, airlines could not check in passengers, causing so many stranded travelers. Quite a few of my colleagues were stuck on business trips and could not get home for a few days. It was complete chaos at the airport. A colleague sent me this photo:
This could happen to anyone, not only software vendors delivering patches to their customers, but also internal development teams releasing new functionalities to production, especially when the whole industry encourages quicker software releases and more developers embrace CI/CD.
A few weeks ago I visited one of our manufacturing customers, who told me they were affected by this same update that caused a global impact to the travel industry. This particular customer experienced outages in several of their factories. It’s easy to overlook that many factories still depend heavily on industrial PCs (IPCs) running Windows OS for tasks ranging from engineering tools and protocol translation to directly interacting with the production process. Downtime of these IPCs may impact a critical process, resulting in a production line shutdown.
The customer told me this is a new problem because in the past they did not typically upgrade the OS or applications. After all, these devices were air-gapped, meaning they did not have direct access to the Internet. Things have changed a lot due to Industry 4.0 initiatives. These IPCs running Windows are becoming more connected, and now in addition to bad software patches that can bring the whole system down, malware or ransomware can also cause the production line to stop if it affects the right IPCs.
The whole manufacturing industry already places significant focus on security, primarily through preventative measures like anti-virus, firewall, IDS/IPS, and specialized OT security solutions. However, what about a recovery strategy? How would you quickly recover from a bad software update, malware, or ransomware attack on IPCs, especially when a compromised system could bring the production line to a halt and cause a significant loss of revenue?
The benefits of virtualization
Back to the conversation with my manufacturing customer. I said, this is a solved problem in IT. I recommended virtualizing these IPCs—in other words, running the Windows system as a VM until you can containerize your factory applications. By replacing the IPCs with Industrial Thin Clients, the operating system and applications could run as VMs on your edge servers within the factory.
One of the most powerful, yet often overlooked capabilities in virtualization is snapshot and rollback. You keep a copy of the last known good state of the VM and revert to it when something bad happens. Additionally, because virtualization abstracts the hardware from the OS, it enables you to run older versions of Windows required for factory operations on new server hardware. This customer found this idea interesting and mentioned as they build a new factory, this approach may be exactly what we need. They said, “I want to be able to do my tasks remotely, but we have been doing things the same way. Maybe we need to change.”
How Audi is modernizing factory operations
Last week I gave a talk with Audi AG at the VMware Explore 2024 Las Vegas event. Audi Edge Cloud 4 Production (EC4P) follows a very similar idea, and they gain benefits that go beyond just reducing the attack surface. The concept of servers running virtualization workloads is not new, but edge servers supporting both production workloads—including non-real-time workloads like those running on IPCs and real-time workloads— is new.
I believe the capabilities that we are continuously developing for VMware Edge Compute Stack give manufacturers better security, more flexibility, and agility to support evolving demand and use cases as the industry continues to embrace more digitalization and bring more technologies into shop floor operations.
If you want to learn more about how manufacturing can adopt edge computing and edge technologies and how we can help, reach out to us at [email protected].
