Learn why Concourse was key in Cerner’s digital transformation journey.
The market for electronic medical records (EMRs) is a lucrative one and competition is fierce. The market stands at around $28 billion and no one vendor holds more than a fifth of the market. There are hundreds of EMR vendors, each looking for ways to differentiate themselves and grow market share.
Cerner, one such EMR vendor, is betting on speed to market.
“Our market is getting extremely competitive due to regulations and expectations,” said Rob Rose, a lead architect at Cerner. “A lot of the hospitals domestically have gone digital already [and] they have electronic records, so at this point a competitive advantage is to actually differentiate by delivering [software] faster.”
That means developing and releasing software to production in smaller batches more frequently. Of course, many firms are adopting this model. But it’s only really effective when you listen more closely to users, and letting their feedback guide the features and applications that actually get built. This is a rare combination in the EMR market. And here’s proof: frustrated clinicians report spending nearly twice as much time struggling with clunky EMR software than interacting with patients. In a crowded market, that’s a chance for Cerner to stand out.
Cerner started to modernize its development process a few years ago. Software engineers adopted a number of agile development practices, including writing user stories to track new feature development. They also reorganized into cross-functional teams. As a result, the pace of feature development at Cerner has increased significantly. But the company was still stuck delivering new software to production once or twice a year. New features reached code completion, but then just sat in a staging environment until the next release.
How did this sticky situation come about? While Cerner modernized its software development process, it hadn’t revamped its IT infrastructure and operations processes. Cerner’s ops team still maintained its infrastructure of virtual machines and bare metal servers individually. Pushing software to production required a coordinated, manual effort.
“That can create some problems when it comes to continuous deployment,” said Greg Meyer, a distinguished engineer at Cerner, speaking at SpringOne Platform in December. “We got a lot of things from our operations side like, ’You can’t deploy anything because we’ve got to go do an OS upgrade, or we’ve got to go do a CDE patch, or its Christmas, everybody’s taking off so we can’t do any deployments for the next four weeks.’”
In order to close the last mile to production, Cerner needed a platform to sit on top of its virtual machines. This approach would allow its ops team to significantly increase efficiency, by reducing the manual steps required to push software to production. These requirements led the company to select and deploy Pivotal Cloud Foundry, a modern platform for developing and running cloud-native applications.
Livestock Versus Pets
Pivotal Cloud Foundry is well known for boosting developer productivity by abstracting away the underlying complexity of infrastructure. The product also supports operational efficiency, by enabling ops teams to adopt “immutable infrastructure” principles. Here, operators treat infrastructure as a uniform fleet of machines, rather than individual servers and VMs. Put another way, PCF allows ops teams to treat infrastructure as “livestock” rather than “pets.”
When you treat infrastructure as pets, ops teams tend to each server or VM with tender, loving care. It can take weeks or even months to procure new VMs. Each one is manually maintained and monitored. And if a VM fails, everything comes to a halt until the ops team revives it. That’s a great way to treat a your cat or dog. But it’s not conducive to high velocity development,as Cerner knew firsthand.
By contrast, when you treat infrastructure like cattle, “everything becomes a lot more commoditized,” said Meyer. In a herd of cattle, if one heifer wanders away, the herd keeps on functioning. The rancher doesn’t spend time chasing it down. Likewise, treating infrastructure like cattle means VMs are mutable and if one goes down, you just spin up another one.
Ultimately, treating infrastructure like livestock removes many of the bottlenecks that slow down software development and delivery. With PCF, for example, there’s no more waiting for VMs to be provisioned. Developers can spin up new environments via self-service, with just a few clicks. The platform is also self-healing. When a VM fails — and they always do — a new one is automatically spun up to take its place.
The cattle approach also allows ops teams to roll out zero-downtime upgrades to PCF itself. “If we need to do an upgrade … we throw in a brand new VM and away we go,” said Meyer.
Concourse and Compliance
Cerner has the right culture and organizational structure. And a highly automated platform like PCF is the centerpiece of a high velocity team. But Cerner needed another component: Concourse.
Concourse is a continuous integration/continuous deployment system that works natively in PCF that manages automated test and delivery pipelines. This includes testing software on different IaaS platforms, testing software compatibility with multiple platform versions, and ultimately packaging and delivering software to production. It is used by developers to move their code through the development process. Ops teams use Concourse to update and maintain PCF as well.
As Meyer put it, “Concourse is very, very ephemeral in how it does everything. And these pipelines are very, very declarative in nature. You actually use yaml to define what your pipelines look like. So it’s all written with code.”
In addition to supporting release velocity and high availability with immutable infrastructure, it turns out Concourse is also helping mitigate the effects of compliance requirements.
Like most companies in the healthcare field, Cerner must comply with numerous regulations. This includes ISO 9000, which requires companies to implement and document a number of quality management and quality assurance policies and procedures throughout the manufacturing process. When applied to software, IT staff must perform dozens/scores/hundreds of manual checks and sign-offs throughout the development process.
You can imagine how this plays out in practice. One person forgets to send an email, or someone delays a compliance check, development comes to a screeching halt. With Concourse, Cerner automates the process of notifying the appropriate people that a particular step is complete. The person then performs their ISO 9000 compliance tasks. When the check is complete, Concourse automatically kicks off the next step in the process. Code moves to the next step on the pipeline, be it a dev/test environment, QA, staging, or even production.
Concourse, said Meyer, “really makes the ISO process just another first-class citizen inside of our pipeline, without really having to change what we’re doing.”
Keep the Pipelines Flowing
Today, Cerner is in the midst of piloting PCF with a handful of teams, who are now delivering software in days and weeks instead of months. Ultimately, the platform enhances the agile development practices teams were already adopting. Now the company is reaping the rewards of improved ops efficiency and reduced the friction for compliance checks.
With a full-fledged rollout of PCF on AWS coming in the next few months, Cerner’s ops team is ready to keep the build pipelines flowing for developers and applications throughout the company. Doctors, nurses and, ultimately, patients will benefits too, in the form of a steady flow of new, useful EMR feature.
