An edited version of this article appeared on The New Stack on June 28th, 2014. You can read it here.
Once a week, twice a week, every day. These stats are not uncommon when discussing new code or feature delivery cadence, because today’s dynamic business environment demands accelerated value delivery. And while those companies who recognize this urgency have done much work to modernize their software development and delivery processes, one crucial aspect of the process continues to be sidelined – upgrading existing code running in production.
Undeniably, retracing the steps required to update and maintain source code is a daunting task for many organizations, particularly those relying on OSS technologies with limited support windows. As time goes on it becomes harder to remember what’s what, what version things are, and why code was written the way it was in the first place. It’s not a problem at first but as technical debt builds up it becomes increasingly difficult to go back and remediate potential issues or CVEs. Not to mention as OSS frameworks and tools evolve the lift required to move to the latest version increases.
These modern technologies also pose increasing challenges for organizations who operate within highly regulated industries. In order to adhere to mandated standards, upgrades for certain technologies act as a critical checkpoint which must be completed as policy dictates or as CVEs arise. When these upgrades are only performed as one off events when mandated, this adds another barrier into an often already complicated and process heavy application delivery pipeline.
We saw this first hand, late last year when the end of OSS support for versions of Spring Boot (2.x) had many organizations looking for ways to carve out room on their roadmap to make the upgrade or to extend their support window via commercial support.
This begs the question – what makes updating source code, dependencies, and frameworks such a formidable challenge for organizations and why is it not built into application development culture by default?
Business Value Always Triumphs
Let’s return to a time when delivering new code required months of work, where code was left untouched for half a year, running smoothly in the data center. Back then, the mantra was simple – “if it ain’t broke, don’t fix it”. Software updates were a luxury, with quarterly releases being the best-case scenario. More often than not, updates would only occur twice a year, sometimes even annually.
Anytime an opportunity to update code arose, the goal was entirely focused on delivering new functionality to the business. The relentless pursuit of value meant that once a feature was deployed, it was straight onto the next one with little time spared for addressing technical debt.
Security and stability, unless explicitly requested by end users, were often overlooked with this approach. Engineers and software architects would struggle to advocate for these crucial elements, only to see their concerns snowball into larger issues over time. As such, patching old code turned into a Herculean task as developers kept their distance from the aging codebase.
Today, this reluctance has fostered an all-too-common perception: that updating source code is a complicated, difficult, and low business-value process. Of course, upgrading source code does have its technical challenges but at its core many organizations simply tend to avoid it until they are forced to act. Continually updating source code has yet to find its place in the heart of organizational culture and many companies are holding their breath, resorting to any measure that might help them evade the looming threat of CVEs.
Why Upgrades Matter – a look at the Spring/Java ecosystem
To illustrate why continual code upgrades matter let’s use the Java/Spring Ecosystem as an example. Java continues to be one of the most widely used programming languages. In its 25 year history, there haven’t been many updates that actually broke existing code. An incredible feat but a double edged sword. While it’s possible that code written in the 90s will still run on the latest version of Java, if none of the patching, none of the maintenance, and none of the dependencies were updated along the way, you’re not getting the most out of your Java-based applications and systems.
Though it’s been possible to carry on with Java support provided by Oracle, many organizations are leaving potential savings and benefits on the table by not upgrading to the latest versions. For example, Java 21, considered by many as one the most significant releases in Java’s history, includes features that shorten startup times, reduce application memory footprints and, and generally make applications less vulnerable to threats and performance issues. Yet until this year, most Java applications running in production were running on Java 11 according to New Relic’s most recent State of Java Ecosystem report! For more about all the goodness in Java 21 check out Josh Long’s article about it here.
With all this in mind there’s never been a better time than ever to keep pace with the Spring Framework release train. Now is the time to create a culture of continuous upgrades in your organization.
What a Culture of Continuous Upgrades Actually Looks Like
When software took 6+ months to deploy usually meant lots of new features which is exciting if not hard to absorb all at once and not to mention increased risk of things not working or breaking. The shift to smaller more frequent releases helps with system availability, feature utilization, and release velocity. A culture of continuous upgrades is an evolution of this philosophy. By reducing the volume of upgrading you need to carry out in one go, you inherently decrease the threat to your development process with little to no impact on your ability to keep up with the feature delivery pace of the business.
Though there is a persistent perception that the older version is more stable, this often isn’t the case. For example, patches for both Spring Boot 3.1 and 3.2 are released on the same day and fix the same things, there is nothing that makes one inherently more stable. You are, however, missing out on the new features introduced in 3.2.
When upgrades become a normal part of your software development life cycle, it no longer takes multiple sprints to upgrade, you don’t need to tell the PM that you can’t deliver a feature because you’re working through a list of technical chores. The path forward isn’t how cool and fancy you can get, it’s how maintainable your code is, it’s about having confidence that in two or three years that code will be maintained rather than adding to the pile of technical debt already incurred.
The end goal is creating a culture where continuous upgrades become so embedded in your process that they don’t require a second thought. It’s the reassurance that what you ship out into the market today will stand the test of time, remaining safe and secure in the years to come and ensures you meet compliance requirements.This sounds great, but there is an investment required to get there!
Building Your Upgrade Muscle
For organizations that haven’t been in the process of taking upgrades this is not always an easy muscle to turn on. There is no cheat code to skip the training and turn this on at scale. Every organization is going to have patterns that are hard or something special they did that can’t simply be plugged into an existing recipe. Those cases will require code changes. Here are three tips to consider for building your upgrade culture.
1. Start with Leadership Buy In
Like any major change to the way things are done within an organization, cultural change championed by leadership is critical to success. It’s critical to ensure that the people who are responsible for more than just the functionality of applications are on board before moving forward. This could mean the CIO, CSO, Enterprise Architects, all the way down to the application leaders who own the portfolios. Really it’s getting the support from anyone who might have said “ I wish I could have worked on that, but I need to deliver the features”.
Software leaders can no longer afford to bet their business (and their customer’s business) on their code being safe and secure, the time from a CVE being announced to code patched is now critical.
2. Prioritize Release Planning and Portfolio Visibility
Above all else, a good plan and visibility are the critical prerequisites before getting hands on with any code. This starts with understanding what releases are headed your way. Looking again at the Spring Framework, in 2023, there were only 4 weeks in the year where the Spring team didn’t have a release. The Spring team is pulling in third party dependencies so everything under each release is patched, has been tested against each other and is validated to work.
Having awareness of what is coming up and understanding the release train of those dependencies is valuable as you will always know everything has been patched.
Equally important, your CSO also needs to have a list of all the known CVEs that are coming in and visibility into what’s actually running in production within your applications portfolio. This may seem trivial, of course you know what’s running inside your own portfolio but as the features and code-base grows, it’s easy for things to get lost. Developers are always looking for solutions to fix their problems, and maybe along the way they inserted a library here and there to help them solve what they needed.
3. Leverage Tools and Automation
Scanning tools and application discovery can help identify where the biggest risks in your portfolio are, bucket apps into easy opportunities to upgrade, and identify what’s going to require a bit more effort. When enhanced with automation capabilities these tools can help you bolster your continuous upgrade culture.
When you’re ready to actually start upgrading your apps, you need to determine what a reasonable starting point should be that won’t impact or interrupt your existing development pipeline. It might not be sustainable to have all your teams start individually tackling upgrades, but instead a small team and some automation might be a good starting point.
For the apps that your analysis identified as being easy to upgrade or requiring less of a white-glove approach, tools such as Open-Rewrite offer a simple way to start automating your refactoring and remediation. Open-Rewrite consists of an auto-refactoring engine that runs prepackaged, open-source refactoring recipes for common framework migrations, security fixes, and stylistic consistency tasks.
Once you have tackled the easy to upgrade applications, you can start to tackle the complex applications that don’t have existing recipes to pull from. As you learn more about your apps that need to be upgraded you can develop your own recipes to scale out your upgrade efforts and establish your own framework going forward.
One question that might come up when looking to upgrade source code is how to manage your code running in production, while simultaneously working on the upgrade. Do you take the application down, while upgrading? Application platforms shine in this scenarios
As we saw in the annual state of cloud native application platforms report, enterprises are looking for a single platform experience that supports multiple app types and deployment patterns. A cloud native app platform allows you to run multiple instances that can be upgraded or patched while other instances are running, or change the OS layer for something running in a cloud-native buildpack in production.
Rinse, Repeat, and Learn
To truly get to a state of continuous upgrades there is an aspect of “you must be this tall to ride the ride” but once you are tall enough you need to continuously get back in line and back on the ride. By working through the upgrade process, you will learn to figure out what patterns will work for your organization, how to better leverage tools like Open-Rewrite, and create recipes and your own frameworks to help apply them at scale.
Today, everything is moving faster, there are more opportunities for bad actors, but also more possibilities and innovations to take advantage of than ever before. So don’t bet your business on the way things have always been done, start implementing small changes today and work towards building out your own culture of continuous upgrades.
