The Built to Adapt Benchmark was a quantitative framework of indicators that gauged how well an organization builds, deploys, and operates software. Pivotal collaborated with Longitude Research and Ovum in 2018 to assess the world’s top organizations with more than 1,600 respondents across six countries and five industries. The research explored how organizations align their product development and IT operations with business goals to build software that delivers desirable outcomes.
Here are some of the key findings from the Benchmark:
By Country
Australia
Overall, Australian enterprises ranked fourth out of the six markets surveyed. The results of the commissioned research report indicate that:
Speed:
-
Australian developer teams are more balanced than anywhere else, with 63% of respondents saying they have few or no ‘star’ developers, and rely on equal contributions from across the team. In the UK, just 41% of firms said the same.
-
Australian firms are among the fastest to deploy software updates following user feedback, taking an average of 7.1 days to deploy a simple change, compared with 7.6 in the UK and 7.8 in Singapore.
Stability:
-
Globally, firms report that approximately 19% of software launches and upgrades are delayed due to defects. Australian firms are slightly ahead of US firms with 19% opposed to 22% delays reported, respectively.
Scalability:
-
Australian firms are ahead of their neighbors in transitioning to the cloud. Around 38% of Australian applications are built or refactored to run in the cloud, second only to the US (45%).
-
Australian firms lack confidence on scalability. 20% would feel confident scaling their existing architecture with minimal structural changes if they had to double the workloads on current applications. In the US, 55% of firms said they could do the same.
Security:
-
23% of Australian firms report having experienced more than 11 security-driven service suspensions or delays over the past year. The US and Singapore had more firms experience as much disruption at 33% and 28%, respectively.
Savings:
-
On average, Australian firms surveyed have more applications per developer than any other country (3.7 applications per developer). This indicates Australian developer productivity is high, compared with the global average, where around 2.4 applications are resourced to each developer.
-
Despite high developer productivity, Australian firms require far more IT operators, with 90% of Australian firms reporting one to every 5 or fewer developers, compared to the US, where 79% of firms report 5 or more developers per operator.
Germany
Speed:
-
German firms are second only to their US counterparts when it comes to seeking customer feedback—43% of German firms seek customer feedback at least daily (compared with 51% in the US).
-
German firms manage to move mission critical applications from deployment to production in just over one day (.9 days), far outpacing the global average of 7.3 days.
-
It takes German firms 4.8 days to integrate a new feature (e.g. credit card payment with biometric authentication) into a customer facing application—approximately half the global average (9 days).
-
German firms are shipping code most frequently, with almost 3 in 4 pushing software code at least once a day (72%).
-
German telecoms are the fastest overall, with 82% deploying code at least once a day.
-
Stability:
-
German firms report just 19% of launches or upgrades are delayed due to defects. German telecoms (17%) and retailers (15%) are particularly diligent when it comes to writing high-quality software.
Scalability:
-
Cloud adoption is lagging behind in Germany, with German firms having the lowest rates of cloud-native infrastructure of any country surveyed: German firms report having less than one third of applications built or refactored to run in the cloud (31%), compared to levels as high as 45% in the US.
-
German retailers are performing especially poorly—with firms running less than a quarter of applications in cloud environments (24%). In sharp contrast, US retailers run up to 54% of applications in the cloud.
-
Security:
-
Germany reported the least software disruptions due to security. Only 12% of German firms had more than 11 development delays in the last year due to security concerns, compared to 33% in the US.
Savings:
-
German firms achieve the highest rates of budget flexibility, with 23% of firms reporting that IT budgets are “highly flexible,” can adapt throughout the year, and that initiating new projects is easy—compared to only 16% of US firms and a global average of 19%.
-
On average, German firms resource 3.1 to each developer, compared with .7 in the US and 1.5 in the UK, demonstrating high developer productivity in Germany.
-
German developers spend 37% of their time on writing code for new products and features, with the remainder dedicated to maintenance and fixing old code. In contrast, US developers spend 42% of their time writing new code.
Japan
Speed:
-
In the product development process, Japanese firms are ranked second in the global comparison when it comes to achieving a good distribution of skills among teams. 56% of Japanese companies are able to report that they rely on few or no ‘star’ developers, second only to Australia (63%).
-
Japanese firms are particularly efficient when it comes to implementing changes based on user feedback, taking on average 6.3 days to make simple changes—beating the international competition by almost a day.
-
Only 5% of Japanese organizations have adopted continuous, hourly, or daily deployment practices. For comparison, the top-performing country, Germany, has almost three quarters of organizations deploying code continuously, hourly, or daily (72%).
Stability:
-
Falling roughly within the global average (20%), Japanese firms report that around 21% of software launches or upgrades are delayed due to defects. In Germany, where firms perform best on this particular indicator, defect rates still amount to 19%.
Scalability:
-
Japanese firms spend a slightly higher proportion of IT budget on developing or refactoring software rather than operating and maintaining IT systems (22%) than Singapore (14%) or Australia (15%), but lag far behind the US (51%).
-
Japanese firms lack confidence on scalability, but also report less disruption than the US. 23% of firms in Japan would feel confident scaling their existing architecture with minimal structural changes if they had to double the workloads on current applications. In the US, 55% firms said they could do the same. However, US firms report the highest levels of disruptions for doing so at 32%, while far fewer Japanese firms report extreme levels of disruption (7%).
Security:
-
17% of Japanese companies reported having new or existing apps paused more than 11 times in the past year due to security concerns, as opposed to 33% in the US.
Savings:
-
Less than a third of developer time is spent writing new products or features (28%), with the rest dedicated to maintaining old or existing code. That means a significant percentage of engineering time is still dedicated to maintenance tasks or fixing legacy code. That compares with 35% globally, and up to 42% in the US time spent writing new products or features.
-
Japanese companies lag behind in terms of automating elements of the software development process. In areas including infrastructure provisioning (15%), software build (30%), testing (28%), and deployment (28%), not even a third of firms can claim that their processes are either fully or mostly automated.
Singapore
Speed:
-
Just 2% of Singaporean firms are deploying code on a continuous, hourly, or daily basis, compared with almost three quarters of German firms (72%) and more than half of US firms (55%). Notably, only 5% and 6%, of Japanese and Australian firms, respectively deploy code on at least a daily basis.
-
Firms in Singapore are gathering little insight into users. Just 12% are gathering continuous or daily feedback on their applications, compared with over half of US firms.
-
Firms in Singapore are taking longer than elsewhere to move applications from deployment to production—around 19 days on average, compared with a global average of 7 days.
Stability:
-
Falling within the global average (20%), Singaporean firms report that around 20% of software launches or upgrades are delayed due to defects. In Germany, where firms perform best on this particular indicator, defect rates still amount to 19%.
Scalability:
-
Singaporean firms lack confidence on scalability, but also report less disruption than the US. 18% of firms in Singapore would feel confident scaling their existing architecture with minimal structural changes if they had to double the workloads on current applications. In the US, 55% firms said they could do the same. However, US firms report the highest levels of disruptions for doing so at 32% while far fewer Singaporean firms report extreme levels of disruption (13%).
Security:
-
Over a quarter of Singaporean companies report having suffered 11 or more security related suspensions of service in the past year (28%), compared with 17% in Japan. Only the US had more firms report the same number of security suspensions (33%).
Savings:
-
However, less than a third of developer time is spent writing new products or features (29%), with the rest dedicated to maintaining old or existing code. That compares with 35% globally, and up to 42% in the US of developer time spent writing new products or features.
United Kingdom
Speed:
-
UK firms are taking 5.3 days to integrate a new customer-facing feature (e.g. biometric ID), compared with a global average of more than 9 days.
-
UK firms are among the slowest to respond to user feedback, taking an average of 7.6 days to deploy reactive changes, compared with just 6.3 days in Japan.
-
UK businesses are taking .8 days to move applications from deployment to production, compared with a global average of more than 7.3 days.
Stability:
-
Falling above the global average (20%), UK firms report that around 21% of software launches or upgrades are delayed due to defects. In Germany, where firms perform best on this particular indicator, defect rates still amount to 19%.
Scalability:
-
UK businesses are building scalable technologies. 40% said doubling the workloads on existing applications would result in little or no disruption, compared with 32% in Germany and 33% in the US.
Security:
-
14% of UK businesses suffered more than 11 security-driven disruptions in the last year compared with 33% in the US.
Savings:
-
On average, UK firms resource 1.5 to each developer, compared with .7 in the US and 3.1 in Germany, demonstrating high developer productivity in Germany.
-
43% of UK firms are managing to spend more of their IT budgets on developing new software and refactoring legacy software than they are on maintaining existing IT systems, compared with a global average of 36% across all markets. Thus, UK firms are leading in delivering software value.
United States
Speed:
-
US firms are most likely to get continuous or daily feedback from customers (51%) followed by 43% in Germany and 38% in the UK.
-
55% of US firms are deploying code on a continuous, hourly, or daily basis. That fares well against the global average of 39%, but well behind Germany, where 72% of firms are deploying on a continuous, hourly, or daily basis.
Stability:
-
US firms report approximately 22% of their software launches or upgrades are delayed due to defects.
Scalability:
-
45% of applications run by US firms have been built or refactored to run in the cloud, compared with between 31% (Germany) and 38% (Australia) 37% (both Japan and Singapore), across other markets surveyed.
-
Only in the US are half of firms spending more on developing new and/or refactoring legacy software (51%) than they are on maintaining existing IT systems. In the UK and Germany, just 43% and 44%, respectively, of firms are managing the same.
-
US firms offer a mixed picture on scalability.
-
More than half of US firms stated scaling their infrastructure to accommodate a doubling of workloads would require minimal structural changes (55%), compared with an average of 39% across all markets.
-
However, one-third of those firms say doing so would result in extremely high levels of service disruption (32%), compared with an average of 17% across other markets.
-
Security:
-
33% of US companies reported having new or existing apps paused 11 or more times in the past year due to security concerns, as opposed to only 12% in Germany or 14% in UK.
Savings:
-
US IT budgets are among the least flexible, with half of firms saying budgets are fully committed at the start of the year (50%), making it difficult to initiate projects mid-cycle.
-
US firms are leading the way on automation, with more firms than anywhere else:
-
Performance monitoring is mostly or fully automated at 70% of US firms, compared with an average of 54% across all markets.
-
Infrastructure provisioning is mostly or fully automated at 58% of US firms, compared with an average of 38% across all markets.
-
Software testing is fully or mostly automated at 56% of US firms, compared with an average of 43% across all markets.
-
-
US software teams have an average of more than 6 software developers to every operations/QA team member—the highest of any country—compared with a global average of 5.
-
US developers are spending more time than elsewhere writing code for new products and features (42%), as opposed to carrying out maintenance or troubleshooting old code.
By Sector
Overall
- On average, 39% of organizations polled report deploying code on a continuous, hourly, or daily basis. A whopping 40% only deploy code on a monthly, quarterly, or annual basis.
- Building for the cloud allows applications to scale easily. Of the organizations polled globally, only 37% of apps were built, or have been refactored, to run in the cloud.
- Organizations polled report an average of 20% of software launches and upgrades were delayed due to defects.
- In the last year, 33% of US organizations surveyed delayed application releases more than 11 times due to security concerns, as opposed to 21% globally.
- Of organizations polled globally, only 35% of developers' time is spent writing code for new products or features to deliver value as opposed to maintenance or fixing old code.
Automotive
- Auto developers polled are spending 65% of their time on maintaining or fixing old code, rather than building new customer or internal applications.
- Car companies surveyed say they are moving apps from deployment to production in under a week, compared with 10–12 days across other sectors.
- 14% automakers polled managed to avoid security-related disruption altogether, and almost half managed fewer than four such disruptions over the past year (46%).
- German and US auto companies surveyed are leaders in continuously collecting customer feedback on their applications at 26% and 23%, respectively, compared with 17% globally.
Banking
- Banks polled scored lowest on 5 out of 16 indicators, specifically the level of investment in new development and level of automation in software deployments were below average for survey respondents.
- 36% of banks polled are spending more on maintaining IT systems than new software development, and 32% are spending equally on both.
- Banks polled in the UK report that developers spend 40% of their time writing code for new products or features, in contrast to 28% for Australian banks.
- Of organizations polled, only 34% of banking apps are built to run in the cloud.
- Only a third of banks polled are collecting data continuously or daily on how customers use apps (34%).
Insurance
- Insurers polled take fewer than 8.3 days to integrate biometric ID into a consumer app. Only automotive companies said they’d be faster (7.6 days).
- Insurance companies polled globally are resourcing 3 applications on average for every developer, the highest of any sector. Compared with 1 application per developer in Automotive or 2.4 average across all sectors globally.
- 39% of insurers polled report deploying code at least daily, compared with 29% across all sectors globally.
- Insurance companies polled report the highest disruption to services should workloads double. 71% of insurance companies report high or extremely high disruption, compared with 66% across industries.
Retail
- 63% of US retailers polled dedicate more than half of their IT budgets to building new software compared with the global retail average of 37%.
- US retailers polled report it takes 4 days on average to implement a new product feature compared to 9 days for retailers globally.
- 62% of US retailers say their developers' skills are evenly distributed and that they rely on a few 'star' developers compared to 54% retailers polled globally.
- Almost 9 in 10 retailers polled globally are gathering data on how customers use apps at least every two weeks (87%).
- Among retailers polled globally, only 18% of software rollouts are delayed due to defects.
Telecoms
- 23% of telecoms companies polled globally are deploying new code on a continuous basis, as opposed to 19% on average across sectors.
- 37% of telecoms polled globally are gathering continuous or daily customer data on how applications are being used. Collecting daily customer data is the norm among (67%) of US and (54%) of German telecoms companies.
- A quarter of telecoms companies polled globally estimate that doubling user workloads would cause ‘extremely high’ disruption (24%). Among US-based telecoms, it’s 56%.
- German and US Telecom companies polled lead in the proportion of developers' time spent on writing code for new products or features in contrast to fixing old code at 45 % and 44 % respectively.
