“If you can’t measure it, you can’t manage it” – we took that quote from the business management disciplines and decided to apply it to open source. However, there is an important disclaimer that we want to start with, which is true for both the corporate world and even more for open source because of its role as an incubator for innovation. We shouldn’t over-quantify our work because the most important portion is usually hard or impossible to measure.
So why would we bother to try to measure open source?
Open source is already established as a standard for many evolving technologies and there is almost no project that is not based on it or using it in some aspect. There are great ideas and valuable efforts that have been brought to life as open source projects. It is important to not let these important works fail – rather, we should do our best to improve and grow them further.
Based on a survey held last year by opensourcesurvey.io, 72% of the respondents have recently left or neglected a project. Even though this is a natural process and not every project proves worthy of resource investment, many great ideas might have been abandoned as well. Thus, it is important to notice and nourish those ideas and projects that are really good and could bring value. That is why measuring and monitoring “open source project health” is so important from the very start of a project.
Project health indicators
We asked open source practitioners what are the first reasons that come to mind when they hear about an unsuccessful open source project and noticed some repetitive indicators.
Here are some of the “red flags” we notice:
- It’s common for key contributors to lose interest or lack the time or resources, which results in low maintenance of the project.
- legal issues
- competition or alternative solutions
- security concerns
- team conflicts
All of the above can be transformed into several key areas related to open source project health:
Governance
This can include the type of ownership like company, foundation or individual contributor, as it’s important to assess certain risks. Additionally, there are maintenance specifics like who is making the decisions on the project, is the project evolving new potential leaders and is there clear growth processes?
Contributor risk
This indicates the probability of a project to become fully unsupported. It’s not only about the number of project owners and maintainers, but also the tendency to give such responsibilities to new people and to share knowledge. For example, we have defined a metric to monitor the percentage of commits and to highlight them if more than 70% are done by less than three people.
Diversity and Inclusion
An unfriendly environment increases the risk of governance issues, as well as forces people to step back from a project. This is why all company, regional, gender and ethnic diversity are significant factors when evaluating project’s health.
Activity
This includes the frequency of new commits, PRs, PR reviews, issue responses, issues being addressed and new releases.
Security
A well-maintained project requires an established security protocol. This includes documentation, advertising and how to report security issues. It is crucially important that those are addressed and released on time.
Balancing it All
Being an open source project maintainer means a higher level of responsibility for the project leadership. The health indicators are an important lever to not only balance all the parallel tasks but to justify the investment of time and resources.
One major role of an Open Source Program Office (OSPO) is to guide projects as they become sustainable and recognized as reliable for adoption. There is no easy approach and every OSPO has their own “magic.” Our experience has taught us some of the following when dealing with a large number of projects:
- Shorten the list. Archive or clean up any inactive projects so that you have better visibility of your portfolio.
- Group them by common criteria depending on the metrics you are applying.
- Keep close contact with the maintainers, so that you are prepared for any changes and can also provide enough guidance and support.
- Make it easy to collect feedback from the maintainers and their teams by using automated forms, surveys, slack channels, etc.
- And last but not least, use tooling to monitor all relevant metrics.
The CHAOSS project’s Augur tool has been developed to enable this type of data collection and metrics measurement. Other measuring tools include GrimoireLab, Devstats and other proprietary and internally deployed tools.
Part of the CHAOSS project’s work is defining the concrete metrics for each of the health indicators. Each individual project can adjust them to its specific needs.
As with implementing any new methodology, start slowly. While using numbers and metrics to measure the value of a project, it is always important to know that any indicator will drive behavior. Be careful when setting expectations and try, rather, to simply assess risk while not giving in to judgement or unhealthy labels. And remember, do not over-quantify the metrics – it’s all about people and communities in open source and in life.
