Behind Pivotal’s no whiteboards, no tricks, no brain teasers, language-agnostic interview process
As a developer, you’ve probably been asked a brain teaser during an interview. Something about being very small and trapped in a blender might ring a bell. In response, you might’ve paused and constructed a thoughtful answer intended to show how you think. But what did it really accomplish? “You might recognize it’s someone smart, but that doesn’t tell you about their practical skills as a developer,” said Edward Hieatt, Senior Vice President at Pivotal. “Are they really going to succeed with us? I don’t know.”
Hieatt, along with his colleagues on Pivotal’s leadership team, believe the traditional methods many companies use to identify and hire engineering talent is broken. Commonly, a company will set up multiple interviews for a candidate to watch him or her write code on a whiteboard or solve puzzles. If one interviewer isn’t impressed, they often go back to the drawing board. Worse, interviewers are often seeking a “superstar” developer, missing the opportunity to find strong coders who can work effectively with others — the skill Pivotal sees as most necessary to building great software.
10x-ing Your Team: The End of Superstar Developer Culture
As a result, Pivotal began developing an interview process years ago now colloquially known as the RPI, short for “Rob’s Pairing Interview.” This approach stemmed from Pivotal CEO Rob Mee’s days as an engineering consultant. When evaluating talent, Mee would ask a team of developers to start building something that generally all programmers knew about, like collections in modern programming languages. “You have containers for objects, including a list, a map, and a set. I would tell them, ‘These things that you understand really well, let’s imagine they don’t exist and let’s build them all,’” Mee said. “I quickly discovered this exercise was really informative. This very simple, structured process would show how good someone was: how they interacted, whether they had a good sense of design, whether they were easy to collaborate with, did they think quickly, did they have a good sense of abstraction? So I started using a modified version of that to interview programmers.”
Today’s Pivotal RPI is a 45-minute coding session that candidates complete together with a Pivotal employee instead of standing in front of a whiteboard. The goal of the session is to select candidates who have basic engineering competence and the ability to think quickly and interact with somebody they don’t know. Many of the test’s questions focus on the social dynamic during the pair programming exercise. Besides fundamental programming capability, the test helps Pivotal look for speed, abstract thinking, empathy, and communication skills. Those who successfully complete the test are invited to come back and do multiple pair programming exercises with different teams. The company sees interviewing as a two-way street: you’re evaluating them as much as they are evaluating you.
Part of the problem companies face with hiring developers is that it can be difficult to find a scaleable, repeatable format that accurately evaluates the skill level and is applicable across all of programmers’ expertise. Programmers will often say they specialize in a coding language, and therefore, a test with a different language would be ‘outside their wheelhouse.’ Perhaps, more than that, if you think about how people interview and the settings that most programmers find themselves in, there’s nothing formalized about the interview.
“Because programming is a new enough industry, there isn’t a ‘here’s how you do software engineering’ hiring process that hasn’t proven to be disastrous. For many companies, they’re still trying to figure out how you interview engineers. What does teamwork look like for them? What does a good team mean?” said Hieatt. “These things are still up for discussion, and we’ve just come up with a strong point of view backed up by an argument and data.”
We believe coding is a social activity. In fact, we’re testing you to see how social you can be around coding.
Hieatt attributes broken developer hiring methods to the academic experience of most hiring managers. “I think the academic setting is the only setting that many managers know in terms of computers. They relate the job interview to their academic experience, which I think is flawed in lots of way,” he said. “One, computer science is only an element of software engineering. Two, I think programming is still, outside of our ecosystem, seen as a solo activity that brilliant people do in isolation. There’s a mystique around it. It’s like managers say, ‘Just leave those smart people alone in the corner, give them hamburgers and money and leave them alone.’ We believe coding is a social activity. In fact, we’re testing you to see how social you can be around coding.”
There are no trick questions with the RPI. During the interview, candidates are given multiple chances. Once they get comfortable with the exercise, they have a chance to show what they can do. If they answer a question incorrectly, they get multiple chances to recover, to learn as they’re doing it and show that they are able to adapt, which is important for working at Pivotal.
Though passing the RPI doesn’t guarantee an offer, it says a lot about a candidate. Pivotal’s London office found that there was a strong correlation, weighting to how well applicants scored on the RPI, to how likely they were to pass the second round of interviews. People who scored a 97 or 98 on the RPI had an excellent chance of getting through the secondary interviews and getting offered a position.
The RPI has also been critical to Pivotal’s success, helping the company to hire thousands of employees in just four years in large part due to the test’s quick and scalable template. It’s even helped some Pivotal customers refine their recruitment approach over the years. Recounting one company, Mee said, “The CTO came into this test with me, and he was so taken with it, he kind of just lost it. Within a month he had done it 40 times. He said, ‘I don’t have to filter anybody anymore, I just get every single candidate I can and just run them through this because it’s so obvious whether or not they are good.’ Before, he used to agonize over every resume.”
This is the third part in a series, Developers at Work, exploring the changing role of developers in today’s workplace.
The Developer Hiring Process is Broken was originally published in Built to Adapt on Medium, where people are continuing the conversation by highlighting and responding to this story.
