Evaluating abilities of hiring candidates is challenging. The candidate highlights what is important to them and their experience on a resume or portfolio of some kind. As an interviewer or screener, you pick out things from what you’re given and imagine how it meshes with your needs and vision. Validating and verifying is another challenge: How do you know they’re not making up stuff? How can we compare the skillsets of many candidates?
To verify, we use interviews to ask searching questions. You might use the STAR method to peel back the onion of various real-world experiences. You might have a standard set of questions that you ask of every candidate to make it easier to compare and rate candidates.
How can you speed up this process or avoid wasting everyone’s time in the first place? How can you get more of what you want instead of being fed what the candidate thinks you want to hear? Automate it!
You could use an online form like Google Forms to present a quiz. Even an old-school paper application could serve as a type of automation to gather simple insights.
For our ideal candidates, we expect that they’ll feel much more comfortable on the command line. Our command-line automation allows us to not only see that they can solve complex problems, but how they go about it. We can review their command history. Even if they don’t solve the problem, we can see what they tried. We’ve also had candidates who didn’t know how to login with SSH. That is informative too.
How we built our automated hiring tool
We have automated a portion of our process using cloud-config and home-grown bash scripts to present the candidate with a lab environment. We present them with tasks common to the role being considered. The tool allows us to immediately evaluate complicated tasks and send those results to Slack for review.
Let’s dig into the technical details. Wait, there’s one more critical, non-technical step! You must identify problems that you want the candidate to solve. We currently present eight tasks, ranging from finding disk space issues to cloning git repos and configuring PHP. These are the types of skills used in day-to-day activities by the Seller Labs DevOps team.
On to the good stuff: the technical details
- A message of the day script runs on startup to give instructions for the lab.
- Another script proxies any candidate questions from the command line to our Slack channel.
- A check script runs when the user submits the lab for grading. Results are sent to Slack, along with a note about how to connect to the instance for inspection of the machine for partial credit for missed answers.
We run our lab environments on a stand-alone Digital Ocean account for simplicity and security. That is, we don’t allow candidates directly only our VPC in AWS where we run most of our workload. Cloud-config will work on most cloud providers. It’s been around a long time. It’s a nice fit for a simple deploy that can be shared and possibly collaborated on.
Among other things, the cloud-config sets up users, formats disks, installs packages, deploys the bash scripts, and sets an auto-shutdown time.
Spinning up the machine
We spin up an environment on Digital Ocean using the
doctl CLI tool for each candidate after our recruiters refer them to our team. We try to avoid wasting candidates’ time by setting the expectation that most candidates should only spend an hour or two to complete the lab, and we automatically terminate the lab after 72 hours. We only start the lab after getting a good idea of when the candidate might reasonably start working. The candidate uses their own computer to take the lab on their own time, with no one looking over their shoulder. That is, it’s an open-book test; the complexity of questions can mitigate simply “Googling” direct answers.
Automating parts of the hiring process helps us quickly get an idea of a candidate’s ability to solve problems that are common in our environment. And since it’s codified and documented, it’s easy to iterate, change, and improve. Let us know what you can come up with by forking the repo and writing a few tests. Happy hiring!