The Science Behind Hiring “10X Engineers”

Wouldn’t it be great if we could find the perfect person for every position every time?

Yes, it would

We all want that, but are we geared towards doing that?

I am not questioning intentions here, merely questioning the means

Before I ponder further, I would like us to be on the same page as to who falls under the 10X Engineer category

10X Engineer a.k.a the coding Ninja a.k.a rockstar developer is an engineer who contributes 10X times the value of one engineer

Don’t get me wrong. They are not developers who can type out the code at 10X speed but those who can build solutions that are 10X effective

10X Engineers build and love to build better software

I love the way Joel Spolsky, CEO, and co-founder of StackOverflow puts this into perspective:

"Five Antonio Salieris won’t produce Mozart’s Requiem. Ever. Not if they work for 100 years"

Joel Spolsky, CEO, and co-founder of StackOverflow

Now that we have established the idea of a 10X Engineer, keep in mind that a 10X Engineer that suits my team might not be the 10X engineer that suits yours

Why do we fail in hiring 10X Engineers?

We fail because we are assessing things the wrong way. We rely too much on resumes, textbook-oriented assessments, unstructured interviews, and the like

The good news is that you and I are not alone; even the best of companies including Google have chinks in their recruitment process. Here is a famous example of Google rejecting Max Howell, creator of HomeBrew, as he could not invert a binary tree on a whiteboard

Fortunately for us, Frank L Schmidt and John E Hunter chose 19 assessment methods and did a detailed study on how accurate each of the assessment methods is in predicting candidates’ on-the-job performance

I accept the research does have some limitations and the data was not only from the tech fraternity.  But is anyway better than relying on opinions, anecdotes, and hunches

This is a table of the top 12 assessment methods reverse sorted based on their validity scores on how well each test predicts a candidate’s on-the-job performance

Assessment typeValidity ScoreWork sample assessments0.54General mental ability tests0.51Structured Interviews0.51Peer Ratings0.49Job knowledge tests0.48Integrity tests0.41Unstructured Interviews0.38Assessment centers0.37Biographical data measures0.35Conscientiousness tests0.31Reference checks0.26Job experience (years)0.18Years of education0.10Interests0.10

The right way!

Of course, using all these tests will help us get better clarity on identifying our unicorn - the 10X Engineer.  But owing to the limited time, money, and other resources, we can’t use them all. We need to choose

Our all-time favorites: unstructured interviews and resume highlights (Job experience in years, education, interests, etc.) are seen in the bottom half

And clearly, Work sample assessments have the best individual performance, followed by Structured Interviews, and General Mental Ability(GMA) tests.

We are overconfident about our expertise and experience in judging people that we dislike, realigning ourselves to more structured approaches fearing that these assessments might outsource human judgment to a machine

But one thing we seem to miss is these structured evaluations and automation are there to supplement human judgment with data and not replace them.

One big issue

GMA tests tend to attract legal issues, at least in the US, as it is biased against minorities. Here is one such example - Griggs vs. Duke Power Co.

By using GMA, your company might not only have to pay a hefty penalty but would also run the risk of being biased.

Our quest for hiring 10X Engineers might get severely derailed if we are perceived as a biased organization

So we now have the top two:

1. Work sample assessments and
2. Structured Interviews

Getting started

Before getting started with the assessments, conduct a detailed job analysis that will help us implement these assessments.

1. Work sample assessments

Work sample assessments are designed to mirror a situation parallel to that at work, under realistic and standardized conditions. For example, deploying an API, debugging code, or building a machine learning model

Here is a what Iris Bohnet, the author of What Works: Gender Equality by Design, has to say about work sample assessments

These are ideally designed as projects and are implemented as take-home projects that the candidates work on

Even with such overwhelming validity, work sample assessments are being used only by a handful of companies because

• Traditionally, work sample assessments incur high cost to implement as it involves one engineering source for every candidate code evaluation.

• Most companies who rely on work sample tests have to use Google Drive, Dropbox, or any other file sharing solution to collect projects

• There are no standard evaluation engines on which projects from different candidates can be evaluated to get insights on a comparative scale

This might seem out of place for a start-up, mid-sized or even large-sized enterprises. But with the advent of automated work sample assessments, running costs can be reduced to as low as 10$

So, head over here to try them out

2. Structured interviews

Structured interviews consist of a specific set of questions that are designed to assess the critical skills and abilities required for a job. Each question is structured to provide standard rating criteria so that it helps us to rank interviewee performance accordingly

Structured interviews are twice as effective as unstructured interviews. Granted, they can take more time to prepare for. But if you get it right once, you can boost your chances of making the right hire for every future position

Apart from technical prowess, Structured interview helps us to evaluate candidates’ soft skills such as communication skills, leadership, planning, and adaptability among others

A lack of questions mapped to key skills and abilities we look for in our candidates force managers to resort to unstructured interviews that fall short

Now with live coding platforms, paired coding is slowly becoming the norm in the industry. With preloaded questions that are mapped to test specific skills, TalScale’s live coding platform makes it easy for you to conduct remote structured interviews, thereby saving time and money for both the candidate and your company before you engage them for an onsite interview

Key TakeAways

• Hiring 10x Engineers is easier said than done. Assessing them in the traditional route like resume shortlisting and unstructured interviews will not yield results
• Work sample assessments have proved to be the best way to recruit top tech talent as they mirror the hands-on work that a candidate does at a company
• Talscale is one of the few companies in the world that offer such assessments. Our platform can significantly help in your quest for hiring 10x Engineers.

‍