Andrew Chen Archives

Why do people interview technical and nontechnical people differently? Nerds vs jocks

Interviewing engineers and objective tests for competence
Recently, I’ve been meeting a number of engineers to find more people I might want to work with. I think as a whole, most tech companies do a pretty good job interviewing engineers because at the very least, there are objectively correct answers to programming questions. In particular, you can give engineering questions which result in code that either runs or doesn’t, and having an interview candidate code with you for an hour is pretty enlightening.

(Note that ultimately, there’s still lots of gray area, and some solutions are better than others, but there’s at least a minimum bar for objectivity. You can never get rid of human judgement, of course)

Because there’s a strong signal for competence, as a result, you can create a series of “can you tie your shoelaces?” type questions which quickly suss out their level of expertise, and you can do this all in one phone call (or at the very least at the end of a couple hours together). In fact, if you search for “programming interview questions” you can get a sense for how concrete these interviews get – I think this is great.

Nontechnical interviews and weak signals for competence
Now let’s compare this to nontechnical interviews, which, in my expertise at least, generate weak signals for competence: Almost every interview process I’ve ever been involved with, whether I’m the interviewer or the interviewee, seems to lack the level of rigor that most engineers go through. Why is that? I imagine that much of it has to do with the fact that in nontechnical positions, it’s harder to decide objectively what’s “good” or “bad” – people often disagree on strategy, design, and it’s hard to figure out if you’re actually competent or not.

As a result, many of the nontechnical interviews I’ve seen tend degenerate into descriptions of previous work, or soft skills, or very subjective conversations around “how would you improve X or Y?” That’s not to say that these discussions aren’t useful, but for me personally, I’ve seen far too many people with 10 years of experience in some area that turn out to not be able to tie their shoelaces. The question is, how do you find this out sooner rather than later?

In fact, it may be that this entire discussion isn’t really about interviewing for technical people versus nontechnical, bur rather thorough interviewing versus sucky processes. Even then, I’d mostly argue that there’s a real issue that you can use objective tests in the engineering world to create strong signals of competence, whereas it’s much harder for marketing and product roles.

Crafting concrete interview questions for nontechnical roles
So what would a series of concrete tests look like for nontechnical roles? I would argue that you can rigorously test a couple key areas such as:

1. Can you create the deliverables that are part of the day-to-day role?
2. Are you familiar with previous relevant work in your area (whether you follow it or not)?
3. Can you demonstrate that you can do the thing you’re being hired to do?

Let’s drive into each one of these areas, as a thought experiment of what it’d look like to do an interview structure that’s as concrete as what most engineers have to go through:

Part 1: Can you create the deliverables that are part of the day-to-day role?
This is probably the closest that you can get to a coding question, at least for nontechnical people. The point is, most nontechnical jobs still do provide deliverables to other people in the company – for some, they will be spreadsheets, or documented product roadmaps, or launch schedules, or powerpoints, or whatever. The question is, can you have them sit down and craft a basic version of whatever deliverables they’ll be expected to create on the job?

Here’s an example: Let’s say that you were going to hire a product manager who needs to have a strong background in user acquisition via search engine marketing. Ideally, you should be able to sit them in front of a blank spreadsheet and they should be able to model out the user acquisition process from start to end. This means they’ll know how to think about the problem like a funnel, show the different steps, be able to roughly approximate what the numbers might be, and then calculate the cost per acquisition.

Or, if you have you interviewing for sales, they should be able to sketch out the basics of an RFP response, or build out a sales pipeline document, or make a list of sales collateral they might need, or whatever. If it’s someone from the marcomm world, then they should be able to sit down with you and craft a budget for making a splash at a tradeshow, or creating a schedule for a product launch, or whatever.

The point of all of this is that it’s a “tie your shoelaces” exercise that complements the soft skills discussion and meandering conversation about previous work.

Part 2: Are you familiar with previous relevant work in your area (whether you follow it or not)?
The next set of questions can be asked around how engaged they are in previous work in their area, regardless of whether or not they follow it. This area I’m often torn about, since there are often talented people who don’t know anything about historical precedence – but I do think that it demonstrates competence in the main. In concrete terms, I think that you can test for a couple specific things:

• Are they familiar with industry jargon in their field?
• Do they understand the theoretical underpinnings for what they’re doing?
• Have they read relevant books and blogs, attended conferences, or otherwise engaged in the discussion?

So for example, a product manager who focuses on go-to-market strategy should ideally be familiar with books like Crossing the Chasm or be aware of previous successes/failures in the tech industry. If the product manager is involved in the development process, you’d want them to be familiar with Scrum or Agile development, and ideas like the man-month. If they involved in product design, they would ideally know terms like visual language or affordances or Fitt’s Law.

As with the caveat in the previous section, I would ask these questions primarily to suss out expertise level and while it would contribute to a final hire/no-hire decision, it wouldn’t be the overriding factor. Ultimately some people are amazing decision makers on products without having formal training, but as entrepreneurship has a long history of failures, you’d ideally find people who were familiar with other situations that led to success or failure.

Part 3: Can you demonstrate that you can do the thing you’re being hired to do?
Similar to a programming interview to test programming skills, ideally you’d have the applicant tested in a way that most resembles their actual day to day job. That way you are testing them for their actual skills, rather than their self-reported skills. The trick to this, I think, is to break down the actual job description into specific areas that define the success or failure for them in the role.

For example, let’s take hiring a technical recruiter, whose day-to-day role you might break into:

• Prospecting (finding new candidates)
• Making contact and selling
• Evaluating skillsets
• Scheduling and interview coordination
• etc.

Ideally, the applicant would be tested using the real tools that they would use. If they are prospecting and using Linkedin, you’d give them a job description and ask them to pull up the site. Then you’d have them go through and try to find good candidates for you. Similarly, you’d ask them to pick a particular candidate and draft a high-quality, personalized request for them to come in. And so on.

When my girlfriend interviewed for IDEO, the global product design consultancy, they had her present her resume to a group of people in slide format and then take questions from a group. This is really smart, because of course a lot of her job is to take ideas, synthesize them down, and present it in groups. So I think that having her do that is a great test for competence in this area.

What’s next here?
I think the next step in this blog discussion would be to actually post some job titles and give rough formats for interviewing for that type. If I have time over the next couple weeks, I’ll write something up.

Has your company created a rigorous process for selecting non-technical hires? If so, I’d enjoy hearing more – please write a comment.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Leave this field empty if you're human: