Andrew Chen Archives

Why do people interview technical and nontechnical people differently? Nerds vs jocks

Interviewing engineers and objective tests for competence
Recently, I’ve been meeting a number of engineers to find more people I might want to work with. I think as a whole, most tech companies do a pretty good job interviewing engineers because at the very least, there are objectively correct answers to programming questions. In particular, you can give engineering questions which result in code that either runs or doesn’t, and having an interview candidate code with you for an hour is pretty enlightening.

(Note that ultimately, there’s still lots of gray area, and some solutions are better than others, but there’s at least a minimum bar for objectivity. You can never get rid of human judgement, of course)

Because there’s a strong signal for competence, as a result, you can create a series of “can you tie your shoelaces?” type questions which quickly suss out their level of expertise, and you can do this all in one phone call (or at the very least at the end of a couple hours together). In fact, if you search for “programming interview questions” you can get a sense for how concrete these interviews get – I think this is great.

Nontechnical interviews and weak signals for competence
Now let’s compare this to nontechnical interviews, which, in my expertise at least, generate weak signals for competence: Almost every interview process I’ve ever been involved with, whether I’m the interviewer or the interviewee, seems to lack the level of rigor that most engineers go through. Why is that? I imagine that much of it has to do with the fact that in nontechnical positions, it’s harder to decide objectively what’s “good” or “bad” – people often disagree on strategy, design, and it’s hard to figure out if you’re actually competent or not.

As a result, many of the nontechnical interviews I’ve seen tend degenerate into descriptions of previous work, or soft skills, or very subjective conversations around “how would you improve X or Y?” That’s not to say that these discussions aren’t useful, but for me personally, I’ve seen far too many people with 10 years of experience in some area that turn out to not be able to tie their shoelaces. The question is, how do you find this out sooner rather than later?

In fact, it may be that this entire discussion isn’t really about interviewing for technical people versus nontechnical, bur rather thorough interviewing versus sucky processes. Even then, I’d mostly argue that there’s a real issue that you can use objective tests in the engineering world to create strong signals of competence, whereas it’s much harder for marketing and product roles.

Crafting concrete interview questions for nontechnical roles
So what would a series of concrete tests look like for nontechnical roles? I would argue that you can rigorously test a couple key areas such as:

1. Can you create the deliverables that are part of the day-to-day role?
2. Are you familiar with previous relevant work in your area (whether you follow it or not)?
3. Can you demonstrate that you can do the thing you’re being hired to do?

Let’s drive into each one of these areas, as a thought experiment of what it’d look like to do an interview structure that’s as concrete as what most engineers have to go through:

Part 1: Can you create the deliverables that are part of the day-to-day role?
This is probably the closest that you can get to a coding question, at least for nontechnical people. The point is, most nontechnical jobs still do provide deliverables to other people in the company – for some, they will be spreadsheets, or documented product roadmaps, or launch schedules, or powerpoints, or whatever. The question is, can you have them sit down and craft a basic version of whatever deliverables they’ll be expected to create on the job?

Here’s an example: Let’s say that you were going to hire a product manager who needs to have a strong background in user acquisition via search engine marketing. Ideally, you should be able to sit them in front of a blank spreadsheet and they should be able to model out the user acquisition process from start to end. This means they’ll know how to think about the problem like a funnel, show the different steps, be able to roughly approximate what the numbers might be, and then calculate the cost per acquisition.

Or, if you have you interviewing for sales, they should be able to sketch out the basics of an RFP response, or build out a sales pipeline document, or make a list of sales collateral they might need, or whatever. If it’s someone from the marcomm world, then they should be able to sit down with you and craft a budget for making a splash at a tradeshow, or creating a schedule for a product launch, or whatever.

The point of all of this is that it’s a “tie your shoelaces” exercise that complements the soft skills discussion and meandering conversation about previous work.

Part 2: Are you familiar with previous relevant work in your area (whether you follow it or not)?
The next set of questions can be asked around how engaged they are in previous work in their area, regardless of whether or not they follow it. This area I’m often torn about, since there are often talented people who don’t know anything about historical precedence – but I do think that it demonstrates competence in the main. In concrete terms, I think that you can test for a couple specific things:

• Are they familiar with industry jargon in their field?
• Do they understand the theoretical underpinnings for what they’re doing?
• Have they read relevant books and blogs, attended conferences, or otherwise engaged in the discussion?

So for example, a product manager who focuses on go-to-market strategy should ideally be familiar with books like Crossing the Chasm or be aware of previous successes/failures in the tech industry. If the product manager is involved in the development process, you’d want them to be familiar with Scrum or Agile development, and ideas like the man-month. If they involved in product design, they would ideally know terms like visual language or affordances or Fitt’s Law.

As with the caveat in the previous section, I would ask these questions primarily to suss out expertise level and while it would contribute to a final hire/no-hire decision, it wouldn’t be the overriding factor. Ultimately some people are amazing decision makers on products without having formal training, but as entrepreneurship has a long history of failures, you’d ideally find people who were familiar with other situations that led to success or failure.

Part 3: Can you demonstrate that you can do the thing you’re being hired to do?
Similar to a programming interview to test programming skills, ideally you’d have the applicant tested in a way that most resembles their actual day to day job. That way you are testing them for their actual skills, rather than their self-reported skills. The trick to this, I think, is to break down the actual job description into specific areas that define the success or failure for them in the role.

For example, let’s take hiring a technical recruiter, whose day-to-day role you might break into:

• Prospecting (finding new candidates)
• Making contact and selling
• Evaluating skillsets
• Scheduling and interview coordination
• etc.

Ideally, the applicant would be tested using the real tools that they would use. If they are prospecting and using Linkedin, you’d give them a job description and ask them to pull up the site. Then you’d have them go through and try to find good candidates for you. Similarly, you’d ask them to pick a particular candidate and draft a high-quality, personalized request for them to come in. And so on.

When my girlfriend interviewed for IDEO, the global product design consultancy, they had her present her resume to a group of people in slide format and then take questions from a group. This is really smart, because of course a lot of her job is to take ideas, synthesize them down, and present it in groups. So I think that having her do that is a great test for competence in this area.

What’s next here?
I think the next step in this blog discussion would be to actually post some job titles and give rough formats for interviewing for that type. If I have time over the next couple weeks, I’ll write something up.

Has your company created a rigorous process for selecting non-technical hires? If so, I’d enjoy hearing more – please write a comment.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

I am often pitched with new blog widgets and services to try out. Most of the time, I won’t try it, unless it falls into one of two categories:

1. It helps me build my readership and audience, particularly email and RSS subscribers or Twitter followers
2. Or, it helps me understand my audience, and even better, creates a way for me to start a dialog with them

The best tools do both. If you have one like that, then send me a note at voodoo at gmail. I am likely to try it out!

Money doesn’t matter
Note that monetization doesn’t factor into my decision above – and that’s because I didn’t start this blog to make money ;-) Instead, I am writing it because I enjoy the process, it helps me structure my thoughts, and I often meet interesting people through my blog.

When I’m thinking about a problem and am willing to make it public, a quick post or two about the topic often brings world class people to the forefront, and it’s a lot easier to learn from people who are much smarter than me than to try to figure things out from first principles in a vacuum. 

Plus, making money from writing a niche blog is hard, with the only viable method being direct monetization via ebooks or something similar. You definitely need scale to make advertising work.

For the above reason, over time, I’ve come to trust on a number of tools that I can’t live without:

1) Feedburner, for email subscriptions
There are certainly better email feed managers, but I built my initial audience with this, and it’s working well enough. Every time someone subscribes by email, they enter in their address. This becomes a valuable clue and I often will google email addresses or look them up on Linkedin, just to know. And sometimes, it becomes a trigger for me to send an email and introduce myself. That’s pretty good. The biggest plus would be if there was a way to

2) Twitter, for follower bios
Twitter has a similar affect, except people are much more likely to follow on Twitter than give you their email address. Similarly, Twitter has a nice “bio” that you can eyeball for details about the person. I’ve also recently discovered Tweepsearch, which lets me do things like search for any of my followers who do iPhone work and contact them. The point is, it makes it so that blogging becomes a great tool for me to massively (but passively) build a big business network – and the corners of this network may not be helpful today, but maybe one day ;-)

3) LinkedIn, for business backgrounds
LinkedIn has some of the same properties as above, but also has detailed info. On the other hand, it absolutely skews towards marketing and business professionals. I often find that a lot of engineers don’t have Linkedin accounts.

And by the way, Google Friend Connect sucks
I recently tried out Google Friend Connect, and it definitely sucks using the values above. If you go to my blog now, at the bottom there’s a “Social Bar” where you can “join” this blog (whatever that means). Since implementing it last night, I’ve had a couple people join, but it’s basically worthless. There’s really no profile to speak of, and there’s no way for me to reach out to interesting people, even if I could figure out whether they were interesting or not. What’s the point of having people join then, if I can’t do anything with the audience I build?

Disqus has a little bit of this also – people often comment on stuff, and sometimes their comments will be really interesting. But when I go to a Disqus profile, there’s often little to know information, and certainly no way to contact them. Instead, I just have to reply in the public comments, which is pretty kludgey and certainly not a “passive” way to go about doing this.

Stop focusing on monetization, at least for 99% of bloggers!
My uber point on this, ultimately, is that 99% of bloggers don’t make any money from their blogs, and the rewards for their work are meeting interesting people as well as building up a following. Things that help pierce the veil of anonymity (with reader consent, of course) are the most useful since it helps on all the non-monetization goals.

Anyway, if you have a useful widget or plugin for me to try that fulfills any of the above, please shoot me an email.

Hi everyone,

Hopefully all of you are enjoying the blog although I’ve only been writing about once a week now. We recently passed 7,000 subscribers, which is fantastic. The blog continues to grow and seems to be a useful reference for people.

Anyway, as some of you know, I’ve been working on some of my own startup projects – still quite early – and am now looking for recommendations for smart engineers.

In particular, I’m looking for advice on:

• Engineers that are interested in consumer internet startups
• Have experience with Ruby (or PHP/Python) including frameworks like Rails or Django
• Can work with me on customer-centered product development
• Have a BS/MS in Computer Science or equivalent experience

If you have recommendations for interesting people for me to talk to, please shoot me a note at voodoo [at] gmail.

I’d hugely appreciate it!

Thanks,
Andrew

Answer this question honestly…
When’s the last time you spoke to your target customer? Like really talked to them?

If it’s been more than a month, then shame on you!

Consumer internet companies are often overly dependent on quantitative data like Google Analytics, but without understanding the qualitative parts – the consumer psychology that actually goes into making purchase decisions. It’s a good idea to balance out the data aspects, particularly if you are not your target customer.

If you haven’t finished developing your product yet, that’s no excuse! After all, there are many methods of doing qualitative user research without writing a single line of code. In fact, in many ways talking to your customer and understanding them great detail is often much more powerful before you even go through the product development process.

How to recruit target customers to talk to, in 5 easy steps
It’s very very easy to talk to people on the internet. You really don’t have to do much work. Here’s what I will often do, in order to get some opinions about a particular set of products, or to deeply understand user behavior (like gifting! or decorating), or to get a better picture of what people do day to day.

Step 1: Write a recruiting survey
First off, go to Wufoo.com or a similar site (Surveymonkey.com works well too).

The most important part is to title the survey “Get a $20 Amazon gift certificate for 1 hour on the phone” or something similar.

Make a survey that includes the following questions:

• First name (text)
• Phone number (phone number)
• Email so we can send you a gift certificate (text)
• Best time to call, morning, afternoon, evening, weekend (multiple choice)
• Tell me about yourself! (textarea)

That is usually a good base, and you should make all the entries required. Then you also want to provide a couple questions that can help you screen or otherwise prioritize your questions. For example, for a Facebook app you might ask:

• What types of games do you like (multiple choice) 
• What kind of phone do you have? (multiple choice)
• Why do you like game X? (textarea)
• Have you ever spent money on a game? (multiple choice)
• etc.

Anyway, you get the point. I usually try to keep these pretty short.

Step 2: Recruit your participants
Now that you have a survey set up, then you can take the URL and start getting people to fill it out. There are a couple obvious areas to recruit people, and I typically do the following:

• Link the survey from your product (if it’s out there)
• Buy ads on Facebook and send traffic to your link
• Post your survey on Craigslist
• Buy ads on Google Adwords and send clicks to your survey

For the ad-based solutions, I will usually limit the buy to $50 per day, and spend $0.50 or so per click. I usually find that it costs about $1-2 per survey completion. After I recruit a couple dozen, then you can start moving forward with the call.

Step 3: Do your phone interviews and learn something!
This where you’ll learn the most – you can just pick up the phone and start talking. I usually structure the interviews into a couple distinct sections, depending on what I’m trying to learn.

The first section I usually try to learn about basic internet usage:

• Tell me about yourself
• What’s your typical day like?
• Tell me about your computer setup – what do you have? When do you use it?
• What are your favorite internet sites? What sites do you use every day?

Then depending on the topic, I’ll usually drill into 3 or 4 different areas with a couple questions each. The entire point is to ask open-ended questions without leading them too much. I will do as many of these calls as makes sense until I am hearing the same information over and over. Then I’ll start tweaking things and changing the interview to adjust.

Also, I will usually not show them a product unless the entire discussion is focused on that – the point of these conversations for me is usually qualitative understanding, not usability. Having them thoroughly test competitive products can be interesting also. You want to use this information to drive product strategy, and not be reactive.

I guarantee you’ll learn something!

Step 4: Buy your interviewees a gift card
When you’re done, don’t forget to send your interviewees a gift certificate – $20 card from Amazon is a good idea – to thank them for their time.

One of the best things is that once you get some relationships going with the best interviewees, you can go back to them for updates or to identify some of the most extreme cases.

Conclusion
The point is, it’s easy to talk to people, and it’s this type of detective work that separates customer-focused companies from technology-driven ones. There’s even a fun tool to suggest a bunch of other methodologies like this also – the IDEO method card deck.

If you have other additions to this, please suggest in the comments!

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.