AI vs Human Interviews: Which is better for win-loss?

AI technology has come on so much in a couple of short years that AI interviews are now a very viable tool for win-loss – but their success depends on your business’s goals, your ICP, deal volume and many other factors. This article helps you evaluate the benefits and drawbacks of both AI and human interviewers, and where both can be best deployed for maximum effectiveness.

1. Overview: AI vs Human interviewing
2. Where human interviewing is still significantly better
3. Where AI wins
4. Some recommendations
5. Conclusion

Overview: AI vs Human interviewing

AI interviewing has come a very long way in a very short space of time. Only a couple of years ago, the idea of asking a B2B buyer to have a conversation with an AI interviewer would probably have seemed either gimmicky or slightly ridiculous. Today, AI interview tools can ask questions conversationally, respond to what an interviewee has said, probe answers and generate a usable transcript and analysis afterwards.

Unsurprisingly, this has generated a lot of interest in the win-loss world, and leading vendors such as Clozd and Klue now actively offer AI interviews – often referred to as ‘flex’ interviews or similar – alongside human interviews as part of their overall win-loss packages. In fact, for Clozd, AI interviewing is now increasingly the leading feature of their product proposition.

Given AI interviews have become so mainstream, to save you scrolling through the rest of the detail in this article, here’s a summary table:

Summary: Human-led vs AI-led interviews

(Source: Clearsighted conversations with win-loss buyers including Product Marketers and B2B Sales leaders, plus secondary research from public sources).

Where human interviewing is still significantly better

Better quality and depth of insights overall

Although AI is evolving rapidly, human interviewing still leads the way in terms of the quality of insights uncovered. This is because:

• Humans still respond better to other humans asking questions
• Humans are still more skilled interviewers than AIs (especially when it comes to probing, judging reactions and hesitations, and asking difficult questions).
• Able to pivot quicker, and adapt lines of questioning to distinctive personas, industries and more.

Critical to this, of course, is making sure your win-loss provider has strong HUMINT expertise and track record, as well as knowing they are well briefed on your business goals and blindspots.

More tailored to the business and product – including current assumptions and future goals

A good win-loss provider should take the time to set up any program properly, including gathering current assumptions about reasons for wins and losses, current blindspots and future goals.

The human interviewer can then proactively weave ideas into their lines of questioning that uncover answers for these. Plus, they’re able to challenge where necessary on why a decision was really made, or what the implications of X action were.

More actionable, continuously improving

Good human interviewers/ win-loss analysts not only present the insights from human research, but make clear and tailored recommendations for improvement. These recommendations are based purely on your own data, without the data of other businesses influencing things.

There’s the added bonus that you can question, ask for clarification or even challenge your human interviewer on results – and discuss how the quality of the program can be continuously improved.

Use human interviews for…

• Gathering deep, nuanced detail from important (ARR) or strategic deals
• If you have a small total sales volume, but these are all long sales cycle, complex enterprise deals
• Sensitive or senior buyers
• Areas where you know you have a blindspot, and need that extra layer of probing or challenge

Where AI interviews win

Cost-effectiveness

This is a big one… the biggest. AI interviews can cost as little as $20 per interview – substantially lower than a human-led interview which takes into account paying both the interviewer plus (usually) an incentive for the interviewee. If cost is your main consideration, AI wins; a win-loss program comprised entirely of AI interviewing will be far cheaper than one where you have a human analyst on call, and there are some decent tools out there to achieve this, including Outset and Conveo.

Note: Some of these platforms use digital twins or synesthetic users. There’s very mixed data out there currently about the efficacy of synthetic users, including from the Neilsen Norman group and Martech.

Speed

Because AI interviews can be completed whenever the respondent has a spare few minutes, and at their total convenience, feedback can be gathered quickly rather than having to go through the rounds of finding 45mins in a busy schedule. This means quite often feedback can be gathered quickly, often very close to when the deal has closed and not weeks later.

While it shouldn’t make a big difference if it’s the day after or a couple of weeks after (your buyer shouldn’t forget their feedback that soon!), I do sometimes come across win-loss agencies that are speaking to buyers months and months – usually 6-10 months – after a deal has closed. Obviously, that’s far from optimal, especially if you’re after deep feedback on the product features or demo.

Breadth of pipeline

Because they are cheap, take little time and invites can be sent out en-masse, AI interviews can be useful for plugging the gaps between human-led interviews and gathering feedback cost-effectively from lower priority deals. They can be ideal for smaller ARR deals, less strategic deals, or deals that were less well-qualified or stalled earlier in your pipeline (these respondents may have less interesting or more one-dimensional feedback, anyway).

Be aware that response rates will still vary enormously. A number of buyers I’ve spoken to say that AI interviews typically generate, on average, 5 minutes of respondent talk time. Across lots and lots of deals, this can provide very real and useful insights. Across a smaller volume, it probably amounts to something that’s not driving a lot of value.

Some recommendations

Don’t defer very senior or commercially-sensitive buyers to AI

If your buyer is Director-level, VP or C-suite – especially at a large company – get a human to do it. Buyers like this are exceptionally busy, and offering them a human interview first and foremost shows that you really value their time and expertise. If they’re too busy to speak for 45mins, an AI interview will be a great fallback – but you have to understand you’ll get ten minutes’ worth of insights at most from them.

From experience, if you want the answers to difficult questions which may be somewhat sensitive (e.g. pricing, contractual terms), you’re always better off using a human to build rapport, gain trust, and ask these questions in a skilled and considerate manner. This way, you are much more likely to get the honest answers.

If you’re using a vendor that has CRM automations that send out AI invites as soon as deal has closed, you might want to discuss how you can tailor what kind of buyer these get sent out to.

Experiment with AI before a proper roll out, or use it as a great backup option

Just as with human interviews, response rates with AI can vary a lot. I’ve spoken to some Product Marketers and sales execs who are really happy with the results they’re getting, but others complain of interviews that last less then 5 minutes, where follow-up email requests for ‘more detail’ are subsequently ignored.

If you’re serious about using AI interviewing, it can be a great idea to try it for a trial period as attitudes towards AI (some buyers will be happy to chat and find it a more convenient option, others will not touch it with a barge pole) can vary substantially.

Adjust analysis for human vs AI interview output

The insights gained from both methods will be different. You’ll need to think about how best to present these findings, and make sure that data is being routed back into the business in an actionable way.

AI will gain lighter/ more shallow insights over a greater breadth of your pipeline. These insights may be best presented in a graphical format: for example, graphs and charts (with supporting verbatims), visual representations of themes or dashboards so you can track things at a big picture level.

For human interviews, insights will be deeper and more nuanced but more suited to executive readouts or thematic analysis with supporting verbatims. Actionability probably looks like AE deal reviews, getting recommendations into sales enablement cadences, roadmapping sessions or board reports.

Conclusion

As with anything, it’s a case of using the right tools for the job. Businesses must balance the need for great quality win-loss interviews that uncover the full range of insights on offer from buyers, versus the requirement to touch a higher number of closed deals, in a cost effective way. To repeat my earlier recommendations, a mix of the two can work well:

Human-led interviews for strategic deals (high ARR, important logo etc), enterprise of sensitive ICPs, or complex technical products. Use ideally for deals that have made it far down your pipeline – for example, if you were in the final two vendors vs a competitor.

AI interviews are great for less important, lower ARR, less well-qualified or earlier in the sales process deals. Also good for SMB and mid-market deals, where decision making and budget approval is less complex. Hopefully, if you get a good response rate, this should yield insights with a decent quality vs cost ratio.

If you’d like to discuss where a targeted set of human interviews can work and generate substantial value, talk to us.