Don’t Use Generative AI to Replace Discovery with Real Humans

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

I’m disappointed to see the rise of generative AI tools that are designed to replace discovery with real humans.

Don’t get me wrong. I’m a big fan of generative AI. I use it daily in both my personal life and at work.

But when we use generative AI to replace customer interviews, to generate opportunity solution trees, or to do our thinking for us, we fundamentally misunderstand the purpose of discovery.

When we use generative AI to replace customer interviews, to generate opportunity solution trees, or to do our thinking for us, we fundamentally misunderstand the purpose of discovery. – Tweet This

So I want to take some time to review why we do discovery. I’ll then share how and where I think generative AI can help, and clearly identify what we should avoid.

I also want to note that the world of generative AI is moving quickly. My advice in this article may not stand the test of time. I’ll do my best to update it as the technology evolves. But this is where I stand today.

Understanding Why We Do Discovery: It’s Not About Checking Boxes

We do discovery so that we can increase our hit rate for creating customer value while creating business value.

I break the underlying structure of discovery into three parts:

1. Starting with a clear desired outcome—in other words, starting with a clear definition of success.
2. Discovering unmet customer needs, pain points, and desires—AKA opportunities.
3. Discovering solutions that address those opportunities in a way that drives business value.

The outcomes represent business value. The opportunities represent customer value. Our job is to make sure that our solutions satisfy both.

I recommend teams conduct story-based customer interviews to discover opportunities and run assumption testing to discover the right solutions. And I like to use opportunity solution trees to visualize our work to help us stay aligned on what we think we know.

Interviewing helps us build empathy for our customers. When we interview continuously, we are continuously investing in our understanding of the humans who use our products. This understanding, over time, becomes a competitive advantage. The better we understand our customers, the better we can meet their needs, and the more we can differentiate our product from the competitors.

Assumption testing helps us ensure that we build the right solutions—solutions that fully satisfy an opportunity (creating customer value) in a way that drives our outcome (building customer value). We test desirability, usability, feasibility, usability, and ethical assumptions. With each test, we remove risk from our solutions and increase the likelihood they will work for both our customers and our business.

The goal is to ship value to customers in a way that creates value for our business. Everything we do in discovery is in service of that.

Along the way, we might create interview snapshots, experience maps, opportunity solution trees, story maps, KPI trees, or other visual artifacts. But these artifacts are not the end goal. They are tools that help us think through what we know. They help us uncover customer value and business value. They help us explore the diverse perspectives on our cross-functional team and then align around a shared understanding. These visuals are not the discovery work; they are the by-product of discovery work.

Too many generative AI tools miss this point. They think the artifact is the end goal. It’s why we now have tools that generate interview transcripts from synthetic customers, fictional interview snapshots and opportunity solution trees, and detailed story maps.

Again, I’m not a generative AI Luddite. I use these tools every day. I’m even using them in my own discovery work. So I want to share four guidelines to keep in mind when choosing generative AI tools to support your discovery efforts.

1. Don’t Let Generative AI Tools Replace Talking to Real Humans

When we use generative AI to generate interview snapshots from interviews with synthetic customers, we get a nice summary of pretend people. We learn nothing about the real humans who might use our product some day.

We don’t interview to learn about what people generally want. We interview customers to learn about their unique, specific stories about their real past behavior. These stories help us understand their actual unmet needs, pain points, and desires.

As we previously reviewed, interviewing helps us build empathy for the people who might use our products or services. We interview so that we can learn how we can positively intervene in their lives. The more we interview, the more we identify gaps between their expectations and the products they currently use. These gaps allow us to differentiate our products and services from our competitors’.

When we interview a synthetic user or create an AI-generated interview snapshot, we get generalizations. It’s very challenging to get specifics. Even when we do, they are made up. They don’t represent the needs, pain points, or desires of real humans.

When we interview a synthetic user or create an AI-generated interview snapshot, we get generalizations. It’s very challenging to get specifics. Even when we do, they are made up. – Tweet This

As long as you are building products for real humans, generative AI can’t replace interviewing real humans.

Still not convinced? Let’s look at some examples.

To start, I asked ChatGPT to tell me what are the common needs, pain points, and desires of Netflix customers. Here’s what it told me:

Okay, this might be helpful if I know absolutely nothing about streaming entertainment. But it’s not specific enough to be actionable.

I gave ChatGPT the benefit of the doubt and I asked for more specifics: What are some specific user interface issues that someone might encounter?

Here’s an excerpt from ChatGPT’s reply:

Take a minute and suppose you were on the search team. Is this helpful feedback?

Of course not. We need to know exactly what the customer was searching for and why the results weren’t relevant in order for this type of feedback to be actionable.

In other words, we need to know what a real human did that led to unsatisfactory search results and why those results were unsatisfactory.

I tried again. I asked: What are some reasons or use cases why the search results might be irrelevant?

If I was brand-new to search, this information might be helpful, as it can help me get up to speed on common challenges with search interfaces. But it’s not nearly specific enough to replace interviews.

Okay, let’s try another tactic. I asked ChatGPT: You are an engaged Netflix customer who enjoys watching TV at the end of your work day. Tell me about the last time you watched Netflix.

Me again: Set the scene for me. Where were you?

Does this sound like a real human to you? It sounds like the scene from a Crate & Barrel showroom. “A glass of my favorite drink?” Come on!

But I know there are doubters out there, so I continued: Okay, so you sat in your cozy living room. You turned on your TV. How did you get to Stranger Things?

And of course, ChatGPT obliged:

Me again: Tell me about your experience watching the episode.

Me still playing along: Did you encounter any challenges?

Okay, I have to say, ChatGPT is pretty good at writing fiction. But I’m not interested in fiction. I’m interested in real human stories.

ChatGPT is pretty good at writing fiction. But I’m not interested in fiction. I’m interested in real human stories. – Tweet This

I’ll spare you the details, but I asked for ChatGPT to change its persona and tell me another story. In total, I generated five stories all representing different “characters.”

In every story, the customer:

• Was happy with their Netflix experience.
• Loved the show they were watching.
• Had minor internet hiccups and were interrupted by something (e.g. a neighbor, their dog).

I pushed ChatGPT for more details, for more variation, and it struggled. Here’s why: These aren’t real people. They are fictional characters based on what ChatGPT can glean from its training data about what people like and dislike about Netflix. It’s not specific enough. It’s not real.

Even if it was, it doesn’t build empathy and it doesn’t help us differentiate from our competitors, because our competitors can get access to the exact same information.

Just in case you still need convincing, let’s look at a cleaned-up version of a real story I collected about a real human watching TV:

This is a human story. Humans are messy. We get distracted. We make mistakes. We aren’t overly positive about everything.

I’m going to say it again: If you are building products for humans, don’t replace interviewing real humans with interviewing generative AIs.

2. Don’t Use AI to Replace Thinking

I have now seen generative AI tools that will generate an interview snapshot, an opportunity solution tree, an experience map, and a story map from the click of a button. In case my opinion isn’t clear: I do not recommend using these tools.

We create these artifacts to examine our thinking, to align as a team, and to facilitate the synthesis process. The goal of these documents is not to create the document. The goal is to have the conversation, to do the examination, to do the thinking and the aligning. The goal is the process.

We create artifacts like interview snapshots to examine our thinking, to align as a team, and to facilitate the synthesis process. The goal of these documents is not to create the document. – Tweet This

If you create an opportunity solution tree from the click of a button, what value does it create? Does anyone on your team have any idea where the opportunities came from? Do they represent real customer needs? Does everyone agree with the outcome, the structure, the framing of each opportunity?

I know from arguing with people on social media that some of you are wondering, “But can’t I use a tool like this to solve the blank page problem—to get to a first draft—and then we can refine it as a team from there?”

My short answer is no. When we let AI do the first draft, we let AI dominate the conversation.

We’ve all been there. You are sitting in a team meeting, maybe you are tired or just not that interested in the conversation, you don’t really want to put in the work, so you let the most vocal person dictate the conversation. We all do it. There’s even an academic term for it—social loafing. Someone on your team is running with it, they have energy around it, what’s the harm in letting them take the lead?

The harm is that we don’t take the time to explore the diverse perspectives on our team. We don’t leverage everyone’s unique knowledge and expertise. We defer to the extrovert, the person who is willing to take the lead, the person who has a fresh coffee. Whatever the reason, it rarely leads to the best outcome.

Most of the work—most of the thinking—happens when we go from a blank page to a first draft. When we outsource this to generative AI, we let generative AI do all the thinking for us. And we’ve already seen generative AI isn’t specific enough, doesn’t know about real humans, can’t (yet) replace high-quality discovery work. When we rely on generative AI to generate the first draft, everyone on the team falls prey to social loafing. Don’t do this.

Most of the work—most of the thinking—happens when we go from a blank page to a first draft. When we outsource this to generative AI, we let generative AI do all the thinking for us. – Tweet This

3. Use AI to Analyze Large Data Sets (But Check Its Work)

You can use generative AI to help you make sense of large data sets. It can help you find insights and patterns in your behavioral analytics. It can do a sentiment analysis on your NPS verbatims. It can help you summarize and categorize your support tickets.

But it will make mistakes. So check its work.

I recently used ChatGPT-4o to help me analyze responses to a qualitative survey question. I had over 300 open-ended answers and I wanted to group and sort related responses together to get a better understanding of the most common themes.

First, I fed ChatGPT-4o the full list and asked it to find common themes. It did a mediocre job at this. The themes were ambiguous, they overlapped, and many didn’t make sense to me at all.

But it did solve the blank page problem for me. Seeing the half-baked themes helped me to start to identify my own themes. I identified six to eight themes myself and asked ChatGPT to sort all of the responses according to those themes. I also asked it to list any responses that didn’t match those themes. It did a pretty good job of this. I was then able to look at the ones that didn’t fit and identify additional themes.

Once I had a complete list of themes, I fed ChatGPT the complete list of themes and the individual responses and asked it to group them again. This worked great. I ended up with a list of responses grouped by theme.

To check its work, I made sure that the number of responses matched my original list and that there were no duplicates across the themes. This was fairly easy to do using Google Sheets and asking it to remove duplicates and then checking the counts.

In total, this took about one hour. Had I tried to group all of the responses myself, I easily would have spent three to four hours on this task.

But notice that I didn’t outsource the work to ChatGPT-4o. I collaborated with it. I iterated through several attempts until I was happy with the quality. I was engaged. I didn’t outsource my thinking. I used ChatGPT-4o to help with the tedious elements, but I still put in the work.

Let’s look at a second example. I recently used ChatGPT to analyze the engagement I get on my LinkedIn posts. I exported my top 50 posts on LinkedIn and added engagement data to them (comments, reactions, reposts, new followers). I categorized each post by topic and by type. I then fed this data to ChatGPT and asked it to create a heat map showing which posts were most successful at driving impressions, engagement, and new followers.

ChatGPT gave me a beautiful heat map, but I quickly realized my most successful posts were the types that I share most frequently. I realized that ChatGPT was using raw counts. I then asked it to redo the heat maps using averages for category and type.

This is an example where I still have to know what to ask for to get reliable analysis. Even with averages, the heat map was hard to read because the numbers were all over the place. So then I asked for normalized averages and I got heat maps that clearly showed my most successful categories and post types.

To check its work, I reviewed the Python code it used to generate the heat maps to make sure there were no mistakes in the math. And I manually calculated a couple of the values to make sure it matched my expectations. With the help of ChatGPT-4o, this took about two hours of focused effort. Had I done this on my own, it would have taken a day or two at least.

You can even use generative AI to help you get up to speed on a new market or a new customer segment. As we saw in the Netflix example above, if I was brand-new to streaming entertainment or to search, those initial responses would have helped me build my foundation. But don’t let it replace customer interviews with real humans—that’s where you’ll learn how you can differentiate from everyone else in the market.

Remember, good continuous discovery is a competitive advantage that compounds over time. Don’t outsource it to a tool that everyone has access to.

Remember, good continuous discovery is a competitive advantage that compounds over time. Don’t outsource it to a tool that everyone has access to. – Tweet This

4. Engage with AI Like It’s Another Member of Your Team

Discovery is a team sport. It works best when we work as a cross-functional team from the very beginning when deciding what to build. This is why I advocate for the product trio to do discovery together.

However, we only get the benefits of cross-functional collaboration when we leverage the different knowledge, expertise, and experience on the team. If we let one person do all of the work, dominate the conversation, or make all of the decisions, we don’t get the benefits of cross-functional collaboration.

In all of our courses, we teach teams how to collaborate cross-functionally by following a simple pattern. We have everyone do the work individually, before they meet as a team to discuss a team version. This requires that everyone puts in the effort and develops an individual perspective.

We then teach teams how to explore the diverse perspectives on their team. We encourage them to pay particular attention to the differences across the unique perspectives. Rather than advocating for one point of view, we encourage them to share how they developed their unique perspectives. Then we ask them to co-create a new team perspective.

This allows the team to leverage each of the unique perspectives, do the work to build a shared understanding, and then develop a stronger team perspective to work from moving forward.

For teams that follow this pattern, it can be extremely helpful to treat generative AI as if it’s another member of the team. Generative AI tools can help us brainstorm and they are reasonably good at generating story maps and identifying critical assumptions. They can even help us identify opportunities in interview transcripts.

But for each of these tasks, I don’t want to outsource the team work, but rather I want to treat the generative AI as if it’s another member of the team. So rather than blindly moving forward with its answer, we treat it like one of the many perspectives on our team and integrate it with the other perspectives on our team.

I use generative AI for these types of tasks almost every day. It helps me get out of my own head and see that there are other options. The same can be true for cross-functional teams.

It’s easy to get fixated on a single direction, have blindspots to our own assumptions, or miss a salient point in an interview. Generative AI can help us catch some of these errors. But again, I have to reiterate this: Don’t outsource your thinking to generative AI. Embrace it as a helpful member of your team, but do the work to figure out what you think first, and then use generative AI to augment your and your team’s perspective.

It’s easy to get fixated on a single direction, have blindspots to our own assumptions, or miss a salient point in an interview. Generative AI can help us catch some of these errors. –Tweet This

Final Thoughts

Don’t let this article discourage you from using generative AI in your work. I will reiterate that I use these tools every day. But do be thoughtful about how and when you use them. We don’t want to forget that as long as we are building for humans, we need to be engaging with other humans in our discovery efforts. Technology needs more humanity; not less.

As we are building for humans, we need to be engaging with other humans in our discovery efforts. Technology needs more humanity; not less. – Tweet This