For those of you following along, I’ve been conducting interviews to understand the life of a product manager.
If you need to catch up, you can see my initial call for participants here and my breakdown of how I am selecting participants here.
I’ll be blogging about both how I conduct the research and the trends and patterns that come out of it.
One of the biggest challenges with qualitative research is capturing the overwhelming volume of data you collect throughout the process.
To Record or Not
I have often recommended that people not record their interviews.
It’s rare that an industry research team has the time (or the money) to do full transcriptions and recordings are rarely listened to. So it felt unnecessary.
It also often led to lazy habits when it came to sharing the research with the rest of the team. Too often, I see teams share links to the raw videos rather than synthesizing key learnings.
While these limitations and risks are still true, I’ve recently changed my mind. I now always record my interviews and recommend that others do the same.
When I don’t record, I try to take verbatim notes. I want to capture exactly what the participant said as it’s my only record.
But the challenge (as we’ll see in a later section), is that we can’t synthesize while taking verbatim notes. If I can’t synthesize, I can’t ask good follow-up questions.
So I started recording sessions as a way of easing the burden of verbatim note-taking and of increasing my ability to stay present with the participant.
Record your interview sessions to ease the burden of taking verbatim notes. – Tweet This
Audio vs. Video Recordings
I only record the audio of my interviews. I find that video cameras are intrusive. They interfere with the comfort of the participant. When the participant isn’t comfortable, they don’t share as much.
Audio recorders are less intrusive and are quickly forgotten. However, I never hide the fact that I’m recording the audio from the participant. This is unethical.
In some states, the law requires that you tell a participant that the session is being recorded. But don’t limit yourself to the law—do the right thing and always tell your participants if you are recording.
Video recorders can be intrusive during interviews, whereas audio recorders are quickly forgotten. – Tweet This
For in-person interviews, I use this flash-drive recorder. It’s good enough to pick up a conversation in a conference room or at someone’s house. I like that it’s small and easily forgotten. It also has a long battery life and large storage capacity, so I don’t have to worry about running out of either.
For virtual interviews, I use Call Recorder for Skype. It does mean that I run the risk of having connection issues, but it’s far easier to record a Skype session than to record a phone call, so I deal with the connection issues.
Handwritten vs. Typed Notes
Even though I record the session, I still take notes during the interview.
Before I started recording sessions, I always took notes on my computer. It was the only way that I could capture verbatim notes.
But this introduced a new problem. It’s hard to build rapport with a participant when you are hiding behind a laptop. The participant will often pause to give you time to type, which means they are thinking about you instead of thinking about what they have to say.
Now that I record my sessions, I don’t take verbatim notes and I’ve switched to taking notes with pen and paper.
Even with pen and paper, you have to work to maintain frequent eye contact. I find this to be much easier in person than on virtual calls, but I work to maintain frequent eye contact regardless of the medium.
Regardless of how you take notes, maintain frequent eye contact with your participant. – Tweet This
I like these large sketch books and use whichever pen I can get my hands on.
I find that writing gives me freedom that typing does not. I can use space to set topics apart, use arrows to connect related topics, and can star pivotal moments.
Handwriting Notes Might Help You Synthesize More than Typing
Pam Mueller, a recent PhD student in social psychology at Princeton, and Daniel Oppenheimer, psychology faculty at Princeton, compared the impact of taking notes with pen and paper vs. typing on a computer.
They found that students who handwrite their notes did better on conceptual questions—meaning they were better able to process the information.
Mueller and Oppenheimer speculated, however, that this result was not necessarily due to the medium itself, but due to the fact that students who typed their notes tended to capture verbatim notes. While they captured more content, they did less work to capture it during the note-taking process, and thus processed less of it.
Because handwriting is slower, you have to decide what to capture, and this forces you to process the information. – Tweet This
In a follow-up study, they tried to instruct computer-using students to not take verbatim notes, as a way of teasing apart the impact of the medium vs. the note-taking style, but unfortunately the participants did not heed this instruction.
If you want to learn more about Mueller and Oppenheimer’s work, check out this Freakonomics episode.
I’m skeptical that the difference in comprehension is due to typing vs. handwriting and suspect that it is more tied to verbatim vs. synthesis. Regardless, there are enough benefits to handwriting for me that in interviews, I’ll continue to handwrite.
Since I’m not capturing verbatim notes, let’s turn to what I am capturing.
Capture Enough to Ask Good Questions
I need to write to think. If you’ve ever had a conversation with me in a room with a whiteboard, odds are I started scribbling all over it. Seeing helps me synthesize.
During an interview, I try to capture the structure of what’s being said. My goal is to look for gaps or inconsistencies, as these are areas I want to follow up on.
For example, just last week, I was talking to a product manager who said he was one of five product managers. He then walked me through how the product team was organized—who worked on which products. But he only described four product areas.
Since I had captured that he was one of five product managers and because I was drawing the structure of his org chart as he was describing it to me, I could ask him what the fifth product manager did.
I would have never caught this if I hadn’t been writing notes.
Capture the structure of what is being said. Look for gaps and inconsistencies. – Tweet This
By the way, I do this during phone conversations, too. I need to see to synthesize. My brain wants to outline. I want to see the structure to understand. And I find it immensely useful.
Capture Enough to Reflect Back What You Heard
A key tenet of active listening is to reflect back to the person what you heard. This does two things. First, it shows that you are listening, and second, it allows the participant to clarify any misunderstandings.
I don’t have a good short-term memory. My brain likes to jump around. I’m constantly trying to connect the dots and that means that sometimes I get jumbled up between what I just heard and my own thoughts.
Capturing the structure of what I hear helps me to stay focused on your part of the conversation. It also gives me a handy outline to refer back to when reflecting back what I heard.
With a glance at my notes, I can quickly summarize at any point in the conversation, before jumping into the next topic. This might look something like, “You’ve told me about your current role, how you got there, and what you like about product management. Tell me what you aspire to. Do you have any career goals?” Of course, if I’m doing this in an interview, I’ll include some specific details, so that the participant knows that I’m listening.
I don’t do this just when I’m switching topics, but also when I need clarification on something. I’ll start with what I heard, what I expect that to mean, and then ask about the part that I’m confused about.
For example, I might say, “You mentioned earlier that you do a lot reading at home, almost every day, but several times since you’ve mentioned that you don’t have time for other activities. What’s different about reading?”
Again, this shows that I’m listening and it allows the participant to clarify what they mean, and it helps me to ensure that I’m not misinterpreting what they said.
Reflect back what you heard. It shows that you are listening and allows the participant to clarify. – Tweet This
Capture What Needs Follow Up
I also capture what I want to come back to later so that I don’t have to interrupt.
For example, I ask every product manager to tell me about how an idea goes from concept to launch. Suppose someone says, “Ideas can come from anywhere. We capture them all in a Google Spreadsheet and prioritize them. For the top ideas we start to do research…”
I already have a series of questions.
- What are some examples of where ideas come from?
- How do you prioritize them?
- Who prioritizes them?
- What do you mean by research?
But I don’t want to interrupt the participant. He or she is on a roll and I don’t want to interrupt their thought process.
Thankfully, I’m sketching this process as they describe it, so it’s easy for me to return to the places where I need more information.
For something that isn’t as linear, I might just jot down a note in the margins to remind myself to come back to it when there’s an appropriate time in the interview.
Don’t interrupt your participant. Capture your thoughts and come back to them later. – Tweet This
Highlight Key Moments (And Other Notations)
With time, I’ve developed my own style of notations.
I put notable phrases in quotations. If the participant mentions an aside, I usually put it in parentheses.
Some of this is just good punctuation, but it’s easy to get sloppy in note-taking, and it’s important to know what your own punctuation means.
Oftentimes, in the midst of an interview something jumps out at me. Maybe it’s particularly salient, or maybe it’s unique.
I don’t want to just capture that moment. I want to highlight it so that it doesn’t get lost in the shuffle.
I usually draw a star next to a key moment. Again, something that is easier to do with pen and paper than with computer.
I also make note of the time so that it’s easier for me to find it on the audio recording later.
I also note when something was said in response to a specific prompt.
For example, if I ask you about the people on your team and you say, “We have five developers and one QA person,” and then I respond with, “What about UX?”, I’m going to capture that like this to indicate that I had to prompt for UX.:
– 5 developers
– 1 QA
– UX? Oh yeah, 1 UX
It tells me that you might not consider UX as part of your team.
Develop your own notation to denote key moments in the interview. – Tweet This
Denote Source Material from Your Own Thoughts
The most important notation I use is to distinguish my own thoughts from what the person said.
For example, I talked about using parentheses to note an aside from the participant. I could just as easily use parentheses for my own thoughts. But I don’t want to confuse these.
Because I used to code, I use // to start my own thoughts. This makes it crystal clear what came from my head vs. what the participant said.
So take the following notes:
“we do all the right things”
customer interviews, co-creation exercises
it leads to no “tangible product output” // outcomes?
This is a snippet from an interview where the person is explaining that they do all the “right things” when it comes to product discovery, but that they get no “tangible product output.” I put that phrase in quotes because it surprised me. I was expecting him to say outcomes (captured in my comment), because product discovery is usually tied to outcomes not output.
Note the mix of summary (the second line) and exact quotes (the first and third lines).
Clearly denote source material (what the participant said) from your own thoughts and reactions. – Tweet This
Review and Extend Your Notes Immediately
It’s easy to misinterpret your notes later. So I always take a few minutes to review my notes as soon as possible after the interview and add any missing detail.
Sometimes my handwriting is illegible and I need to rewrite some words.
Sometimes I already can’t tell what I meant by my note and I’ll make a note to go back and listen to that part of the interview again.
I try to do this immediately after conducting an interview. But interviewing can be tiring, so sometimes I need a break and forget to do it right away.
However, I always do this before my next interview. I don’t want new content to interfere with the interpretation of my old notes.
Review your notes immediately following the interview. Add any missing detail. – Tweet This
Catalog Notes for Later Retrieval
Cataloging your source material is just as important as taking good notes.
I use a post-interview checklist to make sure that I can find everything I need later. It includes:
- Create a note in Evernote for this interview
- Add a photo of the participant
- Capture the three or four most salient things from the interview
- Add photos of handwritten notes to the note
- Add any screen grabs or other artifacts from the interview
- Add the audio file to the note
- Send the participant details about his or her incentive
It’s been a lot of fun doing primary research again. I’ve missed it. I’ll keep blogging about the process, and eventually about the results. To follow along, subscribe to the Product Talk mailing list.
Zoe Tai says
Hi Teresa, thanks for sharing your experience so far! I really like that you included your own best practises as “how-to” guides can be very theoretical and difficult to apply without detailed examples. I particularly like your point about “Denote Source Material from Your Own Thoughts” as it’s not something I currently do but I can see how it would be useful in the future to bring up actual user quotes in internal conversations.
Teresa Torres says
Thanks for the comment. I’m glad you found it helpful.
Mike Savory says
This is a really good article with a lot of solid, practical advice. Regarding the video-recording of interviews, I like to have video so I can share compelling clips with those who didn’t attend, especially decision makers. Video tends to be much more persuasive than written notes.
Teresa Torres says
I can definitely see the value of this. But I think it’s a trade-off between how you want to use the research afterwards and what you might learn from it. I’ve seen too many times that video makes people uncomfortable and thus not as open. I wonder if you could get the same benefit by showing a photo of the person and playing audio.
Joe Kleinwaechter says
Great stuff, Teresa. I have found that facial expressions say so much more than words can convey in expressing emotional elements and give some great visual clues to the cognitive processes. If I have enough observers, I will typically pick one person out to focus only on taking notes on body language and facial expressions.
While I definitely prefer video, less obtrusive video like a built-in webcam or Skype (as you pointed out), where they are not constantly reminded of the camera works much better than a physical camera they can see.
Teresa Torres says
I agree. Facial expressions, tone of voice, body language communicate so much. If we can capture on video great, if not, I think it’s important that the interviewer is not taking verbatim notes so that they have the mental capacity to notice all of these things and properly interpret what they are hearing.
Diana Tobey says
Something I’ve found to be effective is recording audio, and then capturing video of key things such as:
– introducing themselves
– retelling a poignant story
– answering a specific prompt that’s relevant to the research
This makes it really easy to edit these sound bites into a short video that gives non-attendees a feel for who that person is even though they couldn’t attend in person.
I find that including one more person in the interview process as notetaker is key. Doing point of interest notes or quick diagrams for your own clarification and processing is great but I find trying to do meaningful facilitation while taking anything more then very light notes is impossible. Sure you can do it but the quality on both ends suffer.
Having a second team member also starts to foster the idea of design collaboration and shared understanding. A developer will hear things differently then you do and provide valuable insight.
Discussing the interview immediately after the interviewee leaves, with a person who was focused on note taking, really ups the clarity of analysis. You get to see where you clearly agree and where you each may have missed important nuggets of gold.
Lastly you can also switch roles between interviews as facilitation is exhausting.
I find by adding this note taker the use case for recording drops dramatically (I still do audio recording with my phone but almost never go back or have ever seen a client actually go back and listen). It is nice to have the customer voice if you really want to pull something out to validate an assumption with absolute clarity with the team or stakeholders.
Just found the blog and am loving it so far. Good work.
Teresa Torres says
This is a GREAT point. My recent batch of research has been done on my own. But when I was an employee (and as a coach I encourage the same), I always include others in the research process. And having them take notes is a great way to keep them engaged and ease the burden of facilitating.
I also love that you mentioned discussing the interviews after, as each will hear different things and you want to capture all of this.
Thanks for the great comment.
I agree with Mike, two people allows for a dedicated note taker and you can switch periodically which is more energizing for the conversation and gives you a chance to de-brief immediately after the interview. I also find it useful to send a summary to the interview subject; they will often elaborate on points they made, point out mistakes in your note or understanding, or clarify what they intended to say. See http://www.skmurphy.com/blog/2011/10/19/tips-for-b2b-customer-development-interviews/ and http://www.skmurphy.com/blog/2014/11/21/customer-interviews-how-to-organize-findings/ for some more suggestions for how to conduct interviews and organize findings
Laure (@laurex) says
I take some handwritten notes, mostly to remind myself of things I’d like to explore further in the conversation. Otherwise, I do record and run the recording (audio) through rev.com for a transcript. This has changed my life (no, really). Now, I can listen much more naturally and carefully and not worry about missing something or being biased by my thoughts as things are unfolding it when I look at my notes later. I’ve definitely noticed that interviews feel much more comfortable this way. Most of my interviews are via Zoom, which has recording built in, and I use Evernote to record audio in person.
Thank you for the tip on clearly separating notes from your own thoughts! Very valuable!