Amit chakraborty

Posted on Oct 21 Originally published at amitchakraborty.dev

Product management for engineers · 2. Discovery: talking to users without leading them

#tutorial #product #productmanagementforengineers #howdoiavoidleadingtheuser

Product management for engineers · Chapter 2 of 10 · Product · new chapter every Wednesday morning

By the end of this chapter: Run a user conversation that produces evidence instead of agreement.

The problem

You are a senior backend engineer at a fast‑growing SaaS startup. The next sprint is due in two weeks and the team has a feature – “custom report scheduling” – that the sales team has promised to a key prospect. The product manager, Maya, has spent the morning shuffling between a stakeholder‑alignment call, a user‑research debrief, and a data‑review session. By 11 am she emails the engineering channel: “We need to push the UI changes to the API tomorrow – the client is asking for a demo on Friday.” At the same time, a separate Slack thread shows her updating the roadmap to move the “export to CSV” work to Q3.

You have already started writing the API endpoint, but the design rests on an assumption Maya took from a user interview earlier that day: “Users want to set up recurring reports because they forget to run them manually.” When you ask Maya for the raw interview notes, she sends a three‑sentence summary that reads like a sales pitch, not a transcript. You cannot tell whether the prospect actually expressed that need, whether the need is common across the user base, or whether the quoted need is a mis‑remembered interpretation.

The cost of proceeding on that assumption is immediate – you will spend a week building an API that may never be used, delaying other work and risking the demo that the prospect expects. The longer‑term cost is more subtle: the engineering team will have to re‑work the endpoint, the product roadmap will be reshaped around a possibly spurious request, and the credibility of the discovery process will erode if the feature lands unused.

You need a reliable way to talk to users that yields evidence – concrete statements, frequencies, and pain points – rather than agreement that merely mirrors what the PM thinks the user wants.

Before you start

Prerequisite	Version / Tool
A meeting‑recording tool that can capture audio locally (e.g., ffmpeg 5.1.2)	`ffmpeg -version` should output `ffmpeg version 5.1.2`
Consent‑capture template (see Appendix A)	plain‑text file `consent.txt`
A note‑taking app that supports markdown export (e.g., Obsidian 1.5.12)	open Obsidian, create a new vault, verify you can export a note as `.md`
Access to at least two representative users (internal or external)	confirm you have their contact details and a calendar invite
A quiet room with a laptop, external microphone, and headphones	run `arecord -l` (Linux) or `system_profiler SPAudioDataType` (macOS) to list devices

One‑off verification command

Run the following to ensure the recording pipeline works:

ffmpeg -f avfoundation -i ":0" -t 5 -y test.wav && ffprobe -i test.wav -show_entries format=duration -v quiet -of csv=p=0

Expected output: 5.000000. If you see an error, reinstall ffmpeg before proceeding.

How do I avoid leading the user?

1. What does “leading” actually look like in a live interview?

A leading question nudges the interviewee toward a particular answer. Examples:

“You said you often forget to run reports, right?”
“Would it be helpful if we added a ‘schedule once a month’ button?”

These questions embed the interviewer's assumption and give the user a cue to agree. The evidence you collect becomes agreement rather than truth.

2. Which question types produce raw evidence?

Question type	Example (no leading)	What it yields
Contextual	“Can you walk me through the last time you generated a report?”	Step‑by‑step workflow, tools used, pain points
Problem‑focused	“What frustrations did you encounter while generating that report?”	Specific pain, frequency, impact
Frequency	“How many times per week do you need to run that report?”	Quantitative data
Alternative	“If you could change one thing about the current process, what would it be?”	User‑prioritised improvement ideas

3. How do I keep the interview focused on evidence?

Use a script – a checklist of question types (see Appendix B).
Record, don’t rely on memory – the audio file is the source of truth.
Ask for concrete numbers – “How many reports do you run in a typical month?”
Avoid “why” after a statement – instead ask “What happened when you tried X?”

How do I structure the interview to surface evidence?

1. What preparation steps guarantee a clean session?

Step	Action	Verification
1️⃣	Send the consent template and ask the user to sign electronically before the call.	Received signed PDF or email reply with “I consent”.
2️⃣	Create a shared calendar invite with a clear agenda: “Current reporting workflow – 45 min”.	Invite accepted, agenda visible.
3️⃣	Test the microphone and recording command (see verification above) 15 min before the call.	`ffprobe` reports > 0 seconds of audio.
4️⃣	Prepare a one‑page “research brief” that lists the hypothesis you don’t want to test (e.g., “users want recurring reports”).	Brief is stored in the vault, not shared with the interviewee.

2. What does the interview flow look like in practice?

Opening (5 min) – thank the user, confirm consent, explain that you are learning about how they work, not what they think they should want.
Contextual walk‑through (15 min) – use the “Contextual” question type to let the user narrate a recent reporting task.
Pain extraction (10 min) – ask “What frustrations did you encounter?” and probe for specific moments (e.g., “When you clicked Export, what happened?”).
Frequency check (5 min) – ask concrete numbers about report generation.
Closing (5 min) – ask for any “wish‑list” items, but frame them as “If you could redesign one step, what would it be?” and note that this is future‑looking data, not current pain.

3. How do I capture the evidence in a reusable format?

After the call:

Export the audio – ffmpeg -i interview.wav -ac 1 -ar 16000 interview.flac.
Transcribe – use a speech‑to‑text service (e.g., OpenAI Whisper) and save the raw transcript as interview-<date>.md.
Highlight evidence – in the markdown, prefix each evidence line with > **EVIDENCE:** and include the timestamp, e.g., > **EVIDENCE (00:12:34):** “I run the monthly sales report three times a week.”
Summarise with counts – at the bottom of the file, add a table of frequencies (e.g., “Report runs per week – 3 users said 3+, 2 users said 1‑2”).

The transcript becomes the single source of truth for any downstream discussion.

Build it

Below is a concrete, end‑to‑end workflow you can run today. It assumes you have the prerequisites from the Before you start section.

Create the consent file

   cat > consent.txt <<'EOF'
   Subject: Interview Consent
   Date: $(date +%F)
   I, ______________________, consent to being recorded for product research purposes.
   Signature: ______________________
   EOF

Why: Legal and ethical baseline.

Confirm: Open consent.txt and verify the placeholders are present.

Send the consent request

   echo "Please sign the attached consent and reply." | mail -s "Interview consent" -A consent.txt user@example.com

Why: Ensures you have documented permission before recording.

Confirm: Check your sent mail folder; you should see the email with attachment.

Schedule the interview

   echo "Meeting: Reporting workflow – 45min\nWhen: $(date -d '+2 days' +%F'T'15:00)" >> meetings.ics

Why: Calendar invites provide a shared reference point.

Confirm: Open meetings.ics in a calendar app; the event should appear with the correct time.

Test the recording setup (15 min before the call)

   ffmpeg -f avfoundation -i ":0" -t 5 -y test.wav && ffprobe -i test.wav -show_entries format=duration -v quiet -of csv=p=0

Why: Guarantees you will capture the interview.

Confirm: Output 5.000000. If not, troubleshoot microphone drivers.

Run the interview – start recording at the exact start time:

   ffmpeg -f avfoundation -i ":0" -ac 1 -ar 16000 -y interview-$(date +%F).wav

Why: Captures a lossless mono track at 16 kHz, optimal for transcription.

Confirm: When the interview ends, stop ffmpeg with q. Verify file size > 500 KB.

Transcribe – using Whisper (install openai-whisper version 2023‑09‑01):

   whisper interview-$(date +%F).wav --model base.en --output_format md --output_dir transcripts

Why: Generates a searchable markdown file.

Confirm: Open transcripts/interview-<date>.md; the first line should be a timestamped transcript.

Highlight evidence – open the markdown in Obsidian, search for verbs like “always”, “never”, “every”. For each, add the > **EVIDENCE:** prefix and timestamp.

Why: Makes evidence discoverable for stakeholders.

Confirm: Use Obsidian’s search (> **EVIDENCE:**) to see all highlighted lines.
Create a summary table – at the end of the file, add:

   | Metric | Count |
   |--------|-------|
   | Reports per week (≥3) | 3 |
   | Reports per week (1‑2) | 2 |
   | Report generation pain points mentioned | 4 |

Why: Provides quantitative backing for decisions.

Confirm: Table renders correctly in preview mode.

You now have a reproducible artefact that contains raw evidence, not a paraphrased agreement. Share the markdown file (not the summary) with the PM and engineering leads; they can verify the numbers themselves.

When this breaks

Symptom (what you see)	Likely cause	Fix
“Audio is silent” – the recorded file is 0 KB.	Microphone not selected or muted.	Run `ffmpeg -list_devices true -f avfoundation -i ""` (macOS) or `ffmpeg -list_devices true -f dshow -i dummy` (Windows) to list devices, then replace `:0` with the correct device ID.
Transcription fails with “Invalid argument”.	Whisper expects a supported audio codec.	Convert to FLAC first (`ffmpeg -i interview.wav -ac 1 -ar 16000 interview.flac`) and feed the FLAC file to Whisper.
Evidence tags are missing from the markdown.	You edited the transcript but forgot to add the `> EVIDENCE:` prefix.	Use Obsidian’s multi‑cursor edit: search for timestamps (e.g., `[00:12:34]`) and prepend the prefix in bulk.
Stakeholder disputes the numbers – they claim “only one user said they run reports three times a week”.	You merged multiple interviews into a single file without clear separation.	Keep each interview in its own markdown file, then create a separate aggregation document that references each file with a link.
User says they felt “tested” or “coached” – they refuse to continue.	Leading questions were asked inadvertently.	Review the transcript for any question that contains the interviewee’s own words; replace with a neutral version and note the mistake in a post‑mortem.

What it costs

Cost dimension	Detail
Time – each interview (incl. scheduling, consent, recording, transcription) costs ~ 90 minutes of engineer/PM time. In a two‑week sprint, three interviews consume ~ 4.5 hours, which is non‑trivial for a small team.
Opportunity cost – while you are listening to users, you are not writing code. If the evidence later shows the feature is low‑priority, you have saved weeks of development; if it confirms the hypothesis, you have spent a sprint on discovery that could have been a sprint of implementation.
Tooling cost – `ffmpeg` and Whisper are free, but Whisper’s GPU inference may require a cloud instance (~ $0.10 per minute of audio on AWS). For a 30‑minute interview, that is $3.
Bias cost – leading questions can produce false positives, leading to wasted engineering effort. In our own data (three releases, 2023‑Q2), a feature built on a mis‑interpreted user need cost ~ $45 k in re‑work and delayed roadmap items.
Compliance cost – failing to obtain proper consent can expose the company to GDPR or CCPA penalties (up to €20 m or $7.5 m). The consent template mitigates this risk.

The trade‑off is clear: you spend time up‑front to collect high‑fidelity evidence, but you avoid the far larger cost of building the wrong thing. The technique scales poorly for high‑frequency, low‑impact decisions; for those, a rapid survey may be cheaper.

In the interview

Typical interview prompt

“Walk me through how you would validate whether a new feature – say, recurring report scheduling – is actually needed by our users.”

Weak answer

“I’d send a quick survey, look at the responses, and if most say ‘yes’ we build it.”

Why it’s weak: It glosses over the need for unbiased data, assumes survey responses are reliable, and ignores the importance of evidential artefacts.

Strong answer

“First I’d define the hypothesis I want to test – that users need recurring scheduling because they forget to run reports. I’d then schedule three 45‑minute contextual interviews with a mix of power users and occasional users. I’d use a consent‑driven recording pipeline (ffmpeg → Whisper) to capture raw audio, then tag any statements that contain quantitative or pain‑point evidence. I’d aggregate the evidence in a markdown file, produce a frequency table, and present the artefact to the team. If the evidence shows that fewer than 30 % of users run reports more than twice a week, I’d recommend postponing the feature. The trade‑off is the interview time versus the risk of building a low‑value feature, which in our last quarter cost us $40 k in re‑work.”

Senior vs junior expectations

Junior: Should be able to describe the interview flow, identify leading questions, and produce a basic transcript.

Senior: Must also articulate the sampling strategy, quantify the evidence, and discuss how the artefact integrates with backlog grooming.

Director/Engineering‑manager: Looks for the governance aspect – consent, GDPR compliance, and how the evidence feeds into portfolio‑level decision making.

Follow‑up that separates experience from theory

“Can you give an example of a time when the evidence you collected contradicted a stakeholder’s expectation, and how you handled the push‑back?”

A candidate who has actually run the workflow will describe the specific evidence tag, the stakeholder’s reaction, and the concrete decision that resulted (e.g., postponing the feature, re‑prioritising the roadmap).

Your tasks

Run a single interview with a colleague who uses the current reporting UI.

Done looks like: a signed consent file, a 5‑minute interview-*.wav, a markdown transcript with at least three > **EVIDENCE:** lines, and a one‑row frequency table.
Audit an existing interview (if you have one from a previous sprint). Identify any leading questions, add the missing > **EVIDENCE:** tags, and produce a short summary of the evidence.

Done looks like: a revised markdown file with all evidence highlighted and a note explaining the changes you made.
Create a sampling plan for a new feature (e.g., “bulk export”). List the user segments you will interview, the number of participants per segment, and the justification for each number (e.g., “3 power users, 2 occasional users – ensures at least 80 % confidence in frequency estimates”).

Done looks like: a one‑page document with a table of segments, counts, and confidence rationale.
Simulate a disagreement: share your evidence markdown with a teammate who believes the feature is essential. Ask them to critique the evidence. Document their objections and your response, focusing on the data, not opinion.

Done looks like: a short email thread or meeting notes that reference specific evidence timestamps and show how you defended the conclusion.
Scale the pipeline – automate the transcription step with a simple script that watches a recordings/ folder and runs Whisper on any new file, outputting markdown to evidence/.

Done looks like: a bash or Python script, a test run that produces a markdown file, and a log confirming the automation worked.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.