How to analyze feedback themes with LLMs
Reading a thousand support tickets used to be an intern project or a vendor contract. An LLM does the reading part in minutes, and does it well. What it doesn't do is decide what corpus to read, what themes are allowed to exist, or whether its output is trustworthy — and those three decisions are where LLM feedback analysis succeeds or quietly produces plausible garbage.
Here's the pipeline, in the order the decisions actually get made.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in NotionStep 1: assemble the corpus before you prompt anything
An LLM analysis of your Zendesk tickets is an analysis of your Zendesk tickets — not of your customers. If half your feedback lives in Slack threads and Gong calls, a support-only analysis will systematically miss what your largest accounts say, because they complain to their CSM, not to the ticket form.
So the first step is boring: pull feedback from every channel into one dataset, with metadata attached to each item — source, date, account, plan tier. The metadata matters as much as the text; without it, step 5 (quantifying) is impossible. This is standard feedback aggregation, and it's the step most tutorials skip.
Step 2: choose deductive, inductive, or both
Thematic analysis comes in two modes, and LLMs support both (Zonka's overview covers the distinction):
- Deductive: you define the taxonomy ("billing, performance, onboarding, integrations, feature requests…") and the model classifies each item into it. Output is consistent and countable. Risk: you only find what you already named.
- Inductive: the model reads a sample and proposes themes from scratch. Output surfaces things you didn't know to look for. Risk: theme names drift between runs, so counts don't compare over time.
The practical pattern is both: run inductive quarterly to discover candidate themes, promote the useful ones into a stable deductive taxonomy, and run deductive classification continuously. Keep the taxonomy under ~25 themes with one-line definitions; vague taxonomies produce vague classifications.
Step 3: prompt for classification, evidence, and an escape hatch
The classification prompt needs three properties:
- Constrained output: the model must answer with theme IDs from your list (plus severity or sentiment if you want it), in a structured format you can parse.
- Quoted evidence: require the model to include the exact sentence that justifies each tag. This makes step 4 possible and hallucination visible.
- An "other" escape hatch: force-fitting every item into the taxonomy pollutes your counts. Let the model say "no theme fits," and review the other-bucket weekly — it's where new themes announce themselves.
Process items individually or in small batches rather than dumping the whole corpus into one context window; long-context summarization loses tail themes, and the tail is often the point. Chattermill's guide to LLM feedback analysis covers the input-quality caveats well; a worked blueprint for app-review analysis shows the same architecture end to end.
Step 4: validate before you trust a single chart
Take a random sample — 50 to 100 classified items — and have a human label them blind. Compare. You're looking for two failure types: systematic confusion (the model files "slow dashboard" under performance when you meant dashboards) and evidence mismatch (the quoted sentence doesn't support the tag). Fix confusions by tightening theme definitions in the prompt, and re-run the sample after any model upgrade — newer models are not automatically better at your specific taxonomy, a trap Thematic has documented.
Repeat a small validation monthly. It's an hour that keeps the dashboard honest.
Step 5: turn themes into numbers that mean something
A theme count alone ("142 items tagged performance") mostly measures channel volume. The useful numbers come from joining themes to the metadata from step 1: distinct accounts per theme, segment concentration, revenue represented, and trend direction over time. That arithmetic is its own discipline — covered in how to quantify feedback — but it's the reason the metadata had to travel with the text.
This whole pipeline — capture, dedupe, classify, count, with requesters preserved — is what we build Modem to do continuously rather than as a batch project, so read this section knowing we sell the automated version. What the pipeline accumulates is a context graph — themes linked to people, companies, and the original quotes — which an agent can query over MCP instead of re-running classification over a raw export each time a question comes up. The build-it-yourself route is entirely viable, especially if your feedback volume is modest or your taxonomy is unusual; where Modem specifically doesn't help is one-off research corpora like interview transcripts from a discovery study, where a notebook and a model API are the right tool.
Step 6: route the output somewhere that causes action
An LLM theme report that lands in a slide deck changes nothing. Wire the output to consequences: top themes reviewed in roadmap planning, spike detection alerting the owning team, individual classified requests becoming tracked issues with requesters attached so shipping them can close the loop. The test of the pipeline is not classification accuracy; it's whether anything different got built.
The smallest version this week
Export one month of support tickets, write a 10-theme taxonomy with one-line definitions, and classify the batch with any capable model, requiring quoted evidence per tag. Hand-check 30 results. If the accuracy convinces you, you've validated the approach for a few hours' work — and the taxonomy you wrote is reusable in whatever you build or buy next.
