How to analyze open-text PostHog survey responses at scale
PostHog's survey results page will chart your NPS or CSAT scores and let you filter responses by what someone answered, but it has no view that groups two hundred open-text "why" answers into themes. You get a scrollable list of individual rows. Getting from that list to "here are the six things people are actually saying" is a manual job PostHog leaves to you, and two of its own open feature requests confirm the gap is still there: better results analysis with metric templates for NPS and CSAT, and promoter/passive/detractor filtering on rating questions, both still open.
The workaround most teams land on is exporting the response table and doing the grouping somewhere else: a spreadsheet, a notebook, or a prompt pasted into an LLM. That gets you from raw text to themes. It doesn't survive past a few dozen responses a month without becoming its own maintenance job, and it's worth knowing exactly where that happens before you build a process around it.
What PostHog's survey results actually show you
Set up a rating question (PostHog surveys support NPS, PMF, churn-reason, and plain open-text formats) and the results page gives you: how many people saw the survey, how many dismissed it, response and skip counts, and a chart per question. For rating questions you can click a bar to filter to just that score, and open that filtered view as a saved insight. You can filter by date range, by property, by cohort, or by HogQL expression, and there's a "filter by response" option for exact-match filtering.
None of that gets you a computed NPS number, a promoter/passive/detractor split, or a theme list for a free-text field. What you get for open text is the filtered list of raw answers. Reading fifteen of those is fine. Reading two hundred is a different task, and PostHog's own UI doesn't offer to do it for you on that page.
There's a second, newer path worth ruling out precisely because it sounds like it should cover this. PostHog's Self-driving feature, currently in open beta, is a general product-monitoring agent aimed at catching regressions and abandonment spikes and opening pull requests against them, not a customer-feedback tool. Its documented signal sources run well past error tracking and session replay: logs, product analytics, and health checks inside PostHog, plus more than two dozen external integrations spanning issue trackers (GitHub, Jira, Linear, GitLab), support tools (Zendesk, Freshdesk, Front), error trackers (Sentry, Rollbar, Bugsnag), and even a few third-party feedback and NPS tools like Canny and Retently. Survey responses aren't on that list anywhere, and neither page mentions clustering open-text answers into themes. If you were hoping Self-driving already closes the gap this guide is about, it doesn't: it doesn't currently ingest PostHog's native Surveys data at all, and separately, it doesn't read Slack or sales calls either.
The manual path: export, then group
Until you're past the beta or the volume outgrows either approach, the practical loop is:
- Export the response set. Filter to the question and date range you want, then pull the rows. PostHog exposes each answer as an event property (
$survey_response_{id}per question), so a CSV export or a HogQL query againsteventsgets you the raw text plus whatever person properties you want alongside it. - Paste into a grouping pass. A spreadsheet with manual tags works under a hundred rows. Past that, most teams paste batches into an LLM with a prompt like "group these into themes and count each" and treat the output as a draft, not a final answer. A human still checks that "checkout confusion" and "billing confusion" didn't get merged into one bucket.
- Re-attach the score. Because the theme-grouping step usually happens outside PostHog, you have to manually rejoin each theme back to the NPS or CSAT score it came from, and to who said it, if you want to say "detractors, specifically, keep mentioning onboarding."
That third step is where the process usually breaks first, because the rejoin is manual and nobody wants to do it twice.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in NotionA worked example
Dana Kessler runs product at Ledgerly, a small accounting SaaS with about 400 monthly active customers. Ledgerly ships a quarterly NPS survey through PostHog with a required "why" field, and by the third quarter of running it, she had 240 open-text responses sitting in the results table.
She exported the responses tied to scores of 6 or below (detractors), about 90 rows, and pasted them in batches of 30 into an LLM with a grouping prompt. The output surfaced five recurring complaints, the largest being "reconciliation takes too long" at 22 mentions. She spot-checked a sample of that bucket against the raw text and found two rows that had been miscategorized (one was actually about invoice formatting) and fixed the grouping by hand.
The process didn't tell her that four of those 22 reconciliation complaints came from accounts that had also filed the same complaint in a support ticket that quarter, or that one had come up twice on a sales call with a renewal at risk. That context existed. It just lived in three different systems, and connecting it back to the survey theme meant Dana remembering to go look, which she didn't, until a renewal call surfaced it after the fact.
Where the manual path stops working
The export-and-group loop holds up for a quarterly survey with a few hundred responses reviewed by one person. It stops working at a predictable point:
- The rejoin never happens automatically. Every survey cycle, someone has to remember to cross-check the new themes against tickets, Slack, and calls, and at volume that step gets skipped.
- Grouping consistency degrades across cycles. The taxonomy an LLM produces in Q1 isn't guaranteed to match the one it produces in Q3, so quarter-over-quarter trend lines on themes require someone actively holding the taxonomy steady.
- Detractor-specific analysis is entirely manual, because PostHog doesn't compute the NPS breakdown by response for you (see issue 28813), so "what are our detractors saying, specifically" means filtering by score by hand every time.
That's the point where teams either build (and staff) an internal script to do the export-group-rejoin loop, or bring in a tool that already treats the survey text as one input among several.
We build Modem for that second case, so weigh this section accordingly. Modem's PostHog integration already pairs product analytics, feature flags, and error data with what customers are saying, and PostHog's own survey notification feature can push each response, including the open-text answer, to a Slack channel or a webhook the moment it's submitted. Point that at a channel Modem is already reading, and every survey response lands in the same context graph as the Slack messages, support tickets, and call transcripts about the same account, deduped and counted into a theme automatically instead of exported and re-grouped by hand each quarter.
Pricing is unlimited users on every plan, with usage-based volume beyond what's included, not a per-response or per-seat charge for reading survey text.
That doesn't replace PostHog for anything analytics does well: feature flags, session replay, and product usage stay there. It closes the specific gap this guide is about, a "why" field with two hundred answers in it and no native way to read them as one thing instead of two hundred.
The rejoin problem above isn't unique to survey text; it's the same tax described in the real cost of manual feedback triage. And if PostHog is also where you're pairing usage data against what customers say more broadly, best tools to pair product analytics with customer feedback covers that comparison directly.
The smallest version you can start this week
Export your last quarter's detractor responses (score 6 or below), group them by hand or with one careful LLM pass, and check the resulting counts against any support tickets or Slack threads you remember from the same accounts. If that cross-check turns up matches you'd have otherwise missed, that's the signal the manual loop isn't going to hold much longer.
