How Accurate Does AI Ticket Triage Need To Be Before Engineers Trust It?
Canny's own case study says Autopilot "identified 93% of the feature requests correctly" in a batch of Typeform's support tickets. Read that sentence closely and the 93% is a detection number. It measures whether the tool noticed that a ticket contained a feature request at all, checked against what a human reviewer confirmed afterward. It says nothing about whether the specific category, priority, or team Autopilot assigned to that request was the right one, and it says nothing about a different customer's ticket queue. The same post reports a separate 98.3% figure for something else entirely, how often the team accepted Autopilot's suggested duplicate merges.
There is no single answer to "how good is good enough," because the bar isn't set by the number, it's set by what happens automatically once the label is applied. A triage call that only feeds a dashboard can tolerate real error. A triage call that closes a ticket, merges a report, or skips a human review needs to be right close to every time, because the cost of one miss lands on a customer who thinks they were heard and weren't.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in NotionWhat one accuracy number is actually counting
Vendors publish the number that makes the best headline, and it usually comes from one measurement on one job, not an audited average across everything the tool does. Canny's post is a good example because it discloses three different numbers from the same experiment, and they measure three different things:
- 93%, detection: did Autopilot flag a ticket as containing a feature request that a human agreed was one.
- 63%, the comparison point: how often Typeform's live support team caught the same requests without the tool, which is what makes "30% better" a fair relative claim.
- 98.3%, dedupe acceptance: how often the team accepted Autopilot's suggestion to merge two requests together, a narrower, easier judgment than deciding what a ticket is about in the first place.
None of those numbers tells you how accurate the tool's category or priority assignment is on your ticket queue, because that wasn't what got measured. The move when a vendor cites one flattering percentage is to ask which of the three jobs it covers, then assume the others are unmeasured until you check for yourself.
The bar moves with the action, not the topic
Sort triage actions by what happens if the label is wrong, and the accuracy bar falls out of that, not out of the topic being triaged:
- Suggestion only. The tool proposes a category or priority and a person still reads the ticket before doing anything. Wrong suggestions cost a few seconds of correction. 80 to 90% is genuinely fine here, because the human is still the last check.
- Routing without closing. The tool assigns an owner or a queue automatically, but the ticket stays open and visible until someone works it. A misroute costs a delay, not a loss, since the ticket doesn't disappear, it just sits in the wrong inbox until someone notices the queue looks light.
- Auto-close or auto-merge. The tool closes the ticket, merges it into another, or silences future alerts on it. This is the one that needs near-certainty, because a wrong call here doesn't produce a correction later, it produces a customer whose report just stopped existing anywhere a human would look.
Most teams end up mixing all three without deciding to: suggestions for new-ticket triage, automatic routing to the right engineer, and auto-close for anything the model scores above a confidence threshold. The mistake is applying one vendor's aggregate accuracy number to all three, when only the middle band ever tolerates being wrong.
Where spot-checking the aggregate stops working
At low volume, the fix is easy. Read every ticket the bot touches, which is what most teams are already doing without calling it verification. That works fine at ten issues a week. It stops working somewhere past thirty or forty, because reading every ticket in full defeats the point of triage in the first place, and the team is back to the manual bottleneck the bot was supposed to remove.
The usual next step is a monthly spot-check. Pull a random sample of the bot's labels, read the underlying tickets, and calculate an informal accuracy rate. That catches systemic drift, a model getting worse over time, or a new class of ticket the bot handles badly. It does not catch the one mislabeled P0 sitting in this week's queue, because a sample by definition skips most of the tickets, and the ticket you actually need to check is whichever one is in front of an engineer right now, not whichever ones landed in last month's sample.
Two different questions hide in there: "the tool is 93% accurate on average" and "is this specific ticket one of the 7%." An aggregate number can't answer the second question. Only the source conversation can, and the friction of finding it, digging back into GitHub, cross-referencing a Slack thread, checking whether the same bug got reported somewhere else, is exactly what makes engineers stop checking and start trusting the label instead.
Where Modem fits
Modem keeps the source conversation attached to every triaged topic instead of collapsing it into a label. Modem's GitHub integration captures issues alongside Slack, support tickets, and other channels, groups differently worded reports of the same problem into one topic, and keeps the original quotes attached to it, so an engineer looking at a severity call can open the exact ticket text behind it in the same click instead of taking the label on faith or going back to GitHub to reconstruct it. That doesn't make the triage model itself more accurate. It makes the cost of checking any single call low enough that engineers actually do it on the ticket in front of them, rather than trusting a vendor's aggregate stat because verifying is too much friction. Disclosure: this is our product, and the DIY route above costs nothing. The alternatives are compared in our guide to AI triage tools for engineering teams lines up by what each one actually triages. If your question is really about a coding agent's fix quality rather than its classification accuracy, what Sentry's Seer accuracy number actually measures covers that related but separate gap.
Ask which action the accuracy number was measured against
Don't ask a vendor for one accuracy number. Ask which action their number was measured against, suggestion, routing, or auto-close, and only let the model touch actions at or below the accuracy it actually demonstrated for that job. Then pick the one category of ticket where a wrong label costs the most, silent failures, cancellation language, anything security-adjacent, and require a human to open the source conversation on that category specifically, regardless of the bot's confidence score. That's a narrower rule than reviewing everything, and it's the one that catches a mislabeled P0 before a customer has to find it for you.
