How Do You Stop Devin From Inventing a Fix When the Ticket Is Vague?
You stop it before the ticket reaches Devin, not after. Devin's own documentation is explicit that subjective, underspecified instructions produce bad outcomes, and its good-vs-bad-instructions guide calls out exactly the kind of ticket a support team writes without thinking twice. Something like "find issues with our codebase and fix them" is flagged as too open-ended, with no success criteria for Devin to check its own work against. Nothing in Devin's public docs describes a step where it stops and asks a human to fill the gap. The gap gets filled with an assumption instead, and the assumption is usually plausible enough that a reviewer has to actually read the diff to notice it's wrong.
That's the mechanism to keep in mind, because it isn't Devin malfunctioning. A vague ticket is a set of constraints with holes in it, and an agent that has to produce something will produce something that satisfies the constraints it has and quietly picks a value for the ones it doesn't. The fix is to close the holes before the ticket ships to Devin. A short, enforced list of what a ticket needs before it's allowed to become a Devin session covers most of it: a repro, the affected version or environment, and a sentence of expected behavior. Below that, the honest limit of doing this by hand, and where a gate like this stops scaling on its own.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in Notion
DevinWhat "inventing a fix" actually looks like
It rarely looks like Devin refusing or erroring out. Independent testing backs this up directly. When Answer.AI ran Devin against 20 real engineering tasks, the tally was 3 successes, 14 failures, and 3 inconclusive results, and the failures mostly weren't loud. Asked to build matching DaisyUI themes, Devin shipped a change that looked complete, until the team tried actually switching themes and found "the theme was doing nothing," because the app was still rendering off the default theme underneath. On a separate task, deploying multiple applications to a single Railway instance, a configuration the report says Railway likely can't support ("it's not actually possible to do this, if I understand correctly"), Devin didn't flag the limitation. It spent over a day trying different approaches and, in the report's words, "hallucinating features that didn't exist," rather than surfacing that the task itself was impossible as specified.
Both failures share a shape. Devin produced something that read as a real fix, in a context where the actual problem was underspecified or fundamentally different from what the instruction implied. A vague customer ticket sets up the same trap on a smaller scale. The ticket names a symptom, Devin proposes a plan that resolves the symptom as literally described, and the plan ships as a PR that looks reviewable because it compiles and touches the right file.
The refund bug that wasn't a rounding problem
Talbrook builds membership billing software for boutique fitness studios, the kind of product that has to get proration and cancellation timing exactly right or a studio owner hears about it from an angry member within the hour. Imani Okoye leads the platform team, and Talbrook runs Devin against tickets filed straight from their support queue into GitHub.
A ticket came in from a support rep after a call:
Support (Priyanka's shift, filed from a call): Member says she was charged twice after canceling mid-cycle. Wants a refund for the extra charge.
Devin scoped it and opened with a plan:
Devin (implementation plan): Reviewed the proration calculation in
billing/cancel.ts. The refund amount for mid-cycle cancellations appears to round in the studio's favor in some cases. Proposed fix: adjust the proration formula to round in the member's favor and add a regression test.
The PR looked complete. It changed a rounding direction, added a test that passed, and the ticket had a plausible one-line explanation for "charged twice" that Devin's plan matched exactly. Nobody had told Devin what "charged twice" meant technically, because the support rep didn't know either, and the ticket had no repro steps and no mention of which billing path the member's studio used. The actual bug was a Stripe cancellation webhook that Talbrook's billing job processed twice under a network retry, with no idempotency key to catch the duplicate, so the member was billed once by the webhook handler and once by the nightly reconciliation job. Rounding had nothing to do with it. The rounding fix shipped, the regression test passed because it tested the wrong mechanism entirely, and the same member was double-charged again on her next cycle.
The ticket wasn't sloppy by the standards support tickets usually get held to. It just didn't contain a repro, an affected version or billing path, or a sentence distinguishing "charged the wrong amount" from "charged twice via two separate transactions," and Devin's plan filled that gap with the nearest plausible mechanism it could find in the code it was pointed at.
Why a vague ticket gets a confident answer instead of a question
Devin's Jira integration docs describe scoping mode as producing "a summary, implementation plan, and confidence estimate" before any code gets written. That estimate is a read on how legible the ticket is, not an independent check on whether the described fix addresses the real problem, and the guidance around it runs one direction only: write more complete instructions up front, not wait for Devin to ask a clarifying question back. A ticket that reads clean, like Talbrook's, can score high confidence while resting on a wrong diagnosis, because nothing in the ticket text contradicted the wrong diagnosis. Confidence measures the document. It has no way to measure the gap between the document and the system underneath it.
The gate: three fields, enforced before a session starts
The fix that actually holds up doesn't ask support to write better prose. It requires three fields before a ticket is allowed to become a Devin task, and blocks the handoff if any are missing:
- Repro steps: the exact sequence that reproduces the symptom, not a paraphrase of what the customer said happened.
- Affected version or path: which environment, plan, or code path the customer is on, when more than one exists.
- Expected behavior: one sentence stating what should happen instead, written by someone who checked the product, not by inference from the complaint.
Talbrook's fix after the Stripe incident was procedural, not clever. A required-fields checklist went into the GitHub issue template, with a rule that a ticket missing any of the three fields gets one round of follow-up with the customer, or gets pulled into a short debugging session, before it's routed to Devin at all. It's the same principle behind building coding agents an executable backlog: an agent fills gaps with assumptions, so the assumptions have to already be written down.
Three cracks in the checklist as volume grows
A checklist enforced by one team on one queue holds up fine at low volume. It stops holding up in three specific ways as a team scales:
- The same bug arrives as five different tickets. Nobody filling out the required fields on ticket three knows tickets one and two already exist, so five people each write a complete, single-ticket version of the same incomplete picture, and nobody notices the pattern that would tell Devin this bug is worse than one report suggests.
- The fields get skipped under pressure. A required-fields template is a convention, and conventions erode exactly when there's a backlog and someone routes a ticket straight to Devin to clear it faster.
- The context that would fill the gaps lives outside the ticket. The Slack thread where an engineer already found the webhook race, or the earlier ticket where a different customer described the same double-charge with more detail, doesn't automatically attach itself to ticket five just because someone wrote a template.
Modem is built for that specific gap. Modem is also the company writing this guide, so read the specifics below and judge the fit against your own backlog rather than the recommendation itself. Modem's Devin integration writes the task brief from the Modem agent, which means the brief can carry the customer's original wording and the repro details customers actually gave. Underneath that, Modem groups repeated mentions of the same symptom, across Slack, support, and prior tickets, into a single topic instead of treating each new report as its own isolated issue, so a brief pulled from that topic already carries whatever history exists. If the second double-charge, the same member's repeat on her next cycle, had come through Modem instead of landing as a fresh ticket, the brief for that second report would have carried forward that a "charged twice" complaint already existed with a fix that had shipped and hadn't held, which is exactly the signal a single ticket, however well templated, can't hold on its own. Below the point where the same bug starts arriving more than once, the three-field template above covers the failure mode that matters most, and there isn't much to add to it.
For the wider set of ways teams get bugs in front of Devin, the six ways to hand customer-reported bugs to Devin compares the integrations, the API, and the manual route side by side. And if you want the full picture of what happens after the ticket, from the first report to a merged fix a customer gets told about, from user report to merged fix walks the whole chain stage by stage.
Fix the ticket in front of you first
Before routing anything else through Devin, add the three required fields to whatever template creates your GitHub or Jira tickets, and make at least one of them, repro steps, a hard block rather than a suggestion. That alone catches the Talbrook-shaped failure, a ticket that reads clean, scores a confident plan, and fixes the wrong mechanism. The pattern-across-tickets problem is real, but it's the second problem. Fix the one ticket in front of you first.
