Why Does the Intercom Companies Scroll API Fail on Large Data Exports?
Intercom's Companies Scroll API returns intermittent errors on exports past roughly 100,000 records because the scroll session backing the export has its own expiration clock, and a slow page, a retry, or a network stall can let that clock run out mid-job. When it does, the next request comes back rejected instead of returning the next page, and the workspace has to restart the whole scroll from the beginning. At a few hundred companies this never surfaces, because the whole export finishes in a handful of requests, well inside the session's window. At six figures of companies, the export takes long enough that timing becomes the thing that decides whether it finishes at all.
This isn't hypothetical. A thread on Intercom's own community forum describes one team's experience pulling over 100,000 company records: intermittent failures on the scroll endpoint, retry logic with delays as long as five minutes between attempts, and the same errors resurfacing anyway. Intercom's community team acknowledged the report and routed it to support rather than posting a public fix, and the thread doesn't show a follow-up resolution. It's one account, not a chorus of them, but the mechanism it describes matches how the scroll endpoint is documented to work.
Why the export needs a scroll in the first place
Intercom's regular company listing endpoint isn't built for a full-workspace pull. Its own API reference says that "there is a limit of 10,000 Companies that can be returned" when paging through the endpoint, after which it stops returning further pages no matter what page parameter you send. For anything beyond that ceiling, the same page is direct about the fix: "If you need to list or iterate on more than 10,000 Companies, please use the Scroll API."
The scroll endpoint trades page numbers for a session. The first request opens the scroll and returns a scroll_param value along with the first batch of companies. Every following request passes that same scroll_param back to fetch the next batch, in order, until the response comes back empty. It's a reasonable design for iterating through a dataset that changes while you're reading it. The trade-off is that the session isn't indefinite. It's tied to a window of continued activity, and a workspace with 100,000+ companies is exactly the case where a single page can be slow enough, or a retry delayed enough, to fall outside that window.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in NotionWhat actually breaks, and why retries don't fix it
The forum thread's error is scroll_exists, Intercom's response when a request arrives for a scroll session it no longer recognizes as the active one. The confusing part, and the reason the reporting team's retry logic didn't help, is that the failure doesn't behave like a rate limit. Intercom's rate-limiting documentation puts the default ceiling at 10,000 API calls per minute per app, with a 25,000-per-minute cap per workspace, well above what a sequential scroll job of any reasonable size would hit. A 429 from that limit is something a backoff loop is built to handle. A scroll_exists error on a session the client believes is still valid isn't a capacity problem, it's a timing mismatch between how long the workspace's export takes to walk each page and how long the session stays valid without one. Waiting longer between retries, which is the fix teams reach for first, doesn't help, because the fix an expired session needs is starting a brand-new scroll, not waiting and asking the old one again.
Three things make the mismatch worse as a workspace grows. Any slow step inside the loop, a database write, a transform, a lookup, counts against the session the same as network latency does, because the session clock doesn't distinguish between time spent waiting on Intercom and time spent doing something else with the page you just got. Retry logic built for transient failures makes it worse rather than better: backing off and re-sending the same scroll_param treats the symptom as a rate limit, when what's actually happened is that the session is already gone and no amount of waiting brings it back. And without a checkpoint recording where the export stopped, a mid-job failure at page 400 of 1,000 means restarting from page one, which makes the next attempt take even longer and hands the timing problem another chance to repeat.
The version of this that's easy to miss
A common way the failure creeps in has nothing to do with the scroll endpoint itself. A nightly job pulls company records into a warehouse to feed some downstream lookup, and it runs clean for months. Then someone adds one more step per page, an enrichment call, a database write, a tier lookup, before the next scroll_param request goes out. Nothing about that change looks risky, and it usually isn't caught in testing, because a workspace with a few thousand companies finishes the whole export in well under a minute no matter what extra work happens per page. It only turns into a problem once the company count is high enough that the export runs long, at which point the added per-page time is exactly the kind of thing that occasionally pushes a request past the session's window. By then the job has usually been running fine for so long that the recent change is the last thing anyone suspects.
The fix isn't a longer retry delay, since the session that expired isn't coming back regardless of how long you wait for it. It's separating the two jobs that got bundled into one loop: pull raw company data as fast as the scroll allows and write it straight to a staging table, then run any lookup or transform as its own step afterward, once the export itself has already finished. That keeps the only thing happening inside the scroll's window the one thing that has to happen there.
Where a nightly scroll job stops being the right design
Checkpointing and a leaner per-page transform buy real headroom, but they're a workaround for a structural mismatch, not a fix for it. A scroll job is trying to move an entire workspace's state in one sitting, and the workspace keeps growing. The same job that takes twenty minutes at 100,000 companies takes longer at 200,000, and every added minute is more surface for a session to expire in the middle. Nothing about chunking the pull changes that; it just delays the day it stops fitting inside the window.
The read is that a periodic full pull is the wrong shape for the problem, not just a fragile implementation of the right one. Intercom already tells you about company changes as they happen, through webhooks, rather than only when you ask for a full scroll. We build Modem around that difference: it reads Intercom company and conversation activity incrementally, as events arrive, and keeps company records current without a job that has to complete a six-figure walk of the whole workspace to stay accurate. There's no daily window to fall outside of, because there's no single request carrying the whole export. We build Modem, and obviously that gives us a reason to like how this comparison lands, so run the arithmetic against your own workspace size rather than taking it on faith: incremental sync grows a few events at a time, while a scroll job that regenerates the entire company table nightly grows with the size of the table itself. For the mechanics of wiring Intercom into a tracker the same incremental way, see how to connect Intercom conversations to Linear issues. If large exports are failing you elsewhere in Intercom too, the CSV side of this has its own scale ceiling, covered in why Intercom's CSV import drops rows on large contact lists.
What to log before this bites you
If your export is still small enough that this hasn't happened to you yet, add the checkpoint now rather than after the first mid-run failure: log the scroll_param and the running count of companies processed after every batch. It costs nothing to add and nothing to keep running, and it's the difference between a failure that loses a few minutes of work and one that sends the whole job back to page one.
