Skip to content

fix: back off and retry when Slack rate limits the channel listing - #73

Merged
sourya-deepsource merged 1 commit into
masterfrom
ENG-4785
Aug 17, 2026
Merged

fix: back off and retry when Slack rate limits the channel listing#73
sourya-deepsource merged 1 commit into
masterfrom
ENG-4785

Conversation

@sourya-deepsource

Copy link
Copy Markdown
Contributor

Problem

The Slack integration cannot be installed at all on a workspace large enough to need several pages of channels.

OAuth completes and we get a valid bot token. GetOptValues then calls conversations.list to populate the channel picker. That is a Slack Tier 2 method (~20 requests/minute), and the pagination loop fires pages back-to-back with no pacing, so Slack returns 429 {"ok":false,"error":"ratelimited"} partway through. The loop discarded every page it had already fetched and returned an error, which surfaces to the caller as a 422 and blocks the install with no indication of why.

This is a regression from #71. Before that, GetChannels made a single request returning up to 1000 channels: a larger workspace got a silently truncated list, but never a rate limit. #71 traded "truncated channel list" for "install completely blocked".

Observed in production, 11 consecutive failed attempts by one customer across three days, each aborting on a different cursor:

GET /providers/slack -> 422 (latency 4.95s)
slack: Error fetching page dGVhbTpDMDYyNlYwSkdRNA==: received 5xx, error={"ok":false,"error":"ratelimited"}
slack: failed to get channels : received 5xx, error={"ok":false,"error":"ratelimited"}
Failed to get options values: received 5xx, error={"ok":false,"error":"ratelimited"}

A cold first attempt of the day also failed, so there is no retry-later workaround for the customer.

Changes

Rate limit handling. A 429 is now distinguished from a server error and retried against the same cursor, honoring Retry-After. Both shapes Slack uses are detected: the 429 status, and ok: false with error: "ratelimited" on a 200.

Bounds, because this runs inside a synchronous OAuth callback and must not hold the request open indefinitely: 2 retries, each wait clamped to 15s, falling back to 5s when Retry-After is absent or unparseable.

Partial results beat a hard failure. Once retries are exhausted, the pages already fetched are returned instead of erroring, so the install can complete with a possibly incomplete picker. Only a rate limit with zero results stays a hard error. A follow-up worth considering is caching the list per workspace or moving the picker to a server-side typeahead, which would take the full enumeration off the install path entirely.

Slack application errors are no longer swallowed. ok: false on a 200 previously fell through and produced an empty channel list, so invalid_auth and missing_scope looked like "this workspace has no channels". They now return an error naming the Slack error code.

handleHTTPFailure fix. The retryable branch was nested under if response.StatusCode > 500 and a dead if err != nil that can never be true at that point, since a non-nil err has already returned above. Every failure was therefore classified permanent and logged as failed with 5xx response code: <nil>, including the 429s. That misleading log line is a large part of why this took a while to find. IsFatal() has no consumers in the codebase, so this is a logging and semantics correction with no behavior change.

A page cap of 200 as a runaway guard, and GetChannels now returns an empty slice rather than nil for a workspace with no channels, so callers iterating the options do not trip over a null.

Testing

17 tests replacing the empty scaffold, covering pagination, retry-on-429 with cursor reuse, the ok: false rate limit variant, partial results on exhaustion, hard failure with no results, application errors not being retried, Retry-After parsing and clamping, and the handleHTTPFailure classification. Full suite, go vet and gofmt all clean.

Note that shipping this needs a new tag. master, v1.2.0 and v1.3.0 all currently point at the same commit (cbe9190), and v1.2.0 is what is deployed.

Ref ENG-4785

🤖 Generated with Claude Code

conversations.list is a Tier 2 method, so paginating a large workspace trips
Slack's rate limit partway through. The listing treated the 429 as a fatal 5xx
and abandoned every page it had already fetched, which made the Slack
integration impossible to install on those workspaces.

Rate limited pages are now retried against the same cursor, honouring
Retry-After within a cap. Once retries are exhausted the pages already fetched
are returned rather than failing, since a partial channel list still lets the
install complete. Slack's `ok: false` application errors on a 200 are also
detected now, where previously they surfaced as an empty channel list.

Also fixes handleHTTPFailure, where a dead `err != nil` branch under
`StatusCode > 500` classified every failure as permanent and logged it as a
5xx, the 429s included.

Ref ENG-4785
@sourya-deepsource
sourya-deepsource merged commit 81dd38e into master Aug 17, 2026
1 check passed
@sourya-deepsource
sourya-deepsource deleted the ENG-4785 branch August 17, 2026 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants