Skip to content

Server responding with html error message causes an exponential number of retries #38

@moritz157

Description

@moritz157

I have my server running behind Apache as a reverse proxy. If the server is running everything is working fine. If I then stop the server the SSE connection dies and after 5 seconds it tries to reconnect. However if the server is still down Apache responds with a regular html error message. The SSEClient then, line by line, tries to parse the response to single events. Of course this fails for every single line. The problem is that if the parser cannot parse a line it calls _retryConnection while completely ignoring if a previous line already triggered a reconnect. Therefore a single response consisting of for example 10 lines triggers 10 connection retries simultaneously. If the server remains offline, each of those retries then trigger another 10 retries since Apache is still responding with an error. One minute of server downtime therefore results in 10^12 = 1 trillion requests at once (retry interval 5 seconds).

I accidentally kinda DOS'ed myself a number of times before I discovered the problem 😅
I think I am not the only one running my server behind a reverse proxy that responds like this if the server is down and I think this should be prevented.

The simplest solution to this would be to just cancel the response line stream subscription if an error occurs while parsing and therefore only allow one retry attempt. One giant downside of this approach though is that this only logs the first line of the response which isn't really helpful for debugging. So we should think about catching non-event-stream responses early and logging them in full before it could be split up line-by-line.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions