Skip to content

[flagd] Should shutdown stop the connectivity watcher? In-process leaves "Cannot invoke RPC: Channel closed!" #419

Description

@aepfli

Asking rather than reporting, because nothing required is unmet here — the shutdown completes, and well within the bound. What it leaves behind is a traceback in the host application's log, and I would like to know whether that is known and intended.

What I see

In-process resolver. After provider.shutdown():

ValueError: Cannot invoke RPC: Channel closed!

raised from gRPC's connectivity poller, after the shutdown call has already returned successfully.

The reading that fits: shutdown() closes the gRPC channel, but the watcher task observing channel connectivity is still running and calls into the closed channel on its next poll. Nothing retries and nothing fails — the provider is down by then and the application is not asking it for anything — so the only symptom is the traceback.

Why it may be worth fixing anyway

An application that shuts down cleanly still gets a stack trace on the way out. That is noise at the exact moment operators are reading logs to find out whether shutdown was clean, and it trains them to ignore tracebacks from this path — which is where a real error would also appear.

It is also the kind of thing that becomes a test failure elsewhere: a suite configured with -W error, or one asserting nothing is logged at ERROR, would fail on a provider that is behaving correctly.

Questions

  1. Is the watcher meant to be stopped before the channel closes, or is the traceback known and considered harmless?
  2. If it should be stopped — is that shutdown()'s job, or should the watcher tolerate a closed channel and exit quietly?
  3. Is there a reason the ordering is hard here that I am not seeing? I have not attempted a fix, because the answer to (2) decides what a fix would look like.

How I found it

Building the cross-language provider conformance suite proposed in
open-feature/spec#417. The relevant detail is that
the scenario passes — it asserts that shutdown completes within a bound, and it does. The
traceback is beside the assertion, not in it.

That is also why it went unnoticed for some time: the Python adoption had not declared the
@lifecycle capability, so the six lifecycle scenarios had never run against flagd at all. Declaring
it was a one-line change and this surfaced on the first run. The other three languages' adoptions
have been running those scenarios throughout and do not show anything equivalent, so this looks
specific to the Python in-process resolver rather than to the shape of the test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Needs TriageThis issue needs to be investigated by a maintainerquestionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions