Skip to content

DOC-404: Document orphaned containers after Docker Compose stop - #894

Open
quetzalliwrites wants to merge 1 commit into
mainfrom
doc-404-doc-faq-addition-of-why-are-some-containers-left-behind
Open

DOC-404: Document orphaned containers after Docker Compose stop#894
quetzalliwrites wants to merge 1 commit into
mainfrom
doc-404-doc-faq-addition-of-why-are-some-containers-left-behind

Conversation

@quetzalliwrites

Copy link
Copy Markdown
Collaborator

Summary

  • Add an FAQ entry explaining why some containers (including k3d containers backing an EKS cluster) can be left behind after stopping LocalStack with Docker Compose, and how to raise stop_grace_period to avoid it.
  • Add a cross-referencing note in the EKS docs, where k3d-backed cluster creation is introduced, pointing to the new FAQ entry.

Linear ticket

https://linear.app/localstack/issue/DOC-404/doc-faq-addition-of-why-are-some-containers-left-behind-after-i-stop

Test plan

  • npx astro build completes successfully (401 pages built)
  • Link validator confirms all internal links, including the new cross-link anchor between eks.mdx and faq.mdx, are valid

Add an FAQ entry explaining that Docker Compose's default 10s
SIGTERM-to-SIGKILL grace period can be too short for LocalStack to
finish tearing down auxiliary containers (including k3d containers
backing an EKS cluster), leaving them behind as orphans, and how to
raise stop_grace_period to avoid it. The CLI is not affected since it
waits for shutdown to finish.

Cross-link the same behavior from the EKS docs where k3d-backed
cluster creation is introduced.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying localstack-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: ef16f22
Status: ✅  Deploy successful!
Preview URL: https://8f8baac4.localstack-docs.pages.dev
Branch Preview URL: https://doc-404-doc-faq-addition-of.localstack-docs.pages.dev

View logs

@quetzalliwrites

Copy link
Copy Markdown
Collaborator Author

@nik-localstack @pinzon heads up, looks like this PR is still pending eng review

@nik-localstack nik-localstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together, the entry describes a real problem, and it came up in an internal discussion recently too.

Some things from that are worth folding in:

  • There's a second setting involved, and it isn't in our docs. Alongside docker-compose stop_grace_period, LocalStack applies its own SHUTDOWN_TIMEOUT (5 seconds by default) to its cleanup, and that one runs out first. Raising only stop_grace_period doesn't give the cleanup more time and it has a side effect worth warning about: docker compose stop then takes the full three minutes before the container exits, whereas raising both settings gets a clean shutdown in well under a minute. The inline suggestions reflect that.SHUTDOWN_TIMEOUT isn't listed in the configuration reference today, so this entry would be the first place we mention it. We should probably expose it in the configuration page.
  • It isn't only EKS. The same shutdown budget is shared by every service that manages containers. ECS is the closest example, where a task's own stop timeout can consume the entire budget by itself.

I think we need more eyes before merging: shutdown behavior is being actively discussed internally right now, and how these settings interact may change. I don't think that blocks the changes but we should consider what should be advertised as a permanent solution and what not.

cc @purcell who was involved in this discussion about shutdown

btw I have added some inline suggestions but it may not be what we want to document in the end, so feel free to disregard them.


If a previous run already left containers behind, remove them before starting LocalStack again.

This applies to Docker Compose specifically. Starting LocalStack with the CLI is not affected, as the CLI waits for LocalStack to finish shutting down.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: What do we refer as CLI here ? the legacy localstack cli or lstk ?


### Why are some containers left behind after I stop LocalStack with Docker Compose?

When LocalStack shuts down, it cleans up the auxiliary containers it started, including k3d containers that back an EKS cluster.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When LocalStack shuts down, it cleans up the auxiliary containers it started, including k3d containers that back an EKS cluster.
When LocalStack shuts down, it cleans up the auxiliary containers it started for services such as Lambda, ECS, RDS or EKS. If it is stopped before that cleanup finishes, those containers are left running.

I don't think we need to specifically mention the k3d here. This can go to the EKS doc.

Comment on lines +161 to +162
Docker Compose sends `SIGTERM` to the container and then, after a grace period of 10 seconds, follows up with `SIGKILL`.
That default window might be too short for LocalStack to finish its cleanup in some scenarios, so it is killed mid-teardown and the containers survive as orphans.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Docker Compose sends `SIGTERM` to the container and then, after a grace period of 10 seconds, follows up with `SIGKILL`.
That default window might be too short for LocalStack to finish its cleanup in some scenarios, so it is killed mid-teardown and the containers survive as orphans.
By default, LocalStack gets only a few seconds. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short.

localstack:
image: localstack/localstack-pro:latest
stop_grace_period: 3m
# ...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# ...
environment:
- SHUTDOWN_TIMEOUT=180
# ...

stop_grace_period: 3m
# ...
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Raise them together. If you raise only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every docker compose stop waits the full three minutes.

# ...
```

If a previous run already left containers behind, remove them before starting LocalStack again.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If a previous run already left containers behind, remove them before starting LocalStack again.
If a previous run already left containers behind, run docker ps and remove the ones LocalStack started before starting it again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants