DOC-404: Document orphaned containers after Docker Compose stop - #894
DOC-404: Document orphaned containers after Docker Compose stop#894quetzalliwrites wants to merge 1 commit into
Conversation
Add an FAQ entry explaining that Docker Compose's default 10s SIGTERM-to-SIGKILL grace period can be too short for LocalStack to finish tearing down auxiliary containers (including k3d containers backing an EKS cluster), leaving them behind as orphans, and how to raise stop_grace_period to avoid it. The CLI is not affected since it waits for shutdown to finish. Cross-link the same behavior from the EKS docs where k3d-backed cluster creation is introduced.
Deploying localstack-docs with
|
| Latest commit: |
ef16f22
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8f8baac4.localstack-docs.pages.dev |
| Branch Preview URL: | https://doc-404-doc-faq-addition-of.localstack-docs.pages.dev |
|
@nik-localstack @pinzon heads up, looks like this PR is still pending eng review |
nik-localstack
left a comment
There was a problem hiding this comment.
Thanks for putting this together, the entry describes a real problem, and it came up in an internal discussion recently too.
Some things from that are worth folding in:
- There's a second setting involved, and it isn't in our docs. Alongside docker-compose stop_grace_period, LocalStack applies its own
SHUTDOWN_TIMEOUT(5 seconds by default) to its cleanup, and that one runs out first. Raising onlystop_grace_perioddoesn't give the cleanup more time and it has a side effect worth warning about: docker compose stop then takes the full three minutes before the container exits, whereas raising both settings gets a clean shutdown in well under a minute. The inline suggestions reflect that.SHUTDOWN_TIMEOUTisn't listed in the configuration reference today, so this entry would be the first place we mention it. We should probably expose it in the configuration page. - It isn't only EKS. The same shutdown budget is shared by every service that manages containers. ECS is the closest example, where a task's own stop timeout can consume the entire budget by itself.
I think we need more eyes before merging: shutdown behavior is being actively discussed internally right now, and how these settings interact may change. I don't think that blocks the changes but we should consider what should be advertised as a permanent solution and what not.
cc @purcell who was involved in this discussion about shutdown
btw I have added some inline suggestions but it may not be what we want to document in the end, so feel free to disregard them.
|
|
||
| If a previous run already left containers behind, remove them before starting LocalStack again. | ||
|
|
||
| This applies to Docker Compose specifically. Starting LocalStack with the CLI is not affected, as the CLI waits for LocalStack to finish shutting down. |
There was a problem hiding this comment.
question: What do we refer as CLI here ? the legacy localstack cli or lstk ?
|
|
||
| ### Why are some containers left behind after I stop LocalStack with Docker Compose? | ||
|
|
||
| When LocalStack shuts down, it cleans up the auxiliary containers it started, including k3d containers that back an EKS cluster. |
There was a problem hiding this comment.
| When LocalStack shuts down, it cleans up the auxiliary containers it started, including k3d containers that back an EKS cluster. | |
| When LocalStack shuts down, it cleans up the auxiliary containers it started for services such as Lambda, ECS, RDS or EKS. If it is stopped before that cleanup finishes, those containers are left running. |
I don't think we need to specifically mention the k3d here. This can go to the EKS doc.
| Docker Compose sends `SIGTERM` to the container and then, after a grace period of 10 seconds, follows up with `SIGKILL`. | ||
| That default window might be too short for LocalStack to finish its cleanup in some scenarios, so it is killed mid-teardown and the containers survive as orphans. |
There was a problem hiding this comment.
| Docker Compose sends `SIGTERM` to the container and then, after a grace period of 10 seconds, follows up with `SIGKILL`. | |
| That default window might be too short for LocalStack to finish its cleanup in some scenarios, so it is killed mid-teardown and the containers survive as orphans. | |
| By default, LocalStack gets only a few seconds. Docker Compose sends `SIGTERM` to the container and follows up with `SIGKILL` once `stop_grace_period` expires (10 seconds by default), and LocalStack stops waiting for its own cleanup after `SHUTDOWN_TIMEOUT` (5 seconds by default). Whichever window runs out first, LocalStack is stopped mid-cleanup and the containers survive as orphans. Cleanup that takes longer than a few seconds is the most likely to be cut short. |
| localstack: | ||
| image: localstack/localstack-pro:latest | ||
| stop_grace_period: 3m | ||
| # ... |
There was a problem hiding this comment.
| # ... | |
| environment: | |
| - SHUTDOWN_TIMEOUT=180 | |
| # ... |
| stop_grace_period: 3m | ||
| # ... | ||
| ``` | ||
|
|
There was a problem hiding this comment.
| Raise them together. If you raise only `stop_grace_period`, LocalStack still gives up on its cleanup after 5 seconds and the container then stays up until the grace period runs out, so every docker compose stop waits the full three minutes. | |
| # ... | ||
| ``` | ||
|
|
||
| If a previous run already left containers behind, remove them before starting LocalStack again. |
There was a problem hiding this comment.
| If a previous run already left containers behind, remove them before starting LocalStack again. | |
| If a previous run already left containers behind, run docker ps and remove the ones LocalStack started before starting it again. |
Summary
stop_grace_periodto avoid it.Linear ticket
https://linear.app/localstack/issue/DOC-404/doc-faq-addition-of-why-are-some-containers-left-behind-after-i-stop
Test plan
npx astro buildcompletes successfully (401 pages built)eks.mdxandfaq.mdx, are valid