Add rawsetenv message type for provider plugins by rajyan · Pull Request #13742 · docker/compose

rajyan · 2026-04-16T08:20:26Z

What I did

Added a new rawsetenv message type to the provider plugin protocol. This allows providers to inject environment variables into dependent services without the automatic service name prefix.

Currently, setenv always prefixes variables with the service name (e.g., URL becomes DATABASE_URL). This works well for connection strings, but some applications require exact variable names that cannot be altered. There is no way to inject these as-is today.

With this change, providers can choose per-variable whether to use the prefixed (setenv) or unprefixed (rawsetenv) behavior:

{"type": "setenv", "message": "URL=https://example.com"}
{"type": "rawsetenv", "message": "SECRET_KEY=xxx"}

Changes

pkg/compose/plugins.go — Add RawSetEnvType constant and pluginVariables struct to separate prefixed/raw variables
docs/extension.md — Document rawsetenv in the protocol specification
docs/examples/provider.go — Add rawsetenv usage to the example provider
pkg/e2e/providers_test.go — Test for single-provider rawsetenv and multi-provider rawsetenv
pkg/e2e/fixtures/providers/rawsetenv.yaml — Test fixture

Related issue

resolves #13727

phillias · 2026-06-14T16:50:20Z

this patch worked for my issue with mariadb requiring MYSQL_ROOT_PASSWORD with no override, not LOCKET_MYSQL_ROOT_PASSWORD as the compose provider was passing.

glours

Few things to address before we can merge.
Also we need to add a sibling fixture to pkg/e2e/fixtures/providers/rawsetenv.yaml that sets environment: { CLOUD_REGION: user-value } on the test service, and assert that the resulting value matches the documented behavior (skipped write, warning, or overwrite, depending on which resolution is chosen).

glours · 2026-06-15T13:25:43Z

 	}

-	variables, err := s.executePlugin(cmd, command, service)
+	vars, err := s.executePlugin(cmd, command, service)


Why this rename?

There's no good reason for it. Reverted to variables.

glours · 2026-06-15T13:29:50Z

 				s.Environment[prefix+key] = &val
 			}
+			for key, val := range vars.raw {
+				s.Environment[key] = &val


rawsetenv silently overwrites user-defined env vars

If a user sets environment: { CLOUD_REGION: eu-west-1 } and a provider emits rawsetenv CLOUD_REGION=us-east-1, the user value is silently clobbered. setenv was immune thanks to the prefix.
Please either skip writes when the key already exists, or we need to document this precedence explicitly.

@ndeloof what is you preference here? I'm in favor of the overwrite + warning message in logs, and you?

Went with overwrite + warning as suggested. One clarification on the framing though: setenv normally doesn't overwrite anything, since the prefix gives each provider its own namespace and collisions are rare. But the write has no existence check either, so if a service does define the exact prefixed key (e.g. a user-set SECRETS_URL and a provider named "secret" set URL), setenv silently overwrites it the same way. I kept the warning on rawsetenv only since prefixed collisions are unlikely, but can extend it to setenv if you'd prefer consistency.

glours · 2026-06-15T13:34:36Z

+This is useful when injecting secrets or configuration values that must match exact variable names expected by
+applications or frameworks. Unlike `setenv`, which avoids collisions through automatic prefixing, `rawsetenv` keys
+are the provider's responsibility to keep unique. If multiple providers emit the same `rawsetenv` key, the last one
+to run will overwrite previous values.


executor.go runs plan nodes concurrently via errgroup. Two providers without a mutual dependency run in parallel; mux serializes the writes but not their order. The doc says "the last one to run will overwrite previous values", literally true, but readers will assume declaration order wins.
Either tighten the doc, or detect cross-provider rawsetenv collisions and fail.

Reworded. Dropped the "last one to run will overwrite" phrasing and now state that providers without a depends_on link may run concurrently, so the resulting value is non-deterministic on collision, and that an override logs a warning.

glours · 2026-06-15T13:36:58Z

 	env := getEnv(res.Combined(), false)
 	assert.Check(t, slices.Contains(env, "PROVIDER1_URL=https://magic.cloud/provider1"), env)
 	assert.Check(t, slices.Contains(env, "PROVIDER2_URL=https://magic.cloud/provider2"), env)
+	assert.Check(t, slices.Contains(env, "CLOUD_REGION=us-east-1"), env)


both providers emit the same CLOUD_REGION value, so the conflict path is never exercised. Please add a test where a rawsetenv key collides with a user-defined env var

Removed the CLOUD_REGION assertion from TestDependsOnMultipleProviders since, as you noted, both providers emit the same value so it never exercised a conflict, and a real cross-provider conflict can't be asserted deterministically (independent providers run concurrently, so it would be flaky). The deterministic conflict is now covered by TestProviderRawSetEnvOverridesUserEnv, which asserts the provider value wins over a user-defined one and that the override is logged as a warning rather than happening silently. The non-deterministic cross-provider case is documented in extension.md.

rajyan · 2026-06-22T06:39:34Z

Hi @glours, I've addressed all the review comments. Would appreciate another look when you get a chance. Thanks!

glours

Almost there, thanks for the iteration on the collision handling. A couple of follow-ups before we can merge:

The collision fix covers KEY: value but not the - KEY / KEY: (inherit-from-shell) form — see inline comment on plugins.go. Same class of bug as the one I flagged previously, just exposed by a different YAML syntax.
extension.md needs a couple of doc tweaks so the protocol contract is consistent — the stop section still mentions only setenv, and the mermaid sequence diagram doesn't show rawsetenv.

glours · 2026-06-22T13:15:42Z

 				s.Environment[prefix+key] = &val
 			}
+			for key, val := range variables.raw {
+				if existing, ok := s.Environment[key]; ok && existing != nil && *existing != val {


Follow-up on the previous collision fix: the existing != nil guard short-circuits when the user declares the env var without a value, e.g.:

environment: - CLOUD_REGION # or environment: CLOUD_REGION:

MappingWithEquals stores that as *string == nil, meaning "inherit from the shell environment at runtime". A rawsetenv for the same key then overwrites it silently — no warning fires. Same class of bug as the case I flagged before, just exposed
by a different YAML form (and arguably the more common form when users want the host shell value to flow through).

Could you extend the condition to cover the nil case as well?

Suggested change

if existing, ok := s.Environment[key]; ok && existing != nil && *existing != val {

if existing, ok := s.Environment[key]; ok && (existing != nil || *existing != val) {

An additional e2e fixture that declares environment: [- CLOUD_REGION] and asserts the warning fires would lock the behaviour in.

Thanks for catching the nil-pointer case — I hadn't considered the - KEY / KEY: form. Fixed the condition to existing == nil || *existing != val so the warning fires for shell-inherited env vars as well.

Added e2e tests for both list-style (- KEY) and map-style (KEY:) declarations that assert the warning message, which fail against the previous code and pass with the fix.

glours · 2026-06-22T13:17:56Z

 When the user runs `docker compose stop`, Compose invokes `<provider> compose --project-name <NAME> stop <SERVICE>` for each
 provider-backed service in reverse dependency order. The provider should pause the resource without releasing it, so a later
 `docker compose up` can resume it (note that `docker compose start` only restarts existing containers and does not invoke
 provider hooks). Any `setenv` JSON message returned during `stop` is ignored, since dependent services are also stopping.


Now that rawsetenv is a defined message type, this sentence should mention it too. The code at plugins.go:84-86 ignores both setenv and rawsetenv when command == "stop", but the doc reads as if only setenv is dropped.

Suggested wording: Any setenvorrawsetenvJSON message returned duringstop is ignored, since dependent services are also stopping.

glours · 2026-06-22T13:21:03Z

 `type` can be either:
 - `info`: Reports status updates to the user. Compose will render message as the service state in the progress UI
 - `error`: Lets the user know something went wrong with details about the error. Compose will render the message as the reason for the service failure.
- `setenv`: Lets the plugin tell Compose how dependent services can access the created resource. See next section for further details.
+- `setenv`: Lets the plugin tell Compose how dependent services can access the created resource. The variable is automatically prefixed with the service name. See next section for further details.
+- `rawsetenv`: Same as `setenv`, but the variable is injected as-is without the service name prefix. Useful when applications require exact variable names that cannot be altered.
 - `debug`: Those messages could help debugging the provider, but are not rendered to the user by default. They are rendered when Compose is started with `--verbose` flag.

 ```mermaid
 sequenceDiagram
    Shell->>Compose: docker compose up
    Compose->>Provider: compose up --project-name=xx --foo=bar "database"
    Provider--)Compose: json { "info": "pulling 25%" }
    Compose-)Shell: pulling 25%
    Provider--)Compose: json { "info": "pulling 50%" }
    Compose-)Shell: pulling 50%
    Provider--)Compose: json { "info": "pulling 75%" }
    Compose-)Shell: pulling 75%
    Provider--)Compose: json { "setenv": "URL=http://cloud.com/abcd:1234" }
    Compose-)Compose: set DATABASE_URL
    Provider-)Compose: EOF (command complete) exit 0
    Compose-)Shell: service started
 ```

 ## Connection to a service managed by a provider

 A service in the Compose application can declare dependency on a service managed by an external provider: 

 ```yaml
 services:
  app:
    image: myapp 
    depends_on:
       - database

  database:
    provider:
      type: awesomecloud
 ```

 When the provider command sends a `setenv` JSON message, Compose injects the specified variable into any dependent service,
 automatically prefixing it with the service name. For example, if `awesomecloud compose up` returns:
 ```json
 {"type": "setenv", "message": "URL=https://awesomecloud.com/db:1234"}
 ```
 Then the `app` service, which depends on the service managed by the provider, will receive a `DATABASE_URL` environment variable injected
 into its runtime environment.

+When the provider command sends a `rawsetenv` JSON message, Compose injects the variable as-is without any prefix:
+```json
+{"type": "rawsetenv", "message": "SECRET_KEY=xxx"}
+```
+The `app` service will receive `SECRET_KEY` exactly as specified, regardless of the provider service name.
+This is useful when injecting secrets or configuration values that must match exact variable names expected by
+applications or frameworks.
+
+Unlike `setenv`, which avoids collisions through automatic prefixing, `rawsetenv` keys are the provider's
+responsibility to keep unique. If a `rawsetenv` key collides with a variable already set on the dependent service,
+the existing value is overwritten and Compose logs a warning. This includes variables declared by the user in the
+service `environment` section as well as values emitted by other providers. Providers that are not linked by a
+`depends_on` relationship may run concurrently, so when several of them emit the same `rawsetenv` key the resulting
+value is not deterministic.
+
 > __Note:__  The `compose up` provider command _MUST_ be idempotent. If resource is already running, the command _MUST_ set
 > the same environment variables to ensure consistent configuration of dependent services.

 ## Down lifecycle

 `down` lifecycle is equivalent to `up` with the `<provider> compose --project-name <NAME> down <SERVICE>` command.
 The provider is responsible for releasing all resources associated with the service.

 ## Stop lifecycle

 When the user runs `docker compose stop`, Compose invokes `<provider> compose --project-name <NAME> stop <SERVICE>` for each
 provider-backed service in reverse dependency order. The provider should pause the resource without releasing it, so a later
 `docker compose up` can resume it (note that `docker compose start` only restarts existing containers and does not invoke
 provider hooks). Any `setenv` JSON message returned during `stop` is ignored, since dependent services are also stopping.

 The `stop` hook is opt-in: Compose invokes it only when the provider declares a `stop` block in its `metadata` subcommand
 output. Providers that do not advertise `stop` in metadata (or do not implement the `metadata` subcommand at all) are
 silently skipped during `docker compose stop`, preserving backward compatibility with providers that pre-date this hook.

 The `--timeout` flag of `docker compose stop` applies only to container services; provider stop hooks are not subject to
 this timeout and are responsible for managing their own shutdown duration.

 ## Provide metadata about options

 Compose extensions *MAY* optionally implement a `metadata` subcommand to provide information about the parameters accepted by the `up` and `down` commands.  

 The `metadata` subcommand takes no parameters and returns a JSON structure on the `stdout` channel that describes the parameters accepted by both the `up` and `down` commands, including whether each parameter is mandatory or optional.

 ```console
 awesomecloud compose metadata
 ```

 The expected JSON output format is:
 ```json
 {
  "description": "Manage services on AwesomeCloud",
  "up": {
    "parameters": [
      {
        "name": "type",
        "description": "Database type (mysql, postgres, etc.)",
        "required": true,
        "type": "string"
      },
      {
        "name": "size",
        "description": "Database size in GB",
        "required": false,
        "type": "integer",
        "default": "10"
      },
      {
        "name": "name",
        "description": "Name of the database to be created",
        "required": true,
        "type": "string"
      }
    ]
  },
  "down": {
    "parameters": [
      {
        "name": "name",
        "description": "Name of the database to be removed",
        "required": true,
        "type": "string"


Could you extend the sequence diagram to show rawsetenv as well? Right now plugin authors reading the diagram alone would not know it exists.

rajyan · 2026-06-23T01:45:37Z

@glours Thanks for the review! Addressed the follow-ups:

Fixed the nil-pointer guard for - KEY / KEY: form and added e2e tests.
Updated the stop section to mention rawsetenv and added it to the mermaid sequence diagram.

glours

We're good to go! Thank you very much for this contribution @rajyan 🙏

edit: It seems the CI validation step disagree with me 😬

codecov · 2026-06-23T07:28:02Z

Codecov Report

❌ Patch coverage is 56.52174% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/compose/plugins.go	56.52%	9 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

rajyan · 2026-06-23T08:07:15Z

@glours The lint CI is failing with unparam on the getEnv function in pkg/e2e/providers_test.go:

pkg/e2e/providers_test.go:158:25: getEnv - run always receives false (unparam)

This getEnv(out string, run bool) function already existed on main with the run parameter always being false, but lint passes on main — likely because there's only one call site, so unparam doesn't flag it. My PR added more test cases calling getEnv(..., false), which seems to have triggered the detection.

Since this is a pre-existing issue, how would you prefer I handle it? Should I remove the run parameter and the dead code branch in this PR, or would you prefer a separate fix?

glours · 2026-06-23T08:17:58Z

@rajyan I did a PR to fix it, I'll let you know when merged so you could rebase your branch

rajyan · 2026-06-23T08:20:41Z

@glours Thanks for the quick fix! I'll rebase once it's merged.

I also noticed the Codecov check is failing. It looks like it might be due to not having the latest main changes, so rebasing should fix that as well.

The run parameter was always passed as false at the single call site and the run==true branch was dead code. Remove it so unparam stops flagging callers added by PR #13742. Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>

glours · 2026-06-23T08:44:34Z

@rajyan it's merged, you can rebase now

Providers can now send rawsetenv messages to inject environment variables into dependent services without the automatic service name prefix. This enables use cases where applications require exact variable names that cannot be altered. Closes docker#13727 Signed-off-by: Yohta Kimura <38206553+rajyan@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

rawsetenv injects provider variables without the service-name prefix, so a key can collide with a value already set on the dependent service, whether declared by the user in environment or emitted by another provider. Log a warning and overwrite on collision, document the precedence and the non-deterministic ordering between concurrent providers, and cover the user-environment override with an e2e test. Signed-off-by: Yohta Kimura <38206553+rajyan@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The stop section only mentioned setenv being ignored; rawsetenv is handled identically by the code but was missing from the docs. The mermaid sequence diagram also lacked a rawsetenv step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Yohta Kimura <38206553+rajyan@users.noreply.github.com>

When a service declares an env var without a value (e.g. `- KEY` or `KEY:`), MappingWithEquals stores it as a nil *string. The previous condition `existing != nil && ...` skipped the warning for this case, allowing silent overwrites. Change to `existing == nil || ...` so the warning fires for both nil (shell-inherit) and value-mismatch cases. Add e2e tests for both list-style (`- KEY`) and map-style (`KEY:`) YAML forms to lock in the behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Yohta Kimura <38206553+rajyan@users.noreply.github.com>

rajyan · 2026-06-23T12:22:48Z

@glours
Thank you! Rebased with the latest main.

glours

LGTM, thanks @rajyan

rajyan requested a review from a team as a code owner April 16, 2026 08:20

rajyan requested review from glours and ndeloof April 16, 2026 08:20

rajyan force-pushed the 13727-rawsetenv branch from ba605f9 to 3ab9b49 Compare May 13, 2026 06:33

rajyan mentioned this pull request May 13, 2026

Provider services: cannot inject environment variables without service name prefix #13727

Closed

rajyan force-pushed the 13727-rawsetenv branch from 3ab9b49 to 101a41c Compare June 3, 2026 12:43

glours requested changes Jun 15, 2026

View reviewed changes

glours requested changes Jun 22, 2026

View reviewed changes

rajyan force-pushed the 13727-rawsetenv branch from 72c444f to 48cc0b5 Compare June 23, 2026 01:42

glours previously approved these changes Jun 23, 2026

View reviewed changes

glours mentioned this pull request Jun 23, 2026

pkg/e2e: drop unused run param from getEnv #13867

Merged

rajyan and others added 4 commits June 23, 2026 21:16

rajyan dismissed glours’s stale review via 451bb97 June 23, 2026 12:18

rajyan force-pushed the 13727-rawsetenv branch from 48cc0b5 to 451bb97 Compare June 23, 2026 12:18

glours approved these changes Jun 23, 2026

View reviewed changes

glours merged commit 43922d5 into docker:main Jun 23, 2026
42 checks passed

rajyan deleted the 13727-rawsetenv branch June 23, 2026 13:24

	if existing, ok := s.Environment[key]; ok && existing != nil && *existing != val {
	if existing, ok := s.Environment[key]; ok && (existing != nil \|\| *existing != val) {

Conversation

rajyan commented Apr 16, 2026

What I did

Changes

Related issue

Uh oh!

phillias commented Jun 14, 2026

Uh oh!

glours left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rajyan commented Jun 22, 2026

Uh oh!

glours left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rajyan commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glours left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jun 23, 2026

Codecov Report

Uh oh!

rajyan commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glours commented Jun 23, 2026

Uh oh!

rajyan commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glours commented Jun 23, 2026

Uh oh!

rajyan commented Jun 23, 2026

Uh oh!

glours left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rajyan commented Jun 23, 2026 •

edited

Loading

glours left a comment •

edited

Loading

rajyan commented Jun 23, 2026 •

edited

Loading

rajyan commented Jun 23, 2026 •

edited

Loading