Skip to content

python-sdk: report only what generation produced - #18

Merged
eunomie merged 2 commits into
dagger:mainfrom
eunomie:python-sdk-generate-context-changeset
Aug 25, 2026
Merged

python-sdk: report only what generation produced#18
eunomie merged 2 commits into
dagger:mainfrom
eunomie:python-sdk-generate-context-changeset

Conversation

@eunomie

@eunomie eunomie commented Aug 21, 2026

Copy link
Copy Markdown
Member

Follow-up to #14, which merged before this landed.

Generation used moduleSource(...).generate(ws).changes(ws). On v1.0.0-beta.10
that loses the diff baseline: the module's whole generated context comes back as
added instead of only what generation changed. Every generate and
generateAll claimed to add the module's own dagger.json, which the engine
owns and had not touched — and once a module already has generated output on
disk, the files codegen rewrites byte for byte are reported as added too.

Root cause

Reproduced on v1.0.0-beta.10 here. It is not workspaceChangesBetween, as an
earlier revision of this PR said — that function applies the overlay to both
sides, identically on beta.10 and on main.

A Changeset reports the paths whose content differs, but
Directory.withChanges (and Changeset.layer / Changeset.export) project the
structural before/after diff, which a fresh mtime alone puts a path into —
and codegen gives a fresh mtime to every file it rewrites. A host-backed
workspace then sizes the baseline it compares against from the reported paths,
so the surplus files diff against nothing and read as newly added.

generatedContextChangeset.addedPaths   →  .gitattributes, .gitignore, sdk/**   ← correct
generatedContextChangeset.layer        →  ...the same, plus dagger.json         ← surplus
ws.withChanges(cs).changes(ws)         →  ...the surplus, reported as added

On a re-run, with generated output already on disk and one generated file
drifted, the old idiom reports .gitattributes, .gitignore and dagger.json
as added while the correct answer is "one file modified".

Fixed by dagger/dagger#13947, which is open and unmerged: not in
v1.0.0-beta.10, and not on dagger/dagger@main.

This is the same defect for both shapes python-sdk sees — the engine-owned
dagger.json on a first generate, and already-on-disk generated files on a
re-generate. Both leak through Changeset.layer / Directory.withChanges
projecting the structural diff, so #13947's
TestWorkspaceChangesAgainstExistingBaseline covers them; nothing here needs a
separate engine report. (Changeset.export has the same gap and reproduces
here too: exporting the generated context rewrote the fixture's dagger.json
though it was never a reported path.)

Fix

Write the generated context as a directory rather than threading a workspace
through ModuleSource.generate — the shape go-sdk settled on in
dagger/go-sdk#30. A directory overlay diffs correctly, so Workspace.changes
still does the cwd rooting and the outside-the-cwd guard, and nothing here has
to be reverted once dagger/dagger#13947 lands.

generatedContextDirectory does not resolve the local dependency closure the way
generate(ws) did, so generation stages it explicitly, and generateAll merges
each module's changeset rather than threading one workspace through all of them.

The generate fixture now commits the .gitattributes generation emits, so
e-2-e:generate-skips-existing-files-check catches a generator returning its
whole context; every other file there is genuinely new on a fresh module, so only
an already-present one can catch it. On the previous idiom it fails with:

generate re-added files that already exist:
  .dagger/modules/e2e/fixtures/generate/app/dagger.json,
  .dagger/modules/e2e/fixtures/generate/app/.gitattributes

sdk-sdk

Also bumps sdk-sdk to 3344489 (dagger/sdk-sdk#17), which adds a monorepo
check group for a workspace config in a subdirectory of the git root. sdk-sdk
reported three of those red against the SDK they were written with; all six pass
here on beta.10, which carries the engine-side fix for dagger/dagger#13889.

dagger check: 43/43 (was 37/37).

Picks up dagger/sdk-sdk#17, a `monorepo` check group covering a workspace
config in a subdirectory of the git root: the layout where the caller's cwd
sits below the workspace root, so module paths cross the engine <-> SDK
boundary root-relative while init changesets are applied at the root.

sdk-sdk reported three of those checks red against the SDK it was written with;
all six pass here on v1.0.0-beta.10, which carries the engine-side fix for
dagger/dagger#13889. `dagger check`: 43/43, up from 37/37.

The golang:1.26-alpine digest is a transitive pin the workspace already
resolves at check time; recording it keeps a check run from dirtying the tree.

Signed-off-by: Yves Brissaud <yves@dagger.io>
Generation threaded the workspace through ModuleSource.generate and diffed it
back with Workspace.changes. On v1.0.0-beta.10 that loses the baseline: the
module's whole generated context comes back as added rather than only what
generation changed. Every generate and generateAll claimed to add the module's
own dagger.json, which the engine owns and had not touched, and once a module
has generated output on disk, files codegen rewrites byte for byte are reported
as added too.

The engine defect is in applying a changeset, not in comparing workspaces:
withChanges projects the structural before/after diff, which a fresh mtime alone
puts a path into, rather than the content-based paths the changeset reports.
dagger/dagger#13947 fixes it.

Write the generated context as a directory instead, the shape go-sdk settled on
in dagger/go-sdk#30. A directory overlay diffs correctly, so Workspace.changes
still does the cwd rooting and the outside-the-cwd guard, and nothing here has
to be reverted once the engine fix lands.

generatedContextDirectory does not resolve the local dependency closure the way
generate(ws) did, so generation stages it explicitly, and generateAll merges
each module's changeset rather than threading one workspace through all of them.

The generate fixture now commits the .gitattributes that generation emits, so
e-2-e:generate-skips-existing-files-check catches a generator returning its
whole context: every other file there is genuinely new on a fresh module.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
eunomie force-pushed the python-sdk-generate-context-changeset branch from 842e489 to c724b03 Compare August 25, 2026 14:15
@eunomie
eunomie marked this pull request as ready for review August 25, 2026 14:26
@eunomie
eunomie merged commit 21701fe into dagger:main Aug 25, 2026
44 checks passed
@eunomie
eunomie deleted the python-sdk-generate-context-changeset branch August 25, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant