diff --git a/.agents/rules/template-versions.md b/.agents/rules/template-versions.md new file mode 100644 index 00000000000..e9bbd9ec75f --- /dev/null +++ b/.agents/rules/template-versions.md @@ -0,0 +1,24 @@ +--- +description: Version pins in bundle template versions.tmpl files (DBR, DB Connect, serverless env, Python) +globs: + - "libs/template/templates/**/library/versions.tmpl" +paths: + - "libs/template/templates/**/library/versions.tmpl" +--- + +# Bundle template version pins + +Each bundle template pins the runtime versions a freshly initialized project +ships with in `library/versions.tmpl`. The `default` template pins the full set +— `latest_lts_dbr_version`, `conservative_db_connect_version_spec`, +`serverless_environment_version`, `python_version_spec`, and +`default_python_version`; the SQL templates (`dbt-sql`, `default-sql`) define +only a subset. + +**RULE: Keep `conservative_db_connect_version_spec` (in `default/`) at the lowest version that still works. Bump it only when the pinned DBR release falls out of support — never just to match the newest serverless environment version.** The DB Connect client is only forward-compatible: it reaches compute of its own version and higher. The lowest working pin therefore maximizes the range of DBR versions a customer can connect to, whereas a high pin rules out customers on older DBR. Customers can upgrade themselves after initializing the template. This is why the pin typically lags the newest release. See PR #3897 and PR #6378 for prior history. + +**RULE: In `default/`, keep `serverless_environment_version`, the Python pins (`python_version_spec` / `default_python_version`), and `conservative_db_connect_version_spec` mutually compatible.** They form one set: a serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. For example, environment version 5 uses Python 3.12, and DB Connect 16.4 supports Python 3.12. When you change any one, cross-check the other two against the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements). + +**RULE: `serverless_environment_version` is the one macro that must stay in sync across templates — it is pinned to the same value (`5`) in both `default/` and `dbt-sql/`, so bump it in both.** Other macros that appear in more than one template hold intentionally different values and must NOT be synced: `latest_lts_dbr_version` is `16.4` in `default/` but `15.4` in the SQL templates, and each SQL template pins its own `latest_lts_db_connect_version_spec` (a distinct macro from `default/`'s `conservative_db_connect_version_spec`). Change only the templates that define a given macro, and only for the same reason. + +Changing a version pin changes rendered template output, so regenerate the acceptance goldens afterward (see [auto-generated-files.md](auto-generated-files.md)). diff --git a/.agents/skills/bump-serverless-env-version/SKILL.md b/.agents/skills/bump-serverless-env-version/SKILL.md new file mode 100644 index 00000000000..0c9f946259d --- /dev/null +++ b/.agents/skills/bump-serverless-env-version/SKILL.md @@ -0,0 +1,111 @@ +--- +name: bump-serverless-env-version +description: "Use when bumping or upgrading the default serverless environment version shipped by Databricks bundle templates, including its coupled Python and DB Connect pins." +user-invocable: true +allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion +--- + +# Bump the serverless environment version + +Read `.agents/rules/template-versions.md` first for compatibility and +synchronization policy. Never hand-edit generated acceptance output. + +## 1. Resolve a compatible version set + +Use the requested environment version, or the newest published version when none +was specified. Confirm its runtime Python version from the +[environment release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) +and [databricks/environments](https://github.com/databricks/environments). + +Apply `.agents/rules/template-versions.md` to the environment, Python, and DB +Connect pins. Check Python compatibility in the +[DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements). +Advance `conservative_db_connect_version_spec` only when its DBR line is no longer +supported, using the supported-LTS table in the +[Databricks Runtime release notes](https://docs.databricks.com/aws/en/release-notes/runtime/). +Do not advance it merely to match the environment version. + +## 2. Update every template source + +Find both macro definitions and hardcoded literals; trust the search results: + +```bash +grep -rn 'environment_version\|environment-version' libs/template/templates/ +``` + +Inspect every hit and change only values that pin a version. Re-run the command +after editing and confirm every version-bearing template hit uses the target. +If all sources and coupled pins already match the resolved set and the worktree has +no partial bump, report that no changes are needed and stop. + +Current sources include: + +- `serverless_environment_version` definitions in `default/library/versions.tmpl` + and `dbt-sql/library/versions.tmpl`; +- a hardcoded value in the `default-scala` job template; +- three hardcoded values in `lakeflow-integrations`, including its + `--environment-version` argument. + +In `default/library/versions.tmpl`, update `python_version_spec` and +`default_python_version` when the runtime Python version changes. Update +`conservative_db_connect_version_spec` only under the support rule above. Keep the +version-specific compatibility comments accurate even when their pin is unchanged. +Do not synchronize unrelated DBR or SQL-template DB Connect macros. + +Update version-specific examples in `.agents/rules/template-versions.md` so its +policy remains accurate; do not change the policy itself as part of the bump. + +Also inspect `defaultServerlessVersion` in `libs/localenv/envkey.go`. It is the +product-spec fallback for `databricks environments setup-local`, not a template +pin; do not infer that it should move with the templates. Change it only when the +user or current product specification also requires the fallback to move, and first +confirm `python/serverless/serverless-v{N}/pyproject.toml` exists in +`databricks/environments`. Then update default-version help, error, and test +expectations under `libs/localenv` and `cmd/environments`; verify with +`go test ./libs/localenv ./cmd/environments`, then update and verify with: + +```bash +go test ./acceptance -run '^TestAccept/localenv' -update -timeout=60m +go test ./acceptance -run '^TestAccept/localenv' -timeout=60m +``` + +Otherwise leave it unchanged and record why. Do not change the intentionally older +SSH fixtures in `acceptance/ssh/connect-serverless-*`. + +## 3. Regenerate and verify targeted goldens + +Update and verify both template acceptance trees: + +```bash +./task test-update-templates +go test ./acceptance -run '^TestAccept/pipelines' -update -timeout=60m + +go test ./acceptance -run '^TestAccept/bundle/templates' -timeout=60m +go test ./acceptance -run '^TestAccept/pipelines' -timeout=60m +``` + +Both non-update commands must pass. Update mode selects covering `EnvMatrix` +variants; the non-update runs verify every variant against the regenerated goldens. + +## 4. Add the changelog fragment + +Add `.nextchanges/bundles/serverless-environment-version-v{N}.md`. Follow the +`pr-checklist` skill's changelog conventions. Describe a benefit stated in the +target version's release notes, link the actual version page, and mention the DB +Connect version only if it changed. + +Cross-check the final source and generated-output footprint against the prior +template bumps in [PR #3897](https://github.com/databricks/cli/pull/3897) and +[PR #6378](https://github.com/databricks/cli/pull/6378). Explain material +differences in the final handoff or PR description. Prior PRs are not sources of +truth: revalidate their compatibility decisions, wording, URLs, and file lists. + +## 5. Finish only when requested + +**Required sub-skill:** use `pr-checklist`, run its checks, and inspect the final +diff. If formatting or linting changes generated acceptance files, fix the source +and regenerate them. + +Commit, push, or create/update a PR only when the user explicitly requests that +operation. When requested, follow `pr-checklist` rather than duplicating its commit +and PR-body instructions here. diff --git a/libs/template/templates/default/library/versions.tmpl b/libs/template/templates/default/library/versions.tmpl index 2c58247bc6c..b41af62ce73 100644 --- a/libs/template/templates/default/library/versions.tmpl +++ b/libs/template/templates/default/library/versions.tmpl @@ -5,19 +5,20 @@ 16.4.x-scala2.12 {{- end}} -{{/* A conservative version of DB Connect for local development. +{{/* A conservative DB Connect pin for local development. * - * DB Connect is only forward-compatible: a client connects to compute of the - * same version and higher. We keep this conservative (low) so a freshly - * initialized project can reach the widest range of DBR versions; a high - * version would rule out customers on older DBR. We use 16.4 rather than an - * older release because DBR 15 is no longer supported and used Python 3.11, - * whereas 16.4 uses Python 3.12 (matching the serverless environment and the - * template's Python pin). Customers can move to a newer version themselves - * after initializing the template. + * The DB Connect client is only forward-compatible (it reaches compute of its + * own version and higher), so keep this at the LOWEST version that still + * works: bump it only when the pinned DBR release falls out of support, never + * to match the newest serverless environment version. A higher pin rules out + * customers on older DBR; they can upgrade themselves after init. * - * See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements - * for DB Connect release notes and version compatibility. + * Current floor is 16.4 (DBR 15 is out of support and ran Python 3.11; 16.4 + * runs Python 3.12, matching serverless env 5 and python_version_spec). + * + * See .agents/rules/template-versions.md for the full upgrade rule and the + * env-version / Python / DB Connect consistency invariant. + * https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements */}} {{define "conservative_db_connect_version_spec" -}} >=16.4,<16.5