Add bump-serverless-env-version skill

.agents/rules/template-versions.md

@@ -0,0 +1,24 @@
+---
+description: Version pins in bundle template versions.tmpl files (DBR, DB Connect, serverless env, Python)
+globs:
+  - "libs/template/templates/**/library/versions.tmpl"
+paths:
+  - "libs/template/templates/**/library/versions.tmpl"
+---
+
+# Bundle template version pins
+
+Each bundle template pins the runtime versions a freshly initialized project
+ships with in `library/versions.tmpl`. The `default` template pins the full set
+— `latest_lts_dbr_version`, `conservative_db_connect_version_spec`,
+`serverless_environment_version`, `python_version_spec`, and
+`default_python_version`; the SQL templates (`dbt-sql`, `default-sql`) define
+only a subset.
+
+**RULE: Keep `conservative_db_connect_version_spec` (in `default/`) at the lowest version that still works. Bump it only when the pinned DBR release falls out of support — never just to match the newest serverless environment version.** The DB Connect client is only forward-compatible: it reaches compute of its own version and higher. The lowest working pin therefore maximizes the range of DBR versions a customer can connect to, whereas a high pin rules out customers on older DBR. Customers can upgrade themselves after initializing the template. This is why the pin typically lags the newest release. See PR #3897 and PR #6378 for prior history.
+
+**RULE: In `default/`, keep `serverless_environment_version`, the Python pins (`python_version_spec` / `default_python_version`), and `conservative_db_connect_version_spec` mutually compatible.** They form one set: a serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. For example, environment version 5 uses Python 3.12, and DB Connect 16.4 supports Python 3.12. When you change any one, cross-check the other two against the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements).
+
+**RULE: `serverless_environment_version` is the one macro that must stay in sync across templates — it is pinned to the same value (`5`) in both `default/` and `dbt-sql/`, so bump it in both.** Other macros that appear in more than one template hold intentionally different values and must NOT be synced: `latest_lts_dbr_version` is `16.4` in `default/` but `15.4` in the SQL templates, and each SQL template pins its own `latest_lts_db_connect_version_spec` (a distinct macro from `default/`'s `conservative_db_connect_version_spec`). Change only the templates that define a given macro, and only for the same reason.
+
+Changing a version pin changes rendered template output, so regenerate the acceptance goldens afterward (see [auto-generated-files.md](auto-generated-files.md)).

.agents/skills/bump-serverless-env-version/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: bump-serverless-env-version
+description: "Use when bumping or upgrading the default serverless environment version shipped by Databricks bundle templates, including its coupled Python and DB Connect pins."
+user-invocable: true
+allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion
+---
+
+# Bump the serverless environment version
+
+Read `.agents/rules/template-versions.md` first for compatibility and
+synchronization policy. Never hand-edit generated acceptance output.
+
+## 1. Resolve a compatible version set
+
+Use the requested environment version, or the newest published version when none
+was specified. Confirm its runtime Python version from the
+[environment release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/)
+and [databricks/environments](https://github.com/databricks/environments).
+
+Apply `.agents/rules/template-versions.md` to the environment, Python, and DB
+Connect pins. Check Python compatibility in the
+[DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements).
+Advance `conservative_db_connect_version_spec` only when its DBR line is no longer
+supported, using the supported-LTS table in the
+[Databricks Runtime release notes](https://docs.databricks.com/aws/en/release-notes/runtime/).
+Do not advance it merely to match the environment version.
+
+## 2. Update every template source
+
+Find both macro definitions and hardcoded literals; trust the search results:
+
+```bash
+grep -rn 'environment_version\|environment-version' libs/template/templates/
+```
+
+Inspect every hit and change only values that pin a version. Re-run the command
+after editing and confirm every version-bearing template hit uses the target.
+If all sources and coupled pins already match the resolved set and the worktree has
+no partial bump, report that no changes are needed and stop.
+
+Current sources include:
+
+- `serverless_environment_version` definitions in `default/library/versions.tmpl`
+  and `dbt-sql/library/versions.tmpl`;
+- a hardcoded value in the `default-scala` job template;
+- three hardcoded values in `lakeflow-integrations`, including its
+  `--environment-version` argument.
+
+In `default/library/versions.tmpl`, update `python_version_spec` and
+`default_python_version` when the runtime Python version changes. Update
+`conservative_db_connect_version_spec` only under the support rule above. Keep the
+version-specific compatibility comments accurate even when their pin is unchanged.
+Do not synchronize unrelated DBR or SQL-template DB Connect macros.
+
+Update version-specific examples in `.agents/rules/template-versions.md` so its
+policy remains accurate; do not change the policy itself as part of the bump.
+
+Also inspect `defaultServerlessVersion` in `libs/localenv/envkey.go`. It is the
+product-spec fallback for `databricks environments setup-local`, not a template
+pin; do not infer that it should move with the templates. Change it only when the
+user or current product specification also requires the fallback to move, and first
+confirm `python/serverless/serverless-v{N}/pyproject.toml` exists in
+`databricks/environments`. Then update default-version help, error, and test
+expectations under `libs/localenv` and `cmd/environments`; verify with
+`go test ./libs/localenv ./cmd/environments`, then update and verify with:
+
+```bash
+go test ./acceptance -run '^TestAccept/localenv' -update -timeout=60m
+go test ./acceptance -run '^TestAccept/localenv' -timeout=60m
+```
+
+Otherwise leave it unchanged and record why. Do not change the intentionally older
+SSH fixtures in `acceptance/ssh/connect-serverless-*`.
+
+## 3. Regenerate and verify targeted goldens
+
+Update and verify both template acceptance trees:
+
+```bash
+./task test-update-templates
+go test ./acceptance -run '^TestAccept/pipelines' -update -timeout=60m
+
+go test ./acceptance -run '^TestAccept/bundle/templates' -timeout=60m
+go test ./acceptance -run '^TestAccept/pipelines' -timeout=60m
+```
+
+Both non-update commands must pass. Update mode selects covering `EnvMatrix`
+variants; the non-update runs verify every variant against the regenerated goldens.
+
+## 4. Add the changelog fragment
+
+Add `.nextchanges/bundles/serverless-environment-version-v{N}.md`. Follow the
+`pr-checklist` skill's changelog conventions. Describe a benefit stated in the
+target version's release notes, link the actual version page, and mention the DB
+Connect version only if it changed.
+
+Cross-check the final source and generated-output footprint against the prior
+template bumps in [PR #3897](https://github.com/databricks/cli/pull/3897) and
+[PR #6378](https://github.com/databricks/cli/pull/6378). Explain material
+differences in the final handoff or PR description. Prior PRs are not sources of
+truth: revalidate their compatibility decisions, wording, URLs, and file lists.
+
+## 5. Finish only when requested
+
+**Required sub-skill:** use `pr-checklist`, run its checks, and inspect the final
+diff. If formatting or linting changes generated acceptance files, fix the source
+and regenerate them.
+
+Commit, push, or create/update a PR only when the user explicitly requests that
+operation. When requested, follow `pr-checklist` rather than duplicating its commit
+and PR-body instructions here.

libs/template/templates/default/library/versions.tmpl

@@ -5,19 +5,20 @@
   16.4.x-scala2.12
 {{- end}}
 
-{{/* A conservative version of DB Connect for local development.
+{{/* A conservative DB Connect pin for local development.
    *
-   * DB Connect is only forward-compatible: a client connects to compute of the
-   * same version and higher. We keep this conservative (low) so a freshly
-   * initialized project can reach the widest range of DBR versions; a high
-   * version would rule out customers on older DBR. We use 16.4 rather than an
-   * older release because DBR 15 is no longer supported and used Python 3.11,
-   * whereas 16.4 uses Python 3.12 (matching the serverless environment and the
-   * template's Python pin). Customers can move to a newer version themselves
-   * after initializing the template.
+   * The DB Connect client is only forward-compatible (it reaches compute of its
+   * own version and higher), so keep this at the LOWEST version that still
+   * works: bump it only when the pinned DBR release falls out of support, never
+   * to match the newest serverless environment version. A higher pin rules out
+   * customers on older DBR; they can upgrade themselves after init.
    *
-   * See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements
-   * for DB Connect release notes and version compatibility.
+   * Current floor is 16.4 (DBR 15 is out of support and ran Python 3.11; 16.4
+   * runs Python 3.12, matching serverless env 5 and python_version_spec).
+   *
+   * See .agents/rules/template-versions.md for the full upgrade rule and the
+   * env-version / Python / DB Connect consistency invariant.
+   * https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements
    */}}
 {{define "conservative_db_connect_version_spec" -}}
   >=16.4,<16.5