Expand reused nested types per-path in direct resources generator - #6618
Conversation
get_field_behaviors threaded a single global `visited` set through the
whole schema walk, so a type reached at two sibling paths was expanded
only at the first. A reused element type — e.g. the destination config
under both config.routing.destinations[*] and
config.routing.fallback.destinations[*] — lost its nested OUTPUT_ONLY /
INPUT_ONLY annotations at the second path, which then surfaced as phantom
drift on those server-populated fields.
Scope `visited` to the current root->node path (pass `visited | {ref}`
per branch) so a self-referential type still terminates the recursion,
but a type reused at sibling paths is expanded at each occurrence.
Regenerating resources.generated.yml recovers the dropped suppressions on
apps, database_instances, and external_locations.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: 2f07b5e
Top 3 slowest tests (at least 2 minutes):
|
janniklasrose
left a comment
There was a problem hiding this comment.
Self-review notes: for each newly-generated suppression, the spec type of the field and where that type was already visited (so the diff is easier to follow). These are all fields whose OUTPUT_ONLY/INPUT_ONLY the global per-type visited set was dropping at a second path; scoping it per-path restores them.
| apps: | ||
|
|
||
| ignore_remote_changes: | ||
| - field: git_repository.caller_credential_id |
There was a problem hiding this comment.
Type: apps.GitRepository.caller_credential_id — INPUT_ONLY (a write-only PAT credential id the server never returns).
New because the fix stopped the global visited set from swallowing it. apps.GitRepository is reached and marked visited earlier, under the OUTPUT_ONLY parents active_deployment (→ AppDeployment.git_source → GitSource.git_repository) and default_git_source. Those parents collapse to a single entry (filter_prefixes), so that first occurrence is invisible in this file — which is why there's no sibling line to compare against. Per-path visited now lets the top-level git_repository branch expand on its own; on this path there's no parent to propagate, so it keeps its intrinsic INPUT_ONLY.
| reason: spec:input_only | ||
| - field: node_count | ||
| reason: spec:input_only | ||
| - field: parent_instance_ref.lsn |
There was a problem hiding this comment.
Type: database.DatabaseInstanceRef.lsn — INPUT_ONLY. parent_instance_ref is the same type as each child_instance_refs[] element (already emitted above).
Previously dropped because child_instance_refs[*] visited DatabaseInstanceRef first. Behavior is path-dependent: under child_instance_refs (OUTPUT_ONLY parent) lsn shows as output_only; here under parent_instance_ref (IMMUTABLE, no propagation) it keeps its intrinsic input_only. Both correct — that path-dependence is exactly what per-path expansion restores.
| reason: spec:output_only | ||
| - field: effective_custom_tags | ||
| reason: spec:output_only | ||
| - field: effective_custom_tags[*].key |
There was a problem hiding this comment.
Type: database.CustomTag.key / .value — these have no intrinsic behavior; they inherit from the parent field. Both custom_tags[] and effective_custom_tags[] are database.CustomTag.
custom_tags[*] (INPUT_ONLY) visited CustomTag first, so these were dropped. They are the OUTPUT_ONLY mirror of the already-present custom_tags[*].key / .value (input_only) — same type, opposite behavior, because the parent differs.
| reason: spec:output_only | ||
| - field: effective_usage_policy_id | ||
| reason: spec:output_only | ||
| - field: parent_instance_ref.effective_lsn |
There was a problem hiding this comment.
Type: database.DatabaseInstanceRef.effective_lsn / .uid — both intrinsically OUTPUT_ONLY. Same DatabaseInstanceRef as child_instance_refs[].
Dropped before because child_instance_refs[*] visited the type first; recovered per-path. Compare the already-present child_instance_refs[*].effective_lsn / .uid above.
| reason: spec:output_only | ||
| - field: effective_file_event_queue | ||
| reason: spec:output_only | ||
| - field: file_event_queue.managed_aqs.managed_resource_id |
There was a problem hiding this comment.
Types: file_event_queue and effective_file_event_queue are both catalog.FileEventQueue; inside it managed_aqs/provided_aqs → catalog.AzureQueueStorage, managed_pubsub/provided_pubsub → catalog.GcpPubsub, managed_sqs/provided_sqs → catalog.AwsSqsQueue. Each queue type's managed_resource_id is OUTPUT_ONLY (server-computed).
Two layers of reuse hid all six before: (1) effective_file_event_queue (OUTPUT_ONLY, earlier field) visited FileEventQueue first and collapses to a single entry, so these were never visible; (2) within one FileEventQueue, the managed_/provided_ pairs share the same cloud type, so even a single expansion dropped the second of each pair. Per-path visited expands file_event_queue and each of the six sub-configs independently.
…the generator change #6618 expands reused nested types per-path in the direct resources generator, which changes the field paths the catalog enumerates for these two types. Both are local_only, so this is their local truth -- verdict counts shift (database_instances' COLLATERAL_DRIFT findings actually decrease, external_locations stays finding-free), no new problems, deterministic. Regenerated to match the new structure. Co-authored-by: Isaac
Integration test reportCommit: 95206fb
461 interesting tests: 453 FAIL, 7 KNOWN, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
Follow-up to the mcp_service support, mirroring #6654 for model_services now that #6618 enabled it. Added catalog.McpService to apitypes.yml so the generator reads field behaviors from the response type (config.* paths), removed the hand-written config.source_connection.is_deleted override from resources.yml, and regenerated resources.generated.yml. The suppression is unchanged, just auto-derived from the OpenAPI schema instead of hand-listed. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up to the mcp_service support, mirroring #6654 for model_services now that #6618 enabled it. Added catalog.McpService to apitypes.yml so the generator reads field behaviors from the response type (config.* paths), removed the hand-written config.source_connection.is_deleted override from resources.yml, and regenerated resources.generated.yml. The suppression is unchanged, just auto-derived from the OpenAPI schema instead of hand-listed. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up to the mcp_service support, mirroring #6654 for model_services now that #6618 enabled it. Added catalog.McpService to apitypes.yml so the generator reads field behaviors from the response type (config.* paths), removed the hand-written config.source_connection.is_deleted override from resources.yml, and regenerated resources.generated.yml. The suppression is unchanged, just auto-derived from the OpenAPI schema instead of hand-listed. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up mirroring #6654 (model_services) and the mcp_services cleanup, now that #6618 enabled generating field behaviors from the response type. Added catalog.ModelProviderService to apitypes.yml, removed the hand-written recreate_on_changes (config.provider_type) and ignore_remote_changes (inference_table + write-only provider credentials) from resources.yml, and regenerated resources.generated.yml. The generator now derives all of these from the OpenAPI schema (and additionally picks up config.anthropic.relayed. plan_type as immutable); no suppression is lost. Co-authored-by: Isaac <no-reply@databricks.com>
…the generator change #6618 expands reused nested types per-path in the direct resources generator, which changes the field paths the catalog enumerates for these two types. Both are local_only, so this is their local truth -- verdict counts shift (database_instances' COLLATERAL_DRIFT findings actually decrease, external_locations stays finding-free), no new problems, deterministic. Regenerated to match the new structure. Co-authored-by: Isaac
…cks#6654) Follow-up to databricks#6525 after databricks#6618 enabled this. Added `catalog.ModelService` to apitypes.yml, removed the override from resources.yml, and generated the rest
) ## What Allow `get_field_behaviors` to exhaustively extract from specs. This discovers `OUTPUT_ONLY` / `INPUT_ONLY` field behaviours. ## Why Follow-up to the discussion on #6525 (#6525 (comment)): the `model_services` resource can't rely on the generator to suppress its output-only config fields, because the destination-config type reused under `config.routing.fallback.destinations[*]` is deduped away. That is a general generator bug, not model_services-specific — it already drops correct suppressions on the three resources above. Fixing it here lets a later `apitypes.yml` pin (`model_services: catalog.ModelService`, riding with #6525) generate the full set instead of hand-classifying it. ## Explanation (There's also inline comments on `resources.generated.yml`) `get_field_behaviors` in `bundle/direct/tools/generate_resources.py` threaded a single global `visited` set through the whole schema walk, deduping by type name across the entire tree. A type reached at two sibling paths was therefore expanded only at the first one, and its nested `OUTPUT_ONLY` / `INPUT_ONLY` annotations were dropped at every later path — surfacing later as phantom drift on those server-populated fields. This scopes `visited` to the current root→node path (each branch gets its own copy via `visited | {ref}`) so a self-referential type still terminates the recursion, but a type reused at sibling paths is expanded at each occurrence. `max_depth` stays as the runaway guard. ## Tests - `go test ./bundle/direct/dresources/` passes, including `TestResourcesYMLNoRedundantRules` (the recovered entries don't collide with hand-written `resources.yml` rules). - Regeneration is idempotent; the script is ruff-clean and formatted with the pinned `ruff@0.15.17`. - No acceptance test references the recovered fields, so there is no golden-output churn. - Simulated #6525 and the `model_services` annotations are picked up This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.17.0 ### Notable Changes * Bump the direct deployment state version to 3. Clients older than v1.8.0 will reject bundles deployed with this release. ([#6713](#6713)) ### CLI * Add an `INVALID_REFRESH_TOKEN` error code to `databricks auth token --output json` failures. ([#6684](#6684)) * Add experimental `databricks auth docker configure` to configure Docker credential helper access for Databricks Artifact Registry. ([#6700](#6700)) * Add experimental `databricks auth docker token` to generate Docker credentials for Databricks Artifact Registry. ([#6699](#6699)) * `databricks environments setup-local` now reports the `E_PROVISION_CONFLICT` error code instead of the generic `E_PROVISION` when `uv sync` fails to resolve a dependency conflict. ([#6666](#6666)) * Preserve SSH sessions across temporary tunnel disconnects, with bounded replay and backpressure for large transfers. ([#6650](#6650)) * Allow OAuth U2M logins to override the CLI client ID with `--client-id`, profile `client_id`, or `DATABRICKS_CLIENT_ID`. ([#6594](#6594)) ### Bundles * direct: Store a dashboard's `serialized_dashboard` in state as a content hash instead of its full contents. ([#6105](#6105)) * direct: Fix pipelines recreation when the whole `ingestion_definition` block is added or removed. ([#6589](#6589)) * `bundle plan`, `deploy`, and `destroy` no longer report removing `permissions`, `grants`, or secret scope ACLs from a bundle as a deletion, since it leaves the resource untouched. ([#6647](#6647)) * `bundle plan` and `deploy` no longer list or count a resource that was already deleted remotely as a deletion, matching `bundle destroy`; applying still cleans up its stale state entry. ([#6675](#6675)) * Fix `bundle run` failing with `expected an int, found a string` when an unrelated resource references another resource that is not deployed. `bundle run` now resolves `${resources.*}` references only within the resource being run. ([#6690](#6690)) * Add grants support for the AI Gateway `model_service`, `mcp_service`, and `model_provider_service` resources (direct engine). ([#6635](#6635)) * Add bundle support for the AI Gateway `mcp_service` resource (direct engine). ([#6633](#6633)) * Add bundle support for the AI Gateway `model_provider_service` resource (direct engine). ([#6634](#6634)) * Add bundle support for the AI Gateway `model_service` resource (direct engine). ([#6525](#6525)) * Prevent resource drift on catalogs if `storage_root` contained a trailing slash in the URL. ([#6622](#6622)) * Fixed a "lineage mismatch in state files" error that could occur after destroying a bundle and redeploying it from another machine. `bundle destroy` now removes the local state file so no stale lineage is left behind, and prunes the state directories it leaves empty (such as `.internal/` and `sync-snapshots/`). ([#6210](#6210), [#6685](#6685)) * direct: `bundle plan` no longer reports a permanent update on a cluster that uses a cluster policy: when the cluster spec sets `policy_id`, a field present in the remote but absent from the bundle config is not treated as drift. ([#6531](#6531)) * `bundle deploy` on the direct engine now reports each resource as soon as it is deployed, instead of listing them all after the deployment finishes. A deploy that fails part way through now reports the resources it did apply. ([#6361](#6361)) * Direct-engine bundles no longer flag phantom drift on server-populated nested fields under reused config types (e.g. `external_locations` file-event-queue resource IDs, `database_instances` parent-instance refs, `apps` git credential ID). ([#6618](#6618)) * `databricks bundle generate app` now reproduces a git-backed app's `git_repository` and `git_source` configuration instead of emitting a workspace `source_code_path`, so generating from a Git-deployed app no longer silently converts it to workspace source. ([#6656](#6656)) * Improved configuration load time for bundles with many included files. ([#6195](#6195)) * `bundle destroy` no longer deletes triggered job runs, leaving them untouched on the backend. ([#6672](#6672)) * direct: resources.job\_runs: new lifecycle.triggers.on\_file\_change setting to restart the run when monitored files change. Can be set to a series of paths or globs. ([#6309](#6309)) * Bundle summary now shows a name for Postgres branches, endpoints, databases, and roles instead of a blank Name field. ([#6663](#6663)) * Added PyDABs (Python) support for cluster policies, dashboards, and Genie spaces. ([#6585](#6585)) * CLI commands no longer imply that a resource whose type has no workspace URL is merely not deployed yet. ([#6583](#6583)) * Capture the implicit dependency a vector search index has on a catalog or schema defined in the same bundle, so the catalog and schema are deployed first. ([#6655](#6655)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6695](#6695)) * Bump `github.com/databricks/databricks-sdk-go` from v0.177.0 to v0.178.0. ([#6673](#6673)) * Bump Terraform provider from v1.131.0 to v1.132.0. ([#6671](#6671))
What
Allow
get_field_behaviorsto exhaustively extract from specs.This discovers
OUTPUT_ONLY/INPUT_ONLYfield behaviours.Why
Follow-up to the discussion on #6525
(#6525 (comment)): the
model_servicesresource can't rely on the generator to suppress itsoutput-only config fields, because the destination-config type reused under
config.routing.fallback.destinations[*]is deduped away. That is a generalgenerator bug, not model_services-specific — it already drops correct
suppressions on the three resources above. Fixing it here lets a later
apitypes.ymlpin (model_services: catalog.ModelService, riding with #6525)generate the full set instead of hand-classifying it.
Explanation
(There's also inline comments on
resources.generated.yml)get_field_behaviorsinbundle/direct/tools/generate_resources.pythreaded asingle global
visitedset through the whole schema walk, deduping by type nameacross the entire tree. A type reached at two sibling paths was therefore
expanded only at the first one, and its nested
OUTPUT_ONLY/INPUT_ONLYannotations were dropped at every later path — surfacing later as phantom drift
on those server-populated fields.
This scopes
visitedto the current root→node path (each branch gets its owncopy via
visited | {ref}) so a self-referential type still terminates therecursion, but a type reused at sibling paths is expanded at each occurrence.
max_depthstays as the runaway guard.Tests
go test ./bundle/direct/dresources/passes, includingTestResourcesYMLNoRedundantRules(the recovered entries don't collide withhand-written
resources.ymlrules).pinned
ruff@0.15.17.golden-output churn.
model_servicesannotations are picked upThis pull request and its description were written by Isaac.