[ENH]: Rust sysdb schema migration service by tanujnay112 · Pull Request #6069 · chroma-core/chroma

tanujnay112 · 2025-12-19T07:17:49Z

Description of changes

Summarize the changes made by this PR.

Improvements & Bug fixes
- Create a rust-sysdb-migration service that bootstraps the Spanner emulator and runs a stub migration service binary.
- Moved spanner emulator configuration to a common crate.
New functionality
- ...

Test plan

How are these changes tested?

Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the _docs section?_

tanujnay112 · 2025-12-19T07:18:06Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-12-19T07:18:21Z

github-actions · 2025-12-19T07:23:35Z

⚠️ The Helm chart was updated without a version bump. Your changes will only be published if the version field in k8s/distributed-chroma/Chart.yaml is updated.

propel-code-bot · 2025-12-19T22:11:38Z

Add rust-based Spanner migration service and centralize emulator config

Introduces a Rust spanner-migrations binary and Kubernetes job that bootstraps the Spanner emulator before schema migrations, while refactoring emulator configuration into the shared chroma-config crate. Updates the Rust workspace, Docker build, and Tilt/Helm assets so the new migration job is built, published, and supplied with configuration separate from the runtime sysdb service.
This release also removes inline emulator bootstrap logic from rust-sysdb, aligns both services on the shared config structures, and promotes the Rust toolchain to 1.88 for the unified builder image.

Key Changes

• Added rust/rust-sysdb/spanner-migrations crate with bootstrap_emulator gRPC admin flow and configuration loader for the migration job
• Moved SpannerConfig and SpannerEmulatorConfig into rust/config/src/spanner.rs and re-exported them from rust-sysdb/src/config.rs
• Updated rust/rust-sysdb/src/spanner.rs to rely on pre-created emulator resources and removed REST bootstrap logic
• Extended rust/Dockerfile (Rust 1.88) to build and package the spanner_migration binary and added a new runtime stage
• Provisioned Helm/Tilt plumbing (k8s/distributed-chroma/templates/rust-sysdb-migration.yaml, Tiltfile, .github/actions/tilt-setup-prebuild/docker-bake.hcl, k8s/distributed-chroma/values.yaml) to deploy the migration job with optional ConfigMap-based configuration

Affected Areas

• rust/rust-sysdb/spanner-migrations/src
• rust/config/src/spanner.rs
• rust/rust-sysdb/src/config.rs
• rust/rust-sysdb/src/spanner.rs
• rust/Dockerfile
• Tiltfile
• k8s/distributed-chroma/templates/rust-sysdb-migration.yaml
• k8s/distributed-chroma/values.yaml
• Cargo workspace metadata

This summary was automatically generated by @propel-code-bot

sanketkedia · 2025-12-20T20:30:25Z

+metadata:
+  name: rust-sysdb-migration-{{ .Values.rustSysdbMigration.image.tag }}
+  namespace: {{ .Values.namespace }}
+  annotations:


do you need annotations?

keeping it because sysdb-migration service has it

right, but why does it need it was my question

sanketkedia · 2025-12-20T20:31:16Z

+        - name: config
+          configMap:
+            name: rust-sysdb-migration-config
+      {{if .Values.rustSysdbMigration.tolerations}}


does it need tolerations and node selector?

keeping since sysdb-migration has

reading the description, it doesn't seem like it's applicable here. would prefer to clean this up in a future change along with sysdb-migration service.

sanketkedia · 2025-12-20T20:34:50Z

@@ -0,0 +1,149 @@
+//! Shared Spanner configuration types.


thoughts on moving this to the chroma-config crate instead of creating one here? In general, I was thinking of moving all the mcmr common config (like topology, regions, etc) into this crate and reuse it across services.

Even the bootstrap function?

why does the bootstrap function need to be shared? shouldn't only the rust-sysdb-migration perform the bootstrap? and rust-sysdb assume that all the entities exist in the db?

sanketkedia · 2025-12-20T20:36:46Z

+  image:
+    repository: 'rust-sysdb-migration'
+    tag: 'latest'
+  project: chroma-dev


why not use the rust/worker/chroma_config.yaml directly. We inject config using that and not helm values. It'd be hard to maintain this with that

create a rust_sysdb_migration section and put it under there. Next week, I'll centralize all of this under mcmr_config section

sanketkedia · 2025-12-20T20:39:11Z

@@ -0,0 +1,10 @@
+[package]


see above. would prefer it to be a part of chroma-config crate

propel-code-bot · 2025-12-21T05:42:08Z

+{{- if .Values.rustSysdbMigration.configuration }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: rust-sysdb-migration-config
+  namespace: {{ .Values.namespace }}
+data:
+  config.yaml: |
+{{ .Values.rustSysdbMigration.configuration | indent 4 }}
+---
+{{- end }}
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: rust-sysdb-migration-{{ .Values.rustSysdbMigration.image.tag }}
+  namespace: {{ .Values.namespace }}
+spec:
+  template:
+    metadata:
+      labels:
+        app: rust-sysdb-migration
+    spec:
+      restartPolicy: OnFailure
+      containers:
+        - name: rust-sysdb-migration
+          image: "{{ .Values.rustSysdbMigration.image.repository }}:{{ .Values.rustSysdbMigration.image.tag }}"
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: CONFIG_PATH
+              value: "/config/config.yaml"
+          volumeMounts:
+            {{- if .Values.rustSysdbMigration.configuration }}
+            - name: config
+              mountPath: /config/
+            {{- end }}
+      {{- if .Values.rustSysdbMigration.configuration }}
+      volumes:
+        - name: config
+          configMap:
+            name: rust-sysdb-migration-config
+      {{- end }}
+      {{- if .Values.rustSysdbMigration.tolerations }}
+      tolerations:
+        {{- toYaml .Values.rustSysdbMigration.tolerations | nindent 8 }}
+      {{- end }}
+      {{- if .Values.rustSysdbMigration.nodeSelector }}
+      nodeSelector:
+        {{- toYaml .Values.rustSysdbMigration.nodeSelector | nindent 8 }}
+      {{- end }}
+---


[Logic] The current Helm template creates the Job resource unconditionally, while the ConfigMap it depends on is only created if .Values.rustSysdbMigration.configuration is provided. This will cause the job to fail at runtime if no configuration is supplied, as the path specified in CONFIG_PATH will not exist.

To prevent deploying a non-functional job, it's better to make the creation of all related resources conditional on the presence of the configuration. Wrapping the entire template in a conditional block ensures that both the ConfigMap and the Job are only created when the necessary configuration is available. This also allows for the removal of redundant inner conditional checks for volumes and volume mounts.

Context for Agents

The current Helm template creates the `Job` resource unconditionally, while the `ConfigMap` it depends on is only created if `.Values.rustSysdbMigration.configuration` is provided. This will cause the job to fail at runtime if no configuration is supplied, as the path specified in `CONFIG_PATH` will not exist. To prevent deploying a non-functional job, it's better to make the creation of all related resources conditional on the presence of the configuration. Wrapping the entire template in a conditional block ensures that both the `ConfigMap` and the `Job` are only created when the necessary configuration is available. This also allows for the removal of redundant inner conditional checks for volumes and volume mounts. File: k8s/distributed-chroma/templates/rust-sysdb-migration.yaml Line: 50

sanketkedia · 2025-12-21T07:15:31Z

      instance: "test-instance"
-      database: "local-database"
+      database: "local-database"
+rust-sysdb-migration:


this is not needed right? Since the schema service is only in chroma2 namespace

Not needed yes, removing.

sanketkedia · 2025-12-21T07:16:14Z

 k8s_yaml(
  local(
-    'helm template --set-file rustFrontendService.configuration=' + rfe_config_file + ',rustLogService.configuration=rust/worker/chroma_config.yaml,heapTenderService.configuration=rust/worker/chroma_config.yaml,compactionService.configuration=rust/worker/chroma_config.yaml,queryService.configuration=rust/worker/chroma_config.yaml,garbageCollector.configuration=rust/worker/chroma_config.yaml,rustSysdbService.configuration=rust/worker/chroma_config.yaml --values ' + distributed_chroma_values + ' k8s/distributed-chroma'
+    'helm template --set-file rustFrontendService.configuration=' + rfe_config_file + ',rustLogService.configuration=rust/worker/chroma_config.yaml,heapTenderService.configuration=rust/worker/chroma_config.yaml,compactionService.configuration=rust/worker/chroma_config.yaml,queryService.configuration=rust/worker/chroma_config.yaml,garbageCollector.configuration=rust/worker/chroma_config.yaml,rustSysdbService.configuration=rust/worker/chroma_config.yaml,rustSysdbMigration.configuration=rust/worker/chroma_config.yaml --values ' + distributed_chroma_values + ' k8s/distributed-chroma'


this is not needed?

it populates

.Values.rustSysdbMigration.configuration

in rust-sysdb-migration.yaml

sanketkedia · 2025-12-21T07:18:44Z

+use std::env;
+
+const CONFIG_PATH_ENV_VAR: &str = "CONFIG_PATH";
+const DEFAULT_CONFIG_PATH: &str = "./chroma_config.yaml";


If this is in chroma2 namespace then should this be chroma_config2.yaml?

sanketkedia · 2025-12-21T07:19:42Z

+    );
+
+    // Set environment variable for clients
+    std::env::set_var("SPANNER_EMULATOR_HOST", emulator.grpc_endpoint());


Is this needed? since you set the config in AdminClientConfig below, I am guessing this is not needed

sanketkedia · 2025-12-21T08:41:30Z

 k8s_yaml(
  local(
-    'helm template --set-file rustFrontendService.configuration=' + rfe_config_file + ',rustLogService.configuration=rust/worker/chroma_config.yaml,heapTenderService.configuration=rust/worker/chroma_config.yaml,compactionService.configuration=rust/worker/chroma_config.yaml,queryService.configuration=rust/worker/chroma_config.yaml,garbageCollector.configuration=rust/worker/chroma_config.yaml,rustSysdbService.configuration=rust/worker/chroma_config.yaml --values ' + distributed_chroma_values + ' k8s/distributed-chroma'
+    'helm template --set-file rustFrontendService.configuration=' + rfe_config_file + ',rustLogService.configuration=rust/worker/chroma_config.yaml,heapTenderService.configuration=rust/worker/chroma_config.yaml,compactionService.configuration=rust/worker/chroma_config.yaml,queryService.configuration=rust/worker/chroma_config.yaml,garbageCollector.configuration=rust/worker/chroma_config.yaml,rustSysdbService.configuration=rust/worker/chroma_config.yaml,rustSysdbMigration.configuration=rust/worker/chroma_config2.yaml --values ' + distributed_chroma_values + ' k8s/distributed-chroma'


nit: I think this change is not needed

it populates

.Values.rustSysdbMigration.configuration

in rust-sysdb-migration.yaml

the service panics without this change as it can't find the configuration path

sanketkedia · 2025-12-21T08:46:09Z

@@ -0,0 +1,50 @@
+{{- if .Values.rustSysdbMigration.configuration }}


ok to do in another PR but would encourage you to think about how to feature flag bringing up the migration service in staging/prod since it would fail currently due to no real spanner present. One idea that I've done for the rust sysdb service is disable this entire manifest conditioned on a value that you set to true in values.dev.yaml and false in values.yaml. i.e. at the top of this file add:

{{- if .Values.rustSysdbMigration.enabled }}

and at the end of this file add:

{{-end }}

And then in values.dev.yaml set
rust-sysdb-migration:
enabled: true

In values.yaml set:
rust-sysdb-migration:
enabled: false

tanujnay112 force-pushed the spanner-changes branch 4 times, most recently from 8a541ce to 8acc93a Compare December 19, 2025 08:04

tanujnay112 mentioned this pull request Dec 19, 2025

[ENH]: Implement spanner migration runner #6070

Merged

1 task

tanujnay112 changed the title ~~[ENH]: Spanner schema migration~~ [ENH]: Rust sysdb schema migration service Dec 19, 2025

tanujnay112 force-pushed the spanner-changes branch from 8acc93a to fa5f0b2 Compare December 19, 2025 11:01

tanujnay112 mentioned this pull request Dec 19, 2025

[ENH]: Checksum manifest for spanner migrations #6071

Merged

1 task

tanujnay112 marked this pull request as ready for review December 19, 2025 22:10

tanujnay112 requested a review from sanketkedia December 19, 2025 22:13

tanujnay112 commented Dec 19, 2025

View reviewed changes

Comment thread Tiltfile Outdated

tanujnay112 commented Dec 19, 2025

View reviewed changes

Comment thread go/Dockerfile.rust-migration Outdated

tanujnay112 force-pushed the spanner-changes branch from fa5f0b2 to 53f2678 Compare December 20, 2025 01:14

propel-code-bot Bot reviewed Dec 20, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-config/src/lib.rs Outdated

tanujnay112 force-pushed the spanner-changes branch 2 times, most recently from 5316f50 to 335646c Compare December 20, 2025 02:02

propel-code-bot Bot reviewed Dec 20, 2025

View reviewed changes

Comment thread .github/actions/tilt-setup-prebuild/docker-bake.hcl Outdated

tanujnay112 force-pushed the spanner-changes branch from 335646c to 60603b1 Compare December 20, 2025 02:16

tanujnay112 requested a review from rescrv December 20, 2025 02:17

propel-code-bot Bot reviewed Dec 20, 2025

View reviewed changes

Comment thread k8s/distributed-chroma/templates/rust-sysdb-migration.yaml Outdated

sanketkedia reviewed Dec 20, 2025

View reviewed changes

Comment thread .github/actions/tilt-setup-prebuild/docker-bake.hcl

sanketkedia reviewed Dec 20, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-config/src/lib.rs Outdated

sanketkedia reviewed Dec 20, 2025

View reviewed changes

tanujnay112 force-pushed the spanner-changes branch 2 times, most recently from ac91667 to 2bac7cf Compare December 20, 2025 22:09

propel-code-bot Bot reviewed Dec 20, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-migrations/src/config.rs Outdated

propel-code-bot Bot reviewed Dec 20, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-config/src/lib.rs Outdated

tanujnay112 force-pushed the spanner-changes branch from 2bac7cf to 77190b0 Compare December 21, 2025 02:07

propel-code-bot Bot reviewed Dec 21, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-migrations/src/config.rs Outdated

propel-code-bot Bot reviewed Dec 21, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-migrations/src/main.rs Outdated

tanujnay112 force-pushed the spanner-changes branch from 77190b0 to 9f65a8f Compare December 21, 2025 05:20

propel-code-bot Bot reviewed Dec 21, 2025

View reviewed changes

Comment thread rust/rust-sysdb/spanner-migrations/src/main.rs

tanujnay112 force-pushed the spanner-changes branch from 9f65a8f to 752979b Compare December 21, 2025 05:40

propel-code-bot Bot reviewed Dec 21, 2025

View reviewed changes

tanujnay112 force-pushed the spanner-changes branch 4 times, most recently from 0a8304b to 76fe609 Compare December 21, 2025 06:16

sanketkedia reviewed Dec 21, 2025

View reviewed changes

Comment thread k8s/distributed-chroma/templates/rust-sysdb-migration.yaml

sanketkedia reviewed Dec 21, 2025

View reviewed changes

tanujnay112 force-pushed the spanner-changes branch from 76fe609 to 3a3b482 Compare December 21, 2025 08:04

[ENH]: Spanner schema migration

d3b78dd

tanujnay112 force-pushed the spanner-changes branch from 3a3b482 to d3b78dd Compare December 21, 2025 08:06

tanujnay112 requested a review from sanketkedia December 21, 2025 08:18

sanketkedia reviewed Dec 21, 2025

View reviewed changes

sanketkedia approved these changes Dec 21, 2025

View reviewed changes

tanujnay112 merged commit 3e2be15 into main Dec 21, 2025
64 checks passed

tanujnay112 mentioned this pull request Dec 22, 2025

[ENH]: Feature flag rust-sysdb-migration service #6074

Merged

1 task

		@@ -0,0 +1,50 @@
		{{- if .Values.rustSysdbMigration.configuration }}

Conversation

tanujnay112 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Test plan

Migration plan

Observability plan

Documentation Changes

Uh oh!

tanujnay112 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Dec 19, 2025

Reviewer Checklist

Testing, Bugs, Errors, Logs, Documentation

System Compatibility

Quality

Uh oh!

github-actions Bot commented Dec 19, 2025

Uh oh!

propel-code-bot Bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanketkedia Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

propel-code-bot Bot Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tanujnay112 commented Dec 19, 2025 •

edited

Loading

tanujnay112 commented Dec 19, 2025 •

edited

Loading

propel-code-bot Bot commented Dec 19, 2025 •

edited

Loading

sanketkedia Dec 21, 2025 •

edited

Loading

sanketkedia Dec 21, 2025 •

edited

Loading