Lakebase DX: a native-search helper, grants bootstrap, and graduating proven cookbook patterns

[saselvan]

Field feedback from building an app on @databricks/appkit 0.66.1 + Lakebase. First, credit where it's due — these saved real work: the Lakebase plugin's auto OAuth refresh, asUser(req) OBO per-user pools, and exposeAsAgentTool (read-only SQL classifier). And ai-search is great for Databricks Vector Search.

The gaps below are all Lakebase-specific. Each is verified against 0.66.1; happy to contribute PRs.

1. A lakebaseSearch() helper (Lakebase-native BM25 + vector ANN)

ai-search targets Vector Search (managed indexes), not Lakebase's native lakebase_bm25 + lakebase_ann (halfvec) indexes. There's no helper for those, so devs write raw SQL and hit avoidable traps:

• the bm25 regclass resolves against search_path, so from an app you must schema-qualify it (to_bm25query(..., 'app.providers_bm25')) or get relation "providers_bm25" does not exist;
• plainto_tsquery, not to_tsquery (the latter crashes on multi-word input);
• halfvec(1024) cosine + ai_query('databricks-gte-large-en', text) embeddings.

Proposed API, mirroring ai-search's shape but pointing at Lakebase indexes:

lakebaseSearch({
  indexes: { providers: { table: "app.providers", bm25Index: "app.providers_bm25",
    annIndex: "app.providers_ann", textColumn: "search_tsv",
    embeddingColumn: "embedding", mode: "hybrid" } }
})
// appkit.lakebaseSearch.query("providers", { text: "home health", numResults: 5 })

(Worth confirming Lakebase Search GA/interface internally before building.)

2. Service-principal grants bootstrap + a clear grant error

The app SP needs databricks_create_role(...) + explicit GRANTs; today that's manual copy-paste, and a missing grant surfaces as a generic permission denied. Proposal: a ensureServicePrincipalGrants(spClientId, { schema, tables }) helper (+ an optional apps init grants.sql scaffold), and map Postgres permission denied → "the app SP lacks GRANT X on Y".

3. Graduate proven Lakebase patterns from the cookbook into AppKit

databricks-solutions/lakebase-cookbook already has battle-tested code for these — the ask is to bring them into the toolkit:

• Migrations: an appkit migrate wrapping the cookbook's checksum/drift migrations_runner.py, made multi-SP-ownership aware (CREATE INDEX IF NOT EXISTS still fails if another SP owns the table).
• Synced-read + writable-ops scaffold: the apps/lakebase-fastapi pattern (read-only synced mirror + writable ops table + derive-state-by-join) as a reusable helper/template.
• Dev-branch selection: a lakebase({ branch }) config so an app targets a dev branch vs production without hand-editing endpoint paths.