Databricks Unity Catalog Metric View converter

.gitignore

@@ -1,2 +1,22 @@
 **/__pycache__/
 **/.venv/
+*.egg-info/
+dist/
+build/
+
+# Hypothesis property-based testing database
+**/.hypothesis/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Test / coverage artifacts
+.coverage
+htmlcov/
+.pytest_cache/

converters/databricks/README.md

@@ -0,0 +1,139 @@
+# osi-databricks
+
+Bidirectional converter between [Databricks Unity Catalog Metric View YAML](https://docs.databricks.com/aws/metric-views/yaml-ref) and the [Open Semantic Interchange (OSI)](../../core-spec/spec.md) format.
+
+Part of the OSI hub-and-spoke converter architecture — see the [converter guide](../index.md) for background.
+
+## Installation
+
+**With uv (recommended):**
+
+```bash
+cd converters/databricks
+uv sync
+```
+
+**With pip:**
+
+```bash
+cd converters/databricks
+pip install -e .
+```
+
+### Dependencies
+
+- Python ≥ 3.11
+- `osi-python >= 0.2.0.dev0`
+- `pydantic >= 2.0`
+- `PyYAML >= 6.0`
+
+## Usage
+
+The package provides a CLI entry point `osi-databricks` with two subcommands.
+
+### Import: Metric View → OSI
+
+Convert a Databricks Metric View YAML file to OSI format:
+
+```bash
+osi-databricks import -i metric_view.yaml -o output_osi.yaml
+
+# Specify a custom model name
+osi-databricks import -i metric_view.yaml -o output_osi.yaml --model-name my_model
+```
+
+### Export: OSI → Metric View
+
+Convert an OSI YAML file to one or more Metric View YAML files (one per dataset):
+
+```bash
+osi-databricks export -i osi_model.yaml -o ./output_dir/
+```
+
+Output files are named `{dataset_name}.yaml`. The output directory is created automatically if it doesn't exist.
+
+### Programmatic Usage
+
+```python
+from osi_databricks.models import MetricViewModel
+from osi_databricks.metric_view_to_osi import metric_view_to_osi
+from osi_databricks.osi_to_metric_view import osi_to_metric_view
+
+# Import
+mv = MetricViewModel.from_yaml(open("metric_view.yaml").read())
+osi_doc = metric_view_to_osi(mv, model_name="my_model")
+print(osi_doc.to_osi_yaml())
+
+# Export
+from osi import OSIDocument
+import yaml
+
+raw = yaml.safe_load(open("osi_model.yaml").read())
+doc = OSIDocument.model_validate(raw)
+results = osi_to_metric_view(doc)
+for name, model in results:
+    open(f"{name}.yaml", "w").write(model.to_yaml())
+```
+
+## Mapping Reference
+
+### Core Constructs
+
+| Metric View YAML | OSI Equivalent | Direction | Notes |
+|---|---|---|---|
+| `source` (three-part name) | `dataset.source` | ↔ | Direct pass-through |
+| `source` (SQL query) | `custom_extension` (DATABRICKS, `source_query`) | ↔ | Detected by SQL keywords |
+| `fields[].name` | `field.name` | ↔ | Direct |
+| `fields[].expr` | `field.expression.dialects[DATABRICKS]` | ↔ | Always stored as DATABRICKS dialect |
+| `measures[].name` | `metric.name` | ↔ | Direct |
+| `measures[].expr` | `metric.expression.dialects[DATABRICKS]` | ↔ | Same dialect logic as fields |
+| `joins[].name` | `relationship.name` | ↔ | Direct |
+| `joins[].source` | `relationship.to` | ↔ | Dataset name extracted from source |
+| `joins[].on` | `relationship.from_columns` / `to_columns` | ↔ | Parsed from `a.col = b.col` pattern |
+| `joins[].using` | `from_columns = to_columns = using` | → OSI | Columns same on both sides |
+| `comment` (top-level) | `semantic_model.description` | ↔ | Direct |
+
+### Metadata Mapping
+
+| Metric View YAML | OSI Equivalent | Direction | Notes |
+|---|---|---|---|
+| `fields[].comment` | `field.description` | ↔ | Direct |
+| `fields[].display_name` | `field.ai_context.synonyms[0]` | ↔ | First synonym = display_name |
+| `fields[].synonyms` | `field.ai_context.synonyms[1:]` | ↔ | Remaining synonyms |
+| `fields[].format` | `field.custom_extension` (DATABRICKS) | ↔ | No OSI core equivalent |
+| `measures[].window` | `metric.custom_extension` (DATABRICKS) | ↔ | No OSI core equivalent |
+| `filter` | `dataset.custom_extension` (DATABRICKS) | ↔ | No OSI core equivalent |
+| `materialization` | `semantic_model.custom_extension` (DATABRICKS) | ↔ | No OSI core equivalent |
+
+### Dialect Handling
+
+On **import** (MV → OSI):
+- Expressions are stored as `DATABRICKS` dialect
+- If the expression uses only standard SQL (no `FILTER()`, `MEASURE()`, `QUALIFY`, `::`), an `ANSI_SQL` entry is also generated
+
+On **export** (OSI → MV):
+- Prefers `DATABRICKS` dialect expression
+- Falls back to `ANSI_SQL` if `DATABRICKS` is unavailable
+- Skips the field/metric with a warning if neither dialect is present
+
+## Development
+
+```bash
+cd converters/databricks
+uv sync
+
+# Run tests
+uv run pytest
+
+# Run linter
+uv run ruff check src/ tests/
+```
+
+## Known Limitations
+
+- **Time dimension inference is heuristic.** The importer infers `is_time` from expression content (looking for DATE, TIME, TIMESTAMP, etc.). This may produce false positives or miss custom time expressions.
+- **ON clause parsing is pattern-based.** Only `alias.column = alias.column` patterns are parsed. Complex ON conditions (functions, nested expressions) are stored in a custom extension with a warning.
+- **Nested joins (snowflake schema) are flattened.** Metric View supports nested join definitions; during import these are flattened into a list of OSI relationships. The nesting structure is not preserved during round-trip.
+- **USING clause becomes ON on round-trip.** Joins defined with `USING` are converted to explicit column pairs in OSI. On export, they are reconstructed as `ON` clauses rather than `USING`.
+- **No model-level description in Metric View.** The top-level `comment` is the closest equivalent. Model-level `ai_context.instructions` is stored in a custom extension.
+- **Custom extensions for other vendors are preserved but not applied.** Extensions for SNOWFLAKE, DBT, etc. pass through without modification.

converters/databricks/pyproject.toml

@@ -0,0 +1,47 @@
+[project]
+name = "osi-databricks"
+version = "0.2.0.dev0"
+description = "Bidirectional converter between Databricks Unity Catalog Metric View YAML and OSI semantic model"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "osi-python>=0.2.0.dev0",
+    "pydantic>=2.0",
+    "pyyaml>=6.0",
+]
+
+[project.scripts]
+osi-databricks = "osi_databricks.cli:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "hypothesis>=6.0",
+    "ruff>=0.11.5",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build]
+include = ["src/osi_databricks/**/*"]
+ignore-vcs = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/osi_databricks"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "W", "C4", "PIE", "SIM"]
+ignore = ["E501"]
+
+[tool.uv.sources]
+osi-python = { path = "../../python", editable = true }
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

converters/databricks/src/osi_databricks/__init__.py

@@ -0,0 +1 @@
+"""osi-databricks: Bidirectional converter between Databricks Metric View YAML and OSI."""

converters/databricks/src/osi_databricks/cli.py

@@ -0,0 +1,87 @@
+"""CLI entry point for the osi-databricks converter.
+
+Usage:
+    osi-databricks import -i metric_view.yaml -o output_osi.yaml
+    osi-databricks export -i osi_model.yaml -o ./output_dir/
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+import yaml
+from osi import OSIDocument
+
+from osi_databricks.metric_view_to_osi import metric_view_to_osi
+from osi_databricks.models import MetricViewModel
+from osi_databricks.osi_to_metric_view import osi_to_metric_view
+
+
+def _cmd_import(args: argparse.Namespace) -> None:
+    """Import: Metric View YAML → OSI YAML."""
+    input_path = Path(args.input)
+    output_path = Path(args.output)
+
+    try:
+        mv_model = MetricViewModel.from_yaml(input_path.read_text())
+    except Exception as e:
+        print(f"Error parsing {input_path}: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    osi_doc = metric_view_to_osi(mv_model, model_name=args.model_name)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(osi_doc.to_osi_yaml())
+    print(f"Written to {output_path}", file=sys.stderr)
+
+
+def _cmd_export(args: argparse.Namespace) -> None:
+    """Export: OSI YAML → Metric View YAML file(s)."""
+    input_path = Path(args.input)
+    output_dir = Path(args.output)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    try:
+        raw = yaml.safe_load(input_path.read_text())
+        document = OSIDocument.model_validate(raw)
+    except Exception as e:
+        print(f"Error parsing {input_path}: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    results = osi_to_metric_view(document)
+    for dataset_name, mv_model in results:
+        out_file = output_dir / f"{dataset_name}.yaml"
+        out_file.write_text(mv_model.to_yaml())
+        print(f"Written {out_file}", file=sys.stderr)
+
+
+def main() -> None:
+    """Entry point for the osi-databricks CLI."""
+    parser = argparse.ArgumentParser(
+        prog="osi-databricks",
+        description="Convert between Databricks Metric View YAML and OSI YAML.",
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    # Import subcommand
+    import_cmd = subparsers.add_parser("import", help="Convert Metric View YAML → OSI YAML")
+    import_cmd.add_argument("-i", "--input", required=True, metavar="FILE", help="Path to Metric View YAML")
+    import_cmd.add_argument("-o", "--output", required=True, metavar="FILE", help="Path for output OSI YAML")
+    import_cmd.add_argument(
+        "--model-name", default="metric_view_model", metavar="NAME",
+        help="OSI semantic model name (default: metric_view_model)",
+    )
+
+    # Export subcommand
+    export_cmd = subparsers.add_parser("export", help="Convert OSI YAML → Metric View YAML")
+    export_cmd.add_argument("-i", "--input", required=True, metavar="FILE", help="Path to OSI YAML")
+    export_cmd.add_argument("-o", "--output", required=True, metavar="DIR", help="Output directory for Metric View files")
+
+    args = parser.parse_args()
+    if args.command == "import":
+        _cmd_import(args)
+    elif args.command == "export":
+        _cmd_export(args)
+
+
+if __name__ == "__main__":
+    main()

converters/databricks/src/osi_databricks/dialect_utils.py

@@ -0,0 +1,59 @@
+"""Dialect selection and ANSI SQL detection utilities.
+
+Provides shared logic for both import (Metric View → OSI) and export (OSI → Metric View)
+directions, including detecting Databricks-specific SQL patterns and selecting the
+best available dialect expression.
+"""
+
+from __future__ import annotations
+
+import re
+
+from osi.models import OSIDialect, OSIDialectExpression
+
+# Databricks-specific SQL patterns not found in ANSI SQL
+_DATABRICKS_ONLY_PATTERNS = [
+    r"\bFILTER\s*\(",
+    r"\bMEASURE\s*\(",
+    r"\bQUALIFY\b",
+    r"::",  # Type casting syntax
+]
+
+
+def is_standard_sql(expr: str) -> bool:
+    """Determine if an expression uses only ANSI-standard SQL syntax.
+
+    Args:
+        expr: SQL expression string to check.
+
+    Returns:
+        True if the expression appears to use only standard SQL constructs.
+    """
+    for pattern in _DATABRICKS_ONLY_PATTERNS:
+        if re.search(pattern, expr, re.IGNORECASE):
+            return False
+    return True
+
+
+def select_dialect_expression(
+    dialects: list[OSIDialectExpression],
+    preferred: OSIDialect = OSIDialect.DATABRICKS,
+    fallback: OSIDialect = OSIDialect.ANSI_SQL,
+) -> str | None:
+    """Select expression string with dialect preference chain.
+
+    Args:
+        dialects: List of dialect expressions to choose from.
+        preferred: First-choice dialect (default: DATABRICKS).
+        fallback: Second-choice dialect (default: ANSI_SQL).
+
+    Returns:
+        The expression string for the best available dialect, or None if
+        neither preferred nor fallback dialect is available.
+    """
+    by_dialect = {d.dialect: d.expression for d in dialects}
+    if preferred in by_dialect:
+        return by_dialect[preferred]
+    if fallback in by_dialect:
+        return by_dialect[fallback]
+    return None