add vortex-geo crate and WKB extension type by a10y · Pull Request #7722 · vortex-data/vortex

a10y · 2026-04-29T20:46:09Z

Summary

Part of the implementation of #7686

This PR adds a new crate, vortex-geo, which will hold all extension types, custom layouts, and encodings necessary to store geospatial vector datasets in Vortex files.

The goal of this crate is to enable integration with DuckDB Spatial, GeoDataFusion, SedonaDB, and Iceberg v3 geo types.

This initial PR implements an extension type for the Well-Known Binary encoding (WKB). This encoding is the most common format for geospatial data for analytics, it's what both GeoParquet and DuckDB use to represent geometry types.

We also wire this into vortex DuckDB extension to support converting Geometry columns between Vortex and DuckDB formats.

API Changes

Adds new crate vortex-geo with extension type WellKnownBinary, (geo.wkb)

Adds support for geometry columns to DuckDB as well, e.g. you can now do something like

SELECT building_id, building_name, ST_Area(geometry) AS area 
FROM read_vortex("buildings.vortex")
ORDER BY area DESC
LIMIT 10;

It won't be very performant yet, not until we support better layouts + stats for geometry.

Testing

We have unit tests for metadata serde, round trip conversion between Vortex <> DuckDB geometry format, and an additional E2E test that demonstrates reading a Vortex file with geometry data and providing it to the DuckDB Spatial extension.

codspeed-hq · 2026-04-29T20:52:51Z

Merging this PR will degrade performance by 24.94%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 1167 untouched benchmarks
⏩ 138 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`new_bp_prim_test_between[i64, 32768]`	177.3 µs	236.2 µs	-24.94%
⚡	Simulation	`bitwise_not_vortex_buffer_mut[128]`	275.3 ns	246.1 ns	+11.85%

_{Comparing aduffy/geo-v0 (f499553) with develop (44a6367)}

138 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

a10y · 2026-05-01T16:27:49Z

+duckdb_logical_type duckdb_vx_create_geometry(const char *crs) {
+    D_ASSERT(crs);
+    auto geom =
+        (*crs == '\0') ? duckdb::LogicalType::GEOMETRY() : duckdb::LogicalType::GEOMETRY(std::string(crs));
+    auto copy = duckdb::make_uniq<duckdb::LogicalType>(std::move(geom));
+    return reinterpret_cast<duckdb_logical_type>(copy.release());
+}


These C++ changes are because DuckDB upstream doesn't expose the full geometry type stuff over the C API.

a10y · 2026-05-01T16:28:50Z

+    auto copy = duckdb::make_uniq<duckdb::LogicalType>(std::move(geom));
+    return reinterpret_cast<duckdb_logical_type>(copy.release());


Claude generated this function and the header update, most of the rest of the PR was braincoded

haha braincoded

a10y · 2026-05-01T18:43:11Z

-                let blob = unsafe { cpp::duckdb_get_blob(self.as_ptr()) };
-                let slice =
-                    unsafe { std::slice::from_raw_parts(blob.data.cast::<u8>(), blob.size.as_()) };


in certain situations this could lead to UB depending on the allocator, b/c if duckdb_malloc returns NULL then you'd call slice::from_raw_parts(NULL) which is UB.

see the new take_blob function below

i only fixed this b/c i was going to repeat this for the GEOMETRY branch and figured i'd fix it in both places instead

a10y · 2026-05-01T18:44:20Z

    }

-    pub unsafe fn set_vector_buffer(&self, buffer: &VectorBufferRef) {
+    pub unsafe fn set_vector_buffer(&mut self, buffer: &VectorBufferRef) {


these methods should've always been &mut for soundness reasons, they were just missing before

a10y · 2026-05-01T18:55:29Z

 }
+
+#[test]
+fn test_geometry() {


see this for (rather simple) example of querying geometry data from DuckDB

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

joseph-isaacs · 2026-05-05T09:24:55Z

+    type NativeValue<'a> = Wkb<'a>;
+
+    fn id(&self) -> ExtId {
+        ExtId::new_static("geo.wkb")


lets think about this id a little more.

What namespace do we want?

joseph-isaacs · 2026-05-05T09:26:16Z

+            if ext.ext_dtype().is::<AnyTemporal>() {
+                return temporal::new_exporter(TemporalArray::try_from(ext)?, ctx);
            }
+
+            if ext.ext_dtype().is::<WellKnownBinary>() {
+                return geo::new_wkb_exporter(WellKnownBinaryData::try_from(ext)?, ctx);
+            }
+


shall we move this into a ext mod?

joseph-isaacs · 2026-05-05T09:27:00Z

+        .execute::<Canonical>(ctx)?
+        .into_varbinview();


execute::

Suggested change

.execute::<Canonical>(ctx)?

.into_varbinview();

.execute::<VarBinView>(ctx)?;

joseph-isaacs · 2026-05-05T09:28:41Z


+fn temporal_to_duckdb(temporal: TemporalMetadata) -> VortexResult<LogicalType> {
+    let duckdb_type = match temporal {
+        TemporalMetadata::Timestamp(unit, None) => match unit {
+            TimeUnit::Nanoseconds => DUCKDB_TYPE::DUCKDB_TYPE_TIMESTAMP_NS,
+            TimeUnit::Microseconds => DUCKDB_TYPE::DUCKDB_TYPE_TIMESTAMP,
+            TimeUnit::Milliseconds => DUCKDB_TYPE::DUCKDB_TYPE_TIMESTAMP_MS,
+            TimeUnit::Seconds => DUCKDB_TYPE::DUCKDB_TYPE_TIMESTAMP_S,
+            _ => vortex_bail!("Invalid TimeUnit {} for timestamp", unit),
+        },
+        TemporalMetadata::Timestamp(unit, Some(tz)) => {
+            if tz.as_ref() != "UTC" {
+                vortex_bail!("Invalid timezone for timestamp_tz {tz}, must be UTC");
+            }
+            if unit != &TimeUnit::Microseconds {
+                vortex_bail!(
+                    "Invalid TimeUnit {} for timestamp_tz, must be Microseconds",
+                    unit
+                );
+            }
+            DUCKDB_TYPE::DUCKDB_TYPE_TIMESTAMP_TZ
+        }
+        TemporalMetadata::Date(unit) => match unit {
+            TimeUnit::Days => DUCKDB_TYPE::DUCKDB_TYPE_DATE,
+            _ => vortex_bail!("Invalid TimeUnit {} for date", unit),
+        },
+        TemporalMetadata::Time(unit) => match unit {
+            TimeUnit::Microseconds => DUCKDB_TYPE::DUCKDB_TYPE_TIME,
+            TimeUnit::Nanoseconds => DUCKDB_TYPE::DUCKDB_TYPE_TIME_NS,
+            _ => vortex_bail!("Invalid TimeUnit {} for time", unit),
+        },
+    };
+
+    Ok(LogicalType::new(duckdb_type))
+}
+


shall we also move these into a ext mod?

joseph-isaacs · 2026-05-05T09:30:33Z

+    const auto val = reinterpret_cast<duckdb::Value *>(value);
+    const auto &str = duckdb::StringValue::Get(*val);


Is this a panic or memory unsafe is val is not a string type?

joseph-isaacs · 2026-05-05T09:31:10Z

+///
+/// This bypasses the GEOMETRY -> BLOB default cast (which would require the spatial extension to
+/// be loaded). The returned `data` pointer must be freed with `duckdb_free`. The caller must
+/// ensure `value` is a non-null GEOMETRY value, otherwise behavior is undefined.


memory unsafe??

a10y force-pushed the aduffy/geo-v0 branch 2 times, most recently from a445f7d to 1c99386 Compare April 29, 2026 20:51

a10y added the changelog/feature A new feature label Apr 29, 2026

a10y force-pushed the aduffy/geo-v0 branch 3 times, most recently from 3382e4f to 05dcff9 Compare May 1, 2026 16:02

a10y marked this pull request as ready for review May 1, 2026 16:24

a10y commented May 1, 2026

View reviewed changes

Comment thread vortex-duckdb/src/datasource.rs Outdated

a10y commented May 1, 2026

View reviewed changes

a10y requested review from 0ax1 and joseph-isaacs May 1, 2026 18:51

a10y mentioned this pull request May 1, 2026

Tracking Issue: Geospatial Vector types #7686

Open

5 tasks

a10y commented May 1, 2026

View reviewed changes

a10y added 9 commits May 4, 2026 09:17

add vortex-geo crate and WKB extension type

9da1d71

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

export to DuckDB

260f992

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

fix lockfiles

e064113

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

support duckdb -> vortex WKB

a4dd175

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

fix lockfiles

ba4f867

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

revert change

d5c2e7b

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

fix handling of scalars

a40f548

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

fix comment

b2eff8f

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

formatting

f499553

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/geo-v0 branch from 134b470 to f499553 Compare May 4, 2026 13:17

joseph-isaacs reviewed May 5, 2026

View reviewed changes

a10y mentioned this pull request May 5, 2026

feat[vortex-array]: add ArrowExportPlugin registry for extension types #7796

Closed

		auto copy = duckdb::make_uniq<duckdb::LogicalType>(std::move(geom));
		return reinterpret_cast<duckdb_logical_type>(copy.release());

	.execute::<Canonical>(ctx)?
	.into_varbinview();
	.execute::<VarBinView>(ctx)?;

		const auto val = reinterpret_cast<duckdb::Value *>(value);
		const auto &str = duckdb::StringValue::Get(*val);

Conversation

a10y commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API Changes

Testing

Uh oh!

codspeed-hq Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 24.94%

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

a10y May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

a10y commented Apr 29, 2026 •

edited

Loading

codspeed-hq Bot commented Apr 29, 2026 •

edited

Loading

a10y May 1, 2026 •

edited

Loading