feat(inspect): support selecting columns in metadata table scans by manuzhang · Pull Request #1008 · apache/iceberg-cpp

manuzhang · 2026-10-13T15:42:47Z

What

Add ColumnSelection (column names plus case sensitivity), MetadataTable::ScanColumns(const ColumnSelection&), and TimeTravelMetadataTable::ScanColumns(const SnapshotSelection&, const ColumnSelection&). Names resolve through Schema::Select, as in TableScanBuilder::Select: nested names are dot-concatenated, columns come back in schema order, * selects all columns, and unknown names are ignored (as in Java's Schema.select). An empty list returns rows without columns, which is enough to count rows. Selecting a map's value returns the whole map; selecting only its key is an error.
Scan() is no longer virtual. Tables implement the protected ScanProjection(const Schema& projected_schema) instead, and ScanSnapshot also receives the projected schema.
Add inspect/metadata_table_columns_internal.h, which maps the projected columns to per-column append functions so a table builds only the selected columns. Selecting only some fields of a nested struct column returns NotSupported for now; no current metadata table has a struct column.
SnapshotsTable builds only the selected columns. HistoryTable scans still return NotSupported.

Why

Metadata table scans currently always return every column. Java's metadata tables support newScan().select(...): StaticDataTask projects each row to scan.schema(), so an engine running SELECT name FROM t.refs only gets name. Without selection, a C++ caller that needs one column still pays to build all of them, for example copying every snapshot's summary map.

Behavior change

New ScanColumns methods return only the selected columns. Scan() and Scan(const SnapshotSelection&) return the same rows as before, and Scan({}) on a time-travel table still resolves to Scan(const SnapshotSelection&).
Metadata table implementations override ScanProjection(const Schema&) (or ScanSnapshot(const SnapshotSelection&, const Schema&) for time-travel tables) instead of Scan(). Open metadata table PRs (feat(inspect): add files, partitions, and manifests metadata tables #876, feat: add position_deletes metadata table #920, feat(inspect): add refs metadata table #1007) need the same change.

Testing

Added metadata_table_test cases for SnapshotsTable covering selection order, unknown names, case sensitivity, * and no selection, an empty selection, and map columns (summary.value returns the whole map, summary.key is an error). A recording TimeTravelMetadataTable checks the snapshot selection and projected columns passed by Scan(), ScanColumns, Scan(const SnapshotSelection&), and Scan({}), and another case checks that HistoryTable::ScanColumns returns NotSupported.

🤖 Generated with Claude Code

Add ColumnSelection and Scan overloads that return only the requested columns, matching Java's metadata table select(). Names resolve through Schema::Select, so nested names, case sensitivity, `*`, and unknown names behave as in TableScanBuilder::Select. Tables now implement ScanProjection(const Schema&) and build only the selected columns. SnapshotsTable uses a per-column append table from the new metadata_table_columns_internal.h helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

ICEBERG_CHECK is used for conditions that can be triggered by input/projection mismatch (e.g., a… · New
This assertion hard-codes the expected map size (10), which makes the test brittle if the fixture… · New

What changed in this PR

Adds column projection support to C++ Iceberg metadata table scans (including time travel), aligning behavior with Java (newScan().select(...)) and enabling engines to request only needed columns.

Changes:

Introduces ColumnSelection and overloads of MetadataTable::Scan(...) / TimeTravelMetadataTable::Scan(...) to project columns via Schema::Select.
Refactors scanning to be implemented via ScanProjection(const Schema&) / ScanSnapshot(..., const Schema&) instead of overriding Scan().
Adds internal helpers to map projected schema columns to per-column append functions, and updates SnapshotsTable + tests accordingly.

File	Description
src/iceberg/inspect/metadata_table.h	Adds `ColumnSelection` and new `Scan` overloads; refactors scan implementation to `ScanProjection` / projected-schema time travel.
src/iceberg/inspect/metadata_table.cc	Implements column selection via `Schema::Select` and routes scans through projected schemas.
src/iceberg/inspect/metadata_table_columns_internal.h	New internal helper to select per-column append functions based on projected schema.
src/iceberg/inspect/snapshots_table.h	Updates `SnapshotsTable` to implement `ScanProjection(...)` rather than `Scan()`.
src/iceberg/inspect/snapshots_table.cc	Implements projected-column building for snapshots using selected append functions.
src/iceberg/inspect/history_table.h / .cc	Updates `HistoryTable` to `ScanProjection(...)` while keeping scan unsupported.
src/iceberg/test/snapshots_table_test.cc	Adds tests for column selection semantics on `SnapshotsTable`.
src/iceberg/test/metadata_table_test.cc	Adds tests for time-travel scans with column projection and `HistoryTable` behavior.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    ICEBERG_CHECK(field.has_value(), "Metadata table column '{}' is not in the schema",
+                  projected_field.name());
+    if (*field->get().type() != *projected_field.type()) {
+      return NotSupported("Cannot select nested fields of metadata table column '{}'",
+                          projected_field.name());
+    }
+    auto column = std::ranges::find(columns, projected_field.field_id(),
+                                    &MetadataTableColumn<Row>::field_id);
+    ICEBERG_CHECK(column != columns.end(),
+                  "Metadata table column '{}' has no append function",
+                  projected_field.name());


shangxinli · 2026-10-14T14:09:54Z

-  /// \brief Scan using the current snapshot on the main branch.
-  Result<ArrowArrayStream> Scan() final;
+  // Scans without a snapshot selection read the current snapshot on the main branch.
+  using MetadataTable::Scan;


This pulls in Scan(const ColumnSelection&), which makes an existing call like Scan({}) ambiguous against Scan(const SnapshotSelection&) since both are aggregates. Is that source break intended, or would a distinct name for the column-selection form avoid it?

Renamed the column-selection forms to ScanColumns

Scan(const ColumnSelection&) made Scan({}) on a TimeTravelMetadataTable ambiguous with Scan(const SnapshotSelection&), since both are aggregates. Use a distinct name so existing Scan({}) calls keep resolving to the snapshot-selection overload. Also assert on summary entries instead of the summary map's size, and move the selection rules explained in test comments to the ColumnSelection docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

shangxinli

The ScanColumns rename resolves the ambiguity I raised - Scan({}) still picks the SnapshotSelection overload. LGTM.

wgtmac · 2026-10-16T05:48:00Z

+  std::optional<std::vector<std::string>> names;
+
+  /// \brief Whether names are matched case-sensitively.
+  bool case_sensitive = true;


Why not use a generic scan builder to pass them individually just like the data table scan does?

Copilot AI balanced review requested due to automatic review settings October 13, 2026 15:42

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Copilot stopped reviewing on behalf of manuzhang due to an error October 13, 2026 20:17
Workflow completed but conclusion is not available. This may indicate rate limiting or API issues.

shangxinli reviewed Oct 14, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 15, 2026 11:30

Copilot AI reviewed Oct 15, 2026

shangxinli approved these changes Oct 15, 2026

View reviewed changes

manuzhang requested a review from wgtmac October 15, 2026 14:51

wgtmac reviewed Oct 16, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(inspect): support selecting columns in metadata table scans - #1008

feat(inspect): support selecting columns in metadata table scans#1008
manuzhang wants to merge 2 commits into
apache:mainfrom
manuzhang:metadata-table-select-columns

manuzhang commented Oct 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

shangxinli Oct 14, 2026

Uh oh!

manuzhang Oct 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

shangxinli left a comment

Uh oh!

wgtmac Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

manuzhang commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Behavior change

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

shangxinli Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

manuzhang Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

shangxinli left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manuzhang commented Oct 13, 2026 •

edited

Loading