Skip to content

Conversation

@alexjbuck
Copy link

@alexjbuck alexjbuck commented Dec 16, 2025

Add the ability to create a tag reference atomically in the same transaction that creates a new snapshot via fast_append.

This adds a with_tag() and with_tag_retention() methods to FastAppendAction that allows specifying a tag name and retention. When set, the tag will be created pointing to the newly created snapshot, all within a single atomic catalog update. The default tag retention is None which should mean it inherits the Table's defaults, if specified.

Example usage:

let tx = Transaction::new(&table);
let action = tx
    .fast_append()
    .add_data_files(data_files)
    .with_tag("v1.0.0");
    .with_tag_retention(i64::MAX); // Implies maximum retention, don't let it expire.
let tx = action.apply(tx)?;
let table = tx.commit(&catalog).await?;

Which issue does this PR close?

What changes are included in this PR?

This change adds a method to the FastAppendAction that allows atomic creation of a snapshot reference to a specified tag in the same transaction that creates the snapshot. This enables atomic setting of tags and appending data in a single transaction.

Without the atomic transaction guarantees, it is possible for other processes to add new snapshots and cleanup old snapshots (including the one we just created) before a tag can be added to the snapshot, thus protecting it from most automated snapshot expiration policies.

With the atomic guarantees, we can guarantee that either our data was not committed into the table, or it was and it has a tagged snapshot reference that is protected from expiry.

Are these changes tested?

I created a unit test test_fast_append_with_tag in the src/transaction/append.rs module that
verifies that there are 3 actions in the FastAppendAction when called with .with_tag("tag"), and
that those 3 actions are a TableUpdate::AddSnapshot and two TableUpdate::SetSnapshotRef actions,
with one setting the MAIN_BRANCH tag, and the second setting the tag specified in
.with_tag("tag").

There is another unit test called test_fast_append_with_tag_retention that validates the tag retention behavior as well.

Edit: I'm looking into why 3 tests from the iceberg-catalog-glue test set are failing, namely test_create_table, test_drop_table, and test_load_table. They're all erroring on The specified bucket does not exist which makes me think this is unrelated to my changes.

Edit 2: I rebased off the current main and that resolved the failing tests.

Edit 3: I added the with_tag_retention() method to allow specifying the retention behavior for the tagged snapshot instead of defaulting to the Table defaults.

@alexjbuck alexjbuck force-pushed the alexjbuck/atomic-snapshot-tag branch from 884853f to f1ffb2e Compare December 16, 2025 21:21
Add the ability to create a tag reference atomically in the same
transaction that creates a new snapshot via fast_append.

This adds a `with_tag()` method to `FastAppendAction` that allows
specifying a tag name. When set, the tag will be created pointing
to the newly created snapshot, all within a single atomic catalog
update.

Example usage:
```rust
let tx = Transaction::new(&table);
let action = tx
    .fast_append()
    .add_data_files(data_files)
    .with_tag("v1.0.0");
let tx = action.apply(tx)?;
let table = tx.commit(&catalog).await?;
```
Add `with_tag_retention()` method to FastAppendAction to allow
controlling tag expiration independently from table defaults.

Tags created with `with_tag()` default to using the table's
`history.expire.max-ref-age-ms` property. Users can override this
with `with_tag_retention(i64::MAX)` to make tags never expire, or
specify a custom retention period in milliseconds.

Example usage:
```rust
// Uses table defaults
.with_tag("v1.0.0")

// Never expires
.with_tag("v1.0.0")
.with_tag_retention(i64::MAX)

// Custom 30-day retention
.with_tag("v1.0.0")
.with_tag_retention(30 * 24 * 60 * 60 * 1000)
```
@alexjbuck alexjbuck force-pushed the alexjbuck/atomic-snapshot-tag branch from 11bd4c8 to 99a0143 Compare December 17, 2025 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the ability to transactionally create named references (tags) to a snapshot in the FastAppendAction

1 participant