Skip to content

Align metadata propagation through Physical and Logical casts - #23169

Open
paleolimbot wants to merge 23 commits into
apache:mainfrom
paleolimbot:cast-metadata-cleanup
Open

Align metadata propagation through Physical and Logical casts#23169
paleolimbot wants to merge 23 commits into
apache:mainfrom
paleolimbot:cast-metadata-cleanup

Conversation

@paleolimbot

@paleolimbot paleolimbot commented Jun 24, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

The logical Expr::Cast and Expr::TryCast have a FieldRef target that was added in #18136 so that logical casts can express a cast to an extension type. In combination with a SQL type planner ( #20676 ) and an optimizer rule, this enabled casting to/from extension types with custom semantics to actually occur. The ability to do this was reverted by #20836 (which removed the original test) and I am not sure that ability ever made it into a release. When investigating this issue, it became clear the logical and physical cast behaviour had diverged with respect to the target field.

What changes are included in this PR?

This PR strips specific metadata keys (extension name and extension metadata) when propagating metadata from the source of a cast to the target (because doing so may result in an invalid destination field that consumers could reject), and propagates all metadata from the (logical) cast target field (e.g., so that a cast to an extension type represented by the cast target field will have a to_field() that communicates the extension type).

For the physical cast, this behaviour is replicated exactly (I hope).

Note that actually casting to an extension type can be implemented with an optimizer rule, planner, or by the mechanism I have in the works in #21071 .

Are these changes tested?

Yes

Are there any user-facing changes?

It was in practice not common to create a Expr::Cast with field metadata internally and thus I don't think users will see metadata changes from the inclusion of metadata from the target field. I would be surprised if stripping the extension name/metadata from the source was disruptive (it was more likely to have caused errors).

Superceeds an earlier but similar attempt ( #22162 ).

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates labels Jun 24, 2026
@paleolimbot
paleolimbot force-pushed the cast-metadata-cleanup branch from 7662d3a to c74dd77 Compare June 25, 2026 19:58
Comment on lines +67 to +70
/// Whether to preserve non-extension metadata from the source field.
/// When true (default), source metadata is merged with target metadata.
/// When false, only the target field's metadata is used.
preserve_source_metadata: bool,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little gross feeling to me, but there's at least one place in the code that is counting on Cast::return_field() not to execute its child's return_field(): the virtual row number rewrite requires that the cast strips metadata and does not validate its child's Column (which is invalid, at least where it's tested). There may be a better way here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm maybe rewrite_file_row_index_expr should not use the CastExpr for this purpose but an "intrinsic" UDF? But I guess this is a different issue then. Not sure how to otherwise avoid that (especially, because we would then require this flag for TryCast if we want feature parity).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an alternate suggestion (compute the target metadata needed in the virtual row number code)

@paleolimbot
paleolimbot marked this pull request as ready for review June 26, 2026 19:37
@paleolimbot paleolimbot changed the title Fix extension type metadata propagation through casts Align metadata propagation through Physical and Logical casts Jun 26, 2026
@paleolimbot

Copy link
Copy Markdown
Member Author

@adriangb @tschwarzinger @cyberbeam524 You've all kindly waited for me to finish this...happy to iterate on any of your comments here!

@tschwarzinger tschwarzinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks already great to me! Thank you for finding the time to work on this! 🚀

Sorry, if I have many questions as I am not too familiar with the casting system, even after trying to explore it a bit. Feel free to leave them unanswered.

I think, what would be great is a test that checks the consistency between logical and physical planning. I.e., create a few logical expr and a physical expr pairs with the same casting semantics and check whether they come to the same conclusion. Do you think this would be easily possible?

Comment thread datafusion/expr/src/expr_schema.rs Outdated
}

#[test]
fn test_try_cast_extension_type_metadata() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can combine these two tests and just parameterize them with Expr::TryCast and Expr::Cast ?

Currently, the test for Expr::Cast seems to check that the metadata of the target field is used, while the test for Expr::TryCast does not test that. We can maybe avoid the nullability check in Expr::TryCast as this has nothing to do with extension types and should be done in a separate tests.

However, this could make the test more "complicated" and keeping them apart might be better. Up to you how you want to deal with it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I did this one!

Comment thread datafusion/expr/src/expr_schema.rs Outdated
use arrow_schema::extension::{EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY};

// Start with source metadata
let mut metadata = source_field.metadata().clone();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify for myself:

  • Creating a custom Expr::Cast with custom metadata (e.g., my_custom_key) has never really worked in the sense that my_custom_key becomes part of the output metadata.
  • And it also does not work in this branch as we only consider the extension name / metadata from the target field.

I think this is fine and if we ever need something like this we can track it in a separate ticket with a use case that requires this sort of casts. I just to make sure that we are not removing the ability to include custom metadata for downstream users.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with custom metadata (e.g., my_custom_key) has never really worked in the sense that my_custom_key becomes part of the output metadata.

Do you mean like adding metadata to the Cast expr itself?

#[derive(Clone, PartialEq, Eq, PartialOrd, Hash, Debug)]
pub struct Cast {
    /// The expression being cast
    pub expr: Box<Expr>,
    /// The `DataType` the expression will yield
    pub field: FieldRef,
    /// *** New Field: custom metadata **
    pub metadata: Metadata,
}

We can probably achieve the same thing with a user defined function perhaps

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alias has been able to do this type of metadata reassignment for some time, and I am not sure it is a good use of the Cast to have it messing with metadata (the confusion that led to the current situation is that the cast operator is being used to manipulate metadata, rather than transform a "data type" in the data-type-or-extension-type sense). I'm happy to tweak this to make it consistent...all I personally need is for the target field extension information not to be dropped (so I can insert an optimizer rule that transforms it into a function call).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do personally like the notion that tje field field unambigiously defines the output Field of the expression 🤔

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For arguments sake, and because it seemed a lot cleaner to me, I updated the physical Cast to be explicit about its constituents (i.e., None for metadata passes through from source for backward compatibility, Some(...) forces output, like is required for schema alignment). We were never really casting to a field, we were just using a FieldRef to store three pieces of information.

cast_options: CastOptions<'static>,
/// Whether to preserve non-extension metadata from the source field.
/// When true (default), source metadata is merged with target metadata.
/// When false, only the target field's metadata is used.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we keep this flag, should we use this flag too in to_field too? Sorry I am not too familiar with the casting system. And if the answer is yes, should we merge the metadata or just retain the source metadata like we currently do (I think)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes from one specific use that is using the physical cast specifically to strip metadata. I think your later suggestion to remove that usage (and remove this flag) is a good one (but I'll wait for one more opinion on what to do here before I commit).

Comment on lines +67 to +70
/// Whether to preserve non-extension metadata from the source field.
/// When true (default), source metadata is merged with target metadata.
/// When false, only the target field's metadata is used.
preserve_source_metadata: bool,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm maybe rewrite_file_row_index_expr should not use the CastExpr for this purpose but an "intrinsic" UDF? But I guess this is a different issue then. Not sure how to otherwise avoid that (especially, because we would then require this flag for TryCast if we want feature parity).

metadata.remove(EXTENSION_TYPE_METADATA_KEY);

// Target metadata takes precedence (including extension type metadata)
for (k, v) in self.target_field.metadata() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we use the target metadata fields ("custom metadata") but not in cast_output_field, which confuses me a bit.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I fixed this (and added a test)

EXTENSION_TYPE_NAME_KEY.to_string(),
"source.type".to_string(),
),
("source_key".to_string(), "source_value".to_string()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe adding "target_key" to the target_meta would be beneficial too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I got this one!


[dependencies]
arrow = { workspace = true }
arrow-schema = { workspace = true }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just use the re-exported types from arrow (why add a new dependency)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't figure out how to get arrow_schema::extension::<constants> to show up (I don't think schema is pub exported from arrow). I can also inline the constants for metadata key/value.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine -- probably means we should add some more re-exports to arrow from arrow-schema

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @paleolimbot and @tschwarzinger -- I can see this is somewhat wacky

I realize the current code takes the metadata from the source field, but that is really confusing to me (as I would expect a cast to produce the output in the target field)

I suspect the challenge is that at some higher levels, when casting to a "data type" that the source metadata can be lost

So what if we made the behavior consistent by:

  1. the Cast expr itself propagate metadata directly from the target_field
  2. Have the higher level aps (like cast_to(data_type)) propagate the source metadata to the target they create

So the semantics of what CastExpr is doing would be clear and consistent

Comment thread datafusion/expr/src/expr_schema.rs Outdated
use arrow_schema::extension::{EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY};

// Start with source metadata
let mut metadata = source_field.metadata().clone();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with custom metadata (e.g., my_custom_key) has never really worked in the sense that my_custom_key becomes part of the output metadata.

Do you mean like adding metadata to the Cast expr itself?

#[derive(Clone, PartialEq, Eq, PartialOrd, Hash, Debug)]
pub struct Cast {
    /// The expression being cast
    pub expr: Box<Expr>,
    /// The `DataType` the expression will yield
    pub field: FieldRef,
    /// *** New Field: custom metadata **
    pub metadata: Metadata,
}

We can probably achieve the same thing with a user defined function perhaps

/// Derives the output field for a cast expression from the source field.
/// Derives the output field for a cast expression from the source and target fields.
///
/// Metadata handling:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is an internal function, I don't think this documentation will be easy to find -- that is not to say it isn't good to have. However I think we should probably also make a more authoritative docs

For example, it would be nice to link to this detail from here

/// - **Cast expressions**: determined by the input expression's field metadata handling

@paleolimbot paleolimbot Jul 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't quite get there today, but we can add this to the documentation. (edit: done)

}
}

/// Create a new `CastExpr` with an explicit target `FieldRef`, using the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we instead make this a more builder style API -- like

pub fn with_preserve_source_metadata(self, preserve_source_metadata: bool) -> Self {
  self.preserve_source_metadata = preserve_source_metadata;
  self
}

I think that would make th callsites easier to read

let cast= Cast::new(..)
  .with_preserve_source_metadata(false)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The physical cast changes let me remove this flag, which I think was kind of ugly anyway. The previous constructors were restored (although maybe we want to make them better).

let source = Arc::new(Column::new(row_index_name, row_index_idx));
let target_field = Arc::new(Field::new("file_row_index", DataType::Int64, true));
Ok(Arc::new(CastExpr::new_with_target_field(
Ok(Arc::new(CastExpr::new_with_exact_target_field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a comment here about why it is preservng the source field?

Or is there some way in this code to calculate the desired output metadata here (and attach it to the target field) ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The physical cast changes let me remove this function and this change dissappeared

Comment on lines +67 to +70
/// Whether to preserve non-extension metadata from the source field.
/// When true (default), source metadata is merged with target metadata.
/// When false, only the target field's metadata is used.
preserve_source_metadata: bool,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an alternate suggestion (compute the target metadata needed in the virtual row number code)

Comment thread datafusion/expr/src/expr_schema.rs Outdated
// Start with source metadata
let mut metadata = source_field.metadata().clone();

// Remove any extension type metadata from source - these should not propagate through casts

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry this is not well documented and we may not be consistent across operators

What do you think about adding documentation on what the expectation for metadata / custom types? I feel like if we had a some good refereence we could then evaluate if the code matched the desired behavior. At the moment I don't know what the overall behavior is supposed to be

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do this as a separate PR

if let Some(ext_meta) = target_metadata.get(EXTENSION_TYPE_METADATA_KEY) {
metadata.insert(EXTENSION_TYPE_METADATA_KEY.to_string(), ext_meta.clone());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels strange to me that when casting to a new Field that the metadata is taken from the source

I think I would personally expect it to come from the target Field (as we are casting "to" that field) -- though I see that is not what the current code does

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this to make a bit more sense hopefully...the previous behaviour (pass all metadata through because casts are just modifying the DataType) stays, except we strip extension metadata so that ::VARCHAR has an output type of Utf8. If any metadata is specified, it is authoritative.

Unfortunately updating the storage like I did for the physical cast would be a breaking change here (we maybe shouldn't have used a FieldRef 😬 ), so "the previous behaviour" is a bit cryptically detected (mind you, this cryptic detection previously existed in the physical operator...I just moved it here).

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 1, 2026
@paleolimbot

paleolimbot commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

(I'm missing TryCast tests here, but I updated this a bit to make it a bit more sane, if a little more verbose. Happy to update whatever here, it just seemed like there was an appeteite to make this cleaner than my hacky previous solution)

edit: I added TryCast tests + sqllogic tests for the practical usages of this, which is basically casting extension types to and from their storage

@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jul 2, 2026
@github-actions github-actions Bot added auto detected api change Auto detected API change and removed auto detected api change Auto detected API change labels Aug 31, 2026
@gene-bordegaray

Copy link
Copy Markdown
Contributor

I cannot edit the descriotion. But will be reviwing this over pushing my implmeentation at #24725 , I have an open issue, can you mark this as closing the issue: #24724 . Thank you 🙇

@paleolimbot

Copy link
Copy Markdown
Member Author

Sure! Happy to update as needed. I haven't thought about this in a while and it may not be up to date with recent datafusion changes.

@gene-bordegaray gene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think logical protobuf decoding still drops the cast target metadata added during encoding.

to_proto.rs writes field.metadata() but from_proto.rs rebuilds the field using only arrow_type and nullable.

this should be good after ^ and other comments are addressed. Thank you @paleolimbot 🙇

let source_field = expr.return_field(input_schema)?;
let has_extension_metadata = source_field
.metadata()
.contains_key(EXTENSION_TYPE_NAME_KEY);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be:

   let metadata = source_field.metadata();
   let has_extension_metadata = metadata.contains_key(EXTENSION_TYPE_NAME_KEY
     || metadata.contains_key(EXTENSION_TYPE_METADATA_KEY);

same with the corresponding line in try_cast

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so...if there's extension metadata but no extension name, I think it's just metadata without any special handling (or more likely, the producer screwed up). Happy to add if you feel strongly.

&& !contains_type(target_type, &is_struct)))
{
let Some(target_field) = retain_field_path(cast.target_field(), &field_path)
let cast_target_field = cast.target_field();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use cast_target_field here

@gene-bordegaray

Copy link
Copy Markdown
Contributor

cc: @alamb

@gene-bordegaray gene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgot to correct the breaking change for 55.1, reminded by @timsaucer thank you 🙇

/// a `DataType`.
/// See [`CastExpr::new`] for type-only casts where source metadata should
/// pass through.
pub fn new_with_target_field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missed this, but since this is going to be in 55.1 patch we cant have the breakin change. Wec ould represent this internally in a way to set up the change in #24725 via something like:

enum CastTarget {
  TypeOnly(FieldRef),
  Explicit(FieldRef),
}

impl CastTarget {
  fn field(&self) -> &FieldRef {
    match self {...}
  }

  fn is_explicit(&self) -> bool { ... }
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with Tim's approach, which was slightly simpler. Introducing a CastTarget is a good idea in general, though.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main motivation was to get this backported, which required some simplification.

Restore the two public signatures on `CastExpr` that this branch changed,
so the metadata fixes can be backported to a patch release:

* `CastExpr::new_with_target_field` takes `FieldRef` by value again
* `CastExpr::target_field()` returns `&FieldRef` again

Both are achieved by storing the target `FieldRef` as before and adding a
private `explicit_target` flag that records whether the field was supplied
by the caller or synthesized from a `DataType`. That flag carries exactly
the information the `Option<HashMap<..>>` / `Option<bool>` pair carried, so
`target_metadata()`, `target_nullable()`, `has_explicit_metadata()` and
`has_explicit_nullability()` keep their meaning and are derived from it.
`Hash`/`PartialEq` compare the same components as before, so the field name
still does not participate.

`TryCastExpr` is given the same shape for symmetry. Its metadata-aware API
is new on this branch, so nothing there is a compatibility constraint.

Cast semantics are unchanged: explicit target fields still supply their
metadata and nullability verbatim, type-only casts still pass source
metadata through with the extension type keys stripped, and the output
field name still comes from the source expression.

Comparing the public signatures against the merge base with main, the only
remaining deltas are additions:

    + CastExpr::target_metadata / target_nullable
    + CastExpr::has_explicit_metadata / has_explicit_nullability
    + TryCastExpr::new_with_target_field / target_metadata / target_field
    + expressions::try_cast_with_target_field

`cast_with_target_field` keeps its `&FieldRef` parameter: `mod cast` is
private and it is only re-exported as `pub(crate)`, so it is not part of
the public API.

Verified: `cargo clippy --workspace --all-targets -- -D warnings` clean,
504/504 sqllogictest files pass, and the unit tests for physical-expr,
physical-expr-adapter, physical-plan, pruning, expr, functions and the
datafusion core lib all pass.
Keep cast metadata changes non-breaking
@github-actions github-actions Bot removed the physical-plan Changes to the physical-plan crate label Aug 31, 2026
@github-actions github-actions Bot added the proto Related to proto crate label Aug 31, 2026
@paleolimbot

Copy link
Copy Markdown
Member Author

I think logical protobuf decoding still drops the cast target metadata added during encoding.

Good call! Should be fixed.

I think I got to the comments with Tim's help but happy to update again as well.

@timsaucer

Copy link
Copy Markdown
Member

Ok, everything is in the green for CI and we no longer have API changes. I'm going to give it one more review and if nobody has objections I'll probably merge in the morning.

@timsaucer timsaucer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a lot of work and a lot of valuable feedback across the community. Thank you @paleolimbot for driving it to completion!

@gene-bordegaray gene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restamp, thanks all @timsaucer @paleolimbot

@adriangb

adriangb commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

@paleolimbot thank you for working on this, and thank you @timsaucer for reviewing.

I would like to bring up a question that is probably better asked now, as frustrating as it may be, than after merging: are these really the semantics we want? The main alternative I see is "cast stamps the metadata of the target type", which is what @paleolimbot originally wanted in #22079 and it's what @alamb proposed in #23169 (comment). That's easier to reason about. I was the one that originally pushed us in this direction (merging but stripping extension metadata from the source), but on reflection I don't have any concrete use case for this, I was just defending the status quo (and probably should have spent more time trying to understand the consequences before having @paleolimbot put in all of the work in this PR to work around that). If we wanted to go in this direction (the result field from the cast carries the metadata from the target) I would support that change.

I've put together a series of draft PRs that reach all of the new tests this PR adds, but via the target-stamps rule instead: #24831 -> #24833 -> #24834 / #24835. Stacked, they pass this PR's version of cast_extension_type_metadata.slt verbatim, with the full sqllogictest suite green.

The main con I see to this is that it will be a breaking change in 56. We're somewhat taking a gamble with some use case or user presenting a use case the change broke, but it's a smaller gamble than I expected: across the whole sqllogictest suite the rule change moves 7 assertions, all in metadata.slt, and all of them were added by me in #21390. It's also worth noting #21390 only made try_cast match cast, so half of that surface has only existed since 54. In exchange we are buying ourselves IMO an easier to reason about mental model.

I'm also happy to merge this as is, but I wanted to offer the option in case we are going down a more complex path because of my input.

Separately, and at no fault of this PR, I don't think we should be viewing this as the minimal back port required. Back ports are to fix bugs introduced in that release or recent releases. #24721 does qualify on that basis: the projection metadata loss is a regression found upgrading to 55, which is what I scoped #24831 to (#24721 plus the minimum cast change needed to make the projection fix not regress). But this PR is fixing longstanding inconsistent behavior, which I don't think meets the same bar. I think there's a more narrow version of #24670 which does (see #24670 (comment), #24831).

@timsaucer

Copy link
Copy Markdown
Member

@paleolimbot @adriangb @gene-bordegaray Would you all be available tomorrow for the community meeting so we can discuss in real time? I think Adrian makes a strong point about the scope and what should/shouldn't be in a back port.

@adriangb

adriangb commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Yea, I should be able to join. This does seem like that kind of thing worth discussing live.

@gene-bordegaray

Copy link
Copy Markdown
Contributor

yes sounds, good 👍

@paleolimbot

Copy link
Copy Markdown
Member Author

I would like to bring up a question that is probably better asked now, as frustrating as it may be, than after merging: are these really the semantics we want?

I think the semantics in this PR are a standalone improvement over the current situation, which is not consistent between trycast and cast / prevents using DataFusion's SQL parser to do casts into extension types.

I don't think we should be viewing this as the minimal back port required. Back ports are to fix bugs introduced in that release or recent releases.

I've always considered this (the target field of a cast / try cast is not reflected in its return_field()) a bug. Technically it was a bug that was introduced before the correct behaviour was ever released, it just seems that nobody noticed or cared until now.

As always, happy to update this PR if there's a reasonable chance it will merge.

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the semantics in this PR are a standalone improvement over the current situation, which is not consistent between trycast and cast / prevents using DataFusion's SQL parser to do casts into extension types.

I agree it's an improvement over the current state.
The questions I want to make sure we answer are:

  1. Is it the long term behavior we want, or do we want metadata = target field metadata as the long term behavior? As per above I recall this being your preferred behavior.
  2. Is this PR something we want to backport to 55?

Basically if you @paleolimbot think these (this PR) are the right long term semantics we want or we think we should backport something to 55 and this is our best option then I'm 👍🏻 on it.

@paleolimbot

Copy link
Copy Markdown
Member Author

think these (this PR) are the right long term semantics we want or we think we should backport something to 55 and this is our best option then I'm 👍🏻 on it

Whether the target field obliterates incoming metadata or merges it doesn't matter much (except for the extension name, which this PR handles); passthrough is safer because DataFusion has done it for a long time, and can be changed by an optimizer rule if a database needs it. I would love it if this were merged and backported while we're all here looking at it, and any improvements / breaking changes to code or behaviour can be a separate discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

8 participants