Skip to content

[FEA] Add TIMESTAMP_NANOSECONDS support to GpuTimeZoneDB timezone conversion #4089

@abellina

Description

@abellina

Is your feature request related to a problem? Please describe.
The GpuTimeZoneDB timezone conversion functions (fromTimestampToUtcTimestamp, fromUtcTimestampToTimestamp) currently only support TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS, and TIMESTAMP_MICROSECONDS. TIMESTAMP_NANOSECONDS is not supported and throws an error.

This is likely a blocker for our usecase where the timestamp in nanos is used only inside the UDF and never exported to Spark proper as a Timestamp. I will close this issue if we can work around this by using micros, or deprioritize it.

Describe the solution you'd like
The underlying algorithm (convert_timestamp_tz_functor) is already templated and supports any timestamp type.

The fix might be this simple (other than testing):

case cudf::type_id::TIMESTAMP_NANOSECONDS:  
  return convert_timestamp_tz<cudf::timestamp_ns>(
  input, transitions, tz_index, to_utc, stream, mr);

Describe alternatives you've considered
No alternatives for timezone conversions, they are in spark-rapids-jni.

We may close this issue if micros good enough for the usecase.

Additional context
Spark's TimestampType only supports microseconds, so this is only for usecases touching the java api directly.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions