-
Notifications
You must be signed in to change notification settings - Fork 77
Description
Is your feature request related to a problem? Please describe.
The GpuTimeZoneDB timezone conversion functions (fromTimestampToUtcTimestamp, fromUtcTimestampToTimestamp) currently only support TIMESTAMP_SECONDS, TIMESTAMP_MILLISECONDS, and TIMESTAMP_MICROSECONDS. TIMESTAMP_NANOSECONDS is not supported and throws an error.
This is likely a blocker for our usecase where the timestamp in nanos is used only inside the UDF and never exported to Spark proper as a Timestamp. I will close this issue if we can work around this by using micros, or deprioritize it.
Describe the solution you'd like
The underlying algorithm (convert_timestamp_tz_functor) is already templated and supports any timestamp type.
The fix might be this simple (other than testing):
case cudf::type_id::TIMESTAMP_NANOSECONDS:
return convert_timestamp_tz<cudf::timestamp_ns>(
input, transitions, tz_index, to_utc, stream, mr);
Describe alternatives you've considered
No alternatives for timezone conversions, they are in spark-rapids-jni.
We may close this issue if micros good enough for the usecase.
Additional context
Spark's TimestampType only supports microseconds, so this is only for usecases touching the java api directly.