[SPARK-53154][SQL][PYTHON] Add hmac SQL function

[vinodkc]

What changes were proposed in this pull request?

Add a built-in hmac(key, message[, algorithm]) scalar function that computes a keyed-hash message authentication code (HMAC) using the JVM's javax.crypto.Mac.

API surface added:

• SQL: hmac(key, message) / hmac(key, message, algorithm)
• Scala DataFrame: functions.hmac(key, message) / functions.hmac(key, message, algorithm)
• PySpark (classic + Connect): pyspark.sql.functions.hmac(key, message, algorithm=None)

Key design choices:

• Returns BinaryType (raw MAC bytes), not a hex string — so the output of one
  hmac call feeds directly into the key of the next with no unhex in between.
  This is the critical property for chained key-derivation (e.g. AWS SigV4).
• Argument order is (key, message), matching Python's hmac.new(key, msg) and RFC 2104's HMAC(K, m).
• Default algorithm is SHA-256. Supported: SHA-224, SHA-256, SHA-384, SHA-512, SHA-1, MD5 (case-insensitive; both SHA-256 and SHA256 spellings accepted).

Why are the changes needed?

Spark had no single expression node for HMAC. Users had to hand-build the RFC 2104
formula from sha2, concat, bitwise xor, and unhex primitives. The motivating
real-world use case derives an AWS SigV4 signing key by chaining four HMACs, each step's output becoming
the next step's key:

-- Four-step SigV4 key derivation (AWS docs, "Signature Version 4").
-- Each hmac() output is BinaryType, feeding directly as the next key -- no unhex anywhere.
SELECT hex(
  hmac(
    hmac(
      hmac(
        hmac(concat('AWS4', secret), date),
        region),
      service),
    'aws4_request')
) AS signing_key;

Without this function, each HMAC step is a deep sha2/unhex/concat/xor subtree.
Nesting four of them multiplies the tree size to the point where the Catalyst analyzer/optimizer runs out of memory on realistic inputs. hmac collapses each level to a single StaticInvoke node.

Does this PR introduce any user-facing change?

Yes. Adds a new built-in SQL function hmac and corresponding Scala/PySpark
DataFrame API entries. No existing behavior is changed.

Example:

-- Default algorithm (SHA-256)
SELECT hex(hmac('key', 'message'));
-- 6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A

-- Explicit algorithm
SELECT hex(hmac('key', 'message', 'SHA-512'));
-- E477384D7CA229DD1426E64B63EBF2D36EBD6D7E669A6735424E72EA6C01D3F8B56EB39C36D8232F5427999B8D1A3F9CD1128FC69F4D75B434216810FA367E98

-- Chaining: binary output feeds directly as the next key
SELECT hex(hmac(hmac('key', 'message'), 'message2'));
-- 28042756E0362D954B61661BB4DEC9FF9F6C970D346CAEB6DB36ED7B735AB38C

How was this patch tested?

Added tests in ExpressionImplUtilsSuite, MiscExpressionsSuite , misc-functions.sql

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (sonnet-5)

[Ma77Ball]

Overall, adding an HMAC function sounds like a great addition. I left two minor optional nits below.

[uros-b]

Thank you @vinodkc and @Ma77Ball!

[cloud-fan]

0 blocking, 0 non-blocking, 0 nits.
No actionable findings; the implementation is consistent with Spark's existing cryptographic expression pattern and its declared API contract.

Verification

Reviewed all 16 changed artifacts at head 84784fc64bead0f6abcf9032da00486afb5de79d, traced function registration through Hmac to ExpressionImplUtils.hmac, compared the design and error handling with AES, checked null/error/test coverage, and completed link and text-quality scans. Tests were not run as part of this review.

[cloud-fan]

Merge Summary:

• merged into master 2bb0b35
• merged into branch-4.x be794c0

Posted by merge_spark_pr.py