XOF extends Hash: SHAKE128 and SHAKE256 are hashes, and squeezing gets its own type

[dghgit]

Executive Summary

This is a complete PR to add support for SP 800-185, it's attempting to address two things, the first being that XOF functions are treated in a lot of situations as message digests, and there's even OIDs for the same, the second is how to deal with variable length output across TupleHash, ParallelHash, and KMAC (the last one actually qualifying as a MAC and an XOF...).

After a bit of to-ing and fro-ing, partly due to the lack of overrides in Rust, the best solution to the problem seemed to be the creation of XofOutput, which is a small trait the getting of which disables the use of the parent structure - this allows do_update on the Hash trait to stay infallible, but thanks to the borrow checker, causes a compile time error if XofOutput is created and someone attempts to use the parent Hash. Most of the text here is devoted exclusively to that issue, it also includes the sample code for the trait, and the sample usage below. There's no need to look anywhere else (yet).

First question: given the constraints, does this look okay as a solution? Second question: If so, I guess it should be XOFOutput shouldn't it?

I note this will need to be re-based once #117 goes in.

Comments welcome.

Issue Link

No linked issue.

Summary

XOF now extends Hash, so SHAKE128 and SHAKE256 are hashes and can be used wherever one is
wanted; the squeezing phase becomes its own type, which turns the absorb-then-squeeze rule from a
runtime error into a compile error.

Description

Stacked on #115, so this PR is based on feature/symmetric-cipher and shows four commits:

Commit
e3c31e3	core, sha3: XOF extends Hash; squeezing becomes its own type
d02361e	sha3: pin the SHAKE block_bitlen and output_len values, which three mutants survived
19f00a0	core: XofOutput gains do_final / do_final_out
a7dd1a4	gitignore: ignore editor swap and backup files

The goal. SHAKE128 and SHAKE256 should be usable as hashes. This is the relationship BC Java
draws with Xof extends ExtendedDigest extends Digest, and it is what lets a caller hold "some
hash" and pass a SHAKE.

What blocked it. Hash::do_update is infallible; XOF::absorb returned
HashError::InvalidState for absorb-after-squeeze. BC Java has the same rule but signals it with an
unchecked IllegalStateException from KeccakDigest.absorb, which is not an option here. So the
error had to go somewhere, and the three candidates were: make Hash::do_update fallible
everywhere, panic, or remove the reachable state.

Why the third. Every absorb site in ML-KEM and ML-DSA already read
.expect("absorb before squeeze is infallible") — 117 of them. The callers were unanimous that the
error could not happen in correct code, which is exactly the case QUALITY_AND_STYLE.md says to
redesign rather than wrap in a Result:

If you're returning Result for something the caller can't reasonably hit with valid usage,
redesign the signature instead.

So XOF::into_output consumes the XOF and returns an XofOutput. After output has begun there is
no value left to call do_update on, the error has nothing to report, and all 117 .expect(...)
calls are gone. Err() counts in core code dropped from 314 to 307.

The invariant that makes do_update infallible is that a SHAKEInternal a caller can name has
never squeezed. That holds because every KDF entry point also takes self by value, and it is
pinned by a debug_assert! in do_update plus a compile_fail doctest.

XofOutput

pub trait XofOutput {
    /// Produces the next `num_bytes` bytes of the output stream.
    ///
    /// BC Java's `Xof.doOutput(out, outOff, outLen)`.
    fn do_output(&mut self, num_bytes: usize) -> Vec<u8>;

    /// As [`do_output`](Self::do_output), filling the caller's buffer, which is zeroized first.
    /// Returns the number of bytes written.
    fn do_output_out(&mut self, output: &mut [u8]) -> usize;

    /// The last output: produces `num_bytes` bytes and ends the stream.
    ///
    /// It reads the same bytes [`do_output`](Self::do_output) would at the same point in the
    /// stream; the difference is only that nothing can follow it.
    fn do_final(mut self, num_bytes: usize) -> Vec<u8>
    where
        Self: Sized,
    {
        self.do_output(num_bytes)
    }

    /// As [`do_final`](Self::do_final), filling the caller's buffer, which is zeroized first.
    /// Returns the number of bytes written.
    fn do_final_out(mut self, output: &mut [u8]) -> usize
    where
        Self: Sized,
    {
        self.do_output_out(output)
    }
}

Sample code — all five of these compile and their assertions pass:

use bouncycastle_core::traits::{Hash, XOF, XofOutput};
use bouncycastle_sha3::SHAKE128;

// 1. As a hash. XOF: Hash, so SHAKE is usable wherever a hash is wanted.
let digest: Vec<u8> = SHAKE128::new().hash(b"Hello, world!");
assert_eq!(digest.len(), 32);              // SHAKE128's nominal digest size

// 2. Streaming input, then a fixed-size digest.
let mut shake = SHAKE128::new();
shake.do_update(b"Hello, ");
shake.do_update(b"world!");
assert_eq!(shake.do_final(), digest);

// 3. Extendable output: into_output() ends the input phase and hands back the stream.
let mut shake = SHAKE128::new();
shake.do_update(b"Hello, world!");
let mut out = shake.into_output();
let first  = out.do_output(16);
let second = out.do_final(16);             // last read; the handle is consumed
assert_eq!([first, second].concat(), digest);   // one continuous stream

// 4. One-shot, any length.
let long: Vec<u8> = SHAKE128::new().hash_xof(b"Hello, world!", 1024);
assert_eq!(&long[..32], &digest[..]);

// 5. Into a caller-owned buffer, no allocation.
let mut buf = [0u8; 64];
let mut shake = SHAKE128::new();
shake.do_update(b"Hello, world!");
let n = shake.into_output().do_final_out(&mut buf);
assert_eq!(n, 64);

And the case that no longer compiles, which is the point of the change:

let mut shake = SHAKE128::new();
shake.do_update(b"abc");
let mut out = shake.into_output();
shake.do_update(b"more");   // error[E0382]: use of moved value: `shake`

Method mapping.

BC Java	here
Digest.getDigestSize()	Hash::output_len — 32 / 64, fixedOutputLength / 4 as SHAKEDigest.java:84
ExtendedDigest.getByteLength()	Hash::block_bitlen, in bits — 1344 / 1088, the rate 1600 - 2c
Digest.update	Hash::do_update
Digest.doFinal(out, off)	Hash::do_final
Xof.doOutput(out, off, len)	XOF::into_output then XofOutput::do_output
Xof.doFinal(out, off, len) after doOutput	XofOutput::do_final
Digest.reset()	none — the final methods take self

XofOutput::do_final is a provided method delegating to do_output. SHAKEDigest.java:92-99
shows Java's doFinal(out, off, outLen) is doOutput plus reset(); here taking self by value
is the reset, and dropping the handle zeroizes the sponge through Secret's Drop. It is a name
for "this read is my last", not new behaviour.

Suspendable. The suite suspends a squeezing SHAKE and resumes it, which under the new design
must not hand back an absorbing value. SHAKEOutput therefore has its own Clone and
Suspendable, and each from_suspended rejects the other's phase with InvalidData. This is a
behaviour change: SHAKE128::from_suspended now refuses a state that was suspended mid-output.

Alternatives considered. Making Hash::do_update fallible would have put a Result on SHA-2,
SHA-3 and SM3 for a condition only XOFs can reach. Panicking is against house rules. A shared
metadata-only supertrait for Hash and XOF was considered and rejected: it avoids the
do_update problem but does not give substitutability, which was the whole point.

Scope and Risk

• Impacted: core, core-test-framework, sha3, factory, mlkem, mlkem-lowmemory,
  mldsa, mldsa-lowmemory, cli, mem_usage_benches.
• Runtime behaviour: SHAKE output is unchanged — cross-checked against openssl dgst byte for
  byte for both variants, and the NIST CAVP bit-oriented vectors and the ML-KEM / ML-DSA
  known-answer tests all pass unmodified. The one deliberate behaviour change is
  SHAKE128::from_suspended rejecting a squeezing state.
• Regression risk: low for output correctness (the KATs are the gate), moderate for API churn —
  this is a breaking change to XOF and every call site moved.
• Worst case: a phase transition inserted in the wrong place in a rejection-sampling loop would
  restart the output stream instead of continuing it. That would break the ML-KEM / ML-DSA KATs
  loudly rather than silently, and it did during development.

Validation

cargo test --workspace                                    # 889 pass
cargo build --workspace --all-targets                     # no warnings
cargo fmt --all --check
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --workspace

# SHAKE output is unchanged
printf 'Hello, world!' | openssl dgst -shake128 -xoflen 32
printf 'Hello, world!' | openssl dgst -shake256 -xoflen 64

# mutation testing (--file is silently ignored by this cargo-mutants; -F works)
cargo mutants --test-workspace=true -p bouncycastle-sha3 -F 'shake\.rs' --jobs 3

cargo mutants on shake.rs against the full workspace suite: 126 mutants, 89 caught, 4 missed,
33 unviable. Three of the four survivors were real gaps — nothing pinned the actual values of
block_bitlen or output_len, so 1600 - 2c surviving as 1600 + 2c went unnoticed. d02361e
adds that test and all three now die. The fourth is a genuine equivalent mutant, | versus ^ on
disjoint bit ranges, verified exhaustively over all 2048 (num_bits, partial_byte) pairs and
commented at the site. The 33 unviable are the .cargo/mutants.toml error values not typechecking
against HashError / KDFError / SuspendableError.

AI Usage Statement

Did you use AI in creating this pull request:

• No
• Yes, indirectly - no submitted code was generated by AI (e.g., answering questions, performing a review, suggestions, etc.)
• Yes, trivial code changes were generated by AI (e.g., autocompletion of a single line, reformatting, or spell-checking)
• Yes, non-trivial code changes were generated by AI

If submitted code changes were generated by AI, fill in the following declaration:
Assisted-by: Claude Code:claude-opus-5

[ounsworth]

At a quick first read of the PR description (which is very long, so I didn't read the whole thing), I am not really sure what problem this is solving, so I will need to take an actual read of the code diff to see what's going on.

My immediate thoughts are:

1. XOFs are not hashes. FIPS 202 makes that point very clearly with a whole appendix about why it's dangerous to think of them as hashes -- this boils down to the fact that SHAKE(m, 31) and SHAKE(m, 32) will be the same for the first 31 bytes and that can lead to significant weakening of the collision resistance or pre-image resistance of your protocol, if you're not careful. So the framing of this PR makes me uncomfortable. But I'll need to look at the code to see what you've actually done.

2. Returning output of type XofOutput to hide some types of state errors. I'm aware of that trick but have not used it, on purpose, in bc-rust. I have never really formalized my thinking on this point until now, but I think it's that the raw inputs and outputs to bc-rust functions (plaintexts, ciphertexts, signature values, hash outputs, etc) are things the application will need to hand to other bits of rust ecosystem -- for example, file or network IO, HTTP libs, etc -- so we should treat all inputs / outputs as raw byte types ([u8] or Vec<u8>). My feeling is that wrapping outputs in a bc-rust custom type (even if that custom type coerces to a [u8]) will create at least some confusion and user friction. I think my intuition here is that we should put

• Inputs and outputs to bc-rust functions (plaintexts, ciphertexts, signature values, hash outputs, etc) should be basic rust types (typically [u8] or Vec<u8>) and not custom bc-rust types. Even if a custom type coerces to a basic type, this will add user friction.

in the QUALITY_AND_STYLE.md as a higher priority than turning runtime errors into compile-time errors.
As always, this is my opinion, but I'd love discussion.

[ounsworth]

I think that's extremely weird to have a comment in bc-rust that references bc-java.
I guess the design / architecture discussion that needs to be had here is whether it's a goal to align bc-rust APIs to bc-java APIs, or whether we consider this a greenfield implementation and should be free to do what makes sense, regardless of what other parts of the BC family do.

[dghgit]

Just a guide, I wouldn't commit this, it's still under development though and I'm using the BC implementations of SP 800-185 to guide the process and provide compatibility tests. I'll clean these up before final commit.

[ounsworth]

My initial reaction is that I don't like this.
NIST has explicitly stated that XOFs are not hashes, and must not be used as hashes. FIPS 202 Appendix A.2 is essentially an essay on that topic, so I think this is going to get us into trouble.

I'm open to having my mind changed though.

[dghgit]

Understand how could happen. You'll see the issue when you get to the last paragraph of FIPS 202 Appendix A.2, it starts out like that, but by the end we can see it's all just one big happy family. The issue they talk about at the start is the motivation behind the design of cSHAKE, so I agree it would make things clearer if it was actually here.

So I've had a look further in BC, I think the only example of an XOF being used as both a hash and an XOF together is RSA-PSS. It would be possible to split the two services, although an RSA-PSS implementation would require both an XOF and a digest passed to it, as opposed to just digest that happens to be an XOF. I think everywhere else it's either digest or XOF. PSS might be a sign of things to come though, as in where the XOF is used to replace a KDF/MGF so it really does feel that having XOF extend Hash is better. The real problem we need to address is that Rust allows us to have a method which says "I'm always good" and for an XOF, update as "I'm always good" is not correct...

In that respect the current solution seems like the most "Rusty" one, as activating the XOF side of the service disables it's digest side, but I'm more than happy to entertain alternatives - XofOutput was just the only thing that the Internet, LLM, and myself could come up with that worked.

[ounsworth]

Backing up a step: why would someone want SHAKE to behave as a hash? I guess because you want a fixed-size output at a size larger than 512 bits (cause otherwise you'd just use SHA3_512 and truncate it). But since you don't get any additional security out of that -- it's still only 256-bit collision-resistant -- I'm not entirely sure why someone would want to do that. Probably in 98% of cases where someone wants to do that, they would be better served by using the [KDF] interface that SHAKE already impls.

As you say, the proper way to turn SHAKE into a hash function is to include the output length in the input. Like

SHAKE_X( m ) = SHAKE( u32::to_bytes(X) + m, X) = <X bit output>

Oddly, this is not what cSHAKE is because it doesn't swallow the L param, so cSHAKE is still not a [Hash].

[dghgit]

No, believe it or not, people actually use SHAKE as a digest instead of SHA3-512 or SHA3-256. RSA-PSS is actually an example of this (using SHAKE as both a digest and a mask generator), CMS as well.

[ounsworth]

I defined the XOF trait in the language of sponge constructions (absorb / squeeze) because that's what FIPS 202 does, and sponge functions have a well-defined API.

Do you have a reference (ideally a FIPS or RFC) that defined XOFs in an abstract way using the update / output language?

[dghgit]

https://www.rfc-editor.org/rfc/rfc8702.html is probably the most general one. The gap in it is it misses the -len OIDs, which mean the AlgorithmIdentifier carrying the OID is also carrying an INTEGER saying how long the output should be. There's also https://www.rfc-editor.org/rfc/rfc8692.html. https://www.rfc-editor.org/rfc/rfc8419.html is an example of a standard using -len OIDs. They're starting to appear everywhere now, KMAC as well. Haven't seen much with TupleHash/ParallelHash yet, but I'm guessing it's just a matter of time. You're probably getting the picture though.

[ounsworth]

I don't see any of those three RFCs defining SHAKE with a do_update / do_final API instead of an absorb / squeeze API. In fact, I don't see them describe an API for SHAKE at all?

I do see them using the word "digest".
To me, the word "digest" means to take an arbitrary-length input and turn it into something smaller and there are two types of cryptographic digest functions:

• Hash functions always produce the same length output.
• XOFs produce a variable-length output.

This is why I haven't used the word "digest" in bc-rust, but instead define [Hash] and [XOF] directly to encapsulate their differences in behaviour.

(btw, this kind of discussion is fun 😊 This is setting the theoretical foundations of the library)

[dghgit]

RFC 8702, section 1, second paragraph:

"In the SHA-3 family, two extendable-output functions (SHAKEs), SHAKE128 and SHAKE256, are defined. Four other hash function instances (SHA3-224, SHA3-256, SHA3-384, and SHA3-512) are also defined but are out of scope for this document. A SHAKE is a variable-length hash function defined as SHAKE(M, d) where the output is a d-bit-long digest of message M. The corresponding collision and second-preimage-resistance strengths for SHAKE128 are min(d/2,128) and min(d,128) bits, respectively (see Appendix A.1 of [SHA3]). And the corresponding collision and second-preimage-resistance strengths for SHAKE256 are min(d/2,256) and min(d,256) bits, respectively. In this specification, we use d=256 (for SHAKE128) and d=512 (for SHAKE256)."

The length is based on the calculations in Appendix A.1 of FIPS PUB 202. 256 and 512 give are the minimum lengths required to meet 128 bit and 256 bit security respectively, and as a result encountering an OID for either SHAKE128 or SHAKE256 means a digest of length 256 and 512 bits respectively. If there's an arbitrary length involved the -len OIDs are used. The OIDs are defined here:

https://csrc.nist.gov/projects/computer-security-objects-register/algorithm-registration#Hash

They get used as digests, we've been using them as digests in BC Java/C# since 2015, I know it might seem weird, but they're digests. As I mention somewhere else, in PSS they are even used as both digest and XOF (if SHAKE is the signature digest, the mask-generator function is not MGF1 but SHAKE). Anyway, they're digests.

As an interesting aside, the section of the extendible hashes finishes with:

"If d > r + c/2, then SHAKE128 and SHAKE256 provide more than 128 and 256 bits of preimage
resistance, respectively; moreover, if d > 1600, a preimage probably does not exist."

A very brave statement, especially for NIST!

I guess the other, cautionary note, I should add - these functions all come with padding. Defining the API in terms of squeeze() and absorb() makes it look like an internal API which is being made public, it's likely the CMVP will complain about it, especially as they'll be expecting them to look like digests with a bit extra.

[ounsworth]

Yeah, see, that comment directly and emphatically contradicts FIPS 202 appdx. A.2, and that makes me comfortable.

[dghgit]

See the last paragraph of A.2. The comment is correct.

[ounsworth]

The branch name and PR description says that this PR implements cSHAKE from SP 800-185, but I don't see any cshake in the actual diff. I'm confused.

[dghgit]

cSHAKE, TupleHash, ParallelHash, and KMAC now added.