Skip to content

Conversation

@HJLebbink
Copy link
Member

@HJLebbink HJLebbink commented Nov 21, 2025

Adds checksum support (CRC32, CRC32C, SHA1, SHA256, CRC64NVME) with automatic verification and hardware acceleration.

Key Changes

  • Introduce validated wrapper types for S3 parameters following the "parse, don't validate" pattern
  • Add checksum algorithm support for CRC32, CRC32C, SHA1, SHA256, and CRC64NVME
  • Add ignore parameter to #[minio_macros::test] for conditionally skipping tests

Typed Parameters

New wrapper types in src/s3/types/typed_parameters.rs:

Type Description
BucketName Validated S3 bucket name (3-63 chars, DNS-compliant)
ObjectKey Validated S3 object key (1-1024 chars)
VersionId S3 version identifier
Region AWS region string
UploadId Multipart upload identifier
ETag Entity tag for object versioning
ContentType MIME content type

These types validate input at construction time, ensuring invalid values cannot propagate through the codebase. Once constructed, the value is guaranteed valid.

Performance

(10 runs each, mean ± stddev)
Size                    CRC32               CRC32C            CRC64NVME
----------------------------------------------------------------------
1 KB       13.1 ±  1.2 GiB/s   13.3 ±  0.0 GiB/s   12.8 ±  0.2 GiB/s
10 KB      62.6 ±  1.4 GiB/s   65.9 ±  0.3 GiB/s   61.7 ±  1.1 GiB/s
100 KB     81.9 ±  0.9 GiB/s  101.7 ±  0.3 GiB/s   80.4 ±  1.6 GiB/s
1 MB      110.4 ±  1.9 GiB/s  100.4 ± 16.4 GiB/s  111.1 ±  1.5 GiB/s
10 MB      35.5 ±  0.6 GiB/s   36.2 ±  0.7 GiB/s   37.5 ±  0.2 GiB/s

Size                      MD5                 SHA1               SHA256
----------------------------------------------------------------------
1 KB        587 ±   10 MiB/s    1.6 ±  0.1 GiB/s    1.5 ±  0.0 GiB/s
10 KB       638 ±   16 MiB/s    1.9 ±  0.0 GiB/s    1.7 ±  0.0 GiB/s
100 KB      647 ±   12 MiB/s    2.0 ±  0.0 GiB/s    1.8 ±  0.0 GiB/s
1 MB        661 ±    5 MiB/s    2.0 ±  0.0 GiB/s    1.8 ±  0.0 GiB/s
10 MB       600 ±   39 MiB/s    1.6 ±  0.1 GiB/s    1.7 ±  0.1 GiB/s

@HJLebbink HJLebbink requested a review from Copilot November 21, 2025 15:02
@HJLebbink HJLebbink self-assigned this Nov 21, 2025
@HJLebbink HJLebbink added the enhancement Used in release doc generation label Nov 21, 2025

This comment was marked as outdated.

@HJLebbink HJLebbink force-pushed the feature/hashes branch 2 times, most recently from d878166 to 10bd659 Compare November 21, 2025 15:22
@HJLebbink HJLebbink changed the title added CRC32, CRC32C, SHA1, SHA256 and CRC64NVME added checksum support CRC32, CRC32C, SHA1, SHA256 and CRC64NVME Nov 21, 2025
@HJLebbink HJLebbink requested a review from Copilot November 21, 2025 17:49

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 50 out of 50 changed files in this pull request and generated no new comments.

@HJLebbink HJLebbink marked this pull request as draft November 30, 2025 22:44
@HJLebbink HJLebbink force-pushed the feature/hashes branch 4 times, most recently from e31b006 to 65f88e8 Compare December 1, 2025 11:02
@HJLebbink HJLebbink requested a review from klauspost December 1, 2025 11:03
@HJLebbink HJLebbink marked this pull request as ready for review December 10, 2025 06:38
@HJLebbink HJLebbink marked this pull request as draft December 19, 2025 11:57
@HJLebbink HJLebbink force-pushed the feature/hashes branch 2 times, most recently from 6b07963 to 1c6c4e0 Compare December 19, 2025 13:15
@HJLebbink HJLebbink requested a review from Copilot December 19, 2025 13:15
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

@HJLebbink HJLebbink force-pushed the feature/hashes branch 3 times, most recently from eab43cc to 0811668 Compare December 19, 2025 14:01
@HJLebbink HJLebbink marked this pull request as ready for review December 19, 2025 14:03
@HJLebbink HJLebbink force-pushed the feature/hashes branch 2 times, most recently from c95cf98 to 189de92 Compare December 19, 2025 17:35
@HJLebbink HJLebbink requested a review from Copilot December 22, 2025 18:00
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

@HJLebbink HJLebbink force-pushed the feature/hashes branch 3 times, most recently from 2682f85 to 884a480 Compare December 22, 2025 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Used in release doc generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant