Skip to content

[Testing][QDP] f32 Parquet fidelity tests and benchmark#1422

Merged
rich7420 merged 5 commits into
apache:mainfrom
0lai0:test-1342-Parquet_f32
Jul 4, 2026
Merged

[Testing][QDP] f32 Parquet fidelity tests and benchmark#1422
rich7420 merged 5 commits into
apache:mainfrom
0lai0:test-1342-Parquet_f32

Conversation

@0lai0

@0lai0 0lai0 commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Closes #1342
Part of #1338

Changes

  • Bug fix
  • New feature
  • Refactoring
  • Documentation
  • Test
  • CI/CD pipeline
  • Other

Why

The f32 Parquet pipeline (A1–A3) lacked dedicated end-to-end fidelity coverage and throughput validation. Without tests, regressions in the f32 reader or dispatch path could go undetected, and without a benchmark, the expected 25–35% throughput improvement over the f64 path for native f32 columns remained unmeasured.

How

  • qdp-core/tests/parquet_f32_fidelity.rs: End-to-end f32-vs-f64 fidelity test for the Parquet pipeline.
    • CPU smoke(asserts in CI): f32 and f64 readers return the same logical values up to f64→f32 rounding.
    • GPU fidelity (Linux + CUDA, self-skip on no-CUDA stub via kernel probe): same deterministic data written as native List<Float32> / List<Float64>, read through matching readers, encoded with f32/f64 kernels, compared via fidelity_cross_precision. Thresholds match gpu_fidelity.rs (1e-3 at 8–16 qubits; amplitude f32 norm precision — the issue's "≥0.99999 where applicable" does not apply here).
  • qdp-core/tests/common/mod.rs: Shared List<Float32> / List<Float64> Parquet writers for fidelity fixtures.
  • qdp-python/benchmark/benchmark_parquet_f32.py + README.md: Optional local/GPU throughput benchmark (vec/s + f32/f64 speedup). Default run generates separate native f32 and f64 files so the f32 path exercises zero-copy native read; --parquet PATH benchmarks a single user file at both dtypes. Not wired into CI.
  • .github/workflows/python-testing.yml: Run parquet_f32 (existing [Feature] ParquetReader: f32/f64 columns + Arrow cast #1340 reader acceptance) and parquet_f32_fidelity in the rust-check job under QDP_NO_CUDA=1. CPU smoke asserts; GPU cases skip without a GPU. Scoped to these two binaries — other gpu_*.rs tests lack probe-skip and would fail on the stub.

Checklist

  • Added or updated unit tests for all changes
  • Added or updated documentation for all changes

@0lai0 0lai0 requested a review from 400Ping as a code owner June 28, 2026 14:18
@0lai0

0lai0 commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

cc @rich7420

@rich7420

rich7420 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@0lai0 thanks for the patch!
LGTM overall
pulled it down, tests pass and clippy's clean, GPU cases self-skip nicely. Threshold matching gpu_fidelity.rs (1e-3, not 0.99999) is the right call.

Two small nits, neither blocking:

  1. write_list_parquet_f32/write_list_parquet_f64 are near-identical (only the builder type differs). Not worth forcing a generic over Arrow builders, but figured I'd flag the dupe.

  2. In the benchmark, --parquet vec/s divides metadata num_rows by the time to iterate total_batches batches. Fine for the generated case where they line up, but a bit off for an arbitrary user file where batches*batch_size != num_rows.

@rich7420 rich7420 merged commit 58e4aff into apache:main Jul 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Testing] Parquet f32 tests and benchmark

2 participants