Read compressed CSVs

[VisruthSK]

This PR was generated by Codex.

Submission Checklist

• Run unit tests
• Declare copyright holder and agree to license (see below)

Summary

Support compressed CSVs. Closes #1027.

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Visruth Srimath Kandali

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.43%. Comparing base (b8ba71f) to head (40029b5).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1217      +/-   ##
==========================================
+ Coverage   92.40%   92.43%   +0.02%     
==========================================
  Files          15       15              
  Lines        6570     6568       -2     
==========================================
  Hits         6071     6071              
+ Misses        499      497       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jgabry]

Thanks for keeping this up to date. I haven't had a chance to seriously look at this yet, although I'm still interested in it. Trying to make a push on the compilation/c++ options stuff first

[jgabry]

This looks quite good. I committed a couple of small improvements and the missing doc. But there's one bigger issue we should probably deal with, even though it's probably uncommon. I asked codex about it:

With the current gzip/bzip2 -dc | grep a truncated or damaged archive can produce part of the CSV before the decompressor fails. grep can still exit successfully, so fread() treats the command as successful and read_cmdstan_csv() returns only the rows it received. I reproduced this with both gzip and bzip2.

Maybe we should run gzip -t or bzip2 -t once per compressed file before reading it? And if that fails then we throw an informative error message (and leave the file unmodified)?

That said, this would add a fourth decompression (metadata, sampler diagnostics, variables, and now this check), which is annoying. But we could probably keep it to three by reading all the requested diagnostic and variable columns once. Something like this (although I haven't checked if there are subtleties I'm overlooking):

selected <- c(sampler_diagnostics, variables)
csv_data <- data.table::fread(cmd = fread_cmd, select = selected, ...)

diagnostics <- csv_data[sampler_diagnostics]
draws <- csv_data[variables]

What do you think?

Also, after we merge this we could extend to #1276, which would make this much more convenient. But that should be done separately I think

[VisruthSK]

Agree, that approach seems good too to collapse to three reads. I think it makes sense to stack 1276 with this so they are merged together? Or to do 1276 in this PR too.

[jgabry]

I think I'd prefer stacking rather than combining, but not super strongly, so whatever you prefer is fine