Explain mechanics and use of “Group batching”

[s5k6]

This comes from here, but I have slightly touched up the wording.

I don't get the description of “group batching” in the README.

Reading the text with muy current (assumed) understanding in mind, it
does not seem incorrect, but it does not convey (to me) how “group
batching” works, how it is intended to be used, and how it relates to
“a "cache" functionality”.

By observation, I came up with my own assumption about how this
might be intended to work. I might be wrong, though!

Here's my take:

clc-stackage tries to build all Haskell packages listed in a given
stack snapshot.

To speed things up, clc-stackage groups the packages into batches,
then iterates over them, for each batche trying to build all
contained packages. If one package fails, then the entire batch
fails, and all packages in that batch are marked as failure.

By default (i.e., without using --batch), clc-stackage will put
all packages into one single batch. This is supposed to be
quick, but one failing package makes the entire batch fail, thus
reporting all packages as failed. So this gives you a yes/no
answer, but is not enough to single out the failing packages. To do
that, you will need to modify the batch sizes.

With --batch 100, batches of 100 packages each are created (except
for the last one, I guess). This should give you better performance
than with --batch 1, but may still report excessive failures. If,
say, 5 failing packages exist and are distributed over 2 batches,
then clc-stackage still reports 200 failed packages.

After running with batches of 100, choose a smaller batch size (say,
--batch 10) and also add --retry-failures. This will only retry
the 200 packages reported as failig from the previous run,
rearranged into smaller batches. If 5 failing packages are
distributed over 4 batches-of-10, then only their 40 packages are
reported as failure now. We are narrowing down.

Finally, run with --batch 1 --retry-failures, which will only retry
the 40 failed packages from above, and report on them individually.

Obviously, one can try to optimise the number of steps, and the
batch sizes used.

Is this correct?.

Also, why is working in batches at all supposed to be faster than just
iterating over the packages?

[Bodigrim]

CC @tbidne

[tbidne]

Hi @s5k6, my apologies for taking so long to get to this.

Essentially yes, you are correct. The --batch argument was introduced solely to work around problems encountered when trying to build everything at once. It is never faster, and the output will not tell you which packages actually failed. For that, you will need to check the logs, which you always had to do anyway. How do you feel about changing the summary to this:

Click to expand docs

---

Group batching

By default, the clc-stackage exe tries to build all packages at once i.e. every package is written to generated/generated.cabal. This can cause problems e.g. we do not have enough memory to build everything simultaneously, or we receive an error that gcc has been given too many arguments. Hence we provide the --batch N option, which will split the package set into disjoint groups of size N. Each group is then built sequentially.

The default behavior is:

1. clc-stackage will try to build everything in the same group, even if some package fails (equivalent to cabal's --keep-going flag.). If instead --package-fail-fast is enabled, the first failure will cause the entire group to immediately fail, and we will move onto the next group.

2. clc-stackage will try every group, even if some prior group fails. The --group-fail-fast option changes this so that the first failure will cause clc-stackage to exit.

Each time a group finishes (success or failure), stdout/err will be updated, and then the next group will start. If the group failed to build and we have --write-logs save-failures (the default), then the logs and error output will be in ./output/logs/<pkg>/, where <pkg> is the name of the first package in the group.

When clc-stackage itself finishes (either on its own or via an interrupt like CTRL-C), the results are saved to a cache i.e. we record all successes, failures, and untested packages. This allows us to pick up where we left off with untested packages (including failures if the --retry-failures flag is active).

[!IMPORTANT]

The cache operates at the batch group level, so only packages that have been part of a successful group will be considered successes. Conversely, a package will be considered a failure if it is part of a failing group, even if it was built successfully. Therefore, to see what packages actually failed, we will want to check the logs directory. Alternatively, we can first run clc-stackage initially with a large --batch group (for maximum performance), then run it again with, say, --batch 1.