Update snapshot to 'nightly'

Instead of pinning the snapshot to a specific date, we set it to
'nightly', which should allow more flexibility i.e. users can build
whichever ghc corresponds to the latest snapshot. This should
hopefully decrease the chance of users running into issues with
ghc minor version upgrades e.g. bounds errors.

Snapshot updates that upgrade the ghc major version will almost
certainly require manual intervention here, but failures will likely
be caught by the CI job that builds everything with --dry-run, so
there should be some warning.

Author: tbidne

.github/workflows/ci.yaml

@@ -29,7 +29,9 @@ jobs:
       - uses: actions/checkout@v4
       - uses: haskell-actions/setup@v2
         with:
-          ghc-version: "9.10.3"
+          # Should be the current stackage nightly, though this will likely go
+          # out-of-date eventually, until a problem is reported.
+          ghc-version: "9.12.3"
       - name: Configure
         run: |
           cabal configure --enable-tests --ghc-options -Werror

README.md

@@ -14,7 +14,7 @@ An impact assessment is due when
 
 The procedure is as follows:
 
-1. Rebase changes, mandated by your proposal, atop of `ghc-9.10` branch.
+1. Rebase changes, mandated by your proposal, atop the ghc branch (or tag) that corresponds to the current [stackage nightly](https://www.stackage.org/nightly). For example, if the latest snapshot ghc is `ghc-9.12.3`, we would want to rebase our changes on the `ghc-9.12.3-release` tag. See [GHC / Stack snapshot compatibility](#ghc--stack-snapshot-compatibility) for more info on choosing a good ghc.
 
 2. Compile a patched GHC, say, `~/ghc/_build/stage1/bin/ghc`.
 
@@ -36,6 +36,7 @@ The procedure is as follows:
     * You can interrupt `cabal` at any time and rerun again later.
     * Consider setting `--jobs` to retain free CPU cores for other tasks.
     * Full build requires roughly 7 Gb of free disk space.
+    * If the build fails with an error about max amount of arguments in `gcc`, run again, but with smaller batch size. 250 worked well for me.
 
     To get an idea of the current progress, we can run the following commands
     on the log file:
@@ -59,6 +60,46 @@ The procedure is as follows:
 
 8. When everything finally builds, get back to CLC with a list of packages affected and patches required.
 
+### GHC / Stack snapshot compatibility
+
+#### GHC Minor update
+
+As nightly changes ghc minor versions (e.g. `9.12.1 -> 9.12.3`), package bounds can change. This can be a problem because `clc-stackage` writes the snapshot's packages' version bounds as exact constraints to the generated `./generated/cabal.project.local`, to ensure we (transitively) build the same package version every time. While we attempt to mitigate such issues (boot packages' exact versions are ignored), it is possible for some to slip through.
+
+User mitigations can include:
+
+- Adding the package to [./excluded_pkgs.jsonc](excluded_pkgs.jsonc). The package will still be built as long as it is a (transitive) dependency of some other package in the snapshot, but will not have its exact bounds written to `cabal.project.local`.
+
+- Manually downloading a snapshot (e.g. `https://www.stackage.org/nightly/cabal.config`), changing / removing the offending package(s), and supplying the file with the `--snapshot-path` param. Like `excluded_pkgs.jsonc`, this only impacts what is actually written to `cabal.project.local`.
+
+- Adding constraints to `generated/cabal.project`. Even though we write most package versions as constraints, there is still some room for non-determinancy (hence, errors).
+
+  Boot packages are not written as exact versions, but it might be useful to write _some_ constraint e.g. `constraints: filepath > 1.5`.
+
+  We also might need to set some flags e.g. prevent cabal from setting the wrong automatic flag. For instance, depending on the resolver, cabal might try to build `bson` with the `_old-network` flag, which will not work with this package set, hence we disable it with `constraints: bson -_old-network`.
+
+#### GHC Major update
+
+When a snapshot upgrades the GHC *major* version, there is usually more work involved. In particular,
+
+- New system dependencies will need to be added to the `flake.nix`.
+
+- New packages will need to be added to `excluded_pkgs.jsonc` because e.g. the system deps are unreasonable or the packages are executables.
+
+Until this is done, `clc-stackage` will likely fail. Such failures should hopefully be caught by CI that is run daily, but either way, please open an issue if `clc-stackage` seems out of date with stackage nightly.
+
+While that is being resolved, the mitigations from the [previous section](#ghc-minor-update) may be useful.
+
+#### Misc
+
+The `flake.nix` line:
+
+```nix
+compiler = pkgs.haskell.packages.ghc<vers>;
+```
+
+can be a useful guide as to which GHC was last tested, as CI uses this ghc to build everything with `--dry-run`, which should report solver errors (e.g. bounds) at the very least.
+
 ### The clc-stackage exe
 
 `clc-stackage` is an executable that will:

cabal.project

@@ -1,5 +1,3 @@
-index-state: 2026-03-19T21:41:35Z
-
 packages: .
 
 program-options

dev.md

@@ -75,28 +75,24 @@ The executable that actually runs. This is a very thin wrapper over `runner`, wh
 
 ## Updating to a new shapshot
 
-1. Update to the desired snapshot:
+`clc-stackage` is based on `nightly` -- which changes automatically -- meaning we do not necessarily have to do anything when a new (minor) snapshot is released. On the other hand, *major* snapshot updates will almost certainly bring in new packages that need to be excluded, so there are some general "update steps" we will want to take:
 
-    ```haskell
-    -- CLC.Stackage.Parser.API
-    stackageSnapshot :: String
-    stackageSnapshot = "nightly-yyyy-mm-dd"
-    ```
+1. Modify [excluded_pkgs.json](excluded_pkgs.json) as needed. That is, updating the snapshot major verison will probably bring in some new packages that we do not want. The update process is essentially trial-and-error i.e. run `clc-stackage` as normal, and later add any failing packages that should be excluded.
 
-2. Update the `index-state` in [cabal.project](cabal.project) and [generated/cabal.project](generated/cabal.project).
+2. Update `ghc-version` in [.github/workflows/ci.yaml](.github/workflows/ci.yaml).
 
-3. Modify [excluded_pkgs.json](excluded_pkgs.json) as needed. That is, updating the snapshot will probably bring in some new packages that we do not want. The update process is essentially trial-and-error i.e. run `clc-stackage` as normal, and later add any failing packages that should be excluded.
+3. Update functional tests as needed i.e. exact package versions in `*golden` and `test/functional/snapshot.txt`.
 
-4. Update references to the current ghc e.g.
+4. Optional: Update nix:
 
-    1. `ghc-version` in [.github/workflows/ci.yaml](.github/workflows/ci.yaml).
-    2. [README.md](README.md).
+    - Inputs (`nix flake update`).
+    - GHC: Update the `compiler = pkgs.haskell.packages.ghc<vers>;` line.
+    - Add to the `flake.nix`'s `ldDeps` and `deps` as needed to have the `nix` CI job pass. System libs available on nix can be found here: https://search.nixos.org/packages?channel=unstable.
 
-5. Update functional tests as needed i.e. exact package versions in `*golden` and `test/functional/snapshot.txt`.
 
-6. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
+    This job builds everything with `--dry-run`, so its success is a useful proxy for `clc-stackage`'s health. In other words, if the nix job fails, there is almost certainly a general issue (i.e. either a package should be excluded or new system dep is required), but if it succeeds, the package set is in pretty good shape (there may still be sporadic issues e.g. a package does not properly declare its system dependencies at config time).
 
-7. Optional: Update nix inputs (`nix flake update`).
+5. Optional: Update `clc-stackage.cabal`'s dependencies (i.e. `cabal outdated`).
 
 ### Verifying snapshot
 

excluded_pkgs.jsonc

@@ -270,6 +270,7 @@
     "midi-music-box",
     "misfortune",
     "mmark-cli",
+    "mmark-ext", // ghc-syntax-highlighter
     "moffy-samples-gtk3",
     "moffy-samples-gtk3-run",
     "moffy-samples-gtk4",
@@ -359,6 +360,7 @@
     "sqlcli",
     "sqlcli-odbc",
     "sqlite-simple",
+    "stakhanov", // hasql
     "stack-all",
     "stack-clean-old",
     "stack-templatizer",

generated/cabal.project

@@ -1,16 +1,20 @@
-index-state: 2026-03-19T21:41:35Z
-
 -- should be an absolute path to your custom compiler
 --with-compiler: /home/ghc/_build/stage1/bin/ghc
 
 packages: .
 
 optimization: False
 
-allow-newer:
-  hoogle:crypton-connection,
-  web-rep:transformers,
+-- The generated cabal.project.local has exact constraints for every (non-boot)
+-- package in the snapshot to ensure we build the same packages every time.
+-- We still want this to override individual package constraints e.g.
+-- web-rep has had an overly tight bound that excludes the transformers from
+-- the snapshot. The allow-newer ignores the error.
+allow-newer: *:*
+allow-older: *:*
 
+constraints: bson -_old-network
 constraints: hlint +ghc-lib
 constraints: ghc-lib-parser-ex -auto
+constraints: path +os-string
 constraints: stylish-haskell +ghc-lib

src/CLC/Stackage/Builder.hs

@@ -5,6 +5,7 @@ module CLC.Stackage.Builder
 
     -- * Misc
     Batch.batchPackages,
+    Process.cabalUpdate,
   )
 where
 

src/CLC/Stackage/Builder/Process.hs

@@ -2,6 +2,7 @@
 
 module CLC.Stackage.Builder.Process
   ( buildProject,
+    cabalUpdate,
   )
 where
 
@@ -122,6 +123,26 @@ buildProject env idx pkgs = do
 
     addPackages = Set.union pkgsSet
 
+-- | Runs "cabal update".
+cabalUpdate :: BuildEnv -> IO ()
+cabalUpdate env = do
+  Logging.putTimeInfoStr env.hLogger "Running 'cabal update'"
+  P.readProcessWithExitCode env.cabalPath ["update"] "" >>= \(ec, out, err) ->
+    case ec of
+      ExitSuccess -> pure ()
+      ExitFailure _ -> do
+        let msg =
+              mconcat
+                [ "Failed running 'cabal update': ",
+                  "Out: '",
+                  T.pack out,
+                  "', Err: '",
+                  T.pack err,
+                  "'"
+                ]
+        Logging.putTimeErrStr env.hLogger msg
+        throwIO ec
+
 createCurrentLogsDir :: IO (OsPath, OsPath, OsPath)
 createCurrentLogsDir = do
   let dirPath = Paths.logsDir </> [osp|current-build|]