Tracking: sparse/CSR path follow-ups after #961

[FabianHofmann]

Note

The following content was generated by AI.

Describe the feature you'd like to see

Tracking issue for the CSR follow-ups after #961 (umbrella #756 is closed). Each child names a place where a CSR-backed expression silently densifies, or a gap in the frozen-constraint API. This issue fixes the order in which to address them.

Key observations

• LinearExpression.data (linopy/expressions.py:2358) is where a CSR-backed expression densifies on read. Other paths drop the backing without passing through it: @ returns dense when the input was dense and sparse_groupby is off (expressions.py:2609), CSRConstraint.to_dense()/mutable(), the rebuild at model.py:1345, and the return None fallbacks in _sparse_matmul, _try_csr_merge, _aligned and csr_rhs, which lose the reason.
• Grid holds only per-dim indexes. Aux coords live next to it in CSRLinearExpression.coords and are dropped by CSRConstraint.from_csr (constraints.py:1401), which is CSRConstraint drops auxiliary coordinates when a sparse expression becomes a constraint #941.
• Zero policy on the sparse path: every operation keeps explicit zeros (merge, scaling, sum, groupby, selection), so sparse and dense give the same terms up to order; only @/dot prunes them. Cell activeness is carried by const alone. Export culls zeros regardless; Zero culling forces model rebuilding #925 is an export/persistent-diff question and does not block this series.

Plan

Four pull requests, grouped by shared design rather than one per issue:

1. Expression kernel (parts A, C, B, D), feat(csr): keep CSR backing through metadata, constant arithmetic, sum/groupby and selection #973: observability, aux coords in Grid, and all operations that keep the backing. Merged.
2. Boundaries (parts E, F), feat(csr): constraint and export boundaries of the sparse path (PR 2 of #972) #974: constraint side and export. Merged.
3. Sparse soften (part S), feat(csr): native soften for frozen constraints #979 (closes Native sparse soften for frozen constraints (penalty= with freeze=True) #975): native soften / penalty= on frozen constraints, so the model key in PR 4 needs no special case for softening. Merged.
4. Control + docs (parts G, H): the configuration surface discussed in One Model-level switch for the sparse path instead of three independent opt-ins #976, and documentation, once the preserving set is final.

Part	Issues	Content	Depends on
A: observability	#962, #969 (parts 1+2)	Serve shape/sizes/coords/dims/isnull from the CSR store. .is_sparse, repr marker. Opt-in warn_on_densify through one _densify_notice(reason) helper, called at data, at each return None fallback and in CSRConstraint.to_dense. data stays a one-way conversion (no cached dense copy next to _csr, the setters would make it stale). Move aux coords into Grid.	–
C	#965	scaled_by / shifted for non-scalar constants, aligned through _matmul_operand_to_matrix	A
B	#964	sum(dim) by merging rows directly (CSRLinearExpression.aggregated), keeping explicit zeros. Chained groupby from a CSR source; fix the gate at expressions.py:613 to use coord_dims instead of self.data.dims.	A
D	#966, mask= part of #970	Row masking, sel through reindexed, isel as row gather. Keep _try_csr_merge sparse when aux coords differ across grids.	A
E: constraint side	#941, #963, rest of #970	Aux coords on CSRConstraint (trivial once in Grid), incl. netcdf round trip (io.py:1260). Explanatory errors for loc/update/from_rule naming .mutable(). Expression rhs moved to lhs.	D, #806
F: export	#968, then #967	flat/to_polars from the CSR store, modelled on CSRConstraint.to_polars. Sparse objective across objective.py, matrices.py, io.py, persistent/diff.py; decide where the objective name lives (CSRLinearExpression has no attrs).	B
S: sparse soften	#975, PR #979	Soften a CSRConstraint natively (one slack term per active row), drop the penalty + freeze error	E, F
G: control	#969 (part 3), #976	One read-only model key instead of the three opt-ins (semantics="v1", sparse=/sparse_groupby, freeze_constraints), as proposed in #976 (under discussion). Replaces the earlier plan of per-call sparse= on @/dot/merge; removes the coupling of @ to options["sparse_groupby"].	S, #976
H: docs	#971	API entries, user-guide section, list of operations that keep the backing	G

Ordering notes:

• Within PR 1, commit in the order A, C, B, D. PR 1 stays out of constraints.py so fix: support adding/removing variables after adding frozen constraint #806 can land before PR 2.
• Tests extend the existing helpers in test/test_csr.py (assert_frozen_equal, assert_terms_equal, assert_cells_equal, assert_contracted_equal) with a "backing kept" assertion instead of adding a new fixture.
• Out of scope for now: DataArray sign in to_constraint and non-scalar fillna still densify. groupby-sum pads every group to the largest group size #745 is likely covered by sparse groupby; re-check and close or narrow.

Children

• repr() silently and permanently densifies a CSR-backed LinearExpression #962
• Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) #969
• Keep CSR backing for elementwise ops with non-scalar constants #965
• Sparse sum(dim=...) and chained groupby on CSR-backed expressions #964
• Keep CSR backing through where/mask/sel/isel/loc #966
• Widen the CSRConstraint fast path in Model.add_constraints #970
• CSRConstraint drops auxiliary coordinates when a sparse expression becomes a constraint #941
• CSRConstraint lacks loc/update/from_rule and raises bare AttributeError #963
• Emit flat/to_polars directly from the CSR store #968
• Sparse path for the objective #967
• Native sparse soften for frozen constraints (penalty= with freeze=True) #975 (PR feat(csr): native soften for frozen constraints #979)
• One Model-level switch for the sparse path instead of three independent opt-ins #976
• Document the sparse expression path (CSRLinearExpression, sparse=True, sparse_groupby) #971