Skip to content

Integration review: preconditions, support-sized algorithms, and objective weighting#32

Merged
timholy merged 22 commits into
mainfrom
teh/integration-review-consolidated
Jul 19, 2026
Merged

Integration review: preconditions, support-sized algorithms, and objective weighting#32
timholy merged 22 commits into
mainfrom
teh/integration-review-consolidated

Conversation

@timholy

@timholy timholy commented Jul 19, 2026

Copy link
Copy Markdown
Member

This branch is the result of a systematic integration review — reading the
package against the interfaces it depends on (LinearAlgebra, SparseArrays, JuMP,
Ipopt, Unitful, Documenter) and against its own documented contracts. It groups
into four themes, plus documentation.

Each commit is self-contained.

1. Preconditions that were assumed but never checked

The recurring failure mode: input that violates an unstated assumption produced a
plausible-looking wrong answer instead of an error.

  • abs.(A) symmetry at the sym entry points. A symmetric cover reads one
    member of each index pair, so an asymmetric matrix silently yielded a cover of
    some unspecified symmetrization of itself. The check is on magnitudes (so a
    complex Hermitian passes) and allows ASYMMETRY_ULPS = 8, since a matrix
    symmetric in exact arithmetic lands a ULP or two off after ordinary
    floating-point work. Storage that makes the property structural is exempt;
    banded storage is not, and asymmetric banded input goes through the asymmetric
    cover family instead.
  • Solver termination status. A solve that stopped for any reason other than
    success left the model holding a point that does not solve the problem posed,
    and the values were read out anyway. An asymmetric input makes the AbsLog{1}
    LP unbounded, and what came back was the base of a ray.
  • The heuristic covers' penalty slot. Ignored is not unchecked — an
    unannotated slot accepted anything, so a wrong first argument silently produced
    a cover.
  • Affine, context, and fixed Unitful units. An affine unit measures from a
    shifted origin, so no product a[i]*b[j] covers an entry in it. Such input was
    accepted and silently reduced to its underlying atom, leaving Unitful stricter
    than this extension.
  • Axis agreement in cover_objective. Scales shorter than A previously
    scored a sub-grid without complaint.

2. Algorithms sized by the support rather than by the grid

The package documents the sparse path as the one to use when nnz ≪ n², but
five separate layers still scanned the full m×n grid. Each now reads the matrix
through foreach_support / foreach_support_sym, gathered once per call.

Layer Before After (n = 200/400/800, nnz ≈ 7n)
Soft-cover kernels (20 sweeps) 3.3 / 17.3 / 43.7 ms 0.16 / 0.40 / 0.81 ms
:leaveout initializer 3.5 / 11.8 / 44.8 ms 0.20 / 0.50 / 1.08 ms
Native AbsLog{2} (:lsqr, allocation) 1.13 / 3.47 / 12.0 MB 0.71 / 1.5 / 3.02 MB
JuMP/Ipopt model build (gather, n = 200→400) 0.31 → 1.22 MiB 0.119 → 0.269 MiB

The consistent shape of the win is that cost now doubles with nnz instead of
quadrupling with n. Supporting changes: a GroupedSupport compressed
neighbor-list type; shared _edge_list / _sym_edge_list / _degrees helpers in
the main package so both solver extensions use them; and an asymmetric
foreach_support for Symmetric/Hermitian over a sparse parent, without which
those types fell through to the generic full-grid scan.

cover_objective itself also reads through the traversal now, adding the
off-support entries back as a single count-weighted term — every entry outside
the support has r = 0 whatever the scales, so they share one penalty value.

3. One objective-weighting convention

The symmetric objective sums over the full grid: each off-diagonal pair counts
twice, each diagonal entry once. foreach_support_sym reports each pair once, so
a caller must supply w = (i == j) ? 1 : 2 itself. This is now documented on the
traversal and applied uniformly.

The bug it exposed: symcover_min!(::AbsLinear{1}) and (::AbsLinear{2}) in the
Ipopt extension summed over the i <= j triangle, which weights a diagonal entry
the same as an off-diagonal pair even though the pair stands for two entries of
A — effectively double-counting the diagonal. The two conventions have
different minimizers, so symcover_min was ranking its multistart candidates by
an objective none of them had minimized.

Most matrices do not distinguish the two: a binding constraint has zero residual
at the optimum, and a zero residual is weight-independent. The new tests pin a
matrix that does, and check the native and HiGHS solvers against reference models
whose constraints are imposed on the full grid explicitly.

4. Interface compliance and error quality

  • scalar_type is a new public hook. cover_objective took its accumulator
    from a promotion of all three operands, which has no concrete result for
    dimensional element types. Since a cover requires
    unit(A[i,j]) == unit(a[i])*unit(b[j]), every ratio is dimensionless and the
    score is a plain number; the accumulator now comes from that scalar type. The
    Unitful extension supplies the method for matrices whose entries carry
    different units.
  • Tolerances scale by eps of the element type. Float64-scaled literals were
    unreachable in Float32 — every descent ran to maxiter and multistart could
    never switch off its first candidate — and stopped far short of what a wider
    type resolves. At BigFloat, the alternating least squares now converges roughly
    thirteen digits further. Float64 thresholds are unchanged.
  • Complex Hermitian sparse input reached symcover_min but fell back to the
    generic O(n²) getindex walk, since the traversal specialized only on the real
    case. Only abs of a stored value is ever read, so the real bound was never
    required.
  • Mismatched scale element types surfaced as a MethodError naming an
    unexported internal; they are promoted internally now.
  • An empty strategy menu reported "no strategy yields a starting cover of A",
    which describes a different failure. The asymmetric drivers already got this
    right.

5. Documentation

  • Both Julia blocks in the manual called symcover_min on a non-square matrix and
    had never been run. They are jldoctest now, so CI keeps them honest.
  • checkdocs=:public, so the extension hooks — marked public but not exported —
    are actually checked. Falls back to :exports on Julia 1.10, which lacks
    Base.ispublic.
  • A worked example in the README, which had none.
  • Ipopt added to the docs environment.

Compat

JuMP lower bound raised 11.15, for the unified nonlinear interface that
supersedes @NLobjective/@NLconstraint. The legacy macros cannot be mixed with
the unified ones, a hazard symcover_min!(::AbsLinear{1}) was already exposed to.
JuMP 1.15 supports Julia 1.6+, so the LTS is unaffected.

timholy added 22 commits July 19, 2026 01:34
A cover requires unit(A[i,j]) == unit(a[i])*unit(b[j]), so every ratio
cover_objective forms is dimensionless and the score is a plain number.
Take the accumulator type from that scalar type rather than from a common
promotion of the three operands, which for dimensional element types has
no concrete result.

The scalar type is exposed as the public hook scalar_type, since a matrix
whose entries carry different units has an abstract eltype for which real
and oneunit are undefined; the Unitful extension supplies the method. When
eltype is not concrete at all the elements themselves are consulted.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
A solve that stops for any reason other than a solved one leaves the model
holding a point that does not solve the problem posed. Reading values from
it yields a vector that is a minimal cover in name only: an asymmetric
input makes the AbsLog{1} LP unbounded, and the point returned is the base
of a ray. Gate every solve on the termination status instead.

check_solved takes the status rather than the model so both solver
extensions share one definition without the main package depending on JuMP.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The convergence tests are on relative quantities, so their floors are set
by the precision of the element type. Float64-scaled literals cannot be
reached in Float32 -- every descent would run to maxiter, and multistart
selection could never switch off its first candidate -- and stop far short
of what a wider type resolves: at BigFloat the alternating least squares
now converges roughly thirteen digits further.

The multiples keep the Float64 thresholds where they were.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Reserves the slot: no implementation may rely on the return meaning
anything, so a traversal always runs to completion.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Unitful offers no public accessor for the atomic units of a FreeUnits and
no supported way to iterate them, so state the layout the code depends on
and why unit arithmetic cannot replace it.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The heuristic covers ignore the penalty, but ignored is not unchecked: an
unannotated slot accepted any value at all, so a typo'd or wrong first
argument silently produced a cover. The extension contract already requires
custom penalties to subtype AbstractCoverPenalty.

The Unitful extension's mirrored slots are annotated to match, so it does
not accept what the package rejects.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
symcover_min accepts Hermitian{<:Any,<:SparseMatrixCSC}, but the traversal
specialized only on the real case, so complex Hermitian input silently fell
back to the generic O(n^2) getindex walk. Only abs of a stored value is
ever read, and abs(A[i,j]) == abs(conj(A[j,i])), so the real bound was
never required.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The symmetric multistart drivers fell through an empty menu to "no strategy
yields a starting cover of A", which describes a different failure: that
every named strategy was tried and none applied. Check the menu up front,
as the asymmetric drivers already do.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The public entry points take plain AbstractVectors, but several internals
they route through required both scale vectors to share one element type,
so a mismatch surfaced as a MethodError naming an unexported internal
rather than as anything the caller could act on. Promote internally and let
each vector keep its own element type.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The extension hooks are marked public but not exported, so checkdocs=:exports
left them unchecked. :public rests on Base.ispublic, which Julia 1.10 lacks,
so there the check falls back to the exported set.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
An affine unit measures from a shifted origin, so no product a[i]*b[j]
covers an entry expressed in it; such input was accepted and silently
reduced to its underlying atom, leaving Unitful stricter here than this
extension. ContextUnits and FixedUnits are likewise named rather than
reaching an unexported internal as a MethodError.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
A symmetric cover reads one member of each index pair, so an asymmetric
matrix does not fail visibly: it yields a cover of some symmetrization of
itself, plausible-looking and wrong. Check the precondition where the sym
family is entered.

The predicate is on the magnitudes, which is what the traversal reads and
what admits a complex Hermitian, and it allows a few ULPs: a matrix that is
symmetric in exact arithmetic lands slightly off after ordinary floating-
point work, and that input is fine.

Types whose storage makes the property structural are exempt. Banded
storage is not: a Bidiagonal reads one off-diagonal as a structural zero,
so one with a nonzero band is never abs-symmetric, and a Tridiagonal
qualifies only when its two bands agree. Cover either as stored through the
asymmetric `cover` family. The banded traversal keeps its `max`, which now
reports the dominant of two entries already known to agree to within
ASYMMETRY_ULPS.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The symmetric objective sums over the full grid, so each off-diagonal pair
counts twice and each diagonal entry once. Code reading a matrix through
foreach_support_sym, which reports each pair once, must supply the factor
of 2 itself. The coverage constraints need no such correction, since
a[i]*a[j] >= |A[i,j]| and its transpose are the same constraint, so
imposing it on the i <= j triangle is equivalent to imposing it everywhere.

Bidiagonal and Tridiagonal are the exception: foreach_support_sym defines
their symmetrization as max(|A[i,j]|, |A[j,i]|), so for unequal bands the
sym solvers minimize the symmetrization's objective while cover_objective
scores the bands as stored. The cover is valid under either reading.

Test that the native and HiGHS solvers reach the same optimum as reference
models whose coverage constraints are imposed on the full grid explicitly,
which pins both halves of the convention. One test matrix is chosen so the
two weightings have different minimizers; for most symmetric matrices the
diagonal residuals vanish at the optimum and the two agree.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
cover_objective walked the full index grid, so scoring a sparse cover
cost O(length(A)) however small its support. It now reads A through
foreach_support and adds the off-support entries back as a single
count-weighted term: every entry outside the support has r = 0 whatever
its scales, so they share one penalty value, which AbsLog makes zero and
AbsLinear a nonzero constant.

That term is skipped entirely when the support is full, since a penalty
infinite at r = 0 would otherwise turn a dense matrix into 0 * Inf.

Also require eachindex(a) and eachindex(b) to match A's axes, as iscover
already does. Scales shorter than A previously scored a sub-grid without
complaint, and the entry count the zero term is derived from is only
meaningful once they match.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The seven descent kernels scanned the full m×n grid on every sweep, so a
sparse matrix cost O(n²) per sweep where its support is O(nnz). Each now
reads a GroupedSupport: a compressed per-group neighbor list built once per
call from foreach_support/foreach_support_sym.

_sym_support enters each off-diagonal pair in both orientations and the
diagonal once, so a group is a full symmetric row. That is the multiplicity
the full-grid objective convention requires, so the symmetric kernels need
no explicit off-diagonal weight and their arithmetic is unchanged.

The symmetric kernels also gain the eachindex(a) == axes(A, 1) check their
asymmetric counterparts already had: the gather is addressed by the matrix's
axes, so a mismatch must fail at the entry rather than inside a sweep.

On symmetric sparse input with nnz ≈ 7n, 20 sweeps cost 0.16/0.40/0.81 ms at
n = 200/400/800, against 3.3/17.3/43.7 ms for the scan they replace.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The workers behind symcover_min and cover_min built dense C and S arrays
covering the full grid, plus a dense W, so the :lsqr path allocated O(n²)
regardless of nnz — despite being the path documented for nnz ≪ n².

Both now build a flat edge list with a parallel cvals vector of log|A_ij|
from the support gather, and nothing n×n is allocated on that path. W was
removable outright: it only carried a weight between two loops over the same
edge list, so the loops merged. The feasibility boost and the balance shift,
which also scanned the grid through S, now walk the edges and a pair of
row/column support counts.

Symmetric and Hermitian over a sparse parent gain an asymmetric
foreach_support; without it they fell through to the generic full-grid scan
and saw none of this.

Allocation on symmetric sparse input with nnz ≈ 7n: 0.71/1.5/3.02 MB at
n = 200/400/800, against 1.13/3.47/12.0 MB before.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The :leaveout and :diagfeasible strategies scanned the full grid, so they
were O(n^2) on a sparse matrix while the strategies around them were not.
_leaveout_logmean_init! now takes one _sym_support gather and reads its
residual scans and Gauss-Seidel sweeps from it; boost_feasible_seq! gathers
off-diagonal pairs through foreach_support_sym and sorts them by (offset,
row), reproducing the propagation order it is defined by.

init_feasible_diag! fills the diagonal from the traversal rather than by
getindex, and the multistart start-set test for a zero entry counts the
support instead of scanning. Both restructured functions now check that A's
axes match a's, which the gather makes load-bearing.

At nnz ~ 7n, :leaveout runs 0.20/0.50/1.08 ms at n = 200/400/800 in place of
3.5/11.8/44.8 ms.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Every JuMP and Ipopt model builder materialized a dense Apos in position
space before extracting the support it actually needed, so building a model
was O(length(A)) regardless of nnz. They now build from a flat edge list,
with log-magnitudes in a vector parallel to the endpoints rather than in a
Dict keyed by position.

_edge_list, _sym_edge_list, and _degrees live in the main package so both
extensions share them. The symmetric list is the full-grid reading; a model
wanting the triangle takes its ei <= ej half, which is also the constraint
set.

At n = 200 -> 400 on symmetric sparse input the gather goes from 0.31 -> 1.22
MiB to 0.119 -> 0.269 MiB, doubling with nnz in place of quadrupling with n.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
symcover_min!(::AbsLinear{1}) and symcover_min!(::AbsLinear{2}) summed their
objective over the i <= j triangle, while soft_symcover_min!, cover_objective,
and every other solver in the package sum over the full grid. Per unordered
pair the triangle weights an off-diagonal pair and a diagonal entry alike, but
the pair stands for two entries of A, so the diagonal was effectively counted
twice. The two conventions have different minimizers, so symcover_min was
ranking its multistart candidates by an objective none of them had minimized.

All four symmetric builders now share one idiom: _triangle pairs the i <= j
half of the gather with the multiplicity w = (i == j) ? 1 : 2 each entry
stands for, the rule foreach_support_sym documents for callers. Carrying the
weight is the same objective as emitting both orientations at half the terms,
so the two soft builders shrink as well.

Most matrices do not distinguish the conventions: a binding constraint has
zero residual at the optimum, and a zero residual is weight-independent. The
new test pins one that does, and the Abasin fixture moved to a matrix that
still separates the :geomean and :hardcover basins under the corrected
objective.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
JuMP 1.15 extended @objective and @constraint to nonlinear expressions,
superseding @NLobjective and @NLconstraint. The legacy macros cannot be
mixed with the unified ones, a hazard symcover_min!(::AbsLinear{1}) was
already exposed to: it paired @NLconstraint with a plain @objective.

Raise the JuMP compat lower bound to 1.15 accordingly. That release
supports Julia 1.6 and later, so the LTS is unaffected.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
`set_silent` does not suppress the banner Ipopt prints from its C++ core
on the first solve of a session; the `sb` option is what covers it. Build
every model through a helper that sets both.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The two Julia blocks in the manual were never run by the doctest pass,
and neither worked as written: both called `symcover_min` on a matrix
that is not square. Give each its own symmetric matrix and convert them
to `jldoctest`, so the doctest run in CI keeps them honest. Add Ipopt to
the docs environment, which those blocks need, and raise the docs `JuMP`
compat to match the package's own floor.

Solver output is shown through `round`: truncating instead would sit a
digit away from flipping, since values like 4.999999990219944 and
5.000000001 truncate differently but round the same.

Add a worked example to the README, which had none.

Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
@timholy
timholy merged commit 330d02a into main Jul 19, 2026
3 checks passed
@timholy
timholy deleted the teh/integration-review-consolidated branch July 19, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant