Fix predict_data problem size so prediction works when test points >= training points#59
Open
fonzie42 wants to merge 1 commit into
Open
Fix predict_data problem size so prediction works when test points >= training points#59fonzie42 wants to merge 1 commit into
fonzie42 wants to merge 1 commit into
Conversation
…crash when test >= train
PredictionSetupHelper set ProblemSize from the training locations only
(data->GetLocations()->GetSize() == N_train). The combined observed+missing
system actually has N_train + N_test points, so the Z descriptor was
undersized; with N_test >= N_train the derived observed count goes negative,
aborting Chameleon ("Too many tiles" / malloc failure).
Set ProblemSize per operation in the shared helper:
- prediction / MLOE-MMOM / IDW -> train + test (observed + missing)
- Fisher -> train (observed only)
Using train+test for Fisher would over-size its covariance and break the
Cholesky factorization; train preserves the original released Fisher value
exactly, so Fisher cannot regress. The prediction path now matches the
behavior validated across the thesis's cross-machine runs.
Adds TestRPredictProblemSize.cpp: predicts with more test points than
training points and checks the kriging interpolation property (prediction
at an observed location recovers the observed value).
46dd176 to
25788f5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
predict_data(and the other R prediction entry points) aborts insideChameleon when the number of test (missing) locations is comparable to or
larger than the number of training (observed) locations. With 9 training points
and 12 test points it fails with:
The existing R-adapter test never triggers this because it uses 16 training
points and only 2 test points (
test << train).Root cause
PredictionSetupHelper(src/Rcpp-adapters/FunctionsAdapter.cpp) sets theproblem size from the training locations only:
The R adapter passes train and test as two independent datasets, so the
invariant
ProblemSize = N_observed + N_missingis broken. The observed countis later derived as
GetProblemSize() - GetUnknownObservationsNb()=N_train - N_test, which goes negative whenN_test > N_train. Thenegative count becomes a huge unsigned descriptor dimension → "Too many tiles" /
malloc failure. (For
N_test < N_trainit is silently too small rather thannegative, which under-sizes the Z descriptor.)
Why a blanket
train + testis not enoughThe same
ProblemSizeis consumed under two different conventions by theoperations that share this helper:
predict/mloe_mmom/idwsize their Z descriptors over the combinedobserved + missing set, so they need
train + test.fisherbuilds a covariance over the observed locations only(
InitiateFisherDescriptorsusesGetProblemSize()directly), so it needstrain. Feeding ittrain + testover-sizes the covariance and breaks theCholesky factorization (
CHAMELEON_dpotrf_Tile Failed, Matrix is not positive definite).Fix
Set the problem size per operation in the shared helper:
The Fisher branch sets
ProblemSize = train_data_size, which is identical tothe original value (
data->GetLocations()->GetSize()returns the trainingcount, because only training locations are loaded into
data). Fisher's sizingis therefore unchanged from the released behavior and cannot regress.
Test
Adds
TestRPredictProblemSize.cpp: predicts with more test points thantraining points (9 train / 12 test) and checks the fundamental kriging
interpolation property: prediction at an observed location must recover the
observed value for an exact, nugget-free model. It uses the non-nugget
univariate_matern_stationarykernel so it is independent of any otherkernel-specific behavior.
values.
Validation
The fix follows directly from the descriptor-sizing code paths:
predict/mloe_mmom/idw, the Z descriptor must span the combinedobserved + missing set, so
ProblemSize = train + testis the correct size;the original
train-only value under-sizes it (and goes negative whentest >= train).fisher, the branch setsProblemSize = train_data_size, which is thesame value the original code produced (
data->GetLocations()->GetSize()returns the training count). Fisher's sizing is therefore unchanged and
cannot regress.
Note
Companion fix: the
UnivariateMaternNuggetsStationaryindex-order fix (#58). The two are independent code changes (disjoint files); both are neededtogether for fully correct end-to-end kriging predictions with the nuggets
kernel.