Fix/mem alloc mismatch by sameehj · Pull Request #11397 · wolfSSL/wolfssl

sameehj · 2026-09-08T09:37:35Z

Description

Fixes spurious Free/Alloc mismatch failures reported by the mem-fail nightly (WOLFSSL_MEM_FAIL_COUNT). Two independent fixes:

memory.c — don't count a failed allocation. wolfSSL_Malloc() calls wc_MemFailCount_AllocMem() (which increments mem_fail_allocs) before invoking the registered allocator. When a caller-installed failing allocator (wolfSSL_SetAllocators(), used by several OOM unit tests) or a genuine OOM returns NULL, the allocation was counted but no block exists to free, so Total ends up one higher than Frees. Because this happens in the baseline counting phase, it reproduces identically in every shard. The fix undoes the count when the allocator returns NULL. All changes are under #ifdef WOLFSSL_MEM_FAIL_COUNT and do not affect production builds.
test_ProcessChainOCSPRequest_bounds — restore SSL-owned cert buffers. The first vector nulled ssl->buffers.certChain/certificate to drive the "chain == NULL" path, but the SSL owns the certificate DER copy (weOwnCert), so nulling it outright leaked 1536 bytes (a real leak, confirmed with leaks). The buffers are now saved, cleared for the call under test, and restored so wolfSSL_free() releases them.

Affected Class A tests now balanced: test_TLSX_SecureRenegotiation_parse, test_TLSX_CSR_parse, test_TLSX_SupportedGroups_parse, test_TLSX_KeyShare_gen, test_tls_msgtype_tca_new_alloc, test_TLSX_CSR_Parse_bounds, test_ProcessChainOCSPRequest_bounds.

Fixes zd# (N/A — tracked via the mem-fail nightly / jenkins-supervisor; add the ticket number if there is one)

Testing

Built with a near-nightly config and the mem-fail flag:

./configure --disable-shared --enable-static --enable-all \
    --enable-secure-renegotiation --enable-debug \
    C_EXTRA_FLAGS="-DWOLFSSL_MEM_FAIL_COUNT"
make -j tests/unit.test

Reproduced each mismatch on the unpatched tree (e.g. test_TLSX_SecureRenegotiation_parse → Total 184 / Frees 183).
After fix: all 7 tests report Total == Frees; unrelated tests unaffected (8/8, 282/282, 19/19).
Injection loop (MEM_FAIL_CNT=1..MAX) on a fixed test: every iteration balanced, no new crashes.
Leak check (macOS leaks --atExit) on test_ProcessChainOCSPRequest_bounds: 1 leak (1536 bytes) before → 0 leaks after.

Checklist

added tests — N/A; fixes existing test/instrumentation, no new test needed
updated/added doxygen — N/A
updated appropriate READMEs — N/A
Updated manual and documentation — N/A

wc_MemFailCount_AllocMem() increments mem_fail_allocs before the registered allocator runs. When a caller-installed failing allocator (via wolfSSL_SetAllocators(), used by several OOM unit tests) or a genuine out-of-memory returns NULL, the allocation is counted but no block exists to free. The mem-fail nightly then reports a spurious "Free/Alloc mismatch" (Total one higher than Frees), even in the baseline counting phase, which is why it repeats identically across shards. Undo the count when the allocator returns NULL so Total stays balanced with Frees. The injected-failure path returns before the allocator is called and so is unaffected. All changes are under WOLFSSL_MEM_FAIL_COUNT and do not affect production builds. Signed-off-by: Sameeh Jubran <sameeh@wolfssl.com>

…_bounds The first test vector nulls ssl->buffers.certChain and ssl->buffers.certificate to drive the "chain == NULL" path of ProcessChainOCSPRequest(). The SSL owns the certificate DER copy that wolfSSL_new() allocated (weOwnCert), so clearing the pointer outright leaked 1536 bytes and the mem-fail nightly reported a Free/Alloc mismatch. Save the owned buffers, clear them for the call under test, then restore them so wolfSSL_free() releases them. Signed-off-by: Sameeh Jubran <sameeh@wolfssl.com>

github-actions · 2026-09-08T09:54:46Z

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

FLASH: .text +576 B (+0.3%, 187,196 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

FLASH: .rodata +8 B, .text +896 B (+0.1%, 783,780 B / 1,048,576 B, total: 75% used)

gcc-arm-cortex-m4-pq

FLASH: .text +448 B (+0.1%, 302,088 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

FLASH: .text +448 B (+0.1%, 332,896 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-tls13

FLASH: .text +512 B (+0.2%, 242,639 B / 262,144 B, total: 93% used)

gcc-arm-cortex-m7-pq

FLASH: .text +448 B (+0.1%, 303,048 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

FLASH: .text +448 B (+0.2%, 242,639 B / 262,144 B, total: 93% used)

linuxkm-pie

Data: __patchable_function_entries +40 B (+0.1%, 27,040 B)

linuxkm-standard

Data: __patchable_function_entries +64 B (+0.1%, 49,848 B)

stm32-sim-stm32h753

FLASH: .text +24 B (+0.0%, 188,604 B / 2,097,152 B, total: 9% used)
No memory changes detected for:
gcc-arm-cortex-m0plus
gcc-arm-cortex-m3
gcc-arm-cortex-m4
gcc-arm-cortex-m4-baremetal
gcc-arm-cortex-m4-crypto-only
gcc-arm-cortex-m4-min-ecc
gcc-arm-cortex-m4-pkcs7
gcc-arm-cortex-m4-sp-math
gcc-arm-cortex-m4-tls12
gcc-arm-cortex-m7

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #11397

Scan targets checked: wolfcrypt-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src
Findings: 1

Required changes (1)

WOLFSSL_TRAP_MALLOC_SZ early return still leaves a counted allocation

File: wolfcrypt/src/memory.c:374
Function: wolfSSL_Malloc
Category: Incorrect error handling

The WOLFSSL_TRAP_MALLOC_SZ oversize check returns NULL directly, bypassing both the new res == NULL decrement at line 430 and the WOLFSSL_FORCE_MALLOC_FAIL_TEST one, so mem_fail_allocs keeps a count for a block that never exists. The Total/Frees mismatch this PR fixes persists in WOLFSSL_TRAP_MALLOC_SZ + WOLFSSL_MEM_FAIL_COUNT builds.

Related known finding #9957 (similar but distinct): Both affect wolfSSL_Malloc error paths under optional memory-test configurations, but this candidate's trap-size early return omits the allocation-failure count decrement, while #9957 frees an adjusted interior pointer in the forced-failure path with memory-zero checking. The operations, root causes, and required patches differ.

Recommendation: Call wc_MemFailCount_AllocFailed() under #ifdef WOLFSSL_MEM_FAIL_COUNT before this return NULL, or route the trap check through the shared NULL-return path.

Referenced code: wolfcrypt/src/memory.c:374-377 (4 lines)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #11397

Fenrir already completed a review of this PR at commit a1dd273bc8c8 (run 2333); its findings are the review threads on the PR. Push new commits to get a re-review of what changed, or comment @wolfSSL-Fenrir-bot review force to run the full review again at this commit.

wolfSSL_Malloc() counts the allocation via wc_MemFailCount_AllocMem() before invoking the allocator. The WOLFSSL_TRAP_MALLOC_SZ oversize check returns NULL after that count, bypassing the res == NULL and WOLFSSL_FORCE_MALLOC_FAIL_TEST decrements, so a WOLFSSL_TRAP_MALLOC_SZ + WOLFSSL_MEM_FAIL_COUNT build keeps a count for a block that never exists and Total ends up one higher than Frees. Undo the count on this path too. Guarded by WOLFSSL_MEM_FAIL_COUNT; no effect on production builds. Signed-off-by: Sameeh Jubran <sameeh@wolfssl.com>

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #11397

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-bugs

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

wolfSSL_Free(NULL) releases nothing, yet wc_MemFailCount_FreeMem() was incrementing mem_fail_frees unconditionally. Now that a failed wolfSSL_Malloc() no longer counts an allocation, a NULL free would push Frees above Total. Guard the free counter with ptr != NULL, matching the existing guard already used on the wolfSSL_Realloc() path. Under WOLFSSL_MEM_FAIL_COUNT only; no effect on production builds.

sameehj · 2026-09-11T09:42:40Z

retest this please

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #11397

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs
Unchanged since last review (not re-run): wolfssl-bugs

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

wolfSSL-Fenrir-bot · 2026-09-11T09:56:38Z

+     * releases nothing (ISO/IEC 9899:2018 7.22.3.3: free(NULL) is a no-op),
+     * and a failed wolfSSL_Malloc() no longer counts an allocation, so a NULL
+     * free must not be counted either or Frees would exceed Total. */
+    if (ptr != NULL) {


NULL-free fix leaves failed realloc count unbalanced · Logic errors

wolfSSL_Realloc(NULL, n) still counts a failed positive-size allocation. The new NULL-free guard removes the compensating count, so valid cleanup reports Total > Frees; known #9956 instead covers the memory-zero realloc crash.

Related earlier finding on this pull request (similar but distinct): Both concern false memory-failure counter mismatches caused by treating failed allocations and NULL frees inconsistently. However, the historical issue faults in wolfSSL_Malloc()/wolfSSL_Free(NULL), while this candidate faults in wolfSSL_Realloc(NULL, size) after failed allocation; its missing realloc failure-count update requires a distinct patch.

Suggested fix: When wolfSSL_Realloc(NULL, size) returns NULL for positive size, call wc_MemFailCount_AllocFailed() before returning.
Basis: POSIX realloc(): when ptr is null, realloc is equivalent to malloc, and allocation failure returns a null pointer.

Fixed as suggested.

wolfSSL_Realloc() counts an allocation via wc_MemFailCount_AllocMem() before calling realloc. realloc(NULL, n) is malloc; when it returns NULL for a positive size, no block exists to free. The NULL-free guard no longer compensates, so Total stays one higher than Frees. Undo the count on this path. Do not undo a failed realloc of a live pointer: that path still counts a free and the original block remains. Guarded by WOLFSSL_MEM_FAIL_COUNT; no effect on production builds. Co-authored-by: Cursor <cursoragent@cursor.com>

sameehj added 2 commits September 8, 2026 12:36

sameehj requested a review from wolfSSL-Fenrir-bot September 8, 2026 09:40

sameehj self-assigned this Sep 8, 2026

wolfSSL-Fenrir-bot requested changes Sep 8, 2026

View reviewed changes

sameehj assigned wolfSSL-Bot and unassigned sameehj Sep 8, 2026

sameehj mentioned this pull request Sep 8, 2026

tests: reset freed pointers in mem-fail TLS tests to avoid double-free #11398

Open

4 tasks

sameehj requested a review from wolfSSL-Fenrir-bot September 9, 2026 15:08

wolfSSL-Fenrir-bot reviewed Sep 9, 2026

View reviewed changes

JacobBarthelmeh previously approved these changes Sep 9, 2026

View reviewed changes

JacobBarthelmeh requested a review from wolfSSL-Fenrir-bot September 9, 2026 21:01

wolfSSL-Fenrir-bot requested changes Sep 9, 2026

View reviewed changes

Comment thread wolfcrypt/src/memory.c

sameehj dismissed JacobBarthelmeh’s stale review via a0cc34a September 10, 2026 08:55

sameehj requested review from JacobBarthelmeh and wolfSSL-Fenrir-bot September 11, 2026 09:45

wolfSSL-Fenrir-bot requested changes Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

Fix/mem alloc mismatch - #11397

Fix/mem alloc mismatch#11397
sameehj wants to merge 5 commits into
wolfSSL:masterfrom
sameehj:fix/mem-alloc-mismatch

sameehj commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

wolfSSL-Fenrir-bot left a comment

Uh oh!

wolfSSL-Fenrir-bot left a comment

Uh oh!

wolfSSL-Fenrir-bot left a comment

Uh oh!

Uh oh!

sameehj commented Sep 11, 2026

Uh oh!

wolfSSL-Fenrir-bot left a comment

Uh oh!

wolfSSL-Fenrir-bot Sep 11, 2026

Uh oh!

sameehj Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sameehj commented Sep 8, 2026

Description

Testing

Checklist

Uh oh!

github-actions Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MemBrowse Memory Report

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #11397

Required changes (1)

WOLFSSL_TRAP_MALLOC_SZ early return still leaves a counted allocation

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #11397

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #11397

Uh oh!

Uh oh!

sameehj commented Sep 11, 2026

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #11397

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Sep 8, 2026 •

edited

Loading