Skip to content

Make dlsym O(1) in the number of exports - #28125

Open
basvandijk wants to merge 1 commit into
emscripten-core:mainfrom
basvandijk:dlsym-no-export-enumeration
Open

basvandijk wants to merge 1 commit into
emscripten-core:mainfrom
basvandijk:dlsym-no-export-enumeration

Conversation

@basvandijk

@basvandijk basvandijk commented Nov 1, 2026 •

Copy link
Copy Markdown

_dlsym_js enumerated every export of the library on each call, Object.keys(lib.exports).indexOf(symbol), before doing anything else. For RTLD_DEFAULT on a main module that is wasmImports, which holds every export of the main module, so each dlsym call cost O(N log N) in the number of exported symbols (V8 sorts the keys of a dictionary-mode object on every Object.keys), whether the symbol exists or not. The index is only needed when addFunction allocates a new table slot and the other threads replay the insertion by index (_dlsym_catchup_js, #18590).

This change tests membership with Object.prototype.hasOwnProperty.call, which gives the same answer as Object.keys(...).indexOf(symbol) != -1 for these plain objects, and computes the enumeration index only in the branch that stores it into symbolIndex. The thread catch-up protocol is unchanged.

Measured with a main module of N exported functions calling dlsym(RTLD_DEFAULT, name) (emcc -O2 -pthread -sMAIN_MODULE=1 -sEXIT_RUNTIME=1, Node 26.13.0, ms per call, hits):

exports before after
10,000 1.70 0.008
250,000 80.9 0.009

Lean 4's lean.js (247,834 exports) makes 2,630 dlsym calls at startup, which took 210 s of a 236 s run and take 17 ms with this change;

Fixes #28124.

Tests: test/runner core0.test_dlfcn_self core0.test_dylink_basics core0.test_pthread_dlsym core0.test_pthread_dylink core0.test_pthread_dylink_basics core0.test_pthread_dylink_main_module_1 other.test_dlsym_rtld_default other.test_dlsym_rtld_default_js_symbol other.test_dylink_pthread_static_data pass.

🤖 Generated with Claude Code

`_dlsym_js` enumerated every export of the library on each call,
`Object.keys(lib.exports).indexOf(symbol)`, before doing anything else.
For `RTLD_DEFAULT` on a main module that is `wasmImports`, which holds
every export of the main module, so each `dlsym` call cost O(N log N) in
the number of exported symbols, whether the symbol exists or not. The
index is only needed when `addFunction` allocates a new table slot and
the other threads replay the insertion by index (`_dlsym_catchup_js`).

Test membership with `hasOwnProperty`, which gives the same answer as
`Object.keys(...).indexOf(symbol) != -1` for these plain objects, and
compute the enumeration index only in the branch that stores it into
`symbolIndex`. A main module with 250,000 exported functions goes from
80.9 ms to 0.009 ms per `dlsym` call.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes dlsym in src/lib/libdylink.js by avoiding the enumeration of all library exports on every lookup. Instead, it uses hasOwnProperty for the existence check and only computes the symbol's index when a table slot is actually allocated. The reviewer suggested a further optimization to cache the keys array on the library object to prevent repeated enumeration during multiple allocations.

Comment thread src/lib/libdylink.js
Comment on lines +1353 to +1356
// Other threads replay this insertion by the symbol's enumeration index
// (`_dlsym_catchup_js`). Computing it enumerates every export of the
// library, so only do so when a slot was actually allocated.
{{{ makeSetValue('symbolIndex', 0, 'Object.keys(lib.exports).indexOf(exportName)', '*') }}};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While computing the enumeration index only when a slot is allocated is a great improvement, if a multi-threaded application allocates multiple slots (e.g., during startup or dynamic linking of many functions), calling Object.keys(lib.exports) repeatedly on a library with hundreds of thousands of exports will still cause a significant performance bottleneck (O(M * N) where M is the number of allocations and N is the number of exports).\n\nWe can optimize this to O(N + M) by caching the keys array on the lib object upon the first allocation, and then performing a fast indexOf lookup on the cached array for subsequent allocations.

        // Other threads replay this insertion by the symbol's enumeration index\n        // (_dlsym_catchup_js). Computing it enumerates every export of the\n        // library, so only do so when a slot was actually allocated.\n        if (!lib.exportKeys) {\n          lib.exportKeys = Object.keys(lib.exports);\n        }\n        {{{ makeSetValue('symbolIndex', 0, 'lib.exportKeys.indexOf(exportName)', '*') }}};

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I left the enumeration on this path deliberately. The export list of __main__ is wasmImports, which grows after startup: mergeLibSymbols adds the symbols of a side module loaded with RTLD_GLOBAL, and resolveGlobalSymbol creates invoke_* wrappers lazily (lines 124 and 133). A key list cached on the DSO would therefore go stale, and _dlsym_catchup_js would replay the wrong symbol on the other threads; keeping it correct means invalidating it at every site that adds to wasmImports, and _dlsym_catchup_js would have to use the same list.

The path is also rare today: updateGOT puts every function of the main module into the table at startup, so this branch runs for JS library functions and for symbols that are not in the table yet, and each such allocation already blocks on _emscripten_dlsync_threads, where every other thread enumerates the exports once more in _dlsym_catchup_js. If lazy GOT creation (#27993) lands and first lookups start allocating slots routinely, a key list maintained next to wasmImports and shared by both functions would be the follow-up; I can add it here if the maintainers prefer that.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dlsym enumerates every export of the library on each call (Object.keys(lib.exports).indexOf)

1 participant