Repository navigation
Make dlsym O(1) in the number of exports - #28125
basvandijk wants to merge 1 commit into
Conversation
`_dlsym_js` enumerated every export of the library on each call, `Object.keys(lib.exports).indexOf(symbol)`, before doing anything else. For `RTLD_DEFAULT` on a main module that is `wasmImports`, which holds every export of the main module, so each `dlsym` call cost O(N log N) in the number of exported symbols, whether the symbol exists or not. The index is only needed when `addFunction` allocates a new table slot and the other threads replay the insertion by index (`_dlsym_catchup_js`). Test membership with `hasOwnProperty`, which gives the same answer as `Object.keys(...).indexOf(symbol) != -1` for these plain objects, and compute the enumeration index only in the branch that stores it into `symbolIndex`. A main module with 250,000 exported functions goes from 80.9 ms to 0.009 ms per `dlsym` call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request optimizes dlsym in src/lib/libdylink.js by avoiding the enumeration of all library exports on every lookup. Instead, it uses hasOwnProperty for the existence check and only computes the symbol's index when a table slot is actually allocated. The reviewer suggested a further optimization to cache the keys array on the library object to prevent repeated enumeration during multiple allocations.
| // Other threads replay this insertion by the symbol's enumeration index | ||
| // (`_dlsym_catchup_js`). Computing it enumerates every export of the | ||
| // library, so only do so when a slot was actually allocated. | ||
| {{{ makeSetValue('symbolIndex', 0, 'Object.keys(lib.exports).indexOf(exportName)', '*') }}}; |
There was a problem hiding this comment.
While computing the enumeration index only when a slot is allocated is a great improvement, if a multi-threaded application allocates multiple slots (e.g., during startup or dynamic linking of many functions), calling Object.keys(lib.exports) repeatedly on a library with hundreds of thousands of exports will still cause a significant performance bottleneck (O(M * N) where M is the number of allocations and N is the number of exports).\n\nWe can optimize this to O(N + M) by caching the keys array on the lib object upon the first allocation, and then performing a fast indexOf lookup on the cached array for subsequent allocations.
// Other threads replay this insertion by the symbol's enumeration index\n // (_dlsym_catchup_js). Computing it enumerates every export of the\n // library, so only do so when a slot was actually allocated.\n if (!lib.exportKeys) {\n lib.exportKeys = Object.keys(lib.exports);\n }\n {{{ makeSetValue('symbolIndex', 0, 'lib.exportKeys.indexOf(exportName)', '*') }}};There was a problem hiding this comment.
Thanks. I left the enumeration on this path deliberately. The export list of __main__ is wasmImports, which grows after startup: mergeLibSymbols adds the symbols of a side module loaded with RTLD_GLOBAL, and resolveGlobalSymbol creates invoke_* wrappers lazily (lines 124 and 133). A key list cached on the DSO would therefore go stale, and _dlsym_catchup_js would replay the wrong symbol on the other threads; keeping it correct means invalidating it at every site that adds to wasmImports, and _dlsym_catchup_js would have to use the same list.
The path is also rare today: updateGOT puts every function of the main module into the table at startup, so this branch runs for JS library functions and for symbols that are not in the table yet, and each such allocation already blocks on _emscripten_dlsync_threads, where every other thread enumerates the exports once more in _dlsym_catchup_js. If lazy GOT creation (#27993) lands and first lookups start allocating slots routinely, a key list maintained next to wasmImports and shared by both functions would be the follow-up; I can add it here if the maintainers prefer that.
_dlsym_jsenumerated every export of the library on each call,Object.keys(lib.exports).indexOf(symbol), before doing anything else. ForRTLD_DEFAULTon a main module that iswasmImports, which holds every export of the main module, so eachdlsymcall cost O(N log N) in the number of exported symbols (V8 sorts the keys of a dictionary-mode object on everyObject.keys), whether the symbol exists or not. The index is only needed whenaddFunctionallocates a new table slot and the other threads replay the insertion by index (_dlsym_catchup_js, #18590).This change tests membership with
Object.prototype.hasOwnProperty.call, which gives the same answer asObject.keys(...).indexOf(symbol) != -1for these plain objects, and computes the enumeration index only in the branch that stores it intosymbolIndex. The thread catch-up protocol is unchanged.Measured with a main module of N exported functions calling
dlsym(RTLD_DEFAULT, name)(emcc -O2 -pthread -sMAIN_MODULE=1 -sEXIT_RUNTIME=1, Node 26.13.0, ms per call, hits):Lean 4's
lean.js(247,834 exports) makes 2,630dlsymcalls at startup, which took 210 s of a 236 s run and take 17 ms with this change;Fixes #28124.
Tests:
test/runner core0.test_dlfcn_self core0.test_dylink_basics core0.test_pthread_dlsym core0.test_pthread_dylink core0.test_pthread_dylink_basics core0.test_pthread_dylink_main_module_1 other.test_dlsym_rtld_default other.test_dlsym_rtld_default_js_symbol other.test_dylink_pthread_static_datapass.🤖 Generated with Claude Code