Skip to content

Add haxelib layer: extern + build-macro JSON override for the Lua target#5

Open
jdonaldson wants to merge 1 commit into
HaxeFoundation:masterfrom
jdonaldson:add-haxelib
Open

Add haxelib layer: extern + build-macro JSON override for the Lua target#5
jdonaldson wants to merge 1 commit into
HaxeFoundation:masterfrom
jdonaldson:add-haxelib

Conversation

@jdonaldson

Copy link
Copy Markdown
Member

This makes hx-lua-simdjson installable as a haxelib alongside the existing luarock, so haxe.Json can opt into simdjson on the Lua target without simdjson being a std dependency. Groundwork for HaxeFoundation/haxe#12771 (making the simdjson parser opt-in rather than the Lua default).

What is added

File Purpose
src/hxluasimdjson/Json.hx Extern binding to the native hxsimdjson module (@:luaRequire)
src/hxluasimdjson/Macro.hx Init macro that attaches a @:build to haxe.format.JsonParser and rewrites parse() to call the simdjson extern
extraParams.hxml Runs the init macro automatically when the lib is added with -lib
haxelib.json haxelib metadata (so it can be haxelib submitted)

Usage:

luarocks make hx-lua-simdjson-scm-1.rockspec   # native module
haxelib install hx-lua-simdjson                # once published
haxe -lua out.lua -lib hx-lua-simdjson ...

Why a build macro instead of shadowing the std file

haxe.format.JsonParser could be overridden by shipping a classpath copy of std/haxe/format/JsonParser.hx, but that copies the entire std file and has to be re-synced every time the std parser changes across Haxe versions (raised in HaxeFoundation/haxe#12771). Instead, the macro attaches @:build to the existing std type via Compiler.addGlobalMetadata and rewrites only the parse body -- nothing is shadowed, nothing needs syncing.

Scope / no-op safety

The @:build is attached globally, so patch() also runs when haxe.format.JsonParser is built for the macro/eval context (where the Lua-only extern does not exist). The rewrite is therefore guarded with Context.defined("lua") and is a no-op on every other target and in the macro context. Only parse is affected; haxe.Json.stringify keeps using the std printer.

Testing

Verified against a local Haxe build with the std JsonParser temporarily flipped to the #12771 default (pure-Haxe parser):

  • no lib -> pure-Haxe parser (no hxsimdjson reference in output)
  • -lib hx-lua-simdjson -> haxe.Json.parse compiles to require("hxsimdjson").parse(...)
  • --interp and non-Lua targets -> clean no-op (interp parses correctly via the pure-Haxe fallback)

Runtime parsing through the native module was not exercised here because the hxsimdjson rock is not built in this environment; the change only affects which parse function the Lua codegen emits.

This makes hx-lua-simdjson installable as a haxelib alongside the
existing luarock, so haxe.Json can opt into simdjson on the Lua target
without simdjson being a std dependency.

- src/hxluasimdjson/Json.hx: extern binding to the native hxsimdjson module
- src/hxluasimdjson/Macro.hx: init macro that attaches a @:build to
  haxe.format.JsonParser and rewrites parse() to call the simdjson extern,
  guarded to the Lua target (no-op elsewhere, including the macro/eval context)
- extraParams.hxml: runs the init macro automatically when -lib'd
- haxelib.json: haxelib metadata

The build-macro approach avoids shadowing std/haxe/format/JsonParser.hx,
so it does not need re-syncing when the Haxe std changes across versions.
@tobil4sk

tobil4sk commented Jul 5, 2026

Copy link
Copy Markdown
Member

This approach looks good to me, the build macro is probably the cleanest way to do this without duplicating classes.

Just some thoughts about less crucial decisions:

  • I wonder if it makes sense to have a noop for non lua, maybe an error message would be better? Not sure if there is some precedent to follow regarding this from other target specific libraries
  • I wonder if we should have the haxe side in the same git history as the lua side. It might make updates cleaner to have them separate (maybe different branches?), though maybe it won't save us too much work since there are already regular conflicts to deal with. I have thought in the past about how we could better organise things to avoid too many merge conflicts when pulling updates from upstream, which is also a topic discussed originally in: haxe specific implementation discussion PR (don't merge!) FourierTransformer/lua-simdjson#12. However, I couldn't come up with any better suggestions either.

@jdonaldson

Copy link
Copy Markdown
Member Author

Thanks for the review! On the two points:

No-op vs. error on non-Lua — I looked at how existing target-specific haxelibs handle this. hxnodejs is the closest precedent: it doesn't hard-error when pulled into a non-JS build. Its extraParams.hxml just runs init macros (--macro define('nodejs'), etc.) and the externs are simply inert / fail at the usage site. So the ecosystem norm for target libs seems to be "no-op off-target" rather than a hard error, and a hard #error would also break a shared multi-target hxml (someone building lua + js from one config).

That said, this lib rewrites haxe.Json rather than only adding externs, so a silent no-op could mask "I added the lib but didn't actually get simdjson." If you'd like to surface that, I'd lean toward a soft Context.warning on non-Lua rather than a hard error — informative without breaking multi-target builds. My default would be to keep the plain no-op to match hxnodejs, but happy to add the warning if you prefer.

(Aside: hxnodejs's extraParams.hxml has a comment that define('nodejs') "should behave like other target defines and not be defined in macro context" — that's the same macro/eval-context distinction that made me guard the rewrite with Context.defined("lua"). The @:build otherwise also fires when JsonParser is built for the macro context, where the Lua extern doesn't exist.)

Same vs. separate git history — I think same history is fine and doesn't really add to the upstream-merge conflict surface. The Haxe additions live in separate paths from the vendored sources: haxelib.json / extraParams.hxml at the root and src/hxluasimdjson/*.hx, none of which overlap with the src/simdjson.* / src/hxluasimdjson.cpp files that get synced from upstream. The only shared file I touched is README.md. Keeping one history also keeps the rockspec tag and the haxelib version pointing at the same commit, which matters for haxelib submit.

If you'd like the split to be more explicit without going to separate branches, I can move the Haxe classpath into its own top-level dir (e.g. haxe/) and point classPath there. That keeps the .hx entirely out of the vendored src/ tree (right now there's both a src/hxluasimdjson.cpp file and a src/hxluasimdjson/ dir, which is a little confusing) and out of the packaged haxelib. Let me know and I'll adjust.

@jdonaldson

Copy link
Copy Markdown
Member Author

Another organic comment here, hope you're having a great summer!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants