Alternative SOCKFS in Node.js environment + synchronous socket calls using JSPI

[ryanking13]

Proposal

We (Pyodide and Cloudflare workers) would like to enable socket support without relying on the websocket proxy.
This would allow us to run existing Python code that relies on sockets (e.g., HTTP clients, database clients, etc) without any modification both in client and server.

This is impossible in a Web environment (unless Direct Socket proposal is accepted #16737) as sockets are not supported.
However, in Node.js (or other JS runtimes such as Deno, Bun, or Cloudflare Workers), we can take advantage of it's native socket APIs.

Also, while the Node.js socket APIs are asynchronous, Asyncify (JSPI), can be used to pause the JS function calls to make it compatible to existing socket APIs that expects blocking on select, poll, or recv.

How is would look like

1. Add a flag such as -sUSE_NODESOCKET that replaces the current SOCKFS implementation
   with a new implementation when the runtime environment is detected as Node.js.

2. Update the socket syscalls (e.g., listen, recv, send, etc) to use JSPI or Asyncify (when available)
   to block the calls until the underlying Node.js socket APIs complete.

Discussion Topics & Concerns

1. Ideally, it would be great to detect the JSPI availability at runtime, so that either the synchronous version or the asynchronous version is selected at runtime. But I guess currently Asyncify does not support that.

---

I would be happy to implement and contribute this feature is it makes sense.
Pyodide can patch Emscripten locally for our use case, but it would be great to have this feature available upstream so that other projects can benefit from it as well.

[sbc100]

Would you be OK putting all this behind NODERAWFS ? i.e. are you already using -sNODERAWFS? if so then yes this seems pretty reasonable request.

The tricky part I think will be converting some/all of the __syscall_xxx functions to be JSPI / async.

[ryanking13]

Would you be OK putting all this behind NODERAWFS ? i.e. are you already using -sNODERAWFS?

No we are not using NODERAWFS, and I am afraid NODERAWFS is not suitable for us. Please correct me if I am wrong, but I guess using NODERAWFS would disable all other filesystem like MEMFS? We are relying on Emscripten's mounting approach that allows us to use different file systems together, so only supporting NODERAWFS will not be suitable for our use case.

The tricky part I think will be converting some/all of the __syscall_xxx functions to be JSPI / async.

Yes, I think that part is something that requires a lot of thought in implementation. My idea was to start with __syscall_connect and __syscall_recvfrom that only applies to the socket file descriptors.