Skip to content

chdb: one long-lived server process instead of a Python process per query - #3444

Open
alexey-milovidov wants to merge 1 commit into
mainfrom
chdb-server
Open

alexey-milovidov wants to merge 1 commit into
mainfrom
chdb-server

Conversation

@alexey-milovidov

Copy link
Copy Markdown
Member

./start now runs chdb/server.py: one long-lived Python process with a single chDB instance over .clickbench, accepting concurrent connections on a Unix socket (chdb.sock). Every connection is served by its own thread with its own Session, reused for all queries sent over that connection:

def worker(requests):
    session = Session(DATA)
    try:
        session.query("USE clickbench")
        for request in requests:
            started = time.perf_counter()
            try:
                output = str(session.query(request.sql, "Pretty"))
            except Exception as exc:
                request.fail(str(exc))
            else:
                request.reply(output, time.perf_counter() - started)
    finally:
        session.close()

./query, ./check, ./load and the new ./concurrent-worker are stdlib-only clients (chdb/client.py), so they don't load chDB. The time is measured by the server around session.query.

Motivation: before, every ./query started a new Python process and opened a Session on .clickbench, so concurrent queries failed with Cannot lock file .clickbench/status. In the randomized benchmark group a73b3f6e-b715-480c-b46a-2cb2fd0dbc09 (4 workers), chdb had 3/300 successful queries.

Other changes in the entry:

  • BENCH_RESTARTABLE is now the default yes: a cold run restarts the server.
  • The concurrent QPS test is enabled again (it was off because each query was a separate process); BENCH_CONCURRENT_WORKER=./concurrent-worker keeps one connection, so one session, per worker, and reconnects if the watchdog restarts the server.
  • ./load goes through the server (DROP TABLE IF EXISTS clickbench.hits SYNC, then create.sql and insert.sql), because the server holds the data directory.

Tested locally with chdb 4.5.1 / chdb-core 26.11.1 on a 2M-row generateRandom table: 8 parallel ./query calls all succeeded; 4 concurrent workers ran ~950-990 queries each in 20 s with 0 errors; data survives a stop and start; a warm startup takes 0.3-0.45 s. A full benchmark run has not been done yet.

🤖 Generated with Claude Code

…uery

`./start` now runs `chdb/server.py`: a single Python process with one chDB
instance over `.clickbench`, serving queries on a Unix socket. Every
connection is served by its own thread with its own `Session`, reused for all
queries on that connection. `./query`, `./check`, `./load` and the new
`./concurrent-worker` are stdlib-only clients (`chdb/client.py`).

Before, every `./query` opened a `Session` on `.clickbench` in a fresh
process, so concurrent queries failed with "Cannot lock file
.clickbench/status" (3/300 successful queries with 4 workers in the
randomized benchmark group a73b3f6e-b715-480c-b46a-2cb2fd0dbc09).

The entry is now restartable (a cold run restarts the server), and the
concurrent QPS test is enabled, with one connection (one session) per worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexey-milovidov alexey-milovidov added the machine:all PR benchmark on every machine type label Nov 19, 2026
@alexey-milovidov
alexey-milovidov deployed to benchmark-approval November 19, 2026 19:18 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
benchmark-approval — 4e267b41 Deployed Nov 19, 2026 by alexey-milovidov via launch #1336
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

machine:all PR benchmark on every machine type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant