Skip to content

Commit 117316d

Browse files
Joxx0rclaude
Joxx0r
and
claude
committed
Add Docker containerization of unreal-index service
Replace manual WSL setup (Node 22, Go, Zoekt build, screen) with `docker compose up`. Multi-stage Dockerfile builds Zoekt from Go and compiles native Node modules, producing a ~250MB runtime image. - Dockerfile: 3-stage build (Go builder, Node builder, slim runtime) - docker-compose.yml: named volumes, 4GB mem limit, health check - docker-compose.dev.yml: source mounts with --watch for development - config.docker.json: container paths (/data/), host 0.0.0.0 - docker-entrypoint.sh: data dir setup, config fallback, exec node - index.js: env-var config path, relaxed project validation for Docker - start-service.sh: --docker flag with health polling - test-docker-perf.mjs: 6-phase Docker perf test suite - DOCKER.md: setup guide, architecture, troubleshooting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent e0aa0e3 commit 117316d

10 files changed

Lines changed: 1059 additions & 3 deletions

‎.dockerignore‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
node_modules
2+
data
3+
*.log
4+
config.json
5+
.git
6+
.github
7+
.claude
8+
memory
9+
profiles
10+
NUL
11+
signature*
12+
perf-results-*.json
13+
test-*.mjs
14+
test-*.js
15+
test-*.ps1
16+
perf-stress-test.mjs
17+
*.bat
18+
src/perf-test.js
19+
src/stress-test.js
20+
src/grep-stress-test.js
21+
src/pool-stress-test.js
22+
src/profile.js
23+
src/*.test.js

‎DOCKER.md‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
# Docker Setup for unreal-index
2+
3+
## Prerequisites
4+
5+
- **Docker Desktop for Windows** with WSL 2 backend enabled
6+
- At least **4 GB RAM** allocated to Docker (Settings > Resources > Memory)
7+
- The Windows watcher (`node src\watcher\watcher-client.js`) runs outside the container
8+
9+
## Quick Start
10+
11+
```bash
12+
# Build and start the service
13+
docker compose up -d
14+
15+
# Verify it's running
16+
curl http://localhost:3847/health
17+
18+
# Start the watcher on Windows (separate terminal)
19+
node src\watcher\watcher-client.js
20+
```
21+
22+
Or use the convenience script:
23+
24+
```bash
25+
./start-service.sh --docker
26+
```
27+
28+
## Architecture
29+
30+
```
31+
Windows Host
32+
┌─────────────────────────────────────┐
33+
│ Watcher (node watcher-client.js) │
34+
│ reads P4 workspace files │
35+
│ parses AS/C++/assets │
36+
│ │ │
37+
│ │ POST /internal/ingest │
38+
│ ▼ │
39+
│ ┌─────────────────────────────┐ │
40+
│ │ Docker Container │ │
41+
│ │ ┌───────────────────┐ │ │
42+
│ │ │ Node.js Service │:3847│◄───┼── Claude Code / MCP
43+
│ │ │ (Express API) │ │ │
44+
│ │ │ SQLite + Memory │ │ │
45+
│ │ └───────┬───────────┘ │ │
46+
│ │ │ │ │
47+
│ │ ┌───────▼───────────┐ │ │
48+
│ │ │ Zoekt │ │ │
49+
│ │ │ (index + web) │ │ │
50+
│ │ └───────────────────┘ │ │
51+
│ │ │ │
52+
│ │ Volumes: │ │
53+
│ │ /data/db (SQLite) │ │
54+
│ │ /data/mirror (Zoekt src) │ │
55+
│ │ /data/zoekt-index (shards)│ │
56+
│ └─────────────────────────────┘ │
57+
└─────────────────────────────────────┘
58+
```
59+
60+
## Configuration
61+
62+
The container uses `config.docker.json` as the default config. Key settings:
63+
64+
| Setting | Default | Description |
65+
|---------|---------|-------------|
66+
| `service.host` | `0.0.0.0` | Required for Docker port mapping |
67+
| `service.port` | `3847` | API port |
68+
| `data.dbPath` | `/data/db/index.db` | SQLite database path |
69+
| `data.mirrorDir` | `/data/mirror` | Zoekt mirror directory |
70+
| `data.indexDir` | `/data/zoekt-index` | Zoekt index shards |
71+
| `zoekt.parallelism` | `4` | Zoekt indexing threads |
72+
| `projects` | `[]` | Empty — data arrives via `/internal/ingest` |
73+
74+
### Custom config
75+
76+
Mount a custom config file:
77+
78+
```yaml
79+
# docker-compose.override.yml
80+
services:
81+
unreal-index:
82+
volumes:
83+
- ./my-config.json:/app/config.json:ro
84+
```
85+
86+
### Environment variables
87+
88+
- `UNREAL_INDEX_CONFIG` — path to config file (overrides default)
89+
- `NODE_OPTIONS` — Node.js options (default: `--max-old-space-size=3072`)
90+
91+
## Data Persistence
92+
93+
Data is stored in three Docker named volumes:
94+
95+
| Volume | Contents | Purpose |
96+
|--------|----------|---------|
97+
| `unreal-index-db` | `index.db` | SQLite database |
98+
| `unreal-index-mirror` | Source files | Zoekt mirror for grep |
99+
| `unreal-index-zoekt` | Index shards | Zoekt search index |
100+
101+
### Lifecycle
102+
103+
```bash
104+
# Stop container (data preserved)
105+
docker compose down
106+
107+
# Start again (data still there)
108+
docker compose up -d
109+
110+
# DANGER: Remove data volumes
111+
docker compose down -v
112+
```
113+
114+
### Backup
115+
116+
```bash
117+
# Backup database
118+
docker compose exec unreal-index cp /data/db/index.db /data/db/index.db.bak
119+
120+
# Copy database to host
121+
docker compose cp unreal-index:/data/db/index.db ./backup-index.db
122+
```
123+
124+
### Restore
125+
126+
```bash
127+
# Copy database into container
128+
docker compose cp ./backup-index.db unreal-index:/data/db/index.db
129+
130+
# Restart to load
131+
docker compose restart
132+
```
133+
134+
## Memory Configuration
135+
136+
The container is limited to 4 GB RAM:
137+
- **3 GB** — Node.js heap (`--max-old-space-size=3072`)
138+
- **~1 GB** — Zoekt processes, OS overhead, SQLite mmap
139+
140+
For larger codebases, increase both limits in `docker-compose.yml`:
141+
142+
```yaml
143+
services:
144+
unreal-index:
145+
mem_limit: 6g
146+
memswap_limit: 6g
147+
environment:
148+
- NODE_OPTIONS=--max-old-space-size=4096
149+
```
150+
151+
Also ensure Docker Desktop has sufficient RAM allocated (Settings > Resources).
152+
153+
## Troubleshooting
154+
155+
### Port conflict
156+
157+
```
158+
Error: listen EADDRINUSE: address already in use :::3847
159+
```
160+
161+
Another process is using port 3847. Stop it or change the port mapping:
162+
163+
```yaml
164+
ports:
165+
- "3848:3847" # Use 3848 on host
166+
```
167+
168+
### Container OOM killed
169+
170+
```bash
171+
# Check if container was OOM killed
172+
docker inspect unreal-index | grep -A 5 OOMKilled
173+
```
174+
175+
Increase `mem_limit` in `docker-compose.yml` and Docker Desktop RAM allocation.
176+
177+
### Slow queries after restart
178+
179+
The in-memory index needs to reload from SQLite on startup (~10s for large indexes). Queries during this window may be slow or return empty results. The health check accounts for this with a 30s start period.
180+
181+
### SQLite errors
182+
183+
SQLite runs on a Docker named volume (ext4 filesystem). This avoids the performance issues of bind-mounting from Windows NTFS. If you see locking errors, ensure only one container is running:
184+
185+
```bash
186+
docker compose ps
187+
```
188+
189+
### Viewing logs
190+
191+
```bash
192+
# Follow logs
193+
docker compose logs -f
194+
195+
# Last 100 lines
196+
docker compose logs --tail=100
197+
```
198+
199+
## Development
200+
201+
Use the dev compose override for source mounting with auto-reload:
202+
203+
```bash
204+
docker compose -f docker-compose.yml -f docker-compose.dev.yml up
205+
```
206+
207+
This mounts `src/` and `public/` as read-only volumes and enables Node.js `--watch` mode. The container rebuilds only when dependencies change.
208+
209+
To rebuild after dependency changes:
210+
211+
```bash
212+
docker compose build
213+
```
214+
215+
## Performance Testing
216+
217+
Run the Docker performance test from the host:
218+
219+
```bash
220+
node test-docker-perf.mjs
221+
```
222+
223+
With a baseline comparison:
224+
225+
```bash
226+
node test-docker-perf.mjs --baseline perf-baseline-wsl.json
227+
```
228+
229+
Long-running stability test (30 minutes):
230+
231+
```bash
232+
node test-docker-perf.mjs --long-run
233+
```
234+
235+
## Docker vs WSL Comparison
236+
237+
| Aspect | WSL (current) | Docker |
238+
|--------|---------------|--------|
239+
| **Setup** | Manual: Node 22, Go, Zoekt build, screen | `docker compose up -d` |
240+
| **Networking** | WSL mirrored mode, screen hacks | Standard port mapping |
241+
| **Persistence** | `~/.unreal-index/` on ext4 | Named volumes (ext4) |
242+
| **Updates** | `git pull && npm install` | `docker compose build && up -d` |
243+
| **Memory** | Shared with WSL | Isolated, configurable limit |
244+
| **Startup** | ~10s (warm) | ~15s (warm, includes container overhead) |
245+
| **SQLite perf** | Native ext4 | Named volume ext4 (equivalent) |
246+
| **Background** | Requires `screen` | Built-in container lifecycle |
247+
| **Portability** | Tied to this machine's WSL setup | Any Docker host |

‎Dockerfile‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Stage 1: Build Zoekt binaries
2+
FROM golang:1.24-alpine AS zoekt-builder
3+
RUN apk add --no-cache git
4+
RUN go install github.com/sourcegraph/zoekt/cmd/zoekt-index@latest && \
5+
go install github.com/sourcegraph/zoekt/cmd/zoekt-webserver@latest
6+
7+
# Stage 2: Install Node.js dependencies (compile native modules for Linux)
8+
FROM node:22-slim AS node-builder
9+
RUN apt-get update && apt-get install -y build-essential python3 && rm -rf /var/lib/apt/lists/*
10+
WORKDIR /app
11+
COPY package.json package-lock.json ./
12+
RUN npm ci --omit=dev
13+
14+
# Stage 3: Runtime
15+
FROM node:22-slim
16+
RUN apt-get update && apt-get install -y --no-install-recommends lsof procps && rm -rf /var/lib/apt/lists/*
17+
WORKDIR /app
18+
19+
COPY --from=zoekt-builder /go/bin/zoekt-index /go/bin/zoekt-webserver /usr/local/bin/
20+
COPY --from=node-builder /app/node_modules ./node_modules
21+
COPY src/ ./src/
22+
COPY public/ ./public/
23+
COPY package.json config.docker.json docker-entrypoint.sh ./
24+
25+
RUN chmod +x docker-entrypoint.sh && mkdir -p /data
26+
27+
ENV NODE_ENV=production
28+
ENV NODE_OPTIONS="--max-old-space-size=3072"
29+
30+
EXPOSE 3847
31+
32+
HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 \
33+
CMD node -e "fetch('http://127.0.0.1:3847/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))"
34+
35+
ENTRYPOINT ["./docker-entrypoint.sh"]

‎config.docker.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"projects": [],
3+
"exclude": ["**/Intermediate/**", "**/Binaries/**", "**/ThirdParty/**",
4+
"**/__ExternalActors__/**", "**/__ExternalObjects__/**", "**/Developers/**"],
5+
"service": { "port": 3847, "host": "0.0.0.0" },
6+
"data": {
7+
"dbPath": "/data/db/index.db",
8+
"mirrorDir": "/data/mirror",
9+
"indexDir": "/data/zoekt-index"
10+
},
11+
"zoekt": { "webPort": 6070, "parallelism": 4, "reindexDebounceMs": 5000 },
12+
"watcher": { "debounceMs": 100 }
13+
}

‎docker-compose.dev.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
services:
2+
unreal-index:
3+
volumes:
4+
- ./src:/app/src:ro
5+
- ./public:/app/public:ro
6+
- unreal-index-db:/data/db
7+
- unreal-index-mirror:/data/mirror
8+
- unreal-index-zoekt:/data/zoekt-index
9+
ports:
10+
- "3847:3847"
11+
- "6070:6070"
12+
command: ["node", "--watch", "src/service/index.js"]

‎docker-compose.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
services:
2+
unreal-index:
3+
build: .
4+
container_name: unreal-index
5+
ports:
6+
- "3847:3847"
7+
volumes:
8+
- unreal-index-db:/data/db
9+
- unreal-index-mirror:/data/mirror
10+
- unreal-index-zoekt:/data/zoekt-index
11+
mem_limit: 4g
12+
memswap_limit: 4g
13+
environment:
14+
- NODE_OPTIONS=--max-old-space-size=3072
15+
restart: unless-stopped
16+
stop_grace_period: 30s
17+
healthcheck:
18+
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3847/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))"]
19+
interval: 15s
20+
timeout: 5s
21+
start_period: 30s
22+
retries: 3
23+
24+
volumes:
25+
unreal-index-db:
26+
unreal-index-mirror:
27+
unreal-index-zoekt:

‎docker-entrypoint.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
set -e
3+
mkdir -p /data/db /data/mirror /data/zoekt-index
4+
if [ ! -f /app/config.json ]; then
5+
cp /app/config.docker.json /app/config.json
6+
fi
7+
exec node src/service/index.js

‎src/service/index.js‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ class UnrealIndexService {
4242
}
4343

4444
async loadConfig() {
45-
const configPath = join(__dirname, '..', '..', 'config.json');
45+
const configPath = process.env.UNREAL_INDEX_CONFIG || join(__dirname, '..', '..', 'config.json');
4646

4747
if (!existsSync(configPath)) {
4848
throw new Error(
@@ -69,8 +69,11 @@ class UnrealIndexService {
6969
}
7070

7171
// Validate projects
72-
if (!this.config.projects || !Array.isArray(this.config.projects) || this.config.projects.length === 0) {
73-
throw new Error(`config.json has no projects configured.`);
72+
if (!this.config.projects || !Array.isArray(this.config.projects)) {
73+
this.config.projects = [];
74+
}
75+
if (this.config.projects.length === 0) {
76+
console.log('[Config] No projects configured — service will receive data via /internal/ingest');
7477
}
7578

7679
for (const project of this.config.projects) {

0 commit comments

Comments
 (0)