From 1be615be99553b5a91c8a5f9c8a06862044d6033 Mon Sep 17 00:00:00 2001 From: kappa Date: Sun, 8 Feb 2026 12:03:50 +0900 Subject: [PATCH] Update CLAUDE.md for SQLite architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflects the JSON → SQLite migration: architecture diagram now shows K8s pod + SSH/SCP remote sync, data flow documentation, updated file descriptions, startup sequence, and write flow. Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 124 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 78 insertions(+), 46 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2af82f1..0327320 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,20 +11,24 @@ HAProxy Dynamic Load Balancer Management Suite - a system for runtime management ## Architecture ``` -┌──────────────────────────────────────────────────┐ -│ HAProxy (host network) │ -│ TCP 80/443 (HTTP/1.1, HTTP/2) + UDP 443 (HTTP/3) │ -│ Runtime API: localhost:9999 │ -│ MCP Proxy: mcp.inouter.com → localhost:8000 │ -└──────────────┬───────────────────────────────────┘ - │ - ┌─────▼─────┐ - │ MCP │ - │ Server │ - │ :8000 │ - └───────────┘ +┌─ K8s Pod (MCP Server) ──────────┐ SSH/SCP ┌─ HAProxy Host ──────────────────┐ +│ │ ───────────→ │ │ +│ /app/data/haproxy_mcp.db │ Runtime API │ HAProxy (host network) │ +│ (SQLite - local working copy) │ ───────────→ │ TCP 80/443 + UDP 443 (HTTP/3) │ +│ │ :9999 │ Runtime API: localhost:9999 │ +│ MCP Server :8000 │ │ │ +│ Transport: streamable-http │ SCP sync │ /opt/haproxy/conf/ │ +│ │ ←──────────→ │ haproxy_mcp.db (persistent) │ +│ │ map sync │ domains.map │ +│ │ ───────────→ │ wildcards.map │ +└──────────────────────────────────┘ └─────────────────────────────────┘ ``` +### Data Flow +- **SQLite DB** (`haproxy_mcp.db`): Single source of truth for domains, servers, certificates +- **Map files** (`domains.map`, `wildcards.map`): Generated from SQLite, HAProxy reads directly +- **Runtime API**: Real-time HAProxy configuration (map entries, server addresses) + ### Domain Routing Flow ``` Request → HAProxy Frontend @@ -33,7 +37,7 @@ domains.map lookup (Host header) ↓ pool_N backend (N = 1-100) ↓ -Server (from servers.json, auto-restored on startup) +Server (from SQLite DB, auto-restored on startup) ``` ## Services @@ -58,7 +62,7 @@ journalctl -u haproxy-mcp -f # Logs - Transport: `streamable-http` - Internal: `http://localhost:8000/mcp` - External: `https://mcp.inouter.com/mcp` (via HAProxy) -- **Auto-restore:** Servers restored from `servers.json` on startup +- **Auto-restore:** Servers restored from SQLite DB on startup ### Environment Variables | Variable | Default | Description | @@ -69,7 +73,10 @@ journalctl -u haproxy-mcp -f # Logs | `HAPROXY_PORT` | `9999` | HAProxy Runtime API port | | `HAPROXY_STATE_FILE` | `/opt/haproxy/data/servers.state` | State file path | | `HAPROXY_MAP_FILE` | `/opt/haproxy/conf/domains.map` | Map file path | -| `HAPROXY_SERVERS_FILE` | `/opt/haproxy/conf/servers.json` | Servers config path | +| `HAPROXY_DB_FILE` | `/opt/haproxy/conf/haproxy_mcp.db` | Local SQLite DB path | +| `HAPROXY_REMOTE_DB_FILE` | `/opt/haproxy/conf/haproxy_mcp.db` | Remote SQLite DB path (SCP target) | +| `HAPROXY_SERVERS_FILE` | `/opt/haproxy/conf/servers.json` | Legacy servers config (migration only) | +| `HAPROXY_CERTS_FILE` | `/opt/haproxy/conf/certificates.json` | Legacy certs config (migration only) | | `HAPROXY_POOL_COUNT` | `100` | Number of pool backends | | `HAPROXY_MAX_SLOTS` | `10` | Max servers per pool | | `HAPROXY_CONTAINER` | `haproxy` | Podman container name | @@ -78,10 +85,10 @@ journalctl -u haproxy-mcp -f # Logs ## Zero-Reload Domain Management ### How It Works -1. **domains.map**: Maps domain → pool backend (e.g., `example.com pool_1`) -2. **Pool backends**: 100 pre-configured backends (`pool_1` to `pool_100`) -3. **Runtime API**: Add/remove map entries and configure servers without reload -4. **servers.json**: Persistent server configuration, auto-restored on MCP startup +1. **SQLite DB** (`haproxy_mcp.db`): Single source of truth for all persistent data +2. **domains.map**: Generated from SQLite, maps domain → pool backend +3. **Pool backends**: 100 pre-configured backends (`pool_1` to `pool_100`) +4. **Runtime API**: Add/remove map entries and configure servers without reload ### Adding a Domain @@ -94,11 +101,12 @@ haproxy_add_domain("api.example.com", "10.0.0.1", 8080) ``` This will: -1. Find available pool (e.g., `pool_5`) -2. Add to `domains.map`: `example.com pool_5` -3. Update HAProxy runtime map: `add map ... example.com pool_5` -4. Configure server in pool via Runtime API (HTTP only) -5. Save to `servers.json` for persistence +1. Find available pool (e.g., `pool_5`) via SQLite query +2. Save to SQLite DB (domain, backend, server) +3. Sync `domains.map` from DB (for HAProxy file-based lookup) +4. Update HAProxy runtime map: `add map ... example.com pool_5` +5. Configure server in pool via Runtime API (HTTP only) +6. Upload DB to remote host via SCP (persistence) **No HAProxy reload required!** @@ -120,7 +128,7 @@ haproxy_add_domain("api.example.com", share_with="example.com") **Benefits:** - Saves pool slots (100 pools can serve unlimited domains) - Multiple domains → same backend servers -- Shared domains stored as `_shares` reference in `servers.json` +- Shared domains stored as `shares_with` in SQLite domains table **Behavior:** - Shared domains cannot specify `ip` (use existing servers) @@ -146,9 +154,12 @@ haproxy_add_servers("api.example.com", '[ ### Files | File | Purpose | |------|---------| -| `domains.map` | Domain → Pool mapping | -| `servers.json` | Server IP/port persistence | +| `haproxy_mcp.db` | SQLite DB — single source of truth (domains, servers, certs) | +| `domains.map` | Domain → Pool mapping (generated from DB, HAProxy reads directly) | +| `wildcards.map` | Wildcard domain mapping (generated from DB) | | `haproxy.cfg` | Pool backend definitions (static) | +| `servers.json` | Legacy — used only for initial migration to SQLite | +| `certificates.json` | Legacy — used only for initial migration to SQLite | ## SSL/TLS Certificates @@ -243,7 +254,7 @@ Returns overall system status: "components": { "mcp": {"status": "ok"}, "haproxy": {"status": "ok", "version": "3.3.2", "uptime_sec": 3600}, - "config_files": {"status": "ok", "files": {"map_file": "ok", "servers_file": "ok"}}, + "config_files": {"status": "ok", "files": {"map_file": "ok", "db_file": "ok"}}, "container": {"status": "ok", "state": "running"} } } @@ -342,15 +353,17 @@ api.example.com → pool_6 └─ pool_6_1 (slot 1) ``` -### Persistence -- **domains.map**: Survives HAProxy restart -- **servers.json**: Auto-restored by MCP on startup -- No manual save required - `haproxy_add_server` auto-saves +### Persistence (SQLite) +- **SQLite DB** (`haproxy_mcp.db`): All domains, servers, certificates stored in single file +- **Remote sync**: DB uploaded to HAProxy host via SCP after every write +- **Startup restore**: DB downloaded from remote on pod start, servers restored via Runtime API +- **Migration**: First startup auto-migrates from legacy JSON/map files (one-time, idempotent) +- **WAL mode**: Write-Ahead Logging for concurrent access, checkpointed before SCP upload ### Safety Features -- **Atomic file writes**: Temp file + rename prevents corruption -- **File locking**: Prevents race conditions on concurrent operations -- **Disk-first pattern**: Config saved before HAProxy update, rollback on failure +- **Atomic file writes**: Temp file + rename prevents corruption (map files) +- **SQLite transactions**: All DB writes are atomic +- **DB-first pattern**: Config saved to SQLite before HAProxy update, rollback on failure - **Command validation**: HAProxy responses checked for errors - **Input validation**: Domain format, IP (v4/v6), port range, slot limits - **Bulk limits**: Max 10 servers per bulk add, 10KB JSON size limit @@ -365,7 +378,7 @@ api.example.com → pool_6 - **Runtime API**: Certificates loaded/updated without HAProxy reload - `new ssl cert` → `set ssl cert` → `commit ssl cert` - No connection drops during certificate changes -- **Persistence**: `certificates.json` stores domain list +- **Persistence**: Certificate domains stored in SQLite `certificates` table - **Auto-restore**: Certificates reloaded into HAProxy on MCP startup ## HAProxy Runtime API @@ -397,10 +410,12 @@ echo "set server pool_1/pool_1_1 state ready" | nc localhost 9999 ├── haproxy_mcp/ # MCP server package (streamable-http) │ ├── server.py # Main entry point │ ├── config.py # Configuration and constants +│ ├── db.py # SQLite database (single source of truth) +│ ├── ssh_ops.py # SSH/SCP remote execution │ ├── exceptions.py # Exception classes │ ├── validation.py # Input validation │ ├── haproxy_client.py # HAProxy Runtime API client -│ ├── file_ops.py # File I/O operations +│ ├── file_ops.py # File I/O (delegates to db.py) │ ├── utils.py # Parsing utilities │ └── tools/ # MCP tools (29 total) │ ├── domains.py # Domain management (3 tools) @@ -411,9 +426,11 @@ echo "set server pool_1/pool_1_1 state ready" | nc localhost 9999 │ └── certificates.py # Certificate management (7 tools) ├── conf/ │ ├── haproxy.cfg # Main HAProxy config (100 pool backends) -│ ├── domains.map # Domain → Pool mapping -│ ├── servers.json # Server persistence (auto-managed) -│ ├── certificates.json # Certificate domain list (auto-managed) +│ ├── haproxy_mcp.db # SQLite DB (single source of truth) +│ ├── domains.map # Domain → Pool mapping (generated from DB) +│ ├── wildcards.map # Wildcard domain mapping (generated from DB) +│ ├── servers.json # Legacy (migration source only) +│ ├── certificates.json # Legacy (migration source only) │ └── mcp-token.env # Bearer token for MCP auth ├── certs/ # SSL/TLS certificates (HAProxy PEM format) ├── data/ # Legacy state files @@ -433,8 +450,9 @@ echo "set server pool_1/pool_1_1 state ready" | nc localhost 9999 | File | Purpose | |------|---------| | `/opt/haproxy/conf/haproxy.cfg` | HAProxy main config | -| `/opt/haproxy/conf/domains.map` | Domain routing map | -| `/opt/haproxy/conf/servers.json` | Server persistence | +| `/opt/haproxy/conf/haproxy_mcp.db` | SQLite DB (single source of truth) | +| `/opt/haproxy/conf/domains.map` | Domain routing map (generated from DB) | +| `/opt/haproxy/conf/wildcards.map` | Wildcard domain map (generated from DB) | | `/opt/haproxy/certs/*.pem` | SSL certificates (auto-loaded) | | `/etc/containers/systemd/haproxy.container` | Podman Quadlet | | `/etc/systemd/system/haproxy-mcp.service` | MCP service | @@ -451,13 +469,27 @@ echo "set server pool_1/pool_1_1 state ready" | nc localhost 9999 ↓ 3. HAProxy loads domains.map (domain → pool mapping) ↓ -4. systemd starts haproxy-mcp.service +4. systemd starts haproxy-mcp.service (or K8s pod starts) ↓ -5. MCP server reads servers.json & certificates.json +5. init_db(): Download SQLite DB from remote host via SCP + (first time: migrate from legacy JSON/map files → create DB → upload) ↓ -6. MCP server restores servers via Runtime API +6. MCP server restores servers from DB via Runtime API ↓ -7. MCP server loads certificates via Runtime API (zero-downtime) +7. MCP server loads certificates from DB via Runtime API (zero-downtime) ↓ 8. Ready to serve traffic ``` + +### Write Flow (runtime) +``` +MCP tool call (add domain/server/cert) + ↓ +Write to local SQLite DB + ↓ +Sync map files to remote (domains only) + ↓ +Update HAProxy via Runtime API + ↓ +Upload DB to remote host via SCP (persistence) +```