haproxy-mcp/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

HAProxy Dynamic Load Balancer Management Suite - a system for runtime management of HAProxy without restarts. Provides MCP interface for dynamic domain/server management. Frontend supports HTTP/HTTPS/HTTP3, backends use HTTP only (SSL termination at HAProxy).

**Key Feature:** Zero-reload domain management using HAProxy map files and pool backends.

## Architecture

```
┌─ K8s Pod (MCP Server) ──────────┐    SSH/SCP    ┌─ HAProxy Host ──────────────────┐
│                                  │ ───────────→  │                                 │
│  /app/data/haproxy_mcp.db       │   Runtime API  │  HAProxy (host network)         │
│  (SQLite - local working copy)  │ ───────────→   │  TCP 80/443 + UDP 443 (HTTP/3)  │
│                                  │   :9999       │  Runtime API: localhost:9999     │
│  MCP Server :8000               │               │                                 │
│  Transport: streamable-http     │   SCP sync    │  /opt/haproxy/conf/             │
│                                  │ ←──────────→  │    haproxy_mcp.db (persistent)  │
│                                  │   map sync    │    domains.map                  │
│                                  │ ───────────→  │    wildcards.map                │
└──────────────────────────────────┘               └─────────────────────────────────┘
```

### Data Flow
- **SQLite DB** (`haproxy_mcp.db`): Single source of truth for domains, servers, certificates
- **Map files** (`domains.map`, `wildcards.map`): Generated from SQLite, HAProxy reads directly
- **Runtime API**: Real-time HAProxy configuration (map entries, server addresses)

### Domain Routing Flow
```
Request → HAProxy Frontend
    ↓
domains.map lookup (Host header)
    ↓
pool_N backend (N = 1-100)
    ↓
Server (from SQLite DB, auto-restored on startup)
```

## Services

### HAProxy (Podman Quadlet)
```bash
systemctl start haproxy      # Start
systemctl stop haproxy       # Stop
systemctl restart haproxy    # Restart
systemctl status haproxy     # Status
```
- Quadlet config: `/etc/containers/systemd/haproxy.container`
- Network mode: `host` (direct access to host services)

### MCP Server (systemd)
```bash
systemctl start haproxy-mcp     # Start
systemctl stop haproxy-mcp      # Stop
systemctl restart haproxy-mcp   # Restart
journalctl -u haproxy-mcp -f    # Logs
```
- Transport: `streamable-http`
- Internal: `http://localhost:8000/mcp`
- External: `https://mcp.inouter.com/mcp` (via HAProxy)
- **Auto-restore:** Servers restored from SQLite DB on startup

### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `MCP_HOST` | `0.0.0.0` | MCP server bind host |
| `MCP_PORT` | `8000` | MCP server port |
| `HAPROXY_HOST` | `localhost` | HAProxy Runtime API host |
| `HAPROXY_PORT` | `9999` | HAProxy Runtime API port |
| `HAPROXY_STATE_FILE` | `/opt/haproxy/data/servers.state` | State file path |
| `HAPROXY_MAP_FILE` | `/opt/haproxy/conf/domains.map` | Map file path |
| `HAPROXY_DB_FILE` | `/opt/haproxy/conf/haproxy_mcp.db` | Local SQLite DB path |
| `HAPROXY_REMOTE_DB_FILE` | `/opt/haproxy/conf/haproxy_mcp.db` | Remote SQLite DB path (SCP target) |
| `HAPROXY_SERVERS_FILE` | `/opt/haproxy/conf/servers.json` | Legacy servers config (migration only) |
| `HAPROXY_CERTS_FILE` | `/opt/haproxy/conf/certificates.json` | Legacy certs config (migration only) |
| `HAPROXY_POOL_COUNT` | `100` | Number of pool backends |
| `HAPROXY_MAX_SLOTS` | `10` | Max servers per pool |
| `HAPROXY_CONTAINER` | `haproxy` | Podman container name |
| `LOG_LEVEL` | `INFO` | Logging level (DEBUG/INFO/WARNING/ERROR) |

## Zero-Reload Domain Management

### How It Works
1. **SQLite DB** (`haproxy_mcp.db`): Single source of truth for all persistent data
2. **domains.map**: Generated from SQLite, maps domain → pool backend
3. **Pool backends**: 100 pre-configured backends (`pool_1` to `pool_100`)
4. **Runtime API**: Add/remove map entries and configure servers without reload

### Adding a Domain

```bash
haproxy_add_domain("example.com", "10.0.0.1")
# Creates: pool_N_1 → 10.0.0.1:80

haproxy_add_domain("api.example.com", "10.0.0.1", 8080)
# Creates: pool_N_1 → 10.0.0.1:8080
```

This will:
1. Find available pool (e.g., `pool_5`) via SQLite query
2. Save to SQLite DB (domain, backend, server)
3. Sync `domains.map` from DB (for HAProxy file-based lookup)
4. Update HAProxy runtime map: `add map ... example.com pool_5`
5. Configure server in pool via Runtime API (HTTP only)
6. Upload DB to remote host via SCP (persistence)

**No HAProxy reload required!**

### Pool Sharing

Multiple domains can share the same pool/backend servers using `share_with`:

```bash
# First domain gets its own pool
haproxy_add_domain("example.com", "10.0.0.1")
# Creates: example.com → pool_5

# Additional domains share the same pool
haproxy_add_domain("www.example.com", share_with="example.com")
haproxy_add_domain("api.example.com", share_with="example.com")
# Both use: pool_5 (same backend servers)
```

**Benefits:**
- Saves pool slots (100 pools can serve unlimited domains)
- Multiple domains → same backend servers
- Shared domains stored as `shares_with` in SQLite domains table

**Behavior:**
- Shared domains cannot specify `ip` (use existing servers)
- Removing a shared domain keeps servers intact
- Removing the last domain using a pool clears the servers

### Backend Configuration
- Backends always use HTTP (port 80 or custom)
- SSL/TLS termination happens at HAProxy frontend
- Each pool has 10 server slots (pool_N_1 to pool_N_10)
- **IPv6 supported**: Both IPv4 and IPv6 addresses accepted

### Bulk Server Operations
```bash
# Add multiple servers at once
haproxy_add_servers("api.example.com", '[
  {"slot": 1, "ip": "10.0.0.1", "http_port": 80},
  {"slot": 2, "ip": "10.0.0.2", "http_port": 80},
  {"slot": 3, "ip": "2001:db8::1", "http_port": 80}
]')
```

### Files
| File | Purpose |
|------|---------|
| `haproxy_mcp.db` | SQLite DB — single source of truth (domains, servers, certs) |
| `domains.map` | Domain → Pool mapping (generated from DB, HAProxy reads directly) |
| `wildcards.map` | Wildcard domain mapping (generated from DB) |
| `haproxy.cfg` | Pool backend definitions (static) |
| `servers.json` | Legacy — used only for initial migration to SQLite |
| `certificates.json` | Legacy — used only for initial migration to SQLite |

## SSL/TLS Certificates

### Current Setup
| Item | Value |
|------|-------|
| ACME Client | acme.sh (Google Trust Services CA) |
| Cert Directory | `/opt/haproxy/certs/` (auto-loaded by HAProxy) |
| acme.sh Home | `~/.acme.sh/` |

### How It Works
1. HAProxy binds with `crt /etc/haproxy/certs/` (directory, not file)
2. All `.pem` files in directory are auto-loaded
3. SNI selects correct certificate based on domain
4. acme.sh deploy hook auto-creates combined PEM on issue/renewal

### Adding New Certificate
```bash
# Issue certificate (Google Trust Services)
~/.acme.sh/acme.sh --issue \
  --dns dns_cf \
  -d "newdomain.com" -d "*.newdomain.com" \
  --reloadcmd "cat ~/.acme.sh/newdomain.com_ecc/fullchain.cer ~/.acme.sh/newdomain.com_ecc/newdomain.com.key > /opt/haproxy/certs/newdomain.com.pem && podman exec haproxy kill -USR2 1"
```

### Certificate Commands
```bash
# List certificates
~/.acme.sh/acme.sh --list

# Renew all
~/.acme.sh/acme.sh --cron

# Force renew specific domain
~/.acme.sh/acme.sh --renew -d "domain.com" --force

# Check loaded certs in HAProxy
ls -la /opt/haproxy/certs/
```

### Cloudflare Credentials
- File: `~/.secrets/cloudflare.ini`
- Format: `dns_cloudflare_api_token = TOKEN`
- Also export: `export CF_Token="your_token"`

## HTTP/3 (QUIC)

Frontend supports all protocols:
```
TCP 443: HTTP/1.1, HTTP/2
UDP 443: HTTP/3 (QUIC)
```

Clients receive `alt-svc: h3=":443"; ma=86400` header for HTTP/3 upgrade.

## MCP Security

HAProxy proxies MCP with Bearer Token authentication (configured in frontend ACL).

### Access Rules
| Source | Token Required |
|--------|----------------|
| Tailscale (100.64.0.0/10) | No |
| External (CF Worker, etc.) | Yes |

### Token Location
- File: `/opt/haproxy/conf/mcp-token.env`

### Connection Examples

**Tailscale (no auth):**
```bash
claude mcp add --transport http haproxy http://100.108.39.107:8000/mcp
```

**External (with auth):**
```bash
curl -X POST https://mcp.inouter.com/mcp \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}},"id":0}'
```

## Health Check

### System Health (`haproxy_health`)
Returns overall system status:
```json
{
  "status": "healthy",
  "components": {
    "mcp": {"status": "ok"},
    "haproxy": {"status": "ok", "version": "3.3.2", "uptime_sec": 3600},
    "config_files": {"status": "ok", "files": {"map_file": "ok", "db_file": "ok"}},
    "container": {"status": "ok", "state": "running"}
  }
}
```

### Domain Health (`haproxy_domain_health`)
Returns backend server status for a specific domain:
```json
{
  "domain": "api.example.com",
  "backend": "pool_3",
  "status": "healthy",
  "servers": [
    {"name": "pool_3_1", "addr": "10.0.0.1:80", "status": "UP", "check_status": "L4OK"}
  ],
  "healthy_count": 1,
  "total_count": 1
}
```

**Status values:** `healthy` (all UP), `degraded` (partial UP), `down` (all DOWN), `no_servers`

## MCP Tools (29 total)

### Domain Management
| Tool | Description |
|------|-------------|
| `haproxy_list_domains` | List domains (use `include_wildcards=true` for wildcards) |
| `haproxy_add_domain` | Add domain to pool (no reload), supports `share_with` for pool sharing |
| `haproxy_remove_domain` | Remove domain from pool (no reload) |

### Server Management
| Tool | Description |
|------|-------------|
| `haproxy_list_servers` | List servers for a domain |
| `haproxy_add_server` | Add server to slot (1-10 or 0=auto), auto-saved |
| `haproxy_add_servers` | Bulk add servers (JSON array), auto-saved |
| `haproxy_remove_server` | Remove server from slot, auto-saved |
| `haproxy_set_server_state` | Set state: ready/drain/maint |
| `haproxy_set_server_weight` | Set weight (0-256) |
| `haproxy_set_domain_state` | Set all servers of domain to ready/drain/maint |
| `haproxy_wait_drain` | Wait for connections to drain (timeout configurable) |

### Health Check
| Tool | Description |
|------|-------------|
| `haproxy_health` | System health (MCP, HAProxy, config files) |
| `haproxy_domain_health` | Domain-specific health with server status |
| `haproxy_get_server_health` | Get UP/DOWN/MAINT status for all servers |

### Monitoring
| Tool | Description |
|------|-------------|
| `haproxy_stats` | HAProxy statistics |
| `haproxy_backends` | List backends |
| `haproxy_list_frontends` | List frontends with status |
| `haproxy_get_connections` | Active connections per server |

### Configuration
| Tool | Description |
|------|-------------|
| `haproxy_reload` | Validate, reload config, auto-restore servers |
| `haproxy_check_config` | Validate config syntax |
| `haproxy_save_state` | Save server state to disk (legacy) |
| `haproxy_restore_state` | Restore state from disk (legacy) |

### Certificate Management (Zero-Downtime via Runtime API)
| Tool | Description |
|------|-------------|
| `haproxy_list_certs` | List all certificates with expiry info |
| `haproxy_cert_info` | Get detailed certificate info (expiry, issuer, SANs) |
| `haproxy_issue_cert` | Issue new certificate via acme.sh + Cloudflare DNS |
| `haproxy_renew_cert` | Renew specific certificate (force option available) |
| `haproxy_renew_all_certs` | Renew all certificates due for renewal |
| `haproxy_delete_cert` | Delete certificate from acme.sh and HAProxy |
| `haproxy_load_cert` | Load/reload certificate into HAProxy (manual trigger) |

## Key Conventions

### Pool-Based Routing
- Domains map to pools: `example.com` → `pool_N`
- 100 pools available: `pool_1` to `pool_100`
- Each pool has 10 server slots

### Server Naming
Pool backends use simple naming: `pool_N_1` to `pool_N_10`

**Examples:**
```
example.com → pool_5
  └─ pool_5_1 (slot 1)
  └─ pool_5_2 (slot 2)
  └─ ... up to pool_5_10

api.example.com → pool_6
  └─ pool_6_1 (slot 1)
```

### Persistence (SQLite)
- **SQLite DB** (`haproxy_mcp.db`): All domains, servers, certificates stored in single file
- **Remote sync**: DB uploaded to HAProxy host via SCP after every write
- **Startup restore**: DB downloaded from remote on pod start, servers restored via Runtime API
- **Migration**: First startup auto-migrates from legacy JSON/map files (one-time, idempotent)
- **WAL mode**: Write-Ahead Logging for concurrent access, checkpointed before SCP upload

### Safety Features
- **Atomic file writes**: Temp file + rename prevents corruption (map files)
- **SQLite transactions**: All DB writes are atomic
- **DB-first pattern**: Config saved to SQLite before HAProxy update, rollback on failure
- **Command validation**: HAProxy responses checked for errors
- **Input validation**: Domain format, IP (v4/v6), port range, slot limits
- **Bulk limits**: Max 10 servers per bulk add, 10KB JSON size limit

### Performance Optimization
- **Command batching**: Multiple HAProxy commands sent in single TCP connection
  - Server config (addr + state): 1 connection instead of 2
  - Startup restore: All servers restored in 1 connection (was 2×N for N servers)
  - Example: 7 servers restored = 1 connection (was 14 connections)

### Zero-Downtime Certificate Management
- **Runtime API**: Certificates loaded/updated without HAProxy reload
  - `new ssl cert` → `set ssl cert` → `commit ssl cert`
  - No connection drops during certificate changes
- **Persistence**: Certificate domains stored in SQLite `certificates` table
- **Auto-restore**: Certificates reloaded into HAProxy on MCP startup

## HAProxy Runtime API

```bash
# Show domain mappings
echo "show map /usr/local/etc/haproxy/domains.map" | nc localhost 9999

# Add domain mapping (runtime only)
echo "add map /usr/local/etc/haproxy/domains.map example.com pool_5" | nc localhost 9999

# Show server state
echo "show servers state" | nc localhost 9999

# Show backends
echo "show backend" | nc localhost 9999

# Set server address
echo "set server pool_1/pool_1_1 addr 10.0.0.1 port 80" | nc localhost 9999

# Enable server
echo "set server pool_1/pool_1_1 state ready" | nc localhost 9999
```

## Directory Structure

```
/opt/haproxy/
├── haproxy_mcp/         # MCP server package (streamable-http)
│   ├── server.py        # Main entry point
│   ├── config.py        # Configuration and constants
│   ├── db.py            # SQLite database (single source of truth)
│   ├── ssh_ops.py       # SSH/SCP remote execution
│   ├── exceptions.py    # Exception classes
│   ├── validation.py    # Input validation
│   ├── haproxy_client.py # HAProxy Runtime API client
│   ├── file_ops.py      # File I/O (delegates to db.py)
│   ├── utils.py         # Parsing utilities
│   └── tools/           # MCP tools (29 total)
│       ├── domains.py   # Domain management (3 tools)
│       ├── servers.py   # Server management (7 tools)
│       ├── health.py    # Health checks (3 tools)
│       ├── monitoring.py # Monitoring (4 tools)
│       ├── configuration.py # Config management (4 tools)
│       └── certificates.py # Certificate management (7 tools)
├── conf/
│   ├── haproxy.cfg      # Main HAProxy config (100 pool backends)
│   ├── haproxy_mcp.db   # SQLite DB (single source of truth)
│   ├── domains.map      # Domain → Pool mapping (generated from DB)
│   ├── wildcards.map    # Wildcard domain mapping (generated from DB)
│   ├── servers.json     # Legacy (migration source only)
│   ├── certificates.json # Legacy (migration source only)
│   └── mcp-token.env    # Bearer token for MCP auth
├── certs/               # SSL/TLS certificates (HAProxy PEM format)
├── data/                # Legacy state files
├── scripts/             # Utility scripts
└── run/                 # Runtime socket files

~/.acme.sh/
├── *_ecc/               # Certificate directories (one per domain)
└── acme.sh              # ACME client script

/etc/containers/systemd/
└── haproxy.container    # Podman Quadlet config
```

## Configuration Files

| File | Purpose |
|------|---------|
| `/opt/haproxy/conf/haproxy.cfg` | HAProxy main config |
| `/opt/haproxy/conf/haproxy_mcp.db` | SQLite DB (single source of truth) |
| `/opt/haproxy/conf/domains.map` | Domain routing map (generated from DB) |
| `/opt/haproxy/conf/wildcards.map` | Wildcard domain map (generated from DB) |
| `/opt/haproxy/certs/*.pem` | SSL certificates (auto-loaded) |
| `/etc/containers/systemd/haproxy.container` | Podman Quadlet |
| `/etc/systemd/system/haproxy-mcp.service` | MCP service |
| `/opt/haproxy/conf/mcp-token.env` | MCP auth token |
| `~/.acme.sh/` | acme.sh certificates and config |
| `~/.secrets/cloudflare.ini` | Cloudflare API token |

## Startup Sequence

```
1. systemd starts haproxy.service
   ↓
2. HAProxy loads haproxy.cfg (100 pool backends)
   ↓
3. HAProxy loads domains.map (domain → pool mapping)
   ↓
4. systemd starts haproxy-mcp.service (or K8s pod starts)
   ↓
5. init_db(): Download SQLite DB from remote host via SCP
   (first time: migrate from legacy JSON/map files → create DB → upload)
   ↓
6. MCP server restores servers from DB via Runtime API
   ↓
7. MCP server loads certificates from DB via Runtime API (zero-downtime)
   ↓
8. Ready to serve traffic
```

### Write Flow (runtime)
```
MCP tool call (add domain/server/cert)
   ↓
Write to local SQLite DB
   ↓
Sync map files to remote (domains only)
   ↓
Update HAProxy via Runtime API
   ↓
Upload DB to remote host via SCP (persistence)
```