ThinkWatch Deployment Guide#

1. Prerequisites#

Hardware Requirements#

The server process itself is lightweight (Rust binary, ~50 MB RSS typical). Most resource consumption comes from PostgreSQL, Redis, and ClickHouse.

Software Requirements#

For development:

For production (Docker-only):

---

2. Development Setup#

2.1 Clone and Install Dependencies#

git clone <repository-url> ThinkWatch
cd ThinkWatch

# Install Rust toolchain (reads from rust-toolchain.toml)
rustup show

# Install web UI dependencies
cd web && pnpm install && cd ..

2.2 Start Infrastructure Services#

make infra

This starts:

• PostgreSQL on port 5432 (user: postgres, password: postgres, db: think_watch)
• Redis on port 6379
• ClickHouse (audit log storage) on port 8123 (HTTP) / 9000 (native TCP)
• Zitadel (OIDC SSO) on port 8080

Wait for all services to become healthy:

make infra-down   # to stop services

2.3 Start the Server#

# From the project root
cargo run

The server reads configuration from environment variables (with sensible development defaults). It will:

1. Connect to PostgreSQL and run migrations automatically.
2. Connect to Redis.
3. Start the gateway on http://localhost:3000.
4. Start the console on http://localhost:3001.

2.4 Start the Web UI (Development)#

cd web
pnpm dev

The Vite dev server starts on http://localhost:5173 and proxies API requests to the console server at localhost:3001.

2.5 Creating the First Admin User (Setup Wizard)#

On a fresh database, ThinkWatch provides a setup wizard accessible through the Web UI or the API. The setup wizard creates the first admin user and optionally configures the first AI provider in a single step.

Via the Web UI:

Open http://localhost:5173 in your browser. If the system has not been initialized, the UI will automatically show the setup wizard.

Via the API:

Check whether setup is needed:

curl http://localhost:3001/api/setup/status
# Returns: {"initialized": false, "needs_setup": true}

Initialize the system:

curl -X POST http://localhost:3001/api/setup/initialize \
  -H "Content-Type: application/json" \
  -d '{
    "admin": {
      "email": "admin@example.com",
      "display_name": "Admin",
      "password": "your-secure-password"
    },
    "provider": {
      "name": "openai-prod",
      "display_name": "OpenAI",
      "provider_type": "openai",
      "base_url": "https://api.openai.com/v1",
      "api_key": "sk-..."
    }
  }'

The provider field is optional. You can add providers later through the admin UI.

Note: The setup endpoint is rate-limited (5 requests per minute) and is disabled after the first admin user is created. See the security documentation for details.

Legacy method (manual):

Alternatively, register via POST /api/auth/register and then assign the super_admin role directly in PostgreSQL:

INSERT INTO user_roles (user_id, role_id, scope)
SELECT u.id, r.id, 'global'
FROM users u, roles r
WHERE u.email = 'admin@example.com' AND r.name = 'super_admin';

Subsequent users can be promoted to admin roles through the web UI by the super admin.

2.6 Configuring Zitadel SSO (Development)#

The dev compose file starts a Zitadel instance at http://localhost:8080.

1. Log in to Zitadel at http://localhost:8080 with username admin and password Admin1234!.
2. Create a new project and an OIDC application of type “Web” with:
   • Redirect URI: http://localhost:3001/api/auth/oidc/callback
   • Post-logout redirect: http://localhost:5173
3. Copy the client ID and client secret.
4. Set environment variables before starting the server:

export OIDC_ISSUER_URL=http://localhost:8080
export OIDC_CLIENT_ID=<your-client-id>
export OIDC_CLIENT_SECRET=<your-client-secret>
export OIDC_REDIRECT_URL=http://localhost:3001/api/auth/oidc/callback

Or add them to a .env file in the project root (loaded automatically by dotenvy).

---

3. Docker Compose Production Deployment#

3.1 Create Production Environment File#

Create a .env.production file with real secrets. Never commit this file to version control.

# Database
DB_USER=thinkwatch
DB_PASSWORD=<generate-a-strong-password>
DB_NAME=think_watch

# Authentication
JWT_SECRET=<generate-a-64-char-hex-string>
ENCRYPTION_KEY=<generate-a-64-char-hex-string>

# Ports
GATEWAY_PORT=3000
WEB_PORT=80

# CORS — set to your actual console domain
CORS_ORIGINS=https://console.yourdomain.com

# ClickHouse
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=<generate-a-strong-password>

# Redis
REDIS_PASSWORD=<generate-a-strong-password>

# OIDC SSO (optional)
OIDC_ISSUER_URL=https://auth.yourdomain.com
OIDC_CLIENT_ID=<client-id>
OIDC_CLIENT_SECRET=<client-secret>
OIDC_REDIRECT_URL=https://console.yourdomain.com/api/auth/oidc/callback

# Logging
RUST_LOG=info,think_watch=info

3.2 Generate Secure Secrets#

# JWT_SECRET (64 hex characters = 256 bits)
openssl rand -hex 32

# ENCRYPTION_KEY (64 hex characters = 256 bits, used for AES-256-GCM)
openssl rand -hex 32

# Database and Redis passwords
openssl rand -base64 24

3.3 Deploy#

Pull the pre-built images from GitHub Container Registry and start all services:

docker compose -f deploy/docker-compose.yml --env-file .env.production pull
docker compose -f deploy/docker-compose.yml --env-file .env.production up -d

To pin a release instead of latest, set IMAGE_TAG to its version, or to the SHA of a commit on main:

IMAGE_TAG=3.7.0 docker compose -f deploy/docker-compose.yml --env-file .env.production up -d

This starts:

• server — The ThinkWatch Rust binary (gateway on port 3000, console on port 3001 internal only)
• web — nginx serving the built React SPA (port 80)
• postgres — PostgreSQL 18 with persistent volume
• redis — Redis 8 with persistent volume
• clickhouse — ClickHouse columnar database for audit logs

3.4 Verify Health#

# Gateway health
curl http://localhost:3000/health

# Console health (from within the Docker network, or via the web container)
docker compose -f deploy/docker-compose.yml exec server curl http://localhost:3001/api/health

3.5 ClickHouse Configuration#

ClickHouse stores audit logs in a columnar format optimized for analytical queries. Configure the connection via these environment variables in .env.production:

# ClickHouse connection
CLICKHOUSE_URL=http://clickhouse:8123
CLICKHOUSE_DB=think_watch
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=<your-clickhouse-password>

ThinkWatch creates six tables in ClickHouse:

• audit_logs — Security audit trail (90-day TTL)
• gateway_logs — AI API request logs (90-day TTL)
• mcp_logs — MCP tool invocation logs (90-day TTL)
• platform_logs — Platform management operations (90-day TTL)
• access_logs — HTTP access logs (30-day TTL)
• app_logs — Application runtime logs (30-day TTL)

Tables are automatically created on first startup via deploy/clickhouse/init.sql.

3.6 Set Up a Reverse Proxy#

In production, place a reverse proxy in front of the Docker services to handle TLS termination. Only the gateway port (3000) and the web UI port (80) need to be reachable by end users.

Important: The console port (3001) should NOT be exposed to the public internet. Access it only through an internal network, VPN, or via the web container’s nginx that reverse-proxies /api/* to port 3001.

Example: nginx reverse proxy#

# Gateway — public-facing
server {
    listen 443 ssl;
    server_name gateway.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/gateway.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/gateway.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Required for SSE streaming
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
    }
}

# Console Web UI — restricted access
server {
    listen 443 ssl;
    server_name console.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/console.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/console.yourdomain.com/privkey.pem;

    # Optionally restrict by IP
    # allow 10.0.0.0/8;
    # deny all;

    location / {
        proxy_pass http://127.0.0.1:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Example: traefik (via Docker labels)#

Add labels to the server and web services in your compose override to configure Traefik routing and TLS with Let’s Encrypt automatically.

---

4. Kubernetes Deployment#

A Helm chart is provided at deploy/helm/think-watch/.

4.1 Images#

Each release publishes its images to GitHub Container Registry, tagged with the version and, for a stable release, latest. Every push to main also publishes images tagged with the commit SHA:

ghcr.io/thinkwatchproject/think-watch-server:<version>
ghcr.io/thinkwatchproject/think-watch-server:latest
ghcr.io/thinkwatchproject/think-watch-server:<git-sha>

ghcr.io/thinkwatchproject/think-watch-web:<version>
ghcr.io/thinkwatchproject/think-watch-web:latest
ghcr.io/thinkwatchproject/think-watch-web:<git-sha>

No authentication is needed — the packages are public.

4.2 Install with Helm#

helm install think-watch deploy/helm/think-watch \
  --set secrets.jwtSecret=$(openssl rand -hex 32) \
  --set secrets.encryptionKey=$(openssl rand -hex 32) \
  --set config.corsOrigins="https://console.internal.example.com"

The chart runs PostgreSQL, Redis and ClickHouse alongside the server. Secrets left empty are generated on the first install and kept across upgrades. To use managed databases instead, see 4.6 External PostgreSQL and Redis.

The chart deploys the images of its own version (appVersion in Chart.yaml). To deploy another tag, such as a commit on main:

helm install think-watch deploy/helm/think-watch \
  --set image.server.tag=<git-sha> \
  --set image.web.tag=<git-sha> \
  ...

4.3 Configure Ingress#

The Helm chart supports dual-host Ingress. Enable and configure in values.yaml or via --set:

ingress:
  enabled: true
  className: nginx
  gateway:
    host: gateway.yourdomain.com        # Public-facing
    tls:
      - secretName: gateway-tls
        hosts:
          - gateway.yourdomain.com
  console:
    host: console.internal.yourdomain.com  # Internal only
    tls:
      - secretName: console-tls
        hosts:
          - console.internal.yourdomain.com

For the console Ingress, use an internal ingress class or add annotations to restrict access:

ingress:
  console:
    annotations:
      nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,172.16.0.0/12"

4.4 External Secrets#

For production, use the External Secrets Operator instead of passing secrets via --set. The chart always creates its own Secret, <release>-secrets (think-watch-secrets for the release above), and the server reads its variables from it, so the ExternalSecret merges values into that Secret instead of creating one: install the chart first, then apply:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: think-watch
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend  # or aws-secrets-manager, etc.
    kind: SecretStore
  target:
    name: think-watch-secrets  # <release>-secrets, created by the chart
    creationPolicy: Merge
  data:
    - secretKey: JWT_SECRET
      remoteRef:
        key: think-watch/jwt-secret
    - secretKey: ENCRYPTION_KEY
      remoteRef:
        key: think-watch/encryption-key

On later upgrades the chart reads JWT_SECRET and ENCRYPTION_KEY back from the Secret, so the values from the store are kept; leave secrets.jwtSecret and secrets.encryptionKey unset, since values set there take precedence over the Secret. The server reads the Secret only at start, so restart it once after the first sync: kubectl rollout restart deployment/think-watch-server. DATABASE_URL and REDIS_URL cannot come from the store, because the chart writes them from postgres.externalUrl and redis.externalUrl on every upgrade (see 4.6).

4.5 Horizontal Pod Autoscaling#

Enable HPA in values.yaml:

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

The server is stateless (all state is in PostgreSQL and Redis), so it scales horizontally without issue.

4.6 External PostgreSQL and Redis#

A managed PostgreSQL or Redis can replace the bundled one. In the Helm chart, set bundled: false and externalUrl for that service; outside the chart, the URLs go in DATABASE_URL and REDIS_URL.

postgres:
  bundled: false
  externalUrl: postgres://user:pass@pg.example.com:5432/think_watch?sslmode=require
redis:
  bundled: false
  externalUrl: rediss://:pass@my-cache.example.com:6379

PostgreSQL behind a connection pooler. Each server instance sets up the schema when it starts, holding a session-level advisory lock so that instances starting together take turns. The URL must therefore reach PostgreSQL directly or through a pooler in session mode, never one in transaction mode (PgBouncer pool_mode = transaction, or the transaction-mode port of a managed pooler): there the lock can stay held on a server connection the pooler keeps, and every later start waits for it indefinitely. Schema setup uses the connections of DATABASE_URL; there is no separate URL for it.

Redis Cluster. Give the URL the redis-cluster:// scheme and name one node or more; the server finds the rest of the cluster from them:

redis-cluster://:pass@redis-0.redis:6379?node=redis-1.redis:6379&node=redis-2.redis:6379

Every node must be reachable from the server at the address it announces to the cluster (cluster-announce-ip / -port). A cluster has only database 0, so the URL names no /<db>.

Redis over TLS. Managed Redis services usually require TLS, for example ElastiCache with in-transit encryption, Upstash, Azure Cache for Redis and Redis Cloud. Use the rediss:// scheme, or rediss-cluster:// for a cluster, with the host name the service gives and the port it uses for TLS (a URL without a port means 6379; Azure Cache for Redis uses 6380). The certificate is checked against the public CAs and the host name in the URL; in a cluster, each node’s certificate must name the address that node announces.

A Redis whose certificate a private CA signed needs that CA: set REDIS_CA_CERT to the path of its PEM certificate, and the server then trusts only the certificates in that file for Redis. With the Helm chart, put the certificate in a Secret and name it under redis.caSecret; the chart mounts it and sets REDIS_CA_CERT:

kubectl -n thinkwatch create secret generic redis-ca --from-file=ca.crt=./ca.crt

redis:
  bundled: false
  externalUrl: rediss://:pass@redis.internal:6379
  caSecret:
    name: redis-ca
    key: ca.crt

The file is read at start, so restart the server after changing it. Client certificates (mutual TLS) are not supported: such a Redis needs tls-auth-clients no and password authentication. With networkPolicy.enabled, ports that no URL names, such as cluster nodes announcing other ports, go in networkPolicy.extraEgress.

---

5. SSL/TLS#

Using Let’s Encrypt with a Reverse Proxy#

The recommended approach is to terminate TLS at the reverse proxy layer. ThinkWatch itself serves plain HTTP on ports 3000 and 3001.

With certbot (standalone nginx):

certbot certonly --nginx -d gateway.yourdomain.com -d console.yourdomain.com

With Kubernetes cert-manager:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

Then reference the issuer in your Ingress annotations:

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-prod

Configuring CORS for HTTPS#

When the console is served over HTTPS, update the CORS_ORIGINS environment variable:

CORS_ORIGINS=https://console.yourdomain.com

Multiple origins can be comma-separated if needed.

---

6. Backup and Recovery#

6.1 PostgreSQL#

PostgreSQL contains all critical application state: users, API keys, provider configurations, usage records, and audit logs.

Scheduled pg_dump (simple):

# Daily backup
pg_dump -h localhost -U thinkwatch -d think_watch -Fc > backup_$(date +%Y%m%d).dump

# Restore
pg_restore -h localhost -U thinkwatch -d think_watch -c backup_20260401.dump

WAL archiving (point-in-time recovery):

For production deployments, configure PostgreSQL WAL archiving to an S3 bucket or network storage. This allows restoring to any point in time, not just the last dump. Use tools like pgBackRest or wal-g for automated WAL management.

Managed databases: If using a managed PostgreSQL service (AWS RDS, Google Cloud SQL, etc.), automated backups and point-in-time recovery are typically included.

6.2 Redis#

Redis stores ephemeral data (rate limit counters, session state, OIDC flow state). Losing Redis data is not catastrophic — rate limits reset and users need to log in again.

That said, to preserve rate limit state across restarts, enable Redis persistence:

# redis.conf
appendonly yes
appendfsync everysec

The Docker Compose files mount a persistent volume for Redis data by default.

6.3 ClickHouse#

ClickHouse stores audit logs in columnar tables. The Docker Compose files mount a persistent volume for ClickHouse data.

For disaster recovery:

• Back up the ClickHouse data volume, or use clickhouse-client to export data.
• Since audit logs are also written to PostgreSQL (audit_logs table), ClickHouse data can be re-inserted from the primary database if needed.

---

7. Monitoring#

7.1 Health Endpoints#

Recommended probe configuration for Kubernetes:

livenessProbe:
  httpGet:
    path: /health/live
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health/ready
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 5

Use /health/live for liveness probes (restart the pod if the process is stuck) and /health/ready for readiness probes (remove the pod from the load balancer if dependencies are down).

7.2 Structured Logging#

ThinkWatch uses tracing with configurable output via the RUST_LOG environment variable:

# Default: info level
RUST_LOG=info,think_watch=info

# Debug for ThinkWatch crates, info for dependencies
RUST_LOG=info,think_watch=debug

# Trace-level for detailed request/response debugging
RUST_LOG=info,think_watch=trace

# JSON-formatted logs (configured in tracing-subscriber setup)

Logs are written to stdout and can be collected by any log aggregation system (Datadog, Loki, CloudWatch, etc.).

7.3 Audit Log Forwarding#

ThinkWatch supports real-time forwarding of audit events to external systems via four channels:

Configure via the admin console at Admin > Log Forwarding:

• Multiple channels can run in parallel
• Each channel can be independently enabled/disabled
• Built-in connection testing and delivery statistics
• Automatic retry with error counting

Audit events are always written to PostgreSQL and ClickHouse (if configured). Forwarding channels provide additional delivery to external systems.

7.4 Prometheus Metrics#

ThinkWatch exposes Prometheus-compatible metrics at GET /metrics on the gateway port (3000). This endpoint is unauthenticated to allow standard Prometheus scraping.

Available metrics include:

• Request latency histograms (http_request_duration_seconds) — by endpoint, method, and status
• Token throughput counters (llm_tokens_total) — prompt and completion tokens by model
• Active connection gauges (http_active_connections) — current number of open connections
• Error rate counters (http_requests_errors_total) — by endpoint and error type

Prometheus scrape configuration:

scrape_configs:
  - job_name: 'thinkwatch'
    scrape_interval: 15s
    static_configs:
      - targets: ['thinkwatch:3000']
    metrics_path: /metrics

Security: The /metrics endpoint is unauthenticated. In production, restrict access via network policies or a reverse proxy to prevent exposing internal metrics to untrusted networks.

7.5 Background Tasks#

ThinkWatch runs several background tasks that operate automatically:

These tasks run within the server process and do not require external schedulers (e.g., cron). They are automatically active when the server starts.

---

8. Client Configuration Guide#

ThinkWatch includes a built-in Configuration Guide page in the web console at /guide. This page provides copy-paste setup instructions for connecting various AI client tools to your gateway and auto-detects the gateway URL.

Supported Client Tools#

The Configuration Guide provides ready-to-use instructions for:

• Claude Code — set ANTHROPIC_BASE_URL to your gateway’s /v1 endpoint (uses the Anthropic Messages API at /v1/messages)
• Cursor — configure the OpenAI-compatible endpoint in Cursor settings (uses /v1/chat/completions)
• Continue — configure the provider in ~/.continue/config.json (uses /v1/chat/completions)
• Cline — configure the API base URL in Cline settings (uses /v1/chat/completions)
• OpenAI SDK — set OPENAI_BASE_URL and OPENAI_API_KEY environment variables (uses /v1/chat/completions or /v1/responses)
• Anthropic SDK — set ANTHROPIC_BASE_URL and ANTHROPIC_API_KEY environment variables (uses /v1/messages)
• cURL — example commands for all three API formats
• ThinkWatch Lite — the Add to ThinkWatch Lite button adds the gateway to the desktop app (see below)

Adding the Gateway to ThinkWatch Lite#

The Add to ThinkWatch Lite button opens the New upstream dialog of ThinkWatch Lite with the gateway’s address and the key already filled in, so a user connects the desktop app in one step. It appears after a key is created or rotated, at the end of the first-run setup, and on the ThinkWatch Lite tab of the Configuration Guide.

The button needs ThinkWatch Lite 2026.10.16 or later. The address comes from the Public Gateway URL setting, and the link format is described in Import links. A gateway served over plain http:// on the company network works too; ThinkWatch Lite states that the key travels unencrypted.

Gateway API Endpoints#

The gateway (port 3000) serves three API formats on a single port:

All endpoints accept tw- API keys via the Authorization: Bearer header.

---

9. Upgrading#

9.1 Database Migrations#

Migrations are run automatically when the server starts. The sqlx migrate system tracks which migrations have been applied in a _sqlx_migrations table and only runs new ones.

Important: Always back up your database before upgrading to a new version that includes schema changes.

9.2 Rolling Updates (Kubernetes)#

The server is stateless, so rolling updates work out of the box:

# Update the image tag
helm upgrade think-watch deploy/helm/think-watch \
  --set image.server.tag=3.7.0 \
  --set image.web.tag=3.7.0 \
  --reuse-values

Kubernetes will perform a rolling update, starting new pods before terminating old ones. The first new pod to start will run any pending database migrations.

Tip: Set maxSurge: 1 and maxUnavailable: 0 in your deployment strategy to ensure zero downtime during upgrades.

9.3 Docker Compose Updates#

# Pull new images or rebuild
docker compose -f deploy/docker-compose.yml --env-file .env.production build

# Restart with new images (database migrations run on startup)
docker compose -f deploy/docker-compose.yml --env-file .env.production up -d

9.4 Breaking Changes Policy#

• Patch versions (0.1.x): Bug fixes only. No migration changes. Safe to upgrade without review.
• Minor versions (0.x.0): May include new migrations that add tables or columns. Always backward-compatible. Review the changelog.
• Major versions (x.0.0): May include breaking API changes, destructive migrations, or configuration changes. Read the upgrade guide carefully before proceeding.

Check the project changelog or release notes for migration details before upgrading.