Migrate analytics from SQLite to ClickHouse
SQLite was too slow for analytical aggregations on traffic_events and waf_events (millions of rows, GROUP BY, COUNT DISTINCT). ClickHouse is a columnar OLAP database purpose-built for this workload. - Add ClickHouse container to Docker Compose with health check - Create src/lib/clickhouse/client.ts with singleton client, table DDL, insert helpers, and all analytics query functions - Update log-parser.ts and waf-log-parser.ts to write to ClickHouse - Remove purgeOldEntries — ClickHouse TTL handles 90-day retention - Rewrite analytics-db.ts and waf-events.ts to query ClickHouse - Remove trafficEvents/wafEvents from SQLite schema, add migration - CLICKHOUSE_PASSWORD is required (no hardcoded default) - Update .env.example, README, and test infrastructure API response shapes are unchanged — no frontend modifications needed. Parse state (file offsets) remains in SQLite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,7 @@ services:
|
||||
SESSION_SECRET: "test-session-secret-32chars!xxxY"
|
||||
ADMIN_USERNAME: testadmin
|
||||
ADMIN_PASSWORD: "TestPassword2026!"
|
||||
CLICKHOUSE_PASSWORD: "test-clickhouse-password-2026"
|
||||
BASE_URL: http://localhost:3000
|
||||
NEXTAUTH_URL: http://localhost:3000
|
||||
# OAuth via Dex OIDC provider
|
||||
@@ -16,6 +17,9 @@ services:
|
||||
OAUTH_TOKEN_URL: http://dex:5556/dex/token
|
||||
OAUTH_USERINFO_URL: http://dex:5556/dex/userinfo
|
||||
OAUTH_ALLOW_AUTO_LINKING: "true"
|
||||
clickhouse:
|
||||
environment:
|
||||
CLICKHOUSE_PASSWORD: "test-clickhouse-password-2026"
|
||||
caddy:
|
||||
ports:
|
||||
- "80:80"
|
||||
@@ -84,3 +88,5 @@ volumes:
|
||||
name: caddy-logs-test
|
||||
geoip-data:
|
||||
name: geoip-data-test
|
||||
clickhouse-data:
|
||||
name: clickhouse-data-test
|
||||
|
||||
Reference in New Issue
Block a user