Migrate analytics from SQLite to ClickHouse
SQLite was too slow for analytical aggregations on traffic_events and waf_events (millions of rows, GROUP BY, COUNT DISTINCT). ClickHouse is a columnar OLAP database purpose-built for this workload. - Add ClickHouse container to Docker Compose with health check - Create src/lib/clickhouse/client.ts with singleton client, table DDL, insert helpers, and all analytics query functions - Update log-parser.ts and waf-log-parser.ts to write to ClickHouse - Remove purgeOldEntries — ClickHouse TTL handles 90-day retention - Rewrite analytics-db.ts and waf-events.ts to query ClickHouse - Remove trafficEvents/wafEvents from SQLite schema, add migration - CLICKHOUSE_PASSWORD is required (no hardcoded default) - Update .env.example, README, and test infrastructure API response shapes are unchanged — no frontend modifications needed. Parse state (file offsets) remains in SQLite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
12
.env.example
12
.env.example
@@ -107,6 +107,18 @@ OAUTH_ALLOW_AUTO_LINKING=false # Auto-link OAuth to accounts without pas
|
||||
# LOGIN_BLOCK_MS=900000
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# CLICKHOUSE ANALYTICS (OPTIONAL)
|
||||
# =============================================================================
|
||||
|
||||
# ClickHouse is used for analytics data (traffic events, WAF events).
|
||||
# Data is automatically retained for 90 days via ClickHouse TTL.
|
||||
# CLICKHOUSE_PASSWORD is required — generate with: openssl rand -base64 32
|
||||
CLICKHOUSE_PASSWORD=your-clickhouse-password-here
|
||||
# CLICKHOUSE_URL=http://clickhouse:8123
|
||||
# CLICKHOUSE_USER=cpm
|
||||
# CLICKHOUSE_DB=analytics
|
||||
|
||||
# =============================================================================
|
||||
# GEOIP UPDATE (OPTIONAL)
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user