Migrate analytics from SQLite to ClickHouse
SQLite was too slow for analytical aggregations on traffic_events and waf_events (millions of rows, GROUP BY, COUNT DISTINCT). ClickHouse is a columnar OLAP database purpose-built for this workload. - Add ClickHouse container to Docker Compose with health check - Create src/lib/clickhouse/client.ts with singleton client, table DDL, insert helpers, and all analytics query functions - Update log-parser.ts and waf-log-parser.ts to write to ClickHouse - Remove purgeOldEntries — ClickHouse TTL handles 90-day retention - Rewrite analytics-db.ts and waf-events.ts to query ClickHouse - Remove trafficEvents/wafEvents from SQLite schema, add migration - CLICKHOUSE_PASSWORD is required (no hardcoded default) - Update .env.example, README, and test infrastructure API response shapes are unchanged — no frontend modifications needed. Parse state (file offsets) remains in SQLite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
4
drizzle/0019_drop_analytics_tables.sql
Normal file
4
drizzle/0019_drop_analytics_tables.sql
Normal file
@@ -0,0 +1,4 @@
|
||||
-- Analytics data (traffic_events, waf_events) has been migrated to ClickHouse.
|
||||
DROP TABLE IF EXISTS traffic_events;
|
||||
--> statement-breakpoint
|
||||
DROP TABLE IF EXISTS waf_events;
|
||||
@@ -134,6 +134,13 @@
|
||||
"when": 1775600000000,
|
||||
"tag": "0018_forward_auth_redirect_intents",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 19,
|
||||
"version": "6",
|
||||
"when": 1775700000000,
|
||||
"tag": "0019_drop_analytics_tables",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user