Migrate analytics from SQLite to ClickHouse

SQLite was too slow for analytical aggregations on traffic_events and
waf_events (millions of rows, GROUP BY, COUNT DISTINCT). ClickHouse is
a columnar OLAP database purpose-built for this workload.

- Add ClickHouse container to Docker Compose with health check
- Create src/lib/clickhouse/client.ts with singleton client, table DDL,
  insert helpers, and all analytics query functions
- Update log-parser.ts and waf-log-parser.ts to write to ClickHouse
- Remove purgeOldEntries — ClickHouse TTL handles 90-day retention
- Rewrite analytics-db.ts and waf-events.ts to query ClickHouse
- Remove trafficEvents/wafEvents from SQLite schema, add migration
- CLICKHOUSE_PASSWORD is required (no hardcoded default)
- Update .env.example, README, and test infrastructure

API response shapes are unchanged — no frontend modifications needed.
Parse state (file offsets) remains in SQLite.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
fuomag9
2026-04-10 00:05:38 +02:00
parent 833284efb1
commit e1c97038d4
21 changed files with 819 additions and 763 deletions

View File

@@ -0,0 +1,4 @@
-- Analytics data (traffic_events, waf_events) has been migrated to ClickHouse.
DROP TABLE IF EXISTS traffic_events;
--> statement-breakpoint
DROP TABLE IF EXISTS waf_events;

View File

@@ -134,6 +134,13 @@
"when": 1775600000000,
"tag": "0018_forward_auth_redirect_intents",
"breakpoints": true
},
{
"idx": 19,
"version": "6",
"when": 1775700000000,
"tag": "0019_drop_analytics_tables",
"breakpoints": true
}
]
}