akanealw/Charon
1
0
Fork 0
Code Issues Pull Requests Actions 16 Packages Projects Releases Wiki Activity

chore: clean .gitignore cache

Browse Source
This commit is contained in:
GitHub Actions
2026-01-26 19:21:33 +00:00
parent 1b1b3a70b1
commit e5f0fec5db
1483 changed files with 0 additions and 472793 deletions
Download Patch File Download Diff File Expand all files Collapse all files

426
docs/plans/crowdsec_reconciliation_failure.md
View File

@@ -1,426 +0,0 @@
# CrowdSec Reconciliation Failure Root Cause Analysis
**Date:** December 15, 2025
**Status:** CRITICAL - CrowdSec NOT starting despite 7+ commits attempting fixes
**Location:** `backend/internal/services/crowdsec_startup.go`
## Executive Summary
**The CrowdSec reconciliation function starts but exits silently** because the `security_configs` table **DOES NOT EXIST** in the production database. The table was added to AutoMigrate but the container was never rebuilt/restarted with a fresh database state after the migration code was added.
## The Silent Exit Point
Looking at the container logs:
```
{"bin_path":"crowdsec","data_dir":"/app/data/crowdsec","level":"info","msg":"CrowdSec reconciliation: starting startup check","time":"2025-12-14T20:55:39-05:00"}
```
Then... NOTHING. The function exits silently.
### Why It Exits
In `backend/internal/services/crowdsec_startup.go`, line 33-36:
```go
// Check if SecurityConfig table exists and has a record with CrowdSecMode = "local"
if !db.Migrator().HasTable(&models.SecurityConfig{}) {
logger.Log().Debug("CrowdSec reconciliation skipped: SecurityConfig table not found")
return
}
```
**This guard clause triggers because the table doesn't exist**, but it logs at **DEBUG** level, not INFO/WARN/ERROR. Since the container is running in production mode (not debug), this log message is never shown.
### Database Evidence
```bash
$ sqlite3 data/charon.db ".tables"
access_lists remote_servers
caddy_configs settings
domains ssl_certificates
import_sessions uptime_heartbeats
locations uptime_hosts
proxy_hosts uptime_monitors
notification_providers uptime_notification_events
notifications users
```
**NO `security_configs` TABLE EXISTS.** Yet the code in `backend/internal/api/routes/routes.go` clearly calls:
```go
if err := db.AutoMigrate(
// ... other models ...
&models.SecurityConfig{},
&models.SecurityDecision{},
&models.SecurityAudit{},
&models.SecurityRuleSet{},
// ...
); err != nil {
return fmt.Errorf("auto migrate: %w", err)
}
```
## Why AutoMigrate Didn't Create the Tables
### Theory 1: Database Persistence Across Rebuilds ✅ MOST LIKELY
The `charon.db` file is mounted as a volume in the Docker container:
```yaml
# docker-compose.yml
volumes:
- ./data:/app/data
```
**What happened:**
1. SecurityConfig model was added to AutoMigrate in recent commits
2. Container was rebuilt with `docker build -t charon:local .`
3. Container started with `docker compose up -d`
4. **BUT** the existing `data/charon.db` file (from before the migration code existed) was reused
5. GORM's AutoMigrate is **non-destructive** - it only adds new tables if they don't exist
6. The tables were never created because the database predates the migration code
### Theory 2: AutoMigrate Failed Silently
Looking at the logs, there is **NO** indication that AutoMigrate failed:
```
{"level":"info","msg":"starting Charon backend on version dev","time":"2025-12-14T20:55:39-05:00"}
{"bin_path":"crowdsec","data_dir":"/app/data/crowdsec","level":"info","msg":"CrowdSec reconciliation: starting startup check","time":"2025-12-14T20:55:39-05:00"}
{"level":"info","msg":"starting Charon backend on :8080","time":"2025-12-14T20:55:39-05:00"}
```
If AutoMigrate had failed, we would see an error from `routes.Register()` because it has:
```go
if err := db.AutoMigrate(...); err != nil {
return fmt.Errorf("auto migrate: %w", err)
}
```
Since the server started successfully, AutoMigrate either:
- Ran successfully but found the DB already in sync (no new tables to add)
- Never ran because the DB was opened but the tables already existed from a previous run
## The Cascading Failures
Because `security_configs` doesn't exist:
1. ✅ Reconciliation exits at line 33-36 (HasTable check)
2. ✅ CrowdSec is never started
3. ✅ Frontend shows "CrowdSec is not running" in Console Enrollment
4. ✅ Security page toggle is stuck ON (because there's no DB record to persist the state)
5. ✅ Log viewer shows "disconnected" (CrowdSec process doesn't exist)
6. ✅ All subsequent API calls fail because they expect the table to exist
## Why This Wasn't Caught During Development
Looking at the test files, **EVERY TEST** manually calls AutoMigrate:
```go
// backend/internal/services/crowdsec_startup_test.go:75
err = db.AutoMigrate(&models.SecurityConfig{})
// backend/internal/api/handlers/security_handler_coverage_test.go:25
require.NoError(t, db.AutoMigrate(&models.SecurityConfig{}, ...))
```
So tests **always create the table fresh**, hiding the issue that would occur in production with a persistent database.
## The Fix
### Option 1: Manual Database Migration (IMMEDIATE FIX)
Run this on the production container:
```bash
# Connect to running container
docker exec -it charon /bin/sh
# Run migration command (create a new CLI command in main.go)
./backend migrate
# OR manually create tables with sqlite3
sqlite3 /app/data/charon.db << EOF
CREATE TABLE IF NOT EXISTS security_configs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
name TEXT,
enabled BOOLEAN DEFAULT false,
admin_whitelist TEXT,
break_glass_hash TEXT,
crowdsec_mode TEXT DEFAULT 'disabled',
crowdsec_api_url TEXT,
waf_mode TEXT DEFAULT 'disabled',
waf_rules_source TEXT,
waf_learning BOOLEAN DEFAULT false,
waf_paranoia_level INTEGER DEFAULT 1,
waf_exclusions TEXT,
rate_limit_mode TEXT DEFAULT 'disabled',
rate_limit_enable BOOLEAN DEFAULT false,
rate_limit_burst INTEGER DEFAULT 10,
rate_limit_requests INTEGER DEFAULT 100,
rate_limit_window_sec INTEGER DEFAULT 60,
rate_limit_bypass_list TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS security_decisions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
ip TEXT NOT NULL,
reason TEXT,
action TEXT DEFAULT 'ban',
duration INTEGER,
expires_at DATETIME,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS security_audits (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
event_type TEXT,
ip_address TEXT,
details TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS security_rule_sets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
name TEXT NOT NULL,
type TEXT DEFAULT 'ip_list',
content TEXT,
enabled BOOLEAN DEFAULT true,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS crowdsec_preset_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
name TEXT NOT NULL,
description TEXT,
enabled BOOLEAN DEFAULT false,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS crowdsec_console_enrollments (
id INTEGER PRIMARY KEY AUTOINCREMENT,
uuid TEXT UNIQUE NOT NULL,
enrollment_key TEXT,
organization_id TEXT,
instance_name TEXT,
enrolled_at DATETIME,
status TEXT DEFAULT 'pending',
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
EOF
# Restart container
exit
docker restart charon
```
### Option 2: Add Migration CLI Command (CLEAN SOLUTION)
Add to `backend/cmd/api/main.go`:
```go
// Handle CLI commands
if len(os.Args) > 1 {
switch os.Args[1] {
case "migrate":
cfg, err := config.Load()
if err != nil {
log.Fatalf("load config: %v", err)
}
db, err := database.Connect(cfg.DatabasePath)
if err != nil {
log.Fatalf("connect database: %v", err)
}
logger.Log().Info("Running database migrations...")
if err := db.AutoMigrate(
&models.SecurityConfig{},
&models.SecurityDecision{},
&models.SecurityAudit{},
&models.SecurityRuleSet{},
&models.CrowdsecPresetEvent{},
&models.CrowdsecConsoleEnrollment{},
); err != nil {
log.Fatalf("migration failed: %v", err)
}
logger.Log().Info("Migration completed successfully")
return
case "reset-password":
// existing reset-password code
}
}
```
Then run:
```bash
docker exec charon /app/backend migrate
docker restart charon
```
### Option 3: Nuclear Option - Reset Database (DESTRUCTIVE)
```bash
# BACKUP FIRST
docker exec charon cp /app/data/charon.db /app/data/backups/charon-pre-security-migration.db
# Remove database
rm data/charon.db data/charon.db-shm data/charon.db-wal
# Restart container (will recreate fresh DB with all tables)
docker restart charon
```
## Fix Verification Checklist
After applying any fix, verify:
1. ✅ Check table exists:
```bash
docker exec charon sqlite3 /app/data/charon.db "SELECT name FROM sqlite_master WHERE type='table' AND name='security_configs';"
```
Expected: `security_configs`
2. ✅ Check reconciliation logs:
```bash
docker logs charon 2>&1 | grep -i "crowdsec reconciliation"
```
Expected: "starting CrowdSec" or "already running" (NOT "skipped: SecurityConfig table not found")
3. ✅ Check CrowdSec is running:
```bash
docker exec charon ps aux | grep crowdsec
```
Expected: `crowdsec -c /app/data/crowdsec/config/config.yaml`
4. ✅ Check frontend Console Enrollment:
- Navigate to `/security` page
- Click "Console Enrollment" tab
- Should show CrowdSec status as "Running"
5. ✅ Check toggle state persists:
- Toggle CrowdSec OFF
- Refresh page
- Toggle should remain OFF
## Code Improvements Needed
### 1. Change Debug Log to Warning
**File:** `backend/internal/services/crowdsec_startup.go:35`
```go
// BEFORE (line 35)
logger.Log().Debug("CrowdSec reconciliation skipped: SecurityConfig table not found")
// AFTER
logger.Log().Warn("CrowdSec reconciliation skipped: SecurityConfig table not found - run migrations")
```
**Rationale:** This is NOT a debug-level issue. If the table doesn't exist, it's a critical setup problem that should always be logged, regardless of debug mode.
### 2. Add Startup Migration Check
**File:** `backend/cmd/api/main.go` (after database.Connect())
```go
// Verify critical tables exist before starting server
requiredTables := []interface{}{
&models.SecurityConfig{},
&models.SecurityDecision{},
&models.SecurityAudit{},
&models.SecurityRuleSet{},
}
for _, model := range requiredTables {
if !db.Migrator().HasTable(model) {
logger.Log().Warnf("Missing table for %T - running migration", model)
if err := db.AutoMigrate(model); err != nil {
log.Fatalf("failed to migrate %T: %v", model, err)
}
}
}
```
### 3. Add Health Check for Tables
**File:** `backend/internal/api/handlers/health.go`
```go
func HealthHandler(c *gin.Context) {
db := c.MustGet("db").(*gorm.DB)
health := gin.H{
"status": "healthy",
"database": "connected",
"migrations": checkMigrations(db),
}
c.JSON(200, health)
}
func checkMigrations(db *gorm.DB) map[string]bool {
return map[string]bool{
"security_configs": db.Migrator().HasTable(&models.SecurityConfig{}),
"security_decisions": db.Migrator().HasTable(&models.SecurityDecision{}),
"security_audits": db.Migrator().HasTable(&models.SecurityAudit{}),
"security_rule_sets": db.Migrator().HasTable(&models.SecurityRuleSet{}),
}
}
```
## Related Issues
- Frontend toggle stuck in ON position → Database issue (no table to persist state)
- Console Enrollment says "not running" → CrowdSec never started (reconciliation exits)
- Log viewer disconnected → CrowdSec process doesn't exist
- All 7 previous commits failed because they addressed symptoms, not the root cause
## Lessons Learned
1. **Always log critical guard clauses at WARN level or higher** - Debug logs are invisible in production
2. **Verify database state matches code expectations** - AutoMigrate is non-destructive and won't fix missing tables from before the migration code existed
3. **Add database health checks** - Make missing tables visible in /api/v1/health endpoint
4. **Test with persistent databases** - All unit tests use fresh in-memory DBs, hiding this issue
5. **Add migration CLI command** - Allow operators to manually trigger migrations without container restart
## Recommended Action Plan
1. **IMMEDIATE:** Run Option 2 (Add migrate CLI command) and execute migration
2. **SHORT-TERM:** Apply Code Improvements #1 and #2
3. **LONG-TERM:** Add health check endpoint and integration tests with persistent DBs
4. **DOCUMENTATION:** Update deployment docs to mention migration requirement
## Status
- [x] Root cause identified (missing tables due to persistent DB from before migration code)
- [x] Silent exit point found (HasTable check with DEBUG logging)
- [x] Fix options documented
- [ ] Fix implemented
- [ ] Fix verified
- [ ] Code improvements applied
- [ ] Documentation updated