Charon/docs/troubleshooting/crowdsec.md

CrowdSec Troubleshooting

Keep Cerberus terminology and the Configuration Packages flow in mind while debugging Hub presets.

Quick checks

• Cerberus is enabled and you are signed in with admin scope.
• cscli is available (preferred path); HTTPS CrowdSec Hub endpoints only.
  • Docker images (v1.7.4+): cscli is pre-installed.
  • Bare-metal deployments: install cscli for Hub preset sync or use HTTP fallback with HUB_BASE_URL.
• HUB_BASE_URL points to a JSON hub endpoint (default: https://hub-data.crowdsec.net/api/index.json). Redirects to HTML will be rejected.
• Proxy env is set when required: HTTP(S)_PROXY and NO_PROXY are respected by the hub client.
• For slow or proxied networks, increase HUB_PULL_TIMEOUT_SECONDS (default 25) and HUB_APPLY_TIMEOUT_SECONDS (default 45) to avoid premature timeouts.
• Preset workflow: pull from Hub using cache keys/ETags → preview changes → apply with automatic backup and reload flag.
• Preset pull/apply requires either cscli or cached presets.
• Offline/curated presets remain available at all times.

LAPI Initialization and Timing

Understanding LAPI Startup

When you enable CrowdSec via the GUI toggle, the Local API (LAPI) needs time to initialize before it's ready to accept requests. This is normal behavior.

Typical startup times:

• Initial start: 5-10 seconds
• First start after container restart: 10-15 seconds
• Maximum wait: 30 seconds (with automatic retries)

What happens during startup:

1. CrowdSec process starts
2. Configuration is loaded
3. Database connections are established
4. Parsers and scenarios are loaded
5. LAPI becomes available on port 8085
6. Status changes from "Starting" to "Active"

Expected User Experience

When you toggle CrowdSec ON in the Security dashboard:

1. Loading overlay appears — "Starting CrowdSec... This may take up to 30 seconds"
2. Backend polls LAPI — Checks every 500ms for up to 30 seconds
3. Success toast displays — One of two messages:
   • ✅ "CrowdSec started and LAPI is ready" — You can immediately enroll in Console
   • ⚠️ "CrowdSec started but LAPI is still initializing" — Wait before enrolling

Verifying LAPI Status

Check if LAPI is running:

docker exec charon cscli lapi status

Expected output when ready:

✓ You can successfully interact with Local API (LAPI)

If LAPI is not ready yet:

ERROR: connection refused

Check LAPI health endpoint directly:

docker exec charon curl -s http://localhost:8085/health

Expected response when healthy:

{"status":"up"}

Troubleshooting LAPI Initialization

Problem: LAPI takes longer than 30 seconds

Symptoms:

• Warning message: "LAPI is still initializing"
• Console enrollment fails with "LAPI not available"

Solution 1 - Wait and retry:

# Wait 15 seconds, then check again
sleep 15
docker exec charon cscli lapi status

Solution 2 - Check CrowdSec logs:

docker logs charon | grep -i crowdsec | tail -20

Look for:

• ✅ "CrowdSec Local API listening" — LAPI started successfully
• ✅ "parsers loaded" — Configuration loaded
• ❌ "error" or "fatal" — Initialization problem

Solution 3 - Restart CrowdSec:

1. Go to Security dashboard
2. Toggle CrowdSec OFF
3. Wait 5 seconds
4. Toggle CrowdSec ON
5. Wait 15 seconds
6. Verify status shows "Active"

Problem: LAPI never becomes available

Check if CrowdSec process is running:

docker exec charon ps aux | grep crowdsec

Expected output:

crowdsec  203  0.5  2.3  /usr/local/bin/crowdsec -c /app/data/crowdsec/config/config.yaml

If no process is running:

1. Check config directory exists:

   docker exec charon ls -la /app/data/crowdsec/config
   

2. If directory is missing:

   docker compose restart
   

3. Check for port conflicts:

   docker exec charon netstat -tulpn | grep 8085
   

4. Remove deprecated environment variables from docker-compose.yml (see migration section below)

Problem: LAPI responds but enrollment fails

Check LAPI can process requests:

docker exec charon cscli machines list

Expected output:

Name                     IP Address      Auth Type    Version
charon-local-machine     127.0.0.1      password     v1.x.x

If command fails:

• LAPI is running but database is not ready
• Wait 10 more seconds and retry
• Check logs for database errors

If enrollment still fails:

• Enrollment has automatic retry (3 attempts, 2 seconds apart)
• If all retries fail, toggle CrowdSec OFF/ON and try again
• See Console Enrollment section below for token troubleshooting

Common issues

• Hub unreachable (503): retry once, then Charon falls back to cached Hub data if available; otherwise stay on curated/offline presets until connectivity returns.
• Hub returns HTML/redirect: set HUB_BASE_URL to the JSON endpoint above or install cscli so the index is fetched locally.
• Bad preset slug (400): the slug must match Hub naming; correct the slug before retrying.
• Apply failed: review the apply response and restore from the backup that was taken automatically, then retry after fixing the underlying issue.
• Apply not supported (501): use curated/offline presets; Hub apply will be re-enabled when supported in your environment.
• Security Engine Offline: If your dashboard says "Offline", it means CrowdSec LAPI is not running.
  • Fix: Ensure CrowdSec is enabled via GUI toggle in the Security dashboard. Do NOT use environment variables.
  • Action: Go to Security dashboard, toggle CrowdSec ON, wait 15 seconds, verify status shows "Active".

Tips

• Keep the CrowdSec Hub reachable over HTTPS; HTTP is blocked.
• If you switch to offline mode, clear pending Hub pulls before retrying so cache keys/ETags refresh cleanly.
• After restoring from a backup, re-run preview before applying again to verify changes.

Database Migrations After Upgrade

Problem: CrowdSec not starting after upgrading Charon

Symptoms:

• CrowdSec toggle appears enabled but status shows "Not Running"
• CrowdSec console shows "Starting..." indefinitely
• Container logs show: WARN CrowdSec reconciliation: security tables missing
• Console enrollment fails immediately

Root Cause:

Upgrading from an older version with a persistent database may be missing the new security tables introduced in version 2.0. The database schema needs to be migrated.

Solution: Run Database Migration

1. Execute the migration command:

   docker exec charon /app/charon migrate
   

   Expected output:

   {"level":"info","msg":"Running database migrations for security tables...","time":"..."}
   {"level":"info","msg":"Migration completed successfully","time":"..."}
   

2. Verify tables were created:

   docker exec charon sqlite3 /app/data/charon.db ".tables"
   

   Expected tables include:

   • security_configs
   • security_decisions
   • security_audits
   • security_rule_sets
   • crowdsec_preset_events
   • crowdsec_console_enrollments

3. Restart container to apply changes:

   docker restart charon
   

4. Verify CrowdSec starts automatically:

   If you had CrowdSec enabled before the upgrade:

   # Wait 15 seconds after restart, then check
   docker exec charon cscli lapi status
   

   Expected output:

   ✓ You can successfully interact with Local API (LAPI)
   

5. If CrowdSec doesn't auto-start:

   Enable it manually via the GUI:

   • Go to Security dashboard
   • Toggle CrowdSec ON
   • Wait 15 seconds
   • Verify status shows "Active"

Why This Happens:

Charon version 2.0 moved CrowdSec configuration from environment variables to the database (see Migration Guide). Persistent databases from older versions need the new security tables added via migration.

Prevention:

Future upgrades will run migrations automatically on startup. For now, manual migration is required for existing installations.

Related Documentation:

• Getting Started - Database Migrations
• Migration Guide - CrowdSec Control

---

Console Enrollment

Prerequisites

Before attempting Console enrollment, ensure:

✅ CrowdSec is enabled — Toggle must be ON in Security dashboard ✅ LAPI is running — Check with: docker exec charon cscli lapi status ✅ Feature flag enabled — feature.crowdsec.console_enrollment must be ON ✅ Valid token — Obtain from crowdsec.net

"missing login field" or CAPI errors

Charon automatically attempts to register your instance with CrowdSec's Central API (CAPI) before enrolling. Ensure your server has internet access to api.crowdsec.net.

Enrollment shows "enrolled" but not on crowdsec.net

Root cause: LAPI was not running when enrollment was attempted.

Charon now checks LAPI availability before enrollment and retries automatically (3 attempts with 2-second delays), but in rare cases enrollment may still fail if LAPI is initializing.

Solution:

1. Verify LAPI status:

   docker exec charon cscli lapi status
   

   Expected output when ready:

   ✓ You can successfully interact with Local API (LAPI)
   

   If LAPI is not running:

   ERROR: cannot contact local API
   

2. If LAPI is not running:

   • Go to Security dashboard
   • Toggle CrowdSec OFF
   • Wait 5 seconds
   • Toggle CrowdSec ON
   • Wait 15 seconds (important: LAPI needs time to initialize)
   • Re-check LAPI status

3. Verify LAPI health endpoint:

   docker exec charon curl -s http://localhost:8085/health
   

   Expected response:

   {"status":"up"}
   

4. Re-submit enrollment token:

   • Go to Cerberus → CrowdSec
   • Click Enroll with CrowdSec Console
   • Paste the same enrollment token (tokens are reusable)
   • Click Submit
   • Wait 30-60 seconds for confirmation

5. Verify enrollment on crowdsec.net:

   • Log in to your CrowdSec Console account
   • Navigate to Instances
   • Your Charon instance should appear in the list

Understanding the automatic retry:

Charon automatically retries LAPI checks during enrollment:

• Attempt 1: Immediate check
• Attempt 2: After 2 seconds (if LAPI not ready)
• Attempt 3: After 4 seconds (if still not ready)
• Total: 3 attempts over 6 seconds

This handles most cases where LAPI is still initializing. If all 3 attempts fail, follow the solution above.

CrowdSec won't start via GUI toggle

Solution:

1. Check container logs:

   docker logs charon | grep -i crowdsec
   

   Look for:

   • ✅ "Starting CrowdSec Local API"
   • ✅ "CrowdSec Local API listening on 127.0.0.1:8085"
   • ❌ "failed to start" or "error loading config"

2. Verify config directory:

   docker exec charon ls -la /app/data/crowdsec/config
   

   Expected files:

   • config.yaml — Main configuration
   • local_api_credentials.yaml — LAPI authentication
   • acquis.yaml — Log sources

3. Check for common startup errors:

   Error: "config.yaml not found"

   # Restart container to regenerate config
   docker compose restart
   

   Error: "port 8085 already in use"

   # Check for conflicting services
   docker exec charon netstat -tulpn | grep 8085
   # Stop conflicting service or change CrowdSec LAPI port
   

   Error: "permission denied"

   # Fix ownership (run on host)
   sudo chown -R 1000:1000 ./data/crowdsec
   docker compose restart
   

4. Remove any deprecated environment variables from docker-compose.yml:

   # REMOVE THESE:
   - CHARON_SECURITY_CROWDSEC_MODE=local
   - CERBERUS_SECURITY_CROWDSEC_MODE=local
   - CPM_SECURITY_CROWDSEC_MODE=local
   

5. Restart and try GUI toggle again:

   docker compose restart
   # Wait 30 seconds for container to fully start
   # Then toggle CrowdSec ON in GUI
   

6. Verify CrowdSec is running:

   # Check process
   docker exec charon ps aux | grep crowdsec
   
   # Check LAPI health
   docker exec charon cscli lapi status
   
   # Check LAPI endpoint
   docker exec charon curl -s http://localhost:8085/health
   

Environment Variable Migration

🚨 DEPRECATED: The CHARON_SECURITY_CROWDSEC_MODE environment variable is no longer used.

If you have this in your docker-compose.yml, remove it and use the GUI toggle instead. See Migration Guide for step-by-step instructions.

Configuration File

Charon uses the configuration located in data/crowdsec/config.yaml. Ensure this file exists and is readable if you are manually modifying it.