Charon/docs/plans/crowdsec_lapi_auth_fix.md

CrowdSec LAPI Authentication Fix

Issue Reference: Related to Issue #585 (CrowdSec Web Console Enrollment) Status: Ready for Implementation Priority: P1 (Blocking CrowdSec functionality) Created: 2026-02-03 Estimated Effort: 2-4 hours

Executive Summary

After a container rebuild, the CrowdSec integration fails with "access forbidden" when attempting to connect to the LAPI. This blocks all CrowdSec functionality including IP banning and web console enrollment testing.

Error Observed:

{
  "level": "error",
  "ts": 1770143945.8009417,
  "logger": "crowdsec",
  "msg": "failed to connect to LAPI, retrying in 10s: API error: access forbidden",
  "instance_id": "99c91cc1",
  "address": "http://127.0.0.1:8085"
}

---

Root Cause Analysis

Finding 1: Invalid Static API Key

Problem: User configured CHARON_SECURITY_CROWDSEC_API_KEY=charonbouncerkey2024 in docker-compose.yml.

Why This Fails: CrowdSec bouncer API keys must be generated by CrowdSec via cscli bouncers add <name>. The manually-specified key charonbouncerkey2024 was never registered with CrowdSec LAPI, so the LAPI rejects authentication with "access forbidden".

Evidence:

• registration.go: EnsureBouncerRegistered() first checks for env var API key, returns it if present (without validation)
• register_bouncer.sh: Script generates real API key via cscli bouncers add

Finding 2: Bouncer Not Auto-Registered on Start

Problem: When CrowdSec agent starts, the bouncer is never registered automatically.

Evidence:

• docker-entrypoint.sh: Only machine registration occurs: cscli machines add -a --force
• crowdsec_handler.go#L191-L295: Start() handler starts CrowdSec process but does NOT call bouncer registration
• crowdsec_handler.go#L1432-1478: RegisterBouncer() is a separate API endpoint (POST /api/v1/admin/crowdsec/bouncer/register) that must be called manually

Finding 3: Incorrect API URL Configuration (Minor)

Problem: User configured CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8080.

Why This Is Wrong:

• CrowdSec LAPI listens on port 8085 (configured in entrypoint via sed)
• Port 8080 is used by Charon management API
• Code default is correct: http://127.0.0.1:8085 (see config.go#L60-64)

Evidence from entrypoint:

# Configure CrowdSec LAPI to use port 8085 to avoid conflict with Charon (port 8080)
sed -i 's|listen_uri: 127.0.0.1:8080|listen_uri: 127.0.0.1:8085|g' /etc/crowdsec/config.yaml

Data Flow Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Current (Broken) Flow                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. Container starts                                                         │
│     ├─► docker-entrypoint.sh                                                 │
│     │   ├─► mkdir /app/data/crowdsec                                        │
│     │   ├─► Copy config to persistent storage                               │
│     │   ├─► cscli machines add -a --force  ✅ Machine registered            │
│     │   └─► ❌ NO bouncer registration                                       │
│                                                                              │
│  2. User enables CrowdSec via GUI                                            │
│     ├─► POST /api/v1/admin/crowdsec/start                                   │
│     │   ├─► SecurityConfig.CrowdSecMode = "local"                           │
│     │   ├─► Start crowdsec process                                          │
│     │   ├─► Wait for LAPI ready                                             │
│     │   └─► ❌ NO bouncer registration                                       │
│                                                                              │
│  3. Caddy loads CrowdSec bouncer config                                      │
│     ├─► GenerateConfig() in caddy/config.go                                 │
│     │   ├─► apiKey := getCrowdSecAPIKey()                                   │
│     │   │   └─► Returns "charonbouncerkey2024" from CHARON_SECURITY_...     │
│     │   └─► CrowdSecApp{APIKey: "charonbouncerkey2024", ...}                │
│                                                                              │
│  4. Caddy CrowdSec bouncer tries to connect                                  │
│     ├─► Connect to http://127.0.0.1:8085                                    │
│     ├─► Send API key "charonbouncerkey2024"                                 │
│     └─► ❌ LAPI responds: "access forbidden" (key not registered)           │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

---

Requirements (EARS Notation)

REQ-1: API Key Validation

WHEN a CrowdSec API key is provided via environment variable, THE SYSTEM SHALL validate that the key is actually registered with the LAPI before using it.

Acceptance Criteria:

• Validation occurs during CrowdSec start
• Invalid keys trigger a warning log
• System attempts auto-registration if validation fails

REQ-2: Auto-Registration on Start

WHEN CrowdSec starts and no valid bouncer is registered, THE SYSTEM SHALL automatically register a bouncer and store the generated API key.

Acceptance Criteria:

• Bouncer registration happens automatically after LAPI is ready
• Generated API key is persisted to /etc/crowdsec/bouncers/caddy-bouncer.key
• Caddy config is regenerated with the new API key

REQ-3: Caddy Config Regeneration

WHEN a new bouncer API key is generated, THE SYSTEM SHALL regenerate the Caddy configuration with the updated key.

Acceptance Criteria:

• Caddy config uses the newly generated API key
• No container restart required
• Bouncer connects successfully to LAPI

REQ-4: Upgrade Scenario Handling

WHEN upgrading from a broken state (invalid static key), THE SYSTEM SHALL heal automatically without manual intervention.

Acceptance Criteria:

• Works for both fresh installs and upgrades
• Volume-mounted data is preserved
• No manual cleanup required

---

Technical Design

Solution Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Fixed Flow                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. Container starts (unchanged)                                             │
│     └─► docker-entrypoint.sh                                                 │
│         └─► cscli machines add -a --force                                   │
│                                                                              │
│  2. User enables CrowdSec via GUI                                            │
│     └─► POST /api/v1/admin/crowdsec/start                                   │
│         ├─► Start crowdsec process                                          │
│         ├─► Wait for LAPI ready                                             │
│         ├─► 🆕 EnsureBouncerRegistered()                                     │
│         │   ├─► Check if env key is valid                                   │
│         │   ├─► If invalid, run register_bouncer.sh                         │
│         │   └─► Store key in database/file                                  │
│         ├─► 🆕 Regenerate Caddy config with new API key                     │
│         └─► Return success                                                   │
│                                                                              │
│  3. Caddy loads CrowdSec bouncer config                                      │
│     ├─► getCrowdSecAPIKey()                                                 │
│     │   ├─► Check env var first                                             │
│     │   ├─► 🆕 Check file /etc/crowdsec/bouncers/caddy-bouncer.key         │
│     │   └─► 🆕 Check database settings table                                │
│     └─► CrowdSecApp{APIKey: <valid-key>, ...}                               │
│                                                                              │
│  4. Caddy CrowdSec bouncer connects                                          │
│     └─► ✅ LAPI accepts valid key                                            │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Implementation Changes

Change 1: Update Start() Handler to Register Bouncer

File: backend/internal/api/handlers/crowdsec_handler.go

Location: After LAPI readiness check (line ~290)

// After confirming LAPI is ready, ensure bouncer is registered
if lapiReady {
    // Register bouncer if needed (idempotent)
    apiKey, regErr := h.ensureBouncerRegistration(ctx)
    if regErr != nil {
        logger.Log().WithError(regErr).Warn("Failed to register bouncer, CrowdSec may not enforce decisions")
    } else if apiKey != "" {
        // Store the API key for Caddy config generation
        h.storeAPIKey(ctx, apiKey)

        // Regenerate Caddy config with new API key
        if h.CaddyManager != nil {
            if err := h.CaddyManager.ReloadConfig(ctx); err != nil {
                logger.Log().WithError(err).Warn("Failed to reload Caddy config with new bouncer key")
            }
        }
    }
}

Change 2: Add ensureBouncerRegistration() Method

File: backend/internal/api/handlers/crowdsec_handler.go

// ensureBouncerRegistration checks if bouncer is registered and registers if needed.
// Returns the API key (empty string if already registered via env var).
func (h *CrowdsecHandler) ensureBouncerRegistration(ctx context.Context) (string, error) {
    // First check if env var key is actually valid
    envKey := getLAPIKey()
    if envKey != "" {
        // Validate the key is actually registered
        if h.validateBouncerKey(ctx, envKey) {
            return "", nil // Key is valid, nothing to do
        }
        logger.Log().Warn("Env-provided CrowdSec API key is invalid, will register new bouncer")
    }

    // Check if key file already exists
    keyFile := "/etc/crowdsec/bouncers/caddy-bouncer.key"
    if data, err := os.ReadFile(keyFile); err == nil {
        key := strings.TrimSpace(string(data))
        if key != "" && h.validateBouncerKey(ctx, key) {
            return key, nil // Key file is valid
        }
    }

    // Register new bouncer using script
    scriptPath := "/usr/local/bin/register_bouncer.sh"
    output, err := h.CmdExec.Execute(ctx, "bash", scriptPath)
    if err != nil {
        return "", fmt.Errorf("bouncer registration failed: %w: %s", err, string(output))
    }

    // Extract API key from output (last non-empty line)
    lines := strings.Split(strings.TrimSpace(string(output)), "\n")
    apiKey := ""
    for i := len(lines) - 1; i >= 0; i-- {
        line := strings.TrimSpace(lines[i])
        if len(line) >= 32 && !strings.Contains(line, " ") {
            apiKey = line
            break
        }
    }

    if apiKey == "" {
        return "", fmt.Errorf("bouncer registration output did not contain API key")
    }

    return apiKey, nil
}

// validateBouncerKey checks if an API key is actually registered with CrowdSec.
func (h *CrowdsecHandler) validateBouncerKey(ctx context.Context, key string) bool {
    // Use cscli bouncers list to check if 'caddy-bouncer' exists
    checkCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    output, err := h.CmdExec.Execute(checkCtx, "cscli", "bouncers", "list", "-o", "json")
    if err != nil {
        return false
    }

    // Parse JSON and check for caddy-bouncer
    var bouncers []struct {
        Name   string `json:"name"`
        APIKey string `json:"api_key"`
    }
    if err := json.Unmarshal(output, &bouncers); err != nil {
        return false
    }

    for _, b := range bouncers {
        if b.Name == "caddy-bouncer" {
            // Note: cscli doesn't return the full API key, just that it exists
            // We trust it's valid if the bouncer is registered
            return true
        }
    }

    return false
}

Change 3: Update getCrowdSecAPIKey() to Check Key File

File: backend/internal/caddy/config.go

Location: getCrowdSecAPIKey() function (line ~1129)

// getCrowdSecAPIKey retrieves the CrowdSec bouncer API key.
// Priority: environment variable > key file > empty
func getCrowdSecAPIKey() string {
    // Check environment variables first
    envVars := []string{
        "CROWDSEC_API_KEY",
        "CROWDSEC_BOUNCER_API_KEY",
        "CERBERUS_SECURITY_CROWDSEC_API_KEY",
        "CHARON_SECURITY_CROWDSEC_API_KEY",
        "CPM_SECURITY_CROWDSEC_API_KEY",
    }

    for _, key := range envVars {
        if val := os.Getenv(key); val != "" {
            return val
        }
    }

    // Check key file (generated by register_bouncer.sh)
    keyFile := "/etc/crowdsec/bouncers/caddy-bouncer.key"
    if data, err := os.ReadFile(keyFile); err == nil {
        key := strings.TrimSpace(string(data))
        if key != "" {
            return key
        }
    }

    return ""
}

Change 4: Add CaddyManager to CrowdsecHandler

File: backend/internal/api/handlers/crowdsec_handler.go

This may already exist, but ensure the handler can trigger Caddy config reload:

type CrowdsecHandler struct {
    DB           *gorm.DB
    BinPath      string
    DataDir      string
    Executor     CrowdsecExecutor
    CmdExec      CommandExecutor
    LAPIMaxWait  time.Duration
    LAPIPollInterval time.Duration
    CaddyManager *caddy.Manager  // ADD: For config reload
}

---

Immediate Workaround (User Action)

While the fix is being implemented, the user can:

1. Remove the static API key from docker-compose.yml:

   # REMOVE or comment out this line:
   # - CHARON_SECURITY_CROWDSEC_API_KEY=charonbouncerkey2024
   

2. Fix the API URL:

   # Change from:
   - CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8080
   # To:
   - CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8085
   

3. Manually register bouncer after container starts:

   docker exec -it charon /usr/local/bin/register_bouncer.sh
   

4. Restart container to pick up the new key:

   docker compose restart charon
   

---

Test Scenarios

Scenario 1: Fresh Install

1. Start container with no CrowdSec env vars
2. Enable CrowdSec via GUI
3. Expected: Bouncer auto-registers, no errors in logs

Scenario 2: Upgrade from Invalid Key

1. Start container with CHARON_SECURITY_CROWDSEC_API_KEY=invalid
2. Enable CrowdSec via GUI
3. Expected: System detects invalid key, registers new bouncer, logs warning

Scenario 3: Upgrade with Valid Key File

1. Container has /etc/crowdsec/bouncers/caddy-bouncer.key from previous run
2. Restart container, enable CrowdSec
3. Expected: Uses existing key file, no re-registration

Scenario 4: API URL Misconfiguration

1. Set CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8080 (wrong port)
2. Enable CrowdSec
3. Expected: Uses default 8085 port, logs warning about ignored URL

---

Implementation Checklist

• Task 1: Add validateBouncerKey() method to crowdsec_handler.go
• Task 2: Add ensureBouncerRegistration() method
• Task 3: Update Start() to call bouncer registration after LAPI ready
• Task 4: Update getCrowdSecAPIKey() in caddy/config.go to read from key file
• Task 5: Add integration test for bouncer auto-registration
• Task 6: Update documentation to clarify API key generation

---

Files to Modify

File	Change
backend/internal/api/handlers/crowdsec_handler.go	Add validation and auto-registration
backend/internal/caddy/config.go	Update getCrowdSecAPIKey()
docs/docker-compose.yml (examples)	Remove/update API key examples
README.md or SECURITY.md	Clarify CrowdSec setup

---

Risk Assessment

Risk	Likelihood	Impact	Mitigation
Breaking existing valid keys	Low	Medium	Only re-register if validation fails
register_bouncer.sh not present	Low	High	Check script existence before calling
Caddy reload fails	Low	Medium	Continue without bouncer, log warning
Race condition on startup	Low	Low	CrowdSec must finish starting first

---

References

• CrowdSec bouncer registration
• Caddy CrowdSec Plugin
• Issue #585 - CrowdSec Web Console Enrollment
• register_bouncer.sh
• docker-entrypoint.sh

---

Last Updated: 2026-02-03 Owner: TBD