Charon/docs/plans/archive/crowdsec-hub-bootstrap-spec.md

CrowdSec Hub Bootstrapping on Container Startup

Problem Statement

After a container rebuild, CrowdSec has zero collections installed and a stale hub index. When users (or the backend) attempt to install collections, they encounter hash mismatch errors because the hub index bundled in the image at build time is outdated by the time the container runs.

The root cause is twofold:

1. cscli hub update in the entrypoint only runs if .index.json is missing — not if it is stale.
2. install_hub_items.sh (which does call cscli hub update) is gated behind SECURITY_CROWDSEC_MODE=local, an env var that is deprecated and no longer set by default. The entrypoint checks $SECURITY_CROWDSEC_MODE, but the backend reads CERBERUS_SECURITY_CROWDSEC_MODE / CHARON_SECURITY_CROWDSEC_MODE — a naming mismatch that means the entrypoint gate never opens.

Current State Analysis

Dockerfile (build-time)

Aspect	What Happens
CrowdSec binaries	Built from source in crowdsec-builder stage, copied to /usr/local/bin/{crowdsec,cscli}
Config template	Source config copied to /etc/crowdsec.dist/
Hub index	Not pre-populated — no cscli hub update at build time
Collections	Not installed at build time
Symlink	/etc/crowdsec → /app/data/crowdsec/config created as root before USER charon
Helper scripts	install_hub_items.sh and register_bouncer.sh copied to /usr/local/bin/

Entrypoint (.docker/docker-entrypoint.sh, runtime)

Step	What Happens	Problem
Config init	Copies /etc/crowdsec.dist/* → /app/data/crowdsec/config/ on first run	Works correctly
Symlink verify	Confirms /etc/crowdsec → /app/data/crowdsec/config	Works correctly
LAPI port fix	sed replaces :8080 → :8085 in config files	Works correctly
Hub update (L313-315)	cscli hub update runs only if /etc/crowdsec/hub/.index.json does not exist	Bug: stale index is never refreshed
Machine registration	cscli machines add -a --force	Works correctly
Hub items install (L325-328)	Calls install_hub_items.sh only if $SECURITY_CROWDSEC_MODE = "local"	Bug: env var is deprecated, never set; wrong var name vs backend
Ownership fix	chown -R charon:charon on CrowdSec dirs	Works correctly

install_hub_items.sh (when invoked)

The script itself is well-structured — it calls cscli hub update then installs individual parsers, scenarios, and two collections (crowdsecurity/http-cve, crowdsecurity/base-http-scenarios). It does not install crowdsecurity/caddy.

Backend (crowdsec_startup.go)

ReconcileCrowdSecOnStartup() checks the database for CrowdSec mode and starts the process if needed. It does not call cscli hub update or install collections. The HubService.runCSCLI() method in hub_sync.go does call cscli hub update before individual item installs, but this is only triggered by explicit GUI actions (Pull/Apply), not at startup.

Proposed Changes

File: .docker/docker-entrypoint.sh

Change 1: Always refresh the hub index

Replace the conditional hub update (current L313-315):

# CURRENT (broken):
if [ ! -f "/etc/crowdsec/hub/.index.json" ]; then
    echo "Updating CrowdSec hub index..."
    timeout 60s cscli hub update 2>/dev/null || echo "⚠️ Hub update timed out or failed, continuing..."
fi

With an unconditional update:

# NEW: Always refresh hub index on startup (stale index causes hash mismatch errors)
echo "Updating CrowdSec hub index..."
if ! timeout 60s cscli hub update 2>&1; then
    echo "⚠️ Hub index update failed (network issue?). Collections may fail to install."
    echo "   CrowdSec will still start with whatever index is cached."
fi

Change 2: Always install required collections (remove env var gate)

Replace the conditional hub items install (current L322-328):

# CURRENT (broken — env var never set):
if [ "$SECURITY_CROWDSEC_MODE" = "local" ]; then
    echo "Installing CrowdSec hub items..."
    if [ -x /usr/local/bin/install_hub_items.sh ]; then
        /usr/local/bin/install_hub_items.sh 2>/dev/null || echo "Warning: Some hub items may not have installed"
    fi
fi

With unconditional execution:

# NEW: Always ensure required collections are present.
# This is idempotent — already-installed items are skipped by cscli.
# Collections are needed regardless of whether CrowdSec is GUI-enabled,
# because the user can enable CrowdSec at any time via the dashboard
# and expects it to work immediately.
echo "Ensuring CrowdSec hub items are installed..."
if [ -x /usr/local/bin/install_hub_items.sh ]; then
    /usr/local/bin/install_hub_items.sh || echo "⚠️ Some hub items may not have installed. CrowdSec can still start."
fi

File: configs/crowdsec/install_hub_items.sh

Change 3: Add crowdsecurity/caddy collection

Add after the existing crowdsecurity/base-http-scenarios install:

# Install Caddy collection (parser + scenarios for Caddy access logs)
echo "Installing Caddy collection..."
cscli collections install crowdsecurity/caddy --force 2>/dev/null || true

Change 4: Remove redundant individual parser installs that are included in collections

The crowdsecurity/base-http-scenarios collection already includes crowdsecurity/http-logs and several of the individually installed scenarios. The crowdsecurity/caddy collection includes crowdsecurity/caddy-logs. Keep the individual installs as fallbacks (they are idempotent), but add a comment noting the overlap. No lines need deletion — the --force flag and idempotency make this safe.

Summary of File Changes

File	Change	Lines
.docker/docker-entrypoint.sh	Unconditional cscli hub update	~L313-315
.docker/docker-entrypoint.sh	Remove SECURITY_CROWDSEC_MODE gate on hub items install	~L322-328
configs/crowdsec/install_hub_items.sh	Add crowdsecurity/caddy collection install	After L60

Edge Cases

Scenario	Handling
No network at startup	cscli hub update fails with timeout. The install_hub_items.sh also fails. Entrypoint continues — CrowdSec starts with whatever is cached (or no collections). User can retry via GUI.
Hub CDN returns 5xx	Same as no network — timeout + fallback.
Collections already installed	--force flag makes cscli collections install idempotent. It updates to latest if newer version available.
First boot (no prior data volume)	Config gets copied from .dist, hub update runs, collections install. Clean bootstrap path.
Existing data volume (upgrade)	Config already exists (skips copy), hub update refreshes stale index, collections install/upgrade.
install_hub_items.sh missing/not executable	-x check in entrypoint skips it with a log message. CrowdSec starts without collections.
CrowdSec disabled in GUI	Collections are still installed (they are just config files). No process runs until user enables via GUI. Zero runtime cost.

Startup Time Impact

• cscli hub update: ~2-5s (single HTTPS request to hub CDN)
• install_hub_items.sh: ~10-15s (multiple cscli invocations, each checking/installing)
• Total additional startup time: ~12-20s (first boot) / ~5-10s (subsequent boots, items cached)

This is acceptable for a container that runs long-lived.

Acceptance Criteria

1. After docker compose up with a fresh data volume, cscli collections list shows crowdsecurity/caddy, crowdsecurity/base-http-scenarios, and crowdsecurity/http-cve installed.
2. After docker compose up with an existing data volume (stale index), hub index is refreshed and collections remain installed.
3. If the container starts with no network, CrowdSec initialization logs warnings but does not crash or block startup.
4. No env var (SECURITY_CROWDSEC_MODE) is required for collections to be installed.
5. Startup time increase is < 30 seconds.

Commit Slicing Strategy

Decision: Single PR. Scope is small (3 files, ~15 lines changed), low risk, and all changes are tightly coupled.

PR-1: CrowdSec hub bootstrapping fix

• Scope: .docker/docker-entrypoint.sh, configs/crowdsec/install_hub_items.sh
• Validation: Manual docker rebuild + verify collections with cscli collections list
• Rollback: Revert PR; behavior returns to current (broken) state — no data loss risk