Charon/docs/analysis/crowdsec_integration_failure_analysis.md

CrowdSec Integration Test Failure Analysis

Date: 2026-01-28 PR: #550 - Alpine to Debian Trixie Migration CI Run: https://github.com/Wikid82/Charon/actions/runs/21456678628/job/61799104804 Branch: feature/beta-release

---

Issue Summary

The CrowdSec integration tests are failing after migrating the Dockerfile from Alpine to Debian Trixie base image. The test builds a Docker image and then tests CrowdSec functionality.

---

Potential Root Causes

1. CrowdSec Builder Stage Compatibility

Alpine vs Debian Differences:

• Alpine uses musl libc, Debian uses glibc
• Different package managers: apk (Alpine) vs apt (Debian)
• Different package names and availability

Current Dockerfile (lines 218-270):

FROM --platform=$BUILDPLATFORM golang:1.25.7-trixie AS crowdsec-builder

Dependencies Installed:

RUN apt-get update && apt-get install -y --no-install-recommends \
    git clang lld \
    && rm -rf /var/lib/apt/lists/*
RUN xx-apt install -y gcc libc6-dev

Possible Issues:

• Missing build dependencies: CrowdSec might require additional packages on Debian that were implicitly available on Alpine
• Git clone failures: Network issues or GitHub rate limiting
• Dependency resolution: go mod tidy might behave differently
• Cross-compilation issues: xx-go might need additional setup for Debian

2. CrowdSec Binary Path Issues

Runtime Image (lines 359-365):

# Copy CrowdSec binaries from the crowdsec-builder stage (built with Go 1.25.5+)
COPY --from=crowdsec-builder /crowdsec-out/crowdsec /usr/local/bin/crowdsec
COPY --from=crowdsec-builder /crowdsec-out/cscli /usr/local/bin/cscli
COPY --from=crowdsec-builder /crowdsec-out/config /etc/crowdsec.dist

Possible Issues:

• If the builder stage fails, these COPY commands will fail
• If fallback stage is used (for non-amd64), paths might be wrong

3. CrowdSec Configuration Issues

Entrypoint Script CrowdSec Init (docker-entrypoint.sh):

• Symlink creation from /etc/crowdsec to /app/data/crowdsec/config
• Configuration file generation and substitution
• Hub index updates

Possible Issues:

• Symlink already exists as directory instead of symlink
• Permission issues with non-root user
• Configuration templates missing or incompatible

4. Test Script Environment Issues

Integration Test (crowdsec_integration.sh):

• Builds the image with docker build -t charon:local .
• Starts container and waits for API
• Tests CrowdSec Hub connectivity
• Tests preset pull/apply functionality

Possible Issues:

• Build step timing out or failing silently
• Container failing to start properly
• CrowdSec processes not starting
• API endpoints not responding

---

Diagnostic Steps

Step 1: Check Build Logs

Review the CI build logs for the CrowdSec builder stage:

• Look for git clone errors
• Check for go get or go mod tidy failures
• Verify xx-go build completes successfully
• Confirm xx-verify passes

Step 2: Verify CrowdSec Binaries

Check if CrowdSec binaries are actually present:

docker run --rm charon:local which crowdsec
docker run --rm charon:local which cscli
docker run --rm charon:local cscli version

Step 3: Check CrowdSec Configuration

Verify configuration is properly initialized:

docker run --rm charon:local ls -la /etc/crowdsec
docker run --rm charon:local ls -la /app/data/crowdsec
docker run --rm charon:local cat /etc/crowdsec/config.yaml

Step 4: Test CrowdSec Locally

Run the integration test locally:

# Build image
docker build --no-cache -t charon:local .

# Run integration test
.github/skills/scripts/skill-runner.sh integration-test-crowdsec

---

Recommended Fixes

Fix 1: Add Missing Build Dependencies

If the build is failing due to missing dependencies, add them to the CrowdSec builder:

RUN apt-get update && apt-get install -y --no-install-recommends \
    git clang lld \
    build-essential pkg-config \
    && rm -rf /var/lib/apt/lists/*

Fix 2: Add Build Stage Debugging

Add debugging output to identify where the build fails:

# After git clone
RUN echo "CrowdSec source cloned successfully" && ls -la

# After dependency patching
RUN echo "Dependencies patched" && go mod graph | grep expr-lang

# After build
RUN echo "Build complete" && ls -la /crowdsec-out/

Fix 3: Use CrowdSec Fallback

If the build continues to fail, ensure the fallback stage is working:

# In final stage, use conditional COPY
COPY --from=crowdsec-fallback /crowdsec-out/bin/crowdsec /usr/local/bin/crowdsec || \
COPY --from=crowdsec-builder /crowdsec-out/crowdsec /usr/local/bin/crowdsec

Fix 4: Verify cscli Before Test

Add a verification step in the entrypoint:

if ! command -v cscli >/dev/null; then
    echo "ERROR: CrowdSec not installed properly"
    exit 1
fi

---

Next Steps

1. Access full CI logs to identify the exact failure point
2. Run local build to reproduce the issue
3. Add debugging output to the Dockerfile if needed
4. Verify fallback mechanism is working
5. Update test if CrowdSec behavior changed with new base image

---

Related Files

• Dockerfile (lines 218-310): CrowdSec builder and fallback stages
• .docker/docker-entrypoint.sh (lines 120-230): CrowdSec initialization
• .github/workflows/crowdsec-integration.yml: CI workflow
• scripts/crowdsec_integration.sh: Legacy integration test
• .github/skills/integration-test-crowdsec-scripts/run.sh: Modern test wrapper

---

Status

Current: Investigation in progress Priority: HIGH (CI blocking) Impact: Cannot merge PR #550 until resolved