Charon/docs/reports/qa_final_crowdsec_validation.md

QA Final CrowdSec Validation Report

Date: December 15, 2025 QA Agent: QA_Security Test Environment: Fresh no-cache Docker build

VERDICT: ❌ FAIL

CrowdSec infrastructure is operational but traffic blocking is NOT working.

---

Test Results Summary

✅ PASS: Infrastructure Components

Component	Status	Evidence
CrowdSec Process	✅ RUNNING	PID 67, verified via logs
CrowdSec LAPI	✅ HEALTHY	Listening on 127.0.0.1:8085
Caddy App Config	✅ POPULATED	apps.crowdsec is non-null
Bouncer Registration	✅ REGISTERED	charon-caddy-bouncer active
Bouncer Last Pull	✅ ACTIVE	2025-12-15T18:01:21Z
Environment Variables	✅ SET	All required vars configured

❌ FAIL: Traffic Blocking

Test	Expected	Actual	Result
Banned IP (172.16.0.99)	403 Forbidden	200 OK	❌ FAIL
Normal Traffic	200 OK	200 OK	✅ PASS
Decision in LAPI	Present	Present	✅ PASS
Decision Streamed	Yes	Yes	✅ PASS
Bouncer Blocking	Active	INACTIVE	❌ FAIL

---

Detailed Evidence

1. Database Enable Status

Method: Environment variables in docker-compose.override.yml

- CHARON_SECURITY_CROWDSEC_MODE=local
- CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8080
- CHARON_SECURITY_CROWDSEC_API_KEY=charonbouncerkey2024
- CERBERUS_SECURITY_CERBERUS_ENABLED=true

Status: ✅ Configured correctly

2. App-Level Config Verification

Command: docker exec charon curl -s http://localhost:2019/config/ | jq '.apps.crowdsec'

Output:

{
  "api_key": "charonbouncerkey2024",
  "api_url": "http://127.0.0.1:8085",
  "enable_streaming": true,
  "ticker_interval": "60s"
}

Status: ✅ Non-null and properly configured

3. Bouncer Registration

Command: docker exec charon cscli bouncers list

Output:

-----------------------------------------------------------------------------------------------------
 Name                  IP Address  Valid  Last API pull         Type              Version  Auth Type
-----------------------------------------------------------------------------------------------------
 charon-caddy-bouncer  127.0.0.1   ✔️     2025-12-15T18:01:21Z  caddy-cs-bouncer  v0.9.2   api-key
-----------------------------------------------------------------------------------------------------

Status: ✅ Registered and actively pulling

4. Decision Creation

Command: docker exec charon cscli decisions add --ip 172.16.0.99 --duration 15m --reason "FINAL QA TEST"

Output:

+----+--------+----------------+---------------+--------+---------+----+--------+------------+----------+
| ID | Source |   Scope:Value  |     Reason    | Action | Country | AS | Events | expiration | Alert ID |
+----+--------+----------------+---------------+--------+---------+----+--------+------------+----------+
| 1  | cscli  | Ip:172.16.0.99 | FINAL QA TEST | ban    |         |    | 1      | 14m55s     | 1        |
+----+--------+----------------+---------------+--------+---------+----+--------+------------+----------+

Status: ✅ Decision created successfully

5. Decision Streaming Verification

Command: docker exec charon curl -s 'http://localhost:8085/v1/decisions/stream?startup=true' -H "X-Api-Key: charonbouncerkey2024"

Output:

{"deleted":null,"new":[{"duration":"13m58s","id":1,"origin":"cscli","scenario":"FINAL QA TEST","scope":"Ip","type":"ban","u...

Status: ✅ Decision is being streamed from LAPI

6. Traffic Blocking Test (CRITICAL FAILURE)

Test Command: curl -H "X-Forwarded-For: 172.16.0.99" http://localhost/ -v

Expected Result: HTTP/1.1 403 Forbidden with CrowdSec block message

Actual Result:

< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Alt-Svc: h3=":443"; ma=2592000
< Content-Length: 2367
< Content-Type: text/html; charset=utf-8

Status: ❌ FAIL - Request was NOT blocked

7. Bouncer Handler Verification

Command: docker exec charon curl -s http://localhost:2019/config/ | jq -r '.apps.http.servers | ... | select(.handler == "crowdsec")'

Output: Found crowdsec handler in multiple routes (5+ instances)

Status: ✅ Handler is registered in routes

8. Normal Traffic Test

Command: curl http://localhost/ -v

Result: HTTP/1.1 200 OK

Status: ✅ PASS - Normal traffic flows correctly

---

Root Cause Analysis

Primary Issue: Bouncer Not Transitioning from Startup Mode

Evidence:

• Bouncer continuously polls with startup=true parameter
• Log entries show: GET /v1/decisions/stream?additional_pull=false&community_pull=false&startup=true
• This parameter should only be present during initial bouncer startup
• After initial pull, bouncer should switch to continuous streaming mode

Technical Details:

1. Caddy CrowdSec bouncer initializes in "startup" mode
2. Makes initial pull to get all existing decisions
3. Should transition to streaming mode where it receives decision updates in real-time
4. Actual behavior: Bouncer stays in startup mode indefinitely
5. Because it's in startup mode, it may not be actively applying decisions to traffic

Secondary Issues Identified

1. Decision Application Lag

   • Even though decisions are streamed, there's no evidence they're being applied to the in-memory decision store
   • No blocking logs appear in Caddy access logs
   • No "blocked by CrowdSec" entries in security logs

2. Potential Middleware Ordering

   • CrowdSec handler is present in routes but may be positioned after other handlers
   • Could be bypassed if reverse_proxy handler executes first

3. Client IP Detection

   • Tested with X-Forwarded-For: 172.16.0.99
   • Bouncer may not be reading this header correctly
   • No trusted_proxies configuration present in bouncer config

---

Configuration State

Caddy CrowdSec App Config

{
  "api_key": "charonbouncerkey2024",
  "api_url": "http://127.0.0.1:8085",
  "enable_streaming": true,
  "ticker_interval": "60s"
}

Missing Fields:

• ❌ trusted_proxies - Required for X-Forwarded-For support
• ❌ captcha_provider - Optional but recommended
• ❌ ban_template_path - Custom block page

Environment Variables

CHARON_SECURITY_CROWDSEC_MODE=local
CHARON_SECURITY_CROWDSEC_API_URL=http://localhost:8080  # ⚠️ Should be 8085
CHARON_SECURITY_CROWDSEC_API_KEY=charonbouncerkey2024
CERBERUS_SECURITY_CERBERUS_ENABLED=true

Issue: LAPI URL is set to 8080 (Charon backend) instead of 8085 (CrowdSec LAPI) Impact: Bouncer is connecting correctly because Caddy config uses 127.0.0.1:8085, but environment variable inconsistency could cause issues

---

Pre-Commit Checks

Status: ✅ ALL PASSED (Run at beginning of session)

---

Integration Test

Script: scripts/crowdsec_startup_test.sh Last Run Status: ❌ FAIL (Exit code 1) Note: Integration test was run in previous session; container restart invalidated results

---

ABSOLUTE REQUIREMENTS FOR PASS

Requirement	Status
✅ apps.crowdsec is non-null	PASS
✅ Bouncer registered in cscli bouncers list	PASS
❌ Test IP returns 403 Forbidden	FAIL
✅ Normal traffic returns 200 OK	PASS
❌ Security logs show crowdsec blocks	FAIL (Not tested - blocking doesn't work)
✅ Pre-commit passes 100%	PASS

Overall: 4/6 requirements met = FAIL

---

Recommendation: DO NOT DEPLOY

Critical Blockers

1. Traffic blocking is completely non-functional

   • Despite all infrastructure being operational
   • Decisions are created and streamed but not enforced
   • Zero evidence of middleware intercepting requests

2. Bouncer stuck in startup mode

   • Never transitions to active streaming
   • May be a bug in caddy-cs-bouncer v0.9.2
   • Requires investigation of bouncer implementation

Required Fixes

Immediate Actions

1. Add trusted_proxies configuration to Caddy CrowdSec app

   {
     "api_key": "charonbouncerkey2024",
     "api_url": "http://127.0.0.1:8085",
     "enable_streaming": true,
     "ticker_interval": "60s",
     "trusted_proxies": ["127.0.0.1/32", "172.20.0.0/16"]
   }
   

2. Fix LAPI URL in environment

   • Change CHARON_SECURITY_CROWDSEC_API_URL from http://localhost:8080 to http://localhost:8085

3. Investigate bouncer startup mode persistence

   • Check caddy-cs-bouncer source code for startup mode logic
   • May need to restart Caddy after bouncer initialization
   • Could be a timing issue with LAPI availability

4. Verify middleware ordering

   • Ensure CrowdSec handler executes BEFORE reverse_proxy
   • Check route handler chain in Caddy config
   • Add explicit ordering if necessary

Verification Steps After Fix

1. Add test decision
2. Wait 60 seconds (one ticker interval)
3. Test with curl from banned IP
4. Verify 403 response
5. Check Caddy access logs for "crowdsec" denial
6. Verify security logs show block event

---

Next Steps

1. Backend Team: Investigate Caddy config generation in internal/caddy/config.go

   • Add trusted_proxies field to CrowdSec app config
   • Ensure middleware ordering is correct
   • Add debug logging for bouncer decision application

2. DevOps Team: Consider alternative bouncer implementations

   • Test with different caddy-cs-bouncer version
   • Evaluate fallback to HTTP middleware bouncer
   • Document bouncer version compatibility

3. QA Team: Create blocking verification test suite

   • Automated test that validates actual blocking
   • Part of integration test suite
   • Must run before any security release

---

Evidence Files

• final_block_test.txt - Contains full curl output showing 200 OK response
• Container logs available via docker logs charon
• Caddy config available via http://localhost:2019/config/

---

Summary

While the CrowdSec integration is architecturally sound and all components are operationally healthy, the critical functionality of blocking malicious traffic is completely broken. This is a show-stopper bug that makes the CrowdSec feature unusable in production.

The bouncer registers correctly, pulls decisions successfully, and integrates with Caddy's request pipeline, but fails to enforce any decisions. This represents a complete failure of the security feature's core purpose.

Status: ❌ FAIL - DO NOT DEPLOY

---

Signed: QA_Security Agent Date: 2025-12-15 Session: Final Validation After No-Cache Rebuild