Charon/docs/plans/current_spec.md

Integration Workflow Fix Plan: go-httpbin Port Mismatch & curl-in-Container Remediation

Status: Active Created: 2026-03-14 Branch: feature/beta-release Context: All four integration workflows (cerberus, crowdsec, rate-limit, waf) are failing with "httpbin not starting." Root cause is a compounding port mismatch introduced when kennethreitz/httpbin (port 80) was swapped for mccutchen/go-httpbin (port 8080) without updating port configuration. A secondary issue exists where two scripts still use curl via docker exec inside an Alpine container that only has wget. Previous Plan: Backed up to docs/plans/cve_remediation_spec.md

---

1. Root Cause Analysis

1.1 Primary: go-httpbin Port Mismatch

Commit 042c5ec6 replaced kennethreitz/httpbin with mccutchen/go-httpbin across all four integration scripts. However, it only changed the image name — no port configuration was updated.

Property	kennethreitz/httpbin	mccutchen/go-httpbin
Default listen port	80	8080
Port env var	N/A	PORT
Exposed port (Dockerfile)	80/tcp	8080/tcp

Evidence: docker inspect mccutchen/go-httpbin --format='{{json .Config.ExposedPorts}}' returns {"8080/tcp":{}}. The go-httpbin README confirms the default port is 8080, configurable via -port flag or PORT environment variable.

Failure mechanism: Each integration script has a health check loop like:

for i in {1..45}; do
    if docker exec ${CONTAINER_NAME} sh -c "wget -qO /dev/null http://${BACKEND_CONTAINER}/get" >/dev/null 2>&1; then
        echo "✓ httpbin backend is ready"
        break
    fi
    if [ $i -eq 45 ]; then
        echo "✗ httpbin backend failed to start"
        exit 1
    fi
    sleep 1
done

The wget URL http://${BACKEND_CONTAINER}/get resolves to port 80 (HTTP default). go-httpbin is listening on port 8080. The connection is refused on port 80 for 45 iterations, then the script prints "httpbin backend failed to start" and exits 1.

Additionally, all four scripts create proxy hosts with "forward_port": 80, which would also fail at the Caddy reverse proxy level even if the health check passed.

1.2 Secondary: curl Not Available Inside Container

Commit 58b087bc correctly migrated the four httpbin health checks from curl to wget (busybox). However, two other scripts still use docker exec ... curl to probe services inside the Alpine container, which does not have curl installed (removed as part of CVE remediation):

1. scripts/crowdsec_startup_test.sh L179 — LAPI health check uses docker exec ${CONTAINER_NAME} curl -sf http://127.0.0.1:8085/health. Always returns "FAILED" (soft-pass, not blocking CI, but wrong).
2. scripts/diagnose-test-env.sh L104 — CrowdSec LAPI check uses docker exec charon-e2e curl -sf http://localhost:8090/health. Always fails silently.

1.3 Timeline of Changes

Commit	Change	Effect
042c5ec6	Swap kennethreitz/httpbin → mccutchen/go-httpbin	Introduced port mismatch (8080 vs 80). Not caught because the prior curl-based health checks were already broken by the missing curl.
58b087bc	Replace curl with wget in docker exec httpbin health checks	Correct tool fix for 4 scripts. But exposed the hidden port mismatch — now wget correctly tries port 80 and correctly fails (connection refused), producing the visible "httpbin not starting" error.
4b896c2e	Replace curl with wget in Docker compose healthchecks	Correct, no issues.

Key insight: The curl→wget migration was a correct fix for a real problem. It was applied on top of an earlier, unnoticed bug (port mismatch from the image swap). The wget migration made the port bug visible because wget actually runs inside the container, whereas curl was silently failing (not found) and the health check was timing out for a different reason.

---

2. Impact Assessment

2.1 Affected Scripts — Port Mismatch (PRIMARY)

File	Docker Run Line	Health Check Line	Forward Port Line	Status
scripts/cerberus_integration.sh	L174	L215	L255	BROKEN
scripts/waf_integration.sh	L167	L206	L246	BROKEN
scripts/rate_limit_integration.sh	L188	L191	L231	BROKEN
scripts/coraza_integration.sh	L159	L163	L191	BROKEN

2.2 Affected Scripts — curl Inside Container (SECONDARY)

File	Line	Command	Status
scripts/crowdsec_startup_test.sh	L179	docker exec ${CONTAINER_NAME} curl -sf http://127.0.0.1:8085/health	SILENT FAIL
scripts/diagnose-test-env.sh	L104	docker exec charon-e2e curl -sf http://localhost:8090/health	SILENT FAIL

2.3 NOT Affected

Component	Reason
Dockerfile HEALTHCHECK	Already uses wget — targets Charon API :8080, not httpbin
.docker/docker-entrypoint.sh	Already uses wget — Caddy readiness on :2019
All docker-compose healthchecks	Already use wget
Workflow YAML files	Use host-side curl (installed on ubuntu-latest), not docker exec
scripts/crowdsec_integration.sh	Uses only host-side curl; does not use httpbin at all
scripts/integration-test.sh	Uses whoami image (port 80), not go-httpbin

File: backend/internal/api/routes/routes.go (lines 260-267)

3. Remediation Plan

3.1 Fix Strategy: Add -e PORT=80 to go-httpbin Container

The least invasive fix is to set the PORT environment variable on the go-httpbin container so it listens on port 80, matching all existing health check URLs and proxy host forward_port values. This avoids cascading changes to URLs and Caddy configurations.

Alternative considered: Change all health check URLs and forward_port values to 8080. Rejected because:

• Requires more changes (health check URLs, forward_port, potentially Caddy route expectations)
• The proxy host forward_port is the user-facing API field; port 80 is the natural default for HTTP backends

3.2 Exact Changes — Port Fix (4 files)

scripts/cerberus_integration.sh — Line 174

-docker run -d --name ${BACKEND_CONTAINER} --network containers_default mccutchen/go-httpbin
+docker run -d --name ${BACKEND_CONTAINER} --network containers_default -e PORT=80 mccutchen/go-httpbin

scripts/waf_integration.sh — Line 167

-docker run -d --name ${BACKEND_CONTAINER} --network containers_default mccutchen/go-httpbin
+docker run -d --name ${BACKEND_CONTAINER} --network containers_default -e PORT=80 mccutchen/go-httpbin

scripts/rate_limit_integration.sh — Line 188

-docker run -d --name ${BACKEND_CONTAINER} --network containers_default mccutchen/go-httpbin
+docker run -d --name ${BACKEND_CONTAINER} --network containers_default -e PORT=80 mccutchen/go-httpbin

scripts/coraza_integration.sh — Line 159

-docker run -d --name coraza-backend --network containers_default mccutchen/go-httpbin
+docker run -d --name coraza-backend --network containers_default -e PORT=80 mccutchen/go-httpbin

3.3 Exact Changes — curl→wget Inside Container (2 files)

scripts/crowdsec_startup_test.sh — Line 179

-LAPI_HEALTH=$(docker exec ${CONTAINER_NAME} curl -sf http://127.0.0.1:8085/health 2>/dev/null || echo "FAILED")
+LAPI_HEALTH=$(docker exec ${CONTAINER_NAME} wget -qO - http://127.0.0.1:8085/health 2>/dev/null || echo "FAILED")

Note: Using wget -qO - (output to stdout) instead of wget -qO /dev/null because the response body is captured in $LAPI_HEALTH and checked.

scripts/diagnose-test-env.sh — Line 104

-if docker exec charon-e2e curl -sf http://localhost:8090/health > /dev/null 2>&1; then
+if docker exec charon-e2e wget -qO /dev/null http://localhost:8090/health 2>/dev/null; then

---

4. wget vs curl Flag Mapping Reference

curl Flag	wget Equivalent	Meaning
-s (silent)	-q (quiet)	Suppress progress output
-f (fail silently on HTTP errors)	(default behavior)	wget exits nonzero on HTTP 4xx/5xx
-o FILE (output to file)	-O FILE	Write output to specified file
-o /dev/null	-O /dev/null	Discard response body
> /dev/null 2>&1	2>/dev/null	Suppress stderr (wget is quiet with -q, only stderr needs redirect)
-m SECS / --max-time	-T SECS	Connection timeout
--retry N	-t N+1	Total attempts (wget counts initial try)
-S (show error)	(no equivalent)	wget shows errors by default without -q
-L (follow redirects)	(default behavior)	wget follows redirects by default

if ip.IsLoopback() {
    return true
}

5. Implementation Plan

Phase 1: Playwright Tests (Verification of Expected Behavior)

No new Playwright tests needed. The integration scripts are bash-based CI workflows, not UI features. Existing workflow YAML files already serve as the test harness.

Phase 2: Backend/Script Implementation

Task	File	Change	Complexity
T1	scripts/cerberus_integration.sh	Add -e PORT=80 to docker run	Trivial
T2	scripts/waf_integration.sh	Add -e PORT=80 to docker run	Trivial
T3	scripts/rate_limit_integration.sh	Add -e PORT=80 to docker run	Trivial
T4	scripts/coraza_integration.sh	Add -e PORT=80 to docker run	Trivial
T5	scripts/crowdsec_startup_test.sh	Replace curl -sf with wget -qO -	Trivial
T6	scripts/diagnose-test-env.sh	Replace curl -sf with wget -qO /dev/null	Trivial

Phase 3: Frontend Implementation

N/A — no frontend changes required.

Phase 4: Integration and Testing

1. Local validation: Run each integration script individually against a locally built charon:local image
2. CI validation: Push branch and verify all four integration workflows pass:
   • .github/workflows/cerberus-integration.yml
   • .github/workflows/waf-integration.yml
   • .github/workflows/rate-limit-integration.yml
   • .github/workflows/crowdsec-integration.yml
3. Regression check: Confirm E2E Playwright tests still pass (they don't touch these scripts, but verify no accidental breakage)

Phase 5: Documentation and Deployment

No documentation changes needed. The fix is internal to CI scripts.

---

6. File Change Summary

File	Change	Lines
scripts/cerberus_integration.sh	Add -e PORT=80 to go-httpbin docker run	L174
scripts/waf_integration.sh	Add -e PORT=80 to go-httpbin docker run	L167
scripts/rate_limit_integration.sh	Add -e PORT=80 to go-httpbin docker run	L188
scripts/coraza_integration.sh	Add -e PORT=80 to go-httpbin docker run	L159
scripts/crowdsec_startup_test.sh	Replace curl -sf with wget -qO - in docker exec	L179
scripts/diagnose-test-env.sh	Replace curl -sf with wget -qO /dev/null in docker exec	L104

Total: 6 one-line changes across 6 files.

Test Name	Host	Scheme	Expected Secure	Expected SameSite
TestSetSecureCookie_HTTP_PrivateIP_Insecure	192.168.1.50	http	false	Lax
TestSetSecureCookie_HTTP_10Network_Insecure	10.0.0.5	http	false	Lax
TestSetSecureCookie_HTTP_172Network_Insecure	172.16.0.1	http	false	Lax
TestSetSecureCookie_HTTPS_PrivateIP_Secure	192.168.1.50	https	true	Strict
TestSetSecureCookie_HTTP_PublicIP_Secure	203.0.113.5	http	true	Lax

7. Commit Slicing Strategy

Decision: Single PR

Reasoning:

• All 6 changes are trivial one-line fixes
• Total diff is ~12 lines (6 deletions + 6 additions)
• All changes are logically related (integration test infrastructure fix)
• No cross-domain concerns (all bash scripts in scripts/)
• Review size is well under the 200-line threshold for splitting
• No risk of partial deployment causing issues

PR-1: Fix integration workflow httpbin startup and curl-in-container issues

Scope: All 6 file changes Files: scripts/{cerberus,waf,rate_limit,coraza}_integration.sh, scripts/crowdsec_startup_test.sh, scripts/diagnose-test-env.sh Dependencies: None Validation gate: All four integration workflows pass in CI

Suggested commit message:

fix(ci): add PORT=80 to go-httpbin containers and replace curl with wget in docker exec

mccutchen/go-httpbin defaults to port 8080, but all integration scripts
expected port 80 from the previous kennethreitz/httpbin image. Add
-e PORT=80 to the docker run commands in cerberus, waf, rate_limit,
and coraza integration scripts.

Also replace remaining docker exec curl commands with wget in
crowdsec_startup_test.sh and diagnose-test-env.sh, since curl is not
installed in the Alpine runtime container.

Fixes: httpbin health check timeout ("httpbin not starting") in all
four integration workflows.

Rollback: git revert <commit> — safe and instant. No database migrations, no API changes, no user-facing impact.

---

8. Supporting Files Review

File	Review Result	Action Needed
.gitignore	No changes needed — no new files or artifacts	None
codecov.yml	No changes needed — integration scripts are not coverage-tracked	None
.dockerignore	No changes needed — scripts are not copied into Docker image	None
Dockerfile	Already correct — HEALTHCHECK uses wget, no httpbin references	None
.docker/docker-entrypoint.sh	Already correct — uses wget for Caddy readiness	None
docker-compose*.yml	Already correct — all healthchecks use wget	None
Workflow YAML files	Already correct — use host-side curl for debug dumps	None

---

9. Acceptance Criteria

• scripts/cerberus_integration.sh starts go-httpbin with -e PORT=80
• scripts/waf_integration.sh starts go-httpbin with -e PORT=80
• scripts/rate_limit_integration.sh starts go-httpbin with -e PORT=80
• scripts/coraza_integration.sh starts go-httpbin with -e PORT=80
• scripts/crowdsec_startup_test.sh uses wget instead of curl for LAPI health check
• scripts/diagnose-test-env.sh uses wget instead of curl for CrowdSec LAPI check
• CI workflow cerberus-integration passes
• CI workflow waf-integration passes
• CI workflow rate-limit-integration passes
• CI workflow crowdsec-integration passes
• No regression in E2E Playwright tests
• grep -r "docker exec.*curl" scripts/ returns zero matches (excluding comments/echo hints)