Charon/docs/plans/caddy_config_architecture_investigation.md

Investigation: Caddy Configuration File Analysis

Date: December 20, 2024 Issue: After Charon restart, /app/data/caddy/config.json does not exist Status: ✅ NOT A BUG - SYSTEM WORKING AS DESIGNED

---

Executive Summary

The /app/data/caddy/config.json file does not exist and is not supposed to exist. This is the correct and expected behavior.

Key Finding: Charon uses Caddy's Admin API to dynamically manage configuration, not file-based configuration. The config is stored in Caddy's memory and updated via HTTP POST requests to the Admin API.

---

1. Why the Config File Doesn't Exist

Architecture Overview

Charon uses API-based configuration management for Caddy v2.x:

Database (ProxyHost models)
    ↓
GenerateConfig() → In-Memory Config (Go struct)
    ↓
Admin API (HTTP POST to localhost:2019/config/)
    ↓
Caddy's In-Memory State

Code Evidence

From backend/internal/caddy/manager.go (ApplyConfig workflow):

// 1. Generate config from database models
generatedConfig, err := GenerateConfig(...)

// 2. Push config to Caddy via Admin API (NOT file write)
if err := m.client.Load(ctx, generatedConfig); err != nil {
    // Rollback on failure
    return fmt.Errorf("apply failed: %w", err)
}

// 3. Save snapshot for rollback capability
if err := m.saveSnapshot(generatedConfig); err != nil {
    // Warning only, not a failure
}

The client.Load() method sends the config via HTTP POST to Caddy's Admin API, NOT by writing a file.

---

2. Where Charon Actually Stores/Applies Config

Active Configuration Location

• Primary Storage: Caddy's in-memory state (managed by Caddy Admin API)
• Access Method: Caddy Admin API at http://localhost:2019/config/
• Persistence: Caddy maintains its own state across restarts using its internal storage

Snapshot Files for Rollback

Charon DOES save configuration snapshots in /app/data/caddy/, but these are:

• For rollback purposes only (disaster recovery)
• Named with timestamps: config-<unix-timestamp>.json
• NOT used as the active config source

Current snapshots on disk:

-rw-r--r-- 1 root root 40.4K Dec 18 12:38 config-1766079503.json
-rw-r--r-- 1 root root 40.4K Dec 18 18:52 config-1766101930.json
-rw-r--r-- 1 root root 40.2K Dec 18 18:59 config-1766102384.json
-rw-r--r-- 1 root root 39.8K Dec 18 19:00 config-1766102447.json
-rw-r--r-- 1 root root 40.4K Dec 18 19:01 config-1766102504.json
-rw-r--r-- 1 root root 40.2K Dec 18 19:02 config-1766102535.json
-rw-r--r-- 1 root root 39.5K Dec 18 19:02 config-1766102562.json
-rw-r--r-- 1 root root 39.5K Dec 18 20:04 config-1766106283.json
-rw-r--r-- 1 root root 39.5K Dec 19 01:02 config-1766124161.json
-rw-r--r-- 1 root root 44.7K Dec 19 13:57 config-1766170642.json (LATEST)

Latest snapshot: December 19, 2024 at 13:57 (44.7 KB)

---

3. How to Verify Current Caddy Configuration

Method 1: Query Caddy Admin API

Retrieve full configuration:

curl -s http://localhost:2019/config/ | jq '.'

Check specific routes:

curl -s http://localhost:2019/config/apps/http/servers/srv0/routes | jq '.'

Verify Caddy is responding:

curl -s http://localhost:2019/config/ -w "\nHTTP Status: %{http_code}\n"

Method 2: Check Container Logs

View recent Caddy activity:

docker logs charon --tail 100 2>&1 | grep -i caddy

Monitor real-time logs:

docker logs -f charon

Method 3: Inspect Latest Snapshot

View most recent config snapshot:

docker exec charon cat /app/data/caddy/config-1766170642.json | jq '.'

List all snapshots:

docker exec charon ls -lh /app/data/caddy/config-*.json

---

4. What Logs to Check for Errors

Container Logs Analysis (Last 100 Lines)

Command:

docker logs charon --tail 100 2>&1

Current Status: ✅ Caddy is operational and proxying traffic successfully

Log Evidence:

• Proxy Traffic: Successfully handling requests to nzbget, sonarr, radarr, seerr
• Health Check: GET /api/v1/health returning 200 OK
• HTTP/2 & HTTP/3: Properly negotiating protocols
• Security Headers: All security headers (HSTS, CSP, X-Frame-Options, etc.) are applied correctly
• No Config Errors: Zero errors related to configuration application or Caddy startup

Secondary Issue Detected (Non-Blocking):

{"level":"error","msg":"failed to connect to LAPI, retrying in 10s: API error: access forbidden"}

• Component: CrowdSec bouncer integration
• Impact: Does NOT affect Caddy functionality or proxy operations
• Action: Check CrowdSec API key configuration if CrowdSec integration is required

---

5. Recommended Fix

⚠️ NO FIX NEEDED - System is working as designed.

Why No Action Is Required

1. Caddy is running correctly: All proxy routes are operational
2. Config is being applied: Admin API is managing configuration dynamically
3. Snapshots exist: Rollback capability is functioning (10 snapshots on disk)
4. No errors: Logs show successful request handling with proper security headers

If You Need Static Config File for Reference

If you need a static reference file for debugging or documentation:

Option 1: Export current config from Admin API

curl -s http://localhost:2019/config/ | jq '.' > /tmp/caddy-current-config.json

Option 2: Copy latest snapshot

docker exec charon cat /app/data/caddy/config-1766170642.json > /tmp/caddy-snapshot.json

---

6. Architecture Benefits

Why Caddy Admin API is Superior to File-Based Config

1. Dynamic Updates: Apply config changes without restarting Caddy
2. Atomic Operations: Config updates are all-or-nothing (prevents partial failures)
3. Rollback Capability: Built-in rollback mechanism via snapshots
4. Validation: API validates config before applying
5. Zero Downtime: No service interruption during config updates
6. Programmatic Management: Easy to automate and integrate with applications

---

7. Configuration Flow Diagram

┌─────────────────────────────────────────────────────────────────┐
│                       User Creates Proxy Host                    │
│                    (via Charon Web UI/API)                       │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Database (SQLite/PostgreSQL)                    │
│              Stores: ProxyHost, SSLCert, Security                │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│           Charon: manager.ApplyConfig(ctx) Triggered             │
│              (via API call or scheduled job)                     │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│             caddy.GenerateConfig(...) [In-Memory]                │
│  • Fetch ProxyHost models from DB                                │
│  • Build Caddy JSON config struct                                │
│  • Apply: SSL, CrowdSec, WAF, Rate Limiting, ACL                 │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│        client.Load(ctx, generatedConfig) [Admin API]             │
│            HTTP POST → localhost:2019/config/                    │
│                  (Pushes config to Caddy)                        │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                        ┌───────┴────────┐
                        │                │
                        ▼                ▼
            ┌─────────────────┐   ┌──────────────────────┐
            │  Caddy Accepts  │   │  Caddy Rejects &      │
            │  Configuration  │   │  Returns Error        │
            └────────┬────────┘   └──────────┬───────────┘
                     │                       │
                     │                       ▼
                     │            ┌─────────────────────────┐
                     │            │ manager.rollback(ctx)   │
                     │            │ • Load latest snapshot  │
                     │            │ • Apply to Admin API    │
                     │            └─────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────────┐
│              manager.saveSnapshot(generatedConfig)               │
│         Writes: /app/data/caddy/config-<timestamp>.json          │
│                  (For rollback only, not active config)          │
└─────────────────────────────────────────────────────────────────┘

---

8. Key Takeaways

1. /app/data/caddy/config.json NEVER existed - This is not a regression or bug
2. Charon uses Caddy Admin API - This is the modern, recommended approach for Caddy v2
3. Snapshots are for rollback - They are not the active config source
4. Caddy is working correctly - Logs show successful proxy operations
5. CrowdSec warning is cosmetic - Does not impact Caddy functionality

---

9. References

Code Files Analyzed

• backend/internal/caddy/manager.go - Configuration lifecycle management
• backend/internal/caddy/config.go - Configuration generation (850+ lines)
• backend/internal/caddy/client.go - Admin API HTTP client
• backend/internal/config/config.go - Application settings
• backend/cmd/api/main.go - Application startup

Caddy Documentation

• Caddy Admin API
• Caddy Config Structure
• Caddy Autosave

---

Appendix: Verification Commands Summary

# 1. Check Caddy is running
docker exec charon caddy version

# 2. Query active configuration
curl -s http://localhost:2019/config/ | jq '.'

# 3. List config snapshots
docker exec charon ls -lh /app/data/caddy/config-*.json

# 4. View latest snapshot
docker exec charon cat /app/data/caddy/config-1766170642.json | jq '.'

# 5. Check container logs
docker logs charon --tail 100 2>&1

# 6. Monitor real-time logs
docker logs -f charon

# 7. Test proxy is working (from host)
curl -I https://yourdomain.com

# 8. Check Caddy health via Admin API
curl -s http://localhost:2019/metrics

---

Investigation Complete ✅ Status: System working as designed, no action required.