454 lines
14 KiB
Markdown
454 lines
14 KiB
Markdown
# Duplicate Proxy Host Diagnosis Report
|
|
|
|
**Date:** 2026-01-28
|
|
**Issue:** Charon container unhealthy, all proxy hosts down
|
|
**Error:** `validation failed: invalid route 1 in server charon_server: duplicate host matcher: immaculaterr.hatfieldhosted.com`
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
**Finding:** The database contains NO duplicate entries. There is only **one** proxy_host record for domain `Immaculaterr.hatfieldhosted.com` (ID 24). The duplicate host matcher error from Caddy indicates a **code-level bug** in the configuration generation logic, NOT a database integrity issue.
|
|
|
|
**Impact:**
|
|
- Caddy failed to load configuration at startup
|
|
- All proxy hosts are unreachable
|
|
- Container health check failing
|
|
- Frontend still accessible (direct backend connection)
|
|
|
|
**Root Cause:** Unknown bug in Caddy config generation that produces duplicate host matchers for the same domain, despite deduplication logic being present in the code.
|
|
|
|
---
|
|
|
|
## Investigation Details
|
|
|
|
### 1. Database Analysis
|
|
|
|
#### Active Database Location
|
|
- **Host path:** `/projects/Charon/data/charon.db` (empty/corrupted - 0 bytes)
|
|
- **Container path:** `/app/data/charon.db` (active - 177MB)
|
|
- **Backup:** `/projects/Charon/data/charon.db.backup-20260128-065828` (empty - contains schema but no data)
|
|
|
|
#### Database Integrity Check
|
|
|
|
**Total Proxy Hosts:** 19
|
|
**Query Results:**
|
|
```sql
|
|
-- Check for the problematic domain
|
|
SELECT id, uuid, name, domain_names, enabled, created_at, updated_at
|
|
FROM proxy_hosts
|
|
WHERE domain_names LIKE '%immaculaterr%';
|
|
```
|
|
|
|
**Result:** Only **ONE** entry found:
|
|
```
|
|
ID: 24
|
|
UUID: 4f392485-405b-4a35-b022-e3d16c30bbde
|
|
Name: Immaculaterr
|
|
Domain: Immaculaterr.hatfieldhosted.com (note: capital 'I')
|
|
Forward Host: Immaculaterr
|
|
Forward Port: 5454
|
|
Enabled: true
|
|
Created: 2026-01-16 20:42:59
|
|
Updated: 2026-01-16 20:42:59
|
|
```
|
|
|
|
#### Duplicate Detection Queries
|
|
|
|
**Test 1: Case-insensitive duplicate check**
|
|
```sql
|
|
SELECT COUNT(*), LOWER(domain_names)
|
|
FROM proxy_hosts
|
|
GROUP BY LOWER(domain_names)
|
|
HAVING COUNT(*) > 1;
|
|
```
|
|
**Result:** 0 duplicates found
|
|
|
|
**Test 2: Comma-separated domains check**
|
|
```sql
|
|
SELECT id, name, domain_names
|
|
FROM proxy_hosts
|
|
WHERE domain_names LIKE '%,%';
|
|
```
|
|
**Result:** No multi-domain entries found
|
|
|
|
**Test 3: Locations check (could cause route duplication)**
|
|
```sql
|
|
SELECT ph.id, ph.name, ph.domain_names, COUNT(l.id) as location_count
|
|
FROM proxy_hosts ph
|
|
LEFT JOIN locations l ON l.proxy_host_id = ph.id
|
|
WHERE ph.enabled = 1
|
|
GROUP BY ph.id;
|
|
```
|
|
**Result:** All proxy_hosts have 0 locations, including ID 24
|
|
|
|
**Test 4: Advanced config check**
|
|
```sql
|
|
SELECT id, name, domain_names, advanced_config
|
|
FROM proxy_hosts
|
|
WHERE id = 24;
|
|
```
|
|
**Result:** No advanced_config set (NULL)
|
|
|
|
**Test 5: Soft deletes check**
|
|
```sql
|
|
.schema proxy_hosts | grep -i deleted
|
|
```
|
|
**Result:** No soft delete columns exist
|
|
|
|
**Conclusion:** Database is clean. Only ONE entry for this domain exists.
|
|
|
|
---
|
|
|
|
### 2. Error Analysis
|
|
|
|
#### Error Message from Docker Logs
|
|
```
|
|
{"error":"validation failed: invalid route 1 in server charon_server: duplicate host matcher: immaculaterr.hatfieldhosted.com","level":"error","msg":"Failed to apply initial Caddy config","time":"2026-01-28T13:18:53-05:00"}
|
|
```
|
|
|
|
#### Key Observations:
|
|
1. **"invalid route 1"** - This is the SECOND route (0-indexed), suggesting the first route (index 0) is valid
|
|
2. **Lowercase domain** - Caddy error shows `immaculaterr` (lowercase) but database has `Immaculaterr` (capital I)
|
|
3. **Timing** - Error occurs at initial startup when `ApplyConfig()` is called
|
|
4. **Validation stage** - Error happens in Caddy's validation, not in Charon's generation
|
|
|
|
#### Code Review Findings
|
|
|
|
**File:** `/projects/Charon/backend/internal/caddy/config.go`
|
|
**Function:** `GenerateConfig()` (line 19)
|
|
|
|
**Deduplication Logic Present:**
|
|
- Line 437: `processedDomains := make(map[string]bool)` - Track processed domains
|
|
- Line 469-488: Domain normalization and duplicate detection
|
|
```go
|
|
d = strings.TrimSpace(d)
|
|
d = strings.ToLower(d) // Normalize to lowercase
|
|
if processedDomains[d] {
|
|
logger.Log().WithField("domain", d).Warn("Skipping duplicate domain")
|
|
continue
|
|
}
|
|
processedDomains[d] = true
|
|
```
|
|
- Line 461: Reverse iteration to prefer newer hosts
|
|
```go
|
|
for i := len(hosts) - 1; i >= 0; i--
|
|
```
|
|
|
|
**Expected Behavior:** The deduplication logic SHOULD prevent this error.
|
|
|
|
**Hypothesis:** One of the following is occurring:
|
|
1. **Bug in deduplication logic:** The domain is bypassing the duplicate check
|
|
2. **Multiple code paths:** Domain is added through a different path (e.g., frontend route, locations, advanced config)
|
|
3. **Database query issue:** GORM joins/preloads causing duplicate records in the Go slice
|
|
4. **Race condition:** Config is being generated/applied multiple times simultaneously (unlikely at startup)
|
|
|
|
---
|
|
|
|
### 3. All Proxy Hosts in Database
|
|
|
|
```
|
|
ID Name Domain
|
|
2 FileFlows fileflows.hatfieldhosted.com
|
|
4 Profilarr profilarr.hatfieldhosted.com
|
|
5 HomePage homepage.hatfieldhosted.com
|
|
6 Prowlarr prowlarr.hatfieldhosted.com
|
|
7 Tautulli tautulli.hatfieldhosted.com
|
|
8 TubeSync tubesync.hatfieldhosted.com
|
|
9 Bazarr bazarr.hatfieldhosted.com
|
|
11 Mealie mealie.hatfieldhosted.com
|
|
12 NZBGet nzbget.hatfieldhosted.com
|
|
13 Radarr radarr.hatfieldhosted.com
|
|
14 Sonarr sonarr.hatfieldhosted.com
|
|
15 Seerr seerr.hatfieldhosted.com
|
|
16 Plex plex.hatfieldhosted.com
|
|
17 Charon charon.hatfieldhosted.com
|
|
18 Wizarr wizarr.hatfieldhosted.com
|
|
20 PruneMate prunemate.hatfieldhosted.com
|
|
21 GiftManager giftmanager.hatfieldhosted.com
|
|
22 Dockhand dockhand.hatfieldhosted.com
|
|
24 Immaculaterr Immaculaterr.hatfieldhosted.com ← PROBLEMATIC
|
|
```
|
|
|
|
**Note:** ID 24 is the newest proxy_host (most recent updated_at timestamp).
|
|
|
|
---
|
|
|
|
### 4. Caddy Configuration State
|
|
|
|
**Current Status:** NO configuration loaded (Caddy is running with minimal admin-only config)
|
|
|
|
**Query:** `curl localhost:2019/config/` returns empty/default config
|
|
|
|
**Last Successful Config:**
|
|
- Timestamp: 2026-01-27 19:15:38
|
|
- Config Hash: `a87bd130369d62ab29a1fcf377d855a5b058223c33818eacff6f7312c2c4d6a0`
|
|
- Status: Success (before ID 24 was added)
|
|
|
|
**Recent Config History (from caddy_configs table):**
|
|
```
|
|
ID Hash Applied At Success
|
|
299 a87bd130...c2c4d6a0 2026-01-27 19:15:38 true
|
|
298 a87bd130...c2c4d6a0 2026-01-27 15:40:56 true
|
|
297 a87bd130...c2c4d6a0 2026-01-27 03:34:46 true
|
|
296 dbf4c820...d963b234 2026-01-27 02:01:45 true
|
|
295 dbf4c820...d963b234 2026-01-27 02:01:45 true
|
|
```
|
|
|
|
All recent configs were successful. The failure happened on 2026-01-28 13:18:53 (not recorded in table due to early validation failure).
|
|
|
|
---
|
|
|
|
### 5. Database File Status
|
|
|
|
**Critical Issue:** The host's `/projects/Charon/data/charon.db` file is **empty** (0 bytes).
|
|
|
|
**Timeline:**
|
|
- Original file was likely corrupted or truncated
|
|
- Container is using an in-memory or separate database file
|
|
- Volume mount may be broken or asynchronous
|
|
|
|
**Evidence:**
|
|
```bash
|
|
-rw-r--r-- 1 root root 0 Jan 28 18:24 /projects/Charon/data/charon.db
|
|
-rw-r--r-- 1 root root 177M Jan 28 18:26 /projects/Charon/data/charon.db.investigation
|
|
```
|
|
|
|
The actual database was copied from the container.
|
|
|
|
---
|
|
|
|
## Recommended Remediation Plan
|
|
|
|
### Immediate Short-Term Fix (Workaround)
|
|
|
|
**Option 1: Disable Problematic Proxy Host**
|
|
```sql
|
|
-- Run inside container
|
|
docker exec charon sqlite3 /app/data/charon.db \
|
|
"UPDATE proxy_hosts SET enabled = 0 WHERE id = 24;"
|
|
|
|
-- Restart container to apply
|
|
docker restart charon
|
|
```
|
|
|
|
**Option 2: Delete Duplicate Entry (if acceptable data loss)**
|
|
```sql
|
|
docker exec charon sqlite3 /app/data/charon.db \
|
|
"DELETE FROM proxy_hosts WHERE id = 24;"
|
|
docker restart charon
|
|
```
|
|
|
|
**Option 3: Change Domain to Bypass Duplicate Detection**
|
|
```sql
|
|
-- Temporarily rename the domain to isolate the issue
|
|
docker exec charon sqlite3 /app/data/charon.db \
|
|
"UPDATE proxy_hosts SET domain_names = 'immaculaterr-temp.hatfieldhosted.com' WHERE id = 24;"
|
|
docker restart charon
|
|
```
|
|
|
|
### Medium-Term Fix (Debug & Patch)
|
|
|
|
**Step 1: Enable Debug Logging**
|
|
```bash
|
|
# Set debug logging in container
|
|
docker exec charon sh -c "export CHARON_DEBUG=1; kill -HUP \$(pidof charon)"
|
|
```
|
|
|
|
**Step 2: Generate Config Manually**
|
|
Create a debug script to generate and inspect the Caddy config:
|
|
```go
|
|
// In backend/cmd/debug/main.go
|
|
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"log"
|
|
|
|
"github.com/Wikid82/charon/backend/internal/caddy"
|
|
"github.com/Wikid82/charon/backend/internal/database"
|
|
"github.com/Wikid82/charon/backend/internal/models"
|
|
)
|
|
|
|
func main() {
|
|
db, _ := database.Connect("data/charon.db")
|
|
var hosts []models.ProxyHost
|
|
db.Preload("Locations").Preload("DNSProvider").Find(&hosts)
|
|
|
|
config, err := caddy.GenerateConfig(hosts, "data/caddy/data", "", "frontend/dist", "", false, false, false, false, false, "", nil, nil, nil, nil, nil)
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
|
|
json, _ := json.MarshalIndent(config, "", " ")
|
|
fmt.Println(string(json))
|
|
}
|
|
```
|
|
|
|
Run and inspect:
|
|
```bash
|
|
go run backend/cmd/debug/main.go > /tmp/caddy-config-debug.json
|
|
jq '.apps.http.servers.charon_server.routes[] | select(.match[0].host[] | contains("immaculaterr"))' /tmp/caddy-config-debug.json
|
|
```
|
|
|
|
**Step 3: Add Unit Test**
|
|
```go
|
|
// In backend/internal/caddy/config_test.go
|
|
func TestGenerateConfig_PreventCaseSensitiveDuplicates(t *testing.T) {
|
|
hosts := []models.ProxyHost{
|
|
{UUID: "uuid-1", DomainNames: "Example.com", Enabled: true, ForwardHost: "app1", ForwardPort: 8080}, {UUID: "uuid-2", DomainNames: "example.com", Enabled: true, ForwardHost: "app2", ForwardPort: 8081},
|
|
}
|
|
|
|
config, err := GenerateConfig(hosts, "/tmp/data", "", "", "", false, false, false, false, false, "", nil, nil, nil, nil, nil)
|
|
require.NoError(t, err)
|
|
|
|
// Should only have ONE route for this domain (not two)
|
|
server := config.Apps.HTTP.Servers["charon_server"]
|
|
routes := server.Routes
|
|
|
|
domainCount := 0
|
|
for _, route := range routes {
|
|
for _, match := range route.Match {
|
|
for _, host := range match.Host {
|
|
if strings.ToLower(host) == "example.com" {
|
|
domainCount++
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
assert.Equal(t, 1, domainCount, "Should only have one route for case-insensitive duplicate domain")
|
|
}
|
|
```
|
|
|
|
### Long-Term Fix (Root Cause Prevention)
|
|
|
|
**1. Add Database Constraint**
|
|
```sql
|
|
-- Create unique index on normalized domain names
|
|
CREATE UNIQUE INDEX idx_proxy_hosts_domain_names_lower
|
|
ON proxy_hosts(LOWER(domain_names));
|
|
```
|
|
|
|
**2. Add Pre-Save Validation Hook**
|
|
```go
|
|
// In backend/internal/models/proxy_host.go
|
|
func (p *ProxyHost) BeforeSave(tx *gorm.DB) error {
|
|
// Normalize domain names to lowercase
|
|
p.DomainNames = strings.ToLower(p.DomainNames)
|
|
|
|
// Check for existing domain (case-insensitive)
|
|
var existing ProxyHost
|
|
if err := tx.Where("id != ? AND LOWER(domain_names) = ?",
|
|
p.ID, strings.ToLower(p.DomainNames)).First(&existing).Error; err == nil {
|
|
return fmt.Errorf("domain %s already exists (ID: %d)", p.DomainNames, existing.ID)
|
|
}
|
|
|
|
return nil
|
|
}
|
|
```
|
|
|
|
**3. Add Duplicate Detection to Frontend**
|
|
```typescript
|
|
// In frontend/src/components/ProxyHostForm.tsx
|
|
const checkDomainUnique = async (domain: string) => {
|
|
const response = await api.get(`/api/v1/proxy-hosts?domain=${encodeURIComponent(domain.toLowerCase())}`);
|
|
if (response.data.length > 0) {
|
|
setError(`Domain ${domain} is already in use by "${response.data[0].name}"`);
|
|
return false;
|
|
}
|
|
return true;
|
|
};
|
|
```
|
|
|
|
**4. Add Monitoring/Alerting**
|
|
- Add Prometheus metric for config generation failures
|
|
- Set up alert for repeated validation failures
|
|
- Log full generated config to file for debugging
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
### Immediate Action Required (Choose ONE):
|
|
|
|
**Recommended:** Option 1 (Disable)
|
|
- **Pros:** Non-destructive, can re-enable later, allows investigation
|
|
- **Cons:** Service unavailable until bug is fixed
|
|
- **Command:**
|
|
```bash
|
|
docker exec charon sqlite3 /app/data/charon.db \
|
|
"UPDATE proxy_hosts SET enabled = 0 WHERE id = 24;"
|
|
docker restart charon
|
|
```
|
|
|
|
### Follow-Up Investigation:
|
|
|
|
1. **Check for code-level bug:** Add debug logging to `GenerateConfig()` to print:
|
|
- Total hosts processed
|
|
- Each domain being added to processedDomains map
|
|
- Final route count vs expected count
|
|
|
|
2. **Verify GORM query behavior:** Check if `.Preload()` is causing duplicate records in the slice
|
|
|
|
3. **Test with minimal reproduction:** Create a fresh database with only ID 24, see if error persists
|
|
|
|
4. **Review recent commits:** Check if any recent changes to config.go introduced the bug
|
|
|
|
---
|
|
|
|
## Files Involved
|
|
|
|
- **Database:** `/app/data/charon.db` (inside container)
|
|
- **Backup:** `/projects/Charon/data/charon.db.backup-20260128-065828`
|
|
- **Investigation Copy:** `/projects/Charon/data/charon.db.investigation`
|
|
- **Code:** `/projects/Charon/backend/internal/caddy/config.go` (GenerateConfig function)
|
|
- **Manager:** `/projects/Charon/backend/internal/caddy/manager.go` (ApplyConfig function)
|
|
|
|
---
|
|
|
|
## Appendix: SQL Queries Used
|
|
|
|
```sql
|
|
-- Find all proxy hosts with specific domain
|
|
SELECT id, uuid, name, domain_names, forward_host, forward_port, enabled, created_at, updated_at
|
|
FROM proxy_hosts
|
|
WHERE domain_names LIKE '%immaculaterr.hatfieldhosted.com%'
|
|
ORDER BY created_at;
|
|
|
|
-- Count total hosts
|
|
SELECT COUNT(*) as total FROM proxy_hosts;
|
|
|
|
-- Check for duplicate domains (case-insensitive)
|
|
SELECT COUNT(*), domain_names
|
|
FROM proxy_hosts
|
|
GROUP BY LOWER(domain_names)
|
|
HAVING COUNT(*) > 1;
|
|
|
|
-- Check proxy hosts with locations
|
|
SELECT ph.id, ph.name, ph.domain_names, COUNT(l.id) as location_count
|
|
FROM proxy_hosts ph
|
|
LEFT JOIN locations l ON l.proxy_host_id = ph.id
|
|
WHERE ph.enabled = 1
|
|
GROUP BY ph.id
|
|
ORDER BY ph.id;
|
|
|
|
-- Check recent Caddy config applications
|
|
SELECT * FROM caddy_configs
|
|
ORDER BY applied_at DESC
|
|
LIMIT 5;
|
|
|
|
-- Get all enabled proxy hosts
|
|
SELECT id, name, domain_names, enabled
|
|
FROM proxy_hosts
|
|
WHERE enabled = 1
|
|
ORDER BY id;
|
|
```
|
|
|
|
---
|
|
|
|
**Report Generated By:** GitHub Copilot
|
|
**Investigation Date:** 2026-01-28
|
|
**Status:** Investigation Complete - Awaiting Remediation Decision
|