chore: git cache cleanup
This commit is contained in:
GitHub Actions
453
docs/implementation/validator_fix_diagnosis_20260128.md
Normal file
453
docs/implementation/validator_fix_diagnosis_20260128.md
Normal file
@@ -0,0 +1,453 @@
|
||||
# Duplicate Proxy Host Diagnosis Report
|
||||
|
||||
**Date:** 2026-01-28
|
||||
**Issue:** Charon container unhealthy, all proxy hosts down
|
||||
**Error:** `validation failed: invalid route 1 in server charon_server: duplicate host matcher: immaculaterr.hatfieldhosted.com`
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Finding:** The database contains NO duplicate entries. There is only **one** proxy_host record for domain `Immaculaterr.hatfieldhosted.com` (ID 24). The duplicate host matcher error from Caddy indicates a **code-level bug** in the configuration generation logic, NOT a database integrity issue.
|
||||
|
||||
**Impact:**
|
||||
- Caddy failed to load configuration at startup
|
||||
- All proxy hosts are unreachable
|
||||
- Container health check failing
|
||||
- Frontend still accessible (direct backend connection)
|
||||
|
||||
**Root Cause:** Unknown bug in Caddy config generation that produces duplicate host matchers for the same domain, despite deduplication logic being present in the code.
|
||||
|
||||
---
|
||||
|
||||
## Investigation Details
|
||||
|
||||
### 1. Database Analysis
|
||||
|
||||
#### Active Database Location
|
||||
- **Host path:** `/projects/Charon/data/charon.db` (empty/corrupted - 0 bytes)
|
||||
- **Container path:** `/app/data/charon.db` (active - 177MB)
|
||||
- **Backup:** `/projects/Charon/data/charon.db.backup-20260128-065828` (empty - contains schema but no data)
|
||||
|
||||
#### Database Integrity Check
|
||||
|
||||
**Total Proxy Hosts:** 19
|
||||
**Query Results:**
|
||||
```sql
|
||||
-- Check for the problematic domain
|
||||
SELECT id, uuid, name, domain_names, enabled, created_at, updated_at
|
||||
FROM proxy_hosts
|
||||
WHERE domain_names LIKE '%immaculaterr%';
|
||||
```
|
||||
|
||||
**Result:** Only **ONE** entry found:
|
||||
```
|
||||
ID: 24
|
||||
UUID: 4f392485-405b-4a35-b022-e3d16c30bbde
|
||||
Name: Immaculaterr
|
||||
Domain: Immaculaterr.hatfieldhosted.com (note: capital 'I')
|
||||
Forward Host: Immaculaterr
|
||||
Forward Port: 5454
|
||||
Enabled: true
|
||||
Created: 2026-01-16 20:42:59
|
||||
Updated: 2026-01-16 20:42:59
|
||||
```
|
||||
|
||||
#### Duplicate Detection Queries
|
||||
|
||||
**Test 1: Case-insensitive duplicate check**
|
||||
```sql
|
||||
SELECT COUNT(*), LOWER(domain_names)
|
||||
FROM proxy_hosts
|
||||
GROUP BY LOWER(domain_names)
|
||||
HAVING COUNT(*) > 1;
|
||||
```
|
||||
**Result:** 0 duplicates found
|
||||
|
||||
**Test 2: Comma-separated domains check**
|
||||
```sql
|
||||
SELECT id, name, domain_names
|
||||
FROM proxy_hosts
|
||||
WHERE domain_names LIKE '%,%';
|
||||
```
|
||||
**Result:** No multi-domain entries found
|
||||
|
||||
**Test 3: Locations check (could cause route duplication)**
|
||||
```sql
|
||||
SELECT ph.id, ph.name, ph.domain_names, COUNT(l.id) as location_count
|
||||
FROM proxy_hosts ph
|
||||
LEFT JOIN locations l ON l.proxy_host_id = ph.id
|
||||
WHERE ph.enabled = 1
|
||||
GROUP BY ph.id;
|
||||
```
|
||||
**Result:** All proxy_hosts have 0 locations, including ID 24
|
||||
|
||||
**Test 4: Advanced config check**
|
||||
```sql
|
||||
SELECT id, name, domain_names, advanced_config
|
||||
FROM proxy_hosts
|
||||
WHERE id = 24;
|
||||
```
|
||||
**Result:** No advanced_config set (NULL)
|
||||
|
||||
**Test 5: Soft deletes check**
|
||||
```sql
|
||||
.schema proxy_hosts | grep -i deleted
|
||||
```
|
||||
**Result:** No soft delete columns exist
|
||||
|
||||
**Conclusion:** Database is clean. Only ONE entry for this domain exists.
|
||||
|
||||
---
|
||||
|
||||
### 2. Error Analysis
|
||||
|
||||
#### Error Message from Docker Logs
|
||||
```
|
||||
{"error":"validation failed: invalid route 1 in server charon_server: duplicate host matcher: immaculaterr.hatfieldhosted.com","level":"error","msg":"Failed to apply initial Caddy config","time":"2026-01-28T13:18:53-05:00"}
|
||||
```
|
||||
|
||||
#### Key Observations:
|
||||
1. **"invalid route 1"** - This is the SECOND route (0-indexed), suggesting the first route (index 0) is valid
|
||||
2. **Lowercase domain** - Caddy error shows `immaculaterr` (lowercase) but database has `Immaculaterr` (capital I)
|
||||
3. **Timing** - Error occurs at initial startup when `ApplyConfig()` is called
|
||||
4. **Validation stage** - Error happens in Caddy's validation, not in Charon's generation
|
||||
|
||||
#### Code Review Findings
|
||||
|
||||
**File:** `/projects/Charon/backend/internal/caddy/config.go`
|
||||
**Function:** `GenerateConfig()` (line 19)
|
||||
|
||||
**Deduplication Logic Present:**
|
||||
- Line 437: `processedDomains := make(map[string]bool)` - Track processed domains
|
||||
- Line 469-488: Domain normalization and duplicate detection
|
||||
```go
|
||||
d = strings.TrimSpace(d)
|
||||
d = strings.ToLower(d) // Normalize to lowercase
|
||||
if processedDomains[d] {
|
||||
logger.Log().WithField("domain", d).Warn("Skipping duplicate domain")
|
||||
continue
|
||||
}
|
||||
processedDomains[d] = true
|
||||
```
|
||||
- Line 461: Reverse iteration to prefer newer hosts
|
||||
```go
|
||||
for i := len(hosts) - 1; i >= 0; i--
|
||||
```
|
||||
|
||||
**Expected Behavior:** The deduplication logic SHOULD prevent this error.
|
||||
|
||||
**Hypothesis:** One of the following is occurring:
|
||||
1. **Bug in deduplication logic:** The domain is bypassing the duplicate check
|
||||
2. **Multiple code paths:** Domain is added through a different path (e.g., frontend route, locations, advanced config)
|
||||
3. **Database query issue:** GORM joins/preloads causing duplicate records in the Go slice
|
||||
4. **Race condition:** Config is being generated/applied multiple times simultaneously (unlikely at startup)
|
||||
|
||||
---
|
||||
|
||||
### 3. All Proxy Hosts in Database
|
||||
|
||||
```
|
||||
ID Name Domain
|
||||
2 FileFlows fileflows.hatfieldhosted.com
|
||||
4 Profilarr profilarr.hatfieldhosted.com
|
||||
5 HomePage homepage.hatfieldhosted.com
|
||||
6 Prowlarr prowlarr.hatfieldhosted.com
|
||||
7 Tautulli tautulli.hatfieldhosted.com
|
||||
8 TubeSync tubesync.hatfieldhosted.com
|
||||
9 Bazarr bazarr.hatfieldhosted.com
|
||||
11 Mealie mealie.hatfieldhosted.com
|
||||
12 NZBGet nzbget.hatfieldhosted.com
|
||||
13 Radarr radarr.hatfieldhosted.com
|
||||
14 Sonarr sonarr.hatfieldhosted.com
|
||||
15 Seerr seerr.hatfieldhosted.com
|
||||
16 Plex plex.hatfieldhosted.com
|
||||
17 Charon charon.hatfieldhosted.com
|
||||
18 Wizarr wizarr.hatfieldhosted.com
|
||||
20 PruneMate prunemate.hatfieldhosted.com
|
||||
21 GiftManager giftmanager.hatfieldhosted.com
|
||||
22 Dockhand dockhand.hatfieldhosted.com
|
||||
24 Immaculaterr Immaculaterr.hatfieldhosted.com ← PROBLEMATIC
|
||||
```
|
||||
|
||||
**Note:** ID 24 is the newest proxy_host (most recent updated_at timestamp).
|
||||
|
||||
---
|
||||
|
||||
### 4. Caddy Configuration State
|
||||
|
||||
**Current Status:** NO configuration loaded (Caddy is running with minimal admin-only config)
|
||||
|
||||
**Query:** `curl localhost:2019/config/` returns empty/default config
|
||||
|
||||
**Last Successful Config:**
|
||||
- Timestamp: 2026-01-27 19:15:38
|
||||
- Config Hash: `a87bd130369d62ab29a1fcf377d855a5b058223c33818eacff6f7312c2c4d6a0`
|
||||
- Status: Success (before ID 24 was added)
|
||||
|
||||
**Recent Config History (from caddy_configs table):**
|
||||
```
|
||||
ID Hash Applied At Success
|
||||
299 a87bd130...c2c4d6a0 2026-01-27 19:15:38 true
|
||||
298 a87bd130...c2c4d6a0 2026-01-27 15:40:56 true
|
||||
297 a87bd130...c2c4d6a0 2026-01-27 03:34:46 true
|
||||
296 dbf4c820...d963b234 2026-01-27 02:01:45 true
|
||||
295 dbf4c820...d963b234 2026-01-27 02:01:45 true
|
||||
```
|
||||
|
||||
All recent configs were successful. The failure happened on 2026-01-28 13:18:53 (not recorded in table due to early validation failure).
|
||||
|
||||
---
|
||||
|
||||
### 5. Database File Status
|
||||
|
||||
**Critical Issue:** The host's `/projects/Charon/data/charon.db` file is **empty** (0 bytes).
|
||||
|
||||
**Timeline:**
|
||||
- Original file was likely corrupted or truncated
|
||||
- Container is using an in-memory or separate database file
|
||||
- Volume mount may be broken or asynchronous
|
||||
|
||||
**Evidence:**
|
||||
```bash
|
||||
-rw-r--r-- 1 root root 0 Jan 28 18:24 /projects/Charon/data/charon.db
|
||||
-rw-r--r-- 1 root root 177M Jan 28 18:26 /projects/Charon/data/charon.db.investigation
|
||||
```
|
||||
|
||||
The actual database was copied from the container.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Remediation Plan
|
||||
|
||||
### Immediate Short-Term Fix (Workaround)
|
||||
|
||||
**Option 1: Disable Problematic Proxy Host**
|
||||
```sql
|
||||
-- Run inside container
|
||||
docker exec charon sqlite3 /app/data/charon.db \
|
||||
"UPDATE proxy_hosts SET enabled = 0 WHERE id = 24;"
|
||||
|
||||
-- Restart container to apply
|
||||
docker restart charon
|
||||
```
|
||||
|
||||
**Option 2: Delete Duplicate Entry (if acceptable data loss)**
|
||||
```sql
|
||||
docker exec charon sqlite3 /app/data/charon.db \
|
||||
"DELETE FROM proxy_hosts WHERE id = 24;"
|
||||
docker restart charon
|
||||
```
|
||||
|
||||
**Option 3: Change Domain to Bypass Duplicate Detection**
|
||||
```sql
|
||||
-- Temporarily rename the domain to isolate the issue
|
||||
docker exec charon sqlite3 /app/data/charon.db \
|
||||
"UPDATE proxy_hosts SET domain_names = 'immaculaterr-temp.hatfieldhosted.com' WHERE id = 24;"
|
||||
docker restart charon
|
||||
```
|
||||
|
||||
### Medium-Term Fix (Debug & Patch)
|
||||
|
||||
**Step 1: Enable Debug Logging**
|
||||
```bash
|
||||
# Set debug logging in container
|
||||
docker exec charon sh -c "export CHARON_DEBUG=1; kill -HUP \$(pidof charon)"
|
||||
```
|
||||
|
||||
**Step 2: Generate Config Manually**
|
||||
Create a debug script to generate and inspect the Caddy config:
|
||||
```go
|
||||
// In backend/cmd/debug/main.go
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"github.com/Wikid82/charon/backend/internal/caddy"
|
||||
"github.com/Wikid82/charon/backend/internal/database"
|
||||
"github.com/Wikid82/charon/backend/internal/models"
|
||||
)
|
||||
|
||||
func main() {
|
||||
db, _ := database.Connect("data/charon.db")
|
||||
var hosts []models.ProxyHost
|
||||
db.Preload("Locations").Preload("DNSProvider").Find(&hosts)
|
||||
|
||||
config, err := caddy.GenerateConfig(hosts, "data/caddy/data", "", "frontend/dist", "", false, false, false, false, false, "", nil, nil, nil, nil, nil)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
json, _ := json.MarshalIndent(config, "", " ")
|
||||
fmt.Println(string(json))
|
||||
}
|
||||
```
|
||||
|
||||
Run and inspect:
|
||||
```bash
|
||||
go run backend/cmd/debug/main.go > /tmp/caddy-config-debug.json
|
||||
jq '.apps.http.servers.charon_server.routes[] | select(.match[0].host[] | contains("immaculaterr"))' /tmp/caddy-config-debug.json
|
||||
```
|
||||
|
||||
**Step 3: Add Unit Test**
|
||||
```go
|
||||
// In backend/internal/caddy/config_test.go
|
||||
func TestGenerateConfig_PreventCaseSensitiveDuplicates(t *testing.T) {
|
||||
hosts := []models.ProxyHost{
|
||||
{UUID: "uuid-1", DomainNames: "Example.com", Enabled: true, ForwardHost: "app1", ForwardPort: 8080}, {UUID: "uuid-2", DomainNames: "example.com", Enabled: true, ForwardHost: "app2", ForwardPort: 8081},
|
||||
}
|
||||
|
||||
config, err := GenerateConfig(hosts, "/tmp/data", "", "", "", false, false, false, false, false, "", nil, nil, nil, nil, nil)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should only have ONE route for this domain (not two)
|
||||
server := config.Apps.HTTP.Servers["charon_server"]
|
||||
routes := server.Routes
|
||||
|
||||
domainCount := 0
|
||||
for _, route := range routes {
|
||||
for _, match := range route.Match {
|
||||
for _, host := range match.Host {
|
||||
if strings.ToLower(host) == "example.com" {
|
||||
domainCount++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
assert.Equal(t, 1, domainCount, "Should only have one route for case-insensitive duplicate domain")
|
||||
}
|
||||
```
|
||||
|
||||
### Long-Term Fix (Root Cause Prevention)
|
||||
|
||||
**1. Add Database Constraint**
|
||||
```sql
|
||||
-- Create unique index on normalized domain names
|
||||
CREATE UNIQUE INDEX idx_proxy_hosts_domain_names_lower
|
||||
ON proxy_hosts(LOWER(domain_names));
|
||||
```
|
||||
|
||||
**2. Add Pre-Save Validation Hook**
|
||||
```go
|
||||
// In backend/internal/models/proxy_host.go
|
||||
func (p *ProxyHost) BeforeSave(tx *gorm.DB) error {
|
||||
// Normalize domain names to lowercase
|
||||
p.DomainNames = strings.ToLower(p.DomainNames)
|
||||
|
||||
// Check for existing domain (case-insensitive)
|
||||
var existing ProxyHost
|
||||
if err := tx.Where("id != ? AND LOWER(domain_names) = ?",
|
||||
p.ID, strings.ToLower(p.DomainNames)).First(&existing).Error; err == nil {
|
||||
return fmt.Errorf("domain %s already exists (ID: %d)", p.DomainNames, existing.ID)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**3. Add Duplicate Detection to Frontend**
|
||||
```typescript
|
||||
// In frontend/src/components/ProxyHostForm.tsx
|
||||
const checkDomainUnique = async (domain: string) => {
|
||||
const response = await api.get(`/api/v1/proxy-hosts?domain=${encodeURIComponent(domain.toLowerCase())}`);
|
||||
if (response.data.length > 0) {
|
||||
setError(`Domain ${domain} is already in use by "${response.data[0].name}"`);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
```
|
||||
|
||||
**4. Add Monitoring/Alerting**
|
||||
- Add Prometheus metric for config generation failures
|
||||
- Set up alert for repeated validation failures
|
||||
- Log full generated config to file for debugging
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate Action Required (Choose ONE):
|
||||
|
||||
**Recommended:** Option 1 (Disable)
|
||||
- **Pros:** Non-destructive, can re-enable later, allows investigation
|
||||
- **Cons:** Service unavailable until bug is fixed
|
||||
- **Command:**
|
||||
```bash
|
||||
docker exec charon sqlite3 /app/data/charon.db \
|
||||
"UPDATE proxy_hosts SET enabled = 0 WHERE id = 24;"
|
||||
docker restart charon
|
||||
```
|
||||
|
||||
### Follow-Up Investigation:
|
||||
|
||||
1. **Check for code-level bug:** Add debug logging to `GenerateConfig()` to print:
|
||||
- Total hosts processed
|
||||
- Each domain being added to processedDomains map
|
||||
- Final route count vs expected count
|
||||
|
||||
2. **Verify GORM query behavior:** Check if `.Preload()` is causing duplicate records in the slice
|
||||
|
||||
3. **Test with minimal reproduction:** Create a fresh database with only ID 24, see if error persists
|
||||
|
||||
4. **Review recent commits:** Check if any recent changes to config.go introduced the bug
|
||||
|
||||
---
|
||||
|
||||
## Files Involved
|
||||
|
||||
- **Database:** `/app/data/charon.db` (inside container)
|
||||
- **Backup:** `/projects/Charon/data/charon.db.backup-20260128-065828`
|
||||
- **Investigation Copy:** `/projects/Charon/data/charon.db.investigation`
|
||||
- **Code:** `/projects/Charon/backend/internal/caddy/config.go` (GenerateConfig function)
|
||||
- **Manager:** `/projects/Charon/backend/internal/caddy/manager.go` (ApplyConfig function)
|
||||
|
||||
---
|
||||
|
||||
## Appendix: SQL Queries Used
|
||||
|
||||
```sql
|
||||
-- Find all proxy hosts with specific domain
|
||||
SELECT id, uuid, name, domain_names, forward_host, forward_port, enabled, created_at, updated_at
|
||||
FROM proxy_hosts
|
||||
WHERE domain_names LIKE '%immaculaterr.hatfieldhosted.com%'
|
||||
ORDER BY created_at;
|
||||
|
||||
-- Count total hosts
|
||||
SELECT COUNT(*) as total FROM proxy_hosts;
|
||||
|
||||
-- Check for duplicate domains (case-insensitive)
|
||||
SELECT COUNT(*), domain_names
|
||||
FROM proxy_hosts
|
||||
GROUP BY LOWER(domain_names)
|
||||
HAVING COUNT(*) > 1;
|
||||
|
||||
-- Check proxy hosts with locations
|
||||
SELECT ph.id, ph.name, ph.domain_names, COUNT(l.id) as location_count
|
||||
FROM proxy_hosts ph
|
||||
LEFT JOIN locations l ON l.proxy_host_id = ph.id
|
||||
WHERE ph.enabled = 1
|
||||
GROUP BY ph.id
|
||||
ORDER BY ph.id;
|
||||
|
||||
-- Check recent Caddy config applications
|
||||
SELECT * FROM caddy_configs
|
||||
ORDER BY applied_at DESC
|
||||
LIMIT 5;
|
||||
|
||||
-- Get all enabled proxy hosts
|
||||
SELECT id, name, domain_names, enabled
|
||||
FROM proxy_hosts
|
||||
WHERE enabled = 1
|
||||
ORDER BY id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Report Generated By:** GitHub Copilot
|
||||
**Investigation Date:** 2026-01-28
|
||||
**Status:** Investigation Complete - Awaiting Remediation Decision
|
||||
Reference in New Issue
Block a user