chore: git cache cleanup

2026-03-04 18:34:49 +00:00
parent c32cce2a88
commit 27c252600a
2001 changed files with 683185 additions and 0 deletions
--- a/docs/development/go_version_upgrades.md
+++ b/docs/development/go_version_upgrades.md
@@ -0,0 +1,420 @@
+# Go Version Upgrades
+
+**Last Updated:** 2026-02-12
+
+## The Short Version
+
+When Charon upgrades to a new Go version, your development tools (like golangci-lint) break. Here's how to fix it:
+
+```bash
+# Step 1: Pull latest code
+git pull
+
+# Step 2: Update your Go installation
+.github/skills/scripts/skill-runner.sh utility-update-go-version
+
+# Step 3: Rebuild tools
+./scripts/rebuild-go-tools.sh
+
+# Step 4: Restart your IDE
+# VS Code: Cmd/Ctrl+Shift+P → "Developer: Reload Window"
+```
+
+That's it! Keep reading if you want to understand why.
+
+---
+
+## What's Actually Happening?
+
+### The Problem (In Plain English)
+
+Think of Go tools like a Swiss Army knife. When you upgrade Go, it's like switching from metric to imperial measurements—your old knife still works, but the measurements don't match anymore.
+
+Here's what breaks:
+
+1. **Renovate updates the project** to Go 1.26.0
+2. **Your tools are still using** Go 1.25.6
+3. **Pre-commit hooks fail** with confusing errors
+4. **Your IDE gets confused** and shows red squiggles everywhere
+
+### Why Tools Break
+
+Development tools like golangci-lint are compiled programs. They were built with Go 1.25.6 and expect Go 1.25.6's features. When you upgrade to Go 1.26.0:
+
+- New language features exist that old tools don't understand
+- Standard library functions change
+- Your tools throw errors like: `undefined: someNewFunction`
+
+**The Fix:** Rebuild tools with the new Go version so they match your project.
+
+---
+
+## Step-by-Step Upgrade Guide
+
+### Step 1: Know When an Upgrade Happened
+
+Renovate (our automated dependency manager) will open a PR titled something like:
+
+```
+chore(deps): update golang to v1.26.0
+```
+
+When this gets merged, you'll need to update your local environment.
+
+### Step 2: Pull the Latest Code
+
+```bash
+cd /projects/Charon
+git checkout development
+git pull origin development
+```
+
+### Step 3: Update Your Go Installation
+
+**Option A: Use the Automated Skill (Recommended)**
+
+```bash
+.github/skills/scripts/skill-runner.sh utility-update-go-version
+```
+
+This script:
+- Detects the required Go version from `go.work`
+- Downloads it from golang.org
+- Installs it to `~/sdk/go{version}/`
+- Updates your system symlink to point to it
+- Rebuilds your tools automatically
+
+**Option B: Manual Installation**
+
+If you prefer to install Go manually:
+
+1. Go to [go.dev/dl](https://go.dev/dl/)
+2. Download the version mentioned in the PR (e.g., 1.26.0)
+3. Install it following the official instructions
+4. Verify: `go version` should show the new version
+5. Continue to Step 4
+
+### Step 4: Rebuild Development Tools
+
+Even if you used Option A (which rebuilds automatically), you can always manually rebuild:
+
+```bash
+./scripts/rebuild-go-tools.sh
+```
+
+This rebuilds:
+- **golangci-lint** — Pre-commit linter (critical)
+- **gopls** — IDE language server (critical)
+- **govulncheck** — Security scanner
+- **dlv** — Debugger
+
+**Duration:** About 30 seconds
+
+**Output:** You'll see:
+
+```
+🔧 Rebuilding Go development tools...
+Current Go version: go version go1.26.0 linux/amd64
+
+📦 Installing golangci-lint...
+✅ golangci-lint installed successfully
+
+📦 Installing gopls...
+✅ gopls installed successfully
+
+...
+
+✅ All tools rebuilt successfully!
+```
+
+### Step 5: Restart Your IDE
+
+Your IDE caches the old Go language server (gopls). Reload to use the new one:
+
+**VS Code:**
+- Press `Cmd/Ctrl+Shift+P`
+- Type "Developer: Reload Window"
+- Press Enter
+
+**GoLand or IntelliJ IDEA:**
+- File → Invalidate Caches → Restart
+- Wait for indexing to complete
+
+### Step 6: Verify Everything Works
+
+Run a quick test:
+
+```bash
+# This should pass without errors
+go test ./backend/...
+```
+
+If tests pass, you're done! 🎉
+
+---
+
+## Troubleshooting
+
+### Error: "golangci-lint: command not found"
+
+**Problem:** Your `$PATH` doesn't include Go's binary directory.
+
+**Fix:**
+
+```bash
+# Add to ~/.bashrc or ~/.zshrc
+export PATH="$PATH:$(go env GOPATH)/bin"
+
+# Reload your shell
+source ~/.bashrc  # or source ~/.zshrc
+```
+
+Then rebuild tools:
+
+```bash
+./scripts/rebuild-go-tools.sh
+```
+
+### Error: Pre-commit hook still failing
+
+**Problem:** Pre-commit is using a cached version of the tool.
+
+**Fix 1: Let the hook auto-rebuild**
+
+The pre-commit hook detects version mismatches and rebuilds automatically. Just commit again:
+
+```bash
+git commit -m "your message"
+# Hook detects mismatch, rebuilds tool, and retries
+```
+
+**Fix 2: Manual rebuild**
+
+```bash
+./scripts/rebuild-go-tools.sh
+git commit -m "your message"
+```
+
+### Error: "package X is not in GOROOT"
+
+**Problem:** Your project's `go.work` or `go.mod` specifies a Go version you don't have installed.
+
+**Check required version:**
+
+```bash
+grep '^go ' go.work
+# Output: go 1.26.0
+```
+
+**Install that version:**
+
+```bash
+.github/skills/scripts/skill-runner.sh utility-update-go-version
+```
+
+### IDE showing errors but code compiles fine
+
+**Problem:** Your IDE's language server (gopls) is out of date.
+
+**Fix:**
+
+```bash
+# Rebuild gopls
+go install golang.org/x/tools/gopls@latest
+
+# Restart IDE
+# VS Code: Cmd/Ctrl+Shift+P → "Developer: Reload Window"
+```
+
+### "undefined: someFunction" errors
+
+**Problem:** Your tools were built with an old Go version and don't recognize new standard library functions.
+
+**Fix:**
+
+```bash
+./scripts/rebuild-go-tools.sh
+```
+
+---
+
+## Frequently Asked Questions
+
+### How often do Go versions change?
+
+Go releases **two major versions per year**:
+- February (e.g., Go 1.26.0)
+- August (e.g., Go 1.27.0)
+
+Plus occasional patch releases (e.g., Go 1.26.1) for security fixes.
+
+**Bottom line:** Expect to run `./scripts/rebuild-go-tools.sh` 2-3 times per year.
+
+### Do I need to rebuild tools for patch releases?
+
+**Usually no**, but it doesn't hurt. Patch releases (like 1.26.0 → 1.26.1) rarely break tool compatibility.
+
+**Rebuild if:**
+- Pre-commit hooks start failing
+- IDE shows unexpected errors
+- Tools report version mismatches
+
+### Why don't CI builds have this problem?
+
+CI environments are **ephemeral** (temporary). Every workflow run:
+1. Starts with a fresh container
+2. Installs Go from scratch
+3. Installs tools from scratch
+4. Runs tests
+5. Throws everything away
+
+**Local development** has persistent tool installations that get out of sync.
+
+### Can I use multiple Go versions on my machine?
+
+**Yes!** Go officially supports this via `golang.org/dl`:
+
+```bash
+# Install Go 1.25.6
+go install golang.org/dl/go1.25.6@latest
+go1.25.6 download
+
+# Install Go 1.26.0
+go install golang.org/dl/go1.26.0@latest
+go1.26.0 download
+
+# Use specific version
+go1.25.6 version
+go1.26.0 test ./...
+```
+
+But for Charon development, you only need **one version** (whatever's in `go.work`).
+
+### What if I skip an upgrade?
+
+**Short answer:** Your local tools will be out of sync, but CI will still work.
+
+**What breaks:**
+- Pre-commit hooks fail (but will auto-rebuild)
+- IDE shows phantom errors
+- Manual `go test` might fail locally
+- CI is unaffected (it always uses the correct version)
+
+**When to catch up:**
+- Before opening a PR (CI checks will fail if your code uses old Go features)
+- When local development becomes annoying
+
+### Should I keep old Go versions installed?
+
+**No need.** The upgrade script preserves old versions in `~/sdk/`, but you don't need to do anything special.
+
+If you want to clean up:
+
+```bash
+# See installed versions
+ls ~/sdk/
+
+# Remove old versions
+rm -rf ~/sdk/go1.25.5
+rm -rf ~/sdk/go1.25.6
+```
+
+But they only take ~400MB each, so cleanup is optional.
+
+### Why doesn't Renovate upgrade tools automatically?
+
+Renovate updates **Dockerfile** and **go.work**, but it can't update tools on *your* machine.
+
+**Think of it like this:**
+- Renovate: "Hey team, we're now using Go 1.26.0"
+- Your machine: "Cool, but my tools are still Go 1.25.6. Let me rebuild them."
+
+The rebuild script bridges that gap.
+
+### What's the difference between `go.work`, `go.mod`, and my system Go?
+
+**`go.work`** — Workspace file (multi-module projects like Charon)
+- Specifies minimum Go version for the entire project
+- Used by Renovate to track upgrades
+
+**`go.mod`** — Module file (individual Go modules)
+- Each module (backend, tools) has its own `go.mod`
+- Inherits Go version from `go.work`
+
+**System Go** (`go version`) — What's installed on your machine
+- Must be >= the version in `go.work`
+- Tools are compiled with whatever version this is
+
+**Example:**
+```
+go.work says: "Use Go 1.26.0 or newer"
+go.mod says: "I'm part of the workspace, use its Go version"
+Your machine: "I have Go 1.26.0 installed"
+Tools: "I was built with Go 1.25.6" ❌ MISMATCH
+```
+
+Running `./scripts/rebuild-go-tools.sh` fixes the mismatch.
+
+---
+
+## Advanced: Pre-commit Auto-Rebuild
+
+Charon's pre-commit hook automatically detects and fixes tool version mismatches.
+
+**How it works:**
+
+1. **Check versions:**
+   ```bash
+   golangci-lint version → "built with go1.25.6"
+   go version → "go version go1.26.0"
+   ```
+
+2. **Detect mismatch:**
+   ```
+   ⚠️  golangci-lint Go version mismatch:
+      golangci-lint: 1.25.6
+      system Go:     1.26.0
+   ```
+
+3. **Auto-rebuild:**
+   ```
+   🔧 Rebuilding golangci-lint with current Go version...
+   ✅ golangci-lint rebuilt successfully
+   ```
+
+4. **Retry linting:**
+   Hook runs again with the rebuilt tool.
+
+**What this means for you:**
+
+The first commit after a Go upgrade will be **slightly slower** (~30 seconds for tool rebuild). Subsequent commits are normal speed.
+
+**Disabling auto-rebuild:**
+
+If you want manual control, edit `scripts/pre-commit-hooks/golangci-lint-fast.sh` and remove the rebuild logic. (Not recommended.)
+
+---
+
+## Related Documentation
+
+- **[Go Version Management Strategy](../plans/go_version_management_strategy.md)** — Research and design decisions
+- **[CONTRIBUTING.md](../../CONTRIBUTING.md)** — Quick reference for contributors
+- **[Go Official Docs](https://go.dev/doc/manage-install)** — Official multi-version management guide
+
+---
+
+## Need Help?
+
+**Open a [Discussion](https://github.com/Wikid82/charon/discussions)** if:
+- These instructions didn't work for you
+- You're seeing errors not covered in troubleshooting
+- You have suggestions for improving this guide
+
+**Open an [Issue](https://github.com/Wikid82/charon/issues)** if:
+- The rebuild script crashes
+- Pre-commit auto-rebuild isn't working
+- CI is failing for Go version reasons
+
+---
+
+**Remember:** Go upgrades happen 2-3 times per year. When they do, just run `./scripts/rebuild-go-tools.sh` and you're good to go! 🚀
--- a/docs/development/integration-tests.md
+++ b/docs/development/integration-tests.md
@@ -0,0 +1,53 @@
+# Integration Tests Runbook
+
+## Overview
+
+This runbook describes how to run integration tests locally with the same entrypoints used in CI. It also documents the scope of each integration script, known port bindings, and the local-only Go integration tests.
+
+## Prerequisites
+
+- Docker 24+
+- Docker Compose 2+
+- curl (required by all scripts)
+- jq (required by CrowdSec decisions script)
+
+## CI-Aligned Entry Points
+
+Local runs should follow the same entrypoints used in CI workflows.
+
+- Cerberus full stack: `scripts/cerberus_integration.sh` (skill: `integration-test-cerberus`, wrapper: `.github/skills/integration-test-cerberus-scripts/run.sh`)
+- Coraza WAF: `scripts/coraza_integration.sh` (skill: `integration-test-coraza`, wrapper: `.github/skills/integration-test-coraza-scripts/run.sh`)
+- Rate limiting: `scripts/rate_limit_integration.sh` (skill: `integration-test-rate-limit`, wrapper: `.github/skills/integration-test-rate-limit-scripts/run.sh`)
+- CrowdSec bouncer: `scripts/crowdsec_integration.sh` (skill: `integration-test-crowdsec`, wrapper: `.github/skills/integration-test-crowdsec-scripts/run.sh`)
+- CrowdSec startup: `scripts/crowdsec_startup_test.sh` (skill: `integration-test-crowdsec-startup`, wrapper: `.github/skills/integration-test-crowdsec-startup-scripts/run.sh`)
+- Run all (CI-aligned): `scripts/integration-test-all.sh` (skill: `integration-test-all`, wrapper: `.github/skills/integration-test-all-scripts/run.sh`)
+
+## Local Execution (Preferred)
+
+Use the skill runner to mirror CI behavior:
+
+- `.github/skills/scripts/skill-runner.sh integration-test-all` (wrapper: `.github/skills/integration-test-all-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-cerberus` (wrapper: `.github/skills/integration-test-cerberus-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-coraza` (wrapper: `.github/skills/integration-test-coraza-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-rate-limit` (wrapper: `.github/skills/integration-test-rate-limit-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-crowdsec` (wrapper: `.github/skills/integration-test-crowdsec-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup` (wrapper: `.github/skills/integration-test-crowdsec-startup-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions` (wrapper: `.github/skills/integration-test-crowdsec-decisions-scripts/run.sh`)
+- `.github/skills/scripts/skill-runner.sh integration-test-waf` (legacy WAF path, wrapper: `.github/skills/integration-test-waf-scripts/run.sh`)
+
+## Go Integration Tests (Local-Only)
+
+Go integration tests under `backend/integration/` are build-tagged and are not executed by CI. To run them locally, use `go test -tags=integration ./backend/integration/...`.
+
+## WAF Scope
+
+- Canonical CI entrypoint: `scripts/coraza_integration.sh`
+- Local-only legacy path: `scripts/waf_integration.sh` (skill: `integration-test-waf`)
+
+## Known Port Bindings
+
+- `scripts/cerberus_integration.sh`: API 8480, HTTP 8481, HTTPS 8444, admin 2319
+- `scripts/waf_integration.sh`: API 8380, HTTP 8180, HTTPS 8143, admin 2119
+- `scripts/coraza_integration.sh`: API 8080, HTTP 80, HTTPS 443, admin 2019
+- `scripts/rate_limit_integration.sh`: API 8280, HTTP 8180, HTTPS 8143, admin 2119
+- `scripts/crowdsec_*`: API 8280/8580, HTTP 8180/8480, HTTPS 8143/8443, admin 2119 (varies by script)
--- a/docs/development/plugin-development.md
+++ b/docs/development/plugin-development.md
@@ -0,0 +1,827 @@
+# DNS Provider Plugin Development
+
+This guide covers the technical details of developing custom DNS provider plugins for Charon.
+
+## Overview
+
+Charon uses Go's plugin system to dynamically load DNS provider implementations. Plugins implement the `ProviderPlugin` interface and are compiled as shared libraries (`.so` files).
+
+### Architecture
+
+```
+┌─────────────────────────────────────────┐
+│         Charon Core Process             │
+│  ┌───────────────────────────────────┐  │
+│  │   Global Provider Registry        │  │
+│  ├───────────────────────────────────┤  │
+│  │ Built-in Providers                │  │
+│  │  - Cloudflare                     │  │
+│  │  - DNSimple                       │  │
+│  │  - Route53                        │  │
+│  ├───────────────────────────────────┤  │
+│  │ External Plugins (*.so)           │  │
+│  │  - PowerDNS     [loaded]          │  │
+│  │  - Custom       [loaded]          │  │
+│  └───────────────────────────────────┘  │
+└─────────────────────────────────────────┘
+```
+
+## Platform Requirements
+
+### Supported Platforms
+
+- **Linux:** x86_64, ARM64 (primary target)
+- **macOS:** x86_64, ARM64 (development/testing)
+- **Windows:** Not supported (Go plugin limitation)
+
+### Build Requirements
+
+- **CGO:** Must be enabled (`CGO_ENABLED=1`)
+- **Go Version:** Must match Charon's Go version exactly (currently 1.25.6+)
+- **Compiler:** GCC/Clang for Linux, Xcode tools for macOS
+- **Build Mode:** Must use `-buildmode=plugin`
+
+## Interface Specification
+
+### Interface Version
+
+Current interface version: **v1**
+
+The interface version is defined in `backend/pkg/dnsprovider/plugin.go`:
+
+```go
+const InterfaceVersion = "v1"
+```
+
+### Core Interface
+
+All plugins must implement `dnsprovider.ProviderPlugin`:
+
+```go
+type ProviderPlugin interface {
+    Type() string
+    Metadata() ProviderMetadata
+    Init() error
+    Cleanup() error
+    RequiredCredentialFields() []CredentialFieldSpec
+    OptionalCredentialFields() []CredentialFieldSpec
+    ValidateCredentials(creds map[string]string) error
+    TestCredentials(creds map[string]string) error
+    SupportsMultiCredential() bool
+    BuildCaddyConfig(creds map[string]string) map[string]any
+    BuildCaddyConfigForZone(baseDomain string, creds map[string]string) map[string]any
+    PropagationTimeout() time.Duration
+    PollingInterval() time.Duration
+}
+```
+
+### Method Reference
+
+#### `Type() string`
+
+Returns the unique provider identifier.
+
+- Must be lowercase, alphanumeric with optional underscores
+- Used as the key for registration and lookup
+- Examples: `"powerdns"`, `"custom_dns"`, `"acme_dns"`
+
+#### `Metadata() ProviderMetadata`
+
+Returns descriptive information for UI display:
+
+```go
+type ProviderMetadata struct {
+    Type             string `json:"type"`               // Same as Type()
+    Name             string `json:"name"`               // Display name
+    Description      string `json:"description"`        // Brief description
+    DocumentationURL string `json:"documentation_url"`  // Help link
+    Author           string `json:"author"`             // Plugin author
+    Version          string `json:"version"`            // Plugin version
+    IsBuiltIn        bool   `json:"is_built_in"`        // Always false for plugins
+    GoVersion        string `json:"go_version"`         // Build Go version
+    InterfaceVersion string `json:"interface_version"`  // Plugin interface version
+}
+```
+
+**Required fields:** `Type`, `Name`, `Description`, `IsBuiltIn` (false), `GoVersion`, `InterfaceVersion`
+
+#### `Init() error`
+
+Called after the plugin is loaded, before registration.
+
+Use for:
+
+- Loading configuration files
+- Validating environment
+- Establishing persistent connections
+- Resource allocation
+
+Return an error to prevent registration.
+
+#### `Cleanup() error`
+
+Called before the plugin is unregistered (graceful shutdown).
+
+Use for:
+
+- Closing connections
+- Flushing caches
+- Releasing resources
+
+**Note:** Due to Go runtime limitations, plugin code remains in memory after `Cleanup()`.
+
+#### `RequiredCredentialFields() []CredentialFieldSpec`
+
+Returns credential fields that must be provided.
+
+Example:
+
+```go
+return []dnsprovider.CredentialFieldSpec{
+    {
+        Name:        "api_token",
+        Label:       "API Token",
+        Type:        "password",
+        Placeholder: "Enter your API token",
+        Hint:        "Found in your account settings",
+    },
+}
+```
+
+#### `OptionalCredentialFields() []CredentialFieldSpec`
+
+Returns credential fields that may be provided.
+
+Example:
+
+```go
+return []dnsprovider.CredentialFieldSpec{
+    {
+        Name:        "timeout",
+        Label:       "Timeout (seconds)",
+        Type:        "text",
+        Placeholder: "30",
+        Hint:        "API request timeout",
+    },
+}
+```
+
+#### `ValidateCredentials(creds map[string]string) error`
+
+Validates credential format and presence (no network calls).
+
+Example:
+
+```go
+func (p *PowerDNSProvider) ValidateCredentials(creds map[string]string) error {
+    if creds["api_url"] == "" {
+        return fmt.Errorf("api_url is required")
+    }
+    if creds["api_key"] == "" {
+        return fmt.Errorf("api_key is required")
+    }
+    return nil
+}
+```
+
+#### `TestCredentials(creds map[string]string) error`
+
+Verifies credentials work with the provider API (may make network calls).
+
+Example:
+
+```go
+func (p *PowerDNSProvider) TestCredentials(creds map[string]string) error {
+    if err := p.ValidateCredentials(creds); err != nil {
+        return err
+    }
+
+    // Test API connectivity
+    url := creds["api_url"] + "/api/v1/servers"
+    req, _ := http.NewRequest("GET", url, nil)
+    req.Header.Set("X-API-Key", creds["api_key"])
+
+    client := &http.Client{Timeout: 10 * time.Second}
+    resp, err := client.Do(req)
+    if err != nil {
+        return fmt.Errorf("API connection failed: %w", err)
+    }
+    defer resp.Body.Close()
+
+    if resp.StatusCode != http.StatusOK {
+        return fmt.Errorf("API returned status %d", resp.StatusCode)
+    }
+    return nil
+}
+```
+
+#### `SupportsMultiCredential() bool`
+
+Indicates if the provider supports zone-specific credentials (Phase 3 feature).
+
+Return `false` for most implementations:
+
+```go
+func (p *PowerDNSProvider) SupportsMultiCredential() bool {
+    return false
+}
+```
+
+#### `BuildCaddyConfig(creds map[string]string) map[string]any`
+
+Constructs Caddy DNS challenge configuration.
+
+The returned map is embedded into Caddy's TLS automation policy for ACME DNS-01 challenges.
+
+Example:
+
+```go
+func (p *PowerDNSProvider) BuildCaddyConfig(creds map[string]string) map[string]any {
+    return map[string]any{
+        "name":      "powerdns",
+        "api_url":   creds["api_url"],
+        "api_key":   creds["api_key"],
+        "server_id": creds["server_id"],
+    }
+}
+```
+
+**Caddy Configuration Reference:** See [Caddy DNS Providers](https://github.com/caddy-dns)
+
+#### `BuildCaddyConfigForZone(baseDomain string, creds map[string]string) map[string]any`
+
+Constructs zone-specific configuration (multi-credential mode).
+
+Only called if `SupportsMultiCredential()` returns `true`.
+
+Most plugins can simply delegate to `BuildCaddyConfig()`:
+
+```go
+func (p *PowerDNSProvider) BuildCaddyConfigForZone(baseDomain string, creds map[string]string) map[string]any {
+    return p.BuildCaddyConfig(creds)
+}
+```
+
+#### `PropagationTimeout() time.Duration`
+
+Returns the recommended DNS propagation wait time.
+
+Typical values:
+
+- **Fast providers:** 30-60 seconds (Cloudflare, PowerDNS)
+- **Standard providers:** 60-120 seconds (DNSimple, Route53)
+- **Slow providers:** 120-300 seconds (traditional DNS)
+
+```go
+func (p *PowerDNSProvider) PropagationTimeout() time.Duration {
+    return 60 * time.Second
+}
+```
+
+#### `PollingInterval() time.Duration`
+
+Returns the recommended polling interval for DNS verification.
+
+Typical values: 2-10 seconds
+
+```go
+func (p *PowerDNSProvider) PollingInterval() time.Duration {
+    return 2 * time.Second
+}
+```
+
+## Plugin Structure
+
+### Minimal Plugin Template
+
+```go
+package main
+
+import (
+    "fmt"
+    "runtime"
+    "time"
+
+    "github.com/Wikid82/charon/backend/pkg/dnsprovider"
+)
+
+// Plugin is the exported symbol that Charon looks for
+var Plugin dnsprovider.ProviderPlugin = &MyProvider{}
+
+type MyProvider struct{}
+
+func (p *MyProvider) Type() string {
+    return "myprovider"
+}
+
+func (p *MyProvider) Metadata() dnsprovider.ProviderMetadata {
+    return dnsprovider.ProviderMetadata{
+        Type:             "myprovider",
+        Name:             "My DNS Provider",
+        Description:      "Custom DNS provider implementation",
+        DocumentationURL: "https://example.com/docs",
+        Author:           "Your Name",
+        Version:          "1.0.0",
+        IsBuiltIn:        false,
+        GoVersion:        runtime.Version(),
+        InterfaceVersion: dnsprovider.InterfaceVersion,
+    }
+}
+
+func (p *MyProvider) Init() error {
+    return nil
+}
+
+func (p *MyProvider) Cleanup() error {
+    return nil
+}
+
+func (p *MyProvider) RequiredCredentialFields() []dnsprovider.CredentialFieldSpec {
+    return []dnsprovider.CredentialFieldSpec{
+        {
+            Name:        "api_key",
+            Label:       "API Key",
+            Type:        "password",
+            Placeholder: "Enter your API key",
+            Hint:        "Found in your account settings",
+        },
+    }
+}
+
+func (p *MyProvider) OptionalCredentialFields() []dnsprovider.CredentialFieldSpec {
+    return []dnsprovider.CredentialFieldSpec{}
+}
+
+func (p *MyProvider) ValidateCredentials(creds map[string]string) error {
+    if creds["api_key"] == "" {
+        return fmt.Errorf("api_key is required")
+    }
+    return nil
+}
+
+func (p *MyProvider) TestCredentials(creds map[string]string) error {
+    return p.ValidateCredentials(creds)
+}
+
+func (p *MyProvider) SupportsMultiCredential() bool {
+    return false
+}
+
+func (p *MyProvider) BuildCaddyConfig(creds map[string]string) map[string]any {
+    return map[string]any{
+        "name":    "myprovider",
+        "api_key": creds["api_key"],
+    }
+}
+
+func (p *MyProvider) BuildCaddyConfigForZone(baseDomain string, creds map[string]string) map[string]any {
+    return p.BuildCaddyConfig(creds)
+}
+
+func (p *MyProvider) PropagationTimeout() time.Duration {
+    return 60 * time.Second
+}
+
+func (p *MyProvider) PollingInterval() time.Duration {
+    return 5 * time.Second
+}
+
+func main() {}
+```
+
+### Project Layout
+
+```
+my-provider-plugin/
+├── go.mod
+├── go.sum
+├── main.go
+├── Makefile
+└── README.md
+```
+
+### `go.mod` Requirements
+
+```go
+module github.com/yourname/charon-plugin-myprovider
+
+go 1.25
+
+require (
+    github.com/Wikid82/charon v0.0.0-20240101000000-abcdef123456
+)
+```
+
+**Important:** Use `replace` directive for local development:
+
+```go
+replace github.com/Wikid82/charon => /path/to/charon
+```
+
+## Building Plugins
+
+### Build Command
+
+```bash
+CGO_ENABLED=1 go build -buildmode=plugin -o myprovider.so main.go
+```
+
+### Build Requirements
+
+1. **CGO must be enabled:**
+
+   ```bash
+   export CGO_ENABLED=1
+   ```
+
+2. **Go version must match Charon:**
+
+   ```bash
+   go version
+   # Must match Charon's build Go version
+   ```
+
+3. **Architecture must match:**
+
+   ```bash
+   # For cross-compilation
+   GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -buildmode=plugin
+   ```
+
+### Makefile Example
+
+```makefile
+.PHONY: build clean install
+
+PLUGIN_NAME = myprovider
+OUTPUT = $(PLUGIN_NAME).so
+INSTALL_DIR = /etc/charon/plugins
+
+build:
+ CGO_ENABLED=1 go build -buildmode=plugin -o $(OUTPUT) main.go
+
+clean:
+ rm -f $(OUTPUT)
+
+install: build
+ install -m 755 $(OUTPUT) $(INSTALL_DIR)/
+
+test:
+ go test -v ./...
+
+lint:
+ golangci-lint run
+
+signature:
+ @echo "SHA-256 Signature:"
+ @sha256sum $(OUTPUT)
+```
+
+### Build Script
+
+```bash
+#!/bin/bash
+set -e
+
+PLUGIN_NAME="myprovider"
+GO_VERSION=$(go version | awk '{print $3}')
+CHARON_GO_VERSION="go1.25.6"
+
+# Verify Go version
+if [ "$GO_VERSION" != "$CHARON_GO_VERSION" ]; then
+    echo "Warning: Go version mismatch"
+    echo "  Plugin: $GO_VERSION"
+    echo "  Charon: $CHARON_GO_VERSION"
+    read -p "Continue? (y/n) " -n 1 -r
+    echo
+    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+        exit 1
+    fi
+fi
+
+# Build plugin
+echo "Building $PLUGIN_NAME.so..."
+CGO_ENABLED=1 go build -buildmode=plugin -o "${PLUGIN_NAME}.so" main.go
+
+# Generate signature
+echo "Generating signature..."
+sha256sum "${PLUGIN_NAME}.so" | tee "${PLUGIN_NAME}.so.sha256"
+
+echo "Build complete!"
+```
+
+## Development Workflow
+
+### 1. Set Up Development Environment
+
+```bash
+# Clone plugin template
+git clone https://github.com/yourname/charon-plugin-template my-provider
+cd my-provider
+
+# Install dependencies
+go mod download
+
+# Set up local Charon dependency
+echo 'replace github.com/Wikid82/charon => /path/to/charon' >> go.mod
+go mod tidy
+```
+
+### 2. Implement Provider Interface
+
+Edit `main.go` to implement all required methods.
+
+### 3. Test Locally
+
+```bash
+# Build plugin
+make build
+
+# Copy to Charon plugin directory
+cp myprovider.so /etc/charon/plugins/
+
+# Restart Charon
+systemctl restart charon
+
+# Check logs
+journalctl -u charon -f | grep plugin
+```
+
+### 4. Debug Plugin Loading
+
+Enable debug logging in Charon:
+
+```yaml
+log:
+  level: debug
+```
+
+Check for errors:
+
+```bash
+journalctl -u charon -n 100 | grep -i plugin
+```
+
+### 5. Test Credential Validation
+
+```bash
+curl -X POST http://localhost:8080/api/admin/dns-providers/test \
+  -H "Content-Type: application/json" \
+  -d '{
+    "type": "myprovider",
+    "credentials": {
+      "api_key": "test-key"
+    }
+  }'
+```
+
+### 6. Test DNS Challenge
+
+Configure a test domain to use your provider and request a certificate.
+
+Monitor Caddy logs for DNS challenge execution:
+
+```bash
+docker logs charon-caddy -f | grep dns
+```
+
+## Best Practices
+
+### Security
+
+1. **Validate All Inputs:** Never trust credential data
+2. **Use HTTPS:** Always use TLS for API connections
+3. **Timeout Requests:** Set reasonable timeouts on all HTTP calls
+4. **Sanitize Errors:** Don't leak credentials in error messages
+5. **Log Safely:** Redact sensitive data from logs
+
+### Performance
+
+1. **Minimize Init() Work:** Fast startup is critical
+2. **Connection Pooling:** Reuse HTTP clients and connections
+3. **Efficient Polling:** Use appropriate polling intervals
+4. **Cache When Possible:** Cache provider metadata
+5. **Fail Fast:** Return errors quickly for invalid credentials
+
+### Reliability
+
+1. **Handle Nil Gracefully:** Check for nil maps and slices
+2. **Provide Defaults:** Use sensible defaults for optional fields
+3. **Retry Transient Errors:** Implement exponential backoff
+4. **Graceful Degradation:** Continue working if non-critical features fail
+
+### Maintainability
+
+1. **Document Public APIs:** Use godoc comments
+2. **Version Your Plugin:** Include semantic versioning
+3. **Test Thoroughly:** Unit tests for all methods
+4. **Provide Examples:** Include configuration examples
+
+## Testing
+
+### Unit Tests
+
+```go
+package main
+
+import (
+    "testing"
+
+    "github.com/Wikid82/charon/backend/pkg/dnsprovider"
+    "github.com/stretchr/testify/assert"
+)
+
+func TestValidateCredentials(t *testing.T) {
+    provider := &MyProvider{}
+
+    tests := []struct {
+        name      string
+        creds     map[string]string
+        expectErr bool
+    }{
+        {
+            name:      "valid credentials",
+            creds:     map[string]string{"api_key": "test-key"},
+            expectErr: false,
+        },
+        {
+            name:      "missing api_key",
+            creds:     map[string]string{},
+            expectErr: true,
+        },
+    }
+
+    for _, tt := range tests {
+        t.Run(tt.name, func(t *testing.T) {
+            err := provider.ValidateCredentials(tt.creds)
+            if tt.expectErr {
+                assert.Error(t, err)
+            } else {
+                assert.NoError(t, err)
+            }
+        })
+    }
+}
+
+func TestMetadata(t *testing.T) {
+    provider := &MyProvider{}
+    meta := provider.Metadata()
+
+    assert.Equal(t, "myprovider", meta.Type)
+    assert.NotEmpty(t, meta.Name)
+    assert.False(t, meta.IsBuiltIn)
+    assert.Equal(t, dnsprovider.InterfaceVersion, meta.InterfaceVersion)
+}
+```
+
+### Integration Tests
+
+```go
+func TestRealAPIConnection(t *testing.T) {
+    if testing.Short() {
+        t.Skip("Skipping integration test")
+    }
+
+    provider := &MyProvider{}
+    creds := map[string]string{
+        "api_key": os.Getenv("TEST_API_KEY"),
+    }
+
+    err := provider.TestCredentials(creds)
+    assert.NoError(t, err)
+}
+```
+
+Run integration tests:
+
+```bash
+go test -v ./... -count=1
+```
+
+## Troubleshooting
+
+### Common Build Errors
+
+#### `plugin was built with a different version of package`
+
+**Cause:** Dependency version mismatch
+
+**Solution:**
+
+```bash
+go clean -cache
+go mod tidy
+go build -buildmode=plugin
+```
+
+#### `cannot use -buildmode=plugin`
+
+**Cause:** CGO not enabled
+
+**Solution:**
+
+```bash
+export CGO_ENABLED=1
+```
+
+#### `undefined: dnsprovider.ProviderPlugin`
+
+**Cause:** Missing or incorrect import
+
+**Solution:**
+
+```go
+import "github.com/Wikid82/charon/backend/pkg/dnsprovider"
+```
+
+### Runtime Errors
+
+#### `plugin was built with a different version of Go`
+
+**Cause:** Go version mismatch between plugin and Charon
+
+**Solution:** Rebuild plugin with matching Go version
+
+#### `symbol not found: Plugin`
+
+**Cause:** Plugin variable not exported
+
+**Solution:**
+
+```go
+// Must be exported (capitalized)
+var Plugin dnsprovider.ProviderPlugin = &MyProvider{}
+```
+
+#### `interface version mismatch`
+
+**Cause:** Plugin built against incompatible interface
+
+**Solution:** Update plugin to match Charon's interface version
+
+## Publishing Plugins
+
+### Release Checklist
+
+- [ ] All methods implemented and tested
+- [ ] Go version matches current Charon release
+- [ ] Interface version set correctly
+- [ ] Documentation includes usage examples
+- [ ] README includes installation instructions
+- [ ] LICENSE file included
+- [ ] Changelog maintained
+- [ ] GitHub releases with binaries for all platforms
+
+### Distribution
+
+1. **GitHub Releases:**
+
+   ```bash
+   # Tag release
+   git tag -a v1.0.0 -m "Release v1.0.0"
+   git push origin v1.0.0
+
+   # Build for multiple platforms
+   make build-all
+
+   # Create GitHub release and attach binaries
+   ```
+
+2. **Signature File:**
+
+   ```bash
+   sha256sum *.so > SHA256SUMS
+   gpg --sign SHA256SUMS
+   ```
+
+3. **Documentation:**
+   - Include README with installation instructions
+   - Provide configuration examples
+   - List required Charon version
+   - Include troubleshooting section
+
+## Resources
+
+### Reference Implementation
+
+- **PowerDNS Plugin:** [`plugins/powerdns/main.go`](../../plugins/powerdns/main.go)
+- **Built-in Providers:** [`backend/pkg/dnsprovider/builtin/`](../../backend/pkg/dnsprovider/builtin/)
+- **Plugin Interface:** [`backend/pkg/dnsprovider/plugin.go`](../../backend/pkg/dnsprovider/plugin.go)
+
+### External Documentation
+
+- [Go Plugin Package](https://pkg.go.dev/plugin)
+- [Caddy DNS Providers](https://github.com/caddy-dns)
+- [ACME DNS-01 Challenge](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge)
+
+### Community
+
+- **GitHub Discussions:** <https://github.com/Wikid82/charon/discussions>
+- **Plugin Registry:** <https://github.com/Wikid82/charon-plugins>
+- **Issue Tracker:** <https://github.com/Wikid82/charon/issues>
+
+## See Also
+
+- [Custom Plugin Installation Guide](../features/custom-plugins.md)
+- [DNS Provider Configuration](../features/dns-providers.md)
+- [Contributing Guidelines](../../CONTRIBUTING.md)
--- a/docs/development/running-e2e.md
+++ b/docs/development/running-e2e.md
@@ -0,0 +1,70 @@
+# Running Playwright E2E (headed and headless)
+
+This document explains how to run Playwright tests using a real browser (headed) on Linux machines and in the project's Docker E2E environment.
+
+## Key points
+- Playwright's interactive Test UI (--ui) requires an X server (a display). On headless CI or servers, use Xvfb.
+- Prefer the project's E2E Docker image for integration-like runs; use the local `--ui` flow for manual debugging.
+
+## Quick commands (local Linux)
+- Headless (recommended for CI / fast runs):
+  ```bash
+  npm run e2e
+  ```
+
+- Headed UI on a headless machine (auto-starts Xvfb):
+  ```bash
+  npm run e2e:ui:headless-server
+  # or, if you prefer manual control:
+  xvfb-run --auto-servernum --server-args='-screen 0 1280x720x24' npx playwright test --ui
+  ```
+
+- Headed UI on a workstation with an X server already running:
+  ```bash
+  npx playwright test --ui
+  ```
+
+- Open the running Docker E2E app in your system browser (one-step via VS Code task):
+  - Run the VS Code task: **Open: App in System Browser (Docker E2E)**
+  - This will rebuild the E2E container (if needed), wait for http://localhost:8080 to respond, and open your system browser automatically.
+
+- Open the running Docker E2E app in VS Code Simple Browser:
+  - Run the VS Code task: **Open: App in Simple Browser (Docker E2E)**
+  - Then use the command palette: `Simple Browser: Open URL` → paste `http://localhost:8080`
+
+## Using the project's E2E Docker image (recommended for parity with CI)
+1. Rebuild/start the E2E container (this sets up the full test environment):
+   ```bash
+   .github/skills/scripts/skill-runner.sh docker-rebuild-e2e
+   ```
+  If you need a clean rebuild after integration alignment changes:
+  ```bash
+  .github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean --no-cache
+  ```
+2. Run the UI against the container (you still need an X server on your host):
+   ```bash
+   PLAYWRIGHT_BASE_URL=http://localhost:8080 npm run e2e:ui:headless-server
+   ```
+
+## CI guidance
+- Do not run Playwright `--ui` in CI. Use headless runs or the E2E Docker image and collect traces/videos for failures.
+- For coverage, use the provided skill: `.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage`
+
+## Troubleshooting
+- Playwright error: "Looks like you launched a headed browser without having a XServer running." → run `npm run e2e:ui:headless-server` or install Xvfb.
+- If `npm run e2e:ui:headless-server` fails with an exit code like `148`:
+  - Inspect Xvfb logs: `tail -n 200 /tmp/xvfb.playwright.log`
+  - Ensure no permission issues on `/tmp/.X11-unix`: `ls -la /tmp/.X11-unix`
+  - Try starting Xvfb manually: `Xvfb :99 -screen 0 1280x720x24 &` then `export DISPLAY=:99` and re-run `npx playwright test --ui`.
+- If running inside Docker, prefer the skill-runner which provisions the required services; the UI still needs host X (or use VNC).
+
+## Developer notes (what we changed)
+- Added `scripts/run-e2e-ui.sh` — wrapper that auto-starts Xvfb when DISPLAY is unset.
+- Added `npm run e2e:ui:headless-server` to run the Playwright UI on headless machines.
+- Playwright config now auto-starts Xvfb when `--ui` is requested locally and prints an actionable error if Xvfb is not available.
+
+## Security & hygiene
+- Playwright auth artifacts are ignored by git (`playwright/.auth/`). Do not commit credentials.
+
+---
+If you'd like, I can open a PR with these changes (scripts + config + docs) and add a short CI note to `.github/` workflows.