akanealw/Charon
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

fix: complete geoip2-golang v2 migration

Browse Source 
- Update import paths to github.com/oschwald/geoip2-golang/v2
- Handle API breaking changes (net.IP → netip.Addr, IsoCode → ISOCode)
- Fix VERSION.md to match git tag (0.7.13)
- Resolves CI failure in benchmark workflow
This commit is contained in:
GitHub Actions
2025-12-14 08:06:32 +00:00
parent 8489394bbc
commit 7f82df80b7
9 changed files with 1130 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

407
docs/plans/current_spec.md
View File

@@ -1,45 +1,380 @@
# Fix CrowdSec Persistence & Offline Status
# CI/CD Failure Diagnosis Report
## Goal Description
The CrowdSec Security Engine is reported as "Offline" on the dashboard. This is caused by the lack of data persistence in the Docker container.
The `docker-entrypoint.sh` and `Dockerfile` currently configure CrowdSec to use ephemeral paths (`/etc/crowdsec` and `/var/lib/crowdsec/data`) which are not linked to the persistent volume `/app/data/crowdsec`.
Consequently, every container restart generates a new Machine ID and loses enrollment credentials, causing the dashboard to see the old instance as offline.
**Date**: December 14, 2025
**GitHub Actions Run**: [#20204673793](https://github.com/Wikid82/Charon/actions/runs/20204673793)
**Workflow**: `benchmark.yml` (Go Benchmark)
**Status**: ❌ Failed
**Commit**: `8489394` - Merge pull request #396
## User Review Required
> [!IMPORTANT]
> **Re-Enrollment Required**: After this fix is applied, the user will need to re-enroll their instance once. The new identity will persist across future restarts.
> **Mode Configuration**: The user must ensure `CERBERUS_SECURITY_CROWDSEC_MODE` is set to `local` in their environment or `docker-compose.yml`.
---
## Proposed Changes
## Executive Summary
### Docker & Scripts
#### [MODIFY] [docker-entrypoint.sh](file:///projects/Charon/docker-entrypoint.sh)
- Update CrowdSec initialization logic to map runtime directories to persistence:
- Check for `/app/data/crowdsec/config` and `/app/data/crowdsec/data`.
- If missing, populate from `/etc/crowdsec` (defaults).
- Use symbolic links or environment variables (`DATA`) to point to `/app/data/crowdsec/...`.
- Ensure `cscli` commands operate on the persistent configuration.
The CI/CD failure is caused by an **incomplete Go module migration** from `github.com/oschwald/geoip2-golang` v1 to v2. The Renovate bot PR #396 updated `go.mod` to use v2 of the package, but:
#### [MODIFY] [docker-compose.yml](file:///projects/Charon/docker-compose.yml)
- Update comments to explicitly recommend setting `CERBERUS_SECURITY_CROWDSEC_MODE=local` to avoid confusion.
1. The actual source code still imports the v1 package path (without `/v2`)
2. This created a mismatch where `go.mod` declares v2 but the code imports v1
3. The module resolution system cannot find the v1 package because it's been removed from `go.mod`
## Verification Plan
**Root Cause**: Import path incompatibility between major versions in Go modules. When upgrading from v1 to v2 of a Go module, both the `go.mod` AND the import statements in source files must be updated to include the `/v2` suffix.
### Manual Verification
1. **Persistence Test**:
- Deploy the updated container.
- Enter container: `docker exec -it charon sh`.
- Run `cscli machines list` and note the Machine ID.
- Modify a file in `/etc/crowdsec` (e.g., `touch /etc/crowdsec/test_persist`).
- Restart container: `docker restart charon`.
- Enter container again.
- Verify `cscli machines list` shows the **SAME** Machine ID.
- Verify `/etc/crowdsec/test_persist` still exists.
---
2. **Online Enrollment Test**:
- Enroll the instance: `cscli console enroll <enroll-key>`.
- Restart container.
- Check `cscli console status` (if available) or verify on Dashboard that it remains "Online".
## Workflow Description
### Automated Tests
- None (requires Docker runtime test, which is manual in this context).
### What the Failing Workflow Does
The `benchmark.yml` workflow (`Go Benchmark`) performs:
1. **Checkout** repository code
2. **Set up Go** environment (v1.25.5)
3. **Run benchmarks** on backend code using `go test -bench=.`
4. **Store benchmark results** (only on pushes to main branch)
5. **Run performance assertions** to catch regressions
**Purpose**: Continuous performance monitoring to detect regressions before they reach production.
**Trigger**: Runs on push/PR to `main` or `development` branches when backend files change.
---
## Failing Step Details
### Step: "Performance Regression Check"
**Error Messages** (9 identical errors):
```
no required module provides package github.com/oschwald/geoip2-golang; to add it:
go get github.com/oschwald/geoip2-golang
```
**Exit Code**: 1 (compilation failure)
**Phase**: Build/compilation phase during `go test` execution
**Affected Files**:
- `/projects/Charon/backend/internal/services/geoip_service.go` (line 9)
- `/projects/Charon/backend/internal/services/geoip_service_test.go` (line 10)
---
## Renovate Changes Analysis
### PR #396: Update github.com/oschwald/geoip2-golang to v2
**Branch**: `renovate/github.com-oschwald-geoip2-golang-2.x`
**Merge Commit**: `8489394` into `development`
**Changes Made by Renovate**:
```diff
# backend/go.mod
- github.com/oschwald/geoip2-golang v1.13.0
+ github.com/oschwald/geoip2-golang/v2 v2.0.1
```
**Issue**: Renovate added the v2 dependency but also left a duplicate entry, resulting in:
```go
require (
// ... other deps ...
github.com/oschwald/geoip2-golang/v2 v2.0.1 // ← ADDED BY RENOVATE
github.com/oschwald/geoip2-golang/v2 v2.0.1 // ← DUPLICATE!
// ... other deps ...
)
```
The v1 dependency was **removed** from `go.mod`.
**Related Commits**:
- `8489394`: Merge PR #396
- `dd9a559`: Renovate branch with geoip2 v2 update
- `6469c6a`: Previous development state (had v1)
---
## Root Cause Analysis
### The Problem
Go modules use [semantic import versioning](https://go.dev/blog/v2-go-modules). For major version 2 and above, the import path **must** include the major version:
**v1 (or unversioned)**:
```go
import "github.com/oschwald/geoip2-golang"
```
**v2+**:
```go
import "github.com/oschwald/geoip2-golang/v2"
```
### What Happened
1. **Before PR #396**:
- `go.mod`: contained `github.com/oschwald/geoip2-golang v1.13.0`
- Source code: imports `github.com/oschwald/geoip2-golang`
- ✅ Everything aligned and working
2. **After PR #396 (Renovate)**:
- `go.mod`: contains `github.com/oschwald/geoip2-golang/v2 v2.0.1` (duplicate entry)
- Source code: **still** imports `github.com/oschwald/geoip2-golang` (v1 path)
- ❌ Mismatch: code wants v1, but only v2 is available
3. **Go Module Resolution**:
- When Go sees `import "github.com/oschwald/geoip2-golang"`, it looks for a module matching that path
- `go.mod` only has `github.com/oschwald/geoip2-golang/v2`
- These are **different module paths** in Go's eyes
- Result: "no required module provides package"
### Verification
Running `go mod tidy` shows:
```
go: finding module for package github.com/oschwald/geoip2-golang
go: found github.com/oschwald/geoip2-golang in github.com/oschwald/geoip2-golang v1.13.0
unused github.com/oschwald/geoip2-golang/v2
```
This confirms:
- Go finds v1 when analyzing imports
- v2 is declared but unused
- The imports and go.mod are out of sync
---
## Impact Assessment
### Directly Affected
- ✅ **security-weekly-rebuild.yml** (the file currently open in editor): NOT affected
- This workflow builds Docker images and doesn't run Go tests directly
- It will succeed if the Docker build process works
- ❌ **benchmark.yml**: FAILING
- Cannot compile backend code
- Blocks performance regression checks
### Potentially Affected
All workflows that compile or test backend Go code:
- `go-build.yml` or similar build workflows
- `go-test.yml` or test workflows
- Any integration tests that compile the backend
- Docker builds that include `go build` steps inside the container
---
## Why Renovate Didn't Handle This
**Renovate's Behavior**:
- Renovate excels at updating dependency **declarations** (in `go.mod`, `package.json`, etc.)
- It updates version numbers and dependency paths in configuration files
- However, it **does not** modify source code imports automatically
**Why Import Updates Are Manual**:
1. Import path changes are **code changes**, not config changes
2. Requires semantic understanding of the codebase
3. May involve API changes that need human review
4. Risk of breaking changes in major version bumps
**Expected Workflow for Major Go Module Updates**:
1. Renovate creates PR updating `go.mod` with v2 path
2. Human reviewer identifies this requires import changes
3. Developer manually updates all import statements
4. Tests confirm everything works with v2 API
5. PR is merged
**What Went Wrong**:
- Renovate was configured for automerge on patch updates
- This appears to have been a major version update (v1 → v2)
- Either automerge rules were too permissive, or manual review was skipped
- The duplicate entry in `go.mod` suggests a merge conflict or incomplete update
---
## Recommended Fix Approach
### Step 1: Update Import Statements
Replace all occurrences of v1 import path with v2:
**Files to Update**:
- `backend/internal/services/geoip_service.go` (line 9)
- `backend/internal/services/geoip_service_test.go` (line 10)
**Change**:
```go
// FROM:
import "github.com/oschwald/geoip2-golang"
// TO:
import "github.com/oschwald/geoip2-golang/v2"
```
### Step 2: Remove Duplicate go.mod Entry
**File**: `backend/go.mod`
**Issue**: Line 13 and 14 both have:
```go
github.com/oschwald/geoip2-golang/v2 v2.0.1
github.com/oschwald/geoip2-golang/v2 v2.0.1 // ← DUPLICATE
```
**Fix**: Remove one duplicate entry.
### Step 3: Run go mod tidy
```bash
cd backend
go mod tidy
```
This will:
- Clean up any unused dependencies
- Update `go.sum` with correct checksums for v2
- Verify all imports are satisfied
### Step 4: Verify the Build
```bash
cd backend
go build ./...
go test ./...
```
### Step 5: Check for API Changes
**IMPORTANT**: Major version bumps may include breaking API changes.
Review the [geoip2-golang v2.0.0 release notes](https://github.com/oschwald/geoip2-golang/releases/tag/v2.0.0) for:
- Renamed functions or types
- Changed function signatures
- Deprecated features
Update code accordingly if the API has changed.
### Step 6: Test Affected Workflows
Trigger the benchmark workflow to confirm it passes:
```bash
git push origin development
```
---
## Prevention Recommendations
### 1. Update Renovate Configuration
Add a rule to prevent automerge on major version updates for Go modules:
```json
{
"packageRules": [
{
"description": "Manual review required for Go major version updates",
"matchManagers": ["gomod"],
"matchUpdateTypes": ["major"],
"automerge": false,
"labels": ["dependencies", "go", "manual-review", "breaking-change"]
}
]
}
```
This ensures major updates wait for human review to handle import path changes.
### 2. Add Pre-merge CI Check
Ensure the benchmark workflow (or a build workflow) runs on PRs to `development`:
```yaml
# benchmark.yml already has this
pull_request:
branches:
- main
- development
```
This would have caught the issue before merge.
### 3. Document Major Update Process
Create a checklist for major Go module updates:
- [ ] Update `go.mod` version
- [ ] Update import paths in all source files (add `/v2`, `/v3`, etc.)
- [ ] Run `go mod tidy`
- [ ] Review release notes for breaking changes
- [ ] Update code for API changes
- [ ] Run full test suite
- [ ] Verify benchmarks pass
### 4. Go Module Update Script
Create a helper script to automate import path updates:
```bash
# scripts/update-go-major-version.sh
# Usage: ./scripts/update-go-major-version.sh github.com/oschwald/geoip2-golang 2
```
---
## Additional Context
### Go Semantic Import Versioning
From [Go Modules v2+ documentation](https://go.dev/blog/v2-go-modules):
> If a module is version v2 or higher, the major version of the module must be included as a /vN at the end of the module paths used in go.mod files and in the package import path.
This is a **fundamental requirement** of Go modules, not a limitation or bug. It ensures:
- Clear indication of major version in code
- Ability to import multiple major versions simultaneously
- Explicit acknowledgment of breaking changes
### Similar Past Issues
This is a common pitfall when updating Go modules. Other examples in the Go ecosystem:
- `gopkg.in` packages (use `/v2`, `/v3` suffixes)
- `github.com/go-chi/chi` → `github.com/go-chi/chi/v5`
- `github.com/gorilla/mux` → `github.com/gorilla/mux/v2` (if they release one)
### Why the Duplicate Entry?
The duplicate in `go.mod` likely occurred because:
1. Renovate added the v2 dependency
2. A merge conflict or concurrent edit preserved an old v2 entry
3. `go mod tidy` was not run after the merge
4. The duplicate doesn't cause an error (Go just ignores duplicates)
However, the real issue is the import path mismatch, not the duplicate.
---
## Conclusion
This is a **textbook case** of incomplete Go module major version migration. The fix is straightforward but requires manual code changes that automation tools like Renovate cannot safely perform.
**Estimated Time to Fix**: 10-15 minutes
**Risk Level**: Low (fix is well-defined and testable)
**Priority**: High (blocks CI/CD and potentially other workflows)
---
## References
- [Go Modules: v2 and Beyond](https://go.dev/blog/v2-go-modules)
- [Go Module Reference](https://go.dev/ref/mod)
- [geoip2-golang v2 Release Notes](https://github.com/oschwald/geoip2-golang/releases/tag/v2.0.0)
- [Renovate Go Modules Documentation](https://docs.renovatebot.com/modules/manager/gomod/)
- [Failed GitHub Actions Run](https://github.com/Wikid82/Charon/actions/runs/20204673793)
- [PR #396: Update geoip2-golang to v2](https://github.com/Wikid82/Charon/pull/396)
---
*Report generated by GitHub Copilot (Claude Sonnet 4.5)*