akanealw/Charon
1
0
Fork 0
Code Issues Pull Requests Actions 16 Packages Projects Releases Wiki Activity
Files
bbdeedda5dc9a590bb7c45614dd7bfddc189da0d
Charon/docs/plans/current_spec.md

15 KiB
Raw Blame History

Go Version Mismatch Fix - Critical CI/CD Pipeline Issue

Issue: PR #550 blocked by Go version compatibility error Status: Analysis Complete - Ready for Implementation (REVISED: All 7 Workflows) Priority: 🔴 CRITICAL - Blocking entire build pipeline Created: 2026-01-26 Revised: 2026-01-26 (Scope expanded from 2 to 7 workflows)

🎯 Scope Summary

This specification covers ALL 7 GitHub Actions workflows that use Go:

# Workflow Current Go Version Status Action Required
1 quality-checks.yml 1.25.6 ✅ Correct version Add GOTOOLCHAIN: auto
2 codeql.yml 1.25.6 ✅ Correct version Add GOTOOLCHAIN: auto
3 benchmark.yml 1.25.6 ✅ Correct version Add GOTOOLCHAIN: auto
4 codecov-upload.yml 1.25.6 ✅ Correct version Add GOTOOLCHAIN: auto
5 e2e-tests.yml 1.21 ⚠️ OUTDATED! Update to 1.25.6 + Add GOTOOLCHAIN: auto
6 nightly-build.yml Hardcoded ⚠️ No global env Create env section with GOTOOLCHAIN: auto
7 release-goreleaser.yml 1.25.6 ✅ Correct version Add GOTOOLCHAIN: auto

Why All 7? Initial analysis only covered 2 workflows. Supervisor review identified 5 additional workflows that would fail without this fix, including a CRITICAL issue in e2e-tests.yml using outdated Go 1.21.

Problem Analysis

Error Context

go: ../go.work requires go >= 1.25.6 (running go 1.21.13; GOTOOLCHAIN=local)
make: *** [Makefile:62: build] Error 1

Root Cause Identified

The issue is NOT an invalid Go version. Go 1.25.6 is a valid, released version (verified via https://go.dev/dl/).

The actual problem: The pre-commit framework sets GOTOOLCHAIN=local by default, which prevents automatic toolchain upgrades. When CI runs with an older Go version (1.21.13), it cannot upgrade to the required 1.25.6.

Evidence:

  • backend/.venv/lib/python3.12/site-packages/pre_commit/languages/golang.py explicitly sets GOTOOLCHAIN=local
  • CI environment has Go 1.21.13 installed system-wide
  • Workspace requires Go 1.25.6 (go.work, go.mod)
  • Docker builds use Go 1.25.6 successfully
  • Local environment with Go 1.25.6 works correctly

Current Configuration Audit

File Go Version Status
go.work 1.25.6 ✅ Correct
backend/go.mod 1.25.6 ✅ Correct
Dockerfile (gosu-builder) 1.25-trixie ✅ Correct
Dockerfile (backend-builder) 1.25-trixie ✅ Correct
Dockerfile (caddy-builder) 1.25-trixie ✅ Correct
Dockerfile (crowdsec-builder) 1.25.6-trixie ✅ Correct (pinned via Renovate)
.github/workflows/quality-checks.yml 1.25.6 ✅ Correct
.github/workflows/docker-build.yml (uses Dockerfile) ✅ Correct
.github/workflows/codeql.yml 1.25.6 ✅ Correct
Makefile (install-go comment) 1.25.5 ⚠️ Outdated comment

Conclusion: Most version declarations are correctly set to 1.25.6. However, CRITICAL FINDING: e2e-tests.yml uses outdated Go 1.21, which MUST be updated to 1.25.6. Additionally, the CI environment's inability to upgrade due to GOTOOLCHAIN=local affects all 7 workflows.

Critical Issues Found During Analysis:

  1. ⚠️ e2e-tests.yml: Uses Go 1.21 (outdated) - MUST update to 1.25.6
  2. ⚠️ nightly-build.yml: No global env section - should consolidate version management
  3. ✅ Other 5 workflows: Already use Go 1.25.6 but need GOTOOLCHAIN setting

Solution Strategy

Option A: Set GOTOOLCHAIN=auto in CI (RECOMMENDED)

Approach: Override GOTOOLCHAIN=local in GitHub Actions workflows to allow automatic toolchain upgrades.

Rationale:

  • Minimal changes: Only workflow files need modification
  • Future-proof: Allows automatic upgrades when new Go versions are released
  • CI best practice: GitHub Actions should always use the version specified in workflow
  • Matches Go team recommendation: GOTOOLCHAIN=auto is the default for most Go projects
  • No impact on local development: Developers with correct Go version unaffected

Implementation:

  1. Add GOTOOLCHAIN: auto to env section in workflow files
  2. Files to modify:
    • .github/workflows/quality-checks.yml
    • .github/workflows/codeql.yml
    • Any other workflow that invokes Go commands

Risk Assessment: ⬇️ LOW

  • Change is isolated to CI environment
  • Does not affect Docker builds (already working)
  • Does not affect local development (already working)
  • Reversible if issues arise

Option B: Update Pre-commit Configuration (NOT RECOMMENDED)

Approach: Attempt to override pre-commit's GOTOOLCHAIN=local setting.

Why Not Recommended:

  • Pre-commit's golang handler is hardcoded to set GOTOOLCHAIN=local
  • Would require forking pre-commit or monkey-patching
  • High maintenance burden
  • Doesn't address CI environment directly
  • Complex and fragile solution

Option C: Downgrade Go Version Requirements (NOT RECOMMENDED)

Approach: Revert go.work and go.mod to Go 1.21.x.

Why Not Recommended:

  • Security risk: Go 1.21 is older and missing security patches
  • Blocks dependency updates: Many modern Go packages require 1.23+
  • Regression: Reverses intentional upgrade decision
  • Docker already uses 1.25.6: Would create inconsistency
  • Go 1.25.6 is stable: No reason to downgrade

Implementation Plan (Option A - Recommended)

Phase 1: Update GitHub Actions Workflows

Files to Modify: 7 workflow files (ALL workflows that use Go)

1. .github/workflows/quality-checks.yml

Location: Line 18 (env section) Current Go Version: 1.25.6 ✅

Change:

env:
  GO_VERSION: '1.25.6'
  NODE_VERSION: '24.12.0'
  GOTOOLCHAIN: auto  # ← ADD THIS LINE

Justification: Allows setup-go action to download and use Go 1.25.6 even if system has older version.

2. .github/workflows/codeql.yml

Location: Line 15 (env section) Current Go Version: 1.25.6 ✅

Change:

env:
  GO_VERSION: '1.25.6'
  GOTOOLCHAIN: auto  # ← ADD THIS LINE

Justification: Ensures CodeQL analysis uses correct Go version for accurate results.

3. .github/workflows/benchmark.yml

Location: Line 21 (env section) Current Go Version: 1.25.6 ✅

Change:

env:
  GO_VERSION: '1.25.6'
  GOTOOLCHAIN: auto  # ← ADD THIS LINE

Justification: Benchmark tests compile and run Go code. Requires correct toolchain version for accurate performance measurements.

4. .github/workflows/codecov-upload.yml

Location: Line 17 (env section) Current Go Version: 1.25.6 ✅

Change:

env:
  GO_VERSION: '1.25.6'
  NODE_VERSION: '24.12.0'
  GOTOOLCHAIN: auto  # ← ADD THIS LINE

Justification: Runs backend tests with coverage collection. Must use correct Go version to ensure accurate coverage metrics.

5. .github/workflows/e2e-tests.yml

Location: Line 60 (env section) Current Go Version: 1.21 ⚠️ OUTDATED!

Change:

env:
  NODE_VERSION: '20'
  GO_VERSION: '1.25.6'  # ← UPDATE FROM 1.21
  GOTOOLCHAIN: auto     # ← ADD THIS LINE
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository_owner }}/charon

Justification: E2E tests build Docker images containing Go backend. The outdated 1.21 version causes build failures. This is a CRITICAL fix.

6. .github/workflows/nightly-build.yml

Location: Line 17 (existing env section) Current State: Has global env section with registry config, missing Go version variables

Change (ADD TO EXISTING):

env:
  GO_VERSION: '1.25.6'      # ← ADD THIS LINE
  NODE_VERSION: '24.12.0'   # ← ADD THIS LINE (consistent with other workflows)
  GOTOOLCHAIN: auto         # ← ADD THIS LINE
  GHCR_REGISTRY: ghcr.io    # ← KEEP EXISTING
  DOCKERHUB_REGISTRY: docker.io  # ← KEEP EXISTING
  IMAGE_NAME: wikid82/charon     # ← KEEP EXISTING

Justification: Nightly build workflow already has an env section with registry config. We need to ADD Go-related variables to it, not create a new section.

7. .github/workflows/release-goreleaser.yml

Location: Line 13 (env section) Current Go Version: 1.25.6 ✅

Change:

env:
  GO_VERSION: '1.25.6'
  NODE_VERSION: '24.12.0'
  GOTOOLCHAIN: auto  # ← ADD THIS LINE

Justification: Production releases must use exact Go version specified. Prevents release failures due to CI environment mismatches.

Verification Command

Before Implementation:

# Count workflows using setup-go
grep -l "setup-go" .github/workflows/*.yml | wc -l
# Expected: 7

After Implementation:

# Verify all Go workflows have GOTOOLCHAIN: auto
grep -l "GOTOOLCHAIN: auto" .github/workflows/*.yml | wc -l
# Expected: 7

# List workflows with GOTOOLCHAIN settings
grep -l "GOTOOLCHAIN: auto" .github/workflows/*.yml
# Should show all 7 workflow files

Phase 2: Update Makefile Comment (Optional Cleanup)

File: Makefile

Location: Line 46 (install-go comment)

Change:

# Install Go 1.25.6 system-wide and setup GOPATH/bin
install-go:
	@echo "Installing Go 1.25.6 and gopls (requires sudo)"
	sudo ./scripts/install-go-1.25.6.sh

Note: This is a comment-only change for consistency. Script may not exist or need updating.

Phase 3: Verification & Testing

Verification Steps

  1. Verify Workflow Syntax

    # Check YAML validity
yamllint .github/workflows/quality-checks.yml
yamllint .github/workflows/codeql.yml

  2. Test CI Build

    • Push changes to a test branch
    • Monitor GitHub Actions for successful builds
    • Verify Go 1.25.6 is used in build logs

  3. Verify Docker Builds

    # Ensure Docker builds still work
make docker-build-versioned

  4. Test Local Development

    # Ensure local development unaffected
cd backend && go version
cd backend && go build -o bin/api ./cmd/api

Success Criteria

  • ✅ ALL 7 Go workflows complete without Go version errors:
    • quality-checks.yml
    • codeql.yml
    • benchmark.yml
    • codecov-upload.yml
    • e2e-tests.yml (CRITICAL: version also updated to 1.25.6)
    • nightly-build.yml
    • release-goreleaser.yml
  • ✅ Backend builds successfully in CI
  • ✅ CodeQL analysis completes without errors
  • ✅ Docker image builds successfully
  • ✅ E2E tests pass with correct Go version
  • ✅ Nightly builds use consistent Go version
  • ✅ Release builds complete without toolchain errors
  • ✅ Local development environment unaffected
  • ✅ PR #550 can proceed

Risk Mitigation

Potential Issues

  1. Issue: setup-go action may not support GOTOOLCHAIN override

    • Mitigation: setup-go@v6 respects environment variables; tested in Go 1.20+
    • Fallback: Explicitly set GOTOOLCHAIN=auto in workflow steps

  2. Issue: Older Go version cached in CI

    • Mitigation: setup-go action's cache is version-specific; will download 1.25.6
    • Fallback: Manually clear cache or use cache: false temporarily

  3. Issue: Pre-commit still enforces GOTOOLCHAIN=local

    • Mitigation: This only affects local pre-commit hooks, not CI
    • Fallback: Skip pre-commit in CI or run with GOTOOLCHAIN=auto override

Best Practices for Go Version Management

Recommendations for Future

  1. Use GOTOOLCHAIN=auto by default in CI

    • Allows automatic upgrades to compatible Go versions
    • Prevents version mismatch errors
    • Aligns with Go team's recommendation

  2. Keep Go version consistent across all files

    • go.work, go.mod, Dockerfile, CI workflows should all use same major.minor version
    • Use Renovate to keep versions synchronized

  3. Pin exact Go version in security-critical builds

    • Use golang:1.25.6-trixie (exact version) for production Docker images
    • Use golang:1.25-trixie (latest patch) for development

  4. Document Go version requirements

    • Add to README.md: "Requires Go 1.25.6 or later"
    • Update CONTRIBUTING.md with setup instructions

  5. Monitor Go releases

    • Subscribe to Go release notes: https://go.dev/dl/
    • Plan upgrades within 1 month of stable release
    • Test in development branch before merging to main

Alternative: GOTOOLCHAIN=auto by Default (Future Enhancement)

Proposal: Set GOTOOLCHAIN=auto as repository default.

Method: Create .go-env file or export in shell profile.

Benefits:

  • Prevents version mismatch issues across environments
  • Aligns with Go's recommended default
  • Reduces CI configuration complexity

Drawbacks:

  • Requires all developers to update local environment
  • May cause unexpected upgrades in local development
  • Not standard practice (most projects don't set this)

Recommendation: ⏸️ DEFER - Implement Option A first, revisit if issues persist.

Timeline

Phase Duration Dependencies
Phase 1: Update Workflows (7 files) 25-30 min None
Phase 2: Update Makefile 5 min Phase 1 complete
Phase 3: Verification 30-45 min Phase 1+2 complete
Total ~1.5 hours

References

Decision Record

Decision: Implement Option A - Set GOTOOLCHAIN=auto in GitHub Actions workflows

Rationale:

  1. Comprehensive fix: Addresses all 7 workflows that use Go (not just 2)
  2. Fixes critical version mismatch: Updates e2e-tests.yml from Go 1.21 to 1.25.6
  3. Minimal invasive changes: Only 1-2 line additions per workflow file
  4. Immediate resolution: Unblocks PR #550 and future builds across entire CI/CD pipeline
  5. Future-proof: Prevents similar issues with future Go upgrades in all workflows
  6. Aligns with Go best practices: Official recommendation is GOTOOLCHAIN=auto
  7. No regression risk: Does not affect Docker builds or local development
  8. Standardizes build environment: Ensures consistency across quality checks, security scans, tests, and releases

Alternatives Considered:

  • ❌ Option B (Pre-commit override): Too complex, high maintenance burden
  • ❌ Option C (Downgrade Go): Security risk, blocks dependency updates

Impact:

  • ✅ Positive: Unblocks CI/CD pipeline immediately
  • ✅ Positive: Future Go version upgrades will be seamless
  • ⚠️ Neutral: Minimal impact on local development
  • ✅ Positive: Aligns with industry best practices

Review Schedule: Post-implementation verification within 24 hours

Next Steps

  1. Supervisor Review: Review and approve this specification
  2. Implementation: Apply changes to workflow files
  3. Testing: Push to test branch and verify CI success
  4. Deployment: Merge to main and unblock PR #550
  5. Documentation: Update README.md with Go version requirements
  6. Monitoring: Watch for any regressions in next 3 builds

Specification Complete - Ready for Implementation Estimated Time to Resolution: 1.5 hours (revised from 1 hour) Confidence Level: HIGH (98% - increased from 95% after comprehensive workflow analysis) Workflows Covered: 7 of 7 (100% of Go workflows identified and documented)

Reference in New Issue View Git Blame Copy Permalink