Charon/docs/plans/archive/nightly_branch_implementation.md

Nightly Branch Implementation - Complete Specification

Date: 2026-01-13 Version: 1.0 Status: Planning Phase

---

Table of Contents

1. Executive Summary
2. Current State Analysis
3. Proposed Architecture
4. Implementation Plan (7 Phases)
5. Detailed Workflow Specifications
6. Configuration Files
7. Testing Strategy
8. Rollback Procedures
9. Monitoring & Alerting
10. Migration Checklist
11. Troubleshooting Guide
12. Appendices

---

Executive Summary

Objective

Add a nightly branch between development and main to provide a stabilization layer with automated builds and package creation.

Branch Hierarchy

feature/* → development → nightly → main (tagged releases)

Key Benefits

• Stability Layer: Code stabilizes in nightly before reaching main
• Daily Testing: Automated builds catch integration issues early
• Package Availability: Regular nightly releases for testing
• Reduced Main Breakage: Main branch stays stable for production releases

Timeline

Total Effort: ~10 hours Duration: 3-4 days with testing

---

Current State Analysis

Existing Workflows

1. propagate-changes.yml

Current Issues:

• Line 149: Incorrect third parameter 'nightly' in createPR call
• Lines 151-152: Commented out development → nightly propagation logic

Current Structure:

on:
  push:
    branches:
      - main
      - development
      - nightly  # Already present but not used

jobs:
  propagate:
    steps:
      # Issue on line 149:
      - createPR('main', 'development', 'nightly')  # Third param should not exist

      # Lines 151-152 (currently commented):
      # - name: Propagate development to nightly
      #   run: createPR('development', 'nightly')

2. docker-build.yml

Current Triggers:

on:
  push:
    branches:
      - main
      - development
      - 'feature/**'
      - 'beta-release/**'
  # nightly is MISSING

Tag Strategy:

• main → latest, v{version}
• development → dev, dev-{sha}
• feature/* → pr-{number}
• Missing: nightly → should produce nightly, nightly-{date}, nightly-{sha}

3. auto-versioning.yml

Current Behavior:

• Only triggers on main branch pushes
• Creates semantic version tags (v1.2.3)
• Creates GitHub releases

No Changes Needed: Nightly should NOT auto-version; only main gets version tags.

4. release-goreleaser.yml

Current Behavior:

• Triggers on v* tag pushes
• Builds cross-platform binaries
• Creates GitHub release with artifacts

No Changes Needed: Only tag-triggered; nightly builds handled separately.

5. supply-chain-verify.yml

Current Issues:

• Missing nightly branch in tag determination logic

Current Tag Logic:

- name: Determine tag
  run: |
    if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
      echo "TAG=latest" >> $GITHUB_ENV
    elif [[ "${{ github.ref }}" == "refs/heads/development" ]]; then
      echo "TAG=dev" >> $GITHUB_ENV
    # Missing nightly case
    fi

Configuration Files Status

✅ .gitignore

Status: Properly configured, no changes needed

• Ignores *.cover, *.sarif, *.txt test artifacts
• Ignores test-results/, playwright-report/
• Ignores Docker overrides

✅ .dockerignore

Status: Properly configured, no changes needed

• Excludes .git, node_modules, test files
• Includes frontend/dist in build context

✅ Dockerfile

Status: Supports VERSION build arg, no changes needed

ARG VERSION=dev
# ... multi-stage build with Caddy, CrowdSec, Go backend, Node frontend

⚠️ propagate-config.yml

Current: Defines sensitive paths that block auto-merge Recommendation: Review if nightly needs different sensitivity rules

---

Proposed Architecture

Branch Flow Diagram

┌──────────────┐
│  feature/*   │
└──────┬───────┘
       │ PR (manual review)
       ▼
┌──────────────┐
│ development  │
└──────┬───────┘
       │ Auto-merge (workflow)
       ▼
┌──────────────┐
│   nightly    │◄────┐ Daily scheduled build (02:00 UTC)
└──────┬───────┘     │
       │ Manual PR   │
       ▼             │
┌──────────────┐     │
│     main     │─────┘ Tagged releases
└──────────────┘

Automation Rules

Source	Target	Trigger	Automation Level	Review Required
feature/*	development	PR open	Manual	Yes
development	nightly	Push to dev	Automatic	No (unless sensitive files)
nightly	main	Manual	Manual	Yes (full review)
nightly	-	Schedule/Push	Build only	-

Package Creation Strategy

Docker Images (via nightly-build.yml)

Frequency: Daily at 02:00 UTC + on every nightly push Tags:

• nightly (rolling latest nightly)
• nightly-YYYY-MM-DD (date-stamped)
• nightly-{short-sha} (commit-specific)

Platforms: linux/amd64, linux/arm64

Security:

• Trivy vulnerability scan
• SBOM generation (CycloneDX format)
• Grype CVE scanning
• CVE-2025-68156 verification

Binary Releases (via nightly-build.yml)

Platforms:

• Linux: amd64, arm64, arm (with CGO via Zig)
• Windows: amd64, arm64
• macOS: amd64, arm64

Formats:

• Archives: tar.gz (Linux/macOS), zip (Windows)
• Packages: deb, rpm

Artifacts: Uploaded to GitHub Actions artifacts (not GitHub Releases)

---

Implementation Plan

Phase 1: Update Propagate Workflow ⚡ URGENT

Priority: P0 Effort: 30 minutes File: .github/workflows/propagate-changes.yml

Changes Required

Change 1: Fix Line 149

# BEFORE (line 149):
await createPR('main', 'development', 'nightly');

# AFTER:
await createPR('main', 'development');

Change 2: Enable Lines 151-152

# BEFORE (lines 151-152):
# - name: Propagate development to nightly
#   run: |
#     await createPR('development', 'nightly');

# AFTER:
- name: Propagate development to nightly
  run: |
    await createPR('development', 'nightly');

Testing

1. Create test branch from development
2. Push to development
3. Verify PR created from development to nightly
4. Verify sensitive file blocking still works

---

Phase 2: Create Nightly Build Workflow

Priority: P1 Effort: 2 hours File: .github/workflows/nightly-build.yml (NEW)

Complete Workflow Specification

name: Nightly Build & Package

on:
  push:
    branches:
      - nightly
  schedule:
    # Daily at 02:00 UTC
    - cron: '0 2 * * *'
  workflow_dispatch:

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push-nightly:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    outputs:
      version: ${{ steps.meta.outputs.version }}
      tags: ${{ steps.meta.outputs.tags }}
      digest: ${{ steps.build.outputs.digest }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=raw,value=nightly
            type=raw,value=nightly-{{date 'YYYY-MM-DD'}}
            type=sha,prefix=nightly-,format=short
          labels: |
            org.opencontainers.image.title=Charon Nightly
            org.opencontainers.image.description=Nightly build of Charon

      - name: Build and push Docker image
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            VERSION=nightly-${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true
          sbom: true

      - name: Generate SBOM
        uses: anchore/sbom-action@v0.21.1
        with:
          image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:nightly
          format: cyclonedx-json
          output-file: sbom-nightly.json

      - name: Upload SBOM artifact
        uses: actions/upload-artifact@v4
        with:
          name: sbom-nightly
          path: sbom-nightly.json
          retention-days: 30

  test-nightly-image:
    needs: build-and-push-nightly
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Pull nightly image
        run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:nightly

      - name: Run container smoke test
        run: |
          docker run --name charon-nightly -d \
            -p 8080:8080 \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:nightly

          # Wait for container to start
          sleep 10

          # Check container is running
          docker ps | grep charon-nightly

          # Basic health check
          curl -f http://localhost:8080/health || exit 1

          # Cleanup
          docker stop charon-nightly
          docker rm charon-nightly

  build-nightly-release:
    needs: test-nightly-image
    runs-on: ubuntu-latest
    permissions:
      contents: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: "1.26.1"

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Set up Zig (for cross-compilation)
        uses: goto-bus-stop/setup-zig@v2
        with:
          version: 0.11.0

      - name: Build frontend
        working-directory: ./frontend
        run: |
          npm ci
          npm run build

      - name: Run GoReleaser (snapshot mode)
        uses: goreleaser/goreleaser-action@v6
        with:
          distribution: goreleaser
          version: '~> v2'
          args: release --snapshot --skip=publish --clean
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload nightly binaries
        uses: actions/upload-artifact@v4
        with:
          name: nightly-binaries
          path: dist/*
          retention-days: 30

  verify-nightly-supply-chain:
    needs: build-and-push-nightly
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Download SBOM
        uses: actions/download-artifact@v4
        with:
          name: sbom-nightly

      - name: Scan with Grype
        uses: anchore/scan-action@v4
        with:
          sbom: sbom-nightly.json
          fail-build: false
          severity-cutoff: high

      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@0.33.1
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:nightly
          format: 'sarif'
          output: 'trivy-nightly.sarif'

      - name: Upload Trivy results
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-nightly.sarif'
          category: 'trivy-nightly'

      - name: Check for critical CVEs
        run: |
          if grep -q "CRITICAL" trivy-nightly.sarif; then
            echo "❌ Critical vulnerabilities found in nightly build"
            exit 1
          fi
          echo "✅ No critical vulnerabilities found"

---

Phase 3: Update Docker Build Workflow

Priority: P1 Effort: 30 minutes File: .github/workflows/docker-build.yml

Change 1: Add nightly to triggers

# BEFORE:
on:
  push:
    branches:
      - main
      - development
      - 'feature/**'
      - 'beta-release/**'

# AFTER:
on:
  push:
    branches:
      - main
      - development
      - nightly
      - 'feature/**'
      - 'beta-release/**'

Change 2: Update metadata action

# BEFORE:
- name: Extract metadata
  id: meta
  uses: docker/metadata-action@v5
  with:
    tags: |
      type=ref,event=branch
      type=ref,event=pr
      type=semver,pattern={{version}}

# AFTER:
- name: Extract metadata
  id: meta
  uses: docker/metadata-action@v5
  with:
    tags: |
      type=ref,event=branch
      type=ref,event=pr
      type=semver,pattern={{version}}
      type=raw,value=nightly,enable=${{ github.ref == 'refs/heads/nightly' }}
      type=raw,value=nightly-{{date 'YYYY-MM-DD'}},enable=${{ github.ref == 'refs/heads/nightly' }}

Change 3: Update test-image tag determination

# BEFORE:
- name: Determine test tag
  id: test-tag
  run: |
    if [[ "$GITHUB_REF" == "refs/heads/main" ]]; then
      echo "tag=latest" >> $GITHUB_OUTPUT
    elif [[ "$GITHUB_REF" == "refs/heads/development" ]]; then
      echo "tag=dev" >> $GITHUB_OUTPUT
    else
      echo "tag=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
    fi

# AFTER:
- name: Determine test tag
  id: test-tag
  run: |
    if [[ "$GITHUB_REF" == "refs/heads/main" ]]; then
      echo "tag=latest" >> $GITHUB_OUTPUT
    elif [[ "$GITHUB_REF" == "refs/heads/development" ]]; then
      echo "tag=dev" >> $GITHUB_OUTPUT
    elif [[ "$GITHUB_REF" == "refs/heads/nightly" ]]; then
      echo "tag=nightly" >> $GITHUB_OUTPUT
    else
      echo "tag=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
    fi

---

Phase 4: Update Supply Chain Verification

Priority: P2 Effort: 30 minutes File: .github/workflows/supply-chain-verify.yml

Change: Add nightly to tag determination

# BEFORE:
- name: Determine tag to verify
  id: tag
  run: |
    if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
      echo "tag=latest" >> $GITHUB_ENV
    elif [[ "${{ github.ref }}" == "refs/heads/development" ]]; then
      echo "tag=dev" >> $GITHUB_ENV
    elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
      echo "tag=pr-${{ github.event.pull_request.number }}" >> $GITHUB_ENV
    fi

# AFTER:
- name: Determine tag to verify
  id: tag
  run: |
    if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
      echo "tag=latest" >> $GITHUB_ENV
    elif [[ "${{ github.ref }}" == "refs/heads/development" ]]; then
      echo "tag=dev" >> $GITHUB_ENV
    elif [[ "${{ github.ref }}" == "refs/heads/nightly" ]]; then
      echo "tag=nightly" >> $GITHUB_ENV
    elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
      echo "tag=pr-${{ github.event.pull_request.number }}" >> $GITHUB_ENV
    fi

---

Phase 5: Configuration File Updates

Priority: P3 Effort: 1 hour

Optional: Create codecov.yml

File: codecov.yml (NEW) Purpose: Configure Codecov behavior for nightly branch

coverage:
  status:
    project:
      default:
        target: 85%
        threshold: 2%
        branches:
          - main
          - development
          - nightly
    patch:
      default:
        target: 85%
        branches:
          - main
          - development
          - nightly

ignore:
  - "**/*_test.go"
  - "tools/**"
  - "scripts/**"

comment:
  behavior: default
  require_changes: false
  branches:
    - nightly

Optional: Update propagate-config.yml

File: .github/propagate-config.yml Current: Lists sensitive paths that block auto-propagation

Review Question: Should nightly have same sensitivity rules as main?

Recommendation: Keep same rules; nightly should be as cautious as main.

---

Phase 6: Branch Protection Configuration

Priority: P1 Effort: 30 minutes

Step 1: Create Nightly Branch

# From development branch
git checkout development
git pull origin development
git checkout -b nightly
git push -u origin nightly

Step 2: Configure Branch Protection Rules

Via GitHub UI or API:

{
  "protection": {
    "required_status_checks": {
      "strict": true,
      "contexts": [
        "build-and-push",
        "test-image",
        "playwright-e2e-tests",
        "verify-supply-chain"
      ]
    },
    "enforce_admins": false,
    "required_pull_request_reviews": null,
    "restrictions": null,
    "allow_force_pushes": true,
    "allow_deletions": false,
    "required_linear_history": false
  }
}

Key Settings:

• ✅ Require status checks to pass
• ✅ Allow force pushes (for auto-merge workflow)
• ❌ No required reviewers (auto-merge)
• ❌ No restrictions on who can push

Alternative: GitHub CLI Script

#!/bin/bash
# scripts/setup-nightly-branch-protection.sh

REPO="owner/charon"
BRANCH="nightly"

gh api -X PUT "/repos/$REPO/branches/$BRANCH/protection" \
  --input - <<EOF
{
  "required_status_checks": {
    "strict": true,
    "contexts": [
      "build-and-push",
      "test-image",
      "playwright-e2e-tests",
      "verify-supply-chain"
    ]
  },
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null,
  "allow_force_pushes": true,
  "allow_deletions": false
}
EOF

---

Phase 7: Documentation Updates

Priority: P3 Effort: 1 hour

File 1: README.md

Add Section:

## Branch Strategy

Charon uses a multi-tier branching strategy:

- **main**: Stable production releases (tagged)
- **nightly**: Daily automated builds for testing
- **development**: Integration branch for features
- **feature/***: Individual feature branches

### Getting Nightly Builds

Docker images:
```bash
docker pull ghcr.io/owner/charon:nightly
docker pull ghcr.io/owner/charon:nightly-2026-01-13

Binary downloads available in GitHub Actions artifacts.

Contributing

1. Fork the repository
2. Create feature branch from development
3. Submit PR to development
4. Code auto-merges to nightly after review
5. Manual PR from nightly to main for releases

#### File 2: VERSION.md
**Add Section:**

```markdown
## Nightly Builds

Nightly builds are created daily at 02:00 UTC from the `nightly` branch.

**Version Format:** `nightly-{date}` or `nightly-{sha}`

**Examples:**
- `nightly-2026-01-13`
- `nightly-a1b2c3d`

**Stability:** Nightly builds are more stable than `dev` but less stable than tagged releases.

**Use Cases:**
- Early testing of upcoming features
- Integration testing with other systems
- Bug reproduction before release

**Not Recommended For:**
- Production deployments
- Long-term support
- Critical infrastructure

File 3: CONTRIBUTING.md

Update Workflow Section:

## Development Workflow

1. **Create Feature Branch**
   ```bash
   git checkout development
   git pull origin development
   git checkout -b feature/your-feature-name

1. Develop and Test

   • Write code with tests
   • Run make test locally
   • Commit with conventional commit messages

2. Submit Pull Request

   • Target: development branch
   • Fill PR template completely
   • Request reviews

3. Automated Propagation

   • After merge to development, code auto-merges to nightly
   • Nightly builds run automatically
   • Monitor for integration issues

4. Release to Main

   • Manual PR from nightly to main
   • Full review and approval required
   • Tagged release created automatically

---

## Testing Strategy

### Pre-Deployment Testing

#### 1. Propagate Workflow Testing
```bash
# Test development → nightly propagation
1. Create test branch: `test/nightly-propagation`
2. Push to development
3. Verify PR created automatically
4. Check PR title/body format
5. Verify sensitive file blocking
6. Merge PR manually
7. Verify nightly updated correctly

2. Nightly Build Testing

# Test nightly build workflow
1. Trigger workflow manually via UI
2. Monitor all 4 jobs completion
3. Verify Docker image pushed with all 3 tags
4. Download and test binary artifacts
5. Review SBOM and vulnerability reports
6. Test container smoke test passes

3. Integration Testing

# Test full integration flow
1. Make change in feature branch
2. PR to development
3. Merge to development
4. Verify auto-PR to nightly
5. Verify nightly build triggered
6. Check all packages created
7. Verify supply chain verification

Post-Deployment Monitoring

Metrics to Track

• Propagation Success Rate: Should be >98%
• Build Success Rate: Should be >95%
• Build Duration: Should be <25 minutes
• Image Size: Monitor for bloat
• Vulnerability Count: Should trend downward
• Auto-merge Failures: Should be <2% (sensitive files only)

Alerting Thresholds

• ❌ Critical: Build failure 3 times in a row
• ⚠️ Warning: Build duration >30 minutes
• ⚠️ Warning: Critical CVE found
• ⚠️ Warning: Auto-merge failure rate >5%

---

Rollback Procedures

Scenario 1: Nightly Build Workflow Broken

# Immediate action
1. Disable nightly-build.yml workflow in GitHub UI
2. Investigate logs and identify issue
3. Fix issue in feature branch
4. Test fix in feature branch
5. Merge to development
6. Re-enable workflow

# If urgent
- Use previous nightly image tag
- Document known issues in README

Scenario 2: Auto-Merge Creating Bad PRs

# Immediate action
1. Close problematic PR
2. Disable propagate-changes.yml workflow
3. Manually sync nightly with development:
   git checkout nightly
   git reset --hard development
   git push --force origin nightly
4. Fix propagate workflow
5. Re-enable workflow

Scenario 3: Nightly Branch Corrupted

# Recovery steps
1. Backup current nightly:
   git checkout nightly
   git branch nightly-backup-$(date +%Y%m%d)
   git push origin nightly-backup-$(date +%Y%m%d)

2. Reset from development:
   git reset --hard origin/development
   git push --force origin nightly

3. Investigate corruption cause
4. Update branch protection if needed

Scenario 4: Need to Revert Entire Feature

# If feature merged to nightly but has critical bug
1. Identify commit range to revert
2. Create revert PR:
   git checkout -b revert/feature-name
   git revert <commit-range>
   git push origin revert/feature-name
3. PR to nightly (bypasses development)
4. After verification, backport revert to development

---

Monitoring & Alerting

GitHub Actions Monitoring

Workflow Run Status

# Example monitoring query (GitHub API)
GET /repos/:owner/:repo/actions/workflows/nightly-build.yml/runs
?status=failure
&created=>2026-01-01

Key Metrics

• Total runs per day: Should be ≥1 (scheduled)
• Success rate: Should be ≥95%
• Average duration: Baseline ~20 minutes
• Artifact upload rate: Should be 100%

External Monitoring

Docker Image Availability

# Cron job to verify nightly image pullable
#!/bin/bash
docker pull ghcr.io/owner/charon:nightly >/dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "❌ Nightly image pull failed" | notify-alerting-system
fi

Vulnerability Tracking

# Daily Grype scan comparison
grype ghcr.io/owner/charon:nightly -o json > scan-$(date +%Y%m%d).json
python scripts/compare-vuln-scans.py scan-yesterday.json scan-today.json

Alerting Channels

1. GitHub Actions Email: Built-in workflow failure notifications
2. Slack Integration: Webhook for build status
3. PagerDuty: Critical failures only (3+ consecutive)

---

Migration Checklist

Pre-Migration (Day 0)

• Review all workflow files locally
• Verify no syntax errors in new nightly-build.yml
• Test propagate-changes.yml fixes in feature branch
• Create implementation branch: feature/nightly-branch-automation
• Document rollback procedures
• Set up monitoring baseline

Phase 1: Propagate Workflow (Day 1 Morning)

• Update .github/workflows/propagate-changes.yml (2 lines)
• Commit and push to feature branch
• Create PR to development
• Test propagation in PR checks
• Merge to development
• Monitor for auto-PR creation

Phase 2: Nightly Build Workflow (Day 1 Afternoon)

• Create .github/workflows/nightly-build.yml
• Commit to same feature branch
• Test workflow syntax with act or GitHub Actions locally
• Push to feature branch
• Merge PR to development

Phase 3-4: Docker & Supply Chain (Day 2 Morning)

• Update .github/workflows/docker-build.yml (3 changes)
• Update .github/workflows/supply-chain-verify.yml (1 change)
• Test in feature branch
• Merge to development

Phase 5: Configuration (Day 2 Afternoon)

• Create codecov.yml (if desired)
• Review .github/propagate-config.yml
• Test configuration changes

Phase 6: Branch Creation (Day 3 Morning)

• Create nightly branch from development
• Push to GitHub
• Configure branch protection rules
• Test force push capability
• Verify status check requirements

Phase 7: Documentation (Day 3 Afternoon)

• Update README.md
• Update VERSION.md
• Update CONTRIBUTING.md
• Create PR with documentation changes
• Merge to development

Post-Migration Testing (Day 3-4)

• Trigger manual nightly build
• Verify all 4 jobs complete successfully
• Check Docker images published with all tags
• Download and test binary artifacts
• Verify SBOM generated correctly
• Check vulnerability scan reports
• Test auto-merge from development
• Verify scheduled build runs at 02:00 UTC

Monitoring Period (Day 5-14)

• Monitor daily build success rate
• Track auto-merge success rate
• Review vulnerability trends
• Check artifact retention working
• Verify alerting triggers correctly
• Document any issues encountered

Sign-Off (Day 15)

• Review all metrics meet targets
• Document lessons learned
• Update troubleshooting guide
• Archive implementation artifacts
• Celebrate success! 🎉

---

Troubleshooting Guide

Issue 1: Auto-Merge PR Not Created

Symptoms:

• Push to development completes
• No PR created from development to nightly

Diagnosis:

# Check workflow run logs
gh run list --workflow=propagate-changes.yml --limit=1
gh run view <run-id> --log

# Check if workflow triggered
gh api /repos/:owner/:repo/actions/workflows/propagate-changes.yml/runs

Common Causes:

1. Workflow file syntax error
2. GitHub token permissions insufficient
3. Sensitive file detected (expected behavior)
4. Branch protection blocking push

Solutions:

1. Validate workflow YAML syntax
2. Check GITHUB_TOKEN permissions in workflow
3. Review .github/propagate-config.yml for false positives
4. Verify nightly branch allows force pushes

Issue 2: Nightly Build Failing

Symptoms:

• Workflow triggered but fails
• Red X on GitHub Actions

Diagnosis:

# View failure details
gh run list --workflow=nightly-build.yml --status=failure --limit=5
gh run view <run-id> --log-failed

# Check specific job
gh run view <run-id> --job=<job-id>

Common Causes:

1. Docker build timeout (>25 minutes)
2. Test failure in smoke test
3. Dependency download failure
4. Registry authentication failure

Solutions:

1. Check Docker build cache; may need --no-cache
2. Review test logs; may be transient network issue
3. Verify actions/checkout fetched correctly
4. Re-run workflow; transient failures common

Issue 3: Nightly Image Not Pullable

Symptoms:

• Build succeeds but docker pull fails
• Image not in ghcr.io registry

Diagnosis:

# Check if image pushed
gh api /user/packages/container/charon/versions

# Try pulling with debug
docker pull ghcr.io/owner/charon:nightly --debug

# Check registry permissions
gh api /user/packages/container/charon

Common Causes:

1. Package visibility set to private
2. GitHub token expired during push
3. Multi-arch manifest not created
4. Tag overwrite issue

Solutions:

1. Make package public in GitHub UI
2. Check docker/login-action in workflow
3. Verify docker/build-push-action platforms
4. Check for conflicting tag push

Issue 4: Binary Build Fails

Symptoms:

• Docker build succeeds
• build-nightly-release job fails

Diagnosis:

# Check GoReleaser logs
gh run view <run-id> --job=build-nightly-release --log

# Test locally
goreleaser release --snapshot --skip=publish --clean

Common Causes:

1. Frontend build missing
2. Zig not installed correctly
3. CGO dependency missing
4. .goreleaser.yaml syntax error

Solutions:

1. Verify frontend build step runs before GoReleaser
2. Check goto-bus-stop/setup-zig action version
3. Install build-essential in workflow
4. Validate .goreleaser.yaml with goreleaser check

---

Appendices

Appendix A: Workflow Trigger Matrix

Workflow	main	development	nightly	feature/*	schedule	manual
propagate-changes.yml	✅	✅	✅	❌	❌	❌
docker-build.yml	✅	✅	✅	✅	❌	❌
nightly-build.yml	❌	❌	✅	❌	✅ (daily)	✅
auto-versioning.yml	✅	❌	❌	❌	❌	❌
release-goreleaser.yml	❌	❌	❌	❌	❌	tags only
supply-chain-verify.yml	✅	✅	✅	❌	❌	❌

Appendix B: Tag Naming Convention

Branch	Docker Tags	Binary Version	SBOM Name
main	latest, v1.2.3	v1.2.3	sbom-v1.2.3.json
development	dev, dev-a1b2c3d	dev-a1b2c3d	sbom-dev.json
nightly	nightly, nightly-2026-01-13, nightly-a1b2c3d	nightly-a1b2c3d	sbom-nightly.json
feature/*	pr-123	N/A	sbom-pr-123.json

Appendix C: Resource Requirements

Compute

• Docker Build: 4 vCPU, 8 GB RAM, ~20 minutes
• GoReleaser: 2 vCPU, 4 GB RAM, ~10 minutes
• Tests: 2 vCPU, 4 GB RAM, ~5 minutes
• Total per nightly: ~35 minutes of runner time

Storage

• Docker Images: ~500 MB per arch × 2 = 1 GB per build
• Binary Artifacts: ~200 MB per platform × 6 = 1.2 GB per build
• SBOMs: ~5 MB per build
• Total per day: ~2.2 GB (retained 30 days = ~66 GB)

Actions Minutes

• Free Tier: 2,000 minutes/month
• Nightly Usage: ~35 minutes × 30 days = 1,050 minutes/month
• Buffer: ~950 minutes for other workflows
• Recommendation: Monitor usage; may need paid tier

Appendix D: Security Considerations

Secrets Management

• GITHUB_TOKEN: Scoped to repository, auto-generated
• Registry Access: Uses GITHUB_TOKEN, no extra secrets
• Signing Keys: Store in repository secrets if using Cosign

Vulnerability Response

1. Critical CVE Found:

   • Automatic scan failure
   • Block deployment
   • Create security issue
   • Patch within 24 hours

2. High CVE Found:

   • Log warning
   • Allow deployment
   • Create tracking issue
   • Patch within 7 days

3. Medium/Low CVE:

   • Log for awareness
   • Address in next cycle

Supply Chain Integrity

• SBOM: Always generated, retained 30 days
• Provenance: Docker buildx includes provenance
• Signatures: Optional Cosign signing
• Attestation: GitHub Actions attestation built-in

Appendix E: Future Enhancements

1. Multi-Architecture Binary Testing

   • QEMU-based testing for ARM64
   • Windows container testing
   • macOS binary testing via hosted runners

2. Performance Benchmarking

   • Automated performance regression tests
   • Historical tracking of build times
   • Resource usage profiling

3. Enhanced Notifications

   • Slack integration for build status
   • Email digest of nightly builds
   • RSS feed for release notes

4. Deployment Automation

   • Automated staging deployment from nightly
   • Blue/green deployment testing
   • Automated rollback on failure

5. Advanced Analytics

   • Build success trends
   • Code coverage over time
   • Dependency update frequency
   • Security posture tracking

---

Conclusion

This specification provides a complete implementation plan for adding a nightly branch with automated builds and package creation. The 7-phase approach ensures incremental rollout with testing at each step.

Key Success Factors:

1. Fix propagate workflow issues first (Phase 1)
2. Test thoroughly in feature branch before merging
3. Monitor metrics closely for first 2 weeks
4. Document any deviations or issues
5. Iterate based on real-world usage

Contact for Questions:

• Implementation Team: [link]
• DevOps Channel: [link]
• Documentation: This file

Last Updated: 2026-01-13 Next Review: After Phase 7 completion