Charon/docs/plans/archive/CI_SECRET_DIAGNOSIS.md

CI Secret Diagnosis - CHARON_ENCRYPTION_KEY_TEST

Date: 2026-02-16 Status: ROOT CAUSE IDENTIFIED ✅ Severity: HIGH (Blocking CI)

---

🔴 CRITICAL FINDING: Wrong Key Generation Command Used

Root Cause Analysis

The CI logs reveal two different error patterns:

Pattern 1: Key Not Set (Older/Some Tests)

Warning: RotationService initialization failed, using basic encryption: CHARON_ENCRYPTION_KEY is required

Pattern 2: Invalid Key Length (Most Tests) ⬅️ THE ACTUAL PROBLEM

Warning: RotationService initialization failed, using basic encryption:
failed to load current encryption key: invalid key length: expected 32 bytes, got 48 bytes

The Smoking Gun 🔍

Evidence from terminal history:

Terminal: root@srv599055: /projects/Charon
Last Command: openssl rand -hex 32      # ❌ WRONG!

What happened:

• Command openssl rand -hex 32 generates 64 hexadecimal characters (32 bytes as hex)
• When base64-decoded, 64 characters = 48 bytes of decoded data
• Application expects exactly 32 bytes after base64 decoding

Math:

• openssl rand -hex 32 → 64 hex chars → 48 bytes when base64-decoded ❌
• openssl rand -base64 32 → 44 base64 chars → 32 bytes when decoded ✅

Code Validation

From backend/internal/crypto/encryption.go:32-39:

// NewEncryptionService creates a new encryption service with the provided base64-encoded key.
// The key must be exactly 32 bytes (256 bits) when decoded.
func NewEncryptionService(keyBase64 string) (*EncryptionService, error) {
	key, err := base64.StdEncoding.DecodeString(keyBase64)
	if err != nil {
		return nil, fmt.Errorf("invalid base64 key: %w", err)
	}

	if len(key) != 32 {
		return nil, fmt.Errorf("invalid key length: expected 32 bytes, got %d bytes", len(key))
	}

Expectation: Base64-encoded string that decodes to exactly 32 bytes (AES-256 key)

---

✅ IMMEDIATE FIX

Step 1: Generate Correct Secret

Run this command locally to generate a valid key:

openssl rand -base64 32

Example output (44 characters):

YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=

Step 2: Update GitHub Secret

1. Navigate to: https://github.com/{OWNER}/{REPO}/settings/secrets/actions
2. Find: CHARON_ENCRYPTION_KEY_TEST
3. Click: "Update"
4. Paste: The output from openssl rand -base64 32
5. Save: Click "Update secret"

Step 3: Verify Secret Format

Before saving, verify the secret:

• ✅ Should be ~44 characters long (base64 encoded)
• ✅ Should end with = or == (base64 padding)
• ❌ Should NOT be 64 characters (that's hex, not base64)
• ❌ Should NOT contain only 0-9a-f characters (that's hex)

Step 4: Re-run Failed Workflows

After updating the secret:

1. Go to the failed PR check
2. Click "Re-run jobs"
3. Monitor for the error message:
   • ❌ expected 32 bytes, got 48 bytes = Still wrong
   • ✅ No warnings = Fixed!

---

📋 Verification Checklist

After updating the secret, the CI logs should show:

• No "CHARON_ENCRYPTION_KEY is required" errors
• No "invalid key length: expected 32 bytes, got 48 bytes" errors
• Tests pass without RotationService warnings
• Backend tests complete successfully
• Codecov upload succeeds

---

🔄 WHY This Happened

User confusion: OpenSSL has two similar commands:

• openssl rand -base64 N → Generates N bytes, outputs as base64 (CORRECT)
• openssl rand -hex N → Generates N bytes, outputs as hex (WRONG for this use case)

The hex output looks "more random" with 0-9a-f characters, which may have seemed more secure, but it's the wrong encoding.

---

🛡️ Prevention

Documentation Updates Needed

1. Add to .env.example:

# Generate with: openssl rand -base64 32
# Must be base64-encoded 256-bit (32-byte) key
# DO NOT use -hex, must use -base64!
CHARON_ENCRYPTION_KEY=

2. Add to docs/development/secrets-management.md:

## Encryption Key Generation

**CRITICAL:** Always use `-base64`, never `-hex`:

✅ **CORRECT:**
```bash
openssl rand -base64 32

❌ WRONG:

openssl rand -hex 32  # This will cause "expected 32 bytes, got 48 bytes" error

3. **Add validation script:** `scripts/validate-secrets.sh`
```bash
#!/bin/bash
# Validates CHARON_ENCRYPTION_KEY format

KEY="${CHARON_ENCRYPTION_KEY:-}"
if [ -z "$KEY" ]; then
    echo "❌ CHARON_ENCRYPTION_KEY not set"
    exit 1
fi

# Decode and check length
DECODED=$(echo "$KEY" | base64 -d 2>/dev/null | wc -c)
if [ "$DECODED" -ne 32 ]; then
    echo "❌ Invalid key length: $DECODED bytes (expected 32)"
    echo "   Generate correct key with: openssl rand -base64 32"
    exit 1
fi

echo "✅ CHARON_ENCRYPTION_KEY is valid (32 bytes)"

---

📊 Impact Analysis

Tests Affected: ALL backend tests that initialize services with encryption Workflows Affected:

• quality-checks.yml (Backend Quality)
• codecov-upload.yml (Backend Codecov)
• Any workflow calling scripts/go-test-coverage.sh

False Positives: Some older logs show "CHARON_ENCRYPTION_KEY is required" which indicates the env var wasn't set at all. This may be from before the workflow changes were merged or from different test contexts.

---

⏭️ Next Steps

1. ✅ IMMEDIATE: Regenerate secret with correct command
2. ✅ IMMEDIATE: Update GitHub secret CHARON_ENCRYPTION_KEY_TEST
3. ✅ IMMEDIATE: Re-run failed CI workflows
4. 🔄 FOLLOW-UP: Add validation script to pre-commit hooks
5. 🔄 FOLLOW-UP: Update documentation with clear instructions
6. 🔄 FOLLOW-UP: Add CI check to validate secret format on workflow start

---

🎯 Expected Outcome

After fix, CI logs should show:

✅ Backend tests: All tests passed
✅ No RotationService warnings
✅ Codecov upload: Success
✅ Quality checks: Passed

---

📞 User Action Required

Please execute these commands now:

# 1. Generate correct key
NEW_KEY=$(openssl rand -base64 32)

# 2. Verify it's correct format (should output "32")
echo "$NEW_KEY" | base64 -d | wc -c

# 3. Output the key to copy (will be ~44 chars)
echo "$NEW_KEY"

# 4. Go to GitHub → Settings → Secrets → Actions
# 5. Update CHARON_ENCRYPTION_KEY_TEST with the output from step 3
# 6. Re-run the failed workflow

---

🔐 Security Note

The old (wrong) secret should be considered compromised and should not be reused. Always generate a fresh secret when correcting this issue.