Charon/docs/runbooks/emergency-token-rotation.md

Emergency Token Rotation Runbook

Version: 1.0 Last Updated: January 26, 2026 Purpose: Secure procedure for rotating the emergency break glass token

---

When to Rotate

Rotate the emergency token in these situations:

• Scheduled rotation — Every 90 days (recommended)
• After use — Token was used during an incident
• Suspected compromise — Token may have been exposed in logs, screenshots, or shared insecurely
• Personnel changes — Team member with token access has left
• Security audit — Compliance requirement or security policy mandate

---

Prerequisites

• ✅ Access to Charon configuration (docker-compose.yml or secrets manager)
• ✅ Ability to restart Charon container
• ✅ Write access to secrets management system (if used)
• ✅ Documentation of where token is stored

---

Step-by-Step Rotation Procedure

Step 1: Generate New Token

Generate a cryptographically secure 64-character hex token:

# Using OpenSSL (recommended)
openssl rand -hex 32

# Example output
a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6a7b8c9d0e1f2

# Using /dev/urandom
head -c 32 /dev/urandom | xxd -p -c 64

# Using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

Requirements:

• Minimum 32 characters (produces 64-char hex)
• Use cryptographically secure random generator
• Never reuse old tokens

Step 2: Document Token Securely

Store the new token in your secrets management system:

HashiCorp Vault:

vault kv put secret/charon/emergency-token \
  token='NEW_TOKEN_HERE'

AWS Secrets Manager:

aws secretsmanager update-secret \
  --secret-id charon/emergency-token \
  --secret-string 'NEW_TOKEN_HERE'

Azure Key Vault:

az keyvault secret set \
  --vault-name charon-vault \
  --name emergency-token \
  --value 'NEW_TOKEN_HERE'

Password Manager:

• Store in "Charon Emergency Access" entry
• Add expiration reminder (90 days)
• Share with authorized personnel only

Step 3: Update Docker Compose Configuration

Option A: Environment Variable (Less Secure)

Edit docker-compose.yml:

services:
  charon:
    environment:
      - CHARON_EMERGENCY_TOKEN=NEW_TOKEN_HERE  # <-- Update this

Option B: Docker Secrets (More Secure)

services:
  charon:
    secrets:
      - charon_emergency_token
    environment:
      - CHARON_EMERGENCY_TOKEN_FILE=/run/secrets/charon_emergency_token

secrets:
  charon_emergency_token:
    external: true

Create Docker secret:

echo "NEW_TOKEN_HERE" | docker secret create charon_emergency_token -

Option C: Environment File (Recommended)

Create .env file (add to .gitignore):

# .env
CHARON_EMERGENCY_TOKEN=NEW_TOKEN_HERE

Update docker-compose.yml:

services:
  charon:
    env_file:
      - .env

Step 4: Restart Charon Container

# Using docker-compose
docker-compose down
docker-compose up -d

# Or restart existing container
docker-compose restart charon

# Verify container started successfully
docker logs charon --tail 20

Expected log output:

[INFO] Emergency token configured (64 characters)
[INFO] Emergency bypass middleware enabled
[INFO] Management CIDRs: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

Step 5: Verify New Token Works

Test the new token from an allowed IP:

# Test emergency reset endpoint
curl -X POST https://charon.example.com/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: NEW_TOKEN_HERE" \
  -H "Content-Type: application/json"

# Expected response
{
  "success": true,
  "message": "All security modules have been disabled",
  "disabled_modules": [...]
}

If testing in production: Re-enable security immediately:

# Navigate to Cerberus Dashboard
# Toggle security modules back ON

Step 6: Verify Old Token is Revoked

Test that the old token no longer works:

# Test with old token (should fail)
curl -X POST https://charon.example.com/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: OLD_TOKEN_HERE" \
  -H "Content-Type: application/json"

# Expected response (401 Unauthorized)
{
  "error": "Invalid emergency token",
  "code": 401
}

Step 7: Update Documentation

Update all locations where the token is documented:

• Password manager entry
• Secrets management system
• Runbooks (if token is referenced)
• Team wiki or internal docs
• Incident response procedures
• Backup/recovery documentation

Step 8: Notify Team

Inform authorized personnel:

Subject: [ACTION REQUIRED] Charon Emergency Token Rotated

The Charon emergency break glass token has been rotated as part of our regular security maintenance.

**Action Required:**
- Update your local password manager with the new token
- Retrieve new token from: [secrets management location]
- Old token is no longer valid as of: [timestamp]

**Next Rotation:** [90 days from now]

If you need access to the new token, contact: [security team contact]

---

Emergency Rotation (Compromise Suspected)

If the token has been compromised, follow this expedited procedure:

Immediate Actions (within 1 hour)

1. Rotate token immediately (Steps 1-5 above)
2. Review audit logs for unauthorized emergency access:

# Check for emergency token usage
docker logs charon | grep -i "emergency"

# Check audit logs
curl http://localhost:8080/api/v1/audit-logs | jq '.[] | select(.action | contains("emergency"))'

1. Alert security team if unauthorized access detected
2. Disable compromised accounts that may have used the token

Investigation (within 24 hours)

1. Determine exposure scope:

   • Was token in logs or screenshots?
   • Was token shared via insecure channel (email, Slack)?
   • Who had access to the token?
   • Was token committed to version control?

2. Check for signs of abuse:

   • Review recent configuration changes
   • Check for new proxy hosts or certificates
   • Verify ACL rules haven't been modified
   • Review CrowdSec decision history

3. Document incident:

   • Create incident report
   • Timeline of exposure
   • Impact assessment
   • Remediation actions taken

Remediation

1. Revoke access for compromised accounts
2. Rotate all related secrets (database passwords, API keys)
3. Implement additional controls:
   • Require 2FA for emergency access (future enhancement)
   • Implement emergency token session limits
   • Add approval workflow for emergency access
4. Update policies to prevent future exposure

---

Automation Script

Save this script as rotate-emergency-token.sh:

#!/usr/bin/env bash
set -euo pipefail

# Colors for output
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m' # No Color

echo -e "${GREEN}Charon Emergency Token Rotation Script${NC}"
echo "========================================"
echo ""

# Step 1: Generate new token
echo -e "${YELLOW}Step 1: Generating new token...${NC}"
NEW_TOKEN=$(openssl rand -hex 32)
echo "New token generated: ${NEW_TOKEN:0:16}...${NEW_TOKEN: -16}"
echo ""

# Step 2: Backup old configuration
echo -e "${YELLOW}Step 2: Backing up current configuration...${NC}"
BACKUP_FILE="docker-compose.yml.backup-$(date +%Y%m%d-%H%M%S)"
cp docker-compose.yml "$BACKUP_FILE"
echo "Backup saved to: $BACKUP_FILE"
echo ""

# Step 3: Update docker-compose.yml
echo -e "${YELLOW}Step 3: Updating docker-compose.yml...${NC}"
sed -i.bak "s/CHARON_EMERGENCY_TOKEN=.*/CHARON_EMERGENCY_TOKEN=${NEW_TOKEN}/" docker-compose.yml
echo "Configuration updated"
echo ""

# Step 4: Restart container
echo -e "${YELLOW}Step 4: Restarting Charon container...${NC}"
docker-compose restart charon
sleep 5
echo "Container restarted"
echo ""

# Step 5: Verify new token
echo -e "${YELLOW}Step 5: Verifying new token...${NC}"
RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: ${NEW_TOKEN}" \
  -H "Content-Type: application/json")

if echo "$RESPONSE" | grep -q '"success":true'; then
  echo -e "${GREEN}✓ New token verified successfully${NC}"
else
  echo -e "${RED}✗ Token verification failed${NC}"
  echo "Response: $RESPONSE"
  exit 1
fi
echo ""

# Step 6: Save token securely
echo -e "${YELLOW}Step 6: Token rotation complete${NC}"
echo ""
echo "========================================"
echo -e "${GREEN}NEXT STEPS:${NC}"
echo "1. Save new token to password manager:"
echo "   ${NEW_TOKEN}"
echo ""
echo "2. Update secrets manager (Vault, AWS, Azure)"
echo "3. Notify team of token rotation"
echo "4. Test old token is revoked"
echo "5. Schedule next rotation: $(date -d '+90 days' +%Y-%m-%d)"
echo "========================================"

Make executable:

chmod +x rotate-emergency-token.sh

Run:

./rotate-emergency-token.sh

---

Compliance Checklist

For organizations with compliance requirements:

• Token rotation documented in change log
• Rotation approved by security team
• Old token marked as revoked in secrets manager
• Access to new token limited to authorized personnel
• Token rotation logged in audit trail
• Backup configuration saved securely
• Team notification sent and acknowledged
• Next rotation scheduled (90 days)

---

Troubleshooting

Error: New Token Not Working After Rotation

Symptom: New token returns 401 Unauthorized.

Causes:

1. Token not saved correctly in configuration
2. Container not restarted after update
3. Token contains whitespace or line breaks
4. Environment variable not exported

Solution:

# Verify environment variable
docker exec charon env | grep CHARON_EMERGENCY_TOKEN

# Check logs for token loading
docker logs charon | grep -i "emergency token"

# Restart container
docker-compose restart charon

Error: Container Won't Start After Update

Symptom: Container exits immediately after restart.

Cause: Malformed docker-compose.yml or invalid token format.

Solution:

# Validate docker-compose.yml syntax
docker-compose config

# Restore backup
cp docker-compose.yml.backup docker-compose.yml

# Fix syntax error
vim docker-compose.yml

# Restart
docker-compose up -d

Error: Lost Access to Old Token

Symptom: Need to verify old token is revoked, but don't have it.

Solution:

# Check backup configuration
grep CHARON_EMERGENCY_TOKEN docker-compose.yml.backup-*

# Or check container environment (if not restarted)
docker exec charon env | grep CHARON_EMERGENCY_TOKEN

---

Security Best Practices

1. Never commit tokens to version control

   • Add to .gitignore: .env, docker-compose.override.yml
   • Use pre-commit hooks to scan for secrets
   • Use git-secrets or trufflehog

2. Use secrets management systems

   • HashiCorp Vault
   • AWS Secrets Manager
   • Azure Key Vault
   • Kubernetes Secrets (with encryption at rest)

3. Limit token access

   • Only senior engineers and ops team
   • Require 2FA for secrets manager access
   • Audit who accesses the token

4. Rotate regularly

   • Every 90 days (at minimum)
   • After any security incident
   • When team members leave

5. Monitor emergency token usage

   • Set up alerts for emergency access
   • Review audit logs weekly
   • Investigate any unexpected usage

6. Test recovery procedures

   • Quarterly disaster recovery drills
   • Verify backup token storage works
   • Ensure team knows how to retrieve token

---

Related Documentation

• Emergency Lockout Recovery Runbook
• Security Documentation
• Configuration Guide

---

Version History:

• v1.0 (2026-01-26): Initial release