Charon/docs/guides/local-key-management.md

Local Key Management for Cosign Signing

Overview

This guide provides comprehensive procedures for managing Cosign signing keys in local development environments. It covers key generation, secure storage, rotation, and air-gapped signing workflows.

Audience: Developers, DevOps engineers, Security team Last Updated: 2026-01-10

Table of Contents

1. Key Generation
2. Secure Storage
3. Key Usage
4. Key Rotation
5. Backup and Recovery
6. Air-Gapped Signing
7. Troubleshooting

---

Key Generation

Prerequisites

• Cosign 2.4.0 or higher installed
• Strong password (20+ characters, mixed case, numbers, special characters)
• Secure environment (trusted machine, no malware)

Generate Key Pair

# Navigate to secure directory
cd ~/.cosign

# Generate key pair interactively
cosign generate-key-pair

# You will be prompted for a password
# Enter a strong password (minimum 20 characters recommended)

# This creates two files:
# - cosign.key (PRIVATE KEY - keep secure!)
# - cosign.pub (public key - share freely)

Non-Interactive Generation (for automation)

# Generate with password from environment
export COSIGN_PASSWORD="your-strong-password"
cosign generate-key-pair --output-key-prefix=cosign-dev

# Cleanup environment variable
unset COSIGN_PASSWORD

Key Naming Convention

Use descriptive prefixes for different environments:

cosign-dev.key      # Development environment
cosign-staging.key  # Staging environment
cosign-prod.key     # Production environment (use HSM if possible)

⚠️ WARNING: Never use the same key for multiple environments!

---

Secure Storage

File System Permissions

# Set restrictive permissions on private key
chmod 600 ~/.cosign/cosign.key

# Verify permissions
ls -l ~/.cosign/cosign.key
# Should show: -rw------- (only owner can read/write)

Password Manager Integration

Store private keys in a password manager:

1. 1Password, Bitwarden, or LastPass:

   • Create a secure note
   • Add the private key content
   • Add the password as a separate field
   • Tag as "cosign-key"

2. Retrieve when needed:

   # Example with op (1Password CLI)
   op read "op://Private/cosign-dev-key/private key" > /tmp/cosign.key
   chmod 600 /tmp/cosign.key
   
   # Use the key
   COSIGN_PRIVATE_KEY="$(cat /tmp/cosign.key)" \
   COSIGN_PASSWORD="$(op read 'op://Private/cosign-dev-key/password')" \
   cosign sign --key /tmp/cosign.key charon:local
   
   # Cleanup
   shred -u /tmp/cosign.key
   

Hardware Security Module (HSM)

For production keys, use an HSM or YubiKey:

# Generate key on YubiKey
cosign generate-key-pair --key-slot 9a

# Sign with YubiKey
cosign sign --key yubikey://slot-id charon:latest

Environment Variables (Development Only)

For development convenience:

# Add to ~/.bashrc or ~/.zshrc (NEVER commit this file!)
export COSIGN_PRIVATE_KEY="$(cat ~/.cosign/cosign-dev.key)"
export COSIGN_PASSWORD="your-dev-password"

# Source the file
source ~/.bashrc

⚠️ WARNING: Only use environment variables in trusted development environments!

---

Key Usage

Sign Docker Image

# Export private key and password
export COSIGN_PRIVATE_KEY="$(cat ~/.cosign/cosign-dev.key)"
export COSIGN_PASSWORD="your-password"

# Sign the image
cosign sign --yes --key <(echo "${COSIGN_PRIVATE_KEY}") charon:local

# Or use the Charon skill
.github/skills/scripts/skill-runner.sh security-sign-cosign docker charon:local

# Cleanup
unset COSIGN_PRIVATE_KEY
unset COSIGN_PASSWORD

Sign Release Artifacts

# Sign a binary
cosign sign-blob --yes \
  --key ~/.cosign/cosign-prod.key \
  --output-signature ./dist/charon-linux-amd64.sig \
  ./dist/charon-linux-amd64

# Verify signature
cosign verify-blob ./dist/charon-linux-amd64 \
  --signature ./dist/charon-linux-amd64.sig \
  --key ~/.cosign/cosign-prod.pub

Batch Signing

# Sign all artifacts in a directory
for artifact in ./dist/charon-*; do
  if [[ -f "$artifact" && ! "$artifact" == *.sig ]]; then
    echo "Signing: $(basename $artifact)"
    cosign sign-blob --yes \
      --key ~/.cosign/cosign-prod.key \
      --output-signature "${artifact}.sig" \
      "$artifact"
  fi
done

---

Key Rotation

When to Rotate

• Every 90 days (recommended)
• After any suspected compromise
• When team members with key access leave
• After security incidents
• Before major releases

Rotation Procedure

1. Generate new key pair:

   cd ~/.cosign
   cosign generate-key-pair --output-key-prefix=cosign-prod-v2
   

2. Test new key:

   # Sign test artifact
   cosign sign-blob --yes \
     --key cosign-prod-v2.key \
     --output-signature test.sig \
     test-file
   
   # Verify
   cosign verify-blob test-file \
     --signature test.sig \
     --key cosign-prod-v2.pub
   
   # Cleanup test files
   rm test-file test.sig
   

3. Update documentation:

   • Update README with new public key
   • Update CI/CD secrets (if key-based signing)
   • Notify team members

4. Transition period:

   • Sign new artifacts with new key
   • Keep old key available for verification
   • Document transition date

5. Retire old key:

   • After 30-90 days (all old artifacts verified)
   • Archive old key securely (for historical verification)
   • Delete from active use

6. Archive old key:

   mkdir -p ~/.cosign/archive/$(date +%Y-%m)
   mv cosign-prod.key ~/.cosign/archive/$(date +%Y-%m)/
   chmod 400 ~/.cosign/archive/$(date +%Y-%m)/cosign-prod.key
   

---

Backup and Recovery

Backup Procedure

# Create encrypted backup
cd ~/.cosign
tar czf cosign-keys-backup.tar.gz cosign*.key cosign*.pub

# Encrypt with GPG
gpg --symmetric --cipher-algo AES256 cosign-keys-backup.tar.gz

# This creates: cosign-keys-backup.tar.gz.gpg

# Remove unencrypted backup
shred -u cosign-keys-backup.tar.gz

# Store encrypted backup in:
# - Password manager
# - Encrypted USB drive (stored in safe)
# - Encrypted cloud storage (e.g., Tresorit, ProtonDrive)

Recovery Procedure

# Decrypt backup
gpg --decrypt cosign-keys-backup.tar.gz.gpg > cosign-keys-backup.tar.gz

# Extract keys
tar xzf cosign-keys-backup.tar.gz

# Set permissions
chmod 600 cosign*.key
chmod 644 cosign*.pub

# Verify keys work
cosign sign-blob --yes \
  --key cosign-dev.key \
  --output-signature test.sig \
  <(echo "test")

# Cleanup
rm cosign-keys-backup.tar.gz
shred -u test.sig

Disaster Recovery

If private key is lost:

1. Generate new key pair (see Key Generation)
2. Revoke old public key (update documentation)
3. Re-sign critical artifacts with new key
4. Notify stakeholders of key change
5. Update CI/CD pipelines with new key
6. Document incident for compliance

---

Air-Gapped Signing

For environments without internet access:

Setup

1. On internet-connected machine:

   # Download Cosign binary
   curl -O -L https://github.com/sigstore/cosign/releases/download/v2.4.1/cosign-linux-amd64
   sha256sum cosign-linux-amd64
   
   # Transfer to air-gapped machine via USB
   

2. On air-gapped machine:

   # Install Cosign
   sudo install cosign-linux-amd64 /usr/local/bin/cosign
   
   # Verify installation
   cosign version
   

Signing Without Rekor

# Sign without transparency log
COSIGN_EXPERIMENTAL=0 \
COSIGN_PRIVATE_KEY="$(cat ~/.cosign/cosign-airgap.key)" \
COSIGN_PASSWORD="your-password" \
cosign sign --yes --key ~/.cosign/cosign-airgap.key charon:local

# Note: This disables Rekor transparency log
# Suitable only for internal use or air-gapped environments

Verification (Air-Gapped)

# Verify signature with public key only
cosign verify charon:local --key ~/.cosign/cosign-airgap.pub --insecure-ignore-tlog

⚠️ SECURITY NOTE: Air-gapped signing without Rekor loses public auditability. Use only when necessary and document the decision.

---

Troubleshooting

"cosign: error: signing: getting signer: reading key: decrypt: encrypted: no password provided"

Cause: Missing COSIGN_PASSWORD environment variable Solution:

export COSIGN_PASSWORD="your-password"
cosign sign --key cosign.key charon:local

"cosign: error: signing: getting signer: reading key: decrypt: openpgp: invalid data: private key checksum failure"

Cause: Incorrect password Solution: Verify you're using the correct password for the key

"Error: signing charon:local: uploading signature: PUT https://registry/v2/.../manifests/sha256-...: UNAUTHORIZED"

Cause: Not authenticated with Docker registry Solution:

docker login ghcr.io
# Enter credentials, then retry signing

"Error: verifying charon:local: fetching signatures: getting signature manifest: GET https://registry/...: NOT_FOUND"

Cause: Image not signed yet, or signature not pushed to registry Solution: Sign the image first with cosign sign

Key File Corrupted

Symptoms: Decryption errors, unusual characters in key file Solution:

1. Restore from encrypted backup (see Backup and Recovery)
2. If no backup: Generate new key pair and re-sign artifacts
3. Update documentation and notify stakeholders

Lost Password

Solution:

1. Cannot recover - private key is permanently inaccessible
2. Generate new key pair
3. Revoke old public key from documentation
4. Re-sign all artifacts
5. Consider using password manager to prevent future loss

---

Best Practices Summary

DO

✅ Use strong passwords (20+ characters) ✅ Store keys in password manager or HSM ✅ Set restrictive file permissions (600 on private keys) ✅ Rotate keys every 90 days ✅ Create encrypted backups ✅ Use different keys for different environments ✅ Test keys after generation ✅ Document key rotation dates ✅ Use keyless signing in CI/CD when possible

DON'T

❌ Commit private keys to version control ❌ Share private keys via email or chat ❌ Store keys in plaintext files ❌ Use the same key for multiple environments ❌ Hardcode passwords in scripts ❌ Skip backups ❌ Ignore rotation schedules ❌ Use weak passwords ❌ Store keys on network shares

---

Security Contacts

If you suspect key compromise:

1. Immediately: Stop using the compromised key
2. Notify: Security team at security@example.com
3. Rotate: Generate new key pair
4. Audit: Review all signatures made with compromised key
5. Document: Create incident report

---

References

• Cosign Documentation
• Key Management Best Practices (NIST)
• OpenSSF Security Best Practices
• SLSA Requirements

---

Document Version: 1.0 Last Reviewed: 2026-01-10 Next Review: 2026-04-10 (quarterly)