Charon/docs/plans/current_spec.md

CI Encryption-Key Investigation and Remediation Plan

Context

• Date: 2026-02-17
• Scope: CI failures where backend jobs report encryption key not picked up.
• In-scope files:
  • .github/workflows/quality-checks.yml
  • .github/workflows/codecov-upload.yml
  • scripts/go-test-coverage.sh
  • backend/internal/crypto/rotation_service.go
  • backend/internal/services/dns_provider_service.go
  • backend/internal/services/credential_service.go

Problem Statement

CI backend tests can fail late and ambiguously when CHARON_ENCRYPTION_KEY is missing or malformed. The root causes are context-dependent secret availability, missing preflight validation, and drift between workflow intent and implementation.

Research Findings

Workflow Surface and Risks

Workflow	Job	Key-sensitive step	Current key source	Main risk
.github/workflows/quality-checks.yml	backend-quality	Run Go tests, Run Perf Asserts	${{ secrets.CHARON_ENCRYPTION_KEY_TEST }}	Empty/malformed input not preflighted
.github/workflows/codecov-upload.yml	backend-codecov	Run Go tests with coverage	${{ secrets.CHARON_ENCRYPTION_KEY_TEST }}	Same key-risk as above

Backend Failure Surface

• backend/internal/crypto/rotation_service.go
  • NewRotationService(db *gorm.DB) hard-fails if CHARON_ENCRYPTION_KEY is empty.
• backend/internal/services/dns_provider_service.go
  • NewDNSProviderService(...) depends on NewRotationService(...) and can degrade to warning-based behavior when key input is bad.
• backend/internal/services/credential_service.go
  • NewCredentialService(...) has the same dependency pattern.

Script Failure Mode

• scripts/go-test-coverage.sh currently uses set -euo pipefail but does not pre-validate key shape before go test.
• Empty secret expressions become late runtime failures instead of deterministic preflight failures.

Supervisor-Required Constraints (Preserved)

1. pull_request_target SHALL NOT be used for secret-bearing backend test execution on untrusted code (fork PRs and Dependabot PRs).
2. Same-repo pull_request and workflow_dispatch SHALL require CHARON_ENCRYPTION_KEY_TEST; missing secret SHALL fail fast (no fallback).
3. Fork PRs and Dependabot PRs SHALL use workflow-only ephemeral key fallback for backend test execution.
4. Key material SHALL NEVER be logged.
5. Resolved key SHALL be masked before any potential output path.
6. GITHUB_ENV propagation SHALL use safe delimiter write pattern.
7. Workflow layer SHALL own key resolution/fallback.
8. Script layer SHALL only validate and fail fast; it SHALL NOT generate fallback keys.
9. Anti-drift guard SHALL be added so trigger comments and trigger blocks remain aligned.
10. Known drift SHALL be corrected: comment in quality-checks.yml about codecov-upload.yml trigger behavior must match actual triggers.

EARS Requirements

Ubiquitous

• THE SYSTEM SHALL fail fast with explicit diagnostics when encryption-key input is required and unavailable or malformed.
• THE SYSTEM SHALL prevent secret-value exposure in logs, summaries, and artifacts.

Event-driven

• WHEN workflow context is trusted (same-repo pull_request or workflow_dispatch), THE SYSTEM SHALL require secrets.CHARON_ENCRYPTION_KEY_TEST.
• WHEN workflow context is untrusted (fork PR or Dependabot PR), THE SYSTEM SHALL generate ephemeral key material in workflow preflight only.
• WHEN workflow context is untrusted, THE SYSTEM SHALL NOT use pull_request_target for secret-bearing backend tests.

Unwanted behavior

• IF CHARON_ENCRYPTION_KEY is empty, non-base64, or decoded length is not 32 bytes, THEN THE SYSTEM SHALL stop before running tests.
• IF trigger comments diverge from workflow triggers, THEN THE SYSTEM SHALL fail anti-drift validation.

Technical Design

Workflow Contract

Both backend jobs (backend-quality, backend-codecov) implement the same preflight sequence:

1. Resolve encryption key for backend tests
2. Fail fast when required encryption secret is missing
3. Validate encryption key format

Preflight Resolution Algorithm

1. Detect fork PR context via github.event.pull_request.head.repo.fork.
2. Detect Dependabot PR context (actor/repo metadata check).
3. Trusted context: require secrets.CHARON_ENCRYPTION_KEY_TEST; fail immediately if empty.
4. Untrusted context: generate ephemeral key (openssl rand -base64 32) in workflow only.
5. Mask resolved key via ::add-mask::.
6. Export via delimiter-based GITHUB_ENV write:
   • CHARON_ENCRYPTION_KEY<<EOF
   • <value>
   • EOF

Script Validation Contract

scripts/go-test-coverage.sh adds strict preflight validation:

• Present and non-empty.
• Base64 decodable.
• Decoded length exactly 32 bytes.

Script constraints:

• SHALL NOT generate keys.
• SHALL NOT select key source.
• SHALL only validate and fail fast with deterministic error messages.

Error Handling Matrix

Condition	Detection layer	Outcome
Trusted context + missing secret	Workflow preflight	Immediate failure with explicit message
Untrusted context + no secret access	Workflow preflight	Ephemeral key path (masked)
Malformed key	Script preflight	Immediate failure before go test
Trigger/comment drift	Workflow consistency guard	CI failure until synchronized

Implementation Plan

Phase 1: Workflow Hardening

• Update .github/workflows/quality-checks.yml and .github/workflows/codecov-upload.yml with identical key-resolution and key-validation steps.
• Enforce trusted-context fail-fast and untrusted-context fallback boundaries.
• Add explicit prohibition notes and controls preventing pull_request_target migration for secret-bearing tests.

Phase 2: Script Preflight Hardening

• Update scripts/go-test-coverage.sh to validate key presence/format/length before tests.
• Preserve existing coverage behavior; only harden pre-test guard path.

Phase 3: Anti-Drift Enforcement

• Define one canonical backend-key-bootstrap contract path.
• Add consistency check that enforces trigger/comment parity between quality-checks.yml and codecov-upload.yml.
• Fix known push-only comment mismatch in quality-checks.yml.

Validation Plan

Run these scenarios:

1. Same-repo PR with valid secret.
2. Same-repo PR with missing secret (must fail fast).
3. Same-repo PR with malformed secret (must fail fast before tests).
4. Fork PR with no secret access (must use ephemeral fallback).
5. Dependabot PR with no secret access (must use ephemeral fallback, no pull_request_target).
6. workflow_dispatch with valid secret.

Expected results:

• No late ambiguous key-init failures.
• No secret material logged.
• Deterministic and attributable failure messages.
• Trigger docs and trigger config remain synchronized.

Acceptance Criteria

• Backend jobs in quality-checks.yml and codecov-upload.yml no longer fail ambiguously on encryption-key pickup.
• Trusted contexts fail fast if CHARON_ENCRYPTION_KEY_TEST is missing.
• Untrusted contexts use workflow-only ephemeral fallback.
• scripts/go-test-coverage.sh enforces deterministic key preflight checks.
• pull_request_target is explicitly prohibited for secret-bearing backend tests on untrusted code.
• Never-log-key-material and safe GITHUB_ENV propagation are implemented.
• Workflow/script responsibility boundary is enforced.
• Anti-drift guard is present and known trigger-comment mismatch is resolved.

Handoff to Supervisor

• This document is intentionally single-scope and restricted to CI encryption-key investigation/remediation.
• Legacy multi-topic coverage planning content has been removed from this file to maintain coherence.