Charon/docs/issues/created/20260111-manual_test_ci_workflow_fixes.md

Manual Test Plan: CI Workflow Fixes

Created: 2026-01-11 PR: #461 Feature: CI/CD Workflow Documentation & Supply Chain Fix

Objective

Manually verify that the CI workflow fixes work correctly in production, focusing on finding potential bugs in the Supply Chain Verification orchestration.

Background

What Was Fixed:

1. Removed branches filter from supply-chain-verify.yml to enable workflow_run triggering on all branches
2. Added documentation to explain the GitHub Security warning (false positive)
3. Updated SECURITY.md with comprehensive security scanning documentation

Expected Behavior:

• Supply Chain Verification should now trigger via workflow_run after Docker Build completes on ANY branch
• Previous behavior: Only triggered via pull_request fallback (branch filter prevented workflow_run)

Test Scenarios

Scenario 1: Push to Feature Branch (workflow_run Test)

Goal: Verify workflow_run trigger works on feature branches after fix

Steps:

1. Create a small test commit on feature/beta-release
2. Push the commit
3. Monitor GitHub Actions workflow runs

Expected Results:

• ✅ Docker Build workflow triggers and completes successfully
• ✅ Supply Chain Verification triggers via workflow_run event (not pull_request)
• ✅ Supply Chain completes successfully
• ✅ GitHub Actions logs show event type is workflow_run

How to Verify Event Type:

gh run list --workflow="supply-chain-verify.yml" --limit 1 --json event,conclusion
# Should show: "event": "workflow_run", "conclusion": "success"

Potential Bugs to Watch For:

• ❌ Supply Chain doesn't trigger at all
• ❌ Supply Chain triggers but fails
• ❌ Multiple simultaneous runs (race condition)
• ❌ Timeout or hang in workflow_run chain

---

Scenario 2: PR Synchronization (Fallback Still Works)

Goal: Verify pull_request fallback trigger still works correctly

Steps:

1. With PR #461 open, push another small commit
2. Monitor GitHub Actions workflow runs

Expected Results:

• ✅ Docker Build triggers via pull_request event
• ✅ Supply Chain may trigger via BOTH workflow_run AND pull_request (race condition possible)
• ✅ If both trigger, both should complete successfully without conflict
• ✅ PR should show both workflow checks passing

Potential Bugs to Watch For:

• ❌ Duplicate runs causing conflicts
• ❌ Race condition causing failures
• ❌ PR checks showing "pending" indefinitely
• ❌ One workflow cancels the other

---

Scenario 3: Main Branch Push (Default Branch Behavior)

Goal: Verify fix doesn't break main branch behavior

Steps:

1. After PR #461 merges to main, monitor the merge commit
2. Check GitHub Actions runs

Expected Results:

• ✅ Docker Build runs on main
• ✅ Supply Chain triggers via workflow_run
• ✅ Both complete successfully
• ✅ Weekly scheduled runs continue to work

Potential Bugs to Watch For:

• ❌ Main branch workflows broken
• ❌ Weekly schedule interferes with workflow_run
• ❌ Permissions issues on main branch

---

Scenario 4: Failed Docker Build (Error Handling)

Goal: Verify Supply Chain doesn't trigger when Docker Build fails

Steps:

1. Intentionally break Docker Build (e.g., invalid Dockerfile syntax)
2. Push to a test branch
3. Monitor workflow behavior

Expected Results:

• ✅ Docker Build fails as expected
• ✅ Supply Chain does NOT trigger (workflow_run only fires on completed and success)
• ✅ No cascading failures

Potential Bugs to Watch For:

• ❌ Supply Chain triggers on failed builds
• ❌ Error handling missing
• ❌ Workflow stuck in pending state

---

Scenario 5: Manual Workflow Dispatch

Goal: Verify manual trigger still works

Steps:

1. Go to GitHub Actions → Supply Chain Verification
2. Click "Run workflow"
3. Select feature/beta-release branch
4. Click "Run workflow"

Expected Results:

• ✅ Workflow starts via workflow_dispatch event
• ✅ Completes successfully
• ✅ SBOM and attestations generated

Potential Bugs to Watch For:

• ❌ Manual dispatch broken
• ❌ Branch selector doesn't work
• ❌ Workflow fails with "branch not found"

---

Scenario 6: Weekly Scheduled Run

Goal: Verify scheduled trigger still works

Steps:

1. Wait for next Monday 00:00 UTC
2. Check GitHub Actions for scheduled run

Expected Results:

• ✅ Workflow triggers via schedule event
• ✅ Runs on main branch
• ✅ Completes successfully

Potential Bugs to Watch For:

• ❌ Schedule doesn't fire
• ❌ Wrong branch selected
• ❌ Interference with other workflows

---

Edge Cases to Test

Edge Case 1: Rapid Pushes (Rate Limiting)

Test: Push 3-5 commits rapidly to feature branch Expected: All Docker Builds run, Supply Chain may queue or skip redundant runs Watch For: Workflow queue overflow, cancellations, failures

Edge Case 2: Long-Running Docker Build

Test: Create a commit that makes Docker Build take >10 minutes Expected: Supply Chain waits for completion before triggering Watch For: Timeouts, abandoned runs, state corruption

Edge Case 3: Branch Deletion During Run

Test: Delete feature branch while workflows are running Expected: Workflows complete or cancel gracefully Watch For: Orphaned runs, resource leaks, errors

---

Success Criteria

• All 6 scenarios pass without critical bugs
• workflow_run event type confirmed in logs
• No cascading failures
• PR checks consistently pass
• Error handling works correctly
• Manual and scheduled triggers functional

Bug Severity Guidelines

CRITICAL (Block Merge):

• Supply Chain doesn't run at all
• Cascading failures breaking other workflows
• Security vulnerabilities introduced

HIGH (Fix Before Release):

• Race conditions causing frequent failures
• Resource leaks or orphaned workflows
• Error handling missing

MEDIUM (Fix in Future PR):

• Duplicate runs (but both succeed)
• Inconsistent behavior (works sometimes)
• Minor UX issues

LOW (Document as Known Issue):

• Cosmetic issues in logs
• Non-breaking edge cases
• Timing inconsistencies

---

Notes for Testers

1. Event Type Verification is Critical: The core fix was to enable workflow_run on feature branches. If logs still show only pull_request events, the fix didn't work.

2. False Positives are OK: The GitHub Security warning may persist for 4-8 weeks due to tracking lag. This is expected.

3. Timing Matters: There may be a 1-2 second delay between Docker Build completion and Supply Chain trigger. This is normal.

4. Logs are Essential: Always check the "Event" field in GitHub Actions run details to confirm the trigger type.

---

Reporting Bugs

If bugs are found during manual testing:

1. Create a new issue in docs/issues/bug_*.md

2. Include:

   • Scenario number
   • Exact steps to reproduce
   • Expected vs actual behavior
   • GitHub Actions run ID
   • Event type from logs
   • Severity classification

3. Link to this test plan

4. Assign to appropriate team member