chore: clean .gitignore cache
This commit is contained in:
GitHub Actions
265
docs/issues/created/20260111-manual_test_ci_workflow_fixes.md
Normal file
265
docs/issues/created/20260111-manual_test_ci_workflow_fixes.md
Normal file
@@ -0,0 +1,265 @@
|
||||
# Manual Test Plan: CI Workflow Fixes
|
||||
|
||||
**Created:** 2026-01-11
|
||||
**PR:** #461
|
||||
**Feature:** CI/CD Workflow Documentation & Supply Chain Fix
|
||||
|
||||
## Objective
|
||||
|
||||
Manually verify that the CI workflow fixes work correctly in production, focusing on finding potential bugs in the Supply Chain Verification orchestration.
|
||||
|
||||
## Background
|
||||
|
||||
**What Was Fixed:**
|
||||
|
||||
1. Removed `branches` filter from `supply-chain-verify.yml` to enable `workflow_run` triggering on all branches
|
||||
2. Added documentation to explain the GitHub Security warning (false positive)
|
||||
3. Updated SECURITY.md with comprehensive security scanning documentation
|
||||
|
||||
**Expected Behavior:**
|
||||
|
||||
- Supply Chain Verification should now trigger via `workflow_run` after Docker Build completes on ANY branch
|
||||
- Previous behavior: Only triggered via `pull_request` fallback (branch filter prevented workflow_run)
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### Scenario 1: Push to Feature Branch (workflow_run Test)
|
||||
|
||||
**Goal:** Verify `workflow_run` trigger works on feature branches after fix
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Create a small test commit on `feature/beta-release`
|
||||
2. Push the commit
|
||||
3. Monitor GitHub Actions workflow runs
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Docker Build workflow triggers and completes successfully
|
||||
- ✅ Supply Chain Verification triggers **via workflow_run event** (not pull_request)
|
||||
- ✅ Supply Chain completes successfully
|
||||
- ✅ GitHub Actions logs show event type is `workflow_run`
|
||||
|
||||
**How to Verify Event Type:**
|
||||
|
||||
```bash
|
||||
gh run list --workflow="supply-chain-verify.yml" --limit 1 --json event,conclusion
|
||||
# Should show: "event": "workflow_run", "conclusion": "success"
|
||||
```
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Supply Chain doesn't trigger at all
|
||||
- ❌ Supply Chain triggers but fails
|
||||
- ❌ Multiple simultaneous runs (race condition)
|
||||
- ❌ Timeout or hang in workflow_run chain
|
||||
|
||||
---
|
||||
|
||||
### Scenario 2: PR Synchronization (Fallback Still Works)
|
||||
|
||||
**Goal:** Verify `pull_request` fallback trigger still works correctly
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. With PR #461 open, push another small commit
|
||||
2. Monitor GitHub Actions workflow runs
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Docker Build triggers via `pull_request` event
|
||||
- ✅ Supply Chain may trigger via BOTH `workflow_run` AND `pull_request` (race condition possible)
|
||||
- ✅ If both trigger, both should complete successfully without conflict
|
||||
- ✅ PR should show both workflow checks passing
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Duplicate runs causing conflicts
|
||||
- ❌ Race condition causing failures
|
||||
- ❌ PR checks showing "pending" indefinitely
|
||||
- ❌ One workflow cancels the other
|
||||
|
||||
---
|
||||
|
||||
### Scenario 3: Main Branch Push (Default Branch Behavior)
|
||||
|
||||
**Goal:** Verify fix doesn't break main branch behavior
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. After PR #461 merges to main, monitor the merge commit
|
||||
2. Check GitHub Actions runs
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Docker Build runs on main
|
||||
- ✅ Supply Chain triggers via `workflow_run`
|
||||
- ✅ Both complete successfully
|
||||
- ✅ Weekly scheduled runs continue to work
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Main branch workflows broken
|
||||
- ❌ Weekly schedule interferes with workflow_run
|
||||
- ❌ Permissions issues on main branch
|
||||
|
||||
---
|
||||
|
||||
### Scenario 4: Failed Docker Build (Error Handling)
|
||||
|
||||
**Goal:** Verify Supply Chain doesn't trigger when Docker Build fails
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Intentionally break Docker Build (e.g., invalid Dockerfile syntax)
|
||||
2. Push to a test branch
|
||||
3. Monitor workflow behavior
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Docker Build fails as expected
|
||||
- ✅ Supply Chain **does NOT trigger** (workflow_run only fires on `completed` and `success`)
|
||||
- ✅ No cascading failures
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Supply Chain triggers on failed builds
|
||||
- ❌ Error handling missing
|
||||
- ❌ Workflow stuck in pending state
|
||||
|
||||
---
|
||||
|
||||
### Scenario 5: Manual Workflow Dispatch
|
||||
|
||||
**Goal:** Verify manual trigger still works
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Go to GitHub Actions → Supply Chain Verification
|
||||
2. Click "Run workflow"
|
||||
3. Select `feature/beta-release` branch
|
||||
4. Click "Run workflow"
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Workflow starts via `workflow_dispatch` event
|
||||
- ✅ Completes successfully
|
||||
- ✅ SBOM and attestations generated
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Manual dispatch broken
|
||||
- ❌ Branch selector doesn't work
|
||||
- ❌ Workflow fails with "branch not found"
|
||||
|
||||
---
|
||||
|
||||
### Scenario 6: Weekly Scheduled Run
|
||||
|
||||
**Goal:** Verify scheduled trigger still works
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Wait for next Monday 00:00 UTC
|
||||
2. Check GitHub Actions for scheduled run
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- ✅ Workflow triggers via `schedule` event
|
||||
- ✅ Runs on main branch
|
||||
- ✅ Completes successfully
|
||||
|
||||
**Potential Bugs to Watch For:**
|
||||
|
||||
- ❌ Schedule doesn't fire
|
||||
- ❌ Wrong branch selected
|
||||
- ❌ Interference with other workflows
|
||||
|
||||
---
|
||||
|
||||
## Edge Cases to Test
|
||||
|
||||
### Edge Case 1: Rapid Pushes (Rate Limiting)
|
||||
|
||||
**Test:** Push 3-5 commits rapidly to feature branch
|
||||
**Expected:** All Docker Builds run, Supply Chain may queue or skip redundant runs
|
||||
**Watch For:** Workflow queue overflow, cancellations, failures
|
||||
|
||||
### Edge Case 2: Long-Running Docker Build
|
||||
|
||||
**Test:** Create a commit that makes Docker Build take >10 minutes
|
||||
**Expected:** Supply Chain waits for completion before triggering
|
||||
**Watch For:** Timeouts, abandoned runs, state corruption
|
||||
|
||||
### Edge Case 3: Branch Deletion During Run
|
||||
|
||||
**Test:** Delete feature branch while workflows are running
|
||||
**Expected:** Workflows complete or cancel gracefully
|
||||
**Watch For:** Orphaned runs, resource leaks, errors
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All 6 scenarios pass without critical bugs
|
||||
- [ ] `workflow_run` event type confirmed in logs
|
||||
- [ ] No cascading failures
|
||||
- [ ] PR checks consistently pass
|
||||
- [ ] Error handling works correctly
|
||||
- [ ] Manual and scheduled triggers functional
|
||||
|
||||
## Bug Severity Guidelines
|
||||
|
||||
**CRITICAL** (Block Merge):
|
||||
|
||||
- Supply Chain doesn't run at all
|
||||
- Cascading failures breaking other workflows
|
||||
- Security vulnerabilities introduced
|
||||
|
||||
**HIGH** (Fix Before Release):
|
||||
|
||||
- Race conditions causing frequent failures
|
||||
- Resource leaks or orphaned workflows
|
||||
- Error handling missing
|
||||
|
||||
**MEDIUM** (Fix in Future PR):
|
||||
|
||||
- Duplicate runs (but both succeed)
|
||||
- Inconsistent behavior (works sometimes)
|
||||
- Minor UX issues
|
||||
|
||||
**LOW** (Document as Known Issue):
|
||||
|
||||
- Cosmetic issues in logs
|
||||
- Non-breaking edge cases
|
||||
- Timing inconsistencies
|
||||
|
||||
---
|
||||
|
||||
## Notes for Testers
|
||||
|
||||
1. **Event Type Verification is Critical:** The core fix was to enable `workflow_run` on feature branches. If logs still show only `pull_request` events, the fix didn't work.
|
||||
|
||||
2. **False Positives are OK:** The GitHub Security warning may persist for 4-8 weeks due to tracking lag. This is expected.
|
||||
|
||||
3. **Timing Matters:** There may be a 1-2 second delay between Docker Build completion and Supply Chain trigger. This is normal.
|
||||
|
||||
4. **Logs are Essential:** Always check the "Event" field in GitHub Actions run details to confirm the trigger type.
|
||||
|
||||
---
|
||||
|
||||
## Reporting Bugs
|
||||
|
||||
If bugs are found during manual testing:
|
||||
|
||||
1. Create a new issue in `docs/issues/bug_*.md`
|
||||
2. Include:
|
||||
- Scenario number
|
||||
- Exact steps to reproduce
|
||||
- Expected vs actual behavior
|
||||
- GitHub Actions run ID
|
||||
- Event type from logs
|
||||
- Severity classification
|
||||
|
||||
3. Link to this test plan
|
||||
4. Assign to appropriate team member
|
||||
Reference in New Issue
Block a user