234 lines
6.1 KiB
Markdown
234 lines
6.1 KiB
Markdown
# Manual Test Plan: CI Docker Build Fix Verification
|
|
|
|
**Issue**: Docker image artifact save failing with "reference does not exist" error
|
|
**Fix Date**: 2026-01-12
|
|
**Test Target**: `.github/workflows/docker-build.yml` (Save Docker Image as Artifact step)
|
|
**Test Priority**: HIGH (blocks PR builds and supply chain verification)
|
|
|
|
---
|
|
|
|
## Test Objective
|
|
|
|
Verify that the CI Docker build fix resolves the "reference does not exist" error and enables successful PR builds with artifact generation and supply chain verification.
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
- [ ] Changes merged to a feature branch or development
|
|
- [ ] Ability to create test PRs against the target branch
|
|
- [ ] Access to GitHub Actions logs for the test PR
|
|
- [ ] Understanding of expected workflow behavior
|
|
|
|
---
|
|
|
|
## Test Scenarios
|
|
|
|
### Scenario 1: Standard PR Build (Happy Path)
|
|
|
|
**Objective**: Verify normal PR build succeeds with image artifact save
|
|
|
|
**Steps**:
|
|
|
|
1. Create a test PR with a minor change (e.g., update README.md)
|
|
2. Wait for `docker-build.yml` workflow to trigger
|
|
3. Monitor the workflow execution in GitHub Actions
|
|
|
|
**Expected Results**:
|
|
|
|
- [ ] ✅ `build-and-push` job completes successfully
|
|
- [ ] ✅ "Save Docker Image as Artifact" step completes without errors
|
|
- [ ] ✅ Step output shows: "🔍 Detected image tag: ghcr.io/wikid82/charon:pr-XXX"
|
|
- [ ] ✅ Step output shows: "✅ Artifact created: /tmp/charon-pr-image.tar"
|
|
- [ ] ✅ "Upload Image Artifact" step succeeds
|
|
- [ ] ✅ Artifact `pr-image-XXX` appears in workflow artifacts
|
|
- [ ] ✅ `verify-supply-chain-pr` job starts and uses the artifact
|
|
- [ ] ✅ Supply chain verification completes successfully
|
|
|
|
**Pass Criteria**: All checks pass, no "reference does not exist" errors
|
|
|
|
---
|
|
|
|
### Scenario 2: Metadata Tag Validation
|
|
|
|
**Objective**: Verify defensive validation catches missing or invalid tags
|
|
|
|
**Steps**:
|
|
|
|
1. Review the "Save Docker Image as Artifact" step logs
|
|
2. Check for validation output
|
|
|
|
**Expected Results**:
|
|
|
|
- [ ] ✅ Step logs show: "🔍 Detected image tag: ghcr.io/wikid82/charon:pr-XXX"
|
|
- [ ] ✅ No error messages about missing tags
|
|
- [ ] ✅ Image inspection succeeds (no "not found locally" errors)
|
|
|
|
**Pass Criteria**: Validation steps execute and pass cleanly
|
|
|
|
---
|
|
|
|
### Scenario 3: Supply Chain Verification Integration
|
|
|
|
**Objective**: Verify downstream job receives and processes the artifact correctly
|
|
|
|
**Steps**:
|
|
|
|
1. Wait for `verify-supply-chain-pr` job to start
|
|
2. Check "Download Image Artifact" step
|
|
3. Check "Load Docker Image" step
|
|
4. Check "Verify Loaded Image" step
|
|
|
|
**Expected Results**:
|
|
|
|
- [ ] ✅ Artifact downloads successfully
|
|
- [ ] ✅ Image loads without errors
|
|
- [ ] ✅ Verification step confirms image exists: "✅ Image verified: ghcr.io/wikid82/charon:pr-XXX"
|
|
- [ ] ✅ SBOM generation step uses correct image reference
|
|
- [ ] ✅ Vulnerability scanning completes
|
|
- [ ] ✅ PR comment appears with supply chain verification results
|
|
|
|
**Pass Criteria**: Full supply chain verification pipeline executes end-to-end
|
|
|
|
---
|
|
|
|
### Scenario 4: Error Handling (Edge Case)
|
|
|
|
**Objective**: Verify defensive validation catches actual errors (if possible to trigger)
|
|
|
|
**Note**: This scenario is difficult to test without artificially breaking the build. Monitor for this in production if a natural failure occurs.
|
|
|
|
**Expected Behavior** (if error occurs):
|
|
|
|
- [ ] Step fails fast with clear diagnostics
|
|
- [ ] Error message shows exact issue (missing tag, image not found, etc.)
|
|
- [ ] Available images are listed for debugging
|
|
- [ ] Workflow fails with actionable error message
|
|
|
|
**Pass Criteria**: If error occurs, diagnostics are clear and actionable
|
|
|
|
---
|
|
|
|
## Regression Testing
|
|
|
|
### Check Previous Failure Cases
|
|
|
|
**Steps**:
|
|
|
|
1. Review previous failed PR builds (before fix)
|
|
2. Note the exact error messages
|
|
3. Confirm those errors no longer occur
|
|
|
|
**Expected Results**:
|
|
|
|
- [ ] ✅ No "reference does not exist" errors
|
|
- [ ] ✅ No "image not found" errors during save
|
|
- [ ] ✅ No manual tag reconstruction mismatches
|
|
|
|
**Pass Criteria**: Previous failure patterns are eliminated
|
|
|
|
---
|
|
|
|
## Performance Validation
|
|
|
|
**Objective**: Ensure fix does not introduce performance degradation
|
|
|
|
**Metrics to Monitor**:
|
|
|
|
- [ ] Build time (build-and-push job duration)
|
|
- [ ] Artifact save time
|
|
- [ ] Artifact upload time
|
|
- [ ] Total PR workflow duration
|
|
|
|
**Expected Results**:
|
|
|
|
- Build time: ~10-15 minutes (no significant change)
|
|
- Artifact save: <30 seconds
|
|
- Artifact upload: <1 minute
|
|
- Total workflow: <20 minutes for PR builds
|
|
|
|
**Pass Criteria**: No significant performance regression (±10% acceptable variance)
|
|
|
|
---
|
|
|
|
## Rollback Plan
|
|
|
|
**If Tests Fail**:
|
|
|
|
1. **Immediate Action**:
|
|
- Revert commit fixing the artifact save step
|
|
- Notify team of rollback
|
|
- Create new issue with failure details
|
|
|
|
2. **Investigation**:
|
|
- Capture full workflow logs
|
|
- Check docker images output from failing run
|
|
- Verify metadata action output format
|
|
- Check for platform-specific issues (amd64 vs arm64)
|
|
|
|
3. **Recovery**:
|
|
- Develop alternative fix approach
|
|
- Test in isolated branch
|
|
- Reapply fix after validation
|
|
|
|
---
|
|
|
|
## Test Log Template
|
|
|
|
**Test Execution Date**: [YYYY-MM-DD]
|
|
**Test PR Number**: #XXX
|
|
**Workflow Run**: [Link to GitHub Actions run]
|
|
**Tester**: [Name]
|
|
|
|
### Scenario 1: Standard PR Build
|
|
|
|
- Status: [ ] PASS / [ ] FAIL
|
|
- Notes:
|
|
|
|
### Scenario 2: Metadata Tag Validation
|
|
|
|
- Status: [ ] PASS / [ ] FAIL
|
|
- Notes:
|
|
|
|
### Scenario 3: Supply Chain Verification Integration
|
|
|
|
- Status: [ ] PASS / [ ] FAIL
|
|
- Notes:
|
|
|
|
### Scenario 4: Error Handling
|
|
|
|
- Status: [ ] PASS / [ ] FAIL / [ ] N/A
|
|
- Notes:
|
|
|
|
### Regression Testing
|
|
|
|
- Status: [ ] PASS / [ ] FAIL
|
|
- Notes:
|
|
|
|
### Performance Validation
|
|
|
|
- Status: [ ] PASS / [ ] FAIL
|
|
- Build time: X minutes
|
|
- Artifact save: X seconds
|
|
- Total workflow: X minutes
|
|
- Notes:
|
|
|
|
---
|
|
|
|
## Sign-Off
|
|
|
|
**Test Result**: [ ] PASS / [ ] FAIL
|
|
**Tested By**: _____________________
|
|
**Date**: _____________________
|
|
**Approved By**: _____________________
|
|
**Date**: _____________________
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- Original issue: See `current_spec.md` for root cause analysis
|
|
- Workflow file: `.github/workflows/docker-build.yml`
|
|
- Related fix: Lines 135-167 (Save Docker Image as Artifact step)
|
|
- CHANGELOG entry: See "Fixed" section under "Unreleased"
|