chore: git cache cleanup
This commit is contained in:
GitHub Actions
339
docs/issues/created/20260110-grype_sbom_manual_testing.md
Normal file
339
docs/issues/created/20260110-grype_sbom_manual_testing.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Manual Testing Plan: Grype SBOM Remediation
|
||||
|
||||
**Issue Type**: Manual Testing
|
||||
**Priority**: High
|
||||
**Component**: CI/CD - Supply Chain Verification
|
||||
**Created**: 2026-01-10
|
||||
**Related PR**: #461 (DNS Challenge Support)
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Manually validate the Grype SBOM remediation implementation in real-world CI/CD scenarios to ensure:
|
||||
|
||||
- Workflow operates correctly in all expected conditions
|
||||
- Error handling is robust and user-friendly
|
||||
- No regressions in existing functionality
|
||||
|
||||
---
|
||||
|
||||
## Test Environment
|
||||
|
||||
- **Branch**: `feature/beta-release` (current)
|
||||
- **Workflow File**: `.github/workflows/supply-chain-verify.yml`
|
||||
- **Trigger Events**: `pull_request`, `push to main`, `workflow_dispatch`
|
||||
|
||||
---
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
### Scenario 1: PR Without Docker Image (Skip Path)
|
||||
|
||||
**Objective**: Verify workflow gracefully skips when image doesn't exist (common in PR workflows before docker-build completes).
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
- Create a test PR with code changes
|
||||
- Ensure docker-build workflow has NOT completed yet
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. Create/update PR on feature branch
|
||||
2. Navigate to Actions → Supply Chain Verification workflow
|
||||
3. Wait for workflow to complete
|
||||
|
||||
**Expected Results**:
|
||||
|
||||
- ✅ Workflow completes successfully (green check)
|
||||
- ✅ "Check Image Availability" step shows "Image not found" message
|
||||
- ✅ "Report Skipped Scan" step shows clear skip reason
|
||||
- ✅ PR comment appears with "⏭️ Status: Image not yet available" message
|
||||
- ✅ PR comment explains this is normal for PR workflows
|
||||
- ✅ No false failures or error messages
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] Workflow status: Success (not failed or warning)
|
||||
- [ ] PR comment is clear and helpful
|
||||
- [ ] GitHub Step Summary shows skip reason
|
||||
- [ ] No confusing error messages in logs
|
||||
|
||||
---
|
||||
|
||||
### Scenario 2: Existing Docker Image (Success Path)
|
||||
|
||||
**Objective**: Verify full SBOM generation, validation, and vulnerability scanning when image exists.
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
- Use a branch where docker-build has completed (e.g., `main` or merged PR)
|
||||
- Image exists in GHCR: `ghcr.io/wikid82/charon:latest` or `ghcr.io/wikid82/charon:pr-XXX`
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. Trigger workflow manually via `workflow_dispatch` on main branch
|
||||
2. OR merge a PR and wait for automatic workflow trigger
|
||||
3. Monitor workflow execution
|
||||
|
||||
**Expected Results**:
|
||||
|
||||
- ✅ "Check Image Availability" step finds image
|
||||
- ✅ "Verify SBOM Completeness" step generates CycloneDX SBOM
|
||||
- ✅ Syft version is logged
|
||||
- ✅ "Validate SBOM File" step passes all checks:
|
||||
- jq is available
|
||||
- File exists and non-empty
|
||||
- Valid JSON structure
|
||||
- CycloneDX format confirmed
|
||||
- Components found (count > 0)
|
||||
- ✅ "Upload SBOM Artifact" step succeeds
|
||||
- ✅ SBOM artifact available for download
|
||||
- ✅ "Scan for Vulnerabilities" step:
|
||||
- Grype DB updates successfully
|
||||
- Scan completes without "format not recognized" error
|
||||
- Vulnerability counts reported
|
||||
- Results table displayed
|
||||
- ✅ PR comment (if PR) shows vulnerability summary table
|
||||
- ✅ No "sbom format not recognized" errors
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] Workflow status: Success
|
||||
- [ ] SBOM artifact uploaded and downloadable
|
||||
- [ ] Grype scan completes without format errors
|
||||
- [ ] Vulnerability counts accurate (Critical/High/Medium/Low)
|
||||
- [ ] PR comment shows detailed results (if applicable)
|
||||
- [ ] No false positives
|
||||
|
||||
---
|
||||
|
||||
### Scenario 3: Invalid/Corrupted SBOM (Validation Path)
|
||||
|
||||
**Objective**: Verify SBOM validation catches malformed files before passing to Grype.
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
- Requires temporarily modifying workflow to introduce error (NOT for production testing)
|
||||
- OR wait for natural occurrence (unlikely)
|
||||
|
||||
**Alternative Testing**:
|
||||
This scenario is validated through code review and unit testing of validation logic. Manual testing in production environment is not recommended as it requires intentionally breaking the workflow.
|
||||
|
||||
**Code Review Validation** (Already Completed):
|
||||
|
||||
- ✅ jq availability check (lines 125-130)
|
||||
- ✅ File existence check (lines 133-138)
|
||||
- ✅ Non-empty check (lines 141-146)
|
||||
- ✅ Valid JSON check (lines 149-156)
|
||||
- ✅ CycloneDX format check (lines 159-173)
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] Code review confirms all validation checks present
|
||||
- [ ] Error handling paths use `exit 1` for real errors
|
||||
- [ ] Clear error messages at each validation point
|
||||
|
||||
---
|
||||
|
||||
### Scenario 4: Critical Vulnerabilities Detected
|
||||
|
||||
**Objective**: Verify workflow correctly identifies and reports critical vulnerabilities.
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
- Use an older image tag with known vulnerabilities (if available)
|
||||
- OR wait for vulnerability to be discovered in current image
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. Trigger workflow on image with vulnerabilities
|
||||
2. Monitor vulnerability scan step
|
||||
3. Check PR comment and workflow logs
|
||||
|
||||
**Expected Results**:
|
||||
|
||||
- ✅ Grype scan completes successfully
|
||||
- ✅ Vulnerabilities categorized by severity
|
||||
- ✅ Critical vulnerabilities trigger GitHub annotation/warning
|
||||
- ✅ PR comment shows vulnerability table with non-zero counts
|
||||
- ✅ PR comment includes "⚠️ Action Required" for critical vulns
|
||||
- ✅ Link to full report is provided
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] Vulnerability counts are accurate
|
||||
- [ ] Critical vulnerabilities highlighted
|
||||
- [ ] Clear action guidance provided
|
||||
- [ ] Links to detailed reports work
|
||||
|
||||
---
|
||||
|
||||
### Scenario 5: Workflow Performance
|
||||
|
||||
**Objective**: Verify workflow executes within acceptable time limits.
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. Monitor workflow execution time across multiple runs
|
||||
2. Check individual step durations
|
||||
|
||||
**Expected Results**:
|
||||
|
||||
- ✅ Total workflow time: < 10 minutes
|
||||
- ✅ Image check: < 30 seconds
|
||||
- ✅ SBOM generation: < 2 minutes
|
||||
- ✅ SBOM validation: < 30 seconds
|
||||
- ✅ Grype scan: < 5 minutes
|
||||
- ✅ Artifact upload: < 1 minute
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] Average workflow time within limits
|
||||
- [ ] No significant performance degradation vs. previous implementation
|
||||
- [ ] No timeout failures
|
||||
|
||||
---
|
||||
|
||||
### Scenario 6: Multiple Parallel PRs
|
||||
|
||||
**Objective**: Verify workflow handles concurrent executions without conflicts.
|
||||
|
||||
**Prerequisites**:
|
||||
|
||||
- Create multiple PRs simultaneously
|
||||
- Trigger workflows on multiple branches
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. Create 3-5 PRs from different feature branches
|
||||
2. Wait for workflows to run concurrently
|
||||
3. Monitor all workflow executions
|
||||
|
||||
**Expected Results**:
|
||||
|
||||
- ✅ All workflows complete successfully
|
||||
- ✅ No resource conflicts or race conditions
|
||||
- ✅ Correct image checked for each PR (`pr-XXX` tags)
|
||||
- ✅ Each PR gets its own comment
|
||||
- ✅ Artifact names are unique (include tag)
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] All workflows succeed independently
|
||||
- [ ] No cross-contamination of results
|
||||
- [ ] Artifact names unique and correct
|
||||
|
||||
---
|
||||
|
||||
## Regression Testing
|
||||
|
||||
### Verify No Breaking Changes
|
||||
|
||||
**Test Areas**:
|
||||
|
||||
1. **Other Workflows**: Ensure docker-build.yml, codeql-analysis.yml, etc. still work
|
||||
2. **Existing Releases**: Verify workflow runs successfully on existing release tags
|
||||
3. **Backward Compatibility**: Old PRs can be re-run without issues
|
||||
|
||||
**Pass Criteria**:
|
||||
|
||||
- [ ] No regressions in other workflows
|
||||
- [ ] Existing functionality preserved
|
||||
- [ ] No unexpected failures
|
||||
|
||||
---
|
||||
|
||||
## Bug Hunting Focus Areas
|
||||
|
||||
Based on the implementation, pay special attention to:
|
||||
|
||||
1. **Conditional Logic**:
|
||||
- Verify `if: steps.image-check.outputs.exists == 'true'` works correctly
|
||||
- Check `if: steps.validate-sbom.outputs.valid == 'true'` gates scan properly
|
||||
|
||||
2. **Error Messages**:
|
||||
- Ensure error messages are clear and actionable
|
||||
- Verify debug output is helpful for troubleshooting
|
||||
|
||||
3. **Authentication**:
|
||||
- GHCR authentication succeeds for private repos
|
||||
- Token permissions are sufficient
|
||||
|
||||
4. **Artifact Handling**:
|
||||
- SBOM artifacts upload correctly
|
||||
- Artifact names are unique and descriptive
|
||||
- Retention period is appropriate (30 days)
|
||||
|
||||
5. **PR Comments**:
|
||||
- Comments appear on all PRs
|
||||
- Markdown formatting is correct
|
||||
- Links work and point to correct locations
|
||||
|
||||
6. **Edge Cases**:
|
||||
- Very large images (slow SBOM generation)
|
||||
- Images with many vulnerabilities (large scan output)
|
||||
- Network failures during Grype DB update
|
||||
- Rate limiting from GHCR
|
||||
|
||||
---
|
||||
|
||||
## Issue Reporting Template
|
||||
|
||||
If you find a bug during manual testing, create an issue with:
|
||||
|
||||
```markdown
|
||||
**Title**: [Grype SBOM] Brief description of issue
|
||||
|
||||
**Scenario**: Which test scenario revealed the issue
|
||||
|
||||
**Expected Behavior**: What should happen
|
||||
|
||||
**Actual Behavior**: What actually happened
|
||||
|
||||
**Evidence**:
|
||||
- Workflow run URL
|
||||
- Relevant log excerpts
|
||||
- Screenshots if applicable
|
||||
|
||||
**Severity**: Critical / High / Medium / Low
|
||||
|
||||
**Impact**: Who/what is affected
|
||||
|
||||
**Workaround**: If known
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off Checklist
|
||||
|
||||
After completing manual testing, verify:
|
||||
|
||||
- [ ] Scenario 1 (Skip Path) tested and passed
|
||||
- [ ] Scenario 2 (Success Path) tested and passed
|
||||
- [ ] Scenario 3 (Validation) verified via code review
|
||||
- [ ] Scenario 4 (Vulnerabilities) tested and passed
|
||||
- [ ] Scenario 5 (Performance) verified within limits
|
||||
- [ ] Scenario 6 (Parallel PRs) tested and passed
|
||||
- [ ] Regression testing completed
|
||||
- [ ] Bug hunting completed
|
||||
- [ ] All critical issues resolved
|
||||
- [ ] Documentation reviewed for accuracy
|
||||
|
||||
**Tester Signature**: _________________
|
||||
**Date**: _________________
|
||||
**Status**: ☐ PASS ☐ PASS WITH MINOR ISSUES ☐ FAIL
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- This manual testing plan complements automated CI/CD checks
|
||||
- Focus on user experience and real-world scenarios
|
||||
- Document any unexpected behavior, even if not blocking
|
||||
- Update this plan based on findings for future use
|
||||
|
||||
---
|
||||
|
||||
**Status**: Ready for Manual Testing
|
||||
**Last Updated**: 2026-01-10
|
||||
Reference in New Issue
Block a user