akanealw/Charon

Fork 0

Files

GitHub Actions 27c252600a chore: git cache cleanup

2026-03-04 18:34:49 +00:00

12 KiB

Raw Blame History

Phase 1 Validation Checklist

Date: February 2, 2026 Status: Ready for Validation Phase: Emergency Hotfix + Deep Diagnostics

Pre-Deployment Validation

1. File Integrity Check

.github/workflows/e2e-tests-split.yml created (34KB)
.github/workflows/e2e-tests.yml.backup created (26KB backup)
docs/reports/phase1_analysis.md created (3.8KB)
docs/reports/phase1_diagnostics.md created (18KB)
docs/reports/phase1_complete.md created (11KB)
tests/utils/diagnostic-helpers.ts created (9.7KB)

2. Workflow YAML Validation

# Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/e2e-tests-split.yml'))"
# ✅ PASSED: Workflow YAML syntax is valid

3. Workflow Structure Validation

Expected Jobs:

build - Build Docker image once
e2e-chromium - 4 shards, independent execution
e2e-firefox - 4 shards, independent execution
e2e-webkit - 4 shards, independent execution
upload-coverage - Merge and upload per-browser coverage
test-summary - Generate summary report
comment-results - Post PR comment
e2e-results - Final status check

Total Jobs: 8 (vs 7 in original workflow)

4. Browser Isolation Validation

Dependency Tree:

build
 ├─ e2e-chromium (independent)
 ├─ e2e-firefox (independent)
 └─ e2e-webkit (independent)
      └─ upload-coverage (needs all 3)
           └─ test-summary
                └─ comment-results
                     └─ e2e-results

Validation:

No dependencies between browser jobs
All browsers depend only on build
Chromium failure cannot block Firefox/WebKit
Each browser runs 4 shards in parallel

5. Coverage Strategy Validation

Expected Artifacts:

e2e-coverage-chromium-shard-{1..4} (4 artifacts)
e2e-coverage-firefox-shard-{1..4} (4 artifacts)
e2e-coverage-webkit-shard-{1..4} (4 artifacts)
e2e-coverage-merged (1 artifact with all browsers)

Expected Codecov Flags:

e2e-chromium flag
e2e-firefox flag
e2e-webkit flag

Expected Reports:

playwright-report-{browser}-shard-{1..4} (12 HTML reports)

Local Validation (Pre-Push)

Step 1: Lint Workflow File

# GitHub Actions YAML linter
docker run --rm -v "$PWD:/repo" rhysd/actionlint:latest -color /repo/.github/workflows/e2e-tests-split.yml

Expected: No errors or warnings

Step 2: Test Playwright with Split Projects

# Test Chromium only
npx playwright test --project=chromium --shard=1/4

# Test Firefox only
npx playwright test --project=firefox --shard=1/4

# Test WebKit only
npx playwright test --project=webkit --shard=1/4

# Verify no cross-contamination

Expected: Each browser runs independently without errors

Step 3: Verify Diagnostic Helpers

# Run TypeScript compiler
npx tsc --noEmit tests/utils/diagnostic-helpers.ts

# Expected: No type errors

Expected: Clean compilation (0 errors)

Step 4: Simulate CI Environment

# Rebuild E2E container
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e

# Wait for health check
curl -sf http://localhost:8080/api/v1/health

# Run with CI settings
CI=1 npx playwright test --project=chromium --workers=1 --retries=2 --shard=1/4

Expected: Tests run in CI mode without interruptions

CI Validation (Post-Push)

Step 1: Create Feature Branch

# Create feature branch for Phase 1 hotfix
git checkout -b phase1-browser-split-hotfix

# Add files
git add .github/workflows/e2e-tests-split.yml \
        .github/workflows/e2e-tests.yml.backup \
        docs/reports/phase1_*.md \
        tests/utils/diagnostic-helpers.ts

# Commit with descriptive message
git commit -m "feat(ci): Phase 1 - Split browser jobs for complete isolation

- Split e2e-tests into 3 independent jobs (chromium, firefox, webkit)
- Add per-browser coverage upload with flags (e2e-{browser})
- Create diagnostic helpers for root cause analysis
- Document Phase 1 investigation findings

Fixes: Browser interruptions blocking downstream tests
See: docs/plans/browser_alignment_triage.md Phase 1
Related: PR #609"

# Push to remote
git push origin phase1-browser-split-hotfix

Step 2: Create Pull Request

PR Title: [Phase 1] Emergency Hotfix: Split Browser Jobs for Complete Isolation

PR Description:

## Phase 1: Browser Alignment Triage - Emergency Hotfix

### Problem
Chromium test interruption at test #263 blocks Firefox/WebKit from executing.
Only 10% of E2E tests (263/2,620) were running in CI.

### Solution
Split browser tests into 3 completely independent jobs:
- `e2e-chromium` (4 shards)
- `e2e-firefox` (4 shards)
- `e2e-webkit` (4 shards)

### Benefits
- ✅ **Complete Browser Isolation:** Chromium failure cannot block Firefox/WebKit
- ✅ **Parallel Execution:** All browsers run simultaneously (faster CI)
- ✅ **Independent Failure Analysis:** Each browser has separate HTML reports
- ✅ **Per-Browser Coverage:** Separate flags for Codecov (e2e-chromium, e2e-firefox, e2e-webkit)

### Changes
1. **New Workflow:** `.github/workflows/e2e-tests-split.yml`
   - 3 independent browser jobs (no cross-dependencies)
   - Per-browser coverage upload with flags
   - Enhanced diagnostic logging

2. **Diagnostic Tools:** `tests/utils/diagnostic-helpers.ts`
   - Browser console logging
   - Page state capture
   - Dialog lifecycle tracking
   - Performance monitoring

3. **Documentation:**
   - `docs/reports/phase1_analysis.md` - Test execution order analysis
   - `docs/reports/phase1_diagnostics.md` - Root cause investigation (18KB)
   - `docs/reports/phase1_complete.md` - Phase 1 completion report

### Testing
- [x] YAML syntax validated
- [ ] All 3 browser jobs execute independently in CI
- [ ] Coverage artifacts upload with correct flags
- [ ] Chromium failure does not block Firefox/WebKit

### Next Steps
- Phase 2: Fix root cause (replace `page.waitForTimeout()` anti-patterns)
- Phase 3: Improve coverage to 85%+
- Phase 4: Consolidate back to single job after fix validated

### References
- Triage Plan: `docs/plans/browser_alignment_triage.md`
- Diagnostic Report: `docs/reports/browser_alignment_diagnostic.md`
- Related Issue: #609 (E2E tests blocking PR merge)

Step 3: Monitor CI Execution

Check GitHub Actions:

Navigate to Actions tab → E2E Tests (Split Browsers) workflow
Verify all 8 jobs appear:
- build (1 job)
- e2e-chromium (4 shards)
- e2e-firefox (4 shards)
- e2e-webkit (4 shards)
- upload-coverage (if enabled)
- test-summary
- comment-results
- e2e-results

Expected Behavior:

Build completes in ~5 minutes
All browser shards start simultaneously (after build)
Each shard uploads HTML report on completion
Coverage artifacts uploaded (if PLAYWRIGHT_COVERAGE=1)
Summary comment posted to PR

Step 4: Verify Browser Isolation

Test Chromium Failure Scenario:

Temporarily add test.fail() to a Chromium-only test
Push change and observe CI behavior
Expected: Chromium jobs fail, Firefox/WebKit continue

Validation Command:

# Check workflow run status
gh run view <run-id> --log

# Expected output:
# - e2e-chromium: failure (expected)
# - e2e-firefox: success
# - e2e-webkit: success
# - e2e-results: failure (as expected, Chromium failed)

Step 5: Verify Coverage Upload

Check Codecov Dashboard:

Navigate to Codecov dashboard for the repository
Go to the commit/PR page
Verify flags appear:
- e2e-chromium flag with coverage %
- e2e-firefox flag with coverage %
- e2e-webkit flag with coverage %

Expected:

3 separate flag entries in Codecov
Each flag shows independent coverage percentage
Combined E2E coverage matches or exceeds original

Post-Deployment Validation

Step 1: Monitor PR #609

Expected Behavior:

E2E tests execute for all 3 browsers
No "did not run" status for Firefox/WebKit
Per-shard HTML reports available for download
PR comment shows all 3 browser results

Step 2: Analyze Test Results

Download Artifacts:

playwright-report-chromium-shard-{1..4} (4 reports)
playwright-report-firefox-shard-{1..4} (4 reports)
playwright-report-webkit-shard-{1..4} (4 reports)

Verify:

Each browser ran >800 tests (not 0)
No interruptions detected (check traces)
Shard execution times < 15 minutes each
HTML reports contain test details

Step 3: Validate Coverage Merge

If PLAYWRIGHT_COVERAGE=1 enabled:

Download e2e-coverage-merged artifact
Verify chromium/lcov.info exists
Verify firefox/lcov.info exists
Verify webkit/lcov.info exists
Check Codecov dashboard for 3 flags

If coverage disabled:

No coverage artifacts uploaded
upload-coverage job skipped
No Codecov updates

Rollback Plan

If Phase 1 hotfix causes issues:

Option 1: Revert to Original Workflow

# Restore backup
cp .github/workflows/e2e-tests.yml.backup .github/workflows/e2e-tests.yml

# Commit revert
git add .github/workflows/e2e-tests.yml
git commit -m "revert(ci): rollback to original E2E workflow

Phase 1 hotfix caused issues. Restoring original workflow
while investigating alternative solutions.

See: docs/reports/phase1_rollback.md"

git push origin phase1-browser-split-hotfix

Option 2: Disable Specific Browser

If one browser has persistent issues:

# Add to workflow
jobs:
  e2e-firefox:
    # Temporarily disable Firefox until root cause identified
    if: false

Option 3: Merge Shards

If sharding causes resource contention:

strategy:
  matrix:
    shard: [1]  # Change from [1, 2, 3, 4] to [1]
    total-shards: [1]  # Change from [4] to [1]

Success Criteria

Must Have (Blocking)

Workflow YAML syntax valid
All 3 browser jobs defined
No dependencies between browser jobs
Documentation complete
CI executes all 3 browsers (verify in PR)
Chromium failure does not block Firefox/WebKit (verify in PR)

Should Have (Important)

Per-browser coverage upload configured
Diagnostic helpers created
Backup of original workflow
PR comment shows all 3 browser results (verify in PR)
HTML reports downloadable per shard (verify in PR)

Nice to Have (Optional)

Coverage flags visible in Codecov dashboard
Performance improvement measured (parallel execution)
Phase 2 plan approved by team

Next Steps After Validation

If Validation Passes ✅

Merge Phase 1 PR
- Squash commits or keep history (team preference)
- Update PR #609 to use new workflow
Begin Phase 2
- Create tests/utils/wait-helpers.ts
- Refactor interrupted tests in certificates.spec.ts
- Code review checkpoint after first 2 files
Monitor Production
- Watch for new interruptions
- Track test execution times
- Monitor CI resource usage

If Validation Fails ❌

Analyze Failure
- Download workflow logs
- Check job dependencies
- Verify environment variables
Apply Fix
- Update workflow configuration
- Re-run validation checklist
- Document issue in phase1_rollback.md
Escalate if Needed
- If fix not obvious, revert to original workflow
- Document issues for team discussion
- Schedule Phase 1 retrospective

Approval Sign-Off

Phase 1 Deliverables Validated:

DevOps Lead
QA Lead
Engineering Manager

Date: _________________

Ready for Deployment: YES / NO

Document Control: Version: 1.0 Last Updated: February 2, 2026 Status: Ready for Validation Next Review: After CI validation in PR

12 KiB Raw Blame History