diff --git a/docs/plans/playright_remidiation_2026.02.04.md b/docs/plans/playright_remidiation_2026.02.04.md new file mode 100644 index 00000000..38f17a96 --- /dev/null +++ b/docs/plans/playright_remidiation_2026.02.04.md @@ -0,0 +1,81 @@ +## CI Test Validation Summary (Run #21695576947) ## + +✅ Test Reorganization Working Correctly + +The test isolation strategy is functioning as designed: +- No cross-shard contamination: Security enforcement tests are properly isolated in dedicated jobs +- Cerberus ON/OFF working: Non-security shards show no evidence of unexpected ACL/rate limit blocks +- Emergency token validated: Consistent across all shards (290afd29...0871) + +## ❌ Issues Found (Not Related to Reorganization) ## + +# 1. CRITICAL: Browser Installation Missing (Firefox & WebKit Security Jobs) # +- Impact: 270 failures (135 per browser) +- Cause: Missing npx playwright install step in security enforcement jobs +- Solution: Add installation step to match non-security jobs +- Files: All tests run in Firefox/WebKit security jobs + +# 2. HIGH: Shard 4/4 Timeout (All Browsers) # +- Impact: 3 jobs timing out at 20 minutes +- Cause: Unbalanced test distribution +- Test Distribution: + - Shard 1: ~4-5 minutes + - Shard 2: 8-13 minutes + - Shard 3: 8-11 minutes + - Shard 4: >20 minutes (TIMEOUT) +- Solution: Reduce from 4 to 3 shards per browser + +# 3. PRE-EXISTING: Consistent Test Failures (Shard 1, All Browsers) # +These failures appeared identically across Chromium, Firefox, and WebKit: + +certificates.spec.ts (16 failures total): + +``` +Line 104: "should display empty state when no certificates exist"   + expect(hasEmptyMessage || hasTable).toBeTruthy() received false + +Line 120: "should show loading spinner while fetching data" expect(hasTable || hasEmpty).toBeTruthy() received false +``` + +proxy-hosts.spec.ts (16 failures total): + +``` +Lines 158, 176, 202, 221: Multiple "waitForModal" errors   + waitForModal: Could not find modal dialog or slide-out panel matching "undefined"  at  + utils/wait-helpers.ts:413 + +``` + +Assessment: These are pre-existing UI issues, not caused by test reorganization. + +# 4. FLAKY: crowdsec-import.spec.ts # +- Test: "should handle archive with optional files (acquis.yaml)" +- Frequency: Appears in almost every successful shard across all browsers +- Error: expect(response.ok()).toBeTruthy() received false +- Assessment: Flaky test, needs investigation or retry logic +- Browser-Specific Failures +- Firefox: API key regeneration (line 705), audit-logs user column visibility +- WebKit: page.goto() internal errors in audit-logs, crowdsec-enrollment, account-settings, real-time-logs + +## Local Test Comparison ## +I attempted to extract your local test results but the HTML report format is too complex to parse via command line. However, based on the CI patterns: + +Expected local results should show: + +✅ Same 16certifictes and proxy-hosts failures (if running all browsers) +✅ Same flaky crowdsec-import test +✅ Browser-specific failures only if running that browser locally + +If your local results differ, it would indicate: + +- Tests miscategorized (security vs non-security) +- Environment differences (Docker vs local) +- Timing/race conditions + +## Recommended Actions ## +- Immediate Fix: Add browser installation to security enforcement jobs +- Rebalance Shards: Reduce to 3 shards per browser (12 total jobs) +- Address Pre-existing Failures: Fix certificates and proxy-hosts tests in separate PR +- Fix Flaky Test: Investigate crowdsec-import or add retry logic + +Validation: The test reorganization is correct - failures are consistent with pre-existing issues, not artifacts of the Cerberus ON/OFF strategy.