diff --git a/.vscode/tasks.json b/.vscode/tasks.json index 6e1c8e00..9e2dddbb 100644 --- a/.vscode/tasks.json +++ b/.vscode/tasks.json @@ -164,7 +164,7 @@ { "label": "Test: E2E Playwright (FireFox) - Cerberus: Security Dashboard", "type": "shell", - "command": "cd /projects/Charon && PLAYWRIGHT_HTML_OPEN=never PLAYWRIGHT_SKIP_SECURITY_DEPS=1 npx playwright test --project=firefox tests/security/security-dashboard.spec.ts", + "command": "cd /projects/Charon && PLAYWRIGHT_HTML_OPEN=never PLAYWRIGHT_SKIP_SECURITY_DEPS=1 npx playwright test --project=security-tests tests/security/security-dashboard.spec.ts", "group": "test", "problemMatcher": [], "presentation": { @@ -176,7 +176,7 @@ { "label": "Test: E2E Playwright (FireFox) - Cerberus: Rate Limiting", "type": "shell", - "command": "cd /projects/Charon && PLAYWRIGHT_HTML_OPEN=never PLAYWRIGHT_SKIP_SECURITY_DEPS=1 npx playwright test --project=firefox tests/security/rate-limiting.spec.ts", + "command": "cd /projects/Charon && PLAYWRIGHT_HTML_OPEN=never PLAYWRIGHT_SKIP_SECURITY_DEPS=1 npx playwright test --project=security-tests tests/security/rate-limiting.spec.ts", "group": "test", "problemMatcher": [], "presentation": { diff --git a/docs/plans/CI_REMEDIATION_MASTER_PLAN.md b/docs/plans/CI_REMEDIATION_MASTER_PLAN.md index 30ac8c3c..985d7a56 100644 --- a/docs/plans/CI_REMEDIATION_MASTER_PLAN.md +++ b/docs/plans/CI_REMEDIATION_MASTER_PLAN.md @@ -2,16 +2,17 @@ **Status:** πŸ”΄ **BLOCKED** - CI failures preventing releases **Created:** February 12, 2026 -**Last Updated:** February 12, 2026 +**Last Updated:** February 13, 2026 **Priority:** CRITICAL (P0) --- ## Status Overview -**Target:** 100% Pass Rate (0 failures) -**Current:** 98.3% Pass Rate (36 failures total) -**Blockers:** 8 security + 28 Chromium E2E +**Target:** 100% Pass Rate (0 failures, 0 skipped) +**Current (latest full rerun):** 1500 passed, 62 failed, 50 skipped +**Current (Phase 2 targeted Chromium rerun):** 17 passed, 1 failed +**Blockers:** Cross-browser E2E instability + unresolved skip debt + Phase 2 user lifecycle regression ### Progress Tracker @@ -20,10 +21,14 @@ - [ ] **Phase 3:** Medium-Impact E2E (6 items) - **PRIORITY 2** - Est. 3-5 hours - [ ] **Phase 4:** Low-Impact E2E (5 items) - **PRIORITY 3** - Est. 2-3 hours - [ ] **Phase 5:** Final Validation & CI Approval - **MANDATORY** - Est. 2-3 hours +- [-] **Phase 6:** Fail & Skip Census (Research) - **MANDATORY** - Est. 2-4 hours +- [ ] **Phase 7:** Failure Cluster Remediation (Execution) - **MANDATORY** - Est. 8-16 hours +- [ ] **Phase 8:** Skip Debt Burn-down & Re-enable - **MANDATORY** - Est. 4-8 hours +- [ ] **Phase 9:** Final Re-baseline & CI Gate Freeze - **MANDATORY** - Est. 2-4 hours -**Current Phase:** Phase 1 - Security Fixes -**Estimated Total Time:** 21-31 hours -**Target Completion:** Within 4-5 business days (split across team) +**Current Phase:** Phase 6 - Fail & Skip Census (skip registry created; full skip enumeration pending) +**Estimated Total Time:** 37-68 hours (including new Phases 6-9) +**Target Completion:** Within 7-10 business days (split across team) --- @@ -34,7 +39,7 @@ **Current Pass Rate:** 94.2% (65/69 tests passing) **Target:** 100% (69/69 tests passing) **Owner:** Backend Dev (API) + Frontend Dev (Imports) -**Status:** πŸ”΄ Not Started +**Status:** 🟑 In Progress --- @@ -1018,6 +1023,163 @@ git push origin fix/ci-remediation --- +## Phase 6: Fail & Skip Census (RESEARCH TRACKING) + +### Overview +**Purpose:** Create a deterministic inventory of all failures and skips from latest full rerun and map each to an owner and remediation path. +**Owner:** QA Lead + Playwright Dev +**Status:** πŸ”΄ Not Started +**Estimated Time:** 2-4 hours + +### Inputs (Latest Evidence) +- Full rerun command: + ```bash + npx playwright test --project=firefox --project=chromium --project=webkit + ``` +- Latest result snapshot: + - Passed: `1500` + - Failed: `62` + - Skipped: `50` +- Phase 2 focused Chromium result: + - Passed: `17` + - Failed: `1` (`tests/settings/user-lifecycle.spec.ts` full lifecycle test) + +### Task 6.1: Build Fail/Skip Ledger +**Output File:** `docs/reports/e2e_fail_skip_ledger_2026-02-13.md` + +**Progress:** βœ… Ledger created and committed locally. + +For each failing or skipped test, record: +- Project/browser (`chromium`, `firefox`, `webkit`) +- Test file + test title +- Failure/skip reason category +- Repro command +- Suspected root cause +- Owner (`Backend Dev`, `Frontend Dev`, `Playwright Dev`, `QA`) +- Priority (`P0`, `P1`, `P2`) + +### Task 6.2: Categorize into Clusters +Minimum clusters to track: +1. Auth/session stability (`auth-long-session`, `authentication`, onboarding) +2. Locator strictness & selector ambiguity (`modal-dropdown-triage`, long-running tasks) +3. Navigation/load reliability (`navigation`, account settings) +4. Data/empty-state assertions (`certificates`, list rendering) +5. Browser-engine specific flakiness (`webkit internal error`, detached elements) +6. Skip debt (`test.skip` or project-level skipped suites) + +**Progress:** 🟑 Skip cause registry created: `docs/reports/e2e_skip_registry_2026-02-13.md`. + +### Task 6.3: Prioritized Queue +- Generate top 15 failing tests by impact/frequency. +- Mark blockers for release path separately. +- Identify tests safe for immediate stabilization vs requiring product/contract decisions. + +### Phase 6 Exit Criteria +- [ ] Ledger created and committed +- [ ] Every fail/skip mapped to an owner and priority +- [ ] Clusters documented with root-cause hypotheses +- [ ] Top-15 queue approved for Phase 7 + +--- + +## Phase 7: Failure Cluster Remediation (EXECUTION TRACKING) + +### Overview +**Purpose:** Resolve failures by cluster, not by ad-hoc file edits, and prevent regression spread. +**Owner:** Playwright Dev + Frontend Dev + Backend Dev +**Status:** πŸ”΄ Not Started +**Estimated Time:** 8-16 hours + +### Execution Order +1. **P0 Auth/Session Cluster** + - Target files: `tests/core/auth-long-session.spec.ts`, `tests/core/authentication.spec.ts`, `tests/core/admin-onboarding.spec.ts`, `tests/settings/user-lifecycle.spec.ts` + - First action: fix context/session API misuse and deterministic re-auth flow. +2. **P1 Locator/Modal Cluster** + - Target files: `tests/modal-dropdown-triage.spec.ts`, `tests/tasks/long-running-operations.spec.ts`, related UI forms + - First action: replace broad strict-mode locators with role/name-scoped unique locators. +3. **P1 Navigation/Load Cluster** + - Target files: `tests/core/navigation.spec.ts`, `tests/settings/account-settings.spec.ts`, `tests/integration/import-to-production.spec.ts` + - First action: enforce stable route-ready checks before assertions. +4. **P2 Data/Empty-State Cluster** + - Target files: `tests/core/certificates.spec.ts` + - First action: align empty-state assertions to actual UI contract. + +### Validation Rule (Per Cluster) +- Run only affected files first. +- Then run browser matrix for those files (`chromium`, `firefox`, `webkit`). +- Then run nightly full rerun checkpoint. + +### Phase 7 Exit Criteria +- [ ] P0 cluster fully green in all browsers +- [ ] P1 clusters fully green in all browsers +- [ ] P2 cluster resolved or explicitly deferred with approved issue +- [ ] No new failures introduced in previously green files + +--- + +## Phase 8: Skip Debt Burn-down & Re-enable (TRACKING) + +### Overview +**Purpose:** Eliminate non-justified skipped tests and restore full execution coverage. +**Owner:** QA Lead + Playwright Dev +**Status:** πŸ”΄ Not Started +**Estimated Time:** 4-8 hours + +### Task 8.1: Enumerate Skip Sources +- `test.skip` annotations +- conditional skips by browser/env +- project-level skip patterns +- temporarily disabled suites + +### Task 8.2: Classify Skips +- **Valid contractual skip** (document reason and expiry) +- **Technical debt skip** (must remediate) +- **Obsolete test** (replace/remove via approved change) + +### Task 8.3: Re-enable Plan +For each technical-debt skip: +- define unblock task +- assign owner +- assign ETA +- define re-enable command + +### Phase 8 Exit Criteria +- [x] Skip registry created (`docs/reports/e2e_skip_registry_2026-02-13.md`) +- [ ] All technical-debt skips have remediation tasks +- [ ] No silent skips remain in critical suites +- [ ] Critical-path suites run with zero skips + +--- + +## Phase 9: Final Re-baseline & CI Gate Freeze + +### Overview +**Purpose:** Produce a clean baseline proving remediation completion and freeze test gates for merge. +**Owner:** QA Lead +**Status:** πŸ”΄ Not Started +**Estimated Time:** 2-4 hours + +### Required Runs +```bash +npx playwright test --project=firefox --project=chromium --project=webkit +scripts/go-test-coverage.sh +scripts/frontend-test-coverage.sh +npm run type-check +pre-commit run --all-files +``` + +### Gate Criteria +- [ ] E2E: 0 fails, 0 skips in required suites +- [ ] Coverage thresholds met + patch coverage 100% +- [ ] Typecheck/lint/security scans green +- [ ] CI workflows fully green on PR + +### Freeze Criteria +- [ ] No test-definition changes after baseline without QA approval +- [ ] New failures automatically routed to ledger process (Phase 6 template) + +--- + ### Success Criteria Summary βœ… **All checkboxes above must be checked before PR approval** @@ -1134,9 +1296,13 @@ pre-commit run --all-files | **4.2** | Admin Onboarding Tests | Playwright Dev | 1h | πŸ”΄ Not Started | Phase 3 Complete | | **4.3** | Navigation Mobile Test | Playwright Dev | 0.5h | πŸ”΄ Not Started | Phase 3 Complete | | **5.0** | Final Validation & CI | QA Lead | 2-3h | πŸ”΄ Not Started | Phases 1-4 Complete | +| **6.0** | Fail & Skip Census | QA Lead + Playwright Dev | 2-4h | πŸ”΄ Not Started | Full rerun evidence | +| **7.0** | Failure Cluster Remediation | Playwright/Frontend/Backend | 8-16h | πŸ”΄ Not Started | Phase 6 Complete | +| **8.0** | Skip Debt Burn-down | QA Lead + Playwright Dev | 4-8h | πŸ”΄ Not Started | Phase 7 Complete | +| **9.0** | Final Re-baseline Freeze | QA Lead | 2-4h | πŸ”΄ Not Started | Phase 8 Complete | -**Total Estimated Time:** 21-23 hours -**Critical Path:** Phase 1 β†’ Phase 2 β†’ Phase 3 β†’ Phase 4 β†’ Phase 5 +**Total Estimated Time:** 37-68 hours +**Critical Path:** Phase 1 β†’ Phase 2 β†’ Phase 3 β†’ Phase 4 β†’ Phase 5 β†’ Phase 6 β†’ Phase 7 β†’ Phase 8 β†’ Phase 9 ### Team Resource Allocation @@ -1312,6 +1478,7 @@ pre-commit run --all-files | Version | Date | Changes | Author | |---------|------|---------|--------| | 1.0 | 2026-02-12 | Initial plan creation | GitHub Copilot (Planning Agent) | +| 1.1 | 2026-02-13 | Added Phases 6-9 for fail/skip research, remediation tracking, skip debt burn-down, and final gate freeze; refreshed latest rerun metrics | GitHub Copilot (Management) | --- diff --git a/docs/plans/current_spec.md b/docs/plans/current_spec.md index 3419c79d..0c4a8650 100644 --- a/docs/plans/current_spec.md +++ b/docs/plans/current_spec.md @@ -1,418 +1,381 @@ --- -post_title: Pre-commit Blocker Remediation Plan +post_title: E2E Skip Retarget & Unskip Execution Plan author1: "Charon Team" -post_slug: precommit-blocker-remediation +post_slug: e2e-skip-retarget-unskip-execution-plan categories: - - infrastructure - testing + - infrastructure + - quality tags: + - playwright + - e2e - ci - - typescript - - go - - quick-fix -summary: "Quick fix plan for two critical pre-commit blockers: GolangCI-Lint version mismatch and TypeScript type errors." -post_date: "2026-02-12" + - remediation +summary: "Execution spec to move skipped suites to the correct Playwright project, remove skip directives, and enforce deterministic preconditions so tests run before failure remediation." +post_date: "2026-02-13" --- -# Pre-commit Blocker Remediation Plan +## Introduction -**Status**: Ready for Implementation -**Priority**: Critical (Blocks commits) -**Estimated Time**: 15-20 minutes -**Confidence**: 95% +This specification defines how to move currently skipped E2E suites to the correct Playwright execution environment and remove skip directives so they run deterministically. ---- +Primary objective: get all currently skipped critical-path suites executing in the right project (`security-tests` vs browser projects) with stable preconditions, even if some assertions still fail and continue into Phase 7 remediation. -## 1. Introduction +Policy update (2026-02-13): E2E must be green before QA audit. Dev agents (Backend/Frontend/Playwright) must fix missing features, product bugs, and failing tests first. -Two critical blockers prevent commits: -1. **GolangCI-Lint Configuration**: Go version mismatch (built with 1.25, project uses 1.26) -2. **TypeScript Type Check**: 13 type errors in test file `src/components/__tests__/ProxyHostForm-dropdown-changes.test.tsx` +## Research Findings -This plan provides exact commands, file changes, and verification steps to resolve both issues. +### Current skip inventory (confirmed) ---- +- `tests/manual-dns-provider.spec.ts` + - `test.describe.skip('Manual Challenge UI Display', ...)` + - `test.describe.skip('Copy to Clipboard', ...)` + - `test.describe.skip('Verify Button Interactions', ...)` + - `test.describe.skip('Manual DNS Challenge Component Tests', ...)` + - `test.describe.skip('Manual DNS Provider Error Handling', ...)` + - `test.skip('No copy buttons found - requires DNS challenge records to be visible')` + - `test.skip('should announce status changes to screen readers', ...)` +- `tests/core/admin-onboarding.spec.ts` + - test title: `Emergency token can be generated` + - inline gate: `test.skip(true, 'Cerberus must be enabled to access emergency token generation UI')` -## 2. Issue Analysis +### Playwright project routing (confirmed) -### 2.1 GolangCI-Lint Version Mismatch +- `playwright.config.js` + - `security-tests` project runs `tests/security/**` and `tests/security-enforcement/**`. + - `chromium`, `firefox`, `webkit` explicitly ignore `**/security/**` and `**/security-enforcement/**`. + - Therefore security-dependent assertions must live under security suites, not core/browser suites. -**Error Message:** -``` -Error: can't load config: the Go language version (go1.25) used to build -golangci-lint is lower than the targeted Go version (1.26) +### Existing reusable patterns (confirmed) + +- Deterministic DNS fixture data exists in `tests/fixtures/dns-providers.ts` (`mockManualChallenge`, `mockExpiredChallenge`, `mockVerifiedChallenge`). +- Deterministic creation helpers already exist in `tests/utils/TestDataManager.ts` (`createDNSProvider`) and are used in integration suites. +- Security suites already cover emergency and Cerberus behaviors (`tests/security/emergency-operations.spec.ts`, `tests/security-enforcement/emergency-token.spec.ts`). + +### Routing mismatch requiring plan action + +- `.vscode/tasks.json` contains security suite invocations using `--project=firefox` for files in `tests/security/`. +- This does not match intended project routing and can hide environment mistakes during local triage. + +## Technical Specifications + +### EARS requirements + +- WHEN a suite requires Cerberus/security enforcement, THE SYSTEM SHALL execute it under `security-tests` only. +- WHEN a suite validates UI flows not dependent on Cerberus, THE SYSTEM SHALL execute it under `chromium`, `firefox`, and `webkit` projects. +- WHEN a test previously used `describe.skip` or `test.skip` due to missing challenge state, THE SYSTEM SHALL provide deterministic preconditions so the test executes. +- IF deterministic preconditions cannot be established from existing APIs/fixtures, THEN THE SYSTEM SHALL fail the test with explicit precondition diagnostics instead of skipping. +- WHILE Phase 7 failure remediation is in progress, THE SYSTEM SHALL keep skip count at zero for targeted suites in this plan. + +### Scope boundaries + +- In scope: test routing, skip removal, deterministic setup, task/script routing consistency, validation commands. +- Out of scope: feature behavior fixes needed to make all assertions pass (handled by existing failure remediation phases). + +### Supervisor blocker list (session-mandated) + +The following blockers are mandatory and must be resolved in dev execution before QA audit starts: + +1. `auth/me` readiness failure in `tests/settings/user-lifecycle.spec.ts`. +2. Manual DNS feature wiring gap (`ManualDNSChallenge` into DNSProviders page). +3. Manual DNS test alignment/rework. +4. Security-dashboard soft-skip/skip-reason masking. +5. Deterministic sync for multi-component security propagation. + +### Explicit pre-QA green gate criteria + +QA execution is blocked until all criteria pass: + +1. Supervisor blocker list above is resolved and verified in targeted suites. +2. Targeted E2E suites show zero failures and zero unexpected skips. +3. `tests/settings/user-lifecycle.spec.ts` is green with stable `auth/me` readiness behavior. +4. Manual DNS feature wiring is present in DNSProviders page and validated by passing tests. +5. Security-dashboard skip masking is removed (no soft-skip/skip-reason masking as failure suppression). +6. Deterministic sync is validated in: + - `tests/core/multi-component-workflows.spec.ts` + - `tests/core/data-consistency.spec.ts` +7. Two consecutive targeted reruns are green before QA handoff. + +No-QA-until-green rule: + +- QA agents and QA audit tasks SHALL NOT execute until this gate passes. +- If any criterion fails, continue dev-only remediation loop and do not invoke QA. + +### Files and symbols in planned change set + +- `tests/manual-dns-provider.spec.ts` + - `test.describe('Manual DNS Provider Feature', ...)` + - skipped blocks listed above +- `tests/core/admin-onboarding.spec.ts` + - test: `Emergency token can be generated` +- `tests/security/security-dashboard.spec.ts` (or a new security-only file under `tests/security/`) + - target location for Cerberus-required emergency-token UI assertions +- `.vscode/tasks.json` + - security tasks currently using `--project=firefox` for `tests/security/*` +- Optional script normalization: + - `package.json` (`e2e:*` scripts) if dedicated security command is added + +### Data flow and environment design + +```mermaid +flowchart LR + A[setup project auth.setup.ts] --> B{Project} + B -->|chromium/firefox/webkit| C[Core/UI suites incl. manual-dns-provider] + B -->|security-tests| D[Security + security-enforcement suites] + C --> E[Deterministic DNS preconditions via fixtures/routes/API seed] + D --> F[Cerberus enabled environment] ``` -**Root Cause:** -- GolangCI-Lint binary was built with Go 1.25 -- Project's `go.mod` targets Go 1.26 -- GolangCI-Lint refuses to run when built with older Go version than target +### Deterministic preconditions (minimum required to run) -**Impact:** -- All Go linting blocked -- Cannot verify Go code quality -- Pre-commit hook fails with exit code 3 +#### Manual DNS suite -### 2.2 TypeScript Type Errors +- Precondition M1: authenticated user/session from existing fixture. +- Precondition M2: deterministic manual DNS provider presence (API create if absent via existing fixture/TestDataManager path). +- Precondition M3: deterministic challenge payload availability (use existing mock challenge fixtures and route interception where backend challenge state is non-deterministic). +- Precondition M3.1: DNS route mocks SHALL be test-scoped (inside each test case or a test-scoped helper), not shared across file scope. +- Precondition M3.2: every `page.route(...)` used for DNS challenge mocking SHALL have deterministic cleanup via `page.unroute(...)` (or equivalent scoped helper cleanup) in the same test lifecycle. +- Precondition M4: explicit page-state readiness check before assertions (`waitForLoadingComplete` + stable challenge container locator). -**File:** `frontend/src/components/__tests__/ProxyHostForm-dropdown-changes.test.tsx` +#### Admin onboarding Cerberus token path -**Error Categories:** +- Precondition C1: test must execute in security-enabled project (`security-tests`). +- Precondition C2: Cerberus status asserted from security status API or visible security dashboard state before token assertions. +- Precondition C3: if token UI not available under security-enabled environment, fail with explicit assertion message; do not skip. +- Precondition C4: moved Cerberus-token coverage SHALL capture explicit security-state snapshots both before and after test execution (pre/post) and fail if post-state drifts unexpectedly. -#### Category A: Invalid Property (Lines 92, 104) -Mock `SecurityHeaderProfile` objects use `headers: {}` property that doesn't exist in the type definition. +### No database schema/API contract change required -**Actual Type Definition** (`frontend/src/api/securityHeaders.ts`): -```typescript -export interface SecurityHeaderProfile { - id: number; - uuid: string; - name: string; - hsts_enabled: boolean; - hsts_max_age: number; - // ... (25+ security header properties) - // NO "headers" property exists -} -``` +- This plan relies on existing endpoints and fixtures; no backend schema migration is required for the retarget/unskip objective. -#### Category B: Untyped Vitest Mocks (Lines 158, 202, 243, 281, 345) -Vitest `vi.fn()` calls lack explicit type parameters, resulting in generic `Mock` type that doesn't match expected function signatures. +## Implementation Plan -**Expected Types:** -- `onSaveSuccess`: `(data: Partial) => Promise` -- `onClose`: `() => void` +### Phase 0: Iterative dev-only test loop (mandatory) ---- +This loop is owned by Backend/Frontend/Playwright agents and repeats until the pre-QA green gate passes. -## 3. Solution Specifications - -### 3.1 GolangCI-Lint Fix - -**Command:** -```bash -go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest -``` - -**What it does:** -- Downloads latest golangci-lint source -- Builds with current Go version (1.26) -- Installs to `$GOPATH/bin` or `$HOME/go/bin` - -**Verification:** -```bash -golangci-lint version -``` - -**Expected Output:** -``` -golangci-lint has version 1.xx.x built with go1.26.x from ... -``` - -### 3.2 TypeScript Type Fixes - -#### Fix 1: Remove Invalid `headers` Property - -**Lines 92, 104** - Remove the `headers: {}` property entirely from mock objects. - -**Current (BROKEN):** -```typescript -const profile = { - id: 1, - uuid: 'profile-uuid-1', - name: 'Basic Security', - description: 'Basic security headers', - is_preset: true, - preset_type: 'basic', - security_score: 60, - headers: {}, // ❌ DOESN'T EXIST IN TYPE - created_at: '2024-01-01', - updated_at: '2024-01-01', -} -``` - -**Fixed:** -```typescript -const profile = { - id: 1, - uuid: 'profile-uuid-1', - name: 'Basic Security', - description: 'Basic security headers', - is_preset: true, - preset_type: 'basic', - security_score: 60, - // headers property removed - created_at: '2024-01-01', - updated_at: '2024-01-01', -} -``` - -#### Fix 2: Add Explicit Mock Types - -**Lines 158, 202, 243, 281, 345** - Add type parameters to `vi.fn()` calls. - -**Current Pattern (BROKEN):** -```typescript -onSaveSuccess: vi.fn(), // ❌ Untyped mock -onClose: vi.fn(), // ❌ Untyped mock -``` - -**Fixed Pattern (Option 1 - Type Assertions):** -```typescript -onSaveSuccess: vi.fn() as jest.MockedFunction<(data: Partial) => Promise>, -onClose: vi.fn() as jest.MockedFunction<() => void>, -``` - -**Fixed Pattern (Option 2 - Generic Type Parameters - RECOMMENDED):** -```typescript -onSaveSuccess: vi.fn<[Partial], Promise>(), -onClose: vi.fn<[], void>(), -``` - -**Rationale for Option 2:** -- More explicit and type-safe -- Better IDE autocomplete support -- Matches Vitest conventions -- Less boilerplate than type assertions - ---- - -## 4. Implementation Steps - -### Step 1: Rebuild GolangCI-Lint +Execution commands: ```bash -# Rebuild golangci-lint with Go 1.26 -go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest +# Iteration run: blocker-focused suites +set -a && source .env && set +a +PLAYWRIGHT_COVERAGE=0 PLAYWRIGHT_HTML_OPEN=never npx playwright test \ + tests/settings/user-lifecycle.spec.ts \ + tests/manual-dns-provider.spec.ts \ + tests/core/multi-component-workflows.spec.ts \ + tests/core/data-consistency.spec.ts \ + tests/security/security-dashboard.spec.ts \ + --project=chromium --reporter=line -# Verify version -golangci-lint version +# Security-specific verification run +set -a && source .env && set +a +PLAYWRIGHT_COVERAGE=0 PLAYWRIGHT_HTML_OPEN=never npx playwright test \ + tests/security/security-dashboard.spec.ts \ + tests/security-enforcement/emergency-token.spec.ts \ + --project=security-tests --reporter=line -# Test run (should no longer error on version) -golangci-lint run ./... --timeout=5m +# Gate run (repeat twice; both must be green) +set -a && source .env && set +a +PLAYWRIGHT_COVERAGE=0 PLAYWRIGHT_HTML_OPEN=never npx playwright test \ + tests/settings/user-lifecycle.spec.ts \ + tests/manual-dns-provider.spec.ts \ + tests/core/multi-component-workflows.spec.ts \ + tests/core/data-consistency.spec.ts \ + tests/security/security-dashboard.spec.ts \ + --project=chromium --project=firefox --project=webkit --project=security-tests \ + --reporter=json > /tmp/pre-qa-green-gate.json ``` -**Expected Result:** No version error, linting runs successfully. +Enforcement: -### Step 2: Fix TypeScript Type Errors +- No QA execution until `/tmp/pre-qa-green-gate.json` confirms gate pass and the second confirmation run is also green. -**File:** `frontend/src/components/__tests__/ProxyHostForm-dropdown-changes.test.tsx` +### Phase 1: Playwright Spec Alignment (behavior contract) -**Change 1: Line 92 (Remove `headers` property)** -```typescript -// BEFORE: -const mockHeaderProfiles = [ - { - id: 1, - uuid: 'profile-uuid-1', - name: 'Basic Security', - description: 'Basic security headers', - is_preset: true, - preset_type: 'basic', - security_score: 60, - headers: {}, // REMOVE THIS LINE - created_at: '2024-01-01', - updated_at: '2024-01-01', - }, +1. Enumerate and freeze the skip baseline for targeted files using JSON reporter. +2. Confirm target ownership: + - `manual-dns-provider` => browser projects. + - Cerberus token path => `security-tests`. +3. Define run contract for each moved/unskipped block in this spec before edits. -// AFTER: -const mockHeaderProfiles = [ - { - id: 1, - uuid: 'profile-uuid-1', - name: 'Basic Security', - description: 'Basic security headers', - is_preset: true, - preset_type: 'basic', - security_score: 60, - // headers property removed - created_at: '2024-01-01', - updated_at: '2024-01-01', - }, -``` - -**Change 2: Line 104 (Remove `headers` property from second profile)** -Same change as above for the second profile in the array. - -**Change 3: Lines 158, 202, 243, 281, 345 (Add mock types)** - -Find all occurrences of: -```typescript -onSaveSuccess: vi.fn(), -onClose: vi.fn(), -``` - -Replace with: -```typescript -onSaveSuccess: vi.fn<[Partial], Promise>(), -onClose: vi.fn<[], void>(), -``` - -**Exact Line Changes:** - -**Line 158:** -```typescript -// BEFORE: - - -// Context shows this is part of a render call -// Update the mock definitions above this line: -const mockOnSubmit = vi.fn<[Partial], Promise>(); -const mockOnCancel = vi.fn<[], void>(); -``` - -Apply the same pattern for lines: 202, 243, 281, 345. - -### Step 3: Verify Fixes +Validation commands: ```bash -# Run TypeScript type check -cd /projects/Charon/frontend -npm run type-check - -# Expected: 0 errors - -# Run pre-commit checks -cd /projects/Charon -.github/skills/scripts/skill-runner.sh qa-precommit-all - -# Expected: Exit code 0 (all hooks pass) +npx playwright test tests/manual-dns-provider.spec.ts tests/core/admin-onboarding.spec.ts --project=chromium --reporter=json > /tmp/skip-contract-baseline.json +jq -r '.. | objects | select(.status? == "skipped") | [.projectName,.location.file,.title] | @tsv' /tmp/skip-contract-baseline.json ``` ---- +### Phase 2: Backend/Environment Preconditions (minimal, deterministic) -## 5. Acceptance Criteria +1. Reuse existing fixture/data helpers for manual DNS setup; do not add new backend endpoints. +2. Standardize Cerberus-enabled environment invocation for security project tests. +3. Ensure local task commands don’t misroute security suites to browser projects. -### GolangCI-Lint -- [ ] `golangci-lint version` shows built with Go 1.26.x -- [ ] `golangci-lint run` executes without version errors -- [ ] Pre-commit hook `golangci-lint-fast` passes +Potential task-level updates: -### TypeScript -- [ ] No `headers` property in mock SecurityHeaderProfile objects -- [ ] All `vi.fn()` calls have explicit type parameters -- [ ] `npm run type-check` exits with 0 errors -- [ ] Pre-commit hook `frontend-type-check` passes +- `.vscode/tasks.json` security task commands should use `--project=security-tests` when targeting files under `tests/security/` or `tests/security-enforcement/`. -### Overall -- [ ] `.github/skills/scripts/skill-runner.sh qa-precommit-all` exits code 0 -- [ ] No new type errors introduced -- [ ] All 13 TypeScript errors resolved +Validation commands: ---- - -## 6. Risk Assessment - -**Risks:** Minimal - -1. **GolangCI-Lint rebuild might fail if Go isn't installed** - - Mitigation: Check Go version first (`go version`) - - Expected: Go 1.26.x already installed - -2. **Mock type changes might break test runtime behavior** - - Mitigation: Run tests after type fixes - - Expected: Tests still pass, only types are corrected - -3. **Removing `headers` property might affect test assertions** - - Mitigation: The property was never valid, so no test logic uses it - - Expected: Tests pass without modification - -**Confidence:** 95% - ---- - -## 7. File Change Summary - -### Files Modified - -1. **`frontend/src/components/__tests__/ProxyHostForm-dropdown-changes.test.tsx`** - - Lines 92, 104: Remove `headers: {}` from mock objects - - Lines 158, 202, 243, 281, 345: Add explicit types to `vi.fn()` calls - -### Files NOT Changed - -- All Go source files (no code changes needed) -- `go.mod` (version stays at 1.26) -- GolangCI-Lint config (no changes needed) -- Other TypeScript files (errors isolated to one test file) - ---- - -## 8. Verification Commands - -### Quick Verification ```bash -# 1. Check Go version -go version - -# 2. Rebuild golangci-lint -go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest - -# 3. Verify golangci-lint version -golangci-lint version | grep "go1.26" - -# 4. Fix TypeScript errors (manual edits per Step 2) - -# 5. Run type check -cd /projects/Charon/frontend && npm run type-check - -# 6. Run full pre-commit -cd /projects/Charon -.github/skills/scripts/skill-runner.sh qa-precommit-all +npx playwright test tests/security/security-dashboard.spec.ts --project=security-tests +npx playwright test tests/security-enforcement/emergency-token.spec.ts --project=security-tests ``` -### Expected Output -``` -βœ… golangci-lint has version X.X.X built with go1.26.x -βœ… TypeScript type check: 0 errors -βœ… Pre-commit hooks: All hooks passed (exit code 0) +### Phase 3: Two-Pass Retarget + Unskip Execution + +#### Pass 1: Critical UI flow first + +1. `tests/core/admin-onboarding.spec.ts` + - remove Cerberus-gated skip path from core onboarding suite. + - keep onboarding suite browser-project-safe. +2. `tests/manual-dns-provider.spec.ts` + - unskip critical flow suites first: + - `Provider Selection Flow` + - `Manual Challenge UI Display` + - `Copy to Clipboard` + - `Verify Button Interactions` + - `Accessibility Checks` + - replace inline `test.skip` with deterministic preconditions and hard assertions. +3. Move Cerberus token assertion out of core onboarding and into security suite under `tests/security/**`. + +Pass 1 execution + checkpoint commands: + +```bash +npx playwright test tests/manual-dns-provider.spec.ts tests/core/admin-onboarding.spec.ts \ + --project=chromium --project=firefox --project=webkit \ + --grep "Provider Selection Flow|Manual Challenge UI Display|Copy to Clipboard|Verify Button Interactions|Accessibility Checks|Admin Onboarding & Setup" \ + --grep-invert "Emergency token can be generated" \ + --reporter=json > /tmp/pass1-critical-ui.json + +# Checkpoint A1: zero skip-reason annotations in targeted run +jq -r '.. | objects | select(has("annotations")) | .annotations[]? | select(.type == "skip-reason") | .description' /tmp/pass1-critical-ui.json + +# Checkpoint A2: zero skipped + did-not-run/not-run statuses in targeted run +jq -r '.. | objects | select(.status? != null and (.status|test("^(skipped|didNotRun|did-not-run|not-run|notrun)$"; "i"))) | [.status, (.title // ""), (.location.file // "")] | @tsv' /tmp/pass1-critical-ui.json ``` ---- +#### Pass 2: Component + error suites second -## 9. Time Estimates +1. `tests/manual-dns-provider.spec.ts` + - unskip and execute: + - `Manual DNS Challenge Component Tests` + - `Manual DNS Provider Error Handling` +2. Enforce per-test route mocking + cleanup for DNS mocks (`page.route` + `page.unroute` parity). -| Task | Time | -|------|------| -| Rebuild GolangCI-Lint | 2 min | -| Fix TypeScript errors (remove headers) | 3 min | -| Fix TypeScript errors (add mock types) | 5 min | -| Run verification | 5 min | -| **Total** | **~15 min** | +Pass 2 execution + checkpoint commands: ---- +```bash +npx playwright test tests/manual-dns-provider.spec.ts \ + --project=chromium --project=firefox --project=webkit \ + --grep "Manual DNS Challenge Component Tests|Manual DNS Provider Error Handling" \ + --reporter=json > /tmp/pass2-component-error.json -## 10. Next Steps After Completion +# Checkpoint B1: zero skip-reason annotations in targeted run +jq -r '.. | objects | select(has("annotations")) | .annotations[]? | select(.type == "skip-reason") | .description' /tmp/pass2-component-error.json -1. Commit fixes with message: - ``` - fix: resolve pre-commit blockers (golangci-lint + typescript) +# Checkpoint B2: zero skipped + did-not-run/not-run statuses in targeted run +jq -r '.. | objects | select(.status? != null and (.status|test("^(skipped|didNotRun|did-not-run|not-run|notrun)$"; "i"))) | [.status, (.title // ""), (.location.file // "")] | @tsv' /tmp/pass2-component-error.json - - Rebuild golangci-lint with Go 1.26 - - Remove invalid 'headers' property from SecurityHeaderProfile mocks - - Add explicit types to Vitest mock functions +# Checkpoint B3: DNS mock anti-leakage (route/unroute parity) +ROUTES=$(grep -c "page\\.route(" tests/manual-dns-provider.spec.ts || true) +UNROUTES=$(grep -c "page\\.unroute(" tests/manual-dns-provider.spec.ts || true) +echo "ROUTES=$ROUTES UNROUTES=$UNROUTES" +test "$ROUTES" -eq "$UNROUTES" +``` - Fixes 13 TypeScript errors in ProxyHostForm test - Resolves golangci-lint version mismatch - ``` +### Phase 4: Integration and Remediation Sequencing -2. Run pre-commit again to confirm: - ```bash - .github/skills/scripts/skill-runner.sh qa-precommit-all - ``` +1. Run anti-duplication guard for Cerberus token assertion: + - removed from `tests/core/admin-onboarding.spec.ts`. + - present exactly once in security suite (`tests/security/**`) only. +2. Run explicit security-state pre/post snapshot checks around moved Cerberus token coverage. +3. Re-run skip census for targeted suites and verify `skipped=0` plus `did-not-run/not-run=0` only for intended file/project pairs. +4. Ignore `did-not-run/not-run` records produced by intentionally excluded project/file combinations (for example, browser projects ignoring security suites). +5. Hand off remaining failures (if any) to existing remediation sequence: + - Phase 7: failure cluster remediation. + - Phase 8: skip debt closure check. + - Phase 9: re-baseline freeze. -3. Proceed with normal development workflow +Validation commands: ---- +```bash +npx playwright test tests/manual-dns-provider.spec.ts tests/core/admin-onboarding.spec.ts tests/security/security-dashboard.spec.ts tests/security-enforcement/emergency-token.spec.ts --project=chromium --project=firefox --project=webkit --project=security-tests --reporter=json > /tmp/retarget-unskip-validation.json -## 11. Reference Links +# Anti-duplication: Cerberus token assertion removed from core, present once in security suite only +CORE_COUNT=$(grep -RIn "Emergency token can be generated" tests/core/admin-onboarding.spec.ts | wc -l) +SEC_COUNT=$(grep -RIn --include='*.spec.ts' "Emergency token can be generated" tests/security tests/security-enforcement | wc -l) +echo "CORE_COUNT=$CORE_COUNT SEC_COUNT=$SEC_COUNT" +test "$CORE_COUNT" -eq 0 +test "$SEC_COUNT" -eq 1 -- **Blocker Report:** `docs/reports/precommit_blockers.md` -- **SecurityHeaderProfile Type:** `frontend/src/api/securityHeaders.ts` -- **Test File:** `frontend/src/components/__tests__/ProxyHostForm-dropdown-changes.test.tsx` -- **GolangCI-Lint Docs:** https://golangci-lint.run/welcome/install/ +# Security-state snapshot presence checks around moved security test +jq -r '[.. | objects | select(has("annotations")) | .annotations[]? | select(.type == "security-state-pre")] | length' /tmp/retarget-unskip-validation.json +jq -r '[.. | objects | select(has("annotations")) | .annotations[]? | select(.type == "security-state-post")] | length' /tmp/retarget-unskip-validation.json ---- +# Final JSON census (intent-scoped): skipped + did-not-run/not-run + skip-reason annotations +# - Browser projects (chromium/firefox/webkit): only non-security targeted files +# - security-tests project: only security targeted files +jq -r ' + .. + | objects + | select(.status? != null and .projectName? != null and .location.file? != null) + | select( + ( + (.projectName | test("^(chromium|firefox|webkit)$")) + and + (.location.file | test("^tests/manual-dns-provider\\.spec\\.ts$|^tests/core/admin-onboarding\\.spec\\.ts$")) + ) + or + ( + (.projectName == "security-tests") + and + (.location.file | test("^tests/security/|^tests/security-enforcement/")) + ) + ) + | select(.status | test("^(skipped|didNotRun|did-not-run|not-run|notrun)$"; "i")) + | [.projectName, .location.file, (.title // ""), .status] + | @tsv +' /tmp/retarget-unskip-validation.json +jq -r '.. | objects | select(has("annotations")) | .annotations[]? | select(.type == "skip-reason") | .description' /tmp/retarget-unskip-validation.json +``` -**Plan Status:** βœ… Ready for Implementation -**Review Status:** Pending -**Implementation Agent:** Coding Agent +### Phase 5: Documentation + CI Gate Alignment + +1. Update `docs/reports/e2e_skip_registry_2026-02-13.md` with post-retarget status. +2. Update `docs/plans/CI_REMEDIATION_MASTER_PLAN.md` Phase 8 progress checkboxes with concrete completion state. +3. Ensure CI split jobs continue to run security suites in security context and non-security suites in browser shards. + +## Risks and Mitigations + +- Risk: manual DNS challenge UI is unavailable in normal flow. + - Mitigation: deterministic route/API fixture setup to force visible challenge state for test runtime. +- Risk: duplicated emergency-token coverage across core and security suites. + - Mitigation: single source of truth in security suite; core suite retains only non-Cerberus onboarding checks. +- Risk: local task misrouting causes false confidence. + - Mitigation: update task commands to use `security-tests` for security files. + +## Acceptance Criteria + +- [ ] E2E is green before QA audit starts (hard gate). +- [ ] Dev agents fix missing features, product bugs, and failing tests first. +- [ ] Supervisor blocker list is fully resolved before QA execution. +- [ ] Iterative dev-only loop is used until gate pass is achieved. +- [ ] No QA execution occurs until pre-QA gate criteria pass. +- [ ] No `test.skip`/`describe.skip` remains in `tests/manual-dns-provider.spec.ts` and `tests/core/admin-onboarding.spec.ts` for the targeted paths. +- [ ] Cerberus-dependent emergency token test executes under `security-tests` (not browser projects). +- [ ] Manual DNS suite executes under browser projects with deterministic preconditions. +- [ ] Pass 1 (critical UI flow) completes with zero `skip-reason` annotations and zero skipped/did-not-run/not-run statuses. +- [ ] Pass 2 (component/error suites) completes with zero `skip-reason` annotations and zero skipped/did-not-run/not-run statuses. +- [ ] Cerberus token assertion is removed from `tests/core/admin-onboarding.spec.ts` and appears exactly once under `tests/security/**`. +- [ ] Moved Cerberus token test emits/validates explicit `security-state-pre` and `security-state-post` snapshots. +- [ ] DNS route mocks are per-test scoped and cleaned up deterministically (`page.route`/`page.unroute` parity). +- [ ] Any remaining failures are assertion/behavior failures only and are tracked in Phase 7 remediation queue. + +## Actionable Phase Summary + +1. Normalize routing first (security assertions in `security-tests`, browser-safe assertions in browser projects). +2. Remove skip directives in `manual-dns-provider` and onboarding emergency-token path. +3. Add deterministic preconditions (existing fixtures/routes/helpers only) so tests run consistently. +4. Re-run targeted matrix and verify `skipped=0` for targeted files. +5. Continue with Phase 7 failure remediation for remaining non-skip failures. diff --git a/docs/reports/e2e_fail_skip_ledger_2026-02-13.md b/docs/reports/e2e_fail_skip_ledger_2026-02-13.md new file mode 100644 index 00000000..e9310fde --- /dev/null +++ b/docs/reports/e2e_fail_skip_ledger_2026-02-13.md @@ -0,0 +1,85 @@ +# E2E Fail/Skip Ledger β€” 2026-02-13 + +**Phase:** 6 (Fail & Skip Census) +**Date:** 2026-02-13 +**Source command:** `npx playwright test --project=firefox --project=chromium --project=webkit` +**Latest full-suite totals:** **1500 passed**, **62 failed**, **50 skipped** +**Supporting evidence sampled:** `/tmp/playwright-full-run.txt` (failure signatures and representative failures), `tests/**/*.spec.ts` (skip sources), `playwright.config.js` (project-level execution behavior) + +--- + +## Failure Clusters + +| Browser(s) | Test file | Representative failing tests | Failure signature | Suspected root cause | Owner | Priority | Repro command | +|---|---|---|---|---|---|---|---| +| firefox, chromium | `tests/settings/user-lifecycle.spec.ts` | `Complete user lifecycle: creation to resource access`; `Deleted user cannot login`; `Session isolation after logout and re-login` | `TimeoutError: page.waitForSelector('[data-testid="dashboard-container"], [role="main"]')` | Login/session readiness race before dashboard main region is stable | Playwright Dev | P0 | `npx playwright test tests/settings/user-lifecycle.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/core/multi-component-workflows.spec.ts` | `WAF enforcement applies to newly created proxy`; `Security enforced even on previously created resources` | `TimeoutError: page.waitForSelector('[role="main"]')` | Security toggle + config propagation timing not synchronized with assertions | Playwright Dev + Backend Dev | P0 | `npx playwright test tests/core/multi-component-workflows.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/core/data-consistency.spec.ts` | `Data created via UI is properly stored and readable via API`; `Pagination and sorting produce consistent results`; `Client-side and server-side validation consistent` | Repeated long timeout failures during API↔UI consistency checks | Eventual consistency and reload synchronization gaps in tests | Playwright Dev | P0 | `npx playwright test tests/core/data-consistency.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/tasks/long-running-operations.spec.ts` | `Backup creation does not block other operations`; `Long-running task completion can be verified` | `TimeoutError: page.waitForSelector('[role="main"]')` in `beforeEach` | Setup/readiness gate too strict under background-task load | Playwright Dev | P1 | `npx playwright test tests/tasks/long-running-operations.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/core/admin-onboarding.spec.ts` | `Logout clears session`; `Re-login after logout successful` | Session/onboarding flow intermittency; conditional skip present in file | Session reset and auth state handoff not deterministic | Playwright Dev | P1 | `npx playwright test tests/core/admin-onboarding.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/core/auth-long-session.spec.ts` | `should maintain valid session for 60 minutes with token refresh`; `session should be isolated and not leak to other contexts` | Long-session / refresh assertions fail under timing variance | Token refresh and context isolation are timing-sensitive and cross-context brittle | Backend Dev + Playwright Dev | P1 | `npx playwright test tests/core/auth-long-session.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/core/domain-dns-management.spec.ts` | `Add domain to system`; `Renew SSL certificate for domain`; `Export domains configuration as JSON` | `TimeoutError` on dashboard/main selector in `beforeEach` | Shared setup readiness issue amplified in domain/DNS suite | Playwright Dev | P1 | `npx playwright test tests/core/domain-dns-management.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/modal-dropdown-triage.spec.ts` | `D. Uptime - CreateMonitorModal Type Dropdown` | `Test timeout ... keyboard.press: Target page/context/browser has been closed` | Modal close path and locator strictness under race conditions | Frontend Dev + Playwright Dev | P1 | `npx playwright test tests/modal-dropdown-triage.spec.ts --project=chromium --project=firefox` | +| firefox, chromium | `tests/settings/user-management.spec.ts` | `should copy invite link` | `expect(locator).toBeVisible() ... element(s) not found` for Copy control | Copy button locator not resilient across render states | Frontend Dev | P2 | `npx playwright test tests/settings/user-management.spec.ts --project=chromium --project=firefox --grep "copy invite link"` | +| firefox, chromium | `tests/dns-provider-types.spec.ts` | `should show script path field when Script type is selected` | `expect(locator).toBeVisible() ... element(s) not found` for script path field | Type-dependent field render timing and selector fallback mismatch | Frontend Dev | P2 | `npx playwright test tests/dns-provider-types.spec.ts --project=chromium --project=firefox --grep "Script type"` | +| firefox, chromium | `tests/core/auth-api-enforcement.spec.ts`, `tests/core/authorization-rbac.spec.ts` | Bearer token / RBAC enforcement examples from full-run failed set | Authentication/authorization assertions intermittently fail with suite instability | Upstream auth/session readiness and shared state interference | Backend Dev + Playwright Dev | P1 | `npx playwright test tests/core/auth-api-enforcement.spec.ts tests/core/authorization-rbac.spec.ts --project=chromium --project=firefox` | +| webkit (to confirm exact list next run) | Cross-cutting impacted suites | Engine-specific flakiness noted in Phase 6 planning track | Browser-engine-specific instability (pending exact test IDs) | WebKit-specific timing/render behavior and potential detached-element races | Playwright Dev | P1 | `npx playwright test --project=webkit --reporter=list` | + +--- + +## Skip Tracking + +**Current skipped total (full suite):** **50** + +### Known skip sources + +1. **Explicit `test.skip` / `describe.skip` in test code** + - `tests/manual-dns-provider.spec.ts` contains multiple `test.describe.skip(...)` blocks and individual `test.skip(...)`. + - `tests/core/admin-onboarding.spec.ts` contains conditional `test.skip(true, ...)` for Cerberus-dependent UI path. + +2. **Conditional runtime skips** + - Browser/env dependent test behavior appears in multiple suites (auth/session/security flow gating). + +3. **Project-level non-execution behavior** + - `playwright.config.js` uses dependency/ignore patterns (`skipSecurityDeps`, project `testIgnore` for security suites on browser projects). + - Full-run artifacts can include `did not run` counts in addition to explicit skips. + +### Actions to enumerate exact skip list on next run + +- Run with machine-readable reporter and archive artifact: + - `npx playwright test --project=firefox --project=chromium --project=webkit --reporter=json > /tmp/e2e-full-2026-02-13.json` +- Extract exact skipped tests with reason and browser: + - `jq -r '.. | objects | select(.status? == "skipped") | [.projectName,.location.file,.title,.annotations] | @tsv' /tmp/e2e-full-2026-02-13.json` +- Produce canonical skip registry from the JSON output: + - `docs/reports/e2e_skip_registry_2026-02-13.md` +- Add owner + expiration date for each non-contractual skip before Phase 8 re-enable work. + +--- + +## Top-15 Remediation Queue (Release impact Γ— fixability) + +| Rank | Test / Scope | Browser(s) | Impact | Fixability | Owner | Priority | Immediate next action | +|---:|---|---|---|---|---|---|---| +| 1 | `tests/settings/user-lifecycle.spec.ts` β€” `Complete user lifecycle: creation to resource access` | chromium, firefox | Critical auth/user-flow gate | High | Playwright Dev | P0 | Add deterministic dashboard-ready wait helper and apply to suite `beforeEach` | +| 2 | `tests/settings/user-lifecycle.spec.ts` β€” `Deleted user cannot login` | chromium, firefox | Security correctness | High | Playwright Dev | P0 | Wait on delete response + auth state settle before login assertion | +| 3 | `tests/settings/user-lifecycle.spec.ts` β€” `Session isolation after logout and re-login` | chromium, firefox | Session integrity | Medium | Playwright Dev | P0 | Explicitly clear and verify storage/session before re-login step | +| 4 | `tests/core/multi-component-workflows.spec.ts` β€” `WAF enforcement applies...` | chromium, firefox | Security enforcement contract | Medium | Backend Dev + Playwright Dev | P0 | Gate assertions on config-reload completion signal | +| 5 | `tests/core/multi-component-workflows.spec.ts` β€” `Security enforced even on previously created resources` | chromium, firefox | Security regression risk | Medium | Backend Dev + Playwright Dev | P0 | Add module-enabled verification helper before traffic checks | +| 6 | `tests/core/data-consistency.spec.ts` β€” `Data created via UI ... readable via API` | chromium, firefox | Core CRUD integrity | Medium | Playwright Dev | P0 | Introduce API-response synchronization checkpoints | +| 7 | `tests/core/data-consistency.spec.ts` β€” `Data deleted via UI is removed from API` | chromium, firefox | Data correctness | Medium | Playwright Dev | P0 | Verify deletion response then poll API until terminal state | +| 8 | `tests/core/data-consistency.spec.ts` β€” `Pagination and sorting produce consistent results` | chromium, firefox | User trust in data views | High | Playwright Dev | P0 | Stabilize table wait + deterministic sort verification | +| 9 | `tests/tasks/long-running-operations.spec.ts` β€” `Backup creation does not block other operations` | chromium, firefox | Background task reliability | Medium | Playwright Dev | P1 | Replace fixed waits with condition-based readiness checks | +| 10 | `tests/tasks/long-running-operations.spec.ts` β€” `Long-running task completion can be verified` | chromium, firefox | Operational correctness | Medium | Playwright Dev | P1 | Wait for terminal task-state API response before UI assert | +| 11 | `tests/core/admin-onboarding.spec.ts` β€” `Logout clears session` | chromium, firefox | Login/session contract | High | Playwright Dev | P1 | Ensure logout request completion + redirect settle criteria | +| 12 | `tests/core/auth-long-session.spec.ts` β€” `maintain valid session for 60 minutes` | chromium, firefox | Auth platform stability | Low-Medium | Backend Dev + Playwright Dev | P1 | Isolate token-refresh assertions and instrument refresh timeline | +| 13 | `tests/modal-dropdown-triage.spec.ts` β€” `CreateMonitorModal Type Dropdown` | chromium, firefox | Key form interaction | High | Frontend Dev | P1 | Harden locator strategy and modal-close sequencing | +| 14 | `tests/settings/user-management.spec.ts` β€” `should copy invite link` | chromium, firefox | Invitation UX | High | Frontend Dev | P2 | Provide stable copy-control locator and await render completion | +| 15 | `tests/dns-provider-types.spec.ts` β€” `script path field when Script type selected` | chromium, firefox | Provider config UX | High | Frontend Dev | P2 | Align field visibility assertion with selected provider type state | + +--- + +## Operational Notes + +- This ledger is Phase 6 tracking output and should be updated after each full-suite rerun. +- Next checkpoint: attach exact fail + skip lists from JSON reporter output and reconcile against this queue. +- Phase handoff dependency: Queue approval unlocks Phase 7 cluster remediation execution. diff --git a/docs/reports/e2e_skip_registry_2026-02-13.md b/docs/reports/e2e_skip_registry_2026-02-13.md new file mode 100644 index 00000000..42142e12 --- /dev/null +++ b/docs/reports/e2e_skip_registry_2026-02-13.md @@ -0,0 +1,183 @@ +# E2E Skip Registry (2026-02-13) + +## Objective + +Determine why tests are skipped and classify each skip source as one of: + +- Wrong environment/configuration +- Product bug +- Missing feature/test preconditions +- Intentional test routing (non-bug) + +## Evidence Sources + +1. Full rerun baseline (previous run): `1500 passed / 62 failed / 50 skipped` +2. Targeted runtime census (Chromium): + +```bash +set -a && source .env && set +a && \ +PLAYWRIGHT_COVERAGE=0 PLAYWRIGHT_HTML_OPEN=never \ +npx playwright test tests/manual-dns-provider.spec.ts tests/core/admin-onboarding.spec.ts \ + --project=chromium --reporter=json > /tmp/skip-census-targeted.json 2>&1 +``` + +3. Static skip directive census in tests: + +```bash +grep -RInE "test\\.skip|describe\\.skip|test\\.fixme|describe\\.fixme" tests/ +``` + +4. Project routing behavior from `playwright.config.js`. + +## Confirmed Skip Sources + +### 1) Manual DNS provider suite skips (Confirmed) + +- File: `tests/manual-dns-provider.spec.ts` +- Runtime evidence (Chromium targeted run): `16 skipped` +- Skip type: explicit `test.describe.skip(...)` and `test.skip(...)` +- Classification: **Missing feature/test preconditions (technical debt skip)** +- Why: + - Tests require deterministic DNS challenge records and UI states that are not guaranteed in default E2E flow. + - One skip reason is explicitly tied to absent visible challenge records (`No copy buttons found - requires DNS challenge records to be visible`). +- Owner: **Playwright Dev + Frontend Dev** +- Priority: **P0 for critical-path coverage, P1 for full suite parity** +- Recommended action: + - Create deterministic fixtures/seed path for manual DNS challenge state. + - Re-enable blocks incrementally and validate across all three browser projects. + +### 2) Conditional Cerberus skip in admin onboarding (Confirmed source, condition-dependent runtime) + +- File: `tests/core/admin-onboarding.spec.ts` +- Skip directive: `test.skip(true, 'Cerberus must be enabled to access emergency token generation UI')` +- Classification: **Wrong environment/configuration (when triggered)** +- Why: + - This is a hard environment gate. If Cerberus is disabled or inaccessible, test intentionally skips. +- Owner: **QA + Backend Dev** +- Priority: **P1** +- Recommended action: + - Split tests into: + - Cerberus-required suite (explicit env contract), and + - baseline onboarding suite (no Cerberus dependency). + - Add preflight assertion that reports config mismatch clearly instead of silent skip where possible. + +### 3) Security project routing behavior (Intentional, non-bug) + +- Source: `playwright.config.js` +- Behavior: + - Browser projects (`chromium`, `firefox`, `webkit`) use `testIgnore` for `**/security-enforcement/**` and `**/security/**`. + - Security coverage is handled by dedicated `security-tests` project. +- Classification: **Intentional test routing (non-bug)** +- Why: + - Prevents security suite execution duplication in standard browser projects. +- Owner: **QA** +- Priority: **P2 (documentation only)** +- Recommended action: + - Keep as-is; ensure CI includes explicit `security-tests` project execution in required checks. + +## Current Assessment + +Based on available runtime and source evidence, most observed skips are currently **intentional skip directives in manual DNS provider tests** rather than emergent engine bugs. + +### Distribution (current confirmed) + +- **Missing feature/preconditions debt:** High (manual DNS blocks) +- **Environment-gated skips:** Present (Cerberus-gated onboarding path) +- **Product bug-derived skips:** Not yet confirmed from current skip evidence +- **Config/routing-intentional non-runs:** Present and expected (security project separation) + +## Actions to Close Phase 8.1 + +1. Export full multi-project JSON report and enumerate all `status=skipped` tests with file/title/annotations. +2. Map every skipped test to one of the four classes above. +3. Open remediation tasks for all technical-debt skips (manual DNS first). +4. Define explicit re-enable criteria and target command per skip cluster. + +## Re-enable Queue (Initial) + +1. `tests/manual-dns-provider.spec.ts` skipped blocks + - Unblock by deterministic challenge fixture + stable locators + - Re-enable command: + + ```bash + npx playwright test tests/manual-dns-provider.spec.ts --project=chromium --project=firefox --project=webkit + ``` + +2. Cerberus-gated onboarding checks + - Unblock by environment contract enforcement or test split + - Re-enable command: + + ```bash + npx playwright test tests/core/admin-onboarding.spec.ts --project=chromium --project=firefox --project=webkit + ``` + +## Exit Criteria for This Registry + +- [x] Confirmed dominant skip source with runtime evidence +- [x] Classified skips into environment vs missing feature/test debt vs routing-intentional +- [ ] Full-suite skip list fully enumerated from JSON (all 50) +- [ ] Owner + ETA assigned per skipped test block + +## Post-Edit Validation Status (Phase 3 + relevant Phase 4) + +### Applied changes + +- `tests/manual-dns-provider.spec.ts` + - Removed targeted `describe.skip` / `test.skip` usage so suites execute. + - Added deterministic preconditions using existing DNS fixtures (`mockManualChallenge`, `mockExpiredChallenge`, `mockVerifiedChallenge`). + - Added test-scoped route mocks with cleanup parity (`page.route` + `page.unroute`). +- `tests/core/admin-onboarding.spec.ts` + - Removed Cerberus-dependent `Emergency token can be generated` from browser-safe core onboarding suite. +- `tests/security/security-dashboard.spec.ts` + - Added `Emergency token can be generated` under security suite ownership. + - Added `security-state-pre` / `security-state-post` annotations and pre/post state drift checks. + +### Concrete command results + +1. **Pass 1** + +```bash +npx playwright test tests/manual-dns-provider.spec.ts tests/core/admin-onboarding.spec.ts \ + --project=chromium --project=firefox --project=webkit \ + --grep "Provider Selection Flow|Manual Challenge UI Display|Copy to Clipboard|Verify Button Interactions|Accessibility Checks|Admin Onboarding & Setup" \ + --grep-invert "Emergency token can be generated" --reporter=json +``` + +- Parsed stats: `expected=43`, `unexpected=30`, `skipped=0` +- Intent-scoped skip census (`chromium|firefox|webkit` + targeted files): **0 skipped / 0 did-not-run** +- `skip-reason` annotations in this run: **0** + +2. **Pass 2** + +```bash +npx playwright test tests/manual-dns-provider.spec.ts \ + --project=chromium --project=firefox --project=webkit \ + --grep "Manual DNS Challenge Component Tests|Manual DNS Provider Error Handling" --reporter=json +``` + +- Parsed stats: `expected=1`, `unexpected=15`, `skipped=0` +- Intent-scoped skip census (`chromium|firefox|webkit` + manual DNS file): **0 skipped / 0 did-not-run** +- `skip-reason` annotations in this run: **0** + +3. **Security-suite ownership + anti-duplication** + +```bash +npx playwright test tests/security/security-dashboard.spec.ts \ + --project=security-tests --grep "Emergency token can be generated" --reporter=json +``` + +- Parsed stats: `unexpected=0`, `skipped=0` +- Raw JSON evidence confirms `projectName: security-tests` for emergency token test execution. +- `security-state-pre` and `security-state-post` annotations captured. +- Anti-duplication check: + - `CORE_COUNT=0` in `tests/core/admin-onboarding.spec.ts` + - `SEC_COUNT=1` across `tests/security/**` + `tests/security-enforcement/**` + +4. **Route mock cleanup parity** + +- `tests/manual-dns-provider.spec.ts`: `ROUTES=3`, `UNROUTES=3`. + +### Residual failures (for Phase 7) + +- Skip debt objective for targeted scopes is met (`skipped=0` and `did-not-run=0` in intended combinations). +- Remaining failures are assertion/behavior failures in manual DNS and onboarding flows and should proceed to Phase 7 remediation.