Charon/docs/reports/archive/browser_alignment_diagnostic.md

# Browser Alignment Diagnostic Report
**Date:** February 2, 2026
**Mission:** Comprehensive E2E test analysis across Chromium, Firefox, and WebKit
**Environment:** Local Docker E2E container (charon-e2e)
**Base URL:** http://localhost:8080

---

## Executive Summary

**🔴 CRITICAL FINDING: Firefox and WebKit tests did not execute**

Out of 2,620 total tests across all browser projects:
- **Chromium:** 263 tests executed (234 passed, 2 interrupted, 27 skipped)
- **Firefox:** 0 tests executed (873 tests queued but never started)
- **WebKit:** 0 tests executed (873 tests queued but never started)
- **Skipped/Not Run:** 2,357 tests total

This represents a **90% test execution failure** for non-Chromium browsers, explaining CI discrepancies between local and GitHub Actions results.

---

## Detailed Findings

### 1. Playwright E2E Test Results

#### Environment Validation
✅ **E2E Container Status:** Healthy
✅ **Port Accessibility:**
- Application (8080): ✓ Accessible
- Emergency API (2020): ✓ Healthy
- Caddy Admin (2019): ✓ Healthy

✅ **Emergency Token:** Validated (64 chars, valid hexadecimal)
✅ **Authentication State:** Setup completed successfully
✅ **Global Setup:** Orphaned data cleanup completed

#### Chromium Test Results (Desktop Chrome)
**Project:** chromium
**Status:** Partially completed (interrupted)
**Tests Run:** 263 total
- ✅ **Passed:** 234 tests (6.3 minutes)
- ⚠️ **Interrupted:** 2 tests
  - `tests/core/certificates.spec.ts:788` - Form Accessibility › keyboard navigation
  - `tests/core/certificates.spec.ts:807` - Form Accessibility › Escape key handling
- ⏭️ **Skipped:** 27 tests
- ❌ **Did Not Run:** 2,357 tests (remaining from Firefox/WebKit projects)

**Interrupted Test Details:**
```
Error: browserContext.close: Target page, context or browser has been closed
Error: page.waitForTimeout: Test ended
```

**Sample Passed Tests:**
- Security Dashboard (all ACL, WAF, Rate Limiting, CrowdSec tests)
- Security Headers Configuration (12/12 tests)
- WAF Configuration (16/16 tests)
- ACL Enforcement (security-tests project)
- Emergency Token Break Glass Protocol (8/8 tests)
- Access Lists CRUD Operations (53/53 tests visible)
- SSL Certificates CRUD Operations (partial)
- Audit Logs (16/16 tests)

**Coverage Collection:** Enabled (`@bgotink/playwright-coverage`)

#### Firefox Test Results (Desktop Firefox)
**Project:** firefox
**Status:** ❌ **NEVER STARTED**
**Tests Expected:** ~873 tests (estimated based on chromium × 3 browsers)
**Tests Run:** 0
**Dependency Chain:** setup → security-tests → security-teardown → firefox

**Observation:** When explicitly running Firefox project tests:
```bash
playwright test --project=setup --project=security-tests --project=security-teardown --project=firefox
```
Result: Tests BEGIN execution (982 tests queued, 2 workers allocated), but in the full test suite run, Firefox tests are marked as "did not run."

**Hypothesis:** Possible causes:
1. **Timeout During Chromium Tests:** Chromium tests take 6.3 minutes; if the overall test run times out before reaching Firefox, subsequent browser projects never execute.
2. **Interrupted Dependency:** If `security-teardown` or `chromium` project encounters a critical error, dependent projects (firefox, webkit) may be skipped.
3. **CI vs Local Configuration Mismatch:** Different timeout settings or resource constraints in GitHub Actions may cause earlier interruption.

#### WebKit Test Results (Desktop Safari)
**Project:** webkit
**Status:** ❌ **NEVER STARTED**
**Tests Expected:** ~873 tests
**Tests Run:** 0
**Dependency Chain:** setup → security-tests → security-teardown → webkit

**Same behavior as Firefox:** Tests are queued but never executed in the full suite.

---

### 2. Backend Test Coverage

**Script:** `./scripts/go-test-coverage.sh`
**Status:** ✅ Completed successfully

**Coverage Metrics:**
- **Overall Coverage:** 84.9%
- **Required Threshold:** 85.0%
- **Gap:** -0.1% (BELOW THRESHOLD ⚠️)

**Sample Package Coverage:**
- `pkg/dnsprovider/custom`: 97.5% ✅
- Various modules: Range from 70%-99%

**Filtered Packages:** Excluded packages (vendor, mocks) removed from report

**Recommendation:** Add targeted unit tests to increase coverage by 0.1%+ to meet threshold.

---

### 3. Frontend Test Coverage

**Script:** `npm test -- --run --coverage` (Vitest)
**Status:** ✅ Completed successfully

**Coverage Metrics:**
- **Overall Coverage:** 84.22% (statements)
- **Branch Coverage:** 77.39%
- **Function Coverage:** 79.29%
- **Line Coverage:** 84.81%

**Module Breakdown:**
- `src/api`: 88.45% ✅
- `src/components`: 88.77% ✅
- `src/hooks`: 99.52% ✅ (excellent)
- `src/pages`: 82.59% ⚠️ (needs attention)
  - `Security.tsx`: 65.17% ❌ (lowest)
  - `SecurityHeaders.tsx`: 69.23% ⚠️
  - `Plugins.tsx`: 63.63% ❌
- `src/utils`: 96.49% ✅

**Localization Files:** 0% (expected - JSON translation files not covered by tests)

**Recommendation:** Focus on increasing coverage for `Security.tsx`, `SecurityHeaders.tsx`, and `Plugins.tsx` pages.

---

## Browser-Specific Discrepancies

### Chromium (Passing Locally)
✅ **234 tests passed** in 6.3 minutes
✅ Authentication working
✅ Security module toggles functional
✅ CRUD operations successful
⚠️ 2 tests interrupted (likely resource/timing issues)

### Firefox (Not Running Locally)
❌ **0 tests executed** in full suite
✅ **Tests DO start** when run in isolation with explicit project flags
❓ **Root Cause:** Unknown - requires further investigation

**Potential Causes:**
1. **Sequential Execution Issue:** Playwright project dependencies may not be triggering Firefox execution after Chromium completes/interrupts.
2. **Resource Exhaustion:** Docker container may run out of memory/CPU during Chromium tests, preventing Firefox from starting.
3. **Configuration Mismatch:** playwright.config.js may have an issue with project dependency resolution.
4. **Workers Setting:** `workers: process.env.CI ? 1 : undefined` - local environment may be allocating workers differently.

### WebKit (Not Running Locally)
❌ **0 tests executed** (same as Firefox)
❓ **Root Cause:** Same as Firefox - likely dependency chain issue

---

## Key Differences: Local vs CI

| Aspect | Local Behavior | Expected CI Behavior |
|--------|----------------|----------------------|
| **Chromium Tests** | ✅ 234 passed, 2 interrupted | ❓ Unknown (CI outage) |
| **Firefox Tests** | ❌ Never executed | ❓ Unknown (CI outage) |
| **WebKit Tests** | ❌ Never executed | ❓ Unknown (CI outage) |
| **Test Workers** | `undefined` (auto) | `1` (sequential) |
| **Retries** | 0 | 2 |
| **Execution Mode** | Parallel per project | Sequential (1 worker) |
| **Total Runtime** | 6.3 min (Chromium only) | Unknown |

**Hypothesis:** In CI, Playwright may:
1. Enforce stricter dependency execution (all projects must run sequentially)
2. Have longer timeouts allowing Firefox/WebKit to eventually execute
3. Allocate resources differently (1 worker forces sequential execution)

---

## Test Execution Flow Analysis

### Configured Project Dependencies
```
setup (auth)
   ↓
security-tests (sequential, 1 worker, headless chromium)
   ↓
security-teardown (cleanup)
   ↓
┌──────────┬──────────┬──────────┐
│ chromium │ firefox  │ webkit   │
└──────────┴──────────┴──────────┘
```

### Actual Execution (Local)
```
setup ✅
   ↓
security-tests ✅ (completed)
   ↓
security-teardown ✅
   ↓
chromium ⚠️ (started, 234 passed, 2 interrupted)
   ↓
firefox ❌ (queued but never started)
   ↓
webkit ❌ (queued but never started)
```

**Critical Observation:** The interruption in Chromium tests at test #263 (certificates accessibility tests) may be the trigger that prevents Firefox/WebKit from executing. The error `Target page, context or browser has been closed` suggests resource cleanup or allocation issues.

---

## Raw Test Output Excerpts

### Chromium - Successful Tests
```
[chromium] › tests/security/audit-logs.spec.ts:26:5 › Audit Logs › Page Loading
✓ 26/982 passed (2.9s)

[chromium] › tests/security/crowdsec-config.spec.ts:26:5 › CrowdSec Configuration
✓ 24-29 passed

[chromium] › tests/security-enforcement/acl-enforcement.spec.ts:114:3
✅ Admin whitelist configured for test IP ranges
✓ Cerberus enabled
✓ ACL enabled
✓ 123-127 passed

[chromium] › tests/security-enforcement/emergency-token.spec.ts:198:3
🧪 Testing emergency token bypass with ACL enabled...
  ✓ Confirmed ACL is enabled
  ✓ Emergency token successfully accessed protected endpoint
✅ Test 1 passed: Emergency token bypasses ACL
✓ 141-148 passed
```

### Chromium - Interrupted Tests
```
[chromium] › tests/core/certificates.spec.ts:788:5
Error: browserContext.close: Target page, context or browser has been closed

[chromium] › tests/core/certificates.spec.ts:807:5
Error: page.waitForTimeout: Test ended.
```

### Firefox - Isolation Run (Successful Start)
```
Running 982 tests using 2 workers
[setup] › tests/auth.setup.ts:26:1 › authenticate ✅
[security-tests] › tests/security/audit-logs.spec.ts:26:5 ✅
[security-tests] › tests/security/audit-logs.spec.ts:47:5 ✅
...
[Tests continuing in security-tests project for Firefox]
```

---

## Coverage Data Summary

| Layer | Coverage | Threshold | Status |
|-------|----------|-----------|--------|
| **Backend** | 84.9% | 85.0% | ⚠️ Below (-0.1%) |
| **Frontend** | 84.22% | N/A | ✅ Acceptable |
| **E2E (Chromium)** | Collected | N/A | ✅ V8 coverage enabled |

---

## Recommendations

### Immediate Actions (Priority: CRITICAL)

1. **Investigate Chromium Test Interruption**
   - Analyze why `certificates.spec.ts` tests are interrupted
   - Check for resource leaks or memory issues in test cleanup
   - Review `page.waitForTimeout(500)` usage (anti-pattern - use auto-waiting)

2. **Fix Project Dependency Execution**
   - Verify `playwright.config.js` project dependencies are correctly configured
   - Test if removing `fullyParallel: true` (line 115) affects execution
   - Consider adding explicit timeout settings for long-running test suites

3. **Enable Verbose Logging for Debugging**
   ```bash
   DEBUG=pw:api npx playwright test --reporter=line
   ```
   Capture full execution flow to identify why Firefox/WebKit projects are skipped.

4. **Reproduce CI Behavior Locally**
   ```bash
   CI=1 npx playwright test --workers=1 --retries=2
   ```
   Force sequential execution with retries to match CI configuration.

### Short-Term Actions (Priority: HIGH)

5. **Isolate Browser Test Runs**
   - Run each browser project independently to confirm functionality:
     ```bash
     npx playwright test --project=setup --project=security-tests --project=chromium
     npx playwright test --project=setup --project=security-tests --project=firefox
     npx playwright test --project=setup --project=security-tests --project=webkit
     ```
   - Compare results to identify browser-specific failures.

6. **Increase Backend Coverage by 0.1%**
   - Target packages with coverage gaps (see Backend section)
   - Add unit tests for uncovered edge cases

7. **Improve Frontend Page Coverage**
   - `Security.tsx`: 65.17% → Target 80%+
   - `SecurityHeaders.tsx`: 69.23% → Target 80%+
   - `Plugins.tsx`: 63.63% → Target 80%+

### Long-Term Actions (Priority: MEDIUM)

8. **Refactor Test Dependencies**
   - Evaluate if security-tests MUST run before all browser tests
   - Consider running security-tests only once, store state, and restore for each browser

9. **Implement Test Sharding**
   - Split tests into multiple shards to reduce runtime
   - Run browser projects in parallel across different CI jobs

10. **Monitor Test Stability**
    - Track test interruptions and flaky tests
    - Implement retry logic for known-flaky tests
    - Add test stability metrics to CI

---

## Triage Plan

### Phase 1: Root Cause Analysis (Day 1)
- [ ] Run Chromium tests in isolation with verbose logging
- [ ] Identify exact cause of `certificates.spec.ts` interruption
- [ ] Fix resource leak or timeout issues

### Phase 2: Browser Execution Fix (Day 2)
- [ ] Verify Firefox/WebKit projects can run independently
- [ ] Investigate project dependency resolution in Playwright
- [ ] Apply configuration fixes to enable sequential browser execution

### Phase 3: CI Alignment (Day 3)
- [ ] Reproduce CI environment locally (`CI=1`, `workers=1`, `retries=2`)
- [ ] Compare test results between local and CI configurations
- [ ] Document any remaining discrepancies

### Phase 4: Coverage Improvements (Day 4-5)
- [ ] Add backend unit tests to reach 85% threshold
- [ ] Add frontend tests for low-coverage pages
- [ ] Verify E2E coverage collection is working correctly

---

## Appendix: Test Execution Commands

### Full Suite (As Executed)
```bash
# E2E container rebuild
/projects/Charon/.github/skills/scripts/skill-runner.sh docker-rebuild-e2e

# Full Playwright suite (all browsers)
npx playwright test
```

### Individual Browser Tests
```bash
# Chromium only
npx playwright test --project=setup --project=security-tests --project=security-teardown --project=chromium

# Firefox only
npx playwright test --project=setup --project=security-tests --project=security-teardown --project=firefox

# WebKit only
npx playwright test --project=setup --project=security-tests --project=security-teardown --project=webkit
```

### Backend Coverage
```bash
./scripts/go-test-coverage.sh
```

### Frontend Coverage
```bash
cd frontend && npm test -- --run --coverage
```

---

## Related Documentation

- [Testing Instructions](.github/instructions/testing.instructions.md)
- [Playwright TypeScript Instructions](.github/instructions/playwright-typescript.instructions.md)
- [Playwright Config](playwright.config.js)
- [E2E Rebuild Skill](.github/skills/docker-rebuild-e2e.SKILL.md)

---

**Report Generated By:** GitHub Copilot (QA Security Mode)
**Total Diagnostic Time:** ~25 minutes
**Next Update:** After Phase 1 completion