test(e2e): stabilize Phase 2 runs — disable dev webServer by default, increase API timeouts, retry navigation and harden dialog interactions
This commit is contained in:
@@ -237,6 +237,7 @@ main() {
|
||||
# Set environment variables
|
||||
# IMPORTANT: Use Vite URL (3000) for coverage, not Docker (8080)
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
export PLAYWRIGHT_SKIP_SECURITY_DEPS="${PLAYWRIGHT_SKIP_SECURITY_DEPS:-1}"
|
||||
export PLAYWRIGHT_BASE_URL="${PLAYWRIGHT_BASE_URL:-http://localhost:${VITE_PORT}}"
|
||||
|
||||
# Log configuration
|
||||
|
||||
@@ -194,6 +194,9 @@ main() {
|
||||
|
||||
# Set environment variables
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
export PLAYWRIGHT_SKIP_SECURITY_DEPS="${PLAYWRIGHT_SKIP_SECURITY_DEPS:-1}"
|
||||
# Debug runs should not start the Vite dev server by default
|
||||
export PLAYWRIGHT_COVERAGE="${PLAYWRIGHT_COVERAGE:-0}"
|
||||
set_default_env "PLAYWRIGHT_BASE_URL" "http://localhost:8080"
|
||||
|
||||
# Enable Inspector if requested
|
||||
|
||||
@@ -147,6 +147,9 @@ main() {
|
||||
|
||||
# Set environment variables for non-interactive execution
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
export PLAYWRIGHT_SKIP_SECURITY_DEPS="${PLAYWRIGHT_SKIP_SECURITY_DEPS:-1}"
|
||||
# Ensure non-coverage runs do NOT start the Vite dev server (use Docker in CI/local non-coverage)
|
||||
export PLAYWRIGHT_COVERAGE="${PLAYWRIGHT_COVERAGE:-0}"
|
||||
set_default_env "PLAYWRIGHT_BASE_URL" "http://localhost:8080"
|
||||
|
||||
# Log configuration
|
||||
|
||||
366
E2E_REMEDIATION_CHECKLIST.md
Normal file
366
E2E_REMEDIATION_CHECKLIST.md
Normal file
@@ -0,0 +1,366 @@
|
||||
# E2E Test Remediation Checklist
|
||||
|
||||
**Status**: Active
|
||||
**Plan Reference**: [docs/plans/current_spec.md](docs/plans/current_spec.md)
|
||||
**Last Updated**: 2026-02-09
|
||||
|
||||
---
|
||||
|
||||
## 📋 Phase 1: Foundation & Test Harness Reliability
|
||||
|
||||
**Objective**: Ensure the shared test harness (global setup, auth, emergency server) is stable
|
||||
**Estimated Runtime**: 2-4 minutes
|
||||
**Status**: ✅ PASSED
|
||||
|
||||
### Setup
|
||||
- [x] **docker-rebuild-e2e**: `.github/skills/scripts/skill-runner.sh docker-rebuild-e2e`
|
||||
- Ensures container has latest code and env vars (`CHARON_EMERGENCY_TOKEN`, encryption key)
|
||||
- **Expected**: Container healthy, port 8080 responsive, port 2020 available
|
||||
- **Status**: ✅ Container rebuilt and ready
|
||||
|
||||
### Execution
|
||||
- [x] **Run Phase 1 tests**:
|
||||
```bash
|
||||
cd /projects/Charon
|
||||
npx playwright test tests/global-setup.ts tests/auth.setup.ts --project=firefox
|
||||
```
|
||||
- **Expected**: Both tests pass without re-auth flakes
|
||||
- **Result**: ✅ **PASSED** (1 test in 5.2s)
|
||||
- **Errors found**: None
|
||||
|
||||
### Validation
|
||||
- [x] Storage state (`tests/.auth/*.json`) created successfully
|
||||
- ✅ Auth state saved to `/projects/Charon/playwright/.auth/user.json`
|
||||
- [x] Emergency token validated (check logs for "Emergency token OK")
|
||||
- ✅ Token length: 64 chars (valid), format: Valid hexadecimal
|
||||
- [x] Security reset executed (check logs for "Security teardown complete")
|
||||
- ✅ Emergency reset successful [22ms]
|
||||
- ✅ Security reset complete with 526ms propagation
|
||||
|
||||
### Blocking Issues
|
||||
- [x] **None** - Phase 1 foundational tests all passing
|
||||
|
||||
**Issues Encountered**:
|
||||
- None
|
||||
|
||||
### Port Connectivity Summary
|
||||
- [x] Caddy admin API (port 2019): ✅ Healthy
|
||||
- [x] Emergency server (port 2020): ✅ Healthy
|
||||
- [x] Application UI (port 8080): ✅ Accessible
|
||||
|
||||
---
|
||||
|
||||
## 📋 Phase 2: Core UI, Settings, Tasks, Monitoring
|
||||
|
||||
**Objective**: Remediate highest-traffic user journeys
|
||||
**Estimated Runtime**: 25-40 minutes
|
||||
**Status**: ❌ FAILED
|
||||
|
||||
**Note:** Verified Phase 2 directories for misfiled security-dependent tests — no remaining ACL/CrowdSec/WAF tests were found in `tests/core`, `tests/settings`, `tests/tasks` or `tests/monitoring`. CrowdSec/ACL-specific tests live in the `tests/security` and `tests/security-enforcement` suites as intended. The Caddy import tests remain in Phase 2 (they do not require security to be enabled).
|
||||
|
||||
### Sub-Phase 2A: Core UI (Navigation, Dashboard, CRUD)
|
||||
- [x] **Run tests**:
|
||||
```bash
|
||||
npx playwright test tests/core --project=firefox
|
||||
```
|
||||
- **Expected**: All core CRUD and navigation pass
|
||||
- **Result**: ❌ Fail (9 passed, 2 interrupted, 187 did not run; total 198; exit code 130)
|
||||
- **Comparison**: Previous 2 failed → Now 2 interrupted (187 did not run)
|
||||
- **Errors found**:
|
||||
```
|
||||
1) [firefox] › tests/core/access-lists-crud.spec.ts:261:5 › Access Lists - CRUD Operations › Create Access List › should add client IP addresses
|
||||
Error: page.goto: Test ended.
|
||||
Call log:
|
||||
- navigating to "http://localhost:5173/access-lists", waiting until "load"
|
||||
|
||||
2) [firefox] › tests/core/access-lists-crud.spec.ts:217:5 › Access Lists - CRUD Operations › Create Access List › should create ACL with name only (IP whitelist)
|
||||
Error: Test was interrupted.
|
||||
```
|
||||
|
||||
**Issue Log for Phase 2A**:
|
||||
1. **Issue**: Access list creation tests interrupted by unexpected page close
|
||||
**File**: [tests/core/access-lists-crud.spec.ts](tests/core/access-lists-crud.spec.ts)
|
||||
**Root Cause**: Test run interrupted during navigation (page/context ended)
|
||||
**Fix Applied**: None (per instructions)
|
||||
**Re-test Result**: ❌
|
||||
|
||||
---
|
||||
|
||||
### Sub-Phase 2B: Settings (System, Account, Notifications, Encryption, Users)
|
||||
- [x] **Run tests**:
|
||||
```bash
|
||||
npx playwright test tests/settings --project=firefox
|
||||
```
|
||||
- **Expected**: All settings flows pass
|
||||
- **Result**: ❌ Fail (1 passed, 2 interrupted, 129 did not run; total 132; exit code 130)
|
||||
- **Comparison**: Previous 15 failed → Now 2 interrupted (129 did not run)
|
||||
- **Errors found**:
|
||||
```
|
||||
1) [firefox] › tests/settings/account-settings.spec.ts:37:5 › Account Settings › Profile Management › should display user profile
|
||||
Error: page.goto: Test ended.
|
||||
Call log:
|
||||
- navigating to "http://localhost:5173/settings/account", waiting until "load"
|
||||
|
||||
2) [firefox] › tests/settings/account-settings.spec.ts:63:5 › Account Settings › Profile Management › should update profile name
|
||||
Error: Test was interrupted.
|
||||
```
|
||||
|
||||
**Issue Log for Phase 2B**:
|
||||
1. **Issue**: Settings test run interrupted during account settings navigation
|
||||
**File**: [tests/settings/account-settings.spec.ts](tests/settings/account-settings.spec.ts)
|
||||
**Root Cause**: Test ended unexpectedly during `page.goto`
|
||||
**Fix Applied**: None (per instructions)
|
||||
**Re-test Result**: ❌
|
||||
|
||||
---
|
||||
|
||||
### Sub-Phase 2C: Tasks, Monitoring, Utilities
|
||||
- [x] **Run tests**:
|
||||
```bash
|
||||
npx playwright test tests/tasks --project=firefox
|
||||
npx playwright test tests/monitoring --project=firefox
|
||||
npx playwright test tests/utils/wait-helpers.spec.ts --project=firefox
|
||||
```
|
||||
- **Expected**: All task/monitoring flows and utilities pass
|
||||
- **Result**: ❌ Fail
|
||||
- **Tasks**: 1 passed, 2 interrupted, 94 did not run; total 97; exit code 130
|
||||
- **Monitoring**: 1 passed, 2 interrupted, 44 did not run; total 47; exit code 130
|
||||
- **Wait-helpers**: 0 passed, 0 failed, 22 did not run; total 22; exit code 130
|
||||
- **Comparison**:
|
||||
- Tasks: Previous 16 failed → Now 2 interrupted (94 did not run)
|
||||
- Monitoring: Previous 20 failed → Now 2 interrupted (44 did not run)
|
||||
- Wait-helpers: Previous 1 failed → Now 0 failed (22 did not run)
|
||||
- **Errors found**:
|
||||
```
|
||||
Tasks
|
||||
1) [firefox] › tests/tasks/backups-create.spec.ts:58:5 › Backups Page - Creation and List › Page Layout › should show Create Backup button for admin users
|
||||
Error: browserContext.close: Protocol error (Browser.removeBrowserContext)
|
||||
|
||||
2) [firefox] › tests/tasks/backups-create.spec.ts:50:5 › Backups Page - Creation and List › Page Layout › should display backups page with correct heading
|
||||
Error: browserContext.newPage: Test ended.
|
||||
|
||||
Monitoring
|
||||
1) [firefox] › tests/monitoring/real-time-logs.spec.ts:247:5 › Real-Time Logs Viewer › Page Layout › should display live logs viewer with correct heading
|
||||
Error: page.goto: Test ended.
|
||||
Call log:
|
||||
- navigating to "http://localhost:5173/", waiting until "load"
|
||||
|
||||
2) [firefox] › tests/monitoring/real-time-logs.spec.ts:510:5 › Real-Time Logs Viewer › Filtering › should filter logs by search text
|
||||
Error: page.goto: Target page, context or browser has been closed
|
||||
|
||||
Wait-helpers
|
||||
1) [firefox] › tests/utils/wait-helpers.spec.ts:284:5 › wait-helpers - Phase 2.1 Semantic Wait Functions › waitForNavigation › should wait for URL change with string match
|
||||
Error: Test run interrupted before executing tests (22 did not run).
|
||||
```
|
||||
|
||||
**Issue Log for Phase 2C**:
|
||||
1. **Issue**: Tasks suite interrupted due to browser context teardown error
|
||||
**File**: [tests/tasks/backups-create.spec.ts](tests/tasks/backups-create.spec.ts)
|
||||
**Root Cause**: `Browser.removeBrowserContext` protocol error during teardown
|
||||
**Fix Applied**: None (per instructions)
|
||||
**Re-test Result**: ❌
|
||||
2. **Issue**: Monitoring suite interrupted by page/context closure during navigation
|
||||
**File**: [tests/monitoring/real-time-logs.spec.ts](tests/monitoring/real-time-logs.spec.ts)
|
||||
**Root Cause**: Page closed before navigation completed
|
||||
**Fix Applied**: None (per instructions)
|
||||
**Re-test Result**: ❌
|
||||
3. **Issue**: Wait-helpers suite interrupted before executing tests
|
||||
**File**: [tests/utils/wait-helpers.spec.ts](tests/utils/wait-helpers.spec.ts)
|
||||
**Root Cause**: Test run interrupted before any assertions executed
|
||||
**Fix Applied**: None (per instructions)
|
||||
**Re-test Result**: ❌
|
||||
|
||||
---
|
||||
|
||||
## 📋 Phase 3: Security UI & Enforcement
|
||||
|
||||
**Objective**: Stabilize Cerberus UI and enforcement workflows
|
||||
**Estimated Runtime**: 30-45 minutes
|
||||
**Status**: ⏳ Not Started
|
||||
**⚠️ CRITICAL**: Must use `--workers=1` for security-enforcement (see Phase 3B)
|
||||
|
||||
### Sub-Phase 3A: Security UI (Dashboard, WAF, Headers, Rate Limiting, CrowdSec, Audit Logs)
|
||||
- [ ] **Run tests**:
|
||||
```bash
|
||||
npx playwright test tests/security --project=firefox
|
||||
```
|
||||
- **Expected**: All security UI toggles and pages load
|
||||
- **Result**: ✅ Pass / ❌ Fail
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
**Issue Log for Phase 3A**:
|
||||
1. **Issue**: [Describe]
|
||||
**File**: [tests/security/...]
|
||||
**Root Cause**: [Analyze]
|
||||
**Fix Applied**: [Link]
|
||||
**Re-test Result**: ✅ / ❌
|
||||
|
||||
---
|
||||
|
||||
### Sub-Phase 3B: Security Enforcement (ACL, WAF, CrowdSec, Rate Limits, Emergency Token, Break-Glass)
|
||||
|
||||
⚠️ **SERIAL EXECUTION REQUIRED**: `--workers=1` (enforces zzz-prefixed ordering)
|
||||
|
||||
- [ ] **Run tests WITH SERIAL FLAG**:
|
||||
```bash
|
||||
npx playwright test tests/security-enforcement --project=firefox --workers=1
|
||||
```
|
||||
- **Expected**: All enforcement tests pass with zzz-prefixing order enforced
|
||||
- **Result**: ✅ Pass / ❌ Fail
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
**Critical Ordering Notes**:
|
||||
- `zzz-admin-whitelist-blocking.spec.ts` MUST run last (before break-glass)
|
||||
- `zzzz-break-glass-recovery.spec.ts` MUST finalize cleanup
|
||||
- If tests fail due to ordering, verify `--workers=1` was used
|
||||
|
||||
**Issue Log for Phase 3B**:
|
||||
1. **Issue**: [Describe]
|
||||
**File**: [tests/security-enforcement/...]
|
||||
**Root Cause**: [Analyze - including ordering if relevant]
|
||||
**Fix Applied**: [Link]
|
||||
**Re-test Result**: ✅ / ❌
|
||||
|
||||
---
|
||||
|
||||
## 📋 Phase 4: Integration, Browser-Specific, Debug (Optional)
|
||||
|
||||
**Objective**: Close cross-feature and browser-specific regressions
|
||||
**Estimated Runtime**: 25-40 minutes
|
||||
**Status**: ⏳ Not Started
|
||||
|
||||
### Sub-Phase 4A: Integration Workflows
|
||||
- [ ] **Run tests**:
|
||||
```bash
|
||||
npx playwright test tests/integration --project=firefox
|
||||
```
|
||||
- **Expected**: Cross-feature workflows pass
|
||||
- **Result**: ✅ Pass / ❌ Fail
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
**Issue Log for Phase 4A**:
|
||||
1. **Issue**: [Describe]
|
||||
**File**: [tests/integration/...]
|
||||
**Root Cause**: [Analyze]
|
||||
**Fix Applied**: [Link]
|
||||
**Re-test Result**: ✅ / ❌
|
||||
|
||||
---
|
||||
|
||||
### Sub-Phase 4B: Browser-Specific Regressions (Firefox & WebKit)
|
||||
- [ ] **Run Firefox-specific tests**:
|
||||
```bash
|
||||
npx playwright test tests/firefox-specific --project=firefox
|
||||
```
|
||||
- **Expected**: Firefox import and flow regressions pass
|
||||
- **Result**: ✅ Pass / ❌ Fail
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
- [ ] **Run WebKit-specific tests**:
|
||||
```bash
|
||||
npx playwright test tests/webkit-specific --project=webkit
|
||||
```
|
||||
- **Expected**: WebKit import and flow regressions pass
|
||||
- **Result**: ✅ Pass / ❌ Fail
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
**Issue Log for Phase 4B**:
|
||||
1. **Issue**: [Describe]
|
||||
**File**: [tests/firefox-specific/... or tests/webkit-specific/...]
|
||||
**Root Cause**: [Analyze - may be browser-specific]
|
||||
**Fix Applied**: [Link]
|
||||
**Re-test Result**: ✅ / ❌
|
||||
|
||||
---
|
||||
|
||||
### Sub-Phase 4C: Debug/POC & Gap Coverage (Optional)
|
||||
- [ ] **Run debug diagnostics**:
|
||||
```bash
|
||||
npx playwright test tests/debug --project=firefox
|
||||
npx playwright test tests/tasks/caddy-import-gaps.spec.ts --project=firefox
|
||||
npx playwright test tests/tasks/caddy-import-cross-browser.spec.ts --project=firefox
|
||||
npx playwright test tests/modal-dropdown-triage.spec.ts --project=firefox
|
||||
npx playwright test tests/proxy-host-dropdown-fix.spec.ts --project=firefox
|
||||
```
|
||||
- **Expected**: Debug and gap-coverage tests pass (or are identified as low-priority)
|
||||
- **Result**: ✅ Pass / ❌ Fail / ⏭️ Skip (optional)
|
||||
- **Errors found** (if any):
|
||||
```
|
||||
[Paste errors]
|
||||
```
|
||||
|
||||
**Issue Log for Phase 4C**:
|
||||
1. **Issue**: [Describe]
|
||||
**File**: [tests/debug/... or tests/tasks/...]
|
||||
**Root Cause**: [Analyze]
|
||||
**Fix Applied**: [Link]
|
||||
**Re-test Result**: ✅ / ❌
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Summary & Sign-Off
|
||||
|
||||
### Overall Status
|
||||
- **Phase 1**: ✅ PASSED
|
||||
- **Phase 2**: ❌ FAILED
|
||||
- **Phase 3**: ⏳ Not Started
|
||||
- **Phase 4**: ⏳ Not Started
|
||||
|
||||
### Total Issues Found & Fixed
|
||||
- **Phase 1**: 0 issues
|
||||
- **Phase 2**: [X] issues (all fixed: ✅ / some pending: ❌)
|
||||
- **Phase 3**: [X] issues (all fixed: ✅ / some pending: ❌)
|
||||
- **Phase 4**: [X] issues (all fixed: ✅ / some pending: ❌)
|
||||
|
||||
### Root Causes Identified
|
||||
1. [Issue type] - Occurred in [Phase] - Example: "Flaky WebSocket timeout in monitoring tests"
|
||||
2. [Issue type] - Occurred in [Phase]
|
||||
3. ...
|
||||
|
||||
### Fixes Applied (with Links)
|
||||
1. [Fix description] - [Link to PR/commit]
|
||||
2. [Fix description] - [Link to PR/commit]
|
||||
3. ...
|
||||
|
||||
### Final Validation
|
||||
- [ ] All phases complete (phases 1-3 required; phase 4 optional)
|
||||
- [ ] All blocking issues resolved
|
||||
- [ ] No new regressions introduced
|
||||
- [ ] Ready for CI integration
|
||||
|
||||
---
|
||||
|
||||
## 🔗 References
|
||||
|
||||
- **Plan**: [docs/plans/current_spec.md](docs/plans/current_spec.md)
|
||||
- **Quick Start**: See Quick Start section in plan
|
||||
- **Emergency Server Docs**: Check tests/security-enforcement/emergency-server/
|
||||
- **Port Requirements**: 8080 (UI/API), 2020 (Emergency Server), 2019 (Caddy Admin)
|
||||
- **Critical Flag**: `--workers=1` for Phase 3B (security-enforcement)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
Use this space to document any additional context, blockers, or learnings:
|
||||
|
||||
```
|
||||
Remaining failures (current rerun):
|
||||
- Test infra interruptions: 8 interrupted tests, 476 did not run (Phase 2A/2B/2C)
|
||||
- WebSocket/logs/import verification: not validated in this rerun due to early interruptions
|
||||
```
|
||||
@@ -426,6 +426,17 @@ func (h *ImportHandler) Upload(c *gin.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
session := models.ImportSession{
|
||||
UUID: sid,
|
||||
SourceFile: tempPath,
|
||||
Status: "pending",
|
||||
ParsedData: string(mustMarshal(result)),
|
||||
ConflictReport: string(mustMarshal(result.Conflicts)),
|
||||
}
|
||||
if err := h.db.Create(&session).Error; err != nil {
|
||||
middleware.GetRequestLogger(c).WithError(err).Warn("Import Upload: failed to persist session")
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"session": gin.H{"id": sid, "state": "transient", "source_file": tempPath},
|
||||
"conflict_details": conflictDetails,
|
||||
@@ -643,6 +654,17 @@ func (h *ImportHandler) UploadMulti(c *gin.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
session := models.ImportSession{
|
||||
UUID: sid,
|
||||
SourceFile: mainCaddyfile,
|
||||
Status: "pending",
|
||||
ParsedData: string(mustMarshal(result)),
|
||||
ConflictReport: string(mustMarshal(result.Conflicts)),
|
||||
}
|
||||
if err := h.db.Create(&session).Error; err != nil {
|
||||
middleware.GetRequestLogger(c).WithError(err).Warn("Import UploadMulti: failed to persist session")
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"session": gin.H{"id": sid, "state": "transient", "source_file": mainCaddyfile},
|
||||
"preview": result,
|
||||
|
||||
@@ -209,13 +209,12 @@ func RegisterWithDeps(router *gin.Engine, db *gorm.DB, cfg config.Config, caddyM
|
||||
protected.POST("/backups/:filename/restore", backupHandler.Restore)
|
||||
|
||||
// Logs
|
||||
protected.GET("/logs", logsHandler.List)
|
||||
protected.GET("/logs/:filename", logsHandler.Read)
|
||||
protected.GET("/logs/:filename/download", logsHandler.Download)
|
||||
|
||||
// WebSocket endpoints
|
||||
logsWSHandler := handlers.NewLogsWSHandler(wsTracker)
|
||||
protected.GET("/logs/live", logsWSHandler.HandleWebSocket)
|
||||
protected.GET("/logs", logsHandler.List)
|
||||
protected.GET("/logs/:filename", logsHandler.Read)
|
||||
protected.GET("/logs/:filename/download", logsHandler.Download)
|
||||
|
||||
// WebSocket status monitoring
|
||||
protected.GET("/websocket/connections", wsStatusHandler.GetConnections)
|
||||
|
||||
@@ -26,6 +26,34 @@ func NewLogService(cfg *config.Config) *LogService {
|
||||
return &LogService{LogDir: logDir}
|
||||
}
|
||||
|
||||
func (s *LogService) logDirs() []string {
|
||||
seen := make(map[string]bool)
|
||||
var dirs []string
|
||||
|
||||
addDir := func(dir string) {
|
||||
clean := filepath.Clean(dir)
|
||||
if clean == "." || clean == "" {
|
||||
return
|
||||
}
|
||||
if !seen[clean] {
|
||||
seen[clean] = true
|
||||
dirs = append(dirs, clean)
|
||||
}
|
||||
}
|
||||
|
||||
addDir(s.LogDir)
|
||||
|
||||
if accessLogPath := os.Getenv("CHARON_CADDY_ACCESS_LOG"); accessLogPath != "" {
|
||||
addDir(filepath.Dir(accessLogPath))
|
||||
}
|
||||
|
||||
if _, err := os.Stat("/var/log/caddy"); err == nil {
|
||||
addDir("/var/log/caddy")
|
||||
}
|
||||
|
||||
return dirs
|
||||
}
|
||||
|
||||
type LogFile struct {
|
||||
Name string `json:"name"`
|
||||
Size int64 `json:"size"`
|
||||
@@ -33,42 +61,44 @@ type LogFile struct {
|
||||
}
|
||||
|
||||
func (s *LogService) ListLogs() ([]LogFile, error) {
|
||||
entries, err := os.ReadDir(s.LogDir)
|
||||
if err != nil {
|
||||
// If directory doesn't exist, return empty list instead of error
|
||||
if os.IsNotExist(err) {
|
||||
return []LogFile{}, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var logs []LogFile
|
||||
seen := make(map[string]bool)
|
||||
for _, entry := range entries {
|
||||
hasLogExtension := strings.HasSuffix(entry.Name(), ".log") || strings.Contains(entry.Name(), ".log.")
|
||||
if entry.IsDir() || !hasLogExtension {
|
||||
continue
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
for _, dir := range s.logDirs() {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
// Handle symlinks + deduplicate files (e.g., charon.log and cpmp.log (legacy name) pointing to same file)
|
||||
entryPath := filepath.Join(s.LogDir, entry.Name())
|
||||
resolved, err := filepath.EvalSymlinks(entryPath)
|
||||
if err == nil {
|
||||
if seen[resolved] {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
seen[resolved] = true
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for _, entry := range entries {
|
||||
hasLogExtension := strings.HasSuffix(entry.Name(), ".log") || strings.Contains(entry.Name(), ".log.")
|
||||
if entry.IsDir() || !hasLogExtension {
|
||||
continue
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
// Handle symlinks + deduplicate files (e.g., charon.log and cpmp.log (legacy name) pointing to same file)
|
||||
entryPath := filepath.Join(dir, entry.Name())
|
||||
resolved, err := filepath.EvalSymlinks(entryPath)
|
||||
if err == nil {
|
||||
if seen[resolved] {
|
||||
continue
|
||||
}
|
||||
seen[resolved] = true
|
||||
}
|
||||
logs = append(logs, LogFile{
|
||||
Name: entry.Name(),
|
||||
Size: info.Size(),
|
||||
ModTime: info.ModTime().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
logs = append(logs, LogFile{
|
||||
Name: entry.Name(),
|
||||
Size: info.Size(),
|
||||
ModTime: info.ModTime().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
|
||||
return logs, nil
|
||||
}
|
||||
|
||||
@@ -78,17 +108,21 @@ func (s *LogService) GetLogPath(filename string) (string, error) {
|
||||
if filename != cleanName {
|
||||
return "", fmt.Errorf("invalid filename: path traversal attempt detected")
|
||||
}
|
||||
path := filepath.Join(s.LogDir, cleanName)
|
||||
if !strings.HasPrefix(path, filepath.Clean(s.LogDir)) {
|
||||
return "", fmt.Errorf("invalid filename: path traversal attempt detected")
|
||||
|
||||
for _, dir := range s.logDirs() {
|
||||
baseDir := filepath.Clean(dir)
|
||||
path := filepath.Join(baseDir, cleanName)
|
||||
if !strings.HasPrefix(path, baseDir+string(os.PathSeparator)) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Verify file exists
|
||||
if _, err := os.Stat(path); err == nil {
|
||||
return path, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Verify file exists
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return path, nil
|
||||
return "", os.ErrNotExist
|
||||
}
|
||||
|
||||
// QueryLogs parses and filters logs from a specific file
|
||||
|
||||
@@ -257,6 +257,8 @@ cd /projects/Charon
|
||||
npx playwright test tests/global-setup.ts tests/auth.setup.ts --project=firefox
|
||||
|
||||
# PHASE 2: Core UI, Settings, Tasks, Monitoring
|
||||
# NOTE: PLAYWRIGHT_SKIP_SECURITY_DEPS=1 is automatically set in E2E scripts
|
||||
# Security suites will NOT execute as dependencies
|
||||
npx playwright test tests/core --project=firefox
|
||||
npx playwright test tests/settings --project=firefox
|
||||
npx playwright test tests/tasks --project=firefox
|
||||
|
||||
42
docs/plans/phase_2_failure_analysis.md
Normal file
42
docs/plans/phase_2_failure_analysis.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Phase 2 E2E Failure Analysis (2026-02-09)
|
||||
|
||||
## Overview
|
||||
Phase 2 commands (tests/core, tests/settings, tests/tasks, tests/monitoring) executed security test suites and then hit widespread ACL/auth failures. The failures are consistent across 2A/2B/2C and point to security modules being enabled during Phase 2, despite Phase 1 reporting a successful security reset.
|
||||
|
||||
## Evidence From E2E Remediation Checklist
|
||||
- Phase 2A/2B/2C failures show ACL blocks and auth errors during non-security runs, e.g.:
|
||||
- "Failed to create user: {\"error\":\"Blocked by access control list\"}"
|
||||
- "Failed to get security status: 403 {\"error\":\"Blocked by access control list\"}"
|
||||
- "Failed to get security status: 401 {\"error\":\"Authorization header required\"}"
|
||||
- Phase 2A/2B/2C failure lists are dominated by security suite test files, for example:
|
||||
- tests/security/waf-config.spec.ts
|
||||
- tests/security/security-dashboard.spec.ts
|
||||
- tests/security-enforcement/rate-limit-enforcement.spec.ts
|
||||
|
||||
## Root Cause Hypotheses (Most Likely)
|
||||
1) Playwright dependency chain forces security tests to run for Phase 2
|
||||
- In playwright.config.js, the browser projects (chromium/firefox/webkit) declare dependencies as:
|
||||
- browserDependencies = ['setup', 'security-tests'] unless PLAYWRIGHT_SKIP_SECURITY_DEPS=1
|
||||
- Running `npx playwright test tests/core --project=firefox` triggers dependencies for the firefox project, so Playwright executes the security-tests project (security/ and security-enforcement/ suites) before the targeted Phase 2 directory. This explains why security tests appear during Phase 2 commands.
|
||||
|
||||
2) Security teardown likely did not execute after security tests failed
|
||||
- The security-tests project declares teardown: 'security-teardown'. If security-tests fails early (as reported), teardown may not run, leaving Cerberus/ACL/WAF/rate limiting enabled. That would cause Phase 2 API calls (user creation, security status checks, admin login) to be blocked by ACL, matching the 403/401 errors seen in Phase 2.
|
||||
|
||||
3) Auth state is valid for UI but blocked for API due to active security enforcement
|
||||
- The 401 "Authorization header required" indicates unauthenticated API calls during security enforcement tests. Combined with ACL blocks, this suggests the system was in security-enforced mode, so normal Phase 2 setup actions (creating users, toggling settings) could not proceed. This is consistent with the security suite running unexpectedly and leaving enforcement active.
|
||||
|
||||
## Recommendation
|
||||
Phase 2 is blocked until the Playwright dependency configuration is addressed or bypassed for Phase 2 runs.
|
||||
- Confirm whether PLAYWRIGHT_SKIP_SECURITY_DEPS=1 is required for Phase 2 commands.
|
||||
- Verify that security-teardown ran after the dependency tests or explicitly run the security reset before Phase 2.
|
||||
- Re-run Phase 2 only after validating that security modules are disabled and auth is functional.
|
||||
|
||||
## Next Steps for Engineering Director
|
||||
1) Validate Playwright dependency behavior with the current config:
|
||||
- Confirm that running a single project with --project=firefox triggers security-tests dependencies by design.
|
||||
2) Decide on a formal Phase 2 execution path:
|
||||
- Option A: Require PLAYWRIGHT_SKIP_SECURITY_DEPS=1 for Phase 2 runs.
|
||||
- Option B: Add a dedicated Phase 2 project without security-test dependencies.
|
||||
3) Add a pre-Phase 2 system state verification step:
|
||||
- Ensure security modules are disabled and admin whitelist is clear before running Phase 2.
|
||||
4) Document the dependency requirement in Phase 2 run instructions so this does not repeat.
|
||||
189
docs/plans/phase_2_fix_plan.md
Normal file
189
docs/plans/phase_2_fix_plan.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# Phase 2 E2E Failure Fix Plan
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
This plan analyzes Phase 2 E2E failures from the remediation checklist and
|
||||
prioritizes fixes that unblock the most tests. It focuses on shared root
|
||||
causes, dependency clusters, and ownership for targeted remediation.
|
||||
|
||||
## 2. Research Findings
|
||||
|
||||
### 2.1 Source of Truth
|
||||
|
||||
Primary input: [E2E_REMEDIATION_CHECKLIST.md](../../E2E_REMEDIATION_CHECKLIST.md)
|
||||
(Phase 2A, 2B, 2C failures).
|
||||
|
||||
### 2.2 Failure Clusters
|
||||
|
||||
- Core UI Docker integration: 2 failures on missing/blocked connection source
|
||||
control.
|
||||
- Settings notifications: 7 failures with timeouts or page context closure.
|
||||
- Settings strict-mode collisions: 5 failures from over-broad selectors.
|
||||
- Tasks log viewing: 12 timeouts waiting for log responses.
|
||||
- Caddy import sessions: 3 failures (import results and missing session banner).
|
||||
- Monitoring real-time logs: 19 failures with WebSocket status stuck at
|
||||
Disconnected.
|
||||
- Wait-helpers: 1 failure waiting for URL string match.
|
||||
|
||||
## 3. Root Cause Categorization
|
||||
|
||||
### 3.1 Failure Buckets (54 total)
|
||||
|
||||
| Bucket | Count | Examples |
|
||||
| --- | --- | --- |
|
||||
| Backend API issues | 24 | Notifications CRUD/timeouts, system settings save, Caddy import results, log viewing API timeouts |
|
||||
| Frontend UI issues | 3 | Docker integration control missing, certificate email validation state |
|
||||
| WebSocket issues | 19 | Real-time logs never connect (Disconnected state persists) |
|
||||
| Test infrastructure issues | 6 | Strict-mode collisions (selectors), wait-helpers URL timeout |
|
||||
| Admin access/permissions issues | 2 | Guest visibility of backup button, permissions uncheck disabled |
|
||||
|
||||
### 3.2 Root Cause Patterns
|
||||
|
||||
- Logs viewing failures (12) all timeout on `page.waitForResponse`, indicating
|
||||
a shared logs API endpoint not returning or blocked in Docker mode.
|
||||
- Real-time logs failures (19) all show Disconnected, indicating WebSocket
|
||||
handshake or server-side streaming not established for `/api/v1/logs`.
|
||||
- Caddy import failures cluster on missing import session artifacts (no banner
|
||||
and zero parsed imports), suggesting a shared import-session persistence or
|
||||
retrieval issue.
|
||||
- Settings notifications failures cluster on timeouts and context closure,
|
||||
suggesting API routes or navigation errors when provider lists/templates
|
||||
are queried or mutated.
|
||||
- Strict-mode collisions in settings and monitoring point to test selectors
|
||||
resolving multiple nodes, indicating test infra refinement needed.
|
||||
- Admin access failures show inconsistent RBAC enforcement between UI
|
||||
visibility and server-side enforcement.
|
||||
|
||||
## 4. Technical Specifications
|
||||
|
||||
### 4.1 Priority Ranking (Max Impact First)
|
||||
|
||||
1. WebSocket connection failures for real-time logs (19 tests blocked)
|
||||
2. Logs API timeouts for static log viewing (12 tests blocked)
|
||||
3. Notifications settings API timeouts/context closure (7 tests blocked)
|
||||
4. Caddy import session persistence/results (3 tests blocked)
|
||||
5. Docker integration UI controls missing (2 tests blocked)
|
||||
6. Strict-mode collisions and wait-helpers (6 tests blocked)
|
||||
7. Admin access/permissions mismatches (2 tests blocked)
|
||||
|
||||
### 4.2 Fix Batches
|
||||
|
||||
#### Critical Fixes (Block multiple suites)
|
||||
|
||||
- WebSocket connection / event delivery
|
||||
- Affected tests: 19 (monitoring/real-time-logs)
|
||||
- Root cause: WebSocket never reaches Connected; likely backend
|
||||
upgrade/streaming path or proxy config issue.
|
||||
- Recommendation: Backend Dev
|
||||
|
||||
- Logs API timeouts
|
||||
- Affected tests: 12 (tasks/logs-viewing)
|
||||
- Root cause: log listing endpoints timing out or blocked in container mode.
|
||||
- Recommendation: Backend Dev
|
||||
|
||||
- Notifications settings API timeouts
|
||||
- Affected tests: 7 (settings/notifications)
|
||||
- Root cause: provider/template APIs not responding or UI navigation error
|
||||
closing the page context.
|
||||
- Recommendation: Backend Dev with Frontend Dev support
|
||||
|
||||
- Caddy import session persistence
|
||||
- Affected tests: 3 (tasks/caddy-import-*)
|
||||
- Root cause: import sessions not persisted or banner data not returned.
|
||||
- Recommendation: Backend Dev
|
||||
|
||||
#### Secondary Fixes (Quick wins or infra)
|
||||
|
||||
- Docker integration UI controls
|
||||
- Affected tests: 2 (core/proxy-hosts Docker integration)
|
||||
- Root cause: missing/hidden form control for connection source.
|
||||
- Recommendation: Frontend Dev
|
||||
|
||||
- Strict-mode collisions and wait helpers
|
||||
- Affected tests: 6 (settings + monitoring + wait-helpers)
|
||||
- Root cause: selectors match multiple elements or URL helper too strict.
|
||||
- Recommendation: Playwright Dev
|
||||
|
||||
- Admin access/permissions mismatches
|
||||
- Affected tests: 2 (tasks/backups guest UI, settings permission uncheck)
|
||||
- Root cause: UI visibility vs RBAC mismatch or disabled inputs.
|
||||
- Recommendation: Backend Dev with Frontend Dev support
|
||||
|
||||
## 5. Effort and Impact Estimates
|
||||
|
||||
| Category | Effort | Impact | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| WebSocket connection | L | Very High | Unblocks 19 monitoring tests |
|
||||
| Logs API timeouts | M | High | Unblocks 12 task tests |
|
||||
| Notifications API timeouts | M | High | Unblocks 7 settings tests |
|
||||
| Caddy import sessions | M | Medium | Unblocks 3 task tests |
|
||||
| Docker integration UI | S | Medium | Unblocks 2 core tests |
|
||||
| Strict-mode + wait helpers | S | Medium | Unblocks 6 tests |
|
||||
| Admin access mismatches | S | Low | Unblocks 2 tests |
|
||||
|
||||
## 6. Implementation Plan
|
||||
|
||||
### Phase 1: WebSocket and Logs APIs
|
||||
|
||||
1. Verify `/api/v1/logs` WebSocket handshake and server-side stream starts.
|
||||
2. Validate static logs API endpoints and response time in Docker mode.
|
||||
3. Confirm UI connects to correct WebSocket endpoint for app/security modes.
|
||||
|
||||
### Phase 2: Notifications and Caddy Import Sessions
|
||||
|
||||
1. Validate notification providers CRUD endpoints and template endpoints.
|
||||
2. Ensure notification routes do not crash the page context.
|
||||
3. Validate import-session persistence and banner retrieval endpoints.
|
||||
|
||||
### Phase 3: UI and Test Infrastructure Quick Wins
|
||||
|
||||
1. Restore Docker integration connection source control visibility.
|
||||
2. Tighten selectors in strict-mode failures (system status, user management,
|
||||
uptime monitor).
|
||||
3. Adjust wait-helpers URL matching to handle expected navigation timing.
|
||||
|
||||
### Phase 4: RBAC Consistency
|
||||
|
||||
1. Ensure guest users cannot see Create Backup UI controls.
|
||||
2. Ensure permission management inputs reflect actual capability and are
|
||||
enabled for admin flows.
|
||||
|
||||
## 7. Acceptance Criteria (EARS)
|
||||
|
||||
- WHEN the real-time logs page loads, THE SYSTEM SHALL establish a WebSocket
|
||||
connection and report Connected status within the test timeout.
|
||||
- WHEN static logs are requested, THE SYSTEM SHALL return log data within
|
||||
the test timeout for pagination, filtering, and download flows.
|
||||
- WHEN notification providers/templates are managed, THE SYSTEM SHALL respond
|
||||
to CRUD requests without page context closure or timeouts.
|
||||
- WHEN a Caddy import session exists, THE SYSTEM SHALL return the session
|
||||
banner and import results for review flows.
|
||||
- WHEN a guest user accesses backups, THE SYSTEM SHALL hide Create Backup
|
||||
controls and enforce server-side RBAC.
|
||||
- WHEN strict-mode selectors are used, THE SYSTEM SHALL present a unique
|
||||
element for each targeted control in settings and monitoring pages.
|
||||
|
||||
## 8. Delegation Recommendations
|
||||
|
||||
- Backend Dev
|
||||
- WebSocket connection and streaming
|
||||
- Logs API timeouts
|
||||
- Notifications APIs
|
||||
- Caddy import session persistence
|
||||
- RBAC enforcement for backups and permissions
|
||||
|
||||
- Frontend Dev
|
||||
- Docker integration UI control visibility
|
||||
- UI state handling for notifications if backend responses are valid
|
||||
|
||||
- Playwright Dev
|
||||
- Strict-mode selector refinements
|
||||
- wait-helpers URL matching reliability
|
||||
|
||||
## 9. Confidence Score
|
||||
|
||||
Confidence: 78 percent
|
||||
|
||||
Rationale: Failure clusters are clear and repeated across suites, but root
|
||||
causes still require endpoint-level confirmation in backend logs and
|
||||
WebSocket diagnostics.
|
||||
119
docs/plans/phase_2_interruption_analysis.md
Normal file
119
docs/plans/phase_2_interruption_analysis.md
Normal file
@@ -0,0 +1,119 @@
|
||||
---
|
||||
post_title: "Phase 2 Test Interruption Analysis"
|
||||
author1: "Charon Team"
|
||||
post_slug: "phase-2-test-interruption-analysis"
|
||||
microsoft_alias: "charon-team"
|
||||
featured_image: "https://wikid82.github.io/charon/assets/images/featured/charon.png"
|
||||
categories: ["testing"]
|
||||
tags: ["playwright", "e2e", "phase-2", "diagnostics"]
|
||||
ai_note: "true"
|
||||
summary: "Analysis of Phase 2 Playwright interruptions, with root cause
|
||||
hypotheses and recommended diagnostics to stabilize execution."
|
||||
post_date: "2026-02-09"
|
||||
---
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
This analysis investigates Phase 2 Playwright test interruptions that surface
|
||||
as exit code 130 and errors such as "Target page, context or browser has been
|
||||
closed" or "Test ended." The focus is on the setup/teardown flow, storage
|
||||
state usage, and security-related dependencies that can affect browser
|
||||
lifecycles.
|
||||
|
||||
## 2. Research Findings
|
||||
|
||||
### 2.1 Setup and Auth Storage State Flow
|
||||
|
||||
- Phase 1 authentication is performed in [tests/auth.setup.ts](tests/auth.setup.ts)
|
||||
using the Playwright request context, with storage state persisted to the
|
||||
path defined by [tests/constants.ts](tests/constants.ts).
|
||||
- The auth setup validates cookie domain alignment with `baseURL` and writes
|
||||
the storage state to `playwright/.auth/user.json`, which Phase 2 uses.
|
||||
- Global setup in [tests/global-setup.ts](tests/global-setup.ts) performs
|
||||
preflight health checks, optional emergency security reset, and cleanup.
|
||||
It does not explicitly close browsers, but it can terminate the process if
|
||||
the emergency token is invalid.
|
||||
|
||||
### 2.2 Playwright Project Dependencies and Scheduling
|
||||
|
||||
- [playwright.config.js](playwright.config.js) declares projects for `setup`,
|
||||
`security-tests`, `security-teardown`, and the browser projects.
|
||||
- When `PLAYWRIGHT_SKIP_SECURITY_DEPS` is set (default), browser projects depend
|
||||
only on `setup`, but `security-teardown` remains a standalone project and can
|
||||
still run during the same invocation.
|
||||
- Fully parallel execution is enabled, so independent projects may run in
|
||||
parallel unless constrained.
|
||||
|
||||
### 2.3 Security Teardown Behavior
|
||||
|
||||
- [tests/security-teardown.setup.ts](tests/security-teardown.setup.ts) uses an
|
||||
authenticated API request context and enforces a specific security state:
|
||||
Cerberus enabled, modules enabled, and admin whitelist configured.
|
||||
- This teardown is documented as a post-security-test action, but it is not
|
||||
guarded from running alongside non-security browser tests.
|
||||
|
||||
### 2.4 Test-Level Browser/Context Closure
|
||||
|
||||
- No Phase 2 core/settings/tasks tests explicitly close the shared browser.
|
||||
- A direct `context.close()` exists in
|
||||
[tests/monitoring/real-time-logs.spec.ts](tests/monitoring/real-time-logs.spec.ts),
|
||||
but it is scoped to a context created in `beforeAll` and should not affect
|
||||
other tests.
|
||||
|
||||
## 3. Root Cause Hypotheses (Ranked)
|
||||
|
||||
### H1: Security Teardown Runs Concurrently and Alters State Mid-Run
|
||||
|
||||
Because `security-teardown` is configured as a standalone project, it can run
|
||||
in parallel when security dependencies are skipped. Its API calls modify
|
||||
security settings and admin whitelist configuration. If this overlaps with
|
||||
Phase 2 navigation or API-driven setup, it can lead to transient 401/403
|
||||
responses, blocked routes, or reload events. The resulting timeouts can cause
|
||||
Playwright to end tests early, which produces "Test ended" and "Target page,
|
||||
context or browser has been closed" errors.
|
||||
|
||||
### H2: Storage State Domain Mismatch or Invalid State
|
||||
|
||||
`auth.setup.ts` writes storage state for the `baseURL` domain. If Phase 2 runs
|
||||
with a different `PLAYWRIGHT_BASE_URL` host (for example, `localhost` vs
|
||||
`127.0.0.1`), cookies may not be sent, leading to authentication failures.
|
||||
Tests that assume an authenticated session can then hang in navigation or UI
|
||||
assertions, eventually triggering test end and context closure errors.
|
||||
|
||||
### H3: Global Setup Emergency Reset Leaves Partial State
|
||||
|
||||
Global setup performs an emergency reset and a post-auth reset if storage state
|
||||
exists. If the emergency server or token validation is inconsistent, the reset
|
||||
may partially apply, leaving security modules enabled without the expected
|
||||
whitelist. This can block subsequent API calls and surface as navigation
|
||||
timeouts, especially early in Phase 2.
|
||||
|
||||
### H4: Environment Instability or Resource Pressure
|
||||
|
||||
If the Docker container becomes unresponsive, navigation requests can fail or
|
||||
hang, which can cascade into test termination. No direct evidence is present in
|
||||
this review, but the symptoms are consistent with sudden page closures after a
|
||||
few tests.
|
||||
|
||||
## 4. Recommendations for Next Steps
|
||||
|
||||
- Confirm whether `security-teardown` is executing during Phase 2 runs and
|
||||
whether it overlaps with browser projects. If it does, isolate it to only run
|
||||
when `security-tests` are executed or after them.
|
||||
- Validate storage state consistency by confirming Phase 2 uses the same
|
||||
`PLAYWRIGHT_BASE_URL` host as Phase 1 authentication. Align to a single host
|
||||
(`127.0.0.1` or `localhost`) and keep it consistent across setup and tests.
|
||||
- Capture Playwright project scheduling output and test order to confirm whether
|
||||
any teardown is running concurrently with Phase 2 suites.
|
||||
- Add a lightweight health check in Phase 2 suites (or in the base fixture) to
|
||||
detect server unresponsiveness early and surface actionable failures instead
|
||||
of page-closed errors.
|
||||
|
||||
## 5. Acceptance Criteria
|
||||
|
||||
- Phase 2 suites (core, settings, tasks) run to completion without premature
|
||||
browser/context closure errors.
|
||||
- No occurrences of "page.goto: Target page, context or browser has been closed"
|
||||
or "page.goto: Test ended" in Phase 2 runs.
|
||||
- Project scheduling confirms `security-teardown` does not run concurrently with
|
||||
non-security browser projects when security dependencies are skipped.
|
||||
77
docs/plans/phase_2_test_organization_audit.md
Normal file
77
docs/plans/phase_2_test_organization_audit.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Phase 2 Test Organization Audit
|
||||
|
||||
**Date**: 2026-02-09
|
||||
|
||||
## Scope
|
||||
|
||||
Phase 2 runs with `PLAYWRIGHT_SKIP_SECURITY_DEPS=1`, so security modules are disabled. This audit flags tests in Phase 2 folders that exercise security UI or security-dependent workflows and should be relocated.
|
||||
|
||||
## Findings From Phase 2 Failures
|
||||
|
||||
No Phase 2 failure messages reference ACL blocks, WAF, rate limiting, or CrowdSec enforcement. The recorded failures are interruption/teardown errors, not security enforcement failures. Security-dependent tests are still present in Phase 2 suites and should be relocated to avoid running with security disabled.
|
||||
|
||||
## Misorganized Tests (Relocate)
|
||||
|
||||
### Move to tests/security/ (security UI/config)
|
||||
|
||||
- [tests/core/access-lists-crud.spec.ts](tests/core/access-lists-crud.spec.ts)
|
||||
- **Tests**: `Access Lists - CRUD Operations` (entire file)
|
||||
- **Reason**: Access lists are a Cerberus security feature; these tests validate security configuration UI and should not run with security disabled.
|
||||
|
||||
- [tests/settings/system-settings.spec.ts](tests/settings/system-settings.spec.ts)
|
||||
- **Tests**: `should toggle Cerberus security feature`, `should toggle CrowdSec console enrollment`, `should persist feature toggle changes`, `should handle concurrent toggle operations`, `should retry on 500 Internal Server Error`, `should fail gracefully after max retries exceeded`
|
||||
- **Reason**: These tests explicitly change security feature flags and expect propagation that only makes sense when security is enabled and being exercised.
|
||||
- **Note**: Remaining non-security system settings tests can stay in Phase 2; recommend splitting into a security toggles spec.
|
||||
|
||||
- [tests/settings/encryption-management.spec.ts](tests/settings/encryption-management.spec.ts)
|
||||
- **Tests**: `Encryption Management` (entire file)
|
||||
- **Reason**: Encryption management is a security area under `/security/encryption` and depends on security configuration state.
|
||||
|
||||
- [tests/tasks/import-crowdsec.spec.ts](tests/tasks/import-crowdsec.spec.ts)
|
||||
- **Tests**: `Import CrowdSec Configuration` (entire file)
|
||||
- **Reason**: CrowdSec import is a security configuration workflow; it should run with security enabled.
|
||||
|
||||
- [tests/monitoring/real-time-logs.spec.ts](tests/monitoring/real-time-logs.spec.ts)
|
||||
- **Tests**: `Real-Time Logs Viewer` (entire file)
|
||||
- **Reason**: The suite explicitly requires Cerberus to render the LiveLogViewer and exercises security-mode log streams and filters.
|
||||
- **Note**: If a future split is desired, only the App Logs mode tests should remain in Phase 2.
|
||||
|
||||
### Move to tests/security-enforcement/ (blocking/enforcement)
|
||||
|
||||
- **None identified in Phase 2 suites.**
|
||||
- The Phase 2 failures list does not include enforcement messages like ACL blocks, WAF violations, or rate-limit errors.
|
||||
|
||||
## Phase 2 Tests Likely Failing for Environmental Reasons (Keep)
|
||||
|
||||
- [tests/settings/account-settings.spec.ts](tests/settings/account-settings.spec.ts)
|
||||
- **Failure type**: `page.goto` interrupted / test ended
|
||||
- **Reason**: Interruption/teardown, not security-related.
|
||||
|
||||
- [tests/tasks/backups-create.spec.ts](tests/tasks/backups-create.spec.ts)
|
||||
- **Failure type**: `Browser.removeBrowserContext` / `Test ended`
|
||||
- **Reason**: Browser context teardown, not security-related.
|
||||
|
||||
- [tests/utils/wait-helpers.spec.ts](tests/utils/wait-helpers.spec.ts)
|
||||
- **Failure type**: Suite interrupted before execution
|
||||
- **Reason**: Test run interruption, not security-related.
|
||||
|
||||
## Relocation Summary
|
||||
|
||||
- **Move to tests/security/**: 5 files
|
||||
- Access Lists CRUD
|
||||
- System Settings security toggles (subset of tests)
|
||||
- Encryption Management
|
||||
- Import CrowdSec
|
||||
- Real-Time Logs Viewer
|
||||
|
||||
- **Move to tests/security-enforcement/**: 0 files
|
||||
|
||||
- **Keep in Phase 2** (but investigate interruptions): 3 files
|
||||
|
||||
## Recommended Moves
|
||||
|
||||
1. Move Access Lists CRUD to tests/security/.
|
||||
2. Split System Settings tests so security toggles move to tests/security/.
|
||||
3. Move Encryption Management to tests/security/.
|
||||
4. Move Import CrowdSec to tests/security/.
|
||||
5. Move Real-Time Logs Viewer to tests/security/ (or split to keep App Logs only in Phase 2).
|
||||
@@ -26,7 +26,10 @@ const STORAGE_STATE = join(__dirname, 'playwright/.auth/user.json');
|
||||
* Enabled by default, disable with PLAYWRIGHT_COVERAGE=0
|
||||
*/
|
||||
const enableCoverage = process.env.PLAYWRIGHT_COVERAGE !== '0';
|
||||
const skipSecurityDeps = process.env.PLAYWRIGHT_SKIP_SECURITY_DEPS === '1';
|
||||
// Skip security-test dependencies by default to avoid running them as a
|
||||
// prerequisite for non-security test runs. Set PLAYWRIGHT_SKIP_SECURITY_DEPS=0
|
||||
// to restore the legacy dependency behavior when needed.
|
||||
const skipSecurityDeps = process.env.PLAYWRIGHT_SKIP_SECURITY_DEPS !== '0';
|
||||
const browserDependencies = skipSecurityDeps ? ['setup'] : ['setup', 'security-tests'];
|
||||
|
||||
const coverageReporterConfig = enableCoverage ? defineCoverageReporterConfig({
|
||||
@@ -236,9 +239,11 @@ export default defineConfig({
|
||||
},
|
||||
|
||||
// Security Teardown - Disable ALL security modules
|
||||
// Conditionally disabled when skipSecurityDeps is true
|
||||
// When disabled, tests cannot find this project and it won't run
|
||||
{
|
||||
name: 'security-teardown',
|
||||
testMatch: /security-teardown\.setup\.ts/,
|
||||
testMatch: skipSecurityDeps ? [] : /security-teardown\.setup\.ts/,
|
||||
},
|
||||
|
||||
// Browser projects - standard Playwright pattern
|
||||
|
||||
@@ -37,9 +37,18 @@ test.describe('SSL Certificates - CRUD Operations', () => {
|
||||
test.beforeEach(async ({ page, adminUser }) => {
|
||||
await loginUser(page, adminUser);
|
||||
await waitForLoadingComplete(page);
|
||||
// Navigate to certificates page
|
||||
await page.goto('/certificates');
|
||||
await waitForLoadingComplete(page);
|
||||
// Navigate to certificates page (retry once on transient failures)
|
||||
for (let i = 0; i < 2; i++) {
|
||||
try {
|
||||
await page.goto('/certificates');
|
||||
await waitForLoadingComplete(page);
|
||||
break;
|
||||
} catch (err) {
|
||||
if (i === 1) throw err;
|
||||
// short backoff and retry
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Helper to get the Add Certificate button
|
||||
@@ -263,7 +272,9 @@ test.describe('SSL Certificates - CRUD Operations', () => {
|
||||
const nameInput = dialog.locator('input').first();
|
||||
await expect(nameInput).toBeVisible();
|
||||
|
||||
// Close dialog
|
||||
// Close dialog (guard visibility/enabled to avoid transient flakiness)
|
||||
await expect(getCancelButton(page)).toBeVisible({ timeout: 3000 });
|
||||
await expect(getCancelButton(page)).toBeEnabled({ timeout: 3000 });
|
||||
await getCancelButton(page).click();
|
||||
});
|
||||
});
|
||||
@@ -306,6 +317,8 @@ test.describe('SSL Certificates - CRUD Operations', () => {
|
||||
});
|
||||
|
||||
await test.step('Close dialog', async () => {
|
||||
await expect(getCancelButton(page)).toBeVisible({ timeout: 3000 });
|
||||
await expect(getCancelButton(page)).toBeEnabled({ timeout: 3000 });
|
||||
await getCancelButton(page).click();
|
||||
});
|
||||
});
|
||||
@@ -327,6 +340,8 @@ test.describe('SSL Certificates - CRUD Operations', () => {
|
||||
});
|
||||
|
||||
await test.step('Close dialog', async () => {
|
||||
await expect(getCancelButton(page)).toBeVisible({ timeout: 3000 });
|
||||
await expect(getCancelButton(page)).toBeEnabled({ timeout: 3000 });
|
||||
await getCancelButton(page).click();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -13,10 +13,10 @@
|
||||
* @see /projects/Charon/docs/plans/current_spec.md - Phase 2
|
||||
*/
|
||||
|
||||
import { test, expect, loginUser, TEST_PASSWORD } from '../fixtures/auth-fixtures';
|
||||
import { waitForLoadingComplete, waitForToast, waitForModal, waitForDialog, waitForDebounce } from '../utils/wait-helpers';
|
||||
import { waitForAPIHealth } from '../utils/api-helpers';
|
||||
import { clickSwitch } from '../utils/ui-helpers';
|
||||
import { test, expect, loginUser, TEST_PASSWORD } from '../../fixtures/auth-fixtures';
|
||||
import { waitForLoadingComplete, waitForToast, waitForModal, waitForDialog, waitForDebounce } from '../../utils/wait-helpers';
|
||||
import { waitForAPIHealth } from '../../utils/api-helpers';
|
||||
import { clickSwitch } from '../../utils/ui-helpers';
|
||||
import {
|
||||
allowOnlyAccessList,
|
||||
denyOnlyAccessList,
|
||||
@@ -25,8 +25,8 @@ import {
|
||||
generateAccessList,
|
||||
invalidACLConfigs,
|
||||
type AccessListConfig,
|
||||
} from '../fixtures/access-lists';
|
||||
import { generateUniqueId, generateIPAddress, generateCIDR } from '../fixtures/test-data';
|
||||
} from '../../fixtures/access-lists';
|
||||
import { generateUniqueId, generateIPAddress, generateCIDR } from '../../fixtures/test-data';
|
||||
|
||||
test.describe('Access Lists - CRUD Operations', () => {
|
||||
test.beforeEach(async ({ page, adminUser }) => {
|
||||
@@ -10,8 +10,8 @@
|
||||
* - Import Execution (3 tests): import success, error handling, already exists
|
||||
*/
|
||||
|
||||
import { test, expect, loginUser } from '../fixtures/auth-fixtures';
|
||||
import { waitForToast, waitForLoadingComplete, waitForAPIResponse } from '../utils/wait-helpers';
|
||||
import { test, expect, loginUser } from '../../fixtures/auth-fixtures';
|
||||
import { waitForToast, waitForLoadingComplete, waitForAPIResponse } from '../../utils/wait-helpers';
|
||||
|
||||
/**
|
||||
* Selectors for the Import CrowdSec page
|
||||
@@ -14,8 +14,8 @@
|
||||
* @see /projects/Charon/docs/plans/phase4-settings-plan.md Section 3.5
|
||||
*/
|
||||
|
||||
import { test, expect, loginUser } from '../fixtures/auth-fixtures';
|
||||
import { waitForLoadingComplete, waitForToast } from '../utils/wait-helpers';
|
||||
import { test, expect, loginUser } from '../../fixtures/auth-fixtures';
|
||||
import { waitForLoadingComplete, waitForToast } from '../../utils/wait-helpers';
|
||||
|
||||
test.describe('Encryption Management', () => {
|
||||
test.beforeEach(async ({ page, adminUser }) => {
|
||||
@@ -13,8 +13,8 @@
|
||||
* - Performance (2 tests): high volume logs, buffer limits
|
||||
*/
|
||||
|
||||
import { test, expect, loginUser } from '../fixtures/auth-fixtures';
|
||||
import { waitForToast, waitForLoadingComplete } from '../utils/wait-helpers';
|
||||
import { test, expect, loginUser } from '../../fixtures/auth-fixtures';
|
||||
import { waitForToast, waitForLoadingComplete } from '../../utils/wait-helpers';
|
||||
|
||||
/**
|
||||
* TypeScript interfaces matching the API
|
||||
@@ -54,11 +54,11 @@
|
||||
* @see /projects/Charon/docs/plans/phase4-settings-plan.md
|
||||
*/
|
||||
|
||||
import { test, expect, loginUser } from '../fixtures/auth-fixtures';
|
||||
import { test, expect, loginUser } from '../../fixtures/auth-fixtures';
|
||||
import {
|
||||
waitForLoadingComplete,
|
||||
} from '../utils/wait-helpers';
|
||||
import { getToastLocator } from '../utils/ui-helpers';
|
||||
} from '../../utils/wait-helpers';
|
||||
import { getToastLocator } from '../../utils/ui-helpers';
|
||||
|
||||
test.describe('System Settings', () => {
|
||||
test.beforeEach(async ({ page, adminUser }) => {
|
||||
@@ -70,9 +70,13 @@ test.describe('Backups Page - Creation and List', () => {
|
||||
await page.goto('/tasks/backups');
|
||||
await waitForLoadingComplete(page);
|
||||
|
||||
// Sanity check: verify UI reflects the logged-in guest identity before asserting
|
||||
const userIndicator = page.getByRole('button', { name: new RegExp(guestUser.email.split('@')[0], 'i') }).first();
|
||||
await expect(userIndicator).toBeVisible({ timeout: 5000 });
|
||||
|
||||
// Guest users should not see any Create Backup button
|
||||
const createButton = page.locator(SELECTORS.createBackupButton);
|
||||
await expect(createButton).toHaveCount(0);
|
||||
await expect(createButton).toHaveCount(0, { timeout: 5000 });
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -142,10 +142,10 @@ test.describe('wait-helpers - Phase 2.1 Semantic Wait Functions', () => {
|
||||
`);
|
||||
|
||||
await page.click('#enable-field');
|
||||
await waitForFormFields(page, '#test-field', { shouldBeEnabled: true });
|
||||
await waitForFormFields(page, '#test-field', { shouldBeEnabled: true, timeout: 2000 });
|
||||
|
||||
const field = page.locator('#test-field');
|
||||
await expect(field).toBeEnabled();
|
||||
await expect(field).toBeEnabled({ timeout: 2000 });
|
||||
});
|
||||
|
||||
test('should handle disabled fields when shouldBeEnabled is false', async ({ page }) => {
|
||||
|
||||
@@ -199,7 +199,9 @@ export async function waitForAPIResponse(
|
||||
urlPattern: string | RegExp,
|
||||
options: APIResponseOptions = {}
|
||||
): Promise<Response> {
|
||||
const { status, timeout = 30000 } = options;
|
||||
// Increase default timeout to 60s to tolerate slower CI/backends; individual
|
||||
// tests may override this if they expect faster responses.
|
||||
const { status, timeout = 60000 } = options;
|
||||
|
||||
const responsePromise = page.waitForResponse(
|
||||
(response) => {
|
||||
|
||||
Reference in New Issue
Block a user