diff --git a/docs/issues/validate_e2e_infrastructure.md b/docs/issues/validate_e2e_infrastructure.md new file mode 100644 index 00000000..7d8c794e --- /dev/null +++ b/docs/issues/validate_e2e_infrastructure.md @@ -0,0 +1,11 @@ +# Manual Validation of E2E Test Infrastructure + +- Test the following scenarios manually (or verifying via CI output): + 1. Verify `crowdsec-diagnostics.spec.ts` does NOT run in standard `chromium` shards. + 2. Verify `tests/security/acl-integration.spec.ts` passes consistently (no 401s, no modal errors). + 3. Verify `waitForModal` helper works for both standard dialogs and slide-out panels. + 4. Verify Authentication setup (`auth.setup.ts`) works with `127.0.0.1` domain. + +Status: To Do +Priority: Medium +Assignee: QA Automation Team diff --git a/docs/reports/qa_report.md b/docs/reports/qa_report.md index 89e3fb9a..a6e9cf34 100644 --- a/docs/reports/qa_report.md +++ b/docs/reports/qa_report.md @@ -1,4 +1,92 @@ -# QA Report: Project Health Check +# QA Report - Phase 6 Audit (Playwright Config Update) + +**Date:** February 6, 2026 +**Trigger:** Update of `playwright.config.js` to separate and sequence security tests. +**Auditor:** QA Security Engineer (Gemini 3 Pro) + +## 1. Executive Summary + +The Phase 6 Audit was performed to validate the new Playwright configuration which splits security tests into a separate project that runs prior to standard browser tests. + +**Status:** šŸ”“ **FAILED** + +While the configuration successfully enforced the execution order (security tests ran first), the security tests themselves failed due to authentication issues in the test environment. This failure, combined with the new dependency structure, caused the majority of the standard E2E suite (1964 tests) to be skipped. + +Security scans identified 1 High-severity misconfiguration in the Dockerfile and 2 High-severity vulnerabilities in the container base image. + +## 2. E2E Test Execution Analysis + +### Execution Order Verification +* **Result:** āœ… **Verified** +* **Observation:** The `security-tests` project executed before `chromium`, `firefox`, and `webkit` projects as configured. + +### Test Results +* **Total Tests Run:** 219 +* **Passed:** 201 +* **Failed:** 18 +* **Skipped / Not Run:** 1,964 +* **Pass Rate:** ~9% (of total suite) / 91% (of executed tests) + +### Failure Analysis +The 18 failed tests were all within the `security-tests` project. The failures were consistent `401 Unauthorized` errors during test setup/teardown helpers. + +**Key Error:** +``` +Failed to enable Cerberus: Error: Failed to set cerberus to true: 401 {"error":"Authorization header required"} +``` + +**Impacted Areas:** +1. **Security Helpers:** `setSecurityModuleEnabled()`, `getSecurityStatus()`, `configureAdminWhitelist()` in `tests/utils/security-helpers.ts`. +2. **Tests:** + * `security-enforcement/acl-enforcement.spec.ts` + * `security-enforcement/combined-enforcement.spec.ts` + * `security-enforcement/crowdsec-enforcement.spec.ts` + * `security-enforcement/rate-limit-enforcement.spec.ts` + * `security-enforcement/waf-enforcement.spec.ts` + * `security/acl-integration.spec.ts` (Also failed finding UI modals) + +**Root Cause Hypothesis:** +The test environment (`charon-e2e` container) requires authentication for the management API (`/api/v1/security/*`), but the test helper functions are failing to provide a valid Authorization header or session cookie in the current context. + +**Blocking Issue:** +Because `chromium` etc. depend on `security-tests`, the failure of the security suite prevented the standard browser tests from running. + +## 3. Security Scan Findings + +### Trivy Filesystem Scan +* **Command:** `trivy fs /projects/Charon --skip-dirs .cache` +* **Findings:** + * **Dockerfile:** 1 šŸ”“ HIGH Misconfiguration + * **ID:** DS-0002 + * **Message:** "Image user should not be 'root'" + * **Resolution:** Add `USER ` instruction. + +### Trivy Docker Image Scan +* **Target:** `charon:local` (Debian 13.3) +* **Findings:** + * **Total:** 2 šŸ”“ HIGH Vulnerabilities + * **CVE-2026-0861** (`libc-bin`, `libc6`): Integer overflow in `memalign` leading to heap corruption. + * **Status:** Fix available in upstream Debian (upgrade required). + +## 4. Recommendations & Next Steps + +### Immediate Actions (Blockers) +1. **Fix Test Authentication:** Investigate `tests/utils/security-helpers.ts`. Ensure it properly authenticates (e.g., logs in via UI or uses a valid API token) before attempting to configure security modules. Inspect `.env` usage in the E2E container. +2. **Fix UI Interaction:** Investigate `waitForModal` failures in `acl-integration.spec.ts`. The UI might have changed, breaking the locator `"/edit|proxy/i"`. + +### Security Remediation +1. **Dockerfile Hardening:** implementation of a non-root user in the `Dockerfile`. +2. **Base Image Update:** Re-pull the base image (`debian:bookworm-slim` or equivalent) to pick up the patch for CVE-2026-0861, or ensure `apt-get upgrade` runs during build. + +### Configuration Adjustment +* **Consider Fail-Open for Dev:** While serial execution is good for CI, consider if local development requires `dependencies: ['security-tests']` to be strict, or if we can allow specific headers/tokens to bypass this for easier debugging. + +## 5. Conclusion +The separation of security tests is sound, but the current state of the security test suite is unstable. Prioritize fixing the 401 errors in the security helpers to unblock the rest of the E2E suite. + +--- + +# QA Report: Project Health Check (Previous) **Date**: 2026-02-05 **Version**: v0.18.13 diff --git a/playwright.config.js b/playwright.config.js index 2d16bc28..d88c2855 100644 --- a/playwright.config.js +++ b/playwright.config.js @@ -182,7 +182,8 @@ export default defineConfig({ ...devices['Desktop Chrome'], storageState: STORAGE_STATE, }, - dependencies: ['setup'], + dependencies: ['setup', 'security-tests'], + testIgnore: ['**/frontend/**', '**/node_modules/**', '**/backend/**', '**/security-enforcement/**', '**/security/**'], }, { @@ -192,6 +193,7 @@ export default defineConfig({ storageState: STORAGE_STATE, }, dependencies: ['setup', 'security-tests'], + testIgnore: ['**/frontend/**', '**/node_modules/**', '**/backend/**', '**/security-enforcement/**', '**/security/**'], }, { @@ -201,6 +203,7 @@ export default defineConfig({ storageState: STORAGE_STATE, }, dependencies: ['setup', 'security-tests'], + testIgnore: ['**/frontend/**', '**/node_modules/**', '**/backend/**', '**/security-enforcement/**', '**/security/**'], }, /* Test against mobile viewports. */ diff --git a/tests/security-enforcement/acl-enforcement.spec.ts b/tests/security-enforcement/acl-enforcement.spec.ts index e0ab94c2..09beda20 100644 --- a/tests/security-enforcement/acl-enforcement.spec.ts +++ b/tests/security-enforcement/acl-enforcement.spec.ts @@ -33,7 +33,7 @@ async function configureAdminWhitelist(requestContext: APIRequestContext) { const testWhitelist = '127.0.0.1/32,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8'; const response = await requestContext.patch( - `${process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'}/api/v1/config`, + `${process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'}/api/v1/config`, { data: { security: { @@ -56,7 +56,7 @@ test.describe('ACL Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/security-enforcement/combined-enforcement.spec.ts b/tests/security-enforcement/combined-enforcement.spec.ts index c10f68b9..b2ba69fa 100644 --- a/tests/security-enforcement/combined-enforcement.spec.ts +++ b/tests/security-enforcement/combined-enforcement.spec.ts @@ -37,7 +37,7 @@ async function configureAdminWhitelist(requestContext: APIRequestContext) { const testWhitelist = '127.0.0.1/32,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8'; const response = await requestContext.patch( - `${process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'}/api/v1/config`, + `${process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'}/api/v1/config`, { data: { security: { @@ -60,7 +60,7 @@ test.describe('Combined Security Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); @@ -166,7 +166,7 @@ test.describe('Combined Security Enforcement', () => { // Create a new request context to simulate fresh session const freshContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/security-enforcement/crowdsec-enforcement.spec.ts b/tests/security-enforcement/crowdsec-enforcement.spec.ts index f4fc0243..525a9d7b 100644 --- a/tests/security-enforcement/crowdsec-enforcement.spec.ts +++ b/tests/security-enforcement/crowdsec-enforcement.spec.ts @@ -29,7 +29,7 @@ async function configureAdminWhitelist(requestContext: APIRequestContext) { const testWhitelist = '127.0.0.1/32,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8'; const response = await requestContext.patch( - `${process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'}/api/v1/config`, + `${process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'}/api/v1/config`, { data: { security: { @@ -52,7 +52,7 @@ test.describe('CrowdSec Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/security-enforcement/rate-limit-enforcement.spec.ts b/tests/security-enforcement/rate-limit-enforcement.spec.ts index 82fe3e4a..6776c030 100644 --- a/tests/security-enforcement/rate-limit-enforcement.spec.ts +++ b/tests/security-enforcement/rate-limit-enforcement.spec.ts @@ -32,7 +32,7 @@ async function configureAdminWhitelist(requestContext: APIRequestContext) { const testWhitelist = '127.0.0.1/32,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8'; const response = await requestContext.patch( - `${process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'}/api/v1/config`, + `${process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'}/api/v1/config`, { data: { security: { @@ -55,7 +55,7 @@ test.describe('Rate Limit Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/security-enforcement/security-headers-enforcement.spec.ts b/tests/security-enforcement/security-headers-enforcement.spec.ts index cc298600..755d21f0 100644 --- a/tests/security-enforcement/security-headers-enforcement.spec.ts +++ b/tests/security-enforcement/security-headers-enforcement.spec.ts @@ -19,7 +19,7 @@ test.describe('Security Headers Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); }); diff --git a/tests/security-enforcement/waf-enforcement.spec.ts b/tests/security-enforcement/waf-enforcement.spec.ts index ff9c1d73..5cfb7942 100644 --- a/tests/security-enforcement/waf-enforcement.spec.ts +++ b/tests/security-enforcement/waf-enforcement.spec.ts @@ -40,7 +40,7 @@ async function configureAdminWhitelist(requestContext: APIRequestContext) { const testWhitelist = '127.0.0.1/32,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8'; const response = await requestContext.patch( - `${process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'}/api/v1/config`, + `${process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'}/api/v1/config`, { data: { security: { @@ -63,7 +63,7 @@ test.describe('WAF Enforcement', () => { test.beforeAll(async () => { requestContext = await request.newContext({ - baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/security-enforcement/zzz-admin-whitelist-blocking.spec.ts b/tests/security-enforcement/zzz-admin-whitelist-blocking.spec.ts index 0e771b47..b1d99d58 100644 --- a/tests/security-enforcement/zzz-admin-whitelist-blocking.spec.ts +++ b/tests/security-enforcement/zzz-admin-whitelist-blocking.spec.ts @@ -14,7 +14,7 @@ import { test, expect } from '@playwright/test'; test.describe.serial('Admin Whitelist IP Blocking (RUN LAST)', () => { const EMERGENCY_TOKEN = process.env.CHARON_EMERGENCY_TOKEN; - const BASE_URL = process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'; + const BASE_URL = process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'; test.beforeAll(() => { if (!EMERGENCY_TOKEN) { diff --git a/tests/security-enforcement/zzzz-break-glass-recovery.spec.ts b/tests/security-enforcement/zzzz-break-glass-recovery.spec.ts index f3acac65..27053829 100644 --- a/tests/security-enforcement/zzzz-break-glass-recovery.spec.ts +++ b/tests/security-enforcement/zzzz-break-glass-recovery.spec.ts @@ -33,7 +33,7 @@ import { test, expect } from '@playwright/test'; test.describe.serial('Break Glass Recovery - Universal Bypass', () => { const EMERGENCY_TOKEN = process.env.CHARON_EMERGENCY_TOKEN; const EMERGENCY_URL = 'http://localhost:2020'; - const BASE_URL = process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'; + const BASE_URL = process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'; test.beforeAll(() => { if (!EMERGENCY_TOKEN) { diff --git a/tests/security-teardown.setup.ts b/tests/security-teardown.setup.ts index 639c5ba8..59c02b00 100644 --- a/tests/security-teardown.setup.ts +++ b/tests/security-teardown.setup.ts @@ -29,7 +29,7 @@ teardown('verify-security-state-for-ui-tests', async () => { console.log('\nšŸ” Security Teardown: Verifying state for UI tests...'); console.log(' Expected: Cerberus ON + All modules ON + Universal bypass (0.0.0.0/0)'); - const baseURL = process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:8080'; + const baseURL = process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080'; // Create authenticated request context with storage state const requestContext = await request.newContext({ diff --git a/tests/security/security-dashboard.spec.ts b/tests/security/security-dashboard.spec.ts index 7918d5fd..c0b15985 100644 --- a/tests/security/security-dashboard.spec.ts +++ b/tests/security/security-dashboard.spec.ts @@ -133,7 +133,7 @@ test.describe('Security Dashboard @security', () => { // Create authenticated request context for cleanup (cannot reuse fixture from beforeAll) const cleanupRequest = await request.newContext({ - baseURL: 'http://localhost:8080', + baseURL: process.env.PLAYWRIGHT_BASE_URL || 'http://127.0.0.1:8080', storageState: STORAGE_STATE, }); diff --git a/tests/utils/wait-helpers.ts b/tests/utils/wait-helpers.ts index 8ec67100..9a9e4bba 100644 --- a/tests/utils/wait-helpers.ts +++ b/tests/utils/wait-helpers.ts @@ -413,27 +413,33 @@ export async function waitForModal( const { timeout = 10000 } = options; // Try to find a modal dialog first, then fall back to a slide-out panel with matching heading - const dialogModal = page.locator('[role="dialog"], .modal'); - const slideOutPanel = page.locator('h2, h3').filter({ hasText: titleText }); + // Use .first() to avoid specific strict mode violations if multiple exist in DOM + const dialogModal = page + .locator('[role="dialog"], .modal') + .filter({ hasText: titleText }) + .first(); + + const slideOutPanel = page + .locator('h2, h3') + .filter({ hasText: titleText }) + .first(); // Wait for either the dialog modal or the slide-out panel heading to be visible try { - await expect(dialogModal.or(slideOutPanel)).toBeVisible({ timeout }); - } catch { + // FIX STRICT MODE VIOLATION: + // If we match both the dialog AND the heading inside it, .or() returns 2 elements. + // We strictly want to wait until *at least one* is visible. + // Using .first() on the combined locator prevents 'strict mode violation' when both match. + await expect(dialogModal.or(slideOutPanel).first()).toBeVisible({ timeout }); + } catch (e) { // If neither is found, throw a more helpful error throw new Error( - `waitForModal: Could not find modal dialog or slide-out panel matching "${titleText}"` + `waitForModal: Could not find visible modal dialog or slide-out panel matching "${titleText}". Error: ${e instanceof Error ? e.message : String(e)}` ); } - // If dialog modal is visible, verify its title + // If dialog modal is visible, use it if (await dialogModal.isVisible()) { - if (titleText) { - const titleLocator = dialogModal.locator( - '[role="heading"], .modal-title, .dialog-title, h1, h2, h3' - ); - await expect(titleLocator).toContainText(titleText); - } return dialogModal; }