fix(ci): streamline Playwright configuration and remove preflight setup test
This commit is contained in:
@@ -9,7 +9,6 @@ applyTo: '**'
|
||||
- **Locators**: Prioritize user-facing, role-based locators (`getByRole`, `getByLabel`, `getByText`, etc.) for resilience and accessibility. Use `test.step()` to group interactions and improve test readability and reporting.
|
||||
- **Assertions**: Use auto-retrying web-first assertions. These assertions start with the `await` keyword (e.g., `await expect(locator).toHaveText()`). Avoid `expect(locator).toBeVisible()` unless specifically testing for visibility changes.
|
||||
- **Timeouts**: Rely on Playwright's built-in auto-waiting mechanisms. Avoid hard-coded waits or increased default timeouts.
|
||||
- **Switch/Toggle Components**: Use helper functions from `tests/utils/ui-helpers.ts` (`clickSwitch`, `expectSwitchState`, `toggleSwitch`) for reliable interactions. Never use `{ force: true }` or direct clicks on hidden inputs.
|
||||
- **Clarity**: Use descriptive test and step titles that clearly state the intent. Add comments only to explain complex logic or non-obvious interactions.
|
||||
|
||||
|
||||
@@ -30,123 +29,6 @@ applyTo: '**'
|
||||
- **Element Counts**: Use `toHaveCount` to assert the number of elements found by a locator.
|
||||
- **Text Content**: Use `toHaveText` for exact text matches and `toContainText` for partial matches.
|
||||
- **Navigation**: Use `toHaveURL` to verify the page URL after an action.
|
||||
- **Switch States**: Use `expectSwitchState(locator, boolean)` to verify toggle states. This is more reliable than `toBeChecked()` directly.
|
||||
|
||||
### Switch/Toggle Interaction Patterns
|
||||
|
||||
Switch components use a hidden `<input>` with styled siblings, requiring special handling:
|
||||
|
||||
```typescript
|
||||
import { clickSwitch, expectSwitchState, toggleSwitch } from './utils/ui-helpers';
|
||||
|
||||
// ✅ RECOMMENDED: Click switch with helper
|
||||
const aclSwitch = page.getByRole('switch', { name: /acl/i });
|
||||
await clickSwitch(aclSwitch);
|
||||
|
||||
// ✅ RECOMMENDED: Assert switch state
|
||||
await expectSwitchState(aclSwitch, true); // Checked
|
||||
|
||||
// ✅ RECOMMENDED: Toggle and verify state change
|
||||
const newState = await toggleSwitch(aclSwitch);
|
||||
console.log(`Switch is now ${newState ? 'enabled' : 'disabled'}`);
|
||||
|
||||
// ❌ AVOID: Direct click on hidden input
|
||||
await aclSwitch.click(); // May fail in WebKit/Firefox
|
||||
|
||||
// ❌ AVOID: Force clicking (anti-pattern)
|
||||
await aclSwitch.click({ force: true }); // Bypasses real user behavior
|
||||
|
||||
// ❌ AVOID: Hard-coded waits
|
||||
await page.waitForTimeout(500); // Non-deterministic, slows tests
|
||||
```
|
||||
|
||||
**When to Use**:
|
||||
- Settings pages with enable/disable toggles
|
||||
- Security dashboard module switches (CrowdSec, ACL, WAF, Rate Limiting)
|
||||
- Access lists and configuration toggles
|
||||
- Any UI component using the `Switch` primitive from shadcn/ui
|
||||
|
||||
**References**:
|
||||
- [Helper Implementation](../../tests/utils/ui-helpers.ts)
|
||||
- [QA Report](../../docs/reports/qa_report.md)
|
||||
|
||||
### Testing Scope: E2E vs Integration
|
||||
|
||||
**CRITICAL:** Playwright E2E tests verify **UI/UX functionality** on the Charon management interface (port 8080). They should NOT test middleware enforcement behavior.
|
||||
|
||||
#### What E2E Tests SHOULD Cover
|
||||
|
||||
✅ **User Interface Interactions:**
|
||||
- Form submissions and validation
|
||||
- Navigation and routing
|
||||
- Visual state changes (toggles, badges, status indicators)
|
||||
- Authentication flows (login, logout, session management)
|
||||
- CRUD operations via the management API
|
||||
- Responsive design (mobile vs desktop layouts)
|
||||
- Accessibility (ARIA labels, keyboard navigation)
|
||||
|
||||
✅ **Example E2E Assertions:**
|
||||
```typescript
|
||||
// GOOD: Testing UI state
|
||||
await expect(aclToggle).toBeChecked();
|
||||
await expect(statusBadge).toHaveText('Active');
|
||||
await expect(page).toHaveURL('/proxy-hosts');
|
||||
|
||||
// GOOD: Testing API responses in management interface
|
||||
const response = await request.post('/api/v1/proxy-hosts', { data: hostConfig });
|
||||
expect(response.ok()).toBeTruthy();
|
||||
```
|
||||
|
||||
#### What E2E Tests should NOT Cover
|
||||
|
||||
❌ **Middleware Enforcement Behavior:**
|
||||
- Rate limiting blocking requests (429 responses)
|
||||
- ACL denying access based on IP rules (403 responses)
|
||||
- WAF blocking malicious payloads (SQL injection, XSS)
|
||||
- CrowdSec IP bans
|
||||
|
||||
❌ **Example Wrong E2E Assertions:**
|
||||
```typescript
|
||||
// BAD: Testing middleware behavior (rate limiting)
|
||||
for (let i = 0; i < 6; i++) {
|
||||
await request.post('/api/v1/emergency/reset');
|
||||
}
|
||||
expect(response.status()).toBe(429); // ❌ This tests Caddy middleware
|
||||
|
||||
// BAD: Testing WAF blocking
|
||||
await request.post('/api/v1/data', { data: "'; DROP TABLE users--" });
|
||||
expect(response.status()).toBe(403); // ❌ This tests Coraza WAF
|
||||
```
|
||||
|
||||
#### Integration Tests for Middleware
|
||||
|
||||
Middleware enforcement is verified by **integration tests** in `backend/integration/`:
|
||||
|
||||
- `cerberus_integration_test.go` - Overall security suite behavior
|
||||
- `coraza_integration_test.go` - WAF blocking (SQL injection, XSS)
|
||||
- `crowdsec_integration_test.go` - IP reputation and bans
|
||||
- `rate_limit_integration_test.go` - Request throttling
|
||||
|
||||
These tests run in Docker Compose with full Caddy+Cerberus stack and are executed in separate CI workflows.
|
||||
|
||||
#### When to Skip Tests
|
||||
|
||||
Use `test.skip()` for tests that require middleware enforcement:
|
||||
|
||||
```typescript
|
||||
test('should rate limit after 5 attempts', async ({ request }) => {
|
||||
test.skip(
|
||||
true,
|
||||
'Rate limiting enforced via Cerberus middleware (port 80). Verified in integration tests (backend/integration/).'
|
||||
);
|
||||
// Test body...
|
||||
});
|
||||
```
|
||||
|
||||
**Skip Reason Template:**
|
||||
```
|
||||
"[Behavior] enforced via Cerberus middleware (port 80). Verified in integration tests (backend/integration/)."
|
||||
```
|
||||
|
||||
|
||||
## Example Test Structure
|
||||
@@ -194,11 +76,6 @@ test.describe('Movie Search Feature', () => {
|
||||
4. **Validate**: Ensure tests pass consistently and cover the intended functionality
|
||||
5. **Report**: Provide feedback on test results and any issues discovered
|
||||
|
||||
### Execution Constraints
|
||||
|
||||
- **No Truncation**: Never pipe Playwright test output through `head`, `tail`, or other truncating commands. Playwright runs interactively and requires user input to quit when piped, causing the command to hang indefinitely.
|
||||
- **Full Output**: Always capture the complete test output to analyze failures accurately.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before finalizing tests, ensure:
|
||||
|
||||
@@ -20,13 +20,12 @@ const STORAGE_STATE = join(__dirname, 'playwright/.auth/user.json');
|
||||
|
||||
/**
|
||||
* Coverage reporter configuration for E2E tests
|
||||
* Tracks V8 coverage during Playwright test execution
|
||||
* Only loaded when PLAYWRIGHT_COVERAGE=1
|
||||
*/
|
||||
const coverageReporterConfig = defineCoverageReporterConfig({
|
||||
// Root directory for source file resolution
|
||||
sourceRoot: __dirname,
|
||||
const enableCoverage = process.env.PLAYWRIGHT_COVERAGE === '1';
|
||||
|
||||
// Exclude non-application code from coverage
|
||||
const coverageReporterConfig = enableCoverage ? defineCoverageReporterConfig({
|
||||
sourceRoot: __dirname,
|
||||
exclude: [
|
||||
'**/node_modules/**',
|
||||
'**/playwright/**',
|
||||
@@ -38,86 +37,60 @@ const coverageReporterConfig = defineCoverageReporterConfig({
|
||||
'**/dist/**',
|
||||
'**/build/**',
|
||||
],
|
||||
|
||||
// Output directory for coverage reports
|
||||
resultDir: join(__dirname, 'coverage/e2e'),
|
||||
|
||||
// Generate multiple report formats
|
||||
reports: [
|
||||
// HTML report for visual inspection
|
||||
['html'],
|
||||
// LCOV for Codecov upload
|
||||
['lcovonly', { file: 'lcov.info' }],
|
||||
// JSON for programmatic access
|
||||
['json', { file: 'coverage.json' }],
|
||||
// Text summary in console
|
||||
['text-summary', { file: null }],
|
||||
],
|
||||
|
||||
// Coverage watermarks (visual thresholds in HTML report)
|
||||
watermarks: {
|
||||
statements: [50, 80],
|
||||
branches: [50, 80],
|
||||
functions: [50, 80],
|
||||
lines: [50, 80],
|
||||
},
|
||||
// Path rewriting for source file resolution
|
||||
rewritePath: ({ absolutePath, relativePath }) => {
|
||||
// Handle paths from Docker container
|
||||
rewritePath: ({ absolutePath }) => {
|
||||
if (absolutePath.startsWith('/app/')) {
|
||||
return absolutePath.replace('/app/', `${__dirname}/`);
|
||||
}
|
||||
|
||||
// Handle Vite dev server paths (relative to frontend/src)
|
||||
// Vite serves files like "/src/components/Button.tsx"
|
||||
if (absolutePath.startsWith('/src/')) {
|
||||
return join(__dirname, 'frontend', absolutePath);
|
||||
}
|
||||
|
||||
// If path doesn't start with /, prepend frontend/src
|
||||
if (!absolutePath.startsWith('/') && !absolutePath.includes('/')) {
|
||||
// Bare filenames like "Button.tsx" - try to resolve to frontend/src
|
||||
return join(__dirname, 'frontend/src', absolutePath);
|
||||
}
|
||||
|
||||
return absolutePath;
|
||||
},
|
||||
});
|
||||
|
||||
const enableCoverage = process.env.PLAYWRIGHT_COVERAGE === '1';
|
||||
}) : null;
|
||||
|
||||
/**
|
||||
* @see https://playwright.dev/docs/test-configuration
|
||||
*/
|
||||
export default defineConfig({
|
||||
testDir: './tests',
|
||||
/* Ignore old/deprecated test directories */
|
||||
testIgnore: ['**/frontend/**', '**/node_modules/**', '**/backend/**'],
|
||||
/* Global timeout for each test - increased to 90s for feature flag propagation
|
||||
* CI uses 60s to fail fast in resource-constrained environment (2-core runners)
|
||||
*/
|
||||
|
||||
/* Standard globalSetup - runs once before all tests */
|
||||
globalSetup: './tests/global-setup.ts',
|
||||
|
||||
/* Timeouts */
|
||||
timeout: process.env.CI ? 60000 : 90000,
|
||||
/* Timeout for expect() assertions */
|
||||
expect: {
|
||||
timeout: 5000,
|
||||
},
|
||||
/* Run tests in files in parallel */
|
||||
expect: { timeout: 5000 },
|
||||
|
||||
/* Parallelization */
|
||||
fullyParallel: true,
|
||||
/* Fail the build on CI if you accidentally left test.only in the source code. */
|
||||
forbidOnly: !!process.env.CI,
|
||||
/* Retry on CI only */
|
||||
retries: process.env.CI ? 2 : 0,
|
||||
/* Opt out of parallel tests on CI - single worker to avoid resource starvation */
|
||||
workers: process.env.CI ? 1 : undefined,
|
||||
/* Reporter to use. See https://playwright.dev/docs/test-reporters
|
||||
* CI uses per-shard HTML reports (no blob merging needed).
|
||||
* Each shard uploads its own HTML report for easier debugging.
|
||||
*/
|
||||
|
||||
/* CI settings */
|
||||
forbidOnly: !!process.env.CI,
|
||||
retries: process.env.CI ? 2 : 0,
|
||||
|
||||
/* Reporters - simplified for CI */
|
||||
reporter: [
|
||||
...(process.env.CI ? [['github']] : [['list']]),
|
||||
process.env.CI ? ['github'] : ['list'],
|
||||
['html', { open: process.env.CI ? 'never' : 'on-failure' }],
|
||||
...(enableCoverage ? [['@bgotink/playwright-coverage', coverageReporterConfig]] : []),
|
||||
['./tests/reporters/debug-reporter.ts'],
|
||||
],
|
||||
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
|
||||
use: {
|
||||
@@ -168,25 +141,13 @@ export default defineConfig({
|
||||
|
||||
/* Configure projects for major browsers */
|
||||
projects: [
|
||||
// 1. Setup project - authentication (runs FIRST)
|
||||
// Setup project - authentication (runs FIRST)
|
||||
{
|
||||
name: 'setup',
|
||||
testMatch: /auth\.setup\.ts/,
|
||||
},
|
||||
|
||||
// 2. Preflight setup - runs AFTER auth.setup.ts to ensure storage state exists
|
||||
// This replaces Playwright globalSetup so authenticated setup work can run
|
||||
// deterministically in fresh CI workspaces.
|
||||
{
|
||||
name: 'preflight',
|
||||
testMatch: /preflight\.setup\.ts/,
|
||||
dependencies: ['setup'],
|
||||
fullyParallel: false,
|
||||
workers: 1,
|
||||
},
|
||||
|
||||
// 2. Security Tests - Run WITH security enabled (SEQUENTIAL, headless Chromium)
|
||||
// These tests enable security modules, verify enforcement, then teardown disables all.
|
||||
// Security Tests - Run WITH security enabled (SEQUENTIAL, Chromium only)
|
||||
{
|
||||
name: 'security-tests',
|
||||
testDir: './tests',
|
||||
@@ -196,33 +157,29 @@ export default defineConfig({
|
||||
],
|
||||
dependencies: ['setup'],
|
||||
teardown: 'security-teardown',
|
||||
fullyParallel: false, // Force sequential - modules share state
|
||||
workers: 1, // Force single worker to prevent race conditions on security settings
|
||||
fullyParallel: false,
|
||||
workers: 1,
|
||||
use: {
|
||||
...devices['Desktop Chrome'],
|
||||
headless: true, // Security tests are API-level, don't need headed
|
||||
headless: true,
|
||||
storageState: STORAGE_STATE,
|
||||
},
|
||||
},
|
||||
|
||||
// 3. Security Teardown - Disable ALL security modules after security-tests
|
||||
// Security Teardown - Disable ALL security modules
|
||||
{
|
||||
name: 'security-teardown',
|
||||
testMatch: /security-teardown\.setup\.ts/,
|
||||
},
|
||||
|
||||
// 4. Browser projects - Depend on setup and security-tests (with teardown) for order
|
||||
// Note: Security modules are re-disabled by teardown before these projects execute
|
||||
// TEMPORARY CI FIX: Skip security-tests dependency to unblock pipeline
|
||||
// Re-enable after fixing hanging security test
|
||||
// Browser projects - standard Playwright pattern
|
||||
{
|
||||
name: 'chromium',
|
||||
use: {
|
||||
...devices['Desktop Chrome'],
|
||||
// Use stored authentication state
|
||||
storageState: STORAGE_STATE,
|
||||
},
|
||||
dependencies: ['preflight'], // Temporarily removed 'security-tests'
|
||||
dependencies: ['setup', 'security-tests'],
|
||||
},
|
||||
|
||||
{
|
||||
@@ -231,7 +188,7 @@ export default defineConfig({
|
||||
...devices['Desktop Firefox'],
|
||||
storageState: STORAGE_STATE,
|
||||
},
|
||||
dependencies: ['preflight'], // Temporarily removed 'security-tests'
|
||||
dependencies: ['setup', 'security-tests'],
|
||||
},
|
||||
|
||||
{
|
||||
@@ -240,7 +197,7 @@ export default defineConfig({
|
||||
...devices['Desktop Safari'],
|
||||
storageState: STORAGE_STATE,
|
||||
},
|
||||
dependencies: ['preflight'], // Temporarily removed 'security-tests'
|
||||
dependencies: ['setup', 'security-tests'],
|
||||
},
|
||||
|
||||
/* Test against mobile viewports. */
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
import { test } from './fixtures/test';
|
||||
import globalSetup from './global-setup';
|
||||
|
||||
test('preflight', async () => {
|
||||
await globalSetup();
|
||||
});
|
||||
Reference in New Issue
Block a user