chore: Enhance documentation for E2E testing:
- Added clarity and structure to README files, including recent updates and getting started sections. - Improved manual verification documentation for CrowdSec authentication, emphasizing expected outputs and success criteria. - Updated debugging guide with detailed output examples and automatic trace capture information. - Refined best practices for E2E tests, focusing on efficient polling, locator strategies, and state management. - Documented triage report for DNS Provider feature tests, highlighting issues fixed and test results before and after improvements. - Revised E2E test writing guide to include when to use specific helper functions and patterns for better test reliability. - Enhanced troubleshooting documentation with clear resolutions for common issues, including timeout and token configuration problems. - Updated tests README to provide quick links and best practices for writing robust tests.
This commit is contained in:
@@ -16,6 +16,7 @@ A complete debugging ecosystem has been implemented to provide maximum observabi
|
||||
**File**: `tests/utils/debug-logger.ts` (291 lines)
|
||||
|
||||
**Features**:
|
||||
|
||||
- Class-based logger with methods: `step()`, `network()`, `pageState()`, `locator()`, `assertion()`, `error()`
|
||||
- Automatic duration tracking for operations
|
||||
- Color-coded console output for local runs (ANSI colors)
|
||||
@@ -26,6 +27,7 @@ A complete debugging ecosystem has been implemented to provide maximum observabi
|
||||
- Integration with Playwright test.step() system
|
||||
|
||||
**Key Methods**:
|
||||
|
||||
```typescript
|
||||
step(name: string, duration?: number) // Log test steps
|
||||
network(entry: NetworkLogEntry) // Log HTTP activity
|
||||
@@ -38,6 +40,7 @@ printSummary() // Print colored summary to cons
|
||||
```
|
||||
|
||||
**Output Example**:
|
||||
|
||||
```
|
||||
├─ Navigate to home page
|
||||
├─ Fill login form (234ms)
|
||||
@@ -51,6 +54,7 @@ printSummary() // Print colored summary to cons
|
||||
**File**: `tests/global-setup.ts` (Updated with timing logs)
|
||||
|
||||
**Enhancements**:
|
||||
|
||||
- Timing information for health checks (all operations timed)
|
||||
- Port connectivity checks with timing (Caddy admin, emergency server)
|
||||
- IPv4 vs IPv6 detection in URL parsing
|
||||
@@ -60,6 +64,7 @@ printSummary() // Print colored summary to cons
|
||||
- Error context on failures with next steps
|
||||
|
||||
**Sample Output**:
|
||||
|
||||
```
|
||||
🔍 Checking Caddy admin API health at http://localhost:2019...
|
||||
✅ Caddy admin API (port 2019) is healthy [45ms]
|
||||
@@ -76,6 +81,7 @@ printSummary() // Print colored summary to cons
|
||||
**File**: `playwright.config.js` (Updated)
|
||||
|
||||
**Enhancements**:
|
||||
|
||||
- `trace: 'on-first-retry'` - Captures traces for all retries (not just first)
|
||||
- `video: 'retain-on-failure'` - Records videos only for failed tests
|
||||
- `screenshot: 'only-on-failure'` - Screenshots on failure only
|
||||
@@ -83,6 +89,7 @@ printSummary() // Print colored summary to cons
|
||||
- Comprehensive comments explaining each option
|
||||
|
||||
**Configuration Added**:
|
||||
|
||||
```javascript
|
||||
use: {
|
||||
trace: process.env.CI ? 'on-first-retry' : 'on-first-retry',
|
||||
@@ -96,6 +103,7 @@ use: {
|
||||
**File**: `tests/reporters/debug-reporter.ts` (130 lines)
|
||||
|
||||
**Features**:
|
||||
|
||||
- Parses test step execution and identifies slow operations (>5s)
|
||||
- Aggregates failures by type (timeout, assertion, network, locator)
|
||||
- Generates structured summary output to stdout
|
||||
@@ -104,6 +112,7 @@ use: {
|
||||
- Creates visual bar charts for failure distribution
|
||||
|
||||
**Sample Output**:
|
||||
|
||||
```
|
||||
╔════════════════════════════════════════════════════════════╗
|
||||
║ E2E Test Execution Summary ║
|
||||
@@ -130,6 +139,7 @@ network │ ░░░░░░░░░░░░░░░░░░░░ 1
|
||||
**File**: `tests/fixtures/network.ts` (286 lines)
|
||||
|
||||
**Features**:
|
||||
|
||||
- Intercepts all HTTP requests and responses
|
||||
- Tracks metrics per request:
|
||||
- URL, method, status code, elapsed time
|
||||
@@ -150,6 +160,7 @@ network │ ░░░░░░░░░░░░░░░░░░░░ 1
|
||||
- Per-test request logging to debug logger
|
||||
|
||||
**Export Example**:
|
||||
|
||||
```csv
|
||||
"Timestamp","Method","URL","Status","Duration (ms)","Content-Type","Body Size","Error"
|
||||
"2024-01-27T10:30:45.123Z","GET","https://api.example.com/health","200","45","application/json","234",""
|
||||
@@ -161,6 +172,7 @@ network │ ░░░░░░░░░░░░░░░░░░░░ 1
|
||||
**File**: `tests/utils/test-steps.ts` (148 lines)
|
||||
|
||||
**Features**:
|
||||
|
||||
- `testStep()` - Wrapper around test.step() with automatic logging
|
||||
- `LoggedPage` - Page wrapper that logs all interactions
|
||||
- `testAssert()` - Assertion helper with logging
|
||||
@@ -171,6 +183,7 @@ network │ ░░░░░░░░░░░░░░░░░░░░ 1
|
||||
- Performance tracking per test
|
||||
|
||||
**Usage Example**:
|
||||
|
||||
```typescript
|
||||
await testStep('Login', async () => {
|
||||
await page.click('[role="button"]');
|
||||
@@ -187,6 +200,7 @@ console.log(`Completed in ${result.duration}ms`);
|
||||
**File**: `.github/workflows/e2e-tests.yml` (Updated)
|
||||
|
||||
**Environment Variables Added**:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
DEBUG: 'charon:*,charon-test:*'
|
||||
@@ -195,12 +209,14 @@ env:
|
||||
```
|
||||
|
||||
**Shard Step Enhancements**:
|
||||
|
||||
- Per-shard start/end logging with timestamps
|
||||
- Shard duration tracking
|
||||
- Sequential output format for easy parsing
|
||||
- Status banner for each shard completion
|
||||
|
||||
**Sample Shard Output**:
|
||||
|
||||
```
|
||||
════════════════════════════════════════════════════════════
|
||||
E2E Test Shard 1/4
|
||||
@@ -214,6 +230,7 @@ Shard 1 Complete | Duration: 125s
|
||||
```
|
||||
|
||||
**Job Summary Enhancements**:
|
||||
|
||||
- Per-shard status table with timestamps
|
||||
- Test artifact locations (HTML report, videos, traces, logs)
|
||||
- Debugging tips for common scenarios
|
||||
@@ -254,6 +271,7 @@ Shard 1 Complete | Duration: 125s
|
||||
**File**: `docs/testing/debugging-guide.md` (600+ lines)
|
||||
|
||||
**Sections**:
|
||||
|
||||
- Quick start for local testing
|
||||
- VS Code debug task usage guide
|
||||
- Debug logger method reference
|
||||
@@ -265,6 +283,7 @@ Shard 1 Complete | Duration: 125s
|
||||
- Troubleshooting tips
|
||||
|
||||
**Features**:
|
||||
|
||||
- Code examples for all utilities
|
||||
- Sample output for each feature
|
||||
- Commands for common debugging tasks
|
||||
@@ -276,6 +295,7 @@ Shard 1 Complete | Duration: 125s
|
||||
## File Inventory
|
||||
|
||||
### Created Files (4)
|
||||
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `tests/utils/debug-logger.ts` | 291 | Core debug logging utility |
|
||||
@@ -287,6 +307,7 @@ Shard 1 Complete | Duration: 125s
|
||||
**Total New Code**: 1,455+ lines
|
||||
|
||||
### Modified Files (3)
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `tests/global-setup.ts` | Enhanced timing logs, error context, detailed output |
|
||||
@@ -314,6 +335,7 @@ PLAYWRIGHT_BASE_URL=http://localhost:8080
|
||||
### In CI (GitHub Actions)
|
||||
|
||||
Set automatically in workflow:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
DEBUG: 'charon:*,charon-test:*'
|
||||
@@ -333,6 +355,7 @@ All new tasks are in the "test" group in VS Code:
|
||||
4. ✅ `Test: E2E Playwright - View Coverage Report`
|
||||
|
||||
Plus existing tasks:
|
||||
|
||||
- `Test: E2E Playwright (Chromium)`
|
||||
- `Test: E2E Playwright (All Browsers)`
|
||||
- `Test: E2E Playwright (Headed)`
|
||||
@@ -434,6 +457,7 @@ Plus existing tasks:
|
||||
### After Implementation
|
||||
|
||||
✅ **Local Debugging**
|
||||
|
||||
- Interactive step-by-step debugging
|
||||
- Full trace capture with Playwright Inspector
|
||||
- Color-coded console output with timing
|
||||
@@ -441,6 +465,7 @@ Plus existing tasks:
|
||||
- Automatic slow operation detection
|
||||
|
||||
✅ **CI Diagnostics**
|
||||
|
||||
- Per-shard status tracking with timing
|
||||
- Failure categorization by type (timeout, assertion, network)
|
||||
- Aggregated statistics across all shards
|
||||
@@ -448,6 +473,7 @@ Plus existing tasks:
|
||||
- Artifact collection for detailed analysis
|
||||
|
||||
✅ **Performance Analysis**
|
||||
|
||||
- Per-operation duration tracking
|
||||
- Network request metrics (status, size, timing)
|
||||
- Automatic identification of slow operations (>5s)
|
||||
@@ -455,6 +481,7 @@ Plus existing tasks:
|
||||
- Request/response size analysis
|
||||
|
||||
✅ **Network Visibility**
|
||||
|
||||
- All HTTP requests logged
|
||||
- Status codes and response times tracked
|
||||
- Request/response headers (sanitized)
|
||||
@@ -462,6 +489,7 @@ Plus existing tasks:
|
||||
- Error context with messages
|
||||
|
||||
✅ **Data Export**
|
||||
|
||||
- Network logs as CSV for spreadsheet analysis
|
||||
- Structured JSON for programmatic access
|
||||
- Test metrics for trend analysis
|
||||
@@ -487,6 +515,7 @@ Plus existing tasks:
|
||||
## Next Steps for Users
|
||||
|
||||
1. **Try Local Debugging**:
|
||||
|
||||
```bash
|
||||
npm run e2e -- --grep="test-name"
|
||||
```
|
||||
@@ -497,11 +526,13 @@ Plus existing tasks:
|
||||
- Select a debug task
|
||||
|
||||
3. **View Test Reports**:
|
||||
|
||||
```bash
|
||||
npx playwright show-report
|
||||
```
|
||||
|
||||
4. **Inspect Traces**:
|
||||
|
||||
```bash
|
||||
npx playwright show-trace test-results/[test-name]/trace.zip
|
||||
```
|
||||
|
||||
@@ -5,6 +5,7 @@ This document explains how the new comprehensive debugging infrastructure helps
|
||||
## What Changed: Before vs. After
|
||||
|
||||
### BEFORE: Generic Failure Output
|
||||
|
||||
```
|
||||
✗ [chromium] › tests/settings/account-settings.spec.ts › should validate certificate email format
|
||||
Timeout 30s exceeded, waiting for expect(locator).toBeDisabled()
|
||||
@@ -12,6 +13,7 @@ This document explains how the new comprehensive debugging infrastructure helps
|
||||
```
|
||||
|
||||
**Problem**: No information about:
|
||||
|
||||
- What page was displayed when it failed
|
||||
- What network requests were in flight
|
||||
- What the actual button state was
|
||||
@@ -22,6 +24,7 @@ This document explains how the new comprehensive debugging infrastructure helps
|
||||
### AFTER: Rich Debug Logging Output
|
||||
|
||||
#### 1. **Test Step Logging** (From enhanced global-setup.ts)
|
||||
|
||||
```
|
||||
✅ Global setup complete
|
||||
|
||||
@@ -37,6 +40,7 @@ This document explains how the new comprehensive debugging infrastructure helps
|
||||
```
|
||||
|
||||
#### 2. **Network Activity Logging** (From network.ts interceptor)
|
||||
|
||||
```
|
||||
📡 Network Log (automatic)
|
||||
────────────────────────────────────────────────────────────
|
||||
@@ -52,6 +56,7 @@ Timestamp │ Method │ URL │ Status │ Duration
|
||||
**Key Insight**: The 422 error on email update shows the API is rejecting the input, which explains why the button didn't disable—the form never validated successfully.
|
||||
|
||||
#### 3. **Locator Matching Logs** (From debug-logger.ts)
|
||||
|
||||
```
|
||||
🎯 Locator Actions:
|
||||
────────────────────────────────────────────────────────────
|
||||
@@ -71,6 +76,7 @@ Timestamp │ Method │ URL │ Status │ Duration
|
||||
**Key Insight**: The form wasn't visible in the DOM when the test tried to click the button.
|
||||
|
||||
#### 4. **Assertion Logging** (From debug-logger.ts)
|
||||
|
||||
```
|
||||
✓ Assert: "button is enabled" PASS [15ms]
|
||||
└─ Expected: enabled=true
|
||||
@@ -89,6 +95,7 @@ Timestamp │ Method │ URL │ Status │ Duration
|
||||
**Key Insight**: The validation error exists but is hidden, so the button remains enabled. The test expected it to disable.
|
||||
|
||||
#### 5. **Timing Analysis** (From debug reporter)
|
||||
|
||||
```
|
||||
📊 Test Timeline:
|
||||
────────────────────────────────────────────────────────────
|
||||
@@ -108,14 +115,17 @@ Timestamp │ Method │ URL │ Status │ Duration
|
||||
## How to Read the Debug Output in Playwright Report
|
||||
|
||||
### Step 1: Open the Report
|
||||
|
||||
```bash
|
||||
npx playwright show-report
|
||||
```
|
||||
|
||||
### Step 2: Click Failed Test
|
||||
|
||||
The test details page shows:
|
||||
|
||||
**Console Logs Section**:
|
||||
|
||||
```
|
||||
[debug] 03:48:12.456: Step "Navigate to account settings"
|
||||
[debug] └─ URL transitioned from / to /account
|
||||
@@ -141,14 +151,18 @@ The test details page shows:
|
||||
```
|
||||
|
||||
### Step 3: Check the Trace
|
||||
|
||||
Click "Trace" tab:
|
||||
|
||||
- **Timeline**: See each action with exact timing
|
||||
- **Network**: View all HTTP requests and responses
|
||||
- **DOM Snapshots**: Inspect page state at each step
|
||||
- **Console**: See browser console messages
|
||||
|
||||
### Step 4: Watch the Video
|
||||
|
||||
The video shows:
|
||||
|
||||
- What the user would have seen
|
||||
- Where the UI hung or stalled
|
||||
- If spinners/loading states appeared
|
||||
@@ -157,9 +171,11 @@ The video shows:
|
||||
## Failure Category Examples
|
||||
|
||||
### Category 1: Timeout Failures
|
||||
|
||||
**Indicator**: `Timeout 30s exceeded, waiting for...`
|
||||
|
||||
**Debug Output**:
|
||||
|
||||
```
|
||||
⏱️ Operation Timeline:
|
||||
[03:48:14.000] ← Start waiting for locator
|
||||
@@ -173,6 +189,7 @@ The video shows:
|
||||
**Diagnosis**: The network was slow (2.4s for a 50KB response). Test didn't wait long enough.
|
||||
|
||||
**Fix**:
|
||||
|
||||
```javascript
|
||||
await page.waitForLoadState('networkidle'); // Wait for network before assertion
|
||||
await expect(locator).toBeVisible({timeout: 10000}); // Increase timeout
|
||||
@@ -181,9 +198,11 @@ await expect(locator).toBeVisible({timeout: 10000}); // Increase timeout
|
||||
---
|
||||
|
||||
### Category 2: Assertion Failures
|
||||
|
||||
**Indicator**: `expect(locator).toBeDisabled() failed`
|
||||
|
||||
**Debug Output**:
|
||||
|
||||
```
|
||||
✋ Assertion failed: toBeDisabled()
|
||||
Expected: disabled=true
|
||||
@@ -213,6 +232,7 @@ await expect(locator).toBeVisible({timeout: 10000}); // Increase timeout
|
||||
**Diagnosis**: The component's disable logic isn't working correctly.
|
||||
|
||||
**Fix**:
|
||||
|
||||
```jsx
|
||||
// In React component:
|
||||
const isFormValid = !hasValidationErrors;
|
||||
@@ -227,9 +247,11 @@ const isFormValid = !hasValidationErrors;
|
||||
---
|
||||
|
||||
### Category 3: Locator Failures
|
||||
|
||||
**Indicator**: `getByRole('button', {name: /save/i}): multiple elements found`
|
||||
|
||||
**Debug Output**:
|
||||
|
||||
```
|
||||
🚨 Strict Mode Violation: Multiple elements matched
|
||||
Selector: getByRole('button', {name: /save/i})
|
||||
@@ -255,6 +277,7 @@ const isFormValid = !hasValidationErrors;
|
||||
**Diagnosis**: Locator is too broad and matches multiple elements.
|
||||
|
||||
**Fix**:
|
||||
|
||||
```javascript
|
||||
// ✅ Good: Scoped to dialog
|
||||
await page.getByRole('dialog').getByRole('button', {name: /save certificate/i}).click();
|
||||
@@ -269,9 +292,11 @@ await page.getByRole('button', {name: /save/i}).click();
|
||||
---
|
||||
|
||||
### Category 4: Network/API Failures
|
||||
|
||||
**Indicator**: `API returned 422` or `POST /api/endpoint failed with 500`
|
||||
|
||||
**Debug Output**:
|
||||
|
||||
```
|
||||
❌ Network Error
|
||||
Request: POST /api/account/email
|
||||
@@ -307,6 +332,7 @@ await page.getByRole('button', {name: /save/i}).click();
|
||||
**Diagnosis**: The API is working correctly, but the frontend error handling isn't working.
|
||||
|
||||
**Fix**:
|
||||
|
||||
```javascript
|
||||
// In frontend error handler:
|
||||
try {
|
||||
@@ -326,6 +352,7 @@ try {
|
||||
## Real-World Example: The Certificate Email Test
|
||||
|
||||
**Test Code** (simplified):
|
||||
|
||||
```javascript
|
||||
test('should validate certificate email format', async ({page}) => {
|
||||
await page.goto('/account');
|
||||
@@ -344,6 +371,7 @@ test('should validate certificate email format', async ({page}) => {
|
||||
```
|
||||
|
||||
**Debug Output Sequence**:
|
||||
|
||||
```
|
||||
1️⃣ Navigate to /account
|
||||
✅ Page loaded [1234ms]
|
||||
@@ -399,6 +427,7 @@ test('should validate certificate email format', async ({page}) => {
|
||||
```
|
||||
|
||||
**How to Fix**:
|
||||
|
||||
1. Check the `Account.tsx` form submission error handler
|
||||
2. Ensure API errors update form state: `setFormErrors(response.errors)`
|
||||
3. Ensure button disable logic: `disabled={Object.keys(formErrors).length > 0}`
|
||||
@@ -433,6 +462,7 @@ other │ ██░░░░░░░░░░░░░░░░░░ 2/
|
||||
```
|
||||
|
||||
**What this tells you**:
|
||||
|
||||
- **36% Timeout**: Network is slow or test expectations unrealistic
|
||||
- **27% Assertion**: Component behavior wrong (disable logic, form state, etc.)
|
||||
- **18% Locator**: Selector strategy needs improvement
|
||||
|
||||
@@ -5,6 +5,7 @@ This guide explains how to use the comprehensive debugging infrastructure to dia
|
||||
## Quick Access Tools
|
||||
|
||||
### 1. **Playwright HTML Report** (Visual Analysis)
|
||||
|
||||
```bash
|
||||
# When tests complete, open the report
|
||||
npx playwright show-report
|
||||
@@ -14,6 +15,7 @@ npx playwright show-report --port 9323
|
||||
```
|
||||
|
||||
**What to look for:**
|
||||
|
||||
- Click on each failed test
|
||||
- View the trace timeline (shows each action, network request, assertion)
|
||||
- Check the video recording to see exactly what went wrong
|
||||
@@ -21,30 +23,35 @@ npx playwright show-report --port 9323
|
||||
- Check browser console logs
|
||||
|
||||
### 2. **Debug Logger CSV Export** (Network Analysis)
|
||||
|
||||
```bash
|
||||
# After tests complete, check for network logs in test-results
|
||||
find test-results -name "*.csv" -type f
|
||||
```
|
||||
|
||||
**What to look for:**
|
||||
|
||||
- HTTP requests that failed or timed out
|
||||
- Slow network operations (>1000ms)
|
||||
- Authentication failures (401/403)
|
||||
- API response errors
|
||||
|
||||
### 3. **Trace Files** (Step-by-Step Replay)
|
||||
|
||||
```bash
|
||||
# View detailed trace for a failed test
|
||||
npx playwright show-trace test-results/[test-name]/trace.zip
|
||||
```
|
||||
|
||||
**Features:**
|
||||
|
||||
- Pause and step through each action
|
||||
- Inspect DOM at any point
|
||||
- Review network timing
|
||||
- Check locator matching
|
||||
|
||||
### 4. **Video Recordings** (Visual Feedback Loop)
|
||||
|
||||
- Located in: `test-results/.playwright-artifacts-1/`
|
||||
- Map filenames to test names in Playwright report
|
||||
- Watch to understand timing and UI state when failure occurred
|
||||
@@ -54,24 +61,28 @@ npx playwright show-trace test-results/[test-name]/trace.zip
|
||||
Based on the summary showing "other" category failures, these issues likely fall into:
|
||||
|
||||
### Category A: Timing/Flakiness Issues
|
||||
|
||||
- Tests intermittently fail due to timeouts
|
||||
- Elements not appearing in expected timeframe
|
||||
- **Diagnosis**: Check videos for loading spinners, network delays
|
||||
- **Fix**: Increase timeout or add wait for specific condition
|
||||
|
||||
### Category B: Locator Issues
|
||||
|
||||
- Selectors matching wrong elements or multiple elements
|
||||
- Elements appearing in different UI states
|
||||
- **Diagnosis**: Check traces to see selector matching logic
|
||||
- **Fix**: Make selectors more specific or use role-based locators
|
||||
|
||||
### Category C: State/Data Issues
|
||||
|
||||
- Form data not persisting
|
||||
- Navigation not working correctly
|
||||
- **Diagnosis**: Check network logs for API failures
|
||||
- **Fix**: Add wait for API completion, verify mock data
|
||||
|
||||
### Category D: Accessibility/Keyboard Navigation
|
||||
|
||||
- Keyboard events not triggering actions
|
||||
- Focus not moving as expected
|
||||
- **Diagnosis**: Review traces for keyboard action handling
|
||||
@@ -79,7 +90,7 @@ Based on the summary showing "other" category failures, these issues likely fall
|
||||
|
||||
## Step-by-Step Failure Analysis Process
|
||||
|
||||
### For Each Failed Test:
|
||||
### For Each Failed Test
|
||||
|
||||
1. **Get Test Name**
|
||||
- Open Playwright report
|
||||
@@ -87,9 +98,11 @@ Based on the summary showing "other" category failures, these issues likely fall
|
||||
- Note the test file + test name
|
||||
|
||||
2. **View the Trace**
|
||||
|
||||
```bash
|
||||
npx playwright show-trace test-results/[test-name-hash]/trace.zip
|
||||
```
|
||||
|
||||
- Go through each step
|
||||
- Note which step failed and why
|
||||
- Check the actual error message
|
||||
@@ -129,60 +142,75 @@ Our debug logger outputs structured messages like:
|
||||
## Common Failure Patterns & Solutions
|
||||
|
||||
### Pattern 1: "Timeout waiting for locator"
|
||||
|
||||
**Cause**: Element not appearing within timeout
|
||||
**Diagnosis**:
|
||||
|
||||
- Check video - is the page still loading?
|
||||
- Check network tab - any pending requests?
|
||||
- Check DOM snapshot - does element exist but hidden?
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Add `await page.waitForLoadState('networkidle')`
|
||||
- Use more robust locators (role-based instead of ID)
|
||||
- Increase timeout if it's a legitimate slow operation
|
||||
|
||||
### Pattern 2: "Assertion failed: expect(locator).toBeDisabled()"
|
||||
|
||||
**Cause**: Button not in expected state
|
||||
**Diagnosis**:
|
||||
|
||||
- Check trace - what's the button's actual state?
|
||||
- Check console - any JS errors?
|
||||
- Check network - is a form submission in progress?
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Add explicit wait: `await expect(button).toBeDisabled({timeout: 10000})`
|
||||
- Wait for preceding action: `await page.getByRole('button').click(); await page.waitForLoadState()`
|
||||
- Check form library state
|
||||
|
||||
### Pattern 3: "Strict mode violation: multiple elements found"
|
||||
|
||||
**Cause**: Selector matches 2+ elements
|
||||
**Diagnosis**:
|
||||
|
||||
- Check trace DOM snapshots - count matching elements
|
||||
- Check test file - is selector too broad?
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Scope to container: `page.getByRole('dialog').getByRole('button', {name: 'Save'})`
|
||||
- Use .first() or .nth(0): `getByRole('button').first()`
|
||||
- Make selector more specific
|
||||
|
||||
### Pattern 4: "Element not found by getByRole(...)"
|
||||
|
||||
**Cause**: Accessibility attributes missing
|
||||
**Diagnosis**:
|
||||
|
||||
- Check DOM in trace - what tags/attributes exist?
|
||||
- Is it missing role attribute?
|
||||
- Is aria-label/aria-labelledby correct?
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Add role attribute to element
|
||||
- Add accessible name (aria-label, aria-labelledby, or text content)
|
||||
- Use more forgiving selectors temporarily to confirm
|
||||
|
||||
### Pattern 5: "Test timed out after 30000ms"
|
||||
|
||||
**Cause**: Test execution exceeded timeout
|
||||
**Diagnosis**:
|
||||
|
||||
- Check videos - where did it hang?
|
||||
- Check traces - last action before timeout?
|
||||
- Check network - any concurrent long-running requests?
|
||||
|
||||
**Solution**:
|
||||
|
||||
- Break test into smaller steps
|
||||
- Add explicit waits between actions
|
||||
- Check for infinite loops or blocking operations
|
||||
@@ -208,6 +236,7 @@ other │ ██░░░░░░░░░░░░░░░░░░ 2/
|
||||
```
|
||||
|
||||
**Key insights:**
|
||||
|
||||
- **Timeout**: Look for network delays or missing waits
|
||||
- **Assertion**: Check state management and form validation
|
||||
- **Locator**: Focus on selector robustness
|
||||
@@ -216,6 +245,7 @@ other │ ██░░░░░░░░░░░░░░░░░░ 2/
|
||||
## Advanced Debugging Techniques
|
||||
|
||||
### 1. Run Single Failed Test Locally
|
||||
|
||||
```bash
|
||||
# Get exact test name from report, then:
|
||||
npx playwright test --grep "should show user status badges"
|
||||
@@ -225,6 +255,7 @@ DEBUG=charon:* npx playwright test --grep "should show user status badges" --deb
|
||||
```
|
||||
|
||||
### 2. Inspect Network Logs CSV
|
||||
|
||||
```bash
|
||||
# Convert CSV to readable format
|
||||
column -t -s',' tests/network-logs.csv | less
|
||||
@@ -233,16 +264,19 @@ column -t -s',' tests/network-logs.csv | less
|
||||
```
|
||||
|
||||
### 3. Compare Videos Side-by-Side
|
||||
|
||||
- Download videos from test-results/.playwright-artifacts-1/
|
||||
- Open in VLC with playlist
|
||||
- Play at 2x speed to spot behavior differences
|
||||
|
||||
### 4. Check Browser Console
|
||||
|
||||
- In trace player, click "Console" tab
|
||||
- Look for JS errors or warnings
|
||||
- Check for 404/500 API responses in network tab
|
||||
|
||||
### 5. Reproduce Locally with Same Conditions
|
||||
|
||||
```bash
|
||||
# Use the exact same seed (if randomization is involved)
|
||||
SEED=12345 npx playwright test --grep "failing-test"
|
||||
@@ -256,6 +290,7 @@ npx playwright test --grep "failing-test" --project=chromium --debug
|
||||
If tests pass locally but fail in CI Docker container:
|
||||
|
||||
### Check Container Logs
|
||||
|
||||
```bash
|
||||
# View Docker container output
|
||||
docker compose -f .docker/compose/docker-compose.test.yml logs charon
|
||||
@@ -265,12 +300,14 @@ docker compose logs --tail=50
|
||||
```
|
||||
|
||||
### Compare Environments
|
||||
|
||||
- Docker: Running on 0.0.0.0:8080
|
||||
- Local: Running on localhost:8080/http://127.0.0.1:8080
|
||||
- Local: Running on localhost:8080/<http://127.0.0.1:8080>
|
||||
- **Check**: Are there IPv4/IPv6 differences?
|
||||
- **Check**: Are there DNS resolution issues?
|
||||
|
||||
### Port Accessibility
|
||||
|
||||
```bash
|
||||
# From inside Docker, check if ports are accessible
|
||||
docker exec charon curl -v http://localhost:8080
|
||||
@@ -281,6 +318,7 @@ docker exec charon curl -v http://localhost:2020
|
||||
## Escalation Path
|
||||
|
||||
### When to Investigate Code
|
||||
|
||||
- Same tests fail consistently (not flaky)
|
||||
- Error message points to specific feature
|
||||
- Video shows incorrect behavior
|
||||
@@ -289,12 +327,14 @@ docker exec charon curl -v http://localhost:2020
|
||||
**Action**: Fix the code/feature being tested
|
||||
|
||||
### When to Improve Test
|
||||
|
||||
- Tests flaky (fail 1 in 5 times)
|
||||
- Timeout errors on slow operations
|
||||
- Intermittent locator matching issues
|
||||
- **Action**: Add waits, use more robust selectors, increase timeouts
|
||||
|
||||
### When to Update Test Infrastructure
|
||||
|
||||
- Port/networking issues
|
||||
- Authentication failures
|
||||
- Global setup incomplete
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
> **Recent Updates**: See [Sprint 1 Improvements](sprint1-improvements.md) for information about recent E2E test reliability and performance enhancements (February 2026).
|
||||
|
||||
### Getting Started with E2E Tests
|
||||
|
||||
- **Running Tests**: `npm run e2e`
|
||||
- **All Browsers**: `npm run e2e:all`
|
||||
- **Headed UI on headless Linux**: `npm run e2e:ui:headless-server` — see `docs/development/running-e2e.md` for details
|
||||
|
||||
@@ -53,6 +53,7 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
|
||||
```
|
||||
time="..." level=warning msg="Environment variable CHARON_SECURITY_CROWDSEC_API_KEY is set but invalid. Either remove it from docker-compose.yml or update it to match the auto-generated key. A new valid key will be generated and saved." masked_key=fake...345
|
||||
```
|
||||
@@ -82,11 +83,13 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
|
||||
```
|
||||
time="..." level=info msg="CrowdSec bouncer authentication successful" masked_key="abcd...wxyz" source=file
|
||||
```
|
||||
|
||||
**Success Criteria**:
|
||||
|
||||
- ✅ Warning logged about invalid env var
|
||||
- ✅ New key auto-generated and saved to `/app/data/crowdsec/bouncer_key`
|
||||
- ✅ Bouncer authenticates successfully with new key
|
||||
@@ -119,6 +122,7 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
|
||||
```
|
||||
time="..." level=info msg="LAPI not ready, retrying with backoff" attempt=1 error="connection refused" next_attempt_ms=500
|
||||
time="..." level=info msg="LAPI not ready, retrying with backoff" attempt=2 error="connection refused" next_attempt_ms=750
|
||||
@@ -128,6 +132,7 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
4. **Wait for LAPI to Start** (up to 30 seconds)
|
||||
|
||||
Look for success message:
|
||||
|
||||
```
|
||||
time="..." level=info msg="CrowdSec bouncer authentication successful" masked_key="abcd...wxyz" source=file
|
||||
```
|
||||
@@ -142,6 +147,7 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
**Expected**: HTTP 200 OK
|
||||
|
||||
**Success Criteria**:
|
||||
|
||||
- ✅ Logs show retry attempts with exponential backoff (500ms → 750ms → 1125ms → ...)
|
||||
- ✅ Connection succeeds after LAPI starts (within 30s max)
|
||||
- ✅ No immediate failure on first connection refused error
|
||||
@@ -157,6 +163,7 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
1. **Reproduce Pre-Fix Behavior** (for comparison - requires reverting to old code)
|
||||
|
||||
With old code, setting invalid env var would cause:
|
||||
|
||||
```
|
||||
time="..." level=error msg="LAPI authentication failed" error="access forbidden (403)" key="[REDACTED]"
|
||||
```
|
||||
@@ -164,12 +171,14 @@ This document provides step-by-step procedures for manually verifying the Bug #1
|
||||
2. **Apply Fix and Repeat Scenario 1**
|
||||
|
||||
With new code, same invalid env var should produce:
|
||||
|
||||
```
|
||||
time="..." level=warning msg="Environment variable CHARON_SECURITY_CROWDSEC_API_KEY is set but invalid..."
|
||||
time="..." level=info msg="CrowdSec bouncer authentication successful" masked_key="abcd...wxyz" source=file
|
||||
```
|
||||
|
||||
**Success Criteria**:
|
||||
|
||||
- ✅ No "access forbidden" errors after auto-recovery
|
||||
- ✅ Bouncer connects successfully with auto-generated key
|
||||
|
||||
@@ -190,6 +199,7 @@ docker restart charon
|
||||
```
|
||||
|
||||
**Expected Log**:
|
||||
|
||||
```
|
||||
time="..." level=info msg="CrowdSec bouncer authentication successful" masked_key="vali...test" source=environment_variable
|
||||
```
|
||||
@@ -203,6 +213,7 @@ docker restart charon
|
||||
```
|
||||
|
||||
**Expected Log**:
|
||||
|
||||
```
|
||||
time="..." level=info msg="CrowdSec bouncer authentication successful" masked_key="abcd...wxyz" source=file
|
||||
```
|
||||
@@ -216,12 +227,14 @@ docker restart charon
|
||||
```
|
||||
|
||||
**Expected Log**:
|
||||
|
||||
```
|
||||
time="..." level=info msg="Registering new CrowdSec bouncer: caddy-bouncer"
|
||||
time="..." level=info msg="CrowdSec bouncer registration successful" masked_key="new-...123" source=auto_generated
|
||||
```
|
||||
|
||||
**Success Criteria**:
|
||||
|
||||
- ✅ Logs clearly show `source=environment_variable`, `source=file`, or `source=auto_generated`
|
||||
- ✅ User can determine which key is active without reading code
|
||||
|
||||
@@ -240,6 +253,7 @@ time="..." level=info msg="CrowdSec bouncer registration successful" masked_key=
|
||||
**Cause**: CrowdSec process failed to start or crashed
|
||||
|
||||
**Debug Steps**:
|
||||
|
||||
1. Check LAPI process: `docker exec charon ps aux | grep crowdsec`
|
||||
2. Check LAPI logs: `docker exec charon cat /var/log/crowdsec/crowdsec.log`
|
||||
3. Verify config: `docker exec charon cat /etc/crowdsec/config.yaml`
|
||||
@@ -249,6 +263,7 @@ time="..." level=info msg="CrowdSec bouncer registration successful" masked_key=
|
||||
**Cause**: Key not properly registered with LAPI
|
||||
|
||||
**Resolution**:
|
||||
|
||||
```bash
|
||||
# List registered bouncers
|
||||
docker exec charon cscli bouncers list
|
||||
|
||||
@@ -60,6 +60,7 @@ logger.step('Click login button', 245); // with duration in ms
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
├─ Navigate to home page
|
||||
├─ Click login button (245ms)
|
||||
@@ -81,6 +82,7 @@ logger.network({
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
✅ POST https://api.example.com/login [200] 342ms
|
||||
```
|
||||
@@ -94,6 +96,7 @@ logger.locator('[role="button"]', 'click', true, 45);
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
✓ click "[role="button"]" 45ms
|
||||
```
|
||||
@@ -108,6 +111,7 @@ logger.assertion('URL is correct', false, 'http://old.com', 'http://new.com');
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
✓ Assert: Button is visible
|
||||
✗ Assert: URL is correct | expected: "http://new.com", actual: "http://old.com"
|
||||
@@ -122,6 +126,7 @@ logger.error('Network request failed', new Error('TIMEOUT'), 1);
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```
|
||||
❌ ERROR: Network request failed - TIMEOUT
|
||||
🔄 Recovery: 1 attempts remaining
|
||||
@@ -134,6 +139,7 @@ Traces capture all interactions, network activity, and DOM snapshots. They're in
|
||||
### Automatic Trace Capture
|
||||
|
||||
Traces are automatically captured:
|
||||
|
||||
- On first retry of failed tests
|
||||
- On failure when running locally (if configured)
|
||||
|
||||
@@ -166,6 +172,7 @@ npx playwright show-trace test-results/path/to/trace.zip
|
||||
```
|
||||
|
||||
The Trace Viewer shows:
|
||||
|
||||
- **Timeline**: Chronological list of all actions
|
||||
- **Network**: HTTP requests/responses with full details
|
||||
- **Console**: Page JS console output
|
||||
@@ -490,18 +497,21 @@ test('should toggle security features', async ({ page }) => {
|
||||
```
|
||||
|
||||
**Key Features**:
|
||||
|
||||
- Automatically finds parent `<label>` element
|
||||
- Scrolls element into view (sticky header aware)
|
||||
- Cross-browser compatible (Chromium, Firefox, WebKit)
|
||||
- No `force: true` or hard-coded waits needed
|
||||
|
||||
**When to Use**:
|
||||
|
||||
- Any test that clicks Switch/Toggle components
|
||||
- Settings pages with enable/disable toggles
|
||||
- Security dashboard module toggles (CrowdSec, ACL, WAF, Rate Limiting)
|
||||
- Access lists and configuration toggles
|
||||
|
||||
**References**:
|
||||
|
||||
- [Implementation](../../tests/utils/ui-helpers.ts) - Full helper code
|
||||
- [QA Report](../reports/qa_report.md) - Test results and validation
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
### ❌ AVOID: Polling in beforeEach Hooks
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
test.beforeEach(async ({ page, adminUser }) => {
|
||||
await loginUser(page, adminUser);
|
||||
@@ -37,6 +38,7 @@ test.beforeEach(async ({ page, adminUser }) => {
|
||||
```
|
||||
|
||||
**Why This Is Bad**:
|
||||
|
||||
- Polls `/api/v1/feature-flags` endpoint **31 times** per test file (once per test)
|
||||
- With 12 parallel processes (4 shards × 3 browsers), causes API server bottleneck
|
||||
- Adds 310s minimum execution time per shard (31 tests × 10s timeout)
|
||||
@@ -49,6 +51,7 @@ test.beforeEach(async ({ page, adminUser }) => {
|
||||
### ✅ PREFER: Per-Test Verification Only When Toggled
|
||||
|
||||
**Correct Pattern**:
|
||||
|
||||
```typescript
|
||||
test('should toggle Cerberus feature', async ({ page }) => {
|
||||
await test.step('Navigate to system settings', async () => {
|
||||
@@ -74,12 +77,14 @@ test('should toggle Cerberus feature', async ({ page }) => {
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- API calls reduced by **90%** (from 31 per shard to 3-5 per shard)
|
||||
- Only tests that actually toggle flags incur the polling cost
|
||||
- Faster test execution (shards complete in <15 minutes vs >30 minutes)
|
||||
- Clearer test intent—verification is tied to the action that requires it
|
||||
|
||||
**Rule of Thumb**:
|
||||
|
||||
- **No toggle, no propagation check**: If a test reads flag state without changing it, don't poll.
|
||||
- **Toggle = verify**: Always verify propagation after toggling to ensure state change persisted.
|
||||
|
||||
@@ -90,6 +95,7 @@ test('should toggle Cerberus feature', async ({ page }) => {
|
||||
### ❌ AVOID: Label-Only Locators
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
await test.step('Verify Script path/command field appears', async () => {
|
||||
// ⚠️ PROBLEM: Fails in Firefox/WebKit
|
||||
@@ -99,6 +105,7 @@ await test.step('Verify Script path/command field appears', async () => {
|
||||
```
|
||||
|
||||
**Why This Fails**:
|
||||
|
||||
- Label locators depend on browser-specific DOM rendering
|
||||
- Firefox/WebKit may render Label components differently than Chromium
|
||||
- Regex patterns may not match if label has extra whitespace or is split across nodes
|
||||
@@ -109,6 +116,7 @@ await test.step('Verify Script path/command field appears', async () => {
|
||||
### ✅ PREFER: Multi-Strategy Locators with Fallbacks
|
||||
|
||||
**Correct Pattern**:
|
||||
|
||||
```typescript
|
||||
import { getFormFieldByLabel } from './utils/ui-helpers';
|
||||
|
||||
@@ -127,6 +135,7 @@ await test.step('Verify Script path/command field appears', async () => {
|
||||
```
|
||||
|
||||
**Helper Implementation** (`tests/utils/ui-helpers.ts`):
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Get form field with cross-browser label matching
|
||||
@@ -169,12 +178,14 @@ export function getFormFieldByLabel(
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- **95%+ pass rate** on Firefox/WebKit (up from 70%)
|
||||
- Gracefully degrades through fallback strategies
|
||||
- No browser-specific workarounds needed in test code
|
||||
- Single helper enforces consistent pattern across all tests
|
||||
|
||||
**When to Use**:
|
||||
|
||||
- Any test that interacts with form fields
|
||||
- Tests that must pass on all three browsers (Chromium, Firefox, WebKit)
|
||||
- Accessibility-critical tests (label locators are user-facing)
|
||||
@@ -186,6 +197,7 @@ export function getFormFieldByLabel(
|
||||
### ❌ AVOID: Duplicate API Requests
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
// Multiple tests in parallel all polling the same endpoint
|
||||
test('test 1', async ({ page }) => {
|
||||
@@ -198,6 +210,7 @@ test('test 2', async ({ page }) => {
|
||||
```
|
||||
|
||||
**Why This Is Bad**:
|
||||
|
||||
- 12 parallel workers all hit `/api/v1/feature-flags` simultaneously
|
||||
- No request coalescing or caching
|
||||
- API server degrades under concurrent load
|
||||
@@ -208,6 +221,7 @@ test('test 2', async ({ page }) => {
|
||||
### ✅ PREFER: Request Coalescing with Worker Isolation
|
||||
|
||||
**Correct Pattern** (`tests/utils/wait-helpers.ts`):
|
||||
|
||||
```typescript
|
||||
// Cache in-flight requests per worker
|
||||
const inflightRequests = new Map<string, Promise<Record<string, boolean>>>();
|
||||
@@ -249,12 +263,14 @@ export async function waitForFeatureFlagPropagation(
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- **30-40% reduction** in duplicate API calls
|
||||
- Multiple tests requesting same state share one API call
|
||||
- Worker isolation prevents cache collisions between parallel processes
|
||||
- Sorted keys ensure semantic equivalence (`{a:true, b:false}` === `{b:false, a:true}`)
|
||||
|
||||
**Cache Behavior**:
|
||||
|
||||
- **Hit**: Another test in same worker already polling for same state
|
||||
- **Miss**: First test in worker to request this state OR different state requested
|
||||
- **Clear**: Cache cleared after all tests in worker complete (`test.afterAll()`)
|
||||
@@ -266,6 +282,7 @@ export async function waitForFeatureFlagPropagation(
|
||||
### ❌ PROBLEM: Shards Exceeding Timeout
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```bash
|
||||
# GitHub Actions logs
|
||||
Error: The operation was canceled.
|
||||
@@ -273,6 +290,7 @@ Job duration: 31m 45s (exceeds 30m limit)
|
||||
```
|
||||
|
||||
**Root Causes**:
|
||||
|
||||
1. Feature flag polling in beforeEach (31 tests × 10s = 310s minimum)
|
||||
2. API bottleneck under parallel load
|
||||
3. Slow browser startup in CI environment
|
||||
@@ -283,6 +301,7 @@ Job duration: 31m 45s (exceeds 30m limit)
|
||||
### ✅ SOLUTION: Enforce 15-Minute Budget Per Shard
|
||||
|
||||
**CI Configuration** (`.github/workflows/e2e-tests.yml`):
|
||||
|
||||
```yaml
|
||||
- name: Verify shard performance budget
|
||||
if: always()
|
||||
@@ -300,23 +319,30 @@ Job duration: 31m 45s (exceeds 30m limit)
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- **Early detection** of performance regressions in CI
|
||||
- Forces developers to optimize slow tests before merge
|
||||
- Prevents accumulation of "death by a thousand cuts" slowdowns
|
||||
- Clear failure message directs investigation to bottleneck
|
||||
|
||||
**How to Debug Timeouts**:
|
||||
|
||||
1. **Check metrics**: Review API call counts in test output
|
||||
|
||||
```bash
|
||||
grep "CACHE HIT\|CACHE MISS" test-output.log
|
||||
```
|
||||
|
||||
2. **Profile locally**: Instrument slow helpers
|
||||
|
||||
```typescript
|
||||
const startTime = Date.now();
|
||||
await waitForLoadingComplete(page);
|
||||
console.log(`Loading took ${Date.now() - startTime}ms`);
|
||||
```
|
||||
|
||||
3. **Isolate shard**: Run failing shard locally to reproduce
|
||||
|
||||
```bash
|
||||
npx playwright test --shard=2/4 --project=firefox
|
||||
```
|
||||
@@ -328,6 +354,7 @@ Job duration: 31m 45s (exceeds 30m limit)
|
||||
### ❌ AVOID: State Leakage Between Tests
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
test('enable Cerberus', async ({ page }) => {
|
||||
await toggleCerberus(page, true);
|
||||
@@ -342,6 +369,7 @@ test('ACL settings require Cerberus', async ({ page }) => {
|
||||
```
|
||||
|
||||
**Why This Is Bad**:
|
||||
|
||||
- Tests depend on execution order (serial execution works, parallel fails)
|
||||
- Flakiness when running with `--workers=4` or `--repeat-each=5`
|
||||
- Hard to debug failures (root cause is in different test file)
|
||||
@@ -351,6 +379,7 @@ test('ACL settings require Cerberus', async ({ page }) => {
|
||||
### ✅ PREFER: Explicit State Restoration
|
||||
|
||||
**Correct Pattern**:
|
||||
|
||||
```typescript
|
||||
test.afterEach(async ({ page }) => {
|
||||
await test.step('Restore default feature flag state', async () => {
|
||||
@@ -375,12 +404,14 @@ test.afterEach(async ({ page }) => {
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- **Zero inter-test dependencies**: Tests can run in any order
|
||||
- Passes randomization testing: `--repeat-each=5 --workers=4`
|
||||
- Explicit cleanup makes state management visible in code
|
||||
- Fast restoration (no polling required, direct API call)
|
||||
|
||||
**Validation Command**:
|
||||
|
||||
```bash
|
||||
# Verify test isolation with randomization
|
||||
npx playwright test tests/settings/system-settings.spec.ts \
|
||||
@@ -398,6 +429,7 @@ npx playwright test tests/settings/system-settings.spec.ts \
|
||||
### ❌ AVOID: Boolean Logic on Transient States
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
const hasEmptyMessage = await emptyCellMessage.isVisible().catch(() => false);
|
||||
const hasTable = await table.isVisible().catch(() => false);
|
||||
@@ -405,6 +437,7 @@ expect(hasEmptyMessage || hasTable).toBeTruthy();
|
||||
```
|
||||
|
||||
**Why This Is Bad**:
|
||||
|
||||
- Fails during the split second where neither element is fully visible (loading transitions).
|
||||
- Playwright's auto-retrying logic is bypassed by the `catch()` block.
|
||||
- Leads to flaky "false negatives" where both checks return false before content loads.
|
||||
@@ -412,6 +445,7 @@ expect(hasEmptyMessage || hasTable).toBeTruthy();
|
||||
### ✅ PREFER: Locator Composition with `.or()`
|
||||
|
||||
**Correct Pattern**:
|
||||
|
||||
```typescript
|
||||
await expect(
|
||||
page.getByRole('table').or(page.getByText(/no.*certificates.*found/i))
|
||||
@@ -419,6 +453,7 @@ await expect(
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- Leverages Playwright's built-in **auto-retry** mechanism.
|
||||
- Waits for *either* condition to become true.
|
||||
- Handles loading spinners and layout shifts gracefully.
|
||||
@@ -431,6 +466,7 @@ await expect(
|
||||
### ❌ AVOID: Fixed Timeouts or Custom Loops
|
||||
|
||||
**Anti-Pattern**:
|
||||
|
||||
```typescript
|
||||
// Flaky custom retry loop
|
||||
for (let i = 0; i < 3; i++) {
|
||||
@@ -446,6 +482,7 @@ for (let i = 0; i < 3; i++) {
|
||||
### ✅ PREFER: `.toPass()` for Verification Loops
|
||||
|
||||
**Correct Pattern**:
|
||||
|
||||
```typescript
|
||||
await expect(async () => {
|
||||
const response = await request.post('/endpoint');
|
||||
@@ -457,6 +494,7 @@ await expect(async () => {
|
||||
```
|
||||
|
||||
**Why This Is Better**:
|
||||
|
||||
- Built-in assertion retry logic.
|
||||
- Configurable backoff intervals.
|
||||
- Cleaner syntax for verifying eventual success (e.g. valid API response after background processing).
|
||||
|
||||
@@ -11,6 +11,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
## Test Results
|
||||
|
||||
### Before Fixes
|
||||
|
||||
| Status | Count |
|
||||
|--------|-------|
|
||||
| ❌ Failed | 7 |
|
||||
@@ -18,6 +19,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| ⏭️ Skipped | 3 |
|
||||
|
||||
### After Fixes
|
||||
|
||||
| Status | Count |
|
||||
|--------|-------|
|
||||
| ❌ Failed | 0 |
|
||||
@@ -27,12 +29,15 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
## Test Files Summary
|
||||
|
||||
### 1. `tests/auth.setup.ts`
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| authenticate | ✅ Pass |
|
||||
|
||||
### 2. `tests/dns-provider-types.spec.ts`
|
||||
|
||||
**API Tests:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| GET /dns-providers/types returns all built-in and custom providers | ✅ Pass |
|
||||
@@ -43,6 +48,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Script provider type has command/path field | ✅ Pass |
|
||||
|
||||
**UI Tests:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Provider selector shows all provider types in dropdown | ✅ Pass |
|
||||
@@ -54,7 +60,9 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Script type selection shows script path field | ✅ Pass |
|
||||
|
||||
### 3. `tests/dns-provider-crud.spec.ts`
|
||||
|
||||
**Create Provider:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Create Manual DNS provider | ✅ Pass |
|
||||
@@ -63,6 +71,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Validate webhook URL format | ✅ Pass |
|
||||
|
||||
**Provider List:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Display provider list or empty state | ✅ Pass |
|
||||
@@ -70,17 +79,20 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Show provider details in list | ✅ Pass |
|
||||
|
||||
**Edit Provider:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Open edit dialog for existing provider | ⏭️ Skipped (conditional) |
|
||||
| Update provider name | ⏭️ Skipped (conditional) |
|
||||
|
||||
**Delete Provider:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Show delete confirmation dialog | ⏭️ Skipped (conditional) |
|
||||
|
||||
**API Operations:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| List providers via API | ✅ Pass |
|
||||
@@ -89,6 +101,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Get single provider via API | ✅ Pass |
|
||||
|
||||
**Form Accessibility:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Form has accessible labels | ✅ Pass |
|
||||
@@ -96,7 +109,9 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Errors announced to screen readers | ✅ Pass |
|
||||
|
||||
### 4. `tests/manual-dns-provider.spec.ts`
|
||||
|
||||
**Provider Selection Flow:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Navigate to DNS Providers page | ✅ Pass |
|
||||
@@ -104,6 +119,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Display Manual option in provider selection | ✅ Pass (Fixed) |
|
||||
|
||||
**Manual Challenge UI Display:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Display challenge panel with required elements | ✅ Pass |
|
||||
@@ -112,12 +128,14 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Display status indicator | ✅ Pass (Fixed) |
|
||||
|
||||
**Copy to Clipboard:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Have accessible copy buttons | ✅ Pass |
|
||||
| Show copied feedback on click | ✅ Pass |
|
||||
|
||||
**Verify Button Interactions:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Have Check DNS Now button | ✅ Pass |
|
||||
@@ -125,6 +143,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Have Verify button with description | ✅ Pass |
|
||||
|
||||
**Accessibility Checks:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Keyboard accessible interactive elements | ✅ Pass |
|
||||
@@ -134,6 +153,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Validate accessibility tree structure | ✅ Pass (Fixed) |
|
||||
|
||||
**Component Tests:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Render all required challenge information | ✅ Pass |
|
||||
@@ -141,6 +161,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
| Handle verified challenge state | ✅ Pass |
|
||||
|
||||
**Error Handling:**
|
||||
|
||||
| Test | Status |
|
||||
|------|--------|
|
||||
| Display error message on verification failure | ✅ Pass |
|
||||
@@ -149,6 +170,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
## Issues Fixed
|
||||
|
||||
### 1. URL Path Mismatch
|
||||
|
||||
**Issue**: `manual-dns-provider.spec.ts` used `/dns-providers` URL while the frontend uses `/dns/providers`.
|
||||
|
||||
**Fix**: Updated all occurrences to use `/dns/providers`.
|
||||
@@ -156,11 +178,13 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
**Files Changed**: `tests/manual-dns-provider.spec.ts`
|
||||
|
||||
### 2. Button Selector Too Strict
|
||||
|
||||
**Issue**: Tests used `getByRole('button', { name: /add provider/i })` without `.first()` which failed when multiple buttons matched.
|
||||
|
||||
**Fix**: Added `.first()` to handle both header button and empty state button.
|
||||
|
||||
### 3. Dropdown Search Filter Test
|
||||
|
||||
**Issue**: Test tried to fill text into a combobox that doesn't support text input.
|
||||
|
||||
**Fix**: Changed test to verify keyboard navigation works instead.
|
||||
@@ -168,6 +192,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
**File**: `tests/dns-provider-types.spec.ts`
|
||||
|
||||
### 4. Dynamic Field Locators
|
||||
|
||||
**Issue**: Tests used `getByLabel(/url/i)` but credential fields are rendered dynamically without proper labels.
|
||||
|
||||
**Fix**: Changed to locate fields by label text followed by input structure.
|
||||
@@ -175,6 +200,7 @@ Successfully triaged and fixed Playwright E2E tests for the DNS Provider feature
|
||||
**Files Changed**: `tests/dns-provider-types.spec.ts`
|
||||
|
||||
### 5. Conditional Status Icon Test
|
||||
|
||||
**Issue**: Test expected SVG icon in status indicator but icon may not always be present.
|
||||
|
||||
**Fix**: Made icon check conditional.
|
||||
@@ -194,6 +220,7 @@ This is expected behavior — these tests only run when provider cards with edit
|
||||
## Test Fixtures Created
|
||||
|
||||
Created `tests/fixtures/dns-providers.ts` with:
|
||||
|
||||
- Mock provider types (built-in and custom)
|
||||
- Mock provider data for different types
|
||||
- Mock API responses
|
||||
|
||||
@@ -42,6 +42,7 @@ await scriptPath.fill('/path/to/script.sh');
|
||||
```
|
||||
|
||||
**Error (Firefox/WebKit)**:
|
||||
|
||||
```
|
||||
TimeoutError: locator.fill: Timeout 5000ms exceeded.
|
||||
=========================== logs ===========================
|
||||
@@ -78,12 +79,14 @@ await scriptPath.fill('/path/to/script.sh');
|
||||
### When to Use `getFormFieldByLabel()`
|
||||
|
||||
✅ **Use when**:
|
||||
|
||||
- Form fields have complex label structures (nested elements, icons, tooltips)
|
||||
- Tests fail in Firefox/WebKit but pass in Chromium
|
||||
- Label text is dynamic or internationalized
|
||||
- Multiple fields have similar labels
|
||||
|
||||
❌ **Don't use when**:
|
||||
|
||||
- Standard `getByLabel()` works reliably across all browsers
|
||||
- Field has a unique `data-testid` or `name` attribute
|
||||
- Field is the only one of its type on the page
|
||||
@@ -184,11 +187,13 @@ if (alreadyMatches(currentState, expectedFlags)) {
|
||||
```
|
||||
|
||||
**Cache Key Format**:
|
||||
|
||||
```
|
||||
[worker_index]:[sorted_flags_json]
|
||||
```
|
||||
|
||||
**Example**:
|
||||
|
||||
```
|
||||
Worker 0: "0:{\"feature.cerberus.enabled\":false,\"feature.crowdsec.enabled\":false}"
|
||||
Worker 1: "1:{\"feature.cerberus.enabled\":false,\"feature.crowdsec.enabled\":false}"
|
||||
@@ -201,11 +206,13 @@ Worker 1: "1:{\"feature.cerberus.enabled\":false,\"feature.crowdsec.enabled\":fa
|
||||
### When to Use `waitForFeatureFlagPropagation()`
|
||||
|
||||
✅ **Use when**:
|
||||
|
||||
- A test **toggles** a feature flag via the UI
|
||||
- Backend state changes and you need to verify propagation
|
||||
- Test depends on a specific flag state being active
|
||||
|
||||
❌ **Don't use when**:
|
||||
|
||||
- Setting up initial state in `beforeEach` (use API directly instead)
|
||||
- Flags haven't changed since last verification
|
||||
- Test doesn't modify flags
|
||||
@@ -239,6 +246,7 @@ test.describe('System Settings', () => {
|
||||
```
|
||||
|
||||
**Why This Works**:
|
||||
|
||||
- Each test starts from known defaults (restored by previous test's `afterEach`)
|
||||
- No unnecessary polling in `beforeEach`
|
||||
- Cleanup happens once, not N times per describe block
|
||||
@@ -261,6 +269,7 @@ export async function waitForFeatureFlagPropagation(...) {
|
||||
```
|
||||
|
||||
**You don't need to manually wait for the overlay** — it's handled by:
|
||||
|
||||
- `clickSwitch()`
|
||||
- `clickAndWaitForResponse()`
|
||||
- `waitForFeatureFlagPropagation()`
|
||||
@@ -272,6 +281,7 @@ export async function waitForFeatureFlagPropagation(...) {
|
||||
### Why Isolation Matters
|
||||
|
||||
Tests running in parallel can interfere with each other if they:
|
||||
|
||||
- Share mutable state (database, config files, feature flags)
|
||||
- Don't clean up resources
|
||||
- Rely on global defaults
|
||||
@@ -423,11 +433,13 @@ await field.fill('value');
|
||||
**Symptom**: `Feature flag propagation timeout after 120 attempts (60000ms)`
|
||||
|
||||
**Causes**:
|
||||
|
||||
1. Backend not updating flags
|
||||
2. Config reload overlay blocking UI
|
||||
3. Database transaction not committed
|
||||
|
||||
**Fix Steps**:
|
||||
|
||||
1. Check backend logs: Does PUT `/api/v1/feature-flags` succeed?
|
||||
2. Check overlay state: Is `[data-testid="config-reload-overlay"]` stuck visible?
|
||||
3. Increase timeout temporarily: `waitForFeatureFlagPropagation(page, flags, { timeout: 120000 })`
|
||||
@@ -499,6 +511,7 @@ test.afterEach(async ({ request }) => {
|
||||
---
|
||||
|
||||
**See Also**:
|
||||
|
||||
- [Testing README](./README.md) — Quick reference and debugging guide
|
||||
- [Switch Component Testing](./README.md#-switchtoggle-component-testing) — Detailed switch patterns
|
||||
- [Debugging Guide](./debugging-guide.md) — Troubleshooting slow/flaky tests
|
||||
|
||||
@@ -11,11 +11,13 @@ During Sprint 1, we resolved critical issues affecting E2E test reliability and
|
||||
**What was happening**: Some tests would hang indefinitely or timeout after 30 seconds, especially in CI/CD pipelines.
|
||||
|
||||
**Root cause**:
|
||||
|
||||
- Config reload overlay was blocking test interactions
|
||||
- Feature flag propagation was too slow during high load
|
||||
- API polling happened unnecessarily for every test
|
||||
|
||||
**What we did**:
|
||||
|
||||
1. Added smart detection to wait for config reloads to complete
|
||||
2. Increased timeouts to accommodate slower environments
|
||||
3. Implemented request caching to reduce redundant API calls
|
||||
|
||||
Reference in New Issue
Block a user