Charon/docs/testing/debugging-guide.md

Playwright E2E Test Debugging Guide

This guide explains how to use the enhanced debugging features in the Playwright E2E test suite.

Quick Start

Local Testing with Debug Logging

To run tests with enhanced debug output locally:

# Test with full debug logging and colors
npm run e2e

# Or with more detailed logging
DEBUG=charon:*,charon-test:* npm run e2e

VS Code Debug Tasks

Several new tasks are available in VS Code for debugging:

1. Test: E2E Playwright (Debug Mode - Full Traces)

   • Runs tests in debug mode with full trace capture
   • Opens Playwright Inspector for step-by-step execution
   • Command: Debug=charon:*,charon-test:* npx playwright test --debug --trace=on
   • Use when: You need to step through test execution interactively

2. Test: E2E Playwright (Debug with Logging)

   • Runs tests with enhanced logging output
   • Shows network activity and page state
   • Command: DEBUG=charon:*,charon-test:* PLAYWRIGHT_DEBUG=1 npx playwright test --project=chromium
   • Use when: You want to see detailed logs without interactive debugging

3. Test: E2E Playwright (Trace Inspector)

   • Opens the Playwright Trace Viewer
   • Inspect recorded traces with full DOM/network/console logs
   • Command: npx playwright show-trace <trace.zip>
   • Use when: You've captured traces and want to inspect them

4. Test: E2E Playwright - View Coverage Report

   • Opens the E2E coverage report in browser
   • Shows which code paths were exercised during tests
   • Use when: Analyzing code coverage from E2E tests

Understanding the Debug Logger

The debug logger provides structured logging with multiple methods:

Logger Methods

step(name: string, duration?: number)

Logs a test step with automatic duration tracking.

const logger = new DebugLogger('my-test');
logger.step('Navigate to home page');
logger.step('Click login button', 245); // with duration in ms

Output:

├─ Navigate to home page
├─ Click login button (245ms)

network(entry: NetworkLogEntry)

Logs HTTP requests and responses with timing and status.

logger.network({
  method: 'POST',
  url: 'https://api.example.com/login',
  status: 200,
  elapsedMs: 342,
  responseContentType: 'application/json',
  responseBodySize: 1024
});

Output:

✅ POST https://api.example.com/login [200] 342ms

locator(selector, action, found, elapsedMs)

Logs element interactions and locator resolution.

logger.locator('[role="button"]', 'click', true, 45);

Output:

✓ click "[role="button"]" 45ms

assertion(condition, passed, actual?, expected?)

Logs test assertions with pass/fail status.

logger.assertion('Button is visible', true);
logger.assertion('URL is correct', false, 'http://old.com', 'http://new.com');

Output:

✓ Assert: Button is visible
✗ Assert: URL is correct | expected: "http://new.com", actual: "http://old.com"

error(context, error, recoveryAttempts?)

Logs errors with context and recovery information.

logger.error('Network request failed', new Error('TIMEOUT'), 1);

Output:

❌ ERROR: Network request failed - TIMEOUT
🔄 Recovery: 1 attempts remaining

Local Trace Capture

Traces capture all interactions, network activity, and DOM snapshots. They're invaluable for debugging.

Automatic Trace Capture

Traces are automatically captured:

• On first retry of failed tests
• On failure when running locally (if configured)

Manual Trace Capture

To capture traces for all tests locally:

npx playwright test --trace=on

Or in code:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    trace: 'on', // always capture
  },
});

Viewing Traces

After tests run, view traces with:

npx playwright show-trace test-results/path/to/trace.zip

The Trace Viewer shows:

• Timeline: Chronological list of all actions
• Network: HTTP requests/responses with full details
• Console: Page JS console output
• DOM: DOM snapshot at each step
• Sources: Source code view

CI Debugging

Viewing CI Test Results

When tests fail in CI/CD:

1. Go to the workflow run in GitHub Actions
2. Check the E2E Tests job summary for per-shard status
3. Download artifacts:
   • merged-playwright-report/ - HTML test report
   • traces-*-shard-*/ - Trace files for failures
   • docker-logs-shard-*/ - Application logs
   • test-results-*-shard-*/ - Raw test data

Interpreting CI Logs

Each shard logs its execution with timing:

════════════════════════════════════════════════════════════
E2E Test Shard 1/4
Browser: chromium
Start Time: 2024-01-27T10:30:45Z
════════════════════════════════════════════════════════════
...
════════════════════════════════════════════════════════════
Shard 1 Complete | Duration: 125s
════════════════════════════════════════════════════════════

The merged report summary shows:

╔════════════════════════════════════════════════════════════╗
║              E2E Test Execution Summary                      ║
╠════════════════════════════════════════════════════════════╣
║ Total Tests:        150                                     ║
║ ✅ Passed:          145 (96%)                               ║
║ ❌ Failed:          5                                       ║
║ ⏭️  Skipped:         0                                       ║
╚════════════════════════════════════════════════════════════╝

Failure Analysis

CI logs include failure categorization:

🔍 Failure Analysis by Type:
────────────────────────────────────────────────────────────
timeout      │ ████░░░░░░░░░░░░░░░░░ 2/5 (40%)
assertion    │ ██░░░░░░░░░░░░░░░░░░  2/5 (40%)
network      │ ░░░░░░░░░░░░░░░░░░░░  1/5 (20%)

And slowest tests:

⏱️  Slow Tests (>5s):
────────────────────────────────────────────────────────────
1. Long-running test name               12.43s
2. Another slow test                     8.92s
3. Network-heavy test                    6.15s

Network Debugging

The network interceptor captures all HTTP traffic:

Viewing Network Logs

Network logs appear in console output:

✅ GET https://api.example.com/health [200] 156ms
⚠️ POST https://api.example.com/user [429] 1234ms
❌ GET https://cdn.example.com/asset [timeout] 5000ms

Exporting Network Data

To export network logs for analysis:

import { createNetworkInterceptor } from './fixtures/network';

test('example', async ({ page }) => {
  const interceptor = createNetworkInterceptor(page, logger);

  // ... run test ...

  // Export as CSV
  const csv = interceptor.exportCSV();
  await fs.writeFile('network.csv', csv);

  // Or JSON
  const json = interceptor.exportJSON();
  await fs.writeFile('network.json', JSON.stringify(json));
});

Network Metrics Available

• Request Headers: Sanitized (auth tokens redacted)
• Response Headers: Sanitized
• Status Code: HTTP response code
• Duration: Total request time
• Request Size: Bytes sent
• Response Size: Bytes received
• Content Type: Response MIME type
• Redirect Chain: Followed redirects
• Errors: Network error messages

Debug Output Formats

Local Console Output (Colors)

When running locally, output uses ANSI colors for readability:

• 🔵 Blue: Steps
• 🟢 Green: Successful assertions/locators
• 🟡 Yellow: Warnings (missing locators, slow operations)
• 🔴 Red: Errors
• 🔵 Cyan: Network activity

CI JSON Output

In CI, the same information is formatted as JSON for parsing:

{
  "type": "step",
  "message": "├─ Navigate to home page",
  "timestamp": "2024-01-27T10:30:45.123Z"
}

Common Debugging Scenarios

Test is Timing Out

1. Check traces: Download and inspect with npx playwright show-trace
2. Check logs: Look for "⏳" (waiting) or "⏭️" (skipped) markers
3. Check network: Look for slow network requests in the network CSV
4. Increase timeout: Run with --timeout=60000 locally to get more data

Test is Flaky (Sometimes Fails)

1. Check timing: Look for operations near the 5000ms assertion timeout
2. Check network: Look for variable response times
3. Check logs: Search for race conditions ("expected X but got Y sometimes")
4. Re-run locally: Use npm run e2e -- --grep="flaky test" multiple times

Test Fails on CI but Passes Locally

1. Compare environments: Check if URLs/tokens differ (Check $PLAYWRIGHT_BASE_URL)
2. Check Docker logs: Look for backend errors in docker-logs-*.txt
3. Check timing: CI machines are often slower; increase timeouts
4. Check parallelization: Try running shards sequentially locally

Network Errors in Tests

1. Check network CSV: Export and analyze request times
2. Check status codes: Look for 429 (rate limit), 503 (unavailable), etc.
3. Check headers: Verify auth tokens are being sent correctly (watch for [REDACTED])
4. Check logs: Look for error messages in response bodies

Performance Analysis

Identifying Slow Tests

Tests slower than 5 seconds are automatically highlighted:

npm run e2e  # Shows "Slow Tests (>5s)" in summary

And in CI:

⏱️  Slow Tests (>5s):
────────────────────────────────────────────────────────────
1. test name               12.43s

Analyzing Step Duration

The debug logger tracks step duration:

const logger = new DebugLogger('test-name');
logger.step('Load page', 456);
logger.step('Submit form', 234);

// Slowest operations automatically reported
logger.printSummary(); // Shows per-step breakdown

Network Performance

Check average response times by endpoint:

const interceptor = createNetworkInterceptor(page, logger);
// ... run test ...
const avgTimes = interceptor.getAverageResponseTimeByPattern();
// {
//   'https://api.example.com/login': 234,
//   'https://api.example.com/health': 45,
// }

Environment Variables

Debugging Environment Variables

These can be set to control logging:

# Enable debug namespace logging
DEBUG=charon:*,charon-test:*

# Enable Playwright debugging
PLAYWRIGHT_DEBUG=1

# Set custom base URL
PLAYWRIGHT_BASE_URL=http://localhost:8080

# Set CI log level
CI_LOG_LEVEL=verbose

In GitHub Actions

Environment variables are set automatically for CI runs:

env:
  DEBUG: 'charon:*,charon-test:*'
  PLAYWRIGHT_DEBUG: '1'
  CI_LOG_LEVEL: 'verbose'

Testing Test Utilities Locally

Test the Debug Logger

import { DebugLogger } from '../utils/debug-logger';

const logger = new DebugLogger({
  testName: 'my-test',
  browser: 'chromium',
  file: 'test.spec.ts'
});

logger.step('Step 1', 100);
logger.network({
  method: 'GET',
  url: 'https://example.com',
  status: 200,
  elapsedMs: 156
});
logger.assertion('Check result', true);
logger.printSummary();

Test the Network Interceptor

import { createNetworkInterceptor } from '../fixtures/network';

test('network test', async ({ page }) => {
  const interceptor = createNetworkInterceptor(page);

  await page.goto('https://example.com');

  const csv = interceptor.exportCSV();
  console.log(csv);

  const slowRequests = interceptor.getSlowRequests(1000);
  console.log(`Requests >1s: ${slowRequests.length}`);
});

UI Interaction Helpers

Switch/Toggle Helpers

The tests/utils/ui-helpers.ts file provides helpers for reliable Switch/Toggle interactions.

Problem: Switch components use a hidden <input> with styled siblings, causing Playwright's click() to fail with "pointer events intercepted" errors.

Solution: Use the switch helper functions:

import { clickSwitch, expectSwitchState, toggleSwitch } from '../utils/ui-helpers';

test('should toggle security features', async ({ page }) => {
  await page.goto('/settings');

  // ✅ GOOD: Click switch reliably
  const aclSwitch = page.getByRole('switch', { name: /acl/i });
  await clickSwitch(aclSwitch);

  // ✅ GOOD: Assert switch state
  await expectSwitchState(aclSwitch, true);

  // ✅ GOOD: Toggle and get new state
  const isEnabled = await toggleSwitch(aclSwitch);
  console.log(`ACL is now ${isEnabled ? 'enabled' : 'disabled'}`);

  // ❌ BAD: Direct click (fails in WebKit/Firefox)
  await aclSwitch.click({ force: true }); // Don't use force!
});

Key Features:

• Automatically finds parent <label> element
• Scrolls element into view (sticky header aware)
• Cross-browser compatible (Chromium, Firefox, WebKit)
• No force: true or hard-coded waits needed

When to Use:

• Any test that clicks Switch/Toggle components
• Settings pages with enable/disable toggles
• Security dashboard module toggles (CrowdSec, ACL, WAF, Rate Limiting)
• Access lists and configuration toggles

References:

• Implementation - Full helper code
• QA Report - Test results and validation

Troubleshooting Debug Features

Traces Not Captured

• Ensure trace: 'on-first-retry' or trace: 'on' is set in config
• Check that test-results/ directory exists and is writable
• Verify test fails (traces only captured on retry/failure by default)

Logs Not Appearing

• Check if running in CI (JSON format instead of colored output)
• Set DEBUG=charon:* environment variable
• Ensure CI environment variable is not set for local runs

Reporter Errors

• Verify tests/reporters/debug-reporter.ts exists
• Check TypeScript compilation errors: npx tsc --noEmit
• Run with --reporter=list as fallback

Further Reading

• Playwright Debugging Docs
• Playwright Trace Viewer
• Test Reporters
• Debugging in VS Code