chore: git cache cleanup
This commit is contained in:
GitHub Actions
192
docs/reports/archive/e2e_validation_report.md
Normal file
192
docs/reports/archive/e2e_validation_report.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# E2E Test Validation Report
|
||||
**Date**: 2026-01-27
|
||||
**Objective**: Validate 99% pass rate (157/159 tests) after emergency reset fixes
|
||||
**Status**: ❌ **FAIL**
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Current Status**: 110/159 tests passing (69% - **BELOW TARGET**)
|
||||
**Target**: 157/159 (99%)
|
||||
**Gap**: 47 tests
|
||||
|
||||
### Critical Finding
|
||||
Emergency token configuration issues prevented proper test setup, causing cascading failures across security enforcement test suites.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Issue 1: Emergency Token Mismatch (RESOLVED)
|
||||
- **.env token**: `7b3b8a36...40e2`
|
||||
- **Container token**: `f51dedd6...346b`
|
||||
- **Resolution**: Updated `.env` to match container configuration
|
||||
|
||||
### Issue 2: Emergency Reset Endpoint Configuration (PARTIALLY RESOLVED)
|
||||
**Problems identified**:
|
||||
1. Wrong API path: `/api/v1/emergency/security-reset` → `/emergency/security-reset`
|
||||
2. Missing basic auth credentials (admin:changeme)
|
||||
3. Wrong response field access: `body.disabled` → `body.disabled_modules`
|
||||
4. Emergency server runs on port 2020, not 8080
|
||||
|
||||
**Files Fixed**:
|
||||
- ✅ `tests/security-teardown.setup.ts` - Fixed and validated
|
||||
- ✅ `tests/global-setup.ts` - Fixed but not taking effect
|
||||
|
||||
### Issue 3: Test Execution Timing
|
||||
Security tests fail because ACL is already enabled when they start, suggesting global-setup emergency reset is not executing successfully.
|
||||
|
||||
---
|
||||
|
||||
## Test Results Breakdown
|
||||
|
||||
### Overall Metrics
|
||||
```
|
||||
Total Tests: 159
|
||||
✅ Passed: 110 (69%)
|
||||
❌ Failed: 20
|
||||
⏭️ Skipped: 29
|
||||
```
|
||||
|
||||
### By Category
|
||||
|
||||
#### ✅ Passing Categories
|
||||
| Category | Status | Count |
|
||||
|----------|--------|-------|
|
||||
| Security Teardown | ✅ PASS | 1/1 |
|
||||
| Emergency Reset (Break-Glass) | ✅ PASS | 4/5 |
|
||||
| Security Headers | ✅ PASS | 4/4 |
|
||||
| Browser Tests | ✅ PASS | ~100 |
|
||||
|
||||
#### ❌ Failing Categories (ACL Blocking)
|
||||
| Category | Expected | Actual | Root Cause |
|
||||
|----------|----------|--------|------------|
|
||||
| ACL Enforcement | 5/5 | 0/5 | ACL enabled, blocking test setup |
|
||||
| Combined Enforcement | 5/5 | 0/5 | ACL blocking module enable calls |
|
||||
| CrowdSec Enforcement | 3/3 | 0/3 | ACL blocking beforeAll setup |
|
||||
| Emergency Token Protocol | 8/8 | 0/7 (7 skipped) | Suite setup fails with 404 |
|
||||
| Rate Limit Enforcement | 3/3 | 0/3 | ACL blocking test setup |
|
||||
| WAF Enforcement | 4/4 | 0/4 | ACL blocking test setup |
|
||||
|
||||
---
|
||||
|
||||
## Specific Failure Examples
|
||||
|
||||
### Security Teardown (RESOLVED ✅)
|
||||
```
|
||||
Test: disable-all-security-modules
|
||||
Status: ✅ PASS (was failing with TypeError)
|
||||
Fix: Corrected emergency endpoint, auth, and response handling
|
||||
Output: "Emergency reset successful: feature.cerberus.enabled, security.acl.enabled..."
|
||||
```
|
||||
|
||||
### ACL Enforcement Tests (BLOCKED ❌)
|
||||
```
|
||||
Error: Failed to get security status: 403 {"error":"Blocked by access control list"}
|
||||
Impact: All 5 ACL tests fail
|
||||
Cause: Tests can't capture initial state because ACL is already enabled
|
||||
```
|
||||
|
||||
### Emergency Token Protocol (SETUP FAILURE ❌)
|
||||
```
|
||||
Error: Failed to enable ACL for test suite: 404
|
||||
Impact: Test suite setup fails, 7 tests skipped
|
||||
Cause: Endpoint /api/v1/security/acl not found (correct path unknown)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison: Before vs After
|
||||
|
||||
| Metric | Before (Baseline) | After Fix | Target | Gap |
|
||||
|--------|-------------------|-----------|--------|-----|
|
||||
| Pass Rate | 116/159 (73%) | 110/159 (69%) | 157/159 (99%) | -47 tests |
|
||||
| Security Teardown | ❌ FAIL (TypeError) | ✅ PASS | ✅ PASS | ✅ |
|
||||
| ACL Tests | Status unknown | 0/5 | 5/5 | -5 |
|
||||
| Emergency Token | Status unknown | 1/8 | 7/8 | -6 |
|
||||
|
||||
**Note**: Pass rate decreased slightly because previously-passing tests are now correctly detecting ACL blocking issues.
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate Actions (Required for 99% Target)
|
||||
|
||||
1. **Ensure Global Setup Emergency Reset Works**
|
||||
- Verify `global-setup.ts` changes are loaded (no caching)
|
||||
- Test emergency reset manually: `curl -u admin:changeme -X POST http://localhost:2020/emergency/security-reset ...`
|
||||
- Add debug logging to confirm global-setup execution path
|
||||
|
||||
2. **Fix Emergency Token Test Suite Setup**
|
||||
- Identify correct endpoint for enabling ACL programmatically
|
||||
- Option 1: Use `/api/v1/settings` with `{"key":"security.acl.enabled", "value":"true"}`
|
||||
- Option 2: Use emergency token to bypass, then enable ACL
|
||||
- Add retry logic with emergency reset fallback
|
||||
|
||||
3. **Verify Container State**
|
||||
- Containers may need restart to pick up environment changes
|
||||
- Confirm `.env` token matches all running containers
|
||||
- Check if ACL is enabled by default in container startup
|
||||
|
||||
### Testing Protocol
|
||||
|
||||
Before next test run:
|
||||
```bash
|
||||
# 1. Verify emergency token
|
||||
grep CHARON_EMERGENCY_TOKEN .env
|
||||
|
||||
# 2. Test emergency reset manually
|
||||
curl -u admin:changeme \
|
||||
-H "X-Emergency-Token: f51dedd6a4f2eaa200dcbf4feecae78ff926e06d9094d726f3613729b66d346b" \
|
||||
-X POST http://localhost:2020/emergency/security-reset \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason":"Manual validation"}'
|
||||
|
||||
# 3. Verify security modules disabled
|
||||
curl -u admin:changeme http://localhost:8080/api/v1/security/status
|
||||
|
||||
# 4. Run targeted test
|
||||
npx playwright test tests/security-teardown.setup.ts
|
||||
|
||||
# 5. Run full suite
|
||||
npx playwright test --project=chromium
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
**Priority**: Return to Backend_Dev
|
||||
|
||||
**Required Fixes**:
|
||||
1. Investigate why global-setup emergency reset returns 401 despite correct configuration
|
||||
2. Identify correct API endpoint for programmatically enabling/disabling ACL
|
||||
3. Consider adding container restart to test setup if environment changes require it
|
||||
|
||||
**Alternative Approach** (if current method continues to fail):
|
||||
- Disable ACL in container by default
|
||||
- Have security tests explicitly enable ACL before running
|
||||
- Use emergency reset only as fallback/cleanup
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
**Validation Status**: ❌ **FAIL**
|
||||
**Pass Rate**: 69% (110/159)
|
||||
**Target**: 99% (157/159)
|
||||
**Gap**: 47 tests (30% shortfall)
|
||||
|
||||
**Blocking Issues**:
|
||||
1. Global-setup emergency reset not disabling ACL before tests start
|
||||
2. Emergency token test suite setup failing with 404 error
|
||||
3. All security enforcement tests blocked by ACL (403 errors)
|
||||
|
||||
**Successful Fixes**:
|
||||
- ✅ Security teardown emergency reset now works correctly
|
||||
- ✅ Emergency reset endpoint configuration corrected
|
||||
- ✅ Emergency token matching container configuration
|
||||
|
||||
**Recommendation**: Return to Backend_Dev for remaining fixes before attempting validation again.
|
||||
Reference in New Issue
Block a user