198 lines
8.8 KiB
Markdown
198 lines
8.8 KiB
Markdown
# Phase 2 Test Failure Triage Report
|
|
|
|
**Date**: 2026-02-09
|
|
**Test Run**: Full Phase 2 (Core, Settings, Tasks, Monitoring)
|
|
**Results**: 308 passed, 28 failed (91.7% pass rate)
|
|
**Duration**: 59 minutes
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
Phase 2 achieved 91.7% pass rate. Failures cluster into 5 categories:
|
|
1. **Code Bugs** (12 failures): Notification providers, Proxy Hosts Docker, Uptime monitoring
|
|
2. **Not Yet Tested Physically** (6 failures): User invite and permission flows
|
|
3. **Feature Scope Questions** (12 failures): Log viewer scope (live vs. system logs)
|
|
4. **Minor Issues** (1 failure): Backup button visibility for guests (may be working)
|
|
5. **Database State** (1 failure): Potential data accumulation after 250+ tests
|
|
|
|
---
|
|
|
|
## Failure Categories & Triage
|
|
|
|
### Category 1: Code Bugs Requiring Fixes (12 failures)
|
|
|
|
#### 1.1 Notifications Provider CRUD (6 failures: tests 205-219)
|
|
- **Tests**: #205 (edit provider), #208 (validate URL), #211 (create template), #212 (preview template), #213 (edit template), #219 (persist selections)
|
|
- **Timeout**: 1.5 minutes (90 seconds)
|
|
- **Classification**: ❌ **CODE BUG**
|
|
- **Issue**: Notification provider operations slow or hanging
|
|
- **Triage Notes**:
|
|
- Multiple CRUD operations timing out consistently
|
|
- All take 1.5m each (not simple network lag)
|
|
- Suggests backend processing issue or missing validation
|
|
- **Recommended Action**:
|
|
- Backend Dev should investigate notification provider CRUD endpoints
|
|
- Check for missing indexes, N+1 queries, or validation delays
|
|
- Post-Phase 3: Create PR to fix backend performance
|
|
|
|
#### 1.2 Proxy Hosts - Docker Integration (2 failures: tests 154-155)
|
|
- **Tests**: #154 (show container selector), #155 (show containers dropdown)
|
|
- **Timeout**: 18s - 90s
|
|
- **Classification**: ❌ **CODE BUG**
|
|
- **Issue**: Docker container selector not appearing or loading
|
|
- **Triage Notes**:
|
|
- Tests expect Docker container UI to appear when source is selected
|
|
- UI may not be rendering or Docker API integration is broken
|
|
- **Recommended Action**:
|
|
- Frontend Dev should verify Docker source component exists and renders correctly
|
|
- Check if Docker API integration is implemented
|
|
- Post-Phase 3: Create PR to fix Docker integration or update tests to skip if Docker is not yet implemented
|
|
|
|
#### 1.3 Uptime Monitoring - Ping State (1 failure: test 166)
|
|
- **Test**: #166 (update existing monitor)
|
|
- **Timeout**: 11s
|
|
- **Classification**: ❌ **CODE BUG**
|
|
- **Issue**: Monitor state marking as "down" when no ping has been sent yet
|
|
- **Root Cause** (per user): Monitor listens for ping response without actually sending a ping first
|
|
- **Triage Notes**:
|
|
- Should remain in neutral/pending state until first ping is sent
|
|
- Currently marking as false negative "down"
|
|
- **Recommended Action**:
|
|
- Backend Dev should fix uptime monitor initial state logic
|
|
- Ensure monitor doesn't mark as "down" until it has actually attempted a ping
|
|
- Post-Phase 3: Create PR with fix
|
|
|
|
---
|
|
|
|
### Category 2: Features Not Yet Physically Tested (6 failures)
|
|
|
|
#### 2.1 User Management - Invites & Permissions (6 failures: tests 248, 258, 260, 262, 269-270)
|
|
- **Tests**:
|
|
- #248 (show pending invite status)
|
|
- #258 (update permission mode)
|
|
- #260 (remove permitted hosts)
|
|
- #262 (enable/disable user)
|
|
- #269 (require admin role for access)
|
|
- #270 (show error for regular user access)
|
|
- **Timeout**: 15s - 1.6m
|
|
- **Classification**: ⚠️ **NOT YET TESTED PHYSICALLY**
|
|
- **Issue**: These flows have not been manually tested in the UI yet
|
|
- **Triage Notes**:
|
|
- User invite/permission system may have unimplemented features
|
|
- Tests may be written against spec rather than actual implementation
|
|
- Timeouts suggest either missing endpoints or slow responses
|
|
- **Recommended Action**:
|
|
- Backend Dev should manually test user invite and permission flows in UI
|
|
- Verify endpoints exist and return correct data
|
|
- If features are partially implemented, tests may need to be updated or xfailed
|
|
- Post-Phase 3: Either fix implementation or update tests based on actual behavior
|
|
|
|
---
|
|
|
|
### Category 3: Log Viewer Scope Questions (12 failures: tests 324-335)
|
|
|
|
#### 3.1 Log Viewing Tests - All Timing Out at 66 Seconds (12 failures)
|
|
- **Tests**: #324-#335 (table display, sorting, pagination, filtering, download)
|
|
- **Timeout**: 1.1m (66 seconds)
|
|
- **All timing out with same duration** ← indicates consistent issue, not random flakes
|
|
- **Classification**: ❓ **SCOPE DEPENDENT**
|
|
- **Triage Notes**:
|
|
- **If Live Log Viewer**: Should be moved to Phase 3 (security dashboard feature, runs after security teardown)
|
|
- **If System Static Logs**: Features may not be fully implemented yet
|
|
- User notes: "System logs are just a way to download recent logs at the current state"
|
|
- **Recommended Action**:
|
|
- Clarify: Is this the live log viewer (security dashboard) or system log viewer (static)?
|
|
- If live logs → Move tests to Phase 3B security-enforcement suite (after security modules are enabled)
|
|
- If system logs → Check if all features (sorting, pagination, filtering) are actually implemented
|
|
- If not implemented: Mark tests as xfail with TODO comment
|
|
- If implemented: Debug why all operations timeout uniformly
|
|
|
|
---
|
|
|
|
### Category 4: Minor Issues (1 failure)
|
|
|
|
#### 4.1 Backups - Hide Button for Guest Users (1 failure: test 274)
|
|
- **Test**: #274 (should hide Create Backup button for guest users)
|
|
- **Timeout**: 7.8s
|
|
- **Classification**: ⚠️ **POSSIBLY WORKING (needs manual verification)**
|
|
- **Issue**: Test expects "Create Backup" button to be hidden for guest users
|
|
- **Triage Notes**:
|
|
- Backups feature itself may work fine
|
|
- This specific authorization check may be missing or inverted
|
|
- Could be that button is visible when it shouldn't be, or test selector is wrong
|
|
- **Recommended Action**:
|
|
- Manually test: Log in as guest user → Go to Backups page → Verify "Create Backup" button is NOT visible
|
|
- If button is hidden: Test selector is wrong → Update test
|
|
- If button is visible: Backend is not checking guest permissions → Fix authorization
|
|
- Post-Phase 3: Either fix code or update test based on findings
|
|
|
|
---
|
|
|
|
### Category 5: Database State Accumulation (1 failure)
|
|
|
|
#### 5.1 Test Suite Degradation After ~250 Tests (potential pattern)
|
|
- **Observation**: First failures appear around test #150-154
|
|
- **Pattern**: Tests 324-335 all fail together (log viewer cluster)
|
|
- **Classification**: ⚠️ **POTENTIAL DATABASE/CACHE STATE ISSUE**
|
|
- **Triage Notes**:
|
|
- May be accumulation of test data affecting subsequent test performance
|
|
- Database cleanup between test suites may not be working
|
|
- Or fixture teardown not properly cleaning up auth state/sessions
|
|
- **Recommended Action**:
|
|
- Check if test data cleanup is running between test suites
|
|
- Verify fixture teardown doesn't leave orphaned records
|
|
- Consider adding intermediate `cleanupTestData()` call between suites
|
|
- Monitor if issue repeats on next full Phase 2 run
|
|
|
|
---
|
|
|
|
## Recommended Next Steps
|
|
|
|
### Immediate (Before Phase 3)
|
|
1. **Clarify log viewer scope** with user
|
|
- Move live logs → Phase 3 if needed
|
|
- Mark system logs features as xfail if not implemented
|
|
|
|
### Before Filing PRs (Post-Phase 3)
|
|
1. **Backend Dev**: Fix notification provider CRUD performance
|
|
2. **Backend Dev**: Fix uptime monitor initial ping state logic
|
|
3. **Frontend Dev**: Verify Docker integration or skip tests
|
|
4. **Backend Dev**: Manually test user invite/permission flows
|
|
5. **Frontend/Backend**: Verify backup button authorization for guests
|
|
6. **QA**: Check for test data cleanup issues after ~250 tests
|
|
|
|
### Phase 3 Readiness
|
|
- Current failures do **NOT block Phase 3 execution**
|
|
- Phase 2 at 91.7% is acceptable for proceeding to security enforcement
|
|
- Phase 3 tests can be run concurrently with Phase 2 failure remediation
|
|
|
|
---
|
|
|
|
## Test Failure Summary Table
|
|
|
|
| # | Test | Category | Impact | Fix Type | Owner |
|
|
|---|------|----------|--------|----------|-------|
|
|
| 154-155 | Proxy Hosts Docker | Code Bug | UI Feature | Backend/Frontend | Frontend Dev |
|
|
| 166 | Uptime Monitor | Code Bug | Logic Error | Backend | Backend Dev |
|
|
| 205-219 | Notifications CRUD | Code Bug | Performance | Backend | Backend Dev |
|
|
| 248, 258, 260, 262, 269-270 | User Management | Not Tested | Feature TBD | TBD | Backend Dev |
|
|
| 274 | Backups Auth | Minor | Authorization | Backend | Backend Dev |
|
|
| 324-335 | Logs Viewing | Scope TBD | Feature TBD | TBD | TBD |
|
|
|
|
---
|
|
|
|
## Decision Matrix: Proceed to Phase 3?
|
|
|
|
| Criterion | Status | Notes |
|
|
|-----------|--------|-------|
|
|
| **Pass Rate** | 91.7% | Meets 85%+ threshold for proceeding |
|
|
| **Blocking Issues** | None | No Phase 2 failures block Phase 3 |
|
|
| **Security Tests Ready** | ✅ Yes | Phase 3 test suite is complete |
|
|
| **Triage Complete** | ✅ Yes | All failures categorized |
|
|
| **Recommendation** | ✅ **PROCEED** | Phase 3 can run while Phase 2 fixes happen in parallel |
|
|
|
|
---
|
|
|
|
**Next Action**: Proceed to Phase 3 (Security UI & Enforcement) while scheduling Phase 2 remediation PRs.
|