# Repository Structure Reorganization Plan **Date**: December 15, 2025 **Status**: Proposed **Risk Level**: Medium (requires CI/CD updates, Docker path changes) --- ## Executive Summary The repository root level currently contains **60+ items**, making it difficult to navigate and maintain. This plan proposes moving files into logical directories to achieve a cleaner, more organized structure with only **~15 essential items** at the root level. **Key Benefits**: - Easier navigation for contributors - Clearer separation of concerns - Reduced cognitive load when browsing repository - Better .gitignore and .dockerignore maintenance - Improved CI/CD workflow clarity --- ## Current Root-Level Analysis ### Category Breakdown | Category | Count | Examples | Status | |----------|-------|----------|--------| | **Docker Compose Files** | 5 | `docker-compose.yml`, `docker-compose.dev.yml`, etc. | 🔴 Scattered | | **CodeQL SARIF Files** | 6 | `codeql-go.sarif`, `codeql-results-*.sarif` | 🔴 Build artifacts at root | | **Implementation Docs** | 9 | `BULK_ACL_FEATURE.md`, `IMPLEMENTATION_SUMMARY.md`, etc. | 🔴 Should be in docs/ | | **Config Files** | 8 | `eslint.config.js`, `.pre-commit-config.yaml`, `Makefile`, etc. | 🟡 Mixed - some stay, some move | | **Docker Files** | 3 | `Dockerfile`, `docker-entrypoint.sh`, `DOCKER.md` | 🟡 Could group | | **Core Docs** | 4 | `README.md`, `CONTRIBUTING.md`, `LICENSE`, `VERSION.md` | 🟢 Stay at root | | **Hidden Config** | 15+ | `.github/`, `.vscode/`, `.gitignore`, `.dockerignore`, etc. | 🟢 Stay at root | | **Source Directories** | 7 | `backend/`, `frontend/`, `docs/`, `scripts/`, etc. | 🟢 Stay at root | | **Workspace File** | 1 | `Chiron.code-workspace` | 🟢 Stay at root | | **Build Artifacts** | 3 | `codeql-db/`, `codeql-agent-results/`, `.trivy_logs/` | 🔴 Gitignored but present | **Total Root Items**: ~60 items (files + directories) ### Problem Areas 1. **Docker Compose Sprawl**: 5 files at root when they should be grouped 2. **SARIF Pollution**: 6 CodeQL SARIF files are build artifacts (should be .gitignored) 3. **Documentation Chaos**: 9 implementation/feature docs scattered at root instead of `docs/` 4. **Mixed Purposes**: Docker files, configs, docs, code all at same level --- ## Proposed New Structure ### Root Level (Clean) ``` /projects/Charon/ ├── .github/ # GitHub workflows, templates, agents ├── .vscode/ # VS Code workspace settings ├── backend/ # Go backend source ├── configs/ # Runtime configs (CrowdSec, etc.) ├── data/ # Runtime data (gitignored) ├── docs/ # Documentation (enhanced) ├── frontend/ # React frontend source ├── logs/ # Runtime logs (gitignored) ├── scripts/ # Build/test/integration scripts ├── test-results/ # Test outputs (gitignored) ├── tools/ # Development tools │ ├── .codecov.yml # Codecov configuration ├── .dockerignore # Docker build exclusions ├── .gitattributes # Git attributes ├── .gitignore # Git exclusions ├── .goreleaser.yaml # GoReleaser config ├── .markdownlint.json # Markdown lint config ├── .markdownlintrc # Markdown lint config ├── .pre-commit-config.yaml # Pre-commit hooks ├── .sourcery.yml # Sourcery config ├── Chiron.code-workspace # VS Code workspace ├── CONTRIBUTING.md # Contribution guidelines ├── LICENSE # License file ├── Makefile # Build automation ├── README.md # Project readme ├── VERSION.md # Version documentation ├── eslint.config.js # ESLint config ├── go.work # Go workspace ├── go.work.sum # Go workspace checksums └── package.json # Root package.json (pre-commit, etc.) ``` ### New Directory: `.docker/` **Purpose**: Consolidate all Docker-related files except the primary Dockerfile ``` .docker/ ├── compose/ │ ├── docker-compose.yml # Main compose (moved from root) │ ├── docker-compose.dev.yml # Dev override (moved from root) │ ├── docker-compose.local.yml # Local override (moved from root) │ ├── docker-compose.remote.yml # Remote override (moved from root) │ └── README.md # Compose file documentation ├── docker-entrypoint.sh # Entrypoint script (moved from root) └── README.md # Docker documentation (DOCKER.md renamed) ``` **Why `.docker/` with a dot?** - Keeps it close to root-level Dockerfile (co-location) - Hidden by default in file browsers (reduces clutter) - Common pattern in monorepos (`.github/`, `.vscode/`) **Alternative**: Could use `docker/` without dot, but `.docker/` is preferred for consistency ### Enhanced: `docs/` **New subdirectory**: `docs/implementation/` **Purpose**: Archive completed implementation documents that shouldn't be at root ``` docs/ ├── implementation/ # NEW: Implementation documents │ ├── BULK_ACL_FEATURE.md # Moved from root │ ├── IMPLEMENTATION_SUMMARY.md # Moved from root │ ├── ISSUE_16_ACL_IMPLEMENTATION.md # Moved from root │ ├── QA_AUDIT_REPORT_LOADING_OVERLAYS.md # Moved from root │ ├── QA_MIGRATION_COMPLETE.md # Moved from root │ ├── SECURITY_CONFIG_PRIORITY.md # Moved from root │ ├── SECURITY_IMPLEMENTATION_PLAN.md # Moved from root │ ├── WEBSOCKET_FIX_SUMMARY.md # Moved from root │ └── README.md # Index of implementation docs ├── issues/ # Existing: Issue templates ├── plans/ # Existing: Planning documents │ ├── structure.md # THIS FILE │ └── ... ├── reports/ # Existing: Reports ├── troubleshooting/ # Existing: Troubleshooting guides ├── acme-staging.md ├── api.md ├── ... └── index.md ``` ### Enhanced: `.gitignore` **New entries** to prevent SARIF files at root: ```gitignore # Add to "CodeQL & Security Scanning" section: # ----------------------------------------------------------------------------- # CodeQL & Security Scanning # ----------------------------------------------------------------------------- # ... existing entries ... # Prevent SARIF files at root level /*.sarif /codeql-*.sarif # Explicit gitignore for scattered SARIF files /codeql-go.sarif /codeql-js.sarif /codeql-results-go.sarif /codeql-results-go-backend.sarif /codeql-results-go-new.sarif /codeql-results-js.sarif ``` --- ## File Migration Table ### Docker Compose Files → `.docker/compose/` | Current Path | New Path | Type | |-------------|----------|------| | `/docker-compose.yml` | `/.docker/compose/docker-compose.yml` | Move | | `/docker-compose.dev.yml` | `/.docker/compose/docker-compose.dev.yml` | Move | | `/docker-compose.local.yml` | `/.docker/compose/docker-compose.local.yml` | Move | | `/docker-compose.remote.yml` | `/.docker/compose/docker-compose.remote.yml` | Move | | `/docker-compose.override.yml` | `/.docker/compose/docker-compose.override.yml` | Move (if exists) | **Note**: `docker-compose.override.yml` is gitignored. Include in .gitignore update. ### Docker Support Files → `.docker/` | Current Path | New Path | Type | |-------------|----------|------| | `/docker-entrypoint.sh` | `/.docker/docker-entrypoint.sh` | Move | | `/DOCKER.md` | `/.docker/README.md` | Move + Rename | ### Implementation Docs → `docs/implementation/` | Current Path | New Path | Type | |-------------|----------|------| | `/BULK_ACL_FEATURE.md` | `/docs/implementation/BULK_ACL_FEATURE.md` | Move | | `/IMPLEMENTATION_SUMMARY.md` | `/docs/implementation/IMPLEMENTATION_SUMMARY.md` | Move | | `/ISSUE_16_ACL_IMPLEMENTATION.md` | `/docs/implementation/ISSUE_16_ACL_IMPLEMENTATION.md` | Move | | `/QA_AUDIT_REPORT_LOADING_OVERLAYS.md` | `/docs/implementation/QA_AUDIT_REPORT_LOADING_OVERLAYS.md` | Move | | `/QA_MIGRATION_COMPLETE.md` | `/docs/implementation/QA_MIGRATION_COMPLETE.md` | Move | | `/SECURITY_CONFIG_PRIORITY.md` | `/docs/implementation/SECURITY_CONFIG_PRIORITY.md` | Move | | `/SECURITY_IMPLEMENTATION_PLAN.md` | `/docs/implementation/SECURITY_IMPLEMENTATION_PLAN.md` | Move | | `/WEBSOCKET_FIX_SUMMARY.md` | `/docs/implementation/WEBSOCKET_FIX_SUMMARY.md` | Move | ### CodeQL SARIF Files → Delete (Add to .gitignore) | Current Path | Action | Reason | |-------------|--------|--------| | `/codeql-go.sarif` | Delete + gitignore | Build artifact | | `/codeql-js.sarif` | Delete + gitignore | Build artifact | | `/codeql-results-go.sarif` | Delete + gitignore | Build artifact | | `/codeql-results-go-backend.sarif` | Delete + gitignore | Build artifact | | `/codeql-results-go-new.sarif` | Delete + gitignore | Build artifact | | `/codeql-results-js.sarif` | Delete + gitignore | Build artifact | **Note**: These are generated by CodeQL and should never be committed. ### Files Staying at Root | File | Reason | |------|--------| | `Dockerfile` | Primary Docker build file - standard location | | `Makefile` | Build automation - standard location | | `README.md` | Project entry point - standard location | | `CONTRIBUTING.md` | Contributor guidelines - standard location | | `LICENSE` | License file - standard location | | `VERSION.md` | Version documentation - standard location | | `Chiron.code-workspace` | VS Code workspace - standard location | | `go.work`, `go.work.sum` | Go workspace - required at root | | `package.json` | Root package (pre-commit, etc.) - required at root | | `eslint.config.js` | ESLint config - required at root | | `.codecov.yml` | Codecov config - required at root | | `.goreleaser.yaml` | GoReleaser config - required at root | | `.markdownlint.json` | Markdown lint config - required at root | | `.pre-commit-config.yaml` | Pre-commit config - required at root | | `.sourcery.yml` | Sourcery config - required at root | | All `.git*` files | Git configuration - required at root | | All hidden directories | Standard locations | --- ## Impact Analysis ### Files Requiring Updates #### 1. GitHub Workflows (`.github/workflows/*.yml`) **Files to Update**: 25+ workflow files **Changes Needed**: ```yaml # OLD (scattered references): - 'Dockerfile' - 'docker-compose.yml' - 'docker-entrypoint.sh' - 'DOCKER.md' # NEW (centralized references): - 'Dockerfile' # Stays at root - '.docker/compose/docker-compose.yml' - '.docker/compose/docker-compose.*.yml' - '.docker/docker-entrypoint.sh' - '.docker/README.md' ``` **Specific Files**: - `.github/workflows/docker-lint.yml` - References Dockerfile (no change needed) - `.github/workflows/docker-build.yml` - May reference docker-compose - `.github/workflows/docker-publish.yml` - May reference docker-compose - `.github/workflows/waf-integration.yml` - References Dockerfile (no change needed) **Search Pattern**: `grep -r "docker-compose" .github/workflows/` #### 2. Scripts (`scripts/*.sh`) **Files to Update**: ~5 scripts **Changes Needed**: ```bash # OLD: docker-compose -f docker-compose.local.yml up -d docker compose -f docker-compose.yml -f docker-compose.dev.yml up # NEW: docker-compose -f .docker/compose/docker-compose.local.yml up -d docker compose -f .docker/compose/docker-compose.yml -f .docker/compose/docker-compose.dev.yml up ``` **Specific Files**: - `scripts/coraza_integration.sh` - Uses docker-compose.local.yml - `scripts/crowdsec_integration.sh` - Uses docker-compose files - `scripts/crowdsec_startup_test.sh` - Uses docker-compose files - `scripts/integration-test.sh` - Uses docker-compose files **Search Pattern**: `grep -r "docker-compose" scripts/` #### 3. VS Code Tasks (`.vscode/tasks.json`) **Changes Needed**: ```json // OLD: "docker compose -f docker-compose.override.yml up -d" "docker compose -f docker-compose.local.yml up -d" // NEW: "docker compose -f .docker/compose/docker-compose.override.yml up -d" "docker compose -f .docker/compose/docker-compose.local.yml up -d" ``` **Affected Tasks**: - "Build & Run: Local Docker Image" - "Build & Run: Local Docker Image No-Cache" - "Docker: Start Dev Environment" - "Docker: Stop Dev Environment" - "Docker: Start Local Environment" - "Docker: Stop Local Environment" #### 4. Makefile **Changes Needed**: ```makefile # OLD: docker-compose build docker-compose up -d docker-compose -f docker-compose.yml -f docker-compose.dev.yml up docker-compose down docker-compose logs -f # NEW: docker-compose -f .docker/compose/docker-compose.yml build docker-compose -f .docker/compose/docker-compose.yml up -d docker-compose -f .docker/compose/docker-compose.yml -f .docker/compose/docker-compose.dev.yml up docker-compose -f .docker/compose/docker-compose.yml down docker-compose -f .docker/compose/docker-compose.yml logs -f ``` #### 5. Dockerfile **Changes Needed**: ```dockerfile # OLD: COPY docker-entrypoint.sh /usr/local/bin/ # NEW: COPY .docker/docker-entrypoint.sh /usr/local/bin/ ``` **Line**: Search for `docker-entrypoint.sh` in Dockerfile #### 6. Documentation Files **Files to Update**: - `README.md` - May reference docker-compose files or DOCKER.md - `CONTRIBUTING.md` - May reference docker-compose files - `docs/getting-started.md` - Likely references docker-compose - `docs/debugging-local-container.md` - Likely references docker-compose - Any docs referencing implementation files moved to `docs/implementation/` **Search Pattern**: - `grep -r "docker-compose" docs/` - `grep -r "DOCKER.md" docs/` - `grep -r "BULK_ACL_FEATURE\|IMPLEMENTATION_SUMMARY" docs/` #### 7. .dockerignore **Changes Needed**: ```dockerignore # Add to "Documentation" section: docs/implementation/ # Update Docker Compose exclusion: .docker/ ``` #### 8. .gitignore **Changes Needed**: ```gitignore # Add explicit SARIF exclusions at root: /*.sarif /codeql-*.sarif # Update docker-compose.override.yml path: .docker/compose/docker-compose.override.yml ``` --- ## Migration Steps ### Phase 1: Preparation (No Breaking Changes) 1. **Create new directories**: ```bash mkdir -p .docker/compose mkdir -p docs/implementation ``` 2. **Create README files**: - `.docker/README.md` (content from DOCKER.md + compose guide) - `.docker/compose/README.md` (compose file documentation) - `docs/implementation/README.md` (index of implementation docs) 3. **Update .gitignore** (add SARIF exclusions): ```bash # Add to .gitignore: /*.sarif /codeql-*.sarif .docker/compose/docker-compose.override.yml ``` 4. **Commit preparation**: ```bash git add .docker/ docs/implementation/ .gitignore git commit -m "chore: prepare directory structure for reorganization" ``` ### Phase 2: Move Files (Breaking Changes) **⚠️ WARNING**: This phase will break existing workflows until all references are updated. 1. **Move Docker Compose files**: ```bash git mv docker-compose.yml .docker/compose/ git mv docker-compose.dev.yml .docker/compose/ git mv docker-compose.local.yml .docker/compose/ git mv docker-compose.remote.yml .docker/compose/ # docker-compose.override.yml is gitignored, no need to move ``` 2. **Move Docker support files**: ```bash git mv docker-entrypoint.sh .docker/ git mv DOCKER.md .docker/README.md ``` 3. **Move implementation docs**: ```bash git mv BULK_ACL_FEATURE.md docs/implementation/ git mv IMPLEMENTATION_SUMMARY.md docs/implementation/ git mv ISSUE_16_ACL_IMPLEMENTATION.md docs/implementation/ git mv QA_AUDIT_REPORT_LOADING_OVERLAYS.md docs/implementation/ git mv QA_MIGRATION_COMPLETE.md docs/implementation/ git mv SECURITY_CONFIG_PRIORITY.md docs/implementation/ git mv SECURITY_IMPLEMENTATION_PLAN.md docs/implementation/ git mv WEBSOCKET_FIX_SUMMARY.md docs/implementation/ ``` 4. **Delete SARIF files**: ```bash git rm codeql-go.sarif git rm codeql-js.sarif git rm codeql-results-go.sarif git rm codeql-results-go-backend.sarif git rm codeql-results-go-new.sarif git rm codeql-results-js.sarif ``` 5. **Commit file moves**: ```bash git commit -m "chore: reorganize repository structure - Move docker-compose files to .docker/compose/ - Move docker-entrypoint.sh to .docker/ - Move DOCKER.md to .docker/README.md - Move implementation docs to docs/implementation/ - Delete committed SARIF files (should be gitignored) " ``` ### Phase 3: Update References (Fix Breaking Changes) **Order matters**: Update in this sequence to minimize build failures. 1. **Update Dockerfile**: - Change `docker-entrypoint.sh` → `.docker/docker-entrypoint.sh` - Test: `docker build -t charon:test .` 2. **Update Makefile**: - Change all `docker-compose` commands to use `.docker/compose/docker-compose.yml` - Test: `make docker-build`, `make docker-up` 3. **Update .vscode/tasks.json**: - Change docker-compose paths in all tasks - Test: Run "Docker: Start Local Environment" task 4. **Update scripts/**: - Update `scripts/coraza_integration.sh` - Update `scripts/crowdsec_integration.sh` - Update `scripts/crowdsec_startup_test.sh` - Update `scripts/integration-test.sh` - Test: Run each script 5. **Update .github/workflows/**: - Update all workflows referencing docker-compose files - Test: Trigger workflows or dry-run locally 6. **Update .dockerignore**: - Add `.docker/` exclusion - Add `docs/implementation/` exclusion 7. **Update documentation**: - Update `README.md` - Update `CONTRIBUTING.md` - Update `docs/getting-started.md` - Update `docs/debugging-local-container.md` - Update any docs referencing moved files 8. **Commit all reference updates**: ```bash git add -A git commit -m "chore: update all references to reorganized files - Update Dockerfile to reference .docker/docker-entrypoint.sh - Update Makefile docker-compose paths - Update VS Code tasks with new compose paths - Update scripts with new compose paths - Update GitHub workflows with new paths - Update documentation references - Update .dockerignore and .gitignore " ``` ### Phase 4: Verification 1. **Local build test**: ```bash docker build -t charon:test . docker compose -f .docker/compose/docker-compose.yml build ``` 2. **Local run test**: ```bash docker compose -f .docker/compose/docker-compose.local.yml up -d # Verify Charon starts correctly docker compose -f .docker/compose/docker-compose.local.yml down ``` 3. **Backend tests**: ```bash cd backend && go test ./... ``` 4. **Frontend tests**: ```bash cd frontend && npm run test ``` 5. **Integration tests**: ```bash scripts/integration-test.sh ``` 6. **Pre-commit checks**: ```bash pre-commit run --all-files ``` 7. **VS Code tasks**: - Test "Build & Run: Local Docker Image" - Test "Docker: Start Local Environment" ### Phase 5: CI/CD Monitoring 1. **Push to feature branch**: ```bash git checkout -b chore/reorganize-structure git push origin chore/reorganize-structure ``` 2. **Create PR** with detailed description: - Link to this plan - List all changed files - Note breaking changes - Request review from maintainers 3. **Monitor CI/CD**: - Watch all workflow runs - Fix any failures immediately - Update this plan if new issues discovered 4. **After merge**: - Announce in project channels - Update any external documentation - Monitor for issues in next few days --- ## Risk Assessment ### High Risk Changes | Change | Risk | Mitigation | |--------|------|------------| | **Docker Compose Paths** | CI/CD workflows may break | Test all workflows locally before merge | | **Dockerfile COPY** | Docker build may fail | Test build immediately after change | | **VS Code Tasks** | Local development disrupted | Update tasks before file moves | | **Script References** | Integration tests may fail | Test all scripts after updates | ### Medium Risk Changes | Change | Risk | Mitigation | |--------|------|------------| | **Documentation References** | Broken links | Use find-and-replace, verify all links | | **Makefile Commands** | Local builds may fail | Test all make targets | | **.dockerignore** | Docker image size may change | Compare before/after image sizes | ### Low Risk Changes | Change | Risk | Mitigation | |--------|------|------------| | **Implementation Docs Move** | Internal docs, low impact | Update any cross-references | | **SARIF Deletion** | Already gitignored | None needed | | **.gitignore Updates** | Prevents future pollution | None needed | ### Rollback Plan If critical issues arise after merge: 1. **Immediate**: Revert the merge commit 2. **Analysis**: Identify what was missed in testing 3. **Fix**: Update this plan with new requirements 4. **Re-attempt**: Create new PR with fixes --- ## Success Criteria ✅ **Before Merge**: - [ ] All file moves completed - [ ] All references updated - [ ] Local Docker build succeeds - [ ] Local Docker run succeeds - [ ] Backend tests pass - [ ] Frontend tests pass - [ ] Integration tests pass - [ ] Pre-commit checks pass - [ ] All VS Code tasks work - [ ] Documentation updated - [ ] PR reviewed by maintainers ✅ **After Merge**: - [ ] All CI/CD workflows pass - [ ] Docker images build successfully - [ ] No broken links in documentation - [ ] No regressions reported - [ ] Root level has ~15 items (down from 60+) --- ## Alternative Approaches Considered ### Alternative 1: Keep Docker Files at Root **Pros**: No breaking changes, familiar location **Cons**: Doesn't solve the clutter problem **Decision**: Rejected - doesn't meet goal of cleaning up root ### Alternative 2: Use `docker/` Instead of `.docker/` **Pros**: More visible, no hidden directory **Cons**: Less consistent with `.github/`, `.vscode/` pattern **Decision**: Rejected - prefer hidden directory for consistency ### Alternative 3: Keep Implementation Docs at Root **Pros**: Easier to find for contributors **Cons**: Continues root-level clutter **Decision**: Rejected - docs belong in `docs/`, can add index ### Alternative 4: Move All Config Files to `.config/` **Pros**: Maximum organization **Cons**: Many tools expect configs at root (eslint, pre-commit, etc.) **Decision**: Rejected - tool requirements win over organization ### Alternative 5: Delete Old Implementation Docs **Pros**: Maximum cleanup **Cons**: Loses historical context, implementation notes **Decision**: Rejected - prefer archiving to deletion --- ## Future Enhancements After this reorganization, consider: 1. **`.config/` Directory**: For configs that don't need to be at root 2. **`build/` Directory**: For build artifacts and temporary files 3. **`deployments/` Directory**: For deployment configurations (Kubernetes, etc.) 4. **Submodule for Configs**: If `configs/` grows too large 5. **Documentation Site**: Consider moving docs to dedicated site structure --- ## References - [Twelve-Factor App](https://12factor.net/) - Config management - [GitHub's .github Directory](https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/creating-a-default-community-health-file) - [VS Code Workspace](https://code.visualstudio.com/docs/editor/workspaces) - [Docker Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) --- ## Appendix: Search Commands For agents implementing this plan, use these commands to find all references: ```bash # Find docker-compose references: grep -r "docker-compose\.yml" . --exclude-dir=node_modules --exclude-dir=.git # Find docker-entrypoint.sh references: grep -r "docker-entrypoint\.sh" . --exclude-dir=node_modules --exclude-dir=.git # Find DOCKER.md references: grep -r "DOCKER\.md" . --exclude-dir=node_modules --exclude-dir=.git # Find implementation doc references: grep -r "BULK_ACL_FEATURE\|IMPLEMENTATION_SUMMARY\|ISSUE_16_ACL" . --exclude-dir=node_modules --exclude-dir=.git # Find SARIF references: grep -r "\.sarif" . --exclude-dir=node_modules --exclude-dir=.git ``` --- **End of Plan**