Charon/docs/plans/current_spec.md

History Rewrite: Address Copilot Suggestions (PR #336)
===================================================

Summary
-------
- PR #336 introduced history-rewrite tooling, documentation, and a CI dry-run workflow to detect unwanted large blobs and CodeQL DB artifacts in repository history.
- Copilot left suggestions on the PR asserting a number of robustness, testing, validation, and safety improvements.
- This spec documents how to resolve those suggestions, lists the impacted files and functions, and provides an implementation & QA plan.

Copilot Suggestions (Short Summary)
----------------------------------
- Improve `validate_after_rewrite.sh` to use a defined `backup_branch` variable and fail gracefully when missing.
- Harden `clean_history.sh` and `preview_removals.sh` to handle shallow clones, tags, and refs, validate `git-filter-repo` args, and double-check backups (include tags & annotated refs).
- Add automated script unit tests (shell) for the scripts (preview/dry-run/validate) to make them testable and CI-friendly.
- Add a CI job to run these script tests (e.g., `bats-core`) and trap shallow clones early.
- Expand pre-commit and `.gitignore` coverage (include `data/backups`), validate `backup_branch` push, and refuse running filter-repo on `main`/`master` or non-existent remotes.
- Add more detailed PR checklist validation (tags, backup branch pushed) and update docs/examples.

Files Changed / Impacted
------------------------
Core scripts and CI currently touched by PR #336 and Copilot suggestions (primary targets):
- scripts/history-rewrite/clean_history.sh
  - Functions: `check_requirements`, `timestamp`, `preview_removals` block, local `backup_branch` creation.
  - Behaviors to harden: shallow clone handling; ensure backup branch pushed to remote and tags backed up; refuse to run on `main`/`master`; confirm `git-filter-repo` args are validated; ensure remote tag backup.
- scripts/history-rewrite/preview_removals.sh
  - Behaviors to add: more structured preview output (json or delimited), detect shallow clone and warn, add checks for tags & refs.
- scripts/history-rewrite/validate_after_rewrite.sh
  - Fix bug: `backup_branch` referenced but not set, add env variable or accept `--backup-branch` argument; verify pre-commit location; exit non-zero on failures.
- scripts/ci/dry_run_history_rewrite.sh
  - Add shallow clone detection and early fail with instructions to fetch full history; ensure `git rev-list` does not grow too large on very large repositories (timeout or cap); fail on conditions.
- .github/workflows/dry-run-history-rewrite.yml
  - Behavior: run the new tests; ensure fetch-depth 0; add `bats` runner step or `shellcheck` runner.
- .github/workflows/pr-checklist.yml
  - Behavior: enhance validation of PR body for additional checklist items: ensure `data/backups` log is attached, `tags` backup statement, and maintainers ack for forced rewrite.
- .github/PULL_REQUEST_TEMPLATE/history-rewrite.md
  - Behavior: update the checklist with new checks for tags and `data/backups/` and note `validate_after_rewrite.sh` will fail if not present.
- .gitignore
  - Add `data/backups/` to `.gitignore` to ensure backup logs are not accidentally committed.
- .pre-commit-config.yaml
  - Add a new `block-data-backups-commit` hook to prevent accidental commits to `data/backups`.

Potential Secondary Impact (best-guess; confirm):
- scripts/pre-commit-hooks/block-codeql-db-commits.sh (might need to be more strict): extend to check codeql-db-* and codeql-*.sarif patterns.
- scripts/ci/dry_run_history_rewrite.sh invocation in `.github/workflows/dry-run-history-rewrite.yml`: adjust to ensure `fetch-depth: 0` is set and that `git` is non-shallow.

Implementation Plan (Phases)
--------------------------
PHASE 1 — Script Hardening (2-4 days)
- Goals: fix functional bugs, add validation checks, handle edge cases (shallow clones, tag preservation), make scripts idempotent and testable.
- Tasks:
  1. Update `scripts/history-rewrite/validate_after_rewrite.sh`:
     - Add a command-line argument or `ENV` for `--backup-branch` and fallback to reading `backup_branch` from the log in `data/backups` if present.
     - Ensure it sets `backup_branch` correctly or exits with a clear message.
     - Ensure it currently fails the build on any reported issues (non-zero exit when pre-commit fails in CI mode).
  2. Update `scripts/history-rewrite/clean_history.sh`:
     - Detect shallow clones (if `git rev-parse --is-shallow-repository` returns true) and fail with instructions to `git fetch --unshallow`.
     - When creating `backup_branch`, also include tag backups: `git tag -l | xargs -n1 -I{} git tag -l -n {}...` and push tags to `origin` into `backup/tags/history-YYYY...` namespace OR save them to `data/backups/tags-*.tar`.
     - Validate `git-filter-repo` args are valid—use `git filter-repo --help` to confirm that provided `--strip-blobs-bigger-than` args are numbers and `--paths` exist in repo for the dry-run case.
     - Ensure `backup_branch` is pushed successfully, otherwise abort.
     - Make `read -r confirmation` explicit with `--` or a short timeout to avoid interactive hang; in scripts launched via terminal, interactive fallback is acceptable, but in CI this should not be used. Add `--non-interactive` to skip confirmation in CI with an explicit flag and require maintainers to pass `FORCE=1` in env to proceed.
  3. Update `scripts/history-rewrite/preview_removals.sh`:
     - Add structured `--format` option with `text` (default) and `json` for CI parsing; include commit oids, paths, and sizes in the output.
     - Detect & warn if the repo is shallow.
  4. Add a `scripts/history-rewrite/check_refs.sh` helper:
     - Print current branches, tags, and any remotes pointing to objects in the paths to be removed.
     - Output a tarball `data/backups/tags-YYYYMMDD.tar` with tag refs.

PHASE 2 — Testing & Automation (2-3 days)
- Goals: Add script unit tests and CI steps to run them; add a validation pipeline for maintainers to use.
- Tasks:
  1. Add `bats-core` test harness inside `scripts/history-rewrite/tests/`.
     - `scripts/history-rewrite/tests/preview_removals.bats` — tests ensuring the preview prints commits and objects for specified paths.
     - `scripts/history-rewrite/tests/clean_history.dryrun.bats` — tests that `--dry-run` exits non-zero when repo contains banned paths and that `--force` requires confirmation.
     - `scripts/history-rewrite/tests/validate_after_rewrite.bats` — tests that `validate_after_rewrite.sh` uses `--backup-branch` and fails with the correct non-zero codes when `backup_branch` is missing.
  2. Add a `ci/scripts/test-history-rewrite.yml` workflow to run bats tests in CI and to fail early on shallow clones or missing tools.
  3. Add a script-level `shellcheck` pass and a `bash` minimal lint step; use `shellcheck` GitHub Action or pre-commit hook.

PHASE 3 — PR Pipeline & Pre-commit (1-2 days)
- Goals: Prevent accidental destructive runs and accidental commits of generated backups.
- Tasks:
  1. Update the PR template `.github/PULL_REQUEST_TEMPLATE/history-rewrite.md` adding checklist items: tag backups, confirm `data/backups` tarball included, confirm remote pushed backup branch and tags, optional `CI verification output` from `preview_removals --format json`.
  2. Update `.github/workflows/pr-checklist.yml` to validate: presence of `preview_removals` output in PR body, a check that `data/backups` is attached, and additional keywords like `tag backup` and `backup branch pushed`.
  3. Add `.pre-commit-config.yaml` hook to block commits to `data/backups` and ensure `data/backups` is added to `.gitignore`.
  4. Add `scripts/pre-commit-hooks/validate-backup-branch.sh` which verifies that `backup_branch` exists and points to the expected ref(s).

PHASE 4 — Docs, QA & Rollout (1-2 days)
- Goals: Update docs, add reproducible tests, and provide QA instructions and rollback strategies.
- Tasks:
  1. Update `docs/plans/history_rewrite.md` to include:
     - `backup_branch` naming and tagging policy
     - `data/backups` layout, e.g., `metadata.json`, `tags.tar.gz`, `log` paths
     - Example `preview_removals --format json` output for PR inclusion
  2. Add `docs/plans/current_spec.md` (this file) containing the execution plan and timeline estimate.
  3. QA steps: run `clean_history.sh --dry-run`, `preview_removals.sh` with `--format json` for PR attachments, then proceed with `--force` only after maintainers confirm window; verify via `validate_after_rewrite.sh` and CI.

PHASE 5 — Post-Deploy & Maintenance (1 day)
- Run `git gc` and prune on mirrors; notify downstream consumers; update CI mirrors and caches. Verify repository size decreased within expected tolerance.

Unit & Integration Tests (Files & Functions)
-------------------------------------------
Add these test files to `scripts/history-rewrite/tests/`.
Unit test harness: `bats-core` recommended; tests should run without network and create ephemeral local repositories.

- `scripts/history-rewrite/tests/preview_removals.bats`:
  - test_preview_detects_banned_commits()
  - test_preview_detects_large_blob_sizes()
  - test_preview_outputs_json_when_requested()

- `scripts/history-rewrite/tests/clean_history.dryrun.bats`:
  - test_dry_run_exits_success_when_no_banned_paths()
  - test_dry_run_reports_banned_commits()
  - test_force_requires_confirmation() — simulate interactive confirmation or set `FORCE=1` with `--non-interactive` flag to test non-interactive usage.
  - test_refuse_on_main_branch() — ensures script refuses to run on `main`/`master`.

- `scripts/history-rewrite/tests/validate_after_rewrite.bats`:
  - test_validate_fails_when_backup_branch_missing()
  - test_validate_passes_when_backup_branch_provided_and_all_checks_clear()
  - test_validate_populates_log_and_error_when_precommit_fails()

Integration test (bash / simulated repository): a test that acts as a small git repo containing a `backend/codeql-db` folder and a large fake blob.
- `scripts/history-rewrite/tests/integration_clean_history.bats`:
  - test_integration_end_to_end_preview_then_dry_run(): create a local repo, add a large file under `backend/codeql-db`, commit it, run `preview_removals` to capture output, ensure `clean_history.sh --dry-run` detects it, then run `clean_history.sh --force` but only after backing up repo; verify `git rev-list` no longer returns commits for that path.

Exact tests & names (for maintainers' convenience):
- `scripts/history-rewrite/tests/preview_removals.bats::test_preview_detects_banned_commits`
- `scripts/history-rewrite/tests/preview_removals.bats::test_preview_outputs_json`
- `scripts/history-rewrite/tests/clean_history.dryrun.bats::test_dry_run_reports_banned_commits`
- `scripts/history-rewrite/tests/clean_history.dryrun.bats::test_force_requires_confirmation`
- `scripts/history-rewrite/tests/validate_after_rewrite.bats::test_validate_fails_when_backup_branch_missing`
- `scripts/history-rewrite/tests/integration_clean_history.bats::test_integration_end_to_end_preview_then_dry_run`

CI & Pre-commit Changes
-----------------------
- Add `data/backups/` to `.gitignore` (to avoid accidental commits of backup logs) and ensure `scripts` produce readable `data/backups` logs that can be attached to PRs.
- Add a new pre-commit hook `scripts/pre-commit-hooks/block-data-backups-commit.sh` to block user commits of `data/backups` and `data/backups/*` (mirror `block-codeql-db-commits.sh`).
- Add `shellcheck` to the pre-commit config or add a `scripts/ci/shellcheck_history_rewrite.yml` workflow that ensures scripts pass style checks.
- Create a new CI workflow: `.github/workflows/history-rewrite-tests.yml`
  - Steps: Checkout with `fetch-depth: 0`, install bats-core via apt or package manager, run the `bats` tests, run `shellcheck` for scripts, and run `scripts/ci/dry_run_history_rewrite.sh`.
- Update existing `.github/workflows/dry-run-history-rewrite.yml` to:
  - Ensure `fetch-depth: 0` in `actions/checkout` is set (already the case), and fail early for shallow clones; add a `shellcheck` step and `bats` tests step.

Potential Regressions & Rollback Strategies
-------------------------------------------
- Regressions:
  - Accidental removal of unrelated history entries due to incorrect `--paths` or `--invert-paths` usage.
  - Loss of tags or refs if not properly backed up and pushed to a safe place before rewrite.
  - CI breakage from new pre-commit hooks or failing `bats` tests.
  - Developer pipelines or forks could break from forced `--all --force` push if they do not follow the rollback steps.

- Mitigations & Rollback:
  - **Always create backups**: `backup_branch` and `backup/tags/history-YYYYMMDD` tarball stored outside the working repo (S3/GitHub release) prior to any `--force` push.
  - Maintain a simple rollback command sequence in the docs:
    - `git checkout -b restore/DATE backup/history-YYYYMMDD-HHMMSS`
    - `git push origin restore/DATE` and create a PR to restore the history (or directly replace refs on the remote as maintainers decide)
  - Keep the `data/backups/` tarball outside the repo in a known remote location (this will also help recovery if the `backup_branch` is not visible).
  - Ensure CI `dry-run` workflow is fully functional and fails on shallow clones so maintainers must re-run with a proper clone.
  - Add a section in `docs/plans/history_rewrite.md` to show commands to restore tags if they were mistakenly deleted.

Backwards Compatibility & Maintainers' Notes
-------------------------------------------
- The scripts must remain POSIX-compliant where pragmatic; use `/bin/sh` for portability.
- Avoid automatic `git push --all --force` from scripts; maintainers must perform final coordinated push.
- Scripts will remain safe by default (`--dry-run` or interactive) with `--force` and explicit `I UNDERSTAND` confirmation for destructive operations.

Timeline Estimate (Rough)
------------------------
- Script hardening: 2-4 days
- Tests & CI: 2-3 days
- PR pipeline updates & pre-commit hooks: 1-2 days
- Docs, QA & rollout ( manual coordination): 1-2 days
- Total: 6-11 business days (one-to-two weeks), may vary with availability of maintainers and CR feedback.

Deployment Checklist for Maintainers
----------------------------------
Before scheduling a destructive rewrite:
1. Verify all `bats` tests in `scripts/history-rewrite/tests` pass on CI.
2. Ensure backup branches and tags are pushed to `origin` (and optionally exported to external storage like an S3 bucket).
3. Confirm the PR uses `.github/PULL_REQUEST_TEMPLATE/history-rewrite.md` and the PR automation passes.
4. Run full `scripts/history-rewrite/clean_history.sh --dry-run` and `scripts/history-rewrite/preview_removals.sh --format json` locally and attach outputs to the PR.
5. Have at least two maintainers approve the destructive rewrite before pushing `git push --all --force`.

Development checklist
---------------------
 - [ ] Implement described script and validation changes.
 - [ ] Add `bats` tests and `history-rewrite` test CI workflow.
 - [ ] Add `data/backups/` to `.gitignore` and add pre-commit hooks to block accidental commits.
 - [ ] Update `pr-checklist.yml` to include tag-backup checks, backup logs, and PR content checks.
 - [ ] Add maintainers' docs and rollback examples.

Follow-ups / Outstanding Questions (ask maintainers)
--------------------------------------------------
- Should `data/backups` remain inside repo (but ignored) or be offloaded to a remote store before the rewrite?
- Should `clean_history.sh` create an optional tarball of `refs` and `tags` and push to `origin/backups/` or an alternate remote repository for longer term storage?
- For CI (bats) tests: do we want to install `bats-core` in the main CI image, or depend on an apt install in the `history-rewrite-tests` workflow?
- Is `git-filter-repo` present on official runner images or should we install it in the CI workflow each time? (script currently exits with `Please install git-filter-repo` advisory.)

Appendix: Example `bats` Test Skeleton (preview_removals)
------------------------------------------------------
You can start implementing the tests with `bats` like the following skeleton:

```
#!/usr/bin/env bats

setup() {
  repo_dir="$(mktemp -d)"
  cd "$repo_dir"
  git init -q
  mkdir -p backend/codeql-db
  echo "largefile" > backend/codeql-db/big.txt
  git add -A
  git commit -m "feat: add dummy codeql-db file" || exit 1
}

teardown() {
  rm -rf "$repo_dir"
}

@test "preview_removals reports commits in path" {
  run sh /workspace/scripts/history-rewrite/preview_removals.sh --paths 'backend/codeql-db' --strip-size 1
  [ "$status" -eq 0 ]
  [[ "$output" == *"Commits touching specified paths"* ]]
}
```

This same pattern can be reused to spawn a test repository and run `clean_history.sh --dry-run`, `validate_after_rewrite.sh` and assert expected outputs and exit codes.

Done.
# Investigation and Remediation Plan: CI Failures on feature/beta-release

## 1. Incident Summary
**Issue**: CI builds failing on `feature/beta-release`.
**Symptoms**:
- Frontend build fails due to missing module `../data/crowdsecPresets`.
- Backend coverage check fails (likely due to missing tests or artifacts).
- Docker build fails.
**Root Cause Identified**:
- The file `frontend/src/data/crowdsecPresets.ts` exists locally but was **ignored by git** due to an overly broad pattern in `.gitignore`.
- The pattern `data/` in `.gitignore` (intended for the root `data/` directory) accidentally matched `frontend/src/data/`.

## 2. Diagnosis Details
- **Local Environment**: The file `frontend/src/data/crowdsecPresets.ts` was present, so local `npm run build` and `npm run test:ci` passed.
- **CI Environment**: The file was missing because it was not committed.
- **Git Ignore Analysis**:
  - `.gitignore` contained `data/` under "Caddy Runtime Data".
  - This pattern matches any directory named `data` anywhere in the tree.
  - It matched `frontend/src/data/`, causing `crowdsecPresets.ts` to be ignored.

## 3. Remediation Steps
1.  **Fix `.gitignore`**:
    - Change `data/` to `/data/` to anchor it to the project root.
    - Change `frontend/frontend/` to `/frontend/frontend/` for safety.
2.  **Add Missing File**:
    - Force add or add `frontend/src/data/crowdsecPresets.ts` after fixing `.gitignore`.
3.  **Verify**:
    - Run `git check-ignore` to ensure the file is no longer ignored.
    - Run local build/test to ensure no regressions.

## 4. Verification Results
- **Local Tests**:
  - Backend Coverage: 85.4% (Pass)
  - Frontend Tests: 70 files passed (Pass)
  - Frontend Coverage: 85.97% (Pass)
  - Build: Passed
- **Git Status**:
  - `frontend/src/data/crowdsecPresets.ts` is now staged for commit.
  - `.gitignore` is modified and staged.

## 5. Next Actions
- Commit the changes with message: `fix: resolve CI failures by unignoring frontend data files`.
- Push to `feature/beta-release`.
- Monitor the next CI run.

## 6. Future Prevention
- Use anchored paths (starting with `/`) in `.gitignore` for root-level directories.
- Check `git status` for unexpected ignored files when adding new directories.
- Add a pre-commit check or CI step to verify that all imported modules exist in the git tree (though `tsc` in CI does this, the issue was the discrepancy between local and CI).