Charon/docs/plans/archive/revert_ci_pipeline.md

title, status, scope, notes

title	status	scope	notes
Revert CI Pipeline Consolidation	draft	ci/workflows, integration, e2e, security	Restore per-workflow pull_request triggers, retire ci-pipeline.yml, and reestablish self-contained image builds.

1. Introduction

This plan dismantles the consolidated CI pipeline and restores individual pull_request triggers for component workflows. The goal is to return to a simple, independent workflow model where each integration or test workflow runs on PRs without relying on a central pipeline or shared image artifacts.

Objectives:

• Identify workflows that had pull_request triggers removed or were merged into ci-pipeline.yml.
• Restore per-workflow pull_request triggers for integration, E2E, and build workflows.
• Delete ci-pipeline.yml as the required path to retire the consolidated pipeline.
• Ensure each workflow is self-contained for image availability.

2. Research Findings

2.1 Current Consolidated Pipeline

• .github/workflows/ci-pipeline.yml runs on pull_request and bundles lint, image build, integration tests, E2E, coverage, CodeQL, Trivy, supply-chain scans, and gates.
• The pipeline builds and uploads an image artifact for integration and uses e2e-tests-split.yml via workflow_call.

2.2 Integration Workflows (Current State)

• .github/workflows/cerberus-integration.yml: workflow_dispatch only.
• .github/workflows/crowdsec-integration.yml: workflow_dispatch only.
• .github/workflows/waf-integration.yml: workflow_dispatch only.
• .github/workflows/rate-limit-integration.yml: workflow_dispatch only.
• Each workflow currently pulls a registry image and tags it as charon:local. There is no pull_request trigger and no local build step.

2.3 E2E Workflows (Current State)

• .github/workflows/e2e-tests-split.yml: workflow_call + workflow_dispatch only. No pull_request trigger.
• The build job can build an image locally when invoked directly, but the file is currently only invoked by ci-pipeline.yml.

2.4 Build Workflow (Current State)

• .github/workflows/docker-build.yml: workflow_dispatch only.
• This workflow is designed to be the main build pipeline but is not currently triggered by pull_request.

2.5 Security Workflows (Current State)

• .github/workflows/security-pr.yml: workflow_dispatch only.
• .github/workflows/supply-chain-pr.yml: workflow_dispatch only.
• .github/workflows/codeql.yml: schedule + workflow_dispatch only.
• These workflows include logic for push and pull_request contexts but their triggers do not include pull_request.

2.6 Historical Reference

• .github/workflows/e2e-tests.yml.backup and .github/workflows/e2e-tests-split.yml.backup show prior pull_request trigger patterns and path filters that can be restored.

3. Technical Specifications

3.1 Workflow Inventory and Trigger Restoration

Target workflows to restore pull_request triggers:

• .github/workflows/docker-build.yml
• .github/workflows/cerberus-integration.yml
• .github/workflows/crowdsec-integration.yml
• .github/workflows/waf-integration.yml
• .github/workflows/rate-limit-integration.yml
• .github/workflows/e2e-tests-split.yml
• .github/workflows/security-pr.yml
• .github/workflows/supply-chain-pr.yml
• .github/workflows/codeql.yml (decision point)

Notes:

• e2e-tests-split.yml should run directly on pull_request with the internal build job enabled, not only via workflow_call.
• security-pr.yml and supply-chain-pr.yml must include pull_request triggers so security coverage is not lost.
• codeql.yml needs a decision: re-enable pull_request in codeql.yml or leave CodeQL in a separate PR workflow. The consolidated pipeline is currently the only PR CodeQL path.

3.2 ci-pipeline.yml Decommission Strategy

Decision:

• Option A (required): delete ci-pipeline.yml to fully end the consolidated pipeline and avoid duplicate PR checks.

3.3 Image Availability Strategy (Critical Challenge)

Independent PR workflows cannot rely on a shared image from another workflow unless using artifacts or a registry. The user wants to avoid pipeline complexity.

Required behavior for each integration workflow:

• Restore the "Build Docker image (Local)" step in each integration workflow, reverting any artifact handover dependency.
• Build a local Docker image within the workflow before tests run.
• Tag the image as charon:local for consistency with existing scripts.
• Avoid external registry dependency for PR builds.

Impacted workflows:

• cerberus-integration.yml
• crowdsec-integration.yml
• waf-integration.yml
• rate-limit-integration.yml

E2E workflows:

• e2e-tests-split.yml already supports building an image locally when invoked directly. Ensure pull_request triggers route through this path (not workflow_call).

3.4 Pull Request Trigger Scope and Path Filters

• Use branch filters consistent with prior backups and docker-build.yml usage: main, development, feature/, hotfix/.
• Apply path filters for E2E to avoid unnecessary runs: frontend/, backend/, tests/**, playwright.config.js, .github/workflows/e2e-tests-split.yml.
• Integration workflows typically run on any backend/frontend changes. Consider adding path filters if desired, but default to full PR runs for parity with previous behavior.

3.5 Dependency and Concurrency Rules

• Remove workflow_run coupling to docker-build.yml for integration and E2E workflows. Each workflow should be independently triggered by pull_request.
• Keep job-level concurrency where it prevents duplicate runs on the same PR, but avoid cross-workflow dependencies.

4. Implementation Plan

Phase 1: Baseline Verification (Tests)

• Confirm current CI behavior for PRs: identify which checks are now only running via ci-pipeline.yml.
• Capture baseline PR check set from GitHub Actions UI for comparison after restoration.

Phase 2: Restore PR Triggers (Core Workflows)

• Add pull_request triggers to docker-build.yml with branches including main and development.
• Add pull_request triggers to cerberus-integration.yml, crowdsec-integration.yml, waf-integration.yml, and rate-limit-integration.yml.
• Add pull_request triggers to e2e-tests-split.yml, using the backup trigger block as the source of truth.

Phase 3: Make Integration Workflows Self-Contained

• Restore the "Build Docker image (Local)" step in each integration workflow and remove dependency on ci-pipeline.yml artifacts.
• Remove registry pull steps or make them optional for manual runs.
• Ensure test scripts continue to reference charon:local.

Phase 4: Security Workflow Triggers

• Add pull_request triggers to security-pr.yml and supply-chain-pr.yml as a mandatory requirement to preserve PR security coverage.
• Decide on CodeQL: either add pull_request to codeql.yml or create a dedicated PR CodeQL workflow. If the pipeline is deleted, CodeQL must have an alternative PR trigger.

Phase 5: Decommission ci-pipeline.yml

• Delete ci-pipeline.yml.

Phase 6: Validation and Audit

• Verify that PRs show the restored individual checks instead of a single pipeline job.
• Confirm each integration workflow completes without relying on registry or artifact inputs and includes the restored local build step.
• Validate E2E workflow runs directly on pull_request with build job executed locally.
• Confirm security workflows run on pull_request.

5. Acceptance Criteria (EARS)

• WHEN a pull_request is opened or updated, THE SYSTEM SHALL trigger docker-build.yml directly on pull_request for main and development.
• WHEN a pull_request is opened or updated, THE SYSTEM SHALL trigger cerberus-integration.yml, crowdsec-integration.yml, waf-integration.yml, and rate-limit-integration.yml on pull_request.
• WHEN an integration workflow runs on pull_request, THE SYSTEM SHALL restore and run the "Build Docker image (Local)" step, build a local Docker image, and tag it as charon:local before tests.
• WHEN a pull_request is opened or updated, THE SYSTEM SHALL trigger e2e-tests-split.yml directly on pull_request without relying on ci-pipeline.yml.
• WHEN the consolidated pipeline is retired, THE SYSTEM SHALL NOT run ci-pipeline.yml on pull_request.
• WHEN a pull_request is opened or updated, THE SYSTEM SHALL run security-pr.yml and supply-chain-pr.yml on pull_request.
• WHEN CodeQL is required for pull_request, THE SYSTEM SHALL run a CodeQL workflow on pull_request independent of ci-pipeline.yml.

6. Risks and Mitigations

• Risk: PR checks increase in parallel count and runtime. Mitigation: use path filters for E2E and consider optional filters for integration workflows.
• Risk: Image build duplication increases CI cost. Mitigation: keep builds scoped to workflows that need the image, and avoid registry pushes for PR builds.
• Risk: Security scans or CodeQL no longer run on PR if triggers are not restored. Mitigation: explicitly re-enable PR triggers in security workflows or add a dedicated PR security workflow.

7. Confidence Score

Confidence: 82 percent

Rationale: The workflow inventory and trigger gaps are clear. The main uncertainty is selecting the final CodeQL and security trigger model once ci-pipeline.yml is removed.