Charon/docs/plans/archive/ci_pipeline_fix_spec.md

CI Pipeline Integration and Gate Enforcement Fix Plan

Introduction

This plan addresses two pipeline defects in .github/workflows/ci-pipeline.yml :

• Integration jobs are skipped even when the image build/push is successful.
• Gate jobs report success even when upstream jobs are skipped.

The goal is to make the execution order deterministic and strict: Setup -> Build/Push -> Integration -> Integration Gate -> E2E -> E2E Gate, with gates failing if any required dependency is not successful.

Research Findings

Integration jobs are conditionally skipped

The integration jobs (integration-cerberus, integration-crowdsec, integration-waf, integration-ratelimit) are gated by the same if: expression in .github/workflows/ci-pipeline.yml . That expression requires:

• needs.build.result == 'success'
• needs.build.outputs.image_ref != ''
• the workflow not being explicitly disabled via workflow_dispatch input

This creates two likely skip paths:

1. Image reference availability is tied to Docker Hub only. If the build job does not push or resolve a Docker Hub reference, integration jobs skip even if an image exists elsewhere (e.g., GHCR).
2. Push policy is not part of the integration condition. The build job exposes image_pushed, but integration jobs do not check it. This prevents a predictable decision about whether an image is actually available in a registry the jobs can pull from.

Gate jobs accept skipped dependencies

The gate jobs (integration-gate, coverage-gate, codecov-gate, pipeline-gate) use if: always() and only fail on failure or cancelled. They do not fail on skipped, which allows skipped dependencies to be treated as a success.

Examples in .github/workflows/ci-pipeline.yml :

• integration-gate exits 0 when integration is skipped due to build state or run_integration being false.
• coverage-gate and pipeline-gate do not enforce a strict success-only check across dependencies.

Reusable E2E workflow masks skipped jobs

The reusable workflow .github/workflows/e2e-tests-split.yml includes a final job that explicitly converts skipped to success. That behavior is useful for partial workflow_dispatch runs, but in CI (where browser=all and test_category=all) it allows a silent skip to pass.

Technical Specifications

Requirements (EARS Notation)

• WHEN the build-and-push stage completes and produces a successful push, THE SYSTEM SHALL start all integration jobs.
• WHEN integration is required, THE SYSTEM SHALL fail the integration gate if any integration job result is not success.
• WHEN E2E tests are required, THE SYSTEM SHALL fail the E2E gate if the reusable workflow result is not success.
• WHEN coverage jobs are required, THE SYSTEM SHALL fail the coverage gate if any coverage or E2E dependency is not success.
• WHEN any required gate fails, THE SYSTEM SHALL fail the pipeline gate.
• WHEN a stage is enabled, THE SYSTEM SHALL treat any skipped or missing dependency as a gate failure.
• IF a stage is explicitly disabled via workflow_dispatch or workflow_call input, THEN THE SYSTEM SHALL skip the stage and its gate by using the same stage-enabled condition on the gate job.

Integration job eligibility and image selection

Define a single computed boolean output that decides whether integration should run. This avoids duplicating conditions across jobs, aligns with the image availability policy, and normalizes input booleans across workflow_dispatch and workflow_call.

Definitive architecture:

• Job setup outputs input_run_integration (user intent only).

• Job build-and-push computes final run_integration.

• Computed logic: run_integration = (needs.setup.outputs.input_run_integration == 'true') && (steps.push.outcome == 'success').

• Dependent jobs (integration + gate) use the exact same if expression: ${{ needs.build-and-push.outputs.run_integration == 'true' }}.

• Gate logic fails if any needs is not success.

• run_integration=true if and only if:

  • needs.setup.outputs.input_run_integration is true, and
  • the push step in build-and-push succeeds.

• Integration tests run in a separate job and require the image to be available in a registry. A pull_request event alone does not permit integration to run without a pushed image.

Recommended outputs:

• setup.outputs.input_run_integration: normalized input boolean derived from workflow_dispatch or workflow_call
• build-and-push.outputs.image_ref: resolved image reference with fallback to GHCR
• build-and-push.outputs.image_registry: dockerhub or ghcr
• build-and-push.outputs.image_pushed: true only when a registry push occurred
• build-and-push.outputs.run_integration: computed eligibility boolean

Integration jobs should use the same if: expression based on needs.build-and-push.outputs.run_integration and should pull from the resolved image_ref.

Gate enforcement pattern (fail on skipped or failed)

Use a strict pattern that fails on anything other than success when a stage is required. This should be reusable across integration, coverage, E2E, and pipeline gates. Gate jobs MUST use the same stage-enabled if as the jobs in the stage.

For integration, the gate job if condition must be ${{ needs.build-and-push.outputs.run_integration == 'true' }}.

Gate logic details (explicit YAML/script pattern):

1. Gate job uses the same stage-enabled if as the jobs in the stage.
2. Gate job uses a single verification step that inspects needs via JSON and fails if any required job is not success (including skipped or missing).
3. Gate job is skipped when the stage is intentionally disabled, since the job-level if matches the stage condition.

Reusable pattern (standard block or composite action):

• Inputs:
  • required_jobs: JSON array of job ids in scope for that gate.
• Logic:
  • Iterate required_jobs and fail on any result not equal to success.

Canonical gate step example (for plan reference):

steps:
  - name: Evaluate gate
    env:
      NEEDS_JSON: ${{ toJSON(needs) }}
      REQUIRED_JOBS: ${{ inputs.required_jobs }}
    run: |
      set -euo pipefail
      for job in $(echo "$REQUIRED_JOBS" | jq -r '.[]'); do
        result=$(echo "$NEEDS_JSON" | jq -r --arg job "$job" '.[$job].result // "missing"')
        if [[ "$result" != "success" ]]; then
          echo "::error::Gate failed: $job result is $result"
          exit 1
        fi
      done

Example stage_enabled signals by gate:

• Integration gate: needs.build-and-push.outputs.run_integration == 'true'
• E2E gate: inputs.run_e2e == 'true' (or the equivalent workflow input)
• Coverage gate: inputs.run_coverage == 'true'
• Pipeline gate: always true, but only depends on gates and required security jobs

E2E strictness

In .github/workflows/e2e-tests-split.yml , the final e2e-results job should only convert skipped to success when the skip is intentional (for example, the workflow is manually dispatched with browser or test_category not including that job). For CI runs with browser=all and test_category=all, any skipped job should be treated as a failure.

Integration run logic (must match actual build/push)

Integration jobs must depend on the actual execution of the build/push step and the explicit input toggle. Use a single source of truth from setup and build-and-push outputs:

• setup.outputs.input_run_integration: normalized input boolean derived from workflow_dispatch or workflow_call
• build-and-push.outputs.image_ref: resolved registry reference from the same push
• build-and-push.outputs.image_pushed: true only when a registry push occurred
• build-and-push.outputs.run_integration: computed boolean that validates input enablement and push availability

Integration job if: should be:

if: ${{ needs.build-and-push.outputs.run_integration == 'true' }}

run_integration must be computed using the strict integration requirement:

run_integration: ${{ (needs.setup.outputs.input_run_integration == 'true') && (steps.push.outcome == 'success') }}

Boolean/type safety

• Normalize workflow_dispatch string inputs using fromJSON before comparison.
• Preserve workflow_call boolean inputs as-is, and pass them through inputs.* without string comparisons.
• Use a setup step to emit normalized boolean outputs (for example, inputs.run_integration) so job conditions stay consistent and avoid mixed string/boolean logic.

Fail-fast strategy (efficiency)

Document and enforce a fail-fast strategy to reduce wasted runtime:

• For matrix jobs (E2E, coverage, or any parallel test suites), set strategy.fail-fast: true for CI runs so other matrix jobs stop when one fails.
• Downstream stages must need their gate job to prevent unnecessary execution after a failure.
• Use workflow concurrency with cancel-in-progress: true for CI workflows targeting the same branch to avoid redundant runs.

Sequence enforcement

Ensure the dependency chain is explicit and strict:

1. setup
2. build-and-push
3. integration jobs
4. integration-gate
5. e2e (reusable workflow)
6. e2e-gate (new)
7. coverage jobs
8. coverage-gate
9. codecov-gate
10. security jobs
11. pipeline-gate

Implementation Plan

Phase 1: CI Workflow Validation Plan

• Add or update workflow validation checks to detect skipped jobs in CI mode.
• Update e2e-tests-split.yml so the final e2e-results job fails if any job is skipped when inputs.browser=all and inputs.test_category=all.

Phase 2: Integration Stage Fix

• Add input_run_integration output in setup.
• Add a computed run_integration output in build-and-push using the push step outcome.
• Add a resolved image_ref output that can use GHCR as a fallback if Docker Hub is unavailable.
• Update all integration jobs to use the computed run_integration output and the resolved image_ref.

Phase 3: Gate Standardization

• Add a new e2e-gate job that fails if needs.e2e.result is not success when E2E is required.
• Implement a reusable gate-check block or composite action that accepts required_jobs and stage_enabled inputs.
• Update integration-gate, coverage-gate, codecov-gate, and pipeline-gate to enforce a strict success-only check for required dependencies.

Phase 4: Sequence and Dependency Updates

• Wire dependencies so coverage-backend and coverage-frontend depend on e2e-gate rather than integration-gate directly.
• Ensure pipeline-gate depends on all gates and required security jobs.

Phase 5: Documentation and Verification

• Update this plan with any final implementation decisions once validated.
• Document the new gating behavior in relevant CI documentation if present.

Acceptance Criteria

• Integration jobs run whenever input_run_integration is true and the build/push step succeeds.
• Integration gate fails if any integration job is skipped, failure, or cancelled while integration is required.
• E2E gate fails if the reusable E2E workflow result is not success while E2E is required.
• Coverage gate fails if any coverage or E2E dependency is not success while coverage is required.
• Pipeline gate fails if any required gate or security job is not success.
• The execution order is enforced as: Build -> Integration -> Integration Gate -> E2E -> E2E Gate -> Coverage -> Coverage Gate -> Codecov Gate -> Security -> Pipeline Gate.
• Fail-fast behavior is documented and applied for matrix jobs in CI runs.