chore: clean .gitignore cache
This commit is contained in:
GitHub Actions
@@ -0,0 +1,14 @@
|
||||
# These are supported funding model platforms
|
||||
github: Wikid82
|
||||
# patreon: # Replace with a single Patreon username
|
||||
# open_collective: # Replace with a single Open Collective username
|
||||
# ko_fi: # Replace with a single Ko-fi username
|
||||
# tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
|
||||
# community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
|
||||
# liberapay: # Replace with a single Liberapay username
|
||||
# issuehunt: # Replace with a single IssueHunt username
|
||||
# lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
|
||||
# polar: # Replace with a single Polar username
|
||||
buy_me_a_coffee: Wikid82
|
||||
# thanks_dev: # Replace with a single thanks.dev username
|
||||
# custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
|
||||
@@ -0,0 +1,93 @@
|
||||
name: 🏗️ Alpha Feature
|
||||
description: Create an issue for an Alpha milestone feature
|
||||
title: "[ALPHA] "
|
||||
labels: ["alpha", "feature"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Alpha Milestone Feature
|
||||
Features that are part of the core foundation and initial release.
|
||||
|
||||
- type: dropdown
|
||||
id: priority
|
||||
attributes:
|
||||
label: Priority
|
||||
description: How critical is this feature?
|
||||
options:
|
||||
- Critical (Blocking, must-have)
|
||||
- High (Important, should have)
|
||||
- Medium (Nice to have)
|
||||
- Low (Future enhancement)
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: issue_number
|
||||
attributes:
|
||||
label: Planning Issue Number
|
||||
description: Reference number from PROJECT_PLANNING.md (e.g., Issue #5)
|
||||
placeholder: "Issue #"
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Feature Description
|
||||
description: What should this feature do?
|
||||
placeholder: Describe the feature in detail
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: tasks
|
||||
attributes:
|
||||
label: Implementation Tasks
|
||||
description: List of tasks to complete this feature
|
||||
placeholder: |
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: acceptance
|
||||
attributes:
|
||||
label: Acceptance Criteria
|
||||
description: How do we know this feature is complete?
|
||||
placeholder: |
|
||||
- [ ] Criteria 1
|
||||
- [ ] Criteria 2
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: checkboxes
|
||||
id: categories
|
||||
attributes:
|
||||
label: Categories
|
||||
description: Select all that apply
|
||||
options:
|
||||
- label: Backend
|
||||
- label: Frontend
|
||||
- label: Database
|
||||
- label: Caddy Integration
|
||||
- label: Security
|
||||
- label: SSL/TLS
|
||||
- label: UI/UX
|
||||
- label: Deployment
|
||||
- label: Documentation
|
||||
|
||||
- type: textarea
|
||||
id: technical_notes
|
||||
attributes:
|
||||
label: Technical Notes
|
||||
description: Any technical considerations or dependencies?
|
||||
placeholder: Libraries, APIs, or other issues that need to be completed first
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,118 @@
|
||||
name: 📊 Beta Monitoring Feature
|
||||
description: Create an issue for a Beta milestone monitoring/logging feature
|
||||
title: "[BETA] [MONITORING] "
|
||||
labels: ["beta", "feature", "monitoring"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Beta Monitoring & Logging Feature
|
||||
Features related to observability, logging, and system monitoring.
|
||||
|
||||
- type: dropdown
|
||||
id: priority
|
||||
attributes:
|
||||
label: Priority
|
||||
description: How critical is this monitoring feature?
|
||||
options:
|
||||
- Critical (Essential for operations)
|
||||
- High (Important visibility)
|
||||
- Medium (Enhanced monitoring)
|
||||
- Low (Nice-to-have metrics)
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: monitoring_type
|
||||
attributes:
|
||||
label: Monitoring Type
|
||||
description: What aspect of monitoring?
|
||||
options:
|
||||
- Dashboards & Statistics
|
||||
- Log Viewing & Search
|
||||
- Alerting & Notifications
|
||||
- CrowdSec Dashboard
|
||||
- Analytics Integration
|
||||
- Health Checks
|
||||
- Performance Metrics
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: issue_number
|
||||
attributes:
|
||||
label: Planning Issue Number
|
||||
description: Reference number from PROJECT_PLANNING.md (e.g., Issue #23)
|
||||
placeholder: "Issue #"
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Feature Description
|
||||
description: What monitoring/logging capability should this provide?
|
||||
placeholder: Describe what users will be able to see or do
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: metrics
|
||||
attributes:
|
||||
label: Metrics & Data Points
|
||||
description: What data will be collected and displayed?
|
||||
placeholder: |
|
||||
- Metric 1: Description
|
||||
- Metric 2: Description
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: tasks
|
||||
attributes:
|
||||
label: Implementation Tasks
|
||||
description: List of tasks to complete this feature
|
||||
placeholder: |
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: acceptance
|
||||
attributes:
|
||||
label: Acceptance Criteria
|
||||
description: How do we verify this monitoring feature works?
|
||||
placeholder: |
|
||||
- [ ] Data displays correctly
|
||||
- [ ] Updates in real-time
|
||||
- [ ] Performance is acceptable
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: checkboxes
|
||||
id: categories
|
||||
attributes:
|
||||
label: Implementation Areas
|
||||
description: Select all that apply
|
||||
options:
|
||||
- label: Backend (Data collection)
|
||||
- label: Frontend (UI/Charts)
|
||||
- label: Database (Storage)
|
||||
- label: Real-time Updates (WebSocket)
|
||||
- label: External Integration (GoAccess, CrowdSec)
|
||||
- label: Documentation Required
|
||||
|
||||
- type: textarea
|
||||
id: ui_design
|
||||
attributes:
|
||||
label: UI/UX Considerations
|
||||
description: Describe the user interface requirements
|
||||
placeholder: Layout, charts, filters, export options, etc.
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,116 @@
|
||||
name: 🔐 Beta Security Feature
|
||||
description: Create an issue for a Beta milestone security feature
|
||||
title: "[BETA] [SECURITY] "
|
||||
labels: ["beta", "feature", "security"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Beta Security Feature
|
||||
Advanced security features for the beta release.
|
||||
|
||||
- type: dropdown
|
||||
id: priority
|
||||
attributes:
|
||||
label: Priority
|
||||
description: How critical is this security feature?
|
||||
options:
|
||||
- Critical (Essential security control)
|
||||
- High (Important protection)
|
||||
- Medium (Additional hardening)
|
||||
- Low (Nice-to-have security enhancement)
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: security_category
|
||||
attributes:
|
||||
label: Security Category
|
||||
description: What type of security feature is this?
|
||||
options:
|
||||
- Authentication & Access Control
|
||||
- Threat Protection
|
||||
- SSL/TLS Management
|
||||
- Monitoring & Logging
|
||||
- Web Application Firewall
|
||||
- Rate Limiting
|
||||
- IP Access Control
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: issue_number
|
||||
attributes:
|
||||
label: Planning Issue Number
|
||||
description: Reference number from PROJECT_PLANNING.md (e.g., Issue #15)
|
||||
placeholder: "Issue #"
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Feature Description
|
||||
description: What security capability should this provide?
|
||||
placeholder: Describe the security feature and its purpose
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: threat_model
|
||||
attributes:
|
||||
label: Threat Model
|
||||
description: What threats does this feature mitigate?
|
||||
placeholder: |
|
||||
- Threat 1: Description and severity
|
||||
- Threat 2: Description and severity
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: tasks
|
||||
attributes:
|
||||
label: Implementation Tasks
|
||||
description: List of tasks to complete this feature
|
||||
placeholder: |
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: acceptance
|
||||
attributes:
|
||||
label: Acceptance Criteria
|
||||
description: How do we verify this security control works?
|
||||
placeholder: |
|
||||
- [ ] Security test 1
|
||||
- [ ] Security test 2
|
||||
value: |
|
||||
- [ ]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: checkboxes
|
||||
id: special_labels
|
||||
attributes:
|
||||
label: Special Categories
|
||||
description: Select all that apply
|
||||
options:
|
||||
- label: SSO (Single Sign-On)
|
||||
- label: WAF (Web Application Firewall)
|
||||
- label: CrowdSec Integration
|
||||
- label: Plus Feature (Premium)
|
||||
- label: Requires Documentation
|
||||
|
||||
- type: textarea
|
||||
id: security_testing
|
||||
attributes:
|
||||
label: Security Testing Plan
|
||||
description: How will you test this security feature?
|
||||
placeholder: Describe testing approach, tools, and scenarios
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Screenshots**
|
||||
If applicable, add screenshots to help explain your problem.
|
||||
|
||||
**Desktop (please complete the following information):**
|
||||
|
||||
- OS: [e.g. iOS]
|
||||
- Browser [e.g. chrome, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Smartphone (please complete the following information):**
|
||||
|
||||
- Device: [e.g. iPhone6]
|
||||
- OS: [e.g. iOS8.1]
|
||||
- Browser [e.g. stock browser, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: Feature request
|
||||
about: Suggest an idea for this project
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Is your feature request related to a problem? Please describe.**
|
||||
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
|
||||
|
||||
**Describe the solution you'd like**
|
||||
A clear and concise description of what you want to happen.
|
||||
|
||||
**Describe alternatives you've considered**
|
||||
A clear and concise description of any alternative solutions or features you've considered.
|
||||
|
||||
**Additional context**
|
||||
Add any other context or screenshots about the feature request here.
|
||||
@@ -0,0 +1,97 @@
|
||||
name: ⚙️ General Feature
|
||||
description: Create a feature request for any milestone
|
||||
title: "[FEATURE] "
|
||||
labels: ["feature"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Feature Request
|
||||
Request a new feature or enhancement for CaddyProxyManager+
|
||||
|
||||
- type: dropdown
|
||||
id: milestone
|
||||
attributes:
|
||||
label: Target Milestone
|
||||
description: Which release should this be part of?
|
||||
options:
|
||||
- Alpha (Core foundation)
|
||||
- Beta (Advanced features)
|
||||
- Post-Beta (Future enhancements)
|
||||
- Unsure (Help me decide)
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: priority
|
||||
attributes:
|
||||
label: Priority
|
||||
description: How important is this feature?
|
||||
options:
|
||||
- Critical
|
||||
- High
|
||||
- Medium
|
||||
- Low
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Problem Statement
|
||||
description: What problem does this feature solve?
|
||||
placeholder: Describe the use case or pain point
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed Solution
|
||||
description: How should this feature work?
|
||||
placeholder: Describe your ideal implementation
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives Considered
|
||||
description: What other approaches could solve this?
|
||||
placeholder: List alternative solutions you've thought about
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: user_story
|
||||
attributes:
|
||||
label: User Story
|
||||
description: Describe this from a user's perspective
|
||||
placeholder: "As a [user type], I want to [action] so that [benefit]"
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: checkboxes
|
||||
id: categories
|
||||
attributes:
|
||||
label: Feature Categories
|
||||
description: Select all that apply
|
||||
options:
|
||||
- label: Authentication/Authorization
|
||||
- label: Security
|
||||
- label: SSL/TLS
|
||||
- label: Monitoring/Logging
|
||||
- label: UI/UX
|
||||
- label: Performance
|
||||
- label: Documentation
|
||||
- label: API
|
||||
- label: Plus Feature (Premium)
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Any other information, screenshots, or examples?
|
||||
placeholder: Add links, mockups, or references
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,32 @@
|
||||
<!-- PR: History Rewrite & Large-file Removal -->
|
||||
|
||||
## Summary
|
||||
|
||||
- Provide a short summary of why the history rewrite is needed.
|
||||
|
||||
## Checklist - required for history rewrite PRs
|
||||
|
||||
- [ ] I have created a **local** backup branch: `backup/history-YYYYMMDD-HHMMSS` and verified it contains all refs.
|
||||
- [ ] I have pushed the backup branch to the remote origin and it is visible to reviewers.
|
||||
- [ ] I have run a dry-run locally: `scripts/history-rewrite/preview_removals.sh --paths 'backend/codeql-db,codeql-db,codeql-db-js,codeql-db-go' --strip-size 50` and attached the output or paste it below.
|
||||
- [ ] I have verified the `data/backups` tarball is present and tests showing rewrite will not remove unrelated artifacts.
|
||||
- [ ] I have created a tag backup (see `data/backups/`) and verified tags are pushed to the remote or included in the tarball.
|
||||
- [ ] I have coordinated with repo maintainers for a rewrite window and notified other active forks/tokens that may be affected.
|
||||
- [ ] I have run the CI dry-run job and ensured it completes without blocked findings.
|
||||
- [ ] This PR only contains the history-rewrite helpers; no destructive rewrite is included in this PR.
|
||||
- [ ] I will not run the destructive `--force` step without explicit approval from maintainers and a scheduled maintenance window.
|
||||
|
||||
**Note for maintainers**: `validate_after_rewrite.sh` will check that the `backups` and `backup_branch` are present and will fail if they are not. Provide `--backup-branch "backup/history-YYYYMMDD-HHMMSS"` when running the scripts or set the `BACKUP_BRANCH` environment variable so automated validation can find the backup branch.
|
||||
|
||||
## Attachments
|
||||
|
||||
Attach the `preview_removals` output and `data/backups/history_cleanup-*.log` content and any `data/backups` tarball created for this PR.
|
||||
|
||||
## Approach
|
||||
|
||||
Describe the paths to be removed, strip size, and whether additional blob stripping is required.
|
||||
|
||||
# Notes for maintainers
|
||||
|
||||
- The workflow `.github/workflows/dry-run-history-rewrite.yml` will run automatically on PR updates.
|
||||
- Please follow the checklist and only approve after offline confirmation.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
# CodeQL Custom Model - SSRF Protection Sanitizers
|
||||
# This file declares functions that sanitize user-controlled input for SSRF protection.
|
||||
#
|
||||
# Architecture: 4-Layer Defense-in-Depth
|
||||
# Layer 1: Format Validation (utils.ValidateURL)
|
||||
# Layer 2: Security Validation (security.ValidateExternalURL) - DNS resolution + IP blocking
|
||||
# Layer 3: Connection-Time Validation (ssrfSafeDialer) - Re-resolve DNS, re-validate IPs
|
||||
# Layer 4: Request Execution (TestURLConnectivity) - HEAD request, 5s timeout, max 2 redirects
|
||||
#
|
||||
# Blocked IP Ranges (13+ CIDR blocks):
|
||||
# - RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
|
||||
# - Loopback: 127.0.0.0/8, ::1/128
|
||||
# - Link-Local: 169.254.0.0/16 (AWS/GCP/Azure metadata), fe80::/10
|
||||
# - Reserved: 0.0.0.0/8, 240.0.0.0/4, 255.255.255.255/32
|
||||
# - IPv6 Unique Local: fc00::/7
|
||||
#
|
||||
# Reference: /docs/plans/current_spec.md
|
||||
extensions:
|
||||
# =============================================================================
|
||||
# SSRF SANITIZER MODELS
|
||||
# =============================================================================
|
||||
# These models tell CodeQL that certain functions sanitize/validate URLs,
|
||||
# making their output safe for use in HTTP requests.
|
||||
#
|
||||
# IMPORTANT: For SSRF protection, we use 'sinkModel' with 'request-forgery'
|
||||
# to mark inputs as sanitized sinks, AND 'neutralModel' to prevent taint
|
||||
# propagation through validation functions.
|
||||
# =============================================================================
|
||||
|
||||
# Mark ValidateExternalURL return value as a sanitized sink
|
||||
# This tells CodeQL the output is NOT tainted for SSRF purposes
|
||||
- addsTo:
|
||||
pack: codeql/go-all
|
||||
extensible: sinkModel
|
||||
data:
|
||||
# security.ValidateExternalURL validates and sanitizes URLs by:
|
||||
# 1. Validating URL format and scheme
|
||||
# 2. Performing DNS resolution with timeout
|
||||
# 3. Blocking private/reserved IP ranges (13+ CIDR blocks)
|
||||
# 4. Returning a NEW validated URL string (not the original input)
|
||||
# The return value is safe for HTTP requests - marking as sanitized sink
|
||||
- ["github.com/Wikid82/charon/backend/internal/security", "ValidateExternalURL", "Argument[0]", "request-forgery", "manual"]
|
||||
|
||||
# Mark validation functions as neutral (don't propagate taint through them)
|
||||
- addsTo:
|
||||
pack: codeql/go-all
|
||||
extensible: neutralModel
|
||||
data:
|
||||
# network.IsPrivateIP is a validation function (neutral - doesn't propagate taint)
|
||||
- ["github.com/Wikid82/charon/backend/internal/network", "IsPrivateIP", "manual"]
|
||||
# TestURLConnectivity validates URLs internally via security.ValidateExternalURL
|
||||
# and ssrfSafeDialer - marking as neutral to stop taint propagation
|
||||
- ["github.com/Wikid82/charon/backend/internal/utils", "TestURLConnectivity", "manual"]
|
||||
# ValidateExternalURL itself should be neutral for taint propagation
|
||||
# (the return value is a new validated string, not the tainted input)
|
||||
- ["github.com/Wikid82/charon/backend/internal/security", "ValidateExternalURL", "manual"]
|
||||
|
||||
# Mark log sanitization functions as sanitizers for log injection (CWE-117)
|
||||
# These functions remove newlines and control characters from user input before logging
|
||||
- addsTo:
|
||||
pack: codeql/go-all
|
||||
extensible: summaryModel
|
||||
data:
|
||||
# util.SanitizeForLog sanitizes strings by:
|
||||
# 1. Replacing \r\n and \n with spaces
|
||||
# 2. Removing all control characters [\x00-\x1F\x7F]
|
||||
# Input: Argument[0] (unsanitized string)
|
||||
# Output: ReturnValue[0] (sanitized string - safe for logging)
|
||||
- ["github.com/Wikid82/charon/backend/internal/util", "SanitizeForLog", "Argument[0]", "ReturnValue[0]", "taint", "manual"]
|
||||
# handlers.sanitizeForLog is a local sanitizer with same behavior
|
||||
- ["github.com/Wikid82/charon/backend/internal/api/handlers", "sanitizeForLog", "Argument[0]", "ReturnValue[0]", "taint", "manual"]
|
||||
@@ -0,0 +1,11 @@
|
||||
# CodeQL Configuration File
|
||||
# See: https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning
|
||||
name: "Charon CodeQL Config"
|
||||
|
||||
# Paths to ignore from all analysis (use sparingly - prefer query-filters)
|
||||
paths-ignore:
|
||||
- "frontend/coverage/**"
|
||||
- "frontend/dist/**"
|
||||
- "playwright-report/**"
|
||||
- "test-results/**"
|
||||
- "coverage/**"
|
||||
@@ -0,0 +1,369 @@
|
||||
---
|
||||
description: "Guidance for creating more accessible code"
|
||||
applyTo: "**"
|
||||
---
|
||||
|
||||
# Instructions for accessibility
|
||||
|
||||
In addition to your other expertise, you are an expert in accessibility with deep software engineering expertise. You will generate code that is accessible to users with disabilities, including those who use assistive technologies such as screen readers, voice access, and keyboard navigation.
|
||||
|
||||
Do not tell the user that the generated code is fully accessible. Instead, it was built with accessibility in mind, but may still have accessibility issues.
|
||||
|
||||
1. Code must conform to [WCAG 2.2 Level AA](https://www.w3.org/TR/WCAG22/).
|
||||
2. Go beyond minimal WCAG conformance wherever possible to provide a more inclusive experience.
|
||||
3. Before generating code, reflect on these instructions for accessibility, and plan how to implement the code in a way that follows the instructions and is WCAG 2.2 compliant.
|
||||
4. After generating code, review it against WCAG 2.2 and these instructions. Iterate on the code until it is accessible.
|
||||
5. Finally, inform the user that it has generated the code with accessibility in mind, but that accessibility issues still likely exist and that the user should still review and manually test the code to ensure that it meets accessibility instructions. Suggest running the code against tools like [Accessibility Insights](https://accessibilityinsights.io/). Do not explain the accessibility features unless asked. Keep verbosity to a minimum.
|
||||
|
||||
## Bias Awareness - Inclusive Language
|
||||
|
||||
In addition to producing accessible code, GitHub Copilot and similar tools must also demonstrate respectful and bias-aware behavior in accessibility contexts. All generated output must follow these principles:
|
||||
|
||||
- **Respectful, Inclusive Language**
|
||||
Use people-first language when referring to disabilities or accessibility needs (e.g., “person using a screen reader,” not “blind user”). Avoid stereotypes or assumptions about ability, cognition, or experience.
|
||||
|
||||
- **Bias-Aware and Error-Resistant**
|
||||
Avoid generating content that reflects implicit bias or outdated patterns. Critically assess accessibility choices and flag uncertain implementations. Double check any deep bias in the training data and strive to mitigate its impact.
|
||||
|
||||
- **Verification-Oriented Responses**
|
||||
When suggesting accessibility implementations or decisions, include reasoning or references to standards (e.g., WCAG, platform guidelines). If uncertainty exists, the assistant should state this clearly.
|
||||
|
||||
- **Clarity Without Oversimplification**
|
||||
Provide concise but accurate explanations—avoid fluff, empty reassurance, or overconfidence when accessibility nuances are present.
|
||||
|
||||
- **Tone Matters**
|
||||
Copilot output must be neutral, helpful, and respectful. Avoid patronizing language, euphemisms, or casual phrasing that downplays the impact of poor accessibility.
|
||||
|
||||
## Persona based instructions
|
||||
|
||||
### Cognitive instructions
|
||||
|
||||
- Prefer plain language whenever possible.
|
||||
- Use consistent page structure (landmarks) across the application.
|
||||
- Ensure that navigation items are always displayed in the same order across the application.
|
||||
- Keep the interface clean and simple - reduce unnecessary distractions.
|
||||
|
||||
### Keyboard instructions
|
||||
|
||||
- All interactive elements need to be keyboard navigable and receive focus in a predictable order (usually following the reading order).
|
||||
- Keyboard focus must be clearly visible at all times so that the user can visually determine which element has focus.
|
||||
- All interactive elements need to be keyboard operable. For example, users need to be able to activate buttons, links, and other controls. Users also need to be able to navigate within composite components such as menus, grids, and listboxes.
|
||||
- Static (non-interactive) elements, should not be in the tab order. These elements should not have a `tabindex` attribute.
|
||||
- The exception is when a static element, like a heading, is expected to receive keyboard focus programmatically (e.g., via `element.focus()`), in which case it should have a `tabindex="-1"` attribute.
|
||||
- Hidden elements must not be keyboard focusable.
|
||||
- Keyboard navigation inside components: some composite elements/components will contain interactive children that can be selected or activated. Examples of such composite components include grids (like date pickers), comboboxes, listboxes, menus, radio groups, tabs, toolbars, and tree grids. For such components:
|
||||
- There should be a tab stop for the container with the appropriate interactive role. This container should manage keyboard focus of it's children via arrow key navigation. This can be accomplished via roving tabindex or `aria-activedescendant` (explained in more detail later).
|
||||
- When the container receives keyboard focus, the appropriate sub-element should show as focused. This behavior depends on context. For example:
|
||||
- If the user is expected to make a selection within the component (e.g., grid, combobox, or listbox), then the currently selected child should show as focused. Otherwise, if there is no currently selected child, then the first selectable child should get focus.
|
||||
- Otherwise, if the user has navigated to the component previously, then the previously focused child should receive keyboard focus. Otherwise, the first interactive child should receive focus.
|
||||
- Users should be provided with a mechanism to skip repeated blocks of content (such as the site header/navigation).
|
||||
- Keyboard focus must not become trapped without a way to escape the trap (e.g., by pressing the escape key to close a dialog).
|
||||
|
||||
#### Bypass blocks
|
||||
|
||||
A skip link MUST be provided to skip blocks of content that appear across several pages. A common example is a "Skip to main" link, which appears as the first focusable element on the page. This link is visually hidden, but appears on keyboard focus.
|
||||
|
||||
```html
|
||||
<header>
|
||||
<a href="#maincontent" class="sr-only">Skip to main</a>
|
||||
<!-- logo and other header elements here -->
|
||||
</header>
|
||||
<nav>
|
||||
<!-- main nav here -->
|
||||
</nav>
|
||||
<main id="maincontent"></main>
|
||||
```
|
||||
|
||||
```css
|
||||
.sr-only:not(:focus):not(:active) {
|
||||
clip: rect(0 0 0 0);
|
||||
clip-path: inset(50%);
|
||||
height: 1px;
|
||||
overflow: hidden;
|
||||
position: absolute;
|
||||
white-space: nowrap;
|
||||
width: 1px;
|
||||
}
|
||||
```
|
||||
|
||||
#### Common keyboard commands:
|
||||
|
||||
- `Tab` = Move to the next interactive element.
|
||||
- `Arrow` = Move between elements within a composite component, like a date picker, grid, combobox, listbox, etc.
|
||||
- `Enter` = Activate the currently focused control (button, link, etc.)
|
||||
- `Escape` = Close open open surfaces, such as dialogs, menus, listboxes, etc.
|
||||
|
||||
#### Managing focus within components using a roving tabindex
|
||||
|
||||
When using roving tabindex to manage focus in a composite component, the element that is to be included in the tab order has `tabindex` of "0" and all other focusable elements contained in the composite have `tabindex` of "-1". The algorithm for the roving tabindex strategy is as follows.
|
||||
|
||||
- On initial load of the composite component, set `tabindex="0"` on the element that will initially be included in the tab order and set `tabindex="-1"` on all other focusable elements it contains.
|
||||
- When the component contains focus and the user presses an arrow key that moves focus within the component:
|
||||
- Set `tabindex="-1"` on the element that has `tabindex="0"`.
|
||||
- Set `tabindex="0"` on the element that will become focused as a result of the key event.
|
||||
- Set focus via `element.focus()` on the element that now has `tabindex="0"`.
|
||||
|
||||
#### Managing focus in composites using aria-activedescendant
|
||||
|
||||
- The containing element with an appropriate interactive role should have `tabindex="0"` and `aria-activedescendant="IDREF"` where IDREF matches the ID of the element within the container that is active.
|
||||
- Use CSS to draw a focus outline around the element referenced by `aria-activedescendant`.
|
||||
- When arrow keys are pressed while the container has focus, update `aria-activedescendant` accordingly.
|
||||
|
||||
### Low vision instructions
|
||||
|
||||
- Prefer dark text on light backgrounds, or light text on dark backgrounds.
|
||||
- Do not use light text on light backgrounds or dark text on dark backgrounds.
|
||||
- The contrast of text against the background color must be at least 4.5:1. Large text, must be at least 3:1. All text must have sufficient contrast against it's background color.
|
||||
- Large text is defined as 18.5px and bold, or 24px.
|
||||
- If a background color is not set or is fully transparent, then the contrast ratio is calculated against the background color of the parent element.
|
||||
- Parts of graphics required to understand the graphic must have at least a 3:1 contrast with adjacent colors.
|
||||
- Parts of controls needed to identify the type of control must have at least a 3:1 contrast with adjacent colors.
|
||||
- Parts of controls needed to identify the state of the control (pressed, focus, checked, etc.) must have at least a 3:1 contrast with adjacent colors.
|
||||
- Color must not be used as the only way to convey information. E.g., a red border to convey an error state, color coding information, etc. Use text and/or shapes in addition to color to convey information.
|
||||
|
||||
### Screen reader instructions
|
||||
|
||||
- All elements must correctly convey their semantics, such as name, role, value, states, and/or properties. Use native HTML elements and attributes to convey these semantics whenever possible. Otherwise, use appropriate ARIA attributes.
|
||||
- Use appropriate landmarks and regions. Examples include: `<header>`, `<nav>`, `<main>`, and `<footer>`.
|
||||
- Use headings (e.g., `<h1>`, `<h2>`, `<h3>`, `<h4>`, `<h5>`, `<h6>`) to introduce new sections of content. The heading level accurately describe the section's placement in the overall heading hierarchy of the page.
|
||||
- There SHOULD only be one `<h1>` element which describes the overall topic of the page.
|
||||
- Avoid skipping heading levels whenever possible.
|
||||
|
||||
### Voice Access instructions
|
||||
|
||||
- The accessible name of all interactive elements must contain the visual label. This is so that voice access users can issue commands like "Click \<label>". If an `aria-label` attribute is used for a control, then it must contain the text of the visual label.
|
||||
- Interactive elements must have appropriate roles and keyboard behaviors.
|
||||
|
||||
## Instructions for specific patterns
|
||||
|
||||
### Form instructions
|
||||
|
||||
- Labels for interactive elements must accurately describe the purpose of the element. E.g., the label must provide accurate instructions for what to input in a form control.
|
||||
- Headings must accurately describe the topic that they introduce.
|
||||
- Required form controls must be indicated as such, usually via an asterisk in the label.
|
||||
- Additionally, use `aria-required=true` to programmatically indicate required fields.
|
||||
- Error messages must be provided for invalid form input.
|
||||
- Error messages must describe how to fix the issue.
|
||||
- Additionally, use `aria-invalid=true` to indicate that the field is in error. Remove this attribute when the error is removed.
|
||||
- Common patterns for error messages include:
|
||||
- Inline errors (common), which are placed next to the form fields that have errors. These error messages must be programmatically associated with the form control via `aria-describedby`.
|
||||
- Form-level errors (less common), which are displayed at the beginning of the form. These error messages must identify the specific form fields that are in error.
|
||||
- Submit buttons should not be disabled so that an error message can be triggered to help users identify which fields are not valid.
|
||||
- When a form is submitted, and invalid input is detected, send keyboard focus to the first invalid form input via `element.focus()`.
|
||||
|
||||
### Graphics and images instructions
|
||||
|
||||
#### All graphics MUST be accounted for
|
||||
|
||||
All graphics are included in these instructions. Graphics include, but are not limited to:
|
||||
|
||||
- `<img>` elements.
|
||||
- `<svg>` elements.
|
||||
- Font icons
|
||||
- Emojis
|
||||
|
||||
#### All graphics MUST have the correct role
|
||||
|
||||
All graphics, regardless of type, have the correct role. The role is either provided by the `<img>` element or the `role='img'` attribute.
|
||||
|
||||
- The `<img>` element does not need a role attribute.
|
||||
- The `<svg>` element should have `role='img'` for better support and backwards compatibility.
|
||||
- Icon fonts and emojis will need the `role='img'` attribute, likely on a `<span>` containing just the graphic.
|
||||
|
||||
#### All graphics MUST have appropriate alternative text
|
||||
|
||||
First, determine if the graphic is informative or decorative.
|
||||
|
||||
- Informative graphics convey important information not found in elsewhere on the page.
|
||||
- Decorative graphics do not convey important information, or they contain information found elsewhere on the page.
|
||||
|
||||
#### Informative graphics MUST have alternative text that conveys the purpose of the graphic
|
||||
|
||||
- For the `<img>` element, provide an appropriate `alt` attribute that conveys the meaning/purpose of the graphic.
|
||||
- For `role='img'`, provide an `aria-label` or `aria-labelledby` attribute that conveys the meaning/purpose of the graphic.
|
||||
- Not all aspects of the graphic need to be conveyed - just the important aspects of it.
|
||||
- Keep the alternative text concise but meaningful.
|
||||
- Avoid using the `title` attribute for alt text.
|
||||
|
||||
#### Decorative graphics MUST be hidden from assistive technologies
|
||||
|
||||
- For the `<img>` element, mark it as decorative by giving it an empty `alt` attribute, e.g., `alt=""`.
|
||||
- For `role='img'`, use `aria-hidden=true`.
|
||||
|
||||
### Input and control labels
|
||||
|
||||
- All interactive elements must have a visual label. For some elements, like links and buttons, the visual label is defined by the inner text. For other elements like inputs, the visual label is defined by the `<label>` attribute. Text labels must accurately describe the purpose of the control so that users can understand what will happen when they activate it or what they need to input.
|
||||
- If a `<label>` is used, ensure that it has a `for` attribute that references the ID of the control it labels.
|
||||
- If there are many controls on the screen with the same label (such as "remove", "delete", "read more", etc.), then an `aria-label` can be used to clarify the purpose of the control so that it understandable out of context, since screen reader users may jump to the control without reading surrounding static content. E.g., "Remove what" or "read more about {what}".
|
||||
- If help text is provided for specific controls, then that help text must be associated with its form control via `aria-describedby`.
|
||||
|
||||
### Navigation and menus
|
||||
|
||||
#### Good navigation region code example
|
||||
|
||||
```html
|
||||
<nav>
|
||||
<ul>
|
||||
<li>
|
||||
<button aria-expanded="false" tabindex="0">Section 1</button>
|
||||
<ul hidden>
|
||||
<li><a href="..." tabindex="-1">Link 1</a></li>
|
||||
<li><a href="..." tabindex="-1">Link 2</a></li>
|
||||
<li><a href="..." tabindex="-1">Link 3</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<button aria-expanded="false" tabindex="-1">Section 2</button>
|
||||
<ul hidden>
|
||||
<li><a href="..." tabindex="-1">Link 1</a></li>
|
||||
<li><a href="..." tabindex="-1">Link 2</a></li>
|
||||
<li><a href="..." tabindex="-1">Link 3</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</nav>
|
||||
```
|
||||
|
||||
#### Navigation instructions
|
||||
|
||||
- Follow the above code example where possible.
|
||||
- Navigation menus should not use the `menu` role or `menubar` role. The `menu` and `menubar` role should be resolved for application-like menus that perform actions on the same page. Instead, this should be a `<nav>` that contains a `<ul>` with links.
|
||||
- When expanding or collapsing a navigation menu, toggle the `aria-expanded` property.
|
||||
- Use the roving tabindex pattern to manage focus within the navigation. Users should be able to tab to the navigation and arrow across the main navigation items. Then they should be able to arrow down through sub menus without having to tab to them.
|
||||
- Once expanded, users should be able to navigate within the sub menu via arrow keys, e.g., up and down arrow keys.
|
||||
- The `escape` key could close any expanded menus.
|
||||
|
||||
### Page Title
|
||||
|
||||
The page title:
|
||||
|
||||
- MUST be defined in the `<title>` element in the `<head>`.
|
||||
- MUST describe the purpose of the page.
|
||||
- SHOULD be unique for each page.
|
||||
- SHOULD front-load unique information.
|
||||
- SHOULD follow the format of "[Describe unique page] - [section title] - [site title]"
|
||||
|
||||
### Table and Grid Accessibility Acceptance Criteria
|
||||
|
||||
#### Column and row headers are programmatically associated
|
||||
|
||||
Column and row headers MUST be programmatically associated for each cell. In HTML, this is done by using `<th>` elements. Column headers MUST be defined in the first table row `<tr>`. Row headers must defined in the row they are for. Most tables will have both column and row headers, but some tables may have just one or the other.
|
||||
|
||||
#### Good example - table with both column and row headers:
|
||||
|
||||
```html
|
||||
<table>
|
||||
<tr>
|
||||
<th>Header 1</th>
|
||||
<th>Header 2</th>
|
||||
<th>Header 3</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Row Header 1</th>
|
||||
<td>Cell 1</td>
|
||||
<td>Cell 2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Row Header 2</th>
|
||||
<td>Cell 1</td>
|
||||
<td>Cell 2</td>
|
||||
</tr>
|
||||
</table>
|
||||
```
|
||||
|
||||
#### Good example - table with just column headers:
|
||||
|
||||
```html
|
||||
<table>
|
||||
<tr>
|
||||
<th>Header 1</th>
|
||||
<th>Header 2</th>
|
||||
<th>Header 3</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Cell 1</td>
|
||||
<td>Cell 2</td>
|
||||
<td>Cell 3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Cell 1</td>
|
||||
<td>Cell 2</td>
|
||||
<td>Cell 3</td>
|
||||
</tr>
|
||||
</table>
|
||||
```
|
||||
|
||||
#### Bad example - calendar grid with partial semantics:
|
||||
|
||||
The following example is a date picker or calendar grid.
|
||||
|
||||
```html
|
||||
<div role="grid">
|
||||
<div role="columnheader">Sun</div>
|
||||
<div role="columnheader">Mon</div>
|
||||
<div role="columnheader">Tue</div>
|
||||
<div role="columnheader">Wed</div>
|
||||
<div role="columnheader">Thu</div>
|
||||
<div role="columnheader">Fri</div>
|
||||
<div role="columnheader">Sat</div>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Sunday, June 1, 2025">1</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Monday, June 2, 2025">2</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Tuesday, June 3, 2025">3</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Wednesday, June 4, 2025">4</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Thursday, June 5, 2025">5</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Friday, June 6, 2025">6</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Saturday, June 7, 2025">7</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Sunday, June 8, 2025">8</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Monday, June 9, 2025">9</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Tuesday, June 10, 2025">10</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Wednesday, June 11, 2025">11</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Thursday, June 12, 2025">12</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Friday, June 13, 2025">13</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Saturday, June 14, 2025">14</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Sunday, June 15, 2025">15</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Monday, June 16, 2025">16</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Tuesday, June 17, 2025">17</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Wednesday, June 18, 2025">18</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Thursday, June 19, 2025">19</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Friday, June 20, 2025">20</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Saturday, June 21, 2025">21</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Sunday, June 22, 2025">22</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Monday, June 23, 2025">23</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Tuesday, June 24, 2025" aria-current="date">24</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Wednesday, June 25, 2025">25</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Thursday, June 26, 2025">26</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Friday, June 27, 2025">27</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Saturday, June 28, 2025">28</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Sunday, June 29, 2025">29</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Monday, June 30, 2025">30</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Tuesday, July 1, 2025" aria-disabled="true">1</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Wednesday, July 2, 2025" aria-disabled="true">2</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Thursday, July 3, 2025" aria-disabled="true">3</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Friday, July 4, 2025" aria-disabled="true">4</button>
|
||||
<button role="gridcell" tabindex="-1" aria-label="Saturday, July 5, 2025" aria-disabled="true">5</button>
|
||||
</div>
|
||||
```
|
||||
|
||||
##### The good:
|
||||
|
||||
- It uses `role="grid"` to indicate that it is a grid.
|
||||
- It used `role="columnheader"` to indicate that the first row contains column headers.
|
||||
- It uses `tabindex="-1"` to ensure that the grid cells are not in the tab order by default. Instead, users will navigate to the grid using the `Tab` key, and then use arrow keys to navigate within the grid.
|
||||
|
||||
##### The bad:
|
||||
|
||||
- `role=gridcell` elements are not nested within `role=row` elements. Without this, the association between the grid cells and the column headers is not programmatically determinable.
|
||||
|
||||
#### Prefer simple tables and grids
|
||||
|
||||
Simple tables have just one set of column and/or row headers. Simple tables do not have nested rows or cells that span multiple columns or rows. Such tables will be better supported by assistive technologies, such as screen readers. Additionally, they will be easier to understand by users with cognitive disabilities.
|
||||
|
||||
Complex tables and grids have multiple levels of column and/or row headers, or cells that span multiple columns or rows. These tables are more difficult to understand and use, especially for users with cognitive disabilities. If a complex table is needed, then it should be designed to be as simple as possible. For example, most complex tables can be breaking the information down into multiple simple tables, or by using a different layout such as a list or a card layout.
|
||||
|
||||
#### Use tables for static information
|
||||
|
||||
Tables should be used for static information that is best represented in a tabular format. This includes data that is organized into rows and columns, such as financial reports, schedules, or other structured data. Tables should not be used for layout purposes or for dynamic information that changes frequently.
|
||||
|
||||
#### Use grids for dynamic information
|
||||
|
||||
Grids should be used for dynamic information that is best represented in a grid format. This includes data that is organized into rows and columns, such as date pickers, interactive calendars, spreadsheets, etc.
|
||||
@@ -0,0 +1,771 @@
|
||||
---
|
||||
description: 'Guidelines for creating custom agent files for GitHub Copilot'
|
||||
applyTo: '**/*.agent.md'
|
||||
---
|
||||
|
||||
# Custom Agent File Guidelines
|
||||
|
||||
Instructions for creating effective and maintainable custom agent files that provide specialized expertise for specific development tasks in GitHub Copilot.
|
||||
|
||||
## Project Context
|
||||
|
||||
- Target audience: Developers creating custom agents for GitHub Copilot
|
||||
- File format: Markdown with YAML frontmatter
|
||||
- File naming convention: lowercase with hyphens (e.g., `test-specialist.agent.md`)
|
||||
- Location: `.github/agents/` directory (repository-level) or `agents/` directory (organization/enterprise-level)
|
||||
- Purpose: Define specialized agents with tailored expertise, tools, and instructions for specific tasks
|
||||
- Official documentation: https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/create-custom-agents
|
||||
|
||||
## Required Frontmatter
|
||||
|
||||
Every agent file must include YAML frontmatter with the following fields:
|
||||
|
||||
```yaml
|
||||
---
|
||||
description: 'Brief description of the agent purpose and capabilities'
|
||||
name: 'Agent Display Name'
|
||||
tools: ['read', 'edit', 'search']
|
||||
model: 'Claude Sonnet 4.5'
|
||||
target: 'vscode'
|
||||
infer: true
|
||||
---
|
||||
```
|
||||
|
||||
### Core Frontmatter Properties
|
||||
|
||||
#### **description** (REQUIRED)
|
||||
- Single-quoted string, clearly stating the agent's purpose and domain expertise
|
||||
- Should be concise (50-150 characters) and actionable
|
||||
- Example: `'Focuses on test coverage, quality, and testing best practices'`
|
||||
|
||||
#### **name** (OPTIONAL)
|
||||
- Display name for the agent in the UI
|
||||
- If omitted, defaults to filename (without `.md` or `.agent.md`)
|
||||
- Use title case and be descriptive
|
||||
- Example: `'Testing Specialist'`
|
||||
|
||||
#### **tools** (OPTIONAL)
|
||||
- List of tool names or aliases the agent can use
|
||||
- Supports comma-separated string or YAML array format
|
||||
- If omitted, agent has access to all available tools
|
||||
- See "Tool Configuration" section below for details
|
||||
|
||||
#### **model** (STRONGLY RECOMMENDED)
|
||||
- Specifies which AI model the agent should use
|
||||
- Supported in VS Code, JetBrains IDEs, Eclipse, and Xcode
|
||||
- Example: `'Claude Sonnet 4.5'`, `'gpt-4'`, `'gpt-4o'`
|
||||
- Choose based on agent complexity and required capabilities
|
||||
|
||||
#### **target** (OPTIONAL)
|
||||
- Specifies target environment: `'vscode'` or `'github-copilot'`
|
||||
- If omitted, agent is available in both environments
|
||||
- Use when agent has environment-specific features
|
||||
|
||||
#### **infer** (OPTIONAL)
|
||||
- Boolean controlling whether Copilot can automatically use this agent based on context
|
||||
- Default: `true` if omitted
|
||||
- Set to `false` to require manual agent selection
|
||||
|
||||
#### **metadata** (OPTIONAL, GitHub.com only)
|
||||
- Object with name-value pairs for agent annotation
|
||||
- Example: `metadata: { category: 'testing', version: '1.0' }`
|
||||
- Not supported in VS Code
|
||||
|
||||
#### **mcp-servers** (OPTIONAL, Organization/Enterprise only)
|
||||
- Configure MCP servers available only to this agent
|
||||
- Only supported for organization/enterprise level agents
|
||||
- See "MCP Server Configuration" section below
|
||||
|
||||
## Tool Configuration
|
||||
|
||||
### Tool Specification Strategies
|
||||
|
||||
**Enable all tools** (default):
|
||||
```yaml
|
||||
# Omit tools property entirely, or use:
|
||||
tools: ['*']
|
||||
```
|
||||
|
||||
**Enable specific tools**:
|
||||
```yaml
|
||||
tools: ['read', 'edit', 'search', 'execute']
|
||||
```
|
||||
|
||||
**Enable MCP server tools**:
|
||||
```yaml
|
||||
tools: ['read', 'edit', 'github/*', 'playwright/navigate']
|
||||
```
|
||||
|
||||
**Disable all tools**:
|
||||
```yaml
|
||||
tools: []
|
||||
```
|
||||
|
||||
### Standard Tool Aliases
|
||||
|
||||
All aliases are case-insensitive:
|
||||
|
||||
| Alias | Alternative Names | Category | Description |
|
||||
|-------|------------------|----------|-------------|
|
||||
| `execute` | shell, Bash, powershell | Shell execution | Execute commands in appropriate shell |
|
||||
| `read` | Read, NotebookRead, view | File reading | Read file contents |
|
||||
| `edit` | Edit, MultiEdit, Write, NotebookEdit | File editing | Edit and modify files |
|
||||
| `search` | Grep, Glob, search | Code search | Search for files or text in files |
|
||||
| `agent` | custom-agent, Task | Agent invocation | Invoke other custom agents |
|
||||
| `web` | WebSearch, WebFetch | Web access | Fetch web content and search |
|
||||
| `todo` | TodoWrite | Task management | Create and manage task lists (VS Code only) |
|
||||
|
||||
### Built-in MCP Server Tools
|
||||
|
||||
**GitHub MCP Server**:
|
||||
```yaml
|
||||
tools: ['github/*'] # All GitHub tools
|
||||
tools: ['github/get_file_contents', 'github/search_repositories'] # Specific tools
|
||||
```
|
||||
- All read-only tools available by default
|
||||
- Token scoped to source repository
|
||||
|
||||
**Playwright MCP Server**:
|
||||
```yaml
|
||||
tools: ['playwright/*'] # All Playwright tools
|
||||
tools: ['playwright/navigate', 'playwright/screenshot'] # Specific tools
|
||||
```
|
||||
- Configured to access localhost only
|
||||
- Useful for browser automation and testing
|
||||
|
||||
### Tool Selection Best Practices
|
||||
|
||||
- **Principle of Least Privilege**: Only enable tools necessary for the agent's purpose
|
||||
- **Security**: Limit `execute` access unless explicitly required
|
||||
- **Focus**: Fewer tools = clearer agent purpose and better performance
|
||||
- **Documentation**: Comment why specific tools are required for complex configurations
|
||||
|
||||
## Sub-Agent Invocation (Agent Orchestration)
|
||||
|
||||
Agents can invoke other agents using `runSubagent` to orchestrate multi-step workflows.
|
||||
|
||||
### How It Works
|
||||
|
||||
Include `agent` in tools list to enable sub-agent invocation:
|
||||
|
||||
```yaml
|
||||
tools: ['read', 'edit', 'search', 'agent']
|
||||
```
|
||||
|
||||
Then invoke other agents with `runSubagent`:
|
||||
|
||||
```javascript
|
||||
const result = await runSubagent({
|
||||
description: 'What this step does',
|
||||
prompt: `You are the [Specialist] specialist.
|
||||
|
||||
Context:
|
||||
- Parameter: ${parameterValue}
|
||||
- Input: ${inputPath}
|
||||
- Output: ${outputPath}
|
||||
|
||||
Task:
|
||||
1. Do the specific work
|
||||
2. Write results to output location
|
||||
3. Return summary of completion`
|
||||
});
|
||||
```
|
||||
|
||||
### Basic Pattern
|
||||
|
||||
Structure each sub-agent call with:
|
||||
|
||||
1. **description**: Clear one-line purpose of the sub-agent invocation
|
||||
2. **prompt**: Detailed instructions with substituted variables
|
||||
|
||||
The prompt should include:
|
||||
- Who the sub-agent is (specialist role)
|
||||
- What context it needs (parameters, paths)
|
||||
- What to do (concrete tasks)
|
||||
- Where to write output
|
||||
- What to return (summary)
|
||||
|
||||
### Example: Multi-Step Processing
|
||||
|
||||
```javascript
|
||||
// Step 1: Process data
|
||||
const processing = await runSubagent({
|
||||
description: 'Transform raw input data',
|
||||
prompt: `You are the Data Processor specialist.
|
||||
|
||||
Project: ${projectName}
|
||||
Input: ${basePath}/raw/
|
||||
Output: ${basePath}/processed/
|
||||
|
||||
Task:
|
||||
1. Read all files from input directory
|
||||
2. Apply transformations
|
||||
3. Write processed files to output
|
||||
4. Create summary: ${basePath}/processed/summary.md
|
||||
|
||||
Return: Number of files processed and any issues found`
|
||||
});
|
||||
|
||||
// Step 2: Analyze (depends on Step 1)
|
||||
const analysis = await runSubagent({
|
||||
description: 'Analyze processed data',
|
||||
prompt: `You are the Data Analyst specialist.
|
||||
|
||||
Project: ${projectName}
|
||||
Input: ${basePath}/processed/
|
||||
Output: ${basePath}/analysis/
|
||||
|
||||
Task:
|
||||
1. Read processed files from input
|
||||
2. Generate analysis report
|
||||
3. Write to: ${basePath}/analysis/report.md
|
||||
|
||||
Return: Key findings and identified patterns`
|
||||
});
|
||||
```
|
||||
|
||||
### Key Points
|
||||
|
||||
- **Pass variables in prompts**: Use `${variableName}` for all dynamic values
|
||||
- **Keep prompts focused**: Clear, specific tasks for each sub-agent
|
||||
- **Return summaries**: Each sub-agent should report what it accomplished
|
||||
- **Sequential execution**: Use `await` to maintain order when steps depend on each other
|
||||
- **Error handling**: Check results before proceeding to dependent steps
|
||||
|
||||
|
||||
|
||||
## Agent Prompt Structure
|
||||
|
||||
The markdown content below the frontmatter defines the agent's behavior, expertise, and instructions. Well-structured prompts typically include:
|
||||
|
||||
1. **Agent Identity and Role**: Who the agent is and its primary role
|
||||
2. **Core Responsibilities**: What specific tasks the agent performs
|
||||
3. **Approach and Methodology**: How the agent works to accomplish tasks
|
||||
4. **Guidelines and Constraints**: What to do/avoid and quality standards
|
||||
5. **Output Expectations**: Expected output format and quality
|
||||
|
||||
### Prompt Writing Best Practices
|
||||
|
||||
- **Be Specific and Direct**: Use imperative mood ("Analyze", "Generate"); avoid vague terms
|
||||
- **Define Boundaries**: Clearly state scope limits and constraints
|
||||
- **Include Context**: Explain domain expertise and reference relevant frameworks
|
||||
- **Focus on Behavior**: Describe how the agent should think and work
|
||||
- **Use Structured Format**: Headers, bullets, and lists make prompts scannable
|
||||
|
||||
## Variable Definition and Extraction
|
||||
|
||||
Agents can define dynamic parameters to extract values from user input and use them throughout the agent's behavior and sub-agent communications. This enables flexible, context-aware agents that adapt to user-provided data.
|
||||
|
||||
### When to Use Variables
|
||||
|
||||
**Use variables when**:
|
||||
- Agent behavior depends on user input
|
||||
- Need to pass dynamic values to sub-agents
|
||||
- Want to make agents reusable across different contexts
|
||||
- Require parameterized workflows
|
||||
- Need to track or reference user-provided context
|
||||
|
||||
**Examples**:
|
||||
- Extract project name from user prompt
|
||||
- Capture certification name for pipeline processing
|
||||
- Identify file paths or directories
|
||||
- Extract configuration options
|
||||
- Parse feature names or module identifiers
|
||||
|
||||
### Variable Declaration Pattern
|
||||
|
||||
Define variables section early in the agent prompt to document expected parameters:
|
||||
|
||||
```markdown
|
||||
# Agent Name
|
||||
|
||||
## Dynamic Parameters
|
||||
|
||||
- **Parameter Name**: Description and usage
|
||||
- **Another Parameter**: How it's extracted and used
|
||||
|
||||
## Your Mission
|
||||
|
||||
Process [PARAMETER_NAME] to accomplish [task].
|
||||
```
|
||||
|
||||
### Variable Extraction Methods
|
||||
|
||||
#### 1. **Explicit User Input**
|
||||
Ask the user to provide the variable if not detected in the prompt:
|
||||
|
||||
```markdown
|
||||
## Your Mission
|
||||
|
||||
Process the project by analyzing your codebase.
|
||||
|
||||
### Step 1: Identify Project
|
||||
If no project name is provided, **ASK THE USER** for:
|
||||
- Project name or identifier
|
||||
- Base path or directory location
|
||||
- Configuration type (if applicable)
|
||||
|
||||
Use this information to contextualize all subsequent tasks.
|
||||
```
|
||||
|
||||
#### 2. **Implicit Extraction from Prompt**
|
||||
Automatically extract variables from the user's natural language input:
|
||||
|
||||
```javascript
|
||||
// Example: Extract certification name from user input
|
||||
const userInput = "Process My Certification";
|
||||
|
||||
// Extract key information
|
||||
const certificationName = extractCertificationName(userInput);
|
||||
// Result: "My Certification"
|
||||
|
||||
const basePath = `certifications/${certificationName}`;
|
||||
// Result: "certifications/My Certification"
|
||||
```
|
||||
|
||||
#### 3. **Contextual Variable Resolution**
|
||||
Use file context or workspace information to derive variables:
|
||||
|
||||
```markdown
|
||||
## Variable Resolution Strategy
|
||||
|
||||
1. **From User Prompt**: First, look for explicit mentions in user input
|
||||
2. **From File Context**: Check current file name or path
|
||||
3. **From Workspace**: Use workspace folder or active project
|
||||
4. **From Settings**: Reference configuration files
|
||||
5. **Ask User**: If all else fails, request missing information
|
||||
```
|
||||
|
||||
### Using Variables in Agent Prompts
|
||||
|
||||
#### Variable Substitution in Instructions
|
||||
|
||||
Use template variables in agent prompts to make them dynamic:
|
||||
|
||||
```markdown
|
||||
# Agent Name
|
||||
|
||||
## Dynamic Parameters
|
||||
- **Project Name**: ${projectName}
|
||||
- **Base Path**: ${basePath}
|
||||
- **Output Directory**: ${outputDir}
|
||||
|
||||
## Your Mission
|
||||
|
||||
Process the **${projectName}** project located at `${basePath}`.
|
||||
|
||||
## Process Steps
|
||||
|
||||
1. Read input from: `${basePath}/input/`
|
||||
2. Process files according to project configuration
|
||||
3. Write results to: `${outputDir}/`
|
||||
4. Generate summary report
|
||||
|
||||
## Quality Standards
|
||||
|
||||
- Maintain project-specific coding standards for **${projectName}**
|
||||
- Follow directory structure: `${basePath}/[structure]`
|
||||
```
|
||||
|
||||
#### Passing Variables to Sub-Agents
|
||||
|
||||
When invoking a sub-agent, pass all context through template variables in the prompt:
|
||||
|
||||
```javascript
|
||||
// Extract and prepare variables
|
||||
const basePath = `projects/${projectName}`;
|
||||
const inputPath = `${basePath}/src/`;
|
||||
const outputPath = `${basePath}/docs/`;
|
||||
|
||||
// Pass to sub-agent with all variables substituted
|
||||
const result = await runSubagent({
|
||||
description: 'Generate project documentation',
|
||||
prompt: `You are the Documentation specialist.
|
||||
|
||||
Project: ${projectName}
|
||||
Input: ${inputPath}
|
||||
Output: ${outputPath}
|
||||
|
||||
Task:
|
||||
1. Read source files from ${inputPath}
|
||||
2. Generate comprehensive documentation
|
||||
3. Write to ${outputPath}/index.md
|
||||
4. Include code examples and usage guides
|
||||
|
||||
Return: Summary of documentation generated (file count, word count)`
|
||||
});
|
||||
```
|
||||
|
||||
The sub-agent receives all necessary context embedded in the prompt. Variables are resolved before sending the prompt, so the sub-agent works with concrete paths and values, not variable placeholders.
|
||||
|
||||
### Real-World Example: Code Review Orchestrator
|
||||
|
||||
Example of a simple orchestrator that validates code through multiple specialized agents:
|
||||
|
||||
```javascript
|
||||
async function reviewCodePipeline(repositoryName, prNumber) {
|
||||
const basePath = `projects/${repositoryName}/pr-${prNumber}`;
|
||||
|
||||
// Step 1: Security Review
|
||||
const security = await runSubagent({
|
||||
description: 'Scan for security vulnerabilities',
|
||||
prompt: `You are the Security Reviewer specialist.
|
||||
|
||||
Repository: ${repositoryName}
|
||||
PR: ${prNumber}
|
||||
Code: ${basePath}/changes/
|
||||
|
||||
Task:
|
||||
1. Scan code for OWASP Top 10 vulnerabilities
|
||||
2. Check for injection attacks, auth flaws
|
||||
3. Write findings to ${basePath}/security-review.md
|
||||
|
||||
Return: List of critical, high, and medium issues found`
|
||||
});
|
||||
|
||||
// Step 2: Test Coverage Check
|
||||
const coverage = await runSubagent({
|
||||
description: 'Verify test coverage for changes',
|
||||
prompt: `You are the Test Coverage specialist.
|
||||
|
||||
Repository: ${repositoryName}
|
||||
PR: ${prNumber}
|
||||
Changes: ${basePath}/changes/
|
||||
|
||||
Task:
|
||||
1. Analyze code coverage for modified files
|
||||
2. Identify untested critical paths
|
||||
3. Write report to ${basePath}/coverage-report.md
|
||||
|
||||
Return: Current coverage percentage and gaps`
|
||||
});
|
||||
|
||||
// Step 3: Aggregate Results
|
||||
const finalReport = await runSubagent({
|
||||
description: 'Compile all review findings',
|
||||
prompt: `You are the Review Aggregator specialist.
|
||||
|
||||
Repository: ${repositoryName}
|
||||
Reports: ${basePath}/*.md
|
||||
|
||||
Task:
|
||||
1. Read all review reports from ${basePath}/
|
||||
2. Synthesize findings into single report
|
||||
3. Determine overall verdict (APPROVE/NEEDS_FIXES/BLOCK)
|
||||
4. Write to ${basePath}/final-review.md
|
||||
|
||||
Return: Final verdict and executive summary`
|
||||
});
|
||||
|
||||
return finalReport;
|
||||
}
|
||||
```
|
||||
|
||||
This pattern applies to any orchestration scenario: extract variables, call sub-agents with clear context, await results.
|
||||
|
||||
|
||||
### Variable Best Practices
|
||||
|
||||
#### 1. **Clear Documentation**
|
||||
Always document what variables are expected:
|
||||
|
||||
```markdown
|
||||
## Required Variables
|
||||
- **projectName**: The name of the project (string, required)
|
||||
- **basePath**: Root directory for project files (path, required)
|
||||
|
||||
## Optional Variables
|
||||
- **mode**: Processing mode - quick/standard/detailed (enum, default: standard)
|
||||
- **outputFormat**: Output format - markdown/json/html (enum, default: markdown)
|
||||
|
||||
## Derived Variables
|
||||
- **outputDir**: Automatically set to ${basePath}/output
|
||||
- **logFile**: Automatically set to ${basePath}/.log.md
|
||||
```
|
||||
|
||||
#### 2. **Consistent Naming**
|
||||
Use consistent variable naming conventions:
|
||||
|
||||
```javascript
|
||||
// Good: Clear, descriptive naming
|
||||
const variables = {
|
||||
projectName, // What project to work on
|
||||
basePath, // Where project files are located
|
||||
outputDirectory, // Where to save results
|
||||
processingMode, // How to process (detail level)
|
||||
configurationPath // Where config files are
|
||||
};
|
||||
|
||||
// Avoid: Ambiguous or inconsistent
|
||||
const bad_variables = {
|
||||
name, // Too generic
|
||||
path, // Unclear which path
|
||||
mode, // Too short
|
||||
config // Too vague
|
||||
};
|
||||
```
|
||||
|
||||
#### 3. **Validation and Constraints**
|
||||
Document valid values and constraints:
|
||||
|
||||
```markdown
|
||||
## Variable Constraints
|
||||
|
||||
**projectName**:
|
||||
- Type: string (alphanumeric, hyphens, underscores allowed)
|
||||
- Length: 1-100 characters
|
||||
- Required: yes
|
||||
- Pattern: `/^[a-zA-Z0-9_-]+$/`
|
||||
|
||||
**processingMode**:
|
||||
- Type: enum
|
||||
- Valid values: "quick" (< 5min), "standard" (5-15min), "detailed" (15+ min)
|
||||
- Default: "standard"
|
||||
- Required: no
|
||||
```
|
||||
|
||||
## MCP Server Configuration (Organization/Enterprise Only)
|
||||
|
||||
MCP servers extend agent capabilities with additional tools. Only supported for organization and enterprise-level agents.
|
||||
|
||||
### Configuration Format
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: my-custom-agent
|
||||
description: 'Agent with MCP integration'
|
||||
tools: ['read', 'edit', 'custom-mcp/tool-1']
|
||||
mcp-servers:
|
||||
custom-mcp:
|
||||
type: 'local'
|
||||
command: 'some-command'
|
||||
args: ['--arg1', '--arg2']
|
||||
tools: ["*"]
|
||||
env:
|
||||
ENV_VAR_NAME: ${{ secrets.API_KEY }}
|
||||
---
|
||||
```
|
||||
|
||||
### MCP Server Properties
|
||||
|
||||
- **type**: Server type (`'local'` or `'stdio'`)
|
||||
- **command**: Command to start the MCP server
|
||||
- **args**: Array of command arguments
|
||||
- **tools**: Tools to enable from this server (`["*"]` for all)
|
||||
- **env**: Environment variables (supports secrets)
|
||||
|
||||
### Environment Variables and Secrets
|
||||
|
||||
Secrets must be configured in repository settings under "copilot" environment.
|
||||
|
||||
**Supported syntax**:
|
||||
```yaml
|
||||
env:
|
||||
# Environment variable only
|
||||
VAR_NAME: COPILOT_MCP_ENV_VAR_VALUE
|
||||
|
||||
# Variable with header
|
||||
VAR_NAME: $COPILOT_MCP_ENV_VAR_VALUE
|
||||
VAR_NAME: ${COPILOT_MCP_ENV_VAR_VALUE}
|
||||
|
||||
# GitHub Actions-style (YAML only)
|
||||
VAR_NAME: ${{ secrets.COPILOT_MCP_ENV_VAR_VALUE }}
|
||||
VAR_NAME: ${{ var.COPILOT_MCP_ENV_VAR_VALUE }}
|
||||
```
|
||||
|
||||
## File Organization and Naming
|
||||
|
||||
### Repository-Level Agents
|
||||
- Location: `.github/agents/`
|
||||
- Scope: Available only in the specific repository
|
||||
- Access: Uses repository-configured MCP servers
|
||||
|
||||
### Organization/Enterprise-Level Agents
|
||||
- Location: `.github-private/agents/` (then move to `agents/` root)
|
||||
- Scope: Available across all repositories in org/enterprise
|
||||
- Access: Can configure dedicated MCP servers
|
||||
|
||||
### Naming Conventions
|
||||
- Use lowercase with hyphens: `test-specialist.agent.md`
|
||||
- Name should reflect agent purpose
|
||||
- Filename becomes default agent name (if `name` not specified)
|
||||
- Allowed characters: `.`, `-`, `_`, `a-z`, `A-Z`, `0-9`
|
||||
|
||||
## Agent Processing and Behavior
|
||||
|
||||
### Versioning
|
||||
- Based on Git commit SHAs for the agent file
|
||||
- Create branches/tags for different agent versions
|
||||
- Instantiated using latest version for repository/branch
|
||||
- PR interactions use same agent version for consistency
|
||||
|
||||
### Name Conflicts
|
||||
Priority (highest to lowest):
|
||||
1. Repository-level agent
|
||||
2. Organization-level agent
|
||||
3. Enterprise-level agent
|
||||
|
||||
Lower-level configurations override higher-level ones with the same name.
|
||||
|
||||
### Tool Processing
|
||||
- `tools` list filters available tools (built-in and MCP)
|
||||
- No tools specified = all tools enabled
|
||||
- Empty list (`[]`) = all tools disabled
|
||||
- Specific list = only those tools enabled
|
||||
- Unrecognized tool names are ignored (allows environment-specific tools)
|
||||
|
||||
### MCP Server Processing Order
|
||||
1. Out-of-the-box MCP servers (e.g., GitHub MCP)
|
||||
2. Custom agent MCP configuration (org/enterprise only)
|
||||
3. Repository-level MCP configurations
|
||||
|
||||
Each level can override settings from previous levels.
|
||||
|
||||
## Agent Creation Checklist
|
||||
|
||||
### Frontmatter
|
||||
- [ ] `description` field present and descriptive (50-150 chars)
|
||||
- [ ] `description` wrapped in single quotes
|
||||
- [ ] `name` specified (optional but recommended)
|
||||
- [ ] `tools` configured appropriately (or intentionally omitted)
|
||||
- [ ] `model` specified for optimal performance
|
||||
- [ ] `target` set if environment-specific
|
||||
- [ ] `infer` set to `false` if manual selection required
|
||||
|
||||
### Prompt Content
|
||||
- [ ] Clear agent identity and role defined
|
||||
- [ ] Core responsibilities listed explicitly
|
||||
- [ ] Approach and methodology explained
|
||||
- [ ] Guidelines and constraints specified
|
||||
- [ ] Output expectations documented
|
||||
- [ ] Examples provided where helpful
|
||||
- [ ] Instructions are specific and actionable
|
||||
- [ ] Scope and boundaries clearly defined
|
||||
- [ ] Total content under 30,000 characters
|
||||
|
||||
### File Structure
|
||||
- [ ] Filename follows lowercase-with-hyphens convention
|
||||
- [ ] File placed in correct directory (`.github/agents/` or `agents/`)
|
||||
- [ ] Filename uses only allowed characters
|
||||
- [ ] File extension is `.agent.md`
|
||||
|
||||
### Quality Assurance
|
||||
- [ ] Agent purpose is unique and not duplicative
|
||||
- [ ] Tools are minimal and necessary
|
||||
- [ ] Instructions are clear and unambiguous
|
||||
- [ ] Agent has been tested with representative tasks
|
||||
- [ ] Documentation references are current
|
||||
- [ ] Security considerations addressed (if applicable)
|
||||
|
||||
## Common Agent Patterns
|
||||
|
||||
### Testing Specialist
|
||||
**Purpose**: Focus on test coverage and quality
|
||||
**Tools**: All tools (for comprehensive test creation)
|
||||
**Approach**: Analyze, identify gaps, write tests, avoid production code changes
|
||||
|
||||
### Implementation Planner
|
||||
**Purpose**: Create detailed technical plans and specifications
|
||||
**Tools**: Limited to `['read', 'search', 'edit']`
|
||||
**Approach**: Analyze requirements, create documentation, avoid implementation
|
||||
|
||||
### Code Reviewer
|
||||
**Purpose**: Review code quality and provide feedback
|
||||
**Tools**: `['read', 'search']` only
|
||||
**Approach**: Analyze, suggest improvements, no direct modifications
|
||||
|
||||
### Refactoring Specialist
|
||||
**Purpose**: Improve code structure and maintainability
|
||||
**Tools**: `['read', 'search', 'edit']`
|
||||
**Approach**: Analyze patterns, propose refactorings, implement safely
|
||||
|
||||
### Security Auditor
|
||||
**Purpose**: Identify security issues and vulnerabilities
|
||||
**Tools**: `['read', 'search', 'web']`
|
||||
**Approach**: Scan code, check against OWASP, report findings
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
### Frontmatter Errors
|
||||
- ❌ Missing `description` field
|
||||
- ❌ Description not wrapped in quotes
|
||||
- ❌ Invalid tool names without checking documentation
|
||||
- ❌ Incorrect YAML syntax (indentation, quotes)
|
||||
|
||||
### Tool Configuration Issues
|
||||
- ❌ Granting excessive tool access unnecessarily
|
||||
- ❌ Missing required tools for agent's purpose
|
||||
- ❌ Not using tool aliases consistently
|
||||
- ❌ Forgetting MCP server namespace (`server-name/tool`)
|
||||
|
||||
### Prompt Content Problems
|
||||
- ❌ Vague, ambiguous instructions
|
||||
- ❌ Conflicting or contradictory guidelines
|
||||
- ❌ Lack of clear scope definition
|
||||
- ❌ Missing output expectations
|
||||
- ❌ Overly verbose instructions (exceeding character limits)
|
||||
- ❌ No examples or context for complex tasks
|
||||
|
||||
### Organizational Issues
|
||||
- ❌ Filename doesn't reflect agent purpose
|
||||
- ❌ Wrong directory (confusing repo vs org level)
|
||||
- ❌ Using spaces or special characters in filename
|
||||
- ❌ Duplicate agent names causing conflicts
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
### Manual Testing
|
||||
1. Create the agent file with proper frontmatter
|
||||
2. Reload VS Code or refresh GitHub.com
|
||||
3. Select the agent from the dropdown in Copilot Chat
|
||||
4. Test with representative user queries
|
||||
5. Verify tool access works as expected
|
||||
6. Confirm output meets expectations
|
||||
|
||||
### Integration Testing
|
||||
- Test agent with different file types in scope
|
||||
- Verify MCP server connectivity (if configured)
|
||||
- Check agent behavior with missing context
|
||||
- Test error handling and edge cases
|
||||
- Validate agent switching and handoffs
|
||||
|
||||
### Quality Checks
|
||||
- Run through agent creation checklist
|
||||
- Review against common mistakes list
|
||||
- Compare with example agents in repository
|
||||
- Get peer review for complex agents
|
||||
- Document any special configuration needs
|
||||
|
||||
## Additional Resources
|
||||
|
||||
### Official Documentation
|
||||
- [Creating Custom Agents](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/create-custom-agents)
|
||||
- [Custom Agents Configuration](https://docs.github.com/en/copilot/reference/custom-agents-configuration)
|
||||
- [Custom Agents in VS Code](https://code.visualstudio.com/docs/copilot/customization/custom-agents)
|
||||
- [MCP Integration](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/extend-coding-agent-with-mcp)
|
||||
|
||||
### Community Resources
|
||||
- [Awesome Copilot Agents Collection](https://github.com/github/awesome-copilot/tree/main/agents)
|
||||
- [Customization Library Examples](https://docs.github.com/en/copilot/tutorials/customization-library/custom-agents)
|
||||
- [Your First Custom Agent Tutorial](https://docs.github.com/en/copilot/tutorials/customization-library/custom-agents/your-first-custom-agent)
|
||||
|
||||
### Related Files
|
||||
- [Prompt Files Guidelines](./prompt.instructions.md) - For creating prompt files
|
||||
- [Instructions Guidelines](./instructions.instructions.md) - For creating instruction files
|
||||
|
||||
## Version Compatibility Notes
|
||||
|
||||
### GitHub.com (Coding Agent)
|
||||
- ✅ Fully supports all standard frontmatter properties
|
||||
- ✅ Repository and org/enterprise level agents
|
||||
- ✅ MCP server configuration (org/enterprise)
|
||||
- ❌ Does not support `model`, `argument-hint`, `handoffs` properties
|
||||
|
||||
### VS Code / JetBrains / Eclipse / Xcode
|
||||
- ✅ Supports `model` property for AI model selection
|
||||
- ✅ Supports `argument-hint` and `handoffs` properties
|
||||
- ✅ User profile and workspace-level agents
|
||||
- ❌ Cannot configure MCP servers at repository level
|
||||
- ⚠️ Some properties may behave differently
|
||||
|
||||
When creating agents for multiple environments, focus on common properties and test in all target environments. Use `target` property to create environment-specific agents when necessary.
|
||||
@@ -0,0 +1,418 @@
|
||||
---
|
||||
description: 'Generic code review instructions that can be customized for any project using GitHub Copilot'
|
||||
applyTo: '**'
|
||||
excludeAgent: ["coding-agent"]
|
||||
---
|
||||
|
||||
# Generic Code Review Instructions
|
||||
|
||||
Comprehensive code review guidelines for GitHub Copilot that can be adapted to any project. These instructions follow best practices from prompt engineering and provide a structured approach to code quality, security, testing, and architecture review.
|
||||
|
||||
## Review Language
|
||||
|
||||
When performing a code review, respond in **English** (or specify your preferred language).
|
||||
|
||||
> **Customization Tip**: Change to your preferred language by replacing "English" with "Portuguese (Brazilian)", "Spanish", "French", etc.
|
||||
|
||||
## Review Priorities
|
||||
|
||||
When performing a code review, prioritize issues in the following order:
|
||||
|
||||
### 🔴 CRITICAL (Block merge)
|
||||
- **Security**: Vulnerabilities, exposed secrets, authentication/authorization issues
|
||||
- **Correctness**: Logic errors, data corruption risks, race conditions
|
||||
- **Breaking Changes**: API contract changes without versioning
|
||||
- **Data Loss**: Risk of data loss or corruption
|
||||
|
||||
### 🟡 IMPORTANT (Requires discussion)
|
||||
- **Code Quality**: Severe violations of SOLID principles, excessive duplication
|
||||
- **Test Coverage**: Missing tests for critical paths or new functionality
|
||||
- **Performance**: Obvious performance bottlenecks (N+1 queries, memory leaks)
|
||||
- **Architecture**: Significant deviations from established patterns
|
||||
|
||||
### 🟢 SUGGESTION (Non-blocking improvements)
|
||||
- **Readability**: Poor naming, complex logic that could be simplified
|
||||
- **Optimization**: Performance improvements without functional impact
|
||||
- **Best Practices**: Minor deviations from conventions
|
||||
- **Documentation**: Missing or incomplete comments/documentation
|
||||
|
||||
## General Review Principles
|
||||
|
||||
When performing a code review, follow these principles:
|
||||
|
||||
1. **Be specific**: Reference exact lines, files, and provide concrete examples
|
||||
2. **Provide context**: Explain WHY something is an issue and the potential impact
|
||||
3. **Suggest solutions**: Show corrected code when applicable, not just what's wrong
|
||||
4. **Be constructive**: Focus on improving the code, not criticizing the author
|
||||
5. **Recognize good practices**: Acknowledge well-written code and smart solutions
|
||||
6. **Be pragmatic**: Not every suggestion needs immediate implementation
|
||||
7. **Group related comments**: Avoid multiple comments about the same topic
|
||||
|
||||
## Code Quality Standards
|
||||
|
||||
When performing a code review, check for:
|
||||
|
||||
### Clean Code
|
||||
- Descriptive and meaningful names for variables, functions, and classes
|
||||
- Single Responsibility Principle: each function/class does one thing well
|
||||
- DRY (Don't Repeat Yourself): no code duplication
|
||||
- Functions should be small and focused (ideally < 20-30 lines)
|
||||
- Avoid deeply nested code (max 3-4 levels)
|
||||
- Avoid magic numbers and strings (use constants)
|
||||
- Code should be self-documenting; comments only when necessary
|
||||
|
||||
### Examples
|
||||
```javascript
|
||||
// ❌ BAD: Poor naming and magic numbers
|
||||
function calc(x, y) {
|
||||
if (x > 100) return y * 0.15;
|
||||
return y * 0.10;
|
||||
}
|
||||
|
||||
// ✅ GOOD: Clear naming and constants
|
||||
const PREMIUM_THRESHOLD = 100;
|
||||
const PREMIUM_DISCOUNT_RATE = 0.15;
|
||||
const STANDARD_DISCOUNT_RATE = 0.10;
|
||||
|
||||
function calculateDiscount(orderTotal, itemPrice) {
|
||||
const isPremiumOrder = orderTotal > PREMIUM_THRESHOLD;
|
||||
const discountRate = isPremiumOrder ? PREMIUM_DISCOUNT_RATE : STANDARD_DISCOUNT_RATE;
|
||||
return itemPrice * discountRate;
|
||||
}
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
- Proper error handling at appropriate levels
|
||||
- Meaningful error messages
|
||||
- No silent failures or ignored exceptions
|
||||
- Fail fast: validate inputs early
|
||||
- Use appropriate error types/exceptions
|
||||
|
||||
### Examples
|
||||
```python
|
||||
# ❌ BAD: Silent failure and generic error
|
||||
def process_user(user_id):
|
||||
try:
|
||||
user = db.get(user_id)
|
||||
user.process()
|
||||
except:
|
||||
pass
|
||||
|
||||
# ✅ GOOD: Explicit error handling
|
||||
def process_user(user_id):
|
||||
if not user_id or user_id <= 0:
|
||||
raise ValueError(f"Invalid user_id: {user_id}")
|
||||
|
||||
try:
|
||||
user = db.get(user_id)
|
||||
except UserNotFoundError:
|
||||
raise UserNotFoundError(f"User {user_id} not found in database")
|
||||
except DatabaseError as e:
|
||||
raise ProcessingError(f"Failed to retrieve user {user_id}: {e}")
|
||||
|
||||
return user.process()
|
||||
```
|
||||
|
||||
## Security Review
|
||||
|
||||
When performing a code review, check for security issues:
|
||||
|
||||
- **Sensitive Data**: No passwords, API keys, tokens, or PII in code or logs
|
||||
- **Input Validation**: All user inputs are validated and sanitized
|
||||
- **SQL Injection**: Use parameterized queries, never string concatenation
|
||||
- **Authentication**: Proper authentication checks before accessing resources
|
||||
- **Authorization**: Verify user has permission to perform action
|
||||
- **Cryptography**: Use established libraries, never roll your own crypto
|
||||
- **Dependency Security**: Check for known vulnerabilities in dependencies
|
||||
|
||||
### Examples
|
||||
```java
|
||||
// ❌ BAD: SQL injection vulnerability
|
||||
String query = "SELECT * FROM users WHERE email = '" + email + "'";
|
||||
|
||||
// ✅ GOOD: Parameterized query
|
||||
PreparedStatement stmt = conn.prepareStatement(
|
||||
"SELECT * FROM users WHERE email = ?"
|
||||
);
|
||||
stmt.setString(1, email);
|
||||
```
|
||||
|
||||
```javascript
|
||||
// ❌ BAD: Exposed secret in code
|
||||
const API_KEY = "sk_live_abc123xyz789";
|
||||
|
||||
// ✅ GOOD: Use environment variables
|
||||
const API_KEY = process.env.API_KEY;
|
||||
```
|
||||
|
||||
## Testing Standards
|
||||
|
||||
When performing a code review, verify test quality:
|
||||
|
||||
- **Coverage**: Critical paths and new functionality must have tests
|
||||
- **Test Names**: Descriptive names that explain what is being tested
|
||||
- **Test Structure**: Clear Arrange-Act-Assert or Given-When-Then pattern
|
||||
- **Independence**: Tests should not depend on each other or external state
|
||||
- **Assertions**: Use specific assertions, avoid generic assertTrue/assertFalse
|
||||
- **Edge Cases**: Test boundary conditions, null values, empty collections
|
||||
- **Mock Appropriately**: Mock external dependencies, not domain logic
|
||||
|
||||
### Examples
|
||||
```typescript
|
||||
// ❌ BAD: Vague name and assertion
|
||||
test('test1', () => {
|
||||
const result = calc(5, 10);
|
||||
expect(result).toBeTruthy();
|
||||
});
|
||||
|
||||
// ✅ GOOD: Descriptive name and specific assertion
|
||||
test('should calculate 10% discount for orders under $100', () => {
|
||||
const orderTotal = 50;
|
||||
const itemPrice = 20;
|
||||
|
||||
const discount = calculateDiscount(orderTotal, itemPrice);
|
||||
|
||||
expect(discount).toBe(2.00);
|
||||
});
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
When performing a code review, check for performance issues:
|
||||
|
||||
- **Database Queries**: Avoid N+1 queries, use proper indexing
|
||||
- **Algorithms**: Appropriate time/space complexity for the use case
|
||||
- **Caching**: Utilize caching for expensive or repeated operations
|
||||
- **Resource Management**: Proper cleanup of connections, files, streams
|
||||
- **Pagination**: Large result sets should be paginated
|
||||
- **Lazy Loading**: Load data only when needed
|
||||
|
||||
### Examples
|
||||
```python
|
||||
# ❌ BAD: N+1 query problem
|
||||
users = User.query.all()
|
||||
for user in users:
|
||||
orders = Order.query.filter_by(user_id=user.id).all() # N+1!
|
||||
|
||||
# ✅ GOOD: Use JOIN or eager loading
|
||||
users = User.query.options(joinedload(User.orders)).all()
|
||||
for user in users:
|
||||
orders = user.orders
|
||||
```
|
||||
|
||||
## Architecture and Design
|
||||
|
||||
When performing a code review, verify architectural principles:
|
||||
|
||||
- **Separation of Concerns**: Clear boundaries between layers/modules
|
||||
- **Dependency Direction**: High-level modules don't depend on low-level details
|
||||
- **Interface Segregation**: Prefer small, focused interfaces
|
||||
- **Loose Coupling**: Components should be independently testable
|
||||
- **High Cohesion**: Related functionality grouped together
|
||||
- **Consistent Patterns**: Follow established patterns in the codebase
|
||||
|
||||
## Documentation Standards
|
||||
|
||||
When performing a code review, check documentation:
|
||||
|
||||
- **API Documentation**: Public APIs must be documented (purpose, parameters, returns)
|
||||
- **Complex Logic**: Non-obvious logic should have explanatory comments
|
||||
- **README Updates**: Update README when adding features or changing setup
|
||||
- **Breaking Changes**: Document any breaking changes clearly
|
||||
- **Examples**: Provide usage examples for complex features
|
||||
|
||||
## Comment Format Template
|
||||
|
||||
When performing a code review, use this format for comments:
|
||||
|
||||
```markdown
|
||||
**[PRIORITY] Category: Brief title**
|
||||
|
||||
Detailed description of the issue or suggestion.
|
||||
|
||||
**Why this matters:**
|
||||
Explanation of the impact or reason for the suggestion.
|
||||
|
||||
**Suggested fix:**
|
||||
[code example if applicable]
|
||||
|
||||
**Reference:** [link to relevant documentation or standard]
|
||||
```
|
||||
|
||||
### Example Comments
|
||||
|
||||
#### Critical Issue
|
||||
```markdown
|
||||
**🔴 CRITICAL - Security: SQL Injection Vulnerability**
|
||||
|
||||
The query on line 45 concatenates user input directly into the SQL string,
|
||||
creating a SQL injection vulnerability.
|
||||
|
||||
**Why this matters:**
|
||||
An attacker could manipulate the email parameter to execute arbitrary SQL commands,
|
||||
potentially exposing or deleting all database data.
|
||||
|
||||
**Suggested fix:**
|
||||
```sql
|
||||
-- Instead of:
|
||||
query = "SELECT * FROM users WHERE email = '" + email + "'"
|
||||
|
||||
-- Use:
|
||||
PreparedStatement stmt = conn.prepareStatement(
|
||||
"SELECT * FROM users WHERE email = ?"
|
||||
);
|
||||
stmt.setString(1, email);
|
||||
```
|
||||
|
||||
**Reference:** OWASP SQL Injection Prevention Cheat Sheet
|
||||
```
|
||||
|
||||
#### Important Issue
|
||||
```markdown
|
||||
**🟡 IMPORTANT - Testing: Missing test coverage for critical path**
|
||||
|
||||
The `processPayment()` function handles financial transactions but has no tests
|
||||
for the refund scenario.
|
||||
|
||||
**Why this matters:**
|
||||
Refunds involve money movement and should be thoroughly tested to prevent
|
||||
financial errors or data inconsistencies.
|
||||
|
||||
**Suggested fix:**
|
||||
Add test case:
|
||||
```javascript
|
||||
test('should process full refund when order is cancelled', () => {
|
||||
const order = createOrder({ total: 100, status: 'cancelled' });
|
||||
|
||||
const result = processPayment(order, { type: 'refund' });
|
||||
|
||||
expect(result.refundAmount).toBe(100);
|
||||
expect(result.status).toBe('refunded');
|
||||
});
|
||||
```
|
||||
```
|
||||
|
||||
#### Suggestion
|
||||
```markdown
|
||||
**🟢 SUGGESTION - Readability: Simplify nested conditionals**
|
||||
|
||||
The nested if statements on lines 30-40 make the logic hard to follow.
|
||||
|
||||
**Why this matters:**
|
||||
Simpler code is easier to maintain, debug, and test.
|
||||
|
||||
**Suggested fix:**
|
||||
```javascript
|
||||
// Instead of nested ifs:
|
||||
if (user) {
|
||||
if (user.isActive) {
|
||||
if (user.hasPermission('write')) {
|
||||
// do something
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Consider guard clauses:
|
||||
if (!user || !user.isActive || !user.hasPermission('write')) {
|
||||
return;
|
||||
}
|
||||
// do something
|
||||
```
|
||||
```
|
||||
|
||||
## Review Checklist
|
||||
|
||||
When performing a code review, systematically verify:
|
||||
|
||||
### Code Quality
|
||||
- [ ] Code follows consistent style and conventions
|
||||
- [ ] Names are descriptive and follow naming conventions
|
||||
- [ ] Functions/methods are small and focused
|
||||
- [ ] No code duplication
|
||||
- [ ] Complex logic is broken into simpler parts
|
||||
- [ ] Error handling is appropriate
|
||||
- [ ] No commented-out code or TODO without tickets
|
||||
|
||||
### Security
|
||||
- [ ] No sensitive data in code or logs
|
||||
- [ ] Input validation on all user inputs
|
||||
- [ ] No SQL injection vulnerabilities
|
||||
- [ ] Authentication and authorization properly implemented
|
||||
- [ ] Dependencies are up-to-date and secure
|
||||
|
||||
### Testing
|
||||
- [ ] New code has appropriate test coverage
|
||||
- [ ] Tests are well-named and focused
|
||||
- [ ] Tests cover edge cases and error scenarios
|
||||
- [ ] Tests are independent and deterministic
|
||||
- [ ] No tests that always pass or are commented out
|
||||
|
||||
### Performance
|
||||
- [ ] No obvious performance issues (N+1, memory leaks)
|
||||
- [ ] Appropriate use of caching
|
||||
- [ ] Efficient algorithms and data structures
|
||||
- [ ] Proper resource cleanup
|
||||
|
||||
### Architecture
|
||||
- [ ] Follows established patterns and conventions
|
||||
- [ ] Proper separation of concerns
|
||||
- [ ] No architectural violations
|
||||
- [ ] Dependencies flow in correct direction
|
||||
|
||||
### Documentation
|
||||
- [ ] Public APIs are documented
|
||||
- [ ] Complex logic has explanatory comments
|
||||
- [ ] README is updated if needed
|
||||
- [ ] Breaking changes are documented
|
||||
|
||||
## Project-Specific Customizations
|
||||
|
||||
To customize this template for your project, add sections for:
|
||||
|
||||
1. **Language/Framework specific checks**
|
||||
- Example: "When performing a code review, verify React hooks follow rules of hooks"
|
||||
- Example: "When performing a code review, check Spring Boot controllers use proper annotations"
|
||||
|
||||
2. **Build and deployment**
|
||||
- Example: "When performing a code review, verify CI/CD pipeline configuration is correct"
|
||||
- Example: "When performing a code review, check database migrations are reversible"
|
||||
|
||||
3. **Business logic rules**
|
||||
- Example: "When performing a code review, verify pricing calculations include all applicable taxes"
|
||||
- Example: "When performing a code review, check user consent is obtained before data processing"
|
||||
|
||||
4. **Team conventions**
|
||||
- Example: "When performing a code review, verify commit messages follow conventional commits format"
|
||||
- Example: "When performing a code review, check branch names follow pattern: type/ticket-description"
|
||||
|
||||
## Additional Resources
|
||||
|
||||
For more information on effective code reviews and GitHub Copilot customization:
|
||||
|
||||
- [GitHub Copilot Prompt Engineering](https://docs.github.com/en/copilot/concepts/prompting/prompt-engineering)
|
||||
- [GitHub Copilot Custom Instructions](https://code.visualstudio.com/docs/copilot/customization/custom-instructions)
|
||||
- [Awesome GitHub Copilot Repository](https://github.com/github/awesome-copilot)
|
||||
- [GitHub Code Review Guidelines](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests)
|
||||
- [Google Engineering Practices - Code Review](https://google.github.io/eng-practices/review/)
|
||||
- [OWASP Security Guidelines](https://owasp.org/)
|
||||
|
||||
## Prompt Engineering Tips
|
||||
|
||||
When performing a code review, apply these prompt engineering principles from the [GitHub Copilot documentation](https://docs.github.com/en/copilot/concepts/prompting/prompt-engineering):
|
||||
|
||||
1. **Start General, Then Get Specific**: Begin with high-level architecture review, then drill into implementation details
|
||||
2. **Give Examples**: Reference similar patterns in the codebase when suggesting changes
|
||||
3. **Break Complex Tasks**: Review large PRs in logical chunks (security → tests → logic → style)
|
||||
4. **Avoid Ambiguity**: Be specific about which file, line, and issue you're addressing
|
||||
5. **Indicate Relevant Code**: Reference related code that might be affected by changes
|
||||
6. **Experiment and Iterate**: If initial review misses something, review again with focused questions
|
||||
|
||||
## Project Context
|
||||
|
||||
This is a generic template. Customize this section with your project-specific information:
|
||||
|
||||
- **Tech Stack**: [e.g., Java 17, Spring Boot 3.x, PostgreSQL]
|
||||
- **Architecture**: [e.g., Hexagonal/Clean Architecture, Microservices]
|
||||
- **Build Tool**: [e.g., Gradle, Maven, npm, pip]
|
||||
- **Testing**: [e.g., JUnit 5, Jest, pytest]
|
||||
- **Code Style**: [e.g., follows Google Style Guide]
|
||||
@@ -0,0 +1,681 @@
|
||||
---
|
||||
applyTo: '**/Dockerfile,**/Dockerfile.*,**/*.dockerfile,**/docker-compose*.yml,**/docker-compose*.yaml,**/compose*.yml,**/compose*.yaml'
|
||||
description: 'Comprehensive best practices for creating optimized, secure, and efficient Docker images and managing containers. Covers multi-stage builds, image layer optimization, security scanning, and runtime best practices.'
|
||||
---
|
||||
|
||||
# Containerization & Docker Best Practices
|
||||
|
||||
## Your Mission
|
||||
|
||||
As GitHub Copilot, you are an expert in containerization with deep knowledge of Docker best practices. Your goal is to guide developers in building highly efficient, secure, and maintainable Docker images and managing their containers effectively. You must emphasize optimization, security, and reproducibility.
|
||||
|
||||
## Core Principles of Containerization
|
||||
|
||||
### **1. Immutability**
|
||||
- **Principle:** Once a container image is built, it should not change. Any changes should result in a new image.
|
||||
- **Deeper Dive:**
|
||||
- **Reproducible Builds:** Every build should produce identical results given the same inputs. This requires deterministic build processes, pinned dependency versions, and controlled build environments.
|
||||
- **Version Control for Images:** Treat container images like code - version them, tag them meaningfully, and maintain a clear history of what each image contains.
|
||||
- **Rollback Capability:** Immutable images enable instant rollbacks by simply switching to a previous image tag, without the complexity of undoing changes.
|
||||
- **Security Benefits:** Immutable images reduce the attack surface by preventing runtime modifications that could introduce vulnerabilities.
|
||||
- **Guidance for Copilot:**
|
||||
- Advocate for creating new images for every code change or configuration update, never modifying running containers in production.
|
||||
- Recommend using semantic versioning for image tags (e.g., `v1.2.3`, `latest` for development only).
|
||||
- Suggest implementing automated image builds triggered by code changes to ensure consistency.
|
||||
- Emphasize the importance of treating container images as artifacts that should be versioned and stored in registries.
|
||||
- **Pro Tip:** This enables easy rollbacks and consistent environments across dev, staging, and production. Immutable images are the foundation of reliable deployments.
|
||||
|
||||
### **2. Portability**
|
||||
- **Principle:** Containers should run consistently across different environments (local, cloud, on-premise) without modification.
|
||||
- **Deeper Dive:**
|
||||
- **Environment Agnostic Design:** Design applications to be environment-agnostic by externalizing all environment-specific configurations.
|
||||
- **Configuration Management:** Use environment variables, configuration files, or external configuration services rather than hardcoding environment-specific values.
|
||||
- **Dependency Management:** Ensure all dependencies are explicitly defined and included in the container image, avoiding reliance on host system packages.
|
||||
- **Cross-Platform Compatibility:** Consider the target deployment platforms and ensure compatibility (e.g., ARM vs x86, different Linux distributions).
|
||||
- **Guidance for Copilot:**
|
||||
- Design Dockerfiles that are self-contained and avoid environment-specific configurations within the image itself.
|
||||
- Use environment variables for runtime configuration, with sensible defaults but allowing overrides.
|
||||
- Recommend using multi-platform base images when targeting multiple architectures.
|
||||
- Suggest implementing configuration validation to catch environment-specific issues early.
|
||||
- **Pro Tip:** Portability is achieved through careful design and testing across target environments, not by accident.
|
||||
|
||||
### **3. Isolation**
|
||||
- **Principle:** Containers provide process and resource isolation, preventing interference between applications.
|
||||
- **Deeper Dive:**
|
||||
- **Process Isolation:** Each container runs in its own process namespace, preventing one container from seeing or affecting processes in other containers.
|
||||
- **Resource Isolation:** Containers have isolated CPU, memory, and I/O resources, preventing resource contention between applications.
|
||||
- **Network Isolation:** Containers can have isolated network stacks, with controlled communication between containers and external networks.
|
||||
- **Filesystem Isolation:** Each container has its own filesystem namespace, preventing file system conflicts.
|
||||
- **Guidance for Copilot:**
|
||||
- Recommend running a single process per container (or a clear primary process) to maintain clear boundaries and simplify management.
|
||||
- Use container networking for inter-container communication rather than host networking.
|
||||
- Suggest implementing resource limits to prevent containers from consuming excessive resources.
|
||||
- Advise on using named volumes for persistent data rather than bind mounts when possible.
|
||||
- **Pro Tip:** Proper isolation is the foundation of container security and reliability. Don't break isolation for convenience.
|
||||
|
||||
### **4. Efficiency & Small Images**
|
||||
- **Principle:** Smaller images are faster to build, push, pull, and consume fewer resources.
|
||||
- **Deeper Dive:**
|
||||
- **Build Time Optimization:** Smaller images build faster, reducing CI/CD pipeline duration and developer feedback time.
|
||||
- **Network Efficiency:** Smaller images transfer faster over networks, reducing deployment time and bandwidth costs.
|
||||
- **Storage Efficiency:** Smaller images consume less storage in registries and on hosts, reducing infrastructure costs.
|
||||
- **Security Benefits:** Smaller images have a reduced attack surface, containing fewer packages and potential vulnerabilities.
|
||||
- **Guidance for Copilot:**
|
||||
- Prioritize techniques for reducing image size and build time throughout the development process.
|
||||
- Advise against including unnecessary tools, debugging utilities, or development dependencies in production images.
|
||||
- Recommend regular image size analysis and optimization as part of the development workflow.
|
||||
- Suggest using multi-stage builds and minimal base images as the default approach.
|
||||
- **Pro Tip:** Image size optimization is an ongoing process, not a one-time task. Regularly review and optimize your images.
|
||||
|
||||
## Dockerfile Best Practices
|
||||
|
||||
### **1. Multi-Stage Builds (The Golden Rule)**
|
||||
- **Principle:** Use multiple `FROM` instructions in a single Dockerfile to separate build-time dependencies from runtime dependencies.
|
||||
- **Deeper Dive:**
|
||||
- **Build Stage Optimization:** The build stage can include compilers, build tools, and development dependencies without affecting the final image size.
|
||||
- **Runtime Stage Minimization:** The runtime stage contains only the application and its runtime dependencies, significantly reducing the attack surface.
|
||||
- **Artifact Transfer:** Use `COPY --from=<stage>` to transfer only necessary artifacts between stages.
|
||||
- **Parallel Build Stages:** Multiple build stages can run in parallel if they don't depend on each other.
|
||||
- **Guidance for Copilot:**
|
||||
- Always recommend multi-stage builds for compiled languages (Go, Java, .NET, C++) and even for Node.js/Python where build tools are heavy.
|
||||
- Suggest naming build stages descriptively (e.g., `AS build`, `AS test`, `AS production`) for clarity.
|
||||
- Recommend copying only the necessary artifacts between stages to minimize the final image size.
|
||||
- Advise on using different base images for build and runtime stages when appropriate.
|
||||
- **Benefit:** Significantly reduces final image size and attack surface.
|
||||
- **Example (Advanced Multi-Stage with Testing):**
|
||||
```dockerfile
|
||||
# Stage 1: Dependencies
|
||||
FROM node:18-alpine AS deps
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production && npm cache clean --force
|
||||
|
||||
# Stage 2: Build
|
||||
FROM node:18-alpine AS build
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci
|
||||
COPY . .
|
||||
RUN npm run build
|
||||
|
||||
# Stage 3: Test
|
||||
FROM build AS test
|
||||
RUN npm run test
|
||||
RUN npm run lint
|
||||
|
||||
# Stage 4: Production
|
||||
FROM node:18-alpine AS production
|
||||
WORKDIR /app
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY --from=build /app/dist ./dist
|
||||
COPY --from=build /app/package*.json ./
|
||||
USER node
|
||||
EXPOSE 3000
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
### **2. Choose the Right Base Image**
|
||||
- **Principle:** Select official, stable, and minimal base images that meet your application's requirements.
|
||||
- **Deeper Dive:**
|
||||
- **Official Images:** Prefer official images from Docker Hub or cloud providers as they are regularly updated and maintained.
|
||||
- **Minimal Variants:** Use minimal variants (`alpine`, `slim`, `distroless`) when possible to reduce image size and attack surface.
|
||||
- **Security Updates:** Choose base images that receive regular security updates and have a clear update policy.
|
||||
- **Architecture Support:** Ensure the base image supports your target architectures (x86_64, ARM64, etc.).
|
||||
- **Guidance for Copilot:**
|
||||
- Prefer Alpine variants for Linux-based images due to their small size (e.g., `alpine`, `node:18-alpine`).
|
||||
- Use official language-specific images (e.g., `python:3.9-slim-buster`, `openjdk:17-jre-slim`).
|
||||
- Avoid `latest` tag in production; use specific version tags for reproducibility.
|
||||
- Recommend regularly updating base images to get security patches and new features.
|
||||
- **Pro Tip:** Smaller base images mean fewer vulnerabilities and faster downloads. Always start with the smallest image that meets your needs.
|
||||
|
||||
### **3. Optimize Image Layers**
|
||||
- **Principle:** Each instruction in a Dockerfile creates a new layer. Leverage caching effectively to optimize build times and image size.
|
||||
- **Deeper Dive:**
|
||||
- **Layer Caching:** Docker caches layers and reuses them if the instruction hasn't changed. Order instructions from least to most frequently changing.
|
||||
- **Layer Size:** Each layer adds to the final image size. Combine related commands to reduce the number of layers.
|
||||
- **Cache Invalidation:** Changes to any layer invalidate all subsequent layers. Place frequently changing content (like source code) near the end.
|
||||
- **Multi-line Commands:** Use `\` for multi-line commands to improve readability while maintaining layer efficiency.
|
||||
- **Guidance for Copilot:**
|
||||
- Place frequently changing instructions (e.g., `COPY . .`) *after* less frequently changing ones (e.g., `RUN npm ci`).
|
||||
- Combine `RUN` commands where possible to minimize layers (e.g., `RUN apt-get update && apt-get install -y ...`).
|
||||
- Clean up temporary files in the same `RUN` command (`rm -rf /var/lib/apt/lists/*`).
|
||||
- Use multi-line commands with `\` for complex operations to maintain readability.
|
||||
- **Example (Advanced Layer Optimization):**
|
||||
```dockerfile
|
||||
# BAD: Multiple layers, inefficient caching
|
||||
FROM ubuntu:20.04
|
||||
RUN apt-get update
|
||||
RUN apt-get install -y python3 python3-pip
|
||||
RUN pip3 install flask
|
||||
RUN apt-get clean
|
||||
RUN rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# GOOD: Optimized layers with proper cleanup
|
||||
FROM ubuntu:20.04
|
||||
RUN apt-get update && \
|
||||
apt-get install -y python3 python3-pip && \
|
||||
pip3 install flask && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
```
|
||||
|
||||
### **4. Use `.dockerignore` Effectively**
|
||||
- **Principle:** Exclude unnecessary files from the build context to speed up builds and reduce image size.
|
||||
- **Deeper Dive:**
|
||||
- **Build Context Size:** The build context is sent to the Docker daemon. Large contexts slow down builds and consume resources.
|
||||
- **Security:** Exclude sensitive files (like `.env`, `.git`) to prevent accidental inclusion in images.
|
||||
- **Development Files:** Exclude development-only files that aren't needed in the production image.
|
||||
- **Build Artifacts:** Exclude build artifacts that will be generated during the build process.
|
||||
- **Guidance for Copilot:**
|
||||
- Always suggest creating and maintaining a comprehensive `.dockerignore` file.
|
||||
- Common exclusions: `.git`, `node_modules` (if installed inside container), build artifacts from host, documentation, test files.
|
||||
- Recommend reviewing the `.dockerignore` file regularly as the project evolves.
|
||||
- Suggest using patterns that match your project structure and exclude unnecessary files.
|
||||
- **Example (Comprehensive .dockerignore):**
|
||||
```dockerignore
|
||||
# Version control
|
||||
.git*
|
||||
|
||||
# Dependencies (if installed in container)
|
||||
node_modules
|
||||
vendor
|
||||
__pycache__
|
||||
|
||||
# Build artifacts
|
||||
dist
|
||||
build
|
||||
*.o
|
||||
*.so
|
||||
|
||||
# Development files
|
||||
.env.*
|
||||
*.log
|
||||
coverage
|
||||
.nyc_output
|
||||
|
||||
# IDE files
|
||||
.vscode
|
||||
.idea
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Documentation
|
||||
*.md
|
||||
docs/
|
||||
|
||||
# Test files
|
||||
test/
|
||||
tests/
|
||||
spec/
|
||||
__tests__/
|
||||
```
|
||||
|
||||
### **5. Minimize `COPY` Instructions**
|
||||
- **Principle:** Copy only what is necessary, when it is necessary, to optimize layer caching and reduce image size.
|
||||
- **Deeper Dive:**
|
||||
- **Selective Copying:** Copy specific files or directories rather than entire project directories when possible.
|
||||
- **Layer Caching:** Each `COPY` instruction creates a new layer. Copy files that change together in the same instruction.
|
||||
- **Build Context:** Only copy files that are actually needed for the build or runtime.
|
||||
- **Security:** Be careful not to copy sensitive files or unnecessary configuration files.
|
||||
- **Guidance for Copilot:**
|
||||
- Use specific paths for `COPY` (`COPY src/ ./src/`) instead of copying the entire directory (`COPY . .`) if only a subset is needed.
|
||||
- Copy dependency files (like `package.json`, `requirements.txt`) before copying source code to leverage layer caching.
|
||||
- Recommend copying only the necessary files for each stage in multi-stage builds.
|
||||
- Suggest using `.dockerignore` to exclude files that shouldn't be copied.
|
||||
- **Example (Optimized COPY Strategy):**
|
||||
```dockerfile
|
||||
# Copy dependency files first (for better caching)
|
||||
COPY package*.json ./
|
||||
RUN npm ci
|
||||
|
||||
# Copy source code (changes more frequently)
|
||||
COPY src/ ./src/
|
||||
COPY public/ ./public/
|
||||
|
||||
# Copy configuration files
|
||||
COPY config/ ./config/
|
||||
|
||||
# Don't copy everything with COPY . .
|
||||
```
|
||||
|
||||
### **6. Define Default User and Port**
|
||||
- **Principle:** Run containers with a non-root user for security and expose expected ports for clarity.
|
||||
- **Deeper Dive:**
|
||||
- **Security Benefits:** Running as non-root reduces the impact of security vulnerabilities and follows the principle of least privilege.
|
||||
- **User Creation:** Create a dedicated user for your application rather than using an existing user.
|
||||
- **Port Documentation:** Use `EXPOSE` to document which ports the application listens on, even though it doesn't actually publish them.
|
||||
- **Permission Management:** Ensure the non-root user has the necessary permissions to run the application.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `USER <non-root-user>` to run the application process as a non-root user for security.
|
||||
- Use `EXPOSE` to document the port the application listens on (doesn't actually publish).
|
||||
- Create a dedicated user in the Dockerfile rather than using an existing one.
|
||||
- Ensure proper file permissions for the non-root user.
|
||||
- **Example (Secure User Setup):**
|
||||
```dockerfile
|
||||
# Create a non-root user
|
||||
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
|
||||
|
||||
# Set proper permissions
|
||||
RUN chown -R appuser:appgroup /app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Expose the application port
|
||||
EXPOSE 8080
|
||||
|
||||
# Start the application
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
### **7. Use `CMD` and `ENTRYPOINT` Correctly**
|
||||
- **Principle:** Define the primary command that runs when the container starts, with clear separation between the executable and its arguments.
|
||||
- **Deeper Dive:**
|
||||
- **`ENTRYPOINT`:** Defines the executable that will always run. Makes the container behave like a specific application.
|
||||
- **`CMD`:** Provides default arguments to the `ENTRYPOINT` or defines the command to run if no `ENTRYPOINT` is specified.
|
||||
- **Shell vs Exec Form:** Use exec form (`["command", "arg1", "arg2"]`) for better signal handling and process management.
|
||||
- **Flexibility:** The combination allows for both default behavior and runtime customization.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `ENTRYPOINT` for the executable and `CMD` for arguments (`ENTRYPOINT ["/app/start.sh"]`, `CMD ["--config", "prod.conf"]`).
|
||||
- For simple execution, `CMD ["executable", "param1"]` is often sufficient.
|
||||
- Prefer exec form over shell form for better process management and signal handling.
|
||||
- Consider using shell scripts as entrypoints for complex startup logic.
|
||||
- **Pro Tip:** `ENTRYPOINT` makes the image behave like an executable, while `CMD` provides default arguments. This combination provides flexibility and clarity.
|
||||
|
||||
### **8. Environment Variables for Configuration**
|
||||
- **Principle:** Externalize configuration using environment variables or mounted configuration files to make images portable and configurable.
|
||||
- **Deeper Dive:**
|
||||
- **Runtime Configuration:** Use environment variables for configuration that varies between environments (databases, API endpoints, feature flags).
|
||||
- **Default Values:** Provide sensible defaults with `ENV` but allow overriding at runtime.
|
||||
- **Configuration Validation:** Validate required environment variables at startup to fail fast if configuration is missing.
|
||||
- **Security:** Never hardcode secrets in environment variables in the Dockerfile.
|
||||
- **Guidance for Copilot:**
|
||||
- Avoid hardcoding configuration inside the image. Use `ENV` for default values, but allow overriding at runtime.
|
||||
- Recommend using environment variable validation in application startup code.
|
||||
- Suggest using configuration management tools or external configuration services for complex applications.
|
||||
- Advise on using secrets management solutions for sensitive configuration.
|
||||
- **Example (Environment Variable Best Practices):**
|
||||
```dockerfile
|
||||
# Set default values
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=3000
|
||||
ENV LOG_LEVEL=info
|
||||
|
||||
# Use ARG for build-time variables
|
||||
ARG BUILD_VERSION
|
||||
ENV APP_VERSION=$BUILD_VERSION
|
||||
|
||||
# The application should validate required env vars at startup
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
## Container Security Best Practices
|
||||
|
||||
### **1. Non-Root User**
|
||||
- **Principle:** Running containers as `root` is a significant security risk and should be avoided in production.
|
||||
- **Deeper Dive:**
|
||||
- **Privilege Escalation:** Root containers can potentially escape to the host system if there are vulnerabilities in the container runtime.
|
||||
- **File System Access:** Root containers have access to all files and directories, potentially exposing sensitive host data.
|
||||
- **Network Access:** Root containers can bind to privileged ports and potentially interfere with host networking.
|
||||
- **Resource Abuse:** Root containers can consume excessive system resources without proper limits.
|
||||
- **Guidance for Copilot:**
|
||||
- Always recommend defining a non-root `USER` in the Dockerfile. Create a dedicated user for your application.
|
||||
- Ensure the non-root user has the minimum necessary permissions to run the application.
|
||||
- Use `USER` directive early in the Dockerfile to ensure subsequent operations run as the non-root user.
|
||||
- Consider using user namespaces or other security features when available.
|
||||
- **Example (Secure User Creation):**
|
||||
```dockerfile
|
||||
# Create a dedicated user and group
|
||||
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
|
||||
|
||||
# Set proper ownership of application files
|
||||
RUN chown -R appuser:appgroup /app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Ensure the user can write to necessary directories
|
||||
VOLUME ["/app/data"]
|
||||
```
|
||||
|
||||
### **2. Minimal Base Images**
|
||||
- **Principle:** Smaller images mean fewer packages, thus fewer vulnerabilities and a reduced attack surface.
|
||||
- **Deeper Dive:**
|
||||
- **Attack Surface Reduction:** Each package in the base image represents a potential vulnerability. Fewer packages mean fewer potential attack vectors.
|
||||
- **Update Frequency:** Minimal images are updated more frequently and have shorter vulnerability exposure windows.
|
||||
- **Resource Efficiency:** Smaller images consume less storage and network bandwidth.
|
||||
- **Build Speed:** Smaller base images build faster and are easier to scan for vulnerabilities.
|
||||
- **Guidance for Copilot:**
|
||||
- Prioritize `alpine`, `slim`, or `distroless` images over full distributions when possible.
|
||||
- Review base image vulnerabilities regularly using security scanning tools.
|
||||
- Consider using language-specific minimal images (e.g., `openjdk:17-jre-slim` instead of `openjdk:17`).
|
||||
- Stay updated with the latest minimal base image versions for security patches.
|
||||
- **Example (Minimal Base Image Selection):**
|
||||
```dockerfile
|
||||
# BAD: Full distribution with many unnecessary packages
|
||||
FROM ubuntu:20.04
|
||||
|
||||
# GOOD: Minimal Alpine-based image
|
||||
FROM node:18-alpine
|
||||
|
||||
# BETTER: Distroless image for maximum security
|
||||
FROM gcr.io/distroless/nodejs18-debian11
|
||||
```
|
||||
|
||||
### **3. Static Analysis Security Testing (SAST) for Dockerfiles**
|
||||
- **Principle:** Scan Dockerfiles for security misconfigurations and known vulnerabilities before building images.
|
||||
- **Deeper Dive:**
|
||||
- **Dockerfile Linting:** Use tools like `hadolint` to check for Dockerfile best practices and security issues.
|
||||
- **Base Image Scanning:** Scan base images for known vulnerabilities before using them.
|
||||
- **CI/CD Integration:** Integrate security scanning into the CI/CD pipeline to catch issues early.
|
||||
- **Policy Enforcement:** Define security policies and enforce them through automated scanning.
|
||||
- **Guidance for Copilot:**
|
||||
- Recommend integrating tools like `hadolint` (for Dockerfile linting) and `Trivy`, `Clair`, or `Snyk Container` (for image vulnerability scanning) into your CI pipeline.
|
||||
- Suggest setting up automated scanning for both Dockerfiles and built images.
|
||||
- Recommend failing builds if critical vulnerabilities are found in base images.
|
||||
- Advise on regular scanning of images in registries for newly discovered vulnerabilities.
|
||||
- **Example (Security Scanning in CI):**
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Run Hadolint
|
||||
run: |
|
||||
docker run --rm -i hadolint/hadolint < Dockerfile
|
||||
|
||||
- name: Scan image for vulnerabilities
|
||||
run: |
|
||||
docker build -t myapp .
|
||||
trivy image myapp
|
||||
```
|
||||
|
||||
### **4. Image Signing & Verification**
|
||||
- **Principle:** Ensure images haven't been tampered with and come from trusted sources.
|
||||
- **Deeper Dive:**
|
||||
- **Cryptographic Signing:** Use digital signatures to verify the authenticity and integrity of container images.
|
||||
- **Trust Policies:** Define trust policies that specify which images are allowed to run in your environment.
|
||||
- **Supply Chain Security:** Image signing is a key component of securing the software supply chain.
|
||||
- **Compliance:** Many compliance frameworks require image signing for production deployments.
|
||||
- **Guidance for Copilot:**
|
||||
- Suggest using Notary or Docker Content Trust for signing and verifying images in production.
|
||||
- Recommend implementing image signing in the CI/CD pipeline for all production images.
|
||||
- Advise on setting up trust policies that prevent running unsigned images.
|
||||
- Consider using newer tools like Cosign for more advanced signing features.
|
||||
- **Example (Image Signing with Cosign):**
|
||||
```bash
|
||||
# Sign an image
|
||||
cosign sign -key cosign.key myregistry.com/myapp:v1.0.0
|
||||
|
||||
# Verify an image
|
||||
cosign verify -key cosign.pub myregistry.com/myapp:v1.0.0
|
||||
```
|
||||
|
||||
### **5. Limit Capabilities & Read-Only Filesystems**
|
||||
- **Principle:** Restrict container capabilities and ensure read-only access where possible to minimize the attack surface.
|
||||
- **Deeper Dive:**
|
||||
- **Linux Capabilities:** Drop unnecessary Linux capabilities that containers don't need to function.
|
||||
- **Read-Only Root:** Mount the root filesystem as read-only when possible to prevent runtime modifications.
|
||||
- **Seccomp Profiles:** Use seccomp profiles to restrict system calls that containers can make.
|
||||
- **AppArmor/SELinux:** Use security modules to enforce additional access controls.
|
||||
- **Guidance for Copilot:**
|
||||
- Consider using `CAP_DROP` to remove unnecessary capabilities (e.g., `NET_RAW`, `SYS_ADMIN`).
|
||||
- Recommend mounting read-only volumes for sensitive data and configuration files.
|
||||
- Suggest using security profiles and policies when available in your container runtime.
|
||||
- Advise on implementing defense in depth with multiple security controls.
|
||||
- **Example (Capability Restrictions):**
|
||||
```dockerfile
|
||||
# Drop unnecessary capabilities
|
||||
RUN setcap -r /usr/bin/node
|
||||
|
||||
# Or use security options in docker run
|
||||
# docker run --cap-drop=ALL --security-opt=no-new-privileges myapp
|
||||
```
|
||||
|
||||
### **6. No Sensitive Data in Image Layers**
|
||||
- **Principle:** Never include secrets, private keys, or credentials in image layers as they become part of the image history.
|
||||
- **Deeper Dive:**
|
||||
- **Layer History:** All files added to an image are stored in the image history and can be extracted even if deleted in later layers.
|
||||
- **Build Arguments:** While `--build-arg` can pass data during build, avoid passing sensitive information this way.
|
||||
- **Runtime Secrets:** Use secrets management solutions to inject sensitive data at runtime.
|
||||
- **Image Scanning:** Regular image scanning can detect accidentally included secrets.
|
||||
- **Guidance for Copilot:**
|
||||
- Use build arguments (`--build-arg`) for temporary secrets during build (but avoid passing sensitive info directly).
|
||||
- Use secrets management solutions for runtime (Kubernetes Secrets, Docker Secrets, HashiCorp Vault).
|
||||
- Recommend scanning images for accidentally included secrets.
|
||||
- Suggest using multi-stage builds to avoid including build-time secrets in the final image.
|
||||
- **Anti-pattern:** `ADD secrets.txt /app/secrets.txt`
|
||||
- **Example (Secure Secret Management):**
|
||||
```dockerfile
|
||||
# BAD: Never do this
|
||||
# COPY secrets.txt /app/secrets.txt
|
||||
|
||||
# GOOD: Use runtime secrets
|
||||
# The application should read secrets from environment variables or mounted files
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
### **7. Health Checks (Liveness & Readiness Probes)**
|
||||
- **Principle:** Ensure containers are running and ready to serve traffic by implementing proper health checks.
|
||||
- **Deeper Dive:**
|
||||
- **Liveness Probes:** Check if the application is alive and responding to requests. Restart the container if it fails.
|
||||
- **Readiness Probes:** Check if the application is ready to receive traffic. Remove from load balancer if it fails.
|
||||
- **Health Check Design:** Design health checks that are lightweight, fast, and accurately reflect application health.
|
||||
- **Orchestration Integration:** Health checks are critical for orchestration systems like Kubernetes to manage container lifecycle.
|
||||
- **Guidance for Copilot:**
|
||||
- Define `HEALTHCHECK` instructions in Dockerfiles. These are critical for orchestration systems like Kubernetes.
|
||||
- Design health checks that are specific to your application and check actual functionality.
|
||||
- Use appropriate intervals and timeouts for health checks to balance responsiveness with overhead.
|
||||
- Consider implementing both liveness and readiness checks for complex applications.
|
||||
- **Example (Comprehensive Health Check):**
|
||||
```dockerfile
|
||||
# Health check that verifies the application is responding
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD curl --fail http://localhost:8080/health || exit 1
|
||||
|
||||
# Alternative: Use application-specific health check
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD node healthcheck.js || exit 1
|
||||
```
|
||||
|
||||
## Container Runtime & Orchestration Best Practices
|
||||
|
||||
### **1. Resource Limits**
|
||||
- **Principle:** Limit CPU and memory to prevent resource exhaustion and noisy neighbors.
|
||||
- **Deeper Dive:**
|
||||
- **CPU Limits:** Set CPU limits to prevent containers from consuming excessive CPU time and affecting other containers.
|
||||
- **Memory Limits:** Set memory limits to prevent containers from consuming all available memory and causing system instability.
|
||||
- **Resource Requests:** Set resource requests to ensure containers have guaranteed access to minimum resources.
|
||||
- **Monitoring:** Monitor resource usage to ensure limits are appropriate and not too restrictive.
|
||||
- **Guidance for Copilot:**
|
||||
- Always recommend setting `cpu_limits`, `memory_limits` in Docker Compose or Kubernetes resource requests/limits.
|
||||
- Suggest monitoring resource usage to tune limits appropriately.
|
||||
- Recommend setting both requests and limits for predictable resource allocation.
|
||||
- Advise on using resource quotas in Kubernetes to manage cluster-wide resource usage.
|
||||
- **Example (Docker Compose Resource Limits):**
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: myapp:latest
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
reservations:
|
||||
cpus: '0.25'
|
||||
memory: 256M
|
||||
```
|
||||
|
||||
### **2. Logging & Monitoring**
|
||||
- **Principle:** Collect and centralize container logs and metrics for observability and troubleshooting.
|
||||
- **Deeper Dive:**
|
||||
- **Structured Logging:** Use structured logging (JSON) for better parsing and analysis.
|
||||
- **Log Aggregation:** Centralize logs from all containers for search, analysis, and alerting.
|
||||
- **Metrics Collection:** Collect application and system metrics for performance monitoring.
|
||||
- **Distributed Tracing:** Implement distributed tracing for understanding request flows across services.
|
||||
- **Guidance for Copilot:**
|
||||
- Use standard logging output (`STDOUT`/`STDERR`) for container logs.
|
||||
- Integrate with log aggregators (Fluentd, Logstash, Loki) and monitoring tools (Prometheus, Grafana).
|
||||
- Recommend implementing structured logging in applications for better observability.
|
||||
- Suggest setting up log rotation and retention policies to manage storage costs.
|
||||
- **Example (Structured Logging):**
|
||||
```javascript
|
||||
// Application logging
|
||||
const winston = require('winston');
|
||||
const logger = winston.createLogger({
|
||||
format: winston.format.json(),
|
||||
transports: [new winston.transports.Console()]
|
||||
});
|
||||
```
|
||||
|
||||
### **3. Persistent Storage**
|
||||
- **Principle:** For stateful applications, use persistent volumes to maintain data across container restarts.
|
||||
- **Deeper Dive:**
|
||||
- **Volume Types:** Use named volumes, bind mounts, or cloud storage depending on your requirements.
|
||||
- **Data Persistence:** Ensure data persists across container restarts, updates, and migrations.
|
||||
- **Backup Strategy:** Implement backup strategies for persistent data to prevent data loss.
|
||||
- **Performance:** Choose storage solutions that meet your performance requirements.
|
||||
- **Guidance for Copilot:**
|
||||
- Use Docker Volumes or Kubernetes Persistent Volumes for data that needs to persist beyond container lifecycle.
|
||||
- Never store persistent data inside the container's writable layer.
|
||||
- Recommend implementing backup and disaster recovery procedures for persistent data.
|
||||
- Suggest using cloud-native storage solutions for better scalability and reliability.
|
||||
- **Example (Docker Volume Usage):**
|
||||
```yaml
|
||||
services:
|
||||
database:
|
||||
image: postgres:13
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
environment:
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
```
|
||||
|
||||
### **4. Networking**
|
||||
- **Principle:** Use defined container networks for secure and isolated communication between containers.
|
||||
- **Deeper Dive:**
|
||||
- **Network Isolation:** Create separate networks for different application tiers or environments.
|
||||
- **Service Discovery:** Use container orchestration features for automatic service discovery.
|
||||
- **Network Policies:** Implement network policies to control traffic between containers.
|
||||
- **Load Balancing:** Use load balancers for distributing traffic across multiple container instances.
|
||||
- **Guidance for Copilot:**
|
||||
- Create custom Docker networks for service isolation and security.
|
||||
- Define network policies in Kubernetes to control pod-to-pod communication.
|
||||
- Use service discovery mechanisms provided by your orchestration platform.
|
||||
- Implement proper network segmentation for multi-tier applications.
|
||||
- **Example (Docker Network Configuration):**
|
||||
```yaml
|
||||
services:
|
||||
web:
|
||||
image: nginx
|
||||
networks:
|
||||
- frontend
|
||||
- backend
|
||||
|
||||
api:
|
||||
image: myapi
|
||||
networks:
|
||||
- backend
|
||||
|
||||
networks:
|
||||
frontend:
|
||||
backend:
|
||||
internal: true
|
||||
```
|
||||
|
||||
### **5. Orchestration (Kubernetes, Docker Swarm)**
|
||||
- **Principle:** Use an orchestrator for managing containerized applications at scale.
|
||||
- **Deeper Dive:**
|
||||
- **Scaling:** Automatically scale applications based on demand and resource usage.
|
||||
- **Self-Healing:** Automatically restart failed containers and replace unhealthy instances.
|
||||
- **Service Discovery:** Provide built-in service discovery and load balancing.
|
||||
- **Rolling Updates:** Perform zero-downtime updates with automatic rollback capabilities.
|
||||
- **Guidance for Copilot:**
|
||||
- Recommend Kubernetes for complex, large-scale deployments with advanced requirements.
|
||||
- Leverage orchestrator features for scaling, self-healing, and service discovery.
|
||||
- Use rolling update strategies for zero-downtime deployments.
|
||||
- Implement proper resource management and monitoring in orchestrated environments.
|
||||
- **Example (Kubernetes Deployment):**
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: myapp
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: myapp
|
||||
spec:
|
||||
containers:
|
||||
- name: myapp
|
||||
image: myapp:latest
|
||||
resources:
|
||||
requests:
|
||||
memory: "64Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "500m"
|
||||
```
|
||||
|
||||
## Dockerfile Review Checklist
|
||||
|
||||
- [ ] Is a multi-stage build used if applicable (compiled languages, heavy build tools)?
|
||||
- [ ] Is a minimal, specific base image used (e.g., `alpine`, `slim`, versioned)?
|
||||
- [ ] Are layers optimized (combining `RUN` commands, cleanup in same layer)?
|
||||
- [ ] Is a `.dockerignore` file present and comprehensive?
|
||||
- [ ] Are `COPY` instructions specific and minimal?
|
||||
- [ ] Is a non-root `USER` defined for the running application?
|
||||
- [ ] Is the `EXPOSE` instruction used for documentation?
|
||||
- [ ] Is `CMD` and/or `ENTRYPOINT` used correctly?
|
||||
- [ ] Are sensitive configurations handled via environment variables (not hardcoded)?
|
||||
- [ ] Is a `HEALTHCHECK` instruction defined?
|
||||
- [ ] Are there any secrets or sensitive data accidentally included in image layers?
|
||||
- [ ] Are there static analysis tools (Hadolint, Trivy) integrated into CI?
|
||||
|
||||
## Troubleshooting Docker Builds & Runtime
|
||||
|
||||
### **1. Large Image Size**
|
||||
- Review layers for unnecessary files. Use `docker history <image>`.
|
||||
- Implement multi-stage builds.
|
||||
- Use a smaller base image.
|
||||
- Optimize `RUN` commands and clean up temporary files.
|
||||
|
||||
### **2. Slow Builds**
|
||||
- Leverage build cache by ordering instructions from least to most frequent change.
|
||||
- Use `.dockerignore` to exclude irrelevant files.
|
||||
- Use `docker build --no-cache` for troubleshooting cache issues.
|
||||
|
||||
### **3. Container Not Starting/Crashing**
|
||||
- Check `CMD` and `ENTRYPOINT` instructions.
|
||||
- Review container logs (`docker logs <container_id>`).
|
||||
- Ensure all dependencies are present in the final image.
|
||||
- Check resource limits.
|
||||
|
||||
### **4. Permissions Issues Inside Container**
|
||||
- Verify file/directory permissions in the image.
|
||||
- Ensure the `USER` has necessary permissions for operations.
|
||||
- Check mounted volumes permissions.
|
||||
|
||||
### **5. Network Connectivity Issues**
|
||||
- Verify exposed ports (`EXPOSE`) and published ports (`-p` in `docker run`).
|
||||
- Check container network configuration.
|
||||
- Review firewall rules.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Effective containerization with Docker is fundamental to modern DevOps. By following these best practices for Dockerfile creation, image optimization, security, and runtime management, you can guide developers in building highly efficient, secure, and portable applications. Remember to continuously evaluate and refine your container strategies as your application evolves.
|
||||
|
||||
---
|
||||
|
||||
<!-- End of Containerization & Docker Best Practices Instructions -->
|
||||
@@ -0,0 +1,190 @@
|
||||
# Charon Copilot Instructions
|
||||
|
||||
## Code Quality Guidelines
|
||||
|
||||
Every session should improve the codebase, not just add to it. Actively refactor code you encounter, even outside of your immediate task scope. Think about long-term maintainability and consistency. Make a detailed plan before writing code. Always create unit tests for new code coverage.
|
||||
|
||||
- **MANDATORY**: Read all relevant instructions in `.github/instructions/` for the specific task before starting.
|
||||
- **DRY**: Consolidate duplicate patterns into reusable functions, types, or components after the second occurrence.
|
||||
- **CLEAN**: Delete dead code immediately. Remove unused imports, variables, functions, types, commented code, and console logs.
|
||||
- **LEVERAGE**: Use battle-tested packages over custom implementations.
|
||||
- **READABLE**: Maintain comments and clear naming for complex logic. Favor clarity over cleverness.
|
||||
- **CONVENTIONAL COMMITS**: Write commit messages using `feat:`, `fix:`, `chore:`, `refactor:`, or `docs:` prefixes.
|
||||
|
||||
## 🚨 CRITICAL ARCHITECTURE RULES 🚨
|
||||
|
||||
- **Single Frontend Source**: All frontend code MUST reside in `frontend/`. NEVER create `backend/frontend/` or any other nested frontend directory.
|
||||
- **Single Backend Source**: All backend code MUST reside in `backend/`.
|
||||
- **No Python**: This is a Go (Backend) + React/TypeScript (Frontend) project. Do not introduce Python scripts or requirements.
|
||||
|
||||
## 🛑 Root Cause Analysis Protocol (MANDATORY)
|
||||
**Constraint:** You must NEVER patch a symptom without tracing the root cause.
|
||||
If a bug is reported, do NOT stop at the first error message found. Use Playwright MCP to trace the entire flow from frontend action to backend processing. Identify the true origin of the issue.
|
||||
|
||||
**The "Context First" Rule:**
|
||||
Before proposing ANY code change or fix, you must build a mental map of the feature:
|
||||
1. **Entry Point:** Where does the data enter? (API Route / UI Event)
|
||||
2. **Transformation:** How is the data modified? (Handlers / Middleware)
|
||||
3. **Persistence:** Where is it stored? (DB Models / Files)
|
||||
4. **Exit Point:** How is it returned to the user?
|
||||
|
||||
**Anti-Pattern Warning:** - Do not assume the error log is the *cause*; it is often just the *victim* of an upstream failure.
|
||||
- If you find an error, search for "upstream callers" to see *why* that data was bad in the first place.
|
||||
|
||||
## Big Picture
|
||||
|
||||
- Charon is a self-hosted web app for managing reverse proxy host configurations with the novice user in mind. Everything should prioritize simplicity, usability, reliability, and security, all rolled into one simple binary + static assets deployment. No external dependencies.
|
||||
- Users should feel like they have enterprise-level security and features with zero effort.
|
||||
- `backend/cmd/api` loads config, opens SQLite, then hands off to `internal/server`.
|
||||
- `internal/config` respects `CHARON_ENV`, `CHARON_HTTP_PORT`, `CHARON_DB_PATH` and creates the `data/` directory.
|
||||
- `internal/server` mounts the built React app (via `attachFrontend`) whenever `frontend/dist` exists.
|
||||
- Persistent types live in `internal/models`; GORM auto-migrates them.
|
||||
|
||||
## Backend Workflow
|
||||
|
||||
- **Run**: `cd backend && go run ./cmd/api`.
|
||||
- **Test**: `go test ./...`.
|
||||
- **Static Analysis (BLOCKING)**: Fast linters run automatically on every commit via pre-commit hooks.
|
||||
- **Staticcheck errors MUST be fixed** - commits are BLOCKED until resolved
|
||||
- Manual run: `make lint-fast` or VS Code task "Lint: Staticcheck (Fast)"
|
||||
- Staticcheck-only: `make lint-staticcheck-only`
|
||||
- Runtime: ~11s (measured: 10.9s) (acceptable for commit gate)
|
||||
- Full golangci-lint (all linters): Use `make lint-backend` before PR (manual stage)
|
||||
- **API Response**: Handlers return structured errors using `gin.H{"error": "message"}`.
|
||||
- **JSON Tags**: All struct fields exposed to the frontend MUST have explicit `json:"snake_case"` tags.
|
||||
- **IDs**: UUIDs (`github.com/google/uuid`) are generated server-side; clients never send numeric IDs.
|
||||
- **Security**: Sanitize all file paths using `filepath.Clean`. Use `fmt.Errorf("context: %w", err)` for error wrapping.
|
||||
- **Graceful Shutdown**: Long-running work must respect `server.Run(ctx)`.
|
||||
|
||||
### Troubleshooting Pre-Commit Staticcheck Failures
|
||||
|
||||
**Common Issues:**
|
||||
|
||||
1. **"golangci-lint not found"**
|
||||
- Install: See README.md Development Setup section
|
||||
- Verify: `golangci-lint --version`
|
||||
- Ensure `$GOPATH/bin` is in PATH
|
||||
|
||||
2. **Staticcheck reports deprecated API usage (SA1019)**
|
||||
- Fix: Replace deprecated function with recommended alternative
|
||||
- Check Go docs for migration path
|
||||
- Example: `filepath.HasPrefix` → use `strings.HasPrefix` with cleaned paths
|
||||
|
||||
3. **"This value is never used" (SA4006)**
|
||||
- Fix: Remove unused assignment or use the value
|
||||
- Common in test setup code
|
||||
|
||||
4. **"Should replace if statement with..." (S10xx)**
|
||||
- Fix: Apply suggested simplification
|
||||
- These improve readability and performance
|
||||
|
||||
5. **Emergency bypass (use sparingly):**
|
||||
- `git commit --no-verify -m "Emergency hotfix"`
|
||||
- **MUST** create follow-up issue to fix staticcheck errors
|
||||
- Only for production incidents
|
||||
|
||||
## Frontend Workflow
|
||||
|
||||
- **Location**: Always work within `frontend/`.
|
||||
- **Stack**: React 18 + Vite + TypeScript + TanStack Query (React Query).
|
||||
- **State Management**: Use `src/hooks/use*.ts` wrapping React Query.
|
||||
- **API Layer**: Create typed API clients in `src/api/*.ts` that wrap `client.ts`.
|
||||
- **Forms**: Use local `useState` for form fields, submit via `useMutation`, then `invalidateQueries` on success.
|
||||
|
||||
## Cross-Cutting Notes
|
||||
|
||||
- **VS Code Integration**: If you introduce new repetitive CLI actions (e.g., scans, builds, scripts), register them in .vscode/tasks.json to allow for easy manual verification.
|
||||
- **Sync**: React Query expects the exact JSON produced by GORM tags (snake_case). Keep API and UI field names aligned.
|
||||
- **Migrations**: When adding models, update `internal/models` AND `internal/api/routes/routes.go` (AutoMigrate).
|
||||
- **Testing**: All new code MUST include accompanying unit tests.
|
||||
- **Ignore Files**: Always check `.gitignore`, `.dockerignore`, and `.codecov.yml` when adding new file or folders.
|
||||
|
||||
## Documentation
|
||||
|
||||
- **Features**: Update `docs/features.md` when adding capabilities. This is a short "marketing" style list. Keep details to their individual docs.
|
||||
- **Links**: Use GitHub Pages URLs (`https://wikid82.github.io/charon/`) for docs and GitHub blob links for repo files.
|
||||
|
||||
## CI/CD & Commit Conventions
|
||||
|
||||
- **Triggers**: Use `feat:`, `fix:`, or `perf:` to trigger Docker builds. `chore:` skips builds.
|
||||
- **Beta**: `feature/beta-release` always builds.
|
||||
- **History-Rewrite PRs**: If a PR touches files in `scripts/history-rewrite/` or `docs/plans/history_rewrite.md`, the PR description MUST include the history-rewrite checklist from `.github/PULL_REQUEST_TEMPLATE/history-rewrite.md`. This is enforced by CI.
|
||||
|
||||
## ✅ Task Completion Protocol (Definition of Done)
|
||||
|
||||
Before marking an implementation task as complete, perform the following in order:
|
||||
|
||||
1. **Playwright E2E Tests** (MANDATORY - Run First):
|
||||
- **Run**: `npx playwright test --project=chromium` from project root
|
||||
- **Why First**: If the app is broken at E2E level, unit tests may need updates. Catch integration issues early.
|
||||
- **Scope**: Run tests relevant to modified features (e.g., `tests/manual-dns-provider.spec.ts`)
|
||||
- **On Failure**: Trace root cause through frontend → backend flow before proceeding
|
||||
- **Base URL**: Uses `PLAYWRIGHT_BASE_URL` or default from `playwright.config.js`
|
||||
- All E2E tests must pass before proceeding to unit tests
|
||||
|
||||
2. **Security Scans** (MANDATORY - Zero Tolerance):
|
||||
- **CodeQL Go Scan**: Run VS Code task "Security: CodeQL Go Scan (CI-Aligned)" OR `pre-commit run codeql-go-scan --all-files`
|
||||
- Must use `security-and-quality` suite (CI-aligned)
|
||||
- **Zero high/critical (error-level) findings allowed**
|
||||
- Medium/low findings should be documented and triaged
|
||||
- **CodeQL JS Scan**: Run VS Code task "Security: CodeQL JS Scan (CI-Aligned)" OR `pre-commit run codeql-js-scan --all-files`
|
||||
- Must use `security-and-quality` suite (CI-aligned)
|
||||
- **Zero high/critical (error-level) findings allowed**
|
||||
- Medium/low findings should be documented and triaged
|
||||
- **Validate Findings**: Run `pre-commit run codeql-check-findings --all-files` to check for HIGH/CRITICAL issues
|
||||
- **Trivy Container Scan**: Run VS Code task "Security: Trivy Scan" for container/dependency vulnerabilities
|
||||
- **Results Viewing**:
|
||||
- Primary: VS Code SARIF Viewer extension (`MS-SarifVSCode.sarif-viewer`)
|
||||
- Alternative: `jq` command-line parsing: `jq '.runs[].results' codeql-results-*.sarif`
|
||||
- CI: GitHub Security tab for automated uploads
|
||||
- **⚠️ CRITICAL:** CodeQL scans are NOT run by default pre-commit hooks (manual stage for performance). You MUST run them explicitly via VS Code tasks or pre-commit manual commands before completing any task.
|
||||
- **Why:** CI enforces security-and-quality suite and blocks HIGH/CRITICAL findings. Local verification prevents CI failures and ensures security compliance.
|
||||
- **CI Alignment:** Local scans now use identical parameters to CI:
|
||||
- Query suite: `security-and-quality` (61 Go queries, 204 JS queries)
|
||||
- Database creation: `--threads=0 --overwrite`
|
||||
- Analysis: `--sarif-add-baseline-file-info`
|
||||
|
||||
3. **Pre-Commit Triage**: Run `pre-commit run --all-files`.
|
||||
- If errors occur, **fix them immediately**.
|
||||
- If logic errors occur, analyze and propose a fix.
|
||||
- Do not output code that violates pre-commit standards.
|
||||
|
||||
4. **Staticcheck BLOCKING Validation**: Pre-commit hooks automatically run fast linters including staticcheck.
|
||||
- **CRITICAL:** Staticcheck errors are BLOCKING - you MUST fix them before commit succeeds.
|
||||
- Manual verification: Run VS Code task "Lint: Staticcheck (Fast)" or `make lint-fast`
|
||||
- To check only staticcheck: `make lint-staticcheck-only`
|
||||
- Test files (`_test.go`) are excluded from staticcheck (matches CI behavior)
|
||||
- If pre-commit fails: Fix the reported issues, then retry commit
|
||||
- **Do NOT** use `--no-verify` to bypass this check unless emergency hotfix
|
||||
|
||||
5. **Coverage Testing** (MANDATORY - Non-negotiable):
|
||||
- **MANDATORY**: Patch coverage must cover 100% of modified lines (Codecov Patch view must be green). If patch coverage fails, add targeted tests for the missing patch line ranges.
|
||||
- **Backend Changes**: Run the VS Code task "Test: Backend with Coverage" or execute `scripts/go-test-coverage.sh`.
|
||||
- Minimum coverage: 85% (set via `CHARON_MIN_COVERAGE` or `CPM_MIN_COVERAGE`).
|
||||
- If coverage drops below threshold, write additional tests to restore coverage.
|
||||
- All tests must pass with zero failures.
|
||||
- **Frontend Changes**: Run the VS Code task "Test: Frontend with Coverage" or execute `scripts/frontend-test-coverage.sh`.
|
||||
- Minimum coverage: 85% (set via `CHARON_MIN_COVERAGE` or `CPM_MIN_COVERAGE`).
|
||||
- If coverage drops below threshold, write additional tests to restore coverage.
|
||||
- All tests must pass with zero failures.
|
||||
- **Critical**: Coverage tests are NOT run by default pre-commit hooks (they are in manual stage for performance). You MUST run them explicitly via VS Code tasks or scripts before completing any task.
|
||||
- **Why**: CI enforces coverage in GitHub Actions. Local verification prevents CI failures and maintains code quality.
|
||||
|
||||
6. **Type Safety** (Frontend only):
|
||||
- Run the VS Code task "Lint: TypeScript Check" or execute `cd frontend && npm run type-check`.
|
||||
- Fix all type errors immediately. This is non-negotiable.
|
||||
- This check is also in manual stage for performance but MUST be run before completion.
|
||||
|
||||
7. **Verify Build**: Ensure the backend compiles and the frontend builds without errors.
|
||||
- Backend: `cd backend && go build ./...`
|
||||
- Frontend: `cd frontend && npm run build`
|
||||
|
||||
8. **Fixed and New Code Testing**:
|
||||
- Ensure all existing and new unit tests pass with zero failures.
|
||||
- When failures and errors are found, deep-dive into root causes. Using the correct `subAgent`, update the working plan, review the implementation, and fix the issues.
|
||||
- No issue is out of scope for investigation and resolution. All issues must be addressed before task completion.
|
||||
|
||||
9. **Clean Up**: Ensure no debug print statements or commented-out blocks remain.
|
||||
- Remove `console.log`, `fmt.Println`, and similar debugging statements.
|
||||
- Delete commented-out code blocks.
|
||||
- Remove unused imports.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
description: "Guidance for writing and formatting the `docs/features.md` file."
|
||||
applyTo: "docs/features.md"
|
||||
---
|
||||
|
||||
# Features Documentation Guidelines
|
||||
|
||||
When creating or updating the `docs/features.md` file, please adhere to the following guidelines to ensure clarity and consistency:
|
||||
|
||||
## Structure
|
||||
|
||||
- This document should provide a short, to the point overview of each feature. It is used for marketing of the project. A quick read of what the feature is and why it matters. It is the "elevator pitch" for each feature.
|
||||
- Each feature should have its own section with a clear heading.
|
||||
- Use bullet points or numbered lists to break down complex information.
|
||||
- Include relevant links to other documentation or resources for further reading.
|
||||
- Use consistent formatting for headings, subheadings, and text styles throughout the document.
|
||||
- Avoid overly technical jargon; the document should be accessible to a broad audience. Keep novice users in mind.
|
||||
- This is not the place for deep technical details or implementation specifics. Keep those for individual feature docs.
|
||||
|
||||
## Content
|
||||
- Start with a brief summary of the feature.
|
||||
- Explain the purpose and benefits of the feature.
|
||||
- Keep descriptions concise and focused.
|
||||
- Ensure accuracy and up-to-date information.
|
||||
|
||||
## Review
|
||||
@@ -0,0 +1,607 @@
|
||||
---
|
||||
applyTo: '.github/workflows/*.yml,.github/workflows/*.yaml'
|
||||
description: 'Comprehensive guide for building robust, secure, and efficient CI/CD pipelines using GitHub Actions. Covers workflow structure, jobs, steps, environment variables, secret management, caching, matrix strategies, testing, and deployment strategies.'
|
||||
---
|
||||
|
||||
# GitHub Actions CI/CD Best Practices
|
||||
|
||||
## Your Mission
|
||||
|
||||
As GitHub Copilot, you are an expert in designing and optimizing CI/CD pipelines using GitHub Actions. Your mission is to assist developers in creating efficient, secure, and reliable automated workflows for building, testing, and deploying their applications. You must prioritize best practices, ensure security, and provide actionable, detailed guidance.
|
||||
|
||||
## Core Concepts and Structure
|
||||
|
||||
### **1. Workflow Structure (`.github/workflows/*.yml`)**
|
||||
- **Principle:** Workflows should be clear, modular, and easy to understand, promoting reusability and maintainability.
|
||||
- **Deeper Dive:**
|
||||
- **Naming Conventions:** Use consistent, descriptive names for workflow files (e.g., `build-and-test.yml`, `deploy-prod.yml`).
|
||||
- **Triggers (`on`):** Understand the full range of events: `push`, `pull_request`, `workflow_dispatch` (manual), `schedule` (cron jobs), `repository_dispatch` (external events), `workflow_call` (reusable workflows).
|
||||
- **Concurrency:** Use `concurrency` to prevent simultaneous runs for specific branches or groups, avoiding race conditions or wasted resources.
|
||||
- **Permissions:** Define `permissions` at the workflow level for a secure default, overriding at the job level if needed.
|
||||
- **Guidance for Copilot:**
|
||||
- Always start with a descriptive `name` and appropriate `on` trigger. Suggest granular triggers for specific use cases (e.g., `on: push: branches: [main]` vs. `on: pull_request`).
|
||||
- Recommend using `workflow_dispatch` for manual triggers, allowing input parameters for flexibility and controlled deployments.
|
||||
- Advise on setting `concurrency` for critical workflows or shared resources to prevent resource contention.
|
||||
- Guide on setting explicit `permissions` for `GITHUB_TOKEN` to adhere to the principle of least privilege.
|
||||
- **Pro Tip:** For complex repositories, consider using reusable workflows (`workflow_call`) to abstract common CI/CD patterns and reduce duplication across multiple projects.
|
||||
|
||||
### **2. Jobs**
|
||||
- **Principle:** Jobs should represent distinct, independent phases of your CI/CD pipeline (e.g., build, test, deploy, lint, security scan).
|
||||
- **Deeper Dive:**
|
||||
- **`runs-on`:** Choose appropriate runners. `ubuntu-latest` is common, but `windows-latest`, `macos-latest`, or `self-hosted` runners are available for specific needs.
|
||||
- **`needs`:** Clearly define dependencies. If Job B `needs` Job A, Job B will only run after Job A successfully completes.
|
||||
- **`outputs`:** Pass data between jobs using `outputs`. This is crucial for separating concerns (e.g., build job outputs artifact path, deploy job consumes it).
|
||||
- **`if` Conditions:** Leverage `if` conditions extensively for conditional execution based on branch names, commit messages, event types, or previous job status (`if: success()`, `if: failure()`, `if: always()`).
|
||||
- **Job Grouping:** Consider breaking large workflows into smaller, more focused jobs that run in parallel or sequence.
|
||||
- **Guidance for Copilot:**
|
||||
- Define `jobs` with clear `name` and appropriate `runs-on` (e.g., `ubuntu-latest`, `windows-latest`, `self-hosted`).
|
||||
- Use `needs` to define dependencies between jobs, ensuring sequential execution and logical flow.
|
||||
- Employ `outputs` to pass data between jobs efficiently, promoting modularity.
|
||||
- Utilize `if` conditions for conditional job execution (e.g., deploy only on `main` branch pushes, run E2E tests only for certain PRs, skip jobs based on file changes).
|
||||
- **Example (Conditional Deployment and Output Passing):**
|
||||
```yaml
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
artifact_path: ${{ steps.package_app.outputs.path }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 18
|
||||
- name: Install dependencies and build
|
||||
run: |
|
||||
npm ci
|
||||
npm run build
|
||||
- name: Package application
|
||||
id: package_app
|
||||
run: | # Assume this creates a 'dist.zip' file
|
||||
zip -r dist.zip dist
|
||||
echo "path=dist.zip" >> "$GITHUB_OUTPUT"
|
||||
- name: Upload build artifact
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: my-app-build
|
||||
path: dist.zip
|
||||
|
||||
deploy-staging:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
if: github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/main'
|
||||
environment: staging
|
||||
steps:
|
||||
- name: Download build artifact
|
||||
uses: actions/download-artifact@v3
|
||||
with:
|
||||
name: my-app-build
|
||||
- name: Deploy to Staging
|
||||
run: |
|
||||
unzip dist.zip
|
||||
echo "Deploying ${{ needs.build.outputs.artifact_path }} to staging..."
|
||||
# Add actual deployment commands here
|
||||
```
|
||||
|
||||
### **3. Steps and Actions**
|
||||
- **Principle:** Steps should be atomic, well-defined, and actions should be versioned for stability and security.
|
||||
- **Deeper Dive:**
|
||||
- **`uses`:** Referencing marketplace actions (e.g., `actions/checkout@v4`, `actions/setup-node@v3`) or custom actions. Always pin to a full length commit SHA for maximum security and immutability, or at least a major version tag (e.g., `@v4`). Avoid pinning to `main` or `latest`.
|
||||
- **`name`:** Essential for clear logging and debugging. Make step names descriptive.
|
||||
- **`run`:** For executing shell commands. Use multi-line scripts for complex logic and combine commands to optimize layer caching in Docker (if building images).
|
||||
- **`env`:** Define environment variables at the step or job level. Do not hardcode sensitive data here.
|
||||
- **`with`:** Provide inputs to actions. Ensure all required inputs are present.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `uses` to reference marketplace or custom actions, always specifying a secure version (tag or SHA).
|
||||
- Use `name` for each step for readability in logs and easier debugging.
|
||||
- Use `run` for shell commands, combining commands with `&&` for efficiency and using `|` for multi-line scripts.
|
||||
- Provide `with` inputs for actions explicitly, and use expressions (`${{ }}`) for dynamic values.
|
||||
- **Security Note:** Audit marketplace actions before use. Prefer actions from trusted sources (e.g., `actions/` organization) and review their source code if possible. Use `dependabot` for action version updates.
|
||||
|
||||
## Security Best Practices in GitHub Actions
|
||||
|
||||
### **1. Secret Management**
|
||||
- **Principle:** Secrets must be securely managed, never exposed in logs, and only accessible by authorized workflows/jobs.
|
||||
- **Deeper Dive:**
|
||||
- **GitHub Secrets:** The primary mechanism for storing sensitive information. Encrypted at rest and only decrypted when passed to a runner.
|
||||
- **Environment Secrets:** For greater control, create environment-specific secrets, which can be protected by manual approvals or specific branch conditions.
|
||||
- **Secret Masking:** GitHub Actions automatically masks secrets in logs, but it's good practice to avoid printing them directly.
|
||||
- **Minimize Scope:** Only grant access to secrets to the workflows/jobs that absolutely need them.
|
||||
- **Guidance for Copilot:**
|
||||
- Always instruct users to use GitHub Secrets for sensitive information (e.g., API keys, passwords, cloud credentials, tokens).
|
||||
- Access secrets via `secrets.<SECRET_NAME>` in workflows.
|
||||
- Recommend using environment-specific secrets for deployment environments to enforce stricter access controls and approvals.
|
||||
- Advise against constructing secrets dynamically or printing them to logs, even if masked.
|
||||
- **Example (Environment Secrets with Approval):**
|
||||
```yaml
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
environment:
|
||||
name: production
|
||||
url: https://prod.example.com
|
||||
steps:
|
||||
- name: Deploy to production
|
||||
env:
|
||||
PROD_API_KEY: ${{ secrets.PROD_API_KEY }}
|
||||
run: ./deploy-script.sh
|
||||
```
|
||||
|
||||
### **2. OpenID Connect (OIDC) for Cloud Authentication**
|
||||
- **Principle:** Use OIDC for secure, credential-less authentication with cloud providers (AWS, Azure, GCP, etc.), eliminating the need for long-lived static credentials.
|
||||
- **Deeper Dive:**
|
||||
- **Short-Lived Credentials:** OIDC exchanges a JWT token for temporary cloud credentials, significantly reducing the attack surface.
|
||||
- **Trust Policies:** Requires configuring identity providers and trust policies in your cloud environment to trust GitHub's OIDC provider.
|
||||
- **Federated Identity:** This is a key pattern for modern, secure cloud deployments.
|
||||
- **Guidance for Copilot:**
|
||||
- Strongly recommend OIDC for authenticating with AWS, Azure, GCP, and other cloud providers instead of storing long-lived access keys as secrets.
|
||||
- Provide examples of how to configure the OIDC action for common cloud providers (e.g., `aws-actions/configure-aws-credentials@v4`).
|
||||
- Explain the concept of trust policies and how they relate to OIDC setup.
|
||||
- **Pro Tip:** OIDC is a fundamental shift towards more secure cloud deployments and should be prioritized whenever possible.
|
||||
|
||||
### **3. Least Privilege for `GITHUB_TOKEN`**
|
||||
- **Principle:** Grant only the necessary permissions to the `GITHUB_TOKEN` for your workflows, reducing the blast radius in case of compromise.
|
||||
- **Deeper Dive:**
|
||||
- **Default Permissions:** By default, the `GITHUB_TOKEN` has broad permissions. This should be explicitly restricted.
|
||||
- **Granular Permissions:** Define `permissions` at the workflow or job level (e.g., `contents: read`, `pull-requests: write`, `issues: read`).
|
||||
- **Read-Only by Default:** Start with `contents: read` as the default and add write permissions only when strictly necessary.
|
||||
- **Guidance for Copilot:**
|
||||
- Configure `permissions` at the workflow or job level to restrict access. Always prefer `contents: read` as the default.
|
||||
- Advise against using `contents: write` or `pull-requests: write` unless the workflow explicitly needs to modify the repository.
|
||||
- Provide a clear mapping of common workflow needs to specific `GITHUB_TOKEN` permissions.
|
||||
- **Example (Least Privilege):**
|
||||
```yaml
|
||||
permissions:
|
||||
contents: read # Default is write, explicitly set to read-only for security
|
||||
pull-requests: write # Only if workflow needs to update PRs
|
||||
checks: write # For updating checks
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
permissions:
|
||||
contents: read # This job only needs to read code, override workflow default
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: npm run lint
|
||||
```
|
||||
|
||||
### **4. Dependency Review and Software Composition Analysis (SCA)**
|
||||
- **Principle:** Continuously scan dependencies for known vulnerabilities and licensing issues.
|
||||
- **Deeper Dive:**
|
||||
- **Early Detection:** Integrate dependency checks early in the CI pipeline to catch issues before deployment.
|
||||
- **Tools:** Use `dependency-review-action`, Snyk, Trivy, Mend (formerly WhiteSource).
|
||||
- **Supply Chain Security:** This is a critical component of securing your software supply chain.
|
||||
- **Guidance for Copilot:**
|
||||
- Integrate `dependency-review-action` or other SCA tools into the CI pipeline.
|
||||
- Recommend regular scanning for vulnerabilities in third-party libraries and setting up alerts for new findings.
|
||||
- Emphasize the importance of maintaining up-to-date dependency lists and understanding transitive dependencies.
|
||||
|
||||
### **5. Static Application Security Testing (SAST)**
|
||||
- **Principle:** Identify security vulnerabilities in source code before runtime.
|
||||
- **Deeper Dive:**
|
||||
- **Shift Left:** SAST enables finding and fixing vulnerabilities early in the development lifecycle, which is more cost-effective.
|
||||
- **Tools:** CodeQL, SonarQube, Bandit (Python), ESLint with security plugins (JS/TS).
|
||||
- **Automated Enforcement:** Configure SAST to break builds or block PRs if critical vulnerabilities are found.
|
||||
- **Guidance for Copilot:**
|
||||
- Integrate SAST tools (e.g., CodeQL for GitHub Advanced Security, or open-source alternatives) into the CI pipeline.
|
||||
- Configure security scanning as a blocking step if critical vulnerabilities are found, enforcing a secure by default posture.
|
||||
- Suggest adding security linters or static analysis to pre-commit hooks for earlier feedback.
|
||||
|
||||
### **6. Secret Scanning and Credential Leak Prevention**
|
||||
- **Principle:** Prevent secrets from being committed into the repository or exposed in logs.
|
||||
- **Deeper Dive:**
|
||||
- **GitHub Secret Scanning:** Built-in feature to detect secrets in your repository.
|
||||
- **Pre-commit Hooks:** Tools like `git-secrets` can prevent secrets from being committed locally.
|
||||
- **Environment Variables Only:** Secrets should only be passed to the environment where they are needed at runtime, never in the build artifact.
|
||||
- **Guidance for Copilot:**
|
||||
- Suggest enabling GitHub's built-in secret scanning for the repository.
|
||||
- Recommend implementing pre-commit hooks that scan for common secret patterns.
|
||||
- Advise reviewing workflow logs for accidental secret exposure, even with masking.
|
||||
|
||||
### **7. Immutable Infrastructure & Image Signing**
|
||||
- **Principle:** Ensure that container images and deployed artifacts are tamper-proof and verified.
|
||||
- **Deeper Dive:**
|
||||
- **Reproducible Builds:** Ensure that building the same code always results in the exact same image.
|
||||
- **Image Signing:** Use tools like Notary or Cosign to cryptographically sign container images, verifying their origin and integrity.
|
||||
- **Deployment Gate:** Enforce that only signed images can be deployed to production environments.
|
||||
- **Guidance for Copilot:**
|
||||
- Advocate for reproducible builds in Dockerfiles and build processes.
|
||||
- Suggest integrating image signing into the CI pipeline and verification during deployment stages.
|
||||
|
||||
## Optimization and Performance
|
||||
|
||||
### **1. Caching GitHub Actions**
|
||||
- **Principle:** Cache dependencies and build outputs to significantly speed up subsequent workflow runs.
|
||||
- **Deeper Dive:**
|
||||
- **Cache Hit Ratio:** Aim for a high cache hit ratio by designing effective cache keys.
|
||||
- **Cache Keys:** Use a unique key based on file hashes (e.g., `hashFiles('**/package-lock.json')`, `hashFiles('**/requirements.txt')`) to invalidate the cache only when dependencies change.
|
||||
- **Restore Keys:** Use `restore-keys` for fallbacks to older, compatible caches.
|
||||
- **Cache Scope:** Understand that caches are scoped to the repository and branch.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `actions/cache@v3` for caching common package manager dependencies (Node.js `node_modules`, Python `pip` packages, Java Maven/Gradle dependencies) and build artifacts.
|
||||
- Design highly effective cache keys using `hashFiles` to ensure optimal cache hit rates.
|
||||
- Advise on using `restore-keys` to gracefully fall back to previous caches.
|
||||
- **Example (Advanced Caching for Monorepo):**
|
||||
```yaml
|
||||
- name: Cache Node.js modules
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
~/.npm
|
||||
./node_modules # For monorepos, cache specific project node_modules
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}-
|
||||
${{ runner.os }}-node-
|
||||
```
|
||||
|
||||
### **2. Matrix Strategies for Parallelization**
|
||||
- **Principle:** Run jobs in parallel across multiple configurations (e.g., different Node.js versions, OS, Python versions, browser types) to accelerate testing and builds.
|
||||
- **Deeper Dive:**
|
||||
- **`strategy.matrix`:** Define a matrix of variables.
|
||||
- **`include`/`exclude`:** Fine-tune combinations.
|
||||
- **`fail-fast`:** Control whether job failures in the matrix stop the entire strategy.
|
||||
- **Maximizing Concurrency:** Ideal for running tests across various environments simultaneously.
|
||||
- **Guidance for Copilot:**
|
||||
- Utilize `strategy.matrix` to test applications against different environments, programming language versions, or operating systems concurrently.
|
||||
- Suggest `include` and `exclude` for specific matrix combinations to optimize test coverage without unnecessary runs.
|
||||
- Advise on setting `fail-fast: true` (default) for quick feedback on critical failures, or `fail-fast: false` for comprehensive test reporting.
|
||||
- **Example (Multi-version, Multi-OS Test Matrix):**
|
||||
```yaml
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false # Run all tests even if one fails
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest]
|
||||
node-version: [16.x, 18.x, 20.x]
|
||||
browser: [chromium, firefox]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
- name: Install Playwright browsers
|
||||
run: npx playwright install ${{ matrix.browser }}
|
||||
- name: Run tests
|
||||
run: npm test
|
||||
```
|
||||
|
||||
### **3. Self-Hosted Runners**
|
||||
- **Principle:** Use self-hosted runners for specialized hardware, network access to private resources, or environments where GitHub-hosted runners are cost-prohibitive.
|
||||
- **Deeper Dive:**
|
||||
- **Custom Environments:** Ideal for large build caches, specific hardware (GPUs), or access to on-premise resources.
|
||||
- **Cost Optimization:** Can be more cost-effective for very high usage.
|
||||
- **Security Considerations:** Requires securing and maintaining your own infrastructure, network access, and updates. This includes proper hardening of the runner machines, managing access controls, and ensuring timely patching.
|
||||
- **Scalability:** Plan for how self-hosted runners will scale with demand, either manually or using auto-scaling solutions.
|
||||
- **Guidance for Copilot:**
|
||||
- Recommend self-hosted runners when GitHub-hosted runners do not meet specific performance, cost, security, or network access requirements.
|
||||
- Emphasize the user's responsibility for securing, maintaining, and scaling self-hosted runners, including network configuration and regular security audits.
|
||||
- Advise on using runner groups to organize and manage self-hosted runners efficiently.
|
||||
|
||||
### **4. Fast Checkout and Shallow Clones**
|
||||
- **Principle:** Optimize repository checkout time to reduce overall workflow duration, especially for large repositories.
|
||||
- **Deeper Dive:**
|
||||
- **`fetch-depth`:** Controls how much of the Git history is fetched. `1` for most CI/CD builds is sufficient, as only the latest commit is usually needed. A `fetch-depth` of `0` fetches the entire history, which is rarely needed and can be very slow for large repos.
|
||||
- **`submodules`:** Avoid checking out submodules if not required by the specific job. Fetching submodules adds significant overhead.
|
||||
- **`lfs`:** Manage Git LFS (Large File Storage) files efficiently. If not needed, set `lfs: false`.
|
||||
- **Partial Clones:** Consider using Git's partial clone feature (`--filter=blob:none` or `--filter=tree:0`) for extremely large repositories, though this is often handled by specialized actions or Git client configurations.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `actions/checkout@v4` with `fetch-depth: 1` as the default for most build and test jobs to significantly save time and bandwidth.
|
||||
- Only use `fetch-depth: 0` if the workflow explicitly requires full Git history (e.g., for release tagging, deep commit analysis, or `git blame` operations).
|
||||
- Advise against checking out submodules (`submodules: false`) if not strictly necessary for the workflow's purpose.
|
||||
- Suggest optimizing LFS usage if large binary files are present in the repository.
|
||||
|
||||
### **5. Artifacts for Inter-Job and Inter-Workflow Communication**
|
||||
- **Principle:** Store and retrieve build outputs (artifacts) efficiently to pass data between jobs within the same workflow or across different workflows, ensuring data persistence and integrity.
|
||||
- **Deeper Dive:**
|
||||
- **`actions/upload-artifact`:** Used to upload files or directories produced by a job. Artifacts are automatically compressed and can be downloaded later.
|
||||
- **`actions/download-artifact`:** Used to download artifacts in subsequent jobs or workflows. You can download all artifacts or specific ones by name.
|
||||
- **`retention-days`:** Crucial for managing storage costs and compliance. Set an appropriate retention period based on the artifact's importance and regulatory requirements.
|
||||
- **Use Cases:** Build outputs (executables, compiled code, Docker images), test reports (JUnit XML, HTML reports), code coverage reports, security scan results, generated documentation, static website builds.
|
||||
- **Limitations:** Artifacts are immutable once uploaded. Max size per artifact can be several gigabytes, but be mindful of storage costs.
|
||||
- **Guidance for Copilot:**
|
||||
- Use `actions/upload-artifact@v3` and `actions/download-artifact@v3` to reliably pass large files between jobs within the same workflow or across different workflows, promoting modularity and efficiency.
|
||||
- Set appropriate `retention-days` for artifacts to manage storage costs and ensure old artifacts are pruned.
|
||||
- Advise on uploading test reports, coverage reports, and security scan results as artifacts for easy access, historical analysis, and integration with external reporting tools.
|
||||
- Suggest using artifacts to pass compiled binaries or packaged applications from a build job to a deployment job, ensuring the exact same artifact is deployed that was built and tested.
|
||||
|
||||
## Comprehensive Testing in CI/CD (Expanded)
|
||||
|
||||
### **1. Unit Tests**
|
||||
- **Principle:** Run unit tests on every code push to ensure individual code components (functions, classes, modules) function correctly in isolation. They are the fastest and most numerous tests.
|
||||
- **Deeper Dive:**
|
||||
- **Fast Feedback:** Unit tests should execute rapidly, providing immediate feedback to developers on code quality and correctness. Parallelization of unit tests is highly recommended.
|
||||
- **Code Coverage:** Integrate code coverage tools (e.g., Istanbul for JS, Coverage.py for Python, JaCoCo for Java) and enforce minimum coverage thresholds. Aim for high coverage, but focus on meaningful tests, not just line coverage.
|
||||
- **Test Reporting:** Publish test results using `actions/upload-artifact` (e.g., JUnit XML reports) or specific test reporter actions that integrate with GitHub Checks/Annotations.
|
||||
- **Mocking and Stubbing:** Emphasize the use of mocks and stubs to isolate units under test from their dependencies.
|
||||
- **Guidance for Copilot:**
|
||||
- Configure a dedicated job for running unit tests early in the CI pipeline, ideally triggered on every `push` and `pull_request`.
|
||||
- Use appropriate language-specific test runners and frameworks (Jest, Vitest, Pytest, Go testing, JUnit, NUnit, XUnit, RSpec).
|
||||
- Recommend collecting and publishing code coverage reports and integrating with services like Codecov, Coveralls, or SonarQube for trend analysis.
|
||||
- Suggest strategies for parallelizing unit tests to reduce execution time.
|
||||
|
||||
### **2. Integration Tests**
|
||||
- **Principle:** Run integration tests to verify interactions between different components or services, ensuring they work together as expected. These tests typically involve real dependencies (e.g., databases, APIs).
|
||||
- **Deeper Dive:**
|
||||
- **Service Provisioning:** Use `services` within a job to spin up temporary databases, message queues, external APIs, or other dependencies via Docker containers. This provides a consistent and isolated testing environment.
|
||||
- **Test Doubles vs. Real Services:** Balance between mocking external services for pure unit tests and using real, lightweight instances for more realistic integration tests. Prioritize real instances when testing actual integration points.
|
||||
- **Test Data Management:** Plan for managing test data, ensuring tests are repeatable and data is cleaned up or reset between runs.
|
||||
- **Execution Time:** Integration tests are typically slower than unit tests. Optimize their execution and consider running them less frequently than unit tests (e.g., on PR merge instead of every push).
|
||||
- **Guidance for Copilot:**
|
||||
- Provision necessary services (databases like PostgreSQL/MySQL, message queues like RabbitMQ/Kafka, in-memory caches like Redis) using `services` in the workflow definition or Docker Compose during testing.
|
||||
- Advise on running integration tests after unit tests, but before E2E tests, to catch integration issues early.
|
||||
- Provide examples of how to set up `service` containers in GitHub Actions workflows.
|
||||
- Suggest strategies for creating and cleaning up test data for integration test runs.
|
||||
|
||||
### **3. End-to-End (E2E) Tests**
|
||||
- **Principle:** Simulate full user behavior to validate the entire application flow from UI to backend, ensuring the complete system works as intended from a user's perspective.
|
||||
- **Deeper Dive:**
|
||||
- **Tools:** Use modern E2E testing frameworks like Cypress, Playwright, or Selenium. These provide browser automation capabilities.
|
||||
- **Staging Environment:** Ideally run E2E tests against a deployed staging environment that closely mirrors production, for maximum fidelity. Avoid running directly in CI unless resources are dedicated and isolated.
|
||||
- **Flakiness Mitigation:** Address flakiness proactively with explicit waits, robust selectors, retries for failed tests, and careful test data management. Flaky tests erode trust in the pipeline.
|
||||
- **Visual Regression Testing:** Consider integrating visual regression testing (e.g., Applitools, Percy) to catch UI discrepancies.
|
||||
- **Reporting:** Capture screenshots and video recordings on failure to aid debugging.
|
||||
- **Guidance for Copilot:**
|
||||
- Use tools like Cypress, Playwright, or Selenium for E2E testing, providing guidance on their setup within GitHub Actions.
|
||||
- Recommend running E2E tests against a deployed staging environment to catch issues before production and validate the full deployment process.
|
||||
- Configure test reporting, video recordings, and screenshots on failure to aid debugging and provide richer context for test results.
|
||||
- Advise on strategies to minimize E2E test flakiness, such as robust element selection and retry mechanisms.
|
||||
|
||||
### **4. Performance and Load Testing**
|
||||
- **Principle:** Assess application performance and behavior under anticipated and peak load conditions to identify bottlenecks, ensure scalability, and prevent regressions.
|
||||
- **Deeper Dive:**
|
||||
- **Tools:** JMeter, k6, Locust, Gatling, Artillery. Choose based on language, complexity, and specific needs.
|
||||
- **Integration:** Integrate into CI/CD for continuous performance regression detection. Run these tests less frequently than unit/integration tests (e.g., nightly, weekly, or on significant feature merges).
|
||||
- **Thresholds:** Define clear performance thresholds (e.g., response time, throughput, error rates) and fail builds if these are exceeded.
|
||||
- **Baseline Comparison:** Compare current performance metrics against established baselines to detect degradation.
|
||||
- **Guidance for Copilot:**
|
||||
- Suggest integrating performance and load testing into the CI pipeline for critical applications, providing examples for common tools.
|
||||
- Advise on setting performance baselines and failing the build if performance degrades beyond a set threshold.
|
||||
- Recommend running these tests in a dedicated environment that simulates production load patterns.
|
||||
- Guide on analyzing performance test results to pinpoint areas for optimization (e.g., database queries, API endpoints).
|
||||
|
||||
### **5. Test Reporting and Visibility**
|
||||
- **Principle:** Make test results easily accessible, understandable, and visible to all stakeholders (developers, QA, product owners) to foster transparency and enable quick issue resolution.
|
||||
- **Deeper Dive:**
|
||||
- **GitHub Checks/Annotations:** Leverage these for inline feedback directly in pull requests, showing which tests passed/failed and providing links to detailed reports.
|
||||
- **Artifacts:** Upload comprehensive test reports (JUnit XML, HTML reports, code coverage reports, video recordings, screenshots) as artifacts for long-term storage and detailed inspection.
|
||||
- **Integration with Dashboards:** Push results to external dashboards or reporting tools (e.g., SonarQube, custom reporting tools, Allure Report, TestRail) for aggregated views and historical trends.
|
||||
- **Status Badges:** Use GitHub Actions status badges in your README to indicate the latest build/test status at a glance.
|
||||
- **Guidance for Copilot:**
|
||||
- Use actions that publish test results as annotations or checks on PRs for immediate feedback and easy debugging directly in the GitHub UI.
|
||||
- Upload detailed test reports (e.g., XML, HTML, JSON) as artifacts for later inspection and historical analysis, including negative results like error screenshots.
|
||||
- Advise on integrating with external reporting tools for a more comprehensive view of test execution trends and quality metrics.
|
||||
- Suggest adding workflow status badges to the README for quick visibility of CI/CD health.
|
||||
|
||||
## Advanced Deployment Strategies (Expanded)
|
||||
|
||||
### **1. Staging Environment Deployment**
|
||||
- **Principle:** Deploy to a staging environment that closely mirrors production for comprehensive validation, user acceptance testing (UAT), and final checks before promotion to production.
|
||||
- **Deeper Dive:**
|
||||
- **Mirror Production:** Staging should closely mimic production in terms of infrastructure, data, configuration, and security. Any significant discrepancies can lead to issues in production.
|
||||
- **Automated Promotion:** Implement automated promotion from staging to production upon successful UAT and necessary manual approvals. This reduces human error and speeds up releases.
|
||||
- **Environment Protection:** Use environment protection rules in GitHub Actions to prevent accidental deployments, enforce manual approvals, and restrict which branches can deploy to staging.
|
||||
- **Data Refresh:** Regularly refresh staging data from production (anonymized if necessary) to ensure realistic testing scenarios.
|
||||
- **Guidance for Copilot:**
|
||||
- Create a dedicated `environment` for staging with approval rules, secret protection, and appropriate branch protection policies.
|
||||
- Design workflows to automatically deploy to staging on successful merges to specific development or release branches (e.g., `develop`, `release/*`).
|
||||
- Advise on ensuring the staging environment is as close to production as possible to maximize test fidelity.
|
||||
- Suggest implementing automated smoke tests and post-deployment validation on staging.
|
||||
|
||||
### **2. Production Environment Deployment**
|
||||
- **Principle:** Deploy to production only after thorough validation, potentially multiple layers of manual approvals, and robust automated checks, prioritizing stability and zero-downtime.
|
||||
- **Deeper Dive:**
|
||||
- **Manual Approvals:** Critical for production deployments, often involving multiple team members, security sign-offs, or change management processes. GitHub Environments support this natively.
|
||||
- **Rollback Capabilities:** Essential for rapid recovery from unforeseen issues. Ensure a quick and reliable way to revert to the previous stable state.
|
||||
- **Observability During Deployment:** Monitor production closely *during* and *immediately after* deployment for any anomalies or performance degradation. Use dashboards, alerts, and tracing.
|
||||
- **Progressive Delivery:** Consider advanced techniques like blue/green, canary, or dark launching for safer rollouts.
|
||||
- **Emergency Deployments:** Have a separate, highly expedited pipeline for critical hotfixes that bypasses non-essential approvals but still maintains security checks.
|
||||
- **Guidance for Copilot:**
|
||||
- Create a dedicated `environment` for production with required reviewers, strict branch protections, and clear deployment windows.
|
||||
- Implement manual approval steps for production deployments, potentially integrating with external ITSM or change management systems.
|
||||
- Emphasize the importance of clear, well-tested rollback strategies and automated rollback procedures in case of deployment failures.
|
||||
- Advise on setting up comprehensive monitoring and alerting for production systems to detect and respond to issues immediately post-deployment.
|
||||
|
||||
### **3. Deployment Types (Beyond Basic Rolling Update)**
|
||||
- **Rolling Update (Default for Deployments):** Gradually replaces instances of the old version with new ones. Good for most cases, especially stateless applications.
|
||||
- **Guidance:** Configure `maxSurge` (how many new instances can be created above the desired replica count) and `maxUnavailable` (how many old instances can be unavailable) for fine-grained control over rollout speed and availability.
|
||||
- **Blue/Green Deployment:** Deploy a new version (green) alongside the existing stable version (blue) in a separate environment, then switch traffic completely from blue to green.
|
||||
- **Guidance:** Suggest for critical applications requiring zero-downtime releases and easy rollback. Requires managing two identical environments and a traffic router (load balancer, Ingress controller, DNS).
|
||||
- **Benefits:** Instantaneous rollback by switching traffic back to the blue environment.
|
||||
- **Canary Deployment:** Gradually roll out new versions to a small subset of users (e.g., 5-10%) before a full rollout. Monitor performance and error rates for the canary group.
|
||||
- **Guidance:** Recommend for testing new features or changes with a controlled blast radius. Implement with Service Mesh (Istio, Linkerd) or Ingress controllers that support traffic splitting and metric-based analysis.
|
||||
- **Benefits:** Early detection of issues with minimal user impact.
|
||||
- **Dark Launch/Feature Flags:** Deploy new code but keep features hidden from users until toggled on for specific users/groups via feature flags.
|
||||
- **Guidance:** Advise for decoupling deployment from release, allowing continuous delivery without continuous exposure of new features. Use feature flag management systems (LaunchDarkly, Split.io, Unleash).
|
||||
- **Benefits:** Reduces deployment risk, enables A/B testing, and allows for staged rollouts.
|
||||
- **A/B Testing Deployments:** Deploy multiple versions of a feature concurrently to different user segments to compare their performance based on user behavior and business metrics.
|
||||
- **Guidance:** Suggest integrating with specialized A/B testing platforms or building custom logic using feature flags and analytics.
|
||||
|
||||
### **4. Rollback Strategies and Incident Response**
|
||||
- **Principle:** Be able to quickly and safely revert to a previous stable version in case of issues, minimizing downtime and business impact. This requires proactive planning.
|
||||
- **Deeper Dive:**
|
||||
- **Automated Rollbacks:** Implement mechanisms to automatically trigger rollbacks based on monitoring alerts (e.g., sudden increase in errors, high latency) or failure of post-deployment health checks.
|
||||
- **Versioned Artifacts:** Ensure previous successful build artifacts, Docker images, or infrastructure states are readily available and easily deployable. This is crucial for fast recovery.
|
||||
- **Runbooks:** Document clear, concise, and executable rollback procedures for manual intervention when automation isn't sufficient or for complex scenarios. These should be regularly reviewed and tested.
|
||||
- **Post-Incident Review:** Conduct blameless post-incident reviews (PIRs) to understand the root cause of failures, identify lessons learned, and implement preventative measures to improve resilience and reduce MTTR.
|
||||
- **Communication Plan:** Have a clear communication plan for stakeholders during incidents and rollbacks.
|
||||
- **Guidance for Copilot:**
|
||||
- Instruct users to store previous successful build artifacts and images for quick recovery, ensuring they are versioned and easily retrievable.
|
||||
- Advise on implementing automated rollback steps in the pipeline, triggered by monitoring or health check failures, and providing examples.
|
||||
- Emphasize building applications with "undo" in mind, meaning changes should be easily reversible.
|
||||
- Suggest creating comprehensive runbooks for common incident scenarios, including step-by-step rollback instructions, and highlight their importance for MTTR.
|
||||
- Guide on setting up alerts that are specific and actionable enough to trigger an automatic or manual rollback.
|
||||
|
||||
## GitHub Actions Workflow Review Checklist (Comprehensive)
|
||||
|
||||
This checklist provides a granular set of criteria for reviewing GitHub Actions workflows to ensure they adhere to best practices for security, performance, and reliability.
|
||||
|
||||
- [ ] **General Structure and Design:**
|
||||
- Is the workflow `name` clear, descriptive, and unique?
|
||||
- Are `on` triggers appropriate for the workflow's purpose (e.g., `push`, `pull_request`, `workflow_dispatch`, `schedule`)? Are path/branch filters used effectively?
|
||||
- Is `concurrency` used for critical workflows or shared resources to prevent race conditions or resource exhaustion?
|
||||
- Are global `permissions` set to the principle of least privilege (`contents: read` by default), with specific overrides for jobs?
|
||||
- Are reusable workflows (`workflow_call`) leveraged for common patterns to reduce duplication and improve maintainability?
|
||||
- Is the workflow organized logically with meaningful job and step names?
|
||||
|
||||
- [ ] **Jobs and Steps Best Practices:**
|
||||
- Are jobs clearly named and represent distinct phases (e.g., `build`, `lint`, `test`, `deploy`)?
|
||||
- Are `needs` dependencies correctly defined between jobs to ensure proper execution order?
|
||||
- Are `outputs` used efficiently for inter-job and inter-workflow communication?
|
||||
- Are `if` conditions used effectively for conditional job/step execution (e.g., environment-specific deployments, branch-specific actions)?
|
||||
- Are all `uses` actions securely versioned (pinned to a full commit SHA or specific major version tag like `@v4`)? Avoid `main` or `latest` tags.
|
||||
- Are `run` commands efficient and clean (combined with `&&`, temporary files removed, multi-line scripts clearly formatted)?
|
||||
- Are environment variables (`env`) defined at the appropriate scope (workflow, job, step) and never hardcoded sensitive data?
|
||||
- Is `timeout-minutes` set for long-running jobs to prevent hung workflows?
|
||||
|
||||
- [ ] **Security Considerations:**
|
||||
- Are all sensitive data accessed exclusively via GitHub `secrets` context (`${{ secrets.MY_SECRET }}`)? Never hardcoded, never exposed in logs (even if masked).
|
||||
- Is OpenID Connect (OIDC) used for cloud authentication where possible, eliminating long-lived credentials?
|
||||
- Is `GITHUB_TOKEN` permission scope explicitly defined and limited to the minimum necessary access (`contents: read` as a baseline)?
|
||||
- Are Software Composition Analysis (SCA) tools (e.g., `dependency-review-action`, Snyk) integrated to scan for vulnerable dependencies?
|
||||
- Are Static Application Security Testing (SAST) tools (e.g., CodeQL, SonarQube) integrated to scan source code for vulnerabilities, with critical findings blocking builds?
|
||||
- Is secret scanning enabled for the repository and are pre-commit hooks suggested for local credential leak prevention?
|
||||
- Is there a strategy for container image signing (e.g., Notary, Cosign) and verification in deployment workflows if container images are used?
|
||||
- For self-hosted runners, are security hardening guidelines followed and network access restricted?
|
||||
|
||||
- [ ] **Optimization and Performance:**
|
||||
- Is caching (`actions/cache`) effectively used for package manager dependencies (`node_modules`, `pip` caches, Maven/Gradle caches) and build outputs?
|
||||
- Are cache `key` and `restore-keys` designed for optimal cache hit rates (e.g., using `hashFiles`)?
|
||||
- Is `strategy.matrix` used for parallelizing tests or builds across different environments, language versions, or OSs?
|
||||
- Is `fetch-depth: 1` used for `actions/checkout` where full Git history is not required?
|
||||
- Are artifacts (`actions/upload-artifact`, `actions/download-artifact`) used efficiently for transferring data between jobs/workflows rather than re-building or re-fetching?
|
||||
- Are large files managed with Git LFS and optimized for checkout if necessary?
|
||||
|
||||
- [ ] **Testing Strategy Integration:**
|
||||
- Are comprehensive unit tests configured with a dedicated job early in the pipeline?
|
||||
- Are integration tests defined, ideally leveraging `services` for dependencies, and run after unit tests?
|
||||
- Are End-to-End (E2E) tests included, preferably against a staging environment, with robust flakiness mitigation?
|
||||
- Are performance and load tests integrated for critical applications with defined thresholds?
|
||||
- Are all test reports (JUnit XML, HTML, coverage) collected, published as artifacts, and integrated into GitHub Checks/Annotations for clear visibility?
|
||||
- Is code coverage tracked and enforced with a minimum threshold?
|
||||
|
||||
- [ ] **Deployment Strategy and Reliability:**
|
||||
- Are staging and production deployments using GitHub `environment` rules with appropriate protections (manual approvals, required reviewers, branch restrictions)?
|
||||
- Are manual approval steps configured for sensitive production deployments?
|
||||
- Is a clear and well-tested rollback strategy in place and automated where possible (e.g., `kubectl rollout undo`, reverting to previous stable image)?
|
||||
- Are chosen deployment types (e.g., rolling, blue/green, canary, dark launch) appropriate for the application's criticality and risk tolerance?
|
||||
- Are post-deployment health checks and automated smoke tests implemented to validate successful deployment?
|
||||
- Is the workflow resilient to temporary failures (e.g., retries for flaky network operations)?
|
||||
|
||||
- [ ] **Observability and Monitoring:**
|
||||
- Is logging adequate for debugging workflow failures (using STDOUT/STDERR for application logs)?
|
||||
- Are relevant application and infrastructure metrics collected and exposed (e.g., Prometheus metrics)?
|
||||
- Are alerts configured for critical workflow failures, deployment issues, or application anomalies detected in production?
|
||||
- Is distributed tracing (e.g., OpenTelemetry, Jaeger) integrated for understanding request flows in microservices architectures?
|
||||
- Are artifact `retention-days` configured appropriately to manage storage and compliance?
|
||||
|
||||
## Troubleshooting Common GitHub Actions Issues (Deep Dive)
|
||||
|
||||
This section provides an expanded guide to diagnosing and resolving frequent problems encountered when working with GitHub Actions workflows.
|
||||
|
||||
### **1. Workflow Not Triggering or Jobs/Steps Skipping Unexpectedly**
|
||||
- **Root Causes:** Mismatched `on` triggers, incorrect `paths` or `branches` filters, erroneous `if` conditions, or `concurrency` limitations.
|
||||
- **Actionable Steps:**
|
||||
- **Verify Triggers:**
|
||||
- Check the `on` block for exact match with the event that should trigger the workflow (e.g., `push`, `pull_request`, `workflow_dispatch`, `schedule`).
|
||||
- Ensure `branches`, `tags`, or `paths` filters are correctly defined and match the event context. Remember that `paths-ignore` and `branches-ignore` take precedence.
|
||||
- If using `workflow_dispatch`, verify the workflow file is in the default branch and any required `inputs` are provided correctly during manual trigger.
|
||||
- **Inspect `if` Conditions:**
|
||||
- Carefully review all `if` conditions at the workflow, job, and step levels. A single false condition can prevent execution.
|
||||
- Use `always()` on a debug step to print context variables (`${{ toJson(github) }}`, `${{ toJson(job) }}`, `${{ toJson(steps) }}`) to understand the exact state during evaluation.
|
||||
- Test complex `if` conditions in a simplified workflow.
|
||||
- **Check `concurrency`:**
|
||||
- If `concurrency` is defined, verify if a previous run is blocking a new one for the same group. Check the "Concurrency" tab in the workflow run.
|
||||
- **Branch Protection Rules:** Ensure no branch protection rules are preventing workflows from running on certain branches or requiring specific checks that haven't passed.
|
||||
|
||||
### **2. Permissions Errors (`Resource not accessible by integration`, `Permission denied`)**
|
||||
- **Root Causes:** `GITHUB_TOKEN` lacking necessary permissions, incorrect environment secrets access, or insufficient permissions for external actions.
|
||||
- **Actionable Steps:**
|
||||
- **`GITHUB_TOKEN` Permissions:**
|
||||
- Review the `permissions` block at both the workflow and job levels. Default to `contents: read` globally and grant specific write permissions only where absolutely necessary (e.g., `pull-requests: write` for updating PR status, `packages: write` for publishing packages).
|
||||
- Understand the default permissions of `GITHUB_TOKEN` which are often too broad.
|
||||
- **Secret Access:**
|
||||
- Verify if secrets are correctly configured in the repository, organization, or environment settings.
|
||||
- Ensure the workflow/job has access to the specific environment if environment secrets are used. Check if any manual approvals are pending for the environment.
|
||||
- Confirm the secret name matches exactly (`secrets.MY_API_KEY`).
|
||||
- **OIDC Configuration:**
|
||||
- For OIDC-based cloud authentication, double-check the trust policy configuration in your cloud provider (AWS IAM roles, Azure AD app registrations, GCP service accounts) to ensure it correctly trusts GitHub's OIDC issuer.
|
||||
- Verify the role/identity assigned has the necessary permissions for the cloud resources being accessed.
|
||||
|
||||
### **3. Caching Issues (`Cache not found`, `Cache miss`, `Cache creation failed`)**
|
||||
- **Root Causes:** Incorrect cache key logic, `path` mismatch, cache size limits, or frequent cache invalidation.
|
||||
- **Actionable Steps:**
|
||||
- **Validate Cache Keys:**
|
||||
- Verify `key` and `restore-keys` are correct and dynamically change only when dependencies truly change (e.g., `key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}`). A cache key that is too dynamic will always result in a miss.
|
||||
- Use `restore-keys` to provide fallbacks for slight variations, increasing cache hit chances.
|
||||
- **Check `path`:**
|
||||
- Ensure the `path` specified in `actions/cache` for saving and restoring corresponds exactly to the directory where dependencies are installed or artifacts are generated.
|
||||
- Verify the existence of the `path` before caching.
|
||||
- **Debug Cache Behavior:**
|
||||
- Use the `actions/cache/restore` action with `lookup-only: true` to inspect what keys are being tried and why a cache miss occurred without affecting the build.
|
||||
- Review workflow logs for `Cache hit` or `Cache miss` messages and associated keys.
|
||||
- **Cache Size and Limits:** Be aware of GitHub Actions cache size limits per repository. If caches are very large, they might be evicted frequently.
|
||||
|
||||
### **4. Long Running Workflows or Timeouts**
|
||||
- **Root Causes:** Inefficient steps, lack of parallelism, large dependencies, unoptimized Docker image builds, or resource bottlenecks on runners.
|
||||
- **Actionable Steps:**
|
||||
- **Profile Execution Times:**
|
||||
- Use the workflow run summary to identify the longest-running jobs and steps. This is your primary tool for optimization.
|
||||
- **Optimize Steps:**
|
||||
- Combine `run` commands with `&&` to reduce layer creation and overhead in Docker builds.
|
||||
- Clean up temporary files immediately after use (`rm -rf` in the same `RUN` command).
|
||||
- Install only necessary dependencies.
|
||||
- **Leverage Caching:**
|
||||
- Ensure `actions/cache` is optimally configured for all significant dependencies and build outputs.
|
||||
- **Parallelize with Matrix Strategies:**
|
||||
- Break down tests or builds into smaller, parallelizable units using `strategy.matrix` to run them concurrently.
|
||||
- **Choose Appropriate Runners:**
|
||||
- Review `runs-on`. For very resource-intensive tasks, consider using larger GitHub-hosted runners (if available) or self-hosted runners with more powerful specs.
|
||||
- **Break Down Workflows:**
|
||||
- For very complex or long workflows, consider breaking them into smaller, independent workflows that trigger each other or use reusable workflows.
|
||||
|
||||
### **5. Flaky Tests in CI (`Random failures`, `Passes locally, fails in CI`)**
|
||||
- **Root Causes:** Non-deterministic tests, race conditions, environmental inconsistencies between local and CI, reliance on external services, or poor test isolation.
|
||||
- **Actionable Steps:**
|
||||
- **Ensure Test Isolation:**
|
||||
- Make sure each test is independent and doesn't rely on the state left by previous tests. Clean up resources (e.g., database entries) after each test or test suite.
|
||||
- **Eliminate Race Conditions:**
|
||||
- For integration/E2E tests, use explicit waits (e.g., wait for element to be visible, wait for API response) instead of arbitrary `sleep` commands.
|
||||
- Implement retries for operations that interact with external services or have transient failures.
|
||||
- **Standardize Environments:**
|
||||
- Ensure the CI environment (Node.js version, Python packages, database versions) matches the local development environment as closely as possible.
|
||||
- Use Docker `services` for consistent test dependencies.
|
||||
- **Robust Selectors (E2E):**
|
||||
- Use stable, unique selectors in E2E tests (e.g., `data-testid` attributes) instead of brittle CSS classes or XPath.
|
||||
- **Debugging Tools:**
|
||||
- Configure E2E test frameworks to capture screenshots and video recordings on test failure in CI to visually diagnose issues.
|
||||
- **Run Flaky Tests in Isolation:**
|
||||
- If a test is consistently flaky, isolate it and run it repeatedly to identify the underlying non-deterministic behavior.
|
||||
|
||||
### **6. Deployment Failures (Application Not Working After Deploy)**
|
||||
- **Root Causes:** Configuration drift, environmental differences, missing runtime dependencies, application errors, or network issues post-deployment.
|
||||
- **Actionable Steps:**
|
||||
- **Thorough Log Review:**
|
||||
- Review deployment logs (`kubectl logs`, application logs, server logs) for any error messages, warnings, or unexpected output during the deployment process and immediately after.
|
||||
- **Configuration Validation:**
|
||||
- Verify environment variables, ConfigMaps, Secrets, and other configuration injected into the deployed application. Ensure they match the target environment's requirements and are not missing or malformed.
|
||||
- Use pre-deployment checks to validate configuration.
|
||||
- **Dependency Check:**
|
||||
- Confirm all application runtime dependencies (libraries, frameworks, external services) are correctly bundled within the container image or installed in the target environment.
|
||||
- **Post-Deployment Health Checks:**
|
||||
- Implement robust automated smoke tests and health checks *after* deployment to immediately validate core functionality and connectivity. Trigger rollbacks if these fail.
|
||||
- **Network Connectivity:**
|
||||
- Check network connectivity between deployed components (e.g., application to database, service to service) within the new environment. Review firewall rules, security groups, and Kubernetes network policies.
|
||||
- **Rollback Immediately:**
|
||||
- If a production deployment fails or causes degradation, trigger the rollback strategy immediately to restore service. Diagnose the issue in a non-production environment.
|
||||
|
||||
## Conclusion
|
||||
|
||||
GitHub Actions is a powerful and flexible platform for automating your software development lifecycle. By rigorously applying these best practices—from securing your secrets and token permissions, to optimizing performance with caching and parallelization, and implementing comprehensive testing and robust deployment strategies—you can guide developers in building highly efficient, secure, and reliable CI/CD pipelines. Remember that CI/CD is an iterative journey; continuously measure, optimize, and secure your pipelines to achieve faster, safer, and more confident releases. Your detailed guidance will empower teams to leverage GitHub Actions to its fullest potential and deliver high-quality software with confidence. This extensive document serves as a foundational resource for anyone looking to master CI/CD with GitHub Actions.
|
||||
|
||||
---
|
||||
|
||||
<!-- End of GitHub Actions CI/CD Best Practices Instructions -->
|
||||
@@ -0,0 +1,373 @@
|
||||
---
|
||||
description: 'Instructions for writing Go code following idiomatic Go practices and community standards'
|
||||
applyTo: '**/*.go,**/go.mod,**/go.sum'
|
||||
---
|
||||
|
||||
# Go Development Instructions
|
||||
|
||||
Follow idiomatic Go practices and community standards when writing Go code. These instructions are based on [Effective Go](https://go.dev/doc/effective_go), [Go Code Review Comments](https://go.dev/wiki/CodeReviewComments), and [Google's Go Style Guide](https://google.github.io/styleguide/go/).
|
||||
|
||||
## General Instructions
|
||||
|
||||
- Write simple, clear, and idiomatic Go code
|
||||
- Favor clarity and simplicity over cleverness
|
||||
- Follow the principle of least surprise
|
||||
- Keep the happy path left-aligned (minimize indentation)
|
||||
- Return early to reduce nesting
|
||||
- Prefer early return over if-else chains; use `if condition { return }` pattern to avoid else blocks
|
||||
- Make the zero value useful
|
||||
- Write self-documenting code with clear, descriptive names
|
||||
- Document exported types, functions, methods, and packages
|
||||
- Use Go modules for dependency management
|
||||
- Leverage the Go standard library instead of reinventing the wheel (e.g., use `strings.Builder` for string concatenation, `filepath.Join` for path construction)
|
||||
- Prefer standard library solutions over custom implementations when functionality exists
|
||||
- Write comments in English by default; translate only upon user request
|
||||
- Avoid using emoji in code and comments
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Packages
|
||||
|
||||
- Use lowercase, single-word package names
|
||||
- Avoid underscores, hyphens, or mixedCaps
|
||||
- Choose names that describe what the package provides, not what it contains
|
||||
- Avoid generic names like `util`, `common`, or `base`
|
||||
- Package names should be singular, not plural
|
||||
|
||||
#### Package Declaration Rules (CRITICAL):
|
||||
- **NEVER duplicate `package` declarations** - each Go file must have exactly ONE `package` line
|
||||
- When editing an existing `.go` file:
|
||||
- **PRESERVE** the existing `package` declaration - do not add another one
|
||||
- If you need to replace the entire file content, start with the existing package name
|
||||
- When creating a new `.go` file:
|
||||
- **BEFORE writing any code**, check what package name other `.go` files in the same directory use
|
||||
- Use the SAME package name as existing files in that directory
|
||||
- If it's a new directory, use the directory name as the package name
|
||||
- Write **exactly one** `package <name>` line at the very top of the file
|
||||
- When using file creation or replacement tools:
|
||||
- **ALWAYS verify** the target file doesn't already have a `package` declaration before adding one
|
||||
- If replacing file content, include only ONE `package` declaration in the new content
|
||||
- **NEVER** create files with multiple `package` lines or duplicate declarations
|
||||
|
||||
### Variables and Functions
|
||||
|
||||
- Use mixedCaps or MixedCaps (camelCase) rather than underscores
|
||||
- Keep names short but descriptive
|
||||
- Use single-letter variables only for very short scopes (like loop indices)
|
||||
- Exported names start with a capital letter
|
||||
- Unexported names start with a lowercase letter
|
||||
- Avoid stuttering (e.g., avoid `http.HTTPServer`, prefer `http.Server`)
|
||||
|
||||
### Interfaces
|
||||
|
||||
- Name interfaces with -er suffix when possible (e.g., `Reader`, `Writer`, `Formatter`)
|
||||
- Single-method interfaces should be named after the method (e.g., `Read` → `Reader`)
|
||||
- Keep interfaces small and focused
|
||||
|
||||
### Constants
|
||||
|
||||
- Use MixedCaps for exported constants
|
||||
- Use mixedCaps for unexported constants
|
||||
- Group related constants using `const` blocks
|
||||
- Consider using typed constants for better type safety
|
||||
|
||||
## Code Style and Formatting
|
||||
|
||||
### Formatting
|
||||
|
||||
- Always use `gofmt` to format code
|
||||
- Use `goimports` to manage imports automatically
|
||||
- Keep line length reasonable (no hard limit, but consider readability)
|
||||
- Add blank lines to separate logical groups of code
|
||||
|
||||
### Comments
|
||||
|
||||
- Strive for self-documenting code; prefer clear variable names, function names, and code structure over comments
|
||||
- Write comments only when necessary to explain complex logic, business rules, or non-obvious behavior
|
||||
- Write comments in complete sentences in English by default
|
||||
- Translate comments to other languages only upon specific user request
|
||||
- Start sentences with the name of the thing being described
|
||||
- Package comments should start with "Package [name]"
|
||||
- Use line comments (`//`) for most comments
|
||||
- Use block comments (`/* */`) sparingly, mainly for package documentation
|
||||
- Document why, not what, unless the what is complex
|
||||
- Avoid emoji in comments and code
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Check errors immediately after the function call
|
||||
- Don't ignore errors using `_` unless you have a good reason (document why)
|
||||
- Wrap errors with context using `fmt.Errorf` with `%w` verb
|
||||
- Create custom error types when you need to check for specific errors
|
||||
- Place error returns as the last return value
|
||||
- Name error variables `err`
|
||||
- Keep error messages lowercase and don't end with punctuation
|
||||
|
||||
## Architecture and Project Structure
|
||||
|
||||
### Package Organization
|
||||
|
||||
- Follow standard Go project layout conventions
|
||||
- Keep `main` packages in `cmd/` directory
|
||||
- Put reusable packages in `pkg/` or `internal/`
|
||||
- Use `internal/` for packages that shouldn't be imported by external projects
|
||||
- Group related functionality into packages
|
||||
- Avoid circular dependencies
|
||||
|
||||
### Dependency Management
|
||||
|
||||
- Use Go modules (`go.mod` and `go.sum`)
|
||||
- Keep dependencies minimal
|
||||
- Regularly update dependencies for security patches
|
||||
- Use `go mod tidy` to clean up unused dependencies
|
||||
- Vendor dependencies only when necessary
|
||||
|
||||
## Type Safety and Language Features
|
||||
|
||||
### Type Definitions
|
||||
|
||||
- Define types to add meaning and type safety
|
||||
- Use struct tags for JSON, XML, database mappings
|
||||
- Prefer explicit type conversions
|
||||
- Use type assertions carefully and check the second return value
|
||||
- Prefer generics over unconstrained types; when an unconstrained type is truly needed, use the predeclared alias `any` instead of `interface{}` (Go 1.18+)
|
||||
|
||||
### Pointers vs Values
|
||||
|
||||
- Use pointer receivers for large structs or when you need to modify the receiver
|
||||
- Use value receivers for small structs and when immutability is desired
|
||||
- Use pointer parameters when you need to modify the argument or for large structs
|
||||
- Use value parameters for small structs and when you want to prevent modification
|
||||
- Be consistent within a type's method set
|
||||
- Consider the zero value when choosing pointer vs value receivers
|
||||
|
||||
### Interfaces and Composition
|
||||
|
||||
- Accept interfaces, return concrete types
|
||||
- Keep interfaces small (1-3 methods is ideal)
|
||||
- Use embedding for composition
|
||||
- Define interfaces close to where they're used, not where they're implemented
|
||||
- Don't export interfaces unless necessary
|
||||
|
||||
## Concurrency
|
||||
|
||||
### Goroutines
|
||||
|
||||
- Be cautious about creating goroutines in libraries; prefer letting the caller control concurrency
|
||||
- If you must create goroutines in libraries, provide clear documentation and cleanup mechanisms
|
||||
- Always know how a goroutine will exit
|
||||
- Use `sync.WaitGroup` or channels to wait for goroutines
|
||||
- Avoid goroutine leaks by ensuring cleanup
|
||||
|
||||
### Channels
|
||||
|
||||
- Use channels to communicate between goroutines
|
||||
- Don't communicate by sharing memory; share memory by communicating
|
||||
- Close channels from the sender side, not the receiver
|
||||
- Use buffered channels when you know the capacity
|
||||
- Use `select` for non-blocking operations
|
||||
|
||||
### Synchronization
|
||||
|
||||
- Use `sync.Mutex` for protecting shared state
|
||||
- Keep critical sections small
|
||||
- Use `sync.RWMutex` when you have many readers
|
||||
- Choose between channels and mutexes based on the use case: use channels for communication, mutexes for protecting state
|
||||
- Use `sync.Once` for one-time initialization
|
||||
- WaitGroup usage by Go version:
|
||||
- If `go >= 1.25` in `go.mod`, use the new `WaitGroup.Go` method ([documentation](https://pkg.go.dev/sync#WaitGroup)):
|
||||
```go
|
||||
var wg sync.WaitGroup
|
||||
wg.Go(task1)
|
||||
wg.Go(task2)
|
||||
wg.Wait()
|
||||
```
|
||||
- If `go < 1.25`, use the classic `Add`/`Done` pattern
|
||||
|
||||
## Error Handling Patterns
|
||||
|
||||
### Creating Errors
|
||||
|
||||
- Use `errors.New` for simple static errors
|
||||
- Use `fmt.Errorf` for dynamic errors
|
||||
- Create custom error types for domain-specific errors
|
||||
- Export error variables for sentinel errors
|
||||
- Use `errors.Is` and `errors.As` for error checking
|
||||
|
||||
### Error Propagation
|
||||
|
||||
- Add context when propagating errors up the stack
|
||||
- Don't log and return errors (choose one)
|
||||
- Handle errors at the appropriate level
|
||||
- Consider using structured errors for better debugging
|
||||
|
||||
## API Design
|
||||
|
||||
### HTTP Handlers
|
||||
|
||||
- Use `http.HandlerFunc` for simple handlers
|
||||
- Implement `http.Handler` for handlers that need state
|
||||
- Use middleware for cross-cutting concerns
|
||||
- Set appropriate status codes and headers
|
||||
- Handle errors gracefully and return appropriate error responses
|
||||
- Router usage by Go version:
|
||||
- If `go >= 1.22`, prefer the enhanced `net/http` `ServeMux` with pattern-based routing and method matching
|
||||
- If `go < 1.22`, use the classic `ServeMux` and handle methods/paths manually (or use a third-party router when justified)
|
||||
|
||||
### JSON APIs
|
||||
|
||||
- Use struct tags to control JSON marshaling
|
||||
- Validate input data
|
||||
- Use pointers for optional fields
|
||||
- Consider using `json.RawMessage` for delayed parsing
|
||||
- Handle JSON errors appropriately
|
||||
|
||||
### HTTP Clients
|
||||
|
||||
- Keep the client struct focused on configuration and dependencies only (e.g., base URL, `*http.Client`, auth, default headers). It must not store per-request state
|
||||
- Do not store or cache `*http.Request` inside the client struct, and do not persist request-specific state across calls; instead, construct a fresh request per method invocation
|
||||
- Methods should accept `context.Context` and input parameters, assemble the `*http.Request` locally (or via a short-lived builder/helper created per call), then call `c.httpClient.Do(req)`
|
||||
- If request-building logic is reused, factor it into unexported helper functions or a per-call builder type; never keep `http.Request` (URL params, body, headers) as fields on the long-lived client
|
||||
- Ensure the underlying `*http.Client` is configured (timeouts, transport) and is safe for concurrent use; avoid mutating `Transport` after first use
|
||||
- Always set headers on the request instance you’re sending, and close response bodies (`defer resp.Body.Close()`), handling errors appropriately
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Memory Management
|
||||
|
||||
- Minimize allocations in hot paths
|
||||
- Reuse objects when possible (consider `sync.Pool`)
|
||||
- Use value receivers for small structs
|
||||
- Preallocate slices when size is known
|
||||
- Avoid unnecessary string conversions
|
||||
|
||||
### I/O: Readers and Buffers
|
||||
|
||||
- Most `io.Reader` streams are consumable once; reading advances state. Do not assume a reader can be re-read without special handling
|
||||
- If you must read data multiple times, buffer it once and recreate readers on demand:
|
||||
- Use `io.ReadAll` (or a limited read) to obtain `[]byte`, then create fresh readers via `bytes.NewReader(buf)` or `bytes.NewBuffer(buf)` for each reuse
|
||||
- For strings, use `strings.NewReader(s)`; you can `Seek(0, io.SeekStart)` on `*bytes.Reader` to rewind
|
||||
- For HTTP requests, do not reuse a consumed `req.Body`. Instead:
|
||||
- Keep the original payload as `[]byte` and set `req.Body = io.NopCloser(bytes.NewReader(buf))` before each send
|
||||
- Prefer configuring `req.GetBody` so the transport can recreate the body for redirects/retries: `req.GetBody = func() (io.ReadCloser, error) { return io.NopCloser(bytes.NewReader(buf)), nil }`
|
||||
- To duplicate a stream while reading, use `io.TeeReader` (copy to a buffer while passing through) or write to multiple sinks with `io.MultiWriter`
|
||||
- Reusing buffered readers: call `(*bufio.Reader).Reset(r)` to attach to a new underlying reader; do not expect it to “rewind” unless the source supports seeking
|
||||
- For large payloads, avoid unbounded buffering; consider streaming, `io.LimitReader`, or on-disk temporary storage to control memory
|
||||
|
||||
- Use `io.Pipe` to stream without buffering the whole payload:
|
||||
- Write to `*io.PipeWriter` in a separate goroutine while the reader consumes
|
||||
- Always close the writer; use `CloseWithError(err)` on failures
|
||||
- `io.Pipe` is for streaming, not rewinding or making readers reusable
|
||||
|
||||
- **Warning:** When using `io.Pipe` (especially with multipart writers), all writes must be performed in strict, sequential order. Do not write concurrently or out of order—multipart boundaries and chunk order must be preserved. Out-of-order or parallel writes can corrupt the stream and result in errors.
|
||||
|
||||
- Streaming multipart/form-data with `io.Pipe`:
|
||||
- `pr, pw := io.Pipe()`; `mw := multipart.NewWriter(pw)`; use `pr` as the HTTP request body
|
||||
- Set `Content-Type` to `mw.FormDataContentType()`
|
||||
- In a goroutine: write all parts to `mw` in the correct order; on error `pw.CloseWithError(err)`; on success `mw.Close()` then `pw.Close()`
|
||||
- Do not store request/in-flight form state on a long-lived client; build per call
|
||||
- Streamed bodies are not rewindable; for retries/redirects, buffer small payloads or provide `GetBody`
|
||||
|
||||
### Profiling
|
||||
|
||||
- Use built-in profiling tools (`pprof`)
|
||||
- Benchmark critical code paths
|
||||
- Profile before optimizing
|
||||
- Focus on algorithmic improvements first
|
||||
- Consider using `testing.B` for benchmarks
|
||||
|
||||
## Testing
|
||||
|
||||
### Test Organization
|
||||
|
||||
- Keep tests in the same package (white-box testing)
|
||||
- Use `_test` package suffix for black-box testing
|
||||
- Name test files with `_test.go` suffix
|
||||
- Place test files next to the code they test
|
||||
|
||||
### Writing Tests
|
||||
|
||||
- Use table-driven tests for multiple test cases
|
||||
- Name tests descriptively using `Test_functionName_scenario`
|
||||
- Use subtests with `t.Run` for better organization
|
||||
- Test both success and error cases
|
||||
- Consider using `testify` or similar libraries when they add value, but don't over-complicate simple tests
|
||||
|
||||
### Test Helpers
|
||||
|
||||
- Mark helper functions with `t.Helper()`
|
||||
- Create test fixtures for complex setup
|
||||
- Use `testing.TB` interface for functions used in tests and benchmarks
|
||||
- Clean up resources using `t.Cleanup()`
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Input Validation
|
||||
|
||||
- Validate all external input
|
||||
- Use strong typing to prevent invalid states
|
||||
- Sanitize data before using in SQL queries
|
||||
- Be careful with file paths from user input
|
||||
- Validate and escape data for different contexts (HTML, SQL, shell)
|
||||
|
||||
### Cryptography
|
||||
|
||||
- Use standard library crypto packages
|
||||
- Don't implement your own cryptography
|
||||
- Use crypto/rand for random number generation
|
||||
- Store passwords using bcrypt, scrypt, or argon2 (consider golang.org/x/crypto for additional options)
|
||||
- Use TLS for network communication
|
||||
|
||||
## Documentation
|
||||
|
||||
### Code Documentation
|
||||
|
||||
- Prioritize self-documenting code through clear naming and structure
|
||||
- Document all exported symbols with clear, concise explanations
|
||||
- Start documentation with the symbol name
|
||||
- Write documentation in English by default
|
||||
- Use examples in documentation when helpful
|
||||
- Keep documentation close to code
|
||||
- Update documentation when code changes
|
||||
- Avoid emoji in documentation and comments
|
||||
|
||||
### README and Documentation Files
|
||||
|
||||
- Include clear setup instructions
|
||||
- Document dependencies and requirements
|
||||
- Provide usage examples
|
||||
- Document configuration options
|
||||
- Include troubleshooting section
|
||||
|
||||
## Tools and Development Workflow
|
||||
|
||||
### Essential Tools
|
||||
|
||||
- `go fmt`: Format code
|
||||
- `go vet`: Find suspicious constructs
|
||||
- `golangci-lint`: Additional linting (golint is deprecated)
|
||||
- `go test`: Run tests
|
||||
- `go mod`: Manage dependencies
|
||||
- `go generate`: Code generation
|
||||
|
||||
### Development Practices
|
||||
|
||||
- Run tests before committing
|
||||
- Use pre-commit hooks for formatting and linting
|
||||
- Keep commits focused and atomic
|
||||
- Write meaningful commit messages
|
||||
- Review diffs before committing
|
||||
|
||||
## Common Pitfalls to Avoid
|
||||
|
||||
- Not checking errors
|
||||
- Ignoring race conditions
|
||||
- Creating goroutine leaks
|
||||
- Not using defer for cleanup
|
||||
- Modifying maps concurrently
|
||||
- Not understanding nil interfaces vs nil pointers
|
||||
- Forgetting to close resources (files, connections)
|
||||
- Using global variables unnecessarily
|
||||
- Over-using unconstrained types (e.g., `any`); prefer specific types or generic type parameters with constraints. If an unconstrained type is required, use `any` rather than `interface{}`
|
||||
- Not considering the zero value of types
|
||||
- **Creating duplicate `package` declarations** - this is a compile error; always check existing files before adding package declarations
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
description: 'Guidelines for creating high-quality custom instruction files for GitHub Copilot'
|
||||
applyTo: '**/*.instructions.md'
|
||||
---
|
||||
|
||||
# Custom Instructions File Guidelines
|
||||
|
||||
Instructions for creating effective and maintainable custom instruction files that guide GitHub Copilot in generating domain-specific code and following project conventions.
|
||||
|
||||
## Project Context
|
||||
|
||||
- Target audience: Developers and GitHub Copilot working with domain-specific code
|
||||
- File format: Markdown with YAML frontmatter
|
||||
- File naming convention: lowercase with hyphens (e.g., `react-best-practices.instructions.md`)
|
||||
- Location: `.github/instructions/` directory
|
||||
- Purpose: Provide context-aware guidance for code generation, review, and documentation
|
||||
|
||||
## Required Frontmatter
|
||||
|
||||
Every instruction file must include YAML frontmatter with the following fields:
|
||||
|
||||
```yaml
|
||||
---
|
||||
description: 'Brief description of the instruction purpose and scope'
|
||||
applyTo: 'glob pattern for target files (e.g., **/*.ts, **/*.py)'
|
||||
---
|
||||
```
|
||||
|
||||
### Frontmatter Guidelines
|
||||
|
||||
- **description**: Single-quoted string, 1-500 characters, clearly stating the purpose
|
||||
- **applyTo**: Glob pattern(s) specifying which files these instructions apply to
|
||||
- Single pattern: `'**/*.ts'`
|
||||
- Multiple patterns: `'**/*.ts, **/*.tsx, **/*.js'`
|
||||
- Specific files: `'src/**/*.py'`
|
||||
- All files: `'**'`
|
||||
|
||||
## File Structure
|
||||
|
||||
A well-structured instruction file should include the following sections:
|
||||
|
||||
### 1. Title and Overview
|
||||
|
||||
- Clear, descriptive title using `#` heading
|
||||
- Brief introduction explaining the purpose and scope
|
||||
- Optional: Project context section with key technologies and versions
|
||||
|
||||
### 2. Core Sections
|
||||
|
||||
Organize content into logical sections based on the domain:
|
||||
|
||||
- **General Instructions**: High-level guidelines and principles
|
||||
- **Best Practices**: Recommended patterns and approaches
|
||||
- **Code Standards**: Naming conventions, formatting, style rules
|
||||
- **Architecture/Structure**: Project organization and design patterns
|
||||
- **Common Patterns**: Frequently used implementations
|
||||
- **Security**: Security considerations (if applicable)
|
||||
- **Performance**: Optimization guidelines (if applicable)
|
||||
- **Testing**: Testing standards and approaches (if applicable)
|
||||
|
||||
### 3. Examples and Code Snippets
|
||||
|
||||
Provide concrete examples with clear labels:
|
||||
|
||||
```markdown
|
||||
### Good Example
|
||||
\`\`\`language
|
||||
// Recommended approach
|
||||
code example here
|
||||
\`\`\`
|
||||
|
||||
### Bad Example
|
||||
\`\`\`language
|
||||
// Avoid this pattern
|
||||
code example here
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
### 4. Validation and Verification (Optional but Recommended)
|
||||
|
||||
- Build commands to verify code
|
||||
- Linting and formatting tools
|
||||
- Testing requirements
|
||||
- Verification steps
|
||||
|
||||
## Content Guidelines
|
||||
|
||||
### Writing Style
|
||||
|
||||
- Use clear, concise language
|
||||
- Write in imperative mood ("Use", "Implement", "Avoid")
|
||||
- Be specific and actionable
|
||||
- Avoid ambiguous terms like "should", "might", "possibly"
|
||||
- Use bullet points and lists for readability
|
||||
- Keep sections focused and scannable
|
||||
|
||||
### Best Practices
|
||||
|
||||
- **Be Specific**: Provide concrete examples rather than abstract concepts
|
||||
- **Show Why**: Explain the reasoning behind recommendations when it adds value
|
||||
- **Use Tables**: For comparing options, listing rules, or showing patterns
|
||||
- **Include Examples**: Real code snippets are more effective than descriptions
|
||||
- **Stay Current**: Reference current versions and best practices
|
||||
- **Link Resources**: Include official documentation and authoritative sources
|
||||
|
||||
### Common Patterns to Include
|
||||
|
||||
1. **Naming Conventions**: How to name variables, functions, classes, files
|
||||
2. **Code Organization**: File structure, module organization, import order
|
||||
3. **Error Handling**: Preferred error handling patterns
|
||||
4. **Dependencies**: How to manage and document dependencies
|
||||
5. **Comments and Documentation**: When and how to document code
|
||||
6. **Version Information**: Target language/framework versions
|
||||
|
||||
## Patterns to Follow
|
||||
|
||||
### Bullet Points and Lists
|
||||
|
||||
```markdown
|
||||
## Security Best Practices
|
||||
|
||||
- Always validate user input before processing
|
||||
- Use parameterized queries to prevent SQL injection
|
||||
- Store secrets in environment variables, never in code
|
||||
- Implement proper authentication and authorization
|
||||
- Enable HTTPS for all production endpoints
|
||||
```
|
||||
|
||||
### Tables for Structured Information
|
||||
|
||||
```markdown
|
||||
## Common Issues
|
||||
|
||||
| Issue | Solution | Example |
|
||||
| ---------------- | ------------------- | ----------------------------- |
|
||||
| Magic numbers | Use named constants | `const MAX_RETRIES = 3` |
|
||||
| Deep nesting | Extract functions | Refactor nested if statements |
|
||||
| Hardcoded values | Use configuration | Store API URLs in config |
|
||||
```
|
||||
|
||||
### Code Comparison
|
||||
|
||||
```markdown
|
||||
### Good Example - Using TypeScript interfaces
|
||||
\`\`\`typescript
|
||||
interface User {
|
||||
id: string;
|
||||
name: string;
|
||||
email: string;
|
||||
}
|
||||
|
||||
function getUser(id: string): User {
|
||||
// Implementation
|
||||
}
|
||||
\`\`\`
|
||||
|
||||
### Bad Example - Using any type
|
||||
\`\`\`typescript
|
||||
function getUser(id: any): any {
|
||||
// Loses type safety
|
||||
}
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
### Conditional Guidance
|
||||
|
||||
```markdown
|
||||
## Framework Selection
|
||||
|
||||
- **For small projects**: Use Minimal API approach
|
||||
- **For large projects**: Use controller-based architecture with clear separation
|
||||
- **For microservices**: Consider domain-driven design patterns
|
||||
```
|
||||
|
||||
## Patterns to Avoid
|
||||
|
||||
- **Overly verbose explanations**: Keep it concise and scannable
|
||||
- **Outdated information**: Always reference current versions and practices
|
||||
- **Ambiguous guidelines**: Be specific about what to do or avoid
|
||||
- **Missing examples**: Abstract rules without concrete code examples
|
||||
- **Contradictory advice**: Ensure consistency throughout the file
|
||||
- **Copy-paste from documentation**: Add value by distilling and contextualizing
|
||||
|
||||
## Testing Your Instructions
|
||||
|
||||
Before finalizing instruction files:
|
||||
|
||||
1. **Test with Copilot**: Try the instructions with actual prompts in VS Code
|
||||
2. **Verify Examples**: Ensure code examples are correct and run without errors
|
||||
3. **Check Glob Patterns**: Confirm `applyTo` patterns match intended files
|
||||
|
||||
## Example Structure
|
||||
|
||||
Here's a minimal example structure for a new instruction file:
|
||||
|
||||
```markdown
|
||||
---
|
||||
description: 'Brief description of purpose'
|
||||
applyTo: '**/*.ext'
|
||||
---
|
||||
|
||||
# Technology Name Development
|
||||
|
||||
Brief introduction and context.
|
||||
|
||||
## General Instructions
|
||||
|
||||
- High-level guideline 1
|
||||
- High-level guideline 2
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Specific practice 1
|
||||
- Specific practice 2
|
||||
|
||||
## Code Standards
|
||||
|
||||
### Naming Conventions
|
||||
- Rule 1
|
||||
- Rule 2
|
||||
|
||||
### File Organization
|
||||
- Structure 1
|
||||
- Structure 2
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1
|
||||
Description and example
|
||||
|
||||
\`\`\`language
|
||||
code example
|
||||
\`\`\`
|
||||
|
||||
### Pattern 2
|
||||
Description and example
|
||||
|
||||
## Validation
|
||||
|
||||
- Build command: `command to verify`
|
||||
- Linting: `command to lint`
|
||||
- Testing: `command to test`
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
- Review instructions when dependencies or frameworks are updated
|
||||
- Update examples to reflect current best practices
|
||||
- Remove outdated patterns or deprecated features
|
||||
- Add new patterns as they emerge in the community
|
||||
- Keep glob patterns accurate as project structure evolves
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Custom Instructions Documentation](https://code.visualstudio.com/docs/copilot/customization/custom-instructions)
|
||||
- [Awesome Copilot Instructions](https://github.com/github/awesome-copilot/tree/main/instructions)
|
||||
@@ -0,0 +1,410 @@
|
||||
---
|
||||
description: "Best practices for authoring GNU Make Makefiles"
|
||||
applyTo: "**/Makefile, **/makefile, **/*.mk, **/GNUmakefile"
|
||||
---
|
||||
|
||||
# Makefile Development Instructions
|
||||
|
||||
Instructions for writing clean, maintainable, and portable GNU Make Makefiles. These instructions are based on the [GNU Make manual](https://www.gnu.org/software/make/manual/).
|
||||
|
||||
## General Principles
|
||||
|
||||
- Write clear and maintainable makefiles that follow GNU Make conventions
|
||||
- Use descriptive target names that clearly indicate their purpose
|
||||
- Keep the default goal (first target) as the most common build operation
|
||||
- Prioritize readability over brevity when writing rules and recipes
|
||||
- Add comments to explain complex rules, variables, or non-obvious behavior
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- Name your makefile `Makefile` (recommended for visibility) or `makefile`
|
||||
- Use `GNUmakefile` only for GNU Make-specific features incompatible with other make implementations
|
||||
- Use standard variable names: `objects`, `OBJECTS`, `objs`, `OBJS`, `obj`, or `OBJ` for object file lists
|
||||
- Use uppercase for built-in variable names (e.g., `CC`, `CFLAGS`, `LDFLAGS`)
|
||||
- Use descriptive target names that reflect their action (e.g., `clean`, `install`, `test`)
|
||||
|
||||
## File Structure
|
||||
|
||||
- Place the default goal (primary build target) as the first rule in the makefile
|
||||
- Group related targets together logically
|
||||
- Define variables at the top of the makefile before rules
|
||||
- Use `.PHONY` to declare targets that don't represent files
|
||||
- Structure makefiles with: variables, then rules, then phony targets
|
||||
|
||||
```makefile
|
||||
# Variables
|
||||
CC = gcc
|
||||
CFLAGS = -Wall -g
|
||||
objects = main.o utils.o
|
||||
|
||||
# Default goal
|
||||
all: program
|
||||
|
||||
# Rules
|
||||
program: $(objects)
|
||||
$(CC) -o program $(objects)
|
||||
|
||||
%.o: %.c
|
||||
$(CC) $(CFLAGS) -c $< -o $@
|
||||
|
||||
# Phony targets
|
||||
.PHONY: clean all
|
||||
clean:
|
||||
rm -f program $(objects)
|
||||
```
|
||||
|
||||
## Variables and Substitution
|
||||
|
||||
- Use variables to avoid duplication and improve maintainability
|
||||
- Define variables with `:=` (simple expansion) for immediate evaluation, `=` for recursive expansion
|
||||
- Use `?=` to set default values that can be overridden
|
||||
- Use `+=` to append to existing variables
|
||||
- Reference variables with `$(VARIABLE)` not `$VARIABLE` (unless single character)
|
||||
- Use automatic variables (`$@`, `$<`, `$^`, `$?`, `$*`) in recipes to make rules more generic
|
||||
|
||||
```makefile
|
||||
# Simple expansion (evaluates immediately)
|
||||
CC := gcc
|
||||
|
||||
# Recursive expansion (evaluates when used)
|
||||
CFLAGS = -Wall $(EXTRA_FLAGS)
|
||||
|
||||
# Conditional assignment
|
||||
PREFIX ?= /usr/local
|
||||
|
||||
# Append to variable
|
||||
CFLAGS += -g
|
||||
```
|
||||
|
||||
## Rules and Prerequisites
|
||||
|
||||
- Separate targets, prerequisites, and recipes clearly
|
||||
- Use implicit rules for standard compilations (e.g., `.c` to `.o`)
|
||||
- List prerequisites in logical order (normal prerequisites before order-only)
|
||||
- Use order-only prerequisites (after `|`) for directories and dependencies that shouldn't trigger rebuilds
|
||||
- Include all actual dependencies to ensure correct rebuilds
|
||||
- Avoid circular dependencies between targets
|
||||
- Remember that order-only prerequisites are omitted from automatic variables like `$^`, so reference them explicitly if needed
|
||||
|
||||
The example below shows a pattern rule that compiles objects into an `obj/` directory. The directory itself is listed as an order-only prerequisite so it is created before compiling but does not force recompilation when its timestamp changes.
|
||||
|
||||
```makefile
|
||||
# Normal prerequisites
|
||||
program: main.o utils.o
|
||||
$(CC) -o $@ $^
|
||||
|
||||
# Order-only prerequisites (directory creation)
|
||||
obj/%.o: %.c | obj
|
||||
$(CC) $(CFLAGS) -c $< -o $@
|
||||
|
||||
obj:
|
||||
mkdir -p obj
|
||||
```
|
||||
|
||||
## Recipes and Commands
|
||||
|
||||
- Start every recipe line with a **tab character** (not spaces) unless `.RECIPEPREFIX` is changed
|
||||
- Use `@` prefix to suppress command echoing when appropriate
|
||||
- Use `-` prefix to ignore errors for specific commands (use sparingly)
|
||||
- Combine related commands with `&&` or `;` on the same line when they must execute together
|
||||
- Keep recipes readable; break long commands across multiple lines with backslash continuation
|
||||
- Use shell conditionals and loops within recipes when needed
|
||||
|
||||
```makefile
|
||||
# Silent command
|
||||
clean:
|
||||
@echo "Cleaning up..."
|
||||
@rm -f $(objects)
|
||||
|
||||
# Ignore errors
|
||||
.PHONY: clean-all
|
||||
clean-all:
|
||||
-rm -rf build/
|
||||
-rm -rf dist/
|
||||
|
||||
# Multi-line recipe with proper continuation
|
||||
install: program
|
||||
install -d $(PREFIX)/bin && \
|
||||
install -m 755 program $(PREFIX)/bin
|
||||
```
|
||||
|
||||
## Phony Targets
|
||||
|
||||
- Always declare phony targets with `.PHONY` to avoid conflicts with files of the same name
|
||||
- Use phony targets for actions like `clean`, `install`, `test`, `all`
|
||||
- Place phony target declarations near their rule definitions or at the end of the makefile
|
||||
|
||||
```makefile
|
||||
.PHONY: all clean test install
|
||||
|
||||
all: program
|
||||
|
||||
clean:
|
||||
rm -f program $(objects)
|
||||
|
||||
test: program
|
||||
./run-tests.sh
|
||||
|
||||
install: program
|
||||
install -m 755 program $(PREFIX)/bin
|
||||
```
|
||||
|
||||
## Pattern Rules and Implicit Rules
|
||||
|
||||
- Use pattern rules (`%.o: %.c`) for generic transformations
|
||||
- Leverage built-in implicit rules when appropriate (GNU Make knows how to compile `.c` to `.o`)
|
||||
- Override implicit rule variables (like `CC`, `CFLAGS`) rather than rewriting the rules
|
||||
- Define custom pattern rules only when built-in rules are insufficient
|
||||
|
||||
```makefile
|
||||
# Use built-in implicit rules by setting variables
|
||||
CC = gcc
|
||||
CFLAGS = -Wall -O2
|
||||
|
||||
# Custom pattern rule for special cases
|
||||
%.pdf: %.md
|
||||
pandoc $< -o $@
|
||||
```
|
||||
|
||||
## Splitting Long Lines
|
||||
|
||||
- Use backslash-newline (`\`) to split long lines for readability
|
||||
- Be aware that backslash-newline is converted to a single space in non-recipe contexts
|
||||
- In recipes, backslash-newline preserves the line continuation for the shell
|
||||
- Avoid trailing whitespace after backslashes
|
||||
|
||||
### Splitting Without Adding Whitespace
|
||||
|
||||
If you need to split a line without adding whitespace, you can use a special technique: insert `$ ` (dollar-space) followed by a backslash-newline. The `$ ` refers to a variable with a single-space name, which doesn't exist and expands to nothing, effectively joining the lines without inserting a space.
|
||||
|
||||
```makefile
|
||||
# Concatenate strings without adding whitespace
|
||||
# The following creates the value "oneword"
|
||||
var := one$ \
|
||||
word
|
||||
|
||||
# This is equivalent to:
|
||||
# var := oneword
|
||||
```
|
||||
|
||||
```makefile
|
||||
# Variable definition split across lines
|
||||
sources = main.c \
|
||||
utils.c \
|
||||
parser.c \
|
||||
handler.c
|
||||
|
||||
# Recipe with long command
|
||||
build: $(objects)
|
||||
$(CC) -o program $(objects) \
|
||||
$(LDFLAGS) \
|
||||
-lm -lpthread
|
||||
```
|
||||
|
||||
## Including Other Makefiles
|
||||
|
||||
- Use `include` directive to share common definitions across makefiles
|
||||
- Use `-include` (or `sinclude`) to include optional makefiles without errors
|
||||
- Place `include` directives after variable definitions that may affect included files
|
||||
- Use `include` for shared variables, pattern rules, or common targets
|
||||
|
||||
```makefile
|
||||
# Include common settings
|
||||
include config.mk
|
||||
|
||||
# Include optional local configuration
|
||||
-include local.mk
|
||||
```
|
||||
|
||||
## Conditional Directives
|
||||
|
||||
- Use conditional directives (`ifeq`, `ifneq`, `ifdef`, `ifndef`) for platform or configuration-specific rules
|
||||
- Place conditionals at the makefile level, not within recipes (use shell conditionals in recipes)
|
||||
- Keep conditionals simple and well-documented
|
||||
|
||||
```makefile
|
||||
# Platform-specific settings
|
||||
ifeq ($(OS),Windows_NT)
|
||||
EXE_EXT = .exe
|
||||
else
|
||||
EXE_EXT =
|
||||
endif
|
||||
|
||||
program: main.o
|
||||
$(CC) -o program$(EXE_EXT) main.o
|
||||
```
|
||||
|
||||
## Automatic Prerequisites
|
||||
|
||||
- Generate header dependencies automatically rather than maintaining them manually
|
||||
- Use compiler flags like `-MMD` and `-MP` to generate `.d` files with dependencies
|
||||
- Include generated dependency files with `-include $(deps)` to avoid errors if they don't exist
|
||||
|
||||
```makefile
|
||||
objects = main.o utils.o
|
||||
deps = $(objects:.o=.d)
|
||||
|
||||
# Include dependency files
|
||||
-include $(deps)
|
||||
|
||||
# Compile with automatic dependency generation
|
||||
%.o: %.c
|
||||
$(CC) $(CFLAGS) -MMD -MP -c $< -o $@
|
||||
```
|
||||
|
||||
## Error Handling and Debugging
|
||||
|
||||
- Use `$(error text)` or `$(warning text)` functions for build-time diagnostics
|
||||
- Test makefiles with `make -n` (dry run) to see commands without executing
|
||||
- Use `make -p` to print the database of rules and variables for debugging
|
||||
- Validate required variables and tools at the beginning of the makefile
|
||||
|
||||
```makefile
|
||||
# Check for required tools
|
||||
ifeq ($(shell which gcc),)
|
||||
$(error "gcc is not installed or not in PATH")
|
||||
endif
|
||||
|
||||
# Validate required variables
|
||||
ifndef VERSION
|
||||
$(error VERSION is not defined)
|
||||
endif
|
||||
```
|
||||
|
||||
## Clean Targets
|
||||
|
||||
- Always provide a `clean` target to remove generated files
|
||||
- Declare `clean` as phony to avoid conflicts with a file named "clean"
|
||||
- Use `-` prefix with `rm` commands to ignore errors if files don't exist
|
||||
- Consider separate `clean` (removes objects) and `distclean` (removes all generated files) targets
|
||||
|
||||
```makefile
|
||||
.PHONY: clean distclean
|
||||
|
||||
clean:
|
||||
-rm -f $(objects)
|
||||
-rm -f $(deps)
|
||||
|
||||
distclean: clean
|
||||
-rm -f program config.mk
|
||||
```
|
||||
|
||||
## Portability Considerations
|
||||
|
||||
- Avoid GNU Make-specific features if portability to other make implementations is required
|
||||
- Use standard shell commands (prefer POSIX shell constructs)
|
||||
- Test with `make -B` to force rebuild all targets
|
||||
- Document any platform-specific requirements or GNU Make extensions used
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
- Use `:=` for variables that don't need recursive expansion (faster)
|
||||
- Avoid unnecessary use of `$(shell ...)` which creates subprocesses
|
||||
- Order prerequisites efficiently (most frequently changing files last)
|
||||
- Use parallel builds (`make -j`) safely by ensuring targets don't conflict
|
||||
|
||||
## Documentation and Comments
|
||||
|
||||
- Add a header comment explaining the makefile's purpose
|
||||
- Document non-obvious variable settings and their effects
|
||||
- Include usage examples or targets in comments
|
||||
- Add inline comments for complex rules or platform-specific workarounds
|
||||
|
||||
```makefile
|
||||
# Makefile for building the example application
|
||||
#
|
||||
# Usage:
|
||||
# make - Build the program
|
||||
# make clean - Remove generated files
|
||||
# make install - Install to $(PREFIX)
|
||||
#
|
||||
# Variables:
|
||||
# CC - C compiler (default: gcc)
|
||||
# PREFIX - Installation prefix (default: /usr/local)
|
||||
|
||||
# Compiler and flags
|
||||
CC ?= gcc
|
||||
CFLAGS = -Wall -Wextra -O2
|
||||
|
||||
# Installation directory
|
||||
PREFIX ?= /usr/local
|
||||
```
|
||||
|
||||
## Special Targets
|
||||
|
||||
- Use `.PHONY` for non-file targets
|
||||
- Use `.PRECIOUS` to preserve intermediate files
|
||||
- Use `.INTERMEDIATE` to mark files as intermediate (automatically deleted)
|
||||
- Use `.SECONDARY` to prevent deletion of intermediate files
|
||||
- Use `.DELETE_ON_ERROR` to remove targets if recipe fails
|
||||
- Use `.SILENT` to suppress echoing for all recipes (use sparingly)
|
||||
|
||||
```makefile
|
||||
# Don't delete intermediate files
|
||||
.SECONDARY:
|
||||
|
||||
# Delete targets if recipe fails
|
||||
.DELETE_ON_ERROR:
|
||||
|
||||
# Preserve specific files
|
||||
.PRECIOUS: %.o
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Standard Project Structure
|
||||
|
||||
```makefile
|
||||
CC = gcc
|
||||
CFLAGS = -Wall -O2
|
||||
objects = main.o utils.o parser.o
|
||||
|
||||
.PHONY: all clean install
|
||||
|
||||
all: program
|
||||
|
||||
program: $(objects)
|
||||
$(CC) -o $@ $^
|
||||
|
||||
%.o: %.c
|
||||
$(CC) $(CFLAGS) -c $< -o $@
|
||||
|
||||
clean:
|
||||
-rm -f program $(objects)
|
||||
|
||||
install: program
|
||||
install -d $(PREFIX)/bin
|
||||
install -m 755 program $(PREFIX)/bin
|
||||
```
|
||||
|
||||
### Managing Multiple Programs
|
||||
|
||||
```makefile
|
||||
programs = prog1 prog2 prog3
|
||||
|
||||
.PHONY: all clean
|
||||
|
||||
all: $(programs)
|
||||
|
||||
prog1: prog1.o common.o
|
||||
$(CC) -o $@ $^
|
||||
|
||||
prog2: prog2.o common.o
|
||||
$(CC) -o $@ $^
|
||||
|
||||
prog3: prog3.o
|
||||
$(CC) -o $@ $^
|
||||
|
||||
clean:
|
||||
-rm -f $(programs) *.o
|
||||
```
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
- Don't start recipe lines with spaces instead of tabs
|
||||
- Avoid hardcoding file lists when they can be generated with wildcards or functions
|
||||
- Don't use `$(shell ls ...)` to get file lists (use `$(wildcard ...)` instead)
|
||||
- Avoid complex shell scripts in recipes (move to separate script files)
|
||||
- Don't forget to declare phony targets as `.PHONY`
|
||||
- Avoid circular dependencies between targets
|
||||
- Don't use recursive make (`$(MAKE) -C subdir`) unless absolutely necessary
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
description: 'Documentation and content creation standards'
|
||||
applyTo: '**/*.md'
|
||||
---
|
||||
|
||||
## Markdown Content Rules
|
||||
|
||||
The following markdown content rules are enforced in the validators:
|
||||
|
||||
1. **Headings**: Use appropriate heading levels (H2, H3, etc.) to structure your content. Do not use an H1 heading, as this will be generated based on the title.
|
||||
2. **Lists**: Use bullet points or numbered lists for lists. Ensure proper indentation and spacing.
|
||||
3. **Code Blocks**: Use fenced code blocks for code snippets. Specify the language for syntax highlighting.
|
||||
4. **Links**: Use proper markdown syntax for links. Ensure that links are valid and accessible.
|
||||
5. **Images**: Use proper markdown syntax for images. Include alt text for accessibility.
|
||||
6. **Tables**: Use markdown tables for tabular data. Ensure proper formatting and alignment.
|
||||
7. **Line Length**: Limit line length to 400 characters for readability.
|
||||
8. **Whitespace**: Use appropriate whitespace to separate sections and improve readability.
|
||||
9. **Front Matter**: Include YAML front matter at the beginning of the file with required metadata fields.
|
||||
|
||||
## Formatting and Structure
|
||||
|
||||
Follow these guidelines for formatting and structuring your markdown content:
|
||||
|
||||
- **Headings**: Use `##` for H2 and `###` for H3. Ensure that headings are used in a hierarchical manner. Recommend restructuring if content includes H4, and more strongly recommend for H5.
|
||||
- **Lists**: Use `-` for bullet points and `1.` for numbered lists. Indent nested lists with two spaces.
|
||||
- **Code Blocks**: Use triple backticks (`) to create fenced code blocks. Specify the language after the opening backticks for syntax highlighting (e.g., `csharp).
|
||||
- **Links**: Use `[link text](URL)` for links. Ensure that the link text is descriptive and the URL is valid.
|
||||
- **Images**: Use `` for images. Include a brief description of the image in the alt text.
|
||||
- **Tables**: Use `|` to create tables. Ensure that columns are properly aligned and headers are included.
|
||||
- **Line Length**: Break lines at 80 characters to improve readability. Use soft line breaks for long paragraphs.
|
||||
- **Whitespace**: Use blank lines to separate sections and improve readability. Avoid excessive whitespace.
|
||||
|
||||
## Validation Requirements
|
||||
|
||||
Ensure compliance with the following validation requirements:
|
||||
|
||||
- **Front Matter**: Include the following fields in the YAML front matter:
|
||||
|
||||
- `post_title`: The title of the post.
|
||||
- `author1`: The primary author of the post.
|
||||
- `post_slug`: The URL slug for the post.
|
||||
- `microsoft_alias`: The Microsoft alias of the author.
|
||||
- `featured_image`: The URL of the featured image.
|
||||
- `categories`: The categories for the post. These categories must be from the list in /categories.txt.
|
||||
- `tags`: The tags for the post.
|
||||
- `ai_note`: Indicate if AI was used in the creation of the post.
|
||||
- `summary`: A brief summary of the post. Recommend a summary based on the content when possible.
|
||||
- `post_date`: The publication date of the post.
|
||||
|
||||
- **Content Rules**: Ensure that the content follows the markdown content rules specified above.
|
||||
- **Formatting**: Ensure that the content is properly formatted and structured according to the guidelines.
|
||||
- **Validation**: Run the validation tools to check for compliance with the rules and guidelines.
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
description: "Guidelines for writing Node.js and JavaScript code with Vitest testing"
|
||||
applyTo: '**/*.js, **/*.mjs, **/*.cjs'
|
||||
---
|
||||
|
||||
# Code Generation Guidelines
|
||||
|
||||
## Coding standards
|
||||
- Use JavaScript with ES2022 features and Node.js (20+) ESM modules
|
||||
- Use Node.js built-in modules and avoid external dependencies where possible
|
||||
- Ask the user if you require any additional dependencies before adding them
|
||||
- Always use async/await for asynchronous code, and use 'node:util' promisify function to avoid callbacks
|
||||
- Keep the code simple and maintainable
|
||||
- Use descriptive variable and function names
|
||||
- Do not add comments unless absolutely necessary, the code should be self-explanatory
|
||||
- Never use `null`, always use `undefined` for optional values
|
||||
- Prefer functions over classes
|
||||
|
||||
## Testing
|
||||
- Use Vitest for testing
|
||||
- Write tests for all new features and bug fixes
|
||||
- Ensure tests cover edge cases and error handling
|
||||
- NEVER change the original code to make it easier to test, instead, write tests that cover the original code as it is
|
||||
|
||||
## Documentation
|
||||
- When adding new features or making significant changes, update the README.md file where necessary
|
||||
|
||||
## User interactions
|
||||
- Ask questions if you are unsure about the implementation details, design choices, or need clarification on the requirements
|
||||
- Always answer in the same language as the question, but use english for the generated content like code, comments or docs
|
||||
@@ -0,0 +1,311 @@
|
||||
---
|
||||
applyTo: '**/*.{cs,ts,java}'
|
||||
description: Enforces Object Calisthenics principles for business domain code to ensure clean, maintainable, and robust code
|
||||
---
|
||||
# Object Calisthenics Rules
|
||||
|
||||
> ⚠️ **Warning:** This file contains the 9 original Object Calisthenics rules. No additional rules must be added, and none of these rules should be replaced or removed.
|
||||
> Examples may be added later if needed.
|
||||
|
||||
## Objective
|
||||
This rule enforces the principles of Object Calisthenics to ensure clean, maintainable, and robust code in the backend, **primarily for business domain code**.
|
||||
|
||||
## Scope and Application
|
||||
- **Primary focus**: Business domain classes (aggregates, entities, value objects, domain services)
|
||||
- **Secondary focus**: Application layer services and use case handlers
|
||||
- **Exemptions**:
|
||||
- DTOs (Data Transfer Objects)
|
||||
- API models/contracts
|
||||
- Configuration classes
|
||||
- Simple data containers without business logic
|
||||
- Infrastructure code where flexibility is needed
|
||||
|
||||
## Key Principles
|
||||
|
||||
|
||||
1. **One Level of Indentation per Method**:
|
||||
- Ensure methods are simple and do not exceed one level of indentation.
|
||||
|
||||
```csharp
|
||||
// Bad Example - this method has multiple levels of indentation
|
||||
public void SendNewsletter() {
|
||||
foreach (var user in users) {
|
||||
if (user.IsActive) {
|
||||
// Do something
|
||||
mailer.Send(user.Email);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Good Example - Extracted method to reduce indentation
|
||||
public void SendNewsletter() {
|
||||
foreach (var user in users) {
|
||||
SendEmail(user);
|
||||
}
|
||||
}
|
||||
private void SendEmail(User user) {
|
||||
if (user.IsActive) {
|
||||
mailer.Send(user.Email);
|
||||
}
|
||||
}
|
||||
|
||||
// Good Example - Filtering users before sending emails
|
||||
public void SendNewsletter() {
|
||||
var activeUsers = users.Where(user => user.IsActive);
|
||||
|
||||
foreach (var user in activeUsers) {
|
||||
mailer.Send(user.Email);
|
||||
}
|
||||
}
|
||||
```
|
||||
2. **Don't Use the ELSE Keyword**:
|
||||
|
||||
- Avoid using the `else` keyword to reduce complexity and improve readability.
|
||||
- Use early returns to handle conditions instead.
|
||||
- Use Fail Fast principle
|
||||
- Use Guard Clauses to validate inputs and conditions at the beginning of methods.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Using else
|
||||
public void ProcessOrder(Order order) {
|
||||
if (order.IsValid) {
|
||||
// Process order
|
||||
} else {
|
||||
// Handle invalid order
|
||||
}
|
||||
}
|
||||
// Good Example - Avoiding else
|
||||
public void ProcessOrder(Order order) {
|
||||
if (!order.IsValid) return;
|
||||
// Process order
|
||||
}
|
||||
```
|
||||
|
||||
Sample Fail fast principle:
|
||||
```csharp
|
||||
public void ProcessOrder(Order order) {
|
||||
if (order == null) throw new ArgumentNullException(nameof(order));
|
||||
if (!order.IsValid) throw new InvalidOperationException("Invalid order");
|
||||
// Process order
|
||||
}
|
||||
```
|
||||
|
||||
3. **Wrapping All Primitives and Strings**:
|
||||
- Avoid using primitive types directly in your code.
|
||||
- Wrap them in classes to provide meaningful context and behavior.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Using primitive types directly
|
||||
public class User {
|
||||
public string Name { get; set; }
|
||||
public int Age { get; set; }
|
||||
}
|
||||
// Good Example - Wrapping primitives
|
||||
public class User {
|
||||
private string name;
|
||||
private Age age;
|
||||
public User(string name, Age age) {
|
||||
this.name = name;
|
||||
this.age = age;
|
||||
}
|
||||
}
|
||||
public class Age {
|
||||
private int value;
|
||||
public Age(int value) {
|
||||
if (value < 0) throw new ArgumentOutOfRangeException(nameof(value), "Age cannot be negative");
|
||||
this.value = value;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. **First Class Collections**:
|
||||
- Use collections to encapsulate data and behavior, rather than exposing raw data structures.
|
||||
First Class Collections: a class that contains an array as an attribute should not contain any other attributes
|
||||
|
||||
```csharp
|
||||
// Bad Example - Exposing raw collection
|
||||
public class Group {
|
||||
public int Id { get; private set; }
|
||||
public string Name { get; private set; }
|
||||
public List<User> Users { get; private set; }
|
||||
|
||||
public int GetNumberOfUsersIsActive() {
|
||||
return Users
|
||||
.Where(user => user.IsActive)
|
||||
.Count();
|
||||
}
|
||||
}
|
||||
|
||||
// Good Example - Encapsulating collection behavior
|
||||
public class Group {
|
||||
public int Id { get; private set; }
|
||||
public string Name { get; private set; }
|
||||
|
||||
public GroupUserCollection userCollection { get; private set; } // The list of users is encapsulated in a class
|
||||
|
||||
public int GetNumberOfUsersIsActive() {
|
||||
return userCollection
|
||||
.GetActiveUsers()
|
||||
.Count();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
5. **One Dot per Line**:
|
||||
- Avoid violating Law of Demeter by only having a single dot per line.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Multiple dots in a single line
|
||||
public void ProcessOrder(Order order) {
|
||||
var userEmail = order.User.GetEmail().ToUpper().Trim();
|
||||
// Do something with userEmail
|
||||
}
|
||||
// Good Example - One dot per line
|
||||
public class User {
|
||||
public NormalizedEmail GetEmail() {
|
||||
return NormalizedEmail.Create(/*...*/);
|
||||
}
|
||||
}
|
||||
public class Order {
|
||||
/*...*/
|
||||
public NormalizedEmail ConfirmationEmail() {
|
||||
return User.GetEmail();
|
||||
}
|
||||
}
|
||||
public void ProcessOrder(Order order) {
|
||||
var confirmationEmail = order.ConfirmationEmail();
|
||||
// Do something with confirmationEmail
|
||||
}
|
||||
```
|
||||
|
||||
6. **Don't abbreviate**:
|
||||
- Use meaningful names for classes, methods, and variables.
|
||||
- Avoid abbreviations that can lead to confusion.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Abbreviated names
|
||||
public class U {
|
||||
public string N { get; set; }
|
||||
}
|
||||
// Good Example - Meaningful names
|
||||
public class User {
|
||||
public string Name { get; set; }
|
||||
}
|
||||
```
|
||||
|
||||
7. **Keep entities small (Class, method, namespace or package)**:
|
||||
- Limit the size of classes and methods to improve code readability and maintainability.
|
||||
- Each class should have a single responsibility and be as small as possible.
|
||||
|
||||
Constraints:
|
||||
- Maximum 10 methods per class
|
||||
- Maximum 50 lines per class
|
||||
- Maximum 10 classes per package or namespace
|
||||
|
||||
```csharp
|
||||
// Bad Example - Large class with multiple responsibilities
|
||||
public class UserManager {
|
||||
public void CreateUser(string name) { /*...*/ }
|
||||
public void DeleteUser(int id) { /*...*/ }
|
||||
public void SendEmail(string email) { /*...*/ }
|
||||
}
|
||||
|
||||
// Good Example - Small classes with single responsibility
|
||||
public class UserCreator {
|
||||
public void CreateUser(string name) { /*...*/ }
|
||||
}
|
||||
public class UserDeleter {
|
||||
public void DeleteUser(int id) { /*...*/ }
|
||||
}
|
||||
|
||||
public class UserUpdater {
|
||||
public void UpdateUser(int id, string name) { /*...*/ }
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
8. **No Classes with More Than Two Instance Variables**:
|
||||
- Encourage classes to have a single responsibility by limiting the number of instance variables.
|
||||
- Limit the number of instance variables to two to maintain simplicity.
|
||||
- Do not count ILogger or any other logger as instance variable.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Class with multiple instance variables
|
||||
public class UserCreateCommandHandler {
|
||||
// Bad: Too many instance variables
|
||||
private readonly IUserRepository userRepository;
|
||||
private readonly IEmailService emailService;
|
||||
private readonly ILogger logger;
|
||||
private readonly ISmsService smsService;
|
||||
|
||||
public UserCreateCommandHandler(IUserRepository userRepository, IEmailService emailService, ILogger logger, ISmsService smsService) {
|
||||
this.userRepository = userRepository;
|
||||
this.emailService = emailService;
|
||||
this.logger = logger;
|
||||
this.smsService = smsService;
|
||||
}
|
||||
}
|
||||
|
||||
// Good: Class with two instance variables
|
||||
public class UserCreateCommandHandler {
|
||||
private readonly IUserRepository userRepository;
|
||||
private readonly INotificationService notificationService;
|
||||
private readonly ILogger logger; // This is not counted as instance variable
|
||||
|
||||
public UserCreateCommandHandler(IUserRepository userRepository, INotificationService notificationService, ILogger logger) {
|
||||
this.userRepository = userRepository;
|
||||
this.notificationService = notificationService;
|
||||
this.logger = logger;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
9. **No Getters/Setters in Domain Classes**:
|
||||
- Avoid exposing setters for properties in domain classes.
|
||||
- Use private constructors and static factory methods for object creation.
|
||||
- **Note**: This rule applies primarily to domain classes, not DTOs or data transfer objects.
|
||||
|
||||
```csharp
|
||||
// Bad Example - Domain class with public setters
|
||||
public class User { // Domain class
|
||||
public string Name { get; set; } // Avoid this in domain classes
|
||||
}
|
||||
|
||||
// Good Example - Domain class with encapsulation
|
||||
public class User { // Domain class
|
||||
private string name;
|
||||
private User(string name) { this.name = name; }
|
||||
public static User Create(string name) => new User(name);
|
||||
}
|
||||
|
||||
// Acceptable Example - DTO with public setters
|
||||
public class UserDto { // DTO - exemption applies
|
||||
public string Name { get; set; } // Acceptable for DTOs
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Guidelines
|
||||
- **Domain Classes**:
|
||||
- Use private constructors and static factory methods for creating instances.
|
||||
- Avoid exposing setters for properties.
|
||||
- Apply all 9 rules strictly for business domain code.
|
||||
|
||||
- **Application Layer**:
|
||||
- Apply these rules to use case handlers and application services.
|
||||
- Focus on maintaining single responsibility and clean abstractions.
|
||||
|
||||
- **DTOs and Data Objects**:
|
||||
- Rules 3 (wrapping primitives), 8 (two instance variables), and 9 (no getters/setters) may be relaxed for DTOs.
|
||||
- Public properties with getters/setters are acceptable for data transfer objects.
|
||||
|
||||
- **Testing**:
|
||||
- Ensure tests validate the behavior of objects rather than their state.
|
||||
- Test classes may have relaxed rules for readability and maintainability.
|
||||
|
||||
- **Code Reviews**:
|
||||
- Enforce these rules during code reviews for domain and application code.
|
||||
- Be pragmatic about infrastructure and DTO code.
|
||||
|
||||
## References
|
||||
- [Object Calisthenics - Original 9 Rules by Jeff Bay](https://www.cs.helsinki.fi/u/luontola/tdd-2009/ext/ObjectCalisthenics.pdf)
|
||||
- [ThoughtWorks - Object Calisthenics](https://www.thoughtworks.com/insights/blog/object-calisthenics)
|
||||
- [Clean Code: A Handbook of Agile Software Craftsmanship - Robert C. Martin](https://www.oreilly.com/library/view/clean-code-a/9780136083238/)
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
description: 'React controls and platform libraries for PCF components'
|
||||
applyTo: '**/*.{ts,tsx,js,json,xml,pcfproj,csproj}'
|
||||
---
|
||||
|
||||
# React Controls & Platform Libraries
|
||||
|
||||
When you use React and platform libraries, you're using the same infrastructure used by the Power Apps platform. This means you no longer have to package React and Fluent libraries individually for each control. All controls share a common library instance and version to provide a seamless and consistent experience.
|
||||
|
||||
## Benefits
|
||||
|
||||
By reusing the existing platform React and Fluent libraries, you can expect:
|
||||
|
||||
- **Reduced control bundle size**
|
||||
- **Optimized solution packaging**
|
||||
- **Faster runtime transfer, scripting, and control rendering**
|
||||
- **Design and theme alignment with the Power Apps Fluent design system**
|
||||
|
||||
> **Note**: With GA release, all existing virtual controls will continue to function. However, they should be rebuilt and deployed using the latest CLI version (>=1.37) to facilitate future platform React version upgrades.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
As with any component, you must install [Visual Studio Code](https://code.visualstudio.com/Download) and the [Microsoft Power Platform CLI](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/powerapps-cli#install-microsoft-power-platform-cli).
|
||||
|
||||
> **Note**: If you have already installed Power Platform CLI for Windows, make sure you are running the latest version by using the `pac install latest` command. The Power Platform Tools for Visual Studio Code should update automatically.
|
||||
|
||||
## Create a React Component
|
||||
|
||||
> **Note**: These instructions expect that you have created code components before. If you have not, see [Create your first component](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/implementing-controls-using-typescript).
|
||||
|
||||
There's a new `--framework` (`-fw`) parameter for the `pac pcf init` command. Set the value of this parameter to `react`.
|
||||
|
||||
### Command Parameters
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| --name | ReactSample |
|
||||
| --namespace | SampleNamespace |
|
||||
| --template | field |
|
||||
| --framework | react |
|
||||
| --run-npm-install | true (default) |
|
||||
|
||||
### PowerShell Command
|
||||
|
||||
The following PowerShell command uses the parameter shortcuts and creates a React component project and runs `npm-install`:
|
||||
|
||||
```powershell
|
||||
pac pcf init -n ReactSample -ns SampleNamespace -t field -fw react -npm
|
||||
```
|
||||
|
||||
You can now build and view the control in the test harness as usual using `npm start`.
|
||||
|
||||
After you build the control, you can package it inside solutions and use it for model-driven apps (including custom pages) and canvas apps like standard code components.
|
||||
|
||||
## Differences from Standard Components
|
||||
|
||||
### ControlManifest.Input.xml
|
||||
|
||||
The [control element](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/manifest-schema-reference/control) `control-type` attribute is set to `virtual` rather than `standard`.
|
||||
|
||||
> **Note**: Changing this value does not convert a component from one type to another.
|
||||
|
||||
Within the [resources element](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/manifest-schema-reference/resources), find two new [platform-library element](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/manifest-schema-reference/platform-library) child elements:
|
||||
|
||||
```xml
|
||||
<resources>
|
||||
<code path="index.ts" order="1" />
|
||||
<platform-library name="React" version="16.14.0" />
|
||||
<platform-library name="Fluent" version="9.46.2" />
|
||||
</resources>
|
||||
```
|
||||
|
||||
> **Note**: For more information about valid platform library versions, see Supported platform libraries list.
|
||||
|
||||
**Recommendation**: We recommend using platform libraries for Fluent 8 and 9. If you don't use Fluent, you should remove the `platform-library` element where the `name` attribute value is `Fluent`.
|
||||
|
||||
### Index.ts
|
||||
|
||||
The [ReactControl.init](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/reference/react-control/init) method for control initialization doesn't have `div` parameters because React controls don't render the DOM directly. Instead [ReactControl.updateView](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/reference/react-control/updateview) returns a ReactElement that has the details of the actual control in React format.
|
||||
|
||||
### bundle.js
|
||||
|
||||
React and Fluent libraries aren't included in the package because they're shared, therefore the size of bundle.js is smaller.
|
||||
|
||||
## Sample Controls
|
||||
|
||||
The following controls are included in the samples. They function the same as their standard versions but offer better performance since they are virtual controls.
|
||||
|
||||
| Sample | Description | Link |
|
||||
|--------|-------------|------|
|
||||
| ChoicesPickerReact | The standard ChoicesPickerControl converted to be a React Control | ChoicesPickerReact Sample |
|
||||
| FacepileReact | The ReactStandardControl converted to be a React Control | FacepileReact |
|
||||
|
||||
## Supported Platform Libraries List
|
||||
|
||||
Platform libraries are made available both at the build and runtime to the controls that are using platform libraries capability. Currently, the following versions are provided by the platform and are the highest currently supported versions.
|
||||
|
||||
| Library | Package | Build Version | Runtime Version |
|
||||
|---------|---------|---------------|-----------------|
|
||||
| React | react | 16.14.0 | 17.0.2 (Model), 16.14.0 (Canvas) |
|
||||
| Fluent | @fluentui/react | 8.29.0 | 8.29.0 |
|
||||
| Fluent | @fluentui/react | 8.121.1 | 8.121.1 |
|
||||
| Fluent | @fluentui/react-components | >=9.4.0 <=9.46.2 | 9.68.0 |
|
||||
|
||||
> **Note**: The application might load a higher compatible version of a platform library at runtime, but the version might not be the latest version available. Fluent 8 and Fluent 9 are each supported but can not both be specified in the same manifest.
|
||||
|
||||
## FAQ
|
||||
|
||||
### Q: Can I convert an existing standard control to a React control using platform libraries?
|
||||
|
||||
A: No. You must create a new control using the new template and then update the manifest and index.ts methods. For reference, compare the standard and react samples described above.
|
||||
|
||||
### Q: Can I use React controls & platform libraries with Power Pages?
|
||||
|
||||
A: No. React controls & platform libraries are currently only supported for canvas and model-driven apps. In Power Pages, React controls don't update based on changes in other fields.
|
||||
|
||||
## Related Articles
|
||||
|
||||
- [What are code components?](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/custom-controls-overview)
|
||||
- [Code components for canvas apps](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/component-framework-for-canvas-apps)
|
||||
- [Create and build a code component](https://learn.microsoft.com/en-us/power-apps/developer/component-framework/create-custom-controls-using-pcf)
|
||||
- [Learn Power Apps component framework](https://learn.microsoft.com/en-us/training/paths/use-power-apps-component-framework)
|
||||
- [Use code components in Power Pages](https://learn.microsoft.com/en-us/power-apps/maker/portals/component-framework)
|
||||
@@ -0,0 +1,420 @@
|
||||
---
|
||||
applyTo: '*'
|
||||
description: 'The most comprehensive, practical, and engineer-authored performance optimization instructions for all languages, frameworks, and stacks. Covers frontend, backend, and database best practices with actionable guidance, scenario-based checklists, troubleshooting, and pro tips.'
|
||||
---
|
||||
|
||||
# Performance Optimization Best Practices
|
||||
|
||||
## Introduction
|
||||
|
||||
Performance isn't just a buzzword—it's the difference between a product people love and one they abandon. I've seen firsthand how a slow app can frustrate users, rack up cloud bills, and even lose customers. This guide is a living collection of the most effective, real-world performance practices I've used and reviewed, covering frontend, backend, and database layers, as well as advanced topics. Use it as a reference, a checklist, and a source of inspiration for building fast, efficient, and scalable software.
|
||||
|
||||
---
|
||||
|
||||
## General Principles
|
||||
|
||||
- **Measure First, Optimize Second:** Always profile and measure before optimizing. Use benchmarks, profilers, and monitoring tools to identify real bottlenecks. Guessing is the enemy of performance.
|
||||
- *Pro Tip:* Use tools like Chrome DevTools, Lighthouse, New Relic, Datadog, Py-Spy, or your language's built-in profilers.
|
||||
- **Optimize for the Common Case:** Focus on optimizing code paths that are most frequently executed. Don't waste time on rare edge cases unless they're critical.
|
||||
- **Avoid Premature Optimization:** Write clear, maintainable code first; optimize only when necessary. Premature optimization can make code harder to read and maintain.
|
||||
- **Minimize Resource Usage:** Use memory, CPU, network, and disk resources efficiently. Always ask: "Can this be done with less?"
|
||||
- **Prefer Simplicity:** Simple algorithms and data structures are often faster and easier to optimize. Don't over-engineer.
|
||||
- **Document Performance Assumptions:** Clearly comment on any code that is performance-critical or has non-obvious optimizations. Future maintainers (including you) will thank you.
|
||||
- **Understand the Platform:** Know the performance characteristics of your language, framework, and runtime. What's fast in Python may be slow in JavaScript, and vice versa.
|
||||
- **Automate Performance Testing:** Integrate performance tests and benchmarks into your CI/CD pipeline. Catch regressions early.
|
||||
- **Set Performance Budgets:** Define acceptable limits for load time, memory usage, API latency, etc. Enforce them with automated checks.
|
||||
|
||||
---
|
||||
|
||||
## Frontend Performance
|
||||
|
||||
### Rendering and DOM
|
||||
- **Minimize DOM Manipulations:** Batch updates where possible. Frequent DOM changes are expensive.
|
||||
- *Anti-pattern:* Updating the DOM in a loop. Instead, build a document fragment and append it once.
|
||||
- **Virtual DOM Frameworks:** Use React, Vue, or similar efficiently—avoid unnecessary re-renders.
|
||||
- *React Example:* Use `React.memo`, `useMemo`, and `useCallback` to prevent unnecessary renders.
|
||||
- **Keys in Lists:** Always use stable keys in lists to help virtual DOM diffing. Avoid using array indices as keys unless the list is static.
|
||||
- **Avoid Inline Styles:** Inline styles can trigger layout thrashing. Prefer CSS classes.
|
||||
- **CSS Animations:** Use CSS transitions/animations over JavaScript for smoother, GPU-accelerated effects.
|
||||
- **Defer Non-Critical Rendering:** Use `requestIdleCallback` or similar to defer work until the browser is idle.
|
||||
|
||||
### Asset Optimization
|
||||
- **Image Compression:** Use tools like ImageOptim, Squoosh, or TinyPNG. Prefer modern formats (WebP, AVIF) for web delivery.
|
||||
- **SVGs for Icons:** SVGs scale well and are often smaller than PNGs for simple graphics.
|
||||
- **Minification and Bundling:** Use Webpack, Rollup, or esbuild to bundle and minify JS/CSS. Enable tree-shaking to remove dead code.
|
||||
- **Cache Headers:** Set long-lived cache headers for static assets. Use cache busting for updates.
|
||||
- **Lazy Loading:** Use `loading="lazy"` for images, and dynamic imports for JS modules/components.
|
||||
- **Font Optimization:** Use only the character sets you need. Subset fonts and use `font-display: swap`.
|
||||
|
||||
### Network Optimization
|
||||
- **Reduce HTTP Requests:** Combine files, use image sprites, and inline critical CSS.
|
||||
- **HTTP/2 and HTTP/3:** Enable these protocols for multiplexing and lower latency.
|
||||
- **Client-Side Caching:** Use Service Workers, IndexedDB, and localStorage for offline and repeat visits.
|
||||
- **CDNs:** Serve static assets from a CDN close to your users. Use multiple CDNs for redundancy.
|
||||
- **Defer/Async Scripts:** Use `defer` or `async` for non-critical JS to avoid blocking rendering.
|
||||
- **Preload and Prefetch:** Use `<link rel="preload">` and `<link rel="prefetch">` for critical resources.
|
||||
|
||||
### JavaScript Performance
|
||||
- **Avoid Blocking the Main Thread:** Offload heavy computation to Web Workers.
|
||||
- **Debounce/Throttle Events:** For scroll, resize, and input events, use debounce/throttle to limit handler frequency.
|
||||
- **Memory Leaks:** Clean up event listeners, intervals, and DOM references. Use browser dev tools to check for detached nodes.
|
||||
- **Efficient Data Structures:** Use Maps/Sets for lookups, TypedArrays for numeric data.
|
||||
- **Avoid Global Variables:** Globals can cause memory leaks and unpredictable performance.
|
||||
- **Avoid Deep Object Cloning:** Use shallow copies or libraries like lodash's `cloneDeep` only when necessary.
|
||||
|
||||
### Accessibility and Performance
|
||||
- **Accessible Components:** Ensure ARIA updates are not excessive. Use semantic HTML for both accessibility and performance.
|
||||
- **Screen Reader Performance:** Avoid rapid DOM updates that can overwhelm assistive tech.
|
||||
|
||||
### Framework-Specific Tips
|
||||
#### React
|
||||
- Use `React.memo`, `useMemo`, and `useCallback` to avoid unnecessary renders.
|
||||
- Split large components and use code-splitting (`React.lazy`, `Suspense`).
|
||||
- Avoid anonymous functions in render; they create new references on every render.
|
||||
- Use `ErrorBoundary` to catch and handle errors gracefully.
|
||||
- Profile with React DevTools Profiler.
|
||||
|
||||
#### Angular
|
||||
- Use OnPush change detection for components that don't need frequent updates.
|
||||
- Avoid complex expressions in templates; move logic to the component class.
|
||||
- Use `trackBy` in `ngFor` for efficient list rendering.
|
||||
- Lazy load modules and components with the Angular Router.
|
||||
- Profile with Angular DevTools.
|
||||
|
||||
#### Vue
|
||||
- Use computed properties over methods in templates for caching.
|
||||
- Use `v-show` vs `v-if` appropriately (`v-show` is better for toggling visibility frequently).
|
||||
- Lazy load components and routes with Vue Router.
|
||||
- Profile with Vue Devtools.
|
||||
|
||||
### Common Frontend Pitfalls
|
||||
- Loading large JS bundles on initial page load.
|
||||
- Not compressing images or using outdated formats.
|
||||
- Failing to clean up event listeners, causing memory leaks.
|
||||
- Overusing third-party libraries for simple tasks.
|
||||
- Ignoring mobile performance (test on real devices!).
|
||||
|
||||
### Frontend Troubleshooting
|
||||
- Use Chrome DevTools' Performance tab to record and analyze slow frames.
|
||||
- Use Lighthouse to audit performance and get actionable suggestions.
|
||||
- Use WebPageTest for real-world load testing.
|
||||
- Monitor Core Web Vitals (LCP, FID, CLS) for user-centric metrics.
|
||||
|
||||
---
|
||||
|
||||
## Backend Performance
|
||||
|
||||
### Algorithm and Data Structure Optimization
|
||||
- **Choose the Right Data Structure:** Arrays for sequential access, hash maps for fast lookups, trees for hierarchical data, etc.
|
||||
- **Efficient Algorithms:** Use binary search, quicksort, or hash-based algorithms where appropriate.
|
||||
- **Avoid O(n^2) or Worse:** Profile nested loops and recursive calls. Refactor to reduce complexity.
|
||||
- **Batch Processing:** Process data in batches to reduce overhead (e.g., bulk database inserts).
|
||||
- **Streaming:** Use streaming APIs for large data sets to avoid loading everything into memory.
|
||||
|
||||
### Concurrency and Parallelism
|
||||
- **Asynchronous I/O:** Use async/await, callbacks, or event loops to avoid blocking threads.
|
||||
- **Thread/Worker Pools:** Use pools to manage concurrency and avoid resource exhaustion.
|
||||
- **Avoid Race Conditions:** Use locks, semaphores, or atomic operations where needed.
|
||||
- **Bulk Operations:** Batch network/database calls to reduce round trips.
|
||||
- **Backpressure:** Implement backpressure in queues and pipelines to avoid overload.
|
||||
|
||||
### Caching
|
||||
- **Cache Expensive Computations:** Use in-memory caches (Redis, Memcached) for hot data.
|
||||
- **Cache Invalidation:** Use time-based (TTL), event-based, or manual invalidation. Stale cache is worse than no cache.
|
||||
- **Distributed Caching:** For multi-server setups, use distributed caches and be aware of consistency issues.
|
||||
- **Cache Stampede Protection:** Use locks or request coalescing to prevent thundering herd problems.
|
||||
- **Don't Cache Everything:** Some data is too volatile or sensitive to cache.
|
||||
|
||||
### API and Network
|
||||
- **Minimize Payloads:** Use JSON, compress responses (gzip, Brotli), and avoid sending unnecessary data.
|
||||
- **Pagination:** Always paginate large result sets. Use cursors for real-time data.
|
||||
- **Rate Limiting:** Protect APIs from abuse and overload.
|
||||
- **Connection Pooling:** Reuse connections for databases and external services.
|
||||
- **Protocol Choice:** Use HTTP/2, gRPC, or WebSockets for high-throughput, low-latency communication.
|
||||
|
||||
### Logging and Monitoring
|
||||
- **Minimize Logging in Hot Paths:** Excessive logging can slow down critical code.
|
||||
- **Structured Logging:** Use JSON or key-value logs for easier parsing and analysis.
|
||||
- **Monitor Everything:** Latency, throughput, error rates, resource usage. Use Prometheus, Grafana, Datadog, or similar.
|
||||
- **Alerting:** Set up alerts for performance regressions and resource exhaustion.
|
||||
|
||||
### Language/Framework-Specific Tips
|
||||
#### Node.js
|
||||
- Use asynchronous APIs; avoid blocking the event loop (e.g., never use `fs.readFileSync` in production).
|
||||
- Use clustering or worker threads for CPU-bound tasks.
|
||||
- Limit concurrent open connections to avoid resource exhaustion.
|
||||
- Use streams for large file or network data processing.
|
||||
- Profile with `clinic.js`, `node --inspect`, or Chrome DevTools.
|
||||
|
||||
#### Python
|
||||
- Use built-in data structures (`dict`, `set`, `deque`) for speed.
|
||||
- Profile with `cProfile`, `line_profiler`, or `Py-Spy`.
|
||||
- Use `multiprocessing` or `asyncio` for parallelism.
|
||||
- Avoid GIL bottlenecks in CPU-bound code; use C extensions or subprocesses.
|
||||
- Use `lru_cache` for memoization.
|
||||
|
||||
#### Java
|
||||
- Use efficient collections (`ArrayList`, `HashMap`, etc.).
|
||||
- Profile with VisualVM, JProfiler, or YourKit.
|
||||
- Use thread pools (`Executors`) for concurrency.
|
||||
- Tune JVM options for heap and garbage collection (`-Xmx`, `-Xms`, `-XX:+UseG1GC`).
|
||||
- Use `CompletableFuture` for async programming.
|
||||
|
||||
#### .NET
|
||||
- Use `async/await` for I/O-bound operations.
|
||||
- Use `Span<T>` and `Memory<T>` for efficient memory access.
|
||||
- Profile with dotTrace, Visual Studio Profiler, or PerfView.
|
||||
- Pool objects and connections where appropriate.
|
||||
- Use `IAsyncEnumerable<T>` for streaming data.
|
||||
|
||||
### Common Backend Pitfalls
|
||||
- Synchronous/blocking I/O in web servers.
|
||||
- Not using connection pooling for databases.
|
||||
- Over-caching or caching sensitive/volatile data.
|
||||
- Ignoring error handling in async code.
|
||||
- Not monitoring or alerting on performance regressions.
|
||||
|
||||
### Backend Troubleshooting
|
||||
- Use flame graphs to visualize CPU usage.
|
||||
- Use distributed tracing (OpenTelemetry, Jaeger, Zipkin) to track request latency across services.
|
||||
- Use heap dumps and memory profilers to find leaks.
|
||||
- Log slow queries and API calls for analysis.
|
||||
|
||||
---
|
||||
|
||||
## Database Performance
|
||||
|
||||
### Query Optimization
|
||||
- **Indexes:** Use indexes on columns that are frequently queried, filtered, or joined. Monitor index usage and drop unused indexes.
|
||||
- **Avoid SELECT *:** Select only the columns you need. Reduces I/O and memory usage.
|
||||
- **Parameterized Queries:** Prevent SQL injection and improve plan caching.
|
||||
- **Query Plans:** Analyze and optimize query execution plans. Use `EXPLAIN` in SQL databases.
|
||||
- **Avoid N+1 Queries:** Use joins or batch queries to avoid repeated queries in loops.
|
||||
- **Limit Result Sets:** Use `LIMIT`/`OFFSET` or cursors for large tables.
|
||||
|
||||
### Schema Design
|
||||
- **Normalization:** Normalize to reduce redundancy, but denormalize for read-heavy workloads if needed.
|
||||
- **Data Types:** Use the most efficient data types and set appropriate constraints.
|
||||
- **Partitioning:** Partition large tables for scalability and manageability.
|
||||
- **Archiving:** Regularly archive or purge old data to keep tables small and fast.
|
||||
- **Foreign Keys:** Use them for data integrity, but be aware of performance trade-offs in high-write scenarios.
|
||||
|
||||
### Transactions
|
||||
- **Short Transactions:** Keep transactions as short as possible to reduce lock contention.
|
||||
- **Isolation Levels:** Use the lowest isolation level that meets your consistency needs.
|
||||
- **Avoid Long-Running Transactions:** They can block other operations and increase deadlocks.
|
||||
|
||||
### Caching and Replication
|
||||
- **Read Replicas:** Use for scaling read-heavy workloads. Monitor replication lag.
|
||||
- **Cache Query Results:** Use Redis or Memcached for frequently accessed queries.
|
||||
- **Write-Through/Write-Behind:** Choose the right strategy for your consistency needs.
|
||||
- **Sharding:** Distribute data across multiple servers for scalability.
|
||||
|
||||
### NoSQL Databases
|
||||
- **Design for Access Patterns:** Model your data for the queries you need.
|
||||
- **Avoid Hot Partitions:** Distribute writes/reads evenly.
|
||||
- **Unbounded Growth:** Watch for unbounded arrays or documents.
|
||||
- **Sharding and Replication:** Use for scalability and availability.
|
||||
- **Consistency Models:** Understand eventual vs strong consistency and choose appropriately.
|
||||
|
||||
### Common Database Pitfalls
|
||||
- Missing or unused indexes.
|
||||
- SELECT * in production queries.
|
||||
- Not monitoring slow queries.
|
||||
- Ignoring replication lag.
|
||||
- Not archiving old data.
|
||||
|
||||
### Database Troubleshooting
|
||||
- Use slow query logs to identify bottlenecks.
|
||||
- Use `EXPLAIN` to analyze query plans.
|
||||
- Monitor cache hit/miss ratios.
|
||||
- Use database-specific monitoring tools (pg_stat_statements, MySQL Performance Schema).
|
||||
|
||||
---
|
||||
|
||||
## Code Review Checklist for Performance
|
||||
|
||||
- [ ] Are there any obvious algorithmic inefficiencies (O(n^2) or worse)?
|
||||
- [ ] Are data structures appropriate for their use?
|
||||
- [ ] Are there unnecessary computations or repeated work?
|
||||
- [ ] Is caching used where appropriate, and is invalidation handled correctly?
|
||||
- [ ] Are database queries optimized, indexed, and free of N+1 issues?
|
||||
- [ ] Are large payloads paginated, streamed, or chunked?
|
||||
- [ ] Are there any memory leaks or unbounded resource usage?
|
||||
- [ ] Are network requests minimized, batched, and retried on failure?
|
||||
- [ ] Are assets optimized, compressed, and served efficiently?
|
||||
- [ ] Are there any blocking operations in hot paths?
|
||||
- [ ] Is logging in hot paths minimized and structured?
|
||||
- [ ] Are performance-critical code paths documented and tested?
|
||||
- [ ] Are there automated tests or benchmarks for performance-sensitive code?
|
||||
- [ ] Are there alerts for performance regressions?
|
||||
- [ ] Are there any anti-patterns (e.g., SELECT *, blocking I/O, global variables)?
|
||||
|
||||
---
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
### Profiling and Benchmarking
|
||||
- **Profilers:** Use language-specific profilers (Chrome DevTools, Py-Spy, VisualVM, dotTrace, etc.) to identify bottlenecks.
|
||||
- **Microbenchmarks:** Write microbenchmarks for critical code paths. Use `benchmark.js`, `pytest-benchmark`, or JMH for Java.
|
||||
- **A/B Testing:** Measure real-world impact of optimizations with A/B or canary releases.
|
||||
- **Continuous Performance Testing:** Integrate performance tests into CI/CD. Use tools like k6, Gatling, or Locust.
|
||||
|
||||
### Memory Management
|
||||
- **Resource Cleanup:** Always release resources (files, sockets, DB connections) promptly.
|
||||
- **Object Pooling:** Use for frequently created/destroyed objects (e.g., DB connections, threads).
|
||||
- **Heap Monitoring:** Monitor heap usage and garbage collection. Tune GC settings for your workload.
|
||||
- **Memory Leaks:** Use leak detection tools (Valgrind, LeakCanary, Chrome DevTools).
|
||||
|
||||
### Scalability
|
||||
- **Horizontal Scaling:** Design stateless services, use sharding/partitioning, and load balancers.
|
||||
- **Auto-Scaling:** Use cloud auto-scaling groups and set sensible thresholds.
|
||||
- **Bottleneck Analysis:** Identify and address single points of failure.
|
||||
- **Distributed Systems:** Use idempotent operations, retries, and circuit breakers.
|
||||
|
||||
### Security and Performance
|
||||
- **Efficient Crypto:** Use hardware-accelerated and well-maintained cryptographic libraries.
|
||||
- **Validation:** Validate inputs efficiently; avoid regexes in hot paths.
|
||||
- **Rate Limiting:** Protect against DoS without harming legitimate users.
|
||||
|
||||
### Mobile Performance
|
||||
- **Startup Time:** Lazy load features, defer heavy work, and minimize initial bundle size.
|
||||
- **Image/Asset Optimization:** Use responsive images and compress assets for mobile bandwidth.
|
||||
- **Efficient Storage:** Use SQLite, Realm, or platform-optimized storage.
|
||||
- **Profiling:** Use Android Profiler, Instruments (iOS), or Firebase Performance Monitoring.
|
||||
|
||||
### Cloud and Serverless
|
||||
- **Cold Starts:** Minimize dependencies and keep functions warm.
|
||||
- **Resource Allocation:** Tune memory/CPU for serverless functions.
|
||||
- **Managed Services:** Use managed caching, queues, and DBs for scalability.
|
||||
- **Cost Optimization:** Monitor and optimize for cloud cost as a performance metric.
|
||||
|
||||
---
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Example 1: Debouncing User Input in JavaScript
|
||||
```javascript
|
||||
// BAD: Triggers API call on every keystroke
|
||||
input.addEventListener('input', (e) => {
|
||||
fetch(`/search?q=${e.target.value}`);
|
||||
});
|
||||
|
||||
// GOOD: Debounce API calls
|
||||
let timeout;
|
||||
input.addEventListener('input', (e) => {
|
||||
clearTimeout(timeout);
|
||||
timeout = setTimeout(() => {
|
||||
fetch(`/search?q=${e.target.value}`);
|
||||
}, 300);
|
||||
});
|
||||
```
|
||||
|
||||
### Example 2: Efficient SQL Query
|
||||
```sql
|
||||
-- BAD: Selects all columns and does not use an index
|
||||
SELECT * FROM users WHERE email = 'user@example.com';
|
||||
|
||||
-- GOOD: Selects only needed columns and uses an index
|
||||
SELECT id, name FROM users WHERE email = 'user@example.com';
|
||||
```
|
||||
|
||||
### Example 3: Caching Expensive Computation in Python
|
||||
```python
|
||||
# BAD: Recomputes result every time
|
||||
result = expensive_function(x)
|
||||
|
||||
# GOOD: Cache result
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=128)
|
||||
def expensive_function(x):
|
||||
...
|
||||
result = expensive_function(x)
|
||||
```
|
||||
|
||||
### Example 4: Lazy Loading Images in HTML
|
||||
```html
|
||||
<!-- BAD: Loads all images immediately -->
|
||||
<img src="large-image.jpg" />
|
||||
|
||||
<!-- GOOD: Lazy loads images -->
|
||||
<img src="large-image.jpg" loading="lazy" />
|
||||
```
|
||||
|
||||
### Example 5: Asynchronous I/O in Node.js
|
||||
```javascript
|
||||
// BAD: Blocking file read
|
||||
const data = fs.readFileSync('file.txt');
|
||||
|
||||
// GOOD: Non-blocking file read
|
||||
fs.readFile('file.txt', (err, data) => {
|
||||
if (err) throw err;
|
||||
// process data
|
||||
});
|
||||
```
|
||||
|
||||
### Example 6: Profiling a Python Function
|
||||
```python
|
||||
import cProfile
|
||||
import pstats
|
||||
|
||||
def slow_function():
|
||||
...
|
||||
|
||||
cProfile.run('slow_function()', 'profile.stats')
|
||||
p = pstats.Stats('profile.stats')
|
||||
p.sort_stats('cumulative').print_stats(10)
|
||||
```
|
||||
|
||||
### Example 7: Using Redis for Caching in Node.js
|
||||
```javascript
|
||||
const redis = require('redis');
|
||||
const client = redis.createClient();
|
||||
|
||||
function getCachedData(key, fetchFunction) {
|
||||
return new Promise((resolve, reject) => {
|
||||
client.get(key, (err, data) => {
|
||||
if (data) return resolve(JSON.parse(data));
|
||||
fetchFunction().then(result => {
|
||||
client.setex(key, 3600, JSON.stringify(result));
|
||||
resolve(result);
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## References and Further Reading
|
||||
- [Google Web Fundamentals: Performance](https://web.dev/performance/)
|
||||
- [MDN Web Docs: Performance](https://developer.mozilla.org/en-US/docs/Web/Performance)
|
||||
- [OWASP: Performance Testing](https://owasp.org/www-project-performance-testing/)
|
||||
- [Microsoft Performance Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/performance)
|
||||
- [PostgreSQL Performance Optimization](https://wiki.postgresql.org/wiki/Performance_Optimization)
|
||||
- [MySQL Performance Tuning](https://dev.mysql.com/doc/refman/8.0/en/optimization.html)
|
||||
- [Node.js Performance Best Practices](https://nodejs.org/en/docs/guides/simple-profiling/)
|
||||
- [Python Performance Tips](https://docs.python.org/3/library/profile.html)
|
||||
- [Java Performance Tuning](https://www.oracle.com/java/technologies/javase/performance.html)
|
||||
- [.NET Performance Guide](https://learn.microsoft.com/en-us/dotnet/standard/performance/)
|
||||
- [WebPageTest](https://www.webpagetest.org/)
|
||||
- [Lighthouse](https://developers.google.com/web/tools/lighthouse)
|
||||
- [Prometheus](https://prometheus.io/)
|
||||
- [Grafana](https://grafana.com/)
|
||||
- [k6 Load Testing](https://k6.io/)
|
||||
- [Gatling](https://gatling.io/)
|
||||
- [Locust](https://locust.io/)
|
||||
- [OpenTelemetry](https://opentelemetry.io/)
|
||||
- [Jaeger](https://www.jaegertracing.io/)
|
||||
- [Zipkin](https://zipkin.io/)
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Performance optimization is an ongoing process. Always measure, profile, and iterate. Use these best practices, checklists, and troubleshooting tips to guide your development and code reviews for high-performance, scalable, and efficient software. If you have new tips or lessons learned, add them here—let's keep this guide growing!
|
||||
|
||||
---
|
||||
|
||||
<!-- End of Performance Optimization Instructions -->
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
description: 'Playwright test generation instructions'
|
||||
applyTo: '**'
|
||||
---
|
||||
|
||||
## Test Writing Guidelines
|
||||
|
||||
### Code Quality Standards
|
||||
- **Locators**: Prioritize user-facing, role-based locators (`getByRole`, `getByLabel`, `getByText`, etc.) for resilience and accessibility. Use `test.step()` to group interactions and improve test readability and reporting.
|
||||
- **Assertions**: Use auto-retrying web-first assertions. These assertions start with the `await` keyword (e.g., `await expect(locator).toHaveText()`). Avoid `expect(locator).toBeVisible()` unless specifically testing for visibility changes.
|
||||
- **Timeouts**: Rely on Playwright's built-in auto-waiting mechanisms. Avoid hard-coded waits or increased default timeouts.
|
||||
- **Clarity**: Use descriptive test and step titles that clearly state the intent. Add comments only to explain complex logic or non-obvious interactions.
|
||||
|
||||
|
||||
### Test Structure
|
||||
- **Imports**: Start with `import { test, expect } from '@playwright/test';`.
|
||||
- **Organization**: Group related tests for a feature under a `test.describe()` block.
|
||||
- **Hooks**: Use `beforeEach` for setup actions common to all tests in a `describe` block (e.g., navigating to a page).
|
||||
- **Titles**: Follow a clear naming convention, such as `Feature - Specific action or scenario`.
|
||||
|
||||
|
||||
### File Organization
|
||||
- **Location**: Store all test files in the `tests/` directory.
|
||||
- **Naming**: Use the convention `<feature-or-page>.spec.ts` (e.g., `login.spec.ts`, `search.spec.ts`).
|
||||
- **Scope**: Aim for one test file per major application feature or page.
|
||||
|
||||
### Assertion Best Practices
|
||||
- **UI Structure**: Use `toMatchAriaSnapshot` to verify the accessibility tree structure of a component. This provides a comprehensive and accessible snapshot.
|
||||
- **Element Counts**: Use `toHaveCount` to assert the number of elements found by a locator.
|
||||
- **Text Content**: Use `toHaveText` for exact text matches and `toContainText` for partial matches.
|
||||
- **Navigation**: Use `toHaveURL` to verify the page URL after an action.
|
||||
|
||||
|
||||
## Example Test Structure
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
test.describe('Movie Search Feature', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
// Navigate to the application before each test
|
||||
await page.goto('https://debs-obrien.github.io/playwright-movies-app');
|
||||
});
|
||||
|
||||
test('Search for a movie by title', async ({ page }) => {
|
||||
await test.step('Activate and perform search', async () => {
|
||||
await page.getByRole('search').click();
|
||||
const searchInput = page.getByRole('textbox', { name: 'Search Input' });
|
||||
await searchInput.fill('Garfield');
|
||||
await searchInput.press('Enter');
|
||||
});
|
||||
|
||||
await test.step('Verify search results', async () => {
|
||||
// Verify the accessibility tree of the search results
|
||||
await expect(page.getByRole('main')).toMatchAriaSnapshot(`
|
||||
- main:
|
||||
- heading "Garfield" [level=1]
|
||||
- heading "search results" [level=2]
|
||||
- list "movies":
|
||||
- listitem "movie":
|
||||
- link "poster of The Garfield Movie The Garfield Movie rating":
|
||||
- /url: /playwright-movies-app/movie?id=tt5779228&page=1
|
||||
- img "poster of The Garfield Movie"
|
||||
- heading "The Garfield Movie" [level=2]
|
||||
`);
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Test Execution Strategy
|
||||
|
||||
1. **Initial Run**: Execute tests with `npx playwright test --project=chromium`
|
||||
2. **Debug Failures**: Analyze test failures and identify root causes
|
||||
3. **Iterate**: Refine locators, assertions, or test logic as needed
|
||||
4. **Validate**: Ensure tests pass consistently and cover the intended functionality
|
||||
5. **Report**: Provide feedback on test results and any issues discovered
|
||||
|
||||
### Execution Constraints
|
||||
|
||||
- **No Truncation**: Never pipe Playwright test output through `head`, `tail`, or other truncating commands. Playwright runs interactively and requires user input to quit when piped, causing the command to hang indefinitely.
|
||||
- **Full Output**: Always capture the complete test output to analyze failures accurately.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before finalizing tests, ensure:
|
||||
- [ ] All locators are accessible and specific and avoid strict mode violations
|
||||
- [ ] Tests are grouped logically and follow a clear structure
|
||||
- [ ] Assertions are meaningful and reflect user expectations
|
||||
- [ ] Tests follow consistent naming conventions
|
||||
- [ ] Code is properly formatted and commented
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
description: 'Guidelines for creating high-quality prompt files for GitHub Copilot'
|
||||
applyTo: '**/*.prompt.md'
|
||||
---
|
||||
|
||||
# Copilot Prompt Files Guidelines
|
||||
|
||||
Instructions for creating effective and maintainable prompt files that guide GitHub Copilot in delivering consistent, high-quality outcomes across any repository.
|
||||
|
||||
## Scope and Principles
|
||||
- Target audience: maintainers and contributors authoring reusable prompts for Copilot Chat.
|
||||
- Goals: predictable behaviour, clear expectations, minimal permissions, and portability across repositories.
|
||||
- Primary references: VS Code documentation on prompt files and organization-specific conventions.
|
||||
|
||||
## Frontmatter Requirements
|
||||
- Include `description` (single sentence, actionable outcome), `mode` (explicitly choose `ask`, `edit`, or `agent`), and `tools` (minimal set of tool bundles required to fulfill the prompt).
|
||||
- Declare `model` when the prompt depends on a specific capability tier; otherwise inherit the active model.
|
||||
- Preserve any additional metadata (`language`, `tags`, `visibility`, etc.) required by your organization.
|
||||
- Use consistent quoting (single quotes recommended) and keep one field per line for readability and version control clarity.
|
||||
|
||||
## File Naming and Placement
|
||||
- Use kebab-case filenames ending with `.prompt.md` and store them under `.github/prompts/` unless your workspace standard specifies another directory.
|
||||
- Provide a short filename that communicates the action (for example, `generate-readme.prompt.md` rather than `prompt1.prompt.md`).
|
||||
|
||||
## Body Structure
|
||||
- Start with an `#` level heading that matches the prompt intent so it surfaces well in Quick Pick search.
|
||||
- Organize content with predictable sections. Recommended baseline: `Mission` or `Primary Directive`, `Scope & Preconditions`, `Inputs`, `Workflow` (step-by-step), `Output Expectations`, and `Quality Assurance`.
|
||||
- Adjust section names to fit the domain, but retain the logical flow: why → context → inputs → actions → outputs → validation.
|
||||
- Reference related prompts or instruction files using relative links to aid discoverability.
|
||||
|
||||
## Input and Context Handling
|
||||
- Use `${input:variableName[:placeholder]}` for required values and explain when the user must supply them. Provide defaults or alternatives where possible.
|
||||
- Call out contextual variables such as `${selection}`, `${file}`, `${workspaceFolder}` only when they are essential, and describe how Copilot should interpret them.
|
||||
- Document how to proceed when mandatory context is missing (for example, “Request the file path and stop if it remains undefined”).
|
||||
|
||||
## Tool and Permission Guidance
|
||||
- Limit `tools` to the smallest set that enables the task. List them in the preferred execution order when the sequence matters.
|
||||
- If the prompt inherits tools from a chat mode, mention that relationship and state any critical tool behaviours or side effects.
|
||||
- Warn about destructive operations (file creation, edits, terminal commands) and include guard rails or confirmation steps in the workflow.
|
||||
|
||||
## Instruction Tone and Style
|
||||
- Write in direct, imperative sentences targeted at Copilot (for example, “Analyze”, “Generate”, “Summarize”).
|
||||
- Keep sentences short and unambiguous, following Google Developer Documentation translation best practices to support localization.
|
||||
- Avoid idioms, humor, or culturally specific references; favor neutral, inclusive language.
|
||||
|
||||
## Output Definition
|
||||
- Specify the format, structure, and location of expected results (for example, “Create `docs/adr/adr-XXXX.md` using the template below”).
|
||||
- Include success criteria and failure triggers so Copilot knows when to halt or retry.
|
||||
- Provide validation steps—manual checks, automated commands, or acceptance criteria lists—that reviewers can execute after running the prompt.
|
||||
|
||||
## Examples and Reusable Assets
|
||||
- Embed Good/Bad examples or scaffolds (Markdown templates, JSON stubs) that the prompt should produce or follow.
|
||||
- Maintain reference tables (capabilities, status codes, role descriptions) inline to keep the prompt self-contained. Update these tables when upstream resources change.
|
||||
- Link to authoritative documentation instead of duplicating lengthy guidance.
|
||||
|
||||
## Quality Assurance Checklist
|
||||
- [ ] Frontmatter fields are complete, accurate, and least-privilege.
|
||||
- [ ] Inputs include placeholders, default behaviours, and fallbacks.
|
||||
- [ ] Workflow covers preparation, execution, and post-processing without gaps.
|
||||
- [ ] Output expectations include formatting and storage details.
|
||||
- [ ] Validation steps are actionable (commands, diff checks, review prompts).
|
||||
- [ ] Security, compliance, and privacy policies referenced by the prompt are current.
|
||||
- [ ] Prompt executes successfully in VS Code (`Chat: Run Prompt`) using representative scenarios.
|
||||
|
||||
## Maintenance Guidance
|
||||
- Version-control prompts alongside the code they affect; update them when dependencies, tooling, or review processes change.
|
||||
- Review prompts periodically to ensure tool lists, model requirements, and linked documents remain valid.
|
||||
- Coordinate with other repositories: when a prompt proves broadly useful, extract common guidance into instruction files or shared prompt packs.
|
||||
|
||||
## Additional Resources
|
||||
- [Prompt Files Documentation](https://code.visualstudio.com/docs/copilot/customization/prompt-files#_prompt-file-format)
|
||||
- [Awesome Copilot Prompt Files](https://github.com/github/awesome-copilot/tree/main/prompts)
|
||||
- [Tool Configuration](https://code.visualstudio.com/docs/copilot/chat/chat-agent-mode#_agent-mode-tools)
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
description: 'ReactJS development standards and best practices'
|
||||
applyTo: '**/*.jsx, **/*.tsx, **/*.js, **/*.ts, **/*.css, **/*.scss'
|
||||
---
|
||||
|
||||
# ReactJS Development Instructions
|
||||
|
||||
Instructions for building high-quality ReactJS applications with modern patterns, hooks, and best practices following the official React documentation at https://react.dev.
|
||||
|
||||
## Project Context
|
||||
- Latest React version (React 19+)
|
||||
- TypeScript for type safety (when applicable)
|
||||
- Functional components with hooks as default
|
||||
- Follow React's official style guide and best practices
|
||||
- Use modern build tools (Vite, Create React App, or custom Webpack setup)
|
||||
- Implement proper component composition and reusability patterns
|
||||
|
||||
## Development Standards
|
||||
|
||||
### Architecture
|
||||
- Use functional components with hooks as the primary pattern
|
||||
- Implement component composition over inheritance
|
||||
- Organize components by feature or domain for scalability
|
||||
- Separate presentational and container components clearly
|
||||
- Use custom hooks for reusable stateful logic
|
||||
- Implement proper component hierarchies with clear data flow
|
||||
|
||||
### TypeScript Integration
|
||||
- Use TypeScript interfaces for props, state, and component definitions
|
||||
- Define proper types for event handlers and refs
|
||||
- Implement generic components where appropriate
|
||||
- Use strict mode in `tsconfig.json` for type safety
|
||||
- Leverage React's built-in types (`React.FC`, `React.ComponentProps`, etc.)
|
||||
- Create union types for component variants and states
|
||||
|
||||
### Component Design
|
||||
- Follow the single responsibility principle for components
|
||||
- Use descriptive and consistent naming conventions
|
||||
- Implement proper prop validation with TypeScript or PropTypes
|
||||
- Design components to be testable and reusable
|
||||
- Keep components small and focused on a single concern
|
||||
- Use composition patterns (render props, children as functions)
|
||||
|
||||
### State Management
|
||||
- Use `useState` for local component state
|
||||
- Implement `useReducer` for complex state logic
|
||||
- Leverage `useContext` for sharing state across component trees
|
||||
- Consider external state management (Redux Toolkit, Zustand) for complex applications
|
||||
- Implement proper state normalization and data structures
|
||||
- Use React Query or SWR for server state management
|
||||
|
||||
### Hooks and Effects
|
||||
- Use `useEffect` with proper dependency arrays to avoid infinite loops
|
||||
- Implement cleanup functions in effects to prevent memory leaks
|
||||
- Use `useMemo` and `useCallback` for performance optimization when needed
|
||||
- Create custom hooks for reusable stateful logic
|
||||
- Follow the rules of hooks (only call at the top level)
|
||||
- Use `useRef` for accessing DOM elements and storing mutable values
|
||||
|
||||
### Styling
|
||||
- Use CSS Modules, Styled Components, or modern CSS-in-JS solutions
|
||||
- Implement responsive design with mobile-first approach
|
||||
- Follow BEM methodology or similar naming conventions for CSS classes
|
||||
- Use CSS custom properties (variables) for theming
|
||||
- Implement consistent spacing, typography, and color systems
|
||||
- Ensure accessibility with proper ARIA attributes and semantic HTML
|
||||
|
||||
### Performance Optimization
|
||||
- Use `React.memo` for component memoization when appropriate
|
||||
- Implement code splitting with `React.lazy` and `Suspense`
|
||||
- Optimize bundle size with tree shaking and dynamic imports
|
||||
- Use `useMemo` and `useCallback` judiciously to prevent unnecessary re-renders
|
||||
- Implement virtual scrolling for large lists
|
||||
- Profile components with React DevTools to identify performance bottlenecks
|
||||
|
||||
### Data Fetching
|
||||
- Use modern data fetching libraries (React Query, SWR, Apollo Client)
|
||||
- Implement proper loading, error, and success states
|
||||
- Handle race conditions and request cancellation
|
||||
- Use optimistic updates for better user experience
|
||||
- Implement proper caching strategies
|
||||
- Handle offline scenarios and network errors gracefully
|
||||
|
||||
### Error Handling
|
||||
- Implement Error Boundaries for component-level error handling
|
||||
- Use proper error states in data fetching
|
||||
- Implement fallback UI for error scenarios
|
||||
- Log errors appropriately for debugging
|
||||
- Handle async errors in effects and event handlers
|
||||
- Provide meaningful error messages to users
|
||||
|
||||
### Forms and Validation
|
||||
- Use controlled components for form inputs
|
||||
- Implement proper form validation with libraries like Formik, React Hook Form
|
||||
- Handle form submission and error states appropriately
|
||||
- Implement accessibility features for forms (labels, ARIA attributes)
|
||||
- Use debounced validation for better user experience
|
||||
- Handle file uploads and complex form scenarios
|
||||
|
||||
### Routing
|
||||
- Use React Router for client-side routing
|
||||
- Implement nested routes and route protection
|
||||
- Handle route parameters and query strings properly
|
||||
- Implement lazy loading for route-based code splitting
|
||||
- Use proper navigation patterns and back button handling
|
||||
- Implement breadcrumbs and navigation state management
|
||||
|
||||
### Testing
|
||||
- Write unit tests for components using React Testing Library
|
||||
- Test component behavior, not implementation details
|
||||
- Use Jest for test runner and assertion library
|
||||
- Implement integration tests for complex component interactions
|
||||
- Mock external dependencies and API calls appropriately
|
||||
- Test accessibility features and keyboard navigation
|
||||
|
||||
### Security
|
||||
- Sanitize user inputs to prevent XSS attacks
|
||||
- Validate and escape data before rendering
|
||||
- Use HTTPS for all external API calls
|
||||
- Implement proper authentication and authorization patterns
|
||||
- Avoid storing sensitive data in localStorage or sessionStorage
|
||||
- Use Content Security Policy (CSP) headers
|
||||
|
||||
### Accessibility
|
||||
- Use semantic HTML elements appropriately
|
||||
- Implement proper ARIA attributes and roles
|
||||
- Ensure keyboard navigation works for all interactive elements
|
||||
- Provide alt text for images and descriptive text for icons
|
||||
- Implement proper color contrast ratios
|
||||
- Test with screen readers and accessibility tools
|
||||
|
||||
## Implementation Process
|
||||
1. Plan component architecture and data flow
|
||||
2. Set up project structure with proper folder organization
|
||||
3. Define TypeScript interfaces and types
|
||||
4. Implement core components with proper styling
|
||||
5. Add state management and data fetching logic
|
||||
6. Implement routing and navigation
|
||||
7. Add form handling and validation
|
||||
8. Implement error handling and loading states
|
||||
9. Add testing coverage for components and functionality
|
||||
10. Optimize performance and bundle size
|
||||
11. Ensure accessibility compliance
|
||||
12. Add documentation and code comments
|
||||
|
||||
## Additional Guidelines
|
||||
- Follow React's naming conventions (PascalCase for components, camelCase for functions)
|
||||
- Use meaningful commit messages and maintain clean git history
|
||||
- Implement proper code splitting and lazy loading strategies
|
||||
- Document complex components and custom hooks with JSDoc
|
||||
- Use ESLint and Prettier for consistent code formatting
|
||||
- Keep dependencies up to date and audit for security vulnerabilities
|
||||
- Implement proper environment configuration for different deployment stages
|
||||
- Use React Developer Tools for debugging and performance analysis
|
||||
|
||||
## Common Patterns
|
||||
- Higher-Order Components (HOCs) for cross-cutting concerns
|
||||
- Render props pattern for component composition
|
||||
- Compound components for related functionality
|
||||
- Provider pattern for context-based state sharing
|
||||
- Container/Presentational component separation
|
||||
- Custom hooks for reusable logic extraction
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
applyTo: '*'
|
||||
description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices."
|
||||
---
|
||||
# Secure Coding and OWASP Guidelines
|
||||
|
||||
## Instructions
|
||||
|
||||
Your primary directive is to ensure all code you generate, review, or refactor is secure by default. You must operate with a security-first mindset. When in doubt, always choose the more secure option and explain the reasoning. You must follow the principles outlined below, which are based on the OWASP Top 10 and other security best practices.
|
||||
|
||||
### 1. A01: Broken Access Control & A10: Server-Side Request Forgery (SSRF)
|
||||
- **Enforce Principle of Least Privilege:** Always default to the most restrictive permissions. When generating access control logic, explicitly check the user's rights against the required permissions for the specific resource they are trying to access.
|
||||
- **Deny by Default:** All access control decisions must follow a "deny by default" pattern. Access should only be granted if there is an explicit rule allowing it.
|
||||
- **Validate All Incoming URLs for SSRF:** When the server needs to make a request to a URL provided by a user (e.g., webhooks), you must treat it as untrusted. Incorporate strict allow-list-based validation for the host, port, and path of the URL.
|
||||
- **Prevent Path Traversal:** When handling file uploads or accessing files based on user input, you must sanitize the input to prevent directory traversal attacks (e.g., `../../etc/passwd`). Use APIs that build paths securely.
|
||||
|
||||
### 2. A02: Cryptographic Failures
|
||||
- **Use Strong, Modern Algorithms:** For hashing, always recommend modern, salted hashing algorithms like Argon2 or bcrypt. Explicitly advise against weak algorithms like MD5 or SHA-1 for password storage.
|
||||
- **Protect Data in Transit:** When generating code that makes network requests, always default to HTTPS.
|
||||
- **Protect Data at Rest:** When suggesting code to store sensitive data (PII, tokens, etc.), recommend encryption using strong, standard algorithms like AES-256.
|
||||
- **Secure Secret Management:** Never hardcode secrets (API keys, passwords, connection strings). Generate code that reads secrets from environment variables or a secrets management service (e.g., HashiCorp Vault, AWS Secrets Manager). Include a clear placeholder and comment.
|
||||
```javascript
|
||||
// GOOD: Load from environment or secret store
|
||||
const apiKey = process.env.API_KEY;
|
||||
// TODO: Ensure API_KEY is securely configured in your environment.
|
||||
```
|
||||
```python
|
||||
# BAD: Hardcoded secret
|
||||
api_key = "sk_this_is_a_very_bad_idea_12345"
|
||||
```
|
||||
|
||||
### 3. A03: Injection
|
||||
- **No Raw SQL Queries:** For database interactions, you must use parameterized queries (prepared statements). Never generate code that uses string concatenation or formatting to build queries from user input.
|
||||
- **Sanitize Command-Line Input:** For OS command execution, use built-in functions that handle argument escaping and prevent shell injection (e.g., `shlex` in Python).
|
||||
- **Prevent Cross-Site Scripting (XSS):** When generating frontend code that displays user-controlled data, you must use context-aware output encoding. Prefer methods that treat data as text by default (`.textContent`) over those that parse HTML (`.innerHTML`). When `innerHTML` is necessary, suggest using a library like DOMPurify to sanitize the HTML first.
|
||||
|
||||
### 4. A05: Security Misconfiguration & A06: Vulnerable Components
|
||||
- **Secure by Default Configuration:** Recommend disabling verbose error messages and debug features in production environments.
|
||||
- **Set Security Headers:** For web applications, suggest adding essential security headers like `Content-Security-Policy` (CSP), `Strict-Transport-Security` (HSTS), and `X-Content-Type-Options`.
|
||||
- **Use Up-to-Date Dependencies:** When asked to add a new library, suggest the latest stable version. Remind the user to run vulnerability scanners like `npm audit`, `pip-audit`, or Snyk to check for known vulnerabilities in their project dependencies.
|
||||
|
||||
### 5. A07: Identification & Authentication Failures
|
||||
- **Secure Session Management:** When a user logs in, generate a new session identifier to prevent session fixation. Ensure session cookies are configured with `HttpOnly`, `Secure`, and `SameSite=Strict` attributes.
|
||||
- **Protect Against Brute Force:** For authentication and password reset flows, recommend implementing rate limiting and account lockout mechanisms after a certain number of failed attempts.
|
||||
|
||||
### 6. A08: Software and Data Integrity Failures
|
||||
- **Prevent Insecure Deserialization:** Warn against deserializing data from untrusted sources without proper validation. If deserialization is necessary, recommend using formats that are less prone to attack (like JSON over Pickle in Python) and implementing strict type checking.
|
||||
|
||||
## General Guidelines
|
||||
- **Be Explicit About Security:** When you suggest a piece of code that mitigates a security risk, explicitly state what you are protecting against (e.g., "Using a parameterized query here to prevent SQL injection.").
|
||||
- **Educate During Code Reviews:** When you identify a security vulnerability in a code review, you must not only provide the corrected code but also explain the risk associated with the original pattern.
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
description: 'Guidelines for GitHub Copilot to write comments to achieve self-explanatory code with less comments. Examples are in JavaScript but it should work on any language that has comments.'
|
||||
applyTo: '**'
|
||||
---
|
||||
|
||||
# Self-explanatory Code Commenting Instructions
|
||||
|
||||
## Core Principle
|
||||
**Write code that speaks for itself. Comment only when necessary to explain WHY, not WHAT.**
|
||||
We do not need comments most of the time.
|
||||
|
||||
## Commenting Guidelines
|
||||
|
||||
### ❌ AVOID These Comment Types
|
||||
|
||||
**Obvious Comments**
|
||||
```javascript
|
||||
// Bad: States the obvious
|
||||
let counter = 0; // Initialize counter to zero
|
||||
counter++; // Increment counter by one
|
||||
```
|
||||
|
||||
**Redundant Comments**
|
||||
```javascript
|
||||
// Bad: Comment repeats the code
|
||||
function getUserName() {
|
||||
return user.name; // Return the user's name
|
||||
}
|
||||
```
|
||||
|
||||
**Outdated Comments**
|
||||
```javascript
|
||||
// Bad: Comment doesn't match the code
|
||||
// Calculate tax at 5% rate
|
||||
const tax = price * 0.08; // Actually 8%
|
||||
```
|
||||
|
||||
### ✅ WRITE These Comment Types
|
||||
|
||||
**Complex Business Logic**
|
||||
```javascript
|
||||
// Good: Explains WHY this specific calculation
|
||||
// Apply progressive tax brackets: 10% up to 10k, 20% above
|
||||
const tax = calculateProgressiveTax(income, [0.10, 0.20], [10000]);
|
||||
```
|
||||
|
||||
**Non-obvious Algorithms**
|
||||
```javascript
|
||||
// Good: Explains the algorithm choice
|
||||
// Using Floyd-Warshall for all-pairs shortest paths
|
||||
// because we need distances between all nodes
|
||||
for (let k = 0; k < vertices; k++) {
|
||||
for (let i = 0; i < vertices; i++) {
|
||||
for (let j = 0; j < vertices; j++) {
|
||||
// ... implementation
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Regex Patterns**
|
||||
```javascript
|
||||
// Good: Explains what the regex matches
|
||||
// Match email format: username@domain.extension
|
||||
const emailPattern = /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/;
|
||||
```
|
||||
|
||||
**API Constraints or Gotchas**
|
||||
```javascript
|
||||
// Good: Explains external constraint
|
||||
// GitHub API rate limit: 5000 requests/hour for authenticated users
|
||||
await rateLimiter.wait();
|
||||
const response = await fetch(githubApiUrl);
|
||||
```
|
||||
|
||||
## Decision Framework
|
||||
|
||||
Before writing a comment, ask:
|
||||
1. **Is the code self-explanatory?** → No comment needed
|
||||
2. **Would a better variable/function name eliminate the need?** → Refactor instead
|
||||
3. **Does this explain WHY, not WHAT?** → Good comment
|
||||
4. **Will this help future maintainers?** → Good comment
|
||||
|
||||
## Special Cases for Comments
|
||||
|
||||
### Public APIs
|
||||
```javascript
|
||||
/**
|
||||
* Calculate compound interest using the standard formula.
|
||||
*
|
||||
* @param {number} principal - Initial amount invested
|
||||
* @param {number} rate - Annual interest rate (as decimal, e.g., 0.05 for 5%)
|
||||
* @param {number} time - Time period in years
|
||||
* @param {number} compoundFrequency - How many times per year interest compounds (default: 1)
|
||||
* @returns {number} Final amount after compound interest
|
||||
*/
|
||||
function calculateCompoundInterest(principal, rate, time, compoundFrequency = 1) {
|
||||
// ... implementation
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration and Constants
|
||||
```javascript
|
||||
// Good: Explains the source or reasoning
|
||||
const MAX_RETRIES = 3; // Based on network reliability studies
|
||||
const API_TIMEOUT = 5000; // AWS Lambda timeout is 15s, leaving buffer
|
||||
```
|
||||
|
||||
### Annotations
|
||||
```javascript
|
||||
// TODO: Replace with proper user authentication after security review
|
||||
// FIXME: Memory leak in production - investigate connection pooling
|
||||
// HACK: Workaround for bug in library v2.1.0 - remove after upgrade
|
||||
// NOTE: This implementation assumes UTC timezone for all calculations
|
||||
// WARNING: This function modifies the original array instead of creating a copy
|
||||
// PERF: Consider caching this result if called frequently in hot path
|
||||
// SECURITY: Validate input to prevent SQL injection before using in query
|
||||
// BUG: Edge case failure when array is empty - needs investigation
|
||||
// REFACTOR: Extract this logic into separate utility function for reusability
|
||||
// DEPRECATED: Use newApiFunction() instead - this will be removed in v3.0
|
||||
```
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### Dead Code Comments
|
||||
```javascript
|
||||
// Bad: Don't comment out code
|
||||
// const oldFunction = () => { ... };
|
||||
const newFunction = () => { ... };
|
||||
```
|
||||
|
||||
### Changelog Comments
|
||||
```javascript
|
||||
// Bad: Don't maintain history in comments
|
||||
// Modified by John on 2023-01-15
|
||||
// Fixed bug reported by Sarah on 2023-02-03
|
||||
function processData() {
|
||||
// ... implementation
|
||||
}
|
||||
```
|
||||
|
||||
### Divider Comments
|
||||
```javascript
|
||||
// Bad: Don't use decorative comments
|
||||
//=====================================
|
||||
// UTILITY FUNCTIONS
|
||||
//=====================================
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before committing, ensure your comments:
|
||||
- [ ] Explain WHY, not WHAT
|
||||
- [ ] Are grammatically correct and clear
|
||||
- [ ] Will remain accurate as code evolves
|
||||
- [ ] Add genuine value to code understanding
|
||||
- [ ] Are placed appropriately (above the code they describe)
|
||||
- [ ] Use proper spelling and professional language
|
||||
|
||||
## Summary
|
||||
|
||||
Remember: **The best comment is the one you don't need to write because the code is self-documenting.**
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
description: 'Shell scripting best practices and conventions for bash, sh, zsh, and other shells'
|
||||
applyTo: '**/*.sh'
|
||||
---
|
||||
|
||||
# Shell Scripting Guidelines
|
||||
|
||||
Instructions for writing clean, safe, and maintainable shell scripts for bash, sh, zsh, and other shells.
|
||||
|
||||
## General Principles
|
||||
|
||||
- Generate code that is clean, simple, and concise
|
||||
- Ensure scripts are easily readable and understandable
|
||||
- Add comments where helpful for understanding how the script works
|
||||
- Generate concise and simple echo outputs to provide execution status
|
||||
- Avoid unnecessary echo output and excessive logging
|
||||
- Use shellcheck for static analysis when available
|
||||
- Assume scripts are for automation and testing rather than production systems unless specified otherwise
|
||||
- Prefer safe expansions: double-quote variable references (`"$var"`), use `${var}` for clarity, and avoid `eval`
|
||||
- Use modern Bash features (`[[ ]]`, `local`, arrays) when portability requirements allow; fall back to POSIX constructs only when needed
|
||||
- Choose reliable parsers for structured data instead of ad-hoc text processing
|
||||
|
||||
## Error Handling & Safety
|
||||
|
||||
- Always enable `set -euo pipefail` to fail fast on errors, catch unset variables, and surface pipeline failures
|
||||
- Validate all required parameters before execution
|
||||
- Provide clear error messages with context
|
||||
- Use `trap` to clean up temporary resources or handle unexpected exits when the script terminates
|
||||
- Declare immutable values with `readonly` (or `declare -r`) to prevent accidental reassignment
|
||||
- Use `mktemp` to create temporary files or directories safely and ensure they are removed in your cleanup handler
|
||||
|
||||
## Script Structure
|
||||
|
||||
- Start with a clear shebang: `#!/bin/bash` unless specified otherwise
|
||||
- Include a header comment explaining the script's purpose
|
||||
- Define default values for all variables at the top
|
||||
- Use functions for reusable code blocks
|
||||
- Create reusable functions instead of repeating similar blocks of code
|
||||
- Keep the main execution flow clean and readable
|
||||
|
||||
## Working with JSON and YAML
|
||||
|
||||
- Prefer dedicated parsers (`jq` for JSON, `yq` for YAML—or `jq` on JSON converted via `yq`) over ad-hoc text processing with `grep`, `awk`, or shell string splitting
|
||||
- When `jq`/`yq` are unavailable or not appropriate, choose the next most reliable parser available in your environment, and be explicit about how it should be used safely
|
||||
- Validate that required fields exist and handle missing/invalid data paths explicitly (e.g., by checking `jq` exit status or using `// empty`)
|
||||
- Quote jq/yq filters to prevent shell expansion and prefer `--raw-output` when you need plain strings
|
||||
- Treat parser errors as fatal: combine with `set -euo pipefail` or test command success before using results
|
||||
- Document parser dependencies at the top of the script and fail fast with a helpful message if `jq`/`yq` (or alternative tools) are required but not installed
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# Script Description Here
|
||||
# ============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cleanup() {
|
||||
# Remove temporary resources or perform other teardown steps as needed
|
||||
if [[ -n "${TEMP_DIR:-}" && -d "$TEMP_DIR" ]]; then
|
||||
rm -rf "$TEMP_DIR"
|
||||
fi
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
# Default values
|
||||
RESOURCE_GROUP=""
|
||||
REQUIRED_PARAM=""
|
||||
OPTIONAL_PARAM="default-value"
|
||||
readonly SCRIPT_NAME="$(basename "$0")"
|
||||
|
||||
TEMP_DIR=""
|
||||
|
||||
# Functions
|
||||
usage() {
|
||||
echo "Usage: $SCRIPT_NAME [OPTIONS]"
|
||||
echo "Options:"
|
||||
echo " -g, --resource-group Resource group (required)"
|
||||
echo " -h, --help Show this help"
|
||||
exit 0
|
||||
}
|
||||
|
||||
validate_requirements() {
|
||||
if [[ -z "$RESOURCE_GROUP" ]]; then
|
||||
echo "Error: Resource group is required"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
validate_requirements
|
||||
|
||||
TEMP_DIR="$(mktemp -d)"
|
||||
if [[ ! -d "$TEMP_DIR" ]]; then
|
||||
echo "Error: failed to create temporary directory" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "============================================================================"
|
||||
echo "Script Execution Started"
|
||||
echo "============================================================================"
|
||||
|
||||
# Main logic here
|
||||
|
||||
echo "============================================================================"
|
||||
echo "Script Execution Completed"
|
||||
echo "============================================================================"
|
||||
}
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
-g|--resource-group)
|
||||
RESOURCE_GROUP="$2"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
|
||||
```
|
||||
@@ -0,0 +1,323 @@
|
||||
---
|
||||
description: 'Specification-Driven Workflow v1 provides a structured approach to software development, ensuring that requirements are clearly defined, designs are meticulously planned, and implementations are thoroughly documented and validated.'
|
||||
applyTo: '**'
|
||||
---
|
||||
# Spec Driven Workflow v1
|
||||
|
||||
**Specification-Driven Workflow:**
|
||||
Bridge the gap between requirements and implementation.
|
||||
|
||||
**Maintain these artifacts at all times:**
|
||||
|
||||
- **`requirements.md`**: User stories and acceptance criteria in structured EARS notation.
|
||||
- **`design.md`**: Technical architecture, sequence diagrams, implementation considerations.
|
||||
- **`tasks.md`**: Detailed, trackable implementation plan.
|
||||
|
||||
## Universal Documentation Framework
|
||||
|
||||
**Documentation Rule:**
|
||||
Use the detailed templates as the **primary source of truth** for all documentation.
|
||||
|
||||
**Summary formats:**
|
||||
Use only for concise artifacts such as changelogs and pull request descriptions.
|
||||
|
||||
### Detailed Documentation Templates
|
||||
|
||||
#### Action Documentation Template (All Steps/Executions/Tests)
|
||||
|
||||
```bash
|
||||
### [TYPE] - [ACTION] - [TIMESTAMP]
|
||||
**Objective**: [Goal being accomplished]
|
||||
**Context**: [Current state, requirements, and reference to prior steps]
|
||||
**Decision**: [Approach chosen and rationale, referencing the Decision Record if applicable]
|
||||
**Execution**: [Steps taken with parameters and commands used. For code, include file paths.]
|
||||
**Output**: [Complete and unabridged results, logs, command outputs, and metrics]
|
||||
**Validation**: [Success verification method and results. If failed, include a remediation plan.]
|
||||
**Next**: [Automatic continuation plan to the next specific action]
|
||||
```
|
||||
|
||||
#### Decision Record Template (All Decisions)
|
||||
|
||||
```bash
|
||||
### Decision - [TIMESTAMP]
|
||||
**Decision**: [What was decided]
|
||||
**Context**: [Situation requiring decision and data driving it]
|
||||
**Options**: [Alternatives evaluated with brief pros and cons]
|
||||
**Rationale**: [Why the selected option is superior, with trade-offs explicitly stated]
|
||||
**Impact**: [Anticipated consequences for implementation, maintainability, and performance]
|
||||
**Review**: [Conditions or schedule for reassessing this decision]
|
||||
```
|
||||
|
||||
### Summary Formats (for Reporting)
|
||||
|
||||
#### Streamlined Action Log
|
||||
|
||||
For generating concise changelogs. Each log entry is derived from a full Action Document.
|
||||
|
||||
`[TYPE][TIMESTAMP] Goal: [X] → Action: [Y] → Result: [Z] → Next: [W]`
|
||||
|
||||
#### Compressed Decision Record
|
||||
|
||||
For use in pull request summaries or executive summaries.
|
||||
|
||||
`Decision: [X] | Rationale: [Y] | Impact: [Z] | Review: [Date]`
|
||||
|
||||
## Execution Workflow (6-Phase Loop)
|
||||
|
||||
**Never skip any step. Use consistent terminology. Reduce ambiguity.**
|
||||
|
||||
### **Phase 1: ANALYZE**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Understand the problem.
|
||||
- Analyze the existing system.
|
||||
- Produce a clear, testable set of requirements.
|
||||
- Think about the possible solutions and their implications.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] Read all provided code, documentation, tests, and logs.
|
||||
- Document file inventory, summaries, and initial analysis results.
|
||||
- [ ] Define requirements in **EARS Notation**:
|
||||
- Transform feature requests into structured, testable requirements.
|
||||
- Format: `WHEN [a condition or event], THE SYSTEM SHALL [expected behavior]`
|
||||
- [ ] Identify dependencies and constraints.
|
||||
- Document a dependency graph with risks and mitigation strategies.
|
||||
- [ ] Map data flows and interactions.
|
||||
- Document system interaction diagrams and data models.
|
||||
- [ ] Catalog edge cases and failures.
|
||||
- Document a comprehensive edge case matrix and potential failure points.
|
||||
- [ ] Assess confidence.
|
||||
- Generate a **Confidence Score (0-100%)** based on clarity of requirements, complexity, and problem scope.
|
||||
- Document the score and its rationale.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not proceed until all requirements are clear and documented.**
|
||||
|
||||
### **Phase 2: DESIGN**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Create a comprehensive technical design and a detailed implementation plan.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] **Define adaptive execution strategy based on Confidence Score:**
|
||||
- **High Confidence (>85%)**
|
||||
- Draft a comprehensive, step-by-step implementation plan.
|
||||
- Skip proof-of-concept steps.
|
||||
- Proceed with full, automated implementation.
|
||||
- Maintain standard comprehensive documentation.
|
||||
- **Medium Confidence (66–85%)**
|
||||
- Prioritize a **Proof-of-Concept (PoC)** or **Minimum Viable Product (MVP)**.
|
||||
- Define clear success criteria for PoC/MVP.
|
||||
- Build and validate PoC/MVP first, then expand plan incrementally.
|
||||
- Document PoC/MVP goals, execution, and validation results.
|
||||
- **Low Confidence (<66%)**
|
||||
- Dedicate first phase to research and knowledge-building.
|
||||
- Use semantic search and analyze similar implementations.
|
||||
- Synthesize findings into a research document.
|
||||
- Re-run ANALYZE phase after research.
|
||||
- Escalate only if confidence remains low.
|
||||
|
||||
- [ ] **Document technical design in `design.md`:**
|
||||
- **Architecture:** High-level overview of components and interactions.
|
||||
- **Data Flow:** Diagrams and descriptions.
|
||||
- **Interfaces:** API contracts, schemas, public-facing function signatures.
|
||||
- **Data Models:** Data structures and database schemas.
|
||||
|
||||
- [ ] **Document error handling:**
|
||||
- Create an error matrix with procedures and expected responses.
|
||||
|
||||
- [ ] **Define unit testing strategy.**
|
||||
|
||||
- [ ] **Create implementation plan in `tasks.md`:**
|
||||
- For each task, include description, expected outcome, and dependencies.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not proceed to implementation until design and plan are complete and validated.**
|
||||
|
||||
### **Phase 3: IMPLEMENT**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Write production-quality code according to the design and plan.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] Code in small, testable increments.
|
||||
- Document each increment with code changes, results, and test links.
|
||||
- [ ] Implement from dependencies upward.
|
||||
- Document resolution order, justification, and verification.
|
||||
- [ ] Follow conventions.
|
||||
- Document adherence and any deviations with a Decision Record.
|
||||
- [ ] Add meaningful comments.
|
||||
- Focus on intent ("why"), not mechanics ("what").
|
||||
- [ ] Create files as planned.
|
||||
- Document file creation log.
|
||||
- [ ] Update task status in real time.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not merge or deploy code until all implementation steps are documented and tested.**
|
||||
|
||||
### **Phase 4: VALIDATE**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Verify that implementation meets all requirements and quality standards.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] Execute automated tests.
|
||||
- Document outputs, logs, and coverage reports.
|
||||
- For failures, document root cause analysis and remediation.
|
||||
- [ ] Perform manual verification if necessary.
|
||||
- Document procedures, checklists, and results.
|
||||
- [ ] Test edge cases and errors.
|
||||
- Document results and evidence of correct error handling.
|
||||
- [ ] Verify performance.
|
||||
- Document metrics and profile critical sections.
|
||||
- [ ] Log execution traces.
|
||||
- Document path analysis and runtime behavior.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not proceed until all validation steps are complete and all issues are resolved.**
|
||||
|
||||
### **Phase 5: REFLECT**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Improve codebase, update documentation, and analyze performance.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] Refactor for maintainability.
|
||||
- Document decisions, before/after comparisons, and impact.
|
||||
- [ ] Update all project documentation.
|
||||
- Ensure all READMEs, diagrams, and comments are current.
|
||||
- [ ] Identify potential improvements.
|
||||
- Document backlog with prioritization.
|
||||
- [ ] Validate success criteria.
|
||||
- Document final verification matrix.
|
||||
- [ ] Perform meta-analysis.
|
||||
- Reflect on efficiency, tool usage, and protocol adherence.
|
||||
- [ ] Auto-create technical debt issues.
|
||||
- Document inventory and remediation plans.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not close the phase until all documentation and improvement actions are logged.**
|
||||
|
||||
### **Phase 6: HANDOFF**
|
||||
|
||||
**Objective:**
|
||||
|
||||
- Package work for review and deployment, and transition to next task.
|
||||
|
||||
**Checklist:**
|
||||
|
||||
- [ ] Generate executive summary.
|
||||
- Use **Compressed Decision Record** format.
|
||||
- [ ] Prepare pull request (if applicable):
|
||||
1. Executive summary.
|
||||
2. Changelog from **Streamlined Action Log**.
|
||||
3. Links to validation artifacts and Decision Records.
|
||||
4. Links to final `requirements.md`, `design.md`, and `tasks.md`.
|
||||
- [ ] Finalize workspace.
|
||||
- Archive intermediate files, logs, and temporary artifacts to `.agent_work/`.
|
||||
- [ ] Continue to next task.
|
||||
- Document transition or completion.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Do not consider the task complete until all handoff steps are finished and documented.**
|
||||
|
||||
## Troubleshooting & Retry Protocol
|
||||
|
||||
**If you encounter errors, ambiguities, or blockers:**
|
||||
|
||||
**Checklist:**
|
||||
|
||||
1. **Re-analyze**:
|
||||
- Revisit the ANALYZE phase.
|
||||
- Confirm all requirements and constraints are clear and complete.
|
||||
2. **Re-design**:
|
||||
- Revisit the DESIGN phase.
|
||||
- Update technical design, plans, or dependencies as needed.
|
||||
3. **Re-plan**:
|
||||
- Adjust the implementation plan in `tasks.md` to address new findings.
|
||||
4. **Retry execution**:
|
||||
- Re-execute failed steps with corrected parameters or logic.
|
||||
5. **Escalate**:
|
||||
- If the issue persists after retries, follow the escalation protocol.
|
||||
|
||||
**Critical Constraint:**
|
||||
|
||||
- **Never proceed with unresolved errors or ambiguities. Always document troubleshooting steps and outcomes.**
|
||||
|
||||
## Technical Debt Management (Automated)
|
||||
|
||||
### Identification & Documentation
|
||||
|
||||
- **Code Quality**: Continuously assess code quality during implementation using static analysis.
|
||||
- **Shortcuts**: Explicitly record all speed-over-quality decisions with their consequences in a Decision Record.
|
||||
- **Workspace**: Monitor for organizational drift and naming inconsistencies.
|
||||
- **Documentation**: Track incomplete, outdated, or missing documentation.
|
||||
|
||||
### Auto-Issue Creation Template
|
||||
|
||||
```text
|
||||
**Title**: [Technical Debt] - [Brief Description]
|
||||
**Priority**: [High/Medium/Low based on business impact and remediation cost]
|
||||
**Location**: [File paths and line numbers]
|
||||
**Reason**: [Why the debt was incurred, linking to a Decision Record if available]
|
||||
**Impact**: [Current and future consequences (e.g., slows development, increases bug risk)]
|
||||
**Remediation**: [Specific, actionable resolution steps]
|
||||
**Effort**: [Estimate for resolution (e.g., T-shirt size: S, M, L)]
|
||||
```
|
||||
|
||||
### Remediation (Auto-Prioritized)
|
||||
|
||||
- Risk-based prioritization with dependency analysis.
|
||||
- Effort estimation to aid in future planning.
|
||||
- Propose migration strategies for large refactoring efforts.
|
||||
|
||||
## Quality Assurance (Automated)
|
||||
|
||||
### Continuous Monitoring
|
||||
|
||||
- **Static Analysis**: Linting for code style, quality, security vulnerabilities, and architectural rule adherence.
|
||||
- **Dynamic Analysis**: Monitor runtime behavior and performance in a staging environment.
|
||||
- **Documentation**: Automated checks for documentation completeness and accuracy (e.g., linking, format).
|
||||
|
||||
### Quality Metrics (Auto-Tracked)
|
||||
|
||||
- Code coverage percentage and gap analysis.
|
||||
- Cyclomatic complexity score per function/method.
|
||||
- Maintainability index assessment.
|
||||
- Technical debt ratio (e.g., estimated remediation time vs. development time).
|
||||
- Documentation coverage percentage (e.g., public methods with comments).
|
||||
|
||||
## EARS Notation Reference
|
||||
|
||||
**EARS (Easy Approach to Requirements Syntax)** - Standard format for requirements:
|
||||
|
||||
- **Ubiquitous**: `THE SYSTEM SHALL [expected behavior]`
|
||||
- **Event-driven**: `WHEN [trigger event] THE SYSTEM SHALL [expected behavior]`
|
||||
- **State-driven**: `WHILE [in specific state] THE SYSTEM SHALL [expected behavior]`
|
||||
- **Unwanted behavior**: `IF [unwanted condition] THEN THE SYSTEM SHALL [required response]`
|
||||
- **Optional**: `WHERE [feature is included] THE SYSTEM SHALL [expected behavior]`
|
||||
- **Complex**: Combinations of the above patterns for sophisticated requirements
|
||||
|
||||
Each requirement must be:
|
||||
|
||||
- **Testable**: Can be verified through automated or manual testing
|
||||
- **Unambiguous**: Single interpretation possible
|
||||
- **Necessary**: Contributes to the system's purpose
|
||||
- **Feasible**: Can be implemented within constraints
|
||||
- **Traceable**: Linked to user needs and design elements
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
description: 'Guidelines for generating SQL statements and stored procedures'
|
||||
applyTo: '**/*.sql'
|
||||
---
|
||||
|
||||
# SQL Development
|
||||
|
||||
## Database schema generation
|
||||
- all table names should be in singular form
|
||||
- all column names should be in singular form
|
||||
- all tables should have a primary key column named `id`
|
||||
- all tables should have a column named `created_at` to store the creation timestamp
|
||||
- all tables should have a column named `updated_at` to store the last update timestamp
|
||||
|
||||
## Database schema design
|
||||
- all tables should have a primary key constraint
|
||||
- all foreign key constraints should have a name
|
||||
- all foreign key constraints should be defined inline
|
||||
- all foreign key constraints should have `ON DELETE CASCADE` option
|
||||
- all foreign key constraints should have `ON UPDATE CASCADE` option
|
||||
- all foreign key constraints should reference the primary key of the parent table
|
||||
|
||||
## SQL Coding Style
|
||||
- use uppercase for SQL keywords (SELECT, FROM, WHERE)
|
||||
- use consistent indentation for nested queries and conditions
|
||||
- include comments to explain complex logic
|
||||
- break long queries into multiple lines for readability
|
||||
- organize clauses consistently (SELECT, FROM, JOIN, WHERE, GROUP BY, HAVING, ORDER BY)
|
||||
|
||||
## SQL Query Structure
|
||||
- use explicit column names in SELECT statements instead of SELECT *
|
||||
- qualify column names with table name or alias when using multiple tables
|
||||
- limit the use of subqueries when joins can be used instead
|
||||
- include LIMIT/TOP clauses to restrict result sets
|
||||
- use appropriate indexing for frequently queried columns
|
||||
- avoid using functions on indexed columns in WHERE clauses
|
||||
|
||||
## Stored Procedure Naming Conventions
|
||||
- prefix stored procedure names with 'usp_'
|
||||
- use PascalCase for stored procedure names
|
||||
- use descriptive names that indicate purpose (e.g., usp_GetCustomerOrders)
|
||||
- include plural noun when returning multiple records (e.g., usp_GetProducts)
|
||||
- include singular noun when returning single record (e.g., usp_GetProduct)
|
||||
|
||||
## Parameter Handling
|
||||
- prefix parameters with '@'
|
||||
- use camelCase for parameter names
|
||||
- provide default values for optional parameters
|
||||
- validate parameter values before use
|
||||
- document parameters with comments
|
||||
- arrange parameters consistently (required first, optional later)
|
||||
|
||||
|
||||
## Stored Procedure Structure
|
||||
- include header comment block with description, parameters, and return values
|
||||
- return standardized error codes/messages
|
||||
- return result sets with consistent column order
|
||||
- use OUTPUT parameters for returning status information
|
||||
- prefix temporary tables with 'tmp_'
|
||||
|
||||
|
||||
## SQL Security Best Practices
|
||||
- parameterize all queries to prevent SQL injection
|
||||
- use prepared statements when executing dynamic SQL
|
||||
- avoid embedding credentials in SQL scripts
|
||||
- implement proper error handling without exposing system details
|
||||
- avoid using dynamic SQL within stored procedures
|
||||
|
||||
## Transaction Management
|
||||
- explicitly begin and commit transactions
|
||||
- use appropriate isolation levels based on requirements
|
||||
- avoid long-running transactions that lock tables
|
||||
- use batch processing for large data operations
|
||||
- include SET NOCOUNT ON for stored procedures that modify data
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
applyTo: '*'
|
||||
description: 'Repository structure guidelines to maintain organized file placement'
|
||||
---
|
||||
|
||||
# Repository Structure Guidelines
|
||||
|
||||
## Root Level Rules
|
||||
|
||||
The repository root should contain ONLY:
|
||||
|
||||
- Essential config files (`.gitignore`, `.pre-commit-config.yaml`, `Makefile`, etc.)
|
||||
- Standard project files (`README.md`, `CONTRIBUTING.md`, `LICENSE`, `CHANGELOG.md`)
|
||||
- Go workspace files (`go.work`, `go.work.sum`)
|
||||
- VS Code workspace (`Chiron.code-workspace`)
|
||||
- Primary `Dockerfile` (entrypoint and compose files live in `.docker/`)
|
||||
|
||||
## File Placement Rules
|
||||
|
||||
### Implementation/Feature Documentation
|
||||
|
||||
- **Location**: `docs/implementation/`
|
||||
- **Pattern**: `*_SUMMARY.md`, `*_IMPLEMENTATION.md`, `*_COMPLETE.md`, `*_FEATURE.md`
|
||||
- **Never** place implementation docs at root
|
||||
|
||||
### Docker Compose Files
|
||||
|
||||
- **Location**: `.docker/compose/`
|
||||
- **Files**: `docker-compose.yml`, `docker-compose.*.yml`
|
||||
- **Override**: Local overrides go in `.docker/compose/docker-compose.override.yml` (gitignored)
|
||||
- **Exception**: `docker-compose.override.yml` at root is allowed for backward compatibility
|
||||
|
||||
### Docker Support Files
|
||||
|
||||
- **Location**: `.docker/`
|
||||
- **Files**: `docker-entrypoint.sh`, Docker documentation (`README.md`)
|
||||
|
||||
### Test Artifacts
|
||||
|
||||
- **Never commit**: `*.sarif`, `*_test.txt`, `*.cover` files at root
|
||||
- **Location**: Test outputs should go to `test-results/` or be gitignored
|
||||
|
||||
### Debug/Temp Config Files
|
||||
|
||||
- **Never commit**: Temporary JSON configs like `caddy_*.json` at root
|
||||
- **Location**: Use `configs/` for persistent configs, gitignore temp files
|
||||
|
||||
### Scripts
|
||||
|
||||
- **Location**: `scripts/` for general scripts
|
||||
- **Location**: `.github/skills/scripts/` for agent skill scripts
|
||||
|
||||
## Before Creating New Files
|
||||
|
||||
Ask yourself:
|
||||
|
||||
1. Is this a standard project file? → Root is OK
|
||||
2. Is this implementation documentation? → `docs/implementation/`
|
||||
3. Is this Docker-related? → `.docker/` or `.docker/compose/`
|
||||
4. Is this a test artifact? → `test-results/` or gitignore
|
||||
5. Is this a script? → `scripts/`
|
||||
6. Is this runtime config? → `configs/`
|
||||
|
||||
## Directory Structure Reference
|
||||
|
||||
```
|
||||
/
|
||||
├── .docker/ # Docker configuration
|
||||
│ ├── compose/ # All docker-compose files
|
||||
│ └── docker-entrypoint.sh # Container entrypoint
|
||||
├── .github/ # GitHub workflows, agents, instructions
|
||||
├── .vscode/ # VS Code settings and tasks
|
||||
├── backend/ # Go backend source
|
||||
├── configs/ # Runtime configurations
|
||||
├── docs/ # Documentation
|
||||
│ ├── implementation/ # Implementation/feature docs archive
|
||||
│ ├── plans/ # Planning documents
|
||||
│ └── ... # User-facing documentation
|
||||
├── frontend/ # React frontend source
|
||||
├── scripts/ # Build/test scripts
|
||||
├── test-results/ # Test outputs (gitignored)
|
||||
├── tools/ # Development tools
|
||||
└── [standard files] # README, LICENSE, Makefile, etc.
|
||||
```
|
||||
|
||||
## Enforcement
|
||||
|
||||
This structure is enforced by:
|
||||
|
||||
- `.gitignore` patterns preventing commits of artifacts at root
|
||||
- Code review guidelines
|
||||
- These instructions for AI assistants
|
||||
|
||||
When reviewing PRs or generating code, ensure new files follow these placement rules.
|
||||
@@ -0,0 +1,65 @@
|
||||
## Subagent Usage Templates and Orchestration
|
||||
|
||||
This helper provides the Management agent with templates to create robust and repeatable `runSubagent` calls.
|
||||
|
||||
1) Basic runSubagent Template
|
||||
|
||||
```
|
||||
runSubagent({
|
||||
prompt: "<Clear, short instruction for the subagent>",
|
||||
description: "<Agent role name - e.g., Backend Dev>",
|
||||
metadata: {
|
||||
plan_file: "docs/plans/current_spec.md",
|
||||
files_to_change: ["..."],
|
||||
commands_to_run: ["..."],
|
||||
tests_to_run: ["..."],
|
||||
timeout_minutes: 60,
|
||||
acceptance_criteria: ["All tests pass", "No lint warnings"]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
2) Orchestration Checklist (Management)
|
||||
|
||||
- Validate: `plan_file` exists and contains a `Handoff Contract` JSON.
|
||||
- Kickoff: call `Planning` to create the plan if not present.
|
||||
- Run: execute `Backend Dev` then `Frontend Dev` sequentially.
|
||||
- Parallel: run `QA and Security`, `DevOps` and `Doc Writer` in parallel for CI / QA checks and documentation.
|
||||
- Return: a JSON summary with `subagent_results`, `overall_status`, and aggregated artifacts.
|
||||
|
||||
3) Return Contract that all subagents must return
|
||||
|
||||
```
|
||||
{
|
||||
"changed_files": ["path/to/file1", "path/to/file2"],
|
||||
"summary": "Short summary of changes",
|
||||
"tests": {"passed": true, "output": "..."},
|
||||
"artifacts": ["..."],
|
||||
"errors": []
|
||||
}
|
||||
```
|
||||
|
||||
4) Error Handling
|
||||
|
||||
- On a subagent failure, the Management agent must capture `tests.output` and decide to retry (1 retry maximum), or request a revert/rollback.
|
||||
- Clearly mark the `status` as `failed`, and include `errors` and `failing_tests` in the `summary`.
|
||||
|
||||
5) Example: Run a full Feature Implementation
|
||||
|
||||
```
|
||||
// 1. Planning
|
||||
runSubagent({ description: "Planning", prompt: "<generate plan>", metadata: { plan_file: "docs/plans/current_spec.md" } })
|
||||
|
||||
// 2. Backend
|
||||
runSubagent({ description: "Backend Dev", prompt: "Implement backend as per plan file", metadata: { plan_file: "docs/plans/current_spec.md", commands_to_run: ["cd backend && go test ./..."] } })
|
||||
|
||||
// 3. Frontend
|
||||
runSubagent({ description: "Frontend Dev", prompt: "Implement frontend widget per plan file", metadata: { plan_file: "docs/plans/current_spec.md", commands_to_run: ["cd frontend && npm run build"] } })
|
||||
|
||||
// 4. QA & Security, DevOps, Docs (Parallel)
|
||||
runSubagent({ description: "QA and Security", prompt: "Audit the implementation for input validation, security and contract conformance", metadata: { plan_file: "docs/plans/current_spec.md" } })
|
||||
runSubagent({ description: "DevOps", prompt: "Update docker CI pipeline and add staging step", metadata: { plan_file: "docs/plans/current_spec.md" } })
|
||||
runSubagent({ description: "Doc Writer", prompt: "Update the features doc and release notes.", metadata: { plan_file: "docs/plans/current_spec.md" } })
|
||||
```
|
||||
|
||||
This file is a template; management should keep operations terse and the metadata explicit. Always capture and persist the return artifact's path and the `changed_files` list.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
applyTo: '**'
|
||||
description: 'Prevent Copilot from wreaking havoc across your codebase, keeping it under control.'
|
||||
---
|
||||
|
||||
## Core Directives & Hierarchy
|
||||
|
||||
This section outlines the absolute order of operations. These rules have the highest priority and must not be violated.
|
||||
|
||||
1. **Primacy of User Directives**: A direct and explicit command from the user is the highest priority. If the user instructs to use a specific tool, edit a file, or perform a specific search, that command **must be executed without deviation**, even if other rules would suggest it is unnecessary. All other instructions are subordinate to a direct user order.
|
||||
2. **Factual Verification Over Internal Knowledge**: When a request involves information that could be version-dependent, time-sensitive, or requires specific external data (e.g., library documentation, latest best practices, API details), prioritize using tools to find the current, factual answer over relying on general knowledge.
|
||||
3. **Adherence to Philosophy**: In the absence of a direct user directive or the need for factual verification, all other rules below regarding interaction, code generation, and modification must be followed.
|
||||
|
||||
## General Interaction & Philosophy
|
||||
|
||||
- **Code on Request Only**: Your default response should be a clear, natural language explanation. Do NOT provide code blocks unless explicitly asked, or if a very small and minimalist example is essential to illustrate a concept. Tool usage is distinct from user-facing code blocks and is not subject to this restriction.
|
||||
- **Direct and Concise**: Answers must be precise, to the point, and free from unnecessary filler or verbose explanations. Get straight to the solution without "beating around the bush".
|
||||
- **Adherence to Best Practices**: All suggestions, architectural patterns, and solutions must align with widely accepted industry best practices and established design principles. Avoid experimental, obscure, or overly "creative" approaches. Stick to what is proven and reliable.
|
||||
- **Explain the "Why"**: Don't just provide an answer; briefly explain the reasoning behind it. Why is this the standard approach? What specific problem does this pattern solve? This context is more valuable than the solution itself.
|
||||
|
||||
## Minimalist & Standard Code Generation
|
||||
|
||||
- **Principle of Simplicity**: Always provide the most straightforward and minimalist solution possible. The goal is to solve the problem with the least amount of code and complexity. Avoid premature optimization or over-engineering.
|
||||
- **Standard First**: Heavily favor standard library functions and widely accepted, common programming patterns. Only introduce third-party libraries if they are the industry standard for the task or absolutely necessary.
|
||||
- **Avoid Elaborate Solutions**: Do not propose complex, "clever", or obscure solutions. Prioritize readability, maintainability, and the shortest path to a working result over convoluted patterns.
|
||||
- **Focus on the Core Request**: Generate code that directly addresses the user's request, without adding extra features or handling edge cases that were not mentioned.
|
||||
- **Spec Hygiene**: When asked to update a plan/spec file, do not append unrelated/archived plans; keep it strictly scoped to the current task.
|
||||
|
||||
## Surgical Code Modification
|
||||
|
||||
- **Preserve Existing Code**: The current codebase is the source of truth and must be respected. Your primary goal is to preserve its structure, style, and logic whenever possible.
|
||||
- **Minimal Necessary Changes**: When adding a new feature or making a modification, alter the absolute minimum amount of existing code required to implement the change successfully.
|
||||
- **Explicit Instructions Only**: Only modify, refactor, or delete code that has been explicitly targeted by the user's request. Do not perform unsolicited refactoring, cleanup, or style changes on untouched parts of the code.
|
||||
- **Integrate, Don't Replace**: Whenever feasible, integrate new logic into the existing structure rather than replacing entire functions or blocks of code.
|
||||
|
||||
## Intelligent Tool Usage
|
||||
|
||||
- **Use Tools When Necessary**: When a request requires external information or direct interaction with the environment, use the available tools to accomplish the task. Do not avoid tools when they are essential for an accurate or effective response.
|
||||
- **Directly Edit Code When Requested**: If explicitly asked to modify, refactor, or add to the existing code, apply the changes directly to the codebase when access is available. Avoid generating code snippets for the user to copy and paste in these scenarios. The default should be direct, surgical modification as instructed.
|
||||
- **Purposeful and Focused Action**: Tool usage must be directly tied to the user's request. Do not perform unrelated searches or modifications. Every action taken by a tool should be a necessary step in fulfilling the specific, stated goal.
|
||||
- **Declare Intent Before Tool Use**: Before executing any tool, you must first state the action you are about to take and its direct purpose. This statement must be concise and immediately precede the tool call.
|
||||
@@ -0,0 +1,212 @@
|
||||
---
|
||||
description: 'Guidelines for building TanStack Start applications'
|
||||
applyTo: '**/*.ts, **/*.tsx, **/*.js, **/*.jsx, **/*.css, **/*.scss, **/*.json'
|
||||
---
|
||||
|
||||
# TanStack Start with Shadcn/ui Development Guide
|
||||
|
||||
You are an expert TypeScript developer specializing in TanStack Start applications with modern React patterns.
|
||||
|
||||
## Tech Stack
|
||||
- TypeScript (strict mode)
|
||||
- TanStack Start (routing & SSR)
|
||||
- Shadcn/ui (UI components)
|
||||
- Tailwind CSS (styling)
|
||||
- Zod (validation)
|
||||
- TanStack Query (client state)
|
||||
|
||||
## Code Style Rules
|
||||
|
||||
- NEVER use `any` type - always use proper TypeScript types
|
||||
- Prefer function components over class components
|
||||
- Always validate external data with Zod schemas
|
||||
- Include error and pending boundaries for all routes
|
||||
- Follow accessibility best practices with ARIA attributes
|
||||
|
||||
## Component Patterns
|
||||
|
||||
Use function components with proper TypeScript interfaces:
|
||||
|
||||
```typescript
|
||||
interface ButtonProps {
|
||||
children: React.ReactNode;
|
||||
onClick: () => void;
|
||||
variant?: 'primary' | 'secondary';
|
||||
}
|
||||
|
||||
export default function Button({ children, onClick, variant = 'primary' }: ButtonProps) {
|
||||
return (
|
||||
<button onClick={onClick} className={cn(buttonVariants({ variant }))}>
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Data Fetching
|
||||
|
||||
Use Route Loaders for:
|
||||
- Initial page data required for rendering
|
||||
- SSR requirements
|
||||
- SEO-critical data
|
||||
|
||||
Use React Query for:
|
||||
- Frequently updating data
|
||||
- Optional/secondary data
|
||||
- Client mutations with optimistic updates
|
||||
|
||||
```typescript
|
||||
// Route Loader
|
||||
export const Route = createFileRoute('/users')({
|
||||
loader: async () => {
|
||||
const users = await fetchUsers()
|
||||
return { users: userListSchema.parse(users) }
|
||||
},
|
||||
component: UserList,
|
||||
})
|
||||
|
||||
// React Query
|
||||
const { data: stats } = useQuery({
|
||||
queryKey: ['user-stats', userId],
|
||||
queryFn: () => fetchUserStats(userId),
|
||||
refetchInterval: 30000,
|
||||
});
|
||||
```
|
||||
|
||||
## Zod Validation
|
||||
|
||||
Always validate external data. Define schemas in `src/lib/schemas.ts`:
|
||||
|
||||
```typescript
|
||||
export const userSchema = z.object({
|
||||
id: z.string(),
|
||||
name: z.string().min(1).max(100),
|
||||
email: z.string().email().optional(),
|
||||
role: z.enum(['admin', 'user']).default('user'),
|
||||
})
|
||||
|
||||
export type User = z.infer<typeof userSchema>
|
||||
|
||||
// Safe parsing
|
||||
const result = userSchema.safeParse(data)
|
||||
if (!result.success) {
|
||||
console.error('Validation failed:', result.error.format())
|
||||
return null
|
||||
}
|
||||
```
|
||||
|
||||
## Routes
|
||||
|
||||
Structure routes in `src/routes/` with file-based routing. Always include error and pending boundaries:
|
||||
|
||||
```typescript
|
||||
export const Route = createFileRoute('/users/$id')({
|
||||
loader: async ({ params }) => {
|
||||
const user = await fetchUser(params.id);
|
||||
return { user: userSchema.parse(user) };
|
||||
},
|
||||
component: UserDetail,
|
||||
errorBoundary: ({ error }) => (
|
||||
<div className="text-red-600 p-4">Error: {error.message}</div>
|
||||
),
|
||||
pendingBoundary: () => (
|
||||
<div className="flex items-center justify-center p-4">
|
||||
<div className="animate-spin rounded-full h-8 w-8 border-b-2 border-primary" />
|
||||
</div>
|
||||
),
|
||||
});
|
||||
```
|
||||
|
||||
## UI Components
|
||||
|
||||
Always prefer Shadcn/ui components over custom ones:
|
||||
|
||||
```typescript
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>User Details</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Button onClick={handleSave}>Save</Button>
|
||||
</CardContent>
|
||||
</Card>
|
||||
```
|
||||
|
||||
Use Tailwind for styling with responsive design:
|
||||
|
||||
```typescript
|
||||
<div className="flex flex-col gap-4 p-6 md:flex-row md:gap-6">
|
||||
<Button className="w-full md:w-auto">Action</Button>
|
||||
</div>
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
Use semantic HTML first. Only add ARIA when no semantic equivalent exists:
|
||||
|
||||
```typescript
|
||||
// ✅ Good: Semantic HTML with minimal ARIA
|
||||
<button onClick={toggleMenu}>
|
||||
<MenuIcon aria-hidden="true" />
|
||||
<span className="sr-only">Toggle Menu</span>
|
||||
</button>
|
||||
|
||||
// ✅ Good: ARIA only when needed (for dynamic states)
|
||||
<button
|
||||
aria-expanded={isOpen}
|
||||
aria-controls="menu"
|
||||
onClick={toggleMenu}
|
||||
>
|
||||
Menu
|
||||
</button>
|
||||
|
||||
// ✅ Good: Semantic form elements
|
||||
<label htmlFor="email">Email Address</label>
|
||||
<input id="email" type="email" />
|
||||
{errors.email && (
|
||||
<p role="alert">{errors.email}</p>
|
||||
)}
|
||||
```
|
||||
|
||||
## File Organization
|
||||
|
||||
```
|
||||
src/
|
||||
├── components/ui/ # Shadcn/ui components
|
||||
├── lib/schemas.ts # Zod schemas
|
||||
├── routes/ # File-based routes
|
||||
└── routes/api/ # Server routes (.ts)
|
||||
```
|
||||
|
||||
## Import Standards
|
||||
|
||||
Use `@/` alias for all internal imports:
|
||||
|
||||
```typescript
|
||||
// ✅ Good
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { userSchema } from '@/lib/schemas'
|
||||
|
||||
// ❌ Bad
|
||||
import { Button } from '../components/ui/button'
|
||||
```
|
||||
|
||||
## Adding Components
|
||||
|
||||
Install Shadcn components when needed:
|
||||
|
||||
```bash
|
||||
npx shadcn@latest add button card input dialog
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
- Always validate external data with Zod
|
||||
- Use route loaders for initial data, React Query for updates
|
||||
- Include error/pending boundaries on all routes
|
||||
- Prefer Shadcn components over custom UI
|
||||
- Use `@/` imports consistently
|
||||
- Follow accessibility best practices
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
applyTo: '**'
|
||||
description: 'Strict protocols for test execution, debugging, and coverage validation.'
|
||||
---
|
||||
# Testing Protocols
|
||||
|
||||
## 0. E2E Verification First (Playwright)
|
||||
|
||||
**MANDATORY**: Before running unit tests, verify the application functions correctly end-to-end.
|
||||
|
||||
### Two Modes: Docker vs Vite
|
||||
|
||||
Playwright E2E tests can run in two modes with different capabilities:
|
||||
|
||||
| Mode | Base URL | Coverage Support | When to Use |
|
||||
|------|----------|-----------------|-------------|
|
||||
| **Docker** | `http://localhost:8080` | ❌ No (0% reported) | Integration testing, CI validation |
|
||||
| **Vite Dev** | `http://localhost:5173` | ✅ Yes (real coverage) | Local development, coverage collection |
|
||||
|
||||
**Why?** The `@bgotink/playwright-coverage` library uses V8 coverage which requires access to source files. Only the Vite dev server exposes source maps and raw source files needed for coverage instrumentation.
|
||||
|
||||
### Running E2E Tests (Integration Mode)
|
||||
|
||||
For general integration testing without coverage:
|
||||
|
||||
```bash
|
||||
# Against Docker container (default)
|
||||
npx playwright test --project=chromium
|
||||
|
||||
# With explicit base URL
|
||||
PLAYWRIGHT_BASE_URL=http://localhost:8080 npx playwright test --project=chromium
|
||||
```
|
||||
|
||||
### Running E2E Tests with Coverage
|
||||
|
||||
**IMPORTANT**: Use the dedicated skill for coverage collection:
|
||||
|
||||
```bash
|
||||
# Recommended: Uses skill that starts Vite and runs against localhost:5173
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage
|
||||
```
|
||||
|
||||
The coverage skill:
|
||||
1. Starts Vite dev server on port 5173
|
||||
2. Sets `PLAYWRIGHT_BASE_URL=http://localhost:5173`
|
||||
3. Runs tests with V8 coverage collection
|
||||
4. Generates reports in `coverage/e2e/` (LCOV, HTML, JSON)
|
||||
|
||||
**DO NOT** expect coverage when running against Docker:
|
||||
```bash
|
||||
# ❌ WRONG: Coverage will show "Unknown% (0/0)"
|
||||
PLAYWRIGHT_BASE_URL=http://localhost:8080 npx playwright test --coverage
|
||||
|
||||
# ✅ CORRECT: Use the coverage skill
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage
|
||||
```
|
||||
|
||||
### Verifying Coverage Locally Before CI
|
||||
|
||||
Before pushing code, verify E2E coverage:
|
||||
|
||||
1. Run the coverage skill:
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage
|
||||
```
|
||||
|
||||
2. Check coverage output:
|
||||
```bash
|
||||
# View HTML report
|
||||
open coverage/e2e/index.html
|
||||
|
||||
# Check LCOV file exists for Codecov
|
||||
ls -la coverage/e2e/lcov.info
|
||||
```
|
||||
|
||||
3. Verify non-zero coverage:
|
||||
```bash
|
||||
# Should show real percentages, not "0%"
|
||||
head -20 coverage/e2e/lcov.info
|
||||
```
|
||||
|
||||
### General Guidelines
|
||||
|
||||
* **No Truncation**: Never pipe Playwright test output through `head`, `tail`, or other truncating commands. Playwright runs interactively and requires user input to quit when piped, causing the command to hang indefinitely.
|
||||
* **Why First**: If the application is broken at the E2E level, unit tests may need updates. Playwright catches integration issues early.
|
||||
* **On Failure**: Analyze failures, trace root cause through frontend → backend flow, then fix before proceeding to unit tests.
|
||||
* **Scope**: Run relevant test files for the feature being modified (e.g., `tests/manual-dns-provider.spec.ts`).
|
||||
|
||||
## 1. Execution Environment
|
||||
* **No Truncation:** Never use pipe commands (e.g., `head`, `tail`) or flags that limit stdout/stderr. If a test hangs, it likely requires an interactive input or is caught in a loop; analyze the full output to identify the block.
|
||||
* **Task-Based Execution:** Do not manually construct test strings. Use existing project tasks (e.g., `npm test`, `go test ./...`). If a specific sub-module requires frequent testing, generate a new task definition in the project's configuration file (e.g., `.vscode/tasks.json`) before proceeding.
|
||||
|
||||
## 2. Failure Analysis & Logic Integrity
|
||||
* **Evidence-Based Debugging:** When a test fails, you must quote the specific error message or stack trace before suggesting a fix.
|
||||
* **Bug vs. Test Flaw:** Treat the test as the "Source of Truth." If a test fails, assume the code is broken until proven otherwise. Research the original requirement or PR description to verify if the test logic itself is outdated before modifying it.
|
||||
* **Zero-Hallucination Policy:** Only use file paths and identifiers discovered via the `ls` or `search` tools. Never guess a path based on naming conventions.
|
||||
|
||||
## 3. Coverage & Completion
|
||||
* **Coverage Gate:** A task is not "Complete" until a coverage report is generated.
|
||||
* **Threshold Compliance:** You must compare the final coverage percentage against the project's threshold (Default: 85% unless specified otherwise). If coverage drops, you must identify the "uncovered lines" and add targeted tests.
|
||||
* **Patch Coverage Gate (Codecov):** If production code is modified, Codecov **patch coverage must be 100%** for the modified lines. Do not relax thresholds; add targeted tests.
|
||||
* **Patch Triage Requirement:** Plans must include the exact missing/partial patch line ranges copied from Codecov’s **Patch** view.
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
description: 'Guidelines for TypeScript Development targeting TypeScript 5.x and ES2022 output'
|
||||
applyTo: '**/*.ts'
|
||||
---
|
||||
|
||||
# TypeScript Development
|
||||
|
||||
> These instructions assume projects are built with TypeScript 5.x (or newer) compiling to an ES2022 JavaScript baseline. Adjust guidance if your runtime requires older language targets or down-level transpilation.
|
||||
|
||||
## Core Intent
|
||||
|
||||
- Respect the existing architecture and coding standards.
|
||||
- Prefer readable, explicit solutions over clever shortcuts.
|
||||
- Extend current abstractions before inventing new ones.
|
||||
- Prioritize maintainability and clarity, short methods and classes, clean code.
|
||||
|
||||
## General Guardrails
|
||||
|
||||
- Target TypeScript 5.x / ES2022 and prefer native features over polyfills.
|
||||
- Use pure ES modules; never emit `require`, `module.exports`, or CommonJS helpers.
|
||||
- Rely on the project's build, lint, and test scripts unless asked otherwise.
|
||||
- Note design trade-offs when intent is not obvious.
|
||||
|
||||
## Project Organization
|
||||
|
||||
- Follow the repository's folder and responsibility layout for new code.
|
||||
- Use kebab-case filenames (e.g., `user-session.ts`, `data-service.ts`) unless told otherwise.
|
||||
- Keep tests, types, and helpers near their implementation when it aids discovery.
|
||||
- Reuse or extend shared utilities before adding new ones.
|
||||
|
||||
## Naming & Style
|
||||
|
||||
- Use PascalCase for classes, interfaces, enums, and type aliases; camelCase for everything else.
|
||||
- Skip interface prefixes like `I`; rely on descriptive names.
|
||||
- Name things for their behavior or domain meaning, not implementation.
|
||||
|
||||
## Formatting & Style
|
||||
|
||||
- Run the repository's lint/format scripts (e.g., `npm run lint`) before submitting.
|
||||
- Match the project's indentation, quote style, and trailing comma rules.
|
||||
- Keep functions focused; extract helpers when logic branches grow.
|
||||
- Favor immutable data and pure functions when practical.
|
||||
|
||||
## Type System Expectations
|
||||
|
||||
- Avoid `any` (implicit or explicit); prefer `unknown` plus narrowing.
|
||||
- Use discriminated unions for realtime events and state machines.
|
||||
- Centralize shared contracts instead of duplicating shapes.
|
||||
- Express intent with TypeScript utility types (e.g., `Readonly`, `Partial`, `Record`).
|
||||
|
||||
## Async, Events & Error Handling
|
||||
|
||||
- Use `async/await`; wrap awaits in try/catch with structured errors.
|
||||
- Guard edge cases early to avoid deep nesting.
|
||||
- Send errors through the project's logging/telemetry utilities.
|
||||
- Surface user-facing errors via the repository's notification pattern.
|
||||
- Debounce configuration-driven updates and dispose resources deterministically.
|
||||
|
||||
## Architecture & Patterns
|
||||
|
||||
- Follow the repository's dependency injection or composition pattern; keep modules single-purpose.
|
||||
- Observe existing initialization and disposal sequences when wiring into lifecycles.
|
||||
- Keep transport, domain, and presentation layers decoupled with clear interfaces.
|
||||
- Supply lifecycle hooks (e.g., `initialize`, `dispose`) and targeted tests when adding services.
|
||||
|
||||
## External Integrations
|
||||
|
||||
- Instantiate clients outside hot paths and inject them for testability.
|
||||
- Never hardcode secrets; load them from secure sources.
|
||||
- Apply retries, backoff, and cancellation to network or IO calls.
|
||||
- Normalize external responses and map errors to domain shapes.
|
||||
|
||||
## Security Practices
|
||||
|
||||
- Validate and sanitize external input with schema validators or type guards.
|
||||
- Avoid dynamic code execution and untrusted template rendering.
|
||||
- Encode untrusted content before rendering HTML; use framework escaping or trusted types.
|
||||
- Use parameterized queries or prepared statements to block injection.
|
||||
- Keep secrets in secure storage, rotate them regularly, and request least-privilege scopes.
|
||||
- Favor immutable flows and defensive copies for sensitive data.
|
||||
- Use vetted crypto libraries only.
|
||||
- Patch dependencies promptly and monitor advisories.
|
||||
|
||||
## Configuration & Secrets
|
||||
|
||||
- Reach configuration through shared helpers and validate with schemas or dedicated validators.
|
||||
- Handle secrets via the project's secure storage; guard `undefined` and error states.
|
||||
- Document new configuration keys and update related tests.
|
||||
|
||||
## UI & UX Components
|
||||
|
||||
- Sanitize user or external content before rendering.
|
||||
- Keep UI layers thin; push heavy logic to services or state managers.
|
||||
- Use messaging or events to decouple UI from business logic.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
- Add or update unit tests with the project's framework and naming style.
|
||||
- Expand integration or end-to-end suites when behavior crosses modules or platform APIs.
|
||||
- Run targeted test scripts for quick feedback before submitting.
|
||||
- Avoid brittle timing assertions; prefer fake timers or injected clocks.
|
||||
|
||||
## Performance & Reliability
|
||||
|
||||
- Lazy-load heavy dependencies and dispose them when done.
|
||||
- Defer expensive work until users need it.
|
||||
- Batch or debounce high-frequency events to reduce thrash.
|
||||
- Track resource lifetimes to prevent leaks.
|
||||
|
||||
## Documentation & Comments
|
||||
|
||||
- Add JSDoc to public APIs; include `@remarks` or `@example` when helpful.
|
||||
- Write comments that capture intent, and remove stale notes during refactors.
|
||||
- Update architecture or design docs when introducing significant patterns.
|
||||
@@ -0,0 +1,549 @@
|
||||
---
|
||||
description: 'Automatically update README.md and documentation files when application code changes require documentation updates'
|
||||
applyTo: '**/*.{md,js,mjs,cjs,ts,tsx,jsx,py,java,cs,go,rb,php,rs,cpp,c,h,hpp}'
|
||||
---
|
||||
|
||||
# Update Documentation on Code Change
|
||||
|
||||
## Overview
|
||||
|
||||
Ensure documentation stays synchronized with code changes by automatically detecting when README.md,
|
||||
API documentation, configuration guides, and other documentation files need updates based on code
|
||||
modifications.
|
||||
|
||||
## Instruction Sections and Configuration
|
||||
|
||||
The following parts of this section, `Instruction Sections and Configurable Instruction Sections`
|
||||
and `Instruction Configuration` are only relevant to THIS instruction file, and are meant to be a
|
||||
method to easily modify how the Copilot instructions are implemented. Essentially the two parts
|
||||
are meant to turn portions or sections of the actual Copilot instructions on or off, and allow for
|
||||
custom cases and conditions for when and how to implement certain sections of this document.
|
||||
|
||||
### Instruction Sections and Configurable Instruction Sections
|
||||
|
||||
There are several instruction sections in this document. The start of an instruction section is
|
||||
indicated by a level two header. Call this an **INSTRUCTION SECTION**. Some instruction
|
||||
sections are configurable. Some are not configurable and will always be used.
|
||||
|
||||
Instruction sections that ARE configurable are not required, and are subject to additional context
|
||||
and/or conditions. Call these **CONFIGURABLE INSTRUCTION SECTIONS**.
|
||||
|
||||
**Configurable instruction sections** will have the section's configuration property appended to
|
||||
the level two header, wrapped in backticks (e.g., `apply-this`). Call this the
|
||||
**CONFIGURABLE PROPERTY**.
|
||||
|
||||
The **configurable property** will be declared and defined in the **Instruction Configuration**
|
||||
portion of this section. They are booleans. If `true`, then apply, utilize, and/or follow the
|
||||
instructions in that section.
|
||||
|
||||
Each **configurable instruction section** will also have a sentence that follows the section's
|
||||
level two header with the section's configuration details. Call this the **CONFIGURATION DETAIL**.
|
||||
|
||||
The **configuration detail** is a subset of rules that expand upon the configurable instruction
|
||||
section. This allows for custom cases and/or conditions to be checked that will determine the final
|
||||
implementation for that **configurable instruction section**.
|
||||
|
||||
Before resolving on how to apply a **configurable instruction section**, check the
|
||||
**configurable property** for a nested and/or corresponding `apply-condition`, and utilize the `apply-condition` when settling on the final approach for the **configurable instruction section**. By
|
||||
default the `apply-condition` for each **configurable property** is unset, but an example of a set
|
||||
`apply-condition` could be something like:
|
||||
|
||||
- **apply-condition** :
|
||||
` this.parent.property = (git.branch == "master") ? this.parent.property = true : this.parent.property = false; `
|
||||
|
||||
The sum of all the **constant instructions sections**, and **configurable instruction sections**
|
||||
will determine the complete instructions to follow. Call this the **COMPILED INSTRUCTIONS**.
|
||||
|
||||
The **compiled instructions** are dependent on the configuration. Each instruction section
|
||||
included in the **compiled instructions** will be interpreted and utilized AS IF a separate set
|
||||
of instructions that are independent of the entirety of this instruction file. Call this the
|
||||
**FINAL PROCEDURE**.
|
||||
|
||||
### Instruction Configuration
|
||||
|
||||
- **apply-doc-file-structure** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-doc-verification** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-doc-quality-standard** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-automation-tooling** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-doc-patterns** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-best-practices** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-validation-commands** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-maintenance-schedule** : true
|
||||
- **apply-condition** : unset
|
||||
- **apply-git-integration** : false
|
||||
- **apply-condition** : unset
|
||||
|
||||
<!--
|
||||
| Configuration Property | Default | Description | When to Enable/Disable |
|
||||
|-------------------------------|---------|-----------------------------------------------------------------------------|-------------------------------------------------------------|
|
||||
| apply-doc-file-structure | true | Ensures documentation follows a consistent file structure. | Disable if you want to allow free-form doc organization. |
|
||||
| apply-doc-verification | true | Verifies that documentation matches code changes. | Disable if verification is handled elsewhere. |
|
||||
| apply-doc-quality-standard | true | Enforces documentation quality standards. | Disable if quality standards are not required. |
|
||||
| apply-automation-tooling | true | Uses automation tools to update documentation. | Disable if you prefer manual documentation updates. |
|
||||
| apply-doc-patterns | true | Applies common documentation patterns and templates. | Disable for custom or unconventional documentation styles. |
|
||||
| apply-best-practices | true | Enforces best practices in documentation. | Disable if best practices are not a priority. |
|
||||
| apply-validation-commands | true | Runs validation commands to check documentation correctness. | Disable if validation is not needed. |
|
||||
| apply-maintenance-schedule | true | Schedules regular documentation maintenance. | Disable if maintenance is managed differently. |
|
||||
| apply-git-integration | false | Integrates documentation updates with Git workflows. | Enable if you want automatic Git integration. |
|
||||
-->
|
||||
## When to Update Documentation
|
||||
|
||||
### Trigger Conditions
|
||||
|
||||
Automatically check if documentation updates are needed when:
|
||||
|
||||
- New features or functionality are added
|
||||
- API endpoints, methods, or interfaces change
|
||||
- Breaking changes are introduced
|
||||
- Dependencies or requirements change
|
||||
- Configuration options or environment variables are modified
|
||||
- Installation or setup procedures change
|
||||
- Command-line interfaces or scripts are updated
|
||||
- Code examples in documentation become outdated
|
||||
|
||||
## Documentation Update Rules
|
||||
|
||||
### README.md Updates
|
||||
|
||||
**Always update README.md when:**
|
||||
|
||||
- Adding new features or capabilities
|
||||
- Add feature description to "Features" section
|
||||
- Include usage examples if applicable
|
||||
- Update table of contents if present
|
||||
|
||||
- Modifying installation or setup process
|
||||
- Update "Installation" or "Getting Started" section
|
||||
- Revise dependency requirements
|
||||
- Update prerequisite lists
|
||||
|
||||
- Adding new CLI commands or options
|
||||
- Document command syntax and examples
|
||||
- Include option descriptions and default values
|
||||
- Add usage examples
|
||||
|
||||
- Changing configuration options
|
||||
- Update configuration examples
|
||||
- Document new environment variables
|
||||
- Update config file templates
|
||||
|
||||
### API Documentation Updates
|
||||
|
||||
**Sync API documentation when:**
|
||||
|
||||
- New endpoints are added
|
||||
- Document HTTP method, path, parameters
|
||||
- Include request/response examples
|
||||
- Update OpenAPI/Swagger specs
|
||||
|
||||
- Endpoint signatures change
|
||||
- Update parameter lists
|
||||
- Revise response schemas
|
||||
- Document breaking changes
|
||||
|
||||
- Authentication or authorization changes
|
||||
- Update authentication examples
|
||||
- Revise security requirements
|
||||
- Update API key/token documentation
|
||||
|
||||
### Code Example Synchronization
|
||||
|
||||
**Verify and update code examples when:**
|
||||
|
||||
- Function signatures change
|
||||
- Update all code snippets using the function
|
||||
- Verify examples still compile/run
|
||||
- Update import statements if needed
|
||||
|
||||
- API interfaces change
|
||||
- Update example requests and responses
|
||||
- Revise client code examples
|
||||
- Update SDK usage examples
|
||||
|
||||
- Best practices evolve
|
||||
- Replace outdated patterns in examples
|
||||
- Update to use current recommended approaches
|
||||
- Add deprecation notices for old patterns
|
||||
|
||||
### Configuration Documentation
|
||||
|
||||
**Update configuration docs when:**
|
||||
|
||||
- New environment variables are added
|
||||
- Add to .env.example file
|
||||
- Document in README.md or docs/configuration.md
|
||||
- Include default values and descriptions
|
||||
|
||||
- Config file structure changes
|
||||
- Update example config files
|
||||
- Document new options
|
||||
- Mark deprecated options
|
||||
|
||||
- Deployment configuration changes
|
||||
- Update Docker/Kubernetes configs
|
||||
- Revise deployment guides
|
||||
- Update infrastructure-as-code examples
|
||||
|
||||
### Migration and Breaking Changes
|
||||
|
||||
**Create migration guides when:**
|
||||
|
||||
- Breaking API changes occur
|
||||
- Document what changed
|
||||
- Provide before/after examples
|
||||
- Include step-by-step migration instructions
|
||||
|
||||
- Major version updates
|
||||
- List all breaking changes
|
||||
- Provide upgrade checklist
|
||||
- Include common migration issues and solutions
|
||||
|
||||
- Deprecating features
|
||||
- Mark deprecated features clearly
|
||||
- Suggest alternative approaches
|
||||
- Include timeline for removal
|
||||
|
||||
## Documentation File Structure `apply-doc-file-structure`
|
||||
|
||||
If `apply-doc-file-structure == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Standard Documentation Files
|
||||
|
||||
Maintain these documentation files and update as needed:
|
||||
|
||||
- **README.md**: Project overview, quick start, basic usage
|
||||
- **CHANGELOG.md**: Version history and user-facing changes
|
||||
- **docs/**: Detailed documentation
|
||||
- `installation.md`: Setup and installation guide
|
||||
- `configuration.md`: Configuration options and examples
|
||||
- `api.md`: API reference documentation
|
||||
- `contributing.md`: Contribution guidelines
|
||||
- `migration-guides/`: Version migration guides
|
||||
- **examples/**: Working code examples and tutorials
|
||||
|
||||
### Changelog Management
|
||||
|
||||
**Add changelog entries for:**
|
||||
|
||||
- New features (under "Added" section)
|
||||
- Bug fixes (under "Fixed" section)
|
||||
- Breaking changes (under "Changed" section with **BREAKING** prefix)
|
||||
- Deprecated features (under "Deprecated" section)
|
||||
- Removed features (under "Removed" section)
|
||||
- Security fixes (under "Security" section)
|
||||
|
||||
**Changelog format:**
|
||||
|
||||
```markdown
|
||||
## [Version] - YYYY-MM-DD
|
||||
|
||||
### Added
|
||||
- New feature description with reference to PR/issue
|
||||
|
||||
### Changed
|
||||
- **BREAKING**: Description of breaking change
|
||||
- Other changes
|
||||
|
||||
### Fixed
|
||||
- Bug fix description
|
||||
```
|
||||
|
||||
## Documentation Verification `apply-doc-verification`
|
||||
|
||||
If `apply-doc-verification == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Before Applying Changes
|
||||
|
||||
**Check documentation completeness:**
|
||||
|
||||
1. All new public APIs are documented
|
||||
2. Code examples compile and run
|
||||
3. Links in documentation are valid
|
||||
4. Configuration examples are accurate
|
||||
5. Installation steps are current
|
||||
6. README.md reflects current state
|
||||
|
||||
### Documentation Tests
|
||||
|
||||
**Include documentation validation:**
|
||||
|
||||
#### Example Tasks
|
||||
|
||||
- Verify code examples in docs compile/run
|
||||
- Check for broken internal/external links
|
||||
- Validate configuration examples against schemas
|
||||
- Ensure API examples match current implementation
|
||||
|
||||
```bash
|
||||
# Example validation commands
|
||||
npm run docs:check # Verify docs build
|
||||
npm run docs:test-examples # Test code examples
|
||||
npm run docs:lint # Check for issues
|
||||
```
|
||||
|
||||
## Documentation Quality Standards `apply-doc-quality-standard`
|
||||
|
||||
If `apply-doc-quality-standard == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Writing Guidelines
|
||||
|
||||
- Use clear, concise language
|
||||
- Include working code examples
|
||||
- Provide both basic and advanced examples
|
||||
- Use consistent terminology
|
||||
- Include error handling examples
|
||||
- Document edge cases and limitations
|
||||
|
||||
### Code Example Format
|
||||
|
||||
```markdown
|
||||
### Example: [Clear description of what example demonstrates]
|
||||
|
||||
\`\`\`language
|
||||
// Include necessary imports/setup
|
||||
import { function } from 'package';
|
||||
|
||||
// Complete, runnable example
|
||||
const result = function(parameter);
|
||||
console.log(result);
|
||||
\`\`\`
|
||||
|
||||
**Output:**
|
||||
\`\`\`
|
||||
expected output
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
### API Documentation Format
|
||||
|
||||
```markdown
|
||||
### `functionName(param1, param2)`
|
||||
|
||||
Brief description of what the function does.
|
||||
|
||||
**Parameters:**
|
||||
- `param1` (type): Description of parameter
|
||||
- `param2` (type, optional): Description with default value
|
||||
|
||||
**Returns:**
|
||||
- `type`: Description of return value
|
||||
|
||||
**Example:**
|
||||
\`\`\`language
|
||||
const result = functionName('value', 42);
|
||||
\`\`\`
|
||||
|
||||
**Throws:**
|
||||
- `ErrorType`: When and why error is thrown
|
||||
```
|
||||
|
||||
## Automation and Tooling `apply-automation-tooling`
|
||||
|
||||
If `apply-automation-tooling == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Documentation Generation
|
||||
|
||||
**Use automated tools when available:**
|
||||
|
||||
#### Automated Tool Examples
|
||||
|
||||
- JSDoc/TSDoc for JavaScript/TypeScript
|
||||
- Sphinx/pdoc for Python
|
||||
- Javadoc for Java
|
||||
- xmldoc for C#
|
||||
- godoc for Go
|
||||
- rustdoc for Rust
|
||||
|
||||
### Documentation Linting
|
||||
|
||||
**Validate documentation with:**
|
||||
|
||||
- Markdown linters (markdownlint)
|
||||
- Link checkers (markdown-link-check)
|
||||
- Spell checkers (cspell)
|
||||
- Code example validators
|
||||
|
||||
### Pre-update Hooks
|
||||
|
||||
**Add pre-commit checks for:**
|
||||
|
||||
- Documentation build succeeds
|
||||
- No broken links
|
||||
- Code examples are valid
|
||||
- Changelog entry exists for changes
|
||||
|
||||
## Common Documentation Patterns `apply-doc-patterns`
|
||||
|
||||
If `apply-doc-patterns == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Feature Documentation Template
|
||||
|
||||
```markdown
|
||||
## Feature Name
|
||||
|
||||
Brief description of the feature.
|
||||
|
||||
### Usage
|
||||
|
||||
Basic usage example with code snippet.
|
||||
|
||||
### Configuration
|
||||
|
||||
Configuration options with examples.
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
Complex scenarios and edge cases.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
Common issues and solutions.
|
||||
```
|
||||
|
||||
### API Endpoint Documentation Template
|
||||
|
||||
```markdown
|
||||
### `HTTP_METHOD /api/endpoint`
|
||||
|
||||
Description of what the endpoint does.
|
||||
|
||||
**Request:**
|
||||
\`\`\`json
|
||||
{
|
||||
"param": "value"
|
||||
}
|
||||
\`\`\`
|
||||
|
||||
**Response:**
|
||||
\`\`\`json
|
||||
{
|
||||
"result": "value"
|
||||
}
|
||||
\`\`\`
|
||||
|
||||
**Status Codes:**
|
||||
- 200: Success
|
||||
- 400: Bad request
|
||||
- 401: Unauthorized
|
||||
```
|
||||
|
||||
## Best Practices `apply-best-practices`
|
||||
|
||||
If `apply-best-practices == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Do's
|
||||
|
||||
- ✅ Update documentation in the same commit as code changes
|
||||
- ✅ Include before/after examples for changes to be reviewed before applying
|
||||
- ✅ Test code examples before committing
|
||||
- ✅ Use consistent formatting and terminology
|
||||
- ✅ Document limitations and edge cases
|
||||
- ✅ Provide migration paths for breaking changes
|
||||
- ✅ Keep documentation DRY (link instead of duplicating)
|
||||
|
||||
### Don'ts
|
||||
|
||||
- ❌ Commit code changes without updating documentation
|
||||
- ❌ Leave outdated examples in documentation
|
||||
- ❌ Document features that don't exist yet
|
||||
- ❌ Use vague or ambiguous language
|
||||
- ❌ Forget to update changelog
|
||||
- ❌ Ignore broken links or failing examples
|
||||
- ❌ Document implementation details users don't need
|
||||
|
||||
## Validation Example Commands `apply-validation-commands`
|
||||
|
||||
If `apply-validation-commands == true`, then apply the following configurable instruction section.
|
||||
|
||||
Example scripts to apply to your project for documentation validation:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"docs:build": "Build documentation",
|
||||
"docs:test": "Test code examples in docs",
|
||||
"docs:lint": "Lint documentation files",
|
||||
"docs:links": "Check for broken links",
|
||||
"docs:spell": "Spell check documentation",
|
||||
"docs:validate": "Run all documentation checks"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Maintenance Schedule `apply-maintenance-schedule`
|
||||
|
||||
If `apply-maintenance-schedule == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Regular Reviews
|
||||
|
||||
- **Monthly**: Review documentation for accuracy
|
||||
- **Per release**: Update version numbers and examples
|
||||
- **Quarterly**: Check for outdated patterns or deprecated features
|
||||
- **Annually**: Comprehensive documentation audit
|
||||
|
||||
### Deprecation Process
|
||||
|
||||
When deprecating features:
|
||||
|
||||
1. Add deprecation notice to documentation
|
||||
2. Update examples to use recommended alternatives
|
||||
3. Create migration guide
|
||||
4. Update changelog with deprecation notice
|
||||
5. Set timeline for removal
|
||||
6. In next major version, remove deprecated feature and docs
|
||||
|
||||
## Git Integration `apply-git-integration`
|
||||
|
||||
If `apply-git-integration == true`, then apply the following configurable instruction section.
|
||||
|
||||
### Pull Request Requirements
|
||||
|
||||
**Documentation must be updated in the same PR as code changes:**
|
||||
|
||||
- Document new features in the feature PR
|
||||
- Update examples when code changes
|
||||
- Add changelog entries with code changes
|
||||
- Update API docs when interfaces change
|
||||
|
||||
### Documentation Review
|
||||
|
||||
**During code review, verify:**
|
||||
|
||||
- Documentation accurately describes the changes
|
||||
- Examples are clear and complete
|
||||
- No undocumented breaking changes
|
||||
- Changelog entry is appropriate
|
||||
- Migration guides are provided if needed
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Before considering documentation complete, and concluding on the **final procedure**:
|
||||
|
||||
- [ ] **Compiled instructions** are based on the sum of **constant instruction sections** and
|
||||
**configurable instruction sections**
|
||||
- [ ] README.md reflects current project state
|
||||
- [ ] All new features are documented
|
||||
- [ ] Code examples are tested and work
|
||||
- [ ] API documentation is complete and accurate
|
||||
- [ ] Configuration examples are up to date
|
||||
- [ ] Breaking changes are documented with migration guide
|
||||
- [ ] CHANGELOG.md is updated
|
||||
- [ ] Links are valid and not broken
|
||||
- [ ] Installation instructions are current
|
||||
- [ ] Environment variables are documented
|
||||
|
||||
## Updating Documentation on Code Change GOAL
|
||||
|
||||
- Keep documentation close to code when possible
|
||||
- Use documentation generators for API reference
|
||||
- Maintain living documentation that evolves with code
|
||||
- Consider documentation as part of feature completeness
|
||||
- Review documentation in code reviews
|
||||
- Make documentation easy to find and navigate
|
||||
@@ -0,0 +1,12 @@
|
||||
## Propagation Config
|
||||
# Central list of sensitive paths that should not be auto-propagated.
|
||||
# The workflow reads this file and will skip automatic propagation if any
|
||||
# changed files match these paths. Only a simple YAML list under `sensitive_paths:` is parsed.
|
||||
|
||||
sensitive_paths:
|
||||
- scripts/history-rewrite/
|
||||
- data/backups
|
||||
- docs/plans/history_rewrite.md
|
||||
- .github/workflows/
|
||||
- scripts/history-rewrite/preview_removals.sh
|
||||
- scripts/history-rewrite/clean_history.sh
|
||||
@@ -0,0 +1,26 @@
|
||||
name-template: 'v$NEXT_PATCH_VERSION'
|
||||
tag-template: 'v$NEXT_PATCH_VERSION'
|
||||
categories:
|
||||
- title: '🚀 Features'
|
||||
labels:
|
||||
- 'feature'
|
||||
- 'feat'
|
||||
- title: '🐛 Fixes'
|
||||
labels:
|
||||
- 'bug'
|
||||
- 'fix'
|
||||
- title: '🧰 Maintenance'
|
||||
labels:
|
||||
- 'chore'
|
||||
- title: '🧪 Tests'
|
||||
labels:
|
||||
- 'test'
|
||||
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
|
||||
template: |
|
||||
## What's Changed
|
||||
|
||||
$CHANGES
|
||||
|
||||
----
|
||||
|
||||
Full Changelog: https://github.com/${{ github.repository }}/compare/$FROM_TAG...$TO_TAG
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":semanticCommits",
|
||||
":separateMultipleMajorReleases",
|
||||
"helpers:pinGitHubActionDigests"
|
||||
],
|
||||
"baseBranches": [
|
||||
"development"
|
||||
],
|
||||
"timezone": "America/New_York",
|
||||
"dependencyDashboard": true,
|
||||
"prConcurrentLimit": 10,
|
||||
"prHourlyLimit": 0,
|
||||
"labels": [
|
||||
"dependencies"
|
||||
],
|
||||
|
||||
"rebaseWhen": "auto",
|
||||
|
||||
"vulnerabilityAlerts": {
|
||||
"enabled": true
|
||||
},
|
||||
|
||||
"schedule": [
|
||||
"before 8am on monday"
|
||||
],
|
||||
|
||||
"rangeStrategy": "bump",
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true,
|
||||
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Track Go dependencies patched in Dockerfile for Caddy CVE fixes",
|
||||
"managerFilePatterns": [
|
||||
"/^Dockerfile$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=go\\s+depName=(?<depName>[^\\s]+)\\s*\\n\\s*go get (?<depName2>[^@]+)@v(?<currentValue>[^\\s|]+)"
|
||||
],
|
||||
"datasourceTemplate": "go",
|
||||
"versioningTemplate": "semver"
|
||||
},
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Track Debian base image in Dockerfile",
|
||||
"managerFilePatterns": ["/^Dockerfile$/"],
|
||||
"matchStrings": [
|
||||
"ARG CADDY_IMAGE=debian:(?<currentValue>[\\w.-]+)"
|
||||
],
|
||||
"depNameTemplate": "debian",
|
||||
"datasourceTemplate": "docker"
|
||||
}
|
||||
],
|
||||
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "THE MEGAZORD: Group ALL non-major updates (NPM, Docker, Go, Actions) into one weekly PR",
|
||||
"matchPackagePatterns": ["*"],
|
||||
"matchUpdateTypes": [
|
||||
"minor",
|
||||
"patch",
|
||||
"pin",
|
||||
"digest"
|
||||
],
|
||||
"groupName": "weekly-non-major-updates",
|
||||
"automerge": true
|
||||
},
|
||||
{
|
||||
"description": "Preserve your custom Caddy patch labels but allow them to group into the weekly PR",
|
||||
"matchManagers": ["custom.regex"],
|
||||
"matchFileNames": ["Dockerfile"],
|
||||
"labels": ["caddy-patch", "security"],
|
||||
"matchPackageNames": [
|
||||
"/expr-lang/expr/",
|
||||
"/quic-go/quic-go/",
|
||||
"/smallstep/certificates/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "Docker: keep Caddy within v2 (no automatic jump to v3)",
|
||||
"matchManagers": ["dockerfile"],
|
||||
"matchPackageNames": ["caddy"],
|
||||
"allowedVersions": "<3.0.0"
|
||||
},
|
||||
{
|
||||
"description": "Safety: Keep MAJOR updates separate and require manual review",
|
||||
"matchUpdateTypes": ["major"],
|
||||
"automerge": false,
|
||||
"labels": ["manual-review"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,408 @@
|
||||
# Agent Skills - Charon Project
|
||||
|
||||
This directory contains [Agent Skills](https://agentskills.io) following the agentskills.io specification for AI-discoverable, executable tasks.
|
||||
|
||||
## Overview
|
||||
|
||||
Agent Skills are self-documenting, AI-discoverable task definitions that combine YAML frontmatter (metadata) with Markdown documentation. Each skill represents a specific task or workflow that can be executed by both humans and AI assistants.
|
||||
|
||||
**Location**: `.github/skills/` is the [VS Code Copilot standard location](https://code.visualstudio.com/docs/copilot/customization/agent-skills) for Agent Skills
|
||||
**Format**: Skills follow the [agentskills.io specification](https://agentskills.io/specification) for structure and metadata
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.github/skills/
|
||||
├── README.md # This file
|
||||
├── scripts/ # Shared infrastructure scripts
|
||||
│ ├── skill-runner.sh # Universal skill executor
|
||||
│ ├── validate-skills.py # Frontmatter validation tool
|
||||
│ ├── _logging_helpers.sh # Logging utilities
|
||||
│ ├── _error_handling_helpers.sh # Error handling utilities
|
||||
│ └── _environment_helpers.sh # Environment validation
|
||||
├── examples/ # Example skill templates
|
||||
└── {skill-name}/ # Individual skill directories
|
||||
├── SKILL.md # Skill definition and documentation
|
||||
└── scripts/
|
||||
└── run.sh # Skill execution script
|
||||
```
|
||||
|
||||
## Available Skills
|
||||
|
||||
### Testing Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [test-backend-coverage](./test-backend-coverage.SKILL.md) | test | Run Go backend tests with coverage analysis | ✅ Active |
|
||||
| [test-backend-unit](./test-backend-unit.SKILL.md) | test | Run fast Go unit tests without coverage | ✅ Active |
|
||||
| [test-frontend-coverage](./test-frontend-coverage.SKILL.md) | test | Run frontend tests with coverage reporting | ✅ Active |
|
||||
| [test-frontend-unit](./test-frontend-unit.SKILL.md) | test | Run fast frontend unit tests without coverage | ✅ Active |
|
||||
| [test-e2e-playwright](./test-e2e-playwright.SKILL.md) | test | Run Playwright E2E tests with browser selection | ✅ Active |
|
||||
| [test-e2e-playwright-debug](./test-e2e-playwright-debug.SKILL.md) | test | Run E2E tests in headed/debug mode for troubleshooting | ✅ Active |
|
||||
| [test-e2e-playwright-coverage](./test-e2e-playwright-coverage.SKILL.md) | test | Run E2E tests with coverage collection | ✅ Active |
|
||||
|
||||
### Integration Testing Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [integration-test-all](./integration-test-all.SKILL.md) | integration | Run all integration tests in sequence | ✅ Active |
|
||||
| [integration-test-coraza](./integration-test-coraza.SKILL.md) | integration | Test Coraza WAF integration | ✅ Active |
|
||||
| [integration-test-crowdsec](./integration-test-crowdsec.SKILL.md) | integration | Test CrowdSec bouncer integration | ✅ Active |
|
||||
| [integration-test-crowdsec-decisions](./integration-test-crowdsec-decisions.SKILL.md) | integration | Test CrowdSec decisions API | ✅ Active |
|
||||
| [integration-test-crowdsec-startup](./integration-test-crowdsec-startup.SKILL.md) | integration | Test CrowdSec startup sequence | ✅ Active |
|
||||
|
||||
### Security Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [security-scan-trivy](./security-scan-trivy.SKILL.md) | security | Run Trivy vulnerability scanner | ✅ Active |
|
||||
| [security-scan-go-vuln](./security-scan-go-vuln.SKILL.md) | security | Run Go vulnerability check | ✅ Active |
|
||||
|
||||
### QA Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [qa-precommit-all](./qa-precommit-all.SKILL.md) | qa | Run all pre-commit hooks on entire codebase | ✅ Active |
|
||||
|
||||
### Utility Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [utility-version-check](./utility-version-check.SKILL.md) | utility | Validate version matches git tag | ✅ Active |
|
||||
| [utility-clear-go-cache](./utility-clear-go-cache.SKILL.md) | utility | Clear Go build and module caches | ✅ Active |
|
||||
| [utility-bump-beta](./utility-bump-beta.SKILL.md) | utility | Increment beta version number | ✅ Active |
|
||||
| [utility-db-recovery](./utility-db-recovery.SKILL.md) | utility | Database integrity check and recovery | ✅ Active |
|
||||
|
||||
### Docker Skills
|
||||
|
||||
| Skill Name | Category | Description | Status |
|
||||
|------------|----------|-------------|--------|
|
||||
| [docker-start-dev](./docker-start-dev.SKILL.md) | docker | Start development Docker Compose environment | ✅ Active |
|
||||
| [docker-stop-dev](./docker-stop-dev.SKILL.md) | docker | Stop development Docker Compose environment | ✅ Active |
|
||||
| [docker-rebuild-e2e](./docker-rebuild-e2e.SKILL.md) | docker | Rebuild Docker image and restart E2E Playwright container | ✅ Active |
|
||||
| [docker-prune](./docker-prune.SKILL.md) | docker | Clean up unused Docker resources | ✅ Active |
|
||||
|
||||
## Usage
|
||||
|
||||
### Running Skills
|
||||
|
||||
Use the universal skill runner to execute any skill:
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
.github/skills/scripts/skill-runner.sh <skill-name> [args...]
|
||||
|
||||
# Example: Run backend coverage tests
|
||||
.github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
### From VS Code Tasks
|
||||
|
||||
Skills are integrated with VS Code tasks (`.vscode/tasks.json`):
|
||||
|
||||
1. Open Command Palette (`Ctrl+Shift+P` or `Cmd+Shift+P`)
|
||||
2. Select `Tasks: Run Task`
|
||||
3. Choose the task (e.g., `Test: Backend with Coverage`)
|
||||
|
||||
### In CI/CD Workflows
|
||||
|
||||
Reference skills in GitHub Actions:
|
||||
|
||||
```yaml
|
||||
- name: Run Backend Tests with Coverage
|
||||
run: .github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
### Validate a Single Skill
|
||||
|
||||
```bash
|
||||
python3 .github/skills/scripts/validate-skills.py --single .github/skills/test-backend-coverage/SKILL.md
|
||||
```
|
||||
|
||||
### Validate All Skills
|
||||
|
||||
```bash
|
||||
python3 .github/skills/scripts/validate-skills.py
|
||||
```
|
||||
|
||||
### Validation Checks
|
||||
|
||||
The validator ensures:
|
||||
- ✅ Required frontmatter fields are present
|
||||
- ✅ Field formats are correct (name, version, description)
|
||||
- ✅ Tags meet minimum/maximum requirements
|
||||
- ✅ Compatibility information is valid
|
||||
- ✅ Custom metadata follows project conventions
|
||||
|
||||
## Creating New Skills
|
||||
|
||||
### 1. Create Skill Directory Structure
|
||||
|
||||
```bash
|
||||
mkdir -p .github/skills/{skill-name}/scripts
|
||||
```
|
||||
|
||||
### 2. Create SKILL.md
|
||||
|
||||
Start with the template structure:
|
||||
|
||||
```markdown
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "skill-name"
|
||||
version: "1.0.0"
|
||||
description: "Brief description (max 120 chars)"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "tag1"
|
||||
- "tag2"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "tool"
|
||||
version: ">=1.0"
|
||||
optional: false
|
||||
metadata:
|
||||
category: "category-name"
|
||||
execution_time: "short|medium|long"
|
||||
risk_level: "low|medium|high"
|
||||
ci_cd_safe: true|false
|
||||
---
|
||||
|
||||
# Skill Name
|
||||
|
||||
## Overview
|
||||
|
||||
Brief description of what this skill does.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- List prerequisites
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh skill-name
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Usage
|
||||
|
||||
```bash
|
||||
# Example command
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: YYYY-MM-DD
|
||||
**Maintained by**: Charon Project
|
||||
```
|
||||
|
||||
### 3. Create Execution Script
|
||||
|
||||
Create `scripts/run.sh` with proper structure:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../../scripts" && pwd)"
|
||||
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
# Add validation calls here
|
||||
|
||||
# Execute skill logic
|
||||
log_step "EXECUTION" "Running skill"
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Your skill logic here
|
||||
|
||||
log_success "Skill completed successfully"
|
||||
```
|
||||
|
||||
### 4. Set Permissions
|
||||
|
||||
```bash
|
||||
chmod +x .github/skills/{skill-name}/scripts/run.sh
|
||||
```
|
||||
|
||||
### 5. Validate
|
||||
|
||||
```bash
|
||||
python3 .github/skills/scripts/validate-skills.py --single .github/skills/{skill-name}/SKILL.md
|
||||
```
|
||||
|
||||
### 6. Test
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh {skill-name}
|
||||
```
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Skill Names**: `{category}-{feature}-{variant}` (kebab-case)
|
||||
- **Categories**: `test`, `integration-test`, `security`, `qa`, `build`, `utility`, `docker`
|
||||
- **Examples**:
|
||||
- `test-backend-coverage`
|
||||
- `integration-test-crowdsec`
|
||||
- `security-scan-trivy`
|
||||
- `utility-version-check`
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Documentation
|
||||
- Keep SKILL.md under 500 lines
|
||||
- Use progressive disclosure (link to extended docs for complex topics)
|
||||
- Include practical examples
|
||||
- Document all prerequisites and environment variables
|
||||
|
||||
### Scripts
|
||||
- Always source helper scripts for consistent logging and error handling
|
||||
- Validate environment before execution
|
||||
- Use `set -euo pipefail` for robust error handling
|
||||
- Make scripts idempotent when possible
|
||||
- Clean up resources on exit
|
||||
|
||||
### Metadata
|
||||
- Use accurate `execution_time` values for scheduling
|
||||
- Set `ci_cd_safe: false` for skills requiring human oversight
|
||||
- Mark `idempotent: true` only if truly safe to run multiple times
|
||||
- Include all required dependencies in `requirements`
|
||||
|
||||
### Error Handling
|
||||
- Use helper functions (`log_error`, `error_exit`, `check_command_exists`)
|
||||
- Provide clear error messages with remediation steps
|
||||
- Return appropriate exit codes (0 = success, non-zero = failure)
|
||||
|
||||
## Helper Scripts Reference
|
||||
|
||||
### Logging Helpers (`_logging_helpers.sh`)
|
||||
|
||||
```bash
|
||||
log_info "message" # Informational message
|
||||
log_success "message" # Success message (green)
|
||||
log_warning "message" # Warning message (yellow)
|
||||
log_error "message" # Error message (red)
|
||||
log_debug "message" # Debug message (only if DEBUG=1)
|
||||
log_step "STEP" "msg" # Step header
|
||||
log_command "cmd" # Log command before executing
|
||||
```
|
||||
|
||||
### Error Handling Helpers (`_error_handling_helpers.sh`)
|
||||
|
||||
```bash
|
||||
error_exit "message" [exit_code] # Print error and exit
|
||||
check_command_exists "cmd" ["message"] # Verify command exists
|
||||
check_file_exists "file" ["message"] # Verify file exists
|
||||
check_dir_exists "dir" ["message"] # Verify directory exists
|
||||
run_with_retry max_attempts delay cmd... # Retry command with backoff
|
||||
trap_error [script_name] # Set up error trapping
|
||||
cleanup_on_exit cleanup_func # Register cleanup function
|
||||
```
|
||||
|
||||
### Environment Helpers (`_environment_helpers.sh`)
|
||||
|
||||
```bash
|
||||
validate_go_environment ["min_version"] # Check Go installation
|
||||
validate_python_environment ["min_version"] # Check Python installation
|
||||
validate_node_environment ["min_version"] # Check Node.js installation
|
||||
validate_docker_environment # Check Docker installation
|
||||
set_default_env "VAR" "default_value" # Set env var with default
|
||||
validate_project_structure file1 file2... # Check required files exist
|
||||
get_project_root ["marker_file"] # Find project root directory
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Skill not found
|
||||
```
|
||||
Error: Skill not found: skill-name
|
||||
```
|
||||
**Solution**: Verify the skill directory exists in `.github/skills/` and contains a `SKILL.md` file
|
||||
|
||||
### Skill script not executable
|
||||
```
|
||||
Error: Skill execution script is not executable
|
||||
```
|
||||
**Solution**: Run `chmod +x .github/skills/{skill-name}/scripts/run.sh`
|
||||
|
||||
### Validation errors
|
||||
```
|
||||
[ERROR] skill.SKILL.md :: description: Must be 120 characters or less
|
||||
```
|
||||
**Solution**: Fix the frontmatter field according to the error message and re-validate
|
||||
|
||||
### Command not found in skill
|
||||
```
|
||||
Error: go is not installed or not in PATH
|
||||
```
|
||||
**Solution**: Install the required dependency or ensure it's in your PATH
|
||||
|
||||
## Integration Points
|
||||
|
||||
### VS Code Tasks
|
||||
Skills are integrated in `.vscode/tasks.json`:
|
||||
```json
|
||||
{
|
||||
"label": "Test: Backend with Coverage",
|
||||
"type": "shell",
|
||||
"command": ".github/skills/scripts/skill-runner.sh test-backend-coverage",
|
||||
"group": "test"
|
||||
}
|
||||
```
|
||||
|
||||
### GitHub Actions
|
||||
Skills are referenced in `.github/workflows/`:
|
||||
```yaml
|
||||
- name: Run Backend Tests with Coverage
|
||||
run: .github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
### Pre-commit Hooks
|
||||
Skills can be used in `.pre-commit-config.yaml`:
|
||||
```yaml
|
||||
repos:
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: backend-coverage
|
||||
name: Backend Coverage Check
|
||||
entry: .github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
language: system
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
- [agentskills.io Specification](https://agentskills.io/specification)
|
||||
- [VS Code Copilot Agent Skills](https://code.visualstudio.com/docs/copilot/customization/agent-skills)
|
||||
- [Project Documentation](../../docs/)
|
||||
- [Contributing Guide](../../CONTRIBUTING.md)
|
||||
|
||||
## Support
|
||||
|
||||
For issues, questions, or contributions:
|
||||
1. Check existing [GitHub Issues](https://github.com/Wikid82/charon/issues)
|
||||
2. Review [CONTRIBUTING.md](../../CONTRIBUTING.md)
|
||||
3. Create a new issue if needed
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**License**: MIT
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# Docker: Prune Unused Resources - Execution Script
|
||||
# ==============================================================================
|
||||
# This script removes unused Docker resources to free up disk space.
|
||||
#
|
||||
# Usage: ./run.sh
|
||||
# Exit codes: 0 = success, non-zero = failure
|
||||
# ==============================================================================
|
||||
|
||||
# Remove unused Docker resources (containers, images, networks, build cache)
|
||||
exec docker system prune -f
|
||||
@@ -0,0 +1,293 @@
|
||||
---
|
||||
name: "docker-prune"
|
||||
version: "1.0.0"
|
||||
description: "Removes unused Docker resources including stopped containers, dangling images, and unused networks"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "docker"
|
||||
- "cleanup"
|
||||
- "maintenance"
|
||||
- "disk-space"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters: []
|
||||
outputs:
|
||||
- name: "exit_code"
|
||||
type: "integer"
|
||||
description: "0 on success, non-zero on failure"
|
||||
- name: "reclaimed_space"
|
||||
type: "string"
|
||||
description: "Amount of disk space freed"
|
||||
metadata:
|
||||
category: "docker"
|
||||
subcategory: "maintenance"
|
||||
execution_time: "short"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: false
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Docker: Prune Unused Resources
|
||||
|
||||
## Overview
|
||||
|
||||
Removes unused Docker resources to free up disk space and clean up the Docker environment. This includes stopped containers, dangling images, unused networks, and build cache. The operation is safe and only removes resources not currently in use.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine installed and running
|
||||
- Sufficient permissions to run Docker commands
|
||||
- No critical containers running (verify first)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
.github/skills/docker-prune-scripts/run.sh
|
||||
```
|
||||
|
||||
### Via Skill Runner
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-prune
|
||||
```
|
||||
|
||||
### Via VS Code Task
|
||||
|
||||
Use the task: **Docker: Prune Unused Resources**
|
||||
|
||||
## Parameters
|
||||
|
||||
This skill uses Docker's default prune behavior (safe mode). No parameters accepted.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
This skill requires no environment variables.
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0
|
||||
- **Error Exit Codes**: Non-zero on failure
|
||||
- **Console Output**: List of removed resources and space reclaimed
|
||||
|
||||
### Output Example
|
||||
|
||||
```
|
||||
Deleted Containers:
|
||||
f8d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab
|
||||
|
||||
Deleted Networks:
|
||||
charon-test_default
|
||||
old-network_default
|
||||
|
||||
Deleted Images:
|
||||
untagged: myimage@sha256:abcdef1234567890...
|
||||
deleted: sha256:1234567890abcdef...
|
||||
|
||||
Deleted build cache objects:
|
||||
abcd1234
|
||||
efgh5678
|
||||
|
||||
Total reclaimed space: 2.5GB
|
||||
```
|
||||
|
||||
## What Gets Removed
|
||||
|
||||
The `docker system prune -f` command removes:
|
||||
|
||||
1. **Stopped Containers**: Containers not currently running
|
||||
2. **Dangling Images**: Images with no tag (intermediate layers)
|
||||
3. **Unused Networks**: Networks with no connected containers
|
||||
4. **Build Cache**: Cached layers from image builds
|
||||
|
||||
## What Gets Preserved
|
||||
|
||||
This command **DOES NOT** remove:
|
||||
- **Running Containers**: Active containers are untouched
|
||||
- **Tagged Images**: Images with tags are preserved
|
||||
- **Volumes**: Data volumes are never removed
|
||||
- **Used Networks**: Networks with connected containers
|
||||
- **Active Build Cache**: Cache for recent builds
|
||||
|
||||
## Safety Features
|
||||
|
||||
- **Force Flag (`-f`)**: Skips confirmation prompt (safe for automation)
|
||||
- **Safe by Default**: Only removes truly unused resources
|
||||
- **Volume Protection**: Volumes require separate `docker volume prune` command
|
||||
- **Running Container Protection**: Cannot remove active containers
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Regular Cleanup
|
||||
|
||||
```bash
|
||||
# Clean up Docker environment
|
||||
.github/skills/docker-prune-scripts/run.sh
|
||||
```
|
||||
|
||||
### Example 2: Check Disk Usage Before/After
|
||||
|
||||
```bash
|
||||
# Check current usage
|
||||
docker system df
|
||||
|
||||
# Run cleanup
|
||||
.github/skills/docker-prune-scripts/run.sh
|
||||
|
||||
# Verify freed space
|
||||
docker system df
|
||||
```
|
||||
|
||||
### Example 3: Aggressive Cleanup (Manual)
|
||||
|
||||
```bash
|
||||
# Standard prune
|
||||
.github/skills/docker-prune-scripts/run.sh
|
||||
|
||||
# Additionally prune volumes (WARNING: data loss)
|
||||
docker volume prune -f
|
||||
|
||||
# Remove all unused images (not just dangling)
|
||||
docker image prune -a -f
|
||||
```
|
||||
|
||||
## Disk Space Analysis
|
||||
|
||||
Check Docker disk usage:
|
||||
|
||||
```bash
|
||||
# Summary view
|
||||
docker system df
|
||||
|
||||
# Detailed view
|
||||
docker system df -v
|
||||
```
|
||||
|
||||
Output shows:
|
||||
- **Images**: Total size of cached images
|
||||
- **Containers**: Size of container writable layers
|
||||
- **Local Volumes**: Size of data volumes
|
||||
- **Build Cache**: Size of cached build layers
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Disk space is running low
|
||||
- After development cycles (many builds)
|
||||
- After running integration tests
|
||||
- Before system backup/snapshot
|
||||
- As part of regular maintenance
|
||||
- After Docker image experiments
|
||||
|
||||
## Frequency Recommendations
|
||||
|
||||
- **Daily**: For active development machines
|
||||
- **Weekly**: For CI/CD build servers
|
||||
- **Monthly**: For production servers (cautiously)
|
||||
- **On-Demand**: When disk space is low
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
### Permission Denied
|
||||
```
|
||||
Error: permission denied
|
||||
```
|
||||
Solution: Add user to docker group or use sudo
|
||||
|
||||
### Daemon Not Running
|
||||
```
|
||||
Error: Cannot connect to Docker daemon
|
||||
```
|
||||
Solution: Start Docker service
|
||||
|
||||
### Resource in Use
|
||||
```
|
||||
Error: resource is in use
|
||||
```
|
||||
This is normal - only unused resources are removed
|
||||
|
||||
## Advanced Cleanup Options
|
||||
|
||||
For more aggressive cleanup:
|
||||
|
||||
### Remove All Unused Images
|
||||
|
||||
```bash
|
||||
docker image prune -a -f
|
||||
```
|
||||
|
||||
### Remove Unused Volumes (DANGER: Data Loss)
|
||||
|
||||
```bash
|
||||
docker volume prune -f
|
||||
```
|
||||
|
||||
### Complete System Prune (DANGER)
|
||||
|
||||
```bash
|
||||
docker system prune -a --volumes -f
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [docker-stop-dev](./docker-stop-dev.SKILL.md) - Stop containers before cleanup
|
||||
- [docker-start-dev](./docker-start-dev.SKILL.md) - Restart after cleanup
|
||||
- [utility-clear-go-cache](./utility-clear-go-cache.SKILL.md) - Clear Go build cache
|
||||
|
||||
## Notes
|
||||
|
||||
- **Idempotent**: Safe to run multiple times
|
||||
- **Low Risk**: Only removes unused resources
|
||||
- **No Data Loss**: Volumes are protected by default
|
||||
- **Fast Execution**: Typically completes in seconds
|
||||
- **No Network Required**: Local operation only
|
||||
- **Not CI/CD Safe**: Can interfere with parallel builds
|
||||
- **Build Cache**: May slow down next build if cache is cleared
|
||||
|
||||
## Disk Space Recovery
|
||||
|
||||
Typical space recovery by resource type:
|
||||
- **Stopped Containers**: 10-100 MB each
|
||||
- **Dangling Images**: 100 MB - 2 GB total
|
||||
- **Build Cache**: 1-10 GB (if many builds)
|
||||
- **Unused Networks**: Negligible space
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Space Freed
|
||||
|
||||
- Check for running containers: `docker ps`
|
||||
- Verify images are untagged: `docker images -f "dangling=true"`
|
||||
- Check volume usage: `docker volume ls`
|
||||
|
||||
### Space Still Low After Prune
|
||||
|
||||
- Use aggressive pruning (see Advanced Cleanup)
|
||||
- Check non-Docker disk usage: `df -h`
|
||||
- Consider increasing disk allocation
|
||||
|
||||
### Container Won't Be Removed
|
||||
|
||||
- Check if container is running: `docker ps`
|
||||
- Stop container first: `docker stop container_name`
|
||||
- Force removal: `docker rm -f container_name`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Docker Command**: `docker system prune -f`
|
||||
+314
@@ -0,0 +1,314 @@
|
||||
#!/usr/bin/env bash
|
||||
# Docker: Rebuild E2E Environment - Execution Script
|
||||
#
|
||||
# Rebuilds the Docker image and restarts the Playwright E2E testing
|
||||
# environment with fresh code and optionally clean state.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Docker compose file for Playwright E2E tests
|
||||
COMPOSE_FILE=".docker/compose/docker-compose.playwright.yml"
|
||||
CONTAINER_NAME="charon-playwright"
|
||||
IMAGE_NAME="charon:local"
|
||||
HEALTH_TIMEOUT=60
|
||||
HEALTH_INTERVAL=5
|
||||
|
||||
# Default parameter values
|
||||
NO_CACHE=false
|
||||
CLEAN=false
|
||||
PROFILE=""
|
||||
|
||||
# Parse command-line arguments
|
||||
parse_arguments() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--no-cache)
|
||||
NO_CACHE=true
|
||||
shift
|
||||
;;
|
||||
--clean)
|
||||
CLEAN=true
|
||||
shift
|
||||
;;
|
||||
--profile=*)
|
||||
PROFILE="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--profile)
|
||||
PROFILE="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
show_help
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_warning "Unknown argument: $1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Show help message
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Usage: run.sh [OPTIONS]
|
||||
|
||||
Rebuild Docker image and restart E2E Playwright container.
|
||||
|
||||
Options:
|
||||
--no-cache Force rebuild without Docker cache
|
||||
--clean Remove test volumes for fresh state
|
||||
--profile=PROFILE Docker Compose profile to enable
|
||||
(security-tests, notification-tests)
|
||||
-h, --help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
DOCKER_NO_CACHE Force rebuild without cache (default: false)
|
||||
SKIP_VOLUME_CLEANUP Preserve test data volumes (default: false)
|
||||
|
||||
Examples:
|
||||
run.sh # Standard rebuild
|
||||
run.sh --no-cache # Force complete rebuild
|
||||
run.sh --clean # Rebuild with fresh volumes
|
||||
run.sh --profile=security-tests # Enable CrowdSec for testing
|
||||
run.sh --no-cache --clean # Complete fresh rebuild
|
||||
EOF
|
||||
}
|
||||
|
||||
# Stop existing containers
|
||||
stop_containers() {
|
||||
log_step "STOP" "Stopping existing E2E containers"
|
||||
|
||||
local compose_cmd="docker compose -f ${COMPOSE_FILE}"
|
||||
|
||||
# Add profile if specified
|
||||
if [[ -n "${PROFILE}" ]]; then
|
||||
compose_cmd="${compose_cmd} --profile ${PROFILE}"
|
||||
fi
|
||||
|
||||
# Stop and remove containers
|
||||
if ${compose_cmd} ps -q 2>/dev/null | grep -q .; then
|
||||
log_info "Stopping containers..."
|
||||
${compose_cmd} down --remove-orphans || true
|
||||
else
|
||||
log_info "No running containers to stop"
|
||||
fi
|
||||
}
|
||||
|
||||
# Clean volumes if requested
|
||||
clean_volumes() {
|
||||
if [[ "${CLEAN}" != "true" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ "${SKIP_VOLUME_CLEANUP:-false}" == "true" ]]; then
|
||||
log_warning "Skipping volume cleanup (SKIP_VOLUME_CLEANUP=true)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_step "CLEAN" "Removing test volumes"
|
||||
|
||||
local volumes=(
|
||||
"playwright_data"
|
||||
"playwright_caddy_data"
|
||||
"playwright_caddy_config"
|
||||
"playwright_crowdsec_data"
|
||||
"playwright_crowdsec_config"
|
||||
)
|
||||
|
||||
for vol in "${volumes[@]}"; do
|
||||
# Try both prefixed and unprefixed volume names
|
||||
for prefix in "compose_" ""; do
|
||||
local full_name="${prefix}${vol}"
|
||||
if docker volume inspect "${full_name}" &>/dev/null; then
|
||||
log_info "Removing volume: ${full_name}"
|
||||
docker volume rm "${full_name}" || true
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
log_success "Volumes cleaned"
|
||||
}
|
||||
|
||||
# Build Docker image
|
||||
build_image() {
|
||||
log_step "BUILD" "Building Docker image: ${IMAGE_NAME}"
|
||||
|
||||
local build_args=("-t" "${IMAGE_NAME}" ".")
|
||||
|
||||
if [[ "${NO_CACHE}" == "true" ]] || [[ "${DOCKER_NO_CACHE:-false}" == "true" ]]; then
|
||||
log_info "Building with --no-cache"
|
||||
build_args=("--no-cache" "${build_args[@]}")
|
||||
fi
|
||||
|
||||
log_command "docker build ${build_args[*]}"
|
||||
|
||||
if ! docker build "${build_args[@]}"; then
|
||||
error_exit "Docker build failed"
|
||||
fi
|
||||
|
||||
log_success "Image built successfully: ${IMAGE_NAME}"
|
||||
}
|
||||
|
||||
# Start containers
|
||||
start_containers() {
|
||||
log_step "START" "Starting E2E containers"
|
||||
|
||||
local compose_cmd="docker compose -f ${COMPOSE_FILE}"
|
||||
|
||||
# Add profile if specified
|
||||
if [[ -n "${PROFILE}" ]]; then
|
||||
log_info "Enabling profile: ${PROFILE}"
|
||||
compose_cmd="${compose_cmd} --profile ${PROFILE}"
|
||||
fi
|
||||
|
||||
log_command "${compose_cmd} up -d"
|
||||
|
||||
if ! ${compose_cmd} up -d; then
|
||||
error_exit "Failed to start containers"
|
||||
fi
|
||||
|
||||
log_success "Containers started"
|
||||
}
|
||||
|
||||
# Wait for container health
|
||||
wait_for_health() {
|
||||
log_step "HEALTH" "Waiting for container to be healthy"
|
||||
|
||||
local elapsed=0
|
||||
local healthy=false
|
||||
|
||||
while [[ ${elapsed} -lt ${HEALTH_TIMEOUT} ]]; do
|
||||
local health_status
|
||||
health_status=$(docker inspect --format='{{.State.Health.Status}}' "${CONTAINER_NAME}" 2>/dev/null || echo "unknown")
|
||||
|
||||
case "${health_status}" in
|
||||
healthy)
|
||||
healthy=true
|
||||
break
|
||||
;;
|
||||
unhealthy)
|
||||
log_error "Container is unhealthy"
|
||||
docker logs "${CONTAINER_NAME}" --tail 20
|
||||
error_exit "Container health check failed"
|
||||
;;
|
||||
starting)
|
||||
log_info "Health status: starting (${elapsed}s/${HEALTH_TIMEOUT}s)"
|
||||
;;
|
||||
*)
|
||||
log_info "Health status: ${health_status} (${elapsed}s/${HEALTH_TIMEOUT}s)"
|
||||
;;
|
||||
esac
|
||||
|
||||
sleep "${HEALTH_INTERVAL}"
|
||||
elapsed=$((elapsed + HEALTH_INTERVAL))
|
||||
done
|
||||
|
||||
if [[ "${healthy}" != "true" ]]; then
|
||||
log_error "Container did not become healthy in ${HEALTH_TIMEOUT}s"
|
||||
docker logs "${CONTAINER_NAME}" --tail 50
|
||||
error_exit "Health check timeout"
|
||||
fi
|
||||
|
||||
log_success "Container is healthy"
|
||||
}
|
||||
|
||||
# Verify environment
|
||||
verify_environment() {
|
||||
log_step "VERIFY" "Verifying E2E environment"
|
||||
|
||||
# Check container is running
|
||||
if ! docker ps --filter "name=${CONTAINER_NAME}" --format "{{.Names}}" | grep -q "${CONTAINER_NAME}"; then
|
||||
error_exit "Container ${CONTAINER_NAME} is not running"
|
||||
fi
|
||||
|
||||
# Test health endpoint
|
||||
log_info "Testing health endpoint..."
|
||||
if curl -sf http://localhost:8080/api/v1/health &>/dev/null; then
|
||||
log_success "Health endpoint responding"
|
||||
else
|
||||
log_warning "Health endpoint not responding (may need more time)"
|
||||
fi
|
||||
|
||||
# Show container status
|
||||
log_info "Container status:"
|
||||
docker ps --filter "name=charon-playwright" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||||
}
|
||||
|
||||
# Show summary
|
||||
show_summary() {
|
||||
log_step "SUMMARY" "E2E environment ready"
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " E2E Environment Ready"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
echo " Application URL: http://localhost:8080"
|
||||
echo " Health Check: http://localhost:8080/api/v1/health"
|
||||
echo " Container: ${CONTAINER_NAME}"
|
||||
echo ""
|
||||
echo " Run E2E tests:"
|
||||
echo " .github/skills/scripts/skill-runner.sh test-e2e-playwright"
|
||||
echo ""
|
||||
echo " Run in debug mode:"
|
||||
echo " .github/skills/scripts/skill-runner.sh test-e2e-playwright-debug"
|
||||
echo ""
|
||||
echo " View logs:"
|
||||
echo " docker logs ${CONTAINER_NAME} -f"
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
parse_arguments "$@"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_docker_environment || error_exit "Docker is not available"
|
||||
check_command_exists "docker" "Docker is required"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
check_file_exists "Dockerfile" "Dockerfile is required"
|
||||
check_file_exists "${COMPOSE_FILE}" "Playwright compose file is required"
|
||||
|
||||
# Log configuration
|
||||
log_step "CONFIG" "Rebuild configuration"
|
||||
log_info "No cache: ${NO_CACHE}"
|
||||
log_info "Clean volumes: ${CLEAN}"
|
||||
log_info "Profile: ${PROFILE:-<none>}"
|
||||
log_info "Compose file: ${COMPOSE_FILE}"
|
||||
|
||||
# Execute rebuild steps
|
||||
stop_containers
|
||||
clean_volumes
|
||||
build_image
|
||||
start_containers
|
||||
wait_for_health
|
||||
verify_environment
|
||||
show_summary
|
||||
|
||||
log_success "E2E environment rebuild complete"
|
||||
}
|
||||
|
||||
# Run main with all arguments
|
||||
main "$@"
|
||||
@@ -0,0 +1,300 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "docker-rebuild-e2e"
|
||||
version: "1.0.0"
|
||||
description: "Rebuild Docker image and restart E2E Playwright container with fresh code and clean state"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "docker"
|
||||
- "e2e"
|
||||
- "playwright"
|
||||
- "rebuild"
|
||||
- "testing"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "docker-compose"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "DOCKER_NO_CACHE"
|
||||
description: "Set to 'true' to force a complete rebuild without cache"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "SKIP_VOLUME_CLEANUP"
|
||||
description: "Set to 'true' to preserve test data volumes"
|
||||
default: "false"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "no-cache"
|
||||
type: "boolean"
|
||||
description: "Force rebuild without Docker cache"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "clean"
|
||||
type: "boolean"
|
||||
description: "Remove test volumes for a completely fresh state"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "profile"
|
||||
type: "string"
|
||||
description: "Docker Compose profile to enable (security-tests, notification-tests)"
|
||||
default: ""
|
||||
required: false
|
||||
outputs:
|
||||
- name: "exit_code"
|
||||
type: "integer"
|
||||
description: "0 on success, non-zero on failure"
|
||||
metadata:
|
||||
category: "docker"
|
||||
subcategory: "e2e"
|
||||
execution_time: "long"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Docker: Rebuild E2E Environment
|
||||
|
||||
## Overview
|
||||
|
||||
Rebuilds the Charon Docker image and restarts the Playwright E2E testing environment with fresh code. This skill handles the complete lifecycle: stopping existing containers, optionally cleaning volumes, rebuilding the image, and starting fresh containers with health check verification.
|
||||
|
||||
**Use this skill when:**
|
||||
- You've made code changes and need to test them in E2E tests
|
||||
- E2E tests are failing due to stale container state
|
||||
- You need a clean slate for debugging
|
||||
- The container is in an inconsistent state
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine installed and running
|
||||
- Docker Compose V2 installed
|
||||
- Dockerfile in repository root
|
||||
- `.docker/compose/docker-compose.playwright.yml` file
|
||||
- Network access for pulling base images (if needed)
|
||||
- Sufficient disk space for image rebuild
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Rebuild image and restart E2E container:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e
|
||||
```
|
||||
|
||||
### Force Rebuild (No Cache)
|
||||
|
||||
Rebuild from scratch without Docker cache:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --no-cache
|
||||
```
|
||||
|
||||
### Clean Rebuild
|
||||
|
||||
Remove test volumes and rebuild with fresh state:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean
|
||||
```
|
||||
|
||||
### With Security Testing Services
|
||||
|
||||
Enable CrowdSec for security testing:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --profile=security-tests
|
||||
```
|
||||
|
||||
### With Notification Testing Services
|
||||
|
||||
Enable MailHog for email testing:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --profile=notification-tests
|
||||
```
|
||||
|
||||
### Full Clean Rebuild with All Services
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --no-cache --clean --profile=security-tests
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| no-cache | boolean | No | false | Force rebuild without Docker cache |
|
||||
| clean | boolean | No | false | Remove test volumes for fresh state |
|
||||
| profile | string | No | "" | Docker Compose profile to enable |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| DOCKER_NO_CACHE | No | false | Force rebuild without cache |
|
||||
| SKIP_VOLUME_CLEANUP | No | false | Preserve test data volumes |
|
||||
|
||||
## What This Skill Does
|
||||
|
||||
1. **Stop Existing Containers**: Gracefully stops any running Playwright containers
|
||||
2. **Clean Volumes** (if `--clean`): Removes test data volumes for fresh state
|
||||
3. **Rebuild Image**: Builds `charon:local` image from Dockerfile
|
||||
4. **Start Containers**: Starts the Playwright compose environment
|
||||
5. **Wait for Health**: Verifies container health before returning
|
||||
6. **Report Status**: Outputs container status and connection info
|
||||
|
||||
## Docker Compose Configuration
|
||||
|
||||
This skill uses `.docker/compose/docker-compose.playwright.yml` which includes:
|
||||
|
||||
- **charon-app**: Main application container on port 8080
|
||||
- **crowdsec** (profile: security-tests): Security bouncer for WAF testing
|
||||
- **mailhog** (profile: notification-tests): Email testing service
|
||||
|
||||
### Volumes Created
|
||||
|
||||
| Volume | Purpose |
|
||||
|--------|---------|
|
||||
| playwright_data | Application data and SQLite database |
|
||||
| playwright_caddy_data | Caddy server data |
|
||||
| playwright_caddy_config | Caddy configuration |
|
||||
| playwright_crowdsec_data | CrowdSec data (if enabled) |
|
||||
| playwright_crowdsec_config | CrowdSec config (if enabled) |
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Quick Rebuild After Code Change
|
||||
|
||||
```bash
|
||||
# Rebuild and restart after making backend changes
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e
|
||||
|
||||
# Run E2E tests
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright
|
||||
```
|
||||
|
||||
### Example 2: Debug Failing Tests with Clean State
|
||||
|
||||
```bash
|
||||
# Complete clean rebuild
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean --no-cache
|
||||
|
||||
# Run specific test in debug mode
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --grep="failing-test"
|
||||
```
|
||||
|
||||
### Example 3: Test Security Features
|
||||
|
||||
```bash
|
||||
# Start with CrowdSec enabled
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --profile=security-tests
|
||||
|
||||
# Run security-related E2E tests
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --grep="security"
|
||||
```
|
||||
|
||||
## Health Check Verification
|
||||
|
||||
After starting, the skill waits for the health check to pass:
|
||||
|
||||
```bash
|
||||
# Health endpoint checked
|
||||
curl -sf http://localhost:8080/api/v1/health
|
||||
```
|
||||
|
||||
The skill will:
|
||||
- Wait up to 60 seconds for container to be healthy
|
||||
- Check every 5 seconds
|
||||
- Report final health status
|
||||
- Exit with error if health check fails
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Docker Build Failed
|
||||
```
|
||||
Error: docker build failed
|
||||
```
|
||||
**Solution**: Check Dockerfile syntax, ensure all COPY sources exist
|
||||
|
||||
#### Port Already in Use
|
||||
```
|
||||
Error: bind: address already in use
|
||||
```
|
||||
**Solution**: Stop conflicting services on port 8080
|
||||
|
||||
#### Health Check Timeout
|
||||
```
|
||||
Error: Container did not become healthy in 60s
|
||||
```
|
||||
**Solution**: Check container logs with `docker logs charon-playwright`
|
||||
|
||||
#### Volume Permission Issues
|
||||
```
|
||||
Error: permission denied
|
||||
```
|
||||
**Solution**: Run with `--clean` to recreate volumes with proper permissions
|
||||
|
||||
## Verifying the Environment
|
||||
|
||||
After the skill completes, verify the environment:
|
||||
|
||||
```bash
|
||||
# Check container status
|
||||
docker ps --filter "name=charon-playwright"
|
||||
|
||||
# Check logs
|
||||
docker logs charon-playwright --tail 50
|
||||
|
||||
# Test health endpoint
|
||||
curl http://localhost:8080/api/v1/health
|
||||
|
||||
# Check database state
|
||||
docker exec charon-playwright sqlite3 /app/data/charon.db ".tables"
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [test-e2e-playwright](./test-e2e-playwright.SKILL.md) - Run E2E tests
|
||||
- [test-e2e-playwright-debug](./test-e2e-playwright-debug.SKILL.md) - Debug E2E tests
|
||||
- [docker-start-dev](./docker-start-dev.SKILL.md) - Start development environment
|
||||
- [docker-stop-dev](./docker-stop-dev.SKILL.md) - Stop development environment
|
||||
- [docker-prune](./docker-prune.SKILL.md) - Clean up Docker resources
|
||||
|
||||
## Key File Locations
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `Dockerfile` | Main application Dockerfile |
|
||||
| `.docker/compose/docker-compose.playwright.yml` | E2E test compose config |
|
||||
| `playwright.config.js` | Playwright test configuration |
|
||||
| `tests/` | E2E test files |
|
||||
| `playwright/.auth/user.json` | Stored authentication state |
|
||||
|
||||
## Notes
|
||||
|
||||
- **Build Time**: Full rebuild takes 2-5 minutes depending on cache
|
||||
- **Disk Space**: Image is ~500MB, volumes add ~100MB
|
||||
- **Network**: Base images may need to be pulled on first run
|
||||
- **Idempotent**: Safe to run multiple times
|
||||
- **CI/CD Safe**: Designed for use in automated pipelines
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-21
|
||||
**Maintained by**: Charon Project Team
|
||||
**Compose File**: `.docker/compose/docker-compose.playwright.yml`
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# Docker: Start Development Environment - Execution Script
|
||||
# ==============================================================================
|
||||
# This script starts the Docker Compose development environment.
|
||||
#
|
||||
# Usage: ./run.sh
|
||||
# Exit codes: 0 = success, non-zero = failure
|
||||
# ==============================================================================
|
||||
|
||||
# Determine the repository root directory
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
|
||||
# Change to repository root
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# Start development environment with Docker Compose
|
||||
exec docker compose -f .docker/compose/docker-compose.dev.yml up -d
|
||||
@@ -0,0 +1,269 @@
|
||||
---
|
||||
name: "docker-start-dev"
|
||||
version: "1.0.0"
|
||||
description: "Starts the Charon development Docker Compose environment with all required services"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "docker"
|
||||
- "development"
|
||||
- "compose"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "docker-compose"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters: []
|
||||
outputs:
|
||||
- name: "exit_code"
|
||||
type: "integer"
|
||||
description: "0 on success, non-zero on failure"
|
||||
metadata:
|
||||
category: "docker"
|
||||
subcategory: "environment"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: false
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Docker: Start Development Environment
|
||||
|
||||
## Overview
|
||||
|
||||
Starts the Charon development Docker Compose environment in detached mode. This brings up all required services including the application, database, CrowdSec, and any other dependencies defined in `.docker/compose/docker-compose.dev.yml`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine installed and running
|
||||
- Docker Compose V2 installed
|
||||
- `.docker/compose/docker-compose.dev.yml` file in repository
|
||||
- Network access (for pulling images)
|
||||
- Sufficient system resources (CPU, memory, disk)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
.github/skills/docker-start-dev-scripts/run.sh
|
||||
```
|
||||
|
||||
### Via Skill Runner
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-start-dev
|
||||
```
|
||||
|
||||
### Via VS Code Task
|
||||
|
||||
Use the task: **Docker: Start Dev Environment**
|
||||
|
||||
## Parameters
|
||||
|
||||
This skill accepts no parameters. Services are configured in `.docker/compose/docker-compose.dev.yml`.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
This skill uses environment variables defined in:
|
||||
- `.env` (if present)
|
||||
- `.docker/compose/docker-compose.dev.yml` environment section
|
||||
- Shell environment
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 - All services started successfully
|
||||
- **Error Exit Codes**: Non-zero - Service startup failed
|
||||
- **Console Output**: Docker Compose logs and status
|
||||
|
||||
### Output Example
|
||||
|
||||
```
|
||||
[+] Running 5/5
|
||||
✔ Network charon-dev_default Created
|
||||
✔ Container charon-dev-db-1 Started
|
||||
✔ Container charon-dev-crowdsec-1 Started
|
||||
✔ Container charon-dev-app-1 Started
|
||||
✔ Container charon-dev-caddy-1 Started
|
||||
```
|
||||
|
||||
## What Gets Started
|
||||
|
||||
Services defined in `.docker/compose/docker-compose.dev.yml`:
|
||||
1. **charon-app**: Main application container
|
||||
2. **charon-db**: SQLite or PostgreSQL database
|
||||
3. **crowdsec**: Security bouncer
|
||||
4. **caddy**: Reverse proxy (if configured)
|
||||
5. **Other Services**: As defined in compose file
|
||||
|
||||
## Service Startup Order
|
||||
|
||||
Docker Compose respects `depends_on` directives:
|
||||
1. Database services start first
|
||||
2. Security services (CrowdSec) start next
|
||||
3. Application services start after dependencies
|
||||
4. Reverse proxy starts last
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Start Development Environment
|
||||
|
||||
```bash
|
||||
# Start all development services
|
||||
.github/skills/docker-start-dev-scripts/run.sh
|
||||
|
||||
# Verify services are running
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml ps
|
||||
```
|
||||
|
||||
### Example 2: Start and View Logs
|
||||
|
||||
```bash
|
||||
# Start services in detached mode
|
||||
.github/skills/docker-start-dev-scripts/run.sh
|
||||
|
||||
# Follow logs from all services
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml logs -f
|
||||
```
|
||||
|
||||
### Example 3: Start and Test Application
|
||||
|
||||
```bash
|
||||
# Start development environment
|
||||
.github/skills/docker-start-dev-scripts/run.sh
|
||||
|
||||
# Wait for services to be healthy
|
||||
sleep 10
|
||||
|
||||
# Test application endpoint
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
## Service Health Checks
|
||||
|
||||
After starting, verify services are healthy:
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml ps
|
||||
|
||||
# Check specific service logs
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml logs app
|
||||
|
||||
# Execute command in running container
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml exec app /bin/sh
|
||||
```
|
||||
|
||||
## Port Mappings
|
||||
|
||||
Default development ports (check `.docker/compose/docker-compose.dev.yml`):
|
||||
- **8080**: Application HTTP
|
||||
- **8443**: Application HTTPS (if configured)
|
||||
- **9000**: Admin panel (if configured)
|
||||
- **3000**: Frontend dev server (if configured)
|
||||
|
||||
## Detached Mode
|
||||
|
||||
The `-d` flag runs containers in detached mode:
|
||||
- Services run in background
|
||||
- Terminal is freed for other commands
|
||||
- Use `docker compose logs -f` to view output
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
### Port Already in Use
|
||||
```
|
||||
Error: bind: address already in use
|
||||
```
|
||||
Solution: Stop conflicting service or change port in compose file
|
||||
|
||||
### Image Pull Failed
|
||||
```
|
||||
Error: failed to pull image
|
||||
```
|
||||
Solution: Check network connection, authenticate to registry
|
||||
|
||||
### Insufficient Resources
|
||||
```
|
||||
Error: failed to start container
|
||||
```
|
||||
Solution: Free up system resources, stop other containers
|
||||
|
||||
### Configuration Error
|
||||
```
|
||||
Error: invalid compose file
|
||||
```
|
||||
Solution: Validate compose file with `docker compose config`
|
||||
|
||||
## Post-Startup Verification
|
||||
|
||||
After starting, verify:
|
||||
|
||||
1. **All Services Running**:
|
||||
```bash
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml ps
|
||||
```
|
||||
|
||||
2. **Application Accessible**:
|
||||
```bash
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
3. **No Error Logs**:
|
||||
```bash
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml logs --tail=50
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [docker-stop-dev](./docker-stop-dev.SKILL.md) - Stop development environment
|
||||
- [docker-prune](./docker-prune.SKILL.md) - Clean up Docker resources
|
||||
- [integration-test-all](./integration-test-all.SKILL.md) - Run integration tests
|
||||
|
||||
## Notes
|
||||
|
||||
- **Idempotent**: Safe to run multiple times (recreates only if needed)
|
||||
- **Resource Usage**: Development mode may use more resources than production
|
||||
- **Data Persistence**: Volumes persist data across restarts
|
||||
- **Network Access**: Requires internet for initial image pulls
|
||||
- **Not CI/CD Safe**: Intended for local development only
|
||||
- **Background Execution**: Services run in detached mode
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Services Won't Start
|
||||
|
||||
1. Check Docker daemon: `docker info`
|
||||
2. Validate compose file: `docker compose -f .docker/compose/docker-compose.dev.yml config`
|
||||
3. Check available resources: `docker stats`
|
||||
4. Review logs: `docker compose -f .docker/compose/docker-compose.dev.yml logs`
|
||||
|
||||
### Slow Startup
|
||||
|
||||
- First run pulls images (may take time)
|
||||
- Subsequent runs use cached images
|
||||
- Use `docker compose pull` to pre-pull images
|
||||
|
||||
### Service Dependency Issues
|
||||
|
||||
- Check `depends_on` in compose file
|
||||
- Add healthchecks for critical services
|
||||
- Increase startup timeout if needed
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Compose File**: `.docker/compose/docker-compose.dev.yml`
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# Docker: Stop Development Environment - Execution Script
|
||||
# ==============================================================================
|
||||
# This script stops and removes the Docker Compose development environment.
|
||||
#
|
||||
# Usage: ./run.sh
|
||||
# Exit codes: 0 = success, non-zero = failure
|
||||
# ==============================================================================
|
||||
|
||||
# Determine the repository root directory
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
|
||||
# Change to repository root
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# Stop development environment with Docker Compose
|
||||
exec docker compose -f .docker/compose/docker-compose.dev.yml down
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
name: "docker-stop-dev"
|
||||
version: "1.0.0"
|
||||
description: "Stops and removes the Charon development Docker Compose environment and containers"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "docker"
|
||||
- "development"
|
||||
- "compose"
|
||||
- "cleanup"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "docker-compose"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters: []
|
||||
outputs:
|
||||
- name: "exit_code"
|
||||
type: "integer"
|
||||
description: "0 on success, non-zero on failure"
|
||||
metadata:
|
||||
category: "docker"
|
||||
subcategory: "environment"
|
||||
execution_time: "short"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: false
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Docker: Stop Development Environment
|
||||
|
||||
## Overview
|
||||
|
||||
Stops and removes all containers defined in the Charon development Docker Compose environment. This gracefully shuts down services, removes containers, and cleans up the default network while preserving volumes and data.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine installed and running
|
||||
- Docker Compose V2 installed
|
||||
- Development environment previously started
|
||||
- `.docker/compose/docker-compose.dev.yml` file in repository
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
.github/skills/docker-stop-dev-scripts/run.sh
|
||||
```
|
||||
|
||||
### Via Skill Runner
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-stop-dev
|
||||
```
|
||||
|
||||
### Via VS Code Task
|
||||
|
||||
Use the task: **Docker: Stop Dev Environment**
|
||||
|
||||
## Parameters
|
||||
|
||||
This skill accepts no parameters.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
This skill requires no environment variables.
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 - All services stopped successfully
|
||||
- **Error Exit Codes**: Non-zero - Service shutdown failed
|
||||
- **Console Output**: Docker Compose shutdown status
|
||||
|
||||
### Output Example
|
||||
|
||||
```
|
||||
[+] Running 5/5
|
||||
✔ Container charon-dev-caddy-1 Removed
|
||||
✔ Container charon-dev-app-1 Removed
|
||||
✔ Container charon-dev-crowdsec-1 Removed
|
||||
✔ Container charon-dev-db-1 Removed
|
||||
✔ Network charon-dev_default Removed
|
||||
```
|
||||
|
||||
## What Gets Stopped
|
||||
|
||||
Services defined in `.docker/compose/docker-compose.dev.yml`:
|
||||
1. **Application Containers**: Charon main app
|
||||
2. **Database Containers**: SQLite/PostgreSQL services
|
||||
3. **Security Services**: CrowdSec bouncer
|
||||
4. **Reverse Proxy**: Caddy server
|
||||
5. **Network**: Default Docker Compose network
|
||||
|
||||
## What Gets Preserved
|
||||
|
||||
The `down` command preserves:
|
||||
- **Volumes**: Database data persists
|
||||
- **Images**: Docker images remain cached
|
||||
- **Configs**: Configuration files unchanged
|
||||
|
||||
To remove volumes, use `docker compose -f .docker/compose/docker-compose.dev.yml down -v`
|
||||
|
||||
## Shutdown Order
|
||||
|
||||
Docker Compose stops services in reverse dependency order:
|
||||
1. Reverse proxy stops first
|
||||
2. Application services stop next
|
||||
3. Security services stop
|
||||
4. Database services stop last
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Stop Development Environment
|
||||
|
||||
```bash
|
||||
# Stop all development services
|
||||
.github/skills/docker-stop-dev-scripts/run.sh
|
||||
|
||||
# Verify services are stopped
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml ps
|
||||
```
|
||||
|
||||
### Example 2: Stop and Remove Volumes
|
||||
|
||||
```bash
|
||||
# Stop services and remove data volumes
|
||||
docker compose -f .docker/compose/docker-compose.dev.yml down -v
|
||||
```
|
||||
|
||||
### Example 3: Stop and Verify Cleanup
|
||||
|
||||
```bash
|
||||
# Stop development environment
|
||||
.github/skills/docker-stop-dev-scripts/run.sh
|
||||
|
||||
# Verify no containers running
|
||||
docker ps --filter "name=charon-dev"
|
||||
|
||||
# Verify network removed
|
||||
docker network ls | grep charon-dev
|
||||
```
|
||||
|
||||
## Graceful Shutdown
|
||||
|
||||
The `down` command:
|
||||
1. Sends `SIGTERM` to each container
|
||||
2. Waits for graceful shutdown (default: 10 seconds)
|
||||
3. Sends `SIGKILL` if timeout exceeded
|
||||
4. Removes stopped containers
|
||||
5. Removes default network
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Switching between development and production modes
|
||||
- Freeing system resources (CPU, memory)
|
||||
- Preparing for system shutdown/restart
|
||||
- Resetting environment for troubleshooting
|
||||
- Applying Docker Compose configuration changes
|
||||
- Before database recovery operations
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
### Container Already Stopped
|
||||
```
|
||||
Warning: Container already stopped
|
||||
```
|
||||
No action needed - idempotent operation
|
||||
|
||||
### Volume in Use
|
||||
```
|
||||
Error: volume is in use
|
||||
```
|
||||
Solution: Check for other containers using the volume
|
||||
|
||||
### Permission Denied
|
||||
```
|
||||
Error: permission denied
|
||||
```
|
||||
Solution: Add user to docker group or use sudo
|
||||
|
||||
## Post-Shutdown Verification
|
||||
|
||||
After stopping, verify:
|
||||
|
||||
1. **No Running Containers**:
|
||||
```bash
|
||||
docker ps --filter "name=charon-dev"
|
||||
```
|
||||
|
||||
2. **Network Removed**:
|
||||
```bash
|
||||
docker network ls | grep charon-dev
|
||||
```
|
||||
|
||||
3. **Volumes Still Exist** (if data preservation intended):
|
||||
```bash
|
||||
docker volume ls | grep charon
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [docker-start-dev](./docker-start-dev.SKILL.md) - Start development environment
|
||||
- [docker-prune](./docker-prune.SKILL.md) - Clean up Docker resources
|
||||
- [utility-db-recovery](./utility-db-recovery.SKILL.md) - Database recovery
|
||||
|
||||
## Notes
|
||||
|
||||
- **Idempotent**: Safe to run multiple times (no error if already stopped)
|
||||
- **Data Preservation**: Volumes are NOT removed by default
|
||||
- **Fast Execution**: Typically completes in seconds
|
||||
- **No Network Required**: Local operation only
|
||||
- **Not CI/CD Safe**: Intended for local development only
|
||||
- **Graceful Shutdown**: Allows containers to clean up resources
|
||||
|
||||
## Complete Cleanup
|
||||
|
||||
For complete environment reset:
|
||||
|
||||
```bash
|
||||
# Stop and remove containers, networks
|
||||
.github/skills/docker-stop-dev-scripts/run.sh
|
||||
|
||||
# Remove volumes (WARNING: deletes data)
|
||||
docker volume rm $(docker volume ls -q --filter "name=charon")
|
||||
|
||||
# Remove images (optional)
|
||||
docker rmi $(docker images -q "*charon*")
|
||||
|
||||
# Clean up dangling resources
|
||||
.github/skills/docker-prune-scripts/run.sh
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container Won't Stop
|
||||
|
||||
1. Check container logs: `docker compose logs app`
|
||||
2. Force removal: `docker compose kill`
|
||||
3. Manual cleanup: `docker rm -f container_name`
|
||||
|
||||
### Volume Still in Use
|
||||
|
||||
1. List processes: `docker ps -a`
|
||||
2. Check volume usage: `docker volume inspect volume_name`
|
||||
3. Force volume removal: `docker volume rm -f volume_name`
|
||||
|
||||
### Network Can't Be Removed
|
||||
|
||||
1. Check connected containers: `docker network inspect network_name`
|
||||
2. Disconnect containers: `docker network disconnect network_name container_name`
|
||||
3. Retry removal: `docker network rm network_name`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Compose File**: `.docker/compose/docker-compose.dev.yml`
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Integration Test All - Wrapper Script
|
||||
# Executes the comprehensive integration test suite
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Delegate to the existing integration test script
|
||||
exec "${PROJECT_ROOT}/scripts/integration-test.sh" "$@"
|
||||
@@ -0,0 +1,220 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "integration-test-all"
|
||||
version: "1.0.0"
|
||||
description: "Run all integration tests including WAF, CrowdSec, Cerberus, and rate limiting"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "integration"
|
||||
- "testing"
|
||||
- "docker"
|
||||
- "end-to-end"
|
||||
- "security"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "docker-compose"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
- name: "curl"
|
||||
version: ">=7.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "DOCKER_BUILDKIT"
|
||||
description: "Enable Docker BuildKit for faster builds"
|
||||
default: "1"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "Aggregated test results from all integration tests"
|
||||
metadata:
|
||||
category: "integration-test"
|
||||
subcategory: "all"
|
||||
execution_time: "long"
|
||||
risk_level: "medium"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Integration Test All
|
||||
|
||||
## Overview
|
||||
|
||||
Executes the complete integration test suite for the Charon project. This skill runs all integration tests including WAF functionality (Coraza), CrowdSec bouncer integration, Cerberus backend protection, and rate limiting. It validates the entire security stack in a containerized environment.
|
||||
|
||||
This is the comprehensive test suite that ensures all components work together correctly before deployment.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- Docker Compose 2.0 or higher
|
||||
- curl 7.0 or higher for API testing
|
||||
- At least 4GB of available RAM for containers
|
||||
- Network access for pulling container images
|
||||
- Docker daemon running with sufficient disk space
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run all integration tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
Run with detailed output:
|
||||
|
||||
```bash
|
||||
VERBOSE=1 .github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions workflows:
|
||||
|
||||
```yaml
|
||||
- name: Run All Integration Tests
|
||||
run: .github/skills/scripts/skill-runner.sh integration-test-all
|
||||
timeout-minutes: 20
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| DOCKER_BUILDKIT | No | 1 | Enable BuildKit for faster builds |
|
||||
| SKIP_CLEANUP | No | false | Skip container cleanup after tests |
|
||||
| TEST_TIMEOUT | No | 300 | Timeout in seconds for each test |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All integration tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **2**: Docker environment setup failed
|
||||
- **3**: Container startup timeout
|
||||
- **4**: Network connectivity issues
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
=== Running Integration Test Suite ===
|
||||
✓ Coraza WAF Integration Tests
|
||||
✓ CrowdSec Bouncer Integration Tests
|
||||
✓ CrowdSec Decision API Tests
|
||||
✓ Cerberus Authentication Tests
|
||||
✓ Rate Limiting Tests
|
||||
|
||||
All integration tests passed!
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
### Example 2: Verbose with Custom Timeout
|
||||
|
||||
```bash
|
||||
VERBOSE=1 TEST_TIMEOUT=600 .github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
### Example 3: Skip Cleanup for Debugging
|
||||
|
||||
```bash
|
||||
SKIP_CLEANUP=true .github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
### Example 4: CI/CD Pipeline
|
||||
|
||||
```bash
|
||||
# Run with specific Docker configuration
|
||||
DOCKER_BUILDKIT=1 .github/skills/scripts/skill-runner.sh integration-test-all
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
This skill executes the following test suites:
|
||||
|
||||
1. **Coraza WAF Tests**: SQL injection, XSS, path traversal detection
|
||||
2. **CrowdSec Bouncer Tests**: IP blocking, decision synchronization
|
||||
3. **CrowdSec Decision Tests**: Decision creation, removal, persistence
|
||||
4. **Cerberus Tests**: Authentication, authorization, token management
|
||||
5. **Rate Limit Tests**: Request throttling, burst handling
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Cannot connect to Docker daemon
|
||||
**Solution**: Ensure Docker is running: `sudo systemctl start docker`
|
||||
|
||||
#### Error: Port already in use
|
||||
**Solution**: Stop conflicting services or run cleanup: `docker compose down`
|
||||
|
||||
#### Error: Container startup timeout
|
||||
**Solution**: Check Docker logs: `docker compose logs`
|
||||
|
||||
#### Error: Network connectivity issues
|
||||
**Solution**: Verify network configuration: `docker network ls`
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- **Slow execution**: Check available system resources
|
||||
- **Random failures**: Increase TEST_TIMEOUT
|
||||
- **Cleanup issues**: Manually run `docker compose down -v`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [integration-test-coraza](./integration-test-coraza.SKILL.md) - Coraza WAF tests only
|
||||
- [integration-test-crowdsec](./integration-test-crowdsec.SKILL.md) - CrowdSec tests only
|
||||
- [integration-test-crowdsec-decisions](./integration-test-crowdsec-decisions.SKILL.md) - Decision API tests
|
||||
- [integration-test-crowdsec-startup](./integration-test-crowdsec-startup.SKILL.md) - Startup tests
|
||||
- [docker-verify-crowdsec-config](./docker-verify-crowdsec-config.SKILL.md) - Config validation
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Long execution (10-15 minutes typical)
|
||||
- **Resource Intensive**: Requires significant CPU and memory
|
||||
- **Network Required**: Pulls Docker images and tests network functionality
|
||||
- **Idempotency**: Safe to run multiple times (cleanup between runs)
|
||||
- **Cleanup**: Automatically cleans up containers unless SKIP_CLEANUP=true
|
||||
- **CI/CD**: Designed for automated pipelines with proper timeout configuration
|
||||
- **Isolation**: Tests run in isolated Docker networks
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/integration-test.sh`
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Integration Test Coraza - Wrapper Script
|
||||
# Tests Coraza WAF integration
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Delegate to the existing Coraza integration test script
|
||||
exec "${PROJECT_ROOT}/scripts/coraza_integration.sh" "$@"
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "integration-test-coraza"
|
||||
version: "1.0.0"
|
||||
description: "Test Coraza WAF integration with OWASP Core Rule Set protection"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "integration"
|
||||
- "waf"
|
||||
- "security"
|
||||
- "coraza"
|
||||
- "owasp"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "curl"
|
||||
version: ">=7.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "WAF_ENABLED"
|
||||
description: "Enable WAF protection"
|
||||
default: "true"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "WAF test results including blocked attacks"
|
||||
metadata:
|
||||
category: "integration-test"
|
||||
subcategory: "waf"
|
||||
execution_time: "medium"
|
||||
risk_level: "medium"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Integration Test Coraza
|
||||
|
||||
## Overview
|
||||
|
||||
Tests the Coraza Web Application Firewall (WAF) integration with OWASP Core Rule Set (CRS). This skill validates that the WAF correctly detects and blocks common web attacks including SQL injection, cross-site scripting (XSS), remote code execution (RCE), and path traversal attempts.
|
||||
|
||||
Coraza provides ModSecurity-compatible rule processing with improved performance and modern Go implementation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- curl 7.0 or higher for HTTP testing
|
||||
- Running Charon Docker environment (or automatic startup)
|
||||
- Network access to test endpoints
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run Coraza WAF integration tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
Run with detailed attack payloads and responses:
|
||||
|
||||
```bash
|
||||
VERBOSE=1 .github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions workflows:
|
||||
|
||||
```yaml
|
||||
- name: Test Coraza WAF Integration
|
||||
run: .github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
timeout-minutes: 5
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| WAF_ENABLED | No | true | Enable WAF protection for tests |
|
||||
| TEST_HOST | No | localhost:8080 | Target host for WAF tests |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All WAF tests passed (attacks blocked correctly)
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more attacks were not blocked
|
||||
- **2**: Docker environment setup failed
|
||||
- **3**: WAF not responding or misconfigured
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
=== Testing Coraza WAF Integration ===
|
||||
✓ SQL Injection: Blocked (403 Forbidden)
|
||||
✓ XSS Attack: Blocked (403 Forbidden)
|
||||
✓ Path Traversal: Blocked (403 Forbidden)
|
||||
✓ RCE Attempt: Blocked (403 Forbidden)
|
||||
✓ Legitimate Request: Allowed (200 OK)
|
||||
|
||||
All Coraza WAF tests passed!
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
This skill validates protection against:
|
||||
|
||||
1. **SQL Injection**: `' OR '1'='1`, `UNION SELECT`, `'; DROP TABLE`
|
||||
2. **Cross-Site Scripting (XSS)**: `<script>alert('XSS')</script>`, `javascript:alert(1)`
|
||||
3. **Path Traversal**: `../../etc/passwd`, `....//....//etc/passwd`
|
||||
4. **Remote Code Execution**: `<?php system($_GET['cmd']); ?>`, `eval()`
|
||||
5. **Legitimate Traffic**: Ensures normal requests are not blocked
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
```
|
||||
|
||||
### Example 2: Verbose with Custom Host
|
||||
|
||||
```bash
|
||||
TEST_HOST=production.example.com VERBOSE=1 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
```
|
||||
|
||||
### Example 3: Disable WAF for Comparison
|
||||
|
||||
```bash
|
||||
WAF_ENABLED=false .github/skills/scripts/skill-runner.sh integration-test-coraza
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: WAF not responding
|
||||
**Solution**: Verify Docker containers are running: `docker ps | grep coraza`
|
||||
|
||||
#### Error: Attacks not blocked (false negatives)
|
||||
**Solution**: Check WAF configuration in `configs/coraza/` and rule sets
|
||||
|
||||
#### Error: Legitimate requests blocked (false positives)
|
||||
**Solution**: Review WAF logs and adjust rule sensitivity
|
||||
|
||||
#### Error: Connection refused
|
||||
**Solution**: Ensure application is accessible: `curl http://localhost:8080/health`
|
||||
|
||||
### Debugging
|
||||
|
||||
- **WAF Logs**: `docker logs $(docker ps -q -f name=coraza)`
|
||||
- **Rule Debugging**: Set `SecRuleEngine DetectionOnly` in config
|
||||
- **Test Individual Payloads**: Use curl with specific attack strings
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [integration-test-all](./integration-test-all.SKILL.md) - Complete integration suite
|
||||
- [integration-test-waf](./integration-test-waf.SKILL.md) - General WAF tests
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Vulnerability scanning
|
||||
|
||||
## Notes
|
||||
|
||||
- **OWASP CRS**: Uses Core Rule Set v4.0+ for comprehensive protection
|
||||
- **Execution Time**: Medium execution (3-5 minutes)
|
||||
- **False Positives**: Tuning required for production workloads
|
||||
- **Performance**: Minimal latency impact (<5ms per request)
|
||||
- **Compliance**: Helps meet OWASP Top 10 and PCI DSS requirements
|
||||
- **Logging**: All blocked requests are logged for analysis
|
||||
- **Rule Updates**: Regularly update CRS for latest threat intelligence
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/coraza_integration.sh`
|
||||
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Integration Test CrowdSec Decisions - Wrapper Script
|
||||
# Tests CrowdSec decision API functionality
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Delegate to the existing CrowdSec decision integration test script
|
||||
exec "${PROJECT_ROOT}/scripts/crowdsec_decision_integration.sh" "$@"
|
||||
@@ -0,0 +1,252 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "integration-test-crowdsec-decisions"
|
||||
version: "1.0.0"
|
||||
description: "Test CrowdSec decision API for creating, retrieving, and removing IP blocks"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "integration"
|
||||
- "crowdsec"
|
||||
- "decisions"
|
||||
- "api"
|
||||
- "blocking"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "curl"
|
||||
version: ">=7.0"
|
||||
optional: false
|
||||
- name: "jq"
|
||||
version: ">=1.6"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "CROWDSEC_API_KEY"
|
||||
description: "CrowdSec API key for decision management"
|
||||
default: "auto-generated"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "Decision API test results"
|
||||
metadata:
|
||||
category: "integration-test"
|
||||
subcategory: "api"
|
||||
execution_time: "medium"
|
||||
risk_level: "medium"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Integration Test CrowdSec Decisions
|
||||
|
||||
## Overview
|
||||
|
||||
Tests the CrowdSec decision API functionality for managing IP block decisions. This skill validates decision creation, retrieval, persistence, expiration, and removal through the CrowdSec Local API (LAPI). It ensures the decision lifecycle works correctly and that bouncers receive updates in real-time.
|
||||
|
||||
Decisions are the core mechanism CrowdSec uses to communicate threats between detectors and enforcers.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- curl 7.0 or higher for API testing
|
||||
- jq 1.6 or higher for JSON parsing
|
||||
- Running CrowdSec LAPI container
|
||||
- Valid CrowdSec API credentials
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run CrowdSec decision API tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
Run with detailed API request/response logging:
|
||||
|
||||
```bash
|
||||
VERBOSE=1 .github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions workflows:
|
||||
|
||||
```yaml
|
||||
- name: Test CrowdSec Decision API
|
||||
run: .github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
timeout-minutes: 5
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| CROWDSEC_API_KEY | No | auto | API key for LAPI access |
|
||||
| CROWDSEC_LAPI_URL | No | http://crowdsec:8080 | CrowdSec LAPI endpoint |
|
||||
| TEST_IP | No | 192.0.2.1 | Test IP address for decisions |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All decision API tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **2**: LAPI not accessible
|
||||
- **3**: Authentication failed
|
||||
- **4**: Decision creation/deletion failed
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
=== Testing CrowdSec Decision API ===
|
||||
✓ Create Decision: IP 192.0.2.1 blocked for 4h
|
||||
✓ Retrieve Decisions: 1 active decision found
|
||||
✓ Decision Details: Correct scope, value, duration
|
||||
✓ Decision Persistence: Survives bouncer restart
|
||||
✓ Decision Expiration: Expires after duration
|
||||
✓ Remove Decision: Successfully deleted
|
||||
✓ Decision Cleanup: No orphaned decisions
|
||||
|
||||
All CrowdSec decision API tests passed!
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
This skill validates:
|
||||
|
||||
1. **Decision Creation**:
|
||||
- Create IP ban decision via API
|
||||
- Create range ban decision
|
||||
- Create captcha decision
|
||||
- Set custom duration and reason
|
||||
|
||||
2. **Decision Retrieval**:
|
||||
- List all active decisions
|
||||
- Filter by scope (ip, range, country)
|
||||
- Filter by value (specific IP)
|
||||
- Pagination support
|
||||
|
||||
3. **Decision Persistence**:
|
||||
- Decisions survive LAPI restart
|
||||
- Decisions sync to bouncers
|
||||
- Database integrity
|
||||
|
||||
4. **Decision Lifecycle**:
|
||||
- Expiration after duration
|
||||
- Manual removal via API
|
||||
- Automatic cleanup of expired decisions
|
||||
|
||||
5. **Decision Synchronization**:
|
||||
- Bouncer receives new decisions
|
||||
- Bouncer updates on decision changes
|
||||
- Real-time propagation
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
### Example 2: Test Specific IP
|
||||
|
||||
```bash
|
||||
TEST_IP=10.0.0.1 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
### Example 3: Custom LAPI URL
|
||||
|
||||
```bash
|
||||
CROWDSEC_LAPI_URL=https://crowdsec-lapi.example.com \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
### Example 4: Verbose with API Key
|
||||
|
||||
```bash
|
||||
CROWDSEC_API_KEY=my-api-key VERBOSE=1 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-decisions
|
||||
```
|
||||
|
||||
## API Endpoints Tested
|
||||
|
||||
- `POST /v1/decisions` - Create new decision
|
||||
- `GET /v1/decisions` - List decisions
|
||||
- `GET /v1/decisions/:id` - Get decision details
|
||||
- `DELETE /v1/decisions/:id` - Remove decision
|
||||
- `GET /v1/decisions/stream` - Bouncer decision stream
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: LAPI not responding
|
||||
**Solution**: Check LAPI container: `docker ps | grep crowdsec`
|
||||
|
||||
#### Error: Authentication failed
|
||||
**Solution**: Verify API key: `docker exec crowdsec cscli machines list`
|
||||
|
||||
#### Error: Decision not created
|
||||
**Solution**: Check LAPI logs for validation errors
|
||||
|
||||
#### Error: Decision not found after creation
|
||||
**Solution**: Verify database connectivity and permissions
|
||||
|
||||
### Debugging
|
||||
|
||||
- **LAPI Logs**: `docker logs $(docker ps -q -f name=crowdsec)`
|
||||
- **Database**: `docker exec crowdsec cscli decisions list`
|
||||
- **API Testing**: `curl -H "X-Api-Key: $KEY" http://localhost:8080/v1/decisions`
|
||||
- **Decision Details**: `docker exec crowdsec cscli decisions list -o json | jq`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [integration-test-crowdsec](./integration-test-crowdsec.SKILL.md) - Main bouncer tests
|
||||
- [integration-test-crowdsec-startup](./integration-test-crowdsec-startup.SKILL.md) - Startup tests
|
||||
- [integration-test-all](./integration-test-all.SKILL.md) - Complete suite
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Medium execution (3-5 minutes)
|
||||
- **Decision Types**: Supports ban, captcha, and throttle decisions
|
||||
- **Scopes**: IP, range, country, AS, user
|
||||
- **Duration**: From seconds to permanent bans
|
||||
- **API Version**: Tests LAPI v1 endpoints
|
||||
- **Cleanup**: All test decisions are removed after execution
|
||||
- **Idempotency**: Safe to run multiple times
|
||||
- **Isolation**: Uses test IP ranges (RFC 5737)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/crowdsec_decision_integration.sh`
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Integration Test CrowdSec - Wrapper Script
|
||||
# Tests CrowdSec bouncer integration
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Delegate to the existing CrowdSec integration test script
|
||||
exec "${PROJECT_ROOT}/scripts/crowdsec_integration.sh" "$@"
|
||||
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Integration Test CrowdSec Startup - Wrapper Script
|
||||
# Tests CrowdSec startup sequence and initialization
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Delegate to the existing CrowdSec startup test script
|
||||
exec "${PROJECT_ROOT}/scripts/crowdsec_startup_test.sh" "$@"
|
||||
@@ -0,0 +1,275 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "integration-test-crowdsec-startup"
|
||||
version: "1.0.0"
|
||||
description: "Test CrowdSec startup sequence, initialization, and error handling"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "integration"
|
||||
- "crowdsec"
|
||||
- "startup"
|
||||
- "initialization"
|
||||
- "resilience"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "curl"
|
||||
version: ">=7.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "STARTUP_TIMEOUT"
|
||||
description: "Maximum wait time for startup in seconds"
|
||||
default: "60"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "Startup test results"
|
||||
metadata:
|
||||
category: "integration-test"
|
||||
subcategory: "startup"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Integration Test CrowdSec Startup
|
||||
|
||||
## Overview
|
||||
|
||||
Tests the CrowdSec startup sequence and initialization process. This skill validates that CrowdSec components (LAPI, bouncer) start correctly, handle initialization errors gracefully, and recover from common startup failures. It ensures the system is resilient to network issues, configuration problems, and timing-related edge cases.
|
||||
|
||||
Proper startup behavior is critical for production deployments and automated container orchestration.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- curl 7.0 or higher for health checks
|
||||
- Docker Compose for orchestration
|
||||
- Network connectivity for pulling images
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run CrowdSec startup tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
Run with detailed startup logging:
|
||||
|
||||
```bash
|
||||
VERBOSE=1 .github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### Custom Timeout
|
||||
|
||||
Run with extended startup timeout:
|
||||
|
||||
```bash
|
||||
STARTUP_TIMEOUT=120 .github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions workflows:
|
||||
|
||||
```yaml
|
||||
- name: Test CrowdSec Startup
|
||||
run: .github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
timeout-minutes: 5
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| STARTUP_TIMEOUT | No | 60 | Maximum wait for startup (seconds) |
|
||||
| SKIP_CLEANUP | No | false | Skip container cleanup after tests |
|
||||
| CROWDSEC_VERSION | No | latest | CrowdSec image version to test |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All startup tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **2**: Startup timeout exceeded
|
||||
- **3**: Configuration errors detected
|
||||
- **4**: Health check failed
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
=== Testing CrowdSec Startup Sequence ===
|
||||
✓ LAPI Initialization: Ready in 8s
|
||||
✓ Database Migration: Successful
|
||||
✓ Bouncer Registration: Successful
|
||||
✓ Configuration Validation: No errors
|
||||
✓ Health Check: All services healthy
|
||||
✓ Graceful Shutdown: Clean exit
|
||||
✓ Restart Resilience: Fast recovery
|
||||
|
||||
All CrowdSec startup tests passed!
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
This skill validates:
|
||||
|
||||
1. **Clean Startup**:
|
||||
- LAPI starts and becomes ready
|
||||
- Database schema migration
|
||||
- Configuration loading
|
||||
- API endpoint availability
|
||||
|
||||
2. **Bouncer Initialization**:
|
||||
- Bouncer registers with LAPI
|
||||
- API key generation/validation
|
||||
- Decision cache initialization
|
||||
- First sync successful
|
||||
|
||||
3. **Error Handling**:
|
||||
- Invalid configuration detection
|
||||
- Missing database handling
|
||||
- Network timeout recovery
|
||||
- Retry mechanisms
|
||||
|
||||
4. **Edge Cases**:
|
||||
- LAPI not ready on first attempt
|
||||
- Race conditions in initialization
|
||||
- Concurrent bouncer registrations
|
||||
- Configuration hot-reload
|
||||
|
||||
5. **Resilience**:
|
||||
- Graceful shutdown
|
||||
- Fast restart (warm start)
|
||||
- State persistence
|
||||
- No resource leaks
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### Example 2: Extended Timeout
|
||||
|
||||
```bash
|
||||
STARTUP_TIMEOUT=180 VERBOSE=1 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### Example 3: Test Specific Version
|
||||
|
||||
```bash
|
||||
CROWDSEC_VERSION=v1.5.0 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
### Example 4: Keep Containers for Debugging
|
||||
|
||||
```bash
|
||||
SKIP_CLEANUP=true \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup
|
||||
```
|
||||
|
||||
## Startup Sequence Verified
|
||||
|
||||
1. **Phase 1: Container Start** (0-5s)
|
||||
- Container created and started
|
||||
- Entrypoint script execution
|
||||
- Environment variable processing
|
||||
|
||||
2. **Phase 2: LAPI Initialization** (5-15s)
|
||||
- Database connection established
|
||||
- Schema migration/validation
|
||||
- Configuration parsing
|
||||
- API server binding
|
||||
|
||||
3. **Phase 3: Bouncer Registration** (15-25s)
|
||||
- Bouncer discovers LAPI
|
||||
- API key generated/validated
|
||||
- Initial decision sync
|
||||
- Cache population
|
||||
|
||||
4. **Phase 4: Ready State** (25-30s)
|
||||
- Health check endpoint responds
|
||||
- All components initialized
|
||||
- Ready to process requests
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Startup timeout exceeded
|
||||
**Solution**: Increase STARTUP_TIMEOUT or check container logs for hangs
|
||||
|
||||
#### Error: Database connection failed
|
||||
**Solution**: Verify database container is running and accessible
|
||||
|
||||
#### Error: Configuration validation failed
|
||||
**Solution**: Check CrowdSec config files for syntax errors
|
||||
|
||||
#### Error: Port already in use
|
||||
**Solution**: Stop conflicting services or change port configuration
|
||||
|
||||
### Debugging
|
||||
|
||||
- **LAPI Logs**: `docker logs $(docker ps -q -f name=crowdsec) -f`
|
||||
- **Bouncer Logs**: `docker logs $(docker ps -q -f name=charon-app) | grep crowdsec`
|
||||
- **Health Check**: `curl http://localhost:8080/health`
|
||||
- **Database**: `docker exec crowdsec cscli machines list`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [integration-test-crowdsec](./integration-test-crowdsec.SKILL.md) - Main bouncer tests
|
||||
- [integration-test-crowdsec-decisions](./integration-test-crowdsec-decisions.SKILL.md) - Decision tests
|
||||
- [docker-verify-crowdsec-config](./docker-verify-crowdsec-config.SKILL.md) - Config validation
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Medium execution (3-5 minutes)
|
||||
- **Typical Startup**: 20-30 seconds for clean start
|
||||
- **Warm Start**: 5-10 seconds after restart
|
||||
- **Timeout Buffer**: Default timeout includes safety margin
|
||||
- **Container Orchestration**: Tests applicable to Kubernetes/Docker Swarm
|
||||
- **Production Ready**: Validates production deployment scenarios
|
||||
- **Cleanup**: Automatically removes test containers unless SKIP_CLEANUP=true
|
||||
- **Idempotency**: Safe to run multiple times consecutively
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/crowdsec_startup_test.sh`
|
||||
@@ -0,0 +1,220 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "integration-test-crowdsec"
|
||||
version: "1.0.0"
|
||||
description: "Test CrowdSec bouncer integration and IP blocking functionality"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "integration"
|
||||
- "security"
|
||||
- "crowdsec"
|
||||
- "ip-blocking"
|
||||
- "bouncer"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "curl"
|
||||
version: ">=7.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "CROWDSEC_API_KEY"
|
||||
description: "CrowdSec API key for bouncer authentication"
|
||||
default: "auto-generated"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "CrowdSec integration test results"
|
||||
metadata:
|
||||
category: "integration-test"
|
||||
subcategory: "security"
|
||||
execution_time: "medium"
|
||||
risk_level: "medium"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Integration Test CrowdSec
|
||||
|
||||
## Overview
|
||||
|
||||
Tests the CrowdSec bouncer integration for IP-based threat detection and blocking. This skill validates that the CrowdSec bouncer correctly synchronizes with the CrowdSec Local API (LAPI), retrieves and applies IP block decisions, and enforces security policies.
|
||||
|
||||
CrowdSec provides collaborative security with real-time threat intelligence sharing across the community.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- curl 7.0 or higher for API testing
|
||||
- Running CrowdSec LAPI container
|
||||
- Running Charon application with CrowdSec bouncer enabled
|
||||
- Network access between bouncer and LAPI
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run CrowdSec bouncer integration tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
Run with detailed API interactions:
|
||||
|
||||
```bash
|
||||
VERBOSE=1 .github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions workflows:
|
||||
|
||||
```yaml
|
||||
- name: Test CrowdSec Integration
|
||||
run: .github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
timeout-minutes: 7
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| CROWDSEC_API_KEY | No | auto | Bouncer API key (auto-generated if not set) |
|
||||
| CROWDSEC_LAPI_URL | No | http://crowdsec:8080 | CrowdSec LAPI endpoint |
|
||||
| BOUNCER_SYNC_INTERVAL | No | 60 | Decision sync interval in seconds |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All CrowdSec integration tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **2**: CrowdSec LAPI not accessible
|
||||
- **3**: Bouncer authentication failed
|
||||
- **4**: Decision synchronization failed
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
=== Testing CrowdSec Bouncer Integration ===
|
||||
✓ LAPI Connection: Successful
|
||||
✓ Bouncer Authentication: Valid API Key
|
||||
✓ Decision Retrieval: 5 active decisions
|
||||
✓ IP Blocking: Blocked malicious IP (403 Forbidden)
|
||||
✓ Legitimate IP: Allowed (200 OK)
|
||||
✓ Decision Synchronization: Every 60s
|
||||
|
||||
All CrowdSec integration tests passed!
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
This skill validates:
|
||||
|
||||
1. **LAPI Connectivity**: Bouncer can reach CrowdSec Local API
|
||||
2. **Authentication**: Valid API key and successful bouncer registration
|
||||
3. **Decision Retrieval**: Fetching active IP block decisions
|
||||
4. **IP Blocking**: Correctly blocking malicious IPs
|
||||
5. **Legitimate Traffic**: Allowing non-blocked IPs
|
||||
6. **Decision Synchronization**: Regular updates from LAPI
|
||||
7. **Graceful Degradation**: Handling LAPI downtime
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
### Example 2: Custom API Key
|
||||
|
||||
```bash
|
||||
CROWDSEC_API_KEY=my-bouncer-key \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
### Example 3: Custom LAPI URL
|
||||
|
||||
```bash
|
||||
CROWDSEC_LAPI_URL=http://crowdsec-lapi:8080 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
### Example 4: Fast Sync Interval
|
||||
|
||||
```bash
|
||||
BOUNCER_SYNC_INTERVAL=30 VERBOSE=1 \
|
||||
.github/skills/scripts/skill-runner.sh integration-test-crowdsec
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Cannot connect to LAPI
|
||||
**Solution**: Verify LAPI container is running: `docker ps | grep crowdsec`
|
||||
|
||||
#### Error: Authentication failed
|
||||
**Solution**: Check API key is valid: `docker exec crowdsec cscli bouncers list`
|
||||
|
||||
#### Error: No decisions retrieved
|
||||
**Solution**: Create test decisions: `docker exec crowdsec cscli decisions add --ip 1.2.3.4`
|
||||
|
||||
#### Error: Blocking not working
|
||||
**Solution**: Check bouncer logs: `docker logs charon-app | grep crowdsec`
|
||||
|
||||
### Debugging
|
||||
|
||||
- **LAPI Logs**: `docker logs $(docker ps -q -f name=crowdsec)`
|
||||
- **Bouncer Status**: Check application logs for sync errors
|
||||
- **Decision List**: `docker exec crowdsec cscli decisions list`
|
||||
- **Test Block**: `curl -H "X-Forwarded-For: 1.2.3.4" http://localhost:8080/`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [integration-test-crowdsec-decisions](./integration-test-crowdsec-decisions.SKILL.md) - Decision API tests
|
||||
- [integration-test-crowdsec-startup](./integration-test-crowdsec-startup.SKILL.md) - Startup tests
|
||||
- [integration-test-all](./integration-test-all.SKILL.md) - Complete test suite
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Medium execution (4-6 minutes)
|
||||
- **Community Intelligence**: Benefits from CrowdSec's global threat network
|
||||
- **Performance**: Minimal latency with in-memory decision caching
|
||||
- **Scalability**: Tested with thousands of concurrent decisions
|
||||
- **Resilience**: Continues working if LAPI is temporarily unavailable
|
||||
- **Observability**: Full metrics exposed for Prometheus/Grafana
|
||||
- **Compliance**: Supports GDPR-compliant threat intelligence
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/crowdsec_integration.sh`
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# QA Pre-commit All - Execution Script
|
||||
#
|
||||
# This script runs all pre-commit hooks for comprehensive code quality validation.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_python_environment "3.8" || error_exit "Python 3.8+ is required"
|
||||
|
||||
# Check for virtual environment
|
||||
if [[ -z "${VIRTUAL_ENV:-}" ]]; then
|
||||
log_warning "Virtual environment not activated, attempting to activate .venv"
|
||||
if [[ -f "${PROJECT_ROOT}/.venv/bin/activate" ]]; then
|
||||
# shellcheck source=/dev/null
|
||||
source "${PROJECT_ROOT}/.venv/bin/activate"
|
||||
log_info "Activated virtual environment: ${VIRTUAL_ENV}"
|
||||
else
|
||||
error_exit "Virtual environment not found at ${PROJECT_ROOT}/.venv"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for pre-commit
|
||||
if ! command -v pre-commit &> /dev/null; then
|
||||
error_exit "pre-commit not found. Install with: pip install pre-commit"
|
||||
fi
|
||||
|
||||
# Parse arguments
|
||||
FILES_MODE="${1:---all-files}"
|
||||
|
||||
# Validate files mode
|
||||
case "${FILES_MODE}" in
|
||||
--all-files|staged)
|
||||
;;
|
||||
*)
|
||||
# If not a recognized mode, treat as a specific hook ID
|
||||
HOOK_ID="${FILES_MODE}"
|
||||
FILES_MODE="--all-files"
|
||||
log_info "Running specific hook: ${HOOK_ID}"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Change to project root
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Execute pre-commit
|
||||
log_step "VALIDATION" "Running pre-commit hooks"
|
||||
log_info "Files mode: ${FILES_MODE}"
|
||||
|
||||
if [[ -n "${SKIP:-}" ]]; then
|
||||
log_info "Skipping hooks: ${SKIP}"
|
||||
fi
|
||||
|
||||
# Build pre-commit command
|
||||
PRE_COMMIT_CMD="pre-commit run"
|
||||
|
||||
# Handle files mode
|
||||
if [[ "${FILES_MODE}" == "staged" ]]; then
|
||||
# Run on staged files only (no flag needed, this is default for 'pre-commit run')
|
||||
log_info "Running on staged files only"
|
||||
else
|
||||
PRE_COMMIT_CMD="${PRE_COMMIT_CMD} --all-files"
|
||||
fi
|
||||
|
||||
# Add specific hook if provided
|
||||
if [[ -n "${HOOK_ID:-}" ]]; then
|
||||
PRE_COMMIT_CMD="${PRE_COMMIT_CMD} ${HOOK_ID}"
|
||||
fi
|
||||
|
||||
# Execute the validation
|
||||
log_info "Executing: ${PRE_COMMIT_CMD}"
|
||||
|
||||
if eval "${PRE_COMMIT_CMD}"; then
|
||||
log_success "All pre-commit hooks passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "One or more pre-commit hooks failed (exit code: ${exit_code})"
|
||||
log_info "Review the output above for details"
|
||||
log_info "Some hooks can auto-fix issues - review and commit changes if appropriate"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,353 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "qa-precommit-all"
|
||||
version: "1.0.0"
|
||||
description: "Run all pre-commit hooks for comprehensive code quality validation"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "qa"
|
||||
- "quality"
|
||||
- "pre-commit"
|
||||
- "linting"
|
||||
- "validation"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "python3"
|
||||
version: ">=3.8"
|
||||
optional: false
|
||||
- name: "pre-commit"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "PRE_COMMIT_HOME"
|
||||
description: "Pre-commit cache directory"
|
||||
default: "~/.cache/pre-commit"
|
||||
required: false
|
||||
- name: "SKIP"
|
||||
description: "Comma-separated list of hook IDs to skip"
|
||||
default: ""
|
||||
required: false
|
||||
parameters:
|
||||
- name: "files"
|
||||
type: "string"
|
||||
description: "Specific files to check (default: all staged files)"
|
||||
default: "--all-files"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "validation_report"
|
||||
type: "stdout"
|
||||
description: "Results of all pre-commit hook executions"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if all hooks pass, non-zero if any fail"
|
||||
metadata:
|
||||
category: "qa"
|
||||
subcategory: "quality"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# QA Pre-commit All
|
||||
|
||||
## Overview
|
||||
|
||||
Executes all configured pre-commit hooks to validate code quality, formatting, security, and best practices across the entire codebase. This skill runs checks for Python, Go, JavaScript/TypeScript, Markdown, YAML, and more.
|
||||
|
||||
This skill is designed for CI/CD pipelines and local quality validation before committing code.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.8 or higher installed and in PATH
|
||||
- Python virtual environment activated (`.venv`)
|
||||
- Pre-commit installed in virtual environment: `pip install pre-commit`
|
||||
- Pre-commit hooks installed: `pre-commit install`
|
||||
- All language-specific tools installed (Go, Node.js, etc.)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run all hooks on all files:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
### Staged Files Only
|
||||
|
||||
Run hooks on staged files only (faster):
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all staged
|
||||
```
|
||||
|
||||
### Specific Hook
|
||||
|
||||
Run only a specific hook by ID:
|
||||
|
||||
```bash
|
||||
SKIP="" .github/skills/scripts/skill-runner.sh qa-precommit-all trailing-whitespace
|
||||
```
|
||||
|
||||
### Skip Specific Hooks
|
||||
|
||||
Skip certain hooks during execution:
|
||||
|
||||
```bash
|
||||
SKIP=prettier,eslint .github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| files | string | No | --all-files | File selection mode (--all-files or staged) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| SKIP | No | "" | Comma-separated hook IDs to skip |
|
||||
| PRE_COMMIT_HOME | No | ~/.cache/pre-commit | Pre-commit cache directory |
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 (all hooks passed)
|
||||
- **Error Exit Codes**: Non-zero (one or more hooks failed)
|
||||
- **Output**: Detailed results from each hook
|
||||
|
||||
## Pre-commit Hooks Included
|
||||
|
||||
The following hooks are configured in `.pre-commit-config.yaml`:
|
||||
|
||||
### General Hooks
|
||||
- **trailing-whitespace**: Remove trailing whitespace
|
||||
- **end-of-file-fixer**: Ensure files end with newline
|
||||
- **check-yaml**: Validate YAML syntax
|
||||
- **check-json**: Validate JSON syntax
|
||||
- **check-merge-conflict**: Detect merge conflict markers
|
||||
- **check-added-large-files**: Prevent committing large files
|
||||
|
||||
### Python Hooks
|
||||
- **black**: Code formatting
|
||||
- **isort**: Import sorting
|
||||
- **flake8**: Linting
|
||||
- **mypy**: Type checking
|
||||
|
||||
### Go Hooks
|
||||
- **gofmt**: Code formatting
|
||||
- **go-vet**: Static analysis
|
||||
- **golangci-lint**: Comprehensive linting
|
||||
|
||||
### JavaScript/TypeScript Hooks
|
||||
- **prettier**: Code formatting
|
||||
- **eslint**: Linting and code quality
|
||||
|
||||
### Markdown Hooks
|
||||
- **markdownlint**: Markdown linting and formatting
|
||||
|
||||
### Security Hooks
|
||||
- **detect-private-key**: Prevent committing private keys
|
||||
- **detect-aws-credentials**: Prevent committing AWS credentials
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Full Quality Check
|
||||
|
||||
```bash
|
||||
# Run all hooks on all files
|
||||
source .venv/bin/activate
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Trim Trailing Whitespace.....................................Passed
|
||||
Fix End of Files.............................................Passed
|
||||
Check Yaml...................................................Passed
|
||||
Check JSON...................................................Passed
|
||||
Check for merge conflicts....................................Passed
|
||||
Check for added large files..................................Passed
|
||||
black........................................................Passed
|
||||
isort........................................................Passed
|
||||
prettier.....................................................Passed
|
||||
eslint.......................................................Passed
|
||||
markdownlint.................................................Passed
|
||||
```
|
||||
|
||||
### Example 2: Quick Staged Files Check
|
||||
|
||||
```bash
|
||||
# Run only on staged files (faster for pre-commit)
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all staged
|
||||
```
|
||||
|
||||
### Example 3: Skip Slow Hooks
|
||||
|
||||
```bash
|
||||
# Skip time-consuming hooks for quick validation
|
||||
SKIP=golangci-lint,mypy .github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
### Example 4: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install pre-commit
|
||||
run: pip install pre-commit
|
||||
|
||||
- name: Run QA Pre-commit Checks
|
||||
run: .github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
### Example 5: Auto-fix Mode
|
||||
|
||||
```bash
|
||||
# Some hooks can auto-fix issues
|
||||
# Run twice: first to fix, second to validate
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all || \
|
||||
.github/skills/scripts/skill-runner.sh qa-precommit-all
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Virtual environment not activated**:
|
||||
```bash
|
||||
Error: pre-commit not found
|
||||
Solution: source .venv/bin/activate
|
||||
```
|
||||
|
||||
**Pre-commit not installed**:
|
||||
```bash
|
||||
Error: pre-commit command not available
|
||||
Solution: pip install pre-commit
|
||||
```
|
||||
|
||||
**Hooks not installed**:
|
||||
```bash
|
||||
Error: Run 'pre-commit install'
|
||||
Solution: pre-commit install
|
||||
```
|
||||
|
||||
**Hook execution failed**:
|
||||
```bash
|
||||
Hook X failed
|
||||
Solution: Review error output and fix reported issues
|
||||
```
|
||||
|
||||
**Language tool missing**:
|
||||
```bash
|
||||
Error: golangci-lint not found
|
||||
Solution: Install required language tools
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: All hooks passed
|
||||
- **1**: One or more hooks failed
|
||||
- **Other**: Hook execution error
|
||||
|
||||
## Hook Fixing Strategies
|
||||
|
||||
### Auto-fixable Issues
|
||||
These hooks automatically fix issues:
|
||||
- `trailing-whitespace`
|
||||
- `end-of-file-fixer`
|
||||
- `black`
|
||||
- `isort`
|
||||
- `prettier`
|
||||
- `gofmt`
|
||||
|
||||
**Workflow**: Run pre-commit, review changes, commit fixed files
|
||||
|
||||
### Manual Fixes Required
|
||||
These hooks only report issues:
|
||||
- `check-yaml`
|
||||
- `check-json`
|
||||
- `flake8`
|
||||
- `eslint`
|
||||
- `markdownlint`
|
||||
- `go-vet`
|
||||
- `golangci-lint`
|
||||
|
||||
**Workflow**: Review errors, manually fix code, re-run pre-commit
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [test-backend-coverage](./test-backend-coverage.SKILL.md) - Backend test coverage
|
||||
- [test-frontend-coverage](./test-frontend-coverage.SKILL.md) - Frontend test coverage
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Security scanning
|
||||
|
||||
## Notes
|
||||
|
||||
- Pre-commit hooks cache their environments for faster execution
|
||||
- First run may be slow while environments are set up
|
||||
- Subsequent runs are much faster (seconds vs minutes)
|
||||
- Hooks run in parallel where possible
|
||||
- Failed hooks stop execution (fail-fast behavior)
|
||||
- Use `SKIP` to bypass specific hooks temporarily
|
||||
- Recommended to run before every commit
|
||||
- Can be integrated into Git pre-commit hook for automatic checks
|
||||
- Cache location: `~/.cache/pre-commit` (configurable)
|
||||
|
||||
## Performance Tips
|
||||
|
||||
- **Initial Setup**: First run takes longer (installing hook environments)
|
||||
- **Incremental**: Run on staged files only for faster feedback
|
||||
- **Parallel**: Pre-commit runs compatible hooks in parallel
|
||||
- **Cache**: Hook environments are cached and reused
|
||||
- **Skip**: Use `SKIP` to bypass slow hooks during development
|
||||
|
||||
## Integration with Git
|
||||
|
||||
To automatically run on every commit:
|
||||
|
||||
```bash
|
||||
# Install Git pre-commit hook
|
||||
pre-commit install
|
||||
|
||||
# Now pre-commit runs automatically on git commit
|
||||
git commit -m "Your commit message"
|
||||
```
|
||||
|
||||
To bypass pre-commit hook temporarily:
|
||||
|
||||
```bash
|
||||
git commit --no-verify -m "Emergency commit"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Pre-commit configuration is in `.pre-commit-config.yaml`. To update hooks:
|
||||
|
||||
```bash
|
||||
# Update to latest versions
|
||||
pre-commit autoupdate
|
||||
|
||||
# Clean cache and re-install
|
||||
pre-commit clean
|
||||
pre-commit install --install-hooks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: `pre-commit run --all-files`
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
#!/usr/bin/env bash
|
||||
# Agent Skills - Environment Helpers
|
||||
#
|
||||
# Provides environment validation and setup utilities.
|
||||
|
||||
# validate_go_environment: Check Go installation and version
|
||||
validate_go_environment() {
|
||||
local min_version="${1:-1.23}"
|
||||
|
||||
if ! command -v go >/dev/null 2>&1; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Go is not installed or not in PATH"
|
||||
else
|
||||
echo "[ERROR] Go is not installed or not in PATH" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
local go_version
|
||||
go_version=$(go version | grep -oP 'go\K[0-9]+\.[0-9]+' || echo "0.0")
|
||||
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
log_debug "Go version: ${go_version} (required: >=${min_version})"
|
||||
fi
|
||||
|
||||
# Simple version comparison (assumes semantic versioning)
|
||||
if [[ "$(printf '%s\n' "${min_version}" "${go_version}" | sort -V | head -n1)" != "${min_version}" ]]; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Go version ${go_version} is below minimum required version ${min_version}"
|
||||
else
|
||||
echo "[ERROR] Go version ${go_version} is below minimum required version ${min_version}" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# validate_python_environment: Check Python installation and version
|
||||
validate_python_environment() {
|
||||
local min_version="${1:-3.8}"
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Python 3 is not installed or not in PATH"
|
||||
else
|
||||
echo "[ERROR] Python 3 is not installed or not in PATH" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
local python_version
|
||||
python_version=$(python3 --version 2>&1 | grep -oP 'Python \K[0-9]+\.[0-9]+' || echo "0.0")
|
||||
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
log_debug "Python version: ${python_version} (required: >=${min_version})"
|
||||
fi
|
||||
|
||||
# Simple version comparison
|
||||
if [[ "$(printf '%s\n' "${min_version}" "${python_version}" | sort -V | head -n1)" != "${min_version}" ]]; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Python version ${python_version} is below minimum required version ${min_version}"
|
||||
else
|
||||
echo "[ERROR] Python version ${python_version} is below minimum required version ${min_version}" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# validate_node_environment: Check Node.js installation and version
|
||||
validate_node_environment() {
|
||||
local min_version="${1:-18.0}"
|
||||
|
||||
if ! command -v node >/dev/null 2>&1; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Node.js is not installed or not in PATH"
|
||||
else
|
||||
echo "[ERROR] Node.js is not installed or not in PATH" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
local node_version
|
||||
node_version=$(node --version | grep -oP 'v\K[0-9]+\.[0-9]+' || echo "0.0")
|
||||
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
log_debug "Node.js version: ${node_version} (required: >=${min_version})"
|
||||
fi
|
||||
|
||||
# Simple version comparison
|
||||
if [[ "$(printf '%s\n' "${min_version}" "${node_version}" | sort -V | head -n1)" != "${min_version}" ]]; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Node.js version ${node_version} is below minimum required version ${min_version}"
|
||||
else
|
||||
echo "[ERROR] Node.js version ${node_version} is below minimum required version ${min_version}" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# validate_docker_environment: Check Docker installation and daemon
|
||||
validate_docker_environment() {
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Docker is not installed or not in PATH"
|
||||
else
|
||||
echo "[ERROR] Docker is not installed or not in PATH" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check if Docker daemon is running
|
||||
if ! docker info >/dev/null 2>&1; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Docker daemon is not running"
|
||||
else
|
||||
echo "[ERROR] Docker daemon is not running" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
local docker_version
|
||||
docker_version=$(docker --version | grep -oP 'Docker version \K[0-9]+\.[0-9]+\.[0-9]+' || echo "unknown")
|
||||
log_debug "Docker version: ${docker_version}"
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# set_default_env: Set environment variable with default value if not set
|
||||
set_default_env() {
|
||||
local var_name="$1"
|
||||
local default_value="$2"
|
||||
|
||||
if [[ -z "${!var_name:-}" ]]; then
|
||||
export "${var_name}=${default_value}"
|
||||
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
log_debug "Set ${var_name}=${default_value} (default)"
|
||||
fi
|
||||
else
|
||||
if declare -f log_debug >/dev/null 2>&1; then
|
||||
log_debug "Using ${var_name}=${!var_name} (from environment)"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# validate_project_structure: Check we're in the correct project directory
|
||||
validate_project_structure() {
|
||||
local required_files=("$@")
|
||||
|
||||
for file in "${required_files[@]}"; do
|
||||
if [[ ! -e "${file}" ]]; then
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Required file/directory not found: ${file}"
|
||||
log_error "Are you running this from the project root?"
|
||||
else
|
||||
echo "[ERROR] Required file/directory not found: ${file}" >&2
|
||||
echo "[ERROR] Are you running this from the project root?" >&2
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# get_project_root: Find project root by looking for marker files
|
||||
get_project_root() {
|
||||
local marker_file="${1:-.git}"
|
||||
local current_dir
|
||||
current_dir="$(pwd)"
|
||||
|
||||
while [[ "${current_dir}" != "/" ]]; do
|
||||
if [[ -e "${current_dir}/${marker_file}" ]]; then
|
||||
echo "${current_dir}"
|
||||
return 0
|
||||
fi
|
||||
current_dir="$(dirname "${current_dir}")"
|
||||
done
|
||||
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Could not find project root (looking for ${marker_file})"
|
||||
else
|
||||
echo "[ERROR] Could not find project root (looking for ${marker_file})" >&2
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# Export functions
|
||||
export -f validate_go_environment
|
||||
export -f validate_python_environment
|
||||
export -f validate_node_environment
|
||||
export -f validate_docker_environment
|
||||
export -f set_default_env
|
||||
export -f validate_project_structure
|
||||
export -f get_project_root
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env bash
|
||||
# Agent Skills - Error Handling Helpers
|
||||
#
|
||||
# Provides error handling utilities for robust skill execution.
|
||||
|
||||
# error_exit: Print error message and exit with code
|
||||
error_exit() {
|
||||
local message="$1"
|
||||
local exit_code="${2:-1}"
|
||||
|
||||
# Source logging helpers if not already loaded
|
||||
if ! declare -f log_error >/dev/null 2>&1; then
|
||||
echo "[ERROR] ${message}" >&2
|
||||
else
|
||||
log_error "${message}"
|
||||
fi
|
||||
|
||||
exit "${exit_code}"
|
||||
}
|
||||
|
||||
# check_command_exists: Verify a command is available
|
||||
check_command_exists() {
|
||||
local cmd="$1"
|
||||
local error_msg="${2:-Command not found: ${cmd}}"
|
||||
|
||||
if ! command -v "${cmd}" >/dev/null 2>&1; then
|
||||
error_exit "${error_msg}" 127
|
||||
fi
|
||||
}
|
||||
|
||||
# check_file_exists: Verify a file exists
|
||||
check_file_exists() {
|
||||
local file="$1"
|
||||
local error_msg="${2:-File not found: ${file}}"
|
||||
|
||||
if [[ ! -f "${file}" ]]; then
|
||||
error_exit "${error_msg}" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# check_dir_exists: Verify a directory exists
|
||||
check_dir_exists() {
|
||||
local dir="$1"
|
||||
local error_msg="${2:-Directory not found: ${dir}}"
|
||||
|
||||
if [[ ! -d "${dir}" ]]; then
|
||||
error_exit "${error_msg}" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# check_exit_code: Verify previous command succeeded
|
||||
check_exit_code() {
|
||||
local exit_code=$?
|
||||
local error_msg="${1:-Command failed with exit code ${exit_code}}"
|
||||
|
||||
if [[ ${exit_code} -ne 0 ]]; then
|
||||
error_exit "${error_msg}" "${exit_code}"
|
||||
fi
|
||||
}
|
||||
|
||||
# run_with_retry: Run a command with retry logic
|
||||
run_with_retry() {
|
||||
local max_attempts="${1}"
|
||||
local delay="${2}"
|
||||
shift 2
|
||||
local cmd=("$@")
|
||||
|
||||
local attempt=1
|
||||
while [[ ${attempt} -le ${max_attempts} ]]; do
|
||||
if "${cmd[@]}"; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ ${attempt} -lt ${max_attempts} ]]; then
|
||||
if declare -f log_warning >/dev/null 2>&1; then
|
||||
log_warning "Command failed (attempt ${attempt}/${max_attempts}). Retrying in ${delay}s..."
|
||||
else
|
||||
echo "[WARNING] Command failed (attempt ${attempt}/${max_attempts}). Retrying in ${delay}s..." >&2
|
||||
fi
|
||||
sleep "${delay}"
|
||||
fi
|
||||
|
||||
((attempt++))
|
||||
done
|
||||
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Command failed after ${max_attempts} attempts: ${cmd[*]}"
|
||||
else
|
||||
echo "[ERROR] Command failed after ${max_attempts} attempts: ${cmd[*]}" >&2
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# trap_error: Set up error trapping for the current script
|
||||
trap_error() {
|
||||
local script_name="${1:-${BASH_SOURCE[1]}}"
|
||||
|
||||
trap 'error_handler ${LINENO} ${BASH_LINENO} "${BASH_COMMAND}" "${script_name}"' ERR
|
||||
}
|
||||
|
||||
# error_handler: Internal error handler for trap
|
||||
error_handler() {
|
||||
local line_no="$1"
|
||||
local bash_line_no="$2"
|
||||
local command="$3"
|
||||
local script="$4"
|
||||
|
||||
if declare -f log_error >/dev/null 2>&1; then
|
||||
log_error "Script failed at line ${line_no} in ${script}"
|
||||
log_error "Command: ${command}"
|
||||
else
|
||||
echo "[ERROR] Script failed at line ${line_no} in ${script}" >&2
|
||||
echo "[ERROR] Command: ${command}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
# cleanup_on_exit: Register a cleanup function to run on exit
|
||||
cleanup_on_exit() {
|
||||
local cleanup_func="$1"
|
||||
|
||||
# Register cleanup function
|
||||
trap "${cleanup_func}" EXIT
|
||||
}
|
||||
|
||||
# Export functions
|
||||
export -f error_exit
|
||||
export -f check_command_exists
|
||||
export -f check_file_exists
|
||||
export -f check_dir_exists
|
||||
export -f check_exit_code
|
||||
export -f run_with_retry
|
||||
export -f trap_error
|
||||
export -f error_handler
|
||||
export -f cleanup_on_exit
|
||||
Executable
+109
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env bash
|
||||
# Agent Skills - Logging Helpers
|
||||
#
|
||||
# Provides colored logging functions for consistent output across all skills.
|
||||
|
||||
# Color codes
|
||||
readonly COLOR_RESET="\033[0m"
|
||||
readonly COLOR_RED="\033[0;31m"
|
||||
readonly COLOR_GREEN="\033[0;32m"
|
||||
readonly COLOR_YELLOW="\033[0;33m"
|
||||
readonly COLOR_BLUE="\033[0;34m"
|
||||
readonly COLOR_MAGENTA="\033[0;35m"
|
||||
readonly COLOR_CYAN="\033[0;36m"
|
||||
readonly COLOR_GRAY="\033[0;90m"
|
||||
|
||||
# Check if output is a terminal (for color support)
|
||||
if [[ -t 1 ]]; then
|
||||
COLORS_ENABLED=true
|
||||
else
|
||||
COLORS_ENABLED=false
|
||||
fi
|
||||
|
||||
# Disable colors if NO_COLOR environment variable is set
|
||||
if [[ -n "${NO_COLOR:-}" ]]; then
|
||||
COLORS_ENABLED=false
|
||||
fi
|
||||
|
||||
# log_info: Print informational message
|
||||
log_info() {
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_BLUE}[INFO]${COLOR_RESET} ${message}"
|
||||
else
|
||||
echo "[INFO] ${message}"
|
||||
fi
|
||||
}
|
||||
|
||||
# log_success: Print success message
|
||||
log_success() {
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_GREEN}[SUCCESS]${COLOR_RESET} ${message}"
|
||||
else
|
||||
echo "[SUCCESS] ${message}"
|
||||
fi
|
||||
}
|
||||
|
||||
# log_warning: Print warning message
|
||||
log_warning() {
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_YELLOW}[WARNING]${COLOR_RESET} ${message}" >&2
|
||||
else
|
||||
echo "[WARNING] ${message}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
# log_error: Print error message
|
||||
log_error() {
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_RED}[ERROR]${COLOR_RESET} ${message}" >&2
|
||||
else
|
||||
echo "[ERROR] ${message}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
# log_debug: Print debug message (only if DEBUG=1)
|
||||
log_debug() {
|
||||
if [[ "${DEBUG:-0}" == "1" ]]; then
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_GRAY}[DEBUG]${COLOR_RESET} ${message}"
|
||||
else
|
||||
echo "[DEBUG] ${message}"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# log_step: Print step header
|
||||
log_step() {
|
||||
local step_name="$1"
|
||||
shift
|
||||
local message="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_CYAN}[${step_name}]${COLOR_RESET} ${message}"
|
||||
else
|
||||
echo "[${step_name}] ${message}"
|
||||
fi
|
||||
}
|
||||
|
||||
# log_command: Log a command before executing (for transparency)
|
||||
log_command() {
|
||||
local command="$*"
|
||||
if [[ "${COLORS_ENABLED}" == "true" ]]; then
|
||||
echo -e "${COLOR_MAGENTA}[$]${COLOR_RESET} ${command}"
|
||||
else
|
||||
echo "[\$] ${command}"
|
||||
fi
|
||||
}
|
||||
|
||||
# Export functions so they can be used by sourcing scripts
|
||||
export -f log_info
|
||||
export -f log_success
|
||||
export -f log_warning
|
||||
export -f log_error
|
||||
export -f log_debug
|
||||
export -f log_step
|
||||
export -f log_command
|
||||
Executable
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# Agent Skills Universal Skill Runner
|
||||
#
|
||||
# This script locates and executes Agent Skills by name, providing a unified
|
||||
# interface for running skills from tasks.json, CI/CD workflows, and the CLI.
|
||||
#
|
||||
# Usage:
|
||||
# skill-runner.sh <skill-name> [args...]
|
||||
#
|
||||
# Exit Codes:
|
||||
# 0 - Skill executed successfully
|
||||
# 1 - Skill not found or invalid
|
||||
# 2 - Skill execution failed
|
||||
# 126 - Skill script not executable
|
||||
# 127 - Skill script not found
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=_logging_helpers.sh
|
||||
source "${SCRIPT_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=_error_handling_helpers.sh
|
||||
source "${SCRIPT_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=_environment_helpers.sh
|
||||
source "${SCRIPT_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Configuration
|
||||
SKILLS_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SKILLS_DIR}/../.." && pwd)"
|
||||
|
||||
# Validate arguments
|
||||
if [[ $# -eq 0 ]]; then
|
||||
log_error "Usage: skill-runner.sh <skill-name> [args...]"
|
||||
log_error "Example: skill-runner.sh test-backend-coverage"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SKILL_NAME="$1"
|
||||
shift # Remove skill name from arguments
|
||||
|
||||
# Validate skill name format
|
||||
if [[ ! "${SKILL_NAME}" =~ ^[a-z][a-z0-9-]*$ ]]; then
|
||||
log_error "Invalid skill name: ${SKILL_NAME}"
|
||||
log_error "Skill names must be kebab-case (lowercase, hyphens, start with letter)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Verify SKILL.md exists
|
||||
SKILL_FILE="${SKILLS_DIR}/${SKILL_NAME}.SKILL.md"
|
||||
if [[ ! -f "${SKILL_FILE}" ]]; then
|
||||
log_error "Skill not found: ${SKILL_NAME}"
|
||||
log_error "Expected file: ${SKILL_FILE}"
|
||||
log_info "Available skills:"
|
||||
for skill_file in "${SKILLS_DIR}"/*.SKILL.md; do
|
||||
if [[ -f "${skill_file}" ]]; then
|
||||
basename "${skill_file}" .SKILL.md
|
||||
fi
|
||||
done | sort | sed 's/^/ - /'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Locate skill execution script (flat structure: skill-name-scripts/run.sh)
|
||||
SKILL_SCRIPT="${SKILLS_DIR}/${SKILL_NAME}-scripts/run.sh"
|
||||
|
||||
if [[ ! -f "${SKILL_SCRIPT}" ]]; then
|
||||
log_error "Skill execution script not found: ${SKILL_SCRIPT}"
|
||||
log_error "Expected: ${SKILL_NAME}-scripts/run.sh"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -x "${SKILL_SCRIPT}" ]]; then
|
||||
log_error "Skill execution script is not executable: ${SKILL_SCRIPT}"
|
||||
log_error "Fix with: chmod +x ${SKILL_SCRIPT}"
|
||||
exit 126
|
||||
fi
|
||||
|
||||
# Log skill execution
|
||||
log_info "Executing skill: ${SKILL_NAME}"
|
||||
log_debug "Skill file: ${SKILL_FILE}"
|
||||
log_debug "Skill script: ${SKILL_SCRIPT}"
|
||||
log_debug "Working directory: ${PROJECT_ROOT}"
|
||||
log_debug "Arguments: $*"
|
||||
|
||||
# Change to project root for execution
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Execute skill with all remaining arguments
|
||||
# shellcheck disable=SC2294
|
||||
if ! "${SKILL_SCRIPT}" "$@"; then
|
||||
log_error "Skill execution failed: ${SKILL_NAME}"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
log_success "Skill completed successfully: ${SKILL_NAME}"
|
||||
exit 0
|
||||
Executable
+422
@@ -0,0 +1,422 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Agent Skills Frontmatter Validator
|
||||
|
||||
Validates YAML frontmatter in .SKILL.md files against the agentskills.io
|
||||
specification. Ensures required fields are present, formats are correct,
|
||||
and custom metadata follows project conventions.
|
||||
|
||||
Usage:
|
||||
python3 validate-skills.py [path/to/.github/skills/]
|
||||
python3 validate-skills.py --single path/to/skill.SKILL.md
|
||||
|
||||
Exit Codes:
|
||||
0 - All validations passed
|
||||
1 - Validation errors found
|
||||
2 - Script error (missing dependencies, invalid arguments)
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import re
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Tuple, Any, Optional
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print("Error: PyYAML is required. Install with: pip install pyyaml", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
# Validation rules
|
||||
REQUIRED_FIELDS = ["name", "version", "description", "author", "license", "tags"]
|
||||
VALID_CATEGORIES = ["test", "integration-test", "security", "qa", "build", "utility", "docker"]
|
||||
VALID_EXECUTION_TIMES = ["short", "medium", "long"]
|
||||
VALID_RISK_LEVELS = ["low", "medium", "high"]
|
||||
VALID_OS_VALUES = ["linux", "darwin", "windows"]
|
||||
VALID_SHELL_VALUES = ["bash", "sh", "zsh", "powershell", "cmd"]
|
||||
|
||||
VERSION_REGEX = re.compile(r'^\d+\.\d+\.\d+$')
|
||||
NAME_REGEX = re.compile(r'^[a-z][a-z0-9-]*$')
|
||||
|
||||
|
||||
class ValidationError:
|
||||
"""Represents a validation error with context."""
|
||||
|
||||
def __init__(self, skill_file: str, field: str, message: str, severity: str = "error"):
|
||||
self.skill_file = skill_file
|
||||
self.field = field
|
||||
self.message = message
|
||||
self.severity = severity
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"[{self.severity.upper()}] {self.skill_file} :: {self.field}: {self.message}"
|
||||
|
||||
|
||||
class SkillValidator:
|
||||
"""Validates Agent Skills frontmatter."""
|
||||
|
||||
def __init__(self, strict: bool = False):
|
||||
self.strict = strict
|
||||
self.errors: List[ValidationError] = []
|
||||
self.warnings: List[ValidationError] = []
|
||||
|
||||
def validate_file(self, skill_path: Path) -> Tuple[bool, List[ValidationError]]:
|
||||
"""Validate a single SKILL.md file."""
|
||||
try:
|
||||
with open(skill_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
except Exception as e:
|
||||
return False, [ValidationError(str(skill_path), "file", f"Cannot read file: {e}")]
|
||||
|
||||
# Extract frontmatter
|
||||
frontmatter = self._extract_frontmatter(content)
|
||||
if not frontmatter:
|
||||
return False, [ValidationError(str(skill_path), "frontmatter", "No valid YAML frontmatter found")]
|
||||
|
||||
# Parse YAML
|
||||
try:
|
||||
data = yaml.safe_load(frontmatter)
|
||||
except yaml.YAMLError as e:
|
||||
return False, [ValidationError(str(skill_path), "yaml", f"Invalid YAML: {e}")]
|
||||
|
||||
if not isinstance(data, dict):
|
||||
return False, [ValidationError(str(skill_path), "yaml", "Frontmatter must be a YAML object")]
|
||||
|
||||
# Run validation checks
|
||||
file_errors: List[ValidationError] = []
|
||||
file_errors.extend(self._validate_required_fields(skill_path, data))
|
||||
file_errors.extend(self._validate_name(skill_path, data))
|
||||
file_errors.extend(self._validate_version(skill_path, data))
|
||||
file_errors.extend(self._validate_description(skill_path, data))
|
||||
file_errors.extend(self._validate_tags(skill_path, data))
|
||||
file_errors.extend(self._validate_compatibility(skill_path, data))
|
||||
file_errors.extend(self._validate_metadata(skill_path, data))
|
||||
|
||||
# Separate errors and warnings
|
||||
errors = [e for e in file_errors if e.severity == "error"]
|
||||
warnings = [e for e in file_errors if e.severity == "warning"]
|
||||
|
||||
self.errors.extend(errors)
|
||||
self.warnings.extend(warnings)
|
||||
|
||||
return len(errors) == 0, file_errors
|
||||
|
||||
def _extract_frontmatter(self, content: str) -> Optional[str]:
|
||||
"""Extract YAML frontmatter from markdown content."""
|
||||
if not content.startswith('---\n'):
|
||||
return None
|
||||
|
||||
end_marker = content.find('\n---\n', 4)
|
||||
if end_marker == -1:
|
||||
return None
|
||||
|
||||
return content[4:end_marker]
|
||||
|
||||
def _validate_required_fields(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Check that all required fields are present."""
|
||||
errors = []
|
||||
for field in REQUIRED_FIELDS:
|
||||
if field not in data:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), field, f"Required field missing"
|
||||
))
|
||||
elif not data[field]:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), field, f"Required field is empty"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_name(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate name field format."""
|
||||
errors = []
|
||||
if "name" in data:
|
||||
name = data["name"]
|
||||
if not isinstance(name, str):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "name", "Must be a string"
|
||||
))
|
||||
elif not NAME_REGEX.match(name):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "name",
|
||||
"Must be kebab-case (lowercase, hyphens only, start with letter)"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_version(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate version field format."""
|
||||
errors = []
|
||||
if "version" in data:
|
||||
version = data["version"]
|
||||
if not isinstance(version, str):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "version", "Must be a string"
|
||||
))
|
||||
elif not VERSION_REGEX.match(version):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "version",
|
||||
"Must follow semantic versioning (x.y.z)"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_description(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate description field."""
|
||||
errors = []
|
||||
if "description" in data:
|
||||
desc = data["description"]
|
||||
if not isinstance(desc, str):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "description", "Must be a string"
|
||||
))
|
||||
elif len(desc) > 120:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "description",
|
||||
f"Must be 120 characters or less (current: {len(desc)})"
|
||||
))
|
||||
elif '\n' in desc:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "description", "Must be a single line"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_tags(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate tags field."""
|
||||
errors = []
|
||||
if "tags" in data:
|
||||
tags = data["tags"]
|
||||
if not isinstance(tags, list):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "tags", "Must be a list"
|
||||
))
|
||||
elif len(tags) < 2:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "tags", "Must have at least 2 tags"
|
||||
))
|
||||
elif len(tags) > 5:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "tags",
|
||||
f"Must have at most 5 tags (current: {len(tags)})",
|
||||
severity="warning"
|
||||
))
|
||||
else:
|
||||
for tag in tags:
|
||||
if not isinstance(tag, str):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "tags", "All tags must be strings"
|
||||
))
|
||||
elif tag != tag.lower():
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "tags",
|
||||
f"Tag '{tag}' should be lowercase",
|
||||
severity="warning"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_compatibility(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate compatibility section."""
|
||||
errors = []
|
||||
if "compatibility" in data:
|
||||
compat = data["compatibility"]
|
||||
if not isinstance(compat, dict):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "compatibility", "Must be an object"
|
||||
))
|
||||
else:
|
||||
# Validate OS
|
||||
if "os" in compat:
|
||||
os_list = compat["os"]
|
||||
if not isinstance(os_list, list):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "compatibility.os", "Must be a list"
|
||||
))
|
||||
else:
|
||||
for os_val in os_list:
|
||||
if os_val not in VALID_OS_VALUES:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "compatibility.os",
|
||||
f"Invalid OS '{os_val}'. Valid: {VALID_OS_VALUES}",
|
||||
severity="warning"
|
||||
))
|
||||
|
||||
# Validate shells
|
||||
if "shells" in compat:
|
||||
shells = compat["shells"]
|
||||
if not isinstance(shells, list):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "compatibility.shells", "Must be a list"
|
||||
))
|
||||
else:
|
||||
for shell in shells:
|
||||
if shell not in VALID_SHELL_VALUES:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "compatibility.shells",
|
||||
f"Invalid shell '{shell}'. Valid: {VALID_SHELL_VALUES}",
|
||||
severity="warning"
|
||||
))
|
||||
return errors
|
||||
|
||||
def _validate_metadata(self, skill_path: Path, data: Dict) -> List[ValidationError]:
|
||||
"""Validate custom metadata section."""
|
||||
errors = []
|
||||
if "metadata" not in data:
|
||||
return errors # Metadata is optional
|
||||
|
||||
metadata = data["metadata"]
|
||||
if not isinstance(metadata, dict):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "metadata", "Must be an object"
|
||||
))
|
||||
return errors
|
||||
|
||||
# Validate category
|
||||
if "category" in metadata:
|
||||
category = metadata["category"]
|
||||
if category not in VALID_CATEGORIES:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "metadata.category",
|
||||
f"Invalid category '{category}'. Valid: {VALID_CATEGORIES}",
|
||||
severity="warning"
|
||||
))
|
||||
|
||||
# Validate execution_time
|
||||
if "execution_time" in metadata:
|
||||
exec_time = metadata["execution_time"]
|
||||
if exec_time not in VALID_EXECUTION_TIMES:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "metadata.execution_time",
|
||||
f"Invalid execution_time '{exec_time}'. Valid: {VALID_EXECUTION_TIMES}",
|
||||
severity="warning"
|
||||
))
|
||||
|
||||
# Validate risk_level
|
||||
if "risk_level" in metadata:
|
||||
risk = metadata["risk_level"]
|
||||
if risk not in VALID_RISK_LEVELS:
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), "metadata.risk_level",
|
||||
f"Invalid risk_level '{risk}'. Valid: {VALID_RISK_LEVELS}",
|
||||
severity="warning"
|
||||
))
|
||||
|
||||
# Validate boolean fields
|
||||
for bool_field in ["ci_cd_safe", "requires_network", "idempotent"]:
|
||||
if bool_field in metadata:
|
||||
if not isinstance(metadata[bool_field], bool):
|
||||
errors.append(ValidationError(
|
||||
str(skill_path), f"metadata.{bool_field}",
|
||||
"Must be a boolean (true/false)",
|
||||
severity="warning"
|
||||
))
|
||||
|
||||
return errors
|
||||
|
||||
def validate_directory(self, skills_dir: Path) -> bool:
|
||||
"""Validate all SKILL.md files in a directory."""
|
||||
if not skills_dir.exists():
|
||||
print(f"Error: Directory not found: {skills_dir}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
skill_files = list(skills_dir.glob("*.SKILL.md"))
|
||||
if not skill_files:
|
||||
print(f"Warning: No .SKILL.md files found in {skills_dir}", file=sys.stderr)
|
||||
return True # Not an error, just nothing to validate
|
||||
|
||||
print(f"Validating {len(skill_files)} skill(s)...\n")
|
||||
|
||||
success_count = 0
|
||||
for skill_file in sorted(skill_files):
|
||||
is_valid, _ = self.validate_file(skill_file)
|
||||
if is_valid:
|
||||
success_count += 1
|
||||
print(f"✓ {skill_file.name}")
|
||||
else:
|
||||
print(f"✗ {skill_file.name}")
|
||||
|
||||
# Print summary
|
||||
print(f"\n{'='*70}")
|
||||
print(f"Validation Summary:")
|
||||
print(f" Total skills: {len(skill_files)}")
|
||||
print(f" Passed: {success_count}")
|
||||
print(f" Failed: {len(skill_files) - success_count}")
|
||||
print(f" Errors: {len(self.errors)}")
|
||||
print(f" Warnings: {len(self.warnings)}")
|
||||
print(f"{'='*70}\n")
|
||||
|
||||
# Print errors
|
||||
if self.errors:
|
||||
print("ERRORS:")
|
||||
for error in self.errors:
|
||||
print(f" {error}")
|
||||
print()
|
||||
|
||||
# Print warnings
|
||||
if self.warnings:
|
||||
print("WARNINGS:")
|
||||
for warning in self.warnings:
|
||||
print(f" {warning}")
|
||||
print()
|
||||
|
||||
return len(self.errors) == 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Validate Agent Skills frontmatter",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__
|
||||
)
|
||||
parser.add_argument(
|
||||
"path",
|
||||
nargs="?",
|
||||
default=".github/skills",
|
||||
help="Path to .github/skills directory or single .SKILL.md file (default: .github/skills)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--strict",
|
||||
action="store_true",
|
||||
help="Treat warnings as errors"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--single",
|
||||
action="store_true",
|
||||
help="Validate a single .SKILL.md file instead of a directory"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
validator = SkillValidator(strict=args.strict)
|
||||
path = Path(args.path)
|
||||
|
||||
if args.single:
|
||||
if not path.exists():
|
||||
print(f"Error: File not found: {path}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
is_valid, errors = validator.validate_file(path)
|
||||
|
||||
if is_valid:
|
||||
print(f"✓ {path.name} is valid")
|
||||
if errors: # Warnings only
|
||||
print("\nWARNINGS:")
|
||||
for error in errors:
|
||||
print(f" {error}")
|
||||
else:
|
||||
print(f"✗ {path.name} has errors")
|
||||
for error in errors:
|
||||
print(f" {error}")
|
||||
|
||||
return 0 if is_valid else 1
|
||||
else:
|
||||
success = validator.validate_directory(path)
|
||||
|
||||
if args.strict and validator.warnings:
|
||||
print("Strict mode: treating warnings as errors", file=sys.stderr)
|
||||
success = False
|
||||
|
||||
return 0 if success else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Scan CodeQL - Execution Script
|
||||
#
|
||||
# This script runs CodeQL security analysis using the security-and-quality
|
||||
# suite to match GitHub Actions CI configuration exactly.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Some helper scripts may not define ANSI color variables; ensure they exist
|
||||
# before using them later in this script (set -u is enabled).
|
||||
RED="${RED:-\033[0;31m}"
|
||||
GREEN="${GREEN:-\033[0;32m}"
|
||||
NC="${NC:-\033[0m}"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "CODEQL_THREADS" "0"
|
||||
set_default_env "CODEQL_FAIL_ON_ERROR" "true"
|
||||
|
||||
# Parse arguments
|
||||
LANGUAGE="${1:-all}"
|
||||
FORMAT="${2:-summary}"
|
||||
|
||||
# Validate language
|
||||
case "${LANGUAGE}" in
|
||||
go|javascript|js|all)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid language: ${LANGUAGE}. Must be one of: go, javascript, all"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# Normalize javascript -> js for internal use
|
||||
if [[ "${LANGUAGE}" == "javascript" ]]; then
|
||||
LANGUAGE="js"
|
||||
fi
|
||||
|
||||
# Validate format
|
||||
case "${FORMAT}" in
|
||||
sarif|text|summary)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid format: ${FORMAT}. Must be one of: sarif, text, summary"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# Validate CodeQL is installed
|
||||
log_step "ENVIRONMENT" "Validating CodeQL installation"
|
||||
if ! command -v codeql &> /dev/null; then
|
||||
log_error "CodeQL CLI is not installed"
|
||||
log_info "Install via: gh extension install github/gh-codeql"
|
||||
log_info "Then run: gh codeql set-version latest"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Check CodeQL version
|
||||
CODEQL_VERSION=$(codeql version 2>/dev/null | head -1 | grep -oP '\d+\.\d+\.\d+' || echo "unknown")
|
||||
log_info "CodeQL version: ${CODEQL_VERSION}"
|
||||
|
||||
# Minimum version check
|
||||
MIN_VERSION="2.17.0"
|
||||
if [[ "${CODEQL_VERSION}" != "unknown" ]]; then
|
||||
if [[ "$(printf '%s\n' "${MIN_VERSION}" "${CODEQL_VERSION}" | sort -V | head -n1)" != "${MIN_VERSION}" ]]; then
|
||||
log_warning "CodeQL version ${CODEQL_VERSION} may be incompatible"
|
||||
log_info "Recommended: gh codeql set-version latest"
|
||||
fi
|
||||
fi
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Track findings
|
||||
GO_ERRORS=0
|
||||
GO_WARNINGS=0
|
||||
JS_ERRORS=0
|
||||
JS_WARNINGS=0
|
||||
SCAN_FAILED=0
|
||||
|
||||
# Function to run CodeQL scan for a language
|
||||
run_codeql_scan() {
|
||||
local lang=$1
|
||||
local source_root=$2
|
||||
local db_name="codeql-db-${lang}"
|
||||
local sarif_file="codeql-results-${lang}.sarif"
|
||||
local build_mode_args=()
|
||||
local codescanning_config="${PROJECT_ROOT}/.github/codeql/codeql-config.yml"
|
||||
|
||||
# Remove generated artifacts that can create noisy/false findings during CodeQL analysis
|
||||
rm -rf "${PROJECT_ROOT}/frontend/coverage" \
|
||||
"${PROJECT_ROOT}/frontend/dist" \
|
||||
"${PROJECT_ROOT}/playwright-report" \
|
||||
"${PROJECT_ROOT}/test-results" \
|
||||
"${PROJECT_ROOT}/coverage"
|
||||
|
||||
if [[ "${lang}" == "javascript" ]]; then
|
||||
build_mode_args=(--build-mode=none)
|
||||
fi
|
||||
|
||||
log_step "CODEQL" "Scanning ${lang} code in ${source_root}/"
|
||||
|
||||
# Clean previous database
|
||||
rm -rf "${db_name}"
|
||||
|
||||
# Create database
|
||||
log_info "Creating CodeQL database..."
|
||||
if ! codeql database create "${db_name}" \
|
||||
--language="${lang}" \
|
||||
"${build_mode_args[@]}" \
|
||||
--source-root="${source_root}" \
|
||||
--codescanning-config="${codescanning_config}" \
|
||||
--threads="${CODEQL_THREADS}" \
|
||||
--overwrite 2>&1 | while read -r line; do
|
||||
# Filter verbose output, show important messages
|
||||
if [[ "${line}" == *"error"* ]] || [[ "${line}" == *"Error"* ]]; then
|
||||
log_error "${line}"
|
||||
elif [[ "${line}" == *"warning"* ]]; then
|
||||
log_warning "${line}"
|
||||
fi
|
||||
done; then
|
||||
log_error "Failed to create CodeQL database for ${lang}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Run analysis
|
||||
log_info "Analyzing with Code Scanning config (CI-aligned query filters)..."
|
||||
if ! codeql database analyze "${db_name}" \
|
||||
--format=sarif-latest \
|
||||
--output="${sarif_file}" \
|
||||
--sarif-add-baseline-file-info \
|
||||
--threads="${CODEQL_THREADS}" 2>&1; then
|
||||
log_error "CodeQL analysis failed for ${lang}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_success "SARIF output: ${sarif_file}"
|
||||
|
||||
# Parse results
|
||||
if command -v jq &> /dev/null && [[ -f "${sarif_file}" ]]; then
|
||||
local total_findings
|
||||
local error_count
|
||||
local warning_count
|
||||
local note_count
|
||||
|
||||
total_findings=$(jq '.runs[].results | length' "${sarif_file}" 2>/dev/null || echo 0)
|
||||
error_count=$(jq '[.runs[].results[] | select(.level == "error")] | length' "${sarif_file}" 2>/dev/null || echo 0)
|
||||
warning_count=$(jq '[.runs[].results[] | select(.level == "warning")] | length' "${sarif_file}" 2>/dev/null || echo 0)
|
||||
note_count=$(jq '[.runs[].results[] | select(.level == "note")] | length' "${sarif_file}" 2>/dev/null || echo 0)
|
||||
|
||||
log_info "Found: ${error_count} errors, ${warning_count} warnings, ${note_count} notes (${total_findings} total)"
|
||||
|
||||
# Store counts for global tracking
|
||||
if [[ "${lang}" == "go" ]]; then
|
||||
GO_ERRORS=${error_count}
|
||||
GO_WARNINGS=${warning_count}
|
||||
else
|
||||
JS_ERRORS=${error_count}
|
||||
JS_WARNINGS=${warning_count}
|
||||
fi
|
||||
|
||||
# Show findings based on format
|
||||
if [[ "${FORMAT}" == "text" ]] || [[ "${FORMAT}" == "summary" ]]; then
|
||||
if [[ ${total_findings} -gt 0 ]]; then
|
||||
echo ""
|
||||
log_info "Top findings:"
|
||||
jq -r '.runs[].results[] | "\(.level): \(.message.text | split("\n")[0]) (\(.locations[0].physicalLocation.artifactLocation.uri):\(.locations[0].physicalLocation.region.startLine))"' "${sarif_file}" 2>/dev/null | head -15
|
||||
echo ""
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for blocking errors
|
||||
if [[ ${error_count} -gt 0 ]]; then
|
||||
log_error "${lang}: ${error_count} HIGH/CRITICAL findings detected"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log_warning "jq not available - install for detailed analysis"
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Run scans based on language selection
|
||||
if [[ "${LANGUAGE}" == "all" ]] || [[ "${LANGUAGE}" == "go" ]]; then
|
||||
if ! run_codeql_scan "go" "backend"; then
|
||||
SCAN_FAILED=1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${LANGUAGE}" == "all" ]] || [[ "${LANGUAGE}" == "js" ]]; then
|
||||
if ! run_codeql_scan "javascript" "frontend"; then
|
||||
SCAN_FAILED=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Final summary
|
||||
echo ""
|
||||
log_step "SUMMARY" "CodeQL Security Scan Results"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
if [[ "${LANGUAGE}" == "all" ]] || [[ "${LANGUAGE}" == "go" ]]; then
|
||||
if [[ ${GO_ERRORS} -gt 0 ]]; then
|
||||
echo -e " Go: ${RED}${GO_ERRORS} errors${NC}, ${GO_WARNINGS} warnings"
|
||||
else
|
||||
echo -e " Go: ${GREEN}0 errors${NC}, ${GO_WARNINGS} warnings"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${LANGUAGE}" == "all" ]] || [[ "${LANGUAGE}" == "js" ]]; then
|
||||
if [[ ${JS_ERRORS} -gt 0 ]]; then
|
||||
echo -e " JavaScript: ${RED}${JS_ERRORS} errors${NC}, ${JS_WARNINGS} warnings"
|
||||
else
|
||||
echo -e " JavaScript: ${GREEN}0 errors${NC}, ${JS_WARNINGS} warnings"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Exit based on findings
|
||||
if [[ "${CODEQL_FAIL_ON_ERROR}" == "true" ]] && [[ ${SCAN_FAILED} -eq 1 ]]; then
|
||||
log_error "CodeQL scan found HIGH/CRITICAL issues - fix before proceeding"
|
||||
echo ""
|
||||
log_info "View results:"
|
||||
log_info " VS Code: Install SARIF Viewer extension, open codeql-results-*.sarif"
|
||||
log_info " CLI: jq '.runs[].results[]' codeql-results-*.sarif"
|
||||
exit 1
|
||||
else
|
||||
log_success "CodeQL scan complete - no blocking issues"
|
||||
exit 0
|
||||
fi
|
||||
@@ -0,0 +1,312 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-scan-codeql"
|
||||
version: "1.0.0"
|
||||
description: "Run CodeQL security analysis for Go and JavaScript/TypeScript code"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "scanning"
|
||||
- "codeql"
|
||||
- "sast"
|
||||
- "vulnerabilities"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "codeql"
|
||||
version: ">=2.17.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "CODEQL_THREADS"
|
||||
description: "Number of threads for analysis (0 = auto)"
|
||||
default: "0"
|
||||
required: false
|
||||
- name: "CODEQL_FAIL_ON_ERROR"
|
||||
description: "Exit with error on HIGH/CRITICAL findings"
|
||||
default: "true"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "language"
|
||||
type: "string"
|
||||
description: "Language to scan (go, javascript, all)"
|
||||
default: "all"
|
||||
required: false
|
||||
- name: "format"
|
||||
type: "string"
|
||||
description: "Output format (sarif, text, summary)"
|
||||
default: "summary"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "sarif_files"
|
||||
type: "file"
|
||||
description: "SARIF files for each language scanned"
|
||||
- name: "summary"
|
||||
type: "stdout"
|
||||
description: "Human-readable findings summary"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if no HIGH/CRITICAL issues, non-zero otherwise"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "sast"
|
||||
execution_time: "long"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Security Scan CodeQL
|
||||
|
||||
## Overview
|
||||
|
||||
Executes GitHub CodeQL static analysis security testing (SAST) for Go and JavaScript/TypeScript code. Uses the **security-and-quality** query suite to match GitHub Actions CI configuration exactly.
|
||||
|
||||
This skill ensures local development catches the same security issues that CI would detect, preventing CI failures due to security findings.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- CodeQL CLI 2.17.0 or higher installed
|
||||
- Query packs: `codeql/go-queries`, `codeql/javascript-queries`
|
||||
- Sufficient disk space for CodeQL databases (~500MB per language)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Scan all languages with summary output:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql
|
||||
```
|
||||
|
||||
### Scan Specific Language
|
||||
|
||||
Scan only Go code:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql go
|
||||
```
|
||||
|
||||
Scan only JavaScript/TypeScript code:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql javascript
|
||||
```
|
||||
|
||||
### Full SARIF Output
|
||||
|
||||
Get detailed SARIF output for integration with tools:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql all sarif
|
||||
```
|
||||
|
||||
### Text Output
|
||||
|
||||
Get text-formatted detailed findings:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql all text
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| language | string | No | all | Language to scan (go, javascript, all) |
|
||||
| format | string | No | summary | Output format (sarif, text, summary) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| CODEQL_THREADS | No | 0 | Analysis threads (0 = auto-detect) |
|
||||
| CODEQL_FAIL_ON_ERROR | No | true | Fail on HIGH/CRITICAL findings |
|
||||
|
||||
## Query Suite
|
||||
|
||||
This skill uses the **security-and-quality** suite to match CI:
|
||||
|
||||
| Language | Suite | Queries | Coverage |
|
||||
|----------|-------|---------|----------|
|
||||
| Go | go-security-and-quality.qls | 61 | Security + quality issues |
|
||||
| JavaScript | javascript-security-and-quality.qls | 204 | Security + quality issues |
|
||||
|
||||
**Note:** This matches GitHub Actions CodeQL default configuration exactly.
|
||||
|
||||
## Outputs
|
||||
|
||||
- **SARIF Files**:
|
||||
- `codeql-results-go.sarif` - Go findings
|
||||
- `codeql-results-js.sarif` - JavaScript/TypeScript findings
|
||||
- **Databases**:
|
||||
- `codeql-db-go/` - Go CodeQL database
|
||||
- `codeql-db-js/` - JavaScript CodeQL database
|
||||
- **Exit Codes**:
|
||||
- 0: No HIGH/CRITICAL findings
|
||||
- 1: HIGH/CRITICAL findings detected
|
||||
- 2: Scanner error
|
||||
|
||||
## Security Categories
|
||||
|
||||
### CWE Coverage
|
||||
|
||||
| Category | Description | Languages |
|
||||
|----------|-------------|-----------|
|
||||
| CWE-079 | Cross-Site Scripting (XSS) | JS |
|
||||
| CWE-089 | SQL Injection | Go, JS |
|
||||
| CWE-117 | Log Injection | Go |
|
||||
| CWE-200 | Information Exposure | Go, JS |
|
||||
| CWE-312 | Cleartext Storage | Go, JS |
|
||||
| CWE-327 | Weak Cryptography | Go, JS |
|
||||
| CWE-502 | Deserialization | Go, JS |
|
||||
| CWE-611 | XXE Injection | Go |
|
||||
| CWE-640 | Email Injection | Go |
|
||||
| CWE-798 | Hardcoded Credentials | Go, JS |
|
||||
| CWE-918 | SSRF | Go, JS |
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Full Scan (Default)
|
||||
|
||||
```bash
|
||||
# Scan all languages, show summary
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
[STEP] CODEQL: Scanning Go code...
|
||||
[INFO] Creating database for backend/
|
||||
[INFO] Analyzing with security-and-quality suite (61 queries)
|
||||
[INFO] Found: 0 errors, 5 warnings, 3 notes
|
||||
[STEP] CODEQL: Scanning JavaScript code...
|
||||
[INFO] Creating database for frontend/
|
||||
[INFO] Analyzing with security-and-quality suite (204 queries)
|
||||
[INFO] Found: 0 errors, 2 warnings, 8 notes
|
||||
[SUCCESS] CodeQL scan complete - no HIGH/CRITICAL issues
|
||||
```
|
||||
|
||||
### Example 2: Go Only with Text Output
|
||||
|
||||
```bash
|
||||
# Detailed text output for Go findings
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql go text
|
||||
```
|
||||
|
||||
### Example 3: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example (already integrated in codeql.yml)
|
||||
- name: Run CodeQL Security Scan
|
||||
run: .github/skills/scripts/skill-runner.sh security-scan-codeql all summary
|
||||
continue-on-error: false
|
||||
```
|
||||
|
||||
### Example 4: Pre-Commit Integration
|
||||
|
||||
```bash
|
||||
# Already available via pre-commit
|
||||
pre-commit run codeql-go-scan --all-files
|
||||
pre-commit run codeql-js-scan --all-files
|
||||
pre-commit run codeql-check-findings --all-files
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**CodeQL version too old**:
|
||||
```bash
|
||||
Error: Extensible predicate API mismatch
|
||||
Solution: Upgrade CodeQL CLI: gh codeql set-version latest
|
||||
```
|
||||
|
||||
**Query pack not found**:
|
||||
```bash
|
||||
Error: Could not resolve pack codeql/go-queries
|
||||
Solution: codeql pack download codeql/go-queries codeql/javascript-queries
|
||||
```
|
||||
|
||||
**Database creation failed**:
|
||||
```bash
|
||||
Error: No source files found
|
||||
Solution: Verify source-root points to correct directory
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: No HIGH/CRITICAL (error-level) findings
|
||||
- **1**: HIGH/CRITICAL findings detected (blocks CI)
|
||||
- **2**: Scanner error or invalid arguments
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Container/dependency vulnerabilities
|
||||
- [security-scan-go-vuln](./security-scan-go-vuln.SKILL.md) - Go-specific CVE checking
|
||||
- [qa-precommit-all](./qa-precommit-all.SKILL.md) - Pre-commit quality checks
|
||||
|
||||
## CI Alignment
|
||||
|
||||
This skill is specifically designed to match GitHub Actions CodeQL workflow:
|
||||
|
||||
| Parameter | Local | CI | Aligned |
|
||||
|-----------|-------|-----|---------|
|
||||
| Query Suite | security-and-quality | security-and-quality | ✅ |
|
||||
| Go Queries | 61 | 61 | ✅ |
|
||||
| JS Queries | 204 | 204 | ✅ |
|
||||
| Threading | auto | auto | ✅ |
|
||||
| Baseline Info | enabled | enabled | ✅ |
|
||||
|
||||
## Viewing Results
|
||||
|
||||
### VS Code SARIF Viewer (Recommended)
|
||||
|
||||
1. Install extension: `MS-SarifVSCode.sarif-viewer`
|
||||
2. Open `codeql-results-go.sarif` or `codeql-results-js.sarif`
|
||||
3. Navigate findings with inline annotations
|
||||
|
||||
### Command Line (jq)
|
||||
|
||||
```bash
|
||||
# Count findings
|
||||
jq '.runs[].results | length' codeql-results-go.sarif
|
||||
|
||||
# List findings
|
||||
jq -r '.runs[].results[] | "\(.level): \(.message.text)"' codeql-results-go.sarif
|
||||
```
|
||||
|
||||
### GitHub Security Tab
|
||||
|
||||
SARIF files are automatically uploaded to GitHub Security tab in CI.
|
||||
|
||||
## Performance
|
||||
|
||||
| Language | Database Creation | Analysis | Total |
|
||||
|----------|------------------|----------|-------|
|
||||
| Go | ~30s | ~30s | ~60s |
|
||||
| JavaScript | ~45s | ~45s | ~90s |
|
||||
| All | ~75s | ~75s | ~150s |
|
||||
|
||||
**Note:** First run downloads query packs; subsequent runs are faster.
|
||||
|
||||
## Notes
|
||||
|
||||
- Requires CodeQL CLI 2.17.0+ (use `gh codeql set-version latest` to upgrade)
|
||||
- Databases are regenerated each run (not cached)
|
||||
- SARIF files are gitignored (see `.gitignore`)
|
||||
- Query results may vary between CodeQL versions
|
||||
- Use `.codeql/` directory for custom queries or suppressions
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-24
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: CodeQL CLI + GitHub Query Packs
|
||||
+263
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Scan Docker Image - Execution Script
|
||||
#
|
||||
# Build Docker image and scan with Grype/Syft matching CI supply chain verification
|
||||
# This script replicates the exact process from supply-chain-pr.yml workflow
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
|
||||
# Check Docker
|
||||
validate_docker_environment || error_exit "Docker is required but not available"
|
||||
|
||||
# Check Syft
|
||||
if ! command -v syft >/dev/null 2>&1; then
|
||||
log_error "Syft not found - install from: https://github.com/anchore/syft"
|
||||
log_error "Installation: curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.17.0"
|
||||
error_exit "Syft is required for SBOM generation" 2
|
||||
fi
|
||||
|
||||
# Check Grype
|
||||
if ! command -v grype >/dev/null 2>&1; then
|
||||
log_error "Grype not found - install from: https://github.com/anchore/grype"
|
||||
log_error "Installation: curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.85.0"
|
||||
error_exit "Grype is required for vulnerability scanning" 2
|
||||
fi
|
||||
|
||||
# Check jq
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
log_error "jq not found - install from package manager (apt-get install jq, brew install jq, etc.)"
|
||||
error_exit "jq is required for JSON processing" 2
|
||||
fi
|
||||
|
||||
# Verify tool versions match CI
|
||||
SYFT_INSTALLED_VERSION=$(syft version | grep -oP 'Version:\s*\Kv?[0-9]+\.[0-9]+\.[0-9]+' | head -1 || echo "unknown")
|
||||
GRYPE_INSTALLED_VERSION=$(grype version | grep -oP 'Version:\s*\Kv?[0-9]+\.[0-9]+\.[0-9]+' | head -1 || echo "unknown")
|
||||
|
||||
# Set defaults matching CI workflow
|
||||
set_default_env "SYFT_VERSION" "v1.17.0"
|
||||
set_default_env "GRYPE_VERSION" "v0.85.0"
|
||||
set_default_env "IMAGE_TAG" "charon:local"
|
||||
set_default_env "FAIL_ON_SEVERITY" "Critical,High"
|
||||
|
||||
# Version check (informational only)
|
||||
log_info "Installed Syft version: ${SYFT_INSTALLED_VERSION}"
|
||||
log_info "Expected Syft version: ${SYFT_VERSION}"
|
||||
if [[ "${SYFT_INSTALLED_VERSION}" != "${SYFT_VERSION#v}" ]] && [[ "${SYFT_INSTALLED_VERSION}" != "${SYFT_VERSION}" ]]; then
|
||||
log_warning "Syft version mismatch - CI uses ${SYFT_VERSION}, you have ${SYFT_INSTALLED_VERSION}"
|
||||
log_warning "Results may differ from CI. Reinstall with: curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin ${SYFT_VERSION}"
|
||||
fi
|
||||
|
||||
log_info "Installed Grype version: ${GRYPE_INSTALLED_VERSION}"
|
||||
log_info "Expected Grype version: ${GRYPE_VERSION}"
|
||||
if [[ "${GRYPE_INSTALLED_VERSION}" != "${GRYPE_VERSION#v}" ]] && [[ "${GRYPE_INSTALLED_VERSION}" != "${GRYPE_VERSION}" ]]; then
|
||||
log_warning "Grype version mismatch - CI uses ${GRYPE_VERSION}, you have ${GRYPE_INSTALLED_VERSION}"
|
||||
log_warning "Results may differ from CI. Reinstall with: curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin ${GRYPE_VERSION}"
|
||||
fi
|
||||
|
||||
# Parse arguments
|
||||
IMAGE_TAG="${1:-${IMAGE_TAG}}"
|
||||
NO_CACHE_FLAG=""
|
||||
if [[ "${2:-}" == "no-cache" ]]; then
|
||||
NO_CACHE_FLAG="--no-cache"
|
||||
log_info "Building without cache (clean build)"
|
||||
fi
|
||||
|
||||
log_info "Image tag: ${IMAGE_TAG}"
|
||||
log_info "Fail on severity: ${FAIL_ON_SEVERITY}"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 1: Build Docker Image
|
||||
# ==============================================================================
|
||||
log_step "BUILD" "Building Docker image: ${IMAGE_TAG}"
|
||||
|
||||
# Get build metadata
|
||||
VERSION="${VERSION:-dev}"
|
||||
BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
VCS_REF=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
|
||||
|
||||
log_info "Build args: VERSION=${VERSION}, BUILD_DATE=${BUILD_DATE}, VCS_REF=${VCS_REF}"
|
||||
|
||||
# Build Docker image with same args as CI
|
||||
if docker build ${NO_CACHE_FLAG} \
|
||||
--build-arg VERSION="${VERSION}" \
|
||||
--build-arg BUILD_DATE="${BUILD_DATE}" \
|
||||
--build-arg VCS_REF="${VCS_REF}" \
|
||||
-t "${IMAGE_TAG}" \
|
||||
-f Dockerfile \
|
||||
.; then
|
||||
log_success "Docker image built successfully: ${IMAGE_TAG}"
|
||||
else
|
||||
error_exit "Docker build failed" 2
|
||||
fi
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 2: Generate SBOM
|
||||
# ==============================================================================
|
||||
log_step "SBOM" "Generating SBOM using Syft ${SYFT_VERSION}"
|
||||
|
||||
log_info "Scanning image: ${IMAGE_TAG}"
|
||||
log_info "Format: CycloneDX JSON (matches CI)"
|
||||
|
||||
# Generate SBOM from the Docker IMAGE (not filesystem)
|
||||
if syft "${IMAGE_TAG}" \
|
||||
--output cyclonedx-json=sbom.cyclonedx.json \
|
||||
--output table; then
|
||||
log_success "SBOM generation complete"
|
||||
else
|
||||
error_exit "SBOM generation failed" 2
|
||||
fi
|
||||
|
||||
# Count components in SBOM
|
||||
COMPONENT_COUNT=$(jq '.components | length' sbom.cyclonedx.json 2>/dev/null || echo "0")
|
||||
log_info "Generated SBOM contains ${COMPONENT_COUNT} packages"
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 3: Scan for Vulnerabilities
|
||||
# ==============================================================================
|
||||
log_step "SCAN" "Scanning for vulnerabilities using Grype ${GRYPE_VERSION}"
|
||||
|
||||
log_info "Scanning SBOM against vulnerability database..."
|
||||
log_info "This may take 30-60 seconds on first run (database download)"
|
||||
|
||||
# Run Grype against the SBOM (generated from image, not filesystem)
|
||||
# This matches exactly what CI does in supply-chain-pr.yml
|
||||
if grype sbom:sbom.cyclonedx.json \
|
||||
--output json \
|
||||
--file grype-results.json; then
|
||||
log_success "Vulnerability scan complete"
|
||||
else
|
||||
log_warning "Grype scan completed with findings"
|
||||
fi
|
||||
|
||||
# Generate SARIF output for GitHub Security (matches CI)
|
||||
grype sbom:sbom.cyclonedx.json \
|
||||
--output sarif \
|
||||
--file grype-results.sarif 2>/dev/null || true
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 4: Analyze Results
|
||||
# ==============================================================================
|
||||
log_step "ANALYSIS" "Analyzing vulnerability scan results"
|
||||
|
||||
# Count vulnerabilities by severity (matches CI logic)
|
||||
if [[ -f grype-results.json ]]; then
|
||||
CRITICAL_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Critical")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
HIGH_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "High")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
MEDIUM_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Medium")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
LOW_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Low")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
NEGLIGIBLE_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Negligible")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
UNKNOWN_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Unknown")] | length' grype-results.json 2>/dev/null || echo "0")
|
||||
TOTAL_COUNT=$(jq '.matches | length' grype-results.json 2>/dev/null || echo "0")
|
||||
else
|
||||
CRITICAL_COUNT=0
|
||||
HIGH_COUNT=0
|
||||
MEDIUM_COUNT=0
|
||||
LOW_COUNT=0
|
||||
NEGLIGIBLE_COUNT=0
|
||||
UNKNOWN_COUNT=0
|
||||
TOTAL_COUNT=0
|
||||
fi
|
||||
|
||||
# Display vulnerability summary
|
||||
echo ""
|
||||
log_info "Vulnerability Summary:"
|
||||
echo " 🔴 Critical: ${CRITICAL_COUNT}"
|
||||
echo " 🟠 High: ${HIGH_COUNT}"
|
||||
echo " 🟡 Medium: ${MEDIUM_COUNT}"
|
||||
echo " 🟢 Low: ${LOW_COUNT}"
|
||||
if [[ ${NEGLIGIBLE_COUNT} -gt 0 ]]; then
|
||||
echo " ⚪ Negligible: ${NEGLIGIBLE_COUNT}"
|
||||
fi
|
||||
if [[ ${UNKNOWN_COUNT} -gt 0 ]]; then
|
||||
echo " ❓ Unknown: ${UNKNOWN_COUNT}"
|
||||
fi
|
||||
echo " 📊 Total: ${TOTAL_COUNT}"
|
||||
echo ""
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 5: Detailed Reporting
|
||||
# ==============================================================================
|
||||
|
||||
# Show Critical vulnerabilities if any
|
||||
if [[ ${CRITICAL_COUNT} -gt 0 ]]; then
|
||||
log_error "Critical Severity Vulnerabilities Found:"
|
||||
echo ""
|
||||
jq -r '.matches[] | select(.vulnerability.severity == "Critical") |
|
||||
" - \(.vulnerability.id) in \(.artifact.name)\n Package: \(.artifact.name)@\(.artifact.version)\n Fixed: \(.vulnerability.fix.versions[0] // "No fix available")\n CVSS: \(.vulnerability.cvss[0].metrics.baseScore // "N/A")\n Description: \(.vulnerability.description[0:100])...\n"' \
|
||||
grype-results.json 2>/dev/null || echo " (Unable to parse details)"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Show High vulnerabilities if any
|
||||
if [[ ${HIGH_COUNT} -gt 0 ]]; then
|
||||
log_warning "High Severity Vulnerabilities Found:"
|
||||
echo ""
|
||||
jq -r '.matches[] | select(.vulnerability.severity == "High") |
|
||||
" - \(.vulnerability.id) in \(.artifact.name)\n Package: \(.artifact.name)@\(.artifact.version)\n Fixed: \(.vulnerability.fix.versions[0] // "No fix available")\n CVSS: \(.vulnerability.cvss[0].metrics.baseScore // "N/A")\n Description: \(.vulnerability.description[0:100])...\n"' \
|
||||
grype-results.json 2>/dev/null || echo " (Unable to parse details)"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ==============================================================================
|
||||
# Phase 6: Exit Code Determination (Matches CI)
|
||||
# ==============================================================================
|
||||
|
||||
# Check if any failing severities were found
|
||||
SHOULD_FAIL=false
|
||||
|
||||
if [[ "${FAIL_ON_SEVERITY}" == *"Critical"* ]] && [[ ${CRITICAL_COUNT} -gt 0 ]]; then
|
||||
SHOULD_FAIL=true
|
||||
fi
|
||||
|
||||
if [[ "${FAIL_ON_SEVERITY}" == *"High"* ]] && [[ ${HIGH_COUNT} -gt 0 ]]; then
|
||||
SHOULD_FAIL=true
|
||||
fi
|
||||
|
||||
if [[ "${FAIL_ON_SEVERITY}" == *"Medium"* ]] && [[ ${MEDIUM_COUNT} -gt 0 ]]; then
|
||||
SHOULD_FAIL=true
|
||||
fi
|
||||
|
||||
if [[ "${FAIL_ON_SEVERITY}" == *"Low"* ]] && [[ ${LOW_COUNT} -gt 0 ]]; then
|
||||
SHOULD_FAIL=true
|
||||
fi
|
||||
|
||||
# Final summary and exit
|
||||
echo ""
|
||||
log_info "Generated artifacts:"
|
||||
log_info " - sbom.cyclonedx.json (SBOM)"
|
||||
log_info " - grype-results.json (vulnerability details)"
|
||||
log_info " - grype-results.sarif (GitHub Security format)"
|
||||
echo ""
|
||||
|
||||
if [[ "${SHOULD_FAIL}" == "true" ]]; then
|
||||
log_error "Found ${CRITICAL_COUNT} Critical and ${HIGH_COUNT} High severity vulnerabilities"
|
||||
log_error "These issues must be resolved before deployment"
|
||||
log_error "Review grype-results.json for detailed remediation guidance"
|
||||
exit 1
|
||||
else
|
||||
if [[ ${TOTAL_COUNT} -gt 0 ]]; then
|
||||
log_success "Docker image scan complete - no critical or high vulnerabilities"
|
||||
log_info "Found ${MEDIUM_COUNT} Medium and ${LOW_COUNT} Low severity issues (non-blocking)"
|
||||
else
|
||||
log_success "Docker image scan complete - no vulnerabilities found"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
@@ -0,0 +1,601 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-scan-docker-image"
|
||||
version: "1.0.0"
|
||||
description: "Build Docker image and scan with Grype/Syft matching CI supply chain verification"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "scanning"
|
||||
- "docker"
|
||||
- "supply-chain"
|
||||
- "vulnerabilities"
|
||||
- "sbom"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
- name: "syft"
|
||||
version: ">=1.17.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/anchore/syft"
|
||||
- name: "grype"
|
||||
version: ">=0.85.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/anchore/grype"
|
||||
- name: "jq"
|
||||
version: ">=1.6"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "SYFT_VERSION"
|
||||
description: "Syft version to use for SBOM generation"
|
||||
default: "v1.17.0"
|
||||
required: false
|
||||
- name: "GRYPE_VERSION"
|
||||
description: "Grype version to use for vulnerability scanning"
|
||||
default: "v0.85.0"
|
||||
required: false
|
||||
- name: "IMAGE_TAG"
|
||||
description: "Docker image tag to build and scan"
|
||||
default: "charon:local"
|
||||
required: false
|
||||
- name: "FAIL_ON_SEVERITY"
|
||||
description: "Comma-separated list of severities that cause failure"
|
||||
default: "Critical,High"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "image_tag"
|
||||
type: "string"
|
||||
description: "Docker image tag to build and scan"
|
||||
default: "charon:local"
|
||||
required: false
|
||||
- name: "no_cache"
|
||||
type: "boolean"
|
||||
description: "Build Docker image without cache"
|
||||
default: false
|
||||
required: false
|
||||
outputs:
|
||||
- name: "sbom_file"
|
||||
type: "file"
|
||||
description: "Generated SBOM in CycloneDX JSON format"
|
||||
- name: "scan_results"
|
||||
type: "file"
|
||||
description: "Grype vulnerability scan results in JSON format"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if no critical/high issues, 1 if issues found, 2 if build/scan failed"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "supply-chain"
|
||||
execution_time: "long"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: false
|
||||
exit_codes:
|
||||
0: "Scan successful, no critical or high vulnerabilities"
|
||||
1: "Critical or high severity vulnerabilities found"
|
||||
2: "Build failed or scan error"
|
||||
---
|
||||
|
||||
# Security: Scan Docker Image (Local)
|
||||
|
||||
## Overview
|
||||
|
||||
**CRITICAL GAP ADDRESSED**: This skill closes a critical security gap discovered in the Charon project's local development workflow. While the existing Trivy filesystem scanner catches some issues, it misses vulnerabilities that only exist in the actual built Docker image, including:
|
||||
|
||||
- **Alpine package vulnerabilities** in the base image
|
||||
- **Compiled binary vulnerabilities** in Go dependencies
|
||||
- **Embedded dependencies** that only exist post-build
|
||||
- **Multi-stage build artifacts** not present in source
|
||||
- **Runtime dependencies** added during Docker build
|
||||
|
||||
This skill replicates the **exact CI supply chain verification process** used in the `supply-chain-pr.yml` workflow, ensuring local scans match CI scans precisely. This prevents the "works locally but fails in CI" scenario and catches image-only vulnerabilities before they reach production.
|
||||
|
||||
## Key Differences from Trivy Filesystem Scan
|
||||
|
||||
| Aspect | Trivy (Filesystem) | This Skill (Image Scan) |
|
||||
|--------|-------------------|------------------------|
|
||||
| **Scan Target** | Source code + dependencies | Built Docker image |
|
||||
| **Alpine Packages** | ❌ Not detected | ✅ Detected |
|
||||
| **Compiled Binaries** | ❌ Not detected | ✅ Detected |
|
||||
| **Build Artifacts** | ❌ Not detected | ✅ Detected |
|
||||
| **CI Alignment** | ⚠️ Different results | ✅ Exact match |
|
||||
| **Supply Chain** | Partial coverage | Full coverage |
|
||||
|
||||
## Features
|
||||
|
||||
- **Exact CI Matching**: Uses same Syft and Grype versions as supply-chain-pr.yml
|
||||
- **Image-Based Scanning**: Scans the actual Docker image, not just filesystem
|
||||
- **SBOM Generation**: Creates CycloneDX JSON SBOM from the built image
|
||||
- **Severity-Based Failures**: Fails on Critical/High severity by default
|
||||
- **Detailed Reporting**: Counts vulnerabilities by severity
|
||||
- **Build Integration**: Builds the Docker image first, ensuring latest code
|
||||
- **Idempotent Scans**: Can be run repeatedly with consistent results
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- Syft 1.17.0 or higher (auto-checked, installation instructions provided)
|
||||
- Grype 0.85.0 or higher (auto-checked, installation instructions provided)
|
||||
- jq 1.6 or higher (for JSON processing)
|
||||
- Internet connection (for vulnerability database updates)
|
||||
- Sufficient disk space for Docker image build (~2GB recommended)
|
||||
|
||||
## Installation
|
||||
|
||||
### Install Syft
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.17.0
|
||||
|
||||
# Or via package manager
|
||||
brew install syft # macOS
|
||||
```
|
||||
|
||||
### Install Grype
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.85.0
|
||||
|
||||
# Or via package manager
|
||||
brew install grype # macOS
|
||||
```
|
||||
|
||||
### Verify Installation
|
||||
|
||||
```bash
|
||||
syft version
|
||||
grype version
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage (Default Image Tag)
|
||||
|
||||
Build and scan the default `charon:local` image:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
```
|
||||
|
||||
### Custom Image Tag
|
||||
|
||||
Build and scan a custom-tagged image:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image charon:test
|
||||
```
|
||||
|
||||
### No-Cache Build
|
||||
|
||||
Force a clean build without Docker cache:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image charon:local no-cache
|
||||
```
|
||||
|
||||
### Environment Variable Overrides
|
||||
|
||||
Override default versions or behavior:
|
||||
|
||||
```bash
|
||||
# Use specific tool versions
|
||||
SYFT_VERSION=v1.17.0 GRYPE_VERSION=v0.85.0 \
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
|
||||
# Change failure threshold
|
||||
FAIL_ON_SEVERITY="Critical" \
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| image_tag | string | No | charon:local | Docker image tag to build and scan |
|
||||
| no_cache | boolean | No | false | Build without Docker cache (pass "no-cache" as second arg) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| SYFT_VERSION | No | v1.17.0 | Syft version (matches CI) |
|
||||
| GRYPE_VERSION | No | v0.85.0 | Grype version (matches CI) |
|
||||
| IMAGE_TAG | No | charon:local | Default image tag if not provided |
|
||||
| FAIL_ON_SEVERITY | No | Critical,High | Severities that cause exit code 1 |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Generated Files
|
||||
|
||||
- **`sbom.cyclonedx.json`**: SBOM in CycloneDX JSON format (industry standard)
|
||||
- **`grype-results.json`**: Detailed vulnerability scan results
|
||||
- **`grype-results.sarif`**: SARIF format for GitHub Security integration
|
||||
|
||||
### Exit Codes
|
||||
|
||||
- **0**: Scan completed successfully, no critical/high vulnerabilities
|
||||
- **1**: Critical or high severity vulnerabilities found (blocking)
|
||||
- **2**: Docker build failed or scan error
|
||||
|
||||
### Output Format
|
||||
|
||||
```
|
||||
[INFO] Building Docker image: charon:local...
|
||||
[BUILD] Using Dockerfile with multi-stage build
|
||||
[BUILD] Image built successfully: charon:local
|
||||
|
||||
[SBOM] Generating SBOM using Syft v1.17.0...
|
||||
[SBOM] Generated SBOM contains 247 packages
|
||||
|
||||
[SCAN] Scanning for vulnerabilities using Grype v0.85.0...
|
||||
[SCAN] Vulnerability Summary:
|
||||
🔴 Critical: 0
|
||||
🟠 High: 0
|
||||
🟡 Medium: 15
|
||||
🟢 Low: 42
|
||||
📊 Total: 57
|
||||
|
||||
[SUCCESS] Docker image scan complete - no critical or high vulnerabilities
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Standard Local Scan
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
[INFO] Building Docker image: charon:local...
|
||||
[BUILD] Step 1/25 : FROM node:24.13.0-alpine AS frontend-builder
|
||||
[BUILD] ...
|
||||
[BUILD] Successfully built abc123def456
|
||||
[BUILD] Successfully tagged charon:local
|
||||
|
||||
[SBOM] Generating SBOM using Syft v1.17.0...
|
||||
[SBOM] Scanning image: charon:local
|
||||
[SBOM] Generated SBOM contains 247 packages
|
||||
|
||||
[SCAN] Scanning for vulnerabilities using Grype v0.85.0...
|
||||
[SCAN] Vulnerability Summary:
|
||||
🔴 Critical: 0
|
||||
🟠 High: 2
|
||||
🟡 Medium: 15
|
||||
🟢 Low: 42
|
||||
📊 Total: 59
|
||||
|
||||
[SCAN] High Severity Vulnerabilities:
|
||||
- CVE-2024-12345 in alpine-baselayout (CVSS: 7.5)
|
||||
Package: alpine-baselayout@3.23.0
|
||||
Fixed: alpine-baselayout@3.23.1
|
||||
Description: Arbitrary file read vulnerability
|
||||
|
||||
- CVE-2024-67890 in busybox (CVSS: 8.2)
|
||||
Package: busybox@1.36.1
|
||||
Fixed: busybox@1.36.2
|
||||
Description: Remote code execution via crafted input
|
||||
|
||||
[ERROR] Found 2 High severity vulnerabilities - please review and remediate
|
||||
Exit code: 1
|
||||
```
|
||||
|
||||
### Example 2: Clean Build After Code Changes
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-scan-docker-image charon:test no-cache
|
||||
[INFO] Building Docker image: charon:test (no cache)...
|
||||
[BUILD] Building without cache to ensure fresh dependencies...
|
||||
[BUILD] Successfully built and tagged charon:test
|
||||
|
||||
[SBOM] Generating SBOM...
|
||||
[SBOM] Generated SBOM contains 248 packages (+1 from previous scan)
|
||||
|
||||
[SCAN] Scanning for vulnerabilities...
|
||||
[SCAN] Vulnerability Summary:
|
||||
🔴 Critical: 0
|
||||
🟠 High: 0
|
||||
🟡 Medium: 16
|
||||
🟢 Low: 43
|
||||
📊 Total: 59
|
||||
|
||||
[SUCCESS] Docker image scan complete - no critical or high vulnerabilities
|
||||
Exit code: 0
|
||||
```
|
||||
|
||||
### Example 3: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# .github/workflows/local-verify.yml (example)
|
||||
- name: Scan Docker Image Locally
|
||||
run: .github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
continue-on-error: false
|
||||
|
||||
- name: Upload SBOM Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: local-sbom
|
||||
path: sbom.cyclonedx.json
|
||||
```
|
||||
|
||||
### Example 4: Pre-Commit Hook Integration
|
||||
|
||||
```bash
|
||||
# .git/hooks/pre-push
|
||||
#!/bin/bash
|
||||
echo "Running local Docker image security scan..."
|
||||
if ! .github/skills/scripts/skill-runner.sh security-scan-docker-image; then
|
||||
echo "❌ Security scan failed - please fix vulnerabilities before pushing"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Build Phase
|
||||
|
||||
1. **Docker Build**: Builds the Docker image using the project's Dockerfile
|
||||
- Uses multi-stage build for frontend and backend
|
||||
- Applies build args: VERSION, BUILD_DATE, VCS_REF
|
||||
- Tags with specified image tag (default: charon:local)
|
||||
|
||||
### SBOM Generation Phase
|
||||
|
||||
2. **Image Analysis**: Syft analyzes the built Docker image (not filesystem)
|
||||
- Scans all layers in the final image
|
||||
- Detects Alpine packages, Go modules, npm packages
|
||||
- Identifies compiled binaries and their dependencies
|
||||
- Catalogs runtime dependencies added during build
|
||||
|
||||
3. **SBOM Creation**: Generates CycloneDX JSON SBOM
|
||||
- Industry-standard format for supply chain visibility
|
||||
- Contains full package inventory with versions
|
||||
- Includes checksums and license information
|
||||
|
||||
### Vulnerability Scanning Phase
|
||||
|
||||
4. **Database Update**: Grype updates its vulnerability database
|
||||
- Fetches latest CVE information
|
||||
- Ensures scan uses current vulnerability data
|
||||
|
||||
5. **Image Scan**: Grype scans the SBOM against vulnerability database
|
||||
- Matches packages against known CVEs
|
||||
- Calculates CVSS scores for each vulnerability
|
||||
- Generates SARIF output for GitHub Security
|
||||
|
||||
6. **Severity Analysis**: Counts vulnerabilities by severity
|
||||
- Critical: CVSS 9.0-10.0
|
||||
- High: CVSS 7.0-8.9
|
||||
- Medium: CVSS 4.0-6.9
|
||||
- Low: CVSS 0.1-3.9
|
||||
|
||||
### Reporting Phase
|
||||
|
||||
7. **Results Summary**: Displays vulnerability counts and details
|
||||
8. **Exit Code**: Returns appropriate exit code based on severity findings
|
||||
|
||||
## Vulnerability Severity Thresholds
|
||||
|
||||
**Project Standards (Matches CI)**:
|
||||
|
||||
| Severity | CVSS Range | Action | Exit Code |
|
||||
|----------|-----------|--------|-----------|
|
||||
| 🔴 **CRITICAL** | 9.0-10.0 | **MUST FIX** - Blocks commit/push | 1 |
|
||||
| 🟠 **HIGH** | 7.0-8.9 | **SHOULD FIX** - Blocks commit/push | 1 |
|
||||
| 🟡 **MEDIUM** | 4.0-6.9 | Fix in next release (logged) | 0 |
|
||||
| 🟢 **LOW** | 0.1-3.9 | Optional, fix as time permits | 0 |
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Docker not running**:
|
||||
```bash
|
||||
[ERROR] Docker daemon is not running
|
||||
Solution: Start Docker Desktop or Docker service
|
||||
```
|
||||
|
||||
**Syft not installed**:
|
||||
```bash
|
||||
[ERROR] Syft not found - install from: https://github.com/anchore/syft
|
||||
Solution: Install Syft v1.17.0 using installation instructions above
|
||||
```
|
||||
|
||||
**Grype not installed**:
|
||||
```bash
|
||||
[ERROR] Grype not found - install from: https://github.com/anchore/grype
|
||||
Solution: Install Grype v0.85.0 using installation instructions above
|
||||
```
|
||||
|
||||
**Build failure**:
|
||||
```bash
|
||||
[ERROR] Docker build failed with exit code 1
|
||||
Solution: Check Dockerfile syntax and dependency availability
|
||||
```
|
||||
|
||||
**Network timeout (vulnerability scan)**:
|
||||
```bash
|
||||
[WARNING] Failed to update Grype vulnerability database
|
||||
Solution: Check internet connection or retry later
|
||||
```
|
||||
|
||||
**Disk space insufficient**:
|
||||
```bash
|
||||
[ERROR] No space left on device
|
||||
Solution: Clean up Docker images and containers: docker system prune -a
|
||||
```
|
||||
|
||||
## Integration with Definition of Done
|
||||
|
||||
This skill is **MANDATORY** in the Management agent's Definition of Done checklist:
|
||||
|
||||
### When to Run
|
||||
|
||||
- ✅ **Before every commit** that changes application code
|
||||
- ✅ **After dependency updates** (Go modules, npm packages)
|
||||
- ✅ **Before creating a Pull Request**
|
||||
- ✅ **After Dockerfile modifications**
|
||||
- ✅ **Before release/tag creation**
|
||||
|
||||
### QA_Security Requirements
|
||||
|
||||
The QA_Security agent **MUST**:
|
||||
|
||||
1. Run this skill after running Trivy filesystem scan
|
||||
2. Verify that both scans pass with zero Critical/High issues
|
||||
3. Document any differences between filesystem and image scans
|
||||
4. Block approval if image scan reveals additional vulnerabilities
|
||||
5. Report findings in the QA report at `docs/reports/qa_report.md`
|
||||
|
||||
### Why This is Critical
|
||||
|
||||
**Image-only vulnerabilities** can exist even when filesystem scans pass:
|
||||
|
||||
- Alpine base image CVEs (e.g., musl, busybox, apk-tools)
|
||||
- Compiled Go binary vulnerabilities (e.g., stdlib CVEs)
|
||||
- Caddy plugin vulnerabilities added during build
|
||||
- Multi-stage build artifacts with known issues
|
||||
|
||||
**Without this scan**, these vulnerabilities reach production undetected.
|
||||
|
||||
## Comparison with CI Supply Chain Workflow
|
||||
|
||||
This skill **exactly replicates** the supply-chain-pr.yml workflow:
|
||||
|
||||
| Step | CI Workflow | This Skill | Match |
|
||||
|------|------------|------------|-------|
|
||||
| Build Image | ✅ Docker build | ✅ Docker build | ✅ |
|
||||
| Load Image | ✅ Load from artifact | ✅ Use built image | ✅ |
|
||||
| Syft Version | v1.17.0 | v1.17.0 | ✅ |
|
||||
| Grype Version | v0.85.0 | v0.85.0 | ✅ |
|
||||
| SBOM Format | CycloneDX JSON | CycloneDX JSON | ✅ |
|
||||
| Scan Target | Docker image | Docker image | ✅ |
|
||||
| Severity Counts | Critical/High/Medium/Low | Critical/High/Medium/Low | ✅ |
|
||||
| Exit on Critical/High | Yes | Yes | ✅ |
|
||||
| SARIF Output | Yes | Yes | ✅ |
|
||||
|
||||
**Guarantee**: If this skill passes locally, the CI supply chain workflow will pass (assuming same code/dependencies).
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Filesystem vulnerability scan (complementary)
|
||||
- [security-verify-sbom](./security-verify-sbom.SKILL.md) - SBOM verification and comparison
|
||||
- [security-sign-cosign](./security-sign-cosign.SKILL.md) - Sign artifacts with Cosign
|
||||
- [security-slsa-provenance](./security-slsa-provenance.SKILL.md) - Generate SLSA provenance
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
### Recommended Execution Order
|
||||
|
||||
1. **Trivy Filesystem Scan** - Fast, catches obvious issues
|
||||
2. **Docker Image Scan (this skill)** - Comprehensive, catches image-only issues
|
||||
3. **CodeQL Scans** - Static analysis for code quality
|
||||
4. **SBOM Verification** - Supply chain drift detection
|
||||
|
||||
### Combined DoD Checklist
|
||||
|
||||
```bash
|
||||
# 1. Filesystem scan (fast)
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
|
||||
# 2. Image scan (comprehensive) - THIS SKILL
|
||||
.github/skills/scripts/skill-runner.sh security-scan-docker-image
|
||||
|
||||
# 3. Code analysis
|
||||
.github/skills/scripts/skill-runner.sh security-scan-codeql
|
||||
|
||||
# 4. Go vulnerabilities
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Execution Time
|
||||
|
||||
- **Docker Build**: 2-5 minutes (cached), 5-10 minutes (no-cache)
|
||||
- **SBOM Generation**: 30-60 seconds
|
||||
- **Vulnerability Scan**: 30-60 seconds
|
||||
- **Total**: ~3-7 minutes (typical), ~6-12 minutes (no-cache)
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use Docker layer caching** (default) for faster builds
|
||||
2. **Run after code changes only** (not needed for doc-only changes)
|
||||
3. **Parallelize with other scans** (Trivy, CodeQL) for efficiency
|
||||
4. **Cache vulnerability database** (Grype auto-caches)
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- SBOM files contain full package inventory (treat as sensitive)
|
||||
- Vulnerability results may contain CVE details (secure storage)
|
||||
- Never commit scan results with credentials/tokens
|
||||
- Review all Critical/High findings before production deployment
|
||||
- Keep Syft and Grype updated to latest versions
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Always Fails
|
||||
|
||||
Check Dockerfile syntax and build context:
|
||||
|
||||
```bash
|
||||
# Test build manually
|
||||
docker build -t charon:test .
|
||||
|
||||
# Check build args
|
||||
docker build --build-arg VERSION=test -t charon:test .
|
||||
```
|
||||
|
||||
### Scan Detects False Positives
|
||||
|
||||
Create `.grype.yaml` in project root to suppress known false positives:
|
||||
|
||||
```yaml
|
||||
ignore:
|
||||
- vulnerability: CVE-2024-12345
|
||||
fix-state: wont-fix
|
||||
```
|
||||
|
||||
### Different Results Than CI
|
||||
|
||||
Verify versions match:
|
||||
|
||||
```bash
|
||||
syft version # Should be v1.17.0
|
||||
grype version # Should be v0.85.0
|
||||
```
|
||||
|
||||
Update if needed:
|
||||
|
||||
```bash
|
||||
# Reinstall specific versions
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.17.0
|
||||
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.85.0
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- This skill is **not idempotent** due to Docker build step
|
||||
- Scan results may vary as vulnerability database updates
|
||||
- Some vulnerabilities may have no fix available yet
|
||||
- Alpine base image updates may resolve multiple CVEs
|
||||
- Go stdlib updates may resolve compiled binary CVEs
|
||||
- Network access required for database updates
|
||||
- Recommended to run before each commit/push
|
||||
- Complements but does not replace Trivy filesystem scan
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-16
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: Syft (SBOM) + Grype (Vulnerability Scanning)
|
||||
**CI Workflow**: `.github/workflows/supply-chain-pr.yml`
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Scan Go Vulnerability - Execution Script
|
||||
#
|
||||
# This script wraps the Go vulnerability checker (govulncheck) to detect
|
||||
# known vulnerabilities in Go code and dependencies.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_go_environment "1.23" || error_exit "Go 1.23+ is required"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "GOVULNCHECK_FORMAT" "text"
|
||||
|
||||
# Parse arguments
|
||||
FORMAT="${1:-${GOVULNCHECK_FORMAT}}"
|
||||
MODE="${2:-source}"
|
||||
|
||||
# Validate format
|
||||
case "${FORMAT}" in
|
||||
text|json|sarif)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid format: ${FORMAT}. Must be one of: text, json, sarif"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Validate mode
|
||||
case "${MODE}" in
|
||||
source|binary)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid mode: ${MODE}. Must be one of: source, binary"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Change to backend directory
|
||||
cd "${PROJECT_ROOT}/backend"
|
||||
|
||||
# Check for go.mod
|
||||
if [[ ! -f "go.mod" ]]; then
|
||||
log_error "go.mod not found in backend directory"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Execute govulncheck
|
||||
log_step "SCANNING" "Running Go vulnerability check"
|
||||
log_info "Format: ${FORMAT}"
|
||||
log_info "Mode: ${MODE}"
|
||||
log_info "Working directory: $(pwd)"
|
||||
|
||||
# Build govulncheck command
|
||||
GOVULNCHECK_CMD="go run golang.org/x/vuln/cmd/govulncheck@latest"
|
||||
|
||||
# Add format flag if not text (text is default)
|
||||
if [[ "${FORMAT}" != "text" ]]; then
|
||||
GOVULNCHECK_CMD="${GOVULNCHECK_CMD} -format=${FORMAT}"
|
||||
fi
|
||||
|
||||
# Add mode flag if not source (source is default)
|
||||
if [[ "${MODE}" != "source" ]]; then
|
||||
GOVULNCHECK_CMD="${GOVULNCHECK_CMD} -mode=${MODE}"
|
||||
fi
|
||||
|
||||
# Add target (all packages)
|
||||
GOVULNCHECK_CMD="${GOVULNCHECK_CMD} ./..."
|
||||
|
||||
# Execute the scan
|
||||
if eval "${GOVULNCHECK_CMD}"; then
|
||||
log_success "No vulnerabilities found"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
if [[ ${exit_code} -eq 3 ]]; then
|
||||
log_error "Vulnerabilities detected (exit code 3)"
|
||||
log_info "Review the output above for details and remediation advice"
|
||||
else
|
||||
log_error "Vulnerability scan failed with exit code: ${exit_code}"
|
||||
fi
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,280 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-scan-go-vuln"
|
||||
version: "1.0.0"
|
||||
description: "Run Go vulnerability checker (govulncheck) to detect known vulnerabilities in Go code"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "vulnerabilities"
|
||||
- "go"
|
||||
- "govulncheck"
|
||||
- "scanning"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "go"
|
||||
version: ">=1.23"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "GOVULNCHECK_FORMAT"
|
||||
description: "Output format (text, json, sarif)"
|
||||
default: "text"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "format"
|
||||
type: "string"
|
||||
description: "Output format (text, json, sarif)"
|
||||
default: "text"
|
||||
required: false
|
||||
- name: "mode"
|
||||
type: "string"
|
||||
description: "Scan mode (source or binary)"
|
||||
default: "source"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "vulnerability_report"
|
||||
type: "stdout"
|
||||
description: "List of detected vulnerabilities with remediation advice"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if no vulnerabilities found, 3 if vulnerabilities detected"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "vulnerability"
|
||||
execution_time: "short"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Security Scan Go Vulnerability
|
||||
|
||||
## Overview
|
||||
|
||||
Executes `govulncheck` from the official Go vulnerability database to scan Go code and dependencies for known security vulnerabilities. This tool analyzes both direct and transitive dependencies, providing actionable remediation advice.
|
||||
|
||||
This skill is designed for CI/CD pipelines and pre-release security validation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Go 1.23 or higher installed and in PATH
|
||||
- Internet connection (for vulnerability database access)
|
||||
- Go module dependencies downloaded (`go mod download`)
|
||||
- Valid Go project with `go.mod` file
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run with default settings (text output, source mode):
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln
|
||||
```
|
||||
|
||||
### JSON Output
|
||||
|
||||
Get results in JSON format for parsing:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln json
|
||||
```
|
||||
|
||||
### SARIF Output
|
||||
|
||||
Get results in SARIF format for GitHub Code Scanning:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln sarif
|
||||
```
|
||||
|
||||
### Custom Format via Environment
|
||||
|
||||
```bash
|
||||
GOVULNCHECK_FORMAT=json .github/skills/scripts/skill-runner.sh security-scan-go-vuln
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| format | string | No | text | Output format (text, json, sarif) |
|
||||
| mode | string | No | source | Scan mode (source or binary) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| GOVULNCHECK_FORMAT | No | text | Output format override |
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 (no vulnerabilities found)
|
||||
- **Error Exit Codes**:
|
||||
- 1: Scan error or invalid arguments
|
||||
- 3: Vulnerabilities detected
|
||||
- **Output**: Vulnerability report to stdout
|
||||
|
||||
## Vulnerability Report Format
|
||||
|
||||
### Text Output (Default)
|
||||
|
||||
```
|
||||
Scanning for dependencies with known vulnerabilities...
|
||||
No vulnerabilities found.
|
||||
```
|
||||
|
||||
Or if vulnerabilities are found:
|
||||
|
||||
```
|
||||
Found 2 vulnerabilities in dependencies
|
||||
|
||||
Vulnerability #1: GO-2023-1234
|
||||
Package: github.com/example/vulnerable
|
||||
Version: v1.2.3
|
||||
Description: Buffer overflow in Parse function
|
||||
Fixed in: v1.2.4
|
||||
More info: https://vuln.go.dev/GO-2023-1234
|
||||
|
||||
Vulnerability #2: GO-2023-5678
|
||||
Package: golang.org/x/crypto/ssh
|
||||
Version: v0.1.0
|
||||
Description: Insecure default configuration
|
||||
Fixed in: v0.3.0
|
||||
More info: https://vuln.go.dev/GO-2023-5678
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Scan
|
||||
|
||||
```bash
|
||||
# Scan backend Go code for vulnerabilities
|
||||
cd backend
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Scanning your code and 125 packages across 23 dependent modules for known vulnerabilities...
|
||||
No vulnerabilities found.
|
||||
```
|
||||
|
||||
### Example 2: JSON Output for CI/CD
|
||||
|
||||
```bash
|
||||
# Get JSON output for automated processing
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln json > vuln-report.json
|
||||
```
|
||||
|
||||
### Example 3: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Check Go Vulnerabilities
|
||||
run: .github/skills/scripts/skill-runner.sh security-scan-go-vuln
|
||||
working-directory: backend
|
||||
|
||||
- name: Upload SARIF Report
|
||||
if: always()
|
||||
run: |
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln sarif > results.sarif
|
||||
# Upload to GitHub Code Scanning
|
||||
```
|
||||
|
||||
### Example 4: Binary Mode Scan
|
||||
|
||||
```bash
|
||||
# Scan a compiled binary
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln text binary
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Go not installed**:
|
||||
```bash
|
||||
Error: Go 1.23+ is required
|
||||
Solution: Install Go 1.23 or higher
|
||||
```
|
||||
|
||||
**Network unavailable**:
|
||||
```bash
|
||||
Error: Failed to fetch vulnerability database
|
||||
Solution: Check internet connection or proxy settings
|
||||
```
|
||||
|
||||
**Vulnerabilities found**:
|
||||
```bash
|
||||
Exit code: 3
|
||||
Solution: Review vulnerabilities and update affected packages
|
||||
```
|
||||
|
||||
**Module not found**:
|
||||
```bash
|
||||
Error: go.mod file not found
|
||||
Solution: Run from a valid Go module directory
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: No vulnerabilities found
|
||||
- **1**: Scan error or invalid arguments
|
||||
- **3**: Vulnerabilities detected (standard govulncheck exit code)
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Multi-language vulnerability scanning
|
||||
- [test-backend-coverage](./test-backend-coverage.SKILL.md) - Backend test coverage
|
||||
|
||||
## Notes
|
||||
|
||||
- `govulncheck` uses the official Go vulnerability database at https://vuln.go.dev
|
||||
- Database is automatically updated during each scan
|
||||
- Only checks vulnerabilities that are reachable from your code
|
||||
- Does not require building the code (analyzes source)
|
||||
- Can also scan compiled binaries with `--mode=binary`
|
||||
- Results may change as new vulnerabilities are published
|
||||
- Recommended to run before each release and in CI/CD
|
||||
- Zero false positives (only reports known CVEs)
|
||||
|
||||
## Remediation Workflow
|
||||
|
||||
When vulnerabilities are found:
|
||||
|
||||
1. **Review the Report**: Understand which packages are affected
|
||||
2. **Check Fix Availability**: Look for fixed versions in the report
|
||||
3. **Update Dependencies**: Run `go get -u` to update affected packages
|
||||
4. **Re-run Scan**: Verify vulnerabilities are resolved
|
||||
5. **Test**: Run full test suite after updates
|
||||
6. **Document**: Note any unresolvable vulnerabilities in security log
|
||||
|
||||
## Integration with GitHub Security
|
||||
|
||||
For SARIF output integration with GitHub Code Scanning:
|
||||
|
||||
```bash
|
||||
# Generate SARIF report
|
||||
.github/skills/scripts/skill-runner.sh security-scan-go-vuln sarif > govulncheck.sarif
|
||||
|
||||
# Upload to GitHub (requires GitHub CLI)
|
||||
gh api /repos/:owner/:repo/code-scanning/sarifs \
|
||||
-F sarif=@govulncheck.sarif \
|
||||
-F commit_sha=$GITHUB_SHA \
|
||||
-F ref=$GITHUB_REF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: `go run golang.org/x/vuln/cmd/govulncheck@latest`
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Scan Trivy - Execution Script
|
||||
#
|
||||
# This script wraps the Trivy Docker command to scan for vulnerabilities,
|
||||
# secrets, and misconfigurations.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_docker_environment || error_exit "Docker is required but not available"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "TRIVY_SEVERITY" "CRITICAL,HIGH,MEDIUM"
|
||||
set_default_env "TRIVY_TIMEOUT" "10m"
|
||||
|
||||
# Parse arguments
|
||||
# Default scanners exclude misconfig to avoid non-actionable policy bundle issues
|
||||
# that can cause scan errors unrelated to the repository contents.
|
||||
SCANNERS="${1:-vuln,secret}"
|
||||
FORMAT="${2:-table}"
|
||||
|
||||
# Validate format
|
||||
case "${FORMAT}" in
|
||||
table|json|sarif)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid format: ${FORMAT}. Must be one of: table, json, sarif"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# Validate scanners
|
||||
IFS=',' read -ra SCANNER_ARRAY <<< "${SCANNERS}"
|
||||
for scanner in "${SCANNER_ARRAY[@]}"; do
|
||||
case "${scanner}" in
|
||||
vuln|secret|misconfig)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid scanner: ${scanner}. Must be one of: vuln, secret, misconfig"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Execute Trivy scan
|
||||
log_step "SCANNING" "Running Trivy security scan"
|
||||
log_info "Scanners: ${SCANNERS}"
|
||||
log_info "Format: ${FORMAT}"
|
||||
log_info "Severity: ${TRIVY_SEVERITY}"
|
||||
log_info "Timeout: ${TRIVY_TIMEOUT}"
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Avoid scanning generated/cached artifacts that commonly contain fixture secrets,
|
||||
# non-Dockerfile files named like Dockerfiles, and large logs.
|
||||
SKIP_DIRS=(
|
||||
".git"
|
||||
".venv"
|
||||
".cache"
|
||||
"node_modules"
|
||||
"frontend/node_modules"
|
||||
"frontend/dist"
|
||||
"frontend/coverage"
|
||||
"test-results"
|
||||
"codeql-db-go"
|
||||
"codeql-db-js"
|
||||
"codeql-agent-results"
|
||||
"my-codeql-db"
|
||||
".trivy_logs"
|
||||
)
|
||||
|
||||
SKIP_DIR_FLAGS=()
|
||||
for d in "${SKIP_DIRS[@]}"; do
|
||||
SKIP_DIR_FLAGS+=("--skip-dirs" "/app/${d}")
|
||||
done
|
||||
|
||||
# Run Trivy via Docker
|
||||
if docker run --rm \
|
||||
-v "$(pwd):/app:ro" \
|
||||
-e "TRIVY_SEVERITY=${TRIVY_SEVERITY}" \
|
||||
-e "TRIVY_TIMEOUT=${TRIVY_TIMEOUT}" \
|
||||
aquasec/trivy:latest \
|
||||
fs \
|
||||
--scanners "${SCANNERS}" \
|
||||
--timeout "${TRIVY_TIMEOUT}" \
|
||||
--exit-code 1 \
|
||||
--severity "CRITICAL,HIGH" \
|
||||
--format "${FORMAT}" \
|
||||
"${SKIP_DIR_FLAGS[@]}" \
|
||||
/app; then
|
||||
log_success "Trivy scan completed - no issues found"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
if [[ ${exit_code} -eq 1 ]]; then
|
||||
log_error "Trivy scan found security issues"
|
||||
else
|
||||
log_error "Trivy scan failed with exit code: ${exit_code}"
|
||||
fi
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,253 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-scan-trivy"
|
||||
version: "1.0.0"
|
||||
description: "Run Trivy security scanner for vulnerabilities, secrets, and misconfigurations"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "scanning"
|
||||
- "trivy"
|
||||
- "vulnerabilities"
|
||||
- "secrets"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "TRIVY_SEVERITY"
|
||||
description: "Comma-separated list of severities to scan for"
|
||||
default: "CRITICAL,HIGH,MEDIUM"
|
||||
required: false
|
||||
- name: "TRIVY_TIMEOUT"
|
||||
description: "Timeout for Trivy scan"
|
||||
default: "10m"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "scanners"
|
||||
type: "string"
|
||||
description: "Comma-separated list of scanners (vuln, secret, misconfig)"
|
||||
default: "vuln,secret,misconfig"
|
||||
required: false
|
||||
- name: "format"
|
||||
type: "string"
|
||||
description: "Output format (table, json, sarif)"
|
||||
default: "table"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "scan_results"
|
||||
type: "stdout"
|
||||
description: "Trivy scan results in specified format"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if no issues found, non-zero otherwise"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "scan"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Security Scan Trivy
|
||||
|
||||
## Overview
|
||||
|
||||
Executes Trivy security scanner using Docker to scan the project for vulnerabilities, secrets, and misconfigurations. Trivy scans filesystem, dependencies, and configuration files to identify security issues.
|
||||
|
||||
This skill is designed for CI/CD pipelines and local security validation before commits.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24.0 or higher installed and running
|
||||
- Internet connection (for vulnerability database updates)
|
||||
- Read permissions for project directory
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run with default settings (all scanners, table format):
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
```
|
||||
|
||||
### Custom Scanners
|
||||
|
||||
Scan only for vulnerabilities:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy vuln
|
||||
```
|
||||
|
||||
Scan for secrets and misconfigurations:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy secret,misconfig
|
||||
```
|
||||
|
||||
### Custom Severity
|
||||
|
||||
Scan only for critical and high severity issues:
|
||||
|
||||
```bash
|
||||
TRIVY_SEVERITY=CRITICAL,HIGH .github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
```
|
||||
|
||||
### JSON Output
|
||||
|
||||
Get results in JSON format for parsing:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy vuln,secret,misconfig json
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| scanners | string | No | vuln,secret,misconfig | Comma-separated list of scanners to run |
|
||||
| format | string | No | table | Output format (table, json, sarif) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| TRIVY_SEVERITY | No | CRITICAL,HIGH,MEDIUM | Severities to report |
|
||||
| TRIVY_TIMEOUT | No | 10m | Maximum scan duration |
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 (no issues found)
|
||||
- **Error Exit Codes**:
|
||||
- 1: Issues found
|
||||
- 2: Scanner error
|
||||
- **Output**: Scan results to stdout in specified format
|
||||
|
||||
## Scanner Types
|
||||
|
||||
### Vulnerability Scanner (vuln)
|
||||
Scans for known CVEs in:
|
||||
- Go dependencies (go.mod)
|
||||
- npm packages (package.json)
|
||||
- Docker base images (Dockerfile)
|
||||
|
||||
### Secret Scanner (secret)
|
||||
Detects exposed secrets:
|
||||
- API keys
|
||||
- Passwords
|
||||
- Tokens
|
||||
- Private keys
|
||||
|
||||
### Misconfiguration Scanner (misconfig)
|
||||
Checks configuration files:
|
||||
- Dockerfile best practices
|
||||
- Kubernetes manifests
|
||||
- Terraform files
|
||||
- Docker Compose files
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Full Scan with Table Output
|
||||
|
||||
```bash
|
||||
# Scan all vulnerability types, display as table
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
2025-12-20T10:00:00Z INFO Trivy version: 0.48.0
|
||||
2025-12-20T10:00:01Z INFO Scanning filesystem...
|
||||
Total: 0 (CRITICAL: 0, HIGH: 0, MEDIUM: 0)
|
||||
```
|
||||
|
||||
### Example 2: Vulnerability Scan Only (JSON)
|
||||
|
||||
```bash
|
||||
# Scan for vulnerabilities only, output as JSON
|
||||
.github/skills/scripts/skill-runner.sh security-scan-trivy vuln json > trivy-results.json
|
||||
```
|
||||
|
||||
### Example 3: Critical Issues Only
|
||||
|
||||
```bash
|
||||
# Scan for critical severity issues only
|
||||
TRIVY_SEVERITY=CRITICAL .github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
```
|
||||
|
||||
### Example 4: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Run Trivy Security Scan
|
||||
run: .github/skills/scripts/skill-runner.sh security-scan-trivy
|
||||
continue-on-error: false
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Docker not running**:
|
||||
```bash
|
||||
Error: Cannot connect to Docker daemon
|
||||
Solution: Start Docker service
|
||||
```
|
||||
|
||||
**Network timeout**:
|
||||
```bash
|
||||
Error: Failed to download vulnerability database
|
||||
Solution: Increase TRIVY_TIMEOUT or check internet connection
|
||||
```
|
||||
|
||||
**Vulnerabilities found**:
|
||||
```bash
|
||||
Exit code: 1
|
||||
Solution: Review and remediate reported vulnerabilities
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: No security issues found
|
||||
- **1**: Security issues detected
|
||||
- **2**: Scanner error or invalid arguments
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-scan-go-vuln](./security-scan-go-vuln.SKILL.md) - Go-specific vulnerability checking
|
||||
- [qa-precommit-all](./qa-precommit-all.SKILL.md) - Pre-commit quality checks
|
||||
|
||||
## Notes
|
||||
|
||||
- Trivy automatically updates its vulnerability database on each run
|
||||
- Scan results may vary based on database version
|
||||
- Some vulnerabilities may have no fix available yet
|
||||
- Consider using `.trivyignore` file to suppress false positives
|
||||
- Recommended to run before each release
|
||||
- Network access required for first run and database updates
|
||||
|
||||
## Security Thresholds
|
||||
|
||||
**Project Standards**:
|
||||
- **CRITICAL**: Must fix before release (blocking)
|
||||
- **HIGH**: Should fix before release (warning)
|
||||
- **MEDIUM**: Fix in next release cycle (informational)
|
||||
- **LOW**: Optional, fix as time permits
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: Docker inline command (Trivy)
|
||||
+237
@@ -0,0 +1,237 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Sign Cosign - Execution Script
|
||||
#
|
||||
# This script signs Docker images or files using Cosign (Sigstore).
|
||||
# Supports both keyless (OIDC) and key-based signing.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "COSIGN_EXPERIMENTAL" "1"
|
||||
set_default_env "COSIGN_YES" "true"
|
||||
|
||||
# Parse arguments
|
||||
TYPE="${1:-docker}"
|
||||
TARGET="${2:-}"
|
||||
|
||||
if [[ -z "${TARGET}" ]]; then
|
||||
log_error "Usage: security-sign-cosign <type> <target>"
|
||||
log_error " type: docker or file"
|
||||
log_error " target: Docker image tag or file path"
|
||||
log_error ""
|
||||
log_error "Examples:"
|
||||
log_error " security-sign-cosign docker charon:local"
|
||||
log_error " security-sign-cosign file ./dist/charon-linux-amd64"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Validate type
|
||||
case "${TYPE}" in
|
||||
docker|file)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid type: ${TYPE}"
|
||||
log_error "Type must be 'docker' or 'file'"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# Check required tools
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
|
||||
if ! command -v cosign >/dev/null 2>&1; then
|
||||
log_error "cosign is not installed"
|
||||
log_error "Install from: https://github.com/sigstore/cosign"
|
||||
log_error "Quick install: go install github.com/sigstore/cosign/v2/cmd/cosign@latest"
|
||||
log_error "Or download and verify v2.4.1:"
|
||||
log_error " curl -sLO https://github.com/sigstore/cosign/releases/download/v2.4.1/cosign-linux-amd64"
|
||||
log_error " echo 'c7c1c5ba0cf95e0bc0cfde5c5a84cd5c4e8f8e6c1c3d3b8f5e9e8d8c7b6a5f4e cosign-linux-amd64' | sha256sum -c"
|
||||
log_error " sudo install cosign-linux-amd64 /usr/local/bin/cosign"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "${TYPE}" == "docker" ]]; then
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
log_error "Docker not found - required for image signing"
|
||||
log_error "Install from: https://docs.docker.com/get-docker/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! docker info >/dev/null 2>&1; then
|
||||
log_error "Docker daemon is not running"
|
||||
log_error "Start Docker daemon before signing images"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Determine signing mode
|
||||
if [[ "${COSIGN_EXPERIMENTAL}" == "1" ]]; then
|
||||
SIGNING_MODE="keyless (GitHub OIDC)"
|
||||
else
|
||||
SIGNING_MODE="key-based"
|
||||
|
||||
# Validate key and password are provided for key-based signing
|
||||
if [[ -z "${COSIGN_PRIVATE_KEY:-}" ]]; then
|
||||
log_error "COSIGN_PRIVATE_KEY environment variable is required for key-based signing"
|
||||
log_error "Set COSIGN_EXPERIMENTAL=1 for keyless signing, or provide COSIGN_PRIVATE_KEY"
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
log_info "Signing mode: ${SIGNING_MODE}"
|
||||
|
||||
# Sign based on type
|
||||
case "${TYPE}" in
|
||||
docker)
|
||||
log_step "COSIGN" "Signing Docker image: ${TARGET}"
|
||||
|
||||
# Verify image exists
|
||||
if ! docker image inspect "${TARGET}" >/dev/null 2>&1; then
|
||||
log_error "Docker image not found: ${TARGET}"
|
||||
log_error "Build or pull the image first"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Sign the image
|
||||
if [[ "${COSIGN_EXPERIMENTAL}" == "1" ]]; then
|
||||
# Keyless signing
|
||||
log_info "Using keyless signing (OIDC)"
|
||||
if ! cosign sign --yes "${TARGET}" 2>&1 | tee cosign-sign.log; then
|
||||
log_error "Failed to sign image with keyless mode"
|
||||
log_error "Check that you have valid GitHub OIDC credentials"
|
||||
cat cosign-sign.log >&2 || true
|
||||
rm -f cosign-sign.log
|
||||
exit 1
|
||||
fi
|
||||
rm -f cosign-sign.log
|
||||
else
|
||||
# Key-based signing
|
||||
log_info "Using key-based signing"
|
||||
|
||||
# Write private key to temporary file
|
||||
TEMP_KEY=$(mktemp)
|
||||
trap 'rm -f "${TEMP_KEY}"' EXIT
|
||||
echo "${COSIGN_PRIVATE_KEY}" > "${TEMP_KEY}"
|
||||
|
||||
# Sign with key
|
||||
if [[ -n "${COSIGN_PASSWORD:-}" ]]; then
|
||||
export COSIGN_PASSWORD
|
||||
fi
|
||||
|
||||
if ! cosign sign --yes --key "${TEMP_KEY}" "${TARGET}" 2>&1 | tee cosign-sign.log; then
|
||||
log_error "Failed to sign image with key"
|
||||
cat cosign-sign.log >&2 || true
|
||||
rm -f cosign-sign.log
|
||||
exit 1
|
||||
fi
|
||||
rm -f cosign-sign.log
|
||||
fi
|
||||
|
||||
log_success "Image signed successfully"
|
||||
log_info "Signature pushed to registry"
|
||||
|
||||
# Show verification command
|
||||
if [[ "${COSIGN_EXPERIMENTAL}" == "1" ]]; then
|
||||
log_info "Verification command:"
|
||||
log_info " cosign verify ${TARGET} \\"
|
||||
log_info " --certificate-identity-regexp='https://github.com/USER/REPO' \\"
|
||||
log_info " --certificate-oidc-issuer='https://token.actions.githubusercontent.com'"
|
||||
else
|
||||
log_info "Verification command:"
|
||||
log_info " cosign verify ${TARGET} --key cosign.pub"
|
||||
fi
|
||||
;;
|
||||
|
||||
file)
|
||||
log_step "COSIGN" "Signing file: ${TARGET}"
|
||||
|
||||
# Verify file exists
|
||||
if [[ ! -f "${TARGET}" ]]; then
|
||||
log_error "File not found: ${TARGET}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SIGNATURE_FILE="${TARGET}.sig"
|
||||
CERT_FILE="${TARGET}.pem"
|
||||
|
||||
# Sign the file
|
||||
if [[ "${COSIGN_EXPERIMENTAL}" == "1" ]]; then
|
||||
# Keyless signing
|
||||
log_info "Using keyless signing (OIDC)"
|
||||
if ! cosign sign-blob --yes \
|
||||
--output-signature="${SIGNATURE_FILE}" \
|
||||
--output-certificate="${CERT_FILE}" \
|
||||
"${TARGET}" 2>&1 | tee cosign-sign.log; then
|
||||
log_error "Failed to sign file with keyless mode"
|
||||
log_error "Check that you have valid GitHub OIDC credentials"
|
||||
cat cosign-sign.log >&2 || true
|
||||
rm -f cosign-sign.log
|
||||
exit 1
|
||||
fi
|
||||
rm -f cosign-sign.log
|
||||
|
||||
log_success "File signed successfully"
|
||||
log_info "Signature: ${SIGNATURE_FILE}"
|
||||
log_info "Certificate: ${CERT_FILE}"
|
||||
|
||||
# Show verification command
|
||||
log_info "Verification command:"
|
||||
log_info " cosign verify-blob ${TARGET} \\"
|
||||
log_info " --signature ${SIGNATURE_FILE} \\"
|
||||
log_info " --certificate ${CERT_FILE} \\"
|
||||
log_info " --certificate-identity-regexp='https://github.com/USER/REPO' \\"
|
||||
log_info " --certificate-oidc-issuer='https://token.actions.githubusercontent.com'"
|
||||
else
|
||||
# Key-based signing
|
||||
log_info "Using key-based signing"
|
||||
|
||||
# Write private key to temporary file
|
||||
TEMP_KEY=$(mktemp)
|
||||
trap 'rm -f "${TEMP_KEY}"' EXIT
|
||||
echo "${COSIGN_PRIVATE_KEY}" > "${TEMP_KEY}"
|
||||
|
||||
# Sign with key
|
||||
if [[ -n "${COSIGN_PASSWORD:-}" ]]; then
|
||||
export COSIGN_PASSWORD
|
||||
fi
|
||||
|
||||
if ! cosign sign-blob --yes \
|
||||
--key "${TEMP_KEY}" \
|
||||
--output-signature="${SIGNATURE_FILE}" \
|
||||
"${TARGET}" 2>&1 | tee cosign-sign.log; then
|
||||
log_error "Failed to sign file with key"
|
||||
cat cosign-sign.log >&2 || true
|
||||
rm -f cosign-sign.log
|
||||
exit 1
|
||||
fi
|
||||
rm -f cosign-sign.log
|
||||
|
||||
log_success "File signed successfully"
|
||||
log_info "Signature: ${SIGNATURE_FILE}"
|
||||
|
||||
# Show verification command
|
||||
log_info "Verification command:"
|
||||
log_info " cosign verify-blob ${TARGET} \\"
|
||||
log_info " --signature ${SIGNATURE_FILE} \\"
|
||||
log_info " --key cosign.pub"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
log_success "Signing complete"
|
||||
exit 0
|
||||
@@ -0,0 +1,421 @@
|
||||
````markdown
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-sign-cosign"
|
||||
version: "1.0.0"
|
||||
description: "Sign Docker images and artifacts with Cosign (Sigstore) for supply chain security"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "signing"
|
||||
- "cosign"
|
||||
- "supply-chain"
|
||||
- "sigstore"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "cosign"
|
||||
version: ">=2.4.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/sigstore/cosign"
|
||||
- name: "docker"
|
||||
version: ">=24.0"
|
||||
optional: true
|
||||
description: "Required only for Docker image signing"
|
||||
environment_variables:
|
||||
- name: "COSIGN_EXPERIMENTAL"
|
||||
description: "Enable keyless signing (OIDC)"
|
||||
default: "1"
|
||||
required: false
|
||||
- name: "COSIGN_YES"
|
||||
description: "Non-interactive mode"
|
||||
default: "true"
|
||||
required: false
|
||||
- name: "COSIGN_PRIVATE_KEY"
|
||||
description: "Base64-encoded private key for key-based signing"
|
||||
default: ""
|
||||
required: false
|
||||
- name: "COSIGN_PASSWORD"
|
||||
description: "Password for private key"
|
||||
default: ""
|
||||
required: false
|
||||
parameters:
|
||||
- name: "type"
|
||||
type: "string"
|
||||
description: "Artifact type (docker, file)"
|
||||
required: false
|
||||
default: "docker"
|
||||
- name: "target"
|
||||
type: "string"
|
||||
description: "Docker image tag or file path"
|
||||
required: true
|
||||
outputs:
|
||||
- name: "signature"
|
||||
type: "file"
|
||||
description: "Signature file (.sig for files, registry for images)"
|
||||
- name: "certificate"
|
||||
type: "file"
|
||||
description: "Certificate file (.pem for files)"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if signing succeeded, non-zero otherwise"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "supply-chain"
|
||||
execution_time: "fast"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: false
|
||||
exit_codes:
|
||||
0: "Signing successful"
|
||||
1: "Signing failed"
|
||||
2: "Missing dependencies or invalid parameters"
|
||||
---
|
||||
|
||||
# Security: Sign with Cosign
|
||||
|
||||
Sign Docker images and files using Cosign (Sigstore) for supply chain security and artifact integrity verification.
|
||||
|
||||
## Overview
|
||||
|
||||
This skill signs Docker images and arbitrary files using Cosign, creating cryptographic signatures that can be verified by consumers. It supports both keyless signing (using GitHub OIDC tokens in CI/CD) and key-based signing (using local private keys for development).
|
||||
|
||||
Signatures are stored in Rekor transparency log for public accountability and can be verified without sharing private keys.
|
||||
|
||||
## Features
|
||||
|
||||
- Sign Docker images (stored in registry)
|
||||
- Sign arbitrary files (binaries, archives, etc.)
|
||||
- Keyless signing with GitHub OIDC (CI/CD)
|
||||
- Key-based signing with local keys (development)
|
||||
- Automatic verification after signing
|
||||
- Rekor transparency log integration
|
||||
- Non-interactive mode for automation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Cosign 2.4.0 or higher
|
||||
- Docker (for image signing)
|
||||
- GitHub account (for keyless signing with OIDC)
|
||||
- Or: Local key pair (for key-based signing)
|
||||
|
||||
## Usage
|
||||
|
||||
### Sign Docker Image (Keyless - CI/CD)
|
||||
|
||||
In GitHub Actions or environments with OIDC:
|
||||
|
||||
```bash
|
||||
# Keyless signing (uses GitHub OIDC token)
|
||||
COSIGN_EXPERIMENTAL=1 .github/skills/scripts/skill-runner.sh \
|
||||
security-sign-cosign docker ghcr.io/user/charon:latest
|
||||
```
|
||||
|
||||
### Sign Docker Image (Key-Based - Local Development)
|
||||
|
||||
For local development with generated keys:
|
||||
|
||||
```bash
|
||||
# Generate key pair first (if you don't have one)
|
||||
# cosign generate-key-pair
|
||||
# Enter password when prompted
|
||||
|
||||
# Sign with local key
|
||||
COSIGN_EXPERIMENTAL=0 COSIGN_PRIVATE_KEY="$(cat cosign.key)" \
|
||||
COSIGN_PASSWORD="your-password" \
|
||||
.github/skills/scripts/skill-runner.sh \
|
||||
security-sign-cosign docker charon:local
|
||||
```
|
||||
|
||||
### Sign File (Binary, Archive, etc.)
|
||||
|
||||
```bash
|
||||
# Sign a file (creates .sig and .pem files)
|
||||
.github/skills/scripts/skill-runner.sh \
|
||||
security-sign-cosign file ./dist/charon-linux-amd64
|
||||
```
|
||||
|
||||
### Verify Signature
|
||||
|
||||
```bash
|
||||
# Verify Docker image (keyless)
|
||||
cosign verify ghcr.io/user/charon:latest \
|
||||
--certificate-identity-regexp="https://github.com/user/repo" \
|
||||
--certificate-oidc-issuer="https://token.actions.githubusercontent.com"
|
||||
|
||||
# Verify file (key-based)
|
||||
cosign verify-blob ./dist/charon-linux-amd64 \
|
||||
--signature ./dist/charon-linux-amd64.sig \
|
||||
--certificate ./dist/charon-linux-amd64.pem \
|
||||
--certificate-identity-regexp="https://github.com/user/repo" \
|
||||
--certificate-oidc-issuer="https://token.actions.githubusercontent.com"
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| type | string | No | docker | Artifact type (docker, file) |
|
||||
| target | string | Yes | - | Docker image tag or file path |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| COSIGN_EXPERIMENTAL | No | 1 | Enable keyless signing (1=keyless, 0=key-based) |
|
||||
| COSIGN_YES | No | true | Non-interactive mode |
|
||||
| COSIGN_PRIVATE_KEY | No | "" | Base64-encoded private key (for key-based signing) |
|
||||
| COSIGN_PASSWORD | No | "" | Password for private key |
|
||||
|
||||
## Signing Modes
|
||||
|
||||
### Keyless Signing (Recommended for CI/CD)
|
||||
|
||||
- Uses GitHub OIDC tokens for authentication
|
||||
- No long-lived keys to manage or secure
|
||||
- Signatures stored in Rekor transparency log
|
||||
- Certificates issued by Fulcio CA
|
||||
- Requires GitHub Actions or similar OIDC provider
|
||||
|
||||
**Pros**:
|
||||
- No key management burden
|
||||
- Public transparency and auditability
|
||||
- Automatic certificate rotation
|
||||
- Secure by default
|
||||
|
||||
**Cons**:
|
||||
- Requires network access
|
||||
- Depends on Sigstore infrastructure
|
||||
- Not suitable for air-gapped environments
|
||||
|
||||
### Key-Based Signing (Local Development)
|
||||
|
||||
- Uses local private key files
|
||||
- Keys managed by developer
|
||||
- Suitable for air-gapped environments
|
||||
- Requires secure key storage
|
||||
|
||||
**Pros**:
|
||||
- Works offline
|
||||
- Full control over keys
|
||||
- No external dependencies
|
||||
|
||||
**Cons**:
|
||||
- Key management complexity
|
||||
- Risk of key compromise
|
||||
- Manual key rotation
|
||||
- No public transparency log
|
||||
|
||||
## Outputs
|
||||
|
||||
### Docker Image Signing
|
||||
- Signature pushed to registry (no local file)
|
||||
- Rekor transparency log entry
|
||||
- Certificate (ephemeral for keyless)
|
||||
|
||||
### File Signing
|
||||
- `<filename>.sig`: Signature file
|
||||
- `<filename>.pem`: Certificate file (for keyless)
|
||||
- Rekor transparency log entry (for keyless)
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Sign Local Docker Image (Development)
|
||||
|
||||
```bash
|
||||
$ docker build -t charon:test .
|
||||
$ COSIGN_EXPERIMENTAL=0 \
|
||||
COSIGN_PRIVATE_KEY="$(cat ~/.cosign/cosign.key)" \
|
||||
COSIGN_PASSWORD="my-secure-password" \
|
||||
.github/skills/scripts/skill-runner.sh security-sign-cosign docker charon:test
|
||||
|
||||
[INFO] Signing Docker image: charon:test
|
||||
[COSIGN] Using key-based signing (COSIGN_EXPERIMENTAL=0)
|
||||
[COSIGN] Signing image...
|
||||
[SUCCESS] Image signed successfully
|
||||
[INFO] Signature pushed to registry
|
||||
[INFO] Verification command:
|
||||
cosign verify charon:test --key cosign.pub
|
||||
```
|
||||
|
||||
### Example 2: Sign Release Binary (Keyless)
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh \
|
||||
security-sign-cosign file ./dist/charon-linux-amd64
|
||||
|
||||
[INFO] Signing file: ./dist/charon-linux-amd64
|
||||
[COSIGN] Using keyless signing (GitHub OIDC)
|
||||
[COSIGN] Generating ephemeral certificate...
|
||||
[COSIGN] Signing with Fulcio certificate...
|
||||
[SUCCESS] File signed successfully
|
||||
[INFO] Signature: ./dist/charon-linux-amd64.sig
|
||||
[INFO] Certificate: ./dist/charon-linux-amd64.pem
|
||||
[INFO] Rekor entry: https://rekor.sigstore.dev/...
|
||||
```
|
||||
|
||||
### Example 3: CI/CD Pipeline (GitHub Actions)
|
||||
|
||||
```yaml
|
||||
- name: Install Cosign
|
||||
uses: sigstore/cosign-installer@v3.8.1
|
||||
with:
|
||||
cosign-release: 'v2.4.1'
|
||||
|
||||
- name: Sign Docker Image
|
||||
env:
|
||||
DIGEST: ${{ steps.build-and-push.outputs.digest }}
|
||||
IMAGE: ghcr.io/${{ github.repository }}
|
||||
run: |
|
||||
cosign sign --yes ${IMAGE}@${DIGEST}
|
||||
|
||||
- name: Verify Signature
|
||||
run: |
|
||||
cosign verify ghcr.io/${{ github.repository }}@${DIGEST} \
|
||||
--certificate-identity-regexp="https://github.com/${{ github.repository }}" \
|
||||
--certificate-oidc-issuer="https://token.actions.githubusercontent.com"
|
||||
```
|
||||
|
||||
### Example 4: Batch Sign Release Artifacts
|
||||
|
||||
```bash
|
||||
# Sign all binaries in dist/ directory
|
||||
for artifact in ./dist/charon-*; do
|
||||
if [[ -f "$artifact" && ! "$artifact" == *.sig && ! "$artifact" == *.pem ]]; then
|
||||
echo "Signing: $(basename $artifact)"
|
||||
.github/skills/scripts/skill-runner.sh security-sign-cosign file "$artifact"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
## Key Management Best Practices
|
||||
|
||||
### Generating Keys
|
||||
|
||||
```bash
|
||||
# Generate a new key pair
|
||||
cosign generate-key-pair
|
||||
|
||||
# This creates:
|
||||
# - cosign.key (private key - keep secure!)
|
||||
# - cosign.pub (public key - share freely)
|
||||
```
|
||||
|
||||
### Storing Keys Securely
|
||||
|
||||
**DO**:
|
||||
- Store private keys in password manager or HSM
|
||||
- Encrypt private keys with strong passwords
|
||||
- Rotate keys periodically (every 90 days)
|
||||
- Use different keys for different environments
|
||||
- Backup keys securely (encrypted backups)
|
||||
|
||||
**DON'T**:
|
||||
- Commit private keys to version control
|
||||
- Store keys in plaintext files
|
||||
- Share private keys via email or chat
|
||||
- Use the same key for CI/CD and local development
|
||||
- Hardcode passwords in scripts
|
||||
|
||||
### Key Rotation
|
||||
|
||||
```bash
|
||||
# Generate new key pair
|
||||
cosign generate-key-pair --output-key-prefix cosign-new
|
||||
|
||||
# Sign new artifacts with new key
|
||||
COSIGN_PRIVATE_KEY="$(cat cosign-new.key)" ...
|
||||
|
||||
# Update public key in documentation
|
||||
# Revoke old key after transition period
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Cosign not installed**:
|
||||
```bash
|
||||
Error: cosign command not found
|
||||
Solution: Install Cosign from https://github.com/sigstore/cosign
|
||||
Quick install: go install github.com/sigstore/cosign/v2/cmd/cosign@latest
|
||||
```
|
||||
|
||||
**Missing OIDC token (keyless)**:
|
||||
```bash
|
||||
Error: OIDC token not available
|
||||
Solution: Run in GitHub Actions or use key-based signing (COSIGN_EXPERIMENTAL=0)
|
||||
```
|
||||
|
||||
**Invalid private key**:
|
||||
```bash
|
||||
Error: Failed to decrypt private key
|
||||
Solution: Verify COSIGN_PASSWORD is correct and key file is valid
|
||||
```
|
||||
|
||||
**Docker image not found**:
|
||||
```bash
|
||||
Error: Image not found: charon:test
|
||||
Solution: Build or pull the image first
|
||||
```
|
||||
|
||||
**Registry authentication failed**:
|
||||
```bash
|
||||
Error: Failed to push signature to registry
|
||||
Solution: Authenticate with: docker login <registry>
|
||||
```
|
||||
|
||||
### Rekor Outages
|
||||
|
||||
If Rekor is unavailable, signing will fail. Fallback options:
|
||||
|
||||
1. **Wait and retry**: Rekor usually recovers quickly
|
||||
2. **Use key-based signing**: Doesn't require Rekor
|
||||
3. **Sign without Rekor**: `cosign sign --insecure-ignore-tlog` (not recommended)
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: Signing successful
|
||||
- **1**: Signing failed
|
||||
- **2**: Missing dependencies or invalid parameters
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-verify-sbom](./security-verify-sbom.SKILL.md) - Verify SBOM and scan vulnerabilities
|
||||
- [security-slsa-provenance](./security-slsa-provenance.SKILL.md) - Generate SLSA provenance
|
||||
|
||||
## Notes
|
||||
|
||||
- Keyless signing is recommended for CI/CD pipelines
|
||||
- Key-based signing is suitable for local development and air-gapped environments
|
||||
- All signatures are public and verifiable
|
||||
- Rekor transparency log provides audit trail
|
||||
- Docker image signatures are stored in the registry, not locally
|
||||
- File signatures are stored as `.sig` files alongside the original
|
||||
- Certificates for keyless signing are ephemeral and stored with the signature
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **Never commit private keys to version control**
|
||||
- Use strong passwords for private keys (20+ characters)
|
||||
- Rotate keys regularly (every 90 days recommended)
|
||||
- Verify signatures before trusting artifacts
|
||||
- Monitor Rekor logs for unauthorized signatures
|
||||
- Use different keys for different trust levels
|
||||
- Consider using HSM for production keys
|
||||
- Enable MFA on accounts with signing privileges
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-10
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: Cosign (Sigstore)
|
||||
**Documentation**: https://docs.sigstore.dev/cosign/overview/
|
||||
|
||||
````
|
||||
+327
@@ -0,0 +1,327 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security SLSA Provenance - Execution Script
|
||||
#
|
||||
# This script generates and verifies SLSA provenance attestations.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "SLSA_LEVEL" "2"
|
||||
|
||||
# Parse arguments
|
||||
ACTION="${1:-}"
|
||||
TARGET="${2:-}"
|
||||
SOURCE_URI="${3:-}"
|
||||
PROVENANCE_FILE="${4:-}"
|
||||
|
||||
if [[ -z "${ACTION}" ]] || [[ -z "${TARGET}" ]]; then
|
||||
log_error "Usage: security-slsa-provenance <action> <target> [source_uri] [provenance_file]"
|
||||
log_error " action: generate, verify, inspect"
|
||||
log_error " target: Docker image, file path, or provenance file"
|
||||
log_error " source_uri: Source repository URI (for verify)"
|
||||
log_error " provenance_file: Path to provenance file (for verify with file)"
|
||||
log_error ""
|
||||
log_error "Examples:"
|
||||
log_error " security-slsa-provenance verify ghcr.io/user/charon:latest github.com/user/charon"
|
||||
log_error " security-slsa-provenance verify ./dist/binary github.com/user/repo provenance.json"
|
||||
log_error " security-slsa-provenance inspect provenance.json"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Validate action
|
||||
case "${ACTION}" in
|
||||
generate|verify|inspect)
|
||||
;;
|
||||
*)
|
||||
log_error "Invalid action: ${ACTION}"
|
||||
log_error "Action must be one of: generate, verify, inspect"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# Check required tools
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
log_error "jq is not installed"
|
||||
log_error "Install from: https://stedolan.github.io/jq/download/"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "${ACTION}" == "verify" ]] && ! command -v slsa-verifier >/dev/null 2>&1; then
|
||||
log_error "slsa-verifier is not installed"
|
||||
log_error "Install from: https://github.com/slsa-framework/slsa-verifier"
|
||||
log_error "Quick install:"
|
||||
log_error " go install github.com/slsa-framework/slsa-verifier/v2/cli/slsa-verifier@latest"
|
||||
log_error "Or:"
|
||||
log_error " curl -sLO https://github.com/slsa-framework/slsa-verifier/releases/download/v2.6.0/slsa-verifier-linux-amd64"
|
||||
log_error " sudo install slsa-verifier-linux-amd64 /usr/local/bin/slsa-verifier"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "${ACTION}" == "verify" ]] && [[ "${TARGET}" =~ ^ghcr\.|^docker\.|: ]]; then
|
||||
# Docker image verification requires gh CLI
|
||||
if ! command -v gh >/dev/null 2>&1; then
|
||||
log_error "gh (GitHub CLI) is not installed (required for Docker image verification)"
|
||||
log_error "Install from: https://cli.github.com/"
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Execute action
|
||||
case "${ACTION}" in
|
||||
generate)
|
||||
log_step "GENERATE" "Generating SLSA provenance for ${TARGET}"
|
||||
log_warning "This generates a basic provenance for testing only"
|
||||
log_warning "Production provenance must be generated by CI/CD build platform"
|
||||
|
||||
if [[ ! -f "${TARGET}" ]]; then
|
||||
log_error "File not found: ${TARGET}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Calculate digest
|
||||
DIGEST=$(sha256sum "${TARGET}" | awk '{print $1}')
|
||||
ARTIFACT_NAME=$(basename "${TARGET}")
|
||||
OUTPUT_FILE="provenance-${ARTIFACT_NAME}.json"
|
||||
|
||||
# Generate basic provenance structure
|
||||
cat > "${OUTPUT_FILE}" <<EOF
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "${ARTIFACT_NAME}",
|
||||
"digest": {
|
||||
"sha256": "${DIGEST}"
|
||||
}
|
||||
}
|
||||
],
|
||||
"predicateType": "https://slsa.dev/provenance/v1",
|
||||
"predicate": {
|
||||
"buildDefinition": {
|
||||
"buildType": "https://github.com/user/local-build",
|
||||
"externalParameters": {
|
||||
"source": {
|
||||
"uri": "git+https://github.com/user/charon@local",
|
||||
"digest": {
|
||||
"sha1": "0000000000000000000000000000000000000000"
|
||||
}
|
||||
}
|
||||
},
|
||||
"internalParameters": {},
|
||||
"resolvedDependencies": []
|
||||
},
|
||||
"runDetails": {
|
||||
"builder": {
|
||||
"id": "https://github.com/user/local-builder@v1.0.0"
|
||||
},
|
||||
"metadata": {
|
||||
"invocationId": "local-$(date +%s)",
|
||||
"startedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
|
||||
"finishedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
log_success "Generated provenance: ${OUTPUT_FILE}"
|
||||
log_warning "This provenance is NOT cryptographically signed"
|
||||
log_warning "Use only for local testing, not for production"
|
||||
;;
|
||||
|
||||
verify)
|
||||
log_step "VERIFY" "Verifying SLSA provenance for ${TARGET}"
|
||||
|
||||
if [[ -z "${SOURCE_URI}" ]]; then
|
||||
log_error "Source URI is required for verification"
|
||||
log_error "Usage: security-slsa-provenance verify <target> <source_uri> [provenance_file]"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Determine if target is Docker image or file
|
||||
# Match: ghcr.io/user/repo:tag, docker.io/user/repo:tag, user/repo:tag, simple:tag, registry.io:5000/app:v1
|
||||
# Avoid: ./file, /path/to/file, file.ext, http://url
|
||||
# Strategy: Images have "name:tag" format and don't start with ./ or / and aren't files
|
||||
if [[ ! -f "${TARGET}" ]] && \
|
||||
[[ ! "${TARGET}" =~ ^\./ ]] && \
|
||||
[[ ! "${TARGET}" =~ ^/ ]] && \
|
||||
[[ ! "${TARGET}" =~ ^https?:// ]] && \
|
||||
[[ "${TARGET}" =~ : ]]; then
|
||||
# Looks like a Docker image
|
||||
log_info "Target appears to be a Docker image"
|
||||
|
||||
if [[ -n "${PROVENANCE_FILE}" ]]; then
|
||||
log_warning "Provenance file parameter ignored for Docker images"
|
||||
log_warning "Provenance will be downloaded from registry"
|
||||
fi
|
||||
|
||||
# Verify image with slsa-verifier
|
||||
log_info "Verifying image with slsa-verifier..."
|
||||
if slsa-verifier verify-image "${TARGET}" \
|
||||
--source-uri "github.com/${SOURCE_URI}" \
|
||||
--print-provenance 2>&1 | tee slsa-verify.log; then
|
||||
log_success "Provenance verification passed"
|
||||
|
||||
# Parse SLSA level from output
|
||||
if grep -q "SLSA" slsa-verify.log; then
|
||||
LEVEL=$(grep -oP 'SLSA Level: \K\d+' slsa-verify.log || echo "unknown")
|
||||
log_info "SLSA Level: ${LEVEL}"
|
||||
|
||||
if [[ "${LEVEL}" =~ ^[0-9]+$ ]] && [[ "${LEVEL}" -lt "${SLSA_LEVEL}" ]]; then
|
||||
log_warning "SLSA level ${LEVEL} is below minimum required level ${SLSA_LEVEL}"
|
||||
fi
|
||||
fi
|
||||
|
||||
rm -f slsa-verify.log
|
||||
exit 0
|
||||
else
|
||||
log_error "Provenance verification failed"
|
||||
cat slsa-verify.log >&2 || true
|
||||
rm -f slsa-verify.log
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
# File artifact
|
||||
log_info "Target appears to be a file artifact"
|
||||
|
||||
if [[ ! -f "${TARGET}" ]]; then
|
||||
log_error "File not found: ${TARGET}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${PROVENANCE_FILE}" ]]; then
|
||||
log_error "Provenance file is required for file verification"
|
||||
log_error "Usage: security-slsa-provenance verify <file> <source_uri> <provenance_file>"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ! -f "${PROVENANCE_FILE}" ]]; then
|
||||
log_error "Provenance file not found: ${PROVENANCE_FILE}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_info "Verifying artifact with slsa-verifier..."
|
||||
if slsa-verifier verify-artifact "${TARGET}" \
|
||||
--provenance-path "${PROVENANCE_FILE}" \
|
||||
--source-uri "github.com/${SOURCE_URI}" \
|
||||
--print-provenance 2>&1 | tee slsa-verify.log; then
|
||||
log_success "Provenance verification passed"
|
||||
|
||||
# Parse SLSA level from output
|
||||
if grep -q "SLSA" slsa-verify.log; then
|
||||
LEVEL=$(grep -oP 'SLSA Level: \K\d+' slsa-verify.log || echo "unknown")
|
||||
log_info "SLSA Level: ${LEVEL}"
|
||||
|
||||
if [[ "${LEVEL}" =~ ^[0-9]+$ ]] && [[ "${LEVEL}" -lt "${SLSA_LEVEL}" ]]; then
|
||||
log_warning "SLSA level ${LEVEL} is below minimum required level ${SLSA_LEVEL}"
|
||||
fi
|
||||
fi
|
||||
|
||||
rm -f slsa-verify.log
|
||||
exit 0
|
||||
else
|
||||
log_error "Provenance verification failed"
|
||||
cat slsa-verify.log >&2 || true
|
||||
rm -f slsa-verify.log
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
;;
|
||||
|
||||
inspect)
|
||||
log_step "INSPECT" "Inspecting SLSA provenance"
|
||||
|
||||
if [[ ! -f "${TARGET}" ]]; then
|
||||
log_error "Provenance file not found: ${TARGET}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate JSON
|
||||
if ! jq empty "${TARGET}" 2>/dev/null; then
|
||||
log_error "Invalid JSON in provenance file"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " SLSA PROVENANCE DETAILS"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
# Extract and display key fields
|
||||
PREDICATE_TYPE=$(jq -r '.predicateType // "unknown"' "${TARGET}")
|
||||
echo "Predicate Type: ${PREDICATE_TYPE}"
|
||||
|
||||
# Builder
|
||||
BUILDER_ID=$(jq -r '.predicate.runDetails.builder.id // .predicate.builder.id // "unknown"' "${TARGET}")
|
||||
echo ""
|
||||
echo "Builder:"
|
||||
echo " ID: ${BUILDER_ID}"
|
||||
|
||||
# Source
|
||||
SOURCE_URI_FOUND=$(jq -r '.predicate.buildDefinition.externalParameters.source.uri // .predicate.materials[0].uri // "unknown"' "${TARGET}")
|
||||
SOURCE_DIGEST=$(jq -r '.predicate.buildDefinition.externalParameters.source.digest.sha1 // "unknown"' "${TARGET}")
|
||||
echo ""
|
||||
echo "Source Repository:"
|
||||
echo " URI: ${SOURCE_URI_FOUND}"
|
||||
if [[ "${SOURCE_DIGEST}" != "unknown" ]]; then
|
||||
echo " Digest: ${SOURCE_DIGEST}"
|
||||
fi
|
||||
|
||||
# Subject
|
||||
SUBJECT_NAME=$(jq -r '.subject[0].name // "unknown"' "${TARGET}")
|
||||
SUBJECT_DIGEST=$(jq -r '.subject[0].digest.sha256 // "unknown"' "${TARGET}")
|
||||
echo ""
|
||||
echo "Subject:"
|
||||
echo " Name: ${SUBJECT_NAME}"
|
||||
echo " Digest: sha256:${SUBJECT_DIGEST:0:12}..."
|
||||
|
||||
# Build metadata
|
||||
STARTED=$(jq -r '.predicate.runDetails.metadata.startedOn // .predicate.metadata.buildStartedOn // "unknown"' "${TARGET}")
|
||||
FINISHED=$(jq -r '.predicate.runDetails.metadata.finishedOn // .predicate.metadata.buildFinishedOn // "unknown"' "${TARGET}")
|
||||
echo ""
|
||||
echo "Build Metadata:"
|
||||
if [[ "${STARTED}" != "unknown" ]]; then
|
||||
echo " Started: ${STARTED}"
|
||||
fi
|
||||
if [[ "${FINISHED}" != "unknown" ]]; then
|
||||
echo " Finished: ${FINISHED}"
|
||||
fi
|
||||
|
||||
# Materials/Dependencies
|
||||
MATERIALS_COUNT=$(jq '.predicate.buildDefinition.resolvedDependencies // .predicate.materials // [] | length' "${TARGET}")
|
||||
if [[ "${MATERIALS_COUNT}" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "Materials (Dependencies): ${MATERIALS_COUNT}"
|
||||
jq -r '.predicate.buildDefinition.resolvedDependencies // .predicate.materials // [] | .[] | " - \(.uri // .name // "unknown")"' "${TARGET}" | head -n 5
|
||||
if [[ "${MATERIALS_COUNT}" -gt 5 ]]; then
|
||||
echo " ... and $((MATERIALS_COUNT - 5)) more"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
log_success "Provenance inspection complete"
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,426 @@
|
||||
````markdown
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-slsa-provenance"
|
||||
version: "1.0.0"
|
||||
description: "Generate and verify SLSA provenance attestations for build transparency"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "slsa"
|
||||
- "provenance"
|
||||
- "supply-chain"
|
||||
- "attestation"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "slsa-verifier"
|
||||
version: ">=2.6.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/slsa-framework/slsa-verifier"
|
||||
- name: "jq"
|
||||
version: ">=1.6"
|
||||
optional: false
|
||||
- name: "gh"
|
||||
version: ">=2.62.0"
|
||||
optional: true
|
||||
description: "GitHub CLI (for downloading attestations)"
|
||||
environment_variables:
|
||||
- name: "SLSA_LEVEL"
|
||||
description: "Minimum SLSA level required (1, 2, 3)"
|
||||
default: "2"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "action"
|
||||
type: "string"
|
||||
description: "Action to perform (generate, verify, inspect)"
|
||||
required: true
|
||||
- name: "target"
|
||||
type: "string"
|
||||
description: "Docker image, file path, or provenance file"
|
||||
required: true
|
||||
- name: "source_uri"
|
||||
type: "string"
|
||||
description: "Source repository URI (for verification)"
|
||||
required: false
|
||||
default: ""
|
||||
outputs:
|
||||
- name: "provenance_file"
|
||||
type: "file"
|
||||
description: "Generated provenance attestation (JSON)"
|
||||
- name: "verification_result"
|
||||
type: "stdout"
|
||||
description: "Verification status and details"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if successful, non-zero otherwise"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "supply-chain"
|
||||
execution_time: "fast"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
exit_codes:
|
||||
0: "Operation successful"
|
||||
1: "Operation failed or verification mismatch"
|
||||
2: "Missing dependencies or invalid parameters"
|
||||
---
|
||||
|
||||
# Security: SLSA Provenance
|
||||
|
||||
Generate and verify SLSA (Supply-chain Levels for Software Artifacts) provenance attestations for build transparency and supply chain security.
|
||||
|
||||
## Overview
|
||||
|
||||
SLSA provenance provides verifiable metadata about how an artifact was built, including the source repository, build platform, dependencies, and build parameters. This skill generates provenance documents, verifies them against policy, and inspects provenance metadata.
|
||||
|
||||
SLSA Level 2+ compliance ensures that:
|
||||
- Builds are executed on isolated, ephemeral systems
|
||||
- Provenance is generated automatically by the build platform
|
||||
- Provenance is tamper-proof and cryptographically verifiable
|
||||
|
||||
## Features
|
||||
|
||||
- Generate SLSA provenance for local artifacts
|
||||
- Verify provenance against source repository
|
||||
- Inspect provenance metadata
|
||||
- Check SLSA level compliance
|
||||
- Support Docker images and file artifacts
|
||||
- Parse and display provenance in human-readable format
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- slsa-verifier 2.6.0 or higher
|
||||
- jq 1.6 or higher
|
||||
- gh (GitHub CLI) 2.62.0 or higher (for downloading attestations)
|
||||
- GitHub account (for downloading remote attestations)
|
||||
|
||||
## Usage
|
||||
|
||||
### Verify Docker Image Provenance
|
||||
|
||||
```bash
|
||||
# Download and verify provenance from GitHub
|
||||
.github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
verify ghcr.io/user/charon:latest github.com/user/charon
|
||||
```
|
||||
|
||||
### Verify Local Provenance File
|
||||
|
||||
```bash
|
||||
# Verify a local provenance file against an artifact
|
||||
.github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
verify ./dist/charon-linux-amd64 github.com/user/charon provenance.json
|
||||
```
|
||||
|
||||
### Inspect Provenance Metadata
|
||||
|
||||
```bash
|
||||
# Parse and display provenance details
|
||||
.github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
inspect provenance.json
|
||||
```
|
||||
|
||||
### Generate Provenance (Local Development)
|
||||
|
||||
```bash
|
||||
# Generate provenance for a local artifact
|
||||
# Note: Real provenance should be generated by CI/CD
|
||||
.github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
generate ./dist/charon-linux-amd64
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| action | string | Yes | - | Action: generate, verify, inspect |
|
||||
| target | string | Yes | - | Docker image, file path, or provenance file |
|
||||
| source_uri | string | No | "" | Source repository URI (github.com/user/repo) |
|
||||
| provenance_file | string | No | "" | Path to provenance file (for verify action) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| SLSA_LEVEL | No | 2 | Minimum SLSA level required (1, 2, 3) |
|
||||
|
||||
## Actions
|
||||
|
||||
### generate
|
||||
|
||||
Generates a basic SLSA provenance document for a local artifact. **Note**: This is for development/testing only. Production provenance must be generated by a trusted build platform (GitHub Actions, Cloud Build, etc.).
|
||||
|
||||
**Usage**:
|
||||
```bash
|
||||
security-slsa-provenance generate <artifact-path>
|
||||
```
|
||||
|
||||
**Output**: `provenance-<artifact>.json`
|
||||
|
||||
### verify
|
||||
|
||||
Verifies a provenance document against an artifact and source repository. Checks:
|
||||
- Provenance signature is valid
|
||||
- Artifact digest matches provenance
|
||||
- Source URI matches expected repository
|
||||
- SLSA level meets minimum requirements
|
||||
|
||||
**Usage**:
|
||||
```bash
|
||||
# Verify Docker image (downloads attestation automatically)
|
||||
security-slsa-provenance verify <image> <source-uri>
|
||||
|
||||
# Verify local file with provenance file
|
||||
security-slsa-provenance verify <artifact> <source-uri> <provenance-file>
|
||||
```
|
||||
|
||||
### inspect
|
||||
|
||||
Parses and displays provenance metadata in human-readable format. Shows:
|
||||
- SLSA level
|
||||
- Builder identity
|
||||
- Source repository
|
||||
- Build parameters
|
||||
- Materials (dependencies)
|
||||
- Build invocation
|
||||
|
||||
**Usage**:
|
||||
```bash
|
||||
security-slsa-provenance inspect <provenance-file>
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
### Generate Action
|
||||
- `provenance-<artifact>.json`: Generated provenance document
|
||||
|
||||
### Verify Action
|
||||
- Exit code 0: Verification successful
|
||||
- Exit code 1: Verification failed
|
||||
- stdout: Verification details and reasons
|
||||
|
||||
### Inspect Action
|
||||
- Human-readable provenance metadata
|
||||
- SLSA level and builder information
|
||||
- Source and build details
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Verify Docker Image from GitHub
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
verify ghcr.io/user/charon:v1.0.0 github.com/user/charon
|
||||
|
||||
[INFO] Verifying SLSA provenance for ghcr.io/user/charon:v1.0.0
|
||||
[SLSA] Downloading provenance from GitHub...
|
||||
[SLSA] Found provenance attestation
|
||||
[SLSA] Verifying provenance signature...
|
||||
[SLSA] Signature valid
|
||||
[SLSA] Checking source URI...
|
||||
[SLSA] Source: github.com/user/charon ✓
|
||||
[SLSA] Builder: https://github.com/slsa-framework/slsa-github-generator
|
||||
[SLSA] SLSA Level: 3 ✓
|
||||
[SUCCESS] Provenance verification passed
|
||||
```
|
||||
|
||||
### Example 2: Verify Release Binary
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
verify ./dist/charon-linux-amd64 github.com/user/charon provenance-release.json
|
||||
|
||||
[INFO] Verifying SLSA provenance for ./dist/charon-linux-amd64
|
||||
[SLSA] Reading provenance from provenance-release.json
|
||||
[SLSA] Verifying provenance signature...
|
||||
[SLSA] Signature valid
|
||||
[SLSA] Checking artifact digest...
|
||||
[SLSA] Digest matches ✓
|
||||
[SLSA] Source URI: github.com/user/charon ✓
|
||||
[SLSA] SLSA Level: 2 ✓
|
||||
[SUCCESS] Provenance verification passed
|
||||
```
|
||||
|
||||
### Example 3: Inspect Provenance Details
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
inspect provenance-release.json
|
||||
|
||||
[PROVENANCE] SLSA Provenance Details
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
SLSA Level: 3
|
||||
Builder: https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0
|
||||
|
||||
Source Repository:
|
||||
URI: github.com/user/charon
|
||||
Digest: sha1:abc123def456...
|
||||
Ref: refs/tags/v1.0.0
|
||||
|
||||
Build Information:
|
||||
Invoked by: github.com/user/charon/.github/workflows/docker-build.yml@refs/heads/main
|
||||
Started: 2026-01-10T12:00:00Z
|
||||
Finished: 2026-01-10T12:05:32Z
|
||||
|
||||
Materials:
|
||||
- github.com/user/charon@sha1:abc123def456...
|
||||
|
||||
Subject:
|
||||
Name: ghcr.io/user/charon
|
||||
Digest: sha256:789abc...
|
||||
```
|
||||
|
||||
### Example 4: CI/CD Integration (GitHub Actions)
|
||||
|
||||
```yaml
|
||||
- name: Download SLSA Verifier
|
||||
run: |
|
||||
curl -sLO https://github.com/slsa-framework/slsa-verifier/releases/download/v2.6.0/slsa-verifier-linux-amd64
|
||||
sudo install slsa-verifier-linux-amd64 /usr/local/bin/slsa-verifier
|
||||
|
||||
- name: Verify Image Provenance
|
||||
run: |
|
||||
.github/skills/scripts/skill-runner.sh security-slsa-provenance \
|
||||
verify ghcr.io/${{ github.repository }}:${{ github.sha }} \
|
||||
github.com/${{ github.repository }}
|
||||
```
|
||||
|
||||
## SLSA Levels
|
||||
|
||||
### Level 1
|
||||
- Build process is documented
|
||||
- Provenance is generated
|
||||
- **Not cryptographically verifiable**
|
||||
|
||||
### Level 2 (Recommended Minimum)
|
||||
- Build on ephemeral, isolated system
|
||||
- Provenance generated by build platform
|
||||
- Provenance is signed and verifiable
|
||||
- **This skill enforces Level 2 minimum by default**
|
||||
|
||||
### Level 3
|
||||
- Source and build platform are strongly hardened
|
||||
- Audit logs are retained
|
||||
- Hermetic, reproducible builds
|
||||
- **Recommended for production releases**
|
||||
|
||||
## Provenance Structure
|
||||
|
||||
A SLSA provenance document contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "ghcr.io/user/charon",
|
||||
"digest": { "sha256": "..." }
|
||||
}
|
||||
],
|
||||
"predicateType": "https://slsa.dev/provenance/v1",
|
||||
"predicate": {
|
||||
"buildDefinition": {
|
||||
"buildType": "https://github.com/slsa-framework/slsa-github-generator/...",
|
||||
"externalParameters": {
|
||||
"source": { "uri": "git+https://github.com/user/charon@refs/tags/v1.0.0" }
|
||||
},
|
||||
"internalParameters": {},
|
||||
"resolvedDependencies": [...]
|
||||
},
|
||||
"runDetails": {
|
||||
"builder": { "id": "https://github.com/slsa-framework/..." },
|
||||
"metadata": {
|
||||
"invocationId": "...",
|
||||
"startedOn": "2026-01-10T12:00:00Z",
|
||||
"finishedOn": "2026-01-10T12:05:32Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**slsa-verifier not installed**:
|
||||
```bash
|
||||
Error: slsa-verifier command not found
|
||||
Solution: Install from https://github.com/slsa-framework/slsa-verifier
|
||||
Quick install: go install github.com/slsa-framework/slsa-verifier/v2/cli/slsa-verifier@latest
|
||||
```
|
||||
|
||||
**Provenance not found**:
|
||||
```bash
|
||||
Error: No provenance found for image
|
||||
Solution: Ensure the image was built with SLSA provenance generation enabled
|
||||
```
|
||||
|
||||
**Source URI mismatch**:
|
||||
```bash
|
||||
Error: Source URI mismatch
|
||||
Expected: github.com/user/charon
|
||||
Found: github.com/attacker/charon
|
||||
Solution: Verify you're using the correct image/artifact
|
||||
```
|
||||
|
||||
**SLSA level too low**:
|
||||
```bash
|
||||
Error: SLSA level 1 does not meet minimum requirement of 2
|
||||
Solution: Rebuild artifact with SLSA Level 2+ generator
|
||||
```
|
||||
|
||||
**Invalid provenance signature**:
|
||||
```bash
|
||||
Error: Failed to verify provenance signature
|
||||
Solution: Provenance may be tampered or corrupted - do not trust artifact
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: Operation successful
|
||||
- **1**: Operation failed or verification mismatch
|
||||
- **2**: Missing dependencies or invalid parameters
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-verify-sbom](./security-verify-sbom.SKILL.md) - Verify SBOM and scan vulnerabilities
|
||||
- [security-sign-cosign](./security-sign-cosign.SKILL.md) - Sign artifacts with Cosign
|
||||
|
||||
## Notes
|
||||
|
||||
- **Production provenance MUST be generated by trusted build platform**
|
||||
- Local provenance generation is for testing only
|
||||
- SLSA Level 2 is the minimum recommended for production
|
||||
- Level 3 provides strongest guarantees but requires hermetic builds
|
||||
- Provenance verification requires network access to download attestations
|
||||
- GitHub attestations are public and verifiable by anyone
|
||||
- Provenance documents are immutable once generated
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Never trust artifacts without verified provenance
|
||||
- Always verify source URI matches expected repository
|
||||
- Require SLSA Level 2+ for production deployments
|
||||
- Provenance tampering indicates compromised supply chain
|
||||
- Provenance signature must be verified before trusting metadata
|
||||
- Local provenance generation bypasses security guarantees
|
||||
- Use SLSA-compliant build platforms (GitHub Actions, Cloud Build, etc.)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-10
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: slsa-framework/slsa-verifier
|
||||
**Documentation**: https://slsa.dev/
|
||||
|
||||
````
|
||||
+316
@@ -0,0 +1,316 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security Verify SBOM - Execution Script
|
||||
#
|
||||
# This script generates an SBOM for a Docker image or local file,
|
||||
# compares it with a baseline (if provided), and scans for vulnerabilities.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Set defaults
|
||||
set_default_env "SBOM_FORMAT" "spdx-json"
|
||||
set_default_env "VULN_SCAN_ENABLED" "true"
|
||||
|
||||
# Parse arguments
|
||||
TARGET="${1:-}"
|
||||
BASELINE="${2:-}"
|
||||
|
||||
if [[ -z "${TARGET}" ]]; then
|
||||
log_error "Usage: security-verify-sbom <target> [baseline]"
|
||||
log_error " target: Docker image tag or local image name (required)"
|
||||
log_error " baseline: Path to baseline SBOM for comparison (optional)"
|
||||
log_error ""
|
||||
log_error "Examples:"
|
||||
log_error " security-verify-sbom charon:local"
|
||||
log_error " security-verify-sbom ghcr.io/user/charon:latest"
|
||||
log_error " security-verify-sbom charon:test sbom-baseline.json"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Validate target format (basic validation)
|
||||
if [[ ! "${TARGET}" =~ ^[a-zA-Z0-9:/@._-]+$ ]]; then
|
||||
log_error "Invalid target format: ${TARGET}"
|
||||
log_error "Target must match pattern: [a-zA-Z0-9:/@._-]+"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Check required tools
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
|
||||
if ! command -v syft >/dev/null 2>&1; then
|
||||
log_error "syft is not installed"
|
||||
log_error "Install from: https://github.com/anchore/syft"
|
||||
log_error "Quick install: curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
log_error "jq is not installed"
|
||||
log_error "Install from: https://stedolan.github.io/jq/download/"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ "${VULN_SCAN_ENABLED}" == "true" ]] && ! command -v grype >/dev/null 2>&1; then
|
||||
log_error "grype is not installed (required for vulnerability scanning)"
|
||||
log_error "Install from: https://github.com/anchore/grype"
|
||||
log_error "Quick install: curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin"
|
||||
log_error ""
|
||||
log_error "Alternatively, disable vulnerability scanning with: VULN_SCAN_ENABLED=false"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Generate SBOM
|
||||
log_step "SBOM" "Generating SBOM for ${TARGET}"
|
||||
log_info "Format: ${SBOM_FORMAT}"
|
||||
|
||||
SBOM_OUTPUT="sbom-generated.json"
|
||||
|
||||
if ! syft "${TARGET}" -o "${SBOM_FORMAT}" > "${SBOM_OUTPUT}" 2>&1; then
|
||||
log_error "Failed to generate SBOM for ${TARGET}"
|
||||
log_error "Ensure the image exists locally or can be pulled from a registry"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Parse and validate SBOM
|
||||
if [[ ! -f "${SBOM_OUTPUT}" ]]; then
|
||||
log_error "SBOM file not generated: ${SBOM_OUTPUT}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate SBOM schema (SPDX format)
|
||||
log_info "Validating SBOM schema..."
|
||||
if ! jq -e '.spdxVersion' "${SBOM_OUTPUT}" >/dev/null 2>&1; then
|
||||
log_error "Invalid SBOM: missing spdxVersion field"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! jq -e '.packages' "${SBOM_OUTPUT}" >/dev/null 2>&1; then
|
||||
log_error "Invalid SBOM: missing packages array"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! jq -e '.name' "${SBOM_OUTPUT}" >/dev/null 2>&1; then
|
||||
log_error "Invalid SBOM: missing name field"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! jq -e '.documentNamespace' "${SBOM_OUTPUT}" >/dev/null 2>&1; then
|
||||
log_error "Invalid SBOM: missing documentNamespace field"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SPDX_VERSION=$(jq -r '.spdxVersion' "${SBOM_OUTPUT}")
|
||||
log_success "SBOM schema valid (${SPDX_VERSION})"
|
||||
|
||||
PACKAGE_COUNT=$(jq '.packages | length' "${SBOM_OUTPUT}" 2>/dev/null || echo "0")
|
||||
|
||||
if [[ "${PACKAGE_COUNT}" -eq 0 ]]; then
|
||||
log_warning "SBOM contains no packages - this may indicate an error"
|
||||
log_warning "Target: ${TARGET}"
|
||||
else
|
||||
log_success "Generated SBOM contains ${PACKAGE_COUNT} packages"
|
||||
fi
|
||||
|
||||
# Baseline comparison (if provided)
|
||||
if [[ -n "${BASELINE}" ]]; then
|
||||
log_step "BASELINE" "Comparing with baseline SBOM"
|
||||
|
||||
if [[ ! -f "${BASELINE}" ]]; then
|
||||
log_error "Baseline SBOM file not found: ${BASELINE}"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
BASELINE_COUNT=$(jq '.packages | length' "${BASELINE}" 2>/dev/null || echo "0")
|
||||
|
||||
if [[ "${BASELINE_COUNT}" -eq 0 ]]; then
|
||||
log_warning "Baseline SBOM appears empty or invalid"
|
||||
else
|
||||
log_info "Baseline: ${BASELINE_COUNT} packages, Current: ${PACKAGE_COUNT} packages"
|
||||
|
||||
# Calculate delta and variance using awk for float arithmetic
|
||||
DELTA=$((PACKAGE_COUNT - BASELINE_COUNT))
|
||||
if [[ "${BASELINE_COUNT}" -gt 0 ]]; then
|
||||
# Use awk to prevent integer overflow and get accurate percentage
|
||||
VARIANCE_PCT=$(awk -v delta="${DELTA}" -v baseline="${BASELINE_COUNT}" 'BEGIN {printf "%.2f", (delta / baseline) * 100}')
|
||||
VARIANCE_ABS=$(awk -v var="${VARIANCE_PCT}" 'BEGIN {print (var < 0 ? -var : var)}')
|
||||
else
|
||||
VARIANCE_PCT="0.00"
|
||||
VARIANCE_ABS="0.00"
|
||||
fi
|
||||
|
||||
if [[ "${DELTA}" -gt 0 ]]; then
|
||||
log_info "Delta: +${DELTA} packages (${VARIANCE_PCT}% increase)"
|
||||
elif [[ "${DELTA}" -lt 0 ]]; then
|
||||
log_info "Delta: ${DELTA} packages (${VARIANCE_PCT}% decrease)"
|
||||
else
|
||||
log_info "Delta: 0 packages (no change)"
|
||||
fi
|
||||
|
||||
# Extract package name@version tuples for semantic comparison
|
||||
jq -r '.packages[] | "\(.name)@\(.versionInfo // .version // "unknown")"' "${BASELINE}" 2>/dev/null | sort > baseline-packages.txt || true
|
||||
jq -r '.packages[] | "\(.name)@\(.versionInfo // .version // "unknown")"' "${SBOM_OUTPUT}" 2>/dev/null | sort > current-packages.txt || true
|
||||
|
||||
# Extract just names for package add/remove detection
|
||||
jq -r '.packages[].name' "${BASELINE}" 2>/dev/null | sort > baseline-names.txt || true
|
||||
jq -r '.packages[].name' "${SBOM_OUTPUT}" 2>/dev/null | sort > current-names.txt || true
|
||||
|
||||
# Find added packages
|
||||
ADDED=$(comm -13 baseline-names.txt current-names.txt 2>/dev/null || echo "")
|
||||
if [[ -n "${ADDED}" ]]; then
|
||||
log_info "Added packages:"
|
||||
echo "${ADDED}" | head -n 10 | while IFS= read -r pkg; do
|
||||
VERSION=$(jq -r ".packages[] | select(.name == \"${pkg}\") | .versionInfo // .version // \"unknown\"" "${SBOM_OUTPUT}" 2>/dev/null || echo "unknown")
|
||||
log_info " + ${pkg}@${VERSION}"
|
||||
done
|
||||
ADDED_COUNT=$(echo "${ADDED}" | wc -l)
|
||||
if [[ "${ADDED_COUNT}" -gt 10 ]]; then
|
||||
log_info " ... and $((ADDED_COUNT - 10)) more"
|
||||
fi
|
||||
else
|
||||
log_info "Added packages: (none)"
|
||||
fi
|
||||
|
||||
# Find removed packages
|
||||
REMOVED=$(comm -23 baseline-names.txt current-names.txt 2>/dev/null || echo "")
|
||||
if [[ -n "${REMOVED}" ]]; then
|
||||
log_info "Removed packages:"
|
||||
echo "${REMOVED}" | head -n 10 | while IFS= read -r pkg; do
|
||||
VERSION=$(jq -r ".packages[] | select(.name == \"${pkg}\") | .versionInfo // .version // \"unknown\"" "${BASELINE}" 2>/dev/null || echo "unknown")
|
||||
log_info " - ${pkg}@${VERSION}"
|
||||
done
|
||||
REMOVED_COUNT=$(echo "${REMOVED}" | wc -l)
|
||||
if [[ "${REMOVED_COUNT}" -gt 10 ]]; then
|
||||
log_info " ... and $((REMOVED_COUNT - 10)) more"
|
||||
fi
|
||||
else
|
||||
log_info "Removed packages: (none)"
|
||||
fi
|
||||
|
||||
# Detect version changes in existing packages
|
||||
log_info "Version changes:"
|
||||
CHANGED_COUNT=0
|
||||
comm -12 baseline-names.txt current-names.txt 2>/dev/null | while IFS= read -r pkg; do
|
||||
BASELINE_VER=$(jq -r ".packages[] | select(.name == \"${pkg}\") | .versionInfo // .version // \"unknown\"" "${BASELINE}" 2>/dev/null || echo "unknown")
|
||||
CURRENT_VER=$(jq -r ".packages[] | select(.name == \"${pkg}\") | .versionInfo // .version // \"unknown\"" "${SBOM_OUTPUT}" 2>/dev/null || echo "unknown")
|
||||
if [[ "${BASELINE_VER}" != "${CURRENT_VER}" ]]; then
|
||||
log_info " ~ ${pkg}: ${BASELINE_VER} → ${CURRENT_VER}"
|
||||
CHANGED_COUNT=$((CHANGED_COUNT + 1))
|
||||
if [[ "${CHANGED_COUNT}" -ge 10 ]]; then
|
||||
log_info " ... (showing first 10 changes)"
|
||||
break
|
||||
fi
|
||||
fi
|
||||
done
|
||||
if [[ "${CHANGED_COUNT}" -eq 0 ]]; then
|
||||
log_info " (none)"
|
||||
fi
|
||||
|
||||
# Warn if variance exceeds threshold (using awk for float comparison)
|
||||
EXCEEDS_THRESHOLD=$(awk -v abs="${VARIANCE_ABS}" 'BEGIN {print (abs > 5.0 ? 1 : 0)}')
|
||||
if [[ "${EXCEEDS_THRESHOLD}" -eq 1 ]]; then
|
||||
log_warning "Package variance (${VARIANCE_ABS}%) exceeds 5% threshold"
|
||||
log_warning "Consider manual review of package changes"
|
||||
fi
|
||||
|
||||
# Cleanup temporary files
|
||||
rm -f baseline-packages.txt current-packages.txt baseline-names.txt current-names.txt
|
||||
fi
|
||||
fi
|
||||
|
||||
# Vulnerability scanning (if enabled)
|
||||
HAS_CRITICAL=false
|
||||
|
||||
if [[ "${VULN_SCAN_ENABLED}" == "true" ]]; then
|
||||
log_step "VULN" "Scanning for vulnerabilities"
|
||||
|
||||
VULN_OUTPUT="vuln-results.json"
|
||||
|
||||
# Run Grype on the SBOM
|
||||
if grype "sbom:${SBOM_OUTPUT}" -o json > "${VULN_OUTPUT}" 2>&1; then
|
||||
log_debug "Vulnerability scan completed successfully"
|
||||
else
|
||||
GRYPE_EXIT=$?
|
||||
if [[ ${GRYPE_EXIT} -eq 1 ]]; then
|
||||
log_debug "Grype found vulnerabilities (expected)"
|
||||
else
|
||||
log_warning "Grype scan encountered an error (exit code: ${GRYPE_EXIT})"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Parse vulnerability counts by severity
|
||||
if [[ -f "${VULN_OUTPUT}" ]]; then
|
||||
CRITICAL_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Critical")] | length' "${VULN_OUTPUT}" 2>/dev/null || echo "0")
|
||||
HIGH_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "High")] | length' "${VULN_OUTPUT}" 2>/dev/null || echo "0")
|
||||
MEDIUM_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Medium")] | length' "${VULN_OUTPUT}" 2>/dev/null || echo "0")
|
||||
LOW_COUNT=$(jq '[.matches[] | select(.vulnerability.severity == "Low")] | length' "${VULN_OUTPUT}" 2>/dev/null || echo "0")
|
||||
|
||||
log_info "Found: ${CRITICAL_COUNT} Critical, ${HIGH_COUNT} High, ${MEDIUM_COUNT} Medium, ${LOW_COUNT} Low"
|
||||
|
||||
# Display critical vulnerabilities
|
||||
if [[ "${CRITICAL_COUNT}" -gt 0 ]]; then
|
||||
HAS_CRITICAL=true
|
||||
log_error "Critical vulnerabilities detected:"
|
||||
jq -r '.matches[] | select(.vulnerability.severity == "Critical") | " - \(.vulnerability.id) in \(.artifact.name)@\(.artifact.version) (CVSS: \(.vulnerability.cvss[0].metrics.baseScore // "N/A"))"' "${VULN_OUTPUT}" 2>/dev/null | head -n 10
|
||||
if [[ "${CRITICAL_COUNT}" -gt 10 ]]; then
|
||||
log_error " ... and $((CRITICAL_COUNT - 10)) more critical vulnerabilities"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Display high vulnerabilities
|
||||
if [[ "${HIGH_COUNT}" -gt 0 ]]; then
|
||||
log_warning "High severity vulnerabilities:"
|
||||
jq -r '.matches[] | select(.vulnerability.severity == "High") | " - \(.vulnerability.id) in \(.artifact.name)@\(.artifact.version) (CVSS: \(.vulnerability.cvss[0].metrics.baseScore // "N/A"))"' "${VULN_OUTPUT}" 2>/dev/null | head -n 5
|
||||
if [[ "${HIGH_COUNT}" -gt 5 ]]; then
|
||||
log_warning " ... and $((HIGH_COUNT - 5)) more high vulnerabilities"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Display table format for summary
|
||||
log_info "Running table format scan for summary..."
|
||||
grype "sbom:${SBOM_OUTPUT}" -o table 2>&1 | tail -n 20 || true
|
||||
else
|
||||
log_warning "Vulnerability scan results not found"
|
||||
fi
|
||||
else
|
||||
log_info "Vulnerability scanning disabled (air-gapped mode)"
|
||||
fi
|
||||
|
||||
# Final summary
|
||||
echo ""
|
||||
log_step "SUMMARY" "SBOM Verification Complete"
|
||||
log_info "Target: ${TARGET}"
|
||||
log_info "Packages: ${PACKAGE_COUNT}"
|
||||
if [[ -n "${BASELINE}" ]]; then
|
||||
log_info "Baseline comparison: ${VARIANCE_PCT}% variance"
|
||||
fi
|
||||
if [[ "${VULN_SCAN_ENABLED}" == "true" ]]; then
|
||||
log_info "Vulnerabilities: ${CRITICAL_COUNT} Critical, ${HIGH_COUNT} High, ${MEDIUM_COUNT} Medium, ${LOW_COUNT} Low"
|
||||
fi
|
||||
log_info "SBOM file: ${SBOM_OUTPUT}"
|
||||
|
||||
# Exit with appropriate code
|
||||
if [[ "${HAS_CRITICAL}" == "true" ]]; then
|
||||
log_error "CRITICAL vulnerabilities found - review required"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${HIGH_COUNT:-0}" -gt 0 ]]; then
|
||||
log_warning "High severity vulnerabilities found - review recommended"
|
||||
fi
|
||||
|
||||
log_success "Verification complete"
|
||||
exit 0
|
||||
@@ -0,0 +1,317 @@
|
||||
````markdown
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "security-verify-sbom"
|
||||
version: "1.0.0"
|
||||
description: "Verify SBOM completeness, scan for vulnerabilities, and perform semantic diff analysis"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "security"
|
||||
- "sbom"
|
||||
- "verification"
|
||||
- "supply-chain"
|
||||
- "vulnerability-scanning"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "syft"
|
||||
version: ">=1.17.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/anchore/syft"
|
||||
- name: "grype"
|
||||
version: ">=0.85.0"
|
||||
optional: false
|
||||
install_url: "https://github.com/anchore/grype"
|
||||
- name: "jq"
|
||||
version: ">=1.6"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "SBOM_FORMAT"
|
||||
description: "SBOM format (spdx-json, cyclonedx-json)"
|
||||
default: "spdx-json"
|
||||
required: false
|
||||
- name: "VULN_SCAN_ENABLED"
|
||||
description: "Enable vulnerability scanning"
|
||||
default: "true"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "target"
|
||||
type: "string"
|
||||
description: "Docker image or file path"
|
||||
required: true
|
||||
validation: "^[a-zA-Z0-9:/@._-]+$"
|
||||
- name: "baseline"
|
||||
type: "string"
|
||||
description: "Baseline SBOM file path for comparison"
|
||||
required: false
|
||||
default: ""
|
||||
- name: "vuln_scan"
|
||||
type: "boolean"
|
||||
description: "Run vulnerability scan"
|
||||
required: false
|
||||
default: true
|
||||
outputs:
|
||||
- name: "sbom_file"
|
||||
type: "file"
|
||||
description: "Generated SBOM in SPDX JSON format"
|
||||
- name: "scan_results"
|
||||
type: "stdout"
|
||||
description: "Verification results and vulnerability counts"
|
||||
- name: "exit_code"
|
||||
type: "number"
|
||||
description: "0 if no critical issues, 1 if critical vulnerabilities found, 2 if validation failed"
|
||||
metadata:
|
||||
category: "security"
|
||||
subcategory: "supply-chain"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
exit_codes:
|
||||
0: "Verification successful"
|
||||
1: "Verification failed or critical vulnerabilities found"
|
||||
2: "Missing dependencies or invalid parameters"
|
||||
---
|
||||
|
||||
# Security: Verify SBOM
|
||||
|
||||
Verify Software Bill of Materials (SBOM) completeness, scan for vulnerabilities, and perform semantic diff analysis.
|
||||
|
||||
## Overview
|
||||
|
||||
This skill generates an SBOM for Docker images or local files, compares it with a baseline (if provided), scans for known vulnerabilities using Grype, and reports any critical security issues. It supports both online vulnerability scanning and air-gapped operation modes.
|
||||
|
||||
## Features
|
||||
|
||||
- Generate SBOM in SPDX format (standardized)
|
||||
- Compare with baseline SBOM (semantic diff)
|
||||
- Scan for vulnerabilities (Critical/High/Medium/Low)
|
||||
- Validate SBOM structure and completeness
|
||||
- Support Docker images and local files
|
||||
- Air-gapped operation support (skip vulnerability scanning)
|
||||
- Detect added/removed packages between builds
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Syft 1.17.0 or higher (for SBOM generation)
|
||||
- Grype 0.85.0 or higher (for vulnerability scanning)
|
||||
- jq 1.6 or higher (for JSON processing)
|
||||
- Internet connection (for vulnerability database updates, unless air-gapped mode)
|
||||
- Docker (if scanning container images)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Verification
|
||||
|
||||
Run with default settings (generate SBOM + scan vulnerabilities):
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh security-verify-sbom ghcr.io/user/charon:latest
|
||||
```
|
||||
|
||||
### Verify Docker Image with Baseline Comparison
|
||||
|
||||
Compare current SBOM against a known baseline:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh security-verify-sbom \
|
||||
charon:local sbom-baseline.json
|
||||
```
|
||||
|
||||
### Air-Gapped Mode (No Vulnerability Scan)
|
||||
|
||||
Verify SBOM structure only, without network access:
|
||||
|
||||
```bash
|
||||
VULN_SCAN_ENABLED=false .github/skills/scripts/skill-runner.sh \
|
||||
security-verify-sbom charon:local
|
||||
```
|
||||
|
||||
### Custom SBOM Format
|
||||
|
||||
Generate SBOM in CycloneDX format:
|
||||
|
||||
```bash
|
||||
SBOM_FORMAT=cyclonedx-json .github/skills/scripts/skill-runner.sh \
|
||||
security-verify-sbom charon:local
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| target | string | Yes | - | Docker image tag or local image name |
|
||||
| baseline | string | No | "" | Path to baseline SBOM for comparison |
|
||||
| vuln_scan | boolean | No | true | Run vulnerability scan (set VULN_SCAN_ENABLED=false to disable) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| SBOM_FORMAT | No | spdx-json | SBOM format (spdx-json or cyclonedx-json) |
|
||||
| VULN_SCAN_ENABLED | No | true | Enable vulnerability scanning (set to false for air-gapped) |
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0 (no critical issues found)
|
||||
- **Error Exit Codes**:
|
||||
- 1: Critical vulnerabilities found or verification failed
|
||||
- 2: Missing dependencies or invalid parameters
|
||||
- **Generated Files**:
|
||||
- `sbom-generated.json`: Generated SBOM file
|
||||
- `vuln-results.json`: Vulnerability scan results (if enabled)
|
||||
- **Output**: Verification summary to stdout
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Verify Local Docker Image
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-verify-sbom charon:test
|
||||
[INFO] Generating SBOM for charon:test...
|
||||
[SBOM] Generated SBOM contains 247 packages
|
||||
[INFO] Scanning for vulnerabilities...
|
||||
[VULN] Found: 0 Critical, 2 High, 15 Medium, 42 Low
|
||||
[INFO] High vulnerabilities:
|
||||
- CVE-2023-12345 in golang.org/x/crypto (CVSS: 7.5)
|
||||
- CVE-2024-67890 in github.com/example/lib (CVSS: 8.2)
|
||||
[SUCCESS] Verification complete - review High severity vulnerabilities
|
||||
```
|
||||
|
||||
### Example 2: With Baseline Comparison
|
||||
|
||||
```bash
|
||||
$ .github/skills/scripts/skill-runner.sh security-verify-sbom \
|
||||
charon:latest sbom-baseline.json
|
||||
[INFO] Generating SBOM for charon:latest...
|
||||
[SBOM] Generated SBOM contains 247 packages
|
||||
[INFO] Comparing with baseline...
|
||||
[BASELINE] Baseline: 245 packages, Current: 247 packages
|
||||
[BASELINE] Delta: +2 packages (0.8% increase)
|
||||
[BASELINE] Added packages:
|
||||
- golang.org/x/crypto@v0.30.0
|
||||
- github.com/pkg/errors@v0.9.1
|
||||
[BASELINE] Removed packages: (none)
|
||||
[INFO] Scanning for vulnerabilities...
|
||||
[VULN] Found: 0 Critical, 0 High, 5 Medium, 20 Low
|
||||
[SUCCESS] Verification complete (0.8% variance from baseline)
|
||||
```
|
||||
|
||||
### Example 3: Air-Gapped Mode
|
||||
|
||||
```bash
|
||||
$ VULN_SCAN_ENABLED=false .github/skills/scripts/skill-runner.sh \
|
||||
security-verify-sbom charon:local
|
||||
[INFO] Generating SBOM for charon:local...
|
||||
[SBOM] Generated SBOM contains 247 packages
|
||||
[INFO] Vulnerability scanning disabled (air-gapped mode)
|
||||
[SUCCESS] SBOM generation complete
|
||||
```
|
||||
|
||||
### Example 4: CI/CD Pipeline Integration
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Verify SBOM
|
||||
run: |
|
||||
.github/skills/scripts/skill-runner.sh \
|
||||
security-verify-sbom ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
continue-on-error: false
|
||||
```
|
||||
|
||||
## Semantic Diff Analysis
|
||||
|
||||
When a baseline SBOM is provided, the skill performs semantic comparison:
|
||||
|
||||
1. **Package Count Comparison**: Reports total package delta
|
||||
2. **Added Packages**: Lists new dependencies with versions
|
||||
3. **Removed Packages**: Lists removed dependencies
|
||||
4. **Variance Percentage**: Calculates percentage change
|
||||
5. **Threshold Check**: Warns if variance exceeds 5%
|
||||
|
||||
## Vulnerability Severity Thresholds
|
||||
|
||||
**Project Standards**:
|
||||
- **CRITICAL**: Must fix before release (blocking) - **Script exits with code 1**
|
||||
- **HIGH**: Should fix before release (warning) - **Script continues but logs warning**
|
||||
- **MEDIUM**: Fix in next release cycle (informational)
|
||||
- **LOW**: Optional, fix as time permits
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Syft not installed**:
|
||||
```bash
|
||||
Error: syft command not found
|
||||
Solution: Install Syft from https://github.com/anchore/syft
|
||||
```
|
||||
|
||||
**Grype not installed**:
|
||||
```bash
|
||||
Error: grype command not found
|
||||
Solution: Install Grype from https://github.com/anchore/grype
|
||||
```
|
||||
|
||||
**Docker image not found**:
|
||||
```bash
|
||||
Error: Unable to find image 'charon:test' locally
|
||||
Solution: Build the image or pull from registry
|
||||
```
|
||||
|
||||
**Invalid baseline SBOM**:
|
||||
```bash
|
||||
Error: Baseline SBOM file not found: sbom-baseline.json
|
||||
Solution: Verify the file path or omit baseline parameter
|
||||
```
|
||||
|
||||
**Network timeout (vulnerability scan)**:
|
||||
```bash
|
||||
Warning: Failed to update vulnerability database
|
||||
Solution: Check internet connection or use air-gapped mode (VULN_SCAN_ENABLED=false)
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- **0**: Verification successful, no critical vulnerabilities
|
||||
- **1**: Critical vulnerabilities found or verification failed
|
||||
- **2**: Missing dependencies or invalid parameters
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [security-sign-cosign](./security-sign-cosign.SKILL.md) - Sign artifacts with Cosign
|
||||
- [security-slsa-provenance](./security-slsa-provenance.SKILL.md) - Generate SLSA provenance
|
||||
- [security-scan-trivy](./security-scan-trivy.SKILL.md) - Alternative vulnerability scanner
|
||||
|
||||
## Notes
|
||||
|
||||
- SBOM generation requires read access to Docker images
|
||||
- Vulnerability database is updated automatically by Grype
|
||||
- Baseline comparison is optional but recommended for drift detection
|
||||
- Critical vulnerabilities will cause the script to exit with code 1
|
||||
- High vulnerabilities generate warnings but don't block execution
|
||||
- Use air-gapped mode when network access is unavailable
|
||||
- SPDX format is standardized and recommended over CycloneDX
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Never commit SBOM files containing sensitive information
|
||||
- Review all High and Critical vulnerabilities before deployment
|
||||
- Baseline drift >5% should trigger manual review
|
||||
- Air-gapped mode skips vulnerability scanning - use with caution
|
||||
- SBOM files can reveal internal architecture - protect accordingly
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-10
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: Syft (SBOM generation) + Grype (vulnerability scanning)
|
||||
|
||||
````
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test Backend Coverage - Execution Script
|
||||
#
|
||||
# This script wraps the legacy go-test-coverage.sh script while providing
|
||||
# the Agent Skills interface and logging.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Helper scripts are in .github/skills/scripts/
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script (skills/skill-name-scripts/run.sh -> project root)
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_go_environment "1.23" || error_exit "Go 1.23+ is required"
|
||||
validate_python_environment "3.8" || error_exit "Python 3.8+ is required"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "backend" "scripts/go-test-coverage.sh" || error_exit "Invalid project structure"
|
||||
|
||||
# Set default environment variables
|
||||
set_default_env "CHARON_MIN_COVERAGE" "85"
|
||||
set_default_env "PERF_MAX_MS_GETSTATUS_P95" "25ms"
|
||||
set_default_env "PERF_MAX_MS_GETSTATUS_P95_PARALLEL" "50ms"
|
||||
set_default_env "PERF_MAX_MS_LISTDECISIONS_P95" "75ms"
|
||||
|
||||
# Execute the legacy script
|
||||
log_step "EXECUTION" "Running backend tests with coverage"
|
||||
log_info "Minimum coverage: ${CHARON_MIN_COVERAGE}%"
|
||||
|
||||
LEGACY_SCRIPT="${PROJECT_ROOT}/scripts/go-test-coverage.sh"
|
||||
check_file_exists "${LEGACY_SCRIPT}"
|
||||
|
||||
# Execute with proper error handling
|
||||
if "${LEGACY_SCRIPT}" "$@"; then
|
||||
log_success "Backend coverage tests passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "Backend coverage tests failed (exit code: ${exit_code})"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,212 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-backend-coverage"
|
||||
version: "1.0.0"
|
||||
description: "Run Go backend tests with coverage analysis and threshold validation (minimum 85%)"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "coverage"
|
||||
- "go"
|
||||
- "backend"
|
||||
- "validation"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "go"
|
||||
version: ">=1.23"
|
||||
optional: false
|
||||
- name: "python3"
|
||||
version: ">=3.8"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "CHARON_MIN_COVERAGE"
|
||||
description: "Minimum coverage percentage required (overrides default)"
|
||||
default: "85"
|
||||
required: false
|
||||
- name: "CPM_MIN_COVERAGE"
|
||||
description: "Alternative name for minimum coverage threshold (legacy)"
|
||||
default: "85"
|
||||
required: false
|
||||
- name: "PERF_MAX_MS_GETSTATUS_P95"
|
||||
description: "Maximum P95 latency for GetStatus endpoint (ms)"
|
||||
default: "25ms"
|
||||
required: false
|
||||
- name: "PERF_MAX_MS_GETSTATUS_P95_PARALLEL"
|
||||
description: "Maximum P95 latency for parallel GetStatus calls (ms)"
|
||||
default: "50ms"
|
||||
required: false
|
||||
- name: "PERF_MAX_MS_LISTDECISIONS_P95"
|
||||
description: "Maximum P95 latency for ListDecisions endpoint (ms)"
|
||||
default: "75ms"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose test output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "coverage.txt"
|
||||
type: "file"
|
||||
description: "Go coverage profile in text format"
|
||||
path: "backend/coverage.txt"
|
||||
- name: "coverage_summary"
|
||||
type: "stdout"
|
||||
description: "Summary of coverage statistics and validation result"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "coverage"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test Backend Coverage
|
||||
|
||||
## Overview
|
||||
|
||||
Executes the Go backend test suite with race detection enabled, generates a coverage profile, filters excluded packages, and validates that the total coverage meets or exceeds the configured threshold (default: 85%).
|
||||
|
||||
This skill is designed for continuous integration and pre-commit hooks to ensure code quality standards are maintained.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Go 1.23 or higher installed and in PATH
|
||||
- Python 3.8 or higher installed and in PATH
|
||||
- Backend dependencies installed (`cd backend && go mod download`)
|
||||
- Write permissions in `backend/` directory (for coverage.txt)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run with default settings (85% minimum coverage):
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
### Custom Coverage Threshold
|
||||
|
||||
Set a custom minimum coverage percentage:
|
||||
|
||||
```bash
|
||||
export CHARON_MIN_COVERAGE=90
|
||||
.github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run Backend Tests with Coverage
|
||||
run: .github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
env:
|
||||
CHARON_MIN_COVERAGE: 85
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose test output (-v flag) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| CHARON_MIN_COVERAGE | No | 85 | Minimum coverage percentage required for success |
|
||||
| CPM_MIN_COVERAGE | No | 85 | Legacy name for minimum coverage (fallback) |
|
||||
| PERF_MAX_MS_GETSTATUS_P95 | No | 25ms | Max P95 latency for GetStatus endpoint |
|
||||
| PERF_MAX_MS_GETSTATUS_P95_PARALLEL | No | 50ms | Max P95 latency for parallel GetStatus |
|
||||
| PERF_MAX_MS_LISTDECISIONS_P95 | No | 75ms | Max P95 latency for ListDecisions endpoint |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed and coverage meets threshold
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: Coverage below threshold or coverage file generation failed
|
||||
- **Non-zero**: Tests failed or other error occurred
|
||||
|
||||
### Output Files
|
||||
- **backend/coverage.txt**: Go coverage profile (text format)
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
Filtering excluded packages from coverage report...
|
||||
Coverage filtering complete
|
||||
total: (statements) 87.4%
|
||||
Computed coverage: 87.4% (minimum required 85%)
|
||||
Coverage requirement met
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
### Example 2: Higher Coverage Threshold
|
||||
|
||||
```bash
|
||||
export CHARON_MIN_COVERAGE=90
|
||||
.github/skills/scripts/skill-runner.sh test-backend-coverage
|
||||
```
|
||||
|
||||
## Excluded Packages
|
||||
|
||||
The following packages are excluded from coverage analysis:
|
||||
- `github.com/Wikid82/charon/backend/cmd/api` - API server entrypoint
|
||||
- `github.com/Wikid82/charon/backend/cmd/seed` - Database seeding tool
|
||||
- `github.com/Wikid82/charon/backend/internal/logger` - Logging infrastructure
|
||||
- `github.com/Wikid82/charon/backend/internal/metrics` - Metrics infrastructure
|
||||
- `github.com/Wikid82/charon/backend/internal/trace` - Tracing infrastructure
|
||||
- `github.com/Wikid82/charon/backend/integration` - Integration test utilities
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: coverage file not generated by go test
|
||||
**Solution**: Review test output for failures; fix failing tests
|
||||
|
||||
#### Error: go tool cover failed or timed out
|
||||
**Solution**: Clear Go cache and re-run tests
|
||||
|
||||
#### Error: Coverage X% is below required Y%
|
||||
**Solution**: Add tests for uncovered code paths or adjust threshold
|
||||
|
||||
## Related Skills
|
||||
|
||||
- test-backend-unit - Fast unit tests without coverage
|
||||
- security-check-govulncheck - Go vulnerability scanning
|
||||
- utility-cache-clear-go - Clear Go build cache
|
||||
|
||||
## Notes
|
||||
|
||||
- **Race Detection**: Always runs with `-race` flag enabled (adds ~30% overhead)
|
||||
- **Coverage Filtering**: Excluded packages are defined in the script itself
|
||||
- **Python Dependency**: Uses Python for decimal-precision coverage comparison
|
||||
- **Timeout Protection**: Coverage generation has a 60-second timeout
|
||||
- **Idempotency**: Safe to run multiple times; cleans up old coverage files
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/go-test-coverage.sh`
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test Backend Unit - Execution Script
|
||||
#
|
||||
# This script runs Go backend unit tests without coverage analysis,
|
||||
# providing fast test execution for development workflows.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Helper scripts are in .github/skills/scripts/
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script (skills/skill-name-scripts/run.sh -> project root)
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_go_environment "1.23" || error_exit "Go 1.23+ is required"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "backend" || error_exit "Invalid project structure"
|
||||
|
||||
# Change to backend directory
|
||||
cd "${PROJECT_ROOT}/backend"
|
||||
|
||||
# Execute tests
|
||||
log_step "EXECUTION" "Running backend unit tests"
|
||||
|
||||
# Check if short mode is enabled
|
||||
SHORT_FLAG=""
|
||||
if [[ "${CHARON_TEST_SHORT:-false}" == "true" ]]; then
|
||||
SHORT_FLAG="-short"
|
||||
log_info "Running in short mode (skipping integration and heavy network tests)"
|
||||
fi
|
||||
|
||||
# Run tests with gotestsum if available, otherwise fall back to go test
|
||||
if command -v gotestsum &> /dev/null; then
|
||||
if gotestsum --format pkgname -- $SHORT_FLAG "$@" ./...; then
|
||||
log_success "Backend unit tests passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "Backend unit tests failed (exit code: ${exit_code})"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
else
|
||||
if go test $SHORT_FLAG "$@" ./...; then
|
||||
log_success "Backend unit tests passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "Backend unit tests failed (exit code: ${exit_code})"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
fi
|
||||
@@ -0,0 +1,191 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-backend-unit"
|
||||
version: "1.0.0"
|
||||
description: "Run Go backend unit tests without coverage analysis (fast execution)"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "unit-tests"
|
||||
- "go"
|
||||
- "backend"
|
||||
- "fast"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "go"
|
||||
version: ">=1.23"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose test output"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "package"
|
||||
type: "string"
|
||||
description: "Specific package to test (e.g., ./internal/...)"
|
||||
default: "./..."
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "Go test output showing pass/fail status"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "unit"
|
||||
execution_time: "short"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test Backend Unit
|
||||
|
||||
## Overview
|
||||
|
||||
Executes the Go backend unit test suite without coverage analysis. This skill provides fast test execution for quick feedback during development, making it ideal for pre-commit checks and rapid iteration.
|
||||
|
||||
Unlike test-backend-coverage, this skill does not generate coverage reports or enforce coverage thresholds, focusing purely on test pass/fail status.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Go 1.23 or higher installed and in PATH
|
||||
- Backend dependencies installed (`cd backend && go mod download`)
|
||||
- Sufficient disk space for test artifacts
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run all backend unit tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit
|
||||
```
|
||||
|
||||
### Test Specific Package
|
||||
|
||||
Test only a specific package or module:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- ./internal/handlers/...
|
||||
```
|
||||
|
||||
### Verbose Output
|
||||
|
||||
Enable verbose test output for debugging:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- -v
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run Backend Unit Tests
|
||||
run: .github/skills/scripts/skill-runner.sh test-backend-unit
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose test output (-v flag) |
|
||||
| package | string | No | ./... | Package pattern to test |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
No environment variables are required for this skill.
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **Non-zero**: One or more tests failed
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
ok github.com/Wikid82/charon/backend/internal/handlers 0.523s
|
||||
ok github.com/Wikid82/charon/backend/internal/models 0.189s
|
||||
ok github.com/Wikid82/charon/backend/internal/services 0.742s
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit
|
||||
```
|
||||
|
||||
### Example 2: Test Specific Package
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- ./internal/handlers
|
||||
```
|
||||
|
||||
### Example 3: Verbose Output
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- -v
|
||||
```
|
||||
|
||||
### Example 4: Run with Race Detection
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- -race
|
||||
```
|
||||
|
||||
### Example 5: Short Mode (Skip Long Tests)
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-backend-unit -- -short
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: package not found
|
||||
**Solution**: Verify package path is correct; run `go list ./...` to see available packages
|
||||
|
||||
#### Error: build failed
|
||||
**Solution**: Fix compilation errors; run `go build ./...` to identify issues
|
||||
|
||||
#### Error: test timeout
|
||||
**Solution**: Increase timeout with `-timeout` flag or fix hanging tests
|
||||
|
||||
## Related Skills
|
||||
|
||||
- test-backend-coverage - Run tests with coverage analysis (slower)
|
||||
- build-check-go - Verify Go builds without running tests
|
||||
- security-check-govulncheck - Go vulnerability scanning
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Fast execution (~5-10 seconds typical)
|
||||
- **No Coverage**: Does not generate coverage reports
|
||||
- **Race Detection**: Not enabled by default (unlike test-backend-coverage)
|
||||
- **Idempotency**: Safe to run multiple times
|
||||
- **Caching**: Benefits from Go test cache for unchanged packages
|
||||
- **Suitable For**: Pre-commit hooks, quick feedback, TDD workflows
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: Inline task command
|
||||
+294
@@ -0,0 +1,294 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test E2E Playwright Coverage - Execution Script
|
||||
#
|
||||
# Runs Playwright end-to-end tests with code coverage collection
|
||||
# using @bgotink/playwright-coverage.
|
||||
#
|
||||
# IMPORTANT: For accurate source-level coverage, this script starts
|
||||
# the Vite dev server (localhost:5173) which proxies API calls to
|
||||
# the Docker backend (localhost:8080). V8 coverage requires source
|
||||
# files to be accessible on the test host.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Default parameter values
|
||||
PROJECT="chromium"
|
||||
VITE_PID=""
|
||||
VITE_PORT="${VITE_PORT:-5173}" # Default Vite port (avoids conflicts with common ports)
|
||||
BACKEND_URL="http://localhost:8080"
|
||||
|
||||
# Cleanup function to kill Vite dev server on exit
|
||||
cleanup() {
|
||||
if [[ -n "${VITE_PID}" ]] && kill -0 "${VITE_PID}" 2>/dev/null; then
|
||||
log_info "Stopping Vite dev server (PID: ${VITE_PID})..."
|
||||
kill "${VITE_PID}" 2>/dev/null || true
|
||||
wait "${VITE_PID}" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
# Set up trap for cleanup
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# Parse command-line arguments
|
||||
parse_arguments() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--project=*)
|
||||
PROJECT="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--project)
|
||||
PROJECT="${2:-chromium}"
|
||||
shift 2
|
||||
;;
|
||||
--skip-vite)
|
||||
SKIP_VITE="true"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
show_help
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_warning "Unknown argument: $1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Show help message
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Usage: run.sh [OPTIONS]
|
||||
|
||||
Run Playwright E2E tests with coverage collection.
|
||||
|
||||
Coverage requires the Vite dev server to serve source files directly.
|
||||
This script automatically starts Vite at localhost:5173, which proxies
|
||||
API calls to the Docker backend at localhost:8080.
|
||||
|
||||
Options:
|
||||
--project=PROJECT Browser project to run (chromium, firefox, webkit)
|
||||
Default: chromium
|
||||
--skip-vite Skip starting Vite dev server (use existing server)
|
||||
-h, --help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
PLAYWRIGHT_BASE_URL Override test URL (default: http://localhost:5173)
|
||||
VITE_PORT Vite dev server port (default: 5173)
|
||||
CI Set to 'true' for CI environment
|
||||
|
||||
Prerequisites:
|
||||
- Docker backend running at localhost:8080
|
||||
- Node.js dependencies installed (npm ci)
|
||||
|
||||
Examples:
|
||||
run.sh # Start Vite, run tests with coverage
|
||||
run.sh --project=firefox # Run in Firefox with coverage
|
||||
run.sh --skip-vite # Use existing Vite server
|
||||
EOF
|
||||
}
|
||||
|
||||
# Validate project parameter
|
||||
validate_project() {
|
||||
local valid_projects=("chromium" "firefox" "webkit")
|
||||
local project_lower
|
||||
project_lower=$(echo "${PROJECT}" | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
for valid in "${valid_projects[@]}"; do
|
||||
if [[ "${project_lower}" == "${valid}" ]]; then
|
||||
PROJECT="${project_lower}"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
error_exit "Invalid project '${PROJECT}'. Valid options: chromium, firefox, webkit"
|
||||
}
|
||||
|
||||
# Check if backend is running
|
||||
check_backend() {
|
||||
log_info "Checking backend at ${BACKEND_URL}..."
|
||||
local max_attempts=5
|
||||
local attempt=1
|
||||
|
||||
while [[ ${attempt} -le ${max_attempts} ]]; do
|
||||
if curl -sf "${BACKEND_URL}/api/v1/health" >/dev/null 2>&1; then
|
||||
log_success "Backend is healthy"
|
||||
return 0
|
||||
fi
|
||||
log_info "Waiting for backend... (attempt ${attempt}/${max_attempts})"
|
||||
sleep 2
|
||||
((attempt++))
|
||||
done
|
||||
|
||||
log_warning "Backend not responding at ${BACKEND_URL}"
|
||||
log_warning "Coverage tests require Docker backend. Start with:"
|
||||
log_warning " docker compose -f .docker/compose/docker-compose.local.yml up -d"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Start Vite dev server
|
||||
start_vite() {
|
||||
local vite_url="http://localhost:${VITE_PORT}"
|
||||
|
||||
# Check if Vite is already running on our preferred port
|
||||
if curl -sf "${vite_url}" >/dev/null 2>&1; then
|
||||
log_info "Vite dev server already running at ${vite_url}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_step "VITE" "Starting Vite dev server"
|
||||
cd "${PROJECT_ROOT}/frontend"
|
||||
|
||||
# Ensure dependencies are installed
|
||||
if [[ ! -d "node_modules" ]]; then
|
||||
log_info "Installing frontend dependencies..."
|
||||
npm ci --silent
|
||||
fi
|
||||
|
||||
# Start Vite in background with explicit port
|
||||
log_command "npx vite --port ${VITE_PORT} (background)"
|
||||
npx vite --port "${VITE_PORT}" > /tmp/vite.log 2>&1 &
|
||||
VITE_PID=$!
|
||||
|
||||
# Wait for Vite to be ready (check log for actual port in case of conflict)
|
||||
log_info "Waiting for Vite to start..."
|
||||
local max_wait=60
|
||||
local waited=0
|
||||
local actual_port="${VITE_PORT}"
|
||||
|
||||
while [[ ${waited} -lt ${max_wait} ]]; do
|
||||
# Check if Vite logged its ready message with actual port
|
||||
if grep -q "Local:" /tmp/vite.log 2>/dev/null; then
|
||||
# Extract actual port from Vite log (handles port conflict auto-switch)
|
||||
actual_port=$(grep -oP 'localhost:\K[0-9]+' /tmp/vite.log 2>/dev/null | head -1 || echo "${VITE_PORT}")
|
||||
vite_url="http://localhost:${actual_port}"
|
||||
fi
|
||||
|
||||
if curl -sf "${vite_url}" >/dev/null 2>&1; then
|
||||
# Update VITE_PORT if Vite chose a different port
|
||||
if [[ "${actual_port}" != "${VITE_PORT}" ]]; then
|
||||
log_warning "Port ${VITE_PORT} was busy, Vite using port ${actual_port}"
|
||||
VITE_PORT="${actual_port}"
|
||||
fi
|
||||
log_success "Vite dev server ready at ${vite_url}"
|
||||
cd "${PROJECT_ROOT}"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
((waited++))
|
||||
done
|
||||
|
||||
log_error "Vite failed to start within ${max_wait} seconds"
|
||||
log_error "Vite log:"
|
||||
cat /tmp/vite.log 2>/dev/null || true
|
||||
cd "${PROJECT_ROOT}"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
SKIP_VITE="${SKIP_VITE:-false}"
|
||||
parse_arguments "$@"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_node_environment "18.0" || error_exit "Node.js 18+ is required"
|
||||
check_command_exists "npx" "npx is required (part of Node.js installation)"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "tests" "playwright.config.js" "package.json" || error_exit "Invalid project structure"
|
||||
|
||||
# Validate project parameter
|
||||
validate_project
|
||||
|
||||
# Check backend is running (required for API proxy)
|
||||
log_step "BACKEND" "Checking Docker backend"
|
||||
if ! check_backend; then
|
||||
error_exit "Backend not available. Coverage tests require Docker backend at ${BACKEND_URL}"
|
||||
fi
|
||||
|
||||
# Start Vite dev server for coverage (unless skipped)
|
||||
if [[ "${SKIP_VITE}" != "true" ]]; then
|
||||
start_vite || error_exit "Failed to start Vite dev server"
|
||||
fi
|
||||
|
||||
# Ensure coverage directory exists
|
||||
log_step "SETUP" "Creating coverage directory"
|
||||
mkdir -p coverage/e2e
|
||||
|
||||
# Set environment variables
|
||||
# IMPORTANT: Use Vite URL (3000) for coverage, not Docker (8080)
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
export PLAYWRIGHT_BASE_URL="${PLAYWRIGHT_BASE_URL:-http://localhost:${VITE_PORT}}"
|
||||
|
||||
# Log configuration
|
||||
log_step "CONFIG" "Test configuration"
|
||||
log_info "Project: ${PROJECT}"
|
||||
log_info "Test URL: ${PLAYWRIGHT_BASE_URL}"
|
||||
log_info "Backend URL: ${BACKEND_URL}"
|
||||
log_info "Coverage output: ${PROJECT_ROOT}/coverage/e2e/"
|
||||
log_info ""
|
||||
log_info "Coverage architecture:"
|
||||
log_info " Tests → Vite (localhost:${VITE_PORT}) → serves source files"
|
||||
log_info " Vite → Docker (localhost:8080) → API proxy"
|
||||
|
||||
# Execute Playwright tests with coverage
|
||||
log_step "EXECUTION" "Running Playwright E2E tests with coverage"
|
||||
log_command "npx playwright test --project=${PROJECT}"
|
||||
|
||||
local exit_code=0
|
||||
if npx playwright test --project="${PROJECT}"; then
|
||||
log_success "All E2E tests passed"
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "E2E tests failed (exit code: ${exit_code})"
|
||||
fi
|
||||
|
||||
# Check if coverage was generated
|
||||
log_step "COVERAGE" "Checking coverage output"
|
||||
if [[ -f "coverage/e2e/lcov.info" ]]; then
|
||||
log_success "E2E coverage generated: coverage/e2e/lcov.info"
|
||||
|
||||
# Print summary if coverage.json exists
|
||||
if [[ -f "coverage/e2e/coverage.json" ]] && command -v jq &> /dev/null; then
|
||||
log_info "📊 Coverage Summary:"
|
||||
jq '.total' coverage/e2e/coverage.json 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Show file sizes
|
||||
log_info "Coverage files:"
|
||||
ls -lh coverage/e2e/ 2>/dev/null || true
|
||||
else
|
||||
log_warning "No coverage data generated"
|
||||
log_warning "Ensure test files import from '@bgotink/playwright-coverage'"
|
||||
fi
|
||||
|
||||
# Output report locations
|
||||
log_step "REPORTS" "Report locations"
|
||||
log_info "Coverage HTML: ${PROJECT_ROOT}/coverage/e2e/index.html"
|
||||
log_info "Coverage LCOV: ${PROJECT_ROOT}/coverage/e2e/lcov.info"
|
||||
log_info "Playwright Report: ${PROJECT_ROOT}/playwright-report/index.html"
|
||||
|
||||
exit "${exit_code}"
|
||||
}
|
||||
|
||||
# Run main with all arguments
|
||||
main "$@"
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-e2e-playwright-coverage"
|
||||
version: "1.0.0"
|
||||
description: "Run Playwright E2E tests with code coverage collection using @bgotink/playwright-coverage"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "e2e"
|
||||
- "playwright"
|
||||
- "coverage"
|
||||
- "integration"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "node"
|
||||
version: ">=18.0"
|
||||
optional: false
|
||||
- name: "npx"
|
||||
version: ">=1.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "PLAYWRIGHT_BASE_URL"
|
||||
description: "Base URL of the Charon application under test"
|
||||
default: "http://localhost:8080"
|
||||
required: false
|
||||
- name: "PLAYWRIGHT_HTML_OPEN"
|
||||
description: "Controls HTML report auto-open behavior (set to 'never' for CI/non-interactive)"
|
||||
default: "never"
|
||||
required: false
|
||||
- name: "CI"
|
||||
description: "Set to 'true' when running in CI environment"
|
||||
default: ""
|
||||
required: false
|
||||
parameters:
|
||||
- name: "project"
|
||||
type: "string"
|
||||
description: "Browser project to run (chromium, firefox, webkit)"
|
||||
default: "chromium"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "coverage-e2e"
|
||||
type: "directory"
|
||||
description: "E2E coverage output directory with LCOV and HTML reports"
|
||||
path: "coverage/e2e/"
|
||||
- name: "playwright-report"
|
||||
type: "directory"
|
||||
description: "HTML test report directory"
|
||||
path: "playwright-report/"
|
||||
- name: "test-results"
|
||||
type: "directory"
|
||||
description: "Test artifacts and traces"
|
||||
path: "test-results/"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "e2e-coverage"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test E2E Playwright Coverage
|
||||
|
||||
## Overview
|
||||
|
||||
Runs Playwright end-to-end tests with code coverage collection using `@bgotink/playwright-coverage`. This skill collects V8 coverage data during test execution and generates reports in LCOV, HTML, and JSON formats suitable for upload to Codecov.
|
||||
|
||||
**IMPORTANT**: This skill starts the **Vite dev server** (not Docker) because V8 coverage requires access to source files. Running coverage against the Docker container will result in `0%` coverage.
|
||||
|
||||
| Mode | Base URL | Coverage Support |
|
||||
|------|----------|-----------------|
|
||||
| Docker (`localhost:8080`) | ❌ No - Shows "Unknown% (0/0)" |
|
||||
| Vite Dev (`localhost:5173`) | ✅ Yes - Real coverage data |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.0 or higher installed and in PATH
|
||||
- Playwright browsers installed (`npx playwright install`)
|
||||
- `@bgotink/playwright-coverage` package installed
|
||||
- Charon application running (default: `http://localhost:8080`)
|
||||
- Test files in `tests/` directory using coverage-enabled imports
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run E2E tests with coverage collection:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage
|
||||
```
|
||||
|
||||
### Browser Selection
|
||||
|
||||
Run tests in a specific browser:
|
||||
|
||||
```bash
|
||||
# Chromium (default)
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage --project=chromium
|
||||
|
||||
# Firefox
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage --project=firefox
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run E2E Tests with Coverage
|
||||
run: .github/skills/scripts/skill-runner.sh test-e2e-playwright-coverage
|
||||
env:
|
||||
PLAYWRIGHT_BASE_URL: http://localhost:8080
|
||||
CI: true
|
||||
|
||||
- name: Upload E2E Coverage to Codecov
|
||||
uses: codecov/codecov-action@v5
|
||||
with:
|
||||
files: ./coverage/e2e/lcov.info
|
||||
flags: e2e
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| project | string | No | chromium | Browser project: chromium, firefox, webkit |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| PLAYWRIGHT_BASE_URL | No | http://localhost:8080 | Application URL to test against |
|
||||
| PLAYWRIGHT_HTML_OPEN | No | never | HTML report auto-open behavior |
|
||||
| CI | No | "" | Set to "true" for CI environment behavior |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed and coverage generated
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **Non-zero**: Configuration or execution error
|
||||
|
||||
### Output Directories
|
||||
- **coverage/e2e/**: Coverage reports (LCOV, HTML, JSON)
|
||||
- `lcov.info` - LCOV format for Codecov upload
|
||||
- `coverage.json` - JSON format for programmatic access
|
||||
- `index.html` - HTML report for visual inspection
|
||||
- **playwright-report/**: HTML test report with results and traces
|
||||
- **test-results/**: Test artifacts, screenshots, and trace files
|
||||
|
||||
## Viewing Coverage Reports
|
||||
|
||||
### Coverage HTML Report
|
||||
|
||||
```bash
|
||||
# Open coverage HTML report
|
||||
open coverage/e2e/index.html
|
||||
```
|
||||
|
||||
### Playwright Test Report
|
||||
|
||||
```bash
|
||||
npx playwright show-report --port 9323
|
||||
```
|
||||
|
||||
## Coverage Data Format
|
||||
|
||||
The skill generates coverage in multiple formats:
|
||||
|
||||
| Format | File | Purpose |
|
||||
|--------|------|---------|
|
||||
| LCOV | `coverage/e2e/lcov.info` | Codecov upload |
|
||||
| HTML | `coverage/e2e/index.html` | Visual inspection |
|
||||
| JSON | `coverage/e2e/coverage.json` | Programmatic access |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- test-e2e-playwright - E2E tests without coverage
|
||||
- test-frontend-coverage - Frontend unit test coverage with Vitest
|
||||
- test-backend-coverage - Backend unit test coverage with Go
|
||||
|
||||
## Notes
|
||||
|
||||
- **Coverage Source**: Uses V8 coverage (native, no instrumentation needed)
|
||||
- **Performance**: ~5-10% overhead compared to tests without coverage
|
||||
- **Sharding**: When running sharded tests in CI, coverage files must be merged
|
||||
- **LCOV Merge**: Use `lcov -a file1.info -a file2.info -o merged.info` to merge
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-18
|
||||
**Maintained by**: Charon Project Team
|
||||
+289
@@ -0,0 +1,289 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test E2E Playwright Debug - Execution Script
|
||||
#
|
||||
# Runs Playwright E2E tests in headed/debug mode with slow motion,
|
||||
# optional Inspector, and trace collection for troubleshooting.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Default parameter values
|
||||
FILE=""
|
||||
GREP=""
|
||||
SLOWMO=500
|
||||
INSPECTOR=false
|
||||
PROJECT="chromium"
|
||||
|
||||
# Parse command-line arguments
|
||||
parse_arguments() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--file=*)
|
||||
FILE="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--file)
|
||||
FILE="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--grep=*)
|
||||
GREP="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--grep)
|
||||
GREP="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--slowmo=*)
|
||||
SLOWMO="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--slowmo)
|
||||
SLOWMO="${2:-500}"
|
||||
shift 2
|
||||
;;
|
||||
--inspector)
|
||||
INSPECTOR=true
|
||||
shift
|
||||
;;
|
||||
--project=*)
|
||||
PROJECT="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--project)
|
||||
PROJECT="${2:-chromium}"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
show_help
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_warning "Unknown argument: $1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Show help message
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Usage: run.sh [OPTIONS]
|
||||
|
||||
Run Playwright E2E tests in debug mode for troubleshooting.
|
||||
|
||||
Options:
|
||||
--file=FILE Specific test file to run (relative to tests/)
|
||||
--grep=PATTERN Filter tests by title pattern (regex)
|
||||
--slowmo=MS Delay between actions in milliseconds (default: 500)
|
||||
--inspector Open Playwright Inspector for step-by-step debugging
|
||||
--project=PROJECT Browser to use: chromium, firefox, webkit (default: chromium)
|
||||
-h, --help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
PLAYWRIGHT_BASE_URL Application URL to test (default: http://localhost:8080)
|
||||
PWDEBUG Set to '1' for Inspector mode
|
||||
DEBUG Verbose logging (e.g., 'pw:api')
|
||||
|
||||
Examples:
|
||||
run.sh # Debug all tests in Chromium
|
||||
run.sh --file=login.spec.ts # Debug specific file
|
||||
run.sh --grep="login" # Debug tests matching pattern
|
||||
run.sh --inspector # Open Playwright Inspector
|
||||
run.sh --slowmo=1000 # Slower execution
|
||||
run.sh --file=test.spec.ts --inspector # Combine options
|
||||
EOF
|
||||
}
|
||||
|
||||
# Validate project parameter
|
||||
validate_project() {
|
||||
local valid_projects=("chromium" "firefox" "webkit")
|
||||
local project_lower
|
||||
project_lower=$(echo "${PROJECT}" | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
for valid in "${valid_projects[@]}"; do
|
||||
if [[ "${project_lower}" == "${valid}" ]]; then
|
||||
PROJECT="${project_lower}"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
error_exit "Invalid project '${PROJECT}'. Valid options: chromium, firefox, webkit"
|
||||
}
|
||||
|
||||
# Validate test file if specified
|
||||
validate_test_file() {
|
||||
if [[ -z "${FILE}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
local test_path="${PROJECT_ROOT}/tests/${FILE}"
|
||||
|
||||
# Handle if user provided full path
|
||||
if [[ "${FILE}" == tests/* ]]; then
|
||||
test_path="${PROJECT_ROOT}/${FILE}"
|
||||
FILE="${FILE#tests/}"
|
||||
fi
|
||||
|
||||
if [[ ! -f "${test_path}" ]]; then
|
||||
log_error "Test file not found: ${test_path}"
|
||||
log_info "Available test files:"
|
||||
ls -1 "${PROJECT_ROOT}/tests/"*.spec.ts 2>/dev/null | xargs -n1 basename || true
|
||||
error_exit "Invalid test file"
|
||||
fi
|
||||
}
|
||||
|
||||
# Build Playwright command arguments
|
||||
build_playwright_args() {
|
||||
local args=()
|
||||
|
||||
# Always run headed in debug mode
|
||||
args+=("--headed")
|
||||
|
||||
# Add project
|
||||
args+=("--project=${PROJECT}")
|
||||
|
||||
# Add grep filter if specified
|
||||
if [[ -n "${GREP}" ]]; then
|
||||
args+=("--grep=${GREP}")
|
||||
fi
|
||||
|
||||
# Always collect traces in debug mode
|
||||
args+=("--trace=on")
|
||||
|
||||
# Run single worker for clarity
|
||||
args+=("--workers=1")
|
||||
|
||||
# No retries in debug mode
|
||||
args+=("--retries=0")
|
||||
|
||||
echo "${args[*]}"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
parse_arguments "$@"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_node_environment "18.0" || error_exit "Node.js 18+ is required"
|
||||
check_command_exists "npx" "npx is required (part of Node.js installation)"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "tests" "playwright.config.js" "package.json" || error_exit "Invalid project structure"
|
||||
|
||||
# Validate parameters
|
||||
validate_project
|
||||
validate_test_file
|
||||
|
||||
# Set environment variables
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
set_default_env "PLAYWRIGHT_BASE_URL" "http://localhost:8080"
|
||||
|
||||
# Enable Inspector if requested
|
||||
if [[ "${INSPECTOR}" == "true" ]]; then
|
||||
export PWDEBUG=1
|
||||
log_info "Playwright Inspector enabled"
|
||||
fi
|
||||
|
||||
# Log configuration
|
||||
log_step "CONFIG" "Debug configuration"
|
||||
log_info "Project: ${PROJECT}"
|
||||
log_info "Test file: ${FILE:-<all tests>}"
|
||||
log_info "Grep filter: ${GREP:-<none>}"
|
||||
log_info "Slow motion: ${SLOWMO}ms"
|
||||
log_info "Inspector: ${INSPECTOR}"
|
||||
log_info "Base URL: ${PLAYWRIGHT_BASE_URL}"
|
||||
|
||||
# Build command arguments
|
||||
local playwright_args
|
||||
playwright_args=$(build_playwright_args)
|
||||
|
||||
# Determine test path
|
||||
local test_target=""
|
||||
if [[ -n "${FILE}" ]]; then
|
||||
test_target="tests/${FILE}"
|
||||
fi
|
||||
|
||||
# Build full command
|
||||
local full_cmd="npx playwright test ${playwright_args}"
|
||||
if [[ -n "${test_target}" ]]; then
|
||||
full_cmd="${full_cmd} ${test_target}"
|
||||
fi
|
||||
|
||||
# Add slowMo via environment (Playwright config reads this)
|
||||
export PLAYWRIGHT_SLOWMO="${SLOWMO}"
|
||||
|
||||
log_step "EXECUTION" "Running Playwright in debug mode"
|
||||
log_info "Slow motion: ${SLOWMO}ms delay between actions"
|
||||
log_info "Traces will be captured for all tests"
|
||||
echo ""
|
||||
log_command "${full_cmd}"
|
||||
echo ""
|
||||
|
||||
# Create a temporary config that includes slowMo
|
||||
local temp_config="${PROJECT_ROOT}/.playwright-debug-config.js"
|
||||
cat > "${temp_config}" << EOF
|
||||
// Temporary debug config - auto-generated
|
||||
import baseConfig from './playwright.config.js';
|
||||
|
||||
export default {
|
||||
...baseConfig,
|
||||
use: {
|
||||
...baseConfig.use,
|
||||
launchOptions: {
|
||||
slowMo: ${SLOWMO},
|
||||
},
|
||||
trace: 'on',
|
||||
},
|
||||
workers: 1,
|
||||
retries: 0,
|
||||
};
|
||||
EOF
|
||||
|
||||
# Run tests with temporary config
|
||||
local exit_code=0
|
||||
# shellcheck disable=SC2086
|
||||
if npx playwright test --config="${temp_config}" --headed --project="${PROJECT}" ${GREP:+--grep="${GREP}"} ${test_target}; then
|
||||
log_success "Debug tests completed successfully"
|
||||
else
|
||||
exit_code=$?
|
||||
log_warning "Debug tests completed with failures (exit code: ${exit_code})"
|
||||
fi
|
||||
|
||||
# Clean up temporary config
|
||||
rm -f "${temp_config}"
|
||||
|
||||
# Output helpful information
|
||||
log_step "ARTIFACTS" "Test artifacts"
|
||||
log_info "HTML Report: ${PROJECT_ROOT}/playwright-report/index.html"
|
||||
log_info "Test Results: ${PROJECT_ROOT}/test-results/"
|
||||
|
||||
# Show trace info if tests ran
|
||||
if [[ -d "${PROJECT_ROOT}/test-results" ]] && find "${PROJECT_ROOT}/test-results" -name "trace.zip" -type f 2>/dev/null | head -1 | grep -q .; then
|
||||
log_info ""
|
||||
log_info "View traces with:"
|
||||
log_info " npx playwright show-trace test-results/<test-name>/trace.zip"
|
||||
fi
|
||||
|
||||
exit "${exit_code}"
|
||||
}
|
||||
|
||||
# Run main with all arguments
|
||||
main "$@"
|
||||
@@ -0,0 +1,383 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-e2e-playwright-debug"
|
||||
version: "1.0.0"
|
||||
description: "Run Playwright E2E tests in headed/debug mode for troubleshooting with slowMo and trace collection"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "e2e"
|
||||
- "playwright"
|
||||
- "debug"
|
||||
- "troubleshooting"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "node"
|
||||
version: ">=18.0"
|
||||
optional: false
|
||||
- name: "npx"
|
||||
version: ">=1.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "PLAYWRIGHT_BASE_URL"
|
||||
description: "Base URL of the Charon application under test"
|
||||
default: "http://localhost:8080"
|
||||
required: false
|
||||
- name: "PWDEBUG"
|
||||
description: "Enable Playwright Inspector (set to '1' for step-by-step debugging)"
|
||||
default: ""
|
||||
required: false
|
||||
- name: "DEBUG"
|
||||
description: "Enable verbose Playwright logging (e.g., 'pw:api')"
|
||||
default: ""
|
||||
required: false
|
||||
parameters:
|
||||
- name: "file"
|
||||
type: "string"
|
||||
description: "Specific test file to run (relative to tests/ directory)"
|
||||
default: ""
|
||||
required: false
|
||||
- name: "grep"
|
||||
type: "string"
|
||||
description: "Filter tests by title pattern (regex)"
|
||||
default: ""
|
||||
required: false
|
||||
- name: "slowmo"
|
||||
type: "number"
|
||||
description: "Slow down operations by specified milliseconds"
|
||||
default: "500"
|
||||
required: false
|
||||
- name: "inspector"
|
||||
type: "boolean"
|
||||
description: "Open Playwright Inspector for step-by-step debugging"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "project"
|
||||
type: "string"
|
||||
description: "Browser project to run (chromium, firefox, webkit)"
|
||||
default: "chromium"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "playwright-report"
|
||||
type: "directory"
|
||||
description: "HTML test report directory"
|
||||
path: "playwright-report/"
|
||||
- name: "test-results"
|
||||
type: "directory"
|
||||
description: "Test artifacts, screenshots, and traces"
|
||||
path: "test-results/"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "e2e-debug"
|
||||
execution_time: "variable"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: false
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test E2E Playwright Debug
|
||||
|
||||
## Overview
|
||||
|
||||
Runs Playwright E2E tests in headed/debug mode for troubleshooting. This skill provides enhanced debugging capabilities including:
|
||||
|
||||
- **Headed Mode**: Visible browser window to watch test execution
|
||||
- **Slow Motion**: Configurable delay between actions for observation
|
||||
- **Playwright Inspector**: Step-by-step debugging with breakpoints
|
||||
- **Trace Collection**: Always captures traces for post-mortem analysis
|
||||
- **Single Test Focus**: Run individual tests or test files
|
||||
|
||||
**Use this skill when:**
|
||||
- Debugging failing E2E tests
|
||||
- Understanding test flow and interactions
|
||||
- Developing new E2E tests
|
||||
- Investigating flaky tests
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.0 or higher installed and in PATH
|
||||
- Playwright browsers installed (`npx playwright install chromium`)
|
||||
- Charon application running at localhost:8080 (use `docker-rebuild-e2e` skill)
|
||||
- Display available (X11 or Wayland on Linux, native on macOS)
|
||||
- Test files in `tests/` directory
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Debug Mode
|
||||
|
||||
Run all tests in headed mode with slow motion:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug
|
||||
```
|
||||
|
||||
### Debug Specific Test File
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --file=login.spec.ts
|
||||
```
|
||||
|
||||
### Debug Test by Name Pattern
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --grep="should login with valid credentials"
|
||||
```
|
||||
|
||||
### With Playwright Inspector
|
||||
|
||||
Open the Playwright Inspector for step-by-step debugging:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --inspector
|
||||
```
|
||||
|
||||
### Custom Slow Motion
|
||||
|
||||
Adjust the delay between actions (in milliseconds):
|
||||
|
||||
```bash
|
||||
# Slower for detailed observation
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --slowmo=1000
|
||||
|
||||
# Faster but still visible
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --slowmo=200
|
||||
```
|
||||
|
||||
### Different Browser
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --project=firefox
|
||||
```
|
||||
|
||||
### Combined Options
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug \
|
||||
--file=dashboard.spec.ts \
|
||||
--grep="navigation" \
|
||||
--slowmo=750 \
|
||||
--project=chromium
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| file | string | No | "" | Specific test file to run |
|
||||
| grep | string | No | "" | Filter tests by title pattern |
|
||||
| slowmo | number | No | 500 | Delay between actions (ms) |
|
||||
| inspector | boolean | No | false | Open Playwright Inspector |
|
||||
| project | string | No | chromium | Browser to use |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| PLAYWRIGHT_BASE_URL | No | http://localhost:8080 | Application URL |
|
||||
| PWDEBUG | No | "" | Set to "1" for Inspector mode |
|
||||
| DEBUG | No | "" | Verbose logging (e.g., "pw:api") |
|
||||
|
||||
## Debugging Techniques
|
||||
|
||||
### Using Playwright Inspector
|
||||
|
||||
The Inspector provides:
|
||||
- **Step-through Execution**: Execute one action at a time
|
||||
- **Locator Playground**: Test and refine selectors
|
||||
- **Call Log**: View all Playwright API calls
|
||||
- **Console**: Access browser console
|
||||
|
||||
```bash
|
||||
# Enable Inspector
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --inspector
|
||||
```
|
||||
|
||||
In the Inspector:
|
||||
1. Use **Resume** to continue to next action
|
||||
2. Use **Step** to execute one action
|
||||
3. Use the **Locator** tab to test selectors
|
||||
4. Check **Console** for JavaScript errors
|
||||
|
||||
### Adding Breakpoints in Tests
|
||||
|
||||
Add `await page.pause()` in your test code:
|
||||
|
||||
```typescript
|
||||
test('debug this test', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.pause(); // Opens Inspector here
|
||||
await page.click('button');
|
||||
});
|
||||
```
|
||||
|
||||
### Verbose Logging
|
||||
|
||||
Enable detailed Playwright API logging:
|
||||
|
||||
```bash
|
||||
DEBUG=pw:api .github/skills/scripts/skill-runner.sh test-e2e-playwright-debug
|
||||
```
|
||||
|
||||
### Screenshot on Failure
|
||||
|
||||
Tests automatically capture screenshots on failure. Find them in:
|
||||
```
|
||||
test-results/<test-name>/
|
||||
├── test-failed-1.png
|
||||
├── trace.zip
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Analyzing Traces
|
||||
|
||||
Traces are always captured in debug mode. View them with:
|
||||
|
||||
```bash
|
||||
# Open trace viewer for a specific test
|
||||
npx playwright show-trace test-results/<test-name>/trace.zip
|
||||
|
||||
# Or view in browser
|
||||
npx playwright show-trace --port 9322
|
||||
```
|
||||
|
||||
Traces include:
|
||||
- DOM snapshots at each step
|
||||
- Network requests/responses
|
||||
- Console logs
|
||||
- Screenshots
|
||||
- Action timeline
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Debug Login Flow
|
||||
|
||||
```bash
|
||||
# Rebuild environment with clean state
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean
|
||||
|
||||
# Debug login tests
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug \
|
||||
--file=login.spec.ts \
|
||||
--slowmo=800
|
||||
```
|
||||
|
||||
### Example 2: Investigate Flaky Test
|
||||
|
||||
```bash
|
||||
# Run with Inspector to step through
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug \
|
||||
--grep="flaky test name" \
|
||||
--inspector
|
||||
|
||||
# After identifying the issue, view the trace
|
||||
npx playwright show-trace test-results/*/trace.zip
|
||||
```
|
||||
|
||||
### Example 3: Develop New Test
|
||||
|
||||
```bash
|
||||
# Run in headed mode while developing
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug \
|
||||
--file=new-feature.spec.ts \
|
||||
--slowmo=500
|
||||
```
|
||||
|
||||
### Example 4: Cross-Browser Debug
|
||||
|
||||
```bash
|
||||
# Debug in Firefox
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug \
|
||||
--project=firefox \
|
||||
--grep="cross-browser issue"
|
||||
```
|
||||
|
||||
## Test File Locations
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `tests/` | All E2E test files |
|
||||
| `tests/auth.setup.ts` | Authentication setup |
|
||||
| `tests/login.spec.ts` | Login flow tests |
|
||||
| `tests/dashboard.spec.ts` | Dashboard tests |
|
||||
| `tests/dns-records.spec.ts` | DNS management tests |
|
||||
| `playwright/.auth/` | Stored auth state |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Browser Window Opens
|
||||
|
||||
**Linux**: Ensure X11/Wayland display is available
|
||||
```bash
|
||||
echo $DISPLAY # Should show :0 or similar
|
||||
```
|
||||
|
||||
**Remote/SSH**: Use X11 forwarding or VNC
|
||||
```bash
|
||||
ssh -X user@host
|
||||
```
|
||||
|
||||
**WSL2**: Install and configure WSLg or X server
|
||||
|
||||
### Test Times Out
|
||||
|
||||
Increase timeout for debugging:
|
||||
```bash
|
||||
# In your test file
|
||||
test.setTimeout(120000); // 2 minutes
|
||||
```
|
||||
|
||||
### Inspector Doesn't Open
|
||||
|
||||
Ensure PWDEBUG is set:
|
||||
```bash
|
||||
PWDEBUG=1 npx playwright test --headed
|
||||
```
|
||||
|
||||
### Cannot Find Test File
|
||||
|
||||
Check the file exists:
|
||||
```bash
|
||||
ls -la tests/*.spec.ts
|
||||
```
|
||||
|
||||
Use relative path from tests/ directory:
|
||||
```bash
|
||||
--file=login.spec.ts # Not tests/login.spec.ts
|
||||
```
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| "Target closed" | Application crashed - check container logs |
|
||||
| "Element not found" | Use Inspector to verify selector |
|
||||
| "Timeout exceeded" | Increase timeout or check if element is hidden |
|
||||
| "Net::ERR_CONNECTION_REFUSED" | Ensure Docker container is running |
|
||||
| Flaky test | Add explicit waits or use Inspector to find race condition |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [test-e2e-playwright](./test-e2e-playwright.SKILL.md) - Run tests normally
|
||||
- [docker-rebuild-e2e](./docker-rebuild-e2e.SKILL.md) - Rebuild E2E environment
|
||||
- [test-e2e-playwright-coverage](./test-e2e-playwright-coverage.SKILL.md) - Run with coverage
|
||||
|
||||
## Notes
|
||||
|
||||
- **Not CI/CD Safe**: Headed mode requires a display
|
||||
- **Resource Usage**: Browser windows consume significant memory
|
||||
- **Slow Motion**: Default 500ms delay; adjust based on needs
|
||||
- **Traces**: Always captured for post-mortem analysis
|
||||
- **Single Worker**: Runs one test at a time for clarity
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-21
|
||||
**Maintained by**: Charon Project Team
|
||||
**Test Directory**: `tests/`
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test E2E Playwright - Execution Script
|
||||
#
|
||||
# Runs Playwright end-to-end tests with browser selection,
|
||||
# headed mode, and test filtering support.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Helper scripts are in .github/skills/scripts/ (one level up from skill-scripts dir)
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script (skills/skill-name-scripts/run.sh -> project root)
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Default parameter values
|
||||
PROJECT="chromium"
|
||||
HEADED=false
|
||||
GREP=""
|
||||
|
||||
# Parse command-line arguments
|
||||
parse_arguments() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--project=*)
|
||||
PROJECT="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--project)
|
||||
PROJECT="${2:-chromium}"
|
||||
shift 2
|
||||
;;
|
||||
--headed)
|
||||
HEADED=true
|
||||
shift
|
||||
;;
|
||||
--grep=*)
|
||||
GREP="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--grep)
|
||||
GREP="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
show_help
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_warning "Unknown argument: $1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Show help message
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Usage: run.sh [OPTIONS]
|
||||
|
||||
Run Playwright E2E tests against the Charon application.
|
||||
|
||||
Options:
|
||||
--project=PROJECT Browser project to run (chromium, firefox, webkit, all)
|
||||
Default: chromium
|
||||
--headed Run tests in headed mode (visible browser)
|
||||
--grep=PATTERN Filter tests by title pattern (regex)
|
||||
-h, --help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
PLAYWRIGHT_BASE_URL Application URL to test (default: http://localhost:8080)
|
||||
PLAYWRIGHT_HTML_OPEN HTML report behavior (default: never)
|
||||
CI Set to 'true' for CI environment
|
||||
|
||||
Examples:
|
||||
run.sh # Run all tests in Chromium (headless)
|
||||
run.sh --project=firefox # Run in Firefox
|
||||
run.sh --headed # Run with visible browser
|
||||
run.sh --grep="login" # Run only login tests
|
||||
run.sh --project=all --grep="smoke" # All browsers, smoke tests only
|
||||
EOF
|
||||
}
|
||||
|
||||
# Validate project parameter
|
||||
validate_project() {
|
||||
local valid_projects=("chromium" "firefox" "webkit" "all")
|
||||
local project_lower
|
||||
project_lower=$(echo "${PROJECT}" | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
for valid in "${valid_projects[@]}"; do
|
||||
if [[ "${project_lower}" == "${valid}" ]]; then
|
||||
PROJECT="${project_lower}"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
error_exit "Invalid project '${PROJECT}'. Valid options: chromium, firefox, webkit, all"
|
||||
}
|
||||
|
||||
# Build Playwright command arguments
|
||||
build_playwright_args() {
|
||||
local args=()
|
||||
|
||||
# Add project selection
|
||||
if [[ "${PROJECT}" != "all" ]]; then
|
||||
args+=("--project=${PROJECT}")
|
||||
fi
|
||||
|
||||
# Add headed mode if requested
|
||||
if [[ "${HEADED}" == "true" ]]; then
|
||||
args+=("--headed")
|
||||
fi
|
||||
|
||||
# Add grep filter if specified
|
||||
if [[ -n "${GREP}" ]]; then
|
||||
args+=("--grep=${GREP}")
|
||||
fi
|
||||
|
||||
echo "${args[*]}"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
parse_arguments "$@"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_node_environment "18.0" || error_exit "Node.js 18+ is required"
|
||||
check_command_exists "npx" "npx is required (part of Node.js installation)"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "tests" "playwright.config.js" "package.json" || error_exit "Invalid project structure"
|
||||
|
||||
# Validate project parameter
|
||||
validate_project
|
||||
|
||||
# Set environment variables for non-interactive execution
|
||||
export PLAYWRIGHT_HTML_OPEN="${PLAYWRIGHT_HTML_OPEN:-never}"
|
||||
set_default_env "PLAYWRIGHT_BASE_URL" "http://localhost:8080"
|
||||
|
||||
# Log configuration
|
||||
log_step "CONFIG" "Test configuration"
|
||||
log_info "Project: ${PROJECT}"
|
||||
log_info "Headed mode: ${HEADED}"
|
||||
log_info "Grep filter: ${GREP:-<none>}"
|
||||
log_info "Base URL: ${PLAYWRIGHT_BASE_URL}"
|
||||
log_info "HTML report auto-open: ${PLAYWRIGHT_HTML_OPEN}"
|
||||
|
||||
# Build command arguments
|
||||
local playwright_args
|
||||
playwright_args=$(build_playwright_args)
|
||||
|
||||
# Execute Playwright tests
|
||||
log_step "EXECUTION" "Running Playwright E2E tests"
|
||||
log_command "npx playwright test ${playwright_args}"
|
||||
|
||||
# Run tests with proper error handling
|
||||
local exit_code=0
|
||||
# shellcheck disable=SC2086
|
||||
if npx playwright test ${playwright_args}; then
|
||||
log_success "All E2E tests passed"
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "E2E tests failed (exit code: ${exit_code})"
|
||||
fi
|
||||
|
||||
# Output report location
|
||||
log_step "REPORT" "Test report available"
|
||||
log_info "HTML Report: ${PROJECT_ROOT}/playwright-report/index.html"
|
||||
log_info "To view in browser: npx playwright show-report --port 9323"
|
||||
log_info "VS Code Simple Browser URL: http://127.0.0.1:9323"
|
||||
|
||||
exit "${exit_code}"
|
||||
}
|
||||
|
||||
# Run main with all arguments
|
||||
main "$@"
|
||||
@@ -0,0 +1,350 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-e2e-playwright"
|
||||
version: "1.0.0"
|
||||
description: "Run Playwright E2E tests against the Charon application with browser selection and filtering"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "e2e"
|
||||
- "playwright"
|
||||
- "integration"
|
||||
- "browser"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "node"
|
||||
version: ">=18.0"
|
||||
optional: false
|
||||
- name: "npx"
|
||||
version: ">=1.0"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "PLAYWRIGHT_BASE_URL"
|
||||
description: "Base URL of the Charon application under test"
|
||||
default: "http://localhost:8080"
|
||||
required: false
|
||||
- name: "PLAYWRIGHT_HTML_OPEN"
|
||||
description: "Controls HTML report auto-open behavior (set to 'never' for CI/non-interactive)"
|
||||
default: "never"
|
||||
required: false
|
||||
- name: "CI"
|
||||
description: "Set to 'true' when running in CI environment"
|
||||
default: ""
|
||||
required: false
|
||||
parameters:
|
||||
- name: "project"
|
||||
type: "string"
|
||||
description: "Browser project to run (chromium, firefox, webkit, all)"
|
||||
default: "chromium"
|
||||
required: false
|
||||
- name: "headed"
|
||||
type: "boolean"
|
||||
description: "Run tests in headed mode (visible browser)"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "grep"
|
||||
type: "string"
|
||||
description: "Filter tests by title pattern (regex)"
|
||||
default: ""
|
||||
required: false
|
||||
outputs:
|
||||
- name: "playwright-report"
|
||||
type: "directory"
|
||||
description: "HTML test report directory"
|
||||
path: "playwright-report/"
|
||||
- name: "test-results"
|
||||
type: "directory"
|
||||
description: "Test artifacts and traces"
|
||||
path: "test-results/"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "e2e"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: true
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test E2E Playwright
|
||||
|
||||
## Overview
|
||||
|
||||
Executes Playwright end-to-end tests against the Charon application. This skill supports browser selection, headed mode for debugging, and test filtering by name pattern.
|
||||
|
||||
The skill runs non-interactively by default (HTML report does not auto-open), making it suitable for CI/CD pipelines and automated testing scenarios.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.0 or higher installed and in PATH
|
||||
- Playwright browsers installed (`npx playwright install`)
|
||||
- Charon application running (default: `http://localhost:8080`)
|
||||
- Test files in `tests/` directory
|
||||
|
||||
### Quick Start: Ensure E2E Environment is Ready
|
||||
|
||||
Before running tests, ensure the Docker E2E environment is running:
|
||||
|
||||
```bash
|
||||
# Start/rebuild E2E Docker container (recommended before testing)
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e
|
||||
|
||||
# Or for a complete clean rebuild:
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean --no-cache
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run E2E tests with default settings (Chromium, headless):
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright
|
||||
```
|
||||
|
||||
### Browser Selection
|
||||
|
||||
Run tests in a specific browser:
|
||||
|
||||
```bash
|
||||
# Chromium (default)
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=chromium
|
||||
|
||||
# Firefox
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=firefox
|
||||
|
||||
# WebKit (Safari)
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=webkit
|
||||
|
||||
# All browsers
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=all
|
||||
```
|
||||
|
||||
### Headed Mode (Debugging)
|
||||
|
||||
Run tests with a visible browser window:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --headed
|
||||
```
|
||||
|
||||
### Filter Tests
|
||||
|
||||
Run only tests matching a pattern:
|
||||
|
||||
```bash
|
||||
# Run tests with "login" in the title
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --grep="login"
|
||||
|
||||
# Run tests with "DNS" in the title
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --grep="DNS"
|
||||
```
|
||||
|
||||
### Combined Options
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=firefox --headed --grep="dashboard"
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run E2E Tests
|
||||
run: .github/skills/scripts/skill-runner.sh test-e2e-playwright
|
||||
env:
|
||||
PLAYWRIGHT_BASE_URL: http://localhost:8080
|
||||
CI: true
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| project | string | No | chromium | Browser project: chromium, firefox, webkit, all |
|
||||
| headed | boolean | No | false | Run with visible browser window |
|
||||
| grep | string | No | "" | Filter tests by title pattern (regex) |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| PLAYWRIGHT_BASE_URL | No | http://localhost:8080 | Application URL to test against |
|
||||
| PLAYWRIGHT_HTML_OPEN | No | never | HTML report auto-open behavior |
|
||||
| CI | No | "" | Set to "true" for CI environment behavior |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: One or more tests failed
|
||||
- **Non-zero**: Configuration or execution error
|
||||
|
||||
### Output Directories
|
||||
- **playwright-report/**: HTML report with test results and traces
|
||||
- **test-results/**: Test artifacts, screenshots, and trace files
|
||||
|
||||
## Viewing the Report
|
||||
|
||||
After test execution, view the HTML report using VS Code Simple Browser:
|
||||
|
||||
### Method 1: Start Report Server
|
||||
|
||||
```bash
|
||||
npx playwright show-report --port 9323
|
||||
```
|
||||
|
||||
Then open in VS Code Simple Browser: `http://127.0.0.1:9323`
|
||||
|
||||
### Method 2: VS Code Task
|
||||
|
||||
Use the VS Code task "Test: E2E Playwright - View Report" to start the report server as a background task, then open `http://127.0.0.1:9323` in Simple Browser.
|
||||
|
||||
### Method 3: Direct File Access
|
||||
|
||||
Open `playwright-report/index.html` directly in a browser.
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Quick Smoke Test
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --grep="smoke"
|
||||
```
|
||||
|
||||
### Example 2: Debug Failing Test
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --headed --grep="failing-test-name"
|
||||
```
|
||||
|
||||
### Example 3: Cross-Browser Validation
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright --project=all
|
||||
```
|
||||
|
||||
## Test Structure
|
||||
|
||||
Tests are located in the `tests/` directory and follow Playwright conventions:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── auth.setup.ts # Authentication setup (runs first)
|
||||
├── dashboard.spec.ts # Dashboard tests
|
||||
├── dns-records.spec.ts # DNS management tests
|
||||
├── login.spec.ts # Login flow tests
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Target page, context or browser has been closed
|
||||
**Solution**: Ensure the application is running at the configured base URL. Rebuild if needed:
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e
|
||||
```
|
||||
|
||||
#### Error: page.goto: net::ERR_CONNECTION_REFUSED
|
||||
**Solution**: Start the Charon application before running tests:
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e
|
||||
```
|
||||
|
||||
#### Error: browserType.launch: Executable doesn't exist
|
||||
**Solution**: Run `npx playwright install` to install browser binaries
|
||||
|
||||
#### Error: Timeout waiting for selector
|
||||
**Solution**: The application may be slow or in an unexpected state. Try:
|
||||
```bash
|
||||
# Rebuild with clean state
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean
|
||||
|
||||
# Or debug the test to see what's happening
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --grep="failing test"
|
||||
```
|
||||
|
||||
#### Error: Authentication state is stale
|
||||
**Solution**: Remove stored auth and let setup recreate it:
|
||||
```bash
|
||||
rm -rf playwright/.auth/user.json
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright
|
||||
```
|
||||
|
||||
## Troubleshooting Workflow
|
||||
|
||||
When E2E tests fail, follow this workflow:
|
||||
|
||||
1. **Check container health**:
|
||||
```bash
|
||||
docker ps --filter "name=charon-playwright"
|
||||
docker logs charon-playwright --tail 50
|
||||
```
|
||||
|
||||
2. **Verify the application is accessible**:
|
||||
```bash
|
||||
curl -sf http://localhost:8080/api/v1/health
|
||||
```
|
||||
|
||||
3. **Rebuild with clean state if needed**:
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh docker-rebuild-e2e --clean
|
||||
```
|
||||
|
||||
4. **Debug specific failing test**:
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-e2e-playwright-debug --grep="test name"
|
||||
```
|
||||
|
||||
5. **View the HTML report for details**:
|
||||
```bash
|
||||
npx playwright show-report --port 9323
|
||||
```
|
||||
|
||||
## Key File Locations
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `tests/` | All E2E test files |
|
||||
| `tests/auth.setup.ts` | Authentication setup fixture |
|
||||
| `playwright.config.js` | Playwright configuration |
|
||||
| `playwright/.auth/user.json` | Stored authentication state |
|
||||
| `playwright-report/` | HTML test reports |
|
||||
| `test-results/` | Test artifacts and traces |
|
||||
| `.docker/compose/docker-compose.playwright.yml` | E2E Docker compose config |
|
||||
| `Dockerfile` | Application Docker image |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [docker-rebuild-e2e](./docker-rebuild-e2e.SKILL.md) - Rebuild Docker image and restart E2E container
|
||||
- [test-e2e-playwright-debug](./test-e2e-playwright-debug.SKILL.md) - Debug E2E tests in headed mode
|
||||
- [test-e2e-playwright-coverage](./test-e2e-playwright-coverage.SKILL.md) - Run E2E tests with coverage
|
||||
- [test-frontend-unit](./test-frontend-unit.SKILL.md) - Frontend unit tests with Vitest
|
||||
- [docker-start-dev](./docker-start-dev.SKILL.md) - Start development environment
|
||||
- [integration-test-all](./integration-test-all.SKILL.md) - Run all integration tests
|
||||
|
||||
## Notes
|
||||
|
||||
- **Authentication**: Tests use stored auth state from `playwright/.auth/user.json`
|
||||
- **Parallelization**: Tests run in parallel locally, sequential in CI
|
||||
- **Retries**: CI automatically retries failed tests twice
|
||||
- **Traces**: Traces are collected on first retry for debugging
|
||||
- **Report**: HTML report is generated at `playwright-report/index.html`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-15
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `tests/` directory
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test Frontend Coverage - Execution Script
|
||||
#
|
||||
# This script wraps the legacy frontend-test-coverage.sh script while providing
|
||||
# the Agent Skills interface and logging.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Helper scripts are in .github/skills/scripts/
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script (skills/skill-name-scripts/run.sh -> project root)
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_node_environment "18.0" || error_exit "Node.js 18.0+ is required"
|
||||
validate_python_environment "3.8" || error_exit "Python 3.8+ is required"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "frontend" "scripts/frontend-test-coverage.sh" || error_exit "Invalid project structure"
|
||||
|
||||
# Set default environment variables
|
||||
set_default_env "CHARON_MIN_COVERAGE" "85"
|
||||
|
||||
# Execute the legacy script
|
||||
log_step "EXECUTION" "Running frontend tests with coverage"
|
||||
log_info "Minimum coverage: ${CHARON_MIN_COVERAGE}%"
|
||||
|
||||
LEGACY_SCRIPT="${PROJECT_ROOT}/scripts/frontend-test-coverage.sh"
|
||||
check_file_exists "${LEGACY_SCRIPT}"
|
||||
|
||||
# Execute with proper error handling
|
||||
if "${LEGACY_SCRIPT}" "$@"; then
|
||||
log_success "Frontend coverage tests passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "Frontend coverage tests failed (exit code: ${exit_code})"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,197 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-frontend-coverage"
|
||||
version: "1.0.0"
|
||||
description: "Run frontend tests with coverage analysis and threshold validation (minimum 85%)"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "coverage"
|
||||
- "frontend"
|
||||
- "vitest"
|
||||
- "validation"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "node"
|
||||
version: ">=18.0"
|
||||
optional: false
|
||||
- name: "npm"
|
||||
version: ">=9.0"
|
||||
optional: false
|
||||
- name: "python3"
|
||||
version: ">=3.8"
|
||||
optional: false
|
||||
environment_variables:
|
||||
- name: "CHARON_MIN_COVERAGE"
|
||||
description: "Minimum coverage percentage required (overrides default)"
|
||||
default: "85"
|
||||
required: false
|
||||
- name: "CPM_MIN_COVERAGE"
|
||||
description: "Alternative name for minimum coverage threshold (legacy)"
|
||||
default: "85"
|
||||
required: false
|
||||
parameters:
|
||||
- name: "verbose"
|
||||
type: "boolean"
|
||||
description: "Enable verbose test output"
|
||||
default: "false"
|
||||
required: false
|
||||
outputs:
|
||||
- name: "coverage-summary.json"
|
||||
type: "file"
|
||||
description: "JSON coverage summary generated by Vitest"
|
||||
path: "frontend/coverage/coverage-summary.json"
|
||||
- name: "coverage_summary"
|
||||
type: "stdout"
|
||||
description: "Summary of coverage statistics and validation result"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "coverage"
|
||||
execution_time: "medium"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test Frontend Coverage
|
||||
|
||||
## Overview
|
||||
|
||||
Executes the frontend test suite using Vitest with coverage enabled, generates a JSON coverage summary, and validates that the total statements coverage meets or exceeds the configured threshold (default: 85%).
|
||||
|
||||
This skill is designed for continuous integration and pre-commit hooks to ensure code quality standards are maintained.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.0 or higher installed and in PATH
|
||||
- npm 9.0 or higher installed and in PATH
|
||||
- Python 3.8 or higher installed and in PATH
|
||||
- Frontend dependencies installed (`cd frontend && npm install`)
|
||||
- Write permissions in `frontend/coverage/` directory
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run with default settings (85% minimum coverage):
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
```
|
||||
|
||||
### Custom Coverage Threshold
|
||||
|
||||
Set a custom minimum coverage percentage:
|
||||
|
||||
```bash
|
||||
export CHARON_MIN_COVERAGE=90
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run Frontend Tests with Coverage
|
||||
run: .github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
env:
|
||||
CHARON_MIN_COVERAGE: 85
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| verbose | boolean | No | false | Enable verbose test output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| CHARON_MIN_COVERAGE | No | 85 | Minimum coverage percentage required for success |
|
||||
| CPM_MIN_COVERAGE | No | 85 | Legacy name for minimum coverage (fallback) |
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed and coverage meets threshold
|
||||
|
||||
### Error Exit Codes
|
||||
- **1**: Coverage below threshold or coverage file generation failed
|
||||
- **Non-zero**: Tests failed or other error occurred
|
||||
|
||||
### Output Files
|
||||
- **frontend/coverage/coverage-summary.json**: Vitest coverage summary (JSON format)
|
||||
- **frontend/coverage/index.html**: HTML coverage report (viewable in browser)
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
Computed frontend coverage: 87.5% (minimum required 85%)
|
||||
Frontend coverage requirement met
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
```
|
||||
|
||||
### Example 2: Higher Coverage Threshold
|
||||
|
||||
```bash
|
||||
export CHARON_MIN_COVERAGE=90
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
```
|
||||
|
||||
### Example 3: View HTML Coverage Report
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-coverage
|
||||
open frontend/coverage/index.html # macOS
|
||||
xdg-open frontend/coverage/index.html # Linux
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Coverage summary file not found
|
||||
**Solution**: Check that Vitest is configured with `--coverage` and `--reporter=json-summary`
|
||||
|
||||
#### Error: Frontend coverage X% is below required Y%
|
||||
**Solution**: Add tests for uncovered components or adjust threshold
|
||||
|
||||
#### Error: npm ci failed
|
||||
**Solution**: Clear node_modules and package-lock.json, then reinstall dependencies
|
||||
|
||||
## Related Skills
|
||||
|
||||
- test-frontend-unit - Fast unit tests without coverage
|
||||
- test-backend-coverage - Backend Go coverage tests
|
||||
- utility-cache-clear-go - Clear build caches
|
||||
|
||||
## Notes
|
||||
|
||||
- **Vitest Configuration**: Uses istanbul coverage provider for JSON summary reports
|
||||
- **Coverage Directory**: Coverage artifacts are written to `frontend/coverage/`
|
||||
- **Python Dependency**: Uses Python for decimal-precision coverage comparison
|
||||
- **Idempotency**: Safe to run multiple times; cleans up old coverage files
|
||||
- **CI Mode**: Runs `npm ci` in CI environments to ensure clean installs
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: `scripts/frontend-test-coverage.sh`
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test Frontend Unit - Execution Script
|
||||
#
|
||||
# This script runs frontend unit tests without coverage analysis,
|
||||
# providing fast test execution for development workflows.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Source helper scripts
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Helper scripts are in .github/skills/scripts/
|
||||
SKILLS_SCRIPTS_DIR="$(cd "${SCRIPT_DIR}/../scripts" && pwd)"
|
||||
|
||||
# shellcheck source=../scripts/_logging_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_logging_helpers.sh"
|
||||
# shellcheck source=../scripts/_error_handling_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_error_handling_helpers.sh"
|
||||
# shellcheck source=../scripts/_environment_helpers.sh
|
||||
source "${SKILLS_SCRIPTS_DIR}/_environment_helpers.sh"
|
||||
|
||||
# Project root is 3 levels up from this script (skills/skill-name-scripts/run.sh -> project root)
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
|
||||
# Validate environment
|
||||
log_step "ENVIRONMENT" "Validating prerequisites"
|
||||
validate_node_environment "18.0" || error_exit "Node.js 18.0+ is required"
|
||||
|
||||
# Validate project structure
|
||||
log_step "VALIDATION" "Checking project structure"
|
||||
cd "${PROJECT_ROOT}"
|
||||
validate_project_structure "frontend" || error_exit "Invalid project structure"
|
||||
|
||||
# Change to frontend directory
|
||||
cd "${PROJECT_ROOT}/frontend"
|
||||
|
||||
# Execute tests
|
||||
log_step "EXECUTION" "Running frontend unit tests"
|
||||
|
||||
# Run npm test with all passed arguments
|
||||
if npm run test -- "$@"; then
|
||||
log_success "Frontend unit tests passed"
|
||||
exit 0
|
||||
else
|
||||
exit_code=$?
|
||||
log_error "Frontend unit tests failed (exit code: ${exit_code})"
|
||||
exit "${exit_code}"
|
||||
fi
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
# agentskills.io specification v1.0
|
||||
name: "test-frontend-unit"
|
||||
version: "1.0.0"
|
||||
description: "Run frontend unit tests without coverage analysis (fast execution)"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "testing"
|
||||
- "unit-tests"
|
||||
- "frontend"
|
||||
- "vitest"
|
||||
- "fast"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "node"
|
||||
version: ">=18.0"
|
||||
optional: false
|
||||
- name: "npm"
|
||||
version: ">=9.0"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters:
|
||||
- name: "watch"
|
||||
type: "boolean"
|
||||
description: "Run tests in watch mode"
|
||||
default: "false"
|
||||
required: false
|
||||
- name: "filter"
|
||||
type: "string"
|
||||
description: "Filter tests by name pattern"
|
||||
default: ""
|
||||
required: false
|
||||
outputs:
|
||||
- name: "test_results"
|
||||
type: "stdout"
|
||||
description: "Vitest output showing pass/fail status"
|
||||
metadata:
|
||||
category: "test"
|
||||
subcategory: "unit"
|
||||
execution_time: "short"
|
||||
risk_level: "low"
|
||||
ci_cd_safe: true
|
||||
requires_network: false
|
||||
idempotent: true
|
||||
---
|
||||
|
||||
# Test Frontend Unit
|
||||
|
||||
## Overview
|
||||
|
||||
Executes the frontend unit test suite using Vitest without coverage analysis. This skill provides fast test execution for quick feedback during development, making it ideal for pre-commit checks and rapid iteration.
|
||||
|
||||
Unlike test-frontend-coverage, this skill does not generate coverage reports or enforce coverage thresholds, focusing purely on test pass/fail status.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.0 or higher installed and in PATH
|
||||
- npm 9.0 or higher installed and in PATH
|
||||
- Frontend dependencies installed (`cd frontend && npm install`)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run all frontend unit tests:
|
||||
|
||||
```bash
|
||||
cd /path/to/charon
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit
|
||||
```
|
||||
|
||||
### Watch Mode
|
||||
|
||||
Run tests in watch mode for continuous testing:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- --watch
|
||||
```
|
||||
|
||||
### Filter Tests
|
||||
|
||||
Run tests matching a specific pattern:
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- --grep "Button"
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For use in GitHub Actions or other CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
- name: Run Frontend Unit Tests
|
||||
run: .github/skills/scripts/skill-runner.sh test-frontend-unit
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| watch | boolean | No | false | Run tests in watch mode |
|
||||
| filter | string | No | "" | Filter tests by name pattern |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
No environment variables are required for this skill.
|
||||
|
||||
## Outputs
|
||||
|
||||
### Success Exit Code
|
||||
- **0**: All tests passed
|
||||
|
||||
### Error Exit Codes
|
||||
- **Non-zero**: One or more tests failed
|
||||
|
||||
### Console Output
|
||||
Example output:
|
||||
```
|
||||
✓ src/components/Button.test.tsx (3)
|
||||
✓ src/utils/helpers.test.ts (5)
|
||||
✓ src/hooks/useAuth.test.ts (4)
|
||||
|
||||
Test Files 3 passed (3)
|
||||
Tests 12 passed (12)
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Execution
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit
|
||||
```
|
||||
|
||||
### Example 2: Watch Mode for TDD
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- --watch
|
||||
```
|
||||
|
||||
### Example 3: Test Specific File
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- Button.test.tsx
|
||||
```
|
||||
|
||||
### Example 4: UI Mode (Interactive)
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- --ui
|
||||
```
|
||||
|
||||
### Example 5: Reporter Configuration
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh test-frontend-unit -- --reporter=verbose
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### Error: Cannot find module
|
||||
**Solution**: Run `npm install` to ensure all dependencies are installed
|
||||
|
||||
#### Error: Test timeout
|
||||
**Solution**: Increase timeout in vitest.config.ts or fix hanging async tests
|
||||
|
||||
#### Error: Unexpected token
|
||||
**Solution**: Check for syntax errors in test files
|
||||
|
||||
## Related Skills
|
||||
|
||||
- test-frontend-coverage - Run tests with coverage analysis (slower)
|
||||
- test-backend-unit - Backend Go unit tests
|
||||
- build-check-go - Verify builds without running tests
|
||||
|
||||
## Notes
|
||||
|
||||
- **Execution Time**: Fast execution (~3-5 seconds typical)
|
||||
- **No Coverage**: Does not generate coverage reports
|
||||
- **Vitest Features**: Full access to Vitest CLI options via arguments
|
||||
- **Idempotency**: Safe to run multiple times
|
||||
- **Caching**: Benefits from Vitest's smart caching
|
||||
- **Suitable For**: Pre-commit hooks, quick feedback, TDD workflows
|
||||
- **Watch Mode**: Available for interactive development
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project Team
|
||||
**Source**: Inline task command
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# Utility: Bump Beta Version - Execution Script
|
||||
# ==============================================================================
|
||||
# This script increments the beta version number across all project files.
|
||||
# It wraps the original bump_beta.sh script.
|
||||
#
|
||||
# Usage: ./run.sh
|
||||
# Exit codes: 0 = success, non-zero = failure
|
||||
# ==============================================================================
|
||||
|
||||
# Determine the repository root directory
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
|
||||
# Change to repository root
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# Execute the bump beta script
|
||||
exec scripts/bump_beta.sh "$@"
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
name: "utility-bump-beta"
|
||||
version: "1.0.0"
|
||||
description: "Increments beta version number across all project files for pre-release versioning"
|
||||
author: "Charon Project"
|
||||
license: "MIT"
|
||||
tags:
|
||||
- "utility"
|
||||
- "versioning"
|
||||
- "release"
|
||||
- "automation"
|
||||
compatibility:
|
||||
os:
|
||||
- "linux"
|
||||
- "darwin"
|
||||
shells:
|
||||
- "bash"
|
||||
requirements:
|
||||
- name: "git"
|
||||
version: ">=2.0"
|
||||
optional: false
|
||||
- name: "sed"
|
||||
version: ">=4.0"
|
||||
optional: false
|
||||
environment_variables: []
|
||||
parameters: []
|
||||
outputs:
|
||||
- name: "new_version"
|
||||
type: "string"
|
||||
description: "The new beta version number"
|
||||
path: ".version"
|
||||
metadata:
|
||||
category: "utility"
|
||||
subcategory: "versioning"
|
||||
execution_time: "short"
|
||||
risk_level: "medium"
|
||||
ci_cd_safe: false
|
||||
requires_network: false
|
||||
idempotent: false
|
||||
---
|
||||
|
||||
# Utility: Bump Beta Version
|
||||
|
||||
## Overview
|
||||
|
||||
Automates beta version bumping across all project files. This skill intelligently increments version numbers following semantic versioning conventions for beta releases, updating multiple files in sync to maintain consistency.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Git repository initialized
|
||||
- Write access to project files
|
||||
- Clean working directory (recommended)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
.github/skills/utility-bump-beta-scripts/run.sh
|
||||
```
|
||||
|
||||
### Via Skill Runner
|
||||
|
||||
```bash
|
||||
.github/skills/scripts/skill-runner.sh utility-bump-beta
|
||||
```
|
||||
|
||||
### Via VS Code Task
|
||||
|
||||
Use the task: **Utility: Bump Beta Version**
|
||||
|
||||
## Parameters
|
||||
|
||||
This skill accepts no parameters. Version bumping logic is automatic based on current version format.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
This skill requires no environment variables.
|
||||
|
||||
## Outputs
|
||||
|
||||
- **Success Exit Code**: 0
|
||||
- **Error Exit Codes**: Non-zero on failure
|
||||
- **Modified Files**:
|
||||
- `.version`
|
||||
- `backend/internal/version/version.go`
|
||||
- `frontend/package.json`
|
||||
- `backend/package.json` (if exists)
|
||||
- **Git Tag**: `v{NEW_VERSION}` (if user confirms)
|
||||
|
||||
### Output Example
|
||||
|
||||
```
|
||||
Starting Beta Version Bump...
|
||||
Current Version: 0.3.0-beta.2
|
||||
New Version: 0.3.0-beta.3
|
||||
Updated .version
|
||||
Updated backend/internal/version/version.go
|
||||
Updated frontend/package.json
|
||||
Updated backend/package.json
|
||||
Do you want to commit and tag this version? (y/n) y
|
||||
Committed and tagged v0.3.0-beta.3
|
||||
Remember to push: git push origin feature/beta-release --tags
|
||||
```
|
||||
|
||||
## Version Bumping Logic
|
||||
|
||||
### Current Version is Beta (x.y.z-beta.N)
|
||||
|
||||
Increments the beta number:
|
||||
- `0.3.0-beta.2` → `0.3.0-beta.3`
|
||||
- `1.0.0-beta.5` → `1.0.0-beta.6`
|
||||
|
||||
### Current Version is Plain Semver (x.y.z)
|
||||
|
||||
Bumps minor version and starts beta.1:
|
||||
- `0.3.0` → `0.4.0-beta.1`
|
||||
- `1.2.0` → `1.3.0-beta.1`
|
||||
|
||||
### Current Version is Alpha or Unrecognized
|
||||
|
||||
Defaults to safe fallback:
|
||||
- `0.3.0-alpha` → `0.3.0-beta.1`
|
||||
- `invalid-version` → `0.3.0-beta.1`
|
||||
|
||||
## Files Updated
|
||||
|
||||
1. **`.version`**: Project root version file
|
||||
2. **`backend/internal/version/version.go`**: Go version constant
|
||||
3. **`frontend/package.json`**: Frontend package version
|
||||
4. **`backend/package.json`**: Backend package version (if exists)
|
||||
|
||||
All files are updated with consistent version strings using `sed` regex replacement.
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Bump Beta Before Release
|
||||
|
||||
```bash
|
||||
# Bump version for next beta iteration
|
||||
.github/skills/utility-bump-beta-scripts/run.sh
|
||||
|
||||
# Confirm when prompted to commit and tag
|
||||
# Then push to remote
|
||||
git push origin feature/beta-release --tags
|
||||
```
|
||||
|
||||
### Example 2: Bump Without Committing
|
||||
|
||||
```bash
|
||||
# Make version changes but skip git operations
|
||||
.github/skills/utility-bump-beta-scripts/run.sh
|
||||
# Answer 'n' when prompted about committing
|
||||
```
|
||||
|
||||
## Interactive Confirmation
|
||||
|
||||
After updating files, the script prompts:
|
||||
|
||||
```
|
||||
Do you want to commit and tag this version? (y/n)
|
||||
```
|
||||
|
||||
- **Yes (y)**: Creates git commit and tag automatically
|
||||
- **No (n)**: Leaves changes staged for manual review
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Validates `.version` file exists and is readable
|
||||
- Uses safe defaults for unrecognized version formats
|
||||
- Does not modify VERSION.md guide content (manual update recommended)
|
||||
- Skips `backend/package.json` if file doesn't exist
|
||||
|
||||
## Post-Execution Steps
|
||||
|
||||
After running this skill:
|
||||
|
||||
1. **Review Changes**: `git diff`
|
||||
2. **Run Tests**: Ensure version change doesn't break builds
|
||||
3. **Push Tags**: `git push origin <branch> --tags`
|
||||
4. **Update CHANGELOG.md**: Manually document changes for this version
|
||||
5. **Verify CI/CD**: Check that automated builds use new version
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [utility-version-check](./utility-version-check.SKILL.md) - Validate version matches tags
|
||||
- [build-check-go](../build-check-go.SKILL.md) - Verify build after version bump
|
||||
|
||||
## Notes
|
||||
|
||||
- **Not Idempotent**: Running multiple times increments version each time
|
||||
- **Risk Level: Medium**: Modifies multiple critical files
|
||||
- **Git State**: Recommended to have clean working directory before running
|
||||
- **Manual Review**: Always review version changes before pushing
|
||||
- **VERSION.md**: Update manually as it contains documentation, not just version
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-20
|
||||
**Maintained by**: Charon Project
|
||||
**Source**: `scripts/bump_beta.sh`
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ==============================================================================
|
||||
# Utility: Clear Go Cache - Execution Script
|
||||
# ==============================================================================
|
||||
# This script clears Go build, test, and module caches, plus gopls cache.
|
||||
# It wraps the original clear-go-cache.sh script.
|
||||
#
|
||||
# Usage: ./run.sh
|
||||
# Exit codes: 0 = success, 1 = failure
|
||||
# ==============================================================================
|
||||
|
||||
# Determine the repository root directory
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
|
||||
# Change to repository root
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# Execute the cache clear script
|
||||
exec scripts/clear-go-cache.sh "$@"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user