Charon/docs/plans/docs_to_issues_workflow.md

docs/issues to GitHub Issues Workflow - Implementation Plan

Version: 1.0 Date: 2025-12-13 Status: 📋 PLANNING

---

Executive Summary

This document provides a comprehensive plan for a GitHub Actions workflow that automatically converts markdown files in docs/issues/ into GitHub Issues, applies labels, and adds them to the project board.

---

1. Research Findings

1.1 Current docs/issues File Analysis

Analyzed 8 existing files in docs/issues/:

File	Structure	Has Frontmatter	Labels Present	Title Pattern
ACL-testing-tasks.md	Free-form + metadata section	❌	Inline suggestion	H1 + metadata block
Additional_Security.md	Markdown lists	❌	❌	H3 title
bulk-acl-subissues.md	Multi-issue spec	❌	Inline per section	Inline titles
bulk-acl-testing.md	Full issue template	❌	Inline section	H1 title
hectate.md	Feature spec	❌	❌	H1 title
orthrus.md	Feature spec	❌	❌	H1 title
plex-remote-access-helper.md	Issue template	❌	Inline section	## Issue Title
rotating-loading-animations.md	Enhancement spec	❌	**Issue Type** line	H1 title

Key Finding: No files use YAML frontmatter. Most have ad-hoc metadata embedded in markdown body.

1.2 Existing Workflow Patterns

From .github/workflows/:

Workflow	Pattern	Relevance
auto-label-issues.yml	actions/github-script for issue manipulation	Label creation, issue API
propagate-changes.yml	Complex github-script with file analysis	File path detection
auto-add-to-project.yml	actions/add-to-project action	Project board integration
create-labels.yml	Label creation/update logic	Label management

1.3 Project Board Configuration

• Project URL stored in secrets.PROJECT_URL
• PAT token: secrets.ADD_TO_PROJECT_PAT
• Uses actions/add-to-project@v1.0.2 for automatic addition

---

2. Markdown File Format Specification

2.1 Required Frontmatter Format

All files in docs/issues/ should use YAML frontmatter:

---
title: "Issue Title Here"
labels:
  - testing
  - feature
  - backend
assignees:
  - username1
  - username2
milestone: "v0.2.0-beta.2"
priority: high          # critical, high, medium, low
type: feature           # feature, bug, enhancement, testing, documentation
parent_issue: 42        # Optional: Link as sub-issue
create_sub_issues: true # Optional: Parse ## sections as separate issues
---

# Issue Title (H1 fallback if frontmatter title missing)

Issue body content starts here...

2.2 Frontmatter Fields

Field	Required	Type	Description
title	Yes*	string	Issue title (*or H1 fallback)
labels	No	array	Labels to apply (created if missing)
assignees	No	array	GitHub usernames
milestone	No	string	Milestone name
priority	No	enum	Maps to priority label
type	No	enum	Maps to type label
parent_issue	No	number	Parent issue number for linking
create_sub_issues	No	boolean	Parse H2 sections as sub-issues

2.3 Sub-Issue Detection

When create_sub_issues: true, each ## Section Title becomes a sub-issue:

---
title: "Main Testing Issue"
labels: [testing]
create_sub_issues: true
---

# Main Testing Issue

Overview text (goes in parent issue).

## Sub-Issue #1: Unit Testing

This section becomes a separate issue titled "Unit Testing"
with body content from this section.

## Sub-Issue #2: Integration Testing

This section becomes another separate issue.

---

3. Workflow Implementation

3.1 Workflow File: .github/workflows/docs-to-issues.yml

name: Convert Docs to Issues

on:
  push:
    branches:
      - main
      - development
    paths:
      - 'docs/issues/**/*.md'
      - '!docs/issues/created/**'

  # Allow manual trigger
  workflow_dispatch:
    inputs:
      dry_run:
        description: 'Dry run (no issues created)'
        required: false
        default: 'false'
        type: boolean
      file_path:
        description: 'Specific file to process (optional)'
        required: false
        type: string

permissions:
  contents: write
  issues: write
  pull-requests: write

jobs:
  convert-docs:
    name: Convert Markdown to Issues
    runs-on: ubuntu-latest
    if: github.actor != 'github-actions[bot]'

    steps:
      - name: Checkout repository
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
        with:
          fetch-depth: 2  # Need previous commit for diff

      - name: Set up Node.js
        uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af # v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install gray-matter

      - name: Detect changed files
        id: changes
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
        with:
          script: |
            const fs = require('fs');
            const path = require('path');

            // Manual file specification
            const manualFile = '${{ github.event.inputs.file_path }}';
            if (manualFile) {
              if (fs.existsSync(manualFile)) {
                core.setOutput('files', JSON.stringify([manualFile]));
                return;
              } else {
                core.setFailed(`File not found: ${manualFile}`);
                return;
              }
            }

            // Get changed files from commit
            const { data: commit } = await github.rest.repos.getCommit({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: context.sha
            });

            const changedFiles = (commit.files || [])
              .filter(f => f.filename.startsWith('docs/issues/'))
              .filter(f => !f.filename.startsWith('docs/issues/created/'))
              .filter(f => f.filename.endsWith('.md'))
              .filter(f => f.status !== 'removed')
              .map(f => f.filename);

            console.log('Changed issue files:', changedFiles);
            core.setOutput('files', JSON.stringify(changedFiles));

      - name: Process issue files
        id: process
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
        env:
          DRY_RUN: ${{ github.event.inputs.dry_run || 'false' }}
        with:
          script: |
            const fs = require('fs');
            const path = require('path');
            const matter = require('gray-matter');

            const files = JSON.parse('${{ steps.changes.outputs.files }}');
            const isDryRun = process.env.DRY_RUN === 'true';
            const createdIssues = [];
            const errors = [];

            if (files.length === 0) {
              console.log('No issue files to process');
              core.setOutput('created_count', 0);
              return;
            }

            // Helper: Ensure label exists
            async function ensureLabel(name) {
              try {
                await github.rest.issues.getLabel({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  name: name
                });
              } catch (e) {
                if (e.status === 404) {
                  // Create label with default color
                  const colors = {
                    testing: 'BFD4F2',
                    feature: 'A2EEEF',
                    enhancement: '84B6EB',
                    bug: 'D73A4A',
                    documentation: '0075CA',
                    backend: '1D76DB',
                    frontend: '5EBEFF',
                    security: 'EE0701',
                    ui: '7057FF',
                    caddy: '1F6FEB',
                    'needs-triage': 'FBCA04',
                    acl: 'C5DEF5',
                    regression: 'D93F0B',
                    'manual-testing': 'BFD4F2',
                    'bulk-acl': '006B75',
                    'error-handling': 'D93F0B',
                    'ui-ux': '7057FF',
                    integration: '0E8A16',
                    performance: 'EDEDED',
                    'cross-browser': '5319E7',
                    plus: 'FFD700',
                    beta: '0052CC',
                    alpha: '5319E7',
                    high: 'D93F0B',
                    medium: 'FBCA04',
                    low: '0E8A16',
                    critical: 'B60205'
                  };
                  await github.rest.issues.createLabel({
                    owner: context.repo.owner,
                    repo: context.repo.repo,
                    name: name,
                    color: colors[name.toLowerCase()] || '666666'
                  });
                  console.log(`Created label: ${name}`);
                }
              }
            }

            // Helper: Parse markdown file
            function parseIssueFile(filePath) {
              const content = fs.readFileSync(filePath, 'utf8');
              const { data: frontmatter, content: body } = matter(content);

              // Extract title: frontmatter > first H1 > filename
              let title = frontmatter.title;
              if (!title) {
                const h1Match = body.match(/^#\s+(.+)$/m);
                title = h1Match ? h1Match[1] : path.basename(filePath, '.md');
              }

              // Build labels array
              const labels = [...(frontmatter.labels || [])];
              if (frontmatter.priority) labels.push(frontmatter.priority);
              if (frontmatter.type) labels.push(frontmatter.type);

              return {
                title,
                body: body.trim(),
                labels: [...new Set(labels)],
                assignees: frontmatter.assignees || [],
                milestone: frontmatter.milestone,
                parent_issue: frontmatter.parent_issue,
                create_sub_issues: frontmatter.create_sub_issues || false
              };
            }

            // Helper: Extract sub-issues from H2 sections
            function extractSubIssues(body, parentLabels) {
              const sections = [];
              const regex = /^##\s+(?:Sub-Issue\s*#?\d*:?\s*)?(.+?)$([\s\S]*?)(?=^##\s|\Z)/gm;
              let match;

              while ((match = regex.exec(body)) !== null) {
                const sectionTitle = match[1].trim();
                const sectionBody = match[2].trim();

                // Extract inline labels like **Labels**: `testing`, `acl`
                const inlineLabels = [];
                const labelMatch = sectionBody.match(/\*\*Labels?\*\*:?\s*[`']?([^`'\n]+)[`']?/i);
                if (labelMatch) {
                  labelMatch[1].split(',').forEach(l => {
                    const label = l.replace(/[`']/g, '').trim();
                    if (label) inlineLabels.push(label);
                  });
                }

                sections.push({
                  title: sectionTitle,
                  body: sectionBody,
                  labels: [...parentLabels, ...inlineLabels]
                });
              }

              return sections;
            }

            // Process each file
            for (const filePath of files) {
              console.log(`\nProcessing: ${filePath}`);

              try {
                const parsed = parseIssueFile(filePath);
                console.log(`  Title: ${parsed.title}`);
                console.log(`  Labels: ${parsed.labels.join(', ')}`);

                if (isDryRun) {
                  console.log('  [DRY RUN] Would create issue');
                  createdIssues.push({ file: filePath, title: parsed.title, dryRun: true });
                  continue;
                }

                // Ensure labels exist
                for (const label of parsed.labels) {
                  await ensureLabel(label);
                }

                // Create the main issue
                const issueResponse = await github.rest.issues.create({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  title: parsed.title,
                  body: parsed.body + `\n\n---\n*Auto-created from [${path.basename(filePath)}](https://github.com/${context.repo.owner}/${context.repo.repo}/blob/${context.sha}/${filePath})*`,
                  labels: parsed.labels,
                  assignees: parsed.assignees
                });

                const issueNumber = issueResponse.data.number;
                console.log(`  Created issue #${issueNumber}`);

                // Handle sub-issues
                if (parsed.create_sub_issues) {
                  const subIssues = extractSubIssues(parsed.body, parsed.labels);
                  for (const sub of subIssues) {
                    for (const label of sub.labels) {
                      await ensureLabel(label);
                    }
                    const subResponse = await github.rest.issues.create({
                      owner: context.repo.owner,
                      repo: context.repo.repo,
                      title: `[${parsed.title}] ${sub.title}`,
                      body: sub.body + `\n\n---\n*Sub-issue of #${issueNumber}*`,
                      labels: sub.labels,
                      assignees: parsed.assignees
                    });
                    console.log(`    Created sub-issue #${subResponse.data.number}: ${sub.title}`);
                  }
                }

                // Link to parent issue if specified
                if (parsed.parent_issue) {
                  await github.rest.issues.createComment({
                    owner: context.repo.owner,
                    repo: context.repo.repo,
                    issue_number: parsed.parent_issue,
                    body: `Sub-issue created: #${issueNumber}`
                  });
                }

                createdIssues.push({
                  file: filePath,
                  title: parsed.title,
                  issueNumber
                });

              } catch (error) {
                console.error(`  Error processing ${filePath}: ${error.message}`);
                errors.push({ file: filePath, error: error.message });
              }
            }

            core.setOutput('created_count', createdIssues.length);
            core.setOutput('created_issues', JSON.stringify(createdIssues));
            core.setOutput('errors', JSON.stringify(errors));

            if (errors.length > 0) {
              core.warning(`${errors.length} file(s) had errors`);
            }

      - name: Move processed files
        if: steps.process.outputs.created_count != '0' && github.event.inputs.dry_run != 'true'
        run: |
          mkdir -p docs/issues/created
          CREATED_ISSUES='${{ steps.process.outputs.created_issues }}'
          echo "$CREATED_ISSUES" | jq -r '.[].file' | while read file; do
            if [ -f "$file" ] && [ ! -z "$file" ]; then
              filename=$(basename "$file")
              timestamp=$(date +%Y%m%d)
              mv "$file" "docs/issues/created/${timestamp}-${filename}"
              echo "Moved: $file -> docs/issues/created/${timestamp}-${filename}"
            fi
          done

      - name: Commit moved files
        if: steps.process.outputs.created_count != '0' && github.event.inputs.dry_run != 'true'
        run: |
          git config --local user.email "github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add docs/issues/
          git diff --staged --quiet || git commit -m "chore: move processed issue files to created/ [skip ci]"
          git push

      - name: Add to project board
        if: steps.process.outputs.created_count != '0' && github.event.inputs.dry_run != 'true'
        continue-on-error: true
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
        env:
          PROJECT_URL: ${{ secrets.PROJECT_URL }}
          ADD_TO_PROJECT_PAT: ${{ secrets.ADD_TO_PROJECT_PAT }}
        with:
          github-token: ${{ secrets.ADD_TO_PROJECT_PAT || secrets.GITHUB_TOKEN }}
          script: |
            const projectUrl = process.env.PROJECT_URL;
            if (!projectUrl) {
              console.log('PROJECT_URL not configured, skipping project board');
              return;
            }

            const createdIssues = JSON.parse('${{ steps.process.outputs.created_issues }}');
            console.log(`Would add ${createdIssues.length} issues to project board`);
            // Note: Actual project board integration requires GraphQL API
            // The actions/add-to-project action handles this automatically
            // for issues created via normal flow

      - name: Summary
        if: always()
        run: |
          echo "## Docs to Issues Summary" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY

          CREATED='${{ steps.process.outputs.created_issues }}'
          ERRORS='${{ steps.process.outputs.errors }}'
          DRY_RUN='${{ github.event.inputs.dry_run }}'

          if [ "$DRY_RUN" = "true" ]; then
            echo "🔍 **Dry Run Mode** - No issues were actually created" >> $GITHUB_STEP_SUMMARY
            echo "" >> $GITHUB_STEP_SUMMARY
          fi

          echo "### Created Issues" >> $GITHUB_STEP_SUMMARY
          if [ -n "$CREATED" ] && [ "$CREATED" != "[]" ]; then
            echo "$CREATED" | jq -r '.[] | "- \(.title) (#\(.issueNumber // "dry-run"))"' >> $GITHUB_STEP_SUMMARY
          else
            echo "_No issues created_" >> $GITHUB_STEP_SUMMARY
          fi

          echo "" >> $GITHUB_STEP_SUMMARY
          echo "### Errors" >> $GITHUB_STEP_SUMMARY
          if [ -n "$ERRORS" ] && [ "$ERRORS" != "[]" ]; then
            echo "$ERRORS" | jq -r '.[] | "- ❌ \(.file): \(.error)"' >> $GITHUB_STEP_SUMMARY
          else
            echo "_No errors_" >> $GITHUB_STEP_SUMMARY
          fi

---

4. File Conversion Templates

4.1 Updated ACL-testing-tasks.md (Example)

---
title: "ACL: Test and validate ACL changes (feature/beta-release)"
labels:
  - testing
  - needs-triage
  - acl
  - regression
priority: high
milestone: "v0.2.0-beta.2"
create_sub_issues: false
---

# ACL: Test and validate ACL changes (feature/beta-release)

**Repository:** Wikid82/Charon
**Branch:** feature/beta-release

## Purpose

Create a tracked issue and sub-tasks to validate ACL-related changes...

[rest of content unchanged]

4.2 Updated bulk-acl-subissues.md (Example with Sub-Issues)

---
title: "Bulk ACL Testing - Sub-Issues"
labels:
  - testing
  - manual-testing
  - bulk-acl
priority: medium
milestone: "v0.2.0-beta.2"
create_sub_issues: true
---

# Bulk ACL Testing - Sub-Issues

## Basic Functionality Testing

**Labels**: `testing`, `manual-testing`, `bulk-acl`

Test the core functionality of the bulk ACL feature...

## ACL Removal Testing

**Labels**: `testing`, `manual-testing`, `bulk-acl`

Test the ability to remove access lists...

[each ## section becomes a sub-issue]

4.3 Template for New Issue Files

Create docs/issues/_TEMPLATE.md:

---
# REQUIRED: Issue title
title: "Your Issue Title"

# OPTIONAL: Labels to apply (will be created if missing)
labels:
  - feature       # feature, bug, enhancement, testing, documentation
  - backend       # backend, frontend, ui, security, caddy, database

# OPTIONAL: Priority (creates matching label)
priority: medium  # critical, high, medium, low

# OPTIONAL: Milestone name
milestone: "v0.2.0-beta.2"

# OPTIONAL: GitHub usernames to assign
assignees: []

# OPTIONAL: Parent issue number for linking
# parent_issue: 42

# OPTIONAL: Parse ## sections as separate sub-issues
# create_sub_issues: true
---

# Issue Title

## Description

Clear description of the issue or feature request.

## Tasks

- [ ] Task 1
- [ ] Task 2
- [ ] Task 3

## Acceptance Criteria

- [ ] Criterion 1
- [ ] Criterion 2

## Related Issues

- #XX - Related issue description

---

*Issue specification created: YYYY-MM-DD*

---

5. Files to Update

5.1 Existing docs/issues Files Needing Frontmatter

File	Action Needed
ACL-testing-tasks.md	Add frontmatter with extracted metadata
Additional_Security.md	Add frontmatter, convert to issue format
bulk-acl-subissues.md	Add frontmatter with create_sub_issues: true
bulk-acl-testing.md	Add frontmatter with extracted metadata
hectate.md	Add frontmatter (feature spec)
orthrus.md	Add frontmatter (feature spec)
plex-remote-access-helper.md	Add frontmatter, already has metadata section
rotating-loading-animations.md	Add frontmatter, extract inline metadata

5.2 New Files to Create

File	Purpose
.github/workflows/docs-to-issues.yml	Main workflow
docs/issues/_TEMPLATE.md	Template for new issues
docs/issues/created/.gitkeep	Directory for processed files
docs/issues/README.md	Documentation for contributors

---

6. Directory Structure

docs/
├── issues/
│   ├── README.md                    # How to create issue files
│   ├── _TEMPLATE.md                 # Template for new issues
│   ├── created/                     # Processed files moved here
│   │   └── .gitkeep
│   ├── ACL-testing-tasks.md         # Pending (needs frontmatter)
│   ├── Additional_Security.md       # Pending
│   ├── bulk-acl-subissues.md        # Pending
│   └── ...

---

7. Duplicate Prevention Strategy

7.1 File-Based Tracking

• Processed files move to docs/issues/created/ with timestamp prefix
• Workflow excludes created/ directory from processing
• Git history provides audit trail of when files were converted

7.2 Issue Tracking

Each created issue includes footer:

---
*Auto-created from [filename.md](link-to-source-commit)*

7.3 Manual Override

• Use workflow_dispatch with specific file path to reprocess
• Files in created/ can be moved back for reprocessing

---

8. Project Board Integration

8.1 Automatic Addition

The workflow leverages the existing auto-add-to-project.yml which triggers on issues: [opened].

When the workflow creates an issue, the auto-add workflow automatically adds it to the project board (if PROJECT_URL secret is configured).

8.2 Manual Configuration

If not using auto-add-to-project.yml, configure in the docs-to-issues workflow:

- name: Add to project
  uses: actions/add-to-project@244f685bbc3b7adfa8466e08b698b5577571133e # v1.0.2
  with:
    project-url: ${{ secrets.PROJECT_URL }}
    github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}

---

9. Testing Strategy

9.1 Dry Run Testing

# Manually trigger with dry_run=true
gh workflow run docs-to-issues.yml -f dry_run=true

# Test specific file
gh workflow run docs-to-issues.yml -f dry_run=true -f file_path=docs/issues/ACL-testing-tasks.md

9.2 Local Testing

# Parse frontmatter locally
node -e "
const matter = require('gray-matter');
const fs = require('fs');
const result = matter(fs.readFileSync('docs/issues/ACL-testing-tasks.md', 'utf8'));
console.log(JSON.stringify(result.data, null, 2));
"

9.3 Validation Checklist

• Frontmatter parses correctly for all files
• Labels are created when missing
• Issue titles are correct
• Issue bodies include full content
• Sub-issues are created correctly (if create_sub_issues: true)
• Files move to created/ after processing
• Auto-add-to-project triggers for new issues
• Commit message includes [skip ci] to prevent loops

---

10. Implementation Phases

Phase 1: Setup (15 min)

1. Create .github/workflows/docs-to-issues.yml
2. Create docs/issues/created/.gitkeep
3. Create docs/issues/_TEMPLATE.md
4. Create docs/issues/README.md

Phase 2: File Migration (30 min)

1. Add frontmatter to existing files (in order of priority)
2. Test with dry_run mode
3. Create one test issue to verify

Phase 3: Validation (15 min)

1. Verify issue creation
2. Verify label creation
3. Verify project board integration
4. Verify file move to created/

---

11. Risk Assessment

Risk	Impact	Mitigation
Duplicate issues	Low	File move + exclude pattern
Label spam	Low	Predefined color map
Rate limiting	Medium	Process files sequentially
Malformed frontmatter	Medium	Try-catch with error logging
Project board auth failure	Low	continue-on-error: true

---

12. Definition of Done

• Workflow file created and committed
• All existing docs/issues files have valid frontmatter
• Dry run succeeds with no errors
• At least one test issue created successfully
• File moved to created/ after processing
• Labels created automatically
• Project board integration verified (if configured)
• Documentation in docs/issues/README.md

---

Appendix A: Label Color Reference

const labelColors = {
  // Priority
  critical: 'B60205',
  high: 'D93F0B',
  medium: 'FBCA04',
  low: '0E8A16',

  // Milestone
  alpha: '5319E7',
  beta: '0052CC',
  'post-beta': '006B75',

  // Type
  feature: 'A2EEEF',
  bug: 'D73A4A',
  enhancement: '84B6EB',
  documentation: '0075CA',
  testing: 'BFD4F2',

  // Component
  backend: '1D76DB',
  frontend: '5EBEFF',
  ui: '7057FF',
  security: 'EE0701',
  caddy: '1F6FEB',
  database: '006B75',

  // Custom
  'needs-triage': 'FBCA04',
  acl: 'C5DEF5',
  'bulk-acl': '006B75',
  'manual-testing': 'BFD4F2',
  regression: 'D93F0B',
  plus: 'FFD700'
};

---

Appendix B: Frontmatter Conversion Examples

B.1 hectate.md (Feature Spec)

---
title: "Hecate: Tunnel & Pathway Manager"
labels:
  - feature
  - backend
  - architecture
priority: medium
milestone: "post-beta"
type: feature
---

B.2 orthrus.md (Feature Spec)

---
title: "Orthrus: Remote Socket Proxy Agent"
labels:
  - feature
  - backend
  - architecture
priority: medium
milestone: "post-beta"
type: feature
---

B.3 plex-remote-access-helper.md

---
title: "Plex Remote Access Helper & CGNAT Solver"
labels:
  - beta
  - feature
  - plus
  - ui
  - caddy
priority: medium
milestone: "beta"
type: feature
parent_issue: 44
---

B.4 rotating-loading-animations.md

---
title: "Enhancement: Rotating Thematic Loading Animations"
labels:
  - enhancement
  - frontend
  - ui
priority: low
type: enhancement
---

B.5 Additional_Security.md

---
title: "Additional Security Threats to Consider"
labels:
  - security
  - documentation
  - architecture
priority: medium
type: documentation
---