managarten/CI_CD_README.md

CI/CD Pipeline Implementation

Complete CI/CD pipeline for the manacore-monorepo with automated testing, building, and deployment.

What's Included

GitHub Actions Workflows (6 workflows)

1. PR Validation (.github/workflows/ci-pull-request.yml)

   • Detects changed projects
   • Runs lint, format, type-check
   • Builds affected projects
   • Runs tests with coverage
   • Docker build validation
   • Security scanning

2. Main Branch CI (.github/workflows/ci-main.yml)

   • Full validation on merge to main
   • Builds and pushes Docker images
   • Triggers staging deployment

3. Staging Deployment (.github/workflows/cd-staging.yml)

   • Automated deployment to staging
   • Zero-downtime rolling updates
   • Health checks
   • Database migrations

4. Production Deployment (.github/workflows/cd-production.yml)

   • Manual trigger with approval gates
   • Pre-deployment backups
   • Rolling updates
   • Extended monitoring
   • Smoke tests

5. Test Coverage (.github/workflows/test-coverage.yml)

   • Runs on PRs and schedule
   • Uploads to Codecov
   • Enforces 50% minimum coverage
   • Generates reports

6. Dependency Updates (.github/workflows/dependency-update.yml)

   • Weekly automated checks
   • Security audits
   • Creates issues for vulnerabilities
   • Updates lock files

Docker Infrastructure

• Templates: Ready-to-use Dockerfiles for NestJS, SvelteKit, and Astro
• Multi-stage builds: Optimized for production
• Security: Non-root users, health checks, resource limits
• docker-compose.staging.yml: Full staging environment
• docker-compose.production.yml: Production configuration

Deployment Scripts

Located in scripts/deploy/:

1. build-and-push.sh: Build and push Docker images
2. deploy-hetzner.sh: Deploy to Hetzner/Coolify servers
3. health-check.sh: Verify service health
4. rollback.sh: Emergency rollback procedures
5. migrate-db.sh: Database migration runner

All scripts include error handling, logging, and safety checks.

Testing Infrastructure

• vitest.config.ts: Unit test configuration
• jest.config.js: Multi-project test setup (backend, mobile, shared)
• playwright.config.ts: E2E test configuration
• tests/e2e/: Example E2E tests

Documentation

• docs/DEPLOYMENT.md: Complete deployment guide (20+ pages)
• docs/CI_CD_SETUP.md: Step-by-step setup instructions
• docs/DOCKER_GUIDE.md: Docker best practices and troubleshooting

Configuration Files

• .dockerignore: Optimized Docker build context
• .github/dependabot.yml: Automated dependency updates
• docker/nginx/: Nginx configurations
• docker/templates/: Dockerfile templates

Quick Start

1. Set Up GitHub Secrets

Follow the checklist in docs/CI_CD_SETUP.md#github-secrets:

# Required secrets (22 minimum for staging + production)
- DOCKER_USERNAME
- DOCKER_PASSWORD
- STAGING_HOST
- STAGING_USER
- STAGING_SSH_KEY
- PRODUCTION_HOST
- PRODUCTION_USER
- PRODUCTION_SSH_KEY
# ... and more

2. Configure Deployment Server

# On your server
sudo adduser deploy
sudo usermod -aG docker deploy
curl -fsSL https://get.docker.com | sh

# Add SSH key
mkdir -p ~/.ssh
echo "ssh-ed25519 YOUR_PUBLIC_KEY" >> ~/.ssh/authorized_keys

# Create deployment directory
mkdir -p ~/manacore-staging

3. Test the Pipeline

# Create test PR
git checkout -b test/ci-pipeline
git push origin test/ci-pipeline

# Watch GitHub Actions tab
# All checks should pass ✅

4. Deploy to Staging

# Merge PR to main
# Staging deployment happens automatically
# Check status:
./scripts/deploy/health-check.sh staging

5. Deploy to Production

# Go to GitHub Actions > CD - Production Deployment
# Click "Run workflow"
# Enter:
#   - Service: all
#   - Environment: production
#   - Confirm: deploy
# Wait for approval gate
# Approve deployment
# Monitor progress

Features

Smart Build Detection

Only builds changed projects using Turborepo filters:

# Detects changes in:
- apps/maerchenzauber/**
- apps/chat/**
- packages/**
# Only builds affected projects

Zero-Downtime Deployments

Rolling update strategy:

# Scale up with new version
docker compose up -d --scale service=2 service
sleep 15
# Scale down to single instance
docker compose up -d --scale service=1 service

Comprehensive Health Checks

Every service has health endpoints:

# Automated health checks after deployment
- mana-core-auth: /api/v1/health
- backends: /health or /api/health
- web apps: / (root)

Automated Backups

Production deployments automatically create backups:

# Pre-deployment backup includes:
- PostgreSQL database dump
- Docker compose configuration
- Environment files
- Current image tags

Security Features

• Dependency scanning (Dependabot)
• Security audits (weekly)
• Non-root Docker users
• Secret scanning
• SSH key rotation guidance

Architecture

┌──────────────┐
│  Pull Request │
└──────┬───────┘
       │
       ▼
┌──────────────────────────────────────┐
│  PR Validation Workflow               │
│  - Detect changes                     │
│  - Lint & format check               │
│  - Type check                        │
│  - Build affected projects           │
│  - Run tests                         │
│  - Docker build validation           │
│  - Security scan                     │
└──────┬───────────────────────────────┘
       │
       ▼
┌──────────────┐
│  Merge to Main│
└──────┬───────┘
       │
       ▼
┌──────────────────────────────────────┐
│  Main Branch CI                       │
│  - Full validation                    │
│  - Build all projects                 │
│  - Build Docker images                │
│  - Push to registry                   │
│  - Trigger staging deployment         │
└──────┬───────────────────────────────┘
       │
       ▼
┌──────────────────────────────────────┐
│  Staging Deployment (Automatic)       │
│  - Pull latest images                 │
│  - Run migrations                     │
│  - Rolling update                     │
│  - Health checks                      │
└───────────────────────────────────────┘
       │
       ▼ (Manual)
┌──────────────────────────────────────┐
│  Production Deployment (Manual)       │
│  - Request approval                   │
│  - Create backup                      │
│  - Pull images                        │
│  - Run migrations                     │
│  - Zero-downtime deployment           │
│  - Extended health checks             │
│  - Smoke tests                        │
│  - Monitor for 5 minutes              │
└───────────────────────────────────────┘

Project Structure

manacore-monorepo/
├── .github/
│   ├── workflows/
│   │   ├── ci-pull-request.yml       # PR validation
│   │   ├── ci-main.yml               # Main branch CI
│   │   ├── cd-staging.yml            # Staging deployment
│   │   ├── cd-production.yml         # Production deployment
│   │   ├── test-coverage.yml         # Coverage tracking
│   │   └── dependency-update.yml     # Dependency management
│   └── dependabot.yml                # Dependabot config
├── docker/
│   ├── templates/
│   │   ├── Dockerfile.nestjs         # NestJS template
│   │   ├── Dockerfile.sveltekit      # SvelteKit template
│   │   └── Dockerfile.astro          # Astro template
│   └── nginx/
│       └── astro.conf                # Nginx config for Astro
├── scripts/
│   └── deploy/
│       ├── build-and-push.sh         # Build images
│       ├── deploy-hetzner.sh         # Deploy to server
│       ├── health-check.sh           # Health verification
│       ├── rollback.sh               # Rollback procedure
│       └── migrate-db.sh             # Database migrations
├── docs/
│   ├── DEPLOYMENT.md                 # Deployment guide
│   ├── CI_CD_SETUP.md               # Setup instructions
│   └── DOCKER_GUIDE.md              # Docker guide
├── tests/
│   └── e2e/
│       └── example.spec.ts           # Example E2E test
├── docker-compose.staging.yml        # Staging orchestration
├── docker-compose.production.yml     # Production orchestration
├── vitest.config.ts                  # Vitest config
├── jest.config.js                    # Jest config
├── playwright.config.ts              # Playwright config
└── .dockerignore                     # Docker build exclusions

Services Deployed

The pipeline handles deployment for:

1. mana-core-auth (Port 3001)

   • Central authentication service
   • JWT token management
   • User authentication

2. maerchenzauber-backend (Port 3002)

   • Story generation service
   • Azure OpenAI integration
   • Character management

3. chat-backend (Port 3002)

   • Chat API service
   • AI conversation handling
   • Message persistence

4. manadeck-backend (Port 3003)

   • Card/deck management
   • Collection handling

5. nutriphi-backend (Port 3004)

   • Nutrition tracking service

6. news-api (Port 3005)

   • News aggregation service

Monitoring and Alerts

Built-in Monitoring

• Health check endpoints
• Docker health checks
• Resource usage tracking
• Log aggregation

Recommended External Tools

• Uptime Monitoring: UptimeRobot, Pingdom
• Error Tracking: Sentry
• Performance: New Relic, Datadog
• Logs: Papertrail, Loggly

Rollback Procedures

Automatic Rollback

# Emergency rollback to previous version
./scripts/deploy/rollback.sh production all

What it does:

1. Confirms with user
2. Checks for backup
3. Stops current services
4. Restores previous configuration
5. Restores database
6. Starts previous version
7. Runs health checks

Manual Rollback

# SSH to server
ssh deploy@api.manacore.app
cd ~/manacore-production

# Find backup
ls -lt backups/

# Restore
cp backups/20250127_120000/docker-compose.yml .
docker compose up -d

Cost Optimization

GitHub Actions Minutes

• Free tier: 2,000 minutes/month
• Smart build detection reduces usage
• Workflow caching saves time
• Estimated usage: ~500-800 minutes/month

Docker Registry

• Docker Hub free tier: 1 organization, unlimited public repos
• Estimated storage: ~10-15GB for all images
• Alternative: GitHub Container Registry (free)

Server Resources

Staging:

• 2 vCPU, 4GB RAM: ~$10-15/month
• Hetzner CX21: €5.83/month

Production:

• 4 vCPU, 8GB RAM: ~$25-35/month
• Hetzner CX31: €11.66/month

Security Considerations

Secrets Management

• Never commit secrets to repository
• Use GitHub Secrets for CI/CD
• Rotate secrets every 90 days
• Use different secrets per environment

Image Security

• Regular base image updates
• Dependabot for dependencies
• Security scanning in CI
• Non-root users in containers

Network Security

• Firewall on servers
• SSL/TLS for all connections
• Reverse proxy for services
• Rate limiting

Troubleshooting

Common Issues

1. Deployment fails: Check GitHub Actions logs
2. Health checks fail: Review service logs
3. Build fails: Test build locally
4. SSH issues: Verify keys and permissions

Full troubleshooting guide: docs/DEPLOYMENT.md#troubleshooting

Next Steps

1. ✅ Review this README
2. ✅ Read docs/CI_CD_SETUP.md
3. ✅ Configure GitHub secrets
4. ✅ Set up deployment server
5. ✅ Test PR workflow
6. ✅ Test staging deployment
7. ✅ Test production deployment
8. ✅ Set up monitoring
9. ✅ Configure alerts
10. ✅ Train team on procedures

Support

For issues or questions:

1. Check documentation in docs/
2. Review GitHub Actions logs
3. Check deployment scripts
4. Contact DevOps team

License

Private - Manacore Team Only