managarten/cicd/PRIVATE_REPO_ACCESS_SOLUTION.md

Private Repository Access Solution for CI/CD

Date: 2025-11-27 Status: READY TO IMPLEMENT Priority: BLOCKER

---

Executive Summary

The CI/CD pipeline fails when building Docker images because it needs access to the private repository github.com/Memo-2023/mana-core-nestjs-package.git. This document analyzes the problem and provides a complete implementation guide for the recommended solution.

---

Problem Analysis

Current Situation

Affected Services:

• apps/maerchenzauber/apps/backend (Storyteller backend)
• apps/manadeck/apps/backend (ManaDeck backend)

What's Happening:

1. Both Dockerfiles clone the private repo mana-core-nestjs-package during build
2. The Dockerfiles already have secret mounting logic: RUN --mount=type=secret,id=github_token
3. GitHub Actions workflow does NOT pass the secret to Docker build
4. Build fails with: git clone failed: exit code 128

Evidence from Dockerfiles:

Both Dockerfiles already include this logic (lines 15-30):

RUN --mount=type=secret,id=github_token \
    if [ -f /run/secrets/github_token ]; then \
        export GITHUB_TOKEN=$(cat /run/secrets/github_token) && \
        echo "Using GitHub token for private repo access" && \
        git clone https://${GITHUB_TOKEN}@github.com/Memo-2023/mana-core-nestjs-package.git /tmp/mana-core; \
    else \
        echo "No GitHub token provided, attempting public clone" && \
        git clone https://github.com/Memo-2023/mana-core-nestjs-package.git /tmp/mana-core; \
    fi

The Gap: The Dockerfiles are ready, but the CI workflow doesn't provide the github_token secret.

---

Solution Options Comparison

Option	Pros	Cons	Cost	Recommendation
A: GitHub Token in CI	✅ Quick implementation ✅ Dockerfiles already configured ✅ Fine-grained access control ✅ No architectural changes	⚠️ Requires PAT management ⚠️ Token rotation needed	FREE	⭐ RECOMMENDED
B: Make Repo Public	✅ Simplest solution ✅ No auth needed	❌ Exposes proprietary code ❌ Security risk ❌ Not acceptable for most orgs	FREE	❌ Not recommended
C: Publish npm Package	✅ Professional approach ✅ Version management ✅ Private npm registry	❌ Complex setup ❌ Ongoing maintenance ❌ Registry costs	$7-29/month	🔮 Future enhancement

---

Recommended Solution: Option A - GitHub Token in CI

Why This Solution?

1. Already 90% implemented - Dockerfiles have the secret mounting logic
2. One-line fix - Just need to pass the secret in CI workflow
3. Battle-tested - Same pattern used in manadeck and maerchenzauber docs
4. Secure - GitHub PAT with fine-grained permissions
5. No code changes - Works with existing Dockerfile architecture

Implementation Complexity

• Estimated Time: 10-15 minutes
• Required Actions: 2 steps
• Risk Level: LOW (proven pattern)
• Testing Required: Build verification only

---

Detailed Implementation Guide

Prerequisites

1. GitHub repository admin access (to create secrets)
2. Ability to create GitHub Personal Access Token (PAT)
3. Access to GitHub Actions workflow files

Step 1: Create GitHub Personal Access Token

Why: GitHub Actions' default GITHUB_TOKEN doesn't have permission to access other private repos.

How:

1. Go to GitHub Settings → Developer Settings → Personal Access Tokens → Fine-grained tokens

2. Click "Generate new token"

3. Configure:

   • Token name: CI-Private-Repo-Access
   • Expiration: 1 year (set calendar reminder to rotate)
   • Repository access: Select "Only select repositories"
     • Choose: Memo-2023/mana-core-nestjs-package
   • Permissions:
     • Repository permissions → Contents → Read (required)
     • Repository permissions → Metadata → Read (auto-selected)

4. Click "Generate token"

5. CRITICAL: Copy the token immediately (can't view again)

Token Format: github_pat_11AAAAAA... (starts with github_pat_)

Step 2: Add Token as GitHub Secret

1. Navigate to https://github.com/Memo-2023/manacore-monorepo/settings/secrets/actions
2. Click "New repository secret"
3. Configure:
   • Name: PRIVATE_REPO_TOKEN
   • Value: Paste the PAT from Step 1
4. Click "Add secret"

Step 3: Update GitHub Actions Workflow

File: .github/workflows/ci-main.yml

Current Code (lines 126-140):

- name: Build and push
  if: steps.check-dockerfile.outputs.exists == 'true'
  uses: docker/build-push-action@v5
  with:
    context: .
    file: ${{ matrix.service.path }}/Dockerfile
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    labels: ${{ steps.meta.outputs.labels }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    build-args: |
      NODE_ENV=production
      PORT=${{ matrix.service.port }}

Updated Code (add 2 lines):

- name: Build and push
  if: steps.check-dockerfile.outputs.exists == 'true'
  uses: docker/build-push-action@v5
  with:
    context: .
    file: ${{ matrix.service.path }}/Dockerfile
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    labels: ${{ steps.meta.outputs.labels }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    build-args: |
      NODE_ENV=production
      PORT=${{ matrix.service.port }}
    secrets: |
      github_token=${{ secrets.PRIVATE_REPO_TOKEN }}

What Changed:

• Added secrets: section with github_token=${{ secrets.PRIVATE_REPO_TOKEN }}
• This passes the token to Docker build as a secret mount
• Dockerfiles already have logic to use this secret

Step 4: Verification Steps

Local Testing (Optional but recommended):

# 1. Export token (use your PAT)
export GITHUB_TOKEN="github_pat_YOUR_TOKEN_HERE"

# 2. Test maerchenzauber backend
cd /Users/wuesteon/dev/mana_universe/manacore-monorepo
docker build \
  --secret id=github_token,env=GITHUB_TOKEN \
  -t test-maerchenzauber \
  -f apps/maerchenzauber/apps/backend/Dockerfile \
  .

# 3. Test manadeck backend
docker build \
  --secret id=github_token,env=GITHUB_TOKEN \
  -t test-manadeck \
  -f apps/manadeck/apps/backend/Dockerfile \
  .

# 4. Verify builds succeeded
docker images | grep test-

CI Testing:

1. Commit the workflow change
2. Push to a test branch
3. Check GitHub Actions logs for:
   • "Using GitHub token for private repo access" (success)
   • OR "No GitHub token provided" (failure - secret not passed)
4. Verify build completes without git clone errors

Success Indicators:

• ✅ Docker build logs show "Using GitHub token for private repo access"
• ✅ Git clone succeeds
• ✅ Tarball creation succeeds: "Mana-core packaged as tarball"
• ✅ Docker image pushed to ghcr.io
• ✅ No "exit code 128" errors

---

Architecture Details

How It Works (Sequence Diagram)

GitHub Actions Workflow
  │
  ├─→ Passes secret to Docker build
  │   secrets: github_token=${{ secrets.PRIVATE_REPO_TOKEN }}
  │
  └─→ Docker Build Process
      │
      ├─→ Mounts secret as file: /run/secrets/github_token
      │
      ├─→ Reads token: export GITHUB_TOKEN=$(cat /run/secrets/github_token)
      │
      ├─→ Clones private repo:
      │   git clone https://${GITHUB_TOKEN}@github.com/Memo-2023/mana-core-nestjs-package.git
      │
      ├─→ Builds and packages:
      │   npm install --force
      │   npm run build
      │   npm pack → mana-core.tgz
      │
      ├─→ Replaces git URL with local tarball in package.json:
      │   "mana-core": "file:../mana-core.tgz"
      │
      └─→ Installs from tarball:
          npm install --legacy-peer-deps
          (No git access needed - fully self-contained)

Security Model

Secret Handling:

• Token stored in GitHub Secrets (encrypted at rest)
• Passed to Docker as build secret (never in logs)
• Docker secret mount (in-memory, not in image layers)
• Token never written to filesystem in final image
• Token only visible during build, not in running containers

Access Control:

• PAT has read-only access to single repo
• Scoped to specific repository
• Can be rotated without code changes
• Can be revoked instantly if compromised

Production Impact

What Gets Built:

• Private package is compiled and bundled into Docker image
• Final image contains mana-core.tgz (built artifact)
• No git dependencies in production
• No authentication needed at runtime

Image Size Impact:

• Minimal (tarball is compressed)
• Same as current local builds

Runtime Performance:

• No impact (package already compiled)
• Faster than git clone at runtime

---

Alternative Options (Future Consideration)

Option B: Make Repository Public

When to Consider:

• If code is not proprietary
• If open-source is part of business model
• If maintaining PAT becomes burdensome

How to Implement:

1. Go to repository settings
2. Change visibility to public
3. Remove secret mounting from Dockerfiles
4. Update CI workflow to remove secrets

Pros: Simplest solution Cons: Not viable for most commercial projects

Option C: Publish as Private npm Package

When to Consider:

• When you have 10+ services using the package
• When version management becomes critical
• When you want professional npm workflow

How to Implement:

Option C1: GitHub Packages (Free)

1. Add .npmrc to mana-core-nestjs-package:

   @memo-2023:registry=https://npm.pkg.github.com
   

2. Update package.json:

   {
   	"name": "@memo-2023/mana-core",
   	"version": "1.0.0",
   	"publishConfig": {
   		"registry": "https://npm.pkg.github.com"
   	}
   }
   

3. Publish:

   npm login --registry=https://npm.pkg.github.com
   npm publish
   

4. Update consuming apps:

   {
   	"dependencies": {
   		"@memo-2023/mana-core": "^1.0.0"
   	}
   }
   

5. Add .npmrc to consuming projects:

   @memo-2023:registry=https://npm.pkg.github.com
   //npm.pkg.github.com/:_authToken=${NPM_TOKEN}
   

Option C2: npm.js Registry (Paid)

• Private packages: $7/month for 1 user, $29/month for teams
• More features: badges, stats, better CDN
• Industry standard for professional packages

Migration Effort: Medium (3-4 hours) Ongoing Maintenance: Low (publish on new versions) Cost: $0 (GitHub) or $7-29/month (npm.js)

---

Troubleshooting Guide

Issue: "No GitHub token provided"

Symptom: Docker build logs show fallback message Cause: Secret not passed to Docker build Fix:

1. Verify secret exists: GitHub repo settings → Secrets → PRIVATE_REPO_TOKEN
2. Check workflow syntax: secrets: | with correct indentation
3. Verify secret name matches exactly

Issue: "git clone failed: exit code 128"

Symptom: Authentication failure during git clone Cause: Invalid or expired PAT Fix:

1. Test PAT manually:

   git clone https://YOUR_PAT@github.com/Memo-2023/mana-core-nestjs-package.git
   

2. If fails: regenerate PAT with correct permissions
3. Update GitHub secret with new PAT

Issue: "Permission denied (publickey)"

Symptom: SSH authentication attempted Cause: Git URL using SSH instead of HTTPS Fix:

• Dockerfiles already use HTTPS URLs
• Verify Dockerfile has https://${GITHUB_TOKEN}@github.com/...

Issue: Build succeeds but package not found

Symptom: npm install fails to find mana-core Cause: Tarball path incorrect or sed replacement failed Fix:

1. Check build logs for "=== Verifying tarball ===" section
2. Verify tarball exists at expected path
3. Check sed replacement worked: grep -n "mana-core" package.json

---

Maintenance Plan

Token Rotation (Every 12 Months)

Why: Security best practice When: Set calendar reminder 1 week before expiration How:

1. Generate new PAT (same permissions)
2. Update GitHub secret: Settings → Secrets → Edit PRIVATE_REPO_TOKEN
3. Trigger test build to verify
4. Delete old PAT

Downtime: None (atomic secret update)

Monitoring

What to Watch:

• GitHub Actions build failures on main branch
• Authentication errors in Docker build logs
• PAT expiration date (GitHub sends email reminders)

Alerts:

• Set GitHub Actions notification to Slack/email
• Monitor build success rate

---

Migration from Current State

What Changes

Modified Files:

• .github/workflows/ci-main.yml (add 2 lines)

New Secrets:

• PRIVATE_REPO_TOKEN (in GitHub repo settings)

No Changes Needed:

• Dockerfiles (already have secret mounting)
• Package.json files
• Application code
• Local development workflow

Rollback Plan

If Implementation Fails:

1. Remove secrets: section from workflow
2. Revert to previous workflow version
3. Builds will fail (same as current state)

Zero Risk: Can't make it worse than current state

---

Success Criteria

Definition of Done

• ✅ GitHub secret PRIVATE_REPO_TOKEN created
• ✅ CI workflow updated with secret passing
• ✅ Test build succeeds on test branch
• ✅ Both maerchenzauber-backend and manadeck-backend images build
• ✅ Docker images pushed to ghcr.io successfully
• ✅ No authentication errors in logs
• ✅ Documentation updated (this file)

Metrics

Before:

• Docker build success rate: 0%
• Manual workarounds: Required
• Developer impact: High (can't merge PRs)

After:

• Docker build success rate: 100%
• Manual workarounds: None
• Developer impact: Zero

---

References

Similar Implementations

• apps/manadeck/apps/backend/SSH_LOCKFILE_SOLUTION.md - Describes the two-layer approach
• apps/maerchenzauber/docs/ci-npm-ssh-fix.md - Documents npm SSH fix pattern
• Docker BuildKit secrets: https://docs.docker.com/build/building/secrets/

Related Documentation

• cicd/IMMEDIATE_FIXES_NEEDED.md - Parent issue tracking document
• cicd/SETUP.md - Overall CI/CD setup guide
• cicd/README.md - CI/CD documentation hub

---

Appendix: Complete Workflow Example

Full ci-main.yml Build Step (After Changes)

- name: Build and push
  if: steps.check-dockerfile.outputs.exists == 'true'
  uses: docker/build-push-action@v5
  with:
    context: .
    file: ${{ matrix.service.path }}/Dockerfile
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    labels: ${{ steps.meta.outputs.labels }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    build-args: |
      NODE_ENV=production
      PORT=${{ matrix.service.port }}
    secrets: |
      github_token=${{ secrets.PRIVATE_REPO_TOKEN }}

Expected Docker Build Log Output

#8 [builder 3/12] RUN --mount=type=secret,id=github_token     if [ -f /run/secrets/github_token ]; then...
#8 0.124 Using GitHub token for private repo access
#8 0.856 Cloning into '/tmp/mana-core'...
#8 12.34 Mana-core packaged as tarball at /app/mana-core.tgz
#8 DONE 12.5s

#9 [builder 4/12] COPY apps/maerchenzauber/apps/backend/package.json ./backend/package.json
#9 DONE 0.1s

#10 [builder 5/12] RUN sed -i 's|"git+https://github.com/...
#10 0.012 === Verifying tarball and package.json ===
#10 0.013 -rw-r--r-- 1 root root 1234567 Nov 27 12:34 mana-core.tgz
#10 0.013 Tarball exists at /app/mana-core.tgz
#10 0.014 Checking package.json replacement:
#10 0.015 26:    "@mana-core/nestjs-integration": "file:../mana-core.tgz",
#10 0.015 === End verification ===
#10 DONE 0.2s

---

Document Version: 1.0 Last Updated: 2025-11-27 Next Review: 2026-11-27 (token rotation)