managarten/maerchenzauber/docs/ci-npm-ssh-fix.md

Summary: Fixing npm ci SSH Authentication in GitHub Actions

🔴 What We Tried (That Didn't Work)

Attempt 1: Git Config URL Rewriting

What we did:

- name: Configure git for private packages
  run: |
    git config --global url."https://${GH_TOKEN}@github.com/".insteadOf "https://github.com/"
    git config --global url."https://${GH_TOKEN}@github.com/".insteadOf "ssh://git@github.com/"
    git config --global url."https://${GH_TOKEN}@github.com/".insteadOf "git@github.com:"

Why it failed:

• npm ci reads URLs directly from package-lock.json
• npm's internal git client doesn't reliably honor git config url.insteadOf rules
• npm fell back to SSH when it couldn't authenticate via HTTPS

Attempt 2: Invalid .npmrc Configuration

What we did:

# backend/.npmrc
git-ssh-url = https://github.com/

Why it failed:

• git-ssh-url is not a valid npm configuration option
• npm doesn't recognize this setting

Attempt 3: Reordering Steps (Git Config Before Node Setup)

What we did:

• Moved git config step before actions/setup-node
• Added npm cache clearing

Why it failed:

• While this helped with caching issues, it didn't solve the root problem
• npm ci still ignored git config and used lockfile URLs directly

Attempt 4: persist-credentials: false

What we did:

- uses: actions/checkout@v4
  with:
    persist-credentials: false

Why it failed (partially):

• This fixed the token override issue (actions/checkout was setting default GITHUB_TOKEN)
• But npm ci still couldn't authenticate because the lockfile URL had no token

Attempt 5: Local Git Config Override

What we did:

# Try to override global SSH rewrites with local HTTPS config
cd project-root
git config --local url."https://github.com/".insteadOf "git@github.com:"
git config --local url."https://github.com/".insteadOf "ssh://git@github.com/"
rm package-lock.json
npm install

Why it failed:

• npm's internal git client doesn't consistently honor local git config during package resolution
• Even with correct local config, npm still generated SSH URLs in package-lock.json
• The git config order of precedence (local > global > system) doesn't reliably apply to npm's git subprocess
• Testing with git ls-remote worked (used HTTPS), but npm install still wrote SSH URLs to lockfile
• Key insight: npm may use a different git execution context that bypasses local config

---

✅ What Finally Worked

The Root Cause

npm ci uses URLs from package-lock.json directly and ignores git config url.insteadOf

Even though:

• ✅ package.json had git+https://github.com/...
• ✅ package-lock.json had git+https://github.com/... (no SSH)
• ✅ git config was properly set

npm would run:

git ls-remote https://github.com/Memo-2023/mana-core-nestjs-package.git

This failed with "Permission denied" because:

1. The URL had no authentication token
2. npm fell back to SSH (which wasn't configured)
3. Git config rewrites were ignored by npm's git subprocess

The Solution: Runtime Token Injection

Step 1: Prevent Token Override

- uses: actions/checkout@v4
  with:
    persist-credentials: false  # Don't let default GITHUB_TOKEN interfere

Step 2a: If lockfile has HTTPS URLs - Verify and Inject Token

- name: Verify no SSH URLs in lockfile
  run: |
    if grep -q "git@github.com\|ssh://git@github.com" package-lock.json; then
      echo "❌ ERROR: SSH URLs found in package-lock.json"
      exit 1
    fi
    echo "✓ No SSH URLs found in package-lock.json"

- name: Inject token into package-lock.json
  env:
    GH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
  run: |
    sed -i "s|https://github.com/Memo-2023/|https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json

Step 2b: If lockfile has SSH URLs - Patch to HTTPS with Token

- name: Patch SSH URLs to HTTPS with token
  env:
    GH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
  run: |
    if grep -q "git+ssh://git@github.com" package-lock.json; then
      echo "⚠️  SSH URLs found - patching to HTTPS with token..."
      sed -i "s|git+ssh://git@github.com/Memo-2023/|git+https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
      echo "✓ Lockfile patched successfully"
    else
      echo "⚠️  No SSH URLs found - patching HTTPS URLs with token..."
      sed -i "s|https://github.com/Memo-2023/|https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
      echo "✓ Token injected successfully"
    fi

Note

: Use Step 2a if you can generate HTTPS lockfiles locally. Use Step 2b if your local git config always generates SSH URLs (accepts either format).

Step 4: Run npm ci

- name: Install dependencies
  run: npm ci

Why This Works

1. Direct URL Authentication: Token is embedded in the URL that npm ci reads
2. No Reliance on Git Config: Doesn't depend on git config which npm ignores
3. Just-in-Time: Token is injected at runtime, never committed to repo
4. Consistent Behavior: Works across npm v7, v8, v9+ and all CI environments

The Execution Order (Critical)

1. Checkout (persist-credentials: false)
2. Configure git (defense in depth, may help in edge cases)
3. Verify no SSH URLs (fail fast if lockfile is corrupted)
4. Setup Node.js (restore npm cache)
5. Inject token into package-lock.json ← THE KEY STEP
6. npm ci (now has authenticated URL from lockfile)

---

📊 Key Learnings

Issue	What We Learned
Git config ignored	npm ci reads package-lock.json directly, bypasses git config
persist-credentials	Default checkout token can override custom PAT config
npm v7+ behavior	Modern npm converts HTTPS to SSH for private repos if auth fails
Token placement	Must inject token into lockfile URL, not just configure git
sed is reliable	Runtime patching with sed is more reliable than git rewrites
Local git config fails	npm's git subprocess doesn't honor local config consistently - even git ls-remote works but npm install still writes SSH
Don't fight it locally	Stop trying to generate HTTPS lockfiles locally - accept SSH and patch in CI

🎯 The One-Liner That Fixed It

For HTTPS lockfiles:

sed -i "s|https://github.com/Memo-2023/|https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json

For SSH lockfiles (if your local git config generates SSH URLs):

sed -i "s|git+ssh://git@github.com/Memo-2023/|git+https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json

These commands ensure npm gets an authenticated HTTPS URL, preventing the SSH fallback entirely.

---

📝 Complete Working Workflow

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false

      - name: Configure git for private packages
        env:
          GH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
        run: |
          git config --global url."https://${GH_TOKEN}@github.com/".insteadOf ssh://git@github.com/
          git config --global url."https://${GH_TOKEN}@github.com/".insteadOf git@github.com:

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: 'backend/package-lock.json'

      - name: Patch package-lock.json with authenticated URLs
        env:
          GH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
        run: |
          cd backend
          # Handle both SSH and HTTPS URLs
          if grep -q "git+ssh://git@github.com" package-lock.json; then
            echo "⚠️  SSH URLs found - patching to HTTPS with token..."
            sed -i "s|git+ssh://git@github.com/Memo-2023/|git+https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
            echo "✓ Lockfile patched successfully"
          else
            echo "⚠️  HTTPS URLs found - injecting token..."
            sed -i "s|https://github.com/Memo-2023/|https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
            echo "✓ Token injected successfully"
          fi

      - name: Clear npm cache (if needed)
        run: npm cache clean --force || true

      - name: Install dependencies
        run: |
          cd backend
          npm ci --loglevel verbose

---

🔧 Required GitHub Secrets

Make sure you have set up in your repository:

• GH_PERSONAL_TOKEN: Personal Access Token with repo access
  • Scope: repo (Full control of private repositories)
  • Used for: Accessing private npm git dependencies

---

🎓 References

• npm/cli Issue #2610 - NPM v7 uses SSH instead of explicit HTTPS
• GitHub Community Discussion #25936 - Installing private GitHub repos via npm in GitHub Actions
• Stack Overflow: npm ignores git+https setting

---

---

🆕 Update: Handling SSH URLs in Lockfile (2025-09-30)

If your local git config always generates SSH URLs in package-lock.json (despite attempts to fix it locally), you have two options:

Option 1: Accept SSH and Patch in CI (Recommended)

1. Stop fighting with local git config - it's unreliable
2. Commit the lockfile with SSH URLs as-is
3. Use the flexible sed command in CI that handles both SSH and HTTPS:

- name: Patch package-lock.json with authenticated URLs
  env:
    GH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
  run: |
    cd backend
    if grep -q "git+ssh://git@github.com" package-lock.json; then
      sed -i "s|git+ssh://git@github.com/Memo-2023/|git+https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
    else
      sed -i "s|https://github.com/Memo-2023/|https://${GH_TOKEN}@github.com/Memo-2023/|g" package-lock.json
    fi

Option 2: Force HTTPS Locally (Fragile)

If you insist on HTTPS lockfiles, you must ensure NO global git config rewrites exist:

# Check for problematic rewrites
git config --global --get-regexp url

# Remove them
git config --global --unset-all url."git@github.com:".insteadOf
git config --global --unset-all url."ssh://git@github.com/".insteadOf

# Regenerate lockfile
rm package-lock.json
npm install

Warning: This changes your global git config and may break other workflows where you prefer SSH.

---

Date Fixed: 2025-09-29 Commit: 5cce473 - fix: inject PAT token into package-lock.json for npm ci

Updated: 2025-09-30 - Added SSH lockfile handling documentation