mirror of
https://github.com/Memo-2023/mana-monorepo.git
synced 2026-05-14 21:01:08 +02:00
Mirror of github.com/Memo-2023/mana-monorepo
45f368f471
The browser tier of @mana/local-llm was running entirely in the main
JS thread. With Gemma 4 E2B that meant ~50-200 ms of synchronous
tensor work per forward pass × ~150 forward passes per generation =
the UI froze for 10-30 seconds during a single chat reply. Scrolling,
clicks, animations all stopped.
Move the actual inference into a Dedicated Web Worker. The main
thread keeps a thin LocalLLMEngine proxy with the same public API
(load / unload / generate / prompt / extractJson / classify /
onStatusChange / isSupported), so existing callers — the /llm-test
page, the playground module, @mana/shared-llm's BrowserBackend, the
Svelte 5 reactive bindings — need NO changes.
File layout after the split:
src/engine.ts — main-thread proxy (lazy worker init,
postMessage protocol, pending request map,
status broadcast handling, convenience
wrappers for prompt/extractJson/classify)
src/worker.ts — Web Worker entry point (typed message
protocol, single LocalLLMEngineImpl instance,
forwards status changes back to main thread)
src/engine-impl.ts — the actual transformers.js engine (renamed
from the previous engine.ts contents). NOT
exported from index.ts — only the worker
imports it. Same two-step tokenization,
aggregated progress reporting, streaming
token handling as before; just running in
a different thread now.
Worker construction uses Vite's documented `new Worker(new URL(
'./worker.ts', import.meta.url), { type: 'module' })` pattern, which
makes Vite split worker.ts (and its transformers.js dep) into its
own bundle chunk at build time. The proxy is lazy-init: the Worker
constructor is never touched at module-import time, so SSR stays
clean (Worker doesn't exist on Node).
Message protocol (typed end-to-end):
Main → Worker:
{ id, type: 'load', modelKey: ModelKey }
{ id, type: 'unload' }
{ id, type: 'generate', opts: SerializableGenerateOptions }
{ id, type: 'isReady' }
Worker → Main:
{ id, type: 'result', data?: unknown }
{ id, type: 'error', message: string }
{ id, type: 'token', token: string } — streaming chunk
{ type: 'status', status: LoadingStatus } — broadcast
The proxy assigns a unique id per request, stores the resolve/reject
+ optional onToken callback in a Map<id, PendingRequest>, and routes
incoming responses by id. Status messages have no id and fire every
registered status listener — same UX as before, just one extra hop.
Streaming: the worker re-attaches the streaming callback on its
side. Each emitted token gets posted back as `{ id, type: 'token',
token }` and the proxy invokes the original `onToken` callback. The
final `result` arrives as a normal response and resolves the
Promise. From the caller's perspective generate() still feels
identical — same async iterable feel via onToken, same return value.
Worker termination on unload: transformers.js doesn't expose a
dispose API, so we terminate the worker after unload and create a
fresh one on the next load. This is the only reliable way to
release VRAM between model swaps.
CSP: no header changes needed. The worker is loaded from a
same-origin URL (Vite emits it as
/_app/immutable/workers/worker.[hash].js), so 'self' in script-src
already covers it. The blob: + cdn.jsdelivr.net + wasm-unsafe-eval
allowlists we added during the original WebLLM/transformers.js
bring-up still apply because the worker still runs the same ONNX
runtime that needed them.
DistributiveOmit type helper: TS's plain `Omit<Union, K>` collapses
discriminated unions to an intersection in some configurations,
which broke the type narrowing at the postRequest call sites for
each request variant. Adding a tiny `DistributiveOmit<T, K>` helper
fixes the type-check without restructuring the protocol.
What this commit deliberately does NOT do:
- Change the public API surface. The whole point is that callers
remain untouched.
- Add multi-tab worker coordination via SharedWorker or
BroadcastChannel. Each tab still spawns its own dedicated worker
with its own copy of the model in VRAM. Multi-tab dedup is
Phase 2.5/Phase 4 work — see the design doc summary in the
previous Phase 1 commit message.
- Add a persistent task queue. Fire-and-forget background tasks
are Phase 4.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .changeset | ||
| .claude | ||
| .github | ||
| .husky | ||
| apps | ||
| docker | ||
| docs | ||
| games | ||
| load-tests | ||
| NewAppIdeas/Roblox Reimagined | ||
| packages | ||
| patches | ||
| scripts | ||
| services | ||
| tests | ||
| .dockerignore | ||
| .editorconfig | ||
| .env.development | ||
| .env.macmini.example | ||
| .env.secrets.example | ||
| .gitignore | ||
| .npmrc | ||
| .nvmrc | ||
| .prettierignore | ||
| .prettierrc.json | ||
| CLAUDE.md | ||
| cloudflared-config.yml | ||
| docker-compose.dev.yml | ||
| docker-compose.macmini.yml | ||
| docker-compose.test.yml | ||
| eslint.config.mjs | ||
| gift-codes-2026-02-14.txt | ||
| lint-staged.config.js | ||
| package.json | ||
| playwright.config.ts | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| README.md | ||
| TROUBLESHOOTING.md | ||
| turbo.json | ||
| vitest.config.ts | ||
Mana Monorepo
Monorepo containing all Mana projects — a self-hosted multi-app ecosystem with shared packages and unified tooling.
Projects
| Project | Description | Apps |
|---|---|---|
| mana | Multi-app ecosystem platform | Expo mobile, SvelteKit web |
| chat | AI chat application | NestJS backend, Expo mobile, SvelteKit web, Astro landing |
| todo | Task management | NestJS backend, SvelteKit web, Astro landing |
| calendar | Calendar & scheduling | NestJS backend, SvelteKit web, Astro landing |
| clock | Pomodoro & time tracking | NestJS backend, SvelteKit web, Astro landing |
| contacts | Contact management | NestJS backend, SvelteKit web |
| picture | AI image generation | NestJS backend, Expo mobile, SvelteKit web, Astro landing |
| cards | Card/deck management | NestJS backend, Expo mobile, SvelteKit web |
| zitare | Daily inspiration quotes | NestJS backend, Expo mobile, SvelteKit web, Astro landing |
| mukke | Music player | NestJS backend, SvelteKit web |
| planta | Plant care tracker | NestJS backend, SvelteKit web |
| storage | Cloud storage | NestJS backend, SvelteKit web |
| questions | Q&A with web search | SvelteKit web |
| skilltree | Skill tree visualization | NestJS backend, SvelteKit web |
| nutriphi | Nutrition tracking | NestJS backend, SvelteKit web |
| citycorners | City guide | NestJS backend, SvelteKit web, Astro landing |
| presi | Presentation tool | NestJS backend, SvelteKit web |
| photos | Photo management | NestJS backend, SvelteKit web |
Getting Started
Prerequisites
- Node.js 20+
- pnpm 9.15.0+
- Docker (for PostgreSQL, Redis, MinIO)
Installation
pnpm install
Development
# Start infrastructure (PostgreSQL, Redis, MinIO)
pnpm docker:up
# Start any app with auto DB setup
pnpm dev:chat:full
pnpm dev:todo:full
pnpm dev:calendar:full
pnpm dev:contacts:full
# Build & quality
pnpm run build
pnpm run type-check
pnpm run format
See CLAUDE.md for comprehensive development documentation.
Architecture
mana-monorepo/
├── apps/ # Product applications
├── services/ # Microservices (auth, search, LLM, bots)
├── packages/ # Shared packages
├── docker/ # Docker configuration
└── scripts/ # Development & deployment scripts
Tooling
- Package Manager: pnpm 9.15.0
- Build System: Turborepo
- Formatting: Prettier (tabs, single quotes, 100 char width)
- Hosting: Mac Mini (self-hosted) via Docker + Cloudflare Tunnel
- Analytics: Umami (stats.mana.how)
License
Private - All rights reserved