mirror of
https://github.com/Memo-2023/mana-monorepo.git
synced 2026-05-16 11:59:39 +02:00
✨ feat(mana-media): add unified media processing platform MVP
- Create mana-media service for centralized media handling - Add upload, processing, and delivery modules - Configure BullMQ for async transcoding jobs - Add S3-compatible storage integration - Create TypeScript client package Features: - Multi-format image/video upload - Async transcoding via ffmpeg - Adaptive streaming (HLS) support - Signed URL delivery Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Till-JS
parent
5c8120f437
commit
cd28a83007
28 changed files with 5318 additions and 0 deletions
215
docs/EXTERNAL_SSD_OPPORTUNITIES.md
Normal file
215
docs/EXTERNAL_SSD_OPPORTUNITIES.md
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
# External 4TB SSD - Opportunities & Usage Guide
|
||||
|
||||
This document outlines the opportunities enabled by the 4TB external SSD connected to the Mac Mini production server.
|
||||
|
||||
## Current Setup
|
||||
|
||||
| Component | Details |
|
||||
|-----------|---------|
|
||||
| **Device** | Mac Mini M4 (16GB RAM) |
|
||||
| **Internal SSD** | 228 GB (30 GB free) |
|
||||
| **External SSD** | 4 TB (3.6 TB free) |
|
||||
| **Mount Point** | `/Volumes/TillJakob-S04` |
|
||||
| **Data Directory** | `/Volumes/TillJakob-S04/ManaData/` |
|
||||
|
||||
## Currently Migrated to SSD
|
||||
|
||||
| Item | Size | Path |
|
||||
|------|------|------|
|
||||
| Ollama Models | ~26 GB | `ManaData/ollama/` (symlink: `~/.ollama`) |
|
||||
| STT Models | ~19 GB | `ManaData/stt-models/` (symlink: `~/STT-models`) |
|
||||
| FLUX.2 | ~15 GB | `ManaData/flux2/` (symlink: `~/FLUX.2`) |
|
||||
| Matrix Media | Variable | `ManaData/matrix-media/` |
|
||||
| PostgreSQL Backups | Variable | `ManaData/backups/postgres/` |
|
||||
|
||||
---
|
||||
|
||||
## Opportunities
|
||||
|
||||
### 1. Larger LLM Models
|
||||
|
||||
With 4TB available, we can host significantly larger and more capable models:
|
||||
|
||||
| Model | Size | Use Case |
|
||||
|-------|------|----------|
|
||||
| `llama3:70b-q4` | ~40 GB | Highest quality general-purpose |
|
||||
| `mixtral:8x7b` | ~26 GB | Fast Mixture of Experts |
|
||||
| `codestral:22b` | ~13 GB | Best code assistant |
|
||||
| `qwen2.5:32b` | ~20 GB | Excellent multilingual (German) |
|
||||
| `deepseek-coder-v2:16b` | ~10 GB | Top coding performance |
|
||||
| `llava:34b` | ~20 GB | Best vision model |
|
||||
|
||||
**Potential:** Host 10-20 specialized models for different tasks, switch dynamically based on use case.
|
||||
|
||||
### 2. RAG / Knowledge Databases
|
||||
|
||||
Enable semantic search and retrieval-augmented generation:
|
||||
|
||||
- **Vector Database** (Qdrant, ChromaDB, Milvus)
|
||||
- Index documents, PDFs, codebases
|
||||
- Build knowledge bases for Chat app
|
||||
- Make company documentation searchable
|
||||
- Estimated storage: 50-200 GB depending on corpus size
|
||||
|
||||
### 3. Local AI Services Expansion
|
||||
|
||||
| Service | Storage | Benefit |
|
||||
|---------|---------|---------|
|
||||
| **Whisper Large-v3** | ~3 GB | Best-in-class speech recognition |
|
||||
| **ComfyUI + Models** | 50-100 GB | Local image generation (SDXL, Flux) |
|
||||
| **MusicGen** | ~10 GB | AI music generation |
|
||||
| **Video Models** | 20-50 GB | Local video AI |
|
||||
| **TTS Models** | ~5 GB | High-quality text-to-speech |
|
||||
|
||||
### 4. Extended Backups & Disaster Recovery
|
||||
|
||||
Comprehensive backup strategy enabled by large storage:
|
||||
|
||||
#### Database Backups
|
||||
- **Daily snapshots** with 90+ day retention
|
||||
- **Point-in-time recovery** capability
|
||||
- **Cross-database consistency** backups
|
||||
- Estimated: 50-100 GB for full history
|
||||
|
||||
#### Docker Infrastructure
|
||||
- **Local image registry** for faster deployments
|
||||
- **Build cache** persistence across restarts
|
||||
- **Container snapshots** before major updates
|
||||
- Estimated: 100-200 GB
|
||||
|
||||
#### Code & Configuration
|
||||
- **Git repository mirrors** (full clone backups)
|
||||
- **Configuration backups** (Docker, Nginx, etc.)
|
||||
- **Secrets backup** (encrypted)
|
||||
- Estimated: 10-50 GB
|
||||
|
||||
#### System Recovery
|
||||
- **Full system snapshots** via Time Machine or rsync
|
||||
- **Bootable recovery** partition
|
||||
- **Quick restore** capability
|
||||
- Estimated: 250-500 GB
|
||||
|
||||
### 5. Media & Content Storage
|
||||
|
||||
Centralized media handling for all applications:
|
||||
|
||||
#### MinIO Expansion
|
||||
| Bucket | Purpose | Est. Size |
|
||||
|--------|---------|-----------|
|
||||
| `picture-storage` | AI-generated images | 100+ GB |
|
||||
| `storage-storage` | User cloud storage | 500+ GB |
|
||||
| `nutriphi-storage` | Meal photos | 50+ GB |
|
||||
| `chat-attachments` | Chat file uploads | 100+ GB |
|
||||
| `presi-assets` | Presentation media | 50+ GB |
|
||||
|
||||
#### Media Processing
|
||||
- **Video transcoding** pipeline with temp storage
|
||||
- **Image optimization** cache
|
||||
- **Thumbnail generation** storage
|
||||
- **Audio processing** workspace
|
||||
|
||||
#### Content Delivery
|
||||
- **Static asset hosting** for all apps
|
||||
- **Game assets** for games/ projects
|
||||
- **Landing page media** (images, videos)
|
||||
- **Documentation assets**
|
||||
|
||||
### 6. New Application Possibilities
|
||||
|
||||
The storage enables entirely new application categories:
|
||||
|
||||
#### Media Applications
|
||||
| App Idea | Storage Need | Description |
|
||||
|----------|--------------|-------------|
|
||||
| **Video Library** | 500+ GB | Local video storage with transcripts, searchable via AI |
|
||||
| **Music Streaming** | 200+ GB | Personal music collection, Spotify-like interface |
|
||||
| **Photo Library** | 500+ GB | iCloud/Google Photos alternative with AI tagging |
|
||||
| **Podcast Archive** | 100+ GB | Download, transcribe, search podcasts |
|
||||
|
||||
#### Document & Knowledge
|
||||
| App Idea | Storage Need | Description |
|
||||
|----------|--------------|-------------|
|
||||
| **Document Vault** | 100+ GB | Encrypted document storage with OCR |
|
||||
| **Research Archive** | 200+ GB | Papers, articles, bookmarks with AI summaries |
|
||||
| **Code Archive** | 50+ GB | Searchable repository of code snippets |
|
||||
| **Learning Library** | 100+ GB | Courses, tutorials, educational content |
|
||||
|
||||
#### AI-Powered Services
|
||||
| App Idea | Storage Need | Description |
|
||||
|----------|--------------|-------------|
|
||||
| **Local AI Studio** | 200+ GB | ComfyUI, training data, generated outputs |
|
||||
| **Voice Clone Lab** | 20+ GB | Custom TTS voices |
|
||||
| **Dataset Hub** | 100+ GB | ML training datasets |
|
||||
|
||||
### 7. Development & Testing
|
||||
|
||||
Enhanced development workflow:
|
||||
|
||||
- **Large test datasets** for ML experiments
|
||||
- **Build cache** for faster CI/CD
|
||||
- **Staging databases** with production-like data
|
||||
- **Log aggregation** (Loki/ELK) with extended retention
|
||||
- **Performance profiling** data storage
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
### Phase 1 (Immediate)
|
||||
- [x] Migrate Ollama models
|
||||
- [x] Migrate STT/FLUX models
|
||||
- [x] Setup PostgreSQL backups
|
||||
- [ ] Download additional LLM models
|
||||
|
||||
### Phase 2 (Short-term)
|
||||
- [ ] Setup local Docker registry
|
||||
- [ ] Expand MinIO to SSD
|
||||
- [ ] Implement extended backup retention
|
||||
- [ ] Add vector database for RAG
|
||||
|
||||
### Phase 3 (Medium-term)
|
||||
- [ ] Setup ComfyUI for local image generation
|
||||
- [ ] Implement media processing pipeline
|
||||
- [ ] Add video/audio transcription service
|
||||
|
||||
### Phase 4 (Long-term)
|
||||
- [ ] Build new media-focused applications
|
||||
- [ ] Implement full disaster recovery
|
||||
- [ ] Create AI training infrastructure
|
||||
|
||||
---
|
||||
|
||||
## Technical Notes
|
||||
|
||||
### Adding Docker File Sharing for SSD
|
||||
|
||||
To enable Docker containers to use SSD storage directly:
|
||||
|
||||
1. Open Docker Desktop → Settings → Resources → File Sharing
|
||||
2. Add `/Volumes/TillJakob-S04/ManaData/`
|
||||
3. Restart Docker Desktop
|
||||
|
||||
### Symlink Pattern
|
||||
|
||||
For applications that don't support custom paths:
|
||||
```bash
|
||||
# Move data to SSD
|
||||
mv ~/original-path /Volumes/TillJakob-S04/ManaData/new-location
|
||||
|
||||
# Create symlink
|
||||
ln -s /Volumes/TillJakob-S04/ManaData/new-location ~/original-path
|
||||
```
|
||||
|
||||
### Monitoring SSD Usage
|
||||
|
||||
```bash
|
||||
# Check SSD usage
|
||||
df -h /Volumes/TillJakob-S04
|
||||
|
||||
# Check ManaData breakdown
|
||||
du -sh /Volumes/TillJakob-S04/ManaData/*
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-02-01*
|
||||
22
services/mana-media/.env.example
Normal file
22
services/mana-media/.env.example
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Server
|
||||
PORT=3050
|
||||
NODE_ENV=development
|
||||
|
||||
# Redis (for job queue)
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
|
||||
# MinIO / S3
|
||||
S3_ENDPOINT=localhost
|
||||
S3_PORT=9000
|
||||
S3_USE_SSL=false
|
||||
S3_ACCESS_KEY=minioadmin
|
||||
S3_SECRET_KEY=minioadmin
|
||||
S3_BUCKET=mana-media
|
||||
S3_PUBLIC_URL=http://localhost:9000/mana-media
|
||||
|
||||
# Public URL for generated links
|
||||
PUBLIC_URL=http://localhost:3050/api/v1
|
||||
|
||||
# CORS
|
||||
CORS_ORIGINS=http://localhost:3000,http://localhost:5173
|
||||
173
services/mana-media/CLAUDE.md
Normal file
173
services/mana-media/CLAUDE.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
# mana-media - Unified Media Platform
|
||||
|
||||
Central media handling service for all ManaCore applications.
|
||||
|
||||
## Overview
|
||||
|
||||
mana-media provides:
|
||||
- **Upload API** - Chunked uploads, deduplication
|
||||
- **Processing** - Thumbnails, WebP conversion, resizing
|
||||
- **Delivery** - Optimized file serving, on-the-fly transforms
|
||||
- **Search** (planned) - Vector-based semantic search
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Start dependencies (Redis + MinIO)
|
||||
docker compose up redis minio -d
|
||||
|
||||
# Install dependencies
|
||||
pnpm install
|
||||
|
||||
# Start development server
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
Service runs on `http://localhost:3050`
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Upload
|
||||
```bash
|
||||
# Upload file
|
||||
curl -X POST http://localhost:3050/api/v1/media/upload \
|
||||
-F "file=@image.jpg" \
|
||||
-F "app=chat" \
|
||||
-F "userId=user123"
|
||||
|
||||
# Response
|
||||
{
|
||||
"id": "abc123",
|
||||
"status": "processing",
|
||||
"urls": {
|
||||
"original": "http://localhost:3050/api/v1/media/abc123/file",
|
||||
"thumbnail": "http://localhost:3050/api/v1/media/abc123/file/thumb"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Get Media
|
||||
```bash
|
||||
# Get metadata
|
||||
GET /api/v1/media/:id
|
||||
|
||||
# Get original file
|
||||
GET /api/v1/media/:id/file
|
||||
|
||||
# Get thumbnail
|
||||
GET /api/v1/media/:id/file/thumb
|
||||
|
||||
# Get medium variant
|
||||
GET /api/v1/media/:id/file/medium
|
||||
|
||||
# On-the-fly transform
|
||||
GET /api/v1/media/:id/transform?w=400&h=300&fit=cover&format=webp
|
||||
```
|
||||
|
||||
### List & Delete
|
||||
```bash
|
||||
# List media
|
||||
GET /api/v1/media?app=chat&limit=50
|
||||
|
||||
# Delete
|
||||
DELETE /api/v1/media/:id
|
||||
```
|
||||
|
||||
## Client Library
|
||||
|
||||
```typescript
|
||||
import { MediaClient } from '@manacore/media-client';
|
||||
|
||||
const media = new MediaClient('http://localhost:3050');
|
||||
|
||||
// Upload
|
||||
const result = await media.upload(file, { app: 'chat' });
|
||||
|
||||
// Wait for processing
|
||||
const ready = await media.waitForReady(result.id);
|
||||
|
||||
// Get URLs
|
||||
const thumbUrl = media.getThumbnailUrl(result.id);
|
||||
const customUrl = media.getTransformUrl(result.id, {
|
||||
width: 400,
|
||||
format: 'webp'
|
||||
});
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ mana-media │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Upload Module │ Handles file uploads │
|
||||
│ Process Module │ Sharp/FFmpeg processing │
|
||||
│ Storage Module │ MinIO abstraction │
|
||||
│ Delivery Module │ File serving + transforms │
|
||||
└─────────────────────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────┐ ┌─────────┐
|
||||
│ Redis │ │ MinIO │
|
||||
│ Queue │ │ Storage │
|
||||
└─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
| File Type | Generated Variants |
|
||||
|-----------|-------------------|
|
||||
| Images | thumb (200x200), medium (800x800), large (1920x1920) |
|
||||
| Videos | (planned) thumbnail, HLS streaming |
|
||||
| Documents | (planned) thumbnail, text extraction |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| PORT | 3050 | API port |
|
||||
| REDIS_HOST | localhost | Redis host |
|
||||
| REDIS_PORT | 6379 | Redis port |
|
||||
| S3_ENDPOINT | localhost | MinIO/S3 endpoint |
|
||||
| S3_PORT | 9000 | MinIO/S3 port |
|
||||
| S3_ACCESS_KEY | minioadmin | S3 access key |
|
||||
| S3_SECRET_KEY | minioadmin | S3 secret key |
|
||||
| S3_BUCKET | mana-media | Storage bucket |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Run with watch mode
|
||||
pnpm dev
|
||||
|
||||
# Type check
|
||||
pnpm type-check
|
||||
|
||||
# Build
|
||||
pnpm build
|
||||
```
|
||||
|
||||
## Storage Layout
|
||||
|
||||
```
|
||||
mana-media bucket/
|
||||
├── originals/ # Original uploads
|
||||
│ └── 2026/02/01/
|
||||
│ └── {id}.{ext}
|
||||
├── processed/ # Generated variants
|
||||
│ └── {id}/
|
||||
│ ├── thumb.webp
|
||||
│ ├── medium.webp
|
||||
│ └── large.webp
|
||||
└── cache/ # Transform cache
|
||||
└── {id}_{params}.webp
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [x] v0.1: Basic upload + thumbnails
|
||||
- [ ] v0.2: Video thumbnails (FFmpeg)
|
||||
- [ ] v0.3: Chunked upload for large files
|
||||
- [ ] v0.4: OCR for documents
|
||||
- [ ] v0.5: Vector search (Qdrant)
|
||||
- [ ] v1.0: Full production ready
|
||||
42
services/mana-media/apps/api/Dockerfile
Normal file
42
services/mana-media/apps/api/Dockerfile
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
FROM node:22-alpine AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install pnpm
|
||||
RUN corepack enable && corepack prepare pnpm@latest --activate
|
||||
|
||||
# Copy package files
|
||||
COPY package.json pnpm-lock.yaml* ./
|
||||
|
||||
# Install dependencies
|
||||
RUN pnpm install --frozen-lockfile
|
||||
|
||||
# Copy source
|
||||
COPY . .
|
||||
|
||||
# Build
|
||||
RUN pnpm build
|
||||
|
||||
# Production stage
|
||||
FROM node:22-alpine AS runner
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install pnpm
|
||||
RUN corepack enable && corepack prepare pnpm@latest --activate
|
||||
|
||||
# Copy package files and install production deps only
|
||||
COPY package.json pnpm-lock.yaml* ./
|
||||
RUN pnpm install --frozen-lockfile --prod
|
||||
|
||||
# Copy built files
|
||||
COPY --from=builder /app/dist ./dist
|
||||
|
||||
# Create non-root user
|
||||
RUN addgroup --system --gid 1001 nodejs && \
|
||||
adduser --system --uid 1001 nestjs
|
||||
USER nestjs
|
||||
|
||||
EXPOSE 3050
|
||||
|
||||
CMD ["node", "dist/main"]
|
||||
5
services/mana-media/apps/api/nest-cli.json
Normal file
5
services/mana-media/apps/api/nest-cli.json
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
{
|
||||
"$schema": "https://json.schemastore.org/nest-cli",
|
||||
"collection": "@nestjs/schematics",
|
||||
"sourceRoot": "src"
|
||||
}
|
||||
36
services/mana-media/apps/api/package.json
Normal file
36
services/mana-media/apps/api/package.json
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
{
|
||||
"name": "@mana-media/api",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "nest start --watch",
|
||||
"build": "nest build",
|
||||
"start": "nest start",
|
||||
"start:prod": "node dist/main",
|
||||
"type-check": "tsc --noEmit",
|
||||
"lint": "eslint 'src/**/*.ts'"
|
||||
},
|
||||
"dependencies": {
|
||||
"@nestjs/bullmq": "^11.0.0",
|
||||
"@nestjs/common": "^11.0.0",
|
||||
"@nestjs/config": "^3.3.0",
|
||||
"@nestjs/core": "^11.0.0",
|
||||
"@nestjs/platform-express": "^11.0.0",
|
||||
"bullmq": "^5.34.0",
|
||||
"mime-types": "^2.1.35",
|
||||
"minio": "^8.0.0",
|
||||
"reflect-metadata": "^0.2.0",
|
||||
"rxjs": "^7.8.0",
|
||||
"sharp": "^0.33.0",
|
||||
"uuid": "^11.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@nestjs/cli": "^11.0.0",
|
||||
"@types/express": "^5.0.0",
|
||||
"@types/mime-types": "^2.1.4",
|
||||
"@types/multer": "^2.0.0",
|
||||
"@types/node": "^22.0.0",
|
||||
"@types/uuid": "^10.0.0",
|
||||
"typescript": "^5.7.0"
|
||||
}
|
||||
}
|
||||
3602
services/mana-media/apps/api/pnpm-lock.yaml
generated
Normal file
3602
services/mana-media/apps/api/pnpm-lock.yaml
generated
Normal file
File diff suppressed because it is too large
Load diff
28
services/mana-media/apps/api/src/app.module.ts
Normal file
28
services/mana-media/apps/api/src/app.module.ts
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
import { Module } from '@nestjs/common';
|
||||
import { ConfigModule } from '@nestjs/config';
|
||||
import { BullModule } from '@nestjs/bullmq';
|
||||
import { UploadModule } from './modules/upload/upload.module';
|
||||
import { StorageModule } from './modules/storage/storage.module';
|
||||
import { ProcessModule } from './modules/process/process.module';
|
||||
import { DeliveryModule } from './modules/delivery/delivery.module';
|
||||
import { HealthController } from './health.controller';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ConfigModule.forRoot({
|
||||
isGlobal: true,
|
||||
}),
|
||||
BullModule.forRoot({
|
||||
connection: {
|
||||
host: process.env.REDIS_HOST || 'localhost',
|
||||
port: parseInt(process.env.REDIS_PORT || '6379'),
|
||||
},
|
||||
}),
|
||||
StorageModule,
|
||||
UploadModule,
|
||||
ProcessModule,
|
||||
DeliveryModule,
|
||||
],
|
||||
controllers: [HealthController],
|
||||
})
|
||||
export class AppModule {}
|
||||
13
services/mana-media/apps/api/src/health.controller.ts
Normal file
13
services/mana-media/apps/api/src/health.controller.ts
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
import { Controller, Get } from '@nestjs/common';
|
||||
|
||||
@Controller()
|
||||
export class HealthController {
|
||||
@Get('health')
|
||||
health() {
|
||||
return {
|
||||
status: 'ok',
|
||||
service: 'mana-media',
|
||||
timestamp: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
}
|
||||
28
services/mana-media/apps/api/src/main.ts
Normal file
28
services/mana-media/apps/api/src/main.ts
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
import { NestFactory } from '@nestjs/core';
|
||||
import { ValidationPipe } from '@nestjs/common';
|
||||
import { AppModule } from './app.module';
|
||||
|
||||
async function bootstrap() {
|
||||
const app = await NestFactory.create(AppModule);
|
||||
|
||||
app.setGlobalPrefix('api/v1');
|
||||
|
||||
app.useGlobalPipes(
|
||||
new ValidationPipe({
|
||||
whitelist: true,
|
||||
transform: true,
|
||||
})
|
||||
);
|
||||
|
||||
app.enableCors({
|
||||
origin: process.env.CORS_ORIGINS?.split(',') || '*',
|
||||
credentials: true,
|
||||
});
|
||||
|
||||
const port = process.env.PORT || 3050;
|
||||
await app.listen(port);
|
||||
|
||||
console.log(`mana-media service running on port ${port}`);
|
||||
}
|
||||
|
||||
bootstrap();
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
import {
|
||||
Controller,
|
||||
Get,
|
||||
Param,
|
||||
Query,
|
||||
Res,
|
||||
NotFoundException,
|
||||
BadRequestException,
|
||||
} from '@nestjs/common';
|
||||
import { Response } from 'express';
|
||||
import { UploadService } from '../upload/upload.service';
|
||||
import { ProcessService } from '../process/process.service';
|
||||
import { StorageService } from '../storage/storage.service';
|
||||
|
||||
type Variant = 'thumb' | 'medium' | 'large';
|
||||
|
||||
@Controller('media')
|
||||
export class DeliveryController {
|
||||
constructor(
|
||||
private uploadService: UploadService,
|
||||
private processService: ProcessService,
|
||||
private storage: StorageService
|
||||
) {}
|
||||
|
||||
@Get(':id/file')
|
||||
async getOriginal(@Param('id') id: string, @Res() res: Response): Promise<void> {
|
||||
const record = await this.uploadService.get(id);
|
||||
if (!record) {
|
||||
throw new NotFoundException('Media not found');
|
||||
}
|
||||
|
||||
await this.streamFile(res, record.keys.original, record.mimeType);
|
||||
}
|
||||
|
||||
@Get(':id/file/:variant')
|
||||
async getVariant(
|
||||
@Param('id') id: string,
|
||||
@Param('variant') variant: Variant,
|
||||
@Res() res: Response
|
||||
): Promise<void> {
|
||||
const record = await this.uploadService.get(id);
|
||||
if (!record) {
|
||||
throw new NotFoundException('Media not found');
|
||||
}
|
||||
|
||||
const variantMap: Record<Variant, string | undefined> = {
|
||||
thumb: record.keys.thumbnail,
|
||||
medium: record.keys.medium,
|
||||
large: record.keys.large,
|
||||
};
|
||||
|
||||
const key = variantMap[variant];
|
||||
if (!key) {
|
||||
// Fallback to original if variant doesn't exist
|
||||
await this.streamFile(res, record.keys.original, record.mimeType);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.streamFile(res, key, 'image/webp');
|
||||
}
|
||||
|
||||
@Get(':id/transform')
|
||||
async transform(
|
||||
@Param('id') id: string,
|
||||
@Query('w') width?: string,
|
||||
@Query('h') height?: string,
|
||||
@Query('fit') fit?: string,
|
||||
@Query('format') format?: string,
|
||||
@Query('q') quality?: string,
|
||||
@Res() res?: Response
|
||||
): Promise<void> {
|
||||
if (!res) return;
|
||||
|
||||
const record = await this.uploadService.get(id);
|
||||
if (!record) {
|
||||
throw new NotFoundException('Media not found');
|
||||
}
|
||||
|
||||
if (!record.mimeType.startsWith('image/')) {
|
||||
throw new BadRequestException('Transform only supported for images');
|
||||
}
|
||||
|
||||
// Download original
|
||||
const originalBuffer = await this.storage.download(record.keys.original);
|
||||
|
||||
// Transform
|
||||
const transformedBuffer = await this.processService.transformImage(originalBuffer, {
|
||||
width: width ? parseInt(width) : undefined,
|
||||
height: height ? parseInt(height) : undefined,
|
||||
fit: (fit as 'cover' | 'contain' | 'fill' | 'inside' | 'outside') || 'inside',
|
||||
format: (format as 'webp' | 'jpeg' | 'png' | 'avif') || 'webp',
|
||||
quality: quality ? parseInt(quality) : 85,
|
||||
});
|
||||
|
||||
const mimeTypes: Record<string, string> = {
|
||||
webp: 'image/webp',
|
||||
jpeg: 'image/jpeg',
|
||||
png: 'image/png',
|
||||
avif: 'image/avif',
|
||||
};
|
||||
|
||||
res.set('Content-Type', mimeTypes[format || 'webp']);
|
||||
res.set('Cache-Control', 'public, max-age=31536000');
|
||||
res.send(transformedBuffer);
|
||||
}
|
||||
|
||||
private async streamFile(res: Response, key: string, contentType: string): Promise<void> {
|
||||
try {
|
||||
const stream = await this.storage.getStream(key);
|
||||
|
||||
res.set('Content-Type', contentType);
|
||||
res.set('Cache-Control', 'public, max-age=31536000');
|
||||
|
||||
stream.pipe(res);
|
||||
} catch (error) {
|
||||
throw new NotFoundException('File not found');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
import { Module } from '@nestjs/common';
|
||||
import { DeliveryController } from './delivery.controller';
|
||||
import { UploadModule } from '../upload/upload.module';
|
||||
import { ProcessModule } from '../process/process.module';
|
||||
|
||||
@Module({
|
||||
imports: [UploadModule, ProcessModule],
|
||||
controllers: [DeliveryController],
|
||||
})
|
||||
export class DeliveryModule {}
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
export const PROCESS_QUEUE = 'media-process';
|
||||
|
||||
export const IMAGE_VARIANTS = {
|
||||
thumbnail: { width: 200, height: 200, fit: 'cover' as const },
|
||||
medium: { width: 800, height: 800, fit: 'inside' as const },
|
||||
large: { width: 1920, height: 1920, fit: 'inside' as const },
|
||||
};
|
||||
|
||||
export const SUPPORTED_IMAGE_TYPES = [
|
||||
'image/jpeg',
|
||||
'image/png',
|
||||
'image/webp',
|
||||
'image/gif',
|
||||
'image/avif',
|
||||
'image/heic',
|
||||
'image/heif',
|
||||
];
|
||||
|
||||
export const SUPPORTED_VIDEO_TYPES = ['video/mp4', 'video/quicktime', 'video/webm', 'video/mpeg'];
|
||||
|
||||
export const SUPPORTED_AUDIO_TYPES = ['audio/mpeg', 'audio/wav', 'audio/ogg', 'audio/webm'];
|
||||
|
||||
export const SUPPORTED_DOCUMENT_TYPES = [
|
||||
'application/pdf',
|
||||
'application/msword',
|
||||
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
];
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
import { Module, forwardRef } from '@nestjs/common';
|
||||
import { BullModule } from '@nestjs/bullmq';
|
||||
import { ProcessService } from './process.service';
|
||||
import { ProcessWorker } from './process.worker';
|
||||
import { PROCESS_QUEUE } from './process.constants';
|
||||
import { UploadModule } from '../upload/upload.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
BullModule.registerQueue({
|
||||
name: PROCESS_QUEUE,
|
||||
}),
|
||||
forwardRef(() => UploadModule),
|
||||
],
|
||||
providers: [ProcessService, ProcessWorker],
|
||||
exports: [ProcessService],
|
||||
})
|
||||
export class ProcessModule {}
|
||||
|
|
@ -0,0 +1,150 @@
|
|||
import { Injectable } from '@nestjs/common';
|
||||
import sharp from 'sharp';
|
||||
import { StorageService } from '../storage/storage.service';
|
||||
import { IMAGE_VARIANTS, SUPPORTED_IMAGE_TYPES } from './process.constants';
|
||||
|
||||
export interface ProcessResult {
|
||||
thumbnail?: string;
|
||||
medium?: string;
|
||||
large?: string;
|
||||
metadata?: {
|
||||
width?: number;
|
||||
height?: number;
|
||||
format?: string;
|
||||
hasAlpha?: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
@Injectable()
|
||||
export class ProcessService {
|
||||
constructor(private storage: StorageService) {}
|
||||
|
||||
async processImage(
|
||||
mediaId: string,
|
||||
originalKey: string,
|
||||
mimeType: string
|
||||
): Promise<ProcessResult> {
|
||||
if (!SUPPORTED_IMAGE_TYPES.includes(mimeType)) {
|
||||
return {};
|
||||
}
|
||||
|
||||
// Download original
|
||||
const originalBuffer = await this.storage.download(originalKey);
|
||||
|
||||
// Get metadata
|
||||
const image = sharp(originalBuffer);
|
||||
const metadata = await image.metadata();
|
||||
|
||||
const result: ProcessResult = {
|
||||
metadata: {
|
||||
width: metadata.width,
|
||||
height: metadata.height,
|
||||
format: metadata.format,
|
||||
hasAlpha: metadata.hasAlpha,
|
||||
},
|
||||
};
|
||||
|
||||
// Generate variants
|
||||
const basePath = originalKey.replace(/^originals\//, 'processed/').replace(/\.[^.]+$/, '');
|
||||
|
||||
// Thumbnail
|
||||
const thumbKey = `${basePath}/thumb.webp`;
|
||||
const thumbBuffer = await sharp(originalBuffer)
|
||||
.resize(IMAGE_VARIANTS.thumbnail.width, IMAGE_VARIANTS.thumbnail.height, {
|
||||
fit: IMAGE_VARIANTS.thumbnail.fit,
|
||||
})
|
||||
.webp({ quality: 80 })
|
||||
.toBuffer();
|
||||
|
||||
await this.storage.upload(thumbKey, thumbBuffer, 'image/webp');
|
||||
result.thumbnail = thumbKey;
|
||||
|
||||
// Medium (only if original is larger)
|
||||
if (
|
||||
(metadata.width || 0) > IMAGE_VARIANTS.medium.width ||
|
||||
(metadata.height || 0) > IMAGE_VARIANTS.medium.height
|
||||
) {
|
||||
const mediumKey = `${basePath}/medium.webp`;
|
||||
const mediumBuffer = await sharp(originalBuffer)
|
||||
.resize(IMAGE_VARIANTS.medium.width, IMAGE_VARIANTS.medium.height, {
|
||||
fit: IMAGE_VARIANTS.medium.fit,
|
||||
withoutEnlargement: true,
|
||||
})
|
||||
.webp({ quality: 85 })
|
||||
.toBuffer();
|
||||
|
||||
await this.storage.upload(mediumKey, mediumBuffer, 'image/webp');
|
||||
result.medium = mediumKey;
|
||||
}
|
||||
|
||||
// Large (only if original is larger)
|
||||
if (
|
||||
(metadata.width || 0) > IMAGE_VARIANTS.large.width ||
|
||||
(metadata.height || 0) > IMAGE_VARIANTS.large.height
|
||||
) {
|
||||
const largeKey = `${basePath}/large.webp`;
|
||||
const largeBuffer = await sharp(originalBuffer)
|
||||
.resize(IMAGE_VARIANTS.large.width, IMAGE_VARIANTS.large.height, {
|
||||
fit: IMAGE_VARIANTS.large.fit,
|
||||
withoutEnlargement: true,
|
||||
})
|
||||
.webp({ quality: 90 })
|
||||
.toBuffer();
|
||||
|
||||
await this.storage.upload(largeKey, largeBuffer, 'image/webp');
|
||||
result.large = largeKey;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
async generateThumbnail(
|
||||
buffer: Buffer,
|
||||
options?: { width?: number; height?: number }
|
||||
): Promise<Buffer> {
|
||||
const width = options?.width || IMAGE_VARIANTS.thumbnail.width;
|
||||
const height = options?.height || IMAGE_VARIANTS.thumbnail.height;
|
||||
|
||||
return sharp(buffer).resize(width, height, { fit: 'cover' }).webp({ quality: 80 }).toBuffer();
|
||||
}
|
||||
|
||||
async transformImage(
|
||||
buffer: Buffer,
|
||||
options: {
|
||||
width?: number;
|
||||
height?: number;
|
||||
fit?: 'cover' | 'contain' | 'fill' | 'inside' | 'outside';
|
||||
format?: 'webp' | 'jpeg' | 'png' | 'avif';
|
||||
quality?: number;
|
||||
}
|
||||
): Promise<Buffer> {
|
||||
let pipeline = sharp(buffer);
|
||||
|
||||
if (options.width || options.height) {
|
||||
pipeline = pipeline.resize(options.width, options.height, {
|
||||
fit: options.fit || 'inside',
|
||||
withoutEnlargement: true,
|
||||
});
|
||||
}
|
||||
|
||||
const format = options.format || 'webp';
|
||||
const quality = options.quality || 85;
|
||||
|
||||
switch (format) {
|
||||
case 'webp':
|
||||
pipeline = pipeline.webp({ quality });
|
||||
break;
|
||||
case 'jpeg':
|
||||
pipeline = pipeline.jpeg({ quality });
|
||||
break;
|
||||
case 'png':
|
||||
pipeline = pipeline.png();
|
||||
break;
|
||||
case 'avif':
|
||||
pipeline = pipeline.avif({ quality });
|
||||
break;
|
||||
}
|
||||
|
||||
return pipeline.toBuffer();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Job } from 'bullmq';
|
||||
import { ProcessService } from './process.service';
|
||||
import { UploadService } from '../upload/upload.service';
|
||||
import { PROCESS_QUEUE, SUPPORTED_IMAGE_TYPES } from './process.constants';
|
||||
|
||||
interface ProcessJobData {
|
||||
mediaId: string;
|
||||
mimeType: string;
|
||||
originalKey: string;
|
||||
}
|
||||
|
||||
@Processor(PROCESS_QUEUE)
|
||||
export class ProcessWorker extends WorkerHost {
|
||||
constructor(
|
||||
private processService: ProcessService,
|
||||
private uploadService: UploadService
|
||||
) {
|
||||
super();
|
||||
}
|
||||
|
||||
async process(job: Job<ProcessJobData>): Promise<void> {
|
||||
const { mediaId, mimeType, originalKey } = job.data;
|
||||
|
||||
console.log(`Processing media ${mediaId} (${mimeType})`);
|
||||
|
||||
try {
|
||||
if (SUPPORTED_IMAGE_TYPES.includes(mimeType)) {
|
||||
await this.processImage(mediaId, originalKey, mimeType);
|
||||
} else {
|
||||
// For unsupported types, just mark as ready
|
||||
await this.uploadService.update(mediaId, { status: 'ready' });
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Failed to process media ${mediaId}:`, error);
|
||||
await this.uploadService.update(mediaId, { status: 'failed' });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private async processImage(
|
||||
mediaId: string,
|
||||
originalKey: string,
|
||||
mimeType: string
|
||||
): Promise<void> {
|
||||
const result = await this.processService.processImage(mediaId, originalKey, mimeType);
|
||||
|
||||
await this.uploadService.update(mediaId, {
|
||||
status: 'ready',
|
||||
keys: {
|
||||
original: originalKey,
|
||||
thumbnail: result.thumbnail,
|
||||
medium: result.medium,
|
||||
large: result.large,
|
||||
},
|
||||
metadata: result.metadata,
|
||||
});
|
||||
|
||||
console.log(
|
||||
`Processed image ${mediaId}: thumbnail=${!!result.thumbnail}, medium=${!!result.medium}, large=${!!result.large}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
import { Module, Global } from '@nestjs/common';
|
||||
import { ConfigModule } from '@nestjs/config';
|
||||
import { StorageService } from './storage.service';
|
||||
|
||||
@Global()
|
||||
@Module({
|
||||
imports: [ConfigModule],
|
||||
providers: [StorageService],
|
||||
exports: [StorageService],
|
||||
})
|
||||
export class StorageModule {}
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
import { Injectable, OnModuleInit } from '@nestjs/common';
|
||||
import { ConfigService } from '@nestjs/config';
|
||||
import * as Minio from 'minio';
|
||||
import { Readable } from 'stream';
|
||||
|
||||
export interface StorageObject {
|
||||
key: string;
|
||||
bucket: string;
|
||||
size: number;
|
||||
contentType: string;
|
||||
etag: string;
|
||||
}
|
||||
|
||||
@Injectable()
|
||||
export class StorageService implements OnModuleInit {
|
||||
private client: Minio.Client;
|
||||
private bucket: string;
|
||||
|
||||
constructor(private config: ConfigService) {
|
||||
this.client = new Minio.Client({
|
||||
endPoint: this.config.get('S3_ENDPOINT', 'localhost'),
|
||||
port: parseInt(this.config.get('S3_PORT', '9000')),
|
||||
useSSL: this.config.get('S3_USE_SSL', 'false') === 'true',
|
||||
accessKey: this.config.get('S3_ACCESS_KEY', 'minioadmin'),
|
||||
secretKey: this.config.get('S3_SECRET_KEY', 'minioadmin'),
|
||||
});
|
||||
this.bucket = this.config.get('S3_BUCKET', 'mana-media');
|
||||
}
|
||||
|
||||
async onModuleInit() {
|
||||
const exists = await this.client.bucketExists(this.bucket);
|
||||
if (!exists) {
|
||||
await this.client.makeBucket(this.bucket);
|
||||
console.log(`Created bucket: ${this.bucket}`);
|
||||
}
|
||||
}
|
||||
|
||||
async upload(
|
||||
key: string,
|
||||
data: Buffer | Readable,
|
||||
contentType: string,
|
||||
metadata?: Record<string, string>
|
||||
): Promise<StorageObject> {
|
||||
const size = Buffer.isBuffer(data) ? data.length : undefined;
|
||||
|
||||
await this.client.putObject(this.bucket, key, data, size, {
|
||||
'Content-Type': contentType,
|
||||
...metadata,
|
||||
});
|
||||
|
||||
const stat = await this.client.statObject(this.bucket, key);
|
||||
|
||||
return {
|
||||
key,
|
||||
bucket: this.bucket,
|
||||
size: stat.size,
|
||||
contentType,
|
||||
etag: stat.etag,
|
||||
};
|
||||
}
|
||||
|
||||
async download(key: string): Promise<Buffer> {
|
||||
const stream = await this.client.getObject(this.bucket, key);
|
||||
const chunks: Buffer[] = [];
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
stream.on('data', (chunk) => chunks.push(chunk));
|
||||
stream.on('end', () => resolve(Buffer.concat(chunks)));
|
||||
stream.on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
async getStream(key: string): Promise<Readable> {
|
||||
return this.client.getObject(this.bucket, key);
|
||||
}
|
||||
|
||||
async delete(key: string): Promise<void> {
|
||||
await this.client.removeObject(this.bucket, key);
|
||||
}
|
||||
|
||||
async exists(key: string): Promise<boolean> {
|
||||
try {
|
||||
await this.client.statObject(this.bucket, key);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async getPresignedUrl(key: string, expiresIn = 3600): Promise<string> {
|
||||
return this.client.presignedGetObject(this.bucket, key, expiresIn);
|
||||
}
|
||||
|
||||
async getUploadUrl(key: string, expiresIn = 3600): Promise<string> {
|
||||
return this.client.presignedPutObject(this.bucket, key, expiresIn);
|
||||
}
|
||||
|
||||
getPublicUrl(key: string): string {
|
||||
const endpoint = this.config.get('S3_PUBLIC_URL', `http://localhost:9000/${this.bucket}`);
|
||||
return `${endpoint}/${key}`;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
import {
|
||||
Controller,
|
||||
Post,
|
||||
Get,
|
||||
Delete,
|
||||
Param,
|
||||
Query,
|
||||
Body,
|
||||
UploadedFile,
|
||||
UseInterceptors,
|
||||
BadRequestException,
|
||||
NotFoundException,
|
||||
} from '@nestjs/common';
|
||||
import { FileInterceptor } from '@nestjs/platform-express';
|
||||
import { UploadService, MediaRecord } from './upload.service';
|
||||
|
||||
interface UploadResponse {
|
||||
id: string;
|
||||
status: MediaRecord['status'];
|
||||
originalName: string;
|
||||
mimeType: string;
|
||||
size: number;
|
||||
urls: {
|
||||
original: string;
|
||||
thumbnail?: string;
|
||||
medium?: string;
|
||||
large?: string;
|
||||
};
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
@Controller('media')
|
||||
export class UploadController {
|
||||
constructor(private uploadService: UploadService) {}
|
||||
|
||||
@Post('upload')
|
||||
@UseInterceptors(
|
||||
FileInterceptor('file', {
|
||||
limits: {
|
||||
fileSize: 100 * 1024 * 1024, // 100 MB
|
||||
},
|
||||
})
|
||||
)
|
||||
async upload(
|
||||
@UploadedFile() file: Express.Multer.File,
|
||||
@Body('app') app?: string,
|
||||
@Body('userId') userId?: string,
|
||||
@Body('skipProcessing') skipProcessing?: string
|
||||
): Promise<UploadResponse> {
|
||||
if (!file) {
|
||||
throw new BadRequestException('No file provided');
|
||||
}
|
||||
|
||||
const record = await this.uploadService.upload(file, {
|
||||
app,
|
||||
userId,
|
||||
skipProcessing: skipProcessing === 'true',
|
||||
});
|
||||
|
||||
return this.toResponse(record);
|
||||
}
|
||||
|
||||
@Get(':id')
|
||||
async get(@Param('id') id: string): Promise<UploadResponse> {
|
||||
const record = await this.uploadService.get(id);
|
||||
if (!record) {
|
||||
throw new NotFoundException('Media not found');
|
||||
}
|
||||
return this.toResponse(record);
|
||||
}
|
||||
|
||||
@Get()
|
||||
async list(
|
||||
@Query('app') app?: string,
|
||||
@Query('userId') userId?: string,
|
||||
@Query('limit') limit?: string
|
||||
): Promise<UploadResponse[]> {
|
||||
const records = await this.uploadService.list({
|
||||
app,
|
||||
userId,
|
||||
limit: limit ? parseInt(limit) : 50,
|
||||
});
|
||||
return records.map((r) => this.toResponse(r));
|
||||
}
|
||||
|
||||
@Delete(':id')
|
||||
async delete(@Param('id') id: string): Promise<{ success: boolean }> {
|
||||
const deleted = await this.uploadService.delete(id);
|
||||
if (!deleted) {
|
||||
throw new NotFoundException('Media not found');
|
||||
}
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
private toResponse(record: MediaRecord): UploadResponse {
|
||||
const baseUrl = process.env.PUBLIC_URL || 'http://localhost:3050/api/v1';
|
||||
|
||||
return {
|
||||
id: record.id,
|
||||
status: record.status,
|
||||
originalName: record.originalName,
|
||||
mimeType: record.mimeType,
|
||||
size: record.size,
|
||||
urls: {
|
||||
original: `${baseUrl}/media/${record.id}/file`,
|
||||
thumbnail: record.keys.thumbnail ? `${baseUrl}/media/${record.id}/file/thumb` : undefined,
|
||||
medium: record.keys.medium ? `${baseUrl}/media/${record.id}/file/medium` : undefined,
|
||||
large: record.keys.large ? `${baseUrl}/media/${record.id}/file/large` : undefined,
|
||||
},
|
||||
createdAt: record.createdAt,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
import { Module } from '@nestjs/common';
|
||||
import { BullModule } from '@nestjs/bullmq';
|
||||
import { UploadController } from './upload.controller';
|
||||
import { UploadService } from './upload.service';
|
||||
import { PROCESS_QUEUE } from '../process/process.constants';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
BullModule.registerQueue({
|
||||
name: PROCESS_QUEUE,
|
||||
}),
|
||||
],
|
||||
controllers: [UploadController],
|
||||
providers: [UploadService],
|
||||
exports: [UploadService],
|
||||
})
|
||||
export class UploadModule {}
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
import { Injectable } from '@nestjs/common';
|
||||
import { InjectQueue } from '@nestjs/bullmq';
|
||||
import { Queue } from 'bullmq';
|
||||
import { v4 as uuid } from 'uuid';
|
||||
import * as mime from 'mime-types';
|
||||
import * as crypto from 'crypto';
|
||||
import { StorageService } from '../storage/storage.service';
|
||||
import { PROCESS_QUEUE } from '../process/process.constants';
|
||||
|
||||
export interface MediaRecord {
|
||||
id: string;
|
||||
originalName: string;
|
||||
mimeType: string;
|
||||
size: number;
|
||||
hash: string;
|
||||
status: 'uploading' | 'processing' | 'ready' | 'failed';
|
||||
app?: string;
|
||||
userId?: string;
|
||||
keys: {
|
||||
original: string;
|
||||
thumbnail?: string;
|
||||
medium?: string;
|
||||
large?: string;
|
||||
};
|
||||
metadata?: Record<string, unknown>;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
// In-memory store for MVP (replace with DB later)
|
||||
const mediaStore = new Map<string, MediaRecord>();
|
||||
|
||||
@Injectable()
|
||||
export class UploadService {
|
||||
constructor(
|
||||
private storage: StorageService,
|
||||
@InjectQueue(PROCESS_QUEUE) private processQueue: Queue
|
||||
) {}
|
||||
|
||||
async upload(
|
||||
file: Express.Multer.File,
|
||||
options?: {
|
||||
app?: string;
|
||||
userId?: string;
|
||||
skipProcessing?: boolean;
|
||||
}
|
||||
): Promise<MediaRecord> {
|
||||
const id = uuid();
|
||||
const ext = mime.extension(file.mimetype) || 'bin';
|
||||
const hash = this.computeHash(file.buffer);
|
||||
|
||||
// Check for duplicate
|
||||
const existing = this.findByHash(hash);
|
||||
if (existing) {
|
||||
return existing;
|
||||
}
|
||||
|
||||
// Generate storage keys
|
||||
const date = new Date();
|
||||
const datePath = `${date.getFullYear()}/${String(date.getMonth() + 1).padStart(2, '0')}/${String(date.getDate()).padStart(2, '0')}`;
|
||||
const originalKey = `originals/${datePath}/${id}.${ext}`;
|
||||
|
||||
// Upload original
|
||||
await this.storage.upload(originalKey, file.buffer, file.mimetype, {
|
||||
'x-amz-meta-original-name': file.originalname,
|
||||
'x-amz-meta-media-id': id,
|
||||
});
|
||||
|
||||
// Create record
|
||||
const record: MediaRecord = {
|
||||
id,
|
||||
originalName: file.originalname,
|
||||
mimeType: file.mimetype,
|
||||
size: file.size,
|
||||
hash,
|
||||
status: options?.skipProcessing ? 'ready' : 'processing',
|
||||
app: options?.app,
|
||||
userId: options?.userId,
|
||||
keys: {
|
||||
original: originalKey,
|
||||
},
|
||||
createdAt: date,
|
||||
updatedAt: date,
|
||||
};
|
||||
|
||||
mediaStore.set(id, record);
|
||||
|
||||
// Queue processing job
|
||||
if (!options?.skipProcessing) {
|
||||
await this.processQueue.add('process-media', {
|
||||
mediaId: id,
|
||||
mimeType: file.mimetype,
|
||||
originalKey,
|
||||
});
|
||||
}
|
||||
|
||||
return record;
|
||||
}
|
||||
|
||||
async get(id: string): Promise<MediaRecord | null> {
|
||||
return mediaStore.get(id) || null;
|
||||
}
|
||||
|
||||
async update(id: string, updates: Partial<MediaRecord>): Promise<MediaRecord | null> {
|
||||
const record = mediaStore.get(id);
|
||||
if (!record) return null;
|
||||
|
||||
const updated = {
|
||||
...record,
|
||||
...updates,
|
||||
updatedAt: new Date(),
|
||||
};
|
||||
mediaStore.set(id, updated);
|
||||
return updated;
|
||||
}
|
||||
|
||||
async delete(id: string): Promise<boolean> {
|
||||
const record = mediaStore.get(id);
|
||||
if (!record) return false;
|
||||
|
||||
// Delete all associated files
|
||||
const keys = Object.values(record.keys).filter(Boolean) as string[];
|
||||
for (const key of keys) {
|
||||
await this.storage.delete(key).catch(() => {});
|
||||
}
|
||||
|
||||
mediaStore.delete(id);
|
||||
return true;
|
||||
}
|
||||
|
||||
async list(options?: { app?: string; userId?: string; limit?: number }): Promise<MediaRecord[]> {
|
||||
let records = Array.from(mediaStore.values());
|
||||
|
||||
if (options?.app) {
|
||||
records = records.filter((r) => r.app === options.app);
|
||||
}
|
||||
if (options?.userId) {
|
||||
records = records.filter((r) => r.userId === options.userId);
|
||||
}
|
||||
|
||||
records.sort((a, b) => b.createdAt.getTime() - a.createdAt.getTime());
|
||||
|
||||
if (options?.limit) {
|
||||
records = records.slice(0, options.limit);
|
||||
}
|
||||
|
||||
return records;
|
||||
}
|
||||
|
||||
private computeHash(buffer: Buffer): string {
|
||||
return crypto.createHash('sha256').update(buffer).digest('hex');
|
||||
}
|
||||
|
||||
private findByHash(hash: string): MediaRecord | undefined {
|
||||
return Array.from(mediaStore.values()).find((r) => r.hash === hash);
|
||||
}
|
||||
}
|
||||
22
services/mana-media/apps/api/tsconfig.json
Normal file
22
services/mana-media/apps/api/tsconfig.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"module": "commonjs",
|
||||
"declaration": true,
|
||||
"removeComments": true,
|
||||
"emitDecoratorMetadata": true,
|
||||
"experimentalDecorators": true,
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"target": "ES2022",
|
||||
"sourceMap": true,
|
||||
"outDir": "./dist",
|
||||
"baseUrl": "./",
|
||||
"incremental": true,
|
||||
"skipLibCheck": true,
|
||||
"strictNullChecks": true,
|
||||
"noImplicitAny": true,
|
||||
"strictBindCallApply": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"esModuleInterop": true
|
||||
}
|
||||
}
|
||||
63
services/mana-media/docker-compose.yml
Normal file
63
services/mana-media/docker-compose.yml
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
services:
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: apps/api/Dockerfile
|
||||
container_name: mana-media-api
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3050:3050"
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- PORT=3050
|
||||
- REDIS_HOST=redis
|
||||
- REDIS_PORT=6379
|
||||
- S3_ENDPOINT=minio
|
||||
- S3_PORT=9000
|
||||
- S3_USE_SSL=false
|
||||
- S3_ACCESS_KEY=${S3_ACCESS_KEY:-minioadmin}
|
||||
- S3_SECRET_KEY=${S3_SECRET_KEY:-minioadmin}
|
||||
- S3_BUCKET=mana-media
|
||||
depends_on:
|
||||
- redis
|
||||
- minio
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:3050/api/v1/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: mana-media-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: mana-media-minio
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
- MINIO_ROOT_USER=${S3_ACCESS_KEY:-minioadmin}
|
||||
- MINIO_ROOT_PASSWORD=${S3_SECRET_KEY:-minioadmin}
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
ports:
|
||||
- "9010:9000"
|
||||
- "9011:9001"
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 30s
|
||||
timeout: 20s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
minio_data:
|
||||
11
services/mana-media/package.json
Normal file
11
services/mana-media/package.json
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"name": "@mana-media/root",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "pnpm --filter @mana-media/api dev",
|
||||
"build": "pnpm --filter @mana-media/api build",
|
||||
"start": "pnpm --filter @mana-media/api start:prod",
|
||||
"type-check": "pnpm -r type-check",
|
||||
"lint": "pnpm -r lint"
|
||||
}
|
||||
}
|
||||
15
services/mana-media/packages/client/package.json
Normal file
15
services/mana-media/packages/client/package.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"name": "@manacore/media-client",
|
||||
"version": "0.1.0",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"type-check": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.7.0",
|
||||
"@types/node": "^22.0.0"
|
||||
}
|
||||
}
|
||||
39
services/mana-media/packages/client/pnpm-lock.yaml
generated
Normal file
39
services/mana-media/packages/client/pnpm-lock.yaml
generated
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
lockfileVersion: '9.0'
|
||||
|
||||
settings:
|
||||
autoInstallPeers: true
|
||||
excludeLinksFromLockfile: false
|
||||
|
||||
importers:
|
||||
|
||||
.:
|
||||
devDependencies:
|
||||
'@types/node':
|
||||
specifier: ^22.0.0
|
||||
version: 22.19.7
|
||||
typescript:
|
||||
specifier: ^5.7.0
|
||||
version: 5.9.3
|
||||
|
||||
packages:
|
||||
|
||||
'@types/node@22.19.7':
|
||||
resolution: {integrity: sha512-MciR4AKGHWl7xwxkBa6xUGxQJ4VBOmPTF7sL+iGzuahOFaO0jHCsuEfS80pan1ef4gWId1oWOweIhrDEYLuaOw==}
|
||||
|
||||
typescript@5.9.3:
|
||||
resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
|
||||
engines: {node: '>=14.17'}
|
||||
hasBin: true
|
||||
|
||||
undici-types@6.21.0:
|
||||
resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
|
||||
|
||||
snapshots:
|
||||
|
||||
'@types/node@22.19.7':
|
||||
dependencies:
|
||||
undici-types: 6.21.0
|
||||
|
||||
typescript@5.9.3: {}
|
||||
|
||||
undici-types@6.21.0: {}
|
||||
203
services/mana-media/packages/client/src/index.ts
Normal file
203
services/mana-media/packages/client/src/index.ts
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
export interface MediaResult {
|
||||
id: string;
|
||||
status: 'uploading' | 'processing' | 'ready' | 'failed';
|
||||
originalName: string;
|
||||
mimeType: string;
|
||||
size: number;
|
||||
urls: {
|
||||
original: string;
|
||||
thumbnail?: string;
|
||||
medium?: string;
|
||||
large?: string;
|
||||
};
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface UploadOptions {
|
||||
app?: string;
|
||||
userId?: string;
|
||||
skipProcessing?: boolean;
|
||||
}
|
||||
|
||||
export interface SearchOptions {
|
||||
type?: 'image' | 'video' | 'audio' | 'document';
|
||||
app?: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface TransformOptions {
|
||||
width?: number;
|
||||
height?: number;
|
||||
fit?: 'cover' | 'contain' | 'fill' | 'inside' | 'outside';
|
||||
format?: 'webp' | 'jpeg' | 'png' | 'avif';
|
||||
quality?: number;
|
||||
}
|
||||
|
||||
export class MediaClient {
|
||||
private baseUrl: string;
|
||||
private apiKey?: string;
|
||||
|
||||
constructor(baseUrl: string, apiKey?: string) {
|
||||
this.baseUrl = baseUrl.replace(/\/$/, '');
|
||||
this.apiKey = apiKey;
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload a file to the media service
|
||||
*/
|
||||
async upload(
|
||||
file: File | Blob | ArrayBuffer,
|
||||
options?: UploadOptions & { filename?: string }
|
||||
): Promise<MediaResult> {
|
||||
const formData = new FormData();
|
||||
|
||||
if (file instanceof ArrayBuffer) {
|
||||
// ArrayBuffer (works in both Node.js and browser)
|
||||
const blob = new Blob([file]);
|
||||
formData.append('file', blob, options?.filename || 'file');
|
||||
} else {
|
||||
// Browser File/Blob
|
||||
formData.append('file', file, options?.filename);
|
||||
}
|
||||
|
||||
if (options?.app) formData.append('app', options.app);
|
||||
if (options?.userId) formData.append('userId', options.userId);
|
||||
if (options?.skipProcessing) formData.append('skipProcessing', 'true');
|
||||
|
||||
const response = await fetch(`${this.baseUrl}/api/v1/media/upload`, {
|
||||
method: 'POST',
|
||||
headers: this.getHeaders(),
|
||||
body: formData,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Upload failed: ${response.statusText}`);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
}
|
||||
|
||||
/**
|
||||
* Get media by ID
|
||||
*/
|
||||
async get(id: string): Promise<MediaResult> {
|
||||
const response = await fetch(`${this.baseUrl}/api/v1/media/${id}`, {
|
||||
headers: this.getHeaders(),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Get failed: ${response.statusText}`);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
}
|
||||
|
||||
/**
|
||||
* List media
|
||||
*/
|
||||
async list(options?: { app?: string; userId?: string; limit?: number }): Promise<MediaResult[]> {
|
||||
const params = new URLSearchParams();
|
||||
if (options?.app) params.append('app', options.app);
|
||||
if (options?.userId) params.append('userId', options.userId);
|
||||
if (options?.limit) params.append('limit', options.limit.toString());
|
||||
|
||||
const response = await fetch(`${this.baseUrl}/api/v1/media?${params}`, {
|
||||
headers: this.getHeaders(),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`List failed: ${response.statusText}`);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete media
|
||||
*/
|
||||
async delete(id: string): Promise<boolean> {
|
||||
const response = await fetch(`${this.baseUrl}/api/v1/media/${id}`, {
|
||||
method: 'DELETE',
|
||||
headers: this.getHeaders(),
|
||||
});
|
||||
|
||||
return response.ok;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get URL for original file
|
||||
*/
|
||||
getOriginalUrl(id: string): string {
|
||||
return `${this.baseUrl}/api/v1/media/${id}/file`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get URL for thumbnail
|
||||
*/
|
||||
getThumbnailUrl(id: string): string {
|
||||
return `${this.baseUrl}/api/v1/media/${id}/file/thumb`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get URL for medium variant
|
||||
*/
|
||||
getMediumUrl(id: string): string {
|
||||
return `${this.baseUrl}/api/v1/media/${id}/file/medium`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get URL for large variant
|
||||
*/
|
||||
getLargeUrl(id: string): string {
|
||||
return `${this.baseUrl}/api/v1/media/${id}/file/large`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get URL for custom transform
|
||||
*/
|
||||
getTransformUrl(id: string, options: TransformOptions): string {
|
||||
const params = new URLSearchParams();
|
||||
if (options.width) params.append('w', options.width.toString());
|
||||
if (options.height) params.append('h', options.height.toString());
|
||||
if (options.fit) params.append('fit', options.fit);
|
||||
if (options.format) params.append('format', options.format);
|
||||
if (options.quality) params.append('q', options.quality.toString());
|
||||
|
||||
return `${this.baseUrl}/api/v1/media/${id}/transform?${params}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for media processing to complete
|
||||
*/
|
||||
async waitForReady(id: string, timeoutMs = 30000, pollIntervalMs = 1000): Promise<MediaResult> {
|
||||
const startTime = Date.now();
|
||||
|
||||
while (Date.now() - startTime < timeoutMs) {
|
||||
const result = await this.get(id);
|
||||
|
||||
if (result.status === 'ready') {
|
||||
return result;
|
||||
}
|
||||
|
||||
if (result.status === 'failed') {
|
||||
throw new Error('Media processing failed');
|
||||
}
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
|
||||
}
|
||||
|
||||
throw new Error('Timeout waiting for media to be ready');
|
||||
}
|
||||
|
||||
private getHeaders(): Record<string, string> {
|
||||
const headers: Record<string, string> = {};
|
||||
|
||||
if (this.apiKey) {
|
||||
headers['Authorization'] = `Bearer ${this.apiKey}`;
|
||||
}
|
||||
|
||||
return headers;
|
||||
}
|
||||
}
|
||||
|
||||
export default MediaClient;
|
||||
14
services/mana-media/packages/client/tsconfig.json
Normal file
14
services/mana-media/packages/client/tsconfig.json
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "commonjs",
|
||||
"declaration": true,
|
||||
"outDir": "./dist",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue