# Vellum — how it works Vellum is a self-hosted media server. It replaces Jellyfin, Navidrome, Audiobookshelf, Stump and Kavita with one program: one Go binary, one SQLite file, one web UI. It handles ebooks, comics, audiobooks, music, photos, games and video. This document explains how the pieces fit together and, more importantly, *why* they fit that way. Where a decision had a real trade-off, there is an ADR in [`decisions/`](decisions/) with the alternatives that were rejected. **Size:** ~11,500 lines of Go (plus ~7,000 lines of tests), ~7,100 lines of frontend, 33 packages, 12 migrations, 64 routes. --- ## 1. The shape ``` one binary one file one UI ┌──────────────────┐ ┌───────────────┐ ┌──────────────┐ │ vellum (Go) │─────▶│ vellum.db │ │ Svelte app │ │ serves API+UI │ │ SQLite + WAL │ │ embedded in │ │ runs the jobs │ └───────────────┘ │ the binary │ └──────────────────┘ └──────────────┘ ``` There is no Redis, no message broker, no second process and no CGO. The SQLite driver is pure Go (`modernc.org/sqlite`), which is why `CGO_ENABLED=0` works and the container is a bare Alpine image. The job queue is a *table*. **"One binary" is a deployment choice, not an architecture.** The code is a modular monolith: 33 packages with enforced boundaries. Splitting it into services would add network failure modes and make debugging harder, not easier. See [ADR-004](decisions/ADR-004-modular-monolith.md). --- ## 2. What happens when you open the app Every request goes through the same chain, in this order: ``` browser │ ├─ withSecurityHeaders outermost, so headers survive a panic ├─ withRecovery panic → 500, never a dead connection ├─ withRequestID X-Request-ID, echoed into every log line ├─ withRequestLogging method, path, status, duration, request_id └─ auth.Middleware cookie → session → User in the request context │ └─ handler → auth.VisibleLibraryIDs / CanAccessLibrary → SQL → JSON ``` The **grant check is the security chokepoint**. Every read path goes through it, including the Jellyfin and Subsonic adapters, which is why an API token can never see more than the web app can. An item you cannot read returns **404, not 403** — a 403 would confirm the item exists and leak the shape of someone else's library. Media bytes take a different path on purpose: `internal/stream` hands off to `http.ServeContent`, which implements HTTP Range correctly. That is why seeking works in every player, and it is why compression is scoped to the static UI only (compressing a Range response would break seeking). --- ## 3. The layers, and the rules that hold them The code is arranged in six layers. A package may import from a lower layer or its own, never from a higher one. | Layer | Packages | What lives here | |---|---|---| | 0 | `core`, `httpx`, `natsort`, `arch`, `migrations` | foundation; knows nothing about the app | | 1 | `auth`, `jobs`, `stream`, `progress`, `probe`, `engine`, `apitoken` | primitives everything is built from | | 2 | `catalog`, `providers` | the data seam, and third-party API clients | | 3 | `library`, `scanner`, `media`, `home`, `search`, `metadata`, `playback`, `modules/*`, `*/dto` | the application | | 4 | `httpapi`, `jellyfin`, `subsonic`, `cli` | adapters: HTTP and foreign protocols | | 5 | `cmd/vellum` | the only place everything is wired together | **These rules are enforced by tests, not by documentation.** `internal/arch` parses every package's imports and fails the build on a violation. There are seven rules: 1. No upward imports across the layers. 2. `internal/core` may depend only on `migrations`. 3. `internal/catalog` must not know about HTTP — or it stops being reusable by a job or a test. 4. Modules must not import each other. 5. **Every package must be placed in a layer**, so the map cannot go stale. 6. The set of packages allowed to import `database/sql` is explicit. 7. Packages already moved onto the `catalog.Store` seam must not reach for `*sql.DB` again. Each of the seven has been deliberately broken and confirmed to fail. A test that cannot fail is decoration. --- ## 4. The data model — the decision everything follows from ``` users ──< library_grants >── libraries │ │ │ └──< media_items ──< media_files │ │ └──< progress (user, item, file) ───────┘ └──< item_views, favorites, api_tokens ``` **Media items are global. Every per-user thing is a join.** The `progress` primary key is `(user_id, item_id, file_id)`. That one choice is why audiobooks needed no new Go module: a twelve-chapter book is one item with twelve files, and "chapter 7, 4 minutes 32 seconds in" is a single row. It is also why the Continue shelf is one query, and why per-user state can never contaminate the shared catalogue. **Deletion is deliberately not deletion.** A file that vanishes is marked `missing = 1` with a `missing_since` timestamp, never removed. Deleting cascades into `progress`, the one thing in the schema that cannot be regenerated — so an unmounted drive would silently wipe your reading history. There is an explicit admin purge for when you really mean it. See [ADR-003](decisions/ADR-003-mark-missing-never-delete.md). --- ## 5. How the library gets in `internal/scanner` walks the library root and reconciles it with the database. ``` walkRoot → files on disk, keyed by relative path buildGroups → one item per file, OR one item per directory batched upsert → 500 groups per transaction missing pass → mark vanished files, never delete ``` Three things worth knowing: - **Transactions are chunked.** It used to hold SQLite's single write lock for the whole library. Now each transaction covers 500 groups, so a progress save from a playing client succeeds *during* a scan. Measured at 20,000 files with concurrent writes all returning 200. - **Change detection is `size` + `mtime`.** Matching files are skipped entirely. `ffprobe` runs only for changed files, and before the write transaction opens, so a slow probe can never hold the write lock. - **Two scans can run at once.** The job queue has four workers, so scanning two libraries concurrently works — the *writes* interleave at batch granularity rather than one library blocking the other. --- ## 6. The job queue `internal/jobs` is a queue implemented as a database table. Claiming a job is a single atomic statement, so several workers are safe with no extra locking: ```sql UPDATE jobs SET state = 'running', attempts = attempts + 1 WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ? ORDER BY run_after, id LIMIT 1) RETURNING id, kind, payload, attempts ``` Four workers with jittered idle waits. A panicking handler is recovered and the job marked failed, so one bad handler cannot kill a worker or strand a row in `running`. `RecoverStale()` runs at boot and requeues anything a dead process left behind — without it, a crash mid-scan left a job `running` forever while the UI polled it forever. Two job kinds: `scan_library` and `enrich_item`. --- ## 7. Playback — direct play only **Vellum never transcodes.** The media server is a headless, GPU-less Xeon with no spare CPU for it. A file either plays in the client or it does not. This is the single most important thing to understand about the playback code, and it came from measurement rather than assumption: | container | video | result | |---|---|---| | mp4 | h264 | plays | | **mkv** | h264 | **plays** — the container is not the problem | | mov | h264 | plays, even though `canPlayType` says "no" | | mp4 | **hevc** | **audio only** — a black picture, no error | | avi | mpeg4 | fails outright | Two conclusions drive the design: 1. **Codec matters, container barely does.** mkv+h264 plays; mp4+HEVC does not. Judging a file by its extension is wrong. 2. **Playability is a property of the CLIENT, not the file.** Chrome cannot decode HEVC; Safari can; Jellyfin's native apps handle far more than either. So the server **never** declares a file playable. It reports the codec facts (from `ffprobe`, stored at scan time) and each client decides. See [ADR-002](decisions/ADR-002-direct-play-only.md). That is what makes the engine registry work. `internal/engine` holds engines that declare what they can play — `html5` ships built in; `mpv`, `vlc` and `mdk` are registered for native clients. Adding one is a declaration, not a rewrite: ``` POST /api/v1/playback/resolve {"item_id":16,"engines":["html5"]} → playable=false, reason="vcodec:hevc" POST /api/v1/playback/resolve {"item_id":16,"engines":["html5","mpv"]} → playable=true, engine="mpv" ``` Same file, different answer, because the client offered a different engine. **The element is always the final authority.** `canPlayType` lies in both directions, so the player uses it only as a hint and confirms by actually loading the file and checking `videoWidth`. For HEVC, Chrome plays audio over a blank frame with no error — the worst possible outcome, and the player detects it and says so rather than pretending. --- ## 8. Three protocol surfaces over one application | surface | who uses it | how it authenticates | |---|---|---| | `/api/v1/*` | the web app | session cookie | | Jellyfin (`/Items`, `/Users/…`) | Jellyfin native apps | `X-Emby-Token`, `?api_key=` | | Subsonic (`/rest/*.view`) | Symfonium, Ultrasonic, play:Sub | `t=md5(key+salt)` | All three resolve to the same `auth.User` and pass through the same grant check. `internal/apitoken` is the shared credential layer, and its migration documents a real asymmetry: Jellyfin tokens are stored **hashed** (like session tokens), while a Subsonic key must be stored **in the clear**, because verifying `t` means computing `md5(key + salt)` server-side. Both adapters are honest about the one thing Vellum cannot do: `PlaybackInfo` reports `SupportsTranscoding: false`, and a client that demands a transcode gets `NoCompatibleStream` rather than a URL that would never resolve. --- ## 9. Where things live ``` cmd/vellum wiring and lifecycle — the only place packages meet internal/ core config, database open, migrations, logger, timestamps httpx small HTTP helpers (JSON, errors) auth users, sessions, argon2id hashing, grants apitoken credentials for non-browser clients catalog THE data seam: read an item, and check you may providers Open Library, Google Books, MusicBrainz, ComicVine, IGDB library library definitions, media-type registry scanner walk the filesystem and reconcile probe ffprobe wrapper (optional dependency) media item, file, cover, progress endpoints home the five home shelves search FTS5 search over titles, paths and authors metadata the enrichment job and its matching policy playback engine selection endpoint engine playback engine registry jobs table-backed job queue progress per-user playback position stream Range-capable file serving modules/ comics, ebooks, games, photos httpapi mux, middleware, security headers, embedded UI jellyfin Jellyfin protocol adapter subsonic Subsonic protocol adapter arch import-rule tests (no code) migrations/ numbered SQL, append-only, tracked by user_version web/ Svelte 5 + Vite frontend ``` --- ## 10. Things to know before changing anything - **`cmd/vellum/main.go` is the only wiring point.** Nothing else should import a module; modules register themselves. - **Migrations are append-only.** Never edit an applied file. The FTS5 authors migration is the cautionary tale: it worked on a fresh database and raised *"database disk image is malformed"* on an upgrade. - **All timestamps go through `core.Stamp`.** `RFC3339Nano` trims trailing zeros, so its strings are not lexicographically sortable — `".5Z" > ".55Z"` while `.500s < .550s`. That silently broke `ORDER BY` and `MAX()` on the Continue shelf. - **Progress is PUT-only**, so `sendBeacon` (POST-only) cannot be used; the teardown flush uses `fetch` with `keepalive: true`. - **`canPlayType` is not a source of truth.** Put `src` on the media element, not `type` on a `` — an explicit type makes Chrome *skip* files it would otherwise play. - **If you need an item, take a `catalog.Store`.** Not `*sql.DB`. The seam is enforced by test rule 6/7. - **Run `go test ./...` after touching a package's imports** — the arch rules will tell you immediately if you crossed a layer. --- ## 11. Commands | Command | What it does | |---|---| | `make dev` | run backend and Vite together | | `go test ./...` | all Go tests, including the arch rules | | `go vet ./...` | static checks | | `cd web && pnpm check` | Svelte type-check | | `vellum user add [--admin]` | create a user | | `vellum user passwd ` | change a password | | `vellum user list` | list users | | `vellum backup ` | safe copy of the database (`VACUUM INTO`) | See [DEPLOY.md](../DEPLOY.md) for deployment, and [`decisions/`](decisions/) for why things are the way they are.