Vellum/docs/ARCHITECTURE.md

Vellum — how it works

Vellum is a self-hosted media server. It replaces Jellyfin, Navidrome, Audiobookshelf, Stump and Kavita with one program: one Go binary, one SQLite file, one web UI. It handles ebooks, comics, audiobooks, music, photos, games and video.

This document explains how the pieces fit together and, more importantly, why they fit that way. Where a decision had a real trade-off, there is an ADR in decisions/ with the alternatives that were rejected.

Size: ~11,500 lines of Go (plus ~7,000 lines of tests), ~7,100 lines of frontend, 33 packages, 12 migrations, 64 routes.

---

1. The shape

        one binary                 one file              one UI
   ┌──────────────────┐      ┌───────────────┐      ┌──────────────┐
   │  vellum (Go)     │─────▶│ vellum.db     │      │ Svelte app   │
   │  serves API+UI   │      │ SQLite + WAL  │      │ embedded in  │
   │  runs the jobs   │      └───────────────┘      │ the binary   │
   └──────────────────┘                             └──────────────┘

There is no Redis, no message broker, no second process and no CGO. The SQLite driver is pure Go (modernc.org/sqlite), which is why CGO_ENABLED=0 works and the container is a bare Alpine image. The job queue is a table.

"One binary" is a deployment choice, not an architecture. The code is a modular monolith: 33 packages with enforced boundaries. Splitting it into services would add network failure modes and make debugging harder, not easier. See ADR-004.

---

2. What happens when you open the app

Every request goes through the same chain, in this order:

browser
  │
  ├─ withSecurityHeaders   outermost, so headers survive a panic
  ├─ withRecovery          panic → 500, never a dead connection
  ├─ withRequestID         X-Request-ID, echoed into every log line
  ├─ withRequestLogging    method, path, status, duration, request_id
  └─ auth.Middleware       cookie → session → User in the request context
       │
       └─ handler  →  auth.VisibleLibraryIDs / CanAccessLibrary  →  SQL  →  JSON

The grant check is the security chokepoint. Every read path goes through it, including the Jellyfin and Subsonic adapters, which is why an API token can never see more than the web app can. An item you cannot read returns 404, not 403 — a 403 would confirm the item exists and leak the shape of someone else's library.

Media bytes take a different path on purpose: internal/stream hands off to http.ServeContent, which implements HTTP Range correctly. That is why seeking works in every player, and it is why compression is scoped to the static UI only (compressing a Range response would break seeking).

---

3. The layers, and the rules that hold them

The code is arranged in six layers. A package may import from a lower layer or its own, never from a higher one.

Layer	Packages	What lives here
0	core, httpx, natsort, arch, migrations	foundation; knows nothing about the app
1	auth, jobs, stream, progress, probe, engine, apitoken	primitives everything is built from
2	catalog, providers	the data seam, and third-party API clients
3	library, scanner, media, home, search, metadata, playback, modules/*, */dto	the application
4	httpapi, jellyfin, subsonic, cli	adapters: HTTP and foreign protocols
5	cmd/vellum	the only place everything is wired together

These rules are enforced by tests, not by documentation. internal/arch parses every package's imports and fails the build on a violation. There are seven rules:

1. No upward imports across the layers.
2. internal/core may depend only on migrations.
3. internal/catalog must not know about HTTP — or it stops being reusable by a job or a test.
4. Modules must not import each other.
5. Every package must be placed in a layer, so the map cannot go stale.
6. The set of packages allowed to import database/sql is explicit.
7. Packages already moved onto the catalog.Store seam must not reach for *sql.DB again.

Each of the seven has been deliberately broken and confirmed to fail. A test that cannot fail is decoration.

---

4. The data model — the decision everything follows from

users ──< library_grants >── libraries
  │                              │
  │                              └──< media_items ──< media_files
  │                                       │
  └──< progress (user, item, file) ───────┘
  └──< item_views, favorites, api_tokens

Media items are global. Every per-user thing is a join.

The progress primary key is (user_id, item_id, file_id). That one choice is why audiobooks needed no new Go module: a twelve-chapter book is one item with twelve files, and "chapter 7, 4 minutes 32 seconds in" is a single row. It is also why the Continue shelf is one query, and why per-user state can never contaminate the shared catalogue.

Deletion is deliberately not deletion. A file that vanishes is marked missing = 1 with a missing_since timestamp, never removed. Deleting cascades into progress, the one thing in the schema that cannot be regenerated — so an unmounted drive would silently wipe your reading history. There is an explicit admin purge for when you really mean it. See ADR-003.

---

5. How the library gets in

internal/scanner walks the library root and reconciles it with the database.

walkRoot        →  files on disk, keyed by relative path
buildGroups     →  one item per file, OR one item per directory
batched upsert  →  500 groups per transaction
missing pass    →  mark vanished files, never delete

Three things worth knowing:

• Transactions are chunked. It used to hold SQLite's single write lock for the whole library. Now each transaction covers 500 groups, so a progress save from a playing client succeeds during a scan. Measured at 20,000 files with concurrent writes all returning 200.
• Change detection is size + mtime. Matching files are skipped entirely. ffprobe runs only for changed files, and before the write transaction opens, so a slow probe can never hold the write lock.
• Two scans can run at once. The job queue has four workers, so scanning two libraries concurrently works — the writes interleave at batch granularity rather than one library blocking the other.

---

6. The job queue

internal/jobs is a queue implemented as a database table. Claiming a job is a single atomic statement, so several workers are safe with no extra locking:

UPDATE jobs SET state = 'running', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
            ORDER BY run_after, id LIMIT 1)
RETURNING id, kind, payload, attempts

Four workers with jittered idle waits. A panicking handler is recovered and the job marked failed, so one bad handler cannot kill a worker or strand a row in running. RecoverStale() runs at boot and requeues anything a dead process left behind — without it, a crash mid-scan left a job running forever while the UI polled it forever.

Two job kinds: scan_library and enrich_item.

---

7. Playback — direct play only

Vellum never transcodes. The media server is a headless, GPU-less Xeon with no spare CPU for it. A file either plays in the client or it does not.

This is the single most important thing to understand about the playback code, and it came from measurement rather than assumption:

container	video	result
mp4	h264	plays
mkv	h264	plays — the container is not the problem
mov	h264	plays, even though canPlayType says "no"
mp4	hevc	audio only — a black picture, no error
avi	mpeg4	fails outright

Two conclusions drive the design:

1. Codec matters, container barely does. mkv+h264 plays; mp4+HEVC does not. Judging a file by its extension is wrong.
2. Playability is a property of the CLIENT, not the file. Chrome cannot decode HEVC; Safari can; Jellyfin's native apps handle far more than either.

So the server never declares a file playable. It reports the codec facts (from ffprobe, stored at scan time) and each client decides. See ADR-002.

That is what makes the engine registry work. internal/engine holds engines that declare what they can play — html5 ships built in; mpv, vlc and mdk are registered for native clients. Adding one is a declaration, not a rewrite:

POST /api/v1/playback/resolve   {"item_id":16,"engines":["html5"]}
  → playable=false, reason="vcodec:hevc"

POST /api/v1/playback/resolve   {"item_id":16,"engines":["html5","mpv"]}
  → playable=true, engine="mpv"

Same file, different answer, because the client offered a different engine.

The element is always the final authority. canPlayType lies in both directions, so the player uses it only as a hint and confirms by actually loading the file and checking videoWidth. For HEVC, Chrome plays audio over a blank frame with no error — the worst possible outcome, and the player detects it and says so rather than pretending.

---

8. Three protocol surfaces over one application

surface	who uses it	how it authenticates
/api/v1/*	the web app	session cookie
Jellyfin (/Items, /Users/…)	Jellyfin native apps	X-Emby-Token, ?api_key=
Subsonic (/rest/*.view)	Symfonium, Ultrasonic, play:Sub	t=md5(key+salt)

All three resolve to the same auth.User and pass through the same grant check. internal/apitoken is the shared credential layer, and its migration documents a real asymmetry: Jellyfin tokens are stored hashed (like session tokens), while a Subsonic key must be stored in the clear, because verifying t means computing md5(key + salt) server-side.

Both adapters are honest about the one thing Vellum cannot do: PlaybackInfo reports SupportsTranscoding: false, and a client that demands a transcode gets NoCompatibleStream rather than a URL that would never resolve.

---

9. Where things live

cmd/vellum          wiring and lifecycle — the only place packages meet
internal/
  core              config, database open, migrations, logger, timestamps
  httpx             small HTTP helpers (JSON, errors)
  auth              users, sessions, argon2id hashing, grants
  apitoken          credentials for non-browser clients
  catalog           THE data seam: read an item, and check you may
  providers         Open Library, Google Books, MusicBrainz, ComicVine, IGDB
  library           library definitions, media-type registry
  scanner           walk the filesystem and reconcile
  probe             ffprobe wrapper (optional dependency)
  media             item, file, cover, progress endpoints
  home              the five home shelves
  search            FTS5 search over titles, paths and authors
  metadata          the enrichment job and its matching policy
  playback          engine selection endpoint
  engine            playback engine registry
  jobs              table-backed job queue
  progress          per-user playback position
  stream            Range-capable file serving
  modules/          comics, ebooks, games, photos
  httpapi           mux, middleware, security headers, embedded UI
  jellyfin          Jellyfin protocol adapter
  subsonic          Subsonic protocol adapter
  arch              import-rule tests (no code)
migrations/         numbered SQL, append-only, tracked by user_version
web/                Svelte 5 + Vite frontend

---

10. Things to know before changing anything

• cmd/vellum/main.go is the only wiring point. Nothing else should import a module; modules register themselves.
• Migrations are append-only. Never edit an applied file. The FTS5 authors migration is the cautionary tale: it worked on a fresh database and raised "database disk image is malformed" on an upgrade.
• All timestamps go through core.Stamp. RFC3339Nano trims trailing zeros, so its strings are not lexicographically sortable — ".5Z" > ".55Z" while .500s < .550s. That silently broke ORDER BY and MAX() on the Continue shelf.
• Progress is PUT-only, so sendBeacon (POST-only) cannot be used; the teardown flush uses fetch with keepalive: true.
• canPlayType is not a source of truth. Put src on the media element, not type on a <source> — an explicit type makes Chrome skip files it would otherwise play.
• If you need an item, take a catalog.Store. Not *sql.DB. The seam is enforced by test rule 6/7.
• Run go test ./... after touching a package's imports — the arch rules will tell you immediately if you crossed a layer.

---

11. Commands

Command	What it does
make dev	run backend and Vite together
go test ./...	all Go tests, including the arch rules
go vet ./...	static checks
cd web && pnpm check	Svelte type-check
vellum user add <name> <pass> [--admin]	create a user
vellum user passwd <name> <new>	change a password
vellum user list	list users
vellum backup <path>	safe copy of the database (VACUUM INTO)

See DEPLOY.md for deployment, and decisions/ for why things are the way they are.