Vellum/docs/ARCHITECTURE.md

# Vellum — how it works

Vellum is a self-hosted media server. It replaces Jellyfin, Navidrome,
Audiobookshelf, Stump and Kavita with one program: one Go binary, one SQLite
file, one web UI. It handles ebooks, comics, audiobooks, music, photos, games
and video.

This document explains how the pieces fit together and, more importantly, *why*
they fit that way. Where a decision had a real trade-off, there is an ADR in
[`decisions/`](decisions/) with the alternatives that were rejected.

**Size:** ~11,500 lines of Go (plus ~7,000 lines of tests), ~7,100 lines of
frontend, 33 packages, 12 migrations, 64 routes.

---

## 1. The shape

```
        one binary                 one file              one UI
   ┌──────────────────┐      ┌───────────────┐      ┌──────────────┐
   │  vellum (Go)     │─────▶│ vellum.db     │      │ Svelte app   │
   │  serves API+UI   │      │ SQLite + WAL  │      │ embedded in  │
   │  runs the jobs   │      └───────────────┘      │ the binary   │
   └──────────────────┘                             └──────────────┘
```

There is no Redis, no message broker, no second process and no CGO. The SQLite
driver is pure Go (`modernc.org/sqlite`), which is why `CGO_ENABLED=0` works and
the container is a bare Alpine image. The job queue is a *table*.

**"One binary" is a deployment choice, not an architecture.** The code is a
modular monolith: 33 packages with enforced boundaries. Splitting it into
services would add network failure modes and make debugging harder, not easier.
See [ADR-004](decisions/ADR-004-modular-monolith.md).

---

## 2. What happens when you open the app

Every request goes through the same chain, in this order:

```
browser
  │
  ├─ withSecurityHeaders   outermost, so headers survive a panic
  ├─ withRecovery          panic → 500, never a dead connection
  ├─ withRequestID         X-Request-ID, echoed into every log line
  ├─ withRequestLogging    method, path, status, duration, request_id
  └─ auth.Middleware       cookie → session → User in the request context
       │
       └─ handler  →  auth.VisibleLibraryIDs / CanAccessLibrary  →  SQL  →  JSON
```

The **grant check is the security chokepoint**. Every read path goes through it,
including the Jellyfin and Subsonic adapters, which is why an API token can never
see more than the web app can. An item you cannot read returns **404, not 403** —
a 403 would confirm the item exists and leak the shape of someone else's library.

Media bytes take a different path on purpose: `internal/stream` hands off to
`http.ServeContent`, which implements HTTP Range correctly. That is why seeking
works in every player, and it is why compression is scoped to the static UI only
(compressing a Range response would break seeking).

---

## 3. The layers, and the rules that hold them

The code is arranged in six layers. A package may import from a lower layer or
its own, never from a higher one.

| Layer | Packages | What lives here |
|---|---|---|
| 0 | `core`, `httpx`, `natsort`, `arch`, `migrations` | foundation; knows nothing about the app |
| 1 | `auth`, `jobs`, `stream`, `progress`, `probe`, `engine`, `apitoken` | primitives everything is built from |
| 2 | `catalog`, `providers` | the data seam, and third-party API clients |
| 3 | `library`, `scanner`, `media`, `home`, `search`, `metadata`, `playback`, `modules/*`, `*/dto` | the application |
| 4 | `httpapi`, `jellyfin`, `subsonic`, `cli` | adapters: HTTP and foreign protocols |
| 5 | `cmd/vellum` | the only place everything is wired together |

**These rules are enforced by tests, not by documentation.** `internal/arch`
parses every package's imports and fails the build on a violation. There are
seven rules:

1. No upward imports across the layers.
2. `internal/core` may depend only on `migrations`.
3. `internal/catalog` must not know about HTTP — or it stops being reusable by a
   job or a test.
4. Modules must not import each other.
5. **Every package must be placed in a layer**, so the map cannot go stale.
6. The set of packages allowed to import `database/sql` is explicit.
7. Packages already moved onto the `catalog.Store` seam must not reach for
   `*sql.DB` again.

Each of the seven has been deliberately broken and confirmed to fail. A test that
cannot fail is decoration.

---

## 4. The data model — the decision everything follows from

```
users ──< library_grants >── libraries
  │                              │
  │                              └──< media_items ──< media_files
  │                                       │
  └──< progress (user, item, file) ───────┘
  └──< item_views, favorites, api_tokens
```

**Media items are global. Every per-user thing is a join.**

The `progress` primary key is `(user_id, item_id, file_id)`. That one choice is
why audiobooks needed no new Go module: a twelve-chapter book is one item with
twelve files, and "chapter 7, 4 minutes 32 seconds in" is a single row. It is
also why the Continue shelf is one query, and why per-user state can never
contaminate the shared catalogue.

**Deletion is deliberately not deletion.** A file that vanishes is marked
`missing = 1` with a `missing_since` timestamp, never removed. Deleting cascades
into `progress`, the one thing in the schema that cannot be regenerated — so an
unmounted drive would silently wipe your reading history. There is an explicit
admin purge for when you really mean it. See
[ADR-003](decisions/ADR-003-mark-missing-never-delete.md).

---

## 5. How the library gets in

`internal/scanner` walks the library root and reconciles it with the database.

```
walkRoot        →  files on disk, keyed by relative path
buildGroups     →  one item per file, OR one item per directory
batched upsert  →  500 groups per transaction
missing pass    →  mark vanished files, never delete
```

Three things worth knowing:

- **Transactions are chunked.** It used to hold SQLite's single write lock for
  the whole library. Now each transaction covers 500 groups, so a progress save
  from a playing client succeeds *during* a scan. Measured at 20,000 files with
  concurrent writes all returning 200.
- **Change detection is `size` + `mtime`.** Matching files are skipped entirely.
  `ffprobe` runs only for changed files, and before the write transaction opens,
  so a slow probe can never hold the write lock.
- **Two scans can run at once.** The job queue has four workers, so scanning two
  libraries concurrently works — the *writes* interleave at batch granularity
  rather than one library blocking the other.

---

## 6. The job queue

`internal/jobs` is a queue implemented as a database table. Claiming a job is a
single atomic statement, so several workers are safe with no extra locking:

```sql
UPDATE jobs SET state = 'running', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
            ORDER BY run_after, id LIMIT 1)
RETURNING id, kind, payload, attempts
```

Four workers with jittered idle waits. A panicking handler is recovered and the
job marked failed, so one bad handler cannot kill a worker or strand a row in
`running`. `RecoverStale()` runs at boot and requeues anything a dead process
left behind — without it, a crash mid-scan left a job `running` forever while the
UI polled it forever.

Two job kinds: `scan_library` and `enrich_item`.

---

## 7. Playback — direct play only

**Vellum never transcodes.** The media server is a headless, GPU-less Xeon with
no spare CPU for it. A file either plays in the client or it does not.

This is the single most important thing to understand about the playback code,
and it came from measurement rather than assumption:

| container | video | result |
|---|---|---|
| mp4 | h264 | plays |
| **mkv** | h264 | **plays** — the container is not the problem |
| mov | h264 | plays, even though `canPlayType` says "no" |
| mp4 | **hevc** | **audio only** — a black picture, no error |
| avi | mpeg4 | fails outright |

Two conclusions drive the design:

1. **Codec matters, container barely does.** mkv+h264 plays; mp4+HEVC does not.
   Judging a file by its extension is wrong.
2. **Playability is a property of the CLIENT, not the file.** Chrome cannot
   decode HEVC; Safari can; Jellyfin's native apps handle far more than either.

So the server **never** declares a file playable. It reports the codec facts
(from `ffprobe`, stored at scan time) and each client decides. See
[ADR-002](decisions/ADR-002-direct-play-only.md).

That is what makes the engine registry work. `internal/engine` holds engines that
declare what they can play — `html5` ships built in; `mpv`, `vlc` and `mdk` are
registered for native clients. Adding one is a declaration, not a rewrite:

```
POST /api/v1/playback/resolve   {"item_id":16,"engines":["html5"]}
  → playable=false, reason="vcodec:hevc"

POST /api/v1/playback/resolve   {"item_id":16,"engines":["html5","mpv"]}
  → playable=true, engine="mpv"
```

Same file, different answer, because the client offered a different engine.

**The element is always the final authority.** `canPlayType` lies in both
directions, so the player uses it only as a hint and confirms by actually loading
the file and checking `videoWidth`. For HEVC, Chrome plays audio over a blank
frame with no error — the worst possible outcome, and the player detects it and
says so rather than pretending.

---

## 8. Three protocol surfaces over one application

| surface | who uses it | how it authenticates |
|---|---|---|
| `/api/v1/*` | the web app | session cookie |
| Jellyfin (`/Items`, `/Users/…`) | Jellyfin native apps | `X-Emby-Token`, `?api_key=` |
| Subsonic (`/rest/*.view`) | Symfonium, Ultrasonic, play:Sub | `t=md5(key+salt)` |

All three resolve to the same `auth.User` and pass through the same grant check.
`internal/apitoken` is the shared credential layer, and its migration documents a
real asymmetry: Jellyfin tokens are stored **hashed** (like session tokens),
while a Subsonic key must be stored **in the clear**, because verifying `t` means
computing `md5(key + salt)` server-side.

Both adapters are honest about the one thing Vellum cannot do: `PlaybackInfo`
reports `SupportsTranscoding: false`, and a client that demands a transcode gets
`NoCompatibleStream` rather than a URL that would never resolve.

---

## 9. Where things live

```
cmd/vellum          wiring and lifecycle — the only place packages meet
internal/
  core              config, database open, migrations, logger, timestamps
  httpx             small HTTP helpers (JSON, errors)
  auth              users, sessions, argon2id hashing, grants
  apitoken          credentials for non-browser clients
  catalog           THE data seam: read an item, and check you may
  providers         Open Library, Google Books, MusicBrainz, ComicVine, IGDB
  library           library definitions, media-type registry
  scanner           walk the filesystem and reconcile
  probe             ffprobe wrapper (optional dependency)
  media             item, file, cover, progress endpoints
  home              the five home shelves
  search            FTS5 search over titles, paths and authors
  metadata          the enrichment job and its matching policy
  playback          engine selection endpoint
  engine            playback engine registry
  jobs              table-backed job queue
  progress          per-user playback position
  stream            Range-capable file serving
  modules/          comics, ebooks, games, photos
  httpapi           mux, middleware, security headers, embedded UI
  jellyfin          Jellyfin protocol adapter
  subsonic          Subsonic protocol adapter
  arch              import-rule tests (no code)
migrations/         numbered SQL, append-only, tracked by user_version
web/                Svelte 5 + Vite frontend
```

---

## 10. Things to know before changing anything

- **`cmd/vellum/main.go` is the only wiring point.** Nothing else should import a
  module; modules register themselves.
- **Migrations are append-only.** Never edit an applied file. The FTS5 authors
  migration is the cautionary tale: it worked on a fresh database and raised
  *"database disk image is malformed"* on an upgrade.
- **All timestamps go through `core.Stamp`.** `RFC3339Nano` trims trailing zeros,
  so its strings are not lexicographically sortable — `".5Z" > ".55Z"` while
  `.500s < .550s`. That silently broke `ORDER BY` and `MAX()` on the Continue
  shelf.
- **Progress is PUT-only**, so `sendBeacon` (POST-only) cannot be used; the
  teardown flush uses `fetch` with `keepalive: true`.
- **`canPlayType` is not a source of truth.** Put `src` on the media element, not
  `type` on a `<source>` — an explicit type makes Chrome *skip* files it would
  otherwise play.
- **If you need an item, take a `catalog.Store`.** Not `*sql.DB`. The seam is
  enforced by test rule 6/7.
- **Run `go test ./...` after touching a package's imports** — the arch rules
  will tell you immediately if you crossed a layer.

---

## 11. Commands

| Command | What it does |
|---|---|
| `make dev` | run backend and Vite together |
| `go test ./...` | all Go tests, including the arch rules |
| `go vet ./...` | static checks |
| `cd web && pnpm check` | Svelte type-check |
| `vellum user add <name> <pass> [--admin]` | create a user |
| `vellum user passwd <name> <new>` | change a password |
| `vellum user list` | list users |
| `vellum backup <path>` | safe copy of the database (`VACUUM INTO`) |

See [DEPLOY.md](../DEPLOY.md) for deployment, and
[`decisions/`](decisions/) for why things are the way they are.