ad3laid3/Vellum
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Documentation, and make a fresh clone actually work

Browse source 
Docs
- docs/ARCHITECTURE.md: how the pieces fit and why. The request path, the six
  enforced layers, the data model, the scanner, the job queue, direct-play
  playback, and the three protocol surfaces over one application.
- docs/decisions/: seven ADRs with the alternatives that were rejected —
  SQLite-with-a-seam, direct-play-only, mark-missing-never-delete, the modular
  monolith, the one item-access chokepoint, the table-backed queue, and
  fixed-width timestamps.
- README: the layout section was stale (it predated catalog, home, scanner,
  providers, engine, probe, playback, apitoken, arch and the dto packages) and
  now lists what exists, plus links to the new docs.

Two real bugs, both found by verifying the docs instead of trusting them

1. The CLI could not run on a fresh install. `Config.Validate` created the data
   and config directories, but the CLI never called it, so `vellum user add` —
   the recovery path when you are locked out — failed with "unable to open
   database file" until the server had booted at least once. Split the directory
   creation into `Config.EnsureDirs`, call it from the CLI, and keep the media
   root check out of it: an admin command must not fail because a library is
   unmounted. The CLI tests now point DBPath inside DataDir, exactly as
   LoadConfig derives it, so they exercise the fresh-install path; removing
   EnsureDirs fails three of them.

2. A fresh clone could not build at all. `//go:embed all:dist` fails to compile
   when web/dist is absent, and web/.gitignore expected a `dist/.gitkeep` that
   was never created. Added it, so a clone builds and the server logs
   "web UI not embedded; run the frontend build to include it" — the intended
   behaviour for someone who only wants the API.

Also: the httpapi tests depended on a built frontend and FAILED rather than
skipping, so the suite was red on any fresh clone. findAsset and readAsset now
skip with a message naming the missing step. They still RUN when a bundle is
present (verified: zero skips here), so they have not been quietly disabled.

Verified from a simulated clone of tracked files only: go build exit 0, 25
packages pass, the server boots and applies all 12 migrations, and the CLI
creates a user and takes a backup. gofmt clean, vet exit 0, svelte-check 0/0.
This commit is contained in:
mithral 2026-10-12 00:18:11 -04:00
parent 129ad80f14
commit 051169a18a
15 changed files with 947 additions and 14 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
README.md
View file

@ -83,35 +83,58 @@ the server logs a warning and refuses to invent a default password.
## Layout
```
cmd/vellum/ main — wiring and lifecycle only
cmd/vellum/ main — wiring and lifecycle only, the one place packages meet
internal/
core/ config, logging, database, migration runner
core/ config, logging, database, migrations, timestamps
httpx/ shared JSON helpers
httpapi/ router, middleware, static SPA serving
httpapi/ router, middleware, security headers, static SPA serving
auth/ argon2id, sessions, grants, bootstrap
library/ library CRUD, media-type registry, scanner
apitoken/ credentials for non-browser clients (Jellyfin, Subsonic)
catalog/ THE data seam: read an item, and check you may
library/ library CRUD and the media-type registry
scanner/ walk the filesystem and reconcile with the database
probe/ ffprobe wrapper — codec facts (optional dependency)
media/ generic item/file/cover/progress endpoints
home/ the five home shelves
search/ FTS5 queries
metadata/ the enrichment job and its matching policy
providers/ Open Library, Google Books, MusicBrainz, ComicVine, IGDB
playback/ playback engine selection endpoint
engine/ playback engine registry (html5, mpv, vlc, mdk)
progress/ the polymorphic per-user progress store
stream/ range-capable file serving
natsort/ natural ordering for pages and tracks
jobs/ table-backed job queue
metadata/ provider interface, Open Library, Google Books, enricher
search/ FTS5 queries
natsort/ natural ordering for pages and tracks
jellyfin/ Jellyfin protocol adapter (+ dto/)
subsonic/ Subsonic protocol adapter (+ dto/)
arch/ import-rule tests; holds no code
modules/
ebooks/ EPUB parsing and cover extraction
comics/ CBZ page listing and extraction
games/ platforms, IGDB, cloud saves, EmulatorJS
photos/ thumbnail generation
web/ Svelte app; dist/ is embedded via web/embed.go
migrations/ embedded SQL, applied in order at boot
docs/ architecture guide and decision records
```
**Module rule:** the core never imports a feature module; modules import the
core and mount their own routes, tables, and scanner rules. Adding a media type
is a new package plus a migration, never a core edit.
**Layer rule:** packages are arranged in six layers and a package may never
import from a higher one. This is enforced by tests in `internal/arch`, not by
convention — see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
**Schema rule:** media items are global; everything user-specific is a join.
Nothing may hold per-user state on a media row.
## Documentation
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how the pieces fit and why.
- [docs/decisions/](docs/decisions/) — the decisions that were expensive to
reverse, with the alternatives that were rejected.
## Database
SQLite in WAL mode, opened through `modernc.org/sqlite` — a pure-Go driver, so

316
docs/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,316 @@
# Vellum — how it works
Vellum is a self-hosted media server. It replaces Jellyfin, Navidrome,
Audiobookshelf, Stump and Kavita with one program: one Go binary, one SQLite
file, one web UI. It handles ebooks, comics, audiobooks, music, photos, games
and video.
This document explains how the pieces fit together and, more importantly, *why*
they fit that way. Where a decision had a real trade-off, there is an ADR in
[`decisions/`](decisions/) with the alternatives that were rejected.
**Size:** ~11,500 lines of Go (plus ~7,000 lines of tests), ~7,100 lines of
frontend, 33 packages, 12 migrations, 64 routes.
---
## 1. The shape
```
one binary one file one UI
┌──────────────────┐ ┌───────────────┐ ┌──────────────┐
│ vellum (Go) │─────▶│ vellum.db │ │ Svelte app │
│ serves API+UI │ │ SQLite + WAL │ │ embedded in │
│ runs the jobs │ └───────────────┘ │ the binary │
└──────────────────┘ └──────────────┘
```
There is no Redis, no message broker, no second process and no CGO. The SQLite
driver is pure Go (`modernc.org/sqlite`), which is why `CGO_ENABLED=0` works and
the container is a bare Alpine image. The job queue is a *table*.
**"One binary" is a deployment choice, not an architecture.** The code is a
modular monolith: 33 packages with enforced boundaries. Splitting it into
services would add network failure modes and make debugging harder, not easier.
See [ADR-004](decisions/ADR-004-modular-monolith.md).
---
## 2. What happens when you open the app
Every request goes through the same chain, in this order:
```
browser
│
├─ withSecurityHeaders outermost, so headers survive a panic
├─ withRecovery panic → 500, never a dead connection
├─ withRequestID X-Request-ID, echoed into every log line
├─ withRequestLogging method, path, status, duration, request_id
└─ auth.Middleware cookie → session → User in the request context
│
└─ handler → auth.VisibleLibraryIDs / CanAccessLibrary → SQL → JSON
```
The **grant check is the security chokepoint**. Every read path goes through it,
including the Jellyfin and Subsonic adapters, which is why an API token can never
see more than the web app can. An item you cannot read returns **404, not 403** —
a 403 would confirm the item exists and leak the shape of someone else's library.
Media bytes take a different path on purpose: `internal/stream` hands off to
`http.ServeContent`, which implements HTTP Range correctly. That is why seeking
works in every player, and it is why compression is scoped to the static UI only
(compressing a Range response would break seeking).
---
## 3. The layers, and the rules that hold them
The code is arranged in six layers. A package may import from a lower layer or
its own, never from a higher one.
| Layer | Packages | What lives here |
|---|---|---|
| 0 | `core`, `httpx`, `natsort`, `arch`, `migrations` | foundation; knows nothing about the app |
| 1 | `auth`, `jobs`, `stream`, `progress`, `probe`, `engine`, `apitoken` | primitives everything is built from |
| 2 | `catalog`, `providers` | the data seam, and third-party API clients |
| 3 | `library`, `scanner`, `media`, `home`, `search`, `metadata`, `playback`, `modules/*`, `*/dto` | the application |
| 4 | `httpapi`, `jellyfin`, `subsonic`, `cli` | adapters: HTTP and foreign protocols |
| 5 | `cmd/vellum` | the only place everything is wired together |
**These rules are enforced by tests, not by documentation.** `internal/arch`
parses every package's imports and fails the build on a violation. There are
seven rules:
1. No upward imports across the layers.
2. `internal/core` may depend only on `migrations`.
3. `internal/catalog` must not know about HTTP — or it stops being reusable by a
job or a test.
4. Modules must not import each other.
5. **Every package must be placed in a layer**, so the map cannot go stale.
6. The set of packages allowed to import `database/sql` is explicit.
7. Packages already moved onto the `catalog.Store` seam must not reach for
`*sql.DB` again.
Each of the seven has been deliberately broken and confirmed to fail. A test that
cannot fail is decoration.
---
## 4. The data model — the decision everything follows from
```
users ──< library_grants >── libraries
│ │
│ └──< media_items ──< media_files
│ │
└──< progress (user, item, file) ───────┘
└──< item_views, favorites, api_tokens
```
**Media items are global. Every per-user thing is a join.**
The `progress` primary key is `(user_id, item_id, file_id)`. That one choice is
why audiobooks needed no new Go module: a twelve-chapter book is one item with
twelve files, and "chapter 7, 4 minutes 32 seconds in" is a single row. It is
also why the Continue shelf is one query, and why per-user state can never
contaminate the shared catalogue.
**Deletion is deliberately not deletion.** A file that vanishes is marked
`missing = 1` with a `missing_since` timestamp, never removed. Deleting cascades
into `progress`, the one thing in the schema that cannot be regenerated — so an
unmounted drive would silently wipe your reading history. There is an explicit
admin purge for when you really mean it. See
[ADR-003](decisions/ADR-003-mark-missing-never-delete.md).
---
## 5. How the library gets in
`internal/scanner` walks the library root and reconciles it with the database.
```
walkRoot → files on disk, keyed by relative path
buildGroups → one item per file, OR one item per directory
batched upsert → 500 groups per transaction
missing pass → mark vanished files, never delete
```
Three things worth knowing:
- **Transactions are chunked.** It used to hold SQLite's single write lock for
the whole library. Now each transaction covers 500 groups, so a progress save
from a playing client succeeds *during* a scan. Measured at 20,000 files with
concurrent writes all returning 200.
- **Change detection is `size` + `mtime`.** Matching files are skipped entirely.
`ffprobe` runs only for changed files, and before the write transaction opens,
so a slow probe can never hold the write lock.
- **Two scans can run at once.** The job queue has four workers, so scanning two
libraries concurrently works — the *writes* interleave at batch granularity
rather than one library blocking the other.
---
## 6. The job queue
`internal/jobs` is a queue implemented as a database table. Claiming a job is a
single atomic statement, so several workers are safe with no extra locking:
```sql
UPDATE jobs SET state = 'running', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
ORDER BY run_after, id LIMIT 1)
RETURNING id, kind, payload, attempts
```
Four workers with jittered idle waits. A panicking handler is recovered and the
job marked failed, so one bad handler cannot kill a worker or strand a row in
`running`. `RecoverStale()` runs at boot and requeues anything a dead process
left behind — without it, a crash mid-scan left a job `running` forever while the
UI polled it forever.
Two job kinds: `scan_library` and `enrich_item`.
---
## 7. Playback — direct play only
**Vellum never transcodes.** The media server is a headless, GPU-less Xeon with
no spare CPU for it. A file either plays in the client or it does not.
This is the single most important thing to understand about the playback code,
and it came from measurement rather than assumption:
| container | video | result |
|---|---|---|
| mp4 | h264 | plays |
| **mkv** | h264 | **plays** — the container is not the problem |
| mov | h264 | plays, even though `canPlayType` says "no" |
| mp4 | **hevc** | **audio only** — a black picture, no error |
| avi | mpeg4 | fails outright |
Two conclusions drive the design:
1. **Codec matters, container barely does.** mkv+h264 plays; mp4+HEVC does not.
Judging a file by its extension is wrong.
2. **Playability is a property of the CLIENT, not the file.** Chrome cannot
decode HEVC; Safari can; Jellyfin's native apps handle far more than either.
So the server **never** declares a file playable. It reports the codec facts
(from `ffprobe`, stored at scan time) and each client decides. See
[ADR-002](decisions/ADR-002-direct-play-only.md).
That is what makes the engine registry work. `internal/engine` holds engines that
declare what they can play — `html5` ships built in; `mpv`, `vlc` and `mdk` are
registered for native clients. Adding one is a declaration, not a rewrite:
```
POST /api/v1/playback/resolve {"item_id":16,"engines":["html5"]}
→ playable=false, reason="vcodec:hevc"
POST /api/v1/playback/resolve {"item_id":16,"engines":["html5","mpv"]}
→ playable=true, engine="mpv"
```
Same file, different answer, because the client offered a different engine.
**The element is always the final authority.** `canPlayType` lies in both
directions, so the player uses it only as a hint and confirms by actually loading
the file and checking `videoWidth`. For HEVC, Chrome plays audio over a blank
frame with no error — the worst possible outcome, and the player detects it and
says so rather than pretending.
---
## 8. Three protocol surfaces over one application
| surface | who uses it | how it authenticates |
|---|---|---|
| `/api/v1/*` | the web app | session cookie |
| Jellyfin (`/Items`, `/Users/…`) | Jellyfin native apps | `X-Emby-Token`, `?api_key=` |
| Subsonic (`/rest/*.view`) | Symfonium, Ultrasonic, play:Sub | `t=md5(key+salt)` |
All three resolve to the same `auth.User` and pass through the same grant check.
`internal/apitoken` is the shared credential layer, and its migration documents a
real asymmetry: Jellyfin tokens are stored **hashed** (like session tokens),
while a Subsonic key must be stored **in the clear**, because verifying `t` means
computing `md5(key + salt)` server-side.
Both adapters are honest about the one thing Vellum cannot do: `PlaybackInfo`
reports `SupportsTranscoding: false`, and a client that demands a transcode gets
`NoCompatibleStream` rather than a URL that would never resolve.
---
## 9. Where things live
```
cmd/vellum wiring and lifecycle — the only place packages meet
internal/
core config, database open, migrations, logger, timestamps
httpx small HTTP helpers (JSON, errors)
auth users, sessions, argon2id hashing, grants
apitoken credentials for non-browser clients
catalog THE data seam: read an item, and check you may
providers Open Library, Google Books, MusicBrainz, ComicVine, IGDB
library library definitions, media-type registry
scanner walk the filesystem and reconcile
probe ffprobe wrapper (optional dependency)
media item, file, cover, progress endpoints
home the five home shelves
search FTS5 search over titles, paths and authors
metadata the enrichment job and its matching policy
playback engine selection endpoint
engine playback engine registry
jobs table-backed job queue
progress per-user playback position
stream Range-capable file serving
modules/ comics, ebooks, games, photos
httpapi mux, middleware, security headers, embedded UI
jellyfin Jellyfin protocol adapter
subsonic Subsonic protocol adapter
arch import-rule tests (no code)
migrations/ numbered SQL, append-only, tracked by user_version
web/ Svelte 5 + Vite frontend
```
---
## 10. Things to know before changing anything
- **`cmd/vellum/main.go` is the only wiring point.** Nothing else should import a
module; modules register themselves.
- **Migrations are append-only.** Never edit an applied file. The FTS5 authors
migration is the cautionary tale: it worked on a fresh database and raised
*"database disk image is malformed"* on an upgrade.
- **All timestamps go through `core.Stamp`.** `RFC3339Nano` trims trailing zeros,
so its strings are not lexicographically sortable — `".5Z" > ".55Z"` while
`.500s < .550s`. That silently broke `ORDER BY` and `MAX()` on the Continue
shelf.
- **Progress is PUT-only**, so `sendBeacon` (POST-only) cannot be used; the
teardown flush uses `fetch` with `keepalive: true`.
- **`canPlayType` is not a source of truth.** Put `src` on the media element, not
`type` on a `<source>` — an explicit type makes Chrome *skip* files it would
otherwise play.
- **If you need an item, take a `catalog.Store`.** Not `*sql.DB`. The seam is
enforced by test rule 6/7.
- **Run `go test ./...` after touching a package's imports** — the arch rules
will tell you immediately if you crossed a layer.
---
## 11. Commands
| Command | What it does |
|---|---|
| `make dev` | run backend and Vite together |
| `go test ./...` | all Go tests, including the arch rules |
| `go vet ./...` | static checks |
| `cd web && pnpm check` | Svelte type-check |
| `vellum user add <name> <pass> [--admin]` | create a user |
| `vellum user passwd <name> <new>` | change a password |
| `vellum user list` | list users |
| `vellum backup <path>` | safe copy of the database (`VACUUM INTO`) |
See [DEPLOY.md](../DEPLOY.md) for deployment, and
[`decisions/`](decisions/) for why things are the way they are.

67
docs/decisions/ADR-001-sqlite-with-a-seam.md Normal file
View file

@ -0,0 +1,67 @@
# ADR-001: SQLite, not PostgreSQL — with a seam for later
## Status
Accepted
## Date
2026-10-12
## Context
Vellum stores everything in one database: users, grants, the catalogue, per-user
progress, the job queue and API tokens. The choice of engine determines the
deployment story, the backup story, and how much concurrent writing is possible.
The original motivation for considering PostgreSQL was concurrency: SQLite allows
only one writer at a time, and a library scan holding the write lock would block
a playing client's progress save.
That problem turned out to be solvable without changing engines, and it was
measured rather than assumed.
## Decision
Use SQLite with WAL, and put a storage seam behind the one package that reads
items, so the engine is swappable later if a real need appears.
## Alternatives Considered
### PostgreSQL
- **Pros:** real concurrent writers; no single-writer bottleneck; mature.
- **Cons:** a second process to run and back up; the deployment story stops being
"one binary and one file"; the SQL is not portable — 186 statements, 193 `?`
placeholders, FTS5 full-text search, `strftime`, `group_concat`, `rowid`, and
`VACUUM INTO` for backups all need reworking.
- **Rejected for now.** The problem it solves was already fixed by other means
(see below), and there is no measured write contention left.
### Fixing the write-lock problem in code instead
Three changes removed the symptom without changing engines:
1. **Chunked scan transactions.** 500 groups per transaction instead of one
transaction over the whole library. Measured: a 20,000-file scan reports
partial progress and a concurrent progress save returns 200 throughout.
2. **Four job workers** with jittered idle waits, so a scan no longer stalls
every other job.
3. **Throttled session writes.** `last_seen` was written on *every* authenticated
request, including every audio Range request. Measured after the fix: 20
requests cause **0** session writes, down from 20.
Measured results after these changes: browse page 1 at **0.03 ms**, the home
screen at ~3 ms, and two concurrent 8,000-file scans interleaving successfully
with every concurrent write returning 200.
## Consequences
- Deployment stays one binary plus one file. Backup is `VACUUM INTO`.
- Writes serialise. This is fine at the measured scale and is the honest ceiling
of this design.
- `internal/catalog` exposes a `Store` interface with a SQLite implementation, so
a PostgreSQL implementation would be a second `Store` rather than a rewrite.
- The seam is guarded: `internal/arch` fails the build if a new package starts
importing `database/sql` without being declared, or if a package already on the
seam reaches for `*sql.DB` again.
- **When to revisit:** multiple Vellum instances sharing a database, or write
throughput beyond what chunked WAL handles. Neither is true today.

102
docs/decisions/ADR-002-direct-play-only.md Normal file
View file

@ -0,0 +1,102 @@
# ADR-002: Direct play only — no transcoding, ever
## Status
Accepted
## Date
2026-10-12
## Context
A media server normally transcodes: it re-encodes a file on the fly when the
client cannot play it. That is how Jellyfin and Plex make an arbitrary library
playable on an arbitrary device.
Vellum runs on a **headless, GPU-less Xeon**. It has no display and no spare CPU
for real-time encoding. A 1080p HEVC transcode on a CPU-only machine runs at a
few times realtime at best; a two-hour film would take tens of minutes, per
viewer, forever.
The question was what to do instead.
## Decision
**Never transcode.** A file either direct-plays in the client or it does not.
The server reports what is *in* the file and the **client** decides whether it
can play it.
## Alternatives Considered
### Transcode on demand, CPU only
- **Pros:** every file plays everywhere.
- **Cons:** on this hardware a single viewer would saturate the CPU for minutes
per film. Two viewers would make it unusable.
- **Rejected:** the hardware cannot support it.
### Remux on demand (`-c copy`), no re-encode
- **Pros:** cheap — no video re-encode, just a container change. Would fix
"mkv will not play".
- **Cons:** the premise turned out to be **false**. Measurement showed Chrome
plays `mkv + h264 + aac` perfectly. The container was never the problem.
- **Rejected:** it solves a problem that does not exist.
### Pre-transcode everything to a normalised format
- **Pros:** paid once, everyone direct-plays forever after.
- **Cons:** doubles disk usage for a whole library, and takes days of CPU for a
large collection. Also irreversible if the settings were wrong.
- **Rejected:** too much cost and risk for the benefit.
## The measurement that decided it
Files with known codecs, played through Vellum's own stream endpoint in Chrome
155 on Linux:
| container | video | audio | result |
|---|---|---|---|
| mp4 | h264 | aac | plays |
| **mkv** | h264 | aac | **plays** |
| mov | h264 | aac | plays |
| webm | vp9 / vp8 | opus / vorbis | plays |
| mp4 | av1 | opus | plays |
| mp4 | **hevc** | aac | **audio only**, `videoWidth` stays 0, no error |
| mkv | **hevc** | flac | **audio only** |
| avi | mpeg4 | mp3 | fails outright |
All seven audio formats play.
**Two conclusions:**
1. **Codec matters, container barely does.** mkv+h264 plays; mp4+HEVC does not.
Judging a file by its extension is wrong.
2. **`canPlayType` is unreliable in both directions.** It says "no" for
`mov+h264` (which plays) and "maybe" for `mkv+hevc` (whose video never
decodes). The only trustworthy test is loading the file and checking
`videoWidth`.
**And the conclusion that shapes the architecture:** playability is a property of
the **client**, not the file. Chrome cannot decode HEVC; Safari can; Jellyfin's
native apps handle far more than either. So a server-side `playable: true/false`
would be wrong for most clients.
## Consequences
- `internal/probe` records codec facts at scan time (optional dependency — the
server works without `ffprobe`, just with less information).
- `internal/engine` holds a registry of playback engines. `html5` ships built in;
`mpv`, `vlc` and `mdk` are registered for native clients. **Adding one is a
declaration, not a re-architecture.**
- `POST /api/v1/playback/resolve` matches a file against the engines the client
declares. The same HEVC file is unplayable with `["html5"]` and playable with
`["html5","mpv"]`.
- The player treats the server as an **advisory** and the media element as the
final authority, because `canPlayType` lies.
- Jellyfin's `PlaybackInfo` reports `SupportsTranscoding: false`, and a client
demanding a transcode gets `NoCompatibleStream` rather than a URL that would
never resolve.
- **The honest UX consequence:** files that cannot direct-play say so clearly,
the same way an unreadable `.mobi` does — rather than a player that loads and
shows nothing. For HEVC, Chrome plays audio over a blank frame with no error,
which is the worst possible outcome, so the player detects exactly that.

65
docs/decisions/ADR-003-mark-missing-never-delete.md Normal file
View file

@ -0,0 +1,65 @@
# ADR-003: A vanished file is marked missing, never deleted
## Status
Accepted
## Date
2026-10-12
## Context
A library root is not always available. Drives unmount, NFS has hiccups, and
people reorganise folders while a scan is running.
The obvious behaviour for a scanner is to delete rows whose files are gone. That
is also the most destructive thing it could do.
## Decision
A scan that cannot find a file marks it `missing = 1` and records
`missing_since`. It never deletes. Removal is a separate, explicit, admin-only
action.
## Alternatives Considered
### Delete on scan
- **Pros:** simple; the database always reflects the disk.
- **Cons:** deleting a `media_items` row cascades into `progress`. If a drive is
unmounted for an hour, every reading position, audiobook chapter and
watch-later marker for that drive is gone permanently.
- **Rejected:** it destroys the one thing in the schema that cannot be
regenerated. Everything else can be rebuilt by rescanning.
### Delete after a grace period, automatically
- **Pros:** reclaims space without anyone thinking about it.
- **Cons:** a drive left disconnected for longer than the period loses its
history silently. The user gets no warning and no undo.
- **Rejected:** the failure mode is silent and irreversible.
### Keep `missing` but never surface it
- **Pros:** none. This was the original state.
- **Cons:** the worst of both worlds — you cannot see what is missing, and it
never goes away. A deleted file was invisible *and* permanent.
- **Rejected:** this is the gap this ADR closes.
## Consequences
- Reads filter on `missing = 0`, so a missing item disappears from every shelf,
list and search result without the row being touched.
- `missing_since` is written **only on the transition** into missing, so the age
says when the file first vanished rather than when it was last rescanned. It is
cleared when a file is restored.
- `GET /api/v1/missing` lists what has vanished, scoped to the caller's visible
libraries.
- `POST /api/v1/libraries/{id}/purge` (admin only) deletes only items that are
missing **and** have no present file. It requires an explicit confirmation in
the UI that names the count and says progress cannot be regenerated.
- The purge cascade — `media_files`, `progress`, `item_people`, `provider_ids`,
`metadata_fetch`, `assets`, `collection_items`, `favorites`, `item_views`,
`game_saves`, child items via `parent_id` — is enumerated in the migration, so
the destructiveness is visible at the point of the decision.
- Verified: purging 19 items leaves 18, only the missing one goes, and a
non-missing item's files are untouched. A second purge returns 0.

57
docs/decisions/ADR-004-modular-monolith.md Normal file
View file

@ -0,0 +1,57 @@
# ADR-004: A modular monolith — one binary, not services
## Status
Accepted
## Date
2026-10-12
## Context
The goal was a codebase that is easy to contribute to and easy to debug. "Split
it into services" is the usual answer to that, and it was explicitly proposed.
## Decision
Keep one process. Get modularity from **enforced package boundaries**, not from
network boundaries.
## Alternatives Considered
### Split into microservices (a scanner service, a metadata service, an API service)
- **Pros:** independent deploys; each service small enough to read in one sitting.
- **Cons:** every call becomes a network call that can time out, retry, duplicate
or partially fail. Debugging a bug becomes "which service dropped this", and
the stack trace is replaced by distributed tracing.
- **Rejected:** it would make the stated goal — *easier to find and fix bugs* —
harder, not easier. Every bug found in this codebase so far has been a code
bug, and none would have been easier to locate across a network boundary.
### One package per concern, no rules
- **Pros:** simplest to start.
- **Cons:** without enforcement, boundaries erode. An upward import does not fail
to compile; it just quietly welds two layers together until nothing can be
moved.
- **Rejected:** documented rules that nobody enforces are rules that break within
a month.
## Consequences
- 33 packages in six layers, with the import rules **enforced by tests** in
`internal/arch`. A violation fails the build.
- Seven rules, each deliberately broken and confirmed to fail: no upward imports;
`core` depends only on `migrations`; `catalog` knows no HTTP; modules do not
import each other; every package must be in a layer; the `database/sql`
importers are an explicit list; packages on the storage seam must not drift
back.
- Rule 5 — *every package must be placed in a layer* — is the one that keeps the
others alive. Without it, a new package quietly has no rules at all. It caught
two unlayered packages during the Phase 2 splits.
- `cmd/vellum/main.go` is the single wiring point. Nothing else imports a module;
modules register themselves.
- Deployment stays one binary, which keeps the operational story simple: no
service discovery, no partial deploys, no version skew between services.
- **Cost accepted:** a change that spans layers needs a deliberate interface,
which is slower than reaching across. That is the point.

77
docs/decisions/ADR-005-one-item-access-chokepoint.md Normal file
View file

@ -0,0 +1,77 @@
# ADR-005: One item-access chokepoint, `internal/catalog`
## Status
Accepted
## Date
2026-10-12
## Context
Five different packages needed the same thing: load a media item, then check the
caller is allowed to read it.
- `internal/media`
- `internal/modules/comics`
- `internal/modules/games`
- `internal/jellyfin` — with its own `dbItem` type and its own SQL
- `internal/subsonic` — with its own per-item queries
Each had its own implementation, and each decided for itself what to do when
access was denied.
## Decision
One package owns item access **and** the access check: `internal/catalog`.
## Alternatives Considered
### Leave the duplication
- **Pros:** no refactor; each package stays self-contained.
- **Cons:** five copies of one authorization rule means four can drift from the
fifth. The failure mode is not a crash — it is silently serving somebody a
library they were never granted. That is a data-leak bug, and it would be found
by accident, if at all.
- **Rejected:** this is the highest-value thing in the codebase to have exactly
one of.
### A shared helper that packages call with their own SQL
- **Pros:** smaller change.
- **Cons:** leaves five copies of the item-loading SQL, so a schema change still
has to be made in five places.
- **Rejected:** it halves the problem instead of removing it.
### A generic repository layer for everything
- **Pros:** one place for all SQL.
- **Cons:** pulls browse lists, home shelves and Subsonic album aggregates into a
generic layer where they do not belong. Those are each package's own
presentation concern, and moving them is a 186-statement rewrite.
- **Rejected:** the chokepoint is the security-critical, most-duplicated path —
not every query.
## The security property
`catalog.ItemFor` returns the **same** `ErrNotFound` for an item that does not
exist and an item the caller may not read.
This is deliberate. If the two were distinguishable, a 403 would confirm that
another user's library contains that item, leaking its shape. The test asserts
the two responses are **byte-identical**, not merely both non-200.
## Consequences
- Five duplicate implementations became one. `internal/jellyfin`'s `dbItem` type
and its own item SQL are gone.
- All four HTTP surfaces — web, modules, Jellyfin, Subsonic — go through the same
check, so an API token can never see more than the web app can.
- An unreadable item is **404, never 403**, everywhere.
- Aggregate and list queries deliberately stayed where they are. `catalog` is
item-centric; Subsonic is file-first (`mf-<id>`) and resolves file→item through
its own projection. A catalog lookup that takes a file id would close that gap.
- The seam is a `Store` interface with a SQLite implementation, so the storage
engine is swappable. One consumer (`internal/playback`) is threaded onto it;
the rest still use the package-level wrappers, and the remaining threading is
mechanical.

62
docs/decisions/ADR-006-table-backed-job-queue.md Normal file
View file

@ -0,0 +1,62 @@
# ADR-006: The job queue is a table, not a broker
## Status
Accepted
## Date
2026-10-12
## Context
Vellum does work outside a request: scanning libraries and enriching metadata
from external providers. That work needs a queue — something has to survive a
restart, retry a failure, and be visible while it runs.
## Decision
Implement the queue as a table in the same SQLite database. No Redis, no broker,
no second process.
## Alternatives Considered
### Redis or a similar broker
- **Pros:** built for queues; fast; familiar.
- **Cons:** a second service to run, back up and monitor. The queue would then
live outside the database, so a backup would capture the catalogue but not the
work in flight.
- **Rejected:** it buys throughput this application does not need and costs the
"one binary, one file" deployment story.
### In-process goroutines with an in-memory queue
- **Pros:** no storage at all.
- **Cons:** a restart loses queued work silently. There is no way to see what is
pending, and no way to retry a failure after a crash.
- **Rejected:** the queue must survive a restart.
## Consequences
Claiming a job is a single atomic statement, so several workers are safe with no
extra locking:
```sql
UPDATE jobs SET state = 'running', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
ORDER BY run_after, id LIMIT 1)
RETURNING id, kind, payload, attempts
```
- Four workers with jittered idle waits. A panicking handler is recovered and the
job marked failed, so one bad handler cannot kill a worker or strand a row in
`running`.
- `RecoverStale()` runs at boot and requeues anything a dead process left behind.
Without it, a crash mid-scan left a job `running` forever while the UI polled it
forever.
- The queue is included in the same backup as everything else.
- Job state is queryable, which is how scan progress reaches the UI
(`GET /libraries/{id}/scan/status`).
- Two job kinds: `scan_library` and `enrich_item`.
- **Cost accepted:** the queue shares SQLite's single writer, so a very large
scan slows enqueueing. Chunked transactions keep each write short, and this is
a deliberate trade against running a second service.

82
docs/decisions/ADR-007-fixed-width-timestamps.md Normal file
View file

@ -0,0 +1,82 @@
# ADR-007: Timestamps are fixed-width, and written through one helper
## Status
Accepted
## Date
2026-10-12
## Context
Timestamps are stored as TEXT and compared as strings: `ORDER BY updated_at`,
`MAX(updated_at)`, `run_after <= ?`, `expires_at < ?`. String comparison is only
correct if the strings are the same width, which is easy to get wrong.
It was wrong. Two formats were in use:
- SQLite's `strftime('%Y-%m-%dT%H:%M:%fZ','now')` — exactly three fractional
digits.
- Go's `time.RFC3339Nano` — **variable** width, because it trims trailing zeros.
## The bug
`RFC3339Nano` produces `.5Z` for 500 milliseconds and `.55Z` for 550. Compared as
strings, `".5Z" > ".55Z"` — but `.500s < .550s`. The ordering is inverted.
This is **not** a cross-format problem. It is wrong even in a column written
100% by Go, because the fraction width varies within that column. Live data
showed 9-digit, 8-digit and 3-digit values in the same column:
```
jobs.run_after ...16:57:11.383879091Z ← 9 digits
jobs.run_after ...16:53:06.28925937Z ← 8 digits, trailing zero trimmed
media_items.added_at ...01:28:19.733Z ← 3 digits, SQLite default
```
It silently affected the **Continue shelf** (`MAX(updated_at)`) and **Most
Viewed** tiebreaks — both user-visible.
## Decision
All Go-side timestamp writes go through `core.Stamp`, which produces a
**fixed-width** format matching SQLite's own default exactly:
```go
t.UTC().Format("2006-01-02T15:04:05.000Z")
```
## Alternatives Considered
### Store Unix nanoseconds as INTEGER
- **Pros:** unambiguously sortable; no format question at all.
- **Cons:** every existing row and every query would need converting, and the
values stop being human-readable in the database — which matters when you are
debugging by hand.
- **Rejected:** too large a migration for a bug that a format change fixes.
### Store RFC3339 with a fixed 6 or 9 digits
- **Pros:** fixed width, more precision.
- **Cons:** would not match SQLite's default, so any row that ever took the
column default would still be a different width — the same bug in a new shape.
- **Rejected:** matching the database's own format is what makes both sources
comparable.
### Sort in Go instead of in SQL
- **Pros:** no format constraint.
- **Cons:** would require loading every row to sort, defeating the indexes.
- **Rejected:** the index does the sort; the format must cooperate.
## Consequences
- All 14 Go-side write sites use `core.Stamp` / `core.Now`. Verified: zero
variable-width writes remain.
- **Read paths were deliberately left alone.** `time.RFC3339Nano` parses both the
old and the new shapes, so existing rows keep working with no migration.
- A test proves the property directly: `Stamp` at .500s sorts **before** `Stamp`
at .550s as a string, and the output matches what SQLite's own default
produces, byte for byte.
- **The general rule:** if a value is ever compared in SQL, its textual form must
be fixed-width.

29
docs/decisions/README.md Normal file
View file

@ -0,0 +1,29 @@
# Decisions
Each record captures a decision that was expensive to reverse, the alternatives
that were rejected, and the reasoning. They exist so nobody has to re-litigate a
settled question — and so an agent does not "improve" something that was
deliberate.
Read the ones relevant to what you are changing.
| ADR | Decision | Rejected alternatives |
|---|---|---|
| [001](ADR-001-sqlite-with-a-seam.md) | SQLite, with a storage seam for later | PostgreSQL; fixing the write lock in code instead |
| [002](ADR-002-direct-play-only.md) | Direct play only, never transcode | On-demand CPU transcode; remux; pre-transcode |
| [003](ADR-003-mark-missing-never-delete.md) | A vanished file is marked, never deleted | Delete on scan; auto-purge; keep it invisible |
| [004](ADR-004-modular-monolith.md) | One binary with enforced package boundaries | Microservices; unenforced conventions |
| [005](ADR-005-one-item-access-chokepoint.md) | One item-access chokepoint, `internal/catalog` | Leave the duplication; a shared helper; a generic repository |
| [006](ADR-006-table-backed-job-queue.md) | The job queue is a table | Redis or a broker; in-memory goroutines |
| [007](ADR-007-fixed-width-timestamps.md) | Fixed-width timestamps via `core.Stamp` | Integer nanoseconds; six/nine digits; sorting in Go |
## Writing a new one
Add `ADR-00N-short-title.md` with **Status**, **Date**, **Context**, **Decision**,
**Alternatives Considered** (with the reason each was rejected) and
**Consequences**. Do not delete an ADR when a decision changes — write a new one
that supersedes it, so the history stays readable.
Two things make an ADR worth writing: the decision would be expensive to reverse,
and a reasonable person could have chosen differently. If neither is true, a
comment in the code is the right place.

10
internal/cli/cli.go
View file

@ -42,6 +42,16 @@ func run(ctx context.Context, args []string, cfg core.Config, log *slog.Logger,
return false, nil
}
// Create the data and config directories before opening the database.
//
// The server does this in Config.Validate, but the CLI is the way back in
// when someone is locked out, so it must work on a fresh install too —
// before the server has ever run and created them. Only the directories:
// an admin command must not fail because a media library is unmounted.
if err := cfg.EnsureDirs(); err != nil {
return true, err
}
db, err := core.OpenDB(cfg.DBPath)
if err != nil {
return true, err

15
internal/cli/cli_test.go
View file

@ -16,9 +16,22 @@ import (
// testConfig points a Config at a fresh database inside a temp dir. The CLI
// itself opens and migrates it, mirroring the auth_test.go setup pattern.
//
// DataDir and ConfigDir are set because the CLI creates them if missing — that
// is what lets `vellum user add` work on a fresh install, before the server has
// ever run. An empty dir here would be an unrealistic config.
func testConfig(t *testing.T) core.Config {
t.Helper()
return core.Config{DBPath: filepath.Join(t.TempDir(), "vellum.db")}
dir := t.TempDir()
data := filepath.Join(dir, "data")
return core.Config{
// DBPath lives INSIDE DataDir, exactly as LoadConfig derives it. That
// matters: it is why the CLI must create the directory, and it is what
// makes this config exercise the fresh-install path.
DBPath: filepath.Join(data, "vellum.db"),
DataDir: data,
ConfigDir: filepath.Join(dir, "config"),
}
}
func testLogger() *slog.Logger {

20
internal/core/config.go
View file

@ -94,15 +94,27 @@ func LoadConfig() (Config, error) {
return c, nil
}
// Validate creates the data and config directories if missing and checks that
// the configured media roots exist. A missing media root is an error, not a
// warning: silently scanning nothing is worse than failing loudly at boot.
func (c Config) Validate() error {
// EnsureDirs creates the data and config directories if missing.
//
// Separate from Validate because the CLI needs the directories but has no
// interest in the media roots: `vellum user add` must work on a fresh install,
// and must not fail because a library happens to be unmounted.
func (c Config) EnsureDirs() error {
for _, dir := range []string{c.DataDir, c.ConfigDir} {
if err := os.MkdirAll(dir, 0o750); err != nil {
return fmt.Errorf("create %s: %w", dir, err)
}
}
return nil
}
// Validate creates the data and config directories if missing and checks that
// the configured media roots exist. A missing media root is an error, not a
// warning: silently scanning nothing is worse than failing loudly at boot.
func (c Config) Validate() error {
if err := c.EnsureDirs(); err != nil {
return err
}
for _, dir := range c.MediaDirs {
info, err := os.Stat(dir)
if err != nil {

12
internal/httpapi/static_test.go
View file

@ -41,7 +41,12 @@ func findAsset(t *testing.T, suffix string) string {
t.Fatalf("walk dist: %v", err)
}
if found == "" {
t.Fatalf("no embedded asset ending in %q", suffix)
// A fresh clone has no built frontend: web/dist holds only .gitkeep, so
// the Go embed compiles and the server logs "web UI not embedded". The
// gzip behaviour cannot be exercised without a real asset, so skip
// rather than fail — the suite must pass on a clone that has not run
// `pnpm build`.
t.Skipf("no embedded asset ending in %q; run `pnpm build` in web/ to exercise this test", suffix)
}
return found
}
@ -54,7 +59,10 @@ func readAsset(t *testing.T, name string) []byte {
}
b, err := fs.ReadFile(sub, name)
if err != nil {
t.Fatalf("read %s: %v", name, err)
// Same reason as findAsset: without `pnpm build` the only thing in dist
// is .gitkeep, so a test that reads a real asset cannot run. Skip so the
// suite is green on a fresh clone.
t.Skipf("embedded %s not present; run `pnpm build` in web/ to exercise this test", name)
}
return b
}

10
web/dist/.gitkeep vendored Normal file
View file

@ -0,0 +1,10 @@
# Keeps web/dist present in a fresh clone.
#
# //go:embed all:dist in web/embed.go fails to compile if the directory is
# absent, so a clone with no frontend build could not run `go build` at all.
# With this file present the build succeeds and the server logs
# "web UI not embedded; run the frontend build to include it", which is the
# intended behaviour for someone who only wants the API.
#
# `pnpm build` overwrites dist/ with the real bundle; the Dockerfile builds the
# frontend first and overlays it.