mithral 051169a18a Documentation, and make a fresh clone actually work

Docs
- docs/ARCHITECTURE.md: how the pieces fit and why. The request path, the six
  enforced layers, the data model, the scanner, the job queue, direct-play
  playback, and the three protocol surfaces over one application.
- docs/decisions/: seven ADRs with the alternatives that were rejected —
  SQLite-with-a-seam, direct-play-only, mark-missing-never-delete, the modular
  monolith, the one item-access chokepoint, the table-backed queue, and
  fixed-width timestamps.
- README: the layout section was stale (it predated catalog, home, scanner,
  providers, engine, probe, playback, apitoken, arch and the dto packages) and
  now lists what exists, plus links to the new docs.

Two real bugs, both found by verifying the docs instead of trusting them

1. The CLI could not run on a fresh install. `Config.Validate` created the data
   and config directories, but the CLI never called it, so `vellum user add` —
   the recovery path when you are locked out — failed with "unable to open
   database file" until the server had booted at least once. Split the directory
   creation into `Config.EnsureDirs`, call it from the CLI, and keep the media
   root check out of it: an admin command must not fail because a library is
   unmounted. The CLI tests now point DBPath inside DataDir, exactly as
   LoadConfig derives it, so they exercise the fresh-install path; removing
   EnsureDirs fails three of them.

2. A fresh clone could not build at all. `//go:embed all:dist` fails to compile
   when web/dist is absent, and web/.gitignore expected a `dist/.gitkeep` that
   was never created. Added it, so a clone builds and the server logs
   "web UI not embedded; run the frontend build to include it" — the intended
   behaviour for someone who only wants the API.

Also: the httpapi tests depended on a built frontend and FAILED rather than
skipping, so the suite was red on any fresh clone. findAsset and readAsset now
skip with a message naming the missing step. They still RUN when a bundle is
present (verified: zero skips here), so they have not been quietly disabled.

Verified from a simulated clone of tracked files only: go build exit 0, 25
packages pass, the server boots and applies all 12 migrations, and the CLI
creates a user and takes a backup. gofmt clean, vet exit 0, svelte-check 0/0.

2026-10-12 00:18:11 -04:00

2.8 KiB

Raw Blame History

ADR-001: SQLite, not PostgreSQL — with a seam for later

Status

Accepted

Date

2026-10-12

Context

Vellum stores everything in one database: users, grants, the catalogue, per-user progress, the job queue and API tokens. The choice of engine determines the deployment story, the backup story, and how much concurrent writing is possible.

The original motivation for considering PostgreSQL was concurrency: SQLite allows only one writer at a time, and a library scan holding the write lock would block a playing client's progress save.

That problem turned out to be solvable without changing engines, and it was measured rather than assumed.

Decision

Use SQLite with WAL, and put a storage seam behind the one package that reads items, so the engine is swappable later if a real need appears.

Alternatives Considered

PostgreSQL

Pros: real concurrent writers; no single-writer bottleneck; mature.
Cons: a second process to run and back up; the deployment story stops being "one binary and one file"; the SQL is not portable — 186 statements, 193 ? placeholders, FTS5 full-text search, strftime, group_concat, rowid, and VACUUM INTO for backups all need reworking.
Rejected for now. The problem it solves was already fixed by other means (see below), and there is no measured write contention left.

Fixing the write-lock problem in code instead

Three changes removed the symptom without changing engines:

Chunked scan transactions. 500 groups per transaction instead of one transaction over the whole library. Measured: a 20,000-file scan reports partial progress and a concurrent progress save returns 200 throughout.
Four job workers with jittered idle waits, so a scan no longer stalls every other job.
Throttled session writes. last_seen was written on every authenticated request, including every audio Range request. Measured after the fix: 20 requests cause 0 session writes, down from 20.

Measured results after these changes: browse page 1 at 0.03 ms, the home screen at ~3 ms, and two concurrent 8,000-file scans interleaving successfully with every concurrent write returning 200.

Consequences

Deployment stays one binary plus one file. Backup is VACUUM INTO.
Writes serialise. This is fine at the measured scale and is the honest ceiling of this design.
internal/catalog exposes a Store interface with a SQLite implementation, so a PostgreSQL implementation would be a second Store rather than a rewrite.
The seam is guarded: internal/arch fails the build if a new package starts importing database/sql without being declared, or if a package already on the seam reaches for *sql.DB again.
When to revisit: multiple Vellum instances sharing a database, or write throughput beyond what chunked WAL handles. Neither is true today.

2.8 KiB Raw Blame History