internal/discovery registers three service types on the listen port:
_vellum._tcp for native clients, and _jellyfin._tcp / _subsonic._tcp
because those ecosystems already have clients that browse the LAN. That
last part is the point: Infuse, Jellyfin Media Player and Symfonium can
find Vellum with no client-side work, because they are already looking for
those names.
TXT records carry api, ver, proto, types, tls and (when set) pub, so a
client that resolves one can tell what it found before connecting.
Definitions is a pure function of config and Start only registers what it
returns, so what is advertised is testable without a socket.
Two behaviours worth knowing:
- VELLUM_DISCOVERY=off stops advertising. It accepts off/no/false/0;
strconv.ParseBool would have silently ignored "off".
- A loopback-only listen address advertises nothing. That is the normal
shape behind a reverse proxy, and advertising it would put a port in the
LAN browse list that refuses every client that dials it. Verified
red-green: defeating the guard fails TestDefinitionsSkipsLoopbackListen.
zeroconf is the only new dependency. It logs to the stdlib logger, so a
host with no multicast-capable interface prints to stderr outside slog.
Not verified: real mDNS on a real LAN. There is no multicast here, so the
wire behaviour of zeroconf is untested - only the definitions it is handed.
ClientIP used r.RemoteAddr and ignored X-Forwarded-For. That was right
about not trusting the header from an untrusted peer, but it is only half
the design: behind a proxy every request arrives from the proxy, so the
limiter key in handlers.go was "<proxy-ip>|<username>" for every client on
the internet. One attacker locked out a username for everyone, an attacker
rotating usernames never hit a limit, and sessions.ip recorded the proxy
so the device list was useless.
- VELLUM_TRUSTED_PROXIES is a CIDR list. X-Forwarded-For is honoured only
when RemoteAddr is inside it, and the chain is walked right-to-left so
the first address that is not itself a trusted proxy wins. Everything to
its left was appended by a client we do not control, so a spoofed
leftmost entry is discarded.
- VELLUM_PUBLIC_URL is the external origin. CookieSecure now defaults to
true when it is https://, so a TLS deployment that forgets
VELLUM_COOKIE_SECURE no longer ships a plaintext session cookie.
- A malformed CIDR fails the boot rather than being skipped: silently
dropping a proxy network widens the trust boundary, which is the exact
failure this setting exists to prevent.
ClientIP became a method on Authenticator so it can see the trusted set,
which is why handlers.go and main.go changed as well.
Verified red-green: neutering the trusted-peer path fails five ClientIP
tests, including the spoofed-leftmost case.
Known gap, pre-existing and not fixed here: this limiter still covers only
/api/v1/auth/login. Jellyfin AuthenticateByName and Subsonic auth call
auth.Authenticate with no limiting, so the same brute force is available
through those two surfaces.
Migrate only skipped migrations whose number was already applied (n <=
current). Point an older binary at a database a newer binary had already
migrated and it started happily, then read columns that had moved. For a
self-hosted app that is the worst failure mode: silent schema confusion
rather than a clean refusal.
Migrate now errors when PRAGMA user_version exceeds the highest embedded
migration, naming both numbers. Both callers already propagate the error,
so the binary exits at startup instead.
Verified red-green: removing the guard fails
TestMigrateRefusesDatabaseNewerThanBinary.
Docs
- docs/ARCHITECTURE.md: how the pieces fit and why. The request path, the six
enforced layers, the data model, the scanner, the job queue, direct-play
playback, and the three protocol surfaces over one application.
- docs/decisions/: seven ADRs with the alternatives that were rejected —
SQLite-with-a-seam, direct-play-only, mark-missing-never-delete, the modular
monolith, the one item-access chokepoint, the table-backed queue, and
fixed-width timestamps.
- README: the layout section was stale (it predated catalog, home, scanner,
providers, engine, probe, playback, apitoken, arch and the dto packages) and
now lists what exists, plus links to the new docs.
Two real bugs, both found by verifying the docs instead of trusting them
1. The CLI could not run on a fresh install. `Config.Validate` created the data
and config directories, but the CLI never called it, so `vellum user add` —
the recovery path when you are locked out — failed with "unable to open
database file" until the server had booted at least once. Split the directory
creation into `Config.EnsureDirs`, call it from the CLI, and keep the media
root check out of it: an admin command must not fail because a library is
unmounted. The CLI tests now point DBPath inside DataDir, exactly as
LoadConfig derives it, so they exercise the fresh-install path; removing
EnsureDirs fails three of them.
2. A fresh clone could not build at all. `//go:embed all:dist` fails to compile
when web/dist is absent, and web/.gitignore expected a `dist/.gitkeep` that
was never created. Added it, so a clone builds and the server logs
"web UI not embedded; run the frontend build to include it" — the intended
behaviour for someone who only wants the API.
Also: the httpapi tests depended on a built frontend and FAILED rather than
skipping, so the suite was red on any fresh clone. findAsset and readAsset now
skip with a message naming the missing step. They still RUN when a bundle is
present (verified: zero skips here), so they have not been quietly disabled.
Verified from a simulated clone of tracked files only: go build exit 0, 25
packages pass, the server boots and applies all 12 migrations, and the CLI
creates a user and takes a backup. gofmt clean, vet exit 0, svelte-check 0/0.
A 50k-file library used to wedge the database. This wave fixes the four
reasons why, plus a timestamp-ordering bug the work exposed.
Chunked scans (internal/library/scan.go)
- One transaction wrapped the whole reconciliation, holding SQLite's single
write lock for the entire scan. Now 500 groups per transaction, and the two
missing-file passes are batched too.
- Verified: a 20,000-file scan reports partial progress (scanned=1500 mid-scan)
and a concurrent progress save returns 200 throughout. Under the old code
that save blocked for the whole scan.
- Behaviour change: a scan is no longer all-or-nothing. A crash now leaves a
consistent partial scan. The scan is idempotent, so re-running finishes it.
- res.Errors is capped at 100 (a broken mount used to append one string per
file, then serialise the lot into the job row); the overflow is counted.
Concurrent job workers (internal/jobs/jobs.go)
- Run() was single-threaded, so a scan stalled every enrichment job behind it.
Now 4 workers with jittered idle waits.
- Handlers run concurrently, so Register/lookup take an RWMutex, and a
panicking handler is recovered and marked failed rather than killing a worker
and stranding a row in 'running'.
- RecoverStale() requeues jobs left 'running' by a dead process; called at boot
from main.go. Previously such a job stayed 'running' forever and the UI
polled it forever.
Explore (internal/media/discover.go)
- Full-scanned and sorted the whole library on every home load, and was
deterministic, so it showed the same 30 items every visit. Now a per-kind
fair-share id range scan with a random start, wrap-around, dedup and
interleaving. The doc comment claimed per-kind sampling before; now it does.
Indexes (migrations/0008_query_indexes.sql)
- Benchmarked with ANALYZE over 60k items: browse page 1 13.37ms -> 0.03ms
(484x, full scan -> covering index), deep page 8x, recently added 3.2x.
- A candidate index for the Continue shelf was built, measured and REJECTED:
the planner never chose it because `finished = 0` selects the majority of a
user's rows, so it had no selectivity and would have cost a write on every
progress save. Recorded in the migration rather than shipped as dead weight.
Canonical timestamps (internal/core/stamp.go)
- time.RFC3339Nano trims trailing zeros, so its fraction width varies and the
strings are not lexicographically sortable: ".5Z" > ".55Z" as strings though
.500s < .550s. That broke every ORDER BY <timestamp> and MAX(timestamp),
including the Continue shelf and Most Viewed. It was wrong even in columns
written 100% by Go, so it was not a cross-format issue.
- All 14 write sites now use core.Stamp, matching SQLite's own default format
exactly. Read paths are untouched: RFC3339Nano parses both shapes, so
existing rows keep working.
Also: busy_timeout 5s -> 15s. Chunked scans still delay a concurrent write, and
measured tail latency on a first scan of 15k new files reached 3.3s, leaving
almost no margin at 5s. Waiting beats failing with SQLITE_BUSY and losing a
playback position. A 10ms inter-batch yield was implemented, measured, found to
have no effect distinguishable from noise (overlapping ranges), and removed.
Verified end to end, not just compiled: orphan recovery logged at boot, scan
returns 202 in 0.7ms, Explore varies across loads, concurrent writes survive a
scan. gofmt/vet clean, 14 packages pass, -race clean on the queue, svelte-check
0/0, fresh install applies all 8 migrations.
The RomM replacement.
- registry: a games media type with ~35 ROM extensions across NES, SNES, GB/GBA,
N64, NDS, Genesis/Master System/Game Gear, PC Engine, Neo Geo Pocket,
WonderSwan, Atari, Coleco, Intellivision, and PlayStation.
- games: platform detection from the file extension, mapping each to an
EmulatorJS core. Platforms with no browser core are recognised but marked
not-playable so the UI offers a download instead of a broken player.
- migration 0005: game_saves keyed by (user, item, slot). Save states are user
data, not media — two people playing the same ROM keep their own progress.
- save endpoints with a 16 MiB cap and a validated slot name; the upload is
read through a LimitReader so an unbounded body is refused rather than
buffered.
- igdb provider: Twitch client-credentials flow with the token cached for its
~2 month lifetime, Apicalypse query body, and cover URLs normalised from
protocol-relative t_thumb to https t_cover_big.
- searchTitle: strips "(USA)", "[!]", "(Rev 1)" and similar release tags before
searching. "Sonic the Hedgehog (USA)" matched nothing; "Sonic the Hedgehog"
matches immediately. Stored titles keep the original.
- web: GamePlayer embeds EmulatorJS and adds Load/Save-to-server buttons.
Verified live against IGDB: Sonic matched (id 316900) with real cover art; a
4 KB save uploaded, listed, downloaded byte-identical, and deleted.
- comicvine provider for comics. ComicVine has no keyless tier, so the
provider is only registered when VELLUM_COMICVINE_API_KEY is set; without it
the server logs a note and comic enrichment completes as a no-op rather than
erroring.
- BaseURL is injectable so the response parsing is covered by a test against a
local server: candidate mapping, the required User-Agent, and the
status_code=100 invalid-key case (ComicVine returns HTTP 200 for that).
- descriptions are stripped of ComicVine's embedded HTML before storage.
Verified against a fake ComicVine server, and live that a comics library
enriches cleanly with no key configured.
- jobs: a table-backed queue in the same SQLite file. No Redis, no second
process, included in the same backup. claim() is a single UPDATE...RETURNING
so a job cannot run twice even with several workers. Failures back off
exponentially; a missing handler fails permanently rather than retrying a
bug.
- metadata: Provider interface plus Open Library and Google Books (both
keyless). The enricher tries providers in registration order, falls back on
empty, and continues past a provider error.
- migration 0004: metadata_fetch records the attempt, not just the match, so a
rescan does not re-hammer a rate-limited API for an item that will never
match. Errors are always retried; ok/notfound are cached for 30 days.
- enrichment writes provider_ids, updates title/year, links authors through
people/item_people, and downloads the cover as an asset. The cover download
runs outside the transaction: a slow image host must not hold a write lock.
- endpoints: queue enrichment per library, and inspect an item's matches.
- web: Enrich button per library.
Verified live against Open Library: "The Hobbit" matched /works/OL27482W,
title and year 1937 written, a 301x500 JPEG cover downloaded and served.
- migration 0003: an FTS5 index over media_items as an external-content table,
so the index stores no copy of the text. Triggers keep it in sync for every
write path, including the scanner's bulk inserts.
- migrate: splitStatements now tracks BEGIN...END depth, so CREATE TRIGGER
bodies survive as one statement. Without this, the trigger's internal
semicolons would tear it apart.
- search: every whitespace term is quoted before it reaches FTS5. Unquoted
input like a bare quote, a hyphen, or the word OR is valid FTS5 syntax and
would either error or silently change the query's meaning. Tested against a
list of hostile inputs, including a DROP TABLE attempt.
- scoped through the same VisibleLibraryIDs chokepoint; an empty library set
returns nothing, never everything.
- web: search box in the topbar and a results view.
Verified live: finds items across libraries, hostile input returns 200 with no
results, index stays in sync on insert/update/delete.
- library: CRUD with root-path resolution (absolute, symlink-free, must be
readable). A library name is unique; the root is validated at creation.
- registry: media types declare their extensions; the core never imports a
module. RegisterType is the extension point.
- scanner: walks a root, reconciles media_items/media_files in one
transaction, skips unchanged files by size+mtime, and marks vanished files
missing rather than deleting them (progress is the one thing here that
cannot be regenerated). A single unreadable file never fails a scan.
- migration 0002: `missing` flags on items and files, plus indexes.
- endpoints: media-types, library list/get/create/delete, per-library items,
and scan. Library reads go through the same grant chokepoint as everything
else; an unauthorised library 404s rather than 403s so its existence is not
leaked.
Multi-user from the first migration, as planned.
- auth: argon2id password hashing (PHC encoded, parameters travel with the
hash), opaque session tokens stored only as SHA-256, login rate limiting,
and account enumeration resistance (unknown user hashes anyway)
- sessions: cookie or Bearer auth, HttpOnly + SameSite=Lax, revocation on
password change
- grants: library_grants is the single access-control chokepoint. Admins see
every library; everyone else sees exactly their can_read grants
- bootstrap: VELLUM_ADMIN_USER/PASSWORD create the first admin only on an
empty database. No default password is ever invented.
- httpapi: mounts modules on a shared mux, session middleware innermost
- httpx: shared JSON helpers, strict body decoding
Gate: two users see different libraries. Tests cover that, plus session
expiry, token-at-rest, grant upsert, and the full login/me/logout flow.
Bootstraps Vellum as a single Go binary with one SQLite database.
- core: env-driven config, slog logging, SQLite (WAL) with pooled pragmas,
embedded migration runner keyed on PRAGMA user_version
- migrations/0001_core.sql: identity, media, user state, infra. Media items
are global; all per-user state (progress, favorites, grants) is a join.
One polymorphic progress table covers every media kind.
- httpapi: request-ID, structured request logging, panic recovery, JSON
endpoints for /healthz and /api/v1/version
- multi-arch Dockerfile (pure-Go SQLite, CGO_ENABLED=0) with an entrypoint
that honours PUID/PGID like the rest of the self-hosted stack
- tests for migration idempotency, FK enforcement, WAL, config, middleware
Media does not exist yet. Nothing is exposed beyond health and version.
