The server could transcode and serve HLS; nothing could play it. Chrome has no
native HLS support, so without this the feature existed only for curl.
hls.js is the only real option for HLS in Chrome, and it is MIT. It is loaded
with a dynamic import, so the ~185 KB gzip is paid only when a transcode
actually happens - a direct play, which is the common case, fetches none of it.
Safari and iOS play HLS natively, so those get the playlist URL directly and
never load the library at all.
Three failure paths all fall back to the direct file URL rather than leaving a
dead player: hls.js not supported, the dynamic import failing, and a fatal
playback error. Transcoding is an optimisation and must never be able to break
playback - a file that would have direct-played has to keep working when the
encoder or the library is unavailable.
The instance is destroyed on source change and on unmount, so seeking between
files does not leave encoders or MediaSource objects behind.
Verified: pnpm check 0/0, the production build succeeds, and hls.js is emitted
as its own lazy chunk rather than folded into the main bundle.
Not verified in a browser - there is no automated browser test in this repo, so
the actual playback of a transcoded stream is unproven end to end. The server
side of that path is verified separately against real hardware.
Fourth and final step of ADR-008's server half. When nothing the client
declared can direct-play a file and this host can re-encode, resolve now
answers playable with engine "transcode". The client sees a string it already
understands, so the frozen JSON contract did not change.
The ordering is the whole point and is asserted directly: transcoding sits
after the direct-play loop, so a file that plays directly never pays for it.
Re-encoding something a client could already decode would burn the GPU and
lose quality for nothing, which is the failure mode ADR-008 rejects by
choosing direct play first. Verified red-green - hoisting the check above the
loop fails with "direct play must always win".
The capability is resolved once at boot and passed in, not read inside a
request: detection spawns ffmpeg to prove the encoder works, and that must
never happen on the request path. It is also passed rather than read from the
transcode package so the tests can drive both answers without hardware.
TranscodeID is deliberately not a registry entry. Every other engine is
something a client declares; this one is the server's own capability, so it
is not advertised in /playback/engines and a client must not name it.
On a host with no usable encoder canTranscode is false and every answer is
byte-identical to what it was before any of this existed - which is this
deployment, and is why ADR-002 still holds for it.
Third step of ADR-008. Two routes inside the item family: the playlist starts
the session lazily and serves what ffmpeg has written so far, so playback can
begin while the rest still encodes; the segment route serves one .ts.
Segment names are validated against the exact shape ffmpeg writes before being
used as a path. Verified red-green: removing the check serves a file from
outside the session directory with a 200.
Two corrections made in review.
1. The manager was being built inside NewHandlers, which meant nothing could
ever shut it down. Its reaper would run for the process lifetime and a
running ffmpeg would outlive the server that started it. It is now created
in main.go beside the job queue and deferred, and passed in - the same
ownership the queue already has.
2. A playlist request could 404 before ffmpeg wrote the file, which a native
HLS player treats as fatal rather than retrying. The playlist handler now
waits up to four seconds for the first write. Segments are never waited for.
Verified end-to-end against a running server with real hardware: a 30-second
file produced a valid EVENT playlist in about a second (the wait doing its
job - no 404), a 658 KB segment with Content-Type video/mp2t, a 404 for a
traversal attempt, and a 401 unauthenticated.
Also confirmed at boot: all four metadata providers report their disabled
state and name their env var, including the new adult one.
Second step of ADR-008. A session spawns ffmpeg with the detected encoder's
args, packages the file as HLS, and is reused while it is alive so two range
requests do not start two ffmpegs.
Three things it has to get right, and all three are about not burning the GPU:
a session nobody is watching is reaped after a few minutes idle; concurrent
sessions are capped rather than queued, because a GPU encodes one or two
streams and a queue would just make everyone wait longer; and Shutdown stops
everything so a restart does not leave ffmpeg running.
Found a real bug in review, and it is the kind that only a real host catches.
The hardware args were placed BEFORE -i. ffmpeg reads options before the input
as INPUT options, so "-c:v h264_nvenc" was looked up as a decoder and failed
with "Unknown decoder 'h264_nvenc'". Transcoding would have produced nothing on
any NVENC host while looking correct in the code.
Verified against the real hardware rather than reasoned about: the same ffmpeg
command produces zero segments with the args before -i and a valid playlist and
segments with them after. The test now asserts the ordering directly and names
the reason, so a future reordering fails with an explanation instead of two arg
lists. Red-green: moving them back fails with "the hardware args must follow
the input".
Verified end-to-end on this host: the Manager started a session against a real
20-second file and produced a 132-byte EVENT playlist and segments.
Known limit: a stopped session leaves its segment directory on disk, so the
cache directory needs pruning out of band.
First step of ADR-008. internal/transcode answers one question: can this host
hardware-transcode, and with which encoder.
The rule the package exists to enforce is that a device node is not proof. A
node can exist while the driver cannot encode, and that produces the worst
outcome ADR-008 describes: a transcode that starts, looks like it works, and
then cannot keep up. So every candidate runs a real two-frame probe encode and
is dropped unless that encode exits cleanly.
This host is a perfect illustration and the reason the rule is not theoretical:
it has /dev/dri/renderD128 AND Nvidia nodes. The render node exists, but VAAPI
fails to initialise against it ("Failed to initialise VAAPI connection"). A
detection that trusted the node would have picked VAAPI and been wrong. The
probe rejected it and selected NVENC, which was verified independently with
ffmpeg outside this code.
The probe is injected, so every branch is testable with no hardware present,
including the important one: device present, probe fails. Verified red-green -
ignoring the probe result fails TestDetectProbeFails with "got vaapi, want no
capability".
One real bug found while proving it: the probe used testsrc=size=64x48, below
NVENC's minimum frame size, so a host with working NVENC would have reported no
capability at all. Raised to 320x240, still two frames.
Also added internal/transcode to the arch layer map as a layer-1 primitive
beside probe and engine, which the arch rule caught immediately.
Not wired into the engine registry or playback yet - that is the next step.
The API could serve tracks but nothing asked for them. The picker lists what
the item has and attaches one <track> at a time, removing the previous element
first so switching never accumulates them.
Default is Off. Auto-selecting a track would turn subtitles on for people who
did not ask and cannot tell why, and the failure mode is worse than a missing
feature: a track that loads over a film you did not want subtitled.
Subtitles never block playback. A failed track list hides the picker, and a
track that fails to load leaves the video playing - the media element's own
state is never touched by any of this.
Not verified by a test: the web package has no test runner, only svelte-check,
so verification here is the type and a11y check (0 errors, 0 warnings) plus
the API end-to-end below.
Verified end-to-end against a running server: a video with "Show.en.srt" and
"Show.pt-BR.srt" beside it produced two sidecar rows with the right languages,
the list route returned both, and the track route served real WebVTT - header,
Content-Type text/vtt, and the SRT commas converted to dots.
Two halves of the same feature: the scanner finds tracks, the media layer
serves them as WebVTT.
Discovery walks each directory for .srt/.ass/.ssa/.vtt/.sub beside a video and
matches them by base name, including the language form "Movie.en.srt" and
"Movie.pt-BR.srt". A sidecar is never itself an item. Matching is a dot-guarded
prefix on the real video base name, so a dotted release name like
"The.Matrix.1999.1080p.en.srt" does not read the year as a language.
Serving adds two routes inside the existing item family, both grant-checked
through catalog.Store (ADR-005) so forbidden and absent are the same 404.
Browsers play WebVTT and nothing else, so .srt is converted (header plus the
comma-to-dot in timestamps), .vtt passes through, and embedded streams and
.ass/.ssa go through ffmpeg. Conversions are cached under ConfigDir, because
converting per request would spawn an ffmpeg on every seek. A track that fails
to convert fails alone - the video and the other tracks are unaffected.
Subtitle rows are a cache of what is on disk, so a vanished sidecar has its row
deleted. That is the opposite of a vanished video, which is marked instead
(ADR-003) because progress hangs off it and cannot be regenerated.
Verified red-green: disabling the sidecar call fails all four scanner tests,
and removing the VTT header fails the conversion test.
Known limits: a "Movie.forced.en.srt" records "forced.en" rather than "en",
and the conversion cache is keyed by track id, so a sidecar replaced under an
unchanged row serves the stale conversion until a rescan recreates the row.
Vellum had no subtitle support of any kind. This is the storage half.
Three sources, and they are genuinely different shapes: a sidecar file beside
the video has a path, an embedded stream has an index into the container and
no file of its own, and a downloaded track has a path but is regenerable.
stream_index is -1 for the file-backed sources rather than NULL, and that is
deliberate: NULL is never equal to NULL in SQL, so a unique index over a
nullable column would happily accept the same sidecar twice. With -1 the
UNIQUE (item_id, source, path, stream_index) actually does its job and a
rescan can upsert.
Nothing here is authoritative about the media. Deleting every row and
rescanning reproduces it exactly, which is what lets a vanished subtitle file
have its row removed - unlike a vanished video, which is marked instead
(ADR-003) because progress hangs off it and cannot be regenerated.
Verified on both paths: a real database at version 13 upgraded to 14 by an
older binary-then-newer sequence, and a fresh database applied 0001..0014.
The adult kind scanned and browsed but could never be enriched, because no
provider claimed it. ThePornDB does: it is the catalogue the Stash ecosystem
uses for scenes and movies, and it is keyed the same way IGDB and TMDB are, so
it registers only when VELLUM_TPDB_API_KEY is set and otherwise logs the env
var at boot.
The API shape came from the live OpenAPI spec rather than the docs, which is
where the search parameter lives: it is `parse`, not `q` or `title`, and the
endpoint is /scenes with a `data` envelope.
Not verified against the live API: it needs a token I do not have, so the
tests run against httptest only. The field names are taken from the published
schema, which is the best available substitute.
The provider also does not use ThePornDB's hash lookup, which is the more
accurate way to match: it needs a file hash computed at scan time, and
matching by parsed title and date works today without that.
AniList is free, needs no key, and carries an isAdult filter, which is what
makes it the right source for this: the same catalogue holds mainstream anime
and the filter is how a query says which side it wants. It fills the gap left
by the adult kinds having no provider at all.
The API facts here were tested against the live service rather than read off
the docs, and one of them would have been wrong otherwise: the year filter is
startDate_like and it needs a wildcard. "2001" returns zero results; "2001%"
returns the right one. A bare year looks correct and silently finds nothing.
Descriptions come back with HTML in them even with asHtml:false, so they are
tag-stripped and entity-decoded before being stored. That reuses the existing
stripHTML from the ComicVine provider rather than adding a second one.
Verified against the live API, not only against a mock: a real search for
"Bible Black" returns id anime/368, year 2001, the correct cover URL, and a
description with no tags in it.
Kinds() is ["hentai"] only. AniList would serve mainstream anime too, but that
is a library kind nobody has asked for yet, and claiming tv here would mean
two providers racing for the same items.
One "video" kind meant every video library behaved the same, so a film shelf
tried to parse episode numbers and a TV library could not be told apart from
it. The kinds are now a declaration rather than a special case.
Five video kinds, all owning the same containers and differing only in how
their files become items:
movies flat tv series/season/episode
adult flat hentai series/season/episode
video series/season/episode (kept: existing libraries already use it)
The scanner dispatches on library.GroupModeFor(lib.Kind) instead of comparing
the kind to "video", so a new episodic kind is a registry entry and nothing in
the scanner changes.
The decision worth recording, because it could have gone the other way:
media_items.kind stays STRUCTURAL. Every video library stores 'video' on its
rows, not its own kind. media_items.kind answers "is this a playable video",
which is what the playback, catalog and Jellyfin paths actually ask, and
migration 0013 already committed to that rule when it decided an episode would
stay kind='video'.
Storing the library kind instead would have meant teaching every query that
says kind='video' about five new values - catalog's episode and next-episode
reads, Jellyfin's episodes-only filter, the item-type mapping, the web
player's routing - and every kind added later would have had to do it again.
So the two jobs are split: the ROW kind says what a thing is, the LIBRARY kind
says how it is classified, and the library kind is what routes metadata. The
enricher now reads the library to choose providers, because a movies library
and an adult library hold identical rows and must not query the same
catalogue.
The adult kinds have no metadata provider. There is no mainstream catalogue
for that content - the ecosystem uses StashDB or local NFO files - so they
scan and browse and stay unenriched until someone registers one. That is
stated in the registry rather than left as a surprise.
Verified end-to-end: the same tree under a movies library stays flat, under a
tv library builds series/season/episodes, and every row in both is stored as
kind='video'. The UI needs no change - it reads /api/v1/media-types, so the
new kinds appeared in the picker by themselves.
Red-green: dispatching on the literal kind fails the tv and hentai cases;
storing the library kind on rows fails the structural-kind assertion.
The scanner builds library -> series -> season -> episode; this makes it
reachable. Three reads, all grant-checked through catalog.Store (ADR-005):
a series' seasons, a season's episodes, and the next unwatched episode.
Each loads the parent through ItemFor first, so forbidden and absent are the
same ErrNotFound, and the child query is scoped to the parent's own
library_id so a corrupt parent_id cannot cross a library boundary. Verified
red-green: swapping ItemFor for Item returns two rows to an ungranted
caller.
Episodes come back in episode order, not insertion order, and the ordering
is verified red-green too - dropping the clause yields [3 1 10 2].
Routes stay inside the existing family: /api/v1/items/{id}/seasons,
/episodes and /next. No parallel route surface.
Jellyfin gains SeriesDto, SeasonDto and EpisodeDto. Jellyfin's wire type is
BaseItemDto, so these carry the folder/file split plus IndexNumber and
ParentIndexNumber. ItemType now takes the item rather than the kind, because
an episode IS kind video - that is the one signature change, and the dto
tests were updated with it.
Verified end-to-end against a running server: a library holding S01E02,
1x02, a bare-number file and a film produced two series, three seasons and
six episodes, with the film left alone; the routes returned the seasons in
order, the episodes in episode order, and the next unwatched episode.
Note for later: Subsonic and the web UI are untouched, so a series browses
there as it did before. The scanner also has no separate 'tv' kind, so a
video library mixing films and shows builds both shapes side by side.
parseEpisode was handed the file's whole relative path, but the bare-number
pattern is anchored at the start of a name. Handed "Show/Season 3/06 -
Title.mkv" it could never match, so every file named that way was filed as a
standalone film at the library root instead of an episode.
Found by running a real scan against a real tree, not by reading the code:
the unit test on parseEpisode passed a bare file name, which is exactly the
input the call site was not providing. The unit test was right and the
caller was wrong, which is a class of bug a helper test cannot see.
parseEpisode now gets path.Base(rel), matching episodeTitle two lines below
it, so the function really is a function of a file name and its directory as
its signature and doc already claimed.
Both scan-level cases are added to the tree test, so the call site is now
covered and not just the helper. Verified red-green: reverting the call site
fails "bare leading number under a Season directory".
The schema landed in 8d6ea6d; this fills it. A video library now builds
library -> series -> season -> episode, with episodes keeping kind='video'
and the two parents standing in for directories that have no file of their
own.
Three naming shapes are recognised, in order:
- a full marker, S01E02 or 1x02
- an episode-only marker, E05 / ep5 / Episode 5, which older rips and anime
use. With no season directory the season is 1, which is what those
libraries mean by it.
- a bare leading number, "05 - Title.mkv", trusted only under a season
directory. Without that anchor it is indistinguishable from a film whose
name starts with a number.
Every marker is delimited by non-alphanumeric characters, which is the guard
that stops "Blade.Runner.1999x264.mkv" reading as season 99 episode 264 -
the failure every release-tagged file would otherwise hit. The episode-only
pattern caps at three digits for the same reason: four would let
"Movie.E.1999.mkv" read as episode 1999, and years have four digits while
episode numbers do not.
Idempotency comes from ON CONFLICT (library_id, path) rather than a lookup,
so a duplicate series row is impossible rather than merely unlikely, and a
rescan leaves the tree byte-identical.
Verified red-green: neutering parseEpisode fails the episode tests, and
removing the episode-only branch fails the four patterns it covers.
Known limits, marked in the code: a multi-episode file (S01E02E03) parses
as its first episode only, and library has no separate 'tv' kind, so a video
library mixing films and shows builds both shapes side by side.
A bare-number episode keeps the cleaned file name as its title rather than
the text after the number. TV metadata replaces the title once the series
lookup lands, so it is not worth guessing at now.
Vellum had no season or episode model. media_items was a flat tree and the
scanner never parsed S01E02, so a library of television was a list of files
in a folder. That is the difference between a movie server and a media
server, and it is also what blocked TV metadata: there was nothing to
attach a series' poster or overview to.
Schema only. The scanner and the API follow in separate commits, so this
one is safe on its own: the columns are additive and nullable, every
existing row is untouched, and a film keeps NULL for both numbers.
Two decisions, stated in the migration because both are cheap now and
expensive later:
- An EPISODE STAYS kind='video'. A new 'episode' kind would mean touching
every query that says video for no gain, since the parent chain already
tells an episode from a film. kind only gains 'series' and 'season'.
- A series and a season have no file, so their path is the directory they
stand for. That satisfies the existing UNIQUE (library_id, path) and
gives the row a stable identity so the scanner can upsert instead of
duplicating a series on every scan.
The tree itself is not new: parent_id already carries albums and audiobook
chapters. This is that same shape applied to television.
Verified on both paths, because only the upgrade can hurt: a database at
version 12 upgraded cleanly to 13 with the columns and both indexes added,
and a fresh database applied 0001..0013 cleanly.
Five providers existed and none covered the "video" kind, so every movie
and show got a title from its filename and nothing else - no poster, no
plot, no cast, no year. That is the gap between "a folder of videos with a
nice web UI" and a media server.
TMDB keeps films and television in separate namespaces behind separate
endpoints, and a Query does not say which one it is. Rather than guessing
from the title, Search asks both at once and merges by TMDB popularity,
which is the only ranking signal available without a second request per
item.
Speed is the point, so the two requests run concurrently over one derived
context on a single shared client, and whichever fails first cancels the
other: one dead endpoint cannot hold the result hostage while the healthy
one is waited out. Verified red-green - serialising the two fetches makes
TestTMDBSearchesConcurrently fail at 2s.
Two corrections made in review, both found by reading the code rather than
trusting the report:
1. ExternalID was the bare TMDB id, so film 603 and show 603 - different
works - would have been stored identically in provider_ids. Namespaced
as movie/603 and tv/603. Nothing resolves an item from an external_id
today, so this was latent rather than broken, but the table is populated
as enrichment runs and correcting it later would have needed a data
migration. Pinned by TestTMDBExternalIDIsNamespaced.
2. Query.Year was added but never populated, so the year filter - the main
way to tell a remake from the original - never applied. catalog.Item
already carried Year; the enricher now passes it.
No cover bytes are fetched here: the provider returns image URLs and the
existing enricher caches what it uses. A missing key logs the env var and
skips, so the feature is discoverable from the boot log.
Not verified: the live TMDB API. Tests run against httptest servers only.
ADR-002 decided never to transcode because the server is a headless,
GPU-less Xeon, and its measurements still hold. The problem is scope:
"this server cannot transcode" became "Vellum does not transcode", and
only the first is forced by the hardware.
Intel Quick Sync has been in most desktop CPUs since 2015, NVENC is in
every recent Nvidia card, and VAAPI covers AMD and Intel on Linux. Capping
the product at the hardware in one closet is the wrong constraint, and the
gap is visible: a client that cannot decode HEVC gets nothing, and in
Chrome that is not even a clean refusal - the video never decodes and
audio plays over a blank frame.
So the engine is added and its registration is gated:
- registered only when usable transcode hardware AND ffmpeg are both present
- off by default; direct play stays the default and the preference
- consulted only when no other engine can direct-play the file, which is
exactly the case that returns "no engine can play this" today
- hardware-accelerated only. No CPU fallback: that is the stall ADR-002
measured, and it fails slowly, which is worse than refusing.
This deployment is unchanged. On a GPU-less Xeon the engine is never
registered and the path is never reached, so ADR-002 holds as written.
ADR-002's status now points here rather than being deleted, per the rule
that a changed decision gets a new record instead of an edited one.
internal/discovery registers three service types on the listen port:
_vellum._tcp for native clients, and _jellyfin._tcp / _subsonic._tcp
because those ecosystems already have clients that browse the LAN. That
last part is the point: Infuse, Jellyfin Media Player and Symfonium can
find Vellum with no client-side work, because they are already looking for
those names.
TXT records carry api, ver, proto, types, tls and (when set) pub, so a
client that resolves one can tell what it found before connecting.
Definitions is a pure function of config and Start only registers what it
returns, so what is advertised is testable without a socket.
Two behaviours worth knowing:
- VELLUM_DISCOVERY=off stops advertising. It accepts off/no/false/0;
strconv.ParseBool would have silently ignored "off".
- A loopback-only listen address advertises nothing. That is the normal
shape behind a reverse proxy, and advertising it would put a port in the
LAN browse list that refuses every client that dials it. Verified
red-green: defeating the guard fails TestDefinitionsSkipsLoopbackListen.
zeroconf is the only new dependency. It logs to the stdlib logger, so a
host with no multicast-capable interface prints to stderr outside slog.
Not verified: real mDNS on a real LAN. There is no multicast here, so the
wire behaviour of zeroconf is untested - only the definitions it is handed.
Many Jellyfin clients probe GET /emby/System/Info/Public during their
handshake. There was no /emby prefix anywhere in the tree, so that 404ed
and the client abandoned discovery and fell back to its manual "enter your
server URL" screen - the exact experience mDNS exists to remove.
Register installs every pattern twice through a mount helper, with the same
handler value both times, so the alias cannot drift from the root surface.
The JSON 404 fallbacks get /emby forms too, plus one subtree handler,
because static.go's isAPIPath knows only /healthz and /api/ and an unknown
/emby/... path would otherwise be served the SPA shell.
No new endpoint, no handler behaviour changed, and the bare root paths are
untouched: Jellyfin's own clients use them.
The limiter covered only /api/v1/auth/login. Jellyfin's AuthenticateByName
and Subsonic's authenticate both called auth.Authenticate with no limiting,
so an attacker brute-forced through those instead: the same password, the
same argon2, no ceiling.
The limiter moves from auth.Handlers onto *Authenticator, which all three
surfaces already hold, so they share one budget and splitting guesses
across surfaces gains nothing. main.go builds one authenticator and hands
it to both compat layers, so the sharing is real rather than intended.
Subsonic needs care: it has no login step, authenticate runs on every
request, so a naive check would charge playback against the login budget
and lock a listener out mid-album. The limiter is consulted only when a
credential was actually supplied, and a success or an uncredentialed
request spends nothing.
Verified red-green on all three: removing the Jellyfin gate fails
TestAuthenticateByNameIsRateLimited with 401 instead of 429; removing the
Subsonic credential guard fails
TestSuccessAndUncredentialedDoNotConsumeTheBudget; neutering AllowLogin or
ResetLogin fails the auth tests.
Known gap, unchanged: the limiter is in-process, so it resets on restart
and does not span replicas. Fine for one binary against one SQLite file.
Gitea Actions is enabled on the instance and the workflow file was
correct, but no act_runner is registered: the push produced zero workflow
runs and the job would queue forever. A workflow that cannot run is worse
than none, because it implies pushes are gated when they are not.
`make gate` runs the same checks locally and is verify-only - gofmt -l must
print nothing and the file is never rewritten. The existing `check` target
stays and keeps its rewriting behaviour; its comment now says so, because
conflating a fixer with a gate is how an unformatted file reaches main.
The workflow is in history at 43116fa if a runner ever appears.
Without a license, default copyright applies and nobody may legally run,
modify or redistribute this. MIT chosen for maximum adoption and minimum
friction.
Copyright holder is "Vellum contributors" - edit if you want a real name.
ClientIP used r.RemoteAddr and ignored X-Forwarded-For. That was right
about not trusting the header from an untrusted peer, but it is only half
the design: behind a proxy every request arrives from the proxy, so the
limiter key in handlers.go was "<proxy-ip>|<username>" for every client on
the internet. One attacker locked out a username for everyone, an attacker
rotating usernames never hit a limit, and sessions.ip recorded the proxy
so the device list was useless.
- VELLUM_TRUSTED_PROXIES is a CIDR list. X-Forwarded-For is honoured only
when RemoteAddr is inside it, and the chain is walked right-to-left so
the first address that is not itself a trusted proxy wins. Everything to
its left was appended by a client we do not control, so a spoofed
leftmost entry is discarded.
- VELLUM_PUBLIC_URL is the external origin. CookieSecure now defaults to
true when it is https://, so a TLS deployment that forgets
VELLUM_COOKIE_SECURE no longer ships a plaintext session cookie.
- A malformed CIDR fails the boot rather than being skipped: silently
dropping a proxy network widens the trust boundary, which is the exact
failure this setting exists to prevent.
ClientIP became a method on Authenticator so it can see the trusted set,
which is why handlers.go and main.go changed as well.
Verified red-green: neutering the trusted-peer path fails five ClientIP
tests, including the spoofed-leftmost case.
Known gap, pre-existing and not fixed here: this limiter still covers only
/api/v1/auth/login. Jellyfin AuthenticateByName and Subsonic auth call
auth.Authenticate with no limiting, so the same brute force is available
through those two surfaces.
Nothing was gated on a push. .gitea/workflows/ci.yml runs gofmt, build,
vet, test and pnpm check on push and pull_request.
web/dist/.gitkeep is tracked on purpose so `go build ./...` succeeds
without a frontend build, so the Go job needs no Node step. The web job
pins pnpm 12: pnpm/action-setup reads packageManager from a root
package.json that does not exist here, and pnpm 10 rejects the allowBuilds
key in web/pnpm-workspace.yaml.
Not verified against a real runner - the first push is the test.
Migrate only skipped migrations whose number was already applied (n <=
current). Point an older binary at a database a newer binary had already
migrated and it started happily, then read columns that had moved. For a
self-hosted app that is the worst failure mode: silent schema confusion
rather than a clean refusal.
Migrate now errors when PRAGMA user_version exceeds the highest embedded
migration, naming both numbers. Both callers already propagate the error,
so the binary exits at startup instead.
Verified red-green: removing the guard fails
TestMigrateRefusesDatabaseNewerThanBinary.
Docs
- docs/ARCHITECTURE.md: how the pieces fit and why. The request path, the six
enforced layers, the data model, the scanner, the job queue, direct-play
playback, and the three protocol surfaces over one application.
- docs/decisions/: seven ADRs with the alternatives that were rejected —
SQLite-with-a-seam, direct-play-only, mark-missing-never-delete, the modular
monolith, the one item-access chokepoint, the table-backed queue, and
fixed-width timestamps.
- README: the layout section was stale (it predated catalog, home, scanner,
providers, engine, probe, playback, apitoken, arch and the dto packages) and
now lists what exists, plus links to the new docs.
Two real bugs, both found by verifying the docs instead of trusting them
1. The CLI could not run on a fresh install. `Config.Validate` created the data
and config directories, but the CLI never called it, so `vellum user add` —
the recovery path when you are locked out — failed with "unable to open
database file" until the server had booted at least once. Split the directory
creation into `Config.EnsureDirs`, call it from the CLI, and keep the media
root check out of it: an admin command must not fail because a library is
unmounted. The CLI tests now point DBPath inside DataDir, exactly as
LoadConfig derives it, so they exercise the fresh-install path; removing
EnsureDirs fails three of them.
2. A fresh clone could not build at all. `//go:embed all:dist` fails to compile
when web/dist is absent, and web/.gitignore expected a `dist/.gitkeep` that
was never created. Added it, so a clone builds and the server logs
"web UI not embedded; run the frontend build to include it" — the intended
behaviour for someone who only wants the API.
Also: the httpapi tests depended on a built frontend and FAILED rather than
skipping, so the suite was red on any fresh clone. findAsset and readAsset now
skip with a message naming the missing step. They still RUN when a bundle is
present (verified: zero skips here), so they have not been quietly disabled.
Verified from a simulated clone of tracked files only: go build exit 0, 25
packages pass, the server boots and applies all 12 migrations, and the CLI
creates a user and takes a backup. gofmt clean, vet exit 0, svelte-check 0/0.
Phase 3 built the seam; nothing stopped it being undone one convenient import at
a time. Two rules, both proven able to fail.
TestDatabaseSQLImportersAreDeclared freezes the blast radius of database/sql.
Twenty packages import it today and that set is now explicit. A new package that
needs its own SQL must be added deliberately, with a reason. The rule also fails
in the other direction — if a listed package stops importing database/sql, the
list must shrink, so it cannot rot into a meaningless allowlist.
TestThreadedConsumersStayOffSQL is the concrete regression guard: internal/
playback was moved onto catalog.Store, and this fails if it reaches for *sql.DB
again.
Both checks read non-test files only, so a package that needs database/sql
purely to build a test database is not forced onto the list.
Proven by breaking each on purpose:
- dropping internal/media from the list -> "imports database/sql but is not
in sqlImporters"
- adding internal/natsort (no SQL) -> "is listed ... but no longer
imports database/sql"
- re-adding database/sql to playback -> "undoes the seam"
All restored afterwards and the files verified unchanged.
Seven arch rules now, each able to fail. gofmt clean, build 0, vet 0, 25
packages pass.
EXPAND ONLY, per the migration order (expand, migrate, verify, contract). No
Postgres implementation, no behaviour change.
internal/catalog now exposes:
type Store interface {
Item / ItemFor / Files / FilesFor / FileByID / FirstFile / FileCount
}
type SQL struct{ db *sql.DB }; func New(db *sql.DB) *SQL
The seven bodies are the old functions moved verbatim. ItemFor still returns the
same ErrNotFound for absent and forbidden, so the security property is
unchanged. A compile-time `var _ Store = (*SQL)(nil)` keeps the implementation
honest.
Deliberately NOT moved: browse lists, home shelves and subsonic album queries.
Those are each package's own presentation concern, and pulling all of them into
the seam is exactly the 186-statement rewrite the seam exists to avoid. The
package doc says so.
Proven substitutable, not just declared: store_test.go implements a fake Store
in memory with no database at all and drives a caller through the interface. A
seam nobody can substitute proves nothing.
The agent's own risk note was right and I acted on it: the seam only pays off
once something actually threads it, and initially nothing did. So internal/
playback — the smallest consumer — now takes a catalog.Store instead of *sql.DB
and is wired with catalog.New(db) at the composition point. It has no database
reference left outside a comment. The remaining consumers (media, jellyfin,
subsonic, metadata) still call the package-level wrappers; threading them is
mechanical, and I would rather do it deliberately than in one sweep.
Verified: gofmt clean, build exit 0, vet exit 0, 25 packages pass, arch rules
pass. End to end against a running server: /playback/resolve still returns
vcodec:hevc for html5 and engine mpv when mpv is offered, an unknown item 404s,
/home and Jellyfin /Items still answer.
Known gap, unchanged by this commit: internal/arch does not enforce "new code
must not take *sql.DB", so the seam can erode by drift. That rule would be the
natural next step if this is to hold.
internal/jellyfin is the most client-visible surface in the tree — real Jellyfin
apps parse its JSON — and it had zero test files. Phase 2 had just moved its
DTOs, so this closes the gap before the next refactor.
Coverage: internal/jellyfin 0.0% -> 35.7%, internal/jellyfin/dto -> 68.9%.
The tests, and why each exists:
- Credential forms. X-Emby-Token, X-MediaBrowser-Token, ?api_key= and
Authorization: MediaBrowser Token="...". The query form matters most: <video>
and <audio> elements cannot set headers, so a client that can only use
?api_key= must still authenticate.
- Grant scoping. A user with no grants sees zero items, and an item they cannot
read returns a response BYTE-IDENTICAL to one that does not exist. A 403 would
confirm the library is there and leak its existence. The test asserts the two
responses are indistinguishable, not merely both non-200.
- The transcode refusal. A client that disables both direct play and direct
stream is demanding a transcode, and gets ErrorCode NoCompatibleStream rather
than a TranscodingUrl that would never resolve. The normal path is asserted to
advertise SupportsTranscoding FALSE, because a client told otherwise waits
forever for a stream that never starts.
- Streaming with ONLY ?api_key= plus a Range header returns 206 with a correct
Content-Range and the right bytes, and a token cannot stream a file it cannot
read.
- AuthenticateByName mints a token that then authenticates, and returns no token
for bad credentials.
- DTO mapping: exact JSON key set, RunTimeTicks as ms x 10000, id round-trip,
ImageTags only with a cover, and golden JSON for both the item and the media
source.
Both halves are proven able to FAIL, because a test that cannot fail is
decoration:
- flipping SupportsTranscoding to true fails TestMediaSourceNeverTranscodes
- disabling the grant check in catalog.LoadFor fails
TestGrantScopingIsIndistinguishableFromAbsent AND TestStreamRefusesUnreadableItem
Both were restored afterwards and the non-test files verified byte-unchanged.
Process note: the agent for this task failed twice (HTTP 400) and left a test
file that did not compile — a bad slices.Delete call and a helper it never
wrote. I fixed those, corrected one wrong expectation (a video item legitimately
carries MediaType; the test's want-list omitted it), and wrote the handler tests
myself since that was the half that never got reached.
gofmt clean, build exit 0, vet exit 0, 25 packages pass, svelte-check 0/0.
Four extractions, each chosen because the two halves have different callers,
different failure modes or different testing needs. Not a line-count exercise.
internal/home out of internal/media
- media serves item RESOURCES (detail, file, cover, progress); home computes
AGGREGATES (the five shelves). The 2328-line package was doing both.
- media keeps POST /items/{id}/view because it is item-scoped and must stay in
the package that authorizes the item; it imports home.RecordView for that one
thing. home does not import media.
- Verified live: all five shelves non-empty, and two consecutive /home calls
return different explore ids, which is the non-determinism the Explore rewrite
exists for.
internal/scanner out of internal/library
- The scanner walks the filesystem and reconciles; library CRUD is an admin HTTP
surface. Different callers (a job vs a request), different failure modes.
- The registry deliberately STAYED in internal/library: three of its four callers
are not the scanner (modules register types, handlers serve the list, metadata
reads the group mode, subsonic calls CleanTitle). It is a property of the kind,
not of the scan. The one genuinely scanner-owned helper became
library.ExtensionsFor.
- Two tests moved with the code rather than being rewritten; they now exercise
the boundary by calling library.ListMissing across it.
- Verified live: first scan updates, rescan reports unchanged=N, so change
detection still short-circuits.
internal/providers out of internal/metadata
- Providers talk to third-party APIs; the enrich job owns the matching policy and
the database writes. The split's whole point is that a provider is testable
WITHOUT a database, so a DB-free test was added and OpenLibrary gained a
BaseURL override to make it possible.
- providers imports zero internal packages — a true leaf.
- The matching policy (searchTitle, artistHint, pickBest) is untouched. It exists
because real false matches were found.
dto packages out of internal/jellyfin and internal/subsonic
- The wire types and their mapping are now separate from the handler logic, so
the contract with clients we do not control is reviewable on its own.
- Both dto packages are PURE: zero references to sql.DB or net/http, verified by
grep. That is what makes them reviewable.
- No field name, tag or error code changed.
internal/arch did its job: it failed the build twice during this phase, once for
each new package that was not yet placed in a layer. That is the rule working,
not an obstacle.
Verified: gofmt clean, build exit 0, vet exit 0, 23 packages pass, svelte-check
0/0, and end to end against a running server — home shelves, a queued scan
reporting unchanged, Jellyfin /Items, and Subsonic ping.
Process note: one agent (dto) stopped mid-task and left the tree red — the dto
packages unlayered and its test still referencing constants that had moved. The
arch test caught it immediately. I finished it by hand: exported constant
references in the test, and both dto packages placed at layer 3.
Still open from Phase 1: internal/jellyfin has no test files, catalog is
item-centric while Subsonic is file-first, and subsonic loadAlbum now scopes to
the item's own library.
Motivation, from the evidence rather than from taste. Five packages each
implemented "load an item, then check the caller may read it": media, comics,
games, jellyfin (with its own dbItem type and its own SQL), and subsonic. Five
copies of one authorization rule means four can drift, and the failure mode is
not a crash — it is silently serving somebody a library they were never granted.
One implementation is one thing to review.
internal/catalog now owns item and file access, and the access check.
LoadUnchecked(ctx, db, id) -- no access check; jobs and tests only
LoadFor(ctx, db, u, id) -- load AND enforce the grant, in one call
Files / FilesFor / FileByID / FirstFile / FileCount
The unchecked variant is deliberately named so that reaching for it by accident
is hard: it is the dangerous call, so it should not have the friendlier name.
This came from the interface-design rule that a good interface makes the right
thing easy and the wrong thing hard.
The security property is tested, not asserted: a forbidden item and an absent
item return byte-identical errors, so no handler can leak the existence of
another user's library through a 403. That is now enforced in one place.
Migrated all five surfaces onto it. media, comics, games and ebooks went from
four duplicate loaders to zero. jellyfin's dbItem and loadDBItem are gone.
subsonic's per-item authorization moved while its album/artist aggregate
projections stayed put, because grouping items into albums is a presentation
concern, not a data-access one. The remaining raw SQL everywhere is browse and
aggregate queries; no per-item load decides access any more.
internal/arch enforces the layering instead of documenting it. Five rules, each
proven able to fail by deliberately breaking it:
- no upward imports across six explicit layers
- internal/core may depend only on migrations
- internal/catalog must not know about HTTP, or it stops being reusable
- modules must not import each other
- every package must be placed in a layer, so the map cannot go stale
A test that only ever passes is decoration, so each rule was verified by
introducing the violation and confirming the failure.
Verified: gofmt clean, go build ./... exit 0, go vet exit 0, 20 packages pass,
svelte-check 0/0. End to end against a running server: the vellum API serves
item detail with codec facts, an unknown item 404s, Jellyfin AuthenticateByName
mints a token and /Items browses, and Subsonic ping returns status ok. The
subsonic migration was additionally A/B'd byte-identical against the previous
implementation across ~15 endpoints, and Jellyfin/Subsonic both still return the
same error for a forbidden item as for an absent one.
Known follow-ups, recorded rather than silently skipped:
- internal/jellyfin has no test files at all. Its wire contract is the most
client-visible surface in the tree and is now verified only by hand.
- catalog is item-centric, but Subsonic addresses songs file-first (mf-<id>),
so it resolves file to item through its own projection. A catalog lookup
that takes a file id would close that.
- subsonic loadAlbum now scopes the projection to the item's own library
rather than all visible audio libraries. Equivalent by construction, since
an item's kind tracks its library's kind, but a hand-edited mismatch would
differ.
Vellum now speaks two foreign protocols, so the clients people already own can
use it. Both are translation layers over queries that already exist, and both go
through the same grant chokepoint as the web app — a token is a second way to be
a user, never a second set of permissions.
Shared foundation (coordinator, written before dispatch)
- migrations/0012_api_tokens.sql + internal/apitoken. Both protocols need a
credential that is not the session cookie, and letting two agents each invent
a token table would have produced two incompatible ones.
- The table stores two shapes for a real reason: a Jellyfin token is a bearer
value so only its hash is kept, while a Subsonic key must be recoverable
because verifying `t` means computing md5(key + salt) server-side. token_plain
is therefore the one column in the schema holding a usable credential. It is a
generated random secret, never a user password, and it is revocable.
Wave 9 - Jellyfin
- /System/Info/Public, /Users/AuthenticateByName, /Users/Me, /Users/{id},
/Items, /Users/{id}/Items, /Items/{id}, /Items/{id}/Images/Primary,
/Items/{id}/PlaybackInfo, /Videos/{id}/stream, /Audio/{id}/stream,
/Sessions/Playing{,/Progress,/Stopped}.
- Auth accepts X-Emby-Token, X-MediaBrowser-Token, ?api_key=, and
Authorization: MediaBrowser Token="...". The ?api_key= form is the one that
matters: <video> elements cannot set headers.
- Honest about the one thing Vellum cannot do. PlaybackInfo reports
SupportsDirectPlay/SupportsDirectStream true and SupportsTranscoding FALSE,
with a DirectStreamUrl carrying the api_key. A client that disables both
direct modes is demanding a transcode and is refused with ErrorCode
"NoCompatibleStream" — rather than handed a TranscodingUrl that would never
resolve, which is how a client hangs forever.
Wave 10 - Subsonic
- /rest/{method} dispatch with ping, getLicense, getMusicFolders, getIndexes,
getArtists/getArtist/getAlbum, getMusicDirectory, getSong, search3, stream,
getCoverArt, getAlbumList2, getRandomSongs, scrobble, getUser, getPlaylists,
plus GET /api/v1/subsonic/credentials to hand a user their key.
- Auth via t=md5(key+salt), p=key, constant-time comparison. A bad credential
returns HTTP 200 with the envelope status "failed" and error code 40 — NOT an
HTTP 401. Subsonic clients read the envelope, and getting this wrong is the
single most common integration failure in that protocol.
- stream supports format=raw and never invokes ffmpeg.
Coordinator wired both into main.go (the line agents were told to leave alone)
and verified against real requests.
Verified end to end, 18 packages pass, gofmt/vet clean, svelte-check 0/0:
- Jellyfin: discovery, AuthenticateByName minting a token, /Items reporting 18,
PlaybackInfo showing SupportsTranscoding=false, and a Range request to
/Videos/{id}/stream using ONLY ?api_key= returning 206 with Content-Range.
- Subsonic: ping as JSON and XML, md5(key+salt) auth succeeding, and a bad token
returning the failed envelope with code 40.
- Scoping, both protocols, with a genuinely non-admin user: 0 items, 404 on an
admin-only item, 404 on its stream, 0 music folders; admin still sees 18.
- The SPA is not shadowed by the root-mounted compat paths: /, /healthz and
/api/v1/version all still answer correctly.
Both agents failed with an HTTP 400 at the REPORT stage, not during the work;
their files were complete and compile, so the work was salvaged and verified
here rather than redone.
Wave 7 - engines, chosen per file by the client's own capabilities
- internal/engine: a registry. html5 is built in and declares a CONSERVATIVE
browser baseline (h264/vp8/vp9/av1, deliberately no hevc and no mpeg4, because
that is what was measured). mpv, vlc and mdk are registered as native engines
declaring the broad support those libraries really have.
- The server never declares a file playable. It reports what is in the file and
matches it against the engines the CLIENT declares. Proven:
engines=["html5"] -> playable=false engine=null reason=vcodec:hevc
engines=["html5","mpv"] -> playable=true engine=mpv
Same file, different answer, because the client offered a different engine.
That is what makes adding a native client a registration rather than a rewrite.
- internal/playback: GET /playback/engines and POST /playback/resolve, behind
auth and the same grant chokepoint. An unreadable item is 404, not 403.
- web/src/lib/capabilities.ts: uses navigator.mediaCapabilities.decodingInfo(),
which returns {supported, smooth, powerEfficient}. That is real client-hardware
detection and it distinguishes "cannot decode" from "can decode but it will
stutter and drain the battery". Feature-detected; degrades to the server
baseline rather than throwing.
- VideoPlayer consults the server as an ADVISORY only. The element remains the
source of truth: the server's note appears first, then the element's own
confirmation replaces it once videoWidth proves the answer. Verified on HEVC —
"Playback may not work here" at 0-906ms, "Video cannot play here" from ~1208ms.
Wave 8 - missing files: visibility and an explicit purge
- migrations/0011 adds missing_since to media_items and media_files.
- It is written only on the TRANSITION into missing, so the age says when the
file first vanished rather than the last rescan. Proven byte-identical across a
rescan, and cleared when a file is restored.
- GET /api/v1/missing (grant-scoped) and POST /api/v1/libraries/{id}/purge
(admin only). The purge deletes only items that are missing AND have no
present file, and its cascade into progress/item_people/item_views/game_saves
is enumerated in a comment — that cascade is exactly why deletion is never
automatic. Proven: 19 items -> 18, only the missing one, the non-missing item
untouched and its files intact, second purge returns 0.
- MissingFiles.svelte in the Manage nav: title, kind, library, path, human age,
and a per-library Purge behind a confirmation that names the count and says
progress cannot be regenerated.
Coordinator wired playback.NewHandlers into main.go (the one line agents were
told to leave alone).
Verified end to end: 17 packages pass, gofmt/vet clean, svelte-check 0/0, and
every claim above checked against the running server.
Follow-up to 785d3ed. The committed version put the codec `type` on a <source>
child. That is the obvious thing to do and it is wrong: an explicit type lets
Chrome SKIP the resource during selection whenever canPlayType disagrees with
reality, and canPlayType is wrong in both directions.
The cost was measurable. mov+h264 has canPlayType('video/quicktime') === "no",
so the browser refused to try it — a file that PLAYS would have been reported as
unplayable. Verified now at 640x360, readyState 4.
Setting src on the element instead makes the browser actually load the file and
report videoWidth / error honestly, which is what playback.ts's measured notes
describe:
mov+h264 -> loads, videoWidth 640 -> plays
mp4+hevc -> loads, videoWidth 0 -> detected by the decode check
avi+mpeg4 -> error code 4 -> detected by the error path
The `type` is kept only as an ignored hint, and the load watchdog still covers
the case where the element says nothing at all. All three detection paths are
now reachable, which is the point: for HEVC, Chrome plays audio over a blank
frame with no error, and that must never be mistaken for working playback.
Process note, against the coordinator: I edited this file while its agent still
owned it, and the edit introduced the mov+h264 regression above. The agent
reverted it and kept the watchdog, which was the useful half. Do not edit a file
an agent owns mid-task; hand the finding over instead.
Direct play only. The server is a headless GPU-less Xeon: nothing is ever
transcoded or remuxed. A file either plays in the client or it does not.
Measured, not assumed (ffmpeg-generated files with known codecs, played through
Vellum's own stream endpoint in Chrome 155 / Linux):
mp4 h264+aac plays mkv h264+aac PLAYS <- container is fine
mov h264+aac plays webm vp9/opus plays
mp4 av1+opus plays webm vp8/vorbis plays
mp4 hevc+aac AUDIO ONLY, videoWidth stays 0, no error
mkv hevc+flac AUDIO ONLY
avi mpeg4+mp3 fails outright
All seven audio formats play. Two conclusions drive the design: codec matters
and container barely does (mkv+h264 plays, mp4+HEVC does not), and canPlayType
is unreliable in both directions — it says "no" for mov+h264 (which plays) and
"maybe" for mkv+hevc (whose video never decodes).
Therefore playability is a property of the CLIENT, not the file. The server
never declares a file playable; it reports codec facts and each client decides.
That is what makes adding mpv/libvlc/MDK later a matter of registering an engine
rather than re-architecting.
Wave 4 - audio depth
- New MusicPlayer.svelte: real queue, shuffle that walks a fixed permutation
(so "previous" means something and nothing repeats early), repeat off/all/one,
per-track rows, album art, next-track preload.
- AudiobookPlayer: MediaSession (OS media keys + lock screen), playback speed
that SURVIVES a chapter change and persists, sleep timer with countdown that a
manual pause cancels, keyboard shortcuts, next-chapter preload.
- Shared mediaSession.ts written by the coordinator before dispatch so both
players consume one implementation instead of inventing two.
Wave 5 - ffprobe facts
- internal/probe (ffprobe wrapper, optional dependency), migration 0010 adding
vcodec/acodec/width/height/probed_at to media_files, scanner hook, API fields.
- Probing runs only for NEW or CHANGED files and BEFORE the write transaction
opens, so it can never hold SQLite's write lock and a rescan never becomes a
full-library ffprobe sweep. Proven: 797ms probing scan, then 1ms rescan with
probed_at byte-identical.
- ffprobe skips attached_pic so an mp3 with cover art reports no video codec.
- Fixed a real hole: .avi is in the video registry but had no MIME entry, so it
was served as application/octet-stream and could never play.
Wave 6 - the video player
- VideoPlayer.svelte with resume, progress + keepalive final flush, MediaSession,
keyboard shortcuts, multi-file picker, and codec facts shown.
- The honest path is the feature: an HEVC file makes Chrome play audio over a
blank frame with no error, which is the worst possible outcome. The player
detects it and says so, naming the codec and pointing at clients that can.
Coordinator fix to Wave 6: the agent's detection hung off `loadedmetadata`, but
an explicit `type` the browser cannot satisfy makes it reject the source during
selection, so NEITHER loadedmetadata NOR error fires — and the player showed a
working-looking 0:00 with a Play button, exactly the silently-broken state the
component exists to prevent. Added a load watchdog: if the element says nothing
within 2.5s, the source was rejected and the honest notice is shown. Verified
per file afterwards (h264/mkv play at 640 wide; hevc and mpeg4 show their
notices).
Verified end to end, not just compiled: 15 packages pass (including the new
internal/probe), gofmt/vet clean, svelte-check 0/0, and every claim above checked
against a running server. Note the demo's own mp3s are 100 KB of random bytes and
cannot decode; the audio players were proven against generated real audio instead.
Authors are searchable (internal/search, migrations/0009)
- The FTS index covered only title and path, so typing an artist or author
found nothing even though people/item_people already held the credits.
- Rather than a second index, the names are denormalised into a real
media_items.authors column, so the existing external-content FTS and its
triggers keep doing the work; item_people/people triggers recompute it, and
the media_items UPDATE trigger re-indexes automatically.
- Verified: searching "daft punk" now returns Random Access Memories, and so
does the partial "punk".
- The agent found and worked around a real FTS5 external-content bug: on an
UPGRADE the backfill UPDATE fires the delete trigger for rowids not yet in
the fresh index, raising "database disk image is malformed". A rebuild before
the backfill fixes it. This would have broken existing databases while a
fresh install looked fine.
gzip for the SPA (internal/httpapi/static.go)
- 456 KB of JS and 31 KB of CSS were served uncompressed on every cold load.
- Scoped to the embedded SPA assets only, never API or media: compressing a
Range response would break seeking, and wrapping every response would undo
Wave 2's Flusher/ReaderFrom forwarding.
- Compresses only when the client accepts gzip, the response is a compressible
type, there is no Range header, and nothing already set Content-Encoding.
Sets Vary: Accept-Encoding and drops the now-wrong Content-Length.
- Verified by sha256 round-trip (identity vs gzip-decompressed are identical),
and by confirming a Range request still returns 206 with no Content-Encoding.
JS 351,835 -> 108,794 bytes (69% smaller); CSS 80% smaller.
epub.js is code-split (Reader.svelte)
- It was a static import, so ~350 KB was downloaded by everyone on first load
even though most never open an ebook. Now loaded on demand.
- Main bundle 458 KB -> 112 KB (gzipped 149 KB -> 39.5 KB).
- Verified at runtime, which pnpm check cannot: only the main bundle loads
initially, the epubjs chunk is fetched when a book is opened, and the reader
still renders.
Home screen tests (internal/media)
- Coverage 23.4% -> 50.2%. (The 17.9% quoted in the brief was a stale reading;
the agent measured the real baseline by moving its own files aside.)
- Covers the five shelves, counts, Explore's invariants (visibility, no
duplicates, every kind represented, varies across calls, wrap-around),
the list offset/limit/total contract, MostViewed ordering, and — most
importantly — grant scoping, which is the security chokepoint.
Fixed a bug the tests found: Suggestions seeded from revoked libraries
- topEngagement selected the user's most recent progress row with no library
filter, so progress from a library whose grant had been revoked still steered
the home screen — a visible item was labelled "More books" based on content
the user could no longer read. Returned items were always correctly scoped, so
this was an information-flow gap rather than a content leak.
- Now scoped to visible libraries, and the test that pinned the old behaviour
flipped to assert the correct reason.
gofmt/vet clean, 14 packages pass, media tests run 3x for flakiness.
app.css carried a `@media (prefers-color-scheme: light)` block that replaced
EVERY design token with a white and generic-purple palette, and index.html
declared `color-scheme: light dark`. So whenever the OS or browser preferred
light, the app silently discarded the brand — which is indistinguishable from
"the design changed by itself" and is exactly what happened.
This was not introduced by the recent hardening work: the block came in with the
original Svelte commit and was present in the brand commits. It is removed here.
- Deleted the light-mode media query. The charcoal/purple tokens are now the
only tokens.
- Added `color-scheme: dark` to :root and changed the meta tag to `dark`, so
native scrollbars, form controls and the caret follow the brand instead of
rendering white against a dark page.
- Left a comment explaining why there is deliberately no light theme. The
comment deliberately does NOT repeat the old hex values, so that grepping the
built CSS for them stays meaningful — my first verification pass was fooled by
exactly that.
Verified with the browser emulating `prefers-color-scheme: light`: --bg #292929,
--surface #303030, --accent #c084fc, --accent-2 #6d7cff, color-scheme dark. The
built CSS contains zero media queries and zero occurrences of the old accent.
TouchSession throttle (internal/auth)
- Middleware wrote sessions.last_seen on EVERY authenticated request, taking
SQLite's single write lock each time. For media that is severe: one seek in
an audio file fires several Range requests, each taking the write lock and
competing with the scanner and progress saves.
- Now touched at most once per 5 minutes, decided from the LastSeen already
loaded by SessionByToken, so no extra query. A zero interval degrades to the
old per-request behaviour, so a hand-built Authenticator cannot silently stop
updating.
- Verified end to end with an AFTER UPDATE trigger: 20 requests now cause 0
session writes, down from 20.
Response writer forwarding (internal/httpapi)
- statusRecorder wrapped every ResponseWriter but implemented only Write and
WriteHeader, so it did not implement http.Flusher. That silently disabled
early flushing (progressive audio/video) and io.ReaderFrom, which is what
lets io.Copy use sendfile for multi-GB files. Now forwards Flush, Hijack and
ReadFrom, and WriteHeader is idempotent.
Security headers (internal/httpapi)
- Added nosniff, X-Frame-Options: DENY, Referrer-Policy and a CSP. Outermost in
the chain so a recovered panic's 500 still carries them.
- CORRECTION to an earlier claim: I said a malicious EPUB was stored XSS.
That was wrong. epub.js sets iframe sandbox to "allow-same-origin" and only
appends "allow-scripts" when settings.allowScriptedContent is true, which
defaults to false and Reader.svelte never sets. EPUB content cannot execute
script. The CSP is still worth having as defence in depth, but that specific
justification does not hold.
Final flush on teardown (AudiobookPlayer, Reader)
- The last playback/reading position was saved with a fire-and-forget fetch in
onDestroy, which the browser cancels on navigate-away — losing exactly the
thing worth persisting. Now persisted via fetch with keepalive: true, wired
to pagehide and visibilitychange(hidden), latched so the overlapping events
fire one request, and reset when the page becomes visible again so a bfcache
restore still flushes later. sendBeacon was not usable: it is POST-only and
progress is PUT-only.
- Also removed the queueMicrotask track-swap race in AudiobookPlayer: play() is
now driven by onLoadedMetadata once the new src has actually loaded.
Honest readers (Reader, ComicReader)
- registry.go advertises .mobi/.azw3/.fb2 under ebooks, but every ebooks item
was handed to epub.js, which cannot parse them. Now only .epub uses epub.js;
other containers show a message naming the format plus a download link. Items
stay catalogued and downloadable rather than silently vanishing.
- .cbr/.rar comics say the archive format is unsupported instead of surfacing
the server's raw 422 (the server reads zip only).
Verified in a real browser, not just compiled: all six reader surfaces render
(epub iframe, cbz images, audio element, photo, pdf canvas) with zero CSP
violations; .mobi shows the honest message with no epub.js iframe. gofmt/vet
clean, 14 packages pass, svelte-check 0/0.
Known issue, NOT introduced here and NOT fixed: the EPUB reader logs
"Error: No Element Provided" x3 and a TypeError in epub.js injectIdentifier
(doc.getElementsByTagName("head")[0] undefined). Proven pre-existing by A/B
against c50b3cf, and not caused by the test file (spec-correct XHTML) or by
caching (cold browser). Non-fatal: the book still renders. Needs its own
investigation.
A 50k-file library used to wedge the database. This wave fixes the four
reasons why, plus a timestamp-ordering bug the work exposed.
Chunked scans (internal/library/scan.go)
- One transaction wrapped the whole reconciliation, holding SQLite's single
write lock for the entire scan. Now 500 groups per transaction, and the two
missing-file passes are batched too.
- Verified: a 20,000-file scan reports partial progress (scanned=1500 mid-scan)
and a concurrent progress save returns 200 throughout. Under the old code
that save blocked for the whole scan.
- Behaviour change: a scan is no longer all-or-nothing. A crash now leaves a
consistent partial scan. The scan is idempotent, so re-running finishes it.
- res.Errors is capped at 100 (a broken mount used to append one string per
file, then serialise the lot into the job row); the overflow is counted.
Concurrent job workers (internal/jobs/jobs.go)
- Run() was single-threaded, so a scan stalled every enrichment job behind it.
Now 4 workers with jittered idle waits.
- Handlers run concurrently, so Register/lookup take an RWMutex, and a
panicking handler is recovered and marked failed rather than killing a worker
and stranding a row in 'running'.
- RecoverStale() requeues jobs left 'running' by a dead process; called at boot
from main.go. Previously such a job stayed 'running' forever and the UI
polled it forever.
Explore (internal/media/discover.go)
- Full-scanned and sorted the whole library on every home load, and was
deterministic, so it showed the same 30 items every visit. Now a per-kind
fair-share id range scan with a random start, wrap-around, dedup and
interleaving. The doc comment claimed per-kind sampling before; now it does.
Indexes (migrations/0008_query_indexes.sql)
- Benchmarked with ANALYZE over 60k items: browse page 1 13.37ms -> 0.03ms
(484x, full scan -> covering index), deep page 8x, recently added 3.2x.
- A candidate index for the Continue shelf was built, measured and REJECTED:
the planner never chose it because `finished = 0` selects the majority of a
user's rows, so it had no selectivity and would have cost a write on every
progress save. Recorded in the migration rather than shipped as dead weight.
Canonical timestamps (internal/core/stamp.go)
- time.RFC3339Nano trims trailing zeros, so its fraction width varies and the
strings are not lexicographically sortable: ".5Z" > ".55Z" as strings though
.500s < .550s. That broke every ORDER BY <timestamp> and MAX(timestamp),
including the Continue shelf and Most Viewed. It was wrong even in columns
written 100% by Go, so it was not a cross-format issue.
- All 14 write sites now use core.Stamp, matching SQLite's own default format
exactly. Read paths are untouched: RFC3339Nano parses both shapes, so
existing rows keep working.
Also: busy_timeout 5s -> 15s. Chunked scans still delay a concurrent write, and
measured tail latency on a first scan of 15k new files reached 3.3s, leaving
almost no margin at 5s. Waiting beats failing with SQLITE_BUSY and losing a
playback position. A 10ms inter-batch yield was implemented, measured, found to
have no effect distinguishable from noise (overlapping ranges), and removed.
Verified end to end, not just compiled: orphan recovery logged at boot, scan
returns 202 in 0.7ms, Explore varies across loads, concurrent writes survive a
scan. gofmt/vet clean, 14 packages pass, -race clean on the queue, svelte-check
0/0, fresh install applies all 8 migrations.
Five tasks, four delegated to agents on disjoint files and one on the critical
path. Board and verification log: ~/.opencode/plan/vellum-beta-tasks.md
T1 Async scans (coordinator)
- migration 0007 adds ref_id + result to jobs, indexed by (kind, ref_id, id),
so a job can be found by what it operates on and can report back.
- jobs: EnqueueRef, JobID(ctx) so a handler can attach a result, SetResult.
- POST /libraries/{id}/scan now enqueues and returns 202; a real library takes
longer than any request timeout, and a hanging request looks like a broken
server. GET /libraries/{id}/scan/status reports the latest run.
- Libraries.svelte polls and shows "Scanning…" instead of freezing.
T2 Admin CLI (agent)
- internal/cli: `user add|passwd|list`, `backup <path>` via VACUUM INTO, with
tests. Wired into main.go, which the coordinator kept.
T3 Deploy (agent)
- compose.yml now mounts all six media kinds read-only with :z and carries the
metadata env keys. DEPLOY.md covers volumes, SELinux, admin bootstrap, TLS,
backup and upgrade.
T4 Pagination (agent)
- GET /items takes offset and returns total; Browse loads 60 at a time with
"Load more" and "Showing X of Y" instead of silently truncating at 300.
T5 Password change UI (agent)
- A Password section in Settings with validation, busy and success states.
Verified end to end: scan returns 202 in 0.7 ms and polls to done; the CLI
created a user, changed a password, and produced a complete 240 KB backup while
the server ran; pagination reports total 8 and pages correctly; compose parses
with every mount :z. go vet and 14 test packages pass; svelte-check 0/0.
- Primary accent is now purple (#c084fc); blue-violet (#6d7cff) becomes the
secondary. The primary button uses dark ink on purple for contrast, since
white on that purple is only about 2.5:1.
- The sidebar gets its own surface (--nav-bg, #212121) one step below the page
(#292929) with a stronger divider, so it reads as a distinct rail rather than
blending into the content. Nav link, count and section colours lifted off
--faint onto --muted for legibility against the darker rail.
- Light mode mirrors the same structure.
Verified: accent resolves to #c084fc, secondary to #6d7cff, nav to #212121 on a
#292929 page.
Replaces the gold experiment with the actual brand.
- Palette: brand charcoal (#303030, sampled from the logo's own background)
with the logo's glow as the primary accent (#6d7cff) and its purple
(#c084fc) as a secondary. Text is the palette's white / light gray. Light
mode re-tuned to the same hues.
- Logo: the source mark keyed off its charcoal background to RGBA, cropped to
its content bounds, and added as web/public/logo.png. It now appears in the
sidebar and on the login screen; a square favicon is derived from it and
wired into index.html.
- The featured kicker uses the purple secondary, so both brand hues are present
without adding a second competing accent.
Verified: logo loads in the app, accent resolves to #6d7cff, secondary to
#c084fc, surface to #303030.
The skeleton, not features. Two things felt rushed: the colour had no point of
view, and type and shape came from ad-hoc values.
Design language
- Palette reworked to warm-neutral dark with a single muted gold accent
(#d9a441). The chrome is deliberately colourless because the artwork is the
colour in this app; gold carries emphasis without competing with covers.
Light mode re-tuned to match.
- A real type scale (--fs-xs … --fs-3xl) replaces the ad-hoc rem values, with
fixed line heights and letter-spacing per level. Headings are 620 weight and
tighter; body is a consistent 15px.
- Radii tightened to a 5/8/12 scale, one shadow pair, and the primary button
now uses dark ink on gold for contrast.
Featured carousel
- The banner rotates through up to six items — in progress first, then most
viewed, then newest — drawn only from the libraries the user was granted, so
it can never show something they cannot open.
- Auto-advances every 8s, pauses on hover and focus, and has dots plus arrows
for manual control. Slides crossfade, and the ambient wash changes with them.
Verified in the built bundle: accent resolves to #d9a441, six slides, rotation
confirmed across three items, and hover pauses it.
Not new features — motion, focus, and depth, kept restrained.
- Motion tokens: one easing curve and three durations, used everywhere, so the
app moves as a single thing rather than component by component. Honours
prefers-reduced-motion by disabling all of it.
- View transitions: navigating fades and rises the content instead of swapping
it, and each view starts scrolled to the top.
- Topbar frosts once the content scrolls, so the header separates from the page
without a permanent bar of chrome.
- Featured area on Home: the item you are most likely to act on (something in
progress, else the newest), with the cover as an ambient blurred wash. The
colour comes from the artwork, not a decorative gradient, so it belongs to
whatever is on screen.
- Cards: lift and scale on hover, a scrim so the action glyph always has
contrast, a kind-appropriate glyph (play / read / expand), press feedback,
and a visible keyboard focus ring.
- Shelves: edge fades and arrow buttons that appear only when the row actually
continues that way; the scrollbar is hidden and the row snaps.
- Item page: ambient wash from its own cover.
- Global :focus-visible ring.
Verified in the built bundle: featured renders the in-progress item, hover
scales a 192px card to 197.8px with the glyph visible, and the item page tints
from its artwork.
- All 50 font-size declarations converted from px to rem, so a single root
value scales headings, labels and body together. The root is now 15px, a
deliberate ~7% increase over the previous 14px.
- Card width and grid column width are now --card-w / --grid-min variables,
driven by --card-scale independently of text. Defaults bumped too: shelf
cards 150 -> 160px, grid columns 148 -> 158px.
- Appearance panel (nav item, visible to everyone): Text size and Card size
as segmented controls, a live preview, and a reset. Settings are stored in
localStorage, per device rather than per account, because a TV across the
room wants bigger text than a laptop and the same person uses both.
- nav: 'settings' was missing from the View union after the home rewrite;
added back along with the Appearance entry.
Verified in the built bundle: root font 15px by default; "Larger" scales it to
19.2px; "Large" cards scale 160 to 192px; both persist across a reload and are
independent of each other.
The landing screen now reads as a product rather than a list.
- migration 0006: item_views, per user. "Most viewed" is personal — what I
re-read is not what you re-read. Counted when an item is opened in a reader,
so the number means consumed, not merely fetched.
- Suggestions: unstarted items related to what the user has been consuming —
same author first, then same kind, then recent as a floor. Each carries the
reason it was chosen ("More by Daft Punk"), because a suggestion nobody can
understand is noise.
- Explore: a mixed sample across every kind, excluding in-progress items so the
zone is discovery rather than a repeat.
- Home fetches all five shelves in one request; the screen never pops in.
- web: Shelf gained subtitle/badge support, Card gained a subtitle, and the
home renders four shelves plus a wrapping Explore grid.
Three bugs found and fixed while verifying:
- Continue showed a multi-track audiobook once per chapter. GROUP BY item_id
with MAX(updated_at) picks one card per item (SQLite takes the bare columns
from the row that produced the max).
- Suggestions and Explore passed their SQL arguments in the wrong order: the IN
list precedes the user id in those queries, so a user id was being bound as a
library id. Covered by a scoping test that would have caught it.
- The sidebar lost its media types on a cold, logged-out load. bootstrap()
fetched media types inside a Promise.all with /auth/me, so the expected 401
rejected the whole thing and nothing refetched after login. Media types are
public and now load independently.
The previous commit's vite.config.ts used process.env without Node types, so
svelte-check failed. Adds @types/node and includes "node" in tsconfig types.
svelte-check is back to 0 errors, 0 warnings.
Running `pnpm dev` alone gave ECONNREFUSED on every request, because nothing
was listening on :7788. Two separate causes:
- the backend simply was not running, and
- Vite bound [::1]:5173 while the proxy target was "localhost", which Node
resolves to ::1 first. The Go server binds IPv4, so even with the backend up
the proxy could fail before falling back.
- vite.config.ts: proxy target is now 127.0.0.1 explicitly and overridable via
VELLUM_API; server.host is true so both stacks are served.
- scripts/dev.sh + `make dev`: builds the backend, waits for /healthz, starts
Vite, and stops the backend on exit. Ctrl-C tears down both.
- README documents that running Vite alone needs the backend already up.
- README status: PDF, music metadata, comic metadata, and games are done.
Six media types work; video remains deferred by design. The stack now covers
Stump, Kavita, Audiobookshelf, Navidrome, and RomM — only Jellyfin is left.
- IGDB environment variables documented.
- Item detail labels the primary action "Play" for games, not "Read".
The RomM replacement.
- registry: a games media type with ~35 ROM extensions across NES, SNES, GB/GBA,
N64, NDS, Genesis/Master System/Game Gear, PC Engine, Neo Geo Pocket,
WonderSwan, Atari, Coleco, Intellivision, and PlayStation.
- games: platform detection from the file extension, mapping each to an
EmulatorJS core. Platforms with no browser core are recognised but marked
not-playable so the UI offers a download instead of a broken player.
- migration 0005: game_saves keyed by (user, item, slot). Save states are user
data, not media — two people playing the same ROM keep their own progress.
- save endpoints with a 16 MiB cap and a validated slot name; the upload is
read through a LimitReader so an unbounded body is refused rather than
buffered.
- igdb provider: Twitch client-credentials flow with the token cached for its
~2 month lifetime, Apicalypse query body, and cover URLs normalised from
protocol-relative t_thumb to https t_cover_big.
- searchTitle: strips "(USA)", "[!]", "(Rev 1)" and similar release tags before
searching. "Sonic the Hedgehog (USA)" matched nothing; "Sonic the Hedgehog"
matches immediately. Stored titles keep the original.
- web: GamePlayer embeds EmulatorJS and adds Load/Save-to-server buttons.
Verified live against IGDB: Sonic matched (id 316900) with real cover art; a
4 KB save uploaded, listed, downloaded byte-identical, and deleted.
