Vellum/migrations/0009_search_authors.sql
mithral 09474b0e08 Wave 3: search depth, transfer size, and the Home screen's tests
Authors are searchable (internal/search, migrations/0009)
- The FTS index covered only title and path, so typing an artist or author
  found nothing even though people/item_people already held the credits.
- Rather than a second index, the names are denormalised into a real
  media_items.authors column, so the existing external-content FTS and its
  triggers keep doing the work; item_people/people triggers recompute it, and
  the media_items UPDATE trigger re-indexes automatically.
- Verified: searching "daft punk" now returns Random Access Memories, and so
  does the partial "punk".
- The agent found and worked around a real FTS5 external-content bug: on an
  UPGRADE the backfill UPDATE fires the delete trigger for rowids not yet in
  the fresh index, raising "database disk image is malformed". A rebuild before
  the backfill fixes it. This would have broken existing databases while a
  fresh install looked fine.

gzip for the SPA (internal/httpapi/static.go)
- 456 KB of JS and 31 KB of CSS were served uncompressed on every cold load.
- Scoped to the embedded SPA assets only, never API or media: compressing a
  Range response would break seeking, and wrapping every response would undo
  Wave 2's Flusher/ReaderFrom forwarding.
- Compresses only when the client accepts gzip, the response is a compressible
  type, there is no Range header, and nothing already set Content-Encoding.
  Sets Vary: Accept-Encoding and drops the now-wrong Content-Length.
- Verified by sha256 round-trip (identity vs gzip-decompressed are identical),
  and by confirming a Range request still returns 206 with no Content-Encoding.
  JS 351,835 -> 108,794 bytes (69% smaller); CSS 80% smaller.

epub.js is code-split (Reader.svelte)
- It was a static import, so ~350 KB was downloaded by everyone on first load
  even though most never open an ebook. Now loaded on demand.
- Main bundle 458 KB -> 112 KB (gzipped 149 KB -> 39.5 KB).
- Verified at runtime, which pnpm check cannot: only the main bundle loads
  initially, the epubjs chunk is fetched when a book is opened, and the reader
  still renders.

Home screen tests (internal/media)
- Coverage 23.4% -> 50.2%. (The 17.9% quoted in the brief was a stale reading;
  the agent measured the real baseline by moving its own files aside.)
- Covers the five shelves, counts, Explore's invariants (visibility, no
  duplicates, every kind represented, varies across calls, wrap-around),
  the list offset/limit/total contract, MostViewed ordering, and — most
  importantly — grant scoping, which is the security chokepoint.

Fixed a bug the tests found: Suggestions seeded from revoked libraries
- topEngagement selected the user's most recent progress row with no library
  filter, so progress from a library whose grant had been revoked still steered
  the home screen — a visible item was labelled "More books" based on content
  the user could no longer read. Returned items were always correctly scoped, so
  this was an information-flow gap rather than a content leak.
- Now scoped to visible libraries, and the test that pinned the old behaviour
  flipped to assert the correct reason.

gofmt/vet clean, 14 packages pass, media tests run 3x for flakiness.
2026-10-11 13:46:18 -04:00

120 lines
5.4 KiB
SQL

-- 0009_search_authors.sql — make credited people (authors, artists, directors)
-- searchable.
--
-- 0003 indexed only `title` and `path`, so typing an author's name found
-- nothing even though `people` and `item_people` already held the credits.
-- Rather than make the search query join the credit tables — which would mean a
-- second MATCH against a second index, or an unindexed LIKE over a join — the
-- names are denormalised into a real column on `media_items`. The existing
-- external-content FTS5 design then keeps working unchanged in spirit: the
-- index still stores no text of its own, and the media_items triggers that
-- already keep it in sync simply gain the new column.
--
-- `authors` is maintained by triggers on item_people/people, so a credit change
-- re-indexes the item no matter which code path writes it — the metadata
-- enricher today, anything else later.
ALTER TABLE media_items ADD COLUMN authors TEXT NOT NULL DEFAULT '';
-- Drop the old two-column index and its triggers before rebuilding.
-- Triggers first: dropping the virtual table does not drop triggers that
-- reference it, and a write landing between the two drops would fire a trigger
-- against a table that no longer exists.
DROP TRIGGER media_items_ai;
DROP TRIGGER media_items_ad;
DROP TRIGGER media_items_au;
DROP TABLE media_search;
CREATE VIRTUAL TABLE media_search USING fts5(
title,
path,
authors,
content='media_items',
content_rowid='id',
tokenize='unicode61 remove_diacritics 2'
);
-- Populate the fresh index from current content before the backfill below.
-- FTS5's external-content 'delete' command assumes the rowid is already in the
-- index; issuing it for an absent rowid raises "database disk image is
-- malformed". The backfill UPDATE fires media_items_au (a delete+insert) for
-- every existing row, so the index must first hold those rows. On a fresh
-- database this indexes nothing and is a no-op.
INSERT INTO media_search(media_search) VALUES ('rebuild');
-- Recreate the media_items triggers verbatim from 0003, with `authors` added,
-- so every direct write to an item keeps the index correct.
CREATE TRIGGER media_items_ai AFTER INSERT ON media_items BEGIN
INSERT INTO media_search(rowid, title, path, authors)
VALUES (new.id, new.title, new.path, new.authors);
END;
CREATE TRIGGER media_items_ad AFTER DELETE ON media_items BEGIN
INSERT INTO media_search(media_search, rowid, title, path, authors)
VALUES ('delete', old.id, old.title, old.path, old.authors);
END;
CREATE TRIGGER media_items_au AFTER UPDATE ON media_items BEGIN
INSERT INTO media_search(media_search, rowid, title, path, authors)
VALUES ('delete', old.id, old.title, old.path, old.authors);
INSERT INTO media_search(rowid, title, path, authors)
VALUES (new.id, new.title, new.path, new.authors);
END;
-- Credit triggers. Each recomputes media_items.authors for the affected item.
-- That UPDATE is itself on media_items, so media_items_au fires and rewrites
-- the FTS row — no FTS-specific code is needed here. (This is a single level of
-- trigger nesting, not recursion; and even with PRAGMA recursive_triggers=1 the
-- chain terminates because the FTS write targets the virtual table, not
-- media_items.)
CREATE TRIGGER item_people_ai AFTER INSERT ON item_people BEGIN
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '')
WHERE id = new.item_id;
END;
CREATE TRIGGER item_people_ad AFTER DELETE ON item_people BEGIN
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '')
WHERE id = old.item_id;
END;
-- An UPDATE can move a credit between items, so both ends are recomputed.
CREATE TRIGGER item_people_au AFTER UPDATE ON item_people BEGIN
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '')
WHERE id = old.item_id;
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '')
WHERE id = new.item_id;
END;
-- A rename must not leave the old spelling in the index: recompute every item
-- credited to this person. people.name is UNIQUE COLLATE NOCASE, so renames are
-- rare, but a stale denormalised column is exactly what a rename would expose.
CREATE TRIGGER people_au AFTER UPDATE ON people BEGIN
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '')
WHERE id IN (SELECT item_id FROM item_people WHERE person_id = old.id);
END;
-- Backfill: the triggers above only fire on future writes, so populate every
-- existing row once. The final rebuild re-derives the whole index from the
-- content table, which also normalises any transient FTS state left by the
-- per-row updates above.
UPDATE media_items SET authors = COALESCE((
SELECT group_concat(p.name, ' ') FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = media_items.id), '');
INSERT INTO media_search(media_search) VALUES ('rebuild');