Vellum/internal/media/discover.go
mithral 09474b0e08 Wave 3: search depth, transfer size, and the Home screen's tests
Authors are searchable (internal/search, migrations/0009)
- The FTS index covered only title and path, so typing an artist or author
  found nothing even though people/item_people already held the credits.
- Rather than a second index, the names are denormalised into a real
  media_items.authors column, so the existing external-content FTS and its
  triggers keep doing the work; item_people/people triggers recompute it, and
  the media_items UPDATE trigger re-indexes automatically.
- Verified: searching "daft punk" now returns Random Access Memories, and so
  does the partial "punk".
- The agent found and worked around a real FTS5 external-content bug: on an
  UPGRADE the backfill UPDATE fires the delete trigger for rowids not yet in
  the fresh index, raising "database disk image is malformed". A rebuild before
  the backfill fixes it. This would have broken existing databases while a
  fresh install looked fine.

gzip for the SPA (internal/httpapi/static.go)
- 456 KB of JS and 31 KB of CSS were served uncompressed on every cold load.
- Scoped to the embedded SPA assets only, never API or media: compressing a
  Range response would break seeking, and wrapping every response would undo
  Wave 2's Flusher/ReaderFrom forwarding.
- Compresses only when the client accepts gzip, the response is a compressible
  type, there is no Range header, and nothing already set Content-Encoding.
  Sets Vary: Accept-Encoding and drops the now-wrong Content-Length.
- Verified by sha256 round-trip (identity vs gzip-decompressed are identical),
  and by confirming a Range request still returns 206 with no Content-Encoding.
  JS 351,835 -> 108,794 bytes (69% smaller); CSS 80% smaller.

epub.js is code-split (Reader.svelte)
- It was a static import, so ~350 KB was downloaded by everyone on first load
  even though most never open an ebook. Now loaded on demand.
- Main bundle 458 KB -> 112 KB (gzipped 149 KB -> 39.5 KB).
- Verified at runtime, which pnpm check cannot: only the main bundle loads
  initially, the epubjs chunk is fetched when a book is opened, and the reader
  still renders.

Home screen tests (internal/media)
- Coverage 23.4% -> 50.2%. (The 17.9% quoted in the brief was a stale reading;
  the agent measured the real baseline by moving its own files aside.)
- Covers the five shelves, counts, Explore's invariants (visibility, no
  duplicates, every kind represented, varies across calls, wrap-around),
  the list offset/limit/total contract, MostViewed ordering, and — most
  importantly — grant scoping, which is the security chokepoint.

Fixed a bug the tests found: Suggestions seeded from revoked libraries
- topEngagement selected the user's most recent progress row with no library
  filter, so progress from a library whose grant had been revoked still steered
  the home screen — a visible item was labelled "More books" based on content
  the user could no longer read. Returned items were always correctly scoped, so
  this was an information-flow gap rather than a content leak.
- Now scoped to visible libraries, and the test that pinned the old behaviour
  flipped to assert the correct reason.

gofmt/vet clean, 14 packages pass, media tests run 3x for flakiness.
2026-10-11 13:46:18 -04:00

391 lines
11 KiB
Go

package media
import (
"context"
"database/sql"
"fmt"
"math/rand"
"strings"
)
// RecordView increments a user's view count for an item.
//
// Called when an item is opened in a reader, so the count reflects consumption
// rather than a detail-page fetch. Opening the same thing repeatedly still
// counts: "most viewed" should reflect habit, not just distinct items.
func RecordView(ctx context.Context, db *sql.DB, userID, itemID int64) (int, error) {
var views int
err := db.QueryRowContext(ctx, `
INSERT INTO item_views (user_id, item_id, views, last_viewed_at)
VALUES (?, ?, 1, ?)
ON CONFLICT (user_id, item_id)
DO UPDATE SET views = views + 1, last_viewed_at = excluded.last_viewed_at
RETURNING views`,
userID, itemID, nowRFC3339()).Scan(&views)
if err != nil {
return 0, fmt.Errorf("media: record view: %w", err)
}
return views, nil
}
// MostViewed returns the user's most-opened items within the given libraries.
func MostViewed(ctx context.Context, db *sql.DB, userID int64, libraryIDs []int64, limit int) ([]shelfItem, error) {
if len(libraryIDs) == 0 {
return []shelfItem{}, nil
}
if limit <= 0 || limit > 100 {
limit = 20
}
in, args := inClause(libraryIDs)
args = append([]any{userID}, args...)
args = append(args, limit)
query := `
SELECT i.id, i.library_id, i.kind, i.title, v.views, v.last_viewed_at
FROM item_views v
JOIN media_items i ON i.id = v.item_id
WHERE v.user_id = ? AND i.missing = 0
AND i.library_id IN (` + in + `)
ORDER BY v.views DESC, v.last_viewed_at DESC
LIMIT ?`
rows, err := db.QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("media: most viewed: %w", err)
}
defer rows.Close()
out := []shelfItem{}
for rows.Next() {
var s shelfItem
if err := rows.Scan(&s.ID, &s.LibraryID, &s.Kind, &s.Title, &s.Views, &s.UpdatedAt); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// itemRef is a lightweight item used by the suggestion and explore queries.
type itemRef struct {
ID int64
LibraryID int64
Kind string
Title string
Year int
}
// engagement describes what a user has been consuming, which is the basis for
// suggestions.
type engagement struct {
Kind string
Title string
Author string
}
// topEngagement finds the kind and author the user most recently engaged with,
// so suggestions can be explained in plain language.
func topEngagement(ctx context.Context, db *sql.DB, userID int64, libraryIDs []int64) (engagement, bool, error) {
var e engagement
// Scoped to libraries the user can still read. Without this, progress from a
// library whose grant was revoked would keep steering the home screen's
// suggestions — confusing on its own, and an information-flow leak about
// content the user no longer has access to.
in, inArgs := inClause(libraryIDs)
args := append([]any{userID}, inArgs...)
err := db.QueryRowContext(ctx, `
SELECT i.kind, i.title,
COALESCE((SELECT p.name FROM item_people ip
JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = i.id AND ip.role = 'author'
ORDER BY ip.ord LIMIT 1), '')
FROM progress p
JOIN media_items i ON i.id = p.item_id
WHERE p.user_id = ? AND i.missing = 0
AND i.library_id IN (`+in+`)
ORDER BY p.updated_at DESC
LIMIT 1`, args...).Scan(&e.Kind, &e.Title, &e.Author)
if err == sql.ErrNoRows {
return engagement{}, false, nil
}
if err != nil {
return engagement{}, false, fmt.Errorf("media: top engagement: %w", err)
}
return e, true, nil
}
// Suggestions returns unstarted items related to what the user has been
// consuming: same author first, then the same kind, then anything recent.
//
// The rule is deliberately legible — a suggestion a person cannot understand is
// just noise. Each result carries the reason it was chosen.
func Suggestions(ctx context.Context, db *sql.DB, userID int64, libraryIDs []int64, limit int) ([]suggestion, error) {
if len(libraryIDs) == 0 {
return []suggestion{}, nil
}
if limit <= 0 || limit > 50 {
limit = 20
}
e, hasEngagement, err := topEngagement(ctx, db, userID, libraryIDs)
if err != nil {
return nil, err
}
in, inArgs := inClause(libraryIDs)
out := []suggestion{}
seen := map[int64]bool{}
add := func(rows *sql.Rows, reason string) error {
defer rows.Close()
for rows.Next() {
var r itemRef
if err := rows.Scan(&r.ID, &r.LibraryID, &r.Kind, &r.Title, &r.Year); err != nil {
return err
}
if seen[r.ID] || len(out) >= limit {
continue
}
seen[r.ID] = true
out = append(out, suggestion{
ID: r.ID, LibraryID: r.LibraryID, Kind: r.Kind,
Title: r.Title, Year: r.Year, Reason: reason,
})
}
return rows.Err()
}
// Unstarted = no progress row for this user. Note the parameter order: the
// IN list comes first in the SQL, then the user id inside NOT EXISTS.
base := `
SELECT i.id, i.library_id, i.kind, i.title, COALESCE(i.year, 0)
FROM media_items i
WHERE i.missing = 0 AND i.library_id IN (` + in + `)
AND NOT EXISTS (SELECT 1 FROM progress p WHERE p.item_id = i.id AND p.user_id = ?)`
if hasEngagement {
// 1. Same author.
if e.Author != "" {
rows, err := db.QueryContext(ctx, base+`
AND EXISTS (SELECT 1 FROM item_people ip JOIN people p ON p.id = ip.person_id
WHERE ip.item_id = i.id AND p.name = ?)
ORDER BY i.added_at DESC LIMIT ?`,
withLibs(inArgs, userID, e.Author, limit)...)
if err != nil {
return nil, fmt.Errorf("media: suggestions by author: %w", err)
}
if err := add(rows, "More by "+e.Author); err != nil {
return nil, err
}
}
// 2. Same kind.
rows, err := db.QueryContext(ctx, base+`
AND i.kind = ? ORDER BY i.added_at DESC LIMIT ?`,
withLibs(inArgs, userID, e.Kind, limit)...)
if err != nil {
return nil, fmt.Errorf("media: suggestions by kind: %w", err)
}
if err := add(rows, "More "+kindNoun(e.Kind)); err != nil {
return nil, err
}
}
// 3. Anything recent, as a floor so the shelf is never empty.
rows, err := db.QueryContext(ctx, base+`
ORDER BY i.added_at DESC LIMIT ?`,
withLibs(inArgs, userID, limit)...)
if err != nil {
return nil, fmt.Errorf("media: suggestions fallback: %w", err)
}
if err := add(rows, "Recently added"); err != nil {
return nil, err
}
return out, nil
}
// Explore returns a mixed sample across every kind the user can see.
//
// Excludes anything already on the Continue shelf so the zone feels like
// discovery rather than a repeat. The shelf is sampled per kind: each kind
// present in the visible libraries gets a fair share of the limit, so one
// large library or format cannot drown out the rest. Within a kind the sample
// starts at a random id and walks an indexed, id-ordered range scan that stops
// once it has enough rows, so repeat visits surface different items without
// scanning or sorting the whole library.
func Explore(ctx context.Context, db *sql.DB, userID int64, libraryIDs []int64, limit int) ([]shelfItem, error) {
if len(libraryIDs) == 0 {
return []shelfItem{}, nil
}
if limit <= 0 || limit > 100 {
limit = 30
}
in, inArgs := inClause(libraryIDs)
// One small grouped query tells us which kinds are actually present, so we
// can give each of them a fair share of the shelf. Kinds with no visible
// items are ignored.
kindRows, err := db.QueryContext(ctx, `
SELECT kind, count(*)
FROM media_items
WHERE missing = 0 AND library_id IN (`+in+`)
GROUP BY kind`, inArgs...)
if err != nil {
return nil, fmt.Errorf("media: explore: %w", err)
}
kinds := []string{}
for kindRows.Next() {
var kind string
var n int64
if err := kindRows.Scan(&kind, &n); err != nil {
kindRows.Close()
return nil, err
}
if n > 0 {
kinds = append(kinds, kind)
}
}
if err := kindRows.Err(); err != nil {
kindRows.Close()
return nil, err
}
kindRows.Close()
if len(kinds) == 0 {
return []shelfItem{}, nil
}
// Divide the limit across the kinds that exist: one each, then spread the
// remainder over the first kinds. If there are more kinds than slots, the
// extras get nothing on this visit.
base := limit / len(kinds)
rem := limit % len(kinds)
seen := map[int64]bool{}
out := []shelfItem{}
// appendRows runs one bounded, id-ordered range scan for a kind and appends
// any rows not already seen. before selects the wrap-around half of the id
// range (id < start) instead of the forward half (id >= start). It returns
// how many new rows were added.
appendRows := func(kind string, start int64, before bool, n int) (int, error) {
if n <= 0 {
return 0, nil
}
cmp := ">="
if before {
cmp = "<"
}
q := `
SELECT i.id, i.library_id, i.kind, i.title, 0, i.added_at
FROM media_items i
WHERE i.missing = 0 AND i.library_id IN (` + in + `)
AND i.kind = ?
AND i.id ` + cmp + ` ?
AND NOT EXISTS (
SELECT 1 FROM progress p
WHERE p.item_id = i.id AND p.user_id = ? AND p.finished = 0
)
ORDER BY i.id
LIMIT ?`
rows, err := db.QueryContext(ctx, q, withLibs(inArgs, kind, start, userID, n)...)
if err != nil {
return 0, fmt.Errorf("media: explore: %w", err)
}
defer rows.Close()
added := 0
for rows.Next() {
var s shelfItem
if err := rows.Scan(&s.ID, &s.LibraryID, &s.Kind, &s.Title, &s.Percent, &s.UpdatedAt); err != nil {
return added, err
}
if seen[s.ID] {
continue
}
seen[s.ID] = true
out = append(out, s)
added++
}
return added, rows.Err()
}
for i, kind := range kinds {
share := base
if i < rem {
share++
}
if share <= 0 {
continue
}
var maxID int64
if err := db.QueryRowContext(ctx, `
SELECT COALESCE(MAX(id), 0)
FROM media_items
WHERE missing = 0 AND library_id IN (`+in+`) AND kind = ?`,
withLibs(inArgs, kind)...).Scan(&maxID); err != nil {
return nil, fmt.Errorf("media: explore: %w", err)
}
if maxID <= 0 {
continue
}
// A random start keeps repeat visits from showing the same slice.
start := rand.Int63n(maxID) + 1
got, err := appendRows(kind, start, false, share)
if err != nil {
return nil, err
}
if got < share {
// The start landed near the end of the id range; wrap to the
// beginning to fill the kind's share.
if _, err := appendRows(kind, start, true, share-got); err != nil {
return nil, err
}
}
}
// Mix the kinds instead of leaving them in contiguous blocks, then trim.
rand.Shuffle(len(out), func(a, b int) { out[a], out[b] = out[b], out[a] })
if len(out) > limit {
out = out[:limit]
}
return out, nil
}
// withLibs builds a query argument list: the IN-list args first (they appear
// first in these queries), then whatever follows.
//
// Written explicitly because the parameter order in the SQL is not the order
// the caller naturally thinks in, and getting it wrong silently swaps a user id
// for a library id.
func withLibs(inArgs []any, rest ...any) []any {
out := make([]any, 0, len(inArgs)+len(rest))
out = append(out, inArgs...)
out = append(out, rest...)
return out
}
// kindNoun turns a media kind into something readable in a sentence.
func kindNoun(kind string) string {
switch kind {
case "ebooks":
return "books"
case "comics":
return "comics"
case "audiobooks":
return "audiobooks"
case "music":
return "music"
case "photos":
return "photos"
case "games":
return "games"
default:
return strings.ReplaceAll(kind, "-", " ")
}
}