Vellum/internal/core/stamp_test.go
mithral c50b3cfb3b Wave 1: survive a real library
A 50k-file library used to wedge the database. This wave fixes the four
reasons why, plus a timestamp-ordering bug the work exposed.

Chunked scans (internal/library/scan.go)
- One transaction wrapped the whole reconciliation, holding SQLite's single
  write lock for the entire scan. Now 500 groups per transaction, and the two
  missing-file passes are batched too.
- Verified: a 20,000-file scan reports partial progress (scanned=1500 mid-scan)
  and a concurrent progress save returns 200 throughout. Under the old code
  that save blocked for the whole scan.
- Behaviour change: a scan is no longer all-or-nothing. A crash now leaves a
  consistent partial scan. The scan is idempotent, so re-running finishes it.
- res.Errors is capped at 100 (a broken mount used to append one string per
  file, then serialise the lot into the job row); the overflow is counted.

Concurrent job workers (internal/jobs/jobs.go)
- Run() was single-threaded, so a scan stalled every enrichment job behind it.
  Now 4 workers with jittered idle waits.
- Handlers run concurrently, so Register/lookup take an RWMutex, and a
  panicking handler is recovered and marked failed rather than killing a worker
  and stranding a row in 'running'.
- RecoverStale() requeues jobs left 'running' by a dead process; called at boot
  from main.go. Previously such a job stayed 'running' forever and the UI
  polled it forever.

Explore (internal/media/discover.go)
- Full-scanned and sorted the whole library on every home load, and was
  deterministic, so it showed the same 30 items every visit. Now a per-kind
  fair-share id range scan with a random start, wrap-around, dedup and
  interleaving. The doc comment claimed per-kind sampling before; now it does.

Indexes (migrations/0008_query_indexes.sql)
- Benchmarked with ANALYZE over 60k items: browse page 1 13.37ms -> 0.03ms
  (484x, full scan -> covering index), deep page 8x, recently added 3.2x.
- A candidate index for the Continue shelf was built, measured and REJECTED:
  the planner never chose it because `finished = 0` selects the majority of a
  user's rows, so it had no selectivity and would have cost a write on every
  progress save. Recorded in the migration rather than shipped as dead weight.

Canonical timestamps (internal/core/stamp.go)
- time.RFC3339Nano trims trailing zeros, so its fraction width varies and the
  strings are not lexicographically sortable: ".5Z" > ".55Z" as strings though
  .500s < .550s. That broke every ORDER BY <timestamp> and MAX(timestamp),
  including the Continue shelf and Most Viewed. It was wrong even in columns
  written 100% by Go, so it was not a cross-format issue.
- All 14 write sites now use core.Stamp, matching SQLite's own default format
  exactly. Read paths are untouched: RFC3339Nano parses both shapes, so
  existing rows keep working.

Also: busy_timeout 5s -> 15s. Chunked scans still delay a concurrent write, and
measured tail latency on a first scan of 15k new files reached 3.3s, leaving
almost no margin at 5s. Waiting beats failing with SQLITE_BUSY and losing a
playback position. A 10ms inter-batch yield was implemented, measured, found to
have no effect distinguishable from noise (overlapping ranges), and removed.

Verified end to end, not just compiled: orphan recovery logged at boot, scan
returns 202 in 0.7ms, Explore varies across loads, concurrent writes survive a
scan. gofmt/vet clean, 14 packages pass, -race clean on the queue, svelte-check
0/0, fresh install applies all 8 migrations.
2026-10-11 13:10:34 -04:00

94 lines
3.1 KiB
Go

package core
import (
"context"
"path/filepath"
"regexp"
"strings"
"testing"
"time"
)
// stampShape is exactly: YYYY-MM-DDTHH:MM:SS.fffZ — three fractional digits and
// a literal Z, which is the shape SQLite's strftime('%Y-%m-%dT%H:%M:%fZ','now')
// emits.
var stampShape = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$`)
// (a) Stamp is fixed-width: exactly 3 fractional digits and a trailing Z, even
// for a fraction RFC3339Nano would trim.
func TestStampFixedWidth(t *testing.T) {
tm := time.Date(2026, 10, 11, 12, 0, 0, 500_000_000, time.UTC)
got := Stamp(tm)
if want := "2026-10-11T12:00:00.500Z"; got != want {
t.Fatalf("Stamp = %q, want %q", got, want)
}
if !strings.HasSuffix(got, "Z") {
t.Errorf("Stamp %q does not end in Z", got)
}
if !stampShape.MatchString(got) {
t.Errorf("Stamp %q is not fixed-width with exactly 3 fractional digits", got)
}
// The premise: RFC3339Nano trims the trailing zeros, which is why it cannot
// be used for a lexicographically sortable column.
if rfc := tm.Format(time.RFC3339Nano); rfc != "2026-10-11T12:00:00.5Z" {
t.Fatalf("RFC3339Nano = %q; test premise (trailing-zero trimming) is wrong", rfc)
}
}
// (b) Two Stamps 1ms apart must compare in the same order as the instants.
func TestStampOrdersChronologically(t *testing.T) {
a := time.Date(2026, 10, 11, 12, 0, 0, 0, time.UTC)
b := a.Add(time.Millisecond)
sa, sb := Stamp(a), Stamp(b)
if !(sa < sb) {
t.Errorf("Stamp %q is not < %q; string order != chronological order", sa, sb)
}
}
// (c) The exact case that was broken: .500s sorts BEFORE .550s as a string.
// With RFC3339Nano the opposite happens (".5Z" > ".55Z").
func TestStampBrokenCaseSortsCorrectly(t *testing.T) {
a := time.Date(2026, 10, 11, 12, 0, 0, 500_000_000, time.UTC) // .500s
b := time.Date(2026, 10, 11, 12, 0, 0, 550_000_000, time.UTC) // .550s
sa, sb := Stamp(a), Stamp(b)
if !(sa < sb) {
t.Errorf("Stamp %q should sort before %q", sa, sb)
}
// Demonstrate that the old format really was misordered, so this test is
// guarding a real regression rather than a tautology.
ra, rb := a.Format(time.RFC3339Nano), b.Format(time.RFC3339Nano)
if !(ra > rb) {
t.Fatalf("premise: RFC3339Nano %q,%q should compare backwards", ra, rb)
}
}
// (d) Stamp's output shape matches SQLite's own default byte-for-byte.
func TestStampMatchesSQLiteDefault(t *testing.T) {
db, err := OpenDB(filepath.Join(t.TempDir(), "stamp.db"))
if err != nil {
t.Fatalf("OpenDB: %v", err)
}
defer db.Close()
// A fixed second keeps the comparison deterministic. The format literal is
// copied verbatim from the column DEFAULTs in the migrations.
var sqliteStamp string
if err := db.QueryRowContext(context.Background(),
`SELECT strftime('%Y-%m-%dT%H:%M:%fZ','2026-10-11 12:00:00.500')`,
).Scan(&sqliteStamp); err != nil {
t.Fatalf("strftime: %v", err)
}
if !stampShape.MatchString(sqliteStamp) {
t.Fatalf("SQLite strftime output %q has an unexpected shape", sqliteStamp)
}
got := Stamp(time.Date(2026, 10, 11, 12, 0, 0, 500_000_000, time.UTC))
if got != sqliteStamp {
t.Fatalf("Stamp = %q, SQLite strftime = %q; they must match byte-for-byte", got, sqliteStamp)
}
}