Vellum/cmd/vellum/main.go
mithral c50b3cfb3b Wave 1: survive a real library
A 50k-file library used to wedge the database. This wave fixes the four
reasons why, plus a timestamp-ordering bug the work exposed.

Chunked scans (internal/library/scan.go)
- One transaction wrapped the whole reconciliation, holding SQLite's single
  write lock for the entire scan. Now 500 groups per transaction, and the two
  missing-file passes are batched too.
- Verified: a 20,000-file scan reports partial progress (scanned=1500 mid-scan)
  and a concurrent progress save returns 200 throughout. Under the old code
  that save blocked for the whole scan.
- Behaviour change: a scan is no longer all-or-nothing. A crash now leaves a
  consistent partial scan. The scan is idempotent, so re-running finishes it.
- res.Errors is capped at 100 (a broken mount used to append one string per
  file, then serialise the lot into the job row); the overflow is counted.

Concurrent job workers (internal/jobs/jobs.go)
- Run() was single-threaded, so a scan stalled every enrichment job behind it.
  Now 4 workers with jittered idle waits.
- Handlers run concurrently, so Register/lookup take an RWMutex, and a
  panicking handler is recovered and marked failed rather than killing a worker
  and stranding a row in 'running'.
- RecoverStale() requeues jobs left 'running' by a dead process; called at boot
  from main.go. Previously such a job stayed 'running' forever and the UI
  polled it forever.

Explore (internal/media/discover.go)
- Full-scanned and sorted the whole library on every home load, and was
  deterministic, so it showed the same 30 items every visit. Now a per-kind
  fair-share id range scan with a random start, wrap-around, dedup and
  interleaving. The doc comment claimed per-kind sampling before; now it does.

Indexes (migrations/0008_query_indexes.sql)
- Benchmarked with ANALYZE over 60k items: browse page 1 13.37ms -> 0.03ms
  (484x, full scan -> covering index), deep page 8x, recently added 3.2x.
- A candidate index for the Continue shelf was built, measured and REJECTED:
  the planner never chose it because `finished = 0` selects the majority of a
  user's rows, so it had no selectivity and would have cost a write on every
  progress save. Recorded in the migration rather than shipped as dead weight.

Canonical timestamps (internal/core/stamp.go)
- time.RFC3339Nano trims trailing zeros, so its fraction width varies and the
  strings are not lexicographically sortable: ".5Z" > ".55Z" as strings though
  .500s < .550s. That broke every ORDER BY <timestamp> and MAX(timestamp),
  including the Continue shelf and Most Viewed. It was wrong even in columns
  written 100% by Go, so it was not a cross-format issue.
- All 14 write sites now use core.Stamp, matching SQLite's own default format
  exactly. Read paths are untouched: RFC3339Nano parses both shapes, so
  existing rows keep working.

Also: busy_timeout 5s -> 15s. Chunked scans still delay a concurrent write, and
measured tail latency on a first scan of 15k new files reached 3.3s, leaving
almost no margin at 5s. Waiting beats failing with SQLITE_BUSY and losing a
playback position. A 10ms inter-batch yield was implemented, measured, found to
have no effect distinguishable from noise (overlapping ranges), and removed.

Verified end to end, not just compiled: orphan recovery logged at boot, scan
returns 202 in 0.7ms, Explore varies across loads, concurrent writes survive a
scan. gofmt/vet clean, 14 packages pass, -race clean on the queue, svelte-check
0/0, fresh install applies all 8 migrations.
2026-10-11 13:10:34 -04:00

200 lines
6.4 KiB
Go

// Command vellum is the unified media server: one process, one database,
// every media type.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/mithral/vellum/internal/auth"
"github.com/mithral/vellum/internal/cli"
"github.com/mithral/vellum/internal/core"
"github.com/mithral/vellum/internal/httpapi"
"github.com/mithral/vellum/internal/jobs"
"github.com/mithral/vellum/internal/library"
"github.com/mithral/vellum/internal/media"
"github.com/mithral/vellum/internal/metadata"
"github.com/mithral/vellum/internal/modules/comics"
"github.com/mithral/vellum/internal/modules/ebooks"
"github.com/mithral/vellum/internal/modules/games"
_ "github.com/mithral/vellum/internal/modules/photos" // registers the thumbnail extractor
"github.com/mithral/vellum/internal/search"
"github.com/mithral/vellum/migrations"
)
// version is overridden at build time with -ldflags "-X main.version=...".
var version = "dev"
func main() {
cfg, err := core.LoadConfig()
if err != nil {
fmt.Fprintln(os.Stderr, "vellum: fatal:", err)
os.Exit(1)
}
log := core.NewLogger(cfg.LogLevel, cfg.LogFormat)
// Administrative subcommands run instead of the server. This is the way
// back in when someone is locked out, and the way to take a backup while
// the server is stopped.
if handled, err := cli.Run(os.Args[1:], cfg, log); handled {
if err != nil {
fmt.Fprintln(os.Stderr, "vellum:", err)
os.Exit(1)
}
return
}
if err := run(cfg, log); err != nil {
fmt.Fprintln(os.Stderr, "vellum: fatal:", err)
os.Exit(1)
}
}
func run(cfg core.Config, log *slog.Logger) error {
log.Info("starting vellum", "version", version, "listen", cfg.Listen)
if err := cfg.Validate(); err != nil {
return err
}
db, err := core.OpenDB(cfg.DBPath)
if err != nil {
return err
}
defer db.Close()
migCtx, cancelMig := context.WithTimeout(context.Background(), 60*time.Second)
err = core.Migrate(migCtx, db, migrations.FS, log)
cancelMig()
if err != nil {
return err
}
api := httpapi.New(db, log, version)
authenticator := auth.NewAuthenticator(db, cfg.CookieName, cfg.SessionTTL, cfg.CookieSecure, log)
if !cfg.CookieSecure {
log.Warn("VELLUM_COOKIE_SECURE is false; session cookies will be sent over plain HTTP. " +
"Set it to true when serving over TLS.")
}
if err := auth.EnsureAdminFromEnv(context.Background(), db, log, cfg.AdminUser, cfg.AdminPassword); err != nil {
return fmt.Errorf("bootstrap admin: %w", err)
}
authHandlers := auth.NewHandlers(authenticator, log)
authHandlers.Register(api.Mux())
api.SetAuth(authenticator)
// Background work: a table-backed queue in the same database. Created
// before the modules so a module's handlers can enqueue work.
queue := jobs.New(db, log)
library.NewHandlers(db, log, authenticator, queue).Register(api.Mux())
media.NewHandlers(db, log, authenticator, cfg.ConfigDir).Register(api.Mux())
ebooks.NewHandlers(db, log, authenticator).Register(api.Mux())
comics.NewHandlers(db, log, authenticator).Register(api.Mux())
games.NewHandlers(db, log, authenticator).Register(api.Mux())
search.NewHandlers(db, log, authenticator).Register(api.Mux())
// Scans run on the queue rather than inside the request: a real library
// takes longer than any sensible timeout, and a hanging request is
// indistinguishable from a broken server. The result is written back so
// the UI can read progress.
queue.Register(library.ScanJobKind, func(ctx context.Context, payload json.RawMessage) error {
var p library.ScanPayload
if err := json.Unmarshal(payload, &p); err != nil {
return fmt.Errorf("scan: bad payload: %w", err)
}
lib, err := library.Get(ctx, db, p.LibraryID)
if err != nil {
return nil // deleted since the job was queued
}
res, err := library.Scan(ctx, db, lib, log)
if jobID, ok := jobs.JobID(ctx); ok {
if serr := jobs.SetResult(ctx, db, jobID, res); serr != nil {
log.Warn("record scan result", "err", serr)
}
}
return err
})
// Providers are registered primary-first; a fallback is only consulted
// when the primary returns nothing.
metadata.Register(metadata.NewOpenLibrary())
metadata.Register(metadata.NewGoogleBooks())
metadata.Register(metadata.NewMusicBrainz())
if cfg.ComicVineAPIKey != "" {
metadata.Register(metadata.NewComicVine(cfg.ComicVineAPIKey))
} else {
log.Info("comic metadata disabled: set VELLUM_COMICVINE_API_KEY to enable ComicVine lookups")
}
if cfg.IGDBClientID != "" && cfg.IGDBClientSecret != "" {
metadata.Register(metadata.NewIGDB(cfg.IGDBClientID, cfg.IGDBClientSecret))
} else {
log.Info("game metadata disabled: set VELLUM_IGDB_CLIENT_ID and VELLUM_IGDB_CLIENT_SECRET to enable IGDB lookups")
}
enricher := metadata.NewEnricher(db, log, cfg.ConfigDir)
queue.Register(metadata.JobKind, enricher.Handle)
metadata.NewHandlers(db, log, authenticator, queue).Register(api.Mux())
// Anything left 'running' belongs to a previous process that died mid-job.
// Requeue before the workers start, or it stays 'running' forever and the
// UI polls a job that will never move.
if n, err := queue.RecoverStale(context.Background()); err != nil {
log.Warn("recover stale jobs", "err", err)
} else if n > 0 {
log.Info("requeued jobs left running by a previous process", "count", n)
}
jobCtx, cancelJobs := context.WithCancel(context.Background())
defer cancelJobs()
go queue.Run(jobCtx)
srv := &http.Server{
Addr: cfg.Listen,
Handler: api.Handler(),
// ReadHeaderTimeout protects against slowloris. WriteTimeout is
// deliberately zero: streaming large media files must not be cut off.
ReadHeaderTimeout: 10 * time.Second,
ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout,
IdleTimeout: cfg.IdleTimeout,
ErrorLog: nil,
}
errCh := make(chan error, 1)
go func() {
log.Info("listening", "addr", cfg.Listen, "data_dir", cfg.DataDir)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
errCh <- err
}
}()
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)
select {
case err := <-errCh:
return fmt.Errorf("http server: %w", err)
case sig := <-stop:
log.Info("shutting down", "signal", sig.String())
}
shutdownCtx, cancel := context.WithTimeout(context.Background(), cfg.ShutdownGrace)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("graceful shutdown: %w", err)
}
cancelJobs()
log.Info("stopped")
return nil
}