diff --git a/DEPLOY.md b/DEPLOY.md new file mode 100644 index 0000000..332ed47 --- /dev/null +++ b/DEPLOY.md @@ -0,0 +1,145 @@ +# Deploying Vellum + +Vellum is a single Go binary in a single container with a single SQLite +database. This document covers a container deployment with `compose.yml`. + +## Prerequisites + +- **Podman or Docker** with the Compose plugin (`podman compose` or + `docker compose`). +- **A media folder** on the host, laid out one directory per media kind + (`ebooks/`, `comics/`, `audiobooks/`, `music/`, `photos/`, `games/`). + Vellum scans these; it never writes to them. +- A host directory for the database and cache (e.g. + `/opt/mediaserver/vellum/{data,config}`). + +The image is built locally with `make docker` (Podman) or +`docker build -t localhost/vellum:0.1.0-dev .`. + +## Configuration: `.env` + +Every setting is an environment variable with a working default, so Vellum +starts with no configuration at all. Copy the relevant values into a `.env` +next to `compose.yml`: + +```dotenv +# --- Media roots (colon-separated, read-only) ------------------------------- +VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics:/media/audiobooks:/media/music:/media/photos:/media/games + +# --- Server ----------------------------------------------------------------- +VELLUM_LISTEN=:7788 +VELLUM_LOG_LEVEL=info + +# --- First admin (only used on an EMPTY database) --------------------------- +VELLUM_ADMIN_USER=admin +VELLUM_ADMIN_PASSWORD=change-me-at-least-8-chars + +# --- Optional metadata providers -------------------------------------------- +VELLUM_COMICVINE_API_KEY= +VELLUM_IGDB_CLIENT_ID= +VELLUM_IGDB_CLIENT_SECRET= + +# --- Set to true once you serve over TLS ------------------------------------ +VELLUM_COOKIE_SECURE=false +``` + +The same names are what the process reads directly; `compose.yml` passes them +through. Defaults, for reference: `VELLUM_LISTEN=:7788`, +`VELLUM_LOG_LEVEL=info`, `VELLUM_COOKIE_SECURE=false`. + +## Volumes and SELinux + +`compose.yml` uses three kinds of mount: + +| Mount | Mode | Purpose | +| --- | --- | --- | +| `/data` | read/write | SQLite database (`vellum.db`) and persistent state | +| `/config` | read/write | Regenerable cache: covers, thumbnails, extracted pages | +| `/media/` | **read-only** | The media library | + +Each media kind is bind-mounted individually and read-only. **Vellum never +writes to `/media`.** If you add a media root, add a matching read-only mount +and a matching entry in `VELLUM_MEDIA_DIRS`. + +Every bind mount carries the `:z` suffix, which is **mandatory on SELinux +hosts** (this stack runs with `getenforce = Enforcing`). Without `:z` the +container cannot read the host paths and Vellum fails to start with permission +errors. `:z` relabels the files so the container's SELinux context may access +them; `:ro` keeps the media read-only. + +## First admin bootstrap + +Set `VELLUM_ADMIN_USER` and `VELLUM_ADMIN_PASSWORD` (minimum 8 characters) +before the first start. On boot, if the users table is empty, Vellum creates +that administrator. + +This happens **only on an empty database**. Setting these on an established +install does nothing — the credentials are ignored. If the database is empty +and both are unset, the server logs a warning and refuses to invent a default +password. Once your admin exists, remove the two variables from `.env`. + +## Reverse proxy and TLS + +Vellum speaks plain HTTP on `:7788`; `compose.yml` binds it to +`127.0.0.1:7788` so it is only reachable from the host. Put a reverse proxy +(nginx, Caddy, Traefik) in front of it to terminate TLS and forward to +`http://127.0.0.1:7788`. + +Once traffic arrives over HTTPS, set: + +```dotenv +VELLUM_COOKIE_SECURE=true +``` + +This marks session cookies `Secure`, so browsers only send them over TLS. When +it is `false` Vellum logs a warning at boot because session cookies can be sent +over plain HTTP. Restart the container after changing it. + +## Backup + +The entire persistent state is the SQLite database at `/data/vellum.db` (in +WAL mode). Two safe options: + +- **Stop the container, then copy the database.** + ```bash + podman compose stop vellum + cp /opt/mediaserver/vellum/data/vellum.db /backup/vellum-$(date +%F).db + podman compose start vellum + ``` +- **Take a consistent snapshot while running** with SQLite's `VACUUM INTO`: + ```bash + podman exec vellum \ + sqlite3 /data/vellum.db "VACUUM INTO '/data/backup.db'" + ``` + `VACUUM INTO` writes a single consistent file without stopping the server; + copy that file off the volume when it is done. + +`/config` holds only regenerable cache and does not need backing up. + +## Upgrading + +1. **Back up first** (see above). Migrations are one-way. +2. Pull or rebuild the image, then recreate the container: + ```bash + make docker + podman compose up -d + ``` + +**Migrations run automatically at boot.** They are embedded and applied in +filename order, recorded by number in `PRAGMA user_version`. Never edit an +applied migration; a new schema change is always a new file. If a migration +fails, the process exits and the container restarts — restore your backup. + +## Logs + +Vellum logs to **stdout/stderr**, so container logs are the log. With the +`json-file` driver configured in `compose.yml` (10 MB per file, 3 files): + +```bash +podman logs -f vellum # follow +podman logs --tail 100 vellum # last 100 lines +``` + +Verbosity is controlled by `VELLUM_LOG_LEVEL` (`debug`, `info`, `warn`, +`error`). Set `VELLUM_LOG_FORMAT=json` for structured output if you ship logs +to a collector. diff --git a/cmd/vellum/main.go b/cmd/vellum/main.go index fa8b72c..7b0f9b5 100644 --- a/cmd/vellum/main.go +++ b/cmd/vellum/main.go @@ -4,8 +4,10 @@ package main import ( "context" + "encoding/json" "errors" "fmt" + "log/slog" "net/http" "os" "os/signal" @@ -13,6 +15,7 @@ import ( "time" "github.com/mithral/vellum/internal/auth" + "github.com/mithral/vellum/internal/cli" "github.com/mithral/vellum/internal/core" "github.com/mithral/vellum/internal/httpapi" "github.com/mithral/vellum/internal/jobs" @@ -31,18 +34,31 @@ import ( var version = "dev" func main() { - if err := run(); err != nil { + cfg, err := core.LoadConfig() + if err != nil { + fmt.Fprintln(os.Stderr, "vellum: fatal:", err) + os.Exit(1) + } + log := core.NewLogger(cfg.LogLevel, cfg.LogFormat) + + // Administrative subcommands run instead of the server. This is the way + // back in when someone is locked out, and the way to take a backup while + // the server is stopped. + if handled, err := cli.Run(os.Args[1:], cfg, log); handled { + if err != nil { + fmt.Fprintln(os.Stderr, "vellum:", err) + os.Exit(1) + } + return + } + + if err := run(cfg, log); err != nil { fmt.Fprintln(os.Stderr, "vellum: fatal:", err) os.Exit(1) } } -func run() error { - cfg, err := core.LoadConfig() - if err != nil { - return err - } - log := core.NewLogger(cfg.LogLevel, cfg.LogFormat) +func run(cfg core.Config, log *slog.Logger) error { log.Info("starting vellum", "version", version, "listen", cfg.Listen) if err := cfg.Validate(); err != nil { @@ -78,17 +94,41 @@ func run() error { authHandlers.Register(api.Mux()) api.SetAuth(authenticator) - library.NewHandlers(db, log, authenticator).Register(api.Mux()) + // Background work: a table-backed queue in the same database. Created + // before the modules so a module's handlers can enqueue work. + queue := jobs.New(db, log) + + library.NewHandlers(db, log, authenticator, queue).Register(api.Mux()) media.NewHandlers(db, log, authenticator, cfg.ConfigDir).Register(api.Mux()) ebooks.NewHandlers(db, log, authenticator).Register(api.Mux()) comics.NewHandlers(db, log, authenticator).Register(api.Mux()) games.NewHandlers(db, log, authenticator).Register(api.Mux()) search.NewHandlers(db, log, authenticator).Register(api.Mux()) - // Background work: a table-backed queue in the same database. Providers - // are registered primary-first; a fallback is only consulted when the - // primary returns nothing. - queue := jobs.New(db, log) + // Scans run on the queue rather than inside the request: a real library + // takes longer than any sensible timeout, and a hanging request is + // indistinguishable from a broken server. The result is written back so + // the UI can read progress. + queue.Register(library.ScanJobKind, func(ctx context.Context, payload json.RawMessage) error { + var p library.ScanPayload + if err := json.Unmarshal(payload, &p); err != nil { + return fmt.Errorf("scan: bad payload: %w", err) + } + lib, err := library.Get(ctx, db, p.LibraryID) + if err != nil { + return nil // deleted since the job was queued + } + res, err := library.Scan(ctx, db, lib, log) + if jobID, ok := jobs.JobID(ctx); ok { + if serr := jobs.SetResult(ctx, db, jobID, res); serr != nil { + log.Warn("record scan result", "err", serr) + } + } + return err + }) + + // Providers are registered primary-first; a fallback is only consulted + // when the primary returns nothing. metadata.Register(metadata.NewOpenLibrary()) metadata.Register(metadata.NewGoogleBooks()) metadata.Register(metadata.NewMusicBrainz()) diff --git a/compose.yml b/compose.yml index 401d5f4..9f57a3c 100644 --- a/compose.yml +++ b/compose.yml @@ -1,6 +1,14 @@ # Vellum — one container, alongside the existing stack. # -# Mounts match the conventions used elsewhere in this stack: +# Volume layout: +# /data the SQLite database (vellum.db) and persistent state — read/write +# /config regenerable cache: covers, thumbnails, extracted pages — read/write +# /media/* the media library, one bind mount per kind, ALL read-only. +# +# Vellum never writes to /media. It only ever reads. If a media mount is not +# read-only, that is a bug in this file, not a feature. +# +# Mount flags match the conventions used elsewhere in this stack: # :z SELinux relabel (mandatory on this host, getenforce = Enforcing) # :ro the media library is read-only; Vellum never writes to it # @@ -29,13 +37,25 @@ services: - TZ=America/New_York - VELLUM_LISTEN=:7788 - VELLUM_LOG_LEVEL=info - # Read-only library roots, colon-separated. Add comics/music/etc. as - # those modules land. - - VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics + # Read-only library roots, colon-separated. Each entry must correspond to + # one of the read-only mounts below. + - VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics:/media/audiobooks:/media/music:/media/photos:/media/games + # Metadata providers (optional; leave blank to disable). + - VELLUM_IGDB_CLIENT_ID= + - VELLUM_IGDB_CLIENT_SECRET= + - VELLUM_COMICVINE_API_KEY= + # Set to true when serving over TLS behind a reverse proxy. + - VELLUM_COOKIE_SECURE=false volumes: - /opt/mediaserver/vellum/data:/data:z - /opt/mediaserver/vellum/config:/config:z - - /mnt/storage/media:/media:ro,z + # Media: one read-only mount per kind. The container never writes here. + - /mnt/storage/media/ebooks:/media/ebooks:ro,z + - /mnt/storage/media/comics:/media/comics:ro,z + - /mnt/storage/media/audiobooks:/media/audiobooks:ro,z + - /mnt/storage/media/music:/media/music:ro,z + - /mnt/storage/media/photos:/media/photos:ro,z + - /mnt/storage/media/games:/media/games:ro,z ports: - 127.0.0.1:7788:7788 logging: diff --git a/internal/cli/cli.go b/internal/cli/cli.go new file mode 100644 index 0000000..ad393f1 --- /dev/null +++ b/internal/cli/cli.go @@ -0,0 +1,149 @@ +// Package cli implements the vellum command-line subcommands that operate +// directly on the database: user administration and backups. +// +// cmd/vellum calls Run before booting the HTTP server. When Run reports +// handled == true the process exits without starting the server; when it +// reports handled == false the caller falls through to its default behaviour. +package cli + +import ( + "context" + "database/sql" + "errors" + "fmt" + "io" + "log/slog" + "os" + + "github.com/mithral/vellum/internal/auth" + "github.com/mithral/vellum/internal/core" + "github.com/mithral/vellum/migrations" +) + +// Run dispatches a CLI subcommand. args is os.Args[1:]. +// +// handled is false when args names no known subcommand, so the caller can +// continue with its normal startup. When handled is true the database has been +// opened and migrated, and a non-nil error is fatal to the invocation. +func Run(args []string, cfg core.Config, log *slog.Logger) (handled bool, err error) { + return run(context.Background(), args, cfg, log, os.Stdout) +} + +// run is the testable core of Run. It opens and migrates the database only +// once a subcommand has actually been matched, so an unrecognised argument +// leaves the database untouched. +func run(ctx context.Context, args []string, cfg core.Config, log *slog.Logger, out io.Writer) (bool, error) { + if len(args) == 0 { + return false, nil + } + switch args[0] { + case "user", "backup": + default: + return false, nil + } + + db, err := core.OpenDB(cfg.DBPath) + if err != nil { + return true, err + } + defer db.Close() + + if err := core.Migrate(ctx, db, migrations.FS, log); err != nil { + return true, err + } + + switch args[0] { + case "user": + return true, userCmd(ctx, db, args[1:], out) + case "backup": + return true, backupCmd(ctx, db, args[1:], out) + } + return false, nil +} + +// ── user ─────────────────────────────────────────────────────────────── + +func userCmd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error { + if len(args) == 0 { + return errors.New("user: missing subcommand (add, passwd, list)") + } + switch args[0] { + case "add": + return userAdd(ctx, db, args[1:], out) + case "passwd": + return userPasswd(ctx, db, args[1:], out) + case "list": + return userList(ctx, db, out) + default: + return fmt.Errorf("user: unknown subcommand %q", args[0]) + } +} + +func userAdd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error { + admin := false + var positional []string + for _, a := range args { + if a == "--admin" { + admin = true + continue + } + positional = append(positional, a) + } + if len(positional) != 2 { + return errors.New("usage: user add [--admin]") + } + + u, err := auth.CreateUser(ctx, db, positional[0], "", positional[1], admin) + if err != nil { + return err + } + _, err = fmt.Fprintf(out, "created user %d %s\n", u.ID, u.Username) + return err +} + +func userPasswd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error { + if len(args) != 2 { + return errors.New("usage: user passwd ") + } + username, password := args[0], args[1] + + u, err := auth.GetUserByUsername(ctx, db, username) + if errors.Is(err, sql.ErrNoRows) { + return fmt.Errorf("user %q not found", username) + } + if err != nil { + return err + } + if err := auth.SetPassword(ctx, db, u.ID, password); err != nil { + return err + } + _, err = fmt.Fprintf(out, "updated password for user %d %s\n", u.ID, u.Username) + return err +} + +func userList(ctx context.Context, db *sql.DB, out io.Writer) error { + users, err := auth.ListUsers(ctx, db) + if err != nil { + return err + } + for _, u := range users { + if _, err := fmt.Fprintf(out, "%d\t%s\tadmin=%t\n", u.ID, u.Username, u.IsAdmin); err != nil { + return err + } + } + return nil +} + +// ── backup ───────────────────────────────────────────────────────────── + +func backupCmd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error { + if len(args) != 1 { + return errors.New("usage: backup ") + } + path := args[0] + if _, err := db.ExecContext(ctx, "VACUUM INTO ?", path); err != nil { + return fmt.Errorf("backup: %w", err) + } + _, err := fmt.Fprintf(out, "backed up database to %s\n", path) + return err +} diff --git a/internal/cli/cli_test.go b/internal/cli/cli_test.go new file mode 100644 index 0000000..04e1f04 --- /dev/null +++ b/internal/cli/cli_test.go @@ -0,0 +1,181 @@ +package cli + +import ( + "bytes" + "context" + "io" + "log/slog" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mithral/vellum/internal/auth" + "github.com/mithral/vellum/internal/core" +) + +// testConfig points a Config at a fresh database inside a temp dir. The CLI +// itself opens and migrates it, mirroring the auth_test.go setup pattern. +func testConfig(t *testing.T) core.Config { + t.Helper() + return core.Config{DBPath: filepath.Join(t.TempDir(), "vellum.db")} +} + +func testLogger() *slog.Logger { + return slog.New(slog.NewTextHandler(io.Discard, nil)) +} + +// runCapturing runs the exported entrypoint with os.Stdout redirected so the +// subcommand's output can be asserted on. +func runCapturing(t *testing.T, args []string, cfg core.Config, log *slog.Logger) (string, bool, error) { + t.Helper() + + old := os.Stdout + r, w, err := os.Pipe() + if err != nil { + t.Fatalf("pipe: %v", err) + } + os.Stdout = w + + handled, runErr := Run(args, cfg, log) + + os.Stdout = old + if err := w.Close(); err != nil { + t.Fatalf("close stdout pipe: %v", err) + } + out, err := io.ReadAll(r) + if err != nil { + t.Fatalf("read stdout: %v", err) + } + _ = r.Close() + return string(out), handled, runErr +} + +func TestRunUserAddAndList(t *testing.T) { + cfg := testConfig(t) + log := testLogger() + + if _, handled, err := runCapturing(t, []string{"user", "add", "alice", "password123"}, cfg, log); !handled || err != nil { + t.Fatalf("user add alice: handled=%v err=%v", handled, err) + } + if _, handled, err := runCapturing(t, []string{"user", "add", "root", "password123", "--admin"}, cfg, log); !handled || err != nil { + t.Fatalf("user add root --admin: handled=%v err=%v", handled, err) + } + + out, handled, err := runCapturing(t, []string{"user", "list"}, cfg, log) + if !handled { + t.Fatal("user list was not handled") + } + if err != nil { + t.Fatalf("user list: %v", err) + } + + lines := strings.Split(strings.TrimSpace(out), "\n") + if len(lines) != 2 { + t.Fatalf("user list printed %d lines, want 2:\n%s", len(lines), out) + } + if !strings.Contains(lines[0], "alice") || strings.Contains(lines[0], "admin=true") { + t.Errorf("alice line = %q, want non-admin alice", lines[0]) + } + if !strings.Contains(lines[1], "root") || !strings.Contains(lines[1], "admin=true") { + t.Errorf("root line = %q, want admin root", lines[1]) + } + + // The password written through the CLI must actually authenticate. + db, err := core.OpenDB(cfg.DBPath) + if err != nil { + t.Fatalf("reopen db: %v", err) + } + defer db.Close() + if _, err := auth.Authenticate(context.Background(), db, "alice", "password123"); err != nil { + t.Fatalf("alice cannot authenticate after CLI add: %v", err) + } +} + +func TestRunUserPasswd(t *testing.T) { + cfg := testConfig(t) + log := testLogger() + ctx := context.Background() + + if _, handled, err := runCapturing(t, []string{"user", "add", "bob", "oldpassword"}, cfg, log); !handled || err != nil { + t.Fatalf("user add bob: handled=%v err=%v", handled, err) + } + + // The password change goes through GetUserByUsername + SetPassword. + if _, handled, err := runCapturing(t, []string{"user", "passwd", "bob", "newpassword"}, cfg, log); !handled || err != nil { + t.Fatalf("user passwd bob: handled=%v err=%v", handled, err) + } + + db, err := core.OpenDB(cfg.DBPath) + if err != nil { + t.Fatalf("reopen db: %v", err) + } + defer db.Close() + if _, err := auth.Authenticate(ctx, db, "bob", "newpassword"); err != nil { + t.Fatalf("new password does not authenticate: %v", err) + } + if _, err := auth.Authenticate(ctx, db, "bob", "oldpassword"); err == nil { + t.Error("old password still authenticates after passwd") + } + + // Unknown users are reported cleanly rather than as sql.ErrNoRows. + if _, handled, err := runCapturing(t, []string{"user", "passwd", "nobody", "whatever"}, cfg, log); !handled || err == nil { + t.Errorf("passwd for unknown user: handled=%v err=%v, want handled error", handled, err) + } +} + +func TestRunBackup(t *testing.T) { + cfg := testConfig(t) + log := testLogger() + ctx := context.Background() + + // Seed a row so the backup has something to carry across. + if _, handled, err := runCapturing(t, []string{"user", "add", "carol", "password123"}, cfg, log); !handled || err != nil { + t.Fatalf("user add carol: handled=%v err=%v", handled, err) + } + + dest := filepath.Join(t.TempDir(), "backup.db") + handled, err := run(ctx, []string{"backup", dest}, cfg, log, io.Discard) + if !handled { + t.Fatal("backup was not handled") + } + if err != nil { + t.Fatalf("backup: %v", err) + } + + info, err := os.Stat(dest) + if err != nil { + t.Fatalf("stat backup: %v", err) + } + if info.Size() == 0 { + t.Fatal("backup file is empty") + } + + // The backup must be a complete, independently usable database. + bdb, err := core.OpenDB(dest) + if err != nil { + t.Fatalf("open backup: %v", err) + } + defer bdb.Close() + if _, err := auth.GetUserByUsername(ctx, bdb, "carol"); err != nil { + t.Fatalf("backup is missing the seeded user: %v", err) + } +} + +func TestRunUnhandled(t *testing.T) { + cfg := testConfig(t) + var buf bytes.Buffer + + handled, err := run(context.Background(), []string{"serve"}, cfg, testLogger(), &buf) + if handled { + t.Error("unknown command reported handled=true") + } + if err != nil { + t.Errorf("unknown command returned error: %v", err) + } + + handled, err = run(context.Background(), nil, cfg, testLogger(), &buf) + if handled || err != nil { + t.Errorf("empty args: handled=%v err=%v, want false,nil", handled, err) + } +} diff --git a/internal/jobs/jobs.go b/internal/jobs/jobs.go index aeeada3..08af1a5 100644 --- a/internal/jobs/jobs.go +++ b/internal/jobs/jobs.go @@ -58,6 +58,29 @@ func (q *Queue) Enqueue(ctx context.Context, kind string, payload any) error { return q.EnqueueAfter(ctx, kind, payload, 0) } +// EnqueueRef adds a job that refers to a specific entity (a library, an item), +// so its state and result can be looked up by that id later. +func (q *Queue) EnqueueRef(ctx context.Context, kind string, refID int64, payload any) (int64, error) { + body := "{}" + if payload != nil { + raw, err := json.Marshal(payload) + if err != nil { + return 0, fmt.Errorf("jobs: marshal payload: %w", err) + } + body = string(raw) + } + var id int64 + err := q.db.QueryRowContext(ctx, ` + INSERT INTO jobs (kind, ref_id, payload, state, run_after) + VALUES (?, ?, ?, ?, ?) + RETURNING id`, + kind, refID, body, StateQueued, time.Now().UTC().Format(time.RFC3339Nano)).Scan(&id) + if err != nil { + return 0, fmt.Errorf("jobs: enqueue %s: %w", kind, err) + } + return id, nil +} + // EnqueueAfter adds a job that will not run before delay has elapsed. func (q *Queue) EnqueueAfter(ctx context.Context, kind string, payload any, delay time.Duration) error { body := "{}" @@ -122,7 +145,7 @@ func (q *Queue) RunOnce(ctx context.Context) (bool, error) { return true, nil } - if err := handler(ctx, job.Payload); err != nil { + if err := handler(withJobID(ctx, job.ID), job.Payload); err != nil { q.log.Warn("job failed", "kind", job.Kind, "id", job.ID, "attempt", job.Attempts, "err", err) q.finish(ctx, job, err) return true, nil @@ -206,6 +229,31 @@ func (q *Queue) failPermanently(ctx context.Context, job claimedJob, err error) } } +// jobCtxKey carries the running job's id to its handler, so a handler can +// attach a result without the queue needing to know the result's shape. +type jobCtxKey struct{} + +func withJobID(ctx context.Context, id int64) context.Context { + return context.WithValue(ctx, jobCtxKey{}, id) +} + +// JobID returns the id of the job currently being handled. +func JobID(ctx context.Context) (int64, bool) { + id, ok := ctx.Value(jobCtxKey{}).(int64) + return id, ok +} + +// SetResult stores a job's result so it can be read back later. Best-effort: +// a handler that produced a result should not fail because recording it did. +func SetResult(ctx context.Context, db *sql.DB, jobID int64, v any) error { + body, err := json.Marshal(v) + if err != nil { + return fmt.Errorf("jobs: marshal result: %w", err) + } + _, err = db.ExecContext(ctx, `UPDATE jobs SET result = ? WHERE id = ?`, string(body), jobID) + return err +} + // Stats reports job counts by state, for the health endpoint and debugging. func (q *Queue) Stats(ctx context.Context) (map[string]int, error) { rows, err := q.db.QueryContext(ctx, `SELECT state, count(*) FROM jobs GROUP BY state`) diff --git a/internal/library/handlers.go b/internal/library/handlers.go index 7c2c45a..7e70b8e 100644 --- a/internal/library/handlers.go +++ b/internal/library/handlers.go @@ -2,6 +2,7 @@ package library import ( "database/sql" + "encoding/json" "errors" "log/slog" "net/http" @@ -9,18 +10,28 @@ import ( "github.com/mithral/vellum/internal/auth" "github.com/mithral/vellum/internal/httpx" + "github.com/mithral/vellum/internal/jobs" ) +// ScanJobKind is the queue job that scans a library. +const ScanJobKind = "scan_library" + +// ScanPayload is the job payload for a scan. +type ScanPayload struct { + LibraryID int64 `json:"library_id"` +} + // Handlers serves library management and scanning endpoints. type Handlers struct { - db *sql.DB - log *slog.Logger - auth *auth.Authenticator + db *sql.DB + log *slog.Logger + auth *auth.Authenticator + queue *jobs.Queue } // NewHandlers builds the library HTTP layer. -func NewHandlers(db *sql.DB, log *slog.Logger, a *auth.Authenticator) *Handlers { - return &Handlers{db: db, log: log, auth: a} +func NewHandlers(db *sql.DB, log *slog.Logger, a *auth.Authenticator, q *jobs.Queue) *Handlers { + return &Handlers{db: db, log: log, auth: a, queue: q} } // Register mounts the routes. @@ -30,6 +41,7 @@ func (h *Handlers) Register(mux *http.ServeMux) { mux.Handle("GET /api/v1/libraries", h.auth.Require(h.handleList)) mux.Handle("GET /api/v1/libraries/{id}", h.auth.Require(h.handleGet)) mux.Handle("GET /api/v1/libraries/{id}/items", h.auth.Require(h.handleListItems)) + mux.Handle("GET /api/v1/libraries/{id}/scan/status", h.auth.RequireAuth(http.HandlerFunc(h.handleScanStatus))) mux.Handle("POST /api/v1/libraries", h.auth.RequireAdmin(http.HandlerFunc(h.handleCreate))) mux.Handle("DELETE /api/v1/libraries/{id}", h.auth.RequireAdmin(http.HandlerFunc(h.handleDelete))) @@ -146,12 +158,48 @@ func (h *Handlers) handleScan(w http.ResponseWriter, r *http.Request) { return } - res, err := Scan(r.Context(), h.db, lib, h.log) + // Queued rather than run inline: a real library takes longer than any + // sensible request timeout, and a hanging request is indistinguishable + // from a broken server. + jobID, err := h.queue.EnqueueRef(r.Context(), ScanJobKind, lib.ID, ScanPayload{LibraryID: lib.ID}) if err != nil { - h.fail(w, "scan library", err) + h.fail(w, "enqueue scan", err) return } - httpx.WriteJSON(w, http.StatusOK, res) + h.log.Info("scan queued", "library", lib.Name, "job", jobID) + httpx.WriteJSON(w, http.StatusAccepted, map[string]any{"queued": 1, "job_id": jobID}) +} + +// handleScanStatus reports the most recent scan for a library, so the UI can +// show progress instead of hanging on the request. +func (h *Handlers) handleScanStatus(w http.ResponseWriter, r *http.Request) { + id, ok := pathInt(w, r, "id") + if !ok { + return + } + + var state, result, errMsg string + err := h.db.QueryRowContext(r.Context(), ` + SELECT state, result, error FROM jobs + WHERE kind = ? AND ref_id = ? + ORDER BY id DESC LIMIT 1`, ScanJobKind, id).Scan(&state, &result, &errMsg) + if errors.Is(err, sql.ErrNoRows) { + httpx.WriteJSON(w, http.StatusOK, map[string]any{"state": "idle"}) + return + } + if err != nil { + h.fail(w, "scan status", err) + return + } + + out := map[string]any{"state": state} + if result != "" { + out["result"] = json.RawMessage(result) + } + if errMsg != "" { + out["error"] = errMsg + } + httpx.WriteJSON(w, http.StatusOK, out) } // handleListItems returns items in a library the caller may read. diff --git a/internal/media/items.go b/internal/media/items.go index bb87495..2c412e9 100644 --- a/internal/media/items.go +++ b/internal/media/items.go @@ -31,18 +31,36 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut h.fail(w, "visible libraries", err) return } + + limit := 300 + if v := r.URL.Query().Get("limit"); v != "" { + if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 1000 { + limit = n + } + } + + offset := 0 + if v := r.URL.Query().Get("offset"); v != "" { + if n, err := strconv.Atoi(v); err == nil && n >= 0 { + offset = n + } + } + if len(libs) == 0 { - httpx.WriteJSON(w, http.StatusOK, map[string]any{"items": []itemRow{}, "total": 0}) + httpx.WriteJSON(w, http.StatusOK, map[string]any{ + "items": []itemRow{}, + "total": 0, + "offset": offset, + "limit": limit, + }) return } in, args := inClause(libs) - query := `SELECT id, library_id, kind, path, title, COALESCE(year, 0) - FROM media_items - WHERE missing = 0 AND library_id IN (` + in + `)` + where := `missing = 0 AND library_id IN (` + in + `)` if kind := r.URL.Query().Get("kind"); kind != "" { - query += ` AND kind = ?` + where += ` AND kind = ?` args = append(args, kind) } if libStr := r.URL.Query().Get("library_id"); libStr != "" { @@ -51,20 +69,24 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut httpx.WriteError(w, http.StatusBadRequest, "invalid library_id") return } - query += ` AND library_id = ?` + where += ` AND library_id = ?` args = append(args, libID) } - limit := 300 - if v := r.URL.Query().Get("limit"); v != "" { - if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 1000 { - limit = n - } + // total counts every matching row, independent of the page window. + var total int + if err := h.db.QueryRowContext(ctx, + `SELECT COUNT(*) FROM media_items WHERE `+where, args...).Scan(&total); err != nil { + h.fail(w, "count items", err) + return } - query += ` ORDER BY sort_title LIMIT ?` - args = append(args, limit) - rows, err := h.db.QueryContext(ctx, query, args...) + query := `SELECT id, library_id, kind, path, title, COALESCE(year, 0) + FROM media_items + WHERE ` + where + ` ORDER BY sort_title LIMIT ? OFFSET ?` + pageArgs := append(append([]any{}, args...), limit, offset) + + rows, err := h.db.QueryContext(ctx, query, pageArgs...) if err != nil { h.fail(w, "list items", err) return @@ -85,5 +107,10 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut return } - httpx.WriteJSON(w, http.StatusOK, map[string]any{"items": items, "total": len(items)}) + httpx.WriteJSON(w, http.StatusOK, map[string]any{ + "items": items, + "total": total, + "offset": offset, + "limit": limit, + }) } diff --git a/migrations/0007_job_ref.sql b/migrations/0007_job_ref.sql new file mode 100644 index 0000000..56a60a6 --- /dev/null +++ b/migrations/0007_job_ref.sql @@ -0,0 +1,10 @@ +-- 0007_job_ref.sql — let a job be found by what it operates on. +-- +-- Scans need their progress read back: the UI asks "what is the state of the +-- scan for library 3?" That requires the job to carry the id it refers to and +-- to store its result, neither of which the queue had. + +ALTER TABLE jobs ADD COLUMN ref_id INTEGER; +ALTER TABLE jobs ADD COLUMN result TEXT NOT NULL DEFAULT ''; + +CREATE INDEX idx_jobs_kind_ref ON jobs(kind, ref_id, id); diff --git a/web/src/lib/api.ts b/web/src/lib/api.ts index 89448b3..fa08744 100644 --- a/web/src/lib/api.ts +++ b/web/src/lib/api.ts @@ -118,7 +118,24 @@ export const api = { deleteLibrary: (id: number) => request('DELETE', `/api/v1/libraries/${id}`), - scanLibrary: (id: number) => request('POST', `/api/v1/libraries/${id}/scan`), + scanLibrary: (id: number) => + request<{ queued: number; job_id: number }>('POST', `/api/v1/libraries/${id}/scan`), + + scanStatus: (id: number) => + request<{ + state: 'queued' | 'running' | 'done' | 'failed' | 'idle' + result?: { + scanned: number + added: number + updated: number + restored: number + missing: number + unchanged: number + duration_ms: number + errors?: string[] + } + error?: string + }>('GET', `/api/v1/libraries/${id}/scan/status`), enrichLibrary: (id: number) => request<{ queued: number }>('POST', `/api/v1/libraries/${id}/enrich`), @@ -154,12 +171,16 @@ export const api = { deleteSave: (id: number, slot = 'default') => request('DELETE', `/api/v1/items/${id}/saves/${slot}`), - listItems: (opts: { kind?: string; libraryId?: number; limit?: number } = {}) => { + listItems: (opts: { kind?: string; libraryId?: number; limit?: number; offset?: number } = {}) => { const p = new URLSearchParams() if (opts.kind) p.set('kind', opts.kind) if (opts.libraryId) p.set('library_id', String(opts.libraryId)) if (opts.limit) p.set('limit', String(opts.limit)) - return request<{ items: Item[]; total: number }>('GET', `/api/v1/items?${p.toString()}`) + if (opts.offset) p.set('offset', String(opts.offset)) + return request<{ items: Item[]; total: number; offset: number; limit: number }>( + 'GET', + `/api/v1/items?${p.toString()}`, + ) }, ebook: (id: number) => request<{ metadata: EbookMetadata }>('GET', `/api/v1/items/${id}/ebook`), diff --git a/web/src/lib/components/Browse.svelte b/web/src/lib/components/Browse.svelte index 7668f60..b477d99 100644 --- a/web/src/lib/components/Browse.svelte +++ b/web/src/lib/components/Browse.svelte @@ -4,13 +4,17 @@ import { go } from '../nav' import Card from './Card.svelte' + const PAGE_SIZE = 60 + let { kind = undefined, libraryId = undefined, }: { kind?: string; libraryId?: number } = $props() let items = $state([]) + let total = $state(0) let loading = $state(true) + let loadingMore = $state(false) let error = $state(null) $effect(() => { @@ -21,8 +25,9 @@ loading = true error = null try { - const res = await api.listItems({ kind: k, libraryId: lib }) + const res = await api.listItems({ kind: k, libraryId: lib, limit: PAGE_SIZE, offset: 0 }) items = res.items + total = res.total } catch (err) { error = err instanceof ApiError ? err.message : 'could not load items' } finally { @@ -30,6 +35,26 @@ } } + async function loadMore() { + if (loadingMore || items.length >= total) return + loadingMore = true + error = null + try { + const res = await api.listItems({ + kind, + libraryId, + limit: PAGE_SIZE, + offset: items.length, + }) + items = [...items, ...res.items] + total = res.total + } catch (err) { + error = err instanceof ApiError ? err.message : 'could not load more items' + } finally { + loadingMore = false + } + } + const types = $derived(($mediaTypes as { Kind: string; Name: string }[])) @@ -48,7 +73,7 @@
{#each Array(12) as _}
{/each}
-{:else if error} +{:else if error && items.length === 0}
{error}
{:else if items.length === 0}
@@ -56,11 +81,22 @@

No items match this section yet.

{:else} +

Showing {items.length} of {total}

{#each items as item (item.id)} {/each}
+ {#if error} +
{error}
+ {/if} + {#if items.length < total} +
+ +
+ {/if} {/if}