Beta blockers: async scans, admin CLI, deploy docs, pagination, password UI

Five tasks, four delegated to agents on disjoint files and one on the critical
path. Board and verification log: ~/.opencode/plan/vellum-beta-tasks.md

T1 Async scans (coordinator)
- migration 0007 adds ref_id + result to jobs, indexed by (kind, ref_id, id),
  so a job can be found by what it operates on and can report back.
- jobs: EnqueueRef, JobID(ctx) so a handler can attach a result, SetResult.
- POST /libraries/{id}/scan now enqueues and returns 202; a real library takes
  longer than any request timeout, and a hanging request looks like a broken
  server. GET /libraries/{id}/scan/status reports the latest run.
- Libraries.svelte polls and shows "Scanning…" instead of freezing.

T2 Admin CLI (agent)
- internal/cli: `user add|passwd|list`, `backup <path>` via VACUUM INTO, with
  tests. Wired into main.go, which the coordinator kept.

T3 Deploy (agent)
- compose.yml now mounts all six media kinds read-only with :z and carries the
  metadata env keys. DEPLOY.md covers volumes, SELinux, admin bootstrap, TLS,
  backup and upgrade.

T4 Pagination (agent)
- GET /items takes offset and returns total; Browse loads 60 at a time with
  "Load more" and "Showing X of Y" instead of silently truncating at 300.

T5 Password change UI (agent)
- A Password section in Settings with validation, busy and success states.

Verified end to end: scan returns 202 in 0.7 ms and polls to done; the CLI
created a user, changed a password, and produced a complete 240 KB backup while
the server ran; pagination reports total 8 and pages correctly; compose parses
with every mount :z. go vet and 14 test packages pass; svelte-check 0/0.
This commit is contained in:
mithral 2026-10-10 18:00:35 -04:00
parent 28d26f18c9
commit f680e1d9f6
13 changed files with 932 additions and 49 deletions

145
DEPLOY.md Normal file
View file

@ -0,0 +1,145 @@
# Deploying Vellum
Vellum is a single Go binary in a single container with a single SQLite
database. This document covers a container deployment with `compose.yml`.
## Prerequisites
- **Podman or Docker** with the Compose plugin (`podman compose` or
`docker compose`).
- **A media folder** on the host, laid out one directory per media kind
(`ebooks/`, `comics/`, `audiobooks/`, `music/`, `photos/`, `games/`).
Vellum scans these; it never writes to them.
- A host directory for the database and cache (e.g.
`/opt/mediaserver/vellum/{data,config}`).
The image is built locally with `make docker` (Podman) or
`docker build -t localhost/vellum:0.1.0-dev .`.
## Configuration: `.env`
Every setting is an environment variable with a working default, so Vellum
starts with no configuration at all. Copy the relevant values into a `.env`
next to `compose.yml`:
```dotenv
# --- Media roots (colon-separated, read-only) -------------------------------
VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics:/media/audiobooks:/media/music:/media/photos:/media/games
# --- Server -----------------------------------------------------------------
VELLUM_LISTEN=:7788
VELLUM_LOG_LEVEL=info
# --- First admin (only used on an EMPTY database) ---------------------------
VELLUM_ADMIN_USER=admin
VELLUM_ADMIN_PASSWORD=change-me-at-least-8-chars
# --- Optional metadata providers --------------------------------------------
VELLUM_COMICVINE_API_KEY=
VELLUM_IGDB_CLIENT_ID=
VELLUM_IGDB_CLIENT_SECRET=
# --- Set to true once you serve over TLS ------------------------------------
VELLUM_COOKIE_SECURE=false
```
The same names are what the process reads directly; `compose.yml` passes them
through. Defaults, for reference: `VELLUM_LISTEN=:7788`,
`VELLUM_LOG_LEVEL=info`, `VELLUM_COOKIE_SECURE=false`.
## Volumes and SELinux
`compose.yml` uses three kinds of mount:
| Mount | Mode | Purpose |
| --- | --- | --- |
| `/data` | read/write | SQLite database (`vellum.db`) and persistent state |
| `/config` | read/write | Regenerable cache: covers, thumbnails, extracted pages |
| `/media/<kind>` | **read-only** | The media library |
Each media kind is bind-mounted individually and read-only. **Vellum never
writes to `/media`.** If you add a media root, add a matching read-only mount
and a matching entry in `VELLUM_MEDIA_DIRS`.
Every bind mount carries the `:z` suffix, which is **mandatory on SELinux
hosts** (this stack runs with `getenforce = Enforcing`). Without `:z` the
container cannot read the host paths and Vellum fails to start with permission
errors. `:z` relabels the files so the container's SELinux context may access
them; `:ro` keeps the media read-only.
## First admin bootstrap
Set `VELLUM_ADMIN_USER` and `VELLUM_ADMIN_PASSWORD` (minimum 8 characters)
before the first start. On boot, if the users table is empty, Vellum creates
that administrator.
This happens **only on an empty database**. Setting these on an established
install does nothing — the credentials are ignored. If the database is empty
and both are unset, the server logs a warning and refuses to invent a default
password. Once your admin exists, remove the two variables from `.env`.
## Reverse proxy and TLS
Vellum speaks plain HTTP on `:7788`; `compose.yml` binds it to
`127.0.0.1:7788` so it is only reachable from the host. Put a reverse proxy
(nginx, Caddy, Traefik) in front of it to terminate TLS and forward to
`http://127.0.0.1:7788`.
Once traffic arrives over HTTPS, set:
```dotenv
VELLUM_COOKIE_SECURE=true
```
This marks session cookies `Secure`, so browsers only send them over TLS. When
it is `false` Vellum logs a warning at boot because session cookies can be sent
over plain HTTP. Restart the container after changing it.
## Backup
The entire persistent state is the SQLite database at `/data/vellum.db` (in
WAL mode). Two safe options:
- **Stop the container, then copy the database.**
```bash
podman compose stop vellum
cp /opt/mediaserver/vellum/data/vellum.db /backup/vellum-$(date +%F).db
podman compose start vellum
```
- **Take a consistent snapshot while running** with SQLite's `VACUUM INTO`:
```bash
podman exec vellum \
sqlite3 /data/vellum.db "VACUUM INTO '/data/backup.db'"
```
`VACUUM INTO` writes a single consistent file without stopping the server;
copy that file off the volume when it is done.
`/config` holds only regenerable cache and does not need backing up.
## Upgrading
1. **Back up first** (see above). Migrations are one-way.
2. Pull or rebuild the image, then recreate the container:
```bash
make docker
podman compose up -d
```
**Migrations run automatically at boot.** They are embedded and applied in
filename order, recorded by number in `PRAGMA user_version`. Never edit an
applied migration; a new schema change is always a new file. If a migration
fails, the process exits and the container restarts — restore your backup.
## Logs
Vellum logs to **stdout/stderr**, so container logs are the log. With the
`json-file` driver configured in `compose.yml` (10 MB per file, 3 files):
```bash
podman logs -f vellum # follow
podman logs --tail 100 vellum # last 100 lines
```
Verbosity is controlled by `VELLUM_LOG_LEVEL` (`debug`, `info`, `warn`,
`error`). Set `VELLUM_LOG_FORMAT=json` for structured output if you ship logs
to a collector.

View file

@ -4,8 +4,10 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"os/signal"
@ -13,6 +15,7 @@ import (
"time"
"github.com/mithral/vellum/internal/auth"
"github.com/mithral/vellum/internal/cli"
"github.com/mithral/vellum/internal/core"
"github.com/mithral/vellum/internal/httpapi"
"github.com/mithral/vellum/internal/jobs"
@ -31,18 +34,31 @@ import (
var version = "dev"
func main() {
if err := run(); err != nil {
cfg, err := core.LoadConfig()
if err != nil {
fmt.Fprintln(os.Stderr, "vellum: fatal:", err)
os.Exit(1)
}
log := core.NewLogger(cfg.LogLevel, cfg.LogFormat)
// Administrative subcommands run instead of the server. This is the way
// back in when someone is locked out, and the way to take a backup while
// the server is stopped.
if handled, err := cli.Run(os.Args[1:], cfg, log); handled {
if err != nil {
fmt.Fprintln(os.Stderr, "vellum:", err)
os.Exit(1)
}
return
}
if err := run(cfg, log); err != nil {
fmt.Fprintln(os.Stderr, "vellum: fatal:", err)
os.Exit(1)
}
}
func run() error {
cfg, err := core.LoadConfig()
if err != nil {
return err
}
log := core.NewLogger(cfg.LogLevel, cfg.LogFormat)
func run(cfg core.Config, log *slog.Logger) error {
log.Info("starting vellum", "version", version, "listen", cfg.Listen)
if err := cfg.Validate(); err != nil {
@ -78,17 +94,41 @@ func run() error {
authHandlers.Register(api.Mux())
api.SetAuth(authenticator)
library.NewHandlers(db, log, authenticator).Register(api.Mux())
// Background work: a table-backed queue in the same database. Created
// before the modules so a module's handlers can enqueue work.
queue := jobs.New(db, log)
library.NewHandlers(db, log, authenticator, queue).Register(api.Mux())
media.NewHandlers(db, log, authenticator, cfg.ConfigDir).Register(api.Mux())
ebooks.NewHandlers(db, log, authenticator).Register(api.Mux())
comics.NewHandlers(db, log, authenticator).Register(api.Mux())
games.NewHandlers(db, log, authenticator).Register(api.Mux())
search.NewHandlers(db, log, authenticator).Register(api.Mux())
// Background work: a table-backed queue in the same database. Providers
// are registered primary-first; a fallback is only consulted when the
// primary returns nothing.
queue := jobs.New(db, log)
// Scans run on the queue rather than inside the request: a real library
// takes longer than any sensible timeout, and a hanging request is
// indistinguishable from a broken server. The result is written back so
// the UI can read progress.
queue.Register(library.ScanJobKind, func(ctx context.Context, payload json.RawMessage) error {
var p library.ScanPayload
if err := json.Unmarshal(payload, &p); err != nil {
return fmt.Errorf("scan: bad payload: %w", err)
}
lib, err := library.Get(ctx, db, p.LibraryID)
if err != nil {
return nil // deleted since the job was queued
}
res, err := library.Scan(ctx, db, lib, log)
if jobID, ok := jobs.JobID(ctx); ok {
if serr := jobs.SetResult(ctx, db, jobID, res); serr != nil {
log.Warn("record scan result", "err", serr)
}
}
return err
})
// Providers are registered primary-first; a fallback is only consulted
// when the primary returns nothing.
metadata.Register(metadata.NewOpenLibrary())
metadata.Register(metadata.NewGoogleBooks())
metadata.Register(metadata.NewMusicBrainz())

View file

@ -1,6 +1,14 @@
# Vellum — one container, alongside the existing stack.
#
# Mounts match the conventions used elsewhere in this stack:
# Volume layout:
# /data the SQLite database (vellum.db) and persistent state — read/write
# /config regenerable cache: covers, thumbnails, extracted pages — read/write
# /media/* the media library, one bind mount per kind, ALL read-only.
#
# Vellum never writes to /media. It only ever reads. If a media mount is not
# read-only, that is a bug in this file, not a feature.
#
# Mount flags match the conventions used elsewhere in this stack:
# :z SELinux relabel (mandatory on this host, getenforce = Enforcing)
# :ro the media library is read-only; Vellum never writes to it
#
@ -29,13 +37,25 @@ services:
- TZ=America/New_York
- VELLUM_LISTEN=:7788
- VELLUM_LOG_LEVEL=info
# Read-only library roots, colon-separated. Add comics/music/etc. as
# those modules land.
- VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics
# Read-only library roots, colon-separated. Each entry must correspond to
# one of the read-only mounts below.
- VELLUM_MEDIA_DIRS=/media/ebooks:/media/comics:/media/audiobooks:/media/music:/media/photos:/media/games
# Metadata providers (optional; leave blank to disable).
- VELLUM_IGDB_CLIENT_ID=
- VELLUM_IGDB_CLIENT_SECRET=
- VELLUM_COMICVINE_API_KEY=
# Set to true when serving over TLS behind a reverse proxy.
- VELLUM_COOKIE_SECURE=false
volumes:
- /opt/mediaserver/vellum/data:/data:z
- /opt/mediaserver/vellum/config:/config:z
- /mnt/storage/media:/media:ro,z
# Media: one read-only mount per kind. The container never writes here.
- /mnt/storage/media/ebooks:/media/ebooks:ro,z
- /mnt/storage/media/comics:/media/comics:ro,z
- /mnt/storage/media/audiobooks:/media/audiobooks:ro,z
- /mnt/storage/media/music:/media/music:ro,z
- /mnt/storage/media/photos:/media/photos:ro,z
- /mnt/storage/media/games:/media/games:ro,z
ports:
- 127.0.0.1:7788:7788
logging:

149
internal/cli/cli.go Normal file
View file

@ -0,0 +1,149 @@
// Package cli implements the vellum command-line subcommands that operate
// directly on the database: user administration and backups.
//
// cmd/vellum calls Run before booting the HTTP server. When Run reports
// handled == true the process exits without starting the server; when it
// reports handled == false the caller falls through to its default behaviour.
package cli
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"log/slog"
"os"
"github.com/mithral/vellum/internal/auth"
"github.com/mithral/vellum/internal/core"
"github.com/mithral/vellum/migrations"
)
// Run dispatches a CLI subcommand. args is os.Args[1:].
//
// handled is false when args names no known subcommand, so the caller can
// continue with its normal startup. When handled is true the database has been
// opened and migrated, and a non-nil error is fatal to the invocation.
func Run(args []string, cfg core.Config, log *slog.Logger) (handled bool, err error) {
return run(context.Background(), args, cfg, log, os.Stdout)
}
// run is the testable core of Run. It opens and migrates the database only
// once a subcommand has actually been matched, so an unrecognised argument
// leaves the database untouched.
func run(ctx context.Context, args []string, cfg core.Config, log *slog.Logger, out io.Writer) (bool, error) {
if len(args) == 0 {
return false, nil
}
switch args[0] {
case "user", "backup":
default:
return false, nil
}
db, err := core.OpenDB(cfg.DBPath)
if err != nil {
return true, err
}
defer db.Close()
if err := core.Migrate(ctx, db, migrations.FS, log); err != nil {
return true, err
}
switch args[0] {
case "user":
return true, userCmd(ctx, db, args[1:], out)
case "backup":
return true, backupCmd(ctx, db, args[1:], out)
}
return false, nil
}
// ── user ───────────────────────────────────────────────────────────────
func userCmd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error {
if len(args) == 0 {
return errors.New("user: missing subcommand (add, passwd, list)")
}
switch args[0] {
case "add":
return userAdd(ctx, db, args[1:], out)
case "passwd":
return userPasswd(ctx, db, args[1:], out)
case "list":
return userList(ctx, db, out)
default:
return fmt.Errorf("user: unknown subcommand %q", args[0])
}
}
func userAdd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error {
admin := false
var positional []string
for _, a := range args {
if a == "--admin" {
admin = true
continue
}
positional = append(positional, a)
}
if len(positional) != 2 {
return errors.New("usage: user add <username> <password> [--admin]")
}
u, err := auth.CreateUser(ctx, db, positional[0], "", positional[1], admin)
if err != nil {
return err
}
_, err = fmt.Fprintf(out, "created user %d %s\n", u.ID, u.Username)
return err
}
func userPasswd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error {
if len(args) != 2 {
return errors.New("usage: user passwd <username> <newpassword>")
}
username, password := args[0], args[1]
u, err := auth.GetUserByUsername(ctx, db, username)
if errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("user %q not found", username)
}
if err != nil {
return err
}
if err := auth.SetPassword(ctx, db, u.ID, password); err != nil {
return err
}
_, err = fmt.Fprintf(out, "updated password for user %d %s\n", u.ID, u.Username)
return err
}
func userList(ctx context.Context, db *sql.DB, out io.Writer) error {
users, err := auth.ListUsers(ctx, db)
if err != nil {
return err
}
for _, u := range users {
if _, err := fmt.Fprintf(out, "%d\t%s\tadmin=%t\n", u.ID, u.Username, u.IsAdmin); err != nil {
return err
}
}
return nil
}
// ── backup ─────────────────────────────────────────────────────────────
func backupCmd(ctx context.Context, db *sql.DB, args []string, out io.Writer) error {
if len(args) != 1 {
return errors.New("usage: backup <path>")
}
path := args[0]
if _, err := db.ExecContext(ctx, "VACUUM INTO ?", path); err != nil {
return fmt.Errorf("backup: %w", err)
}
_, err := fmt.Fprintf(out, "backed up database to %s\n", path)
return err
}

181
internal/cli/cli_test.go Normal file
View file

@ -0,0 +1,181 @@
package cli
import (
"bytes"
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"github.com/mithral/vellum/internal/auth"
"github.com/mithral/vellum/internal/core"
)
// testConfig points a Config at a fresh database inside a temp dir. The CLI
// itself opens and migrates it, mirroring the auth_test.go setup pattern.
func testConfig(t *testing.T) core.Config {
t.Helper()
return core.Config{DBPath: filepath.Join(t.TempDir(), "vellum.db")}
}
func testLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(io.Discard, nil))
}
// runCapturing runs the exported entrypoint with os.Stdout redirected so the
// subcommand's output can be asserted on.
func runCapturing(t *testing.T, args []string, cfg core.Config, log *slog.Logger) (string, bool, error) {
t.Helper()
old := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatalf("pipe: %v", err)
}
os.Stdout = w
handled, runErr := Run(args, cfg, log)
os.Stdout = old
if err := w.Close(); err != nil {
t.Fatalf("close stdout pipe: %v", err)
}
out, err := io.ReadAll(r)
if err != nil {
t.Fatalf("read stdout: %v", err)
}
_ = r.Close()
return string(out), handled, runErr
}
func TestRunUserAddAndList(t *testing.T) {
cfg := testConfig(t)
log := testLogger()
if _, handled, err := runCapturing(t, []string{"user", "add", "alice", "password123"}, cfg, log); !handled || err != nil {
t.Fatalf("user add alice: handled=%v err=%v", handled, err)
}
if _, handled, err := runCapturing(t, []string{"user", "add", "root", "password123", "--admin"}, cfg, log); !handled || err != nil {
t.Fatalf("user add root --admin: handled=%v err=%v", handled, err)
}
out, handled, err := runCapturing(t, []string{"user", "list"}, cfg, log)
if !handled {
t.Fatal("user list was not handled")
}
if err != nil {
t.Fatalf("user list: %v", err)
}
lines := strings.Split(strings.TrimSpace(out), "\n")
if len(lines) != 2 {
t.Fatalf("user list printed %d lines, want 2:\n%s", len(lines), out)
}
if !strings.Contains(lines[0], "alice") || strings.Contains(lines[0], "admin=true") {
t.Errorf("alice line = %q, want non-admin alice", lines[0])
}
if !strings.Contains(lines[1], "root") || !strings.Contains(lines[1], "admin=true") {
t.Errorf("root line = %q, want admin root", lines[1])
}
// The password written through the CLI must actually authenticate.
db, err := core.OpenDB(cfg.DBPath)
if err != nil {
t.Fatalf("reopen db: %v", err)
}
defer db.Close()
if _, err := auth.Authenticate(context.Background(), db, "alice", "password123"); err != nil {
t.Fatalf("alice cannot authenticate after CLI add: %v", err)
}
}
func TestRunUserPasswd(t *testing.T) {
cfg := testConfig(t)
log := testLogger()
ctx := context.Background()
if _, handled, err := runCapturing(t, []string{"user", "add", "bob", "oldpassword"}, cfg, log); !handled || err != nil {
t.Fatalf("user add bob: handled=%v err=%v", handled, err)
}
// The password change goes through GetUserByUsername + SetPassword.
if _, handled, err := runCapturing(t, []string{"user", "passwd", "bob", "newpassword"}, cfg, log); !handled || err != nil {
t.Fatalf("user passwd bob: handled=%v err=%v", handled, err)
}
db, err := core.OpenDB(cfg.DBPath)
if err != nil {
t.Fatalf("reopen db: %v", err)
}
defer db.Close()
if _, err := auth.Authenticate(ctx, db, "bob", "newpassword"); err != nil {
t.Fatalf("new password does not authenticate: %v", err)
}
if _, err := auth.Authenticate(ctx, db, "bob", "oldpassword"); err == nil {
t.Error("old password still authenticates after passwd")
}
// Unknown users are reported cleanly rather than as sql.ErrNoRows.
if _, handled, err := runCapturing(t, []string{"user", "passwd", "nobody", "whatever"}, cfg, log); !handled || err == nil {
t.Errorf("passwd for unknown user: handled=%v err=%v, want handled error", handled, err)
}
}
func TestRunBackup(t *testing.T) {
cfg := testConfig(t)
log := testLogger()
ctx := context.Background()
// Seed a row so the backup has something to carry across.
if _, handled, err := runCapturing(t, []string{"user", "add", "carol", "password123"}, cfg, log); !handled || err != nil {
t.Fatalf("user add carol: handled=%v err=%v", handled, err)
}
dest := filepath.Join(t.TempDir(), "backup.db")
handled, err := run(ctx, []string{"backup", dest}, cfg, log, io.Discard)
if !handled {
t.Fatal("backup was not handled")
}
if err != nil {
t.Fatalf("backup: %v", err)
}
info, err := os.Stat(dest)
if err != nil {
t.Fatalf("stat backup: %v", err)
}
if info.Size() == 0 {
t.Fatal("backup file is empty")
}
// The backup must be a complete, independently usable database.
bdb, err := core.OpenDB(dest)
if err != nil {
t.Fatalf("open backup: %v", err)
}
defer bdb.Close()
if _, err := auth.GetUserByUsername(ctx, bdb, "carol"); err != nil {
t.Fatalf("backup is missing the seeded user: %v", err)
}
}
func TestRunUnhandled(t *testing.T) {
cfg := testConfig(t)
var buf bytes.Buffer
handled, err := run(context.Background(), []string{"serve"}, cfg, testLogger(), &buf)
if handled {
t.Error("unknown command reported handled=true")
}
if err != nil {
t.Errorf("unknown command returned error: %v", err)
}
handled, err = run(context.Background(), nil, cfg, testLogger(), &buf)
if handled || err != nil {
t.Errorf("empty args: handled=%v err=%v, want false,nil", handled, err)
}
}

View file

@ -58,6 +58,29 @@ func (q *Queue) Enqueue(ctx context.Context, kind string, payload any) error {
return q.EnqueueAfter(ctx, kind, payload, 0)
}
// EnqueueRef adds a job that refers to a specific entity (a library, an item),
// so its state and result can be looked up by that id later.
func (q *Queue) EnqueueRef(ctx context.Context, kind string, refID int64, payload any) (int64, error) {
body := "{}"
if payload != nil {
raw, err := json.Marshal(payload)
if err != nil {
return 0, fmt.Errorf("jobs: marshal payload: %w", err)
}
body = string(raw)
}
var id int64
err := q.db.QueryRowContext(ctx, `
INSERT INTO jobs (kind, ref_id, payload, state, run_after)
VALUES (?, ?, ?, ?, ?)
RETURNING id`,
kind, refID, body, StateQueued, time.Now().UTC().Format(time.RFC3339Nano)).Scan(&id)
if err != nil {
return 0, fmt.Errorf("jobs: enqueue %s: %w", kind, err)
}
return id, nil
}
// EnqueueAfter adds a job that will not run before delay has elapsed.
func (q *Queue) EnqueueAfter(ctx context.Context, kind string, payload any, delay time.Duration) error {
body := "{}"
@ -122,7 +145,7 @@ func (q *Queue) RunOnce(ctx context.Context) (bool, error) {
return true, nil
}
if err := handler(ctx, job.Payload); err != nil {
if err := handler(withJobID(ctx, job.ID), job.Payload); err != nil {
q.log.Warn("job failed", "kind", job.Kind, "id", job.ID, "attempt", job.Attempts, "err", err)
q.finish(ctx, job, err)
return true, nil
@ -206,6 +229,31 @@ func (q *Queue) failPermanently(ctx context.Context, job claimedJob, err error)
}
}
// jobCtxKey carries the running job's id to its handler, so a handler can
// attach a result without the queue needing to know the result's shape.
type jobCtxKey struct{}
func withJobID(ctx context.Context, id int64) context.Context {
return context.WithValue(ctx, jobCtxKey{}, id)
}
// JobID returns the id of the job currently being handled.
func JobID(ctx context.Context) (int64, bool) {
id, ok := ctx.Value(jobCtxKey{}).(int64)
return id, ok
}
// SetResult stores a job's result so it can be read back later. Best-effort:
// a handler that produced a result should not fail because recording it did.
func SetResult(ctx context.Context, db *sql.DB, jobID int64, v any) error {
body, err := json.Marshal(v)
if err != nil {
return fmt.Errorf("jobs: marshal result: %w", err)
}
_, err = db.ExecContext(ctx, `UPDATE jobs SET result = ? WHERE id = ?`, string(body), jobID)
return err
}
// Stats reports job counts by state, for the health endpoint and debugging.
func (q *Queue) Stats(ctx context.Context) (map[string]int, error) {
rows, err := q.db.QueryContext(ctx, `SELECT state, count(*) FROM jobs GROUP BY state`)

View file

@ -2,6 +2,7 @@ package library
import (
"database/sql"
"encoding/json"
"errors"
"log/slog"
"net/http"
@ -9,18 +10,28 @@ import (
"github.com/mithral/vellum/internal/auth"
"github.com/mithral/vellum/internal/httpx"
"github.com/mithral/vellum/internal/jobs"
)
// ScanJobKind is the queue job that scans a library.
const ScanJobKind = "scan_library"
// ScanPayload is the job payload for a scan.
type ScanPayload struct {
LibraryID int64 `json:"library_id"`
}
// Handlers serves library management and scanning endpoints.
type Handlers struct {
db *sql.DB
log *slog.Logger
auth *auth.Authenticator
db *sql.DB
log *slog.Logger
auth *auth.Authenticator
queue *jobs.Queue
}
// NewHandlers builds the library HTTP layer.
func NewHandlers(db *sql.DB, log *slog.Logger, a *auth.Authenticator) *Handlers {
return &Handlers{db: db, log: log, auth: a}
func NewHandlers(db *sql.DB, log *slog.Logger, a *auth.Authenticator, q *jobs.Queue) *Handlers {
return &Handlers{db: db, log: log, auth: a, queue: q}
}
// Register mounts the routes.
@ -30,6 +41,7 @@ func (h *Handlers) Register(mux *http.ServeMux) {
mux.Handle("GET /api/v1/libraries", h.auth.Require(h.handleList))
mux.Handle("GET /api/v1/libraries/{id}", h.auth.Require(h.handleGet))
mux.Handle("GET /api/v1/libraries/{id}/items", h.auth.Require(h.handleListItems))
mux.Handle("GET /api/v1/libraries/{id}/scan/status", h.auth.RequireAuth(http.HandlerFunc(h.handleScanStatus)))
mux.Handle("POST /api/v1/libraries", h.auth.RequireAdmin(http.HandlerFunc(h.handleCreate)))
mux.Handle("DELETE /api/v1/libraries/{id}", h.auth.RequireAdmin(http.HandlerFunc(h.handleDelete)))
@ -146,12 +158,48 @@ func (h *Handlers) handleScan(w http.ResponseWriter, r *http.Request) {
return
}
res, err := Scan(r.Context(), h.db, lib, h.log)
// Queued rather than run inline: a real library takes longer than any
// sensible request timeout, and a hanging request is indistinguishable
// from a broken server.
jobID, err := h.queue.EnqueueRef(r.Context(), ScanJobKind, lib.ID, ScanPayload{LibraryID: lib.ID})
if err != nil {
h.fail(w, "scan library", err)
h.fail(w, "enqueue scan", err)
return
}
httpx.WriteJSON(w, http.StatusOK, res)
h.log.Info("scan queued", "library", lib.Name, "job", jobID)
httpx.WriteJSON(w, http.StatusAccepted, map[string]any{"queued": 1, "job_id": jobID})
}
// handleScanStatus reports the most recent scan for a library, so the UI can
// show progress instead of hanging on the request.
func (h *Handlers) handleScanStatus(w http.ResponseWriter, r *http.Request) {
id, ok := pathInt(w, r, "id")
if !ok {
return
}
var state, result, errMsg string
err := h.db.QueryRowContext(r.Context(), `
SELECT state, result, error FROM jobs
WHERE kind = ? AND ref_id = ?
ORDER BY id DESC LIMIT 1`, ScanJobKind, id).Scan(&state, &result, &errMsg)
if errors.Is(err, sql.ErrNoRows) {
httpx.WriteJSON(w, http.StatusOK, map[string]any{"state": "idle"})
return
}
if err != nil {
h.fail(w, "scan status", err)
return
}
out := map[string]any{"state": state}
if result != "" {
out["result"] = json.RawMessage(result)
}
if errMsg != "" {
out["error"] = errMsg
}
httpx.WriteJSON(w, http.StatusOK, out)
}
// handleListItems returns items in a library the caller may read.

View file

@ -31,18 +31,36 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut
h.fail(w, "visible libraries", err)
return
}
limit := 300
if v := r.URL.Query().Get("limit"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 1000 {
limit = n
}
}
offset := 0
if v := r.URL.Query().Get("offset"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n >= 0 {
offset = n
}
}
if len(libs) == 0 {
httpx.WriteJSON(w, http.StatusOK, map[string]any{"items": []itemRow{}, "total": 0})
httpx.WriteJSON(w, http.StatusOK, map[string]any{
"items": []itemRow{},
"total": 0,
"offset": offset,
"limit": limit,
})
return
}
in, args := inClause(libs)
query := `SELECT id, library_id, kind, path, title, COALESCE(year, 0)
FROM media_items
WHERE missing = 0 AND library_id IN (` + in + `)`
where := `missing = 0 AND library_id IN (` + in + `)`
if kind := r.URL.Query().Get("kind"); kind != "" {
query += ` AND kind = ?`
where += ` AND kind = ?`
args = append(args, kind)
}
if libStr := r.URL.Query().Get("library_id"); libStr != "" {
@ -51,20 +69,24 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut
httpx.WriteError(w, http.StatusBadRequest, "invalid library_id")
return
}
query += ` AND library_id = ?`
where += ` AND library_id = ?`
args = append(args, libID)
}
limit := 300
if v := r.URL.Query().Get("limit"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 1000 {
limit = n
}
// total counts every matching row, independent of the page window.
var total int
if err := h.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM media_items WHERE `+where, args...).Scan(&total); err != nil {
h.fail(w, "count items", err)
return
}
query += ` ORDER BY sort_title LIMIT ?`
args = append(args, limit)
rows, err := h.db.QueryContext(ctx, query, args...)
query := `SELECT id, library_id, kind, path, title, COALESCE(year, 0)
FROM media_items
WHERE ` + where + ` ORDER BY sort_title LIMIT ? OFFSET ?`
pageArgs := append(append([]any{}, args...), limit, offset)
rows, err := h.db.QueryContext(ctx, query, pageArgs...)
if err != nil {
h.fail(w, "list items", err)
return
@ -85,5 +107,10 @@ func (h *Handlers) handleListItems(w http.ResponseWriter, r *http.Request, u aut
return
}
httpx.WriteJSON(w, http.StatusOK, map[string]any{"items": items, "total": len(items)})
httpx.WriteJSON(w, http.StatusOK, map[string]any{
"items": items,
"total": total,
"offset": offset,
"limit": limit,
})
}

View file

@ -0,0 +1,10 @@
-- 0007_job_ref.sql — let a job be found by what it operates on.
--
-- Scans need their progress read back: the UI asks "what is the state of the
-- scan for library 3?" That requires the job to carry the id it refers to and
-- to store its result, neither of which the queue had.
ALTER TABLE jobs ADD COLUMN ref_id INTEGER;
ALTER TABLE jobs ADD COLUMN result TEXT NOT NULL DEFAULT '';
CREATE INDEX idx_jobs_kind_ref ON jobs(kind, ref_id, id);

View file

@ -118,7 +118,24 @@ export const api = {
deleteLibrary: (id: number) => request<void>('DELETE', `/api/v1/libraries/${id}`),
scanLibrary: (id: number) => request<ScanResult>('POST', `/api/v1/libraries/${id}/scan`),
scanLibrary: (id: number) =>
request<{ queued: number; job_id: number }>('POST', `/api/v1/libraries/${id}/scan`),
scanStatus: (id: number) =>
request<{
state: 'queued' | 'running' | 'done' | 'failed' | 'idle'
result?: {
scanned: number
added: number
updated: number
restored: number
missing: number
unchanged: number
duration_ms: number
errors?: string[]
}
error?: string
}>('GET', `/api/v1/libraries/${id}/scan/status`),
enrichLibrary: (id: number) =>
request<{ queued: number }>('POST', `/api/v1/libraries/${id}/enrich`),
@ -154,12 +171,16 @@ export const api = {
deleteSave: (id: number, slot = 'default') =>
request<void>('DELETE', `/api/v1/items/${id}/saves/${slot}`),
listItems: (opts: { kind?: string; libraryId?: number; limit?: number } = {}) => {
listItems: (opts: { kind?: string; libraryId?: number; limit?: number; offset?: number } = {}) => {
const p = new URLSearchParams()
if (opts.kind) p.set('kind', opts.kind)
if (opts.libraryId) p.set('library_id', String(opts.libraryId))
if (opts.limit) p.set('limit', String(opts.limit))
return request<{ items: Item[]; total: number }>('GET', `/api/v1/items?${p.toString()}`)
if (opts.offset) p.set('offset', String(opts.offset))
return request<{ items: Item[]; total: number; offset: number; limit: number }>(
'GET',
`/api/v1/items?${p.toString()}`,
)
},
ebook: (id: number) => request<{ metadata: EbookMetadata }>('GET', `/api/v1/items/${id}/ebook`),

View file

@ -4,13 +4,17 @@
import { go } from '../nav'
import Card from './Card.svelte'
const PAGE_SIZE = 60
let {
kind = undefined,
libraryId = undefined,
}: { kind?: string; libraryId?: number } = $props()
let items = $state<Item[]>([])
let total = $state(0)
let loading = $state(true)
let loadingMore = $state(false)
let error = $state<string | null>(null)
$effect(() => {
@ -21,8 +25,9 @@
loading = true
error = null
try {
const res = await api.listItems({ kind: k, libraryId: lib })
const res = await api.listItems({ kind: k, libraryId: lib, limit: PAGE_SIZE, offset: 0 })
items = res.items
total = res.total
} catch (err) {
error = err instanceof ApiError ? err.message : 'could not load items'
} finally {
@ -30,6 +35,26 @@
}
}
async function loadMore() {
if (loadingMore || items.length >= total) return
loadingMore = true
error = null
try {
const res = await api.listItems({
kind,
libraryId,
limit: PAGE_SIZE,
offset: items.length,
})
items = [...items, ...res.items]
total = res.total
} catch (err) {
error = err instanceof ApiError ? err.message : 'could not load more items'
} finally {
loadingMore = false
}
}
const types = $derived(($mediaTypes as { Kind: string; Name: string }[]))
</script>
@ -48,7 +73,7 @@
<div class="grid">
{#each Array(12) as _}<div class="skeleton tile"></div>{/each}
</div>
{:else if error}
{:else if error && items.length === 0}
<div class="alert">{error}</div>
{:else if items.length === 0}
<div class="empty">
@ -56,11 +81,22 @@
<p>No items match this section yet.</p>
</div>
{:else}
<p class="count">Showing {items.length} of {total}</p>
<div class="grid">
{#each items as item (item.id)}
<Card {item} />
{/each}
</div>
{#if error}
<div class="alert">{error}</div>
{/if}
{#if items.length < total}
<div class="more">
<button class="chip load-more" onclick={loadMore} disabled={loadingMore}>
{loadingMore ? 'Loading…' : 'Load more'}
</button>
</div>
{/if}
{/if}
<style>
@ -70,6 +106,11 @@
gap: var(--space-2);
margin-bottom: var(--space-5);
}
.count {
margin: 0 0 var(--space-4);
color: var(--muted);
font-size: var(--fs-sm);
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(var(--grid-min), 1fr));
@ -78,6 +119,15 @@
.skeleton.tile {
aspect-ratio: 2 / 3;
}
.more {
display: flex;
justify-content: center;
margin-top: var(--space-5);
}
.load-more:disabled {
opacity: 0.6;
cursor: default;
}
@media (max-width: 700px) {
.grid {
grid-template-columns: repeat(auto-fill, minmax(calc(var(--grid-min) * 0.75), 1fr));

View file

@ -7,6 +7,7 @@
let showForm = $state(false)
let busy = $state(false)
let scanning = $state<number | null>(null)
let error = $state<string | null>(null)
let notice = $state<string | null>(null)
@ -33,17 +34,43 @@
}
}
// A scan is queued on the server and polled here, because a real library
// takes longer than any single request. The button shows which library is
// working rather than freezing the whole screen.
async function scan(id: number) {
busy = true
scanning = id
error = null
notice = null
try {
const res = await api.scanLibrary(id)
notice = `Scanned ${res.scanned}: ${res.added} added, ${res.updated} updated, ${res.missing} missing.`
await api.scanLibrary(id)
const deadline = Date.now() + 30 * 60 * 1000
for (;;) {
await new Promise((resolve) => setTimeout(resolve, 1000))
const s = await api.scanStatus(id)
if (s.state === 'done') {
const r = s.result
notice = r
? `Scanned ${r.scanned}: ${r.added} added, ${r.updated} updated, ${r.missing} missing.`
: 'Scan complete.'
break
}
if (s.state === 'failed') {
error = s.error || 'scan failed'
break
}
if (Date.now() > deadline) {
error = 'scan is taking longer than expected; check the server logs'
break
}
}
await refreshLibraries()
} catch (err) {
error = err instanceof ApiError ? err.message : 'scan failed'
} finally {
busy = false
scanning = null
}
}
@ -134,7 +161,9 @@
</button>
{#if $user?.is_admin}
<div class="actions">
<button class="btn btn-ghost" onclick={() => scan(lib.id)} disabled={busy}>Scan</button>
<button class="btn btn-ghost" onclick={() => scan(lib.id)} disabled={busy}>
{scanning === lib.id ? 'Scanning…' : 'Scan'}
</button>
<button class="btn btn-ghost" onclick={() => enrich(lib.id)} disabled={busy}>Enrich</button>
<button class="btn btn-ghost danger" onclick={() => remove(lib.id, lib.name)} disabled={busy}>
Delete

View file

@ -1,9 +1,47 @@
<script lang="ts">
import { settings, setSettings, resetSettings, TEXT_STEPS, CARD_STEPS } from '../settings'
import { api, ApiError } from '../api'
function isActive(current: number, value: number): boolean {
return Math.abs(current - value) < 0.001
}
const MIN_PASSWORD = 8
let currentPassword = $state('')
let newPassword = $state('')
let confirmPassword = $state('')
let passwordError = $state<string | null>(null)
let passwordSuccess = $state<string | null>(null)
let passwordBusy = $state(false)
async function changePassword(event: SubmitEvent) {
event.preventDefault()
if (passwordBusy) return
passwordError = null
passwordSuccess = null
if (newPassword.length < MIN_PASSWORD) {
passwordError = `New password must be at least ${MIN_PASSWORD} characters.`
return
}
if (newPassword !== confirmPassword) {
passwordError = 'The new passwords do not match.'
return
}
passwordBusy = true
try {
await api.changePassword(currentPassword, newPassword)
currentPassword = newPassword = confirmPassword = ''
passwordSuccess =
'Password changed. For your security every session was signed out, so you will need to sign in again with your new password.'
} catch (err) {
passwordError = err instanceof ApiError ? err.message : 'Could not reach the server.'
} finally {
passwordBusy = false
}
}
</script>
<div class="wrap">
@ -73,6 +111,68 @@
</p>
</section>
<form class="card group password" onsubmit={changePassword}>
<header>
<h3>Password</h3>
</header>
{#if passwordError}
<div class="alert" role="alert">{passwordError}</div>
{/if}
{#if passwordSuccess}
<div class="ok" role="status">{passwordSuccess}</div>
{/if}
<div class="field">
<label for="pw-current">Current password</label>
<input
id="pw-current"
class="input"
type="password"
autocomplete="current-password"
bind:value={currentPassword}
disabled={passwordBusy}
required
/>
</div>
<div class="field">
<label for="pw-new">New password</label>
<input
id="pw-new"
class="input"
type="password"
autocomplete="new-password"
bind:value={newPassword}
disabled={passwordBusy}
required
/>
<span class="muted hint">At least {MIN_PASSWORD} characters.</span>
</div>
<div class="field">
<label for="pw-confirm">Confirm new password</label>
<input
id="pw-confirm"
class="input"
type="password"
autocomplete="new-password"
bind:value={confirmPassword}
disabled={passwordBusy}
required
/>
</div>
<button
class="btn btn-primary submit"
type="submit"
disabled={passwordBusy || !currentPassword || !newPassword || !confirmPassword}
>
{#if passwordBusy}<span class="spinner"></span>{/if}
{passwordBusy ? 'Changing password' : 'Change password'}
</button>
</form>
<button class="btn" onclick={resetSettings}>Reset to defaults</button>
</div>
@ -175,4 +275,19 @@
max-width: 60ch;
line-height: 1.6;
}
.password {
display: flex;
flex-direction: column;
gap: var(--space-3);
}
.password header {
margin-bottom: 0;
}
.hint {
font-size: var(--fs-xs);
}
.password .submit {
align-self: flex-start;
margin-top: var(--space-1);
}
</style>