051169a18a
Docs - docs/ARCHITECTURE.md: how the pieces fit and why. The request path, the six enforced layers, the data model, the scanner, the job queue, direct-play playback, and the three protocol surfaces over one application. - docs/decisions/: seven ADRs with the alternatives that were rejected — SQLite-with-a-seam, direct-play-only, mark-missing-never-delete, the modular monolith, the one item-access chokepoint, the table-backed queue, and fixed-width timestamps. - README: the layout section was stale (it predated catalog, home, scanner, providers, engine, probe, playback, apitoken, arch and the dto packages) and now lists what exists, plus links to the new docs. Two real bugs, both found by verifying the docs instead of trusting them 1. The CLI could not run on a fresh install. `Config.Validate` created the data and config directories, but the CLI never called it, so `vellum user add` — the recovery path when you are locked out — failed with "unable to open database file" until the server had booted at least once. Split the directory creation into `Config.EnsureDirs`, call it from the CLI, and keep the media root check out of it: an admin command must not fail because a library is unmounted. The CLI tests now point DBPath inside DataDir, exactly as LoadConfig derives it, so they exercise the fresh-install path; removing EnsureDirs fails three of them. 2. A fresh clone could not build at all. `//go:embed all:dist` fails to compile when web/dist is absent, and web/.gitignore expected a `dist/.gitkeep` that was never created. Added it, so a clone builds and the server logs "web UI not embedded; run the frontend build to include it" — the intended behaviour for someone who only wants the API. Also: the httpapi tests depended on a built frontend and FAILED rather than skipping, so the suite was red on any fresh clone. findAsset and readAsset now skip with a message naming the missing step. They still RUN when a bundle is present (verified: zero skips here), so they have not been quietly disabled. Verified from a simulated clone of tracked files only: go build exit 0, 25 packages pass, the server boots and applies all 12 migrations, and the CLI creates a user and takes a backup. gofmt clean, vet exit 0, svelte-check 0/0.
62 lines
2.2 KiB
Markdown
62 lines
2.2 KiB
Markdown
# ADR-006: The job queue is a table, not a broker
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Date
|
|
2026-10-12
|
|
|
|
## Context
|
|
|
|
Vellum does work outside a request: scanning libraries and enriching metadata
|
|
from external providers. That work needs a queue — something has to survive a
|
|
restart, retry a failure, and be visible while it runs.
|
|
|
|
## Decision
|
|
|
|
Implement the queue as a table in the same SQLite database. No Redis, no broker,
|
|
no second process.
|
|
|
|
## Alternatives Considered
|
|
|
|
### Redis or a similar broker
|
|
|
|
- **Pros:** built for queues; fast; familiar.
|
|
- **Cons:** a second service to run, back up and monitor. The queue would then
|
|
live outside the database, so a backup would capture the catalogue but not the
|
|
work in flight.
|
|
- **Rejected:** it buys throughput this application does not need and costs the
|
|
"one binary, one file" deployment story.
|
|
|
|
### In-process goroutines with an in-memory queue
|
|
|
|
- **Pros:** no storage at all.
|
|
- **Cons:** a restart loses queued work silently. There is no way to see what is
|
|
pending, and no way to retry a failure after a crash.
|
|
- **Rejected:** the queue must survive a restart.
|
|
|
|
## Consequences
|
|
|
|
Claiming a job is a single atomic statement, so several workers are safe with no
|
|
extra locking:
|
|
|
|
```sql
|
|
UPDATE jobs SET state = 'running', attempts = attempts + 1
|
|
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
|
|
ORDER BY run_after, id LIMIT 1)
|
|
RETURNING id, kind, payload, attempts
|
|
```
|
|
|
|
- Four workers with jittered idle waits. A panicking handler is recovered and the
|
|
job marked failed, so one bad handler cannot kill a worker or strand a row in
|
|
`running`.
|
|
- `RecoverStale()` runs at boot and requeues anything a dead process left behind.
|
|
Without it, a crash mid-scan left a job `running` forever while the UI polled it
|
|
forever.
|
|
- The queue is included in the same backup as everything else.
|
|
- Job state is queryable, which is how scan progress reaches the UI
|
|
(`GET /libraries/{id}/scan/status`).
|
|
- Two job kinds: `scan_library` and `enrich_item`.
|
|
- **Cost accepted:** the queue shares SQLite's single writer, so a very large
|
|
scan slows enqueueing. Chunked transactions keep each write short, and this is
|
|
a deliberate trade against running a second service.
|