# ADR-006: The job queue is a table, not a broker

## Status
Accepted

## Date
2026-10-12

## Context

Vellum does work outside a request: scanning libraries and enriching metadata
from external providers. That work needs a queue — something has to survive a
restart, retry a failure, and be visible while it runs.

## Decision

Implement the queue as a table in the same SQLite database. No Redis, no broker,
no second process.

## Alternatives Considered

### Redis or a similar broker

- **Pros:** built for queues; fast; familiar.
- **Cons:** a second service to run, back up and monitor. The queue would then
  live outside the database, so a backup would capture the catalogue but not the
  work in flight.
- **Rejected:** it buys throughput this application does not need and costs the
  "one binary, one file" deployment story.

### In-process goroutines with an in-memory queue

- **Pros:** no storage at all.
- **Cons:** a restart loses queued work silently. There is no way to see what is
  pending, and no way to retry a failure after a crash.
- **Rejected:** the queue must survive a restart.

## Consequences

Claiming a job is a single atomic statement, so several workers are safe with no
extra locking:

```sql
UPDATE jobs SET state = 'running', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE state = 'queued' AND run_after <= ?
            ORDER BY run_after, id LIMIT 1)
RETURNING id, kind, payload, attempts
```

- Four workers with jittered idle waits. A panicking handler is recovered and the
  job marked failed, so one bad handler cannot kill a worker or strand a row in
  `running`.
- `RecoverStale()` runs at boot and requeues anything a dead process left behind.
  Without it, a crash mid-scan left a job `running` forever while the UI polled it
  forever.
- The queue is included in the same backup as everything else.
- Job state is queryable, which is how scan progress reaches the UI
  (`GET /libraries/{id}/scan/status`).
- Two job kinds: `scan_library` and `enrich_item`.
- **Cost accepted:** the queue shares SQLite's single writer, so a very large
  scan slows enqueueing. Chunked transactions keep each write short, and this is
  a deliberate trade against running a second service.