JobsBoard Platform — Phase 9 Operational Runbook: Job Acquisition & Ingestion Pipeline

This document establishes operational procedures for monitoring, troubleshooting, scheduling, and adding real job source integrations to JobsBoard.

1. Subsystem Overview

The Phase 9 Job Acquisition system continuously discovers, ingests, normalizes, and manages the lifecycle of job postings from external public ATS boards (Greenhouse, Lever, Ashby) and portal APIs.

The Acquisition Pipeline

Raw Source Board (API / JSON)
              ↓
  Adapter Fetch & Rate Limit Guard
              ↓
  Canonical URL Normalization (Strip UTM/Tracking)
              ↓
  Content Fingerprint Generation (SHA-256)
              ↓
  Multi-Tier Deduplication & Upsert
              ↓
  Freshness & Missing-Scan Lifecycle Tracker
              ↓
  Source Execution Audit Logging

2. Managing Job Sources & Running Ingestions

2.1 Viewing Source Registry & Ingestion Telemetry

Navigate to:

/admin/sources

Or query the API:

curl -s -H "Cookie: <ADMIN_SESSION_COOKIE>" http://localhost:3000/api/admin/sources | jq .metrics

2.2 Triggering an Immediate Ingestion Run

To manually ingest jobs from a company's public ATS board without waiting for the scheduler:

curl -X POST http://localhost:3000/api/admin/sources \
  -H "Content-Type: application/json" \
  -H "Cookie: <ADMIN_SESSION_COOKIE>" \
  -d '{
    "provider": "greenhouse",
    "identifier": "stripe",
    "companyName": "Stripe",
    "limit": 30
  }'

Supported Providers:

greenhouse (e.g. stripe, figma, datadog, ramp, posthog)
lever (e.g. vercel, netflix, spotify, palantir)
ashby (e.g. linear, sentry, retool, openai)

3. Detecting ATS on Unknown Careers Pages

To automatically detect whether a company uses Greenhouse, Lever, or Ashby:

import { detectAts } from "@/lib/sources/detector";

const result = await detectAts("https://linear.app/careers");
console.log(result);
// Output: { provider: "ashby", identifier: "linear", confidence: "CONFIRMED" }

4. Troubleshooting & Operational Failures

Issue A: Source Marked as `FAILED` or `DEGRADED`

Cause: The company's board slug may have changed, or the ATS endpoint returned an HTTP 404/429. Resolution:

Check recent execution logs:

SELECT "status", "errorMessage", "durationMs", "createdAt"
FROM "source_execution_logs"
WHERE "sourceId" = '<SOURCE_ID>'
ORDER BY "createdAt" DESC LIMIT 5;

Verify board endpoint manually:

curl -I "https://boards-api.greenhouse.io/v1/boards/<slug>"

Update slug in database or disable source if discontinued.

Issue B: Job Freshness & Stale Job Expiration

Jobs are NOT expired immediately upon missing from a single scan.
The pipeline requires 3 consecutive successful scans where the job is absent before transitioning lifecycleStatus from ACTIVE to EXPIRED.
If an acquisition scan fails or encounters a network error, missing counts are preserved and no jobs are expired.

3.1 KiB Raw Permalink Blame History