97 lines
3.1 KiB
Markdown
97 lines
3.1 KiB
Markdown
# JobsBoard Platform — Phase 9 Operational Runbook: Job Acquisition & Ingestion Pipeline
|
|
|
|
This document establishes operational procedures for monitoring, troubleshooting, scheduling, and adding real job source integrations to JobsBoard.
|
|
|
|
---
|
|
|
|
## 1. Subsystem Overview
|
|
|
|
The Phase 9 Job Acquisition system continuously discovers, ingests, normalizes, and manages the lifecycle of job postings from external public ATS boards (Greenhouse, Lever, Ashby) and portal APIs.
|
|
|
|
### The Acquisition Pipeline
|
|
```text
|
|
Raw Source Board (API / JSON)
|
|
↓
|
|
Adapter Fetch & Rate Limit Guard
|
|
↓
|
|
Canonical URL Normalization (Strip UTM/Tracking)
|
|
↓
|
|
Content Fingerprint Generation (SHA-256)
|
|
↓
|
|
Multi-Tier Deduplication & Upsert
|
|
↓
|
|
Freshness & Missing-Scan Lifecycle Tracker
|
|
↓
|
|
Source Execution Audit Logging
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Managing Job Sources & Running Ingestions
|
|
|
|
### 2.1 Viewing Source Registry & Ingestion Telemetry
|
|
Navigate to:
|
|
```text
|
|
/admin/sources
|
|
```
|
|
Or query the API:
|
|
```bash
|
|
curl -s -H "Cookie: <ADMIN_SESSION_COOKIE>" http://localhost:3000/api/admin/sources | jq .metrics
|
|
```
|
|
|
|
### 2.2 Triggering an Immediate Ingestion Run
|
|
To manually ingest jobs from a company's public ATS board without waiting for the scheduler:
|
|
```bash
|
|
curl -X POST http://localhost:3000/api/admin/sources \
|
|
-H "Content-Type: application/json" \
|
|
-H "Cookie: <ADMIN_SESSION_COOKIE>" \
|
|
-d '{
|
|
"provider": "greenhouse",
|
|
"identifier": "stripe",
|
|
"companyName": "Stripe",
|
|
"limit": 30
|
|
}'
|
|
```
|
|
|
|
Supported Providers:
|
|
- `greenhouse` (e.g. `stripe`, `figma`, `datadog`, `ramp`, `posthog`)
|
|
- `lever` (e.g. `vercel`, `netflix`, `spotify`, `palantir`)
|
|
- `ashby` (e.g. `linear`, `sentry`, `retool`, `openai`)
|
|
|
|
---
|
|
|
|
## 3. Detecting ATS on Unknown Careers Pages
|
|
|
|
To automatically detect whether a company uses Greenhouse, Lever, or Ashby:
|
|
```typescript
|
|
import { detectAts } from "@/lib/sources/detector";
|
|
|
|
const result = await detectAts("https://linear.app/careers");
|
|
console.log(result);
|
|
// Output: { provider: "ashby", identifier: "linear", confidence: "CONFIRMED" }
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Troubleshooting & Operational Failures
|
|
|
|
### Issue A: Source Marked as `FAILED` or `DEGRADED`
|
|
**Cause**: The company's board slug may have changed, or the ATS endpoint returned an HTTP 404/429.
|
|
**Resolution**:
|
|
1. Check recent execution logs:
|
|
```sql
|
|
SELECT "status", "errorMessage", "durationMs", "createdAt"
|
|
FROM "source_execution_logs"
|
|
WHERE "sourceId" = '<SOURCE_ID>'
|
|
ORDER BY "createdAt" DESC LIMIT 5;
|
|
```
|
|
2. Verify board endpoint manually:
|
|
```bash
|
|
curl -I "https://boards-api.greenhouse.io/v1/boards/<slug>"
|
|
```
|
|
3. Update slug in database or disable source if discontinued.
|
|
|
|
### Issue B: Job Freshness & Stale Job Expiration
|
|
- Jobs are **NOT** expired immediately upon missing from a single scan.
|
|
- The pipeline requires **3 consecutive successful scans** where the job is absent before transitioning `lifecycleStatus` from `ACTIVE` to `EXPIRED`.
|
|
- If an acquisition scan fails or encounters a network error, missing counts are preserved and **no jobs are expired**.
|