Crawler

Overview

The crawler layer has three core responsibilities. First, standardizing different external data formats into a single documents table schema. Second, controlling duplicates by URL while tracking the most recent observation time. Third, making stored documents reliably available to the downstream enrichment pipeline and the read-only API.

Every document produced by a crawler contains the same base fields regardless of source.

{
  "title":          string,   // extracted from source
  "url":            string,   // canonical URL after normalization
  "summary":        string,   // raw excerpt — not LLM-generated
  "content":        string,   // full body if available, else null
  "source":         string,   // logical source identifier
  "content_source": string,   // ingestion method: rss | detail | api
  "created_at":     string    // first seen (ISO 8601)
}

source identifies which logical source a document belongs to. content_source describes how the content was actually obtained — whether only the RSS summary was stored, whether the detail page was fetched and parsed, or whether the content came directly from a structured API response.

Data collection

SynthLink uses three collection strategies depending on the source.

Normalization and filtering

Crawlers do not store raw content as-is. Two steps run before every upsert.

Data flow

After a crawler writes to the documents table, the data moves through two more stages before reaching external consumers.

external source
  → crawler (upsert into documents)
  → insight-worker (LLM enrichment → upsert into insights)
  → public API (/api/v1/documents, /api/v1/insights, /api/v1/combined)

The insight-worker picks up documents that do not yet have an insight, or insight records in a retryable failed state. It uses content as the LLM input when available, falling back to summary. The model output is parsed into llm_summary, keywords, tags, and category, then written to the insights table.

Upserts use url as the conflict key. If a document with the same URL already exists, the existing record is preserved. created_at is never modified after initial insertion.

Automation

Each crawler implements both scheduled() and fetch() handlers, supporting scheduled runs and HTTP-triggered manual runs without any code changes.

Schedules are managed at two levels. Some crawlers define a Cloudflare cron trigger in wrangler.toml. Others are registered with an external scheduler and tracked in the crawlers table alongside metadata such as trigger_url, cron_schedule, and cron_enabled. In practice, the active schedule for any given crawler may differ from what is defined in the repository — always refer to the Status page for the latest run history.

Failure handling

Crawlers are designed to assume external request failures. Each worker uses up to three retries with exponential backoff. Transient errors (429, 5xx) are retried. NVD respects the retry-after header. OpenAI and NASA continue with RSS summary if detail page fetching fails, rather than discarding the item.

The insight-worker applies a separate retry policy for LLM enrichment. Failed insight records with retry_count < 3 are reprocessed with delays of 1, 5, and 15 minutes between attempts. Errors classified as non-retryable — such as missing_openrouter_api_key, empty_source_text, or missing_document — are marked permanently failed without further retries.

Observability

Every crawler writes its run result to the worker_runs table — worker name, success flag, number of processed records, and error message if applicable. This is the data source for the crawler history shown on the Status page.

The insight-worker additionally writes to integrity_checks after each run — recording the count of orphan insights, duplicate URLs, and completed insights with an empty llm_summary. These values are also visible on the Status page.

Sources

Each source section below follows the same format — input method, filter criteria, URL normalization, and operational notes.