observer

This server fronts the harvesting half of the intelligence desk — the Observer machinery that pulls in RSS feeds, X accounts, YouTube channels, and email newsletters on schedule, at zero model cost. Through it an agent manages the source registry, triggers and monitors harvests, adjusts batch and per-source schedules, and works the item pool: list, search, inspect, update, and curate everything that has been collected.

It is one of the fabric's self-hosting servers: it owns the harvest index directly and runs the harvest scheduler inside its own process. Harvested items land as Markdown files plus a local index under your documents tree, which is why everything downstream — the writer agents, the publishing pipeline, your own reading — works from plain files you keep.

How your agents use it

• "Add this feed and pull it now." The agent calls source_add, then harvest_now, and polls harvest_status until the run completes.
• "What came in about open-source models this week?" — items_search over the pool, items_detail for a full item, items_curate to mark what deserves downstream attention.
• "Quiet the overnight batches while I travel." — batch_list to see what is scheduled, schedule_toggle to pause a batch, schedule_update to change its cadence.

Prerequisites

Harvesting itself needs no model key — collection is mechanical. Some source types use optional credentials you configure once: a proxy for transcript-heavy YouTube harvesting and a dedicated newsletter inbox for email sources, both under the observer settings (see the configuration reference). The processing tier above the pool — the writer and analyst agents — runs on your Claude subscription via routines.

Tool reference

Tool	Parameters	What it does
source_add	id*: string, url*: string, type*: string, category*: string, authority_score: number	Register a new feed, API source, or site to harvest.
source_remove	source_id*: string	Unregister a source by ID.
source_list	category: string, type: string	List all configured sources with health status.
harvest_now	batch_id: string, source_id: string, max_lookback_days: integer, latest_count: integer	Harvest a specific batch, source, or all batches. Preferred: use batch_id to harvest a named group of sources together.
harvest_stop	run_id: string	Cancel an in-flight daemon harvest run — the kind shown in daemon_status.active_harvests / harvest_pool. Pass run_id to cancel one specific run; omit it to cancel ALL active + queued runs. The cancel signal is checked at the boundary between sources/videos, so it takes effect within one transcript-fetch timeout (≤45s) rather than instantly. Use to abort a stuck or unwanted harvest.
harvest_backfill_transcripts	limit: integer, source_id: string	Recover YouTube videos previously sealed as 'no captions'. Many were a transient IP-block (VideoUnplayable) misclassified as permanent; this re-attempts each with the fixed retry + yt-dlp fallback logic and persists the outcome (recovered transcript / still-pending / genuinely captionless). Idempotent and re-runnable. Run when no harvest is active (it issues live proxied network calls). Returns a summary with counts + the recovered items.
harvest_status	—	Show harvest health: last harvest time, errors, item counts per source.
process_video	url*: string, force: boolean, max_keyframes: integer, proxy: string	Full-download a single YouTube video on-demand (one-off — NOT a registered source): yt-dlp download → adaptive ffmpeg keyframe extraction → transcript, written as a source_type='youtube_video' item destined for the youtube-video-analysis routine / yt-video-analyzer. This is the MCP proxy for the CC Video tab's POST /api/process-video. Returns IMMEDIATELY with {ok, run_id, status: 'active'|'queued'}; the heavy download runs in the daemon's harvest pool — poll list_videos (row present + frame_count > 0) or daemon_status for completion. force defaults to true (user-initiated: bypass dedup + replace any prior row for this URL). Returns {ok:false, error:'harvest_pool_full'} (429) when the active+queued cap is reached.
list_videos	—	List all full-download youtube_video items (newest first) with per-item companion inventory: on-disk folder, transcript file_path, _metadata.md path, frame files + frame_count, plus status and analysis_status. MCP proxy for GET /api/videos. Use to confirm a process_video download landed (row present, frame_count > 0) and to get each item's folder/paths to hand the yt-video-analyzer.
batch_list	—	List all configured batches with their sources and schedules.
batch_runs	batch_id: string, limit: integer	List recent batch harvest runs with timestamps and item counts.
items_list	category: string, status: string, since: string, min_score: number, batch_id: string, batch_run_id: string, limit: integer	List harvested items with optional filters.
items_search	query*: string, limit: integer	Keyword LIKE search across item title, tags, summary, source_url, and author. Use for fuzzy discovery when you don't have an exact URL or item id. If you have the item's exact URL, prefer items_by_url (zero false positives).
items_by_url	source_url*: string	Exact-match lookup of a single harvested item by source_url. Returns the item's full metadata + content (same shape as items_detail). Preferred drill-down path for the news-writer agent — observer reports list the item's canonical URL in their `Link:` field, so this lets the writer skip fuzzy items_search guessing entirely. Returns {error: 'not found'} when no row matches. Excludes status='duplicate' / 'rejected' / 'orphaned'.
items_detail	item_id*: string	Get full content of a specific item by ID.
items_batch_detail	item_ids*: array, compact: boolean, consolidated_file: boolean	Get content of multiple items in one call. Returns compact format optimized for batch analysis. Use this instead of calling items_detail repeatedly. For batches >20 items, set consolidated_file=true — the server writes all content to a single markdown file and returns only metadata + a file path. Read that file once with the Read tool to get all content. This avoids the tool-result size ceiling that forces slow per-file fallbacks.
items_update	item_id*: string, status: string, summary: string, category: string, tags: string, quality_score: number	Update item fields (status, summary, category, tags, quality_score). Used by observer-agent after AI processing.
items_curate	item_id*: string, action*: string	Approve, reject, or feature an item for publishing.
digest_generate	date: string, category: string	Generate a daily digest from processed/curated items.
stats	—	System overview: item counts, source health, category distribution.
daemon_status	—	Check observer daemon status: uptime, scheduled jobs, current harvest, master schedule state. Requires the daemon HTTP service to be running.
schedule_toggle	batch_id: string, enabled: boolean, all: boolean	Enable or disable scheduled harvesting. Toggle the master schedule (all batches) or a specific batch schedule.
schedule_update	batch_id*: string, schedule*: string	Change the harvest schedule for a batch. Supports interval strings (15min, 30min, 1h, 2h, 4h, 6h, 12h, 24h), daily time (daily:09:00), or cron expressions (0 9 * * *).

Where to go next

• Observer — harvesting — the human surface: sources, batches, and schedules in the app
• Intelligence — publishing — what the pipeline turns harvested items into