Observer — harvesting

The Observer view is the intake half of the intelligence desk: a monitoring engine you point at any sources you care about — blog feeds, X accounts, YouTube channels, your newsletter inbox. On a schedule you set, it fetches what is new, strips it to clean text, throws away duplicates, and files everything on your machine.

The design principle worth knowing up front: harvesting is mechanical and costs no model tokens. Fetching, extraction, and de-duplication run without any AI call and — for the built-in methods — without any paid API key. The AI enters later, as a separate layer that reads the harvested pile (see Intelligence — publishing).

The view at a glance

Seven tabs, ordered the way the pipeline runs. The first three are the mechanical harvest stage; the rest are the analysis stage plus settings.

Four words carry the whole model: a source is one feed; a batch is a named group of sources on a shared schedule; an item is one harvested record (a clean Markdown body plus an index row); a schedule is a per-batch or per-source interval.

Add a source

1. Open Sources and choose New source.
2. Pick the type: RSS, X / Twitter, YouTube, YouTube Video, or Email Newsletter.
3. Paste the URL. You do not need the exact feed address — paste a channel @handle or a site homepage and the resolver discovers the real feed.
4. Click Test feed. The engine probes the source before saving and shows the feed title, item count, and sample headlines — so you never save a source that does not work.
5. Assign a category (create one inline if needed), optionally set a trust weight, and optionally enable a schedule on creation. A scheduled source fires its first harvest about a minute after you save it.

None of the built-in source types needs an API key: X accounts are read through public mirrors, YouTube channels through their native feeds plus transcript extraction, RSS directly, and newsletters through an inbox you own. Your install ships with a curated starter registry of AI and tech sources arranged into ready-made batches — keep them, trim them, or replace them entirely with your own.

Group sources and switch on schedules

Batches are how harvesting becomes unattended. In Batches:

• Group related sources into one collection — for example a competitor's blog, X handle, and YouTube channel in a single batch.
• Give the batch an interval: anywhere from every 15 minutes to every 24 hours, a daily at-a-time run, or a full cron expression. Individual sources can also carry their own schedule.
• Switch the schedule on. Schedules are off on a fresh install — harvesting is a deliberate action you enable, not something running behind your back from day one.

Once on, the engine runs by itself: it fires a warm-up harvest about a minute after you enable a schedule, catches up on overdue batches after your machine was asleep, records a run entry even when nothing new was found, and prunes old items on a retention window. Retiring a batch keeps its configuration but takes it out of rotation — a batch only auto-runs when it is both active and its schedule is switched on.

What a harvest actually does

Every run is deterministic work, no judgment involved:

• Only what is new. Each source remembers the newest thing it saw and skips everything older — by publish date, post ID, video URL, or the unread flag, depending on the type.
• Clean text. Thin feed entries are upgraded to full article text; newsletter HTML is stripped to readable plain text; video transcripts are paragraphed by natural speech pauses.
• Duplicate-proofing, three layers deep. The same story arriving from three feeds is stored once — caught by exact-URL matching, fuzzy title similarity, and content fingerprinting. All plain arithmetic, no AI.
• Local, private storage. Items land as Markdown files in your workspace plus a local search index. Inbox passwords go to the OS keychain. Nothing leaves your machine to harvest.

Review any run in Harvest Runs — each entry lists what came in, what was rejected as duplicate, and any per-source errors, with the item bodies readable in place.

Curate the pile

The AI Processing tab is where the harvested pile meets your agents. From here you dispatch processing runs that read fresh items and score, categorize, summarize, and curate them into dated observer reports — the mid-layer the publishing desk builds on. These runs use your own Claude subscription and only spend when there is genuinely new material; the harvest itself never does.

The Video Analysis tab runs the deepest single workflow on the desk: paste a YouTube URL (or register a channel as a YouTube Video source), let the engine download the video and extract its transcript plus content-aware keyframes, then dispatch an analyzer that writes a structured deep-analysis report — with charts, code, and URLs read off the screen and anchored to timestamps. The video file itself is removed after extraction; the durable artifact is the report, which lands in Reports.

Newsletter inbox and scale

• Newsletters: in Settings, connect an inbox you own (IMAP). A dedicated address you subscribe newsletters to works best — each issue becomes a clean, searchable item, and the read flag is the watermark. No inbox is configured on a fresh install; this is yours to add.
• Heavy harvesting: everything harvests directly out of the box. For high-volume monitoring — many YouTube channels especially — you can optionally route through residential proxies you bring. Nothing is bundled and nothing is required; see Bring your own proxy.

Where to go next

• Intelligence — publishing — turn the harvested pile into briefings, reports, and newsletters
• Routines & automation — how the scheduled layer on top of harvesting works