Observer — harvesting

The Observer view is the intake half of the intelligence desk: a monitoring engine you point at any sources you care about — blog feeds, X accounts, YouTube channels, your newsletter inbox. On a schedule you set, it fetches what is new, strips it to clean text, throws away duplicates, and files everything on your machine.

The design principle worth knowing up front: harvesting is mechanical and costs no model tokens. Fetching, extraction, and de-duplication run without any AI call and — for the built-in methods — without any paid API key. The AI enters later, as a separate layer that reads the harvested pile (see Intelligence — publishing).

The view at a glance

The tabs are ordered the way the pipeline runs. The first three are the mechanical harvest stage; the rest are the analysis stage plus settings.

Four words carry the whole model: a source is one feed; a batch is a named group of sources on a shared schedule; an item is one harvested record (a clean Markdown body plus an index row); a schedule is a per-batch or per-source interval.

Add a source

1. Open Sources and choose New source.
2. Pick the type: RSS, X / Twitter, YouTube, YouTube Video, or Email Newsletter.
3. Paste the URL. You do not need the exact feed address — paste a channel @handle or a site homepage and the resolver discovers the real feed.
4. Click Test feed. The engine probes the source before saving and shows the feed title, item count, and sample headlines — so you never save a source that does not work.
5. Assign a category (create one inline if needed), optionally set a trust weight, and optionally enable a schedule on creation. A scheduled source fires its first harvest about a minute after you save it.

None of the built-in source types needs an API key: X accounts are read through public mirrors, YouTube channels through their native feeds plus transcript extraction, RSS directly, and newsletters through an inbox you own. Your install ships with a curated starter registry of AI and tech sources arranged into ready-made batches — keep them, trim them, or replace them entirely with your own.

Group sources and switch on schedules

Batches are how harvesting becomes unattended. In Batches:

• Group related sources into one collection — for example a competitor's blog, X handle, and YouTube channel in a single batch.
• Give the batch an interval: anywhere from every 15 minutes to every 24 hours, a daily at-a-time run, or a full cron expression. Individual sources can also carry their own schedule.
• Switch the schedule on. Schedules are off on a fresh install — harvesting is a deliberate action you enable, not something running behind your back from day one.

Once on, the engine runs by itself: it fires a warm-up harvest about a minute after you enable a schedule, records a run entry even when nothing new was found, and ages old items out of the working set (see Retention). Retiring a batch keeps its configuration but takes it out of rotation — a batch only auto-runs when it is both active and its schedule is switched on.

Schedules fire while Command Center is running. The harvesting service is started and stopped by the app, so closing Command Center stops the clock — and this is the one place on the desk that makes up for it: at start, any batch whose interval has already elapsed is treated as overdue and harvested, so an overnight gap or a machine that was asleep costs you a delay rather than a day of sources.

What a harvest actually does

Every run is deterministic work, no judgment involved:

• Only what is new. Each source remembers the newest thing it saw and skips everything older — by publish date, post ID, video URL, or the unread flag, depending on the type.
• Clean text. Thin feed entries are upgraded to full article text; newsletter HTML is stripped to readable plain text; video transcripts are paragraphed by natural speech pauses.
• Duplicate-proofing, three layers deep. The same story arriving from three feeds is stored once — caught by exact-URL matching, fuzzy title similarity, and content fingerprinting. All plain arithmetic, no AI.
• Local, private storage. Items land as Markdown files in your workspace plus a local search index. Inbox passwords go to the OS keychain. Nothing leaves your machine to harvest.

Review any run in Harvest Runs — each entry lists what came in, what was rejected as duplicate, and any per-source errors, with the item bodies readable in place.

Curate the pile

The AI Processing tab is where the harvested pile meets your agents. From here you dispatch processing runs that read fresh items and score, categorize, summarize, and curate them into dated observer reports — the mid-layer the publishing desk builds on. These runs spend your own AI subscription, on whichever agent CLI you have configured, and only when there is genuinely new material; the harvest itself never does.

The Video Analysis tab runs the deepest single workflow on the desk: paste a YouTube URL (or register a channel as a YouTube Video source), let the engine download the video and extract its transcript plus content-aware keyframes, then dispatch an analyzer that writes a structured deep-analysis report — with charts, code, and URLs read off the screen and anchored to timestamps. The video file itself is removed after extraction; the durable artifact is the report, which lands in Reports.

Retention and the archive

Harvesting accumulates. The Settings tab's Retention panel is where you decide what happens to items as they age, and it runs two independent clocks — only the second one deletes anything.

The working set. By default an item leaves the live pile after 14 days. Leaving is not deleting: the default disposition is Archive, which moves the item's files into an archive folder inside the harvest tree, where they stay readable. The index row is kept either way, which is what stops a feed re-harvesting the same story after it ages out.

Getting an item back. Archived items are restorable. In Harvest Runs, a run that has archived items carries a Restore action that brings them back into the working set and makes them claimable by the analysis layer again. That is the whole reason archiving is the default disposition rather than deleting — an archive you cannot come back from is a delete with extra steps.

The archive clock. Separately, and off unless you switch it on, the archive can delete its own contents once they have been in there long enough (90 days when you enable it, measured from the archive date). This is the one destructive clock on the desk, and it is the one that ships disabled. The item record survives it, so nothing gets re-harvested; only the content is gone, and there is no second copy.

Three practical notes:

• Preview before you change anything. The panel's Preview button reports exactly what the next sweep would do — how many items would be archived, how many of those were never processed, and how many would be deleted if the archive clock is armed. Run now performs the sweep on demand.
• The first sweep on a backlog moves a lot at once. If you install, harvest for a while and only then look at Retention, the first sweep will act on everything already past the window. Preview it first.
• Items ageing out unprocessed is a signal, not a setting. The preview calls it out when it happens: it means the harvesters are producing and nothing is consuming. Usually the fix is switching on a processing routine, not widening the window.

This clock covers harvested items only. It reads the harvest tree and nothing else — reports, your own documents, and everything else in the workspace are untouched by it.

Newsletter inbox and scale

• Newsletters: in Settings, connect an inbox you own (IMAP). A dedicated address you subscribe newsletters to works best — each issue becomes a clean, searchable item, and the read flag is the watermark. No inbox is configured on a fresh install; this is yours to add. This is the inbox the desk reads; sending your own newsletter is a separate provider, set up in Intelligence — publishing.
• Heavy harvesting: everything harvests directly out of the box. For high-volume monitoring — many YouTube channels especially — you can optionally route through residential proxies you bring. Nothing is bundled and nothing is required. The controls live in this view's Settings ▸ Proxy panel, which ships set to direct; for what to put in them, see Bring your own proxy.

Where to go next

• Intelligence — publishing — turn the harvested pile into briefings, reports, and newsletters
• Routines & automation — how the scheduled layer on top of harvesting works