BYO proxy setup
Bring-your-own proxy — setup, verification, and common pitfalls.
Command Center's Observer (news/feed/YouTube harvesting) works out of the box with no proxy — both transport toggles ship as "No proxy" and all requests go out on your machine's own connection. For light use that is usually fine. Heavy YouTube transcript harvesting or large source lists will eventually hit upstream bot-detection from a single IP; that is what residential proxies solve, and the platform supports any provider.
The two shapes every provider fits
| Shape | What it is | Where it goes |
|---|---|---|
| Rotating gateway | ONE URL your provider rotates exit IPs behind (fresh IP per request). Session pinning, country targeting, etc. are encoded inside the URL's username/password or port — exactly as your provider's dashboard shows it. | Observer ▸ Settings ▸ Proxy ▸ Rotating gateway URL (stored in .env as PROXY_ROTATING_URL) |
| Static pool | A list of fixed residential IPs (host:port per line) plus credentials — either one shared username/password, per-IP inline credentials (host:port:user:pass), or none at all if your provider authorizes your machine's IP (IP-whitelist). | Observer ▸ Settings ▸ Proxy ▸ IP pool (+ pool credentials → .env) |
You don't need both. The tuned "Proxy" routing uses the static pool for feed listings and the rotating gateway for YouTube transcripts (they face opposite bot-detection behavior), and degrades gracefully when only one is configured.
Setup
- Open Observer ▸ Settings ▸ Proxy.
- Pick a Provider: Webshare (preset — just enter username/password; the rotating gateway is derived automatically) or Custom (any provider).
- For Custom: paste your provider's rotating gateway URL (use the Test button before saving — it reports the exit IP + latency) and/or your static IP list + pool credentials. Tick IP-whitelist if your provider authenticates by source IP instead of credentials.
- Flip the Harvest and/or Video processing toggle to Proxy.
- Restart the observer daemon (Dashboard ▸ Platform Health) so credential changes apply.
Provider cheat-sheet
Verify in your provider's dashboard — formats change.
| Provider | Rotating gateway (typical) | Notes |
|---|---|---|
| Webshare | built-in preset | -rotate username suffix, port 80 — handled for you |
| Bright Data | brd.superproxy.io:33335 | zone/country/session tokens in the username (brd-customer-…-zone-…) |
| Oxylabs | pr.oxylabs.io:7777 | username customer-<user>; -cc-XX for country; sticky via ports 10000+ |
| Decodo (Smartproxy) | gate.decodo.com:7000 | port selects rotation: 7000 = rotate, 10001+ = sticky |
| IPRoyal | geo.iproyal.com:12321 | session/country tokens in the password (pass_country-de_session-…) |
| SOAX | proxy.soax.com:5000 | package + session tokens in the username |
HTTP(S) proxies only; enter URLs exactly as your provider documents them.
Expectations
- The battle-tested routing behavior (feed listings on sticky IPs, transcripts on rotating IPs, video downloads pinned to stable IPs) is provider-independent, but observed success rates vary by provider's IP reputation. The built-in retries, rotating fallback, and circuit breaker absorb most variance; the Sources tab shows per-source backoff if a provider underperforms.
- Credentials never leave your machine: they live in your local
.env(never in the repository) and the Settings panel only ever reports whether they are set.
If harvest problems persist after a proxy is configured and tested, work through the daemon-health cluster in Troubleshooting before suspecting the proxy.