Reference
The Prospecting Worker
How the always-on, multi-brand market discovery engine runs — 24/7, 365, in the cloud — and how its output becomes the market globe.
This worker was suspended for DoS on 2026-09-16
It ran at up to 1,400 DNS queries/sec against ~20.8M domains, weighting a local unbound recursor that queried authoritative nameservers directly — added specifically to escape public-resolver rate limits. Google Cloud Trust & Safety classified that as a denial-of-service attack and terminated the VM.
The rate is now a hard cap of 50 q/sec that --rate clamps to, and the local recursor is gone from the pool. Do not raise the cap, do not reintroduce a local resolver, and do not run this in a project anything else depends on. Read Running a job against the outside world before changing anything here.
The Prospecting Worker
The worker finds every company in a brand's total addressable market that proves, in its own public DNS, that it runs the vendor the brand serves. It never stops: it sweeps, archives, pauses a day, and sweeps again — for every brand on its job list. New accounts land straight in Supabase and show up on that brand's market globe.
Info
The rule (mk, 2026-09-12): "we want to run this 24-7, 365 for multiple brands all the time." Market discovery is a permanent cloud service. It never runs on a laptop and never as a one-off script someone remembers to launch.
What it proves, and why that's enough
A company's SPF record is a DNS TXT record its own IT team publishes to say which services may send email as its domain. When a record includes the vendor's mail host — for NetSuite, include:…netsuite.com — the company has authorised that vendor to send its invoices and statements. That is evidence the company published about itself, not a technographic guess.
Proof, not inference
Every account carries its evidence row: the signal, the raw record, the lookup method, the source, and when it was captured. An account with no evidence does not exist.
Unknown is not negative
A DNS timeout says nothing about the company. Unanswered lookups are logged to a retry list, never counted as "doesn't use the vendor".
How it runs
The source
A 2.1 GB Common Crawl domain graph (~95M hosts). The worker streams it line by line and keeps only apex domains in the job's scope.
The lookup
One DNS TXT query per domain, spread across public resolvers, capped at 50 queries/sec. The error rate is sampled continuously: above the ceiling the rate backs off, and it climbs back only as far as the cap.
The write
Every match is inserted into prospect_account for the brand — vendor_tier = A, the SPF evidence in vendor_signals, a batch tag per job — with on conflict do nothing, so re-sweeps never duplicate.
The checkpoint
Progress is saved every 30 seconds. A crash, a dropped database socket, or a reboot resumes from the last checkpoint instead of starting over.
The cycle
When every job is done, the finished checkpoints and hit files are archived by date, the worker pauses a day, and the whole list runs again — so companies that adopt the vendor later still get found.
Where it lives
| Piece | What it is |
|---|---|
| Server | Google Compute Engine VM taqm-sweep — project zero-hour-app, us-central1-a, e2-standard-4 (4 vCPU · 16 GB) |
| Service | taqm-worker.service (systemd) — restarts on crash and on reboot |
| Loop | scripts/prospecting/worker.sh — works through the job list forever |
| Sweep | scripts/prospecting/sweep-all.mjs — one job: brand + vendor + signal + scope |
| Jobs | scripts/prospecting/jobs.json |
| Output | Supabase prospect_account (per brand) → /studio/{slug}/market |
| Cost | ~$100 / month running continuously |
The job list
A job is four things. Adding a brand is adding a line — nothing else changes.
[
{ "brand": "grapevine", "vendor": "netsuite", "spf": "include:[^ ]*netsuite\\.com", "scope": "intl" },
{ "brand": "grapevine", "vendor": "netsuite", "spf": "include:[^ ]*netsuite\\.com", "scope": "na" }
]| Field | Meaning |
|---|---|
brand | The brand slug the accounts belong to |
vendor | Written to prospect_account.vendor |
spf | The pattern in a company's SPF record that proves the vendor |
scope | na = .com .net .org .us .ca .io .co · intl = country codes and their second-level forms (acme.co.uk, acme.com.au) |
Warning
The market is global. The first Grapevine run only accepted North-American domain endings and silently dropped every international company before a single lookup. The intl scope exists because of that — its first minutes found deliveroo.co.uk, cancer.org.au, rugby.com.au. Every vendor job should run both scopes.
From sweep to globe
- Sweep — the worker writes proven accounts to
prospect_account. - Enrich — contact discovery, email verification, headcount and location run against those accounts.
- Place — an account's
country/region(from its own homepage address or country domain) puts it on the map. - Render —
/studio/{slug}/marketshows the brand's potential vs actualized market on one globe. Owner-gated; brands opt into a public, anonymized pipeline withbrand.config.market_globe.pipeline_public.
Safeguards
Operating it
| To… | Do |
|---|---|
| Check it's running | gcloud compute ssh taqm-sweep --zone us-central1-a → systemctl status taqm-worker |
| Watch progress | tail -f /Users/whoismattkohn/0-hr/.data/prospecting/worker.log on the VM |
| Add a brand or vendor | Add a line to jobs.json on the VM, then sudo systemctl restart taqm-worker |
| Pause everything | sudo systemctl stop taqm-worker |
| Resume | sudo systemctl start taqm-worker — every job continues from its checkpoint |
Note
If gcloud says reauthentication failed, the laptop's Google session expired. Run gcloud auth login once — the VM keeps running regardless; only reaching it needs a live login.