Running a job against the outside world

Any job that makes bulk requests to machines we do not own — a DNS sweep, a scrape, a crawl, an enrichment fan-out — is covered by this page. It exists because we got a Google Cloud project suspended for denial-of-service on 2026-09-16, and the suspension was correct.

The rule. Rate and concurrency limits are abuse limits, not tuning knobs. If you are asking "how do I avoid getting throttled", stop. That is a signal to slow down, not a puzzle to solve.

The one-line test

Before the job runs, answer this: what does the target see?

Not what you receive. What a single nameserver, or a single site's operator, sees arriving from us. If the honest answer would alarm them, the job is abusive regardless of what it is called internally or how useful the output is.

Four rules

1. Never bypass a rate limit

A rate limit is the answer, not the obstacle. All of the following are ways of attacking a service while feeling clever, and none of them are permitted:

Pattern	Why it is banned
A local recursive resolver, to skip public-resolver limits	Sends traffic straight at authoritative nameservers, which have no relationship with us and never opted in
Proxy pools, residential proxies, IP rotation	The block was the service declining. Routing around it does not make it consent
Sharding across accounts, projects or regions	Same total load on the target, laundered
Raising a ceiling because the last run "held fine"	The target's tolerance is not measured by whether we got caught

2. Caps are constants, not flags

A limit any operator can raise at 2am is not a limit.

const RATE_CAP = 50;                       // hard cap — not a tuning knob
const rate = Math.min(RATE_CAP, Number(argOf("--rate", 25)));

The flag clamps to the constant. Raising the cap is a code change, reviewed, with a reason written down. Never a ceiling the backoff logic is free to climb back to.

This is the rule that decides blast radius, and the one we actually broke.

	Runs here	Never runs here
Core project — service accounts, Docs/Sheets/Drive, wizards, crons	Business automation	Sweeps, scrapes, crawls, bulk third-party traffic
Disposable project — separate, nothing depends on it	Anything bulk against the outside world	Anything another system needs to be up

A suspension in a disposable project costs a rerun. The same suspension in the core project takes the wizards, document generation and every cron with it. Assume any bulk job may get its host suspended, and place it accordingly.

4. Watch the provider's compliance mail

Abuse mail is a deadline, not a notification. Our warning arrived three days before the suspension and nobody read it. A warning that reaches an unwatched inbox bought us nothing.

Default to slow

The instinct to finish a sweep in hours instead of days is where this goes wrong. Cheap arithmetic, honestly done, usually shows the slow version is fine:

	Aggressive	Capped
Rate	1,400 q/sec	50 q/sec
20.8M domains	~4 hours of query time	~5 days of query time
Outcome	Project suspended, VM terminated, work stopped for days	Finishes

The slow version is faster, because it finishes.

What happened, concretely

For the record, so the next person can recognise the shape of it:

scripts/prospecting/sweep-all.mjs resolved SPF records across ~20.8M domains to find NetSuite customers. It ran at up to 1,400 DNS queries/sec from a Compute Engine VM, weighting a local unbound recursor that queried authoritative nameservers directly — added specifically to escape public-resolver rate limits. Google Cloud Trust & Safety classified it as a denial-of-service attack and suspended the VM.

Nothing about the intent was malicious. Every individual change was someone optimising throughput. The comments in the file read so it has no third-party rate limit and together they saturate the link — written as achievements. If a comment you are writing reads like that, you have already gone wrong.

Before you start the job

What does a single target see? Would it alarm them?
Is the cap a constant that flags clamp to?
Is this running somewhere nothing else depends on?
Am I routing around a limit rather than respecting it?
Is someone watching the provider's abuse mail?