Crawl rate limits belong to site owners

How Oernoe Search works puts politeness in the crawling list next to continuous crawl, privacy-first crawling, and content analysis. The exact line is short: we respect robots.txt files and crawl rate limits set by website owners. That sentence assigns ownership. The rate limit is not a decoration Oernoe invented for marketing. It is a control published by the site that is being visited.

This essay stays on that assignment. It is not the essay about rate limits on the crawler as a product feature in the abstract. It is not the robots-txt-is-not-a-cover-up piece. It is not allow-all-except-api-and-debug. It is not continuous-crawl-is-not-complete-coverage again. Those neighbors matter, but the claim here is narrower: when a website owner publishes crawl limits, those limits belong to them, and Oernoe’s published behavior is to respect them.

## What politeness means in the guide

The crawling process section is a four-bullet list. Continuous crawling means the bots work around the clock to visit pages and stay current with new and updated content. Respecting robots means honoring robots.txt and owner-set crawl rate limits. Privacy-first crawling means the crawl does not store IP addresses, device information, or identifying data as part of that process. Content analysis means reading page content, structure, and links for relevance, without harvesting contact addresses for advertising lists, tracking identities, or storing behavioral signals for ads.

Politeness sits in the second bullet as operational law, not as a vibe. A robots file can disallow paths. A crawl-delay or equivalent rate restraint can ask the crawler to slow down. Site owners publish those signals because their servers, their bandwidth, and their application budgets are theirs. A search company that ignores them is not being thorough; it is being rude, and eventually blocked.

Oernoe’s guide chooses the polite side in public language. That choice is part of the product record anyone can read without a sales call.

## Ownership is the point

“Set by website owners” is the phrase that does the ethical work. Oernoe does not get to redefine another organization’s capacity as Oernoe’s entitlement. Continuous crawl describes Oernoe’s desire to stay current. Rate limits describe the other party’s right to remain upright while that happens. When they conflict, the published rule is that the owner’s limit wins.

This is easy to say and easy to violate in practice. Crawl teams are measured on freshness and coverage. Owners are measured on uptime and cost. Without an explicit norm, freshness pressure eats the owner’s budget. Publishing the norm makes the tradeoff checkable. If Oernoe’s crawler ever treats a published limit as optional, the guide is false, and the fix is engineering and an honest correction—not a softer blog tone.

Robots.txt is the common machine-readable surface for allow and disallow rules. Rate limits may appear as crawl-delay directives where supported, as published policies, or as infrastructure that returns throttling responses. The guide groups robots files and crawl rate limits together because both are owner signals. Respecting one while ignoring the other would be selective hearing.

## Politeness is not secrecy

A separate temptation is to treat robots compliance as a privacy story or a cover story. It is neither. Politeness is about load and permission on public hosts. Privacy-first crawling is about what identifying material the crawler refuses to store. Content analysis is about what public text enters relevance scoring. Advertising disclosures on www are about Google ads on selected publisher pages. Those topics share a company and sometimes a paragraph. They are still different controls.

Search-and-ads notes that robots.txt on the publisher site allows Google’s ads crawlers so an ads bot can see the same writing a person sees. That is a publisher-side allow for verification and ad crawl on www. It does not rewrite Search’s obligation to honor other sites’ robots files and rate limits. Different hosts publish different robots rules. Ownership stays local to each host.

Nor does politeness mean Oernoe hides pages. What we publish explains that some older promotional URLs stay resolvable for old links but leave the index with noindex, and that application shells can send noindex headers. Those are publishing and indexing choices on Oernoe’s own properties. They are not a substitute for respecting someone else’s crawl-delay.

## Continuous crawl without unbounded demand

Continuous crawl sounds absolute if you only read the first bullet. The second bullet exists to stop that misreading. Working around the clock does not mean hammering every host at maximum concurrency. It means the system keeps a schedule of revisits and discovery walks, subject to owner constraints and to the finite nature of any real crawl.

Coverage remains incomplete even when the crawler is polite and busy. Many pages are never linked. Many hosts block bots. Many URLs change faster than any schedule. Journal pieces already warn that continuous crawl is not complete coverage. This essay adds the complementary warning: even the pages you are allowed to fetch must be fetched at a pace the owner accepts.

That is why rate limits “belong” to site owners in the title. Belonging is not a metaphor about feelings. It is a statement about who sets the constraint. Oernoe can choose to be a good citizen relative to that constraint. Oernoe cannot honestly claim the constraint was Oernoe’s idea of generosity if the owner published it first.

## How this interacts with public-link discovery

Crawlers follow public links from pages they already know. Discovery expands the frontier. Rate limits shape how fast that frontier can be walked on any one host. A large site with a strict limit will be revisited more slowly than a small site that allows faster fetch. That unevenness is not a bug in the politeness rule. It is the politeness rule working.

Privacy-first crawling still applies while the bot waits. Slowing down does not create permission to store identifying data. Content analysis still applies to what is fetched: public page text, structure, and links for relevance—not advertising-list harvests. Ranking still treats queries equally unless the account holder saved preferences. Politeness changes pace. It does not rewrite the privacy model.

## Publisher site versus Search crawl

www.oernoe.com is the public publisher site operated by Anoepal. Selected finished pages may load Google AdSense. Search lives at search.oernoe.com. Search-and-ads exists to keep those systems from being mashed into one dishonest privacy sentence. Crawl rate limits belong in the Search crawl story. Ad placement rules belong in the publisher story. Both can mention robots files without becoming the same paragraph.

When Oernoe publishes its own robots.txt, that file speaks for Oernoe’s hosts. When another organization publishes robots.txt, that file speaks for them. Respecting “rate limits set by website owners” means treating foreign robots directives as authoritative for those foreign hosts. It does not mean Oernoe’s own ads-crawler allows somehow license ignoring a newspaper’s crawl-delay.

About and the homepage name the operator, the founder, and the split between publisher writing and private apps. None of that branding overrides a third-party robots rule. Brand is not bandwidth.

## What reviewers and publishers should see

A publisher reading How Search works should find an explicit commitment to robots.txt and owner-set crawl rate limits. A reviewer checking originality should find journal prose that explains the commitment instead of repeating the bullet. A site owner who throttles Oernoe’s bot should expect the bot to back off, not to invent a privileged channel.

Checks are ordinary. Read the guide. Compare crawler behavior to the published line. If logs show sustained disregard for a clear owner limit, the company owes a fix and a correction, not a definitional argument that “continuous” meant “unlimited.”

What we publish reminds us that journal articles need enough original text to be documents, and that product claims get checked against live hostnames and policy pages. This essay’s product claim is the politeness bullet. The policy neighbors are Privacy for logs and ads disclosures, and Search-and-ads for the Search-versus-publisher split. If those sources disagree, the article is wrong.

## What this essay is not doing

It is not claiming robots.txt is cryptography or a privacy guarantee. It is a politeness and permission file for crawlers. It is not claiming Oernoe never keeps technical logs; Search may keep logs needed for abuse and uptime, and those logs are not an ad graph. It is not claiming Search indexes everything allowed. It is not claiming the publisher site has no ads. It is not walking through interface chrome or inventing user counts.

It is also not redefining rate limits as a unique Oernoe invention. Many crawlers speak this language. The journal job is to state what Oernoe publishes about its own bot, in human editor voice, with the ownership clause intact.

## Why ownership language beats feature language

Calling rate limits a “feature” of the crawler centers Oernoe. Calling them limits that belong to site owners centers the host being visited. Both sentences can be true in a narrow sense: the crawler implements backoff, and the owner publishes the rule. The journal prefers the ownership framing because it matches the guide’s grammar—“set by website owners”—and because it resists a marketing slide where Oernoe congratulates itself for inventing restraint.

Feature language also invites a quiet inversion: if the limit is “our feature,” we can tune it for our freshness goals first. Ownership language blocks that inversion. The owner’s published signal is the input. Oernoe’s implementation is the response. When the response drifts from the signal, the bug is on Oernoe’s side.

## Operational meaning without internal chrome

In practice, respecting rate limits means the crawl scheduler treats owner constraints as hard inputs to pacing, not as suggestions to revisit after a growth meeting. Bursts that look like outages are failures even if they temporarily improve index freshness. Coverage metrics should never be the only scoreboard; politeness has to sit beside them.

Internal admin chrome is not the public record. The public record is the guide sentence and the observable behavior at someone else’s robots boundary. If a site owner asks whether Oernoe honors their crawl-delay or equivalent throttle, the answer should match How Search works without a detour into brochure adjectives.

## Shared operator, separate obligations

Anoepal operates Search and the publisher site. Founder credit and contact paths live on About and Contact. None of that corporate fact lets Search borrow authority from www’s robots allows, or lets www’s ad disclosures rewrite crawl pacing on third-party hosts. Separate obligations keep the map readable: publisher ads on selected finished www pages are disclosed; Search crawl politeness is owed to every website owner who publishes a limit; private apps stay off the publisher ad inventory and off the public crawl path.

Search-and-ads warns against mashing systems into one privacy slogan. The same warning applies to mashing politeness into a brand halo. Respecting rate limits is ordinary good-bot behavior. State it, implement it, correct it when it fails. Do not inflate it into a claim that the whole network never touches a third party or never shows ads.

## How this essay stays distinct

Neighboring journal pieces cover capability framing, robots-as-cover-up myths, specific robots postures on Oernoe properties, and the completeness myth of continuous crawl. This piece isolates ownership: who sets the limit, and why Oernoe’s guide assigns that power to website owners rather than to Oernoe’s appetite for freshness.

## Closing the loop

Crawl rate limits belong to site owners because site owners publish them for their own hosts. Oernoe’s How Search works guide says the crawler respects robots.txt and those limits. Continuous crawl keeps trying to stay current. Politeness keeps that effort from becoming a denial-of-service costume. Privacy-first crawling and content analysis still constrain what is stored and what is scored. Keep the ownership clear, and the rest of the crawl story stays honest.