X-Robots-Tag fixes what robots.txt missed

Corrections we have already made tells a mechanical story that is easy to summarize wrongly. robots.txt used to disallow application-shell paths with a trailing slash. The live routes were the bare paths without the trailing slash. Google could fetch the bare URLs, which rendered as near-empty application shells. A reviewer who typed those addresses saw a site padded with empty pages. The fix is the opposite of blocking more paths. Application shells now send `X-Robots-Tag: noindex, nofollow`. robots.txt only disallows /api/ and /debug/, so a crawler can fetch the URL and honour the header. This essay is corrections and Search limits writing about that fix. It is not a remake of the robots-txt-as-politeness essay. It is not a remake of the shells-should-not-rank essay. Those hinges already exist. The hinge here is the header tool itself: why X-Robots-Tag is what robots.txt missed, and what Search limits follow once a publisher stops confusing Disallow with noindex.

## What robots.txt missed on www

A robots.txt Disallow line is an instruction not to fetch. It is not an instruction to drop a URL from an index after a fetch. Corrections’ trailing-slash story is a mismatch between the written Disallow strings and the live route shapes. Paths written as directory-style entries with a trailing slash did not match the bare routes the site actually served. Crawlers that requested the bare routes received pages. Those pages were shells: near-empty application entry points, not finished publisher documents. The politeness file had been asked to do concealment work it could not do, because the strings were wrong.

Even a correctly spelled Disallow can fail the noindex job. If a crawler never fetches the URL, it never receives response headers. A noindex signal that lives on the response cannot arrive. Corrections states the lesson in one sentence worth keeping: a Disallow line that stops Google from reading noindex is how you stay indexed by accident. That sentence is the reason this essay centers X-Robots-Tag rather than politeness theory. Politeness answers whether Search should honour other people’s robots files while crawling the open web. The publisher correction answers how www stopped training reviewers on empty shells. The tool that completed the correction is the header.

What we publish ends with the same mechanical checks in publisher voice. robots.txt should allow Google’s ads crawlers. Application shells send a noindex header; they are not hidden behind a trailing-slash Disallow, because that stopped Google from reading the header. The recovery rule and the Corrections list agree. Fetchability plus noindex beats pretend invisibility when the page must still exist for old links or app entry.

## What X-Robots-Tag is for

`X-Robots-Tag: noindex, nofollow` on a response tells a cooperating crawler that the URL may be fetched and then left out of the index, and that links on that response should not be followed for discovery in the usual way the tag specifies. It is a page-level ranking instruction delivered with the document. It does not require the URL to be absent from the web. It requires the crawler to see the header. That is why narrowing robots.txt to /api/ and /debug/ is part of the fix rather than a strange relaxation. The crawler must be allowed to fetch the shell so the shell can refuse ranking.

Live robots.txt on www.oernoe.com, fetched for this essay, matches that design. Mediapartners-Google and Google-Display-Ads-Bot are allowed across the site so ads crawlers can evaluate publisher inventory. The ordinary user-agent is allowed across the site, with Disallow only for /api/ and /debug/. The file declares the host as https://www.oernoe.com and points at the public listing of URLs the company claims to publish. It does not attempt to hide application shells behind mismatched trailing-slash rules. Shells that still resolve are supposed to carry the header, not hide behind a Disallow string that never matched.

Search limits language belongs here because indexing is a Search-shaped problem even when the subject is the publisher hostname. How Oernoe Search works lists robots respect and crawl rate limits as ordinary crawler behavior on the open web. A crawler that never sees a noindex header cannot honour it. A publisher that wants shells out of foreign indexes must let those crawlers fetch and read the tag. Cover-up thinking wants the opposite: block the fetch and also trust every crawler already knew not to rank the URL. Corrections already showed what that fantasy produced on www: fetchable emptiness and a padded look.

## Disallow, noindex, and rewrite are three tools

Corrections separates several recovery tools across failures. Generic SEO journal addresses still exist so old links do not break, but the documents on them are now Oernoe-specific. That is rewrite. Promotional stubs can stay up for old links while leaving the public index and ads inventory. That is noindex plus de-monetization, described in What we publish. Application shells get the header treatment because they are not candidates for a meaty rewrite. There is nothing honest to expand into a finished journal essay inside an empty console route. robots.txt Disallow remains appropriate for paths that should not be fetched at all, which on the live file means /api/ and /debug/.

Quietly swapping a sentence and pretending the first version never shipped is off the table. Corrections defines a correction as the old wording no longer being the page, or the URL being noindexed, or both. Using robots.txt as a silent cover-up would be the opposite of that definition. Publishing the trailing-slash miss and the X-Robots-Tag fix is how the company keeps the list useful to a reviewer from the Google publisher network.

Ads inventory follows the same honesty. What we publish refuses to monetize login, legal, thin stubs, or the journal listing. Privacy Policy section 4 keeps short or promotional posts, legal pages, error pages, and protected application pages out of AdSense. Cookie and publisher language elsewhere on the site treat application shells as non-inventory. Allowing ads crawlers in robots.txt while noindexing shells is the coherent pair: evaluate finished documents, ignore hollow routes. A trailing-slash Disallow that missed live shells did not protect inventory. It created the low-value look the publisher network is trying to avoid.

## What How Search works contributes without becoming a remake

How Search works explains that Search keeps an index of public web pages it has fetched. Crawlers visit pages they already know about, follow public links, and come back when a page changes. Continuous crawling means the work runs around the clock. Respecting robots means honouring robots.txt files and crawl rate limits set by website owners. Privacy-first crawl claims in that guide say the company does not store identifying data during crawling as advertising harvest. Ranking refuses search-history advertising profiles. Funding is optional premium features and contextual ads on selected publisher pages. Queries are not sold or used to build advertising profiles.

None of that funding language buys the right to ignore robots.txt on the open web. None of it converts a publisher Disallow into a substitute for X-Robots-Tag on www shells. Search honouring other people’s robots files is neighbor behavior. The publisher fixing its own shells with a response header is self-cleanup. Both refuse the fantasy that a robots line is a magic eraser. Both require the written rule to match the live URL shape. The politeness essay already took the neighbor-behavior hinge. This essay takes the header hinge.

Finite index language also helps. Results come from pages actually fetched and kept. Missing is ordinary. A shell that sends noindex should be missing from the public offer even if the URL still resolves for a person who types it. Status on Tracker answers whether named services are up. It does not answer whether a foreign index has finished dropping a noindexed shell. Corrections does not promise same-day disappearance from every cache. It promises a public list that gets longer when the company is wrong.

## Product truth still sits on the homepage

www.oernoe.com is the public publisher: journal, guides, How-To, company pages, and legal. Search lives at search.oernoe.com. Login lives at account.oernoe.com. The live Services list sends people to Search, Health, Chat, AI, Docs, Drive, and Tracker. Unreleased product routes are not sold as joinable homepage products. Corrections already had to unwind a coming-soon capture page that collected addresses for software that did not exist; that route is now a status notice, noindexed, without ads. Whether a stranger today sees a status paragraph or an SSO redirect, the ranking rule does not change. An unfinished route is not a public essay. robots.txt is not how you make it look finished. X-Robots-Tag is how you stop it from ranking as if it were.

About names the same corrections class in company voice: homepage copy that sounded like the whole network had no tracking while www applied for AdSense; a capture page for software that did not exist; generic SEO journal URLs; a robots.txt that missed live application routes. A company page that will not admit those is asking Google to find them. About would rather name them. This essay stays with the robots miss and the header fix so the mechanical lesson does not get lost inside the wider confession.

## How a stranger should check the fix

Open Corrections and read the robots paragraph end to end. Open https://www.oernoe.com/robots.txt and confirm Allow across the site for ordinary agents, Disallow only /api/ and /debug/, ads-bot Allow lines, host declaration, and the public URL listing pointer. Open What we publish’s mechanical checks. Open How Search works if the question is how Oernoe’s own crawler treats other people’s robots files. If you need to confirm a specific shell, fetch the live URL and look for `X-Robots-Tag: noindex, nofollow` on the response rather than assuming a Disallow line still hides it. If a journal line claims empty shells are “blocked by robots” while the live file only disallows /api/ and /debug/, the journal line loses against robots.txt and Corrections. Product claims lose against the homepage. Policy claims lose against Privacy, Cookies, and Terms.

Send a specific, checkable complaint to hello@oernoe.com with the page address and the sentence that is wrong if you find a new mismatch. Corrections does not promise paid corrections, a public comments thread, or a same-day rewrite on every message. It does promise that the list gets longer when the company is wrong rather than shorter when it is embarrassed.

## What this essay refuses to invent

It does not invent crawl volumes, foreign index counts, or a claim that every cached shell has already dropped worldwide. It does not invent a control-by-control tour of a console. It does not treat Status green as proof of header compliance on every historical URL. It does not treat Disallow as moral purity. It trusts the live Corrections page and the live robots file: trailing-slash Disallow missed the shells, X-Robots-Tag is the fix that requires fetchability, and confusing those tools is how a publisher stays indexed by accident.

X-Robots-Tag fixes what robots.txt missed. Search limits include the honesty to let a crawler see the header. Corrections includes the day the publisher wrote that down.