Disallow that hides noindex is a mistake

A robots.txt file can look careful and still leave the wrong pages in the index. That is what happened when our old Disallow lines used trailing-slash paths that did not match the live bare routes. The Corrections guide names the pattern plainly: paths written as /auth/, /console/, and /dashboard/ while the application served /auth, /console, and /dashboard. A crawler that followed the file to the letter could still fetch the bare URLs. Those URLs rendered as near-empty application shells. A reviewer who typed them saw a publisher site padded with empty pages. The mistake was not forgetting to block enough. The mistake was blocking in a way that stopped Google from reading the signal that should have kept those shells out of search results.

The fix we shipped is the opposite of adding more Disallow lines. Application shells now send X-Robots-Tag: noindex, nofollow. robots.txt only disallows /api/ and /debug/, so a crawler can fetch the URL and honour the header. A Disallow line that stops Google from reading noindex is how you stay indexed by accident. That sentence is not a slogan. It is the mechanical lesson of the correction.

## What the old file thought it was doing

Trailing-slash Disallows feel precise when you write them. You imagine a folder tree. You imagine that /auth/ covers every auth surface. Live HTTP routes are not folders. A bare /auth is a different request string from /auth/ if the crawler matches literally and the server does not force a redirect that collapses both forms before robots rules run. In our case the live routes were bare. The Disallow lines were slash-suffixed. The mismatch was enough for Google to fetch the shells.

Those shells were never meant to be journal documents. They were application surfaces: login-adjacent chrome, console scaffolding, dashboard scaffolding. Near-empty HTML is still a fetchable document if the status is 200 and nothing tells the crawler to stay out of the index. A publisher site that wants finished writing in Search results does not want empty shells sitting beside that writing. Reviewers notice empty shells. Readers who land on them from a weird query notice them too. The embarrassment is not theoretical.

People reach for Disallow when they are scared of crawlers. Blocking feels like control. It feels faster than wiring response headers on every shell route. It also fails in the exact way that matters for indexing: if the crawler never fetches the page, it never sees X-Robots-Tag, and it may keep an older discovery path alive longer than you want. Disallow is not noindex. Disallow is do not fetch. Noindex is you may fetch, then leave this out of the index. Confusing those two jobs is how empty shells survive in results after you thought you locked the door.

## Why noindex needs a fetch

X-Robots-Tag only works if the crawler retrieves the response. That is the whole point of putting the instruction on the HTTP response instead of in a separate file. robots.txt that forbids the fetch hides the header. Hidden instructions do not run. You then get the worst of both worlds: you believe the shells are blocked, and the crawler believes it already has a URL worth keeping because nothing authoritative told it otherwise after a successful fetch of the bare route.

The Corrections page is blunt about the direction of the fix. We did not double down on broader Disallow coverage for /auth, /console, and /dashboard. We opened those paths to fetch so the noindex,nofollow header could do its job. robots.txt shrank to the surfaces that should not be fetched as documents at all: /api/ and /debug/. Those are machine endpoints and debug surfaces, not publisher articles. Keeping crawlers out of API noise is a different problem from keeping application shells out of the search index.

nofollow on the same header matters for shells that may still link into deeper application routes. The goal is not to turn a login shell into a hub that passes crawl equity through the product UI. The goal is a clean fetch, a clear noindex, and no accidental invitation to wander the scaffolding. Journal essays and guides remain the documents we want fetched and eligible. Shells remain fetchable only so they can refuse ranking.

## Bare routes, reviewers, and padded sites

A reviewer from an ads network or a careful reader does not need a sophisticated crawl to find /auth, /console, or /dashboard. They type the address. If the page returns a near-empty shell with a 200, the site looks unfinished. If that shell is also indexable, Search can surface it next to real guides. That is how a publisher earns a reputation for thin URLs without meaning to publish thin writing.

Corrections exists because we are willing to put failed patterns in front of a reader instead of quietly pretending the first version never shipped. The robots mismatch belongs on that list for a reason. It was not a marketing typo. It was an operational rule that did not match live URL shapes. Product claims get checked against live hostnames. Policy claims get checked against Privacy, Cookies, and Terms. Crawl rules get checked against the actual path strings the server serves. When those disagree, the article or the robots file loses until we update it.

The same correction mindset applies to what we disallow going forward. We do not treat robots.txt as a dumping ground for every path we dislike. We treat it as a short list of fetch bans, and we treat response headers as the ranking instruction for shells that must stay reachable for people and for crawlers that need to see noindex. That split is boring. Boring is what keeps empty dashboards out of result pages.

## What this is not asking you to do

This essay is not a campaign for maximal openness. /api/ and /debug/ stay disallowed because fetching them is not useful for Search and can be noisy for operators. It is also not a claim that every application hostname should be crawlable end to end. Account surfaces, product apps, and private workspaces have their own rules. The publisher host is www.oernoe.com. The lesson here is about shells that resolve on the publisher host as bare routes, and about the robots file that used to miss them.

It is not a catalogue of every header name in every environment. Formats and edge cases change. The durable rule is the one Corrections already published: shells send X-Robots-Tag noindex,nofollow; robots.txt only disallows /api/ and /debug/ so crawlers can fetch and honour the header; a Disallow that hides noindex is how you stay indexed by accident.

Nor is this a rehash of neighboring journal jobs. Other essays cover why shells should not rank, how rewrite paths differ from noindex paths, why www is a publisher rather than an app shell, why error pages and login screens stay unmonetized, and why old copy on Corrections still matters after the wording changes. Those pieces have their own edges. This one stays on the robots mismatch and the fetch-then-noindex repair.

## How we check the rule without inventing theater

Look at robots.txt and ask what it forbids. It should be short. It should name /api/ and /debug/. It should not pretend that a trailing slash covers a bare shell route. Then fetch /auth, /console, and /dashboard the way a crawler would: bare path, no slash games. Expect X-Robots-Tag: noindex, nofollow on the response. Expect no attempt to dress those shells as journal documents. Expect them out of the public listing of finished writing. If any of those checks fail, the correction is incomplete.

Also check that finished guides and journal essays still allow Google’s ordinary crawlers. A publisher that hides its own documents while obsessing over shells has the priorities upside down. What we publish says www is for original writing, help pages, and the legal and company record. Crawl policy should serve that job. Shells are an exception handled by headers, not a reason to choke the whole host.

If you find a bare shell that still lacks the header, write hello@oernoe.com with the URL. Specific, checkable complaints beat topic-level panic. We do not promise a same-day rewrite on every message. We do promise that a path string you can open in a browser is more useful than a vague claim that robots is wrong.

## Why accidental indexing is worse than an empty Disallow

Empty Disallow lines that miss their targets create false confidence. Teams stop looking at Search discoveries for those URLs because they believe the file already handled it. Meanwhile the bare route returns 200, the shell looks thin, and nothing authoritative says noindex because the crawler never got the header on a path the Disallow actually matched. Accidental indexing is quieter than a public outage and longer lasting than a typo on a homepage.

The repair costs a fetch. That is the trade. We accept that Google may retrieve /auth, /console, and /dashboard. We insist that the retrieval carry noindex,nofollow. We keep /api/ and /debug/ out of the fetch path entirely. That is a narrower, meaner robots file and a stricter header habit. It is also the only combination that matches how indexing actually works.

Publisher ads make the stakes higher, not lower. Selected finished pages on www may load Google AdSense. Empty shells should not sit next to those pages as if they were documents. Hostnames and What we publish both draw a line between finished writing and application chrome. Crawl rules that leave chrome indexable cut across that line. The Corrections entry exists so we do not have to rediscover the cut every quarter.

## Closing the loop with Corrections

Corrections is a public list, not a press release. It grows when we are wrong. The robots.txt mismatch is on it because the old file did not match live URLs, crawlers could fetch near-empty shells, and the durable fix is header-based noindex with a short Disallow list. Readers who want the mechanical claim can open the guide. Readers who want the editorial claim can keep this essay: Disallow that hides noindex is a mistake, and we already paid for learning that the hard way on /auth, /console, and /dashboard.

We will keep writing the same rule until the temptation to just block it in robots stops showing up in drafts. Blocking feels decisive. Fetching with noindex is the decision that actually removes shells from the index. If those two sentences ever disagree on the live site again, the live site is wrong, and Corrections gets another line.