This is the single most common cause of “my whole WordPress site vanished from Google” that I see, and it is one checkbox in Settings, Reading.
It is worth understanding rather than just unticking, because what it does changed years ago and most of the advice written about it is describing the old behaviour.
What the checkbox does now
WordPress documents it plainly in the Reading settings screen: checking “Discourage search engines from indexing this site” asks search engines not to index the site. Then the specifics, which are the interesting part.
Since WordPress 5.3, it causes <meta name='robots' content='noindex,nofollow' /> to be generated into the <head> of your site’s source, provided your theme actually calls wp_head.
Up to and including WordPress 5.2, it did something different: it made requests to robots.txt return User-agent: * and Disallow: /. WordPress notes that this only worked if WordPress was installed in the site root and no real robots.txt file existed, which on a standard install is the normal state: WordPress has no robots.txt file on disk and builds the response per request.
It also does two things nobody mentions. It stops pings to ping-o-matic and any other RPC ping service listed under Update Services, by way of a filter named privacy_ping_filter() that empties the ping list. And it hides the Update Services option in Settings, Writing, entirely, which is why that setting sometimes appears to have gone missing.
Behaviour as documented on the WordPress Reading settings page and in the 5.3 core change note.
Why the change matters
The two mechanisms fail in opposite directions, so the fix is not the same fix.
The old robots.txt behaviour blocked crawling. Blocking a crawl does not remove anything from the index, it just stops Google from seeing the page again, which is the classic way to leave URLs stranded in the results with no snippet. That difference is the whole of noindex versus robots.txt, and it is why a disallow was always a poor tool for the job.
The current behaviour is the correct one. noindex on a crawlable page is the instruction that actually gets a page dropped, because Google has to be able to fetch the page to read the tag telling it to go away.
It also carries a second instruction most people never notice. It is noindex,nofollow, not noindex. Every link on every page of the site is flagged nofollow for as long as that box is ticked. On a staging copy that is irrelevant. On a live site that has been sitting like this for months, it means your internal linking has been switched off as well as your indexing.
How to check whether it is on
Do not trust the admin screen alone, because plugins can override it in both directions. Check the output.
Look at the HTML of a live page and search for a robots meta tag with noindex in it. If you have Search Console access, the faster route is URL Inspection on any URL, which will report the page as excluded by a noindex tag if the tag is being served. Search Console will also list affected pages under the corresponding reason in the page indexing report, and every Search Console indexing status covers how that one reads next to the others.
One warning on the check. The tag only appears if your theme calls wp_head. A theme that does not, or a heavily customised header, can leave the checkbox ticked while nothing is emitted, which means the setting looks active and is doing nothing at all.
How to undo it properly
Untick the box and save. That part takes five seconds. The rest is the part that gets skipped.
Confirm the tag is gone from the live HTML, not just from the settings screen. Caching is the usual culprit here, at the plugin layer or the CDN layer, and a cached copy will keep serving noindex to Googlebot long after you have changed the setting.
Then check whether an SEO plugin is also setting a robots tag, because most of them can, and if one is set to noindex at the plugin level the WordPress checkbox is not your problem.
After that, wait. Google has to recrawl each page to see that the tag is gone, and recrawling is not instant or uniform. Use URL Inspection and request indexing for the important pages rather than for everything, and expect the recovery to arrive gradually rather than all at once. If pages then sit in a fetched-but-not-indexed state, that is a separate problem and it reads like crawled, currently not indexed rather than like a noindex.
The honest caveat
I have described what WordPress core does. I have not tested every theme and plugin combination, and plenty of them add their own robots handling on top. If the tag is still being served with the box unticked, the cause is somewhere else in the stack, and the way to find it is to read the HTML that is actually served rather than to keep changing settings.
Sources
- WordPress documentation, Settings Reading screen, for the checkbox behaviour, the 5.2 robots.txt behaviour, the ping filter and the hidden Update Services option.
- WordPress core development blog, Changes to prevent search engines indexing sites, for the 5.3 change to a meta robots tag.
- Google Search Central, Block search indexing with noindex, for why the page has to remain crawlable for the tag to be read.