h3_html = ‘
cta = ‘
atext = ‘
scdetails = scheader.getElementsByClassName( ‘scdetails’ );
sappendHtml( scdetails, h3_html );
sappendHtml( scdetails, atext );
sappendHtml( scdetails, cta );
sappendHtml( scheader, “http://www.searchenginejournal.com/” );
sc_logo = scheader.getElementsByClassName( ‘sc-logo’ );
logo_html = ‘‘;
sappendHtml( sc_logo, logo_html );
sappendHtml( scheader, ‘
} // endif cat_head_params.sponsor_logo
WordPress introduced an vital change to how it will block engines like google from indexing web sites. This change abandons the normal Robots.txt answer in favor of the Robots Meta Tag method. This change is in keeping with the frequent intent for blocking Google. which is to maintain the blocked pages from exhibiting in Google’s search outcomes.
This is the Robots Meta Tag that WordPress will use:
<meta title=’robots’ content material=’noindex,nofollow’ />
Blocking Google From Indexing
It has lengthy been a regular apply to make use of Robots.txt to dam the “indexing” of an internet site.
The phrase “indexing” meant crawling of the positioning by GoogleBot. By utilizing the Robots.txt blocking characteristic you possibly can cease Google from downloading the required net web page and, it was assumed, Google can be unable to point out your pages within the Search Results.
But that robots.txt directive solely stopped Google from crawling the web page. Google was nonetheless free so as to add it to its index if it was in a position to uncover the URL.
So to dam a web site from showing within the index, a writer would block Google from “indexing” the pages. Which wasn’t constantly efficient.
WordPress 5.3 Will Truly Prevent Indexing
WordPress tailored the Robots.txt method. But that’s altering in model 5.3.
When a writer at the moment selects “discourage engines like google from indexing this web site” what that does is add an entry to the positioning’s robots.txt that prohibits Google from crawling a web site.
Starting with WordPress 5.3, WordPress will undertake the extra dependable Robots Meta Tag method for stopping the indexing of an internet site.
This change will have an effect on the “discourage search engines from indexing this site” setting.
This change is an enchancment. WordPress publishers will be safer in understanding that the blocked net pages won’t be proven in Google’s search outcomes.
Why Did WordPress Use Robots.txt?
WordPress relied on Robots.txt for blocking the indexing of an internet site as a result of that’s how all people stored pages from exhibiting in Google’s search outcomes. That was the usual manner of doing it.
Yet despite the fact that all people did it that manner, as has been defined, it was an unreliable method.
The phrase “indexing” has two meanings:
- Indexing means crawling, as when Googlebot visits and downloads net pages.
- Indexing may imply including an internet web page to Google’s database of net pages (which is named The Index).
Blocking Google from “indexing” an internet web page will preserve it from seeing the online web page however Google may nonetheless index the online web page and add it to Google’s index. Make sense?
Robots.txt Versus Robots Meta Tag
Keeping an internet web page out of Google’s index was not the intent of the Robots.txt answer. Doing that’s the job of the Robots Meta Tag.
So it’s good to see WordPress embrace the Robots Meta Tag as the answer to blocking net pages from exhibiting in the major search engines.
WordPress 5.3 is scheduled to be launched in November 2019.
Read the WordPress announcement:
Changes to Prevent Search Engines Indexing Sites
Read Google’s Authoritative Documentation
- Robots meta tag and X-Robots-Tag HTTP header specs
- Block search indexing with ‘noindex’