Google’s John Mueller responded to a query on LinkedIn to debate the usage of an unsupported noindex directive on the robots.txt of his personal private web site. He defined the professionals and cons of search engine assist for the directive and provided insights into Google’s inside discussions about supporting it.
John Mueller’s Robots.txt
Mueller’s robots.txt has been a subject of dialog for the previous week due to the overall weirdness of the odd and non-standard directives he used inside it.
It was virtually inevitable that Mueller’s robots.txt was scrutinized and went viral within the search advertising neighborhood.
Noindex Directive
Every thing that’s in a robots.txt is named a directive. A directive is a request to an online crawler that it’s obligated to obey (if it obeys robots.txt directives).
There are requirements for write a robots.txt directive and something that doesn’t conform to these these requirements is prone to be ignored. A non-standard directive in Mueller’s robots.txt caught the attention of somebody who determined to put up a query about it to John Mueller through LinkedIn, to know if Google supported the non-standard directive.
It’s an excellent query as a result of it’s straightforward to imagine that if a Googler is utilizing it then perhaps Google helps it.
The non-standard directive was noindex. Noindex is part of the meta robots commonplace however not the robots.txt commonplace. Mueller had not only one occasion of the noindex directive, he had 5,506 noindex directives.
The search engine marketing specialist who requested the query, Mahek Giri, wrote:
“In John Mueller’s robots.txt file,
there’s an uncommon command:
“noindex:”
This command isn’t a part of the usual robots.txt format,
So do you suppose it can have any affect on how search engine indexes his pages?
John Mueller curious to learn about noindex: in robots.txt”
Why Noindex Directive In Robots.txt Is Unsupported By Google
Google’s John Mueller answered that it was unsupported.
Mueller answered:
“That is an unsupported directive, it doesn’t do something.”
Mueller then went on to clarify that Google had at one time thought-about supporting the noindex directive from throughout the robots.txt as a result of it might present a method for publishers to dam Google from each crawling and indexing content material on the identical time.
Proper now it’s attainable to dam crawling in robots.txt or to dam indexing with the meta robots noindex directive. However you may’t block indexing with the meta robots directive and block crawling within the robots.txt on the identical time as a result of a block on the crawl will forestall the crawler from “seeing” the meta robots directive.
Mueller defined why Google determined to not transfer forward with the concept of honoring the noindex directive throughout the robots.txt.
He wrote:
“There have been many discussions about whether or not it needs to be supported as a part of the robots.txt commonplace. The thought behind it was that it might be good to dam each crawling and indexing on the identical time. With robots.txt, you may block crawling, or you may block indexing (with a robots meta tag, when you permit crawling). The concept was that you might have a “noindex” in robots.txt too, and block each.
Sadly, as a result of many individuals copy & paste robots.txt information with out taking a look at them intimately (few individuals look so far as you probably did!), it might be very, very straightforward for somebody to take away vital components of an internet site unintentionally. And so, it was determined that this shouldn’t be a supported director, or part of the robots.txt commonplace… most likely over 10 years in the past at this level.”
Why Was That Noindex In Mueller’s Robots.txt
Mueller made clear that it’s unlikely that Google would assist that tag and that this was confirmed about ten years in the past. The revelation about these inside discussions is attention-grabbing however it’s additionally deepens the sense of weirdness about Mueller’s robots.txt.
See additionally: 8 Widespread Robots.txt Points And How To Repair Them
Featured Picture by Shutterstock/Kues