Hello, A client of mine is going through a bit of a crisis. A developer (at their end) added Disallow: / to the robots.txt file.
I currently have this under my robot txt file: User-agent: * Disallow: /authenticated/ Disallow: /css/ Disallow: /images/ Disallow: /js/
This is a custom result inserted after the second result.
Hello Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt.
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site ...
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.
After a site audit, I find out that my website don't have a robot.txt. Does it hurt my website rankings? One more thing, when I type ...
Moz are reporting the robots.txt file is blocking them from crawling one of our websites. But as far as we can see this file is exactly the ...
Hi guys, When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account ...
When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and ...