Hey, hope you are fine in this Quarantine. If you have a website and the website is hosted on blogger or WordPress, you got the Robot txt error on the website through the mail. If you got his kind of mail, you need to fix it. First, you can cause critical SEO that can also negatively impact your ranking and traffic.
As you can see on the ScreenShot where you can see Fix Coverage issues, this section of Google Console finds some errors Robot txt, crawler issue,404 error, and 505 error today; we will be discussing the robots txt blogger error. In this post, you will learn what a robot txt file is, the importance of a robot txt file, why robot txt shows, and why you need it.
If you want to fix this, read this article till the end, and I will tell you How to fix the robot txt file in blogger and WordPress step by step with the use of screenshots, so Let’s start.
What is Robot.txt
Robot.txt is a file in the form of words and symbols. That resides in the root directory of your website and gives search engine crawlers instructions as to which page they are indexing; crawl while during the indexing and crawlings.
When it’s visiting your website, the first thing they do is check out the website content. Also, check your labels of websites and check and read the URL for indexing the particular websites.
Importance of blogger robots txt file.
Robot.txt file is important because it helps your website with indexing, and crawling and also helps to SEO and ranking of the article.Robot.txt helps your website find your page publicly, and a particular website can be crawled and added to its index.
Suppose you have a big website where crawling and indexing can be very resource-intensive. Crawlers from various search engines will try to crawl and index the whole website, which can be serious crate problems.
Robots Txt File In Blogger
Why does robot.txt show in Google Console
It shows the Robots.txt file in Google console coverage when we do not update the robot’s text file on hosting site websites like Blogger and WordPress.In the case of a blogger, we used labels while posting the article while we added labels on the post. If we added the same labels on the single post, you get this type of problem.
We used the tags and category in the case of word press; if we used the same tags, you could get this type of problem.When we use the same labels, tags, and category on a single article, when the crawler is indexing them and get the three labels to have a single article, you get the duplicate content error. So it’s very important to fix it because it will impact your SEO.
Two Important things you should know about Robot.txt
- The first thing is that adding rules to the robot.txt is the only directive. It means that SEO Search Engine obeys and follows the rules according to a robot txt file.
- The second thing is if you added the if you block any page on the robot.txt, that would be generated and removed or not appear on the web.
How robots.txt works
Robot.txt file is the form of a simple keywords structure. In which some predefined keywords and value combinations you can use
Allow, Disallow, Crawler Delay, Sitemap, and User-agent.
Allow:- You can use the allow on your robot.txt file it gives access to a specific folder on our website, and even the parent directory is disallowed.
For example, If you disallow access to the photos directory but allow you to be located, your blog subfolders are located in subfolders.
User-agent: *
Disallow: /search
Disallow: /category/
Disallow: /tag/
Allow: /
User-Agent:* It helps the specific crawler take the directive’s specific account. You can use * it specify the crawler name, as you can see below.
User-Agent:* this allows crawlers.
User-Agent: this is for the Google bot Only.
Disallow:- it directive that instructs the user agent, not any URL
Sitemap:- It’s the direct major search engine, including Google. It’s very helpful to specify the find the location of the XML file.
The search will be able to find the cause of the XML sitemap. It will be to the bottom of the robot.txt file.
SITEMAP:- https://emample.com/sitemap.xml
If want to add site for blogger so you click on below link:-
How to Fix Blogger Robots txt
Step:-1 First of login to your blogger, which has the Robot.Txt error, and clicks on your blogger’s Setting.
As you can see on the screenshot and follow me by step by step
Setting ＞ Crawlers and indexing
Step:-2 Now you can see the bottom of Crawlers and indexing has Custom robot.txt now enable the 1 Enable custom robots.txt and now Go to google and Denrate the custom eobots.txt file. Click ≻ Blogger site map
Now generating code is a copy on the arrow 2 section and save the section.
Stwp:-3 Now enable the Custom robots header tag and at active all the buttons as according to screen show as you see on the picture.
In Home page tags
all enable
no dp enable
In Archive and search page tags
Noindex enable
nodp enable
Inpost and tags
all enable
nodp enable
Now click on the save and close the blogger.
Step:- 4 Now open the Google console on your browser and click on the coverage on the google console.
See the screenshot I marked by the arrow; your screen also shows like that index through blocked by the robots.txt, so click on this error.
After clicking on this screenshot, your screen will be shown like seen on the bottom screenshot. Now click on start validation, and your error will be remove after some days from the coverage. It will be fix within 15 days after the fix; you will be getting Gmail from the Google Console.
Robots Txt File In WordPress
In WordPress, we need to install one plugin, YOST SEO installs the Yoast SEO.After installing it, open the Yoast SEO, and then your laptop screen will be sho like as below screenshot. ↡↡↡
Now you can see three options are shown on the screen Category, Tag, and format; there are some changes in the three options as you see in the pictures.
In category ➡️ NO
In Tag ➡️ NO
In Format ➡️ Disable
After active, these options save the sitting, and these fills are fixed after some days.
Also Read:-
How to add a responsive table for blogger
How to add Table of Contents in Blogger/Blogspot?
How to Add Reaction Bar Buttons in Blogger and WordPress
How to Add Contact Form in blogger
Free Privacy Policy For Website
Note: If you don’t save all these sitting, you must save all these settings. If you don’t do this can impact your SEO.so read carefully this Robots Txt File In Blogger and WordPress.
Thanks for the reading; please drop the comments if not understand anything; we’ll help within 8hr and share the comment in the comment section about Robots Txt File In Blogger and WordPress.
