All about robots.txt file for website / Blog? 
What is Robots.txt?
The Robots.txt is a text type file, which is created by the webmasters to instruct search engine robots to crawl and index the web pages of their Blogs and Websites. Robots.txt is also known as the robots exclusion protocol (REP).
Where is robots.txt file on your blog?
A REP file is to be placed in the root of your Blog or Website that can be accessed by the Search Engine Robots. This file is included with the commands in the search engines to access your site by the section and also by specific kinds of web crawlers.
Why you need a robots.txt file?
Robots.txt (REP) file is only needed if your website or blog includes the content that you don’t want Typical search engines to crawl and index on their search results and search engines.
Limitation of the REP (Robots.txt)
The commands you included in the robots.txt file on your site does not change the behavior of the crawlers of the search engines. Instead, they act as an official instruction to the web crawlers to access your site.
Most of the respectable web crawlers follow the instruction in a robots.txt file, different web crawlers might understand the instructions differently so make sure you know the proper syntax for different web crawlers.
While most of the web crawlers won’t index the content you blocked by using the robots.txt, but still they can find and index it from other places on the web. Web crawlers can’t prevent the references to your blog or website URL from other sites to index on the search results.
Syntax and examples for creating a robots.txt file
A simple and clean robots.txt file uses words User-agent, Disallow and Allow. Here User-agents are typical search engine robots. Disallow is a command to tell the User-agents to not access a particular folder or specific page on the page URL. You can also use Allow to particular URL which is a child directory where parent directory is disallowed.
Blocking all search engine web crawlers from all content
User-agent: *
Disallow: /
Blocking a search engine robots to crawl a specific folder
User-agent: Googlebot
Disallow: /specific-folder/
Blocking a search engine robots to crawl a specific page
User-agent: Googlebot
Disallow: /specific-folder/blocked-page.html
Sitemap Consideration
User-agent: *
Disallow:
Sitemap: http://www.example.com/location/sitemap.xml
How to add custom robots.txt to a Blogger Blog?
Firstly create a custom robots.txt file for your blog or website. Make sure that the syntax and other words are correctly typed. Copy the text in the file and follow the steps below:
- Go to your blogger.com and login to your account using the correct username and password
- Click on the specific blog you want to add the robots.txt
- On the dashboard on the left side, click on the settings option and select search preferences
- On this page, select yes to Enable custom robots.txt content?
- Now paste the copied robots.txt file content
- Click on save changes button
- Now you have successfully added a robots.txt file to Blogger blog.
How to add Custom roobots.txt to a WordPress blog or website?
Adding a robots.txt to WordPress hosted blog or website is so simple. Below are the steps to add robots.txt file to hosted blog successfully. Go through the details…
- Go the control panel of the hosted blog or website
- Now go to the file manager of the Website
- Click on the root folder
- Now create a file and name it as robots.txt
- And copy the robots.txt file data to this file
- Click on the save button
- Now you have successfully added robots.txt to the WordPress hosted blog or website.
How to Test your websites Robots.txt?
- Go to the webmaster tools of the search engine you want to test the robots.txt
- Just go to the crawl section on the dashboard of the website webmaster tools.
- Select the robots.txt tester tool.
- You can find the live robots.txt file in the text file editor.
- You can also see the robots.txt of your blog or website just by typing the blog/website URL with /robots.txt at
- the end. For example www.yoursite.com/robots.txt.
So this is all about robots.txt file and how to add it to your Blogger and WordPress blogs and websites. This is the tutorial of the day for web aspirants and blog beginners. We have made this tutorial as simple and informative as possible. Even if you have any queries regarding robots.txt file and creating it please feel free to drop your query on the comments section, so that we will reply with details clearly. If you like our tutorial, please support us by sharing this tutorial on your social networks. Thank you guys for visiting our blog, please subscribe to our blog for regular updates.
All about robots.txt on Blogger and WordPress
No comments:
Post a Comment