To edit ROBOTS.txt is an important factor in making a post SEO friendly. There’s a built in ROBOTS.txt in every WordPress blog. But you have to edit robots.txt make your blog SEO friendly. So there’s a significant importance in editing and optimizing the ROBOTS.txt file of your WordPress.
Now the question is where can you find it. To have access to edit ROBOTS.txt file you must have the YOAST SEO plugin. You can install this plugin from your WordPress dashboard. If you are facing trouble in installing this or other plugins check this post How to Install WordPress Plugins.
After you’re done installing and activating yoast seo , there are some easy steps to edit ROBOTS.txt file. I’ll show you how you can edit ROBOTS.txt file and make your blog seo friendly. Before that let’s start with basics.
Robots.txt file is a text file which instructs search engine bots how to crawl and index a site. Whenever a search bot comes to your website to crawl, it find your robots.txt file and follows the given directions by it. Using this file you can specify the bots to crawl to which part of your blog and which part to avoid . If robots.txt doesn’t exist for your blog, it will stop bots to crawl and index your website to google search.
As i said earlier for WordPress blogs, the best plugin is yoast seo for this job. So I have given the steps to edit & optimize robots.txt file with this plugin.You can check your robots.txt by going to http://yourdomain.com/robots.txt. For example, you can check our robots.txt file from here: http://www.bloggertracks.com/robots.txt
There are some specified tags to look for
Defines the name for a search engine bot.* sign means its allowing all the bots to crawl
Instructs search engines not to crawl and index some parts of your site. /wp-admin/ states not to crawl to this part
Instructs search engines to crawl and index which parts you want to index. / states to crawl and index whole website
Now time to edit and optimize yours. with the easiest of steps.
The WordPress plugin you need to do this job is YOAST SEO
Go to WordPress dashboard and click on tools from YOAST SEO as shown above. You will see a new menu.
Then click on file editor. You will find the option of creating a ROBOTS.txt file. As i said earlier there is a default/built in ROBOTS.txt for every WordPress website. But to make it seo Friendly you have to include something more to that.
when you click on create a robots.txt a default Robots.txt will be created and it’ll be something like this :
Basically this robots.txt will still allow google bots and other search bots to crawl your website and index your posts to search. But if you want your site optimized, you have to edit robots.txt a bit. It’ll help search bots to more efficient in crawling and indexing your website. Here’s what you gotta do:
You have to edit robots.txt or replace it with these new contents below
The sitemap tags have to be filled up with your website’s sitemaps. Those are my websites sitemap. To see your sitemaps just go to yoast seo>> XML Sitemaps and you will find your sitemaps their
After editing the robots.txt just click on save changes to robots.txt and your changes would be saved.
GO TO your search console account and find robots.txt tester in the dashboard. Click it and a new page will appear.
just fill up the robots.txt tester with the edited robots.txt from your website. And click on submit.
After clicking on submit you’ll see a pop up box like this one.
Click on submit and google will update the robots.txt for you. Usually it takes a day or two till your robots.txt is updated
once it’s updated you’re good to go.Search engines will able to index what you post. and you will be fascinated seeing that your posts are doing great. To check your sites progress you can add your sitemaps to google search console. You can read How To Add Sitemaps To Google Search Console – Easy Steps
You’re ready to write SEO friendly posts now as you have changed your robots.txt. If you wanna get the total idea about writing a seo friendly post read How To Write a SEO Friendly Blog Post With YOAST SEO Plugin.
I hope this post will help you with robots.txt issues. If you have any confusion feel free to ask in comments or send me a mail . I’ll be always here to help.