How to Easily Edit Robots.txt file For SEO with YOAST SEO

edit robots.txt

To edit ROBOTS.txt is an important factor in making a post SEO friendly. There’s a built in ROBOTS.txt in every WordPress blog. But you have to edit robots.txt make your blog SEO friendly. So there’s a significant importance in editing and optimizing the ROBOTS.txt file of your WordPress.

Now the question is where can you find it. To have access to edit a ROBOTS.txt file you must have the YOAST SEO plugin. You can install this plugin from your WordPress dashboard. If you are facing trouble in installing this or other plugins check this post How to Install WordPress Plugins

After you’re done installing and activating Yoast SEO, there are some easy steps to edit ROBOTS.txt file. I’ll show you how you can edit ROBOTS.txt file and make your blog SEO friendly. Before that let’s start with basics.

What is ROBOTS.txt File?

Robots.txt file is a text file which instructs search engine bots how to crawl and index a site. Whenever a search bot comes to your website to crawl, it finds your robots.txt file and follows the given directions by it. Using this file you can specify the bots to crawl to which part of your blog and which part to avoid. If robots.txt doesn’t exist for your blog, it will stop bots to crawl and index your website to google search.

Understand What The Tags Mean

As I said earlier for WordPress blogs, the best plugin is Yoast SEO for this job. So I have given the steps to edit & optimise robots.txt file with this plugin.You can check your robots.txt by going to http://yourdomain.com/robots.txt.  For example, you can check our robots.txt file from here: http://www.bloggertracks.com/robots.txt

There are some specified tags to look for

  • User-agent: 
    Defines the name for a search engine bot.* sign means its allowing all the bots to crawl
  • Disallow: /wp-admin/

Instructs search engines not to crawl and index some parts of your site. /wp-admin/ states not to crawl to this part

  • Allow: /

Instructs search engines to crawl and index which parts you want to index. / states to crawl and index whole website

Now time to edit and optimize yours. with the easiest of steps.

The WordPress plugin you need to do this job is YOAST SEO

Get Premium Just For $69

Edit and Optimize Your ROBOTS.txt File

edit robots.txt

Go to WordPress dashboard and click on tools from YOAST SEO as shown above. You will see a new menu.

edit robots.txt

Then click on file editor. You will find the option of creating a ROBOTS.txt file. As I said earlier there is a default/built in ROBOTS.txt for every WordPress website. But to make it SEO Friendly you have to include something more to that.

when you click on create a robots.txt a default Robots.txt will be created and it’ll be something like this :
edit robots.txt

Basically this robots.txt will still allow google bots and other search bots to crawl your website and index your posts to search. But if you want your site optimized, you have to edit robots.txt a bit. It’ll help search bots to more efficient in crawling and indexing your website. Here’s what you gotta do:

You have to edit robots.txt or replace it with these new contents below

edit robots.txt

The sitemap tags have to be filled up with your website’s sitemaps. Those are my websites sitemap. To see your sitemaps just go to Yoast SEO>> XML Sitemaps and you will find your sitemaps their

edit robots.txt

After editing the robots.txt just click on save changes to robots.txt and your changes would be saved.

Now The Final Step

GO TO your search console account and find a robots.txt tester in the dashboard. Click it and a new page will appear.

edit robots.txt

just fill up the robots.txt tester with the edited robots.txt from your website. And click on submit.

edit robots.txt

After clicking on submit you’ll see a pop up box like this one.

edit robots.txt

Click on submit and google will update the robots.txt for you. Usually it takes a day or two till your robots.txt is updated

once it’s updated you’re good to go.Search engines will able to index what you post. and you will be fascinated seeing that your posts are doing great. To check your sites progress you can add your sitemaps to google search console. You can read How To Add Sitemaps To Google Search Console – Easy Steps

You’re ready to write SEO friendly posts now as you have changed your robots.txt. If you wanna get the total idea about writing a seo friendly post read How To Write a SEO Friendly Blog Post With YOAST SEO Plugin

I hope this post will help you with  robots.txt issues. If you have any confusion feel free to ask in comments or send me a mail . I’ll be always here to help.

Nishat Mahmud
 

Founder of BloggerTracks.com
I am a full-time blogger. I am always up for helping newcomers to choose the right stuff for their blogs and to set up perfectly.
I wish to lead people on the track of Blogging World.

Click Here to Leave a Comment Below 12 comments
Vijee Djega - January 24, 2017

A really useful post. Am on WordPress, but haven’t done everything you have listed here. Great tips. Am bookmarking this for future reference.

Reply
Michaela Harrison - January 24, 2017

I don’t use WordPress but looking to using it in the future but will reread this in the future

Reply
Blackellis - January 24, 2017

I use wordpress and I know how important this thing is. Very good post

Reply
Neha kumari - January 25, 2017

Looks a really good post. Definitely going to check again when i use wordpress.

Reply
Mizanur Rahaman Mizan - April 14, 2017

Very informative article especially for those who are newbie and don’t want to mess thing up with direct programming but using a plugin like Yoast. Thank you for sharing priceless information with us.

Reply
Kirston - June 26, 2017

Thanks Nishat… if you are interested please try our free Virtual robots.txt plugin too

well anyway thanks for the maybe. cheers mate

Reply

Leave a Reply: