How to Easily Edit Robots.txt File With Yoast SEO – Beginners Guide

edit robots.txt

To edit Robots.txt file is an important factor in making a post SEO friendly. There’s a built-in ROBOTS.txt on every WordPress blog. But you have to edit robots.txt make your blog SEO friendly. So there’s a significant importance in editing and optimizing the ROBOTS.txt file of your WordPress.

Now the question is where can you find it. To have access to edit a ROBOTS.txt file you must have the Yoast SEO plugin. You can install this plugin from your WordPress dashboard. If you are facing trouble in installing this or other plugins check this post How to Install WordPress Plugins

After you’re done installing and activating Yoast SEO, there are some easy steps to edit ROBOTS.txt file. I’ll show you how you can edit ROBOTS.txt file and make your blog SEO friendly. Before that let’s start with basics.

What is ROBOTS.txt File?

Robots.txt file is a text file which instructs search engine bots how to crawl and index a site. Whenever a search bot comes to your website to crawl, it finds your robots.txt file and follows the given directions by it. Using this file you can specify the bots to crawl to which part of your blog and which part to avoid. If robots.txt doesn’t exist for your blog, it will stop bots to crawl and index your website to google search.

Understand What The Tags Mean

As I said earlier for WordPress blogs, the best plugin is Yoast SEO for this job. So I have given the steps to edit & optimize robots.txt file with this plugin.You can check your robots.txt by going to http://yourdomain.com/robots.txt.  For example, you can check our robots.txt file from here: http://www.bloggertracks.com/robots.txt

There are some specified tags to look for

  • User-agent:
    Defines the name for a search engine bot.* sign means its allowing all the bots to crawl
  • Disallow: /wp-admin/

Instructs search engines not to crawl and index some parts of your site. /wp-admin/ states not to crawl to this part

  • Allow: /

Instructs search engines to crawl and index which parts you want to index. / states to crawl and index whole website

Now time to edit and optimize yours. with the easiest of steps.

The WordPress plugin you need to do this job is Yoast SEO Plugin.

How to Easily Edit ROBOTS.txt File

edit robots.txt

Go to WordPress dashboard and click on tools from YOAST SEO as shown above. You will see a new menu.

edit robots.txt

Then click on file editor. You will find the option of creating a ROBOTS.txt file. As I said earlier there is a default/built-in ROBOTS.txt for every WordPress website. But to make it SEO Friendly you have to include something more to that.

when you click on create a robots.txt a default Robots.txt will be created and it’ll be something like this :
edit robots.txt

Basically, this robots.txt will still allow google bots and other search bots to crawl your website and index your posts to search. But if you want your site optimized, you have to edit robots.txt a bit. It’ll help search bots for more efficient in crawling and indexing your website. Here’s what you gotta do:

You have to edit robots.txt or replace it with these new contents below

edit robots.txt

The sitemap tags have to be filled up with your website’s sitemaps. Those are my websites sitemap. To see your sitemaps just go to Yoast SEO → XML Sitemaps and you will find your sitemaps their

edit robots.txt

After editing the robots.txt just click on save changes to robots.txt and your changes would be saved.

Now The Final Step

GO TO your search console account and find a robots.txt tester on the dashboard. Click it and a new page will appear.

edit robots.txt

just fill up the robots.txt tester with the edited robots.txt from your website. And click on submit.

edit robots.txt

After clicking on submit you’ll see a pop-up box like this one.

edit robots.txt

Click on submit and Google will update the robots.txt for you. Usually, it takes a day or two till your robots.txt is updated

Once it’s updated you’re good to go.Search engines will able to index what you post. and you will be fascinated seeing that your posts are doing great. To check your sites progress you can add your sitemaps to google search console. You can read How To Add Sitemaps To Google Search Console – Easy Steps

I hope this post will help you with  robots.txt issues. If you have any confusion feel free to ask in comments or send me us an email. We’ll be always here to help.

Nishat Mahmud
 

Founder of BloggerTracks.com I am a full-time blogger. I am always up for helping newcomers to choose the right stuff for their blogs and to set up perfectly. I wish to lead people on the track of Blogging World.

  • Vijee Djega says:

    A really useful post. Am on WordPress, but haven’t done everything you have listed here. Great tips. Am bookmarking this for future reference.

    • Nishat Mahmud says:

      Thanks for reading mate. 🙂

  • I don’t use WordPress but looking to using it in the future but will reread this in the future

    • Nishat Mahmud says:

      Thanks for reading mate. 🙂

  • Blackellis says:

    I use wordpress and I know how important this thing is. Very good post

    • Nishat Mahmud says:

      Thanks mate. 🙂

  • Neha kumari says:

    Looks a really good post. Definitely going to check again when i use wordpress.

    • Nishat Mahmud says:

      Thanks mate. 🙂

  • Very informative article especially for those who are newbie and don’t want to mess thing up with direct programming but using a plugin like Yoast. Thank you for sharing priceless information with us.

    • Nishat Mahmud says:

      It’s my pleasure brother. 🙂

  • Kirston says:

    Thanks Nishat… if you are interested please try our free Virtual robots.txt plugin too

    well anyway thanks for the maybe. cheers mate

    • Nishat Mahmud says:

      My pleasure mate. 🙂

  • Tillu Ash says:

    Good Day,
    I studied your recent post, really enjoy lot and I hop u will deliver more post such like, I appreciate your ideas, keywords and approach to write above post, meanwhile it reduces my boring and divert my attraction more and more, so I please request you that write more articles in way to attract me and publically good. For more audience I will approach it and market to my community,

    Thanks to given me time.

  • cityceilings says:

    Hi very nice article keep it up the good work Wonderful!! Thank you for posting informative blog. Your posts are more interesting and informative

  • Larry says:

    Thanks Bro, you are a life saver. I have checked the web on how to do this until I found you post. Thanks again it was very help. I have done it for my site

  • >