Robots.txt file is so much important to control what you want to share and what not on SE. So its very important to right the correct codes in to your robots.txt file or else you can down your blog SEO. So here we have a basic codes for newbies that will control his bold security and SEO too like pro.
Table of Contents
What Is Robots.txt?
Robots.txt is common name of a text file that is uploaded to a Web site’s root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
Where Is WordPress Robots.txt File?
Every site should have robots.txt that will help to secure and share his site. The robots.txt file URL should be same for every sites/blogs like at http://www.example.com/robots.txt
You know that the basic codes of robots.txt but why I shared the article here. As after new update, Google want to allow CSS and JS also from your site to fetch by Google as now Google are also seeing theme style to put the site in there result. We wrote a brief article on it at our previous posts under How To Unblock Your CSS And JavaScript From Google Fetching? that you can read there but in this robots.txt, we added all of these codes also.
How To Add Custom Robots.txt Tag In WordPress?
Step 1:) First of all, open http://www.example.com/robots.txt and see that your blog contain a Robots.txt or not.
Step 2:) If yes then go to your FTP and open that folder where you have WordPress core files.
Step 3:) Now see there that is there any file named Robots.txt
Step 4:) If you will find it then edit it and if not find it then create a file with this name there.
Note: Some time, there will be no files with name Robots.txt but there is a file opening at http://www.example.com/robots.txt so it means that you have a SEO plugin that is creating virtual Robots.txt. Dont worry, still follow all the steps.
Step 5:) After creating the file, add the below code there.
User-agent: * Allow: / Allow: /wp-content/uploads/ Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- User-agent: Googlebot # Allow all files ending with these extensions Allow: /*.js$ Allow: /*.css$ Allow: /*.png$ Allow: /*.gif$ Allow: /*.jpg$ Allow: /*.woff$ Sitemap: http://www.example.com/sitemap.xml
Step 6:) Now save that file and you are done.
Customization:
You just have to change http://www.example.com/sitemap.xml with your sitemap URL. Add this line again and again if you have many sitemaps.
How To Check Your Robots.txt File?
You can check this file on your blog by adding /robots.txt at last to your blog URL in the browser like http://www.example.com/robots.txt. Once you visit the robots.txt file URL you will see the entire code which you are using in your custom robots.txt file so its confirmed that you did right.
Last Words:
Here we share complete basic knowledge on custom robots.txt file in WordPress. We tried to make this tutorial effective and simple. But still if you have and doubt or query then feel free to ask me. Think twice before edit it because a single mistake can lead your blog out of SE. Like it than share and comment here…
Be the first to write a comment.