Robots.txt WordPress Editor




The Robots.txt file is a simple text file that is placed in your site’s root directory. This file uses a set of instructions to tell search engine robots which pages on your website they can and cannot crawl.When activated, this plugin will create a robots.txt file and add the following rules:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
This will tell bots (search engines) NOT to crawl wp-admin, wp-includes wp-content/themes, and wp-content/plugins directories.From Settings > robots.txt page you can view content and edit it to suite your needs. For example:To block Bing search engine:
User-agent: Bingbot
Disallow: /
To stop bots from crawling wp/login.php
User-agent: *
Disallow: /wp-login.php
To stop bots from crawling you search results:
User-agent: *
Disallow: /?s=
Disallow: /search/