Skip to content

Backgrounds

How-to Guides

Technical References

Modify the robots.txt file

To modify your robots.txt file, you can hook into the do_robotstxt action, or filter the output by hooking into robots_txt (source).

Example: Mark a directory as “nofollow”

function my_disallow_directory() {
    echo "User-agent: *" . PHP_EOL;
    echo "Disallow: /path/to/your/directory/" . PHP_EOL;
}
add_action( 'do_robotstxt', 'my_disallow_directory' );

Caching

Note that we cache the robots.txt for long periods of time. This means that you’ll need to force the caches to clear after any changes, by going to Settings > Reading within WP-Admin and toggling the Search engine visibility setting saving the changes each time the setting is changed.

On convenience domains

On any convenience domain (e.g., a subdomain of go-vip.net or go-vip.co), the robots.txt output will be hard-coded to return a “Disallow for all user agents” result. This is to prevent search engines from indexing content hosted on development/staging sites.

Last updated: April 09, 2021