Skip to content

Modify the robots.txt file


Environments using a convenience domain (e.g., a subdomain of or, have a hard-coded /robots.txt output that returns a “Disallow for all user agents” result. This is to prevent search engines from indexing content hosted on non-production sites, or unlaunched production sites. In order to modify the output for /robots.txt an environment (production or non-production) must have a custom mapped domain.

To modify the /robots.txt file hook into the do_robotstxt action, or filter the output by hooking into robots_txt filter.

Example: Mark a directory as “nofollow”

function my_disallow_directory() {
    echo "User-agent: *" . PHP_EOL;
    echo "Disallow: /path/to/your/directory/" . PHP_EOL;
add_action( 'do_robotstxt', 'my_disallow_directory' );


The /robots.txt file is cached for long periods of time. In order to force the cache to clear after any changes made to the file, go to Settings > Reading within WP-Admin and toggle the Search engine visibility setting, saving the changes each time the setting is changed.

The page cache for the /robots.txt file can also be flushed using the wp vip cache purge-url WP-CLI command which is available on WordPress environments.

Last updated: June 03, 2022