Block by User Agent

It can be useful to block by User Agent in cases where a tool or bot sends suspicious requests from various IP addresses while retaining the same User Agent. WordPress VIP provides two User Agent blocking methods for WordPress applications.

When blocking by User Agents, take care not to block common browser User Agents, or User Agents that look similar. It is a best practice to search recent HTTP Request Logs to validate that full, or partial matching blocking will only impact the intended requests. Blocking requests from common browser User Agents can result in blocking real user requests.

Block by full User Agent match

The VIP_Request_Block::ua() method can be used when the full text of an unwanted User Agent is known and uses case-sensitive matching.

// Example VIP_Request_Block::ua( string $user_agent );
if ( class_exists( 'VIP_Request_Block' ) ) {
   VIP_Request_Block::ua( 'SuspiciousBot/1.1' );
   VIP_Request_Block::ua( 'AnotherBot/2.1' );
}

Block by partial User Agent match

The VIP_Request_Block::ua_partial_match() method can be used when requests from a User Agent that contains a sub-string need to be blocked. This method does not support regular expressions, and the string matching is case-sensitive.

// Example VIP_Request_Block::ua_partial_match( string $user_agent_substring );
// Will match and block for:
// 	- SuspiciousBot/1.1
// 	- SomewhatSuspiciousBot/1.8 - https://example.com/robot-policy
if ( class_exists( 'VIP_Request_Block' ) ) {
   VIP_Request_Block::ua_partial_match( 'SuspiciousBot/' );
}

Block by partial User Agent match for AI crawlers

To block requests by artificial intelligence (AI) crawlers for a site’s content, a disallow rule should be set in a site’s robots.txt. An additional measure can be put in place to block AI crawlers by their User Agents with the VIP_Request_Block::ua_partial_match() method.

This code example demonstrates a custom function that blocks requests from User Agents of 4 well-known AI crawlers (e.g. OpenAI’s GPTBot) without blocking requests to the site’s robots.txt:

function my_block_ai_user_agents() {
	if ( ! class_exists( 'VIP_Request_Block' ) ) {
		return;
	}

	// Do not block access to robots.txt
	if ( isset( $_SERVER['REQUEST_URI'] )
		&& true === str_contains( $_SERVER['REQUEST_URI'], '/robots.txt' ) ) {
		return;
	}

	// OpenAI GPTBot crawler (https://platform.openai.com/docs/gptbot)
	VIP_Request_Block::ua_partial_match( 'GPTBot/' );

	// OpenAI ChatGPT service (https://platform.openai.com/docs/plugins/bot)
	VIP_Request_Block::ua_partial_match( 'ChatGPT-User/' );

	// Common Crawl crawler (https://commoncrawl.org/faq)
	VIP_Request_Block::ua_partial_match( 'CCBot/' );

	// Google Bard / Gemini crawler (https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
	VIP_Request_Block::ua( 'Google-Extended' );
}

my_block_ai_user_agents();

Only four AI crawlers are included in this code example, though far more exist. Review an environment’s recent HTTP Request Logs to determine which AI crawler User Agents should be disallowed for a site.