Write custom WP-CLI commands at scale
On the VIP Platform, WP-CLI commands run in a container that is separate from a site’s web application, but it shares the Memcached and database containers. It is very important to set protections in place to prevent custom WP-CLI commands from inadvertently affecting your site’s performance.
Avoid memory exhaustion
The WPCOM_VIP_CLI_Command
class, included in VIP MU-plugins, provides helper functions for custom WP-CLI commands that need to run over large datasets. The helper functions can be used by extending this class.
If a large amount of data is being processed on a launched site, make sure to prepare the custom WP-CLI command for processing without exhausting memory and overloading the database by using these helper functions.
WPCOM_VIP_CLI_Command::vip_inmemory_cleanup()
resets the in-memory local WordPress object cache, from the global$wp_object_cache
, without affecting Memcached, and resets the in-memory database query log.- Consider using this to clear memory after having processed 50-100 posts to avoid interruptions, especially when using
get_posts()
orWP_Query
.
- Consider using this to clear memory after having processed 50-100 posts to avoid interruptions, especially when using
WPCOM_VIP_CLI_Command::start_bulk_operation()
defers term counting, so that as individual updates are made to the dataset, expensive term counting is not being triggered each update.- This is important when the command is issuing many writes or changes to the database.
WPCOM_VIP_CLI_Command::end_bulk_operation()
restores and triggers term counting. This should be used as a companion tostart_bulk_operation()
where the pair of functions bookends the command.
- Use sleep() in between updating records or batches of records to help with loads associated with cache re-validation and data replication.
Paginate results and process in batches
- Using a no-LIMIT query can lead to timeout and failure, especially if it takes longer than 30 seconds. Instead, we recommend using smaller queries and paging through the results:
class Test_CLI_Command extends WPCOM_VIP_CLI_Command {
/**
* Publishes all pending posts once they have had their metakeys updated.
*
* Takes a metakey (required) and post category (optional).
*
* @subcommand update-metakey
* @synopsis --meta-key=<meta-key> [--category=<category>] [--dry-run]
*/
public function update_metakey( $args, $assoc_args ) {
// Disable term counting, Elasticsearch indexing, and PushPress.
$this->start_bulk_operation();
$posts_per_page = 100;
$paged = 1;
$count = 0;
// Meta key is required, otherwise an error will be returned.
if ( isset( $assoc_args['meta-key'] ) ) {
$meta_key = $assoc_args['meta-key'];
} else {
// Caution: calling WP_CLI::error stops the execution of the command. Use it only in case you want to stop the execution. Otherwise, use WP_CLI::warning or WP_CLI::line for non-blocking errors.
WP_CLI::error( 'Must have --meta-key attached.' );
}
// Category value is optional.
if ( isset( $assoc_args['category'] ) ) {
$cat = $assoc_args['category'];
} else {
$cat = '';
}
// If --dry-run is not set, then it will default to true. Must set --dry-run explicitly to false to run this command.
if ( isset( $assoc_args['dry-run'] ) ) {
// Passing `--dry-run=false` to the command leads to the `false` value being set to string `'false'`, but casting `'false'` to bool produces `true`. Thus the special handling.
if ( 'false' === $assoc_args['dry-run'] ) {
$dry_run = false;
} else {
$dry_run = (bool) $assoc_args['dry-run'];
}
} else {
$dry_run = true;
}
if ( $dry_run ) {
WP_CLI::line( 'Running in dry-run mode.' );
} else {
WP_CLI::line( 'We\'re doing it live!' );
}
do {
$posts = get_posts(
array(
'posts_per_page' => $posts_per_page,
'paged' => $paged,
'category' => $cat,
'post_status' => 'pending',
'suppress_filters' => 'false',
)
);
foreach ( $posts as $post ) {
if ( ! $dry_run ) {
update_post_meta( $post->ID, $meta_key, 'true' );
wp_update_post( array( 'post_status' => 'publish' ) );
}
$count++;
}
// Pause.
WP_CLI::line( 'Pausing for a breath...' );
sleep( 3 );
// Free up memory.
$this->vip_inmemory_cleanup();
/* At this point, we have to decide whether to increase the value of $paged. In case a value which is being used for querying the posts (like post_status in our example) is being changed via the command, we should keep the WP_Query starting from the beginning in every iteration.
* If the any value used for querying the posts is not being changed, then we need to update the value in order to walk through all the posts. */
// $paged++;
} while ( count( $posts ) );
if ( false === $dry_run ) {
WP_CLI::success( sprintf( '%d posts have successfully been published and had their metakeys updated.', $count ) );
} else {
WP_CLI::success( sprintf( '%d posts will be published and have their metakeys updated.', $count ) );
}
// Trigger a term count as well as trigger bulk indexing of Elasticsearch site.
$this->end_bulk_operation();
}
/**
* Updates terms in that taxonomy by removing the "test-" prefix.
*
* Takes a taxonomy (required).
*
* @subcommand update-terms
* @synopsis --taxonomy=<taxonomy> [--dry_run]
*/
public function update_terms( $args, $assoc_args ) {
$count = 0;
// Disable term counting, Elasticsearch indexing, and PushPress.
$this->start_bulk_operation();
// Taxonomy value is required, otherwise an error will be returned.
if ( isset( $assoc_args['taxonomy'] ) ) {
$taxonomy = $assoc_args['taxonomy'];
} else {
WP_CLI::error( 'Must have a --taxonomy attached.' );
}
if ( isset( $assoc_args['dry-run'] ) ) {
if ( 'false' === $assoc_args['dry-run'] ) {
$dry_run = false;
} else {
$dry_run = (bool) $assoc_args['dry-run'];
}
} else {
$dry_run = true;
}
if ( $dry_run ) {
WP_CLI::line( 'Running in dry-run mode.' );
} else {
WP_CLI::line( 'We\'re doing it live!' );
}
$terms = get_terms( array( 'taxonomy' => $taxonomy ) );
foreach ( $terms as $term ) {
if ( ! $dry_run ) {
$args = array(
'name' => str_replace( 'test ', '', $term->name ),
'slug' => str_replace( 'test-', '', $term->slug ),
);
wp_update_term( $term->term_id, $term->taxonomy, $args );
}
$count++;
}
// Trigger a term count as well as trigger bulk indexing of Elasticsearch site.
$this->end_bulk_operation();
if ( false === $dry_run ) {
WP_CLI::success( sprintf( '%d terms were updated.', $count ) );
} else {
WP_CLI::success( sprintf( '%d terms will be updated.', $count ) );
}
}
}
WP_CLI::add_command( 'test-command', 'Test_CLI_Command' );
Design for restarts
There are many reasons a command may stop mid-execution. Commands dealing with a lot of posts or other long-running actions should be prepared for restart. You might either design them to be idempotent (meaning they can safely be run multiple times) or provide an option to start from a certain point by using offset and limit arguments.
Last updated: August 03, 2023