Skip to content

Backgrounds

How-to Guides

Technical References

Enterprise Search /

Indexing with Enterprise Search

The Elasticsearch (ES) indexes are similar, in a way, to MySQL table indexes – they keep track of all the items they are asked to, and allow the ES engine to quickly resolve a query to the underlying data. But they are also different, because ES doesn’t have the full database; it’s only designed for quick retrieval and sorting/filtering of the items in the index.

If you have more than one index, for example, one index for terms and another (default) index for posts, each index requires separate steps for indexing, re-indexing, and versioning.

Content is normally indexed incrementally, as it is created and updated.

Indexing

The CLI index command initiates indexing activity on a new or existing index starting from the newest content. This is helpful if you’ve just added an allow list entry for a post meta key that wasn’t previously being indexed, for example.

Note

Because these are WP-CLI commands, you’ll need to pass them to the VIP-CLI, as demonstrated in the first example (where our app id is 103). We’ve omitted this prefix from the remaining commands for clarity.

$ vip @103 -- wp vip-search index
Indexing posts...
Processed 500/37674. Last Object ID: 37550
	...
Processed 37500/37674. Last Object ID: 179
Processed 37674/37674. Last Object ID: 1
Number of posts indexed: 37674
Total time elapsed: 605.960
Success: Done!

The amount of time it takes for an index depends on the size of the database. To increase the amount of posts indexed per cycle, use the --per-page option.

Additional Command Options

The index command accepts several additional options. To check which ones are available, use the wp help command:

$ wp help vip-search index

–setup

Values not accepted.

The --setup option completely drops and recreates the index.

This option is useful in situations where you modify the database outside of normal WordPress functionality (i.e. using bulk CLI commands to modify data using raw SQL and the proper hooks do not fire). Only running wp vip-search index may not bring the database and Elasticsearch index back into sync — the only way to sync up is by creating a fresh index and then, indexing the database content into the ES index.

$ wp vip-search index --setup

Warning

Including the --setup option will drop the current index, and create a fresh index. If the site is in production, this is not recommended – please use index versioning instead.

–network-wide

Values not accepted.

This command sequentially indexes all of the subsites in a network.

$ wp vip-search index --network-wide

–version

Accepted values: index version number, next, active, current, or previous

The index version to target the command to — it is used during versioning to build up a new index in parallel with the current active index. Typically used in conjunction with the --indexables parameter.

For example, if we have created a new index version 2 for post content and want to index it:

$ wp vip-search index --version=2 --indexables=post

–indexables

Accepted values: single value Indexable or comma-separated list of Indexables
Default values: all registered Indexables: post, user, term, comment

This option is used to target specific Indexable(s).

$ wp vip-search index --indexables=term

–upper-limit-object-id, –lower-limit-object-id

Accepted values: single value document ID

To limit a subset of IDs when indexing.

A common use case would be to resume indexing after an interruption or timeout. For example, a previous index command was interrupted at post ID 100, so we would trigger a new index operation using --upper-limit-object-id to begin at 100:

$ wp vip-search index --upper-limit-object-id=100

When these options are used together, they scope down the indexing to a range of documents. For example, to index post IDs from 50 to 10:

$ wp vip-search index --upper-limit-object-id=50 --lower-limit-object-id=10

Note

The indexing process goes from highest to lowest ID. E.g. In a range, it starts at --upper-limit-object-id and descends towards --lower-limit-object-id.

–per-page

Accepted values: numerical value of posts per page
Default value: 500 (maximum of 5000)

This option determines the amount of posts to be indexed per cycle.

$ wp vip-search index --per-page=1000

–include

Accepted values: single value object ID or comma-separated list of object IDs

This option targets indexing only the listed ID(s). For example, we only want to index 7 and 20:

$ wp vip-search index --include=7,20

–post-type

Accepted values: single post type or comma-separated list of post types

Defaults to all public post types. This option targets indexing only the specified post type(s).

$ wp vip-search index --post-type=page

–show-bulk-errors

Values not accepted.

This option displays a verbose error output during indexing. If a post fails to index, it will return with the full error message (compared to just the post ID and title).

$ wp vip-search index --show-bulk-errors

–skip-confirm

Values not accepted.

This option skips the Enterprise Search confirmation prompt, “You are about to run a destructive operation. Are you sure?”, for destructive operations. For example, using the option with --setup:

$ wp vip-search index --setup --skip-confirm
Indexing with setup option needs to delete Elasticsearch index first, are you sure you want to delete your Elasticsearch index? [y/n] y

Compared to using the option without:

╰─$ wp vip-search index --setup
⚠️  You are about to run a destructive operation. Are you sure? [y/n] y
Indexing with setup option needs to delete Elasticsearch index first, are you sure you want to delete your Elasticsearch index? [y/n] y

Indexing status

Use the get-indexing-status command if you want to check on the progress of an index command that is running.

$ wp vip-search get-indexing-status

For example, for a site where there is ongoing indexing, the output will return true under "indexing" in JSON formatting (along with how the indexing was actioned, the number of items already indexed and the total number of items needing to be indexed):

{"indexing":true,"method":"cli","items_indexed":1000,"total_items":1015153}

Alternatively, when there is no indexing occurring, the output will be false for "indexing":

{"indexing":false,"method":"none","items_indexed":0,"total_items":-1}

In a scenario where the indexing has unexpectedly failed, you may still see true for "indexing", but the total items will return -1:

{"indexing":true,"method":"cli","items_indexed":0,"total_items":-1}

Stop indexing

If there is indexing currently occurring that needs to be halted, the stop-indexing command can be used:

$ wp vip-search stop-indexing

Troubleshooting

Error: An index is already occurring. Try again later.

If no indexing is currently occurring (i.e. the indexing command unexpectedly stops) but you are receiving an error message that it is, you may need to remove the index lock before re-trying to index:

$ wp vip-search delete-transient

If you are unsure on the status of indexing, use the get-indexing-status command to check on it.

Last updated: September 07, 2021