Skip to content

Backgrounds

How-to Guides

Technical References

Enterprise Search /

Index and version with Enterprise Search

The Elasticsearch (ES) indexes are similar, in a way, to MySQL table indexes – they keep track of all the items they are asked to, and allow the ES engine to quickly resolve a query to the underlying data. But they are also different, because ES doesn’t have the full database; it’s only designed for quick retrieval and sorting/filtering of the items in the index.

If you have more than one index, for example, one index for terms and another (default) index for posts, each index requires separate steps for indexing, re-indexing, and versioning as outlined below.

Indexing and re-indexing

Content is normally indexed incrementally, as it is created and updated.

The CLI index command initiates indexing activity on a new or existing index starting from the newest content. This is helpful if you’ve just added an allow list entry for a post meta key that wasn’t previously being indexed, for example.

Note

Because these are WP-CLI commands, you’ll need to pass them to the VIP-CLI, as demonstrated in the first example (where our app id is 103). We’ve omitted this prefix from the remaining commands for clarity.

$ vip @103 -- wp vip-search index
Indexing posts...
Processed 500/37674. Last Object ID: 37550
	...
Processed 37500/37674. Last Object ID: 179
Processed 37674/37674. Last Object ID: 1
Number of posts indexed: 37674
Total time elapsed: 605.960
Success: Done!

If you modify the database outside of normal WordPress functionality (normal here meaning using the wp-admin to create a post, edit a post, etc.), running wp vip-search index may not bring the database and Elasticsearch index back into sync. This is because there may be items in the index that were deleted or modified with raw SQL that would not be removed from the index.

To completely re-index — for example, following an import or a CLI script that made raw DB queries — you must create a fresh index.

The --setup option drops and recreates the index.

$ wp vip-search index --setup

Warning

Including the --setup option will drop the current index, and create a fresh index. If the site is in production, this is not recommended – use index versioning (below) instead.

A reindex is NOT a sync, for the reasons described above; the only way to bring the database and index back into sync is to create a fresh index and then index the database content into the ES index.

The safest way to do that with a production site is via versioning.

Versioning

If you need to re-index, and do not want to drop the current index, follow these steps to create a new index, ensure it is healthy, and make it active.

1. Register the new index

This will create a new index for post content (this will also work with terms)

$ wp vip-search index-versions add post
Success: Registered and created new index version 2

The response should include the version number added (in this case, 2).

This will add a new index version and creates it (with mapping) on Elasticsearch. NOTE – this does not do any bulk indexing of content, but it will start the indexing of incremental changes to the new version.

The current active version will continue to receive incremental changes as well.

2. Build a new index

Now, build the new index alongside the current one. This may take some time.

In the command, you can specify a version, or use next, active, current and previous as an alias.

$ wp vip-search index --indexables="post" --version="next"
Indexing posts...
Processed 500/37674. Last Object ID: 37550
Processed 1000/37674. Last Object ID: 37050
Processed 1500/37674. Last Object ID: 36550
...
Processed 37674/37674. Last Object ID: 1
Number of posts indexed: 37674
Total time elapsed: 492.615
Success: Done!

3. Health check

Check the status of the indexes (both the current and next)

$ wp vip-search health validate-counts
Validating post count

✅ no inconsistencies found when counting entity: post, type: post, index_version: 1 - (DB: 37498, ES: 37498, Diff: 0)
✅ no inconsistencies found when counting entity: post, type: page, index_version: 1 - (DB: 176, ES: 176, Diff: 0)
✅ no inconsistencies found when counting entity: post, type: post, index_version: 2 - (DB: 37498, ES: 37498, Diff: 0)
✅ no inconsistencies found when counting entity: post, type: page, index_version: 2 - (DB: 176, ES: 176, Diff: 0)

Cannot find indexable 'user', probably the feature is not enabled

4. Activate the new index

Once the new index version shows that it is in sync (“healthy”), it can be activated.

The active index is the one that serves site traffic (queries), so be sure the target index is ready for production before switching.

$ wp vip-search index-versions activate post next
Are you sure you want to activate index version next for type post? [y/n] y
Success: Successfully activated index version next for type post

5. Delete the old index

Once you’ve verified the new index is handling requests correctly, you can delete the old index. (You cannot delete the active index).

$ wp vip-search index-versions delete post previous
⚠️ You are about to run a destructive operation. Are you sure? [y/n] y
Success: Successfully deleted index version 1 for type post

Last updated: May 14, 2021