Index with Enterprise Search

Indexing is the process by which data is compiled and inserted into Elasticsearch. Elasticsearch (ES) is designed for quick retrieval and filtering of items in the index. To accomplish this, ES does not store the full database of a site, and the index will differ from the site’s MySQL table indexes.

How indexing works

The continual indexing of Elasticsearch is dependent on built-in WordPress event action hooks. Normally, the standard WordPress editing process uses event action hooks, and every change made by an editor to a post or page will be quickly reflected in the Elasticsearch index.

Custom code or plugins that use direct database queries or non-standard ways of modifying the database will not sync to the Elasticsearch index.

Content is indexed incrementally, as it is created and updated. Indexing can manually be triggered by opening the particular post or page in the WordPress editor and selecting the button labeled “Update”.

If more than one type of ES index exists (e.g. one for terms and another one for posts), each one requires separate steps for indexing, re-indexing, and versioning.

The index command

The index command initiates indexing activity on a new or existing index (starting from the newest content). Running the index command is necessary to create a new index or if an allow list entry (e.g. post types, taxonomies, or post meta) has been recently modified.

An example of the index command and its output:

$ vip @example-app.develop -- wp vip-search index
Indexing posts...
Processed 500/37674. Last Object ID: 37550
	...
Processed 37500/37674. Last Object ID: 179
Processed 37674/37674. Last Object ID: 1
Number of posts indexed: 37674
Total time elapsed: 605.960
Success: Done!

The amount of time to complete indexing depends on the size of the database. To increase the amount of posts indexed per cycle, use the --per-page option.

Command options

For additional reference on the options available, use the wp help vip-search index command:

vip @example-app.develop -- wp help vip-search index

--setup

Values not accepted.

Completely drops and recreates the index with mappings.

The --setup option is required to create an index the first time. This ensures that mappings are correctly in place.

Recreating an existing index with the --setup option or versioning is necessary to bring a database and its Elasticsearch index back into sync if:

• The database is modified outside of normal WordPress functionality (i.e. database imports, bulk CLI commands that disable the hooks by setting WP_IMPORTING to true, or direct SQL queries that modify data).
• Search tokenization filter settings have been added or modified.

vip @example-app.develop -- wp vip-search index --setup

--blog-ids

Accepted values: blog id or comma-separated list of blog ids

Will target select blog IDs.

For example, to index only blog IDs 3 and 5:

vip @example-app.develop -- wp vip-search index --blog-ids=3,5

--include

Accepted values: single value object ID or comma-separated list of object IDs

Targets indexing only the listed ID(s).

For example, to index only 7 and 20:

vip @example-app.develop -- wp vip-search index --include=7,20

--indexables

Accepted values: single value Indexable or comma-separated list of Indexables
Default values: all registered Indexables (post, user, term, comment)

Targets specific Indexable(s).

For example, to index the term Indexable:

vip @example-app.develop -- wp vip-search index --indexables=term

--network-wide

Values not accepted.

Sequentially index all network sites that exist on a WordPress multisite environment.
Refer to Enterprise Search on multisite for more information.

vip @example-app.develop -- wp vip-search index --network-wide

--per-page

Accepted values: numerical value of posts per page
Default value: 350, maximum of 5000

Determines the amount of posts to be indexed per cycle.

Indexing 350 posts per page is, on average, the most efficient rate for indexing for database content. For databases that are much larger than average, increasing this value could actually prove to slow down the indexing process. Users who choose to customize this value should carefully observe for positive or negative effects of the custom value when indexing.

For example, to increase the indexed posts per cycle to 1000:

vip @example-app.develop -- wp vip-search index --per-page=1000

--post-type

Accepted values: single post type or comma-separated list of post types
Default values: all public post types

Targets indexing only the specified post type(s).

For example, to index only a custom post type event and recipe:

vip @example-app.develop -- wp vip-search index --post-type=event,recipe

--show-errors

Values not accepted.

Displays a verbose error output during indexing. If a post fails to index, it will return with the full error message (compared to just the post ID and title).

vip @example-app.develop -- wp vip-search index --show-errors

--skip-confirm

Values not accepted.

Omits the Enterprise Search confirmation prompt for destructive operations: You are about to run a destructive operation. Are you sure?.

For example, indexing with --setup is a destructive action and will generate the following prompt:

$ vip @example-app.develop -- wp vip-search index --setup
⚠️  You are about to run a destructive operation. Are you sure? [y/n] y
Indexing with setup option needs to delete Elasticsearch index first, are you sure you want to delete your Elasticsearch index? [y/n] y

Including --skip-confirm will omit the destructive operation confirmation prompt:

$ vip @example-app.develop -- wp vip-search index --setup --skip-confirm
Indexing with setup option needs to delete Elasticsearch index first, are you sure you want to delete your Elasticsearch index? [y/n] y

--upper-limit-object-id, --lower-limit-object-id

Accepted values: single value object ID

Limits a subset of object IDs (e.g. post IDs, term IDs, user IDs). IDs are indexed from highest to lowest.

A common use case would be to resume indexing after an interruption or timeout. For example, a previous index command was interrupted at post ID 100, so we would trigger a new index operation using --upper-limit-object-id to begin at 100:

vip @example-app.develop -- wp vip-search index --upper-limit-object-id=100

When these options are used together, they limit the indexing to a range of documents. For example, to index only post IDs from 50 to 10:

vip @example-app.develop -- wp vip-search index --upper-limit-object-id=50 --lower-limit-object-id=10

--using-versions

Values not accepted.

The --using-versions option can be used when re-indexing. Passing the --using-versions option with the wp vip-search index command will automatically:

• Create a new, updated version of the index
• Enable the new version of the index
• Allow the user to delete the previous, inactive index.

vip @example-app.develop -- wp vip-search index --using-versions

--version

Accepted values: index version number, next, active, current, or previous

The index version which the command is targeting. Used during versioning to build up a new index in parallel with the current active index. Typically used in conjunction with the --indexables parameter.

For example, to index version 2 of a posts index:

vip @example-app.develop -- wp vip-search index --version=2 --indexables=post

Monitor or interrupt the indexing process

Subcommands of vip-search are available to retrieve information about the indexing process, stop the process, or unblock the process if necessary.

get-indexing-status

Retrieve information about the progress of a currently running index command with get-indexing-status:

vip @example-app.develop -- wp vip-search get-indexing-status

Output will be returned in JSON format:

• "indexing": true for in process; false for inactive
• "method": how the indexing was actioned
• "items_indexed": number of items already indexed
• "total_items": total number of items not yet indexed

{"indexing":true,"method":"cli","items_indexed":1000,"total_items":1015153}

If the indexing has unexpectedly failed, it is possible that true will be the returned value for "indexing" but the total items will return -1:

{"indexing":true,"method":"cli","items_indexed":0,"total_items":-1}

get-last-indexed-post-id

Use get-last-indexed-post-id to identify the post ID that was last indexed:

vip @example-app.develop -- wp vip-search get-last-indexed-post-id

stop-indexing

Use stop-indexing to stop an active indexing command:

vip @example-app.develop -- wp vip-search stop-indexing

delete-transient

Running the delete-transient command may be necessary if an indexing command process is not currently running—or has unexpectedly stopped—and this error is returned when re-indexing is attempted: Error: An index is already occurring. Try again later.

The delete-transient command removes the index lock will allow re-indexing.

vip @example-app.develop -- wp vip-search delete-transient