Copying production data to your non-production environments

VIP Go has built-in support for syncing/copying data from your production to non-production environments.

This feature facilitates testing and QA of new features and allows teams to accurately reproduce and examine errors in a non-production environment. Our sync process handles the data copying, modifies URLs to match the new environment, and adds a hook for further client-specific customization.

If you have any questions about the data sync process, please contact our Support team.

Running a data sync via the VIP Dashboard

The VIP Dashboard offers a tool to sync data from production to non-production environments. See the What happens during a Data Sync? section below for more details.

Running a data sync via CLI

You will need the VIP-CLI installed on your local machine: Learn more by reading the VIP-CLI installation instructions.

Step 1: Find the application name or ID you wish to sync

You can do this with the vip app list command, which will give you back a list of apps. You want the name or the id.

Step 2: Initiate the sync

Run the sync command with the app name, e.g. vip sync --app=my-app, or the app id, e.g. vip sync --app=000. You will be prompted for the environment you wish to sync to, i.e. the target environment. You can get more information about sync command options by using the help parameter, i.e. vip sync --help.

What happens during a Data Sync?

The operation is designed to not impact the production site in any way; all processing happens on the target environment:

1. The target environment is protected via “maintenance mode”; this prevents access to the site while the data is being restored and manipulated, and ensures no changes can be made (and lost) during this time.
2. The most recent backup from production is loaded into the target environment.
3. We update all references to production domains to match the target environment.
   • The primary production domain is replaced with the primary domain for the target environment; e.g. if your production site is at example.com and your target non-production site is example-com-develop.go-vip.net, then we’ll replace all instances of example.com with example-com-develop.go-vip.net
   • For any additional domains, we use config file with a URL map stored in your site’s code repository. See “domain mapping” below for more details.
4. Caches are flushed.
5. Jetpack is re-connected.
6. Maintenance mode is deactivated.
7. A cron event (vip_go_migration_cleanup) is fired for further cleanup.
   • See below for more detail on this hook, and associated considerations.

Domain mapping config

If your application has multiple domains in use (in a multisite install, for example), Data Sync needs a bit more information to correctly update URL references in your non-production environments. This can be done using config files in your code repository.

The error message “Missing .vip.yml.config” will appear on the Sync section of your App Dashboard if the environment you are syncing to requires a domain mapping config file. Click on the “.vip.yml.config” link to create an automated config file for this environment.

If you don’t have multiple domains, you can skip this section.

Structure

Data Sync relies on YAML-based config files to correctly handle URL replacements. Here’s what the basic structure looks like:

data_sync:
  domain_map:
    prod-url-1: nonprod-url-1
    prod-url-2: nonprod-url-2

To illustrate, here’s an example. Let’s say your production environment uses the following domains:

1. en.example.com
2. fr.example.com

And your non-production environment has the following matching domains:

1. en-preprod.example.com
2. fr-preprod.example.com

The config file would look something like this:

data_sync:
  domain_map:
    en.example.com: en-preprod.example.com
    fr.example.com: fr-preprod.example.com

When Data Sync runs, all instances of en.example.com will be replaced with en-preprod.example.com and all instances of fr.example.com will be replaced with fr-preprod.example.com.

If your environments use a mixture of domain-, subdomain- and/or subdirectory-based sites, you can still specify those via config.

Let’s say we had another site in our example above (example.com/translate) which had a matching non-production site (preprod.example.com/translate). Our config would look something like this:

data_sync:
  domain_map:
    en.example.com: en-preprod.example.com
    fr.example.com: fr-preprod.example.com
    example.com/translate: preprod.example.com/translate

Naming

If you have multiple applications or environments using the same repo, chances are that each of them will need a unique config. We use a specific naming convention for config files to correctly identify the application and environment:

.vip.[app].[environment].yml

The placeholders are:

• [app] is the name/slug used to identify your (production) application
• [environment] is the environment that this config file applies to

Here’s an example: .vip.example.develop.yml (this config file would be used for the example application’s develop environment).

In some (very rare) circumstances, you may have multiple environments using the same environment name (e.g. multiple develop environments). In this case, you can add the unique name of your non-production environment at the end (i.e. .vip.[app].[environment].[child].yml; an example would be .vip.example.develop.instance-01.yml).

The “Sync” section of the VIP Dashboard contains a link that will generate this file in the appropriate place in the repo, with the correct naming, for any child environment that is missing one.

If extra assistance is needed, please let us know and we’ll be happy to help.

Location

Config files for Data Sync should live in a config folder in the root of the branch that tracks your production environment (usually master).

Here’s what the root of your master branch might look like:

|-- client-mu-plugins/
|-- config/
|----- .vip.example.develop.yml
|----- .vip.example.preprod.yml
|-- images/
|-- languages/
|-- plugins/
|-- themes/
|-- vip-config/

Maintenance

Please note that you will need to update the config files if any domains are added, removed, or modified across any of your environments. We’ll be adding tools in the future to help simplify this process.

Custom cleanup operations

The vip_go_migration_cleanup hook can be used for custom cleanup of your application data. Some examples of cleanup operation include removing production API keys or performing brief data manipulation.

Here’s some example code that shows you how to use this cleanup event:

/**
 * Run some custom cleanup after a migration.
 *
 * @uses vip_go_migration_cleanup
 */
function my_action_vip_go_migration_cleanup() {
    // Safety first: Don't do anything in 
    // the production environment
    if ( 'production' === VIP_GO_APP_ENVIRONMENT ) {
        return;
    }
 
    delete_option( 'my_social_api_token' );
}
add_action( 'vip_go_migration_cleanup', 'my_action_vip_go_migration_cleanup' );

On a multisite, the cleanup hook is run individually on each subsite within that multisite.

Media library files and UnionFS

Instead of copying all the media library files from the production environment — a lengthy process for large media libraries of millions of files and hundreds of gigabytes of data — VIP Go provides a service called UnionFS.

UnionFS works by making the files for the production site available to all non-production sites in read-only mode. Files shared by UnionFS in this way are served from the same infrastructure and have the same caching rules applied.

The non-production environment can upload a file of the same name to the same URL, and this will override the file shown on the non-production site (the production file remains untouched).

Caveats:

• The non-production environment cannot delete a file on the production site. The operation will not error, but the production file will still be shared and available from the non-production environment.
• If the file is deleted from production, it will no longer be available to be shared with the non-production environments.