Import media files

Media can be imported to the VIP File System with VIP-CLI. This is especially useful for migrating from an existing WordPress site to a site hosted on the VIP Platform.

Limitations

• Only one import per environment can be run at a single time. Attempts to start a second import while a previous import is still in progress will result in an error.
• Files imported with VIP-CLI will not appear in a site’s WordPress media library unless the attachment data for the files is also imported to the database.
• Media should only be imported to a production environment. Media uploaded to a production environment is immediately made available to a child environment after running a data sync. For this reason, importing media to non-production environments is unnecessary.
• By default, media imports do not overwrite existing files that share the same name and file path. Including the --overwriteExistingFiles option (or -o ) when importing media files will override this behavior and force the silent overwriting of an existing file.
• The maximum size for an individual file imported to VIP’s File System is 2 GB.
• After a media file has been imported, it cannot be deleted from the VIP File System.

Requirements

• Files for import must be archived (.tar.gz, .tgz, or .zip).
• The size of the entire archive that holds the files to be imported cannot be more than 200 GB.
• The archived file must decompress to a directory named uploads, and only contain the media required for a site(s).
• Upload archived files to a publicly accessible URL (e.g. https://www.example.com/uploads.tar.gz).
  • File-sharing services such as Dropbox, Box, or WeTransfer are able to provide publicly accessible URLs.
  • A downloadable Dropbox URL must include a ?dl=1 query parameter.
  • Google Drive URLs are not supported.

If the file for import does not meet the above requirements, create a VIP Support ticket requesting assistance with the import and include a downloadable link for the archived file.

Directory structure

The directory to be imported must be named uploads.

In most cases, media uploaded to a WordPress install is stored in a directory named uploads/, which is a subdirectory of wp-content/. To prepare media files for import to VIP, an archived copy of the uploads/ directory is recommended.

If a non-VIP WordPress site has customized the name of the uploads/ directory, extra steps will be needed to duplicate the directory to a working location and re-name the duplicate directory to uploads before importing the media to VIP.

When an archive’s directory structure contains a directory named uploads/, only the files within the uploads/ directory will be imported. All other files will be ignored.

If no directories named uploads/ exist within an archive, the import tool will import the contents of the archive to an environment’s uploads/ directory with the archive’s existing directory structure.

Multisites

Media import directories for multisites must also be named uploads.

The sub-directory structure of the uploads/ directory for sites belonging to a WordPress multisite will differ from WordPress single sites. The uploads/ file path for the main site (ID 1) of the multisite will beuploads/, but all subsequent sites on a multisite network will be in the uploads/sites/ directory structure, in their own subfolder based on the site number, designated by the database.

Upload paths for all sites other than the main site (ID 1) follow the pattern uploads/sites/{SITE ID}/{YYYY}/{MM}/{FILE NAME}. The values for {SITE ID} do not have any leading 0’s (i.e., use 2, not 02).

Special characters in file names

When files are uploaded through the WordPress media library, WordPress Core sanitizes file names and removes or replaces special characters, including: ~?,[]/\=<>:;'\&$#*()|~`!{}%+ and whitespace ().

When media files are imported to VIP, they bypass the WordPress sanitize_file_name() function, and all special characters are allowed, including: spaces ( ) [ ] ~ + & # % = ' ” \ < > : ; , / ? $ * | ` ! { } . Encoded or alternate whitespace, such as %20or + will be converted to proper spaces during the import.

Intermediate images

Traditional installations of WordPress create intermediate images; an array of smaller versions of each uploaded image. The generated intermediate files are unnecessary on VIP, as the VIP File System is able to dynamically generate and cache any requested dimension of the originally uploaded image. This reduces the amount of storage a site requires and provides image size flexibility at no loss of speed.

If needed, intermediate images can be imported by including the -i flag when running the vip import media command. When including intermediate images for an import, the process will take a longer time to complete.

For site migrations to the VIP Platform: Importing a site’s intermediate files might be necessary to prevent broken links (404s) where links to specific intermediate files exist in the site’s imported database content. After the migration process, links to an image’s intermediate size array will be generated on the fly by the VIP File System.

Unsupported file types

Imports to the VIP File System will fail for the following file types:

• Files without extensions
• Files not included in the list of file types supported by the VIP File System
• Hidden files or files within hidden directories will be ignored. Hidden files will not be included in the count of files that are discovered by the import media command, and they will not be included in the final count of files that are successfully imported.
• File paths that are more than 255 characters in length. This limitation includes the /wp-content/uploads/ prefix in every imported file path.
• File paths that end with a forward-slash (i.e. /).

SVGs

SVG files allow embedded JavaScript, which can pose a security risk. If SVGs exist in the uploads/ directory that will be imported, it is recommended to scan the file contents prior to media import. The file contents of an SVG can be scanned by:

• Examining the contents within a file editor ( e.g. Visual Studio Code).
• A tool such as the SVG Sanitizer Test or VIP’s SVG Sanitizer.
• A PHPCS scan using the WordPress-VIP-Go standard.

All SVG files introduced to a GitHub repository in a pull request will undergo an SVG analysis as part of the automated PHPCS scan using the WordPress-VIP-Go standard from the VIP Code Analysis Bot. Any discovered irregularities will be reported in the automated code review.

Import media files with VIP-CLI

Import media files to the VIP File System with the VIP-CLI command: vip import media [options] [command]

Example command for importing an archived file from the publicly available URL “https://www.example.com/uploads.tar.gz”:

vip @example-app.production import media "https://www.example.com/uploads.tar.gz"

During the import process, the CLI will display the real-time progress of the import. If desired, the real-time progress of an import can be hidden using the Ctrl + C key combination.

When all files in the archive have been processed, a status report for the import will be created within the current working directory. The exact path to this status report will be shown on screen.

Overwrite existing files during import

Including the --overwriteExistingFiles option (or -o ) during import will force the silent overwriting of existing files.

For example:

vip @example-app.production import media "https://www.example.com/uploads.tar.gz" --overwriteExistingFiles

Retrieve error logs

If one or more files fail to import, an error log will be generated and able to be retrieved at the end of an import. Error logs include a list of failed files and the reasons for their failure.

By default, import logs are created in .txt format. Add the --exportFileErrorsToJson option (or -E) to write the error logs to a .json file instead.

vip @example-app.production import media "https://www.example.com/uploads.tar.gz" -E

An error log file can be dowloaded within 15 minutes of the completion of the import process from a URL provided in message output in the terminal. The file can be accessed from a new URL by running the vip import media status command. The file will remain available for up to 7 days after the import or until a new media import is performed.

Abort an import

VIP-CLI command: vip import media abort [options]

Abort an import process in progress. All files imported prior to executing the abort command cannot be deleted.

vip @example-app.production import media abort

Check the status of an import

VIP-CLI command: vip import media status [options]

The status of the current or most recent import can be checked with the vip import media status command.

• If an import is currently running, the output will be for the currently running import, and the status command will continually check and report back on the status of the import process until it is in a finished state (COMPLETED, ABORTED, or FAILED).
• If an import is not currently running, the output will be for the import that ran most recently.

The Imported Files: field displays the number and percentage of files imported successfully.

If any files failed to import, a warning (⚠️) message will appear in the output, and the associated errors will be exported to an error log for review.

vip @example-app.production import media status
=============================================================
Checking the Media import status for your environment...
Imported Files: 95/100 - 95% ✓
=============================================================
Status: COMPLETED ✓ : The imported files should be visible on your App
App: mytestsite (PRODUCTION)
=============================================================

⚠️  5 file error(s) have been extracted

All errors have been exported to /Users/myuser/media-import-mytestsite-1642371503508.txt

The Status: field will display the overall import status with the following possible states:

Import status	Description
INITIALIZING	Contents of the archive file in the URL are downloading.
RUNNING	Files are being imported to the File System.
COMPLETING	The import is finishing up.
COMPLETED	The import is completed.
ABORTING	The import is responding to a vip import media abort request.
ABORTED	The import is aborted.
FAILED**	The import failed.

**The import can be in a FAILED status for a variety of reasons. Review status output, error logs, limitations, and the import checklist before attempting another media import.

Media import actions in the Audit Log

An audit trail of VIP-CLI commands that were executed to start or abort a media import, along with the timestamp of the commands and the VIP user that ran them, is displayed in the Audit Log panel of the VIP Dashboard.