Skip to content

Import media files

Media can be imported to the VIP File System with VIP-CLI. This is especially useful for migrating from an existing WordPress site to a site hosted on the VIP Platform.

Prerequisites

Limitations

  • Media can only be imported to a production environment. Importing media to non-production environments is not necessary, as media uploaded to production is immediately made available to the child environments as well.
  • Only one import per environment can be run at a single time. Attempts to start a second import while a previous import is still in progress will result in an error.
  • Files imported with the VIP-CLI command will not appear in a site’s WordPress media library unless the attachment data for the files is also imported to the database. The vip import media VIP-CLI command only imports files to the VIP File System.
  • After a file has been imported, it cannot be deleted from the VIP File System.

Directory structure

The directory to be imported must be named uploads.

In most cases, media uploaded to a WordPress install is stored in a directory named uploads/, which is a subdirectory of wp-content/. To prepare media files for import to VIP, an archived copy of the uploads/ directory is recommended.

If a non-VIP WordPress site has customized the name of the uploads/ directory, extra steps will be needed to duplicate the directory to a working location and re-name the duplicate directory to uploads before importing the media to VIP.

When an archive’s directory structure contains a directory named uploads/, only the files within the uploads/ directory will be imported. All other files will be ignored.

If no directories named uploads/ exist within an archive, the import tool will import all contents of the archive with their existing directory structure.

Multisites

Media import directories for multisites must also be named uploads.

The sub-directory structure of the uploads/ directory for sites belonging to a WordPress multisite will differ from WordPress single sites. The uploads/ file path for the root site (Site ID 1) of the multisite will be uploads/, but all subsequent sites on a multisite network will be in the uploads/sites/ directory structure, in their own subfolder based on the site number, designated by the database.

Upload paths for non-root sites follow the pattern uploads/sites/{SITE ID}/{YYYY}/{MM}/{FILE NAME}. The values for {SITE ID} do not have any leading 0’s (i.e., use 2, not 02).

Special characters in filenames

WordPress Core sanitizes file names, removing or replacing special characters upon upload to the media library, for example:
 ~?,[]/\=<>:;'\&$#*()|~`!{}%+ and whitespace ().

Media file imports to VIP bypass the WordPress sanitize_file_name() function, and all special characters are allowed, for example:
spaces ( ) [ ] ~ + & # % = ' ” \ < > : ; , / ? $ * | ` ! { } .
Encoded or alternate whitespace, such as %20or + will be converted to proper spaces during the import.

Intermediate images

Traditional installations of WordPress create intermediate images; an array of smaller versions of each uploaded image. The generated intermediate files are unnecessary on VIP, as the VIP File System is able to generate and cache any requested dimension of the originally uploaded image. This reduces the amount of storage a site requires and provides image size flexibility at no loss of speed. It is recommended that intermediate images are excluded from the imported media directory.

If needed, intermediate images can be imported using the -i flag with the import media command. Including intermediate images in an import will cause the process to take a longer time to complete.

Unsupported file types

Imports to the VIP File System will fail for the following file types:

  • Files without extensions
  • Files not included in the list of file types supported by the VIP File System
  • Hidden files or or files within hidden directories will be ignored. Hidden files will not be included in the count of files that are discovered by the import media command, and they will not be included in the final count of files that are successfully imported.

SVGs

SVG files allow embedded JavaScript, which can pose a security risk. If SVGs exist in the uploads/ directory that will be imported, it is recommended to scan the file contents prior to media import. The file contents of an SVG can be scanned by:

  • examining the contents within a file editor (Visual Studio Code, for example)
  • a tool such as http://svg.enshrined.co.uk/ or https://github.com/Automattic/vip-go-svg-sanitizer
  • a PHPCS scan using the WordPress-VIP-Go standard

All SVG files introduced to a GitHub repository in a pull request will undergo an SVG analysis as part of the automated PHPCS scan using the WordPress-VIP-Go standard from the VIP Code Analysis Bot. Any discovered irregularities will be reported in the automated code review.

Validate files and directory structure prior to import

Before the directory is archived and uploaded to an external URL for import, validate the uploads/ directory on a user’s local machine with the VIP-CLI command:
vip import validate-files <path/to/file.sql>

Output from the command will indicate if the directory and any of its contents have incompatibility issues related to invalid filenames, invalid file extensions, intermediate images, or directory structure. Every directory and file within uploads/ is scanned for these potential issues. A summary of the issues found, their status ( PASS, RECOMMENDED, or ERROR ), and suggested actions for resolving the issues will be included in the command output.

In the following example, the output returned by the vip import validate-files command reports that the uploads/ directory structure does not follow the WordPress default, and an intermediate image has been included:

$ vip import validate-files uploads/
Folder: uploads/22/01

✅ File structure: Uploads directory exists
✕ Recommended: Structure your WordPress media files into `uploads/YYYY` directories
✕ Recommended: Structure your WordPress media files into `uploads/YYYY/MM` directories

We recommend the WordPress default folder structure for your media files: 

Single sites:`uploads/year/month/image.png`
 e.g.-`uploads/2020/06/image.png`
Multisites:`uploads/sites/siteID/year/month/image.png`
 e.g.-`uploads/sites/5/2020/06/images.png`

------------------------------------------------------------

✕ Intermediate images: Duplicate files found:
Original file: uploads/22/01/image.png
Intermediate images: uploads/22/01/image-300x300.png

------------------------------------------------------------

 RECOMMENDED  1 folders, 21 folders total
   ERROR      1 intermediate images, 874 files total
    PASS      0 invalid file extensions, 874 files total
    PASS      0 invalid filenames, 874 files total

Import media files with VIP-CLI

Prior to running any import commands, confirm all items in this checklist:

  • Files for import must be archived (.tar.gz, .tgz, or .zip).
  • The size of the entire archive that holds the files to be imported cannot be more than 60 GB.
  • The maximum size for an individual file imported to VIP’s File System is 1GB.
  • The archived file must decompress to a directory named uploads , and only contain the media required for a site(s).
  • Upload archived files to a publicly accessible URL (e.g., https://www.example.com/uploads.tar.gz).
    • File-sharing services such as Dropbox, Box, or WeTransfer are able to provide publicly accessible URLs.
    • A downloadable Dropbox URL must include a ?dl=1 query parameter.
    • Google Drive URLs are not supported.

If the file for import does not meet the above requirements, create a VIP Support ticket requesting assistance with the import and include a downloadable link for the archived file.

Import media

Import media to the VIP File System with the vip import media VIP-CLI command.

Refer to the help menu for a full list of commands, options, and examples.

vip import media --help

VIP-CLI command examples

For demonstration purposes, the <app-alias> value mytestsite and the <env> value production are used in the VIP-CLI command examples below. Read more about how to target environments in VIP-CLI commands.

Example command for importing an archived file available at the URL “https://www.example.com/uploads.tar.gz”:

vip import media @mytestsite.production "https://www.example.com/uploads.tar.gz"

During the import process, the CLI will display the real-time progress of the import. If desired, the real-time progress of an import can be hidden using the Ctrl + C key combination.

When all files in the archive have been processed, a status report for the import will be created within the current working directory. The exact path to this status report will be shown on screen.

Overwrite existing files during import

Including the --overwriteExistingFiles option (or -o ) during import will force the silent overwriting of existing files.

For example:

vip import media @mytestsite.production "https://www.example.com/uploads.tar.gz" --overwriteExistingFiles

Format error logs

If one or more files fail to import, an error log will be created on the local machine at the end of an import.

The file will be created within the path /Users/youruser/ with the file format media-import-<app_alias>-<date>.txt.

Error logs include a list of failed files and the reasons for their failure. Error logs are unable to record more than 250,000 file failures.

By default, import logs are created in .txt format. Add the --exportFileErrorsToJson option (or -E) to write the error logs to a .json file instead.

vip import media @mytestsite.production "https://www.example.com/uploads.tar.gz" -E

Abort an import

The vip import media abort command allows an import process to be aborted after the process has begun. All files imported prior to executing the abort command cannot be deleted.

vip import media abort @mytestsite.production

Check the status of an import

The status of the current or most recent import can be checked with the vip import media status command.

  • If an import is currently running, the output will be for the currently running import, and the status command will continually check and report back on the status of the import process until it is in a finished state (COMPLETED, ABORTED, or FAILED).
  • If an import is not currently running, the output will be for the import that ran most recently.

The Imported Files: field displays the number and percentage of files imported successfully.

If any files failed to import, a warning (⚠️) message will appear in the output, and the associated errors will be exported to an error log for review.

vip import media status @mytestsite.production
=============================================================
Checking the Media import status for your environment...
Imported Files: 95/100 - 95% ✓
=============================================================
Status: COMPLETED ✓ : The imported files should be visible on your App
App: mytestsite (PRODUCTION)
=============================================================

⚠️  5 file error(s) have been extracted

All errors have been exported to /Users/myuser/media-import-mytestsite-1642371503508.txt

The Status: field will display the overall import status with the following possible states:

Import status Description
INITIALIZINGContents of the archive file in the URL are downloading.
RUNNINGFiles are being imported to the File System.
COMPLETINGThe import is finishing up.
COMPLETEDThe import is completed.
ABORTINGThe import is responding to a vip import media abort request.
ABORTEDThe import is aborted.
FAILED**The import failed.

**The import can be in a FAILED status for a variety of reasons. Review status output, error logs, limitations, and the import checklist before attempting another media import.

Import states in the Audit Log

An audit trail of VIP-CLI commands that were executed to start or abort a media import, along with the timestamp of the commands and the VIP user that ran them, is displayed in the Audit Log panel located in the Organization view of the VIP Dashboard.

Last updated: April 18, 2022