Deployment validation for Node.js applications

Harmonia is a Node.js deployment validation testing framework developed by WP VIP to identify issues that could prevent application code from building, deploying, or working as expected before that code is deployed to a production environment.

Two methods of pre-deploy testing and validation powered by Harmonia are available:

• Pull requests: When enabled, an automated suite of tests is run by Harmonia against each pull request made to a branch that deploys to a Node.js production environment, or when a new commit is pushed to a non-production branch.
  To opt in, customers must submit a VIP Support request and indicate which environment(s) should have Harmonia enabled.
• VIP-CLI: Validate a Node.js codebase on a user’s local machine. The VIP-CLI command vip validate preflight can be run from within the local codebase directory, or by passing the location of the directory on the local machine with the --path option. By targeting a specific application and environment in the command, the tests will run with settings that are identical to the targeted environment.

Limitations

Settings for Harmonia are not customizable. There are no available methods at this time to configure Harmonia to ignore failing tests that report false positives for an application.

Execution of tests and the Harmonia build process will not work as expected in pull requests and with VIP-CLI for:

• Applications that depend on a containerized service (e.g. Redis). Harmonia currently only instantiates the main Node.js application container.
• Applications that rely on a third-party service that might limit incoming requests on a specific subnet or IP address.

Execution of tests and the Harmonia build process will not work as expected in pull requests for:

• Applications with private dependencies (e.g. private npm modules or private git submodules).
• Deploying branches that have continuous integration and deployment (CI/CD) enabled. By default, when enabled, Harmonia automatically tests pull requests made to a deploying branch. The deploying branch of an environment with automated build and deploy enabled is the *-built version of the main branch (e.g., main, main-built). Harmonia tests will run in the *-built branch instead of the main branch, and feedback will only be visible in the Github commit status.

Individual tests

Individual Harmonia tests are run against code to check for specific issues. Issues identified by each test are reported based on severity level.

Notice

Issues reported as Notices by tests are intended to provide helpful insights into the test flow and opportunities for code quality improvement.

Warning

Issues that contribute to a lower quality of performance for an application, and may cause additional problems if left unchecked. Warning issue types found by tests will not cause tests to fail, but the test will be marked as a partial success.

Examples of issues that are reported as Warnings:

• Redirects (3XX) and client errors (4XX) on home page health checks
• Cache health check has a delayed response (>100ms but <200ms)
• Wrong node.engines version
• Using npm serve instead of npm start

Error

Issues that impact an application’s functionality, but do not prevent a successful deploy and start-up of the application. Any test that throws at least one Error will cause the final test result to be Failed.

Examples of issues that are reported as Errors:

• /cache-healthcheck? endpoint takes too long to respond (>200ms)
• 5xx status codes returned for home page health checks
• Timeouts during home page health checks

Blocker

An issue that is severe enough to prevent an application from deploying, such as a build error, a starting error, or not responding to the health-check endpoint. Tests that identify an issue of this type will immediately be aborted, and no further tests will be executed. 

Examples of issues that are reported as Blockers:

• start and build scripts not found in package.json
• Docker build fails
• Application does not respond to the /cache-healthcheck? endpoint

Test Suites

A Test Suite is a group of individual tests. Any issue identified by an individual test (e.g., Notice, Warning, Error, Blocker) will affect the final result of the Test Suite it belongs to.

• NPM package.json Test Suite: Tests basic settings such as the presence of npm start and npm build scripts in application code, and some package.json settings that could cause issues.
• Docker Test Suite: Builds and executes the Docker image, and performs a basic health-check to the required /cache-healthcheck? endpoint.
• Health Check Test Suite: Executes a broader and extendible health-check against a site’s home page, the /cache-healthcheck? endpoint, and a few URLs scraped from the home page.

Final test results

Harmonia’s final test result is based on the summary of issues reported by all tests and Test Suites that completed execution. Application code that receives a final test result of Pass or Pass with warnings is considered to be ready to deploy to a production environment.

• Passed: The application passed all tests, and no issues were reported as Warnings, Errors, or Blockers.
• Passed with warnings: The application passed all tests, however at least one issue was reported as a Warning that should be taken into consideration.
• Failed: The application failed at least one test, or one test has been aborted because of an issue reported as a Blocker.