Skip to content

HTTP request Log Shipping

VIP’s Log Shipping automatically saves HTTP request logs to an Amazon Web Services S3 bucket at 5-minute intervals. The logs are then available for storage, process, or analysis. Logs are an important asset for understanding the use of an application’s system, connectivity issues, performance tuning, usage patterns, and in analyzing service interruptions.

HTTP (web) request logs show data about each request made to VIP’s edge network of servers for a site. Enable HTTP request Log Shipping in the VIP Dashboard.

Log contents

The log files are written as a series of gzipped JSON files. An example record in a log file:

{
  "client_site_id": "000",
  "remote_user": "",
  "request_url": "/",
  "wplogin": "-",
  "timestamp": "19/May/2020:17:03:58 +0000",
  "request_type": "GET",
  "scheme": "https",
  "http_referer": "https://example.com/",
  "http_x_forwarded_for": "",
  "true_client_ip": "",
  "remote_addr": "127.0.0.1",
  "asn": "2635",
  "tls_version": "TLSv1.3",
  "ssl_client_verify": "NONE",
  "content_type": "text/html; charset=UTF-8",
  "upstream_country_code": "GB",
  "sent_cache_control": "max-age=300, must-revalidate",
  "timestamp_iso8601": "2020-05-19T17:03:58+00:00",
  "sent_vary": "Accept-Encoding",
  "sent_x_cache": "hit",
  "request_id": "81cd24c5c46d4c1aa0205dd8001cdba6",
  "request_time": "0.001",
  "http_host": "example.com",
  "http_accept_language": "en-US,en;q=0.9",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36",
  "http_version": "HTTP/2.0",
  "body_bytes_sent": "8981",
  "status": "200",
  "private_file": ""
}

Description of fields

body_bytes_sent — Total number of bytes sent to the client.

client_site_id — An internal ID unique to this environment.

content_type — The media type of the resource, e.g. text/html; charset=UTF-8.

http_host — The domain, e.g. example.com.

http_accept_language — The contents of the Accept-Language request HTTP header.

http_user_agent — The contents of the User-Agent request header.

http_version — HTTP protocol version.

http_referer — The Referer request header, if available, containing the purported address of the web page from which a link to the currently requested page was followed.

http_x_forwarded_for — A header that is a means of logging a client’s originating IP address.

remote_user — The username if the request was authenticated with HTTP Basic Authentication (the password is not logged).

request_id — Unique request identifier, this value is set in the X-Request-ID HTTP request header which can be read by a WordPress or Node.js application.

request_url — The path of the resources that was fetched, not including elements that are included elsewhere, e.g. the protocol (e.g. http://, see ‘scheme’), and the domain (e.g. example.com, see http_host).

request_time — The time taken for the request.

request_type — The HTTP method.

sent_cache_control — The contents of the Cache-Control HTTP response header.

sent_x_cache — A header from the VIP platform indicating whether the response was from a cache hit, miss, or pass.

scheme — Either http or https.

sent_vary — The contents of the Vary HTTP response header; note that VIP does not allow free use of the Vary header (e.g. Accept-Encoding).

status — The HTTP response status code, e.g. 200, 404, etc.

timestamp — UTC date and time of the request

timestamp_iso8601 — UTC date and time of the request in ISO format

true_client_ip — A request header commonly set by reverse proxies, including Cloudflare, to indicate the remote address of the client they are forwarding requests for (see also: http_x_forwarded_for).

remote_addr — IP address of the client making the request (see also: true_client_ip and http_x_forwarded_for).

asn — the Autonomous System Number (ASN) of the remote_addr IP address.

tls_version — TLS version used by the client.

ssl_client_verify — The result of client certificate verification: "SUCCESS", "FAILED:reason", and "NONE" if a certificate was not present.

upstream_country_code — All requests are geocoded by country at the edge of the VIP CDN using the incoming IP address, e.g. “US”, “GB”, etc.

wplogin — The login name (i.e. user_login) of the authenticated WordPress user, if any; requests where there is no authenticated WordPress user this field will contain -.

private_file — This will be empty for requests that do not include a /wp-content/uploads/ path. A value of 1 indicates a request to a file restricted by Access-controlled files,  and a  0 for all other files.

Using Log Shipping data

The JSON-formatted log files are human-readable, but ingesting the logged data into another service will provide additional analysis and benefit.

Some available services for ingesting Log Shipping data, depending on the use cases:

Last updated: September 19, 2023

Relevant to

  • Node.js
  • WordPress