HTTP request Log Shipping
VIP’s Log Shipping automatically saves HTTP request logs to an Amazon Web Services S3 bucket at 5-minute intervals. The logs are then available for storage, process, or analysis. Logs are an important asset for understanding the use of an application’s system, connectivity issues, performance tuning, usage patterns, and in analyzing service interruptions.
HTTP (web) request logs show data about each request made to our edge network of servers. HTTP request Log Shipping can be enabled in the VIP Dashboard.
Log contents
The log files are written as a series of gzipped JSON files. Here is a sample record:
{
"client_site_id": "000",
"remote_user": "",
"request_url": "/",
"wplogin": "-",
"timestamp": "19/May/2020:17:03:58 +0000",
"request_type": "GET",
"scheme": "https",
"http_referer": "https://example/",
"http_x_forwarded_for": "",
"true_client_ip": "",
"remote_addr": "REDACTED",
"tls_version": "TLSv1.3",
"content_type": "text/html; charset=UTF-8",
"upstream_country_code": "GB",
"sent_cache_control": "max-age=300, must-revalidate",
"timestamp_iso8601": "2020-05-19T17:03:58+00:00",
"sent_vary": "Accept-Encoding",
"sent_x_cache": "hit",
"request_id": "81cd24c5c46d4c1aa0205dd8001cdba6",
"request_time": "0.001",
"http_host": "example.com",
"http_accept_language": "en-US,en;q=0.9",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36",
"http_version": "HTTP/2.0",
"body_bytes_sent": "8981",
"status": "200"
}
Description of fields
body_bytes_sent
— total number of bytes sent to the client
client_site_id
— an internal ID unique to this environment
content_type
— the media type of the resource, e.g. text/html; charset=UTF-8
http_host
— the domain, e.g. example.com
http_accept_language
— the contents of the Accept-Language
request HTTP header
http_user_agent
— the contents of the User-Agent
request header
http_version
— HTTP protocol version
http_referer
— the Referer request header, if available, containing the purported address of the web page from which a link to the currently requested page was followed
http_x_forwarded_for
— a header that is a means of logging a client’s originating IP address
remote_user
— the username if the request was authenticated with HTTP Basic Authentication (the password is not logged)
request_id
— unique request identifier, this value is set in the X-Request-ID
HTTP request header which can be read by a WordPress or Node.js application
request_url
— the path of the resources that was fetched, not including elements that are included elsewhere, e.g. the protocol (e.g. http://, see ‘scheme’), and the domain (e.g. example.com
, see http_host
)
request_time
— the time taken for the request
request_type
— the HTTP method
sent_cache_control
— the contents of the Cache-Control
HTTP response header
sent_x_cache
— a header from the VIP platform indicating whether the response was from a cache hit
, miss
, or pass
scheme
— either http
or https
sent_vary
— The contents of the Vary HTTP response header; note that VIP does not allow free use of the Vary header (e.g. Accept-Encoding
)
status
— the HTTP response status code, e.g. 200, 404, etc.
timestamp
— UTC date and time of request
timestamp_iso8601
— UTC date and time of request in ISO format
true_client_ip
— a request header commonly set by reverse proxies, including Cloudflare, to indicate the remote address of the client they are forwarding requests for (see also: http_x_forwarded_for
)
remote_addr
— IP address of the client making the request (see also: true_client_ip
and http_x_forwarded_for
)
tls_version
— TLS version used by the client
upstream_country_code
— all requests are geocoded by country at the edge of the VIP CDN using the incoming IP address, e.g. “US”, “GB”, etc.
wplogin
— the login name (i.e. user_login
) of the authenticated WordPress user, if any; requests where there is no authenticated WordPress user this field will contain -
Using Log Shipping data
The JSON formatted log files are human-readable, but the benefits of analyzing logged data are only available by ingesting them into another service.
Some available services for ingesting Log Shipping data, depending on the use cases:
- GoAccess · visualize and analyze (see How To: Analyze HTTP request logs with GoAccess)
- ELK (Elasticsearch, Logstash, Kibana) · filter and view
- Splunk · search, monitor, and analyze
- Data Dog · understand development issues
- Botify · reveal SEO issues
Last updated: February 03, 2023