Istio Access Logs

Overview

The primary source of logs within Istio is derived from the Envoy proxy. As an integral component in the Istio architecture, Envoy functions as the sidecar proxy for microservices. Its logs play a crucial role in offering valuable insights into the traffic dynamics and operational characteristics of the service mesh. Envoy proxy logs encompass information such as traffic metrics, errors, and various operational events, providing a transparent view of communication between services. These logs, which typically feature details about requests and responses, enable comprehensive analysis and diagnostics, contributing to the effective monitoring and optimization of microservices within the Istio deployment.

Log Formats

Istio offers various log formats for Envoy access logs. Log format can be selected on Istio installation by utilizing the IstioOperator Custom Resource Definition (CRD). If the log format is not specified, Istio defaults to using the following format.

[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS%
\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\"
\"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n

Here are few sample log lines of this format:

[2023-11-24T13:39:58.501Z] "DELETE /status/404 HTTP/1.1" 404 - via_upstream - "-" 0 0 1 0 "-" "curl/8.4.0" "9cae0fcf-5171-95d1-bdba-a98b1d772460" "httpbin:8000" "10.244.0.8:80" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.4:48436 10.96.99.71:8000 10.244.0.4:40784 - default
[2023-11-24T13:39:58.596Z] "PUT /status/404 HTTP/1.1" 404 - via_upstream - "-" 0 0 1 1 "-" "curl/8.4.0" "b9077e17-b7c7-95f3-a565-8512299706e7" "httpbin:8000" "10.244.0.8:80" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.4:48436 10.96.99.71:8000 10.244.0.4:40800 - default
[2023-11-24T13:39:58.687Z] "PUT /status/200 HTTP/1.1" 200 - via_upstream - "-" 0 0 1 1 "-" "curl/8.4.0" "8139d00d-72b3-982a-b710-f6fd43bf82d5" "httpbin:8000" "10.244.0.8:80" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.4:45348 10.96.99.71:8000 10.244.0.4:40808 - default
[2023-11-24T13:39:58.806Z] "POST /status/500 HTTP/1.1" 500 - via_upstream - "-" 0 0 3 2 "-" "curl/8.4.0" "21f46c88-ce6d-97aa-b695-b1089a9335b9" "httpbin:8000" "10.244.0.8:80" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.4:45348 10.96.99.71:8000 10.244.0.4:40816 - default
[2023-11-24T13:39:58.887Z] "POST /status/200 HTTP/1.1" 200 - via_upstream - "-" 0 0 1 1 "-" "curl/8.4.0" "640cca62-2ca9-9bd4-b49e-a8e30fba3857" "httpbin:8000" "10.244.0.8:80" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.4:48436 10.96.99.71:8000 10.244.0.4:40832 - default

Parsing Istio Logs

Observo supports transform raw Istio logs into a structured format using Grok parsing. The following is the Grok pattern that enables this conversion:

\[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:(?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?)|%{DATA:full_uri}) %{DATA:protocol}\" %{NUMBER:status_code} %{DATA:response_flags} %{DATA:response_details} %{DATA:termination_details} \"%{DATA:upstream_failure_reason}\" %{NUMBER:bytes_received} %{NUMBER:bytes_sent} %{NUMBER:duration} (?:%{NUMBER:upstream_service_time}|%{DATA:tcp_service_time}) \"%{DATA:forwarded_for}\" \"%{DATA:user_agent}\" \"%{DATA:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_service}\" %{DATA:upstream_cluster} (?:(%{IP:upstream_local_ip}:%{INT:upstream_local_port})|%{DATA:upstream_local}) %{DATA:downstream_local} (?:(%{IP:downstream_remote_ip}:%{INT:downstream_remote_port})|%{DATA:downstream_remote}) %{DATA:requested_server}(?: %{DATA:route_name})?$

Adding Grok Parser for Istio log using Observo

After parsing, the logs will appear as follows:

{
  "timestamp":"2023-11-22T06:12:51.145Z",
  "method":"GET",
  "uri_path":"/status/418",
  "protocol":"HTTP/1.1",
  "status_code":418,
  "response_flags":"-",
  "response_details":"via_upstream",
  "termination_details":"-",
  "upstream_failure_reason":"-",
  "bytes_received":0,
  "bytes_sent":135,
  "duration":8,
  "upstream_service_time":7,
  "forwarded_for":"-",
  "user_agent":"curl/8.4.0",
  "request_id":"a011dfc4-d641-91a4-98a8-eb7b70aabcbd",
  "authority":"httpbin:8000",
  "upstream_service":"10.244.0.3:80",
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "upstream_local_ip":"10.244.0.5",
  "upstream_local_port":35620,
  "downstream_local":"10.96.99.71:8000",
  "downstream_remote_ip":"10.244.0.5",
  "downstream_remote_port":33046,
  "requested_server":"-",
  "route_name":"default"
}

Additional information about the log fields can be found here.

Aggregate Repetitive Success Logs

A significant portion of the log volume often originates from a limited number of hosts, and these hosts typically generate bursts of events within a short timeframe. By employing Observo's Reduce transform, we can consolidate redundant Success logs generated from the same source within a short time span.

Reduce Conditions:

Field Name: "status_code"
Regular Expression Condition: "200"

Field Name: "upstream_failure_reason"
Regular Expression Condition: "-"

Group By:

downstream_local
downstream_remote_ip
upstream_cluster
upstream_local_ip
upstream_service
uri_path
method

Reduction Methods:

upstream_local_port: Flatted array of unique values
downstream_remote_port: Flatted array of unique values
request_id: Keep last value
bytes_received: Sum values
bytes_sent: Sum values