How to enable Envoy access logging in Rancher deployed Istio

• Rancher 2.7 / 2.8
• A Kubernetes cluster Managed by Rancher
• Rancher Monitoring Stack (pre-requisite for Istio) and Rancher Istio

This article details how to enable Envoy's access logging , for Rancher deployed Istio, in Rancher.

Configuring Telemetry API is recommended to enable Access logging for Envoy.

To enable access logging, perform the following steps:

1. With a kubeconfig for the Downstream cluster, create the following Telemetry configuration:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: istio-system
   spec:
     accessLogging:
       - providers:
         - name: envoy

   
   The above uses the default envoy access log provider and only the default settings are configured. A similar configuration can also be applied to an individual namespace, or to an individual workload, to control logging at a fine-grained level.  For more information about using the Telemetry API, see the Telemetry API overview.

2. After enabling access logging, you can test the configuration with the Istio sleep and httpbin sample applications, per the Istio documentation.  Some sample logs after enabling Envoy access logs:

   kubectl logs -l app=sleep -c istio-proxy
   
   2024-07-10T07:58:09.579778Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.420226455s
   2024-07-10T07:58:09.580066Z     info    ads     SDS: PUSH request for node:sleep-78ff5975c6-d9zrk.default resources:1 size:1.1kB resource:ROOTCA
   2024-07-10T07:58:09.580236Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.419767231s
   2024-07-10T07:58:09.588579Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.411432603s
   2024-07-10T07:58:10.622873Z     info    Readiness succeeded in 1.46762116s
   2024-07-10T07:58:10.623674Z     info    Envoy proxy is ready
   [2024-07-10T07:58:46.673Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 14 13 "-" "curl/8.8.0" "62ee6bdb-0afe-495c-83e0-37bc131045c3" "httpbin:8000" "10.42.1.24:8080" outbound|8000||httpbin.default.svc.cluster.local 10.42.1.23:44318 10.43.156.59:8000 10.42.1.23:42030 - default
   [2024-07-10T07:58:53.227Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 10 9 "-" "curl/8.8.0" "addf66ef-fb90-43bd-85e0-35763d7f0fca" "httpbin:8000" "10.42.1.24:8080" outbound|8000||httpbin.default.svc.cluster.local 10.42.1.23:57372 10.43.156.59:8000 10.42.1.23:35794 - default
   [2024-07-10T08:01:56.359Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 23 22 "-" "curl/8.8.0" "385df7b2-9e0a-4506-811d-95d15f1da3ee" "httpbin:8000" "10.42.1.24:8080" outbound|8000||httpbin.default.svc.cluster.local 10.42.1.23:60376 10.43.156.59:8000 10.42.1.23:46754 - default
   [2024-07-10T08:01:57.741Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 11 10 "-" "curl/8.8.0" "e0f34346-032a-4af3-a27c-a40f000e61e1" "httpbin:8000" "10.42.1.24:8080" outbound|8000||httpbin.default.svc.cluster.local 10.42.1.23:35960 10.43.156.59:8000 10.42.1.23:48636 - default
   
   kubectl logs -l app=httpbin -c istio-proxy
   
   2024-07-10T07:58:16.693643Z     info    cache   returned workload certificate from cache        ttl=23h59m59.30636378s
   2024-07-10T07:58:16.693942Z     info    ads     SDS: PUSH request for node:httpbin-54b5c865df-pjjhm.default resources:1 size:4.0kB resource:default
   2024-07-10T07:58:17.883782Z     info    Readiness succeeded in 1.645029716s
   2024-07-10T07:58:17.884493Z     info    Envoy proxy is ready
   [2024-07-10T07:58:46.680Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 2 "-" "curl/8.8.0" "62ee6bdb-0afe-495c-83e0-37bc131045c3" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:42839 10.42.1.24:8080 10.42.1.23:44318 outbound_.8000_._.httpbin.default.svc.cluster.local default
   [2024-07-10T07:58:53.229Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 2 2 "-" "curl/8.8.0" "addf66ef-fb90-43bd-85e0-35763d7f0fca" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:54765 10.42.1.24:8080 10.42.1.23:57372 outbound_.8000_._.httpbin.default.svc.cluster.local default
   [2024-07-10T08:00:17.209Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 2 "-" "curl/8.8.0" "0ea867e3-4e8c-4e45-9655-057d08989f1a" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:54011 10.42.1.24:8080 10.42.1.23:57372 outbound_.8000_._.httpbin.default.svc.cluster.local default
   [2024-07-10T08:00:18.779Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 2 2 "-" "curl/8.8.0" "3bedc3ca-345d-4fba-8269-6ce4ecb1e04b" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:38727 10.42.1.24:8080 10.42.1.23:44318 outbound_.8000_._.httpbin.default.svc.cluster.local default
   [2024-07-10T08:01:56.366Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 4 3 "-" "curl/8.8.0" "385df7b2-9e0a-4506-811d-95d15f1da3ee" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:35665 10.42.1.24:8080 10.42.1.23:60376 outbound_.8000_._.httpbin.default.svc.cluster.local default
   [2024-07-10T08:01:57.745Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 2 2 "-" "curl/8.8.0" "e0f34346-032a-4af3-a27c-a40f000e61e1" "httpbin:8000" "10.42.1.24:8080" inbound|8080|| 127.0.0.6:47731 10.42.1.24:8080 10.42.1.23:35960 outbound_.8000_._.httpbin.default.svc.cluster.local default

    

Further reading

• Istio "Getting Envoy's Access Logs" Documentation
• Envoy Access Logging Documentation