After creating a new Kubernetes cluster, an important task to do is to implement log rotation. Otherwise, the disk space may run out and affect the health of the cluster. We encountered an issue related to log rotation. We were using ingress controller to proxy requests. But the access log of Nginx is enabled by default, which generated lots of log and eventually consumed all disk space.

According to Logging Architecture in Kubernetes doc, if the cluster is created with the kube-up.sh script, logrotate tool is configured by default. If the cluster is created using other tools, like rke, then log rotation should be configured manually.

My solution is to configure logrotate using Ansible. There is a logrotate playbook in Ansible Galaxy, which provides support for configuring logrotate. The Ansible playbook is very simple, which installs logrotate and copy configuration files to logrotate configuration directory /etc/logrotate.d/. Configuration files are generated using logrotate_scripts variable.

The following YAML file shows the logrotate configurations for Kubernetes cluster to be used with Ansible playbook. The configurations were taken from the Kubernetes default setup script, see here.

---
logrotate_scripts:
  - name: allvarlogs
    path: /var/log/*.log
    options:
      - rotate 5
      - daily
      - dateext
      - maxsize 25M
      - missingok
      - compress
      - copytruncate
      - dateformat -%Y%m%d-%s
      - create 0644 root root

  - name: allpodlogs
    path: /var/lib/docker/containers/*/*.log
    options:
      - rotate 5
      - daily
      - copytruncate
      - missingok
      - notifempty
      - compress
      - maxsize 5M
      - dateext
      - dateformat -%Y%m%d-%s
      - create 0644 root root

One thing to note is that the setup script uses /var/log/pods/*/*.log as the file path for pod logs. However, this doesn't work because /var/log/pods/*/*.log files are symbolic links and logrotate doesn't read them due to security risks. So pod logs use the path /var/lib/docker/containers/*/*.log.