After creating a new Kubernetes cluster, an important task to do is to implement log rotation. Otherwise, the disk space may run out and affect the health of the cluster. We encountered an issue related to log rotation. We were using ingress controller to proxy requests. But the access log of Nginx is enabled by default, which generated lots of log and eventually consumed all disk space.
According to Logging Architecture in Kubernetes doc, if the cluster is created with the
kube-up.sh script, logrotate tool is configured by default. If the cluster is created using other tools, like rke, then log rotation should be configured manually.
My solution is to configure
logrotate using Ansible. There is a
logrotate playbook in Ansible Galaxy, which provides support for configuring
logrotate. The Ansible playbook is very simple, which installs
logrotate and copy configuration files to
logrotate configuration directory
/etc/logrotate.d/. Configuration files are generated using
The following YAML file shows the
logrotate configurations for Kubernetes cluster to be used with Ansible playbook. The configurations were taken from the Kubernetes default setup script, see here.
--- logrotate_scripts: - name: allvarlogs path: /var/log/*.log options: - rotate 5 - daily - dateext - maxsize 25M - missingok - compress - copytruncate - dateformat -%Y%m%d-%s - create 0644 root root - name: allpodlogs path: /var/lib/docker/containers/*/*.log options: - rotate 5 - daily - copytruncate - missingok - notifempty - compress - maxsize 5M - dateext - dateformat -%Y%m%d-%s - create 0644 root root
One thing to note is that the setup script uses
/var/log/pods/*/*.log as the file path for pod logs. However, this doesn't work because
/var/log/pods/*/*.log files are symbolic links and
logrotate doesn't read them due to security risks. So pod logs use the path