Application logging in a kubernetes cluster is very easy. In this tutorial we will generate a logging concept for applications deployed on a kubernetes cluster. To save the logs and query them we use elasticsearch, for displaying them we use Kibana and for collecting the logs we use fluentd.

fluentd will be deployed as DaeomSet, so it runs on each node in the cluster and collects logs from the directories /var/log and /var/lib/docker/containers. Therefor hostPath volumes are used that are mounted into the fluentd container. Then fluentd will send the logs to elasticsearch where they are stored in the index logstash-* for queries. Kibana is used to display the logs and visualize them.

It is very easy to deploy it, first we need an elasticsearch server deployment and service (please pay attention this is not production grade logging):

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: es
  namespace: logging
  labels:
    app: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
        env:
          - name: "discovery.type"
            value: "single-node"
          - name: "ES_JAVA_OPTS"
            value: "-Xms256m -Xmx256m"
          - name: "xpack.security.enabled"
            value: "false"
        resources:
          requests:
            memory: 300Mi
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  type: LoadBalancer
  selector:
    app: elasticsearch
  ports:
  - name: http
    port: 9200
    protocol: TCP
  - name: transport
    port: 9300
    protocol: TCP

After the elasticsearch cluster is running in the logging namespace we deploy the fluentd service as DaeomSet and connect it to the elasticsearch Database:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

After this fluentd will watch for new logs and send them to the elasticsearch service.

As last component we need the kibana dashboard to visualize the logs:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: kibana-logging
  namespace: logging
  labels:
    app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana-logging
  template:
    metadata:
      labels:
        app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: docker.elastic.co/kibana/kibana:5.6.2
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
          - name: XPACK_MONITORING_ENABLED
            value: "false"
          - name: XPACK_SECURITY_ENABLED
            value: "false"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana-logging
spec:
  type: LoadBalancer
  selector:
    app: kibana-logging
  ports:
  - name: http
    port: 5601
    protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: logging
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: logging.url.de
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

Now there should be a complete logging environment available that can be accessed, in my case via the ingress url.

After this we only need to add the logstash-* index to Kibana, now we should visualize the logs:

One very interesting part is the available fields list, this allows us to query for specific containers or for example for requests made via nginx-ingress.

One problem of this solution is, that the elasticsearch containers are not persisted, if the container fails all data are lost. One solution might be to add persistentVolume to a container like glusterFS. At the same time, it is important to note that if the cluster fails, the logs may not be accessible.

Björn Wenzel

My name is Björn Wenzel. I’m a Platform Engineer working for Schenker with interests in Kubernetes, CI/CD, Spring and NodeJS.

•