Tuesday, 8 May 2018

Kubernetes Log Shipping to Elasticsearch



By Lokesh Jawane

As we all know while setting up the infrastructure there a lot of tasks involved - configuration, managing the data and files, searching the logs, troubleshooting ,debugging etc.If you have a large and complex infrastructure you have to ensure that your data gets stored properly. This ensures better data filter and analysis which helps in easy troubleshooting and make your infra environment stable.

Let us now talk about Kubernetes. Generally when its comes to Kubernetes we talk about testing, monitoring and configuration management. Let's look at how to collect data about Kunernetes for data/log analysis.
FileBeat version 6.0.0 or later has added processor add_kubernetes_metadata which allows to gather the k8s container logs and send it to Elasticsearch.

PROCESS
add_kubernetes_metadata enriches logs with metadata from the source container, it adds pod name, container name, and image, Kubernetes labels and, optionally, annotations. It works by watching Kubernetes API for pod events to build a local cache of running containers. When a new log line is read, it gets enriched with metadata from the local cache
Configuration with Elasticsearch & Kibana


It’s great  if you have the ElasticSearch & Kibana inplace, if not then not to worry, just follow the below link to setup it.
Note: make sure you have configured ElasticSearch Basic auth for elastic, kibana,logstash users

Now connect to the K8s workstation(kubectl) and download the manifest from : https://raw.githubusercontent.com/elastic/beats/6.0/deploy/kubernetes/filebeat-kubernetes.yaml

Change the auth details in manifest
# Update Elasticsearch connection details
- name: ELASTICSEARCH_HOST
 value: elasticsearch
- name: ELASTICSEARCH_PORT
 value: “9200”
- name: ELASTICSEARCH_USERNAME
 value: elastic
- name: ELASTICSEARCH_PASSWORD
 value: your elastic use password

Now deploy the daemonset using updates manifest.
kubectl create -f filebeat-kubernetes.yaml

Now got to the Kibana Dashboard & configure the index with filebeat-* pattern and within a minute you should see the Kubernetes containers logs.

Cheers!



6 comments:

  1. Nonetheless, public cloud suppliers should adhere to strict compliance protocols and may implement and keep a lot greater safety ranges than on-premise installations as a result of they've extra obtainable assets.This is great blog. If you want to know more about this visit here AWS Cloud Certified.

    ReplyDelete
  2. It's so nice article thank you for sharing a valuable content. google cloud online course

    ReplyDelete
  3. You are not allowed to ship certain items to Brazil. For example, you cannot ship arms, ammunition or dangerous objects to Brazil. https://www.atlanticway.ru/таможенные-склады-в-литве/

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete

Amazon EKS - Kubernetes on AWS

By Komal Devgaonkar Amazon Elastic Container Service for Kubernetes (Amazon EKS), which is highly available and scalable AWS service....