Tuesday, 8 May 2018

Kubernetes Log Shipping to Elasticsearch



By Lokesh Jawane

As we all know while setting up the infrastructure there a lot of tasks involved - configuration, managing the data and files, searching the logs, troubleshooting ,debugging etc.If you have a large and complex infrastructure you have to ensure that your data gets stored properly. This ensures better data filter and analysis which helps in easy troubleshooting and make your infra environment stable.

Let us now talk about Kubernetes. Generally when its comes to Kubernetes we talk about testing, monitoring and configuration management. Let's look at how to collect data about Kunernetes for data/log analysis.
FileBeat version 6.0.0 or later has added processor add_kubernetes_metadata which allows to gather the k8s container logs and send it to Elasticsearch.

PROCESS
add_kubernetes_metadata enriches logs with metadata from the source container, it adds pod name, container name, and image, Kubernetes labels and, optionally, annotations. It works by watching Kubernetes API for pod events to build a local cache of running containers. When a new log line is read, it gets enriched with metadata from the local cache
Configuration with Elasticsearch & Kibana


It’s great  if you have the ElasticSearch & Kibana inplace, if not then not to worry, just follow the below link to setup it.
Note: make sure you have configured ElasticSearch Basic auth for elastic, kibana,logstash users

Now connect to the K8s workstation(kubectl) and download the manifest from : https://raw.githubusercontent.com/elastic/beats/6.0/deploy/kubernetes/filebeat-kubernetes.yaml

Change the auth details in manifest
# Update Elasticsearch connection details
- name: ELASTICSEARCH_HOST
 value: elasticsearch
- name: ELASTICSEARCH_PORT
 value: “9200”
- name: ELASTICSEARCH_USERNAME
 value: elastic
- name: ELASTICSEARCH_PASSWORD
 value: your elastic use password

Now deploy the daemonset using updates manifest.
kubectl create -f filebeat-kubernetes.yaml

Now got to the Kibana Dashboard & configure the index with filebeat-* pattern and within a minute you should see the Kubernetes containers logs.

Cheers!



9 comments:

  1. Nonetheless, public cloud suppliers should adhere to strict compliance protocols and may implement and keep a lot greater safety ranges than on-premise installations as a result of they've extra obtainable assets.This is great blog. If you want to know more about this visit here AWS Cloud Certified.

    ReplyDelete
    Replies
    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. It's so nice article thank you for sharing a valuable content. google cloud online course

    ReplyDelete
  3. You are not allowed to ship certain items to Brazil. For example, you cannot ship arms, ammunition or dangerous objects to Brazil. https://www.atlanticway.ru/таможенные-склады-в-литве/

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Learn Amazon Web Services for excellent job opportunities from Infycle Technologies, the Excellent AWS Training in Chennai. Infycle Technologies gives the most trustworthy AWS course in Chennai, with full hands-on practical training from professional trainers in the field. Along with that, the placement interviews will be arranged for the candidates, so that, they can meet the job interviews without missing them. To transform your career to the next level, call 7502633633 to Infycle Technologies and grab a free demo to know more

    ReplyDelete

Amazon EKS - Kubernetes on AWS

By Komal Devgaonkar Amazon Elastic Container Service for Kubernetes (Amazon EKS), which is highly available and scalable AWS service....