Rohan Balgotra
Grafana Loki & Promtail: Log Management

Grafana Loki & Promtail: Log Management

#day74 of #90daysofdevops

Rohan Balgotra's photo
Rohan Balgotra
·

6 min read

Table of contents

Introduction:)

Welcome to #Day74 of #90DaysOfDevOps! In the previous blog, we explored the installation of Grafana on AWS EC2, laying the foundation for comprehensive monitoring. Today, we're delving into the powerful world of Grafana Loki – a horizontally-scalable log aggregation system, and its seamless integration with Promtail. Get ready to unlock efficient log management and visualization in this step-by-step guide.

What is Grafana Loki?

Grafana Loki is a horizontally-scalable, highly-available, and multi-tenant log aggregation system inspired by Prometheus. It prioritizes cost-effectiveness and ease of operation, designed to index only metadata for logs while leaving original log messages unindexed. This approach makes Loki more cost-effective and efficient.

The Loki project, initiated at Grafana Labs in 2018, operates under the AGPLv3 license.

Benefits of Loki:

  • Indexing only metadata, reducing costs and enhancing efficiency.

  • Cost-effectiveness with storage on object stores like S3 or locally.

  • Multi-tenancy support for storing data separately for each tenant.

  • Optimized for both local/small-scale and horizontal scaling.

Why use Grafana Loki?

  • Easy setup with log acceptance in any format from any source.

  • 100% persistence to object storage for petabyte-scale, high-throughput, and cost-effective storage.

  • Build metrics and alerts directly from log lines.

  • No log formatting requirements during ingestion for enhanced flexibility.

  • Real-time log tailing, log updates, viewing logs for specific dates, etc.

  • Seamless integration with Prometheus, Grafana, and Kubernetes for unified monitoring.

How does Grafana Loki work?

  1. Pull in any logs with Promtail:

    • Promtail, a dedicated logs collector, collects logs using the same service discovery as Prometheus.

  2. Store the logs in Loki:

    • Loki groups entries into streams and indexes them with labels without indexing the text of logs, resulting in cost reduction and rapid query availability.

  3. Use LogQL to explore:

    • Utilize Loki's query language, LogQL, to explore logs directly within Grafana or through LogCLI for command-line enthusiasts.

  4. Alert on your logs:

    • Set up alerting rules in Loki to evaluate incoming log data and route alerts to the appropriate team.

What is Promtail?

Promtail is an agent that ships local log contents to Grafana Loki. It discovers targets, attaches labels to log streams, and pushes them to Loki.

Benefits of Promtail:

  1. Automatic Log Discovery:

    • Finds and collects logs from various sources without manual intervention, simplifying the log collection process.

  2. Flexible Labeling:

    • Allows dynamic labeling of log entries, offering flexibility in categorizing logs based on content or source.

  3. Smooth Integration with Loki:

    • Seamlessly integrates with Grafana Loki, ensuring easy communication and efficient log data transfer.

  4. Scalability and Efficiency:

    • Scales horizontally and optimizes resource usage by collecting logs only from specific targets, reducing unnecessary workload.

  5. Contribution to Unified Monitoring:

    • Works harmoniously with Loki, Prometheus, and Grafana, contributing to a unified monitoring stack for comprehensive system observability.

How Loki and Promtail Work Together:

  • Data Collection:

    • Applications and services generate log data.

    • Promtail collects log data, attaches labels, and sends it to Loki.

  • Log Data Transformation:

    • Promtail adds labels to log entries for organization and searchability.

  • Data Ingestion into Loki:

    • Promtail sends labeled log data to Loki's ingestion service.

  • Log Indexing and Storage:

    • Loki indexes and stores log entries efficiently for queries.

  • Query and Visualization:

    • Users can query log data using LogQL within Grafana, creating dashboards for visualization.

Key Advantages of Loki and Promtail Integration:

  • Dynamic Labeling: Promtail allows dynamic labeling, enabling flexible categorization of log streams.

  • Efficient Storage: Loki's indexing approach reduces storage costs and ensures quick query availability.

  • Unified Monitoring: Seamless integration with Grafana and Prometheus provides a unified platform for metrics, logs, and traces.

  • Scalability: The combined architecture of Loki and Promtail is designed to scale horizontally, catering to growing log volumes.

Installing Loki and Promtail using Docker

Lets Do This Stan Marsh GIF - Lets Do This Stan Marsh South Park GIFs

Step 1: Install Docker on AWS EC2

Ensure Docker is installed on your AWS EC2 instance by executing the following commands:

sudo apt update  # Update the system
sudo apt install docker.io -y  # Install docker.io service

sudo systemctl start docker  # Start the docker service
sudo systemctl enable docker  # Enable the docker service
sudo systemctl status docker  # Check the status of the docker service

sudo usermod -aG docker $USER  # Give permissions to the user over docker
sudo reboot  # Reboot the system to apply the changes

Step 2: Create a Directory for Configurations

Create a dedicated directory to store configurations for Loki and Promtail. This maintains a clean workspace:

mkdir loki-promtail-config
cd loki-promtail-config

Step 3: Download Configurations

Download configurations for Loki and Promtail using the provided cheatsheet commands:

curl -O https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml
curl -O https://raw.githubusercontent.com/grafana/loki/main/cmd/promtail/promtail-docker-config.yaml

Step 4: Run Loki and Promtail Containers

Execute commands below to run Loki and Promtail containers:

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

docker run -d --name promtail -v $(pwd):/etc/promtail -v /var/log:/var/log grafana/promtail:latest -config.file=/etc/promtail/promtail-docker-config.yaml

Step 5: Check Container Status

Verify if the containers are running:

docker ps

Step 6: Open Grafana and Configure Data Source

  • Open Grafana in your browser. ( localhost:3000 )

  • Navigate to the data sources section.

  • Choose Loki from the options available.

  • Enter the connection URL as "localhost:3100/".

    If you want to check loki is install. Just to the browser and paste the "LocalHost:3100/ready" URL

    Just simply paste the URL of the loki on the data source

  • Keep the remaining settings as default, scroll down, and click "Save and test" to verify the connection.

Step 7: Explore Logs in Grafana

  • Click on the "Explore view" link.

  • Under 'Label filters,' select 'job' and 'varlogs'.

  • Click "Run query" to see all system logs displayed on the dashboard.

Step 8: Create Dashboard Panel

  • Click 'Add to Dashboard' to create a new dashboard panel.

  • Optionally, give the panel a custom name.

Funny Dance Dance GIF - Funny Dance Dance Funny Guy GIFs

Conclusion:)

We've successfully set up Loki and Promtail using Docker, integrated them with Grafana, and created a dashboard to monitor system logs.

This is the cheatsheet for the above.

Dancing Funny GIF - Dancing Funny Funny Dance GIFs

Connect with me:)

Thank you for diving into this blog with me! I trust you found the information both helpful and enlightening. To stay updated on the latest in DevOps 🚀, make sure to follow me. Remember, staying informed means staying ahead in the dynamic world of DevOps!

Feel free to connect with me on:

LinkedIn

Twitter

GitHub

For more updates and engaging discussions on DevOps, let's connect! 🚀 #DevOpsCommunity

GrafanaGrafana dashboardGrafana loki#90daysofdevopsTrainWithShubhamDevops