Rohan Balgotra

Monitor Docker Containers with Grafana

Rohan Balgotra's photo
Rohan Balgotra
·

3 min read

Pre-requisites:

  1. Amazon EC2 Instance

    • Ensure the availability of an Amazon EC2 instance to serve as the host for Docker containers and the monitoring tools.

  2. Grafana, Loki, and Promtail Installed

    • Install Grafana, Loki, and Promtail to set up the monitoring and log aggregation stack on the EC2 instance.

        • Grafana: The visualization and monitoring tool.

          • Loki: The log aggregation system.

          • Promtail: The agent responsible for collecting and forwarding logs to Loki.

  3. If the necessary tools are not installed, follow the step-by-step instructions provided in the referenced blog post for comprehensive guidance.

Task: Monitor Docker Containers with Grafana Dashboard📊

Step 1: Configure Promtail to Collect Docker Logs

  • Edit the Promtail configuration file (usually located at /etc/promtail/config.yml). Add the following configuration to collect Docker logs:

      server:
    http_listen_port: 9080
    grpc_listen_port: 0

  positions:
    filename: /tmp/positions.yaml

  clients:
    - url: http://loki:3100/loki/api/v1/push

  scrape_configs:
    - job_name: system
      static_configs:
        - targets:
            - localhost
          labels:
            job: varlogs
            __path__: /var/log/*log
        - targets:
            - localhost
          labels:
            job: dockerlogs
            __path__: /var/lib/docker/containers/*/*.log

Step 2: Restart Promtail docker container

  • After updating the configuration, restart Promtail container to apply the changes:

      docker restart <promtail-container-ID>

Step 3: Create Docker Containers with a Todo App

  • Pull the Docker image from your Docker Hub repository

      docker pull simbaa815/node-todo

Step 4: Create a Docker network

  • Create a Docker network to facilitate communication between containers. You can use the following command:

      docker network create todo-net

    Using a Docker network is not strictly required, but it is a good practice for creating isolated communication channels between containers.

Step 5: Run two containers

  • Run two containers with a basic todo app, each connected to the previously created network.

      docker run -d --network todo-net --name todo-app-1 your-todo-image
  docker run -d --network todo-net --name todo-app-2 your-todo-image

  • You can check the running containers using:

      docker ps

Step 6: Configure Loki as a Data Source in Grafana

  1. Open Grafana in your browser.

  2. Log in using your credentials.

    Note: If you haven't set up Grafana yet, please refer this blog.

  3. Navigate to "Configuration" > "Data Sources."

  4. Click on "Add your first data source."

  5. Choose "Loki" as the data source type.

  6. Set the URL to your Loki instance (e.g., http://localhost:3100).

  7. Click "Save & Test" to ensure the connection is successful.

Step 5: Create a Dashboard in Grafana

  1. Navigate to the "+" icon on the left sidebar and select "Dashboard."

  2. Click on "Add new panel."

  3. Choose "Loki" as the data source.

  4. Use queries to filter logs based on container name or other criteria.

Step 6: Visualize Real-Time Logs

  1. Customize your dashboard by adding panels and specifying log queries.

  2. You can use LogQL queries to filter and visualize logs based on various parameters.

Step 7: Save and Monitor

Save your dashboard, and now you can monitor real-time logs from your Docker containers in Grafana, leveraging Loki for log aggregation.

Congratulations! You've successfully integrated Docker, Loki, and Promtail into Grafana to monitor and visualize real-time logs from your containers. Adjust the configurations based on your specific requirements.

Grafana