Monitoring with Prometheus and Grafana

The Prometheus Pull Model Architecture

Prometheus operates primarily on a pull-based architecture, which fundamentally differs from traditional push-based monitoring systems. In this model, the Prometheus server acts as an active crawler that reaches out to specific HTTP endpoints known as targets to scrape metrics at defined intervals. This design is highly beneficial in dynamic environments because the Prometheus server does not need to know the location of every service instance in advance; it simply requires a service discovery mechanism to populate the list of active targets. By decoupling the collection mechanism from the application logic, the application remains lightweight, focusing solely on exposing a formatted metric endpoint rather than managing complex reporting state. If a target service goes offline, Prometheus naturally detects the missing data point, facilitating immediate alerting. This approach ensures that the central monitoring system dictates the collection frequency, protecting the network from becoming flooded by bursts of uncoordinated pushes from dozens of independent containers.

# A minimal prometheus.yml config to define scrape intervals
global:
  scrape_interval: 15s # Prometheus pulls data every 15 seconds
scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9090'] # Scrapes its own metrics

Exposing Metrics with Exporters

Since applications and infrastructure components like containers or kernels do not natively report metrics in the format Prometheus expects, we utilize 'exporters'. An exporter is a small sidecar process or independent binary that sits alongside the primary service to translate internal performance statistics into a plain-text format that the Prometheus scraper can parse. The reason for this modular design is to maintain a separation of concerns: your core application should not contain the logic for metric management or network transport. Instead, the exporter gathers low-level OS data or application-specific counters and maps them into the Prometheus time-series format. This format is essentially a series of key-value pairs representing labels and numeric values. By using exporters, you can monitor virtually anything—from disk I/O and memory saturation to application request latency—without modifying the source code of the underlying services. This consistency is what allows Prometheus to treat a database metric and a container CPU metric with the same uniform query language.

# Example of a simple Go-based metrics exposure pattern
# This handler provides the endpoint that Prometheus scrapes
func prometheusHandler(w http.ResponseWriter, r *http.Request) {
    // Expose counter metrics in the standard text format
    fmt.Fprintf(w, "http_requests_total{method='GET'} 1024\n")
}

PromQL: Querying Time Series Data

The true power of Prometheus is realized through PromQL, a dedicated functional query language designed specifically for multidimensional time-series data. Unlike standard relational databases that focus on rows and tables, PromQL treats data as a continuous stream of numeric samples associated with a timestamp and a set of labels. Labels are the most critical component here; they allow you to filter and aggregate metrics across multiple dimensions simultaneously. For example, if you want to track error rates, you can filter by service name, environment, or specific container ID. PromQL allows you to perform rate calculations, averages, and offsets over time windows. The reasoning behind this complexity is that infrastructure events are rarely point-in-time incidents; they are trends. By calculating the 'rate' of events over a five-minute window, you transform raw, noisy counter data into meaningful insights, such as requests per second, which effectively smooths out jitter and helps identify legitimate performance degradation versus transient network spikes.

# PromQL query to find request rate per second averaged over 5m
rate(http_requests_total[5m])

Visualization with Grafana Dashboards

While Prometheus excels at storing and querying metrics, it is not optimized for visual presentation. This is where Grafana enters the architecture, serving as the front-end dashboarding engine that connects to Prometheus as a data source. Grafana does not store the data itself; it executes PromQL queries against the Prometheus API and renders the resulting data points into graphical formats like line charts, heatmaps, and gauges. The primary reason for this architectural split is flexibility. Grafana provides a highly customizable interface that allows developers to create hierarchical views of the system, enabling the correlation of high-level metrics (e.g., total system throughput) with low-level metrics (e.g., disk latency) on a single screen. By centralizing visualization, you create a shared mental model for the engineering team. When an incident occurs, a well-constructed Grafana dashboard allows for rapid navigation through layers of abstraction, moving from the alert level down to the specific container causing the contention.

# Representing a dashboard panel configuration in JSON format
{
  "title": "CPU Usage",
  "targets": [{"expr": "rate(container_cpu_usage_seconds_total[1m])"}]
}

Alerting Rules and Operational Maturity

The final stage of a monitoring pipeline is alerting, which turns your collected data into active notifications for on-call engineers. In Prometheus, alerting rules are defined using the same PromQL expressions used for querying. When an expression evaluates to true, an alert is triggered and forwarded to the Alertmanager service. The Alertmanager handles the routing, grouping, and silencing of these alerts, preventing the 'alert fatigue' that occurs when hundreds of individual containers fail and each generates a separate notification. By grouping alerts based on common labels—such as 'cluster' or 'application'—Alertmanager ensures that the human operator receives a consolidated report rather than a stream of repetitive noise. This maturity model is essential for high-availability systems; it transforms the monitoring setup from a passive reporting tool into an active, intelligent notification system that proactively highlights the underlying cause of an issue before it impacts end-users significantly.

# Defining a Prometheus alert rule to trigger on high latency
alerts:
  - alert: HighLatency
    expr: http_request_duration_seconds > 0.5
    for: 2m # Only alert if condition holds for 2 minutes

Interview questions

All Docker & Kubernetes interview questions →

Check yourself

Take the full Docker & Kubernetes quiz →