Prometheus Monitoring: Complete Observability Guide

2025-10-30 · Pavan Y N · prometheus, monitoring, observability, metrics

Published: October 30, 2025
Reading Time: 25 minutes
Tags: Prometheus, Monitoring, Observability, Metrics, DevOps

# Introduction

Prometheus is an open-source monitoring and alerting toolkit that has become the de facto standard for cloud-native monitoring. With its powerful query language, multi-dimensional data model, and seamless Kubernetes integration, Prometheus enables comprehensive observability for modern infrastructure.

This guide covers Prometheus architecture, PromQL, exporters, alerting, and production deployment patterns.

# What is Prometheus?

Prometheus is a time-series database and monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays results, and triggers alerts when specified conditions are met.

## Key Features

→Multi-dimensional Data Model: Time series identified by metric name and key/value pairs
→Powerful Query Language: PromQL for flexible querying and aggregation
→Pull-based Collection: Scrapes metrics from HTTP endpoints
→Service Discovery: Automatic target discovery for dynamic environments
→Alerting: Built-in alerting with Alertmanager integration
→Visualization: Built-in expression browser and Grafana integration

## Prometheus Architecture

code

┌──────────────────────────────────────────────────┐
│              Prometheus Server                    │
│  ┌────────────┐  ┌──────────┐  ┌──────────────┐ │
│  │  Retrieval │→ │  Storage │→ │  HTTP Server │ │
│  └────────────┘  └──────────┘  └──────────────┘ │
│         ↓              ↓              ↑          │
│  ┌────────────┐  ┌──────────┐  ┌──────────────┐ │
│  │Service Disc│  │   TSDB   │  │   PromQL     │ │
│  └────────────┘  └──────────┘  └──────────────┘ │
└──────────────────────────────────────────────────┘
         ↓                              ↑
    ┌────────┐                    ┌──────────┐
    │Targets │                    │ Grafana  │
    │(Scrape)│                    │(Visualize│
    └────────┘                    └──────────┘
         ↓
┌─────────────────┐
│  Alertmanager   │
│  (Notifications)│
└─────────────────┘

# Installation and Setup

## Docker Installation

code

# Create Prometheus configuration
cat > prometheus.yml <<EOF
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    region: 'us-east-1'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'alerts.yml'
  - 'recording_rules.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '\$1'

  - job_name: 'application'
    static_configs:
      - targets: ['app:8080']
    metrics_path: '/metrics'
    scrape_interval: 10s
EOF

# Run Prometheus
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v prometheus-data:/prometheus \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.enable-lifecycle

## Kubernetes Deployment

code

# prometheus-deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    
    scrape_configs:
      - job_name: 'kubernetes-apiservers'
        kubernetes_sd_configs:
          - role: endpoints
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: default;kubernetes;https
      
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
      
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--storage.tsdb.retention.time=30d'
            - '--web.enable-lifecycle'
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: storage
              mountPath: /prometheus
          resources:
            requests:
              cpu: 500m
              memory: 2Gi
            limits:
              cpu: 2000m
              memory: 4Gi
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: storage
          persistentVolumeClaim:
            claimName: prometheus-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - port: 9090
      targetPort: 9090
  type: ClusterIP

# PromQL (Prometheus Query Language)

## Basic Queries

code

# Instant vector - current value
http_requests_total

# Filter by labels
http_requests_total{job="api", method="GET"}

# Multiple label matchers
http_requests_total{job="api", method=~"GET|POST", status!="500"}

# Range vector - values over time
http_requests_total[5m]

# Rate of increase
rate(http_requests_total[5m])

# Increase over time
increase(http_requests_total[1h])

## Aggregation Operators

code

# Sum across all instances
sum(http_requests_total)

# Sum by job
sum by (job) (http_requests_total)

# Average response time
avg(http_request_duration_seconds)

# Maximum memory usage
max(container_memory_usage_bytes)

# Count number of instances
count(up == 1)

# 95th percentile
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

## Advanced Queries

code

# Request rate per second
sum(rate(http_requests_total[5m])) by (job)

# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m])) 
/ 
sum(rate(http_requests_total[5m])) 
* 100

# CPU usage percentage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
/ 
node_memory_MemTotal_bytes 
* 100

# Disk usage percentage
(node_filesystem_size_bytes - node_filesystem_avail_bytes) 
/ 
node_filesystem_size_bytes 
* 100

# Request latency (p95)
histogram_quantile(0.95, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)
)

# Requests per minute
sum(rate(http_requests_total[1m])) * 60

# Top 5 endpoints by request count
topk(5, sum by (endpoint) (rate(http_requests_total[5m])))

# Predict disk full time (linear regression)
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

# Exporters

## Node Exporter (System Metrics)

code

# Run Node Exporter
docker run -d \
  --name node-exporter \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:latest \
  --path.rootfs=/host

Key Metrics:

→node_cpu_seconds_total: CPU usage
→node_memory_MemAvailable_bytes: Available memory
→node_filesystem_avail_bytes: Disk space
→node_network_receive_bytes_total: Network RX
→node_network_transmit_bytes_total: Network TX

## Application Instrumentation

Node.js Example:

code

const express = require('express');
const promClient = require('prom-client');

const app = express();

// Create a Registry
const register = new promClient.Registry();

// Add default metrics
promClient.collectDefaultMetrics({ register });

// Custom metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

const httpRequestTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

const activeConnections = new promClient.Gauge({
  name: 'active_connections',
  help: 'Number of active connections'
});

// Register metrics
register.registerMetric(httpRequestDuration);
register.registerMetric(httpRequestTotal);
register.registerMetric(activeConnections);

// Middleware to track metrics
app.use((req, res, next) => {
  const start = Date.now();
  
  activeConnections.inc();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const labels = {
      method: req.method,
      route: req.route?.path || req.path,
      status_code: res.statusCode
    };
    
    httpRequestDuration.observe(labels, duration);
    httpRequestTotal.inc(labels);
    activeConnections.dec();
  });
  
  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// Business metrics
const orderTotal = new promClient.Counter({
  name: 'orders_total',
  help: 'Total number of orders',
  labelNames: ['status']
});

const orderValue = new promClient.Histogram({
  name: 'order_value_dollars',
  help: 'Order value in dollars',
  buckets: [10, 50, 100, 500, 1000, 5000]
});

register.registerMetric(orderTotal);
register.registerMetric(orderValue);

app.post('/orders', (req, res) => {
  // Process order
  const order = processOrder(req.body);
  
  // Track metrics
  orderTotal.inc({ status: order.status });
  orderValue.observe(order.value);
  
  res.json(order);
});

app.listen(3000);

# Alerting

## Alert Rules

alerts.yml:

code

groups:
  - name: instance_alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
      
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% (current value: {{ $value }}%)"
      
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90% (current value: {{ $value }}%)"
      
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is below 10% (current value: {{ $value }}%)"
      
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is above 5% (current value: {{ $value }}%)"
      
      - alert: HighLatency
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "95th percentile latency is above 2 seconds (current value: {{ $value }}s)"

  - name: application_alerts
    interval: 30s
    rules:
      - alert: ApplicationDown
        expr: up{job="application"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Application is down"
          description: "Application {{ $labels.instance }} has been down for more than 2 minutes"
      
      - alert: HighRequestRate
        expr: sum(rate(http_requests_total[5m])) > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High request rate"
          description: "Request rate is above 1000 req/s (current value: {{ $value }} req/s)"

## Alertmanager Configuration

code

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical-alerts'
      continue: true
    
    - match:
        severity: warning
      receiver: 'warning-alerts'

receivers:
  - name: 'default'
    slack_configs:
      - channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
  
  - name: 'critical-alerts'
    slack_configs:
      - channel: '#critical-alerts'
        title: '🚨 CRITICAL: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'
    email_configs:
      - to: 'oncall@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'alerts@example.com'
        auth_password: 'password'
  
  - name: 'warning-alerts'
    slack_configs:
      - channel: '#warnings'
        title: '⚠️ WARNING: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

# Recording Rules

code

# recording_rules.yml
groups:
  - name: recording_rules
    interval: 30s
    rules:
      # Pre-calculate expensive queries
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)
      
      - record: job:http_request_duration:p95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
      
      - record: instance:node_cpu:utilization
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
      
      - record: instance:node_memory:utilization
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100
      
      - record: instance:node_disk:utilization
        expr: (node_filesystem_size_bytes - node_filesystem_avail_bytes) / node_filesystem_size_bytes * 100

# STAR Approach: Prometheus Implementation

Situation: Company had limited visibility into application performance, relying on reactive troubleshooting and manual log analysis.

Task: Implement comprehensive monitoring solution with Prometheus to enable proactive issue detection and performance optimization.

Action:

•
Infrastructure Setup:
- →Deployed Prometheus on Kubernetes with HA configuration
- →Set up Thanos for long-term storage and global view
- →Configured service discovery for automatic target detection
- →Implemented federation for multi-cluster monitoring
•
Instrumentation:
- →Added Prometheus client libraries to all applications
- →Instrumented 50+ microservices with custom metrics
- →Deployed exporters for infrastructure monitoring
- →Created business metrics for key KPIs
•
Alerting Configuration:
- →Defined 100+ alert rules for various scenarios
- →Set up Alertmanager with multi-channel notifications
- →Implemented alert routing based on severity
- →Created runbooks for common alerts
•
Visualization:
- →Built 30+ Grafana dashboards
- →Created service-level dashboards for each team
- →Implemented SLO tracking dashboards
- →Set up executive dashboards for business metrics
•
Optimization:
- →Implemented recording rules for expensive queries
- →Optimized scrape intervals based on metric importance
- →Set up retention policies for cost optimization
- →Added metric relabeling for cardinality control

Result:

→Mean time to detect (MTTD) reduced from hours to minutes
→Proactive issue detection increased by 80%
→Reduced production incidents by 60%
→Improved application performance by 40%
→Enabled data-driven capacity planning
→Reduced troubleshooting time by 70%

# Best Practices

## 1. Metric Design

→Use meaningful metric names
→Follow naming conventions (unit suffixes)
→Avoid high cardinality labels
→Use histograms for latency metrics

## 2. Query Optimization

→Use recording rules for expensive queries
→Limit time ranges in queries
→Use appropriate aggregation functions
→Avoid regex when possible

## 3. Alerting

→Alert on symptoms, not causes
→Set appropriate thresholds
→Include actionable information
→Avoid alert fatigue

## 4. Resource Management

→Set retention policies
→Monitor Prometheus resource usage
→Use remote storage for long-term data
→Implement metric relabeling

## 5. Security

→Enable authentication
→Use TLS for communication
→Implement network policies
→Regular security updates

# Conclusion

Prometheus has revolutionized monitoring by providing a powerful, flexible, and scalable solution for modern infrastructure. Its pull-based model, powerful query language, and extensive ecosystem make it the ideal choice for cloud-native monitoring.

Whether monitoring a handful of services or thousands of containers, Prometheus provides the tools needed to gain deep insights into system behavior and ensure reliability.

# Additional Resources

Ready to implement Prometheus? Start with basic metrics and gradually add more sophisticated monitoring as you learn!