Grafana Dashboards: Visualization and Alerting

2025-10-30 · Pavan Y N · grafana, dashboards, visualization, alerting

Published: October 30, 2025
Reading Time: 21 minutes
Tags: Grafana, Dashboards, Visualization, Alerting, Monitoring

# Introduction

Grafana is the leading open-source platform for monitoring and observability, providing beautiful dashboards and powerful alerting capabilities. This guide covers dashboard creation, data sources, variables, alerting, and building comprehensive monitoring solutions.

# Getting Started with Grafana

## Installation

Docker:

code

docker run -d \
  --name=grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  -e "GF_SECURITY_ADMIN_PASSWORD=admin" \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel" \
  grafana/grafana:latest

Docker Compose:

code

version: '3.8'

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.example.com
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=smtp.gmail.com:587
      - GF_SMTP_USER=${SMTP_USER}
      - GF_SMTP_PASSWORD=${SMTP_PASSWORD}
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    restart: unless-stopped

volumes:
  grafana-storage:

## Configuration as Code

provisioning/datasources/prometheus.yml:

code

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s
      queryTimeout: 60s
      httpMethod: POST

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false

  - name: CloudWatch
    type: cloudwatch
    jsonData:
      authType: keys
      defaultRegion: us-east-1
    secureJsonData:
      accessKey: ${AWS_ACCESS_KEY_ID}
      secretKey: ${AWS_SECRET_ACCESS_KEY}

provisioning/dashboards/default.yml:

code

apiVersion: 1

providers:
  - name: 'Default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards

# Dashboard Creation

## Basic Dashboard JSON

code

{
  "dashboard": {
    "title": "Application Metrics",
    "tags": ["application", "production"],
    "timezone": "browser",
    "schemaVersion": 38,
    "version": 1,
    "refresh": "30s",
    
    "panels": [
      {
        "id": 1,
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {
          "x": 0,
          "y": 0,
          "w": 12,
          "h": 8
        },
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m])) by (job)",
            "legendFormat": "{{job}}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {
            "format": "reqps",
            "label": "Requests/sec"
          }
        ]
      },
      
      {
        "id": 2,
        "title": "Error Rate",
        "type": "stat",
        "gridPos": {
          "x": 12,
          "y": 0,
          "w": 6,
          "h": 4
        },
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
            "refId": "A"
          }
        ],
        "options": {
          "reduceOptions": {
            "values": false,
            "calcs": ["lastNotNull"]
          },
          "text": {
            "titleSize": 20,
            "valueSize": 40
          },
          "colorMode": "value",
          "graphMode": "area",
          "orientation": "auto"
        },
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {"value": null, "color": "green"},
                {"value": 1, "color": "yellow"},
                {"value": 5, "color": "red"}
              ]
            }
          }
        }
      },
      
      {
        "id": 3,
        "title": "Response Time (p95)",
        "type": "timeseries",
        "gridPos": {
          "x": 0,
          "y": 8,
          "w": 24,
          "h": 8
        },
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))",
            "legendFormat": "{{job}} p95",
            "refId": "A"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))",
            "legendFormat": "{{job}} p99",
            "refId": "B"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "s",
            "custom": {
              "drawStyle": "line",
              "lineInterpolation": "smooth",
              "fillOpacity": 10
            }
          }
        }
      }
    ],
    
    "templating": {
      "list": [
        {
          "name": "datasource",
          "type": "datasource",
          "query": "prometheus"
        },
        {
          "name": "job",
          "type": "query",
          "datasource": "${datasource}",
          "query": "label_values(up, job)",
          "multi": true,
          "includeAll": true
        },
        {
          "name": "instance",
          "type": "query",
          "datasource": "${datasource}",
          "query": "label_values(up{job=~\"$job\"}, instance)",
          "multi": true,
          "includeAll": true
        }
      ]
    },
    
    "annotations": {
      "list": [
        {
          "name": "Deployments",
          "datasource": "Prometheus",
          "expr": "changes(process_start_time_seconds[1m]) > 0",
          "tagKeys": "job,instance",
          "textFormat": "Deployment",
          "iconColor": "blue"
        }
      ]
    }
  }
}

# Advanced Dashboard Features

## Variables

code

{
  "templating": {
    "list": [
      {
        "name": "environment",
        "type": "custom",
        "options": [
          {"text": "Production", "value": "prod"},
          {"text": "Staging", "value": "staging"},
          {"text": "Development", "value": "dev"}
        ],
        "current": {"text": "Production", "value": "prod"}
      },
      
      {
        "name": "namespace",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(kube_pod_info{environment=\"$environment\"}, namespace)",
        "refresh": 1,
        "multi": true,
        "includeAll": true
      },
      
      {
        "name": "interval",
        "type": "interval",
        "options": [
          {"text": "1m", "value": "1m"},
          {"text": "5m", "value": "5m"},
          {"text": "15m", "value": "15m"},
          {"text": "1h", "value": "1h"}
        ],
        "auto": true,
        "auto_count": 30,
        "auto_min": "10s"
      }
    ]
  }
}

## Transformations

code

{
  "transformations": [
    {
      "id": "merge",
      "options": {}
    },
    {
      "id": "organize",
      "options": {
        "excludeByName": {
          "__name__": true,
          "job": false
        },
        "indexByName": {
          "Time": 0,
          "job": 1,
          "Value": 2
        },
        "renameByName": {
          "job": "Service",
          "Value": "Requests/sec"
        }
      }
    },
    {
      "id": "calculateField",
      "options": {
        "mode": "binary",
        "reduce": {
          "reducer": "sum"
        },
        "binary": {
          "left": "Value #A",
          "operator": "/",
          "right": "Value #B"
        },
        "alias": "Success Rate"
      }
    }
  ]
}

# Alerting

## Alert Rules

code

{
  "alert": {
    "name": "High Error Rate",
    "conditions": [
      {
        "evaluator": {
          "params": [5],
          "type": "gt"
        },
        "operator": {
          "type": "and"
        },
        "query": {
          "params": ["A", "5m", "now"]
        },
        "reducer": {
          "params": [],
          "type": "avg"
        },
        "type": "query"
      }
    ],
    "executionErrorState": "alerting",
    "for": "5m",
    "frequency": "1m",
    "handler": 1,
    "message": "Error rate is above 5% for the last 5 minutes",
    "name": "High Error Rate",
    "noDataState": "no_data",
    "notifications": [
      {"uid": "slack-notifications"},
      {"uid": "pagerduty-critical"}
    ]
  }
}

## Notification Channels

Slack:

code

{
  "name": "Slack Alerts",
  "type": "slack",
  "isDefault": false,
  "sendReminder": true,
  "frequency": "15m",
  "settings": {
    "url": "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
    "recipient": "#alerts",
    "username": "Grafana",
    "icon_emoji": ":grafana:",
    "mentionChannel": "here",
    "token": ""
  }
}

PagerDuty:

code

{
  "name": "PagerDuty Critical",
  "type": "pagerduty",
  "isDefault": false,
  "settings": {
    "integrationKey": "YOUR_PAGERDUTY_INTEGRATION_KEY",
    "severity": "critical",
    "autoResolve": true
  }
}

# Complete Dashboard Examples

## Kubernetes Cluster Dashboard

code

{
  "dashboard": {
    "title": "Kubernetes Cluster Overview",
    "panels": [
      {
        "title": "Cluster CPU Usage",
        "targets": [{
          "expr": "sum(rate(container_cpu_usage_seconds_total{container!=\"\"}[5m])) / sum(machine_cpu_cores) * 100"
        }]
      },
      {
        "title": "Cluster Memory Usage",
        "targets": [{
          "expr": "sum(container_memory_working_set_bytes{container!=\"\"}) / sum(machine_memory_bytes) * 100"
        }]
      },
      {
        "title": "Pod Count by Namespace",
        "targets": [{
          "expr": "sum(kube_pod_info) by (namespace)"
        }]
      },
      {
        "title": "Node Status",
        "targets": [{
          "expr": "kube_node_status_condition{condition=\"Ready\",status=\"true\"}"
        }]
      }
    ]
  }
}

# STAR Approach: Grafana Implementation

Situation: Company had limited visibility into application performance with scattered monitoring tools and no centralized dashboards.

Task: Implement comprehensive Grafana monitoring solution with standardized dashboards and alerting.

Action:

•
Infrastructure Setup:
- →Deployed HA Grafana cluster
- →Configured multiple data sources
- →Set up authentication with LDAP
- →Implemented backup strategy
•
Dashboard Development:
- →Created 50+ standardized dashboards
- →Implemented dashboard templates
- →Built service-level dashboards
- →Created executive dashboards
•
Alerting Configuration:
- →Defined 200+ alert rules
- →Set up notification channels
- →Implemented alert routing
- →Created runbooks
•
Automation:
- →Provisioned dashboards as code
- →Automated dashboard deployment
- →Implemented dashboard testing
- →Created dashboard generator
•
Training & Documentation:
- →Conducted training sessions
- →Created comprehensive documentation
- →Built dashboard library
- →Established best practices

Result:

→Centralized monitoring for all services
→Mean time to detect reduced by 70%
→Dashboard creation time reduced from days to hours
→Alert fatigue reduced by 60%
→Improved incident response time by 50%
→Increased system visibility by 90%

# Best Practices

## 1. Dashboard Design

→Keep dashboards focused
→Use consistent layouts
→Implement proper naming
→Add descriptions

## 2. Performance

→Limit panel count
→Optimize queries
→Use appropriate refresh rates
→Implement caching

## 3. Alerting

→Alert on symptoms, not causes
→Set appropriate thresholds
→Avoid alert fatigue
→Include actionable information

## 4. Organization

→Use folders for organization
→Implement tagging
→Version control dashboards
→Regular cleanup

## 5. Security

→Implement RBAC
→Use data source permissions
→Regular access reviews
→Audit logging

# Conclusion

Grafana provides powerful visualization and alerting capabilities that enable comprehensive monitoring solutions. By following best practices and leveraging advanced features, teams can build effective dashboards that provide deep insights into system behavior and enable proactive issue detection.

# Additional Resources

Ready to build Grafana dashboards? Start with basic metrics and gradually add more sophisticated visualizations!