Docker chevron_right Day 19: Debugging and Monitoring Docker Applications

Day 19: Debugging and Monitoring Docker Applications

calendar_today schedule 11 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Implement comprehensive logging strategies for containerized applications
  • Set up monitoring and alerting for Docker environments
  • Debug common container and application issues effectively
  • Use debugging tools and techniques for development and production
  • Create observability dashboards for multi-container applications

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

“You wake up at 3 AM to alerts that your containerized application is down. Users are complaining, but all you see are cryptic container exits and scattered log messages. How do you quickly identify the root cause and get everything back online? Today we’ll learn the detective skills and tools needed to solve these mysteries efficiently.”

Lesson Overview

  • Duration: 90 minutes
  • Prerequisites: Docker Compose, multi-container applications, basic Linux/shell knowledge
  • Materials Needed: Docker Desktop, sample applications with issues, monitoring tools

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Logging Strategies - Centralized logging and log management
  2. Monitoring Tools - Container and application metrics collection
  3. Debugging Techniques - Systematic troubleshooting approaches
  4. Observability - Creating comprehensive visibility into your systems
  5. Alerting and Response - Proactive issue detection and notification

Success Criteria:

By the end of this session, you should be able to diagnose container issues, implement monitoring solutions, and create dashboards that provide actionable insights into your application’s health and performance.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

The Three Pillars of Observability

Modern application monitoring relies on three key pillars:

  1. Logs: Detailed records of application events and errors
  2. Metrics: Numerical measurements of system performance
  3. Traces: Request flows through distributed systems

Real-World Analogy: Think of debugging like being a detective. Logs are witness statements, metrics are the crime scene evidence, and traces are the timeline of events. You need all three to solve the case.

Why This Matters: Without proper observability, identifying issues in containerized applications becomes like finding a needle in a haystack, especially in production environments with multiple services.

Container-Specific Challenges

Containerized applications present unique debugging challenges:

  • Ephemeral Nature: Containers can be recreated, losing debugging context
  • Isolation: Limited visibility into container internals
  • Scale: Multiple instances make issue correlation difficult
  • Networking: Complex inter-service communication paths

Debugging Mindset

Effective debugging follows a systematic approach:

  1. Reproduce: Can you consistently recreate the issue?
  2. Isolate: Which component or layer is causing the problem?
  3. Analyze: What does the evidence (logs, metrics) tell you?
  4. Hypothesize: What could be causing this behavior?
  5. Test: Verify your hypothesis with targeted changes

Key Terminology

  • Telemetry: Automated collection of data from remote systems
  • Observability: The ability to understand system behavior from external outputs
  • APM: Application Performance Monitoring tools and practices
  • SLI/SLO: Service Level Indicators and Objectives for reliability
  • MTTR: Mean Time To Recovery - how quickly issues are resolved

4. Detailed Explanation and Examples (25-30 minutes)

Understanding Container Logging

Docker provides several logging mechanisms, each suitable for different scenarios:

Docker Logging Drivers

  • json-file: Default driver, stores logs as JSON files
  • syslog: Forwards logs to syslog daemon
  • journald: Integrates with systemd journal
  • fluentd: Sends logs to Fluentd collector
  • awslogs: Direct integration with AWS CloudWatch

Log Aggregation Patterns

In multi-container environments, centralized logging is essential:

  1. ELK Stack: Elasticsearch, Logstash, and Kibana
  2. EFK Stack: Elasticsearch, Fluentd, and Kibana
  3. Loki Stack: Grafana Loki with Promtail
  4. Cloud Solutions: AWS CloudWatch, Google Cloud Logging, Azure Monitor

Monitoring and Metrics Collection

Container Metrics

Key metrics to monitor in containerized environments:

  • Resource Usage: CPU, memory, disk, network I/O
  • Container Health: Running/stopped states, restart counts
  • Application Metrics: Response times, error rates, throughput
  • Infrastructure Metrics: Host system resources, Docker daemon health
  • Prometheus + Grafana: Open-source monitoring and visualization
  • DataDog: Commercial APM solution
  • New Relic: Application performance monitoring
  • cAdvisor: Container resource usage and performance metrics

Debugging Techniques and Tools

Container Inspection Commands

Essential commands for container debugging:

# Inspect container configuration and state
docker inspect container_name

# View container processes
docker top container_name

# Execute commands inside running containers
docker exec -it container_name /bin/bash

# Copy files to/from containers
docker cp container_name:/path/to/file ./local/path

Network Debugging

Container networking issues are common and can be tricky to diagnose:

# List networks
docker network ls

# Inspect network configuration
docker network inspect network_name

# Test connectivity between containers
docker exec container1 ping container2

Common Misconceptions

  1. Misconception: Container logs are automatically rotated and managed Reality: Log rotation and retention must be configured to prevent disk space issues

  2. Misconception: Monitoring tools automatically understand your application context Reality: Effective monitoring requires custom metrics and alerting rules tailored to your application

5. Live Coding Demonstration (20-25 minutes)

Example 1: Comprehensive Logging Setup

version: '3.8'

services:
  # Application with structured logging
  app:
    build: ./app
    networks:
      - app-network
    environment:
      - LOG_LEVEL=info
      - LOG_FORMAT=json
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: app.logs
    volumes:
      - app_logs:/var/log/app

  # Fluentd for log aggregation
  fluentd:
    image: fluentd:v1.14
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - fluentd_logs:/var/log/fluentd
    networks:
      - monitoring

  # Elasticsearch for log storage
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - monitoring

  # Kibana for log visualization
  kibana:
    image: docker.elastic.co/kibana/kibana:7.15.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    networks:
      - monitoring
    depends_on:
      - elasticsearch

networks:
  app-network:
  monitoring:

volumes:
  app_logs:
  fluentd_logs:
  es_data:

Fluentd Configuration (fluentd/conf/fluent.conf):

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter app.logs>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

<match app.logs>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name app-logs
  type_name _doc
  <buffer>
    @type file
    path /var/log/fluentd/buffer
    flush_mode interval
    flush_interval 5s
  </buffer>
</match>

Example 2: Monitoring with Prometheus and Grafana

version: '3.8'

services:
  # Application with metrics endpoint
  app:
    build: ./app
    ports:
      - "3000:3000"
    networks:
      - app-network
      - monitoring
    environment:
      - METRICS_ENABLED=true
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Prometheus for metrics collection
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    networks:
      - monitoring

  # Grafana for visualization
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/datasources:/etc/grafana/provisioning/datasources:ro
    networks:
      - monitoring

  # Node Exporter for host metrics
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    networks:
      - monitoring

  # cAdvisor for container metrics
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    networks:
      - monitoring

networks:
  app-network:
  monitoring:

volumes:
  prometheus_data:
  grafana_data:

Prometheus Configuration (prometheus/prometheus.yml):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'app'
    static_configs:
      - targets: ['app:3000']
    metrics_path: '/metrics'
    scrape_interval: 5s

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

Example 3: Debugging Tools and Techniques

# Comprehensive container debugging script
#!/bin/bash

CONTAINER_NAME=$1

if [ -z "$CONTAINER_NAME" ]; then
    echo "Usage: $0 <container_name>"
    exit 1
fi

echo "=== Container Information ==="
docker inspect $CONTAINER_NAME | jq '{
    Id: .Id[0:12],
    State: .State,
    Config: {
        Image: .Config.Image,
        Env: .Config.Env,
        Cmd: .Config.Cmd
    },
    NetworkSettings: .NetworkSettings.Networks
}'

echo -e "\n=== Resource Usage ==="
docker stats $CONTAINER_NAME --no-stream

echo -e "\n=== Recent Logs ==="
docker logs --tail 50 $CONTAINER_NAME

echo -e "\n=== Process Information ==="
docker exec $CONTAINER_NAME ps aux 2>/dev/null || echo "Cannot access processes"

echo -e "\n=== Network Connectivity ==="
docker exec $CONTAINER_NAME netstat -tlnp 2>/dev/null || echo "Cannot access network info"

echo -e "\n=== Health Check Status ==="
docker inspect $CONTAINER_NAME | jq '.State.Health // "No health check configured"'

echo -e "\n=== Environment Variables ==="
docker exec $CONTAINER_NAME env | sort

Application Monitoring Code (Node.js example):

const express = require('express');
const client = require('prom-client');
const app = express();

// Create metrics
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status']
});

const httpRequestTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status']
});

// Middleware for metrics collection
app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const labels = {
      method: req.method,
      route: req.route?.path || req.path,
      status: res.statusCode
    };
    
    httpRequestDuration.observe(labels, duration);
    httpRequestTotal.inc(labels);
  });
  
  next();
});

// Health check endpoint
app.get('/health', (req, res) => {
  res.json({ 
    status: 'healthy', 
    timestamp: new Date().toISOString(),
    uptime: process.uptime()
  });
});

// Metrics endpoint for Prometheus
app.get('/metrics', (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(client.register.metrics());
});

// Error handling with structured logging
app.use((err, req, res, next) => {
  console.error(JSON.stringify({
    timestamp: new Date().toISOString(),
    level: 'error',
    message: err.message,
    stack: err.stack,
    request: {
      method: req.method,
      url: req.url,
      headers: req.headers
    }
  }));
  
  res.status(500).json({ error: 'Internal server error' });
});

const port = process.env.PORT || 3000;
app.listen(port, () => {
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level: 'info',
    message: `Server started on port ${port}`
  }));
});

6. Hands-On Practice (30-40 minutes)

Exercise 1: Debugging a Broken Application (15 minutes)

Objective: Diagnose and fix issues in a multi-container application

Scenario: You have a web application that’s experiencing intermittent failures. Users report slow response times and occasional 500 errors.

Provided Broken Application:

version: '3.8'

services:
  web:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app

  app:
    build: ./broken-app
    environment:
      - DATABASE_URL=postgresql://user:wrongpassword@database:5432/myapp
      - MEMORY_LIMIT=64m  # Too low
    # Missing health check
    
  database:
    image: postgres:13
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=correctpassword
    # Missing volume for data persistence

Instructions:

  1. Identify and document all issues in the provided configuration
  2. Use debugging commands to gather evidence
  3. Fix the configuration and verify the fixes work
  4. Add monitoring and health checks

Exercise 2: Implementing Alerting Rules (15 minutes)

Objective: Create meaningful alerts for common container issues

Requirements:

  • Alert when container restarts exceed threshold
  • Alert on high error rates
  • Alert on resource usage exceeding limits
  • Alert when services become unavailable

Starter Alerting Rules Template:

groups:
- name: container.rules
  rules:
  - alert: ContainerRestartingTooMuch
    expr: # TODO: Write expression for restart count
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Container {{ $labels.container }} is restarting too frequently"
      
  - alert: HighErrorRate
    expr: # TODO: Write expression for error rate
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"

Exercise 3: Log Analysis and Troubleshooting (10 minutes)

Objective: Analyze logs to identify patterns and issues

Challenge: Given sample log files with various error patterns:

  • Parse and identify error categories
  • Find correlation between errors and system events
  • Create log-based alerts for proactive monitoring

Extension Ideas:

  • Implement log aggregation across multiple services
  • Create custom Grafana dashboards for log analysis
  • Set up automated log rotation and archival

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Debugging Mistakes

  1. Problem: Looking at symptoms instead of root causes Cause: Focusing on the immediate error without understanding the underlying issue Solution: Use systematic debugging approaches and trace issues to their source

  2. Problem: Inadequate logging makes debugging difficult Cause: Missing or poorly structured log messages Solution: Implement structured logging with appropriate detail levels

  3. Problem: Monitoring alerts are too noisy or miss important issues Cause: Poorly configured thresholds and alert rules Solution: Tune alerting rules based on actual system behavior and business impact

Container-Specific Debugging Tips

# Check container exit codes and restart patterns
docker-compose ps

# Examine container startup logs
docker-compose logs --tail 100 service_name

# Check resource constraints
docker stats --no-stream

# Inspect container networking
docker network ls
docker exec container_name nslookup other_service

# Debug volume mounts
docker exec container_name ls -la /mounted/path
docker volume inspect volume_name

Performance Investigation Commands

# Monitor system resources in real-time
top
htop
iotop

# Check disk usage and I/O
df -h
du -sh /var/lib/docker/*
iostat -x 1

# Network analysis
netstat -tlnp
ss -tlnp
tcpdump -i any port 80

8. Best Practices and Professional Tips (5 minutes)

Logging Best Practices

  • Structured Logging: Use JSON format for easy parsing and filtering
  • Log Levels: Implement appropriate log levels (DEBUG, INFO, WARN, ERROR)
  • Centralized Logging: Aggregate logs from all containers in one location
  • Log Rotation: Implement proper log rotation to prevent disk space issues
  • Sensitive Data: Never log passwords, tokens, or personal information

Monitoring Strategy

  • Golden Signals: Monitor latency, traffic, errors, and saturation
  • Business Metrics: Track metrics that matter to your business
  • Proactive Alerting: Alert on trends, not just thresholds
  • Dashboard Design: Create dashboards that tell a story and guide action

Production Debugging

  • Minimize Impact: Use non-invasive debugging techniques in production
  • Reproducible Environments: Maintain staging environments that mirror production
  • Rollback Plan: Always have a plan to quickly revert changes
  • Documentation: Document debugging procedures and known issues

9. Summary and Review (5 minutes)

Key Takeaways

  1. Systematic Approach: Effective debugging requires methodical investigation
  2. Observability: Logs, metrics, and traces provide comprehensive system visibility
  3. Proactive Monitoring: Good monitoring prevents issues from becoming outages
  4. Tool Mastery: Knowing your debugging tools saves critical time during incidents

Quick Review Questions

  1. What are the three pillars of observability and why is each important?
  2. How would you debug a container that keeps restarting?
  3. What metrics would you monitor for a web application running in containers?

For Instructors

Teaching Tips

  • Use real debugging scenarios and broken applications for hands-on practice
  • Demonstrate both development and production debugging techniques
  • Emphasize the importance of logging and monitoring from day one
  • Show how proper observability saves time during incidents

Assessment Ideas

  • Present broken multi-container applications for students to debug
  • Design monitoring and alerting strategies for given application scenarios
  • Analyze real log files to identify patterns and issues

Adaptation Notes

  • Beginners: Focus on basic Docker debugging commands and simple monitoring
  • Advanced: Cover distributed tracing, advanced alerting strategies, and performance optimization
  • Time Management: Balance theory with extensive hands-on debugging practice
  • Real-world Focus: Use examples and scenarios from actual production environments