Docker chevron_right Day 17: Compose Networking and Volumes

Day 17: Compose Networking and Volumes

calendar_today schedule 9 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Create and manage custom networks in Docker Compose
  • Configure volumes and bind mounts in Compose files
  • Implement service-to-service communication patterns
  • Design network isolation and security boundaries
  • Manage persistent data across multi-container applications

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

“Think about a complex web application like an e-commerce site. How would you ensure that the database is only accessible by the backend services, while the frontend can communicate with the API, but all services can share configuration files? Today we’ll learn how Docker Compose makes this orchestration elegant and secure.”

Lesson Overview

  • Duration: 90 minutes
  • Prerequisites: Docker Compose basics, understanding of containers and volumes
  • Materials Needed: Docker Desktop, text editor, sample multi-service application

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Custom Networks in Compose - Creating isolated communication channels
  2. Volume Management - Persistent data and shared storage strategies
  3. Service Communication - Best practices for inter-service connectivity
  4. Network Security - Implementing proper isolation boundaries
  5. Real-world Patterns - Common architectures and their implementations

Success Criteria:

By the end of this session, you should be able to design and implement a multi-service application with proper network isolation, persistent data management, and secure service communication.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Docker Compose Networking

Docker Compose automatically creates a default network for your application, but you can define custom networks to control how services communicate. This is like having different VLANs in an office building - some departments need to talk to each other, while others should remain isolated.

Real-World Analogy: Think of your application as a corporate office building. The default network is like having everyone in one big open space, but custom networks are like having separate floors or departments with controlled access between them.

Why This Matters: Proper network design improves security, performance, and maintainability by controlling which services can communicate and how.

Volume Management in Compose

Volumes in Compose allow you to persist data, share files between services, and manage application state. Unlike containers, volumes persist even when containers are removed.

Real-World Analogy: Volumes are like filing cabinets in an office - even if employees (containers) come and go, the important documents (data) remain accessible and organized.

Key Terminology

  • Network Driver: The underlying technology that creates and manages networks (bridge, overlay, etc.)
  • Service Discovery: Automatic resolution of service names to IP addresses within networks
  • Named Volumes: Docker-managed storage that persists independently of containers
  • Bind Mounts: Direct mapping of host filesystem paths into containers
  • External Networks: Networks created outside of the current Compose file

4. Detailed Explanation and Examples (25-30 minutes)

Understanding Compose Networking

Docker Compose networking works on several levels. By default, Compose creates a bridge network named after your project, and all services join this network. Services can communicate using their service names as hostnames.

Network Types and Use Cases

  1. Default Bridge Network: Suitable for simple applications where all services need to communicate
  2. Custom Bridge Networks: Useful for grouping related services and creating isolation boundaries
  3. External Networks: Pre-existing networks that multiple Compose projects can share
  4. Host and None Networks: Special cases for direct host access or complete isolation

Service Discovery and DNS

Compose provides automatic DNS resolution within networks. A service named database can be reached by other services using database as the hostname. This eliminates the need for hardcoded IP addresses and makes applications portable.

Volume Management Strategies

Volume Types in Compose

  1. Named Volumes: Managed by Docker, portable, and easy to backup
  2. Bind Mounts: Direct host path mapping, useful for development
  3. Tmpfs Mounts: In-memory storage for temporary data

Data Persistence Patterns

  • Database Storage: Always use named volumes for database data
  • Application Code: Bind mounts for development, volumes for production
  • Configuration Files: Can use either depending on update frequency
  • Logs: Consider external log management or bind mounts for easy access

Common Misconceptions

  1. Misconception: All services in a Compose file can automatically communicate Reality: Only services on the same network can communicate directly

  2. Misconception: Volumes are automatically backed up Reality: You need to implement backup strategies for volume data

5. Live Coding Demonstration (20-25 minutes)

Example 1: Multi-Network Architecture

version: '3.8'

services:
  # Frontend service - publicly accessible
  frontend:
    image: nginx:alpine
    ports:
      - "80:80"
    networks:
      - public
      - internal
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - frontend_assets:/usr/share/nginx/html

  # API service - internal communication only
  api:
    build: ./api
    networks:
      - internal
      - database
    environment:
      - DATABASE_URL=postgresql://user:password@database:5432/myapp
    volumes:
      - ./api:/app
      - api_logs:/var/log/api
    depends_on:
      - database

  # Database service - isolated network
  database:
    image: postgres:13
    networks:
      - database
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro

networks:
  public:
    driver: bridge
  internal:
    driver: bridge
    internal: true  # No external access
  database:
    driver: bridge
    internal: true

volumes:
  postgres_data:
    driver: local
  frontend_assets:
    driver: local
  api_logs:
    driver: local

Explanation:

  • Network Isolation: Database is only accessible via the database network
  • Frontend Access: Can communicate with API but not directly with database
  • Volume Management: Persistent data for database, logs, and assets
  • Security: Internal networks prevent external access to sensitive services

Example 2: Development vs Production Configuration

# docker-compose.yml (base configuration)
version: '3.8'

services:
  app:
    build: .
    networks:
      - app-network
    volumes:
      - app_data:/data

  database:
    image: postgres:13
    networks:
      - app-network
    volumes:
      - db_data:/var/lib/postgresql/data

networks:
  app-network:

volumes:
  app_data:
  db_data:

# docker-compose.override.yml (development overrides)
version: '3.8'

services:
  app:
    volumes:
      - .:/app  # Bind mount for live code changes
      - /app/node_modules  # Anonymous volume to preserve node_modules
    environment:
      - NODE_ENV=development
    ports:
      - "3000:3000"

  database:
    ports:
      - "5432:5432"  # Expose database for development tools
    environment:
      - POSTGRES_DB=myapp_dev

# docker-compose.prod.yml (production overrides)
version: '3.8'

services:
  app:
    environment:
      - NODE_ENV=production
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure

  database:
    environment:
      - POSTGRES_DB=myapp_prod
    deploy:
      restart_policy:
        condition: on-failure

Example 3: External Network Integration

version: '3.8'

services:
  web:
    image: nginx
    networks:
      - shared-network
      - internal

  app:
    build: .
    networks:
      - internal

networks:
  # Connect to existing shared network
  shared-network:
    external: true
    name: company-network
  
  # Internal network for this application
  internal:
    driver: bridge

Debugging Network Connectivity

# Inspect networks
docker network ls
docker network inspect myproject_default

# Test connectivity between services
docker-compose exec service1 ping service2
docker-compose exec service1 nslookup service2

# Check which networks a container is connected to
docker inspect container_name | grep -A 10 Networks

6. Hands-On Practice (30-40 minutes)

Exercise 1: Three-Tier Architecture (15 minutes)

Objective: Create a web application with proper network isolation

Instructions:

  1. Design a three-tier architecture: web server, application server, database
  2. Create appropriate networks to isolate each tier
  3. Ensure only necessary communication paths exist
  4. Add persistent volumes for database and application logs

Starter Code:

version: '3.8'

services:
  # TODO: Add nginx web server
  # TODO: Add application server (node.js or python)
  # TODO: Add PostgreSQL database
  
  # TODO: Define networks for proper isolation
  # TODO: Add volumes for persistence

networks:
  # TODO: Define your networks

volumes:
  # TODO: Define your volumes

Solution:

version: '3.8'

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    networks:
      - frontend
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app

  app:
    image: node:16-alpine
    working_dir: /app
    command: npm start
    networks:
      - frontend
      - backend
    volumes:
      - ./app:/app
      - app_logs:/var/log/app
    environment:
      - DATABASE_URL=postgresql://user:password@database:5432/myapp
    depends_on:
      - database

  database:
    image: postgres:13
    networks:
      - backend
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true

volumes:
  postgres_data:
  app_logs:

Exercise 2: Shared Storage Architecture (15 minutes)

Objective: Implement file sharing between multiple services

Requirements:

  • Multiple worker services that process files from a shared directory
  • A file upload service that receives files
  • Persistent storage that survives container restarts
  • Proper network isolation

Hints:

  • Use named volumes for shared storage
  • Consider using bind mounts for development
  • Implement health checks for file processing services

Exercise 3: Microservices Communication (10 minutes)

Objective: Design a microservices architecture with service mesh

Challenge: Create a system with:

  • User service
  • Product service
  • Order service
  • API Gateway
  • Shared cache (Redis)
  • Message queue (RabbitMQ)

Extension Ideas:

  • Add monitoring and logging services
  • Implement service discovery patterns
  • Add configuration management

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Services can’t communicate despite being in the same Compose file Cause: Services are on different networks or network isolation is blocking communication Solution: Ensure services share at least one network and check internal network settings

  2. Problem: Volume data is lost when recreating containers Cause: Using anonymous volumes or bind mounts to non-persistent locations Solution: Use named volumes for persistent data and verify volume mounting

  3. Problem: Port conflicts when running multiple Compose projects Cause: Hard-coded port mappings conflict between projects Solution: Use environment variables for ports or remove port mappings for internal services

Debugging Tips

  • Use docker-compose config to validate and view the final configuration
  • Inspect networks with docker network inspect to understand connectivity
  • Use docker-compose logs to track communication issues
  • Test connectivity with docker-compose exec service ping other-service

Network Troubleshooting Commands

# View all networks
docker network ls

# Inspect specific network
docker network inspect network_name

# Check service connectivity
docker-compose exec service1 nslookup service2
docker-compose exec service1 telnet service2 port

# View container network configuration
docker inspect container_name | jq '.[].NetworkSettings'

8. Best Practices and Professional Tips (5 minutes)

Network Design Principles

  • Principle of Least Privilege: Only allow necessary communication paths
  • Defense in Depth: Use multiple layers of network isolation
  • Service Mesh Patterns: Consider using dedicated service mesh for complex architectures
  • Documentation: Clearly document network architecture and communication flows

Volume Management Best Practices

  • Use Named Volumes: For all persistent data in production
  • Backup Strategy: Implement regular backup procedures for volume data
  • Development vs Production: Use bind mounts for development, volumes for production
  • Performance: Consider volume drivers for specific performance requirements

Production Considerations

  • External Networks: Use external networks for shared infrastructure
  • Secret Management: Never put sensitive data in environment variables
  • Resource Limits: Set appropriate memory and CPU limits
  • Health Checks: Implement proper health checks for all services

9. Summary and Review (5 minutes)

Key Takeaways

  1. Network Isolation: Custom networks provide security and organization benefits
  2. Volume Strategy: Choose the right volume type for each use case
  3. Service Communication: Use service names for reliable inter-service communication
  4. Architecture Patterns: Multi-tier and microservices patterns improve maintainability

Quick Review Questions

  1. How would you isolate a database service from external access while allowing application access?
  2. What’s the difference between named volumes and bind mounts in production?
  3. How does Docker Compose handle service discovery within networks?

For Instructors

Teaching Tips

  • Start with simple network examples before introducing complex multi-network architectures
  • Use visual diagrams to illustrate network boundaries and communication flows
  • Emphasize security implications of network design decisions
  • Demonstrate troubleshooting techniques with real connectivity issues

Assessment Ideas

  • Design a secure multi-tier architecture for a given application
  • Troubleshoot network connectivity issues in a broken Compose file
  • Compare volume strategies for different types of data persistence

Adaptation Notes

  • Beginners: Focus on default networking and simple volume usage
  • Advanced: Introduce service mesh concepts and advanced volume drivers
  • Time Management: Prioritize hands-on exercises over theoretical explanations
  • Remote Learning: Use screen sharing for network inspection commands