Day 20: Security and Best Practices

calendar_today July 11, 2025 schedule 11 min read

Learning Objectives: By the end of this lesson, you will be able to:

Implement security best practices for Docker containers and images

Configure proper access controls and user permissions

Scan images for vulnerabilities and manage security risks

Implement secrets management and secure configuration practices

Design secure container architectures and network policies

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

“Imagine your containerized application handles sensitive customer data and financial transactions. One day, you discover that an attacker has gained access to your production environment through a vulnerable container. What security measures could have prevented this breach? Today we’ll learn how to build security into every layer of your containerized applications.”

Lesson Overview

Duration: 90 minutes
Prerequisites: Docker fundamentals, container networking, basic security concepts
Materials Needed: Docker Desktop, vulnerability scanning tools, sample applications

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

Container Security Fundamentals - Understanding the security model
Image Security - Building and scanning secure container images
Runtime Security - Securing running containers and hosts
Access Control - Managing permissions and authentication
Secrets Management - Handling sensitive data securely

Success Criteria:

By the end of this session, you should be able to identify common security vulnerabilities in containerized applications and implement comprehensive security measures to protect your applications in development and production.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Container Security Model

Containers share the host kernel but provide process isolation through namespaces and cgroups. This shared kernel model creates unique security considerations compared to traditional virtual machines.

Real-World Analogy: Think of containers like apartments in a building. While each apartment (container) is separate and private, they all share the same infrastructure (kernel). A security issue in the building’s foundation (kernel) affects all apartments, and poorly secured apartments can impact their neighbors.

Why This Matters: Understanding the container security model helps you make informed decisions about when and how to use containers securely.

Defense in Depth

Container security requires multiple layers of protection:

Image Security: Secure base images and build processes
Runtime Security: Proper container configuration and isolation
Host Security: Securing the underlying infrastructure
Network Security: Controlling communication between containers
Access Control: Managing who can do what with containers

Common Attack Vectors

Understanding common attack patterns helps build better defenses:

Vulnerable Dependencies: Outdated packages with known vulnerabilities
Privilege Escalation: Containers running with unnecessary privileges
Container Escape: Breaking out of container isolation
Supply Chain Attacks: Compromised base images or build tools
Secrets Exposure: Hardcoded passwords or exposed configuration

Key Terminology

CVE: Common Vulnerabilities and Exposures database
RBAC: Role-Based Access Control for managing permissions
Principle of Least Privilege: Granting minimal necessary permissions
Zero Trust: Security model that trusts nothing by default
Supply Chain Security: Protecting the software development and deployment pipeline

4. Detailed Explanation and Examples (25-30 minutes)

Image Security Best Practices

Secure Base Images

Choose base images carefully and keep them updated:

Official Images: Use official images from Docker Hub when possible
Minimal Images: Use distroless or alpine-based images to reduce attack surface
Regular Updates: Keep base images and dependencies current
Image Scanning: Regularly scan images for vulnerabilities

Dockerfile Security

Writing secure Dockerfiles is crucial for image security:

# Use specific, minimal base image
FROM node:16-alpine

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nextjs -u 1001

# Install dependencies as root, then switch to non-root user
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force

# Copy application code and set ownership
COPY --chown=nextjs:nodejs . .

# Switch to non-root user
USER nextjs

# Expose port (not running as root)
EXPOSE 3000

# Use exec form for proper signal handling
CMD ["node", "server.js"]

Runtime Security Configuration

Container Isolation

Proper container configuration prevents privilege escalation and limits potential damage:

version: '3.8'

services:
  app:
    build: .
    # Security configurations
    read_only: true  # Read-only filesystem
    user: "1001:1001"  # Non-root user
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE  # Only necessary capabilities
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp
      - /var/cache
    volumes:
      - app_data:/app/data:rw
      - ./config:/app/config:ro
    networks:
      - app-network
    environment:
      - NODE_ENV=production
    # Resource limits prevent DoS
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

networks:
  app-network:
    driver: bridge
    internal: true  # No external access

volumes:
  app_data:

Network Security

Network Segmentation

Isolate services using custom networks:

version: '3.8'

services:
  frontend:
    image: nginx:alpine
    networks:
      - public
      - internal
    ports:
      - "443:443"

  api:
    build: ./api
    networks:
      - internal
      - database
    # No external ports exposed

  database:
    image: postgres:13
    networks:
      - database  # Isolated network
    environment:
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password

networks:
  public:
    driver: bridge
  internal:
    driver: bridge
    internal: true
  database:
    driver: bridge
    internal: true

secrets:
  db_password:
    file: ./secrets/db_password.txt

Secrets Management

Docker Secrets

Docker provides built-in secrets management for sensitive data:

# Create a secret
echo "my_secret_password" | docker secret create db_password -

# Use secret in service
docker service create \
  --name myapp \
  --secret db_password \
  myapp:latest

Environment-Specific Secrets

version: '3.8'

services:
  app:
    image: myapp:latest
    environment:
      - DATABASE_URL_FILE=/run/secrets/database_url
      - API_KEY_FILE=/run/secrets/api_key
    secrets:
      - database_url
      - api_key

secrets:
  database_url:
    external: true
  api_key:
    external: true

Common Misconceptions

Misconception: Containers are inherently secure Reality: Containers require proper configuration and security practices to be secure
Misconception: Running containers as non-root is sufficient for security Reality: User permissions are just one layer; comprehensive security requires multiple measures

5. Live Coding Demonstration (20-25 minutes)

Example 1: Secure Multi-tier Application

version: '3.8'

services:
  # Web server with security hardening
  nginx:
    image: nginx:1.21-alpine
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
      - nginx_cache:/var/cache/nginx
      - nginx_logs:/var/log/nginx
    networks:
      - frontend
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp
      - /var/run
    deploy:
      resources:
        limits:
          memory: 128M
        reservations:
          memory: 64M

  # Application server with minimal privileges
  app:
    build: 
      context: ./app
      dockerfile: Dockerfile.secure
    networks:
      - frontend
      - backend
    environment:
      - NODE_ENV=production
      - DATABASE_URL_FILE=/run/secrets/database_url
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
    secrets:
      - database_url
      - jwt_secret
    volumes:
      - app_uploads:/app/uploads
    read_only: true
    tmpfs:
      - /tmp
    user: "1001:1001"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '0.5'

  # Database with network isolation
  database:
    image: postgres:13-alpine
    networks:
      - backend
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=appuser
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./database/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    read_only: true
    tmpfs:
      - /tmp
      - /var/run/postgresql
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M

  # Security scanning service
  trivy:
    image: aquasec/trivy:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - trivy_cache:/root/.cache
    command: ["server", "--listen", "0.0.0.0:8080"]
    networks:
      - monitoring

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true
  monitoring:
    driver: bridge
    internal: true

volumes:
  postgres_data:
  nginx_cache:
  nginx_logs:
  app_uploads:
  trivy_cache:

secrets:
  database_url:
    file: ./secrets/database_url.txt
  jwt_secret:
    file: ./secrets/jwt_secret.txt
  db_password:
    file: ./secrets/db_password.txt

Secure Dockerfile (app/Dockerfile.secure):

# Multi-stage build for smaller, more secure images
FROM node:16-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force

FROM node:16-alpine AS runtime

# Install security updates
RUN apk update && apk upgrade && apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nextjs -u 1001 -G nodejs

# Create app directory with correct permissions
WORKDIR /app
RUN chown nextjs:nodejs /app

# Copy dependencies from builder stage
COPY --from=builder --chown=nextjs:nodejs /app/node_modules ./node_modules

# Copy application code
COPY --chown=nextjs:nodejs . .

# Remove unnecessary files
RUN rm -rf .git .gitignore README.md Dockerfile* && \
    npm prune --production

# Switch to non-root user
USER nextjs

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]

Example 2: Security Scanning and Compliance

#!/bin/bash
# Security scanning script

set -e

IMAGE_NAME="myapp:latest"
DOCKERFILE_PATH="./Dockerfile"

echo "=== Docker Security Scanning ==="

# 1. Dockerfile linting with hadolint
echo "Scanning Dockerfile for best practices..."
docker run --rm -i hadolint/hadolint < $DOCKERFILE_PATH

# 2. Image vulnerability scanning with Trivy
echo "Scanning image for vulnerabilities..."
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  -v $(pwd):/root/.cache/ aquasec/trivy:latest image $IMAGE_NAME

# 3. Container configuration scanning
echo "Checking container security configuration..."
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  -v $(pwd):/tmp anchore/grype:latest $IMAGE_NAME

# 4. Secrets scanning
echo "Scanning for exposed secrets..."
docker run --rm -v $(pwd):/path \
  zricethezav/gitleaks:latest detect --source="/path" --verbose

# 5. Custom security checks
echo "Running custom security checks..."

# Check for running containers with privileged mode
PRIVILEGED_CONTAINERS=$(docker ps --filter "label=privileged=true" -q)
if [ ! -z "$PRIVILEGED_CONTAINERS" ]; then
    echo "WARNING: Found containers running in privileged mode:"
    docker ps --filter "label=privileged=true"
fi

# Check for containers running as root
ROOT_CONTAINERS=$(docker ps --format "table {{.Names}}\t{{.Image}}" | while read name image; do
    if [ "$name" != "NAMES" ]; then
        USER=$(docker exec $name whoami 2>/dev/null || echo "unknown")
        if [ "$USER" = "root" ]; then
            echo "$name ($image)"
        fi
    fi
done)

if [ ! -z "$ROOT_CONTAINERS" ]; then
    echo "WARNING: Found containers running as root:"
    echo "$ROOT_CONTAINERS"
fi

echo "Security scanning completed."

Example 3: Secrets Management with External Tools

# Using external secrets management
version: '3.8'

services:
  app:
    image: myapp:latest
    environment:
      # Reference to external secret store
      - DATABASE_URL=vault:secret/data/myapp#DATABASE_URL
      - API_KEY=vault:secret/data/myapp#API_KEY
    volumes:
      - vault_token:/vault/token:ro
    depends_on:
      - vault-agent

  vault-agent:
    image: vault:latest
    command: ["vault", "agent", "-config=/vault/config/agent.hcl"]
    volumes:
      - ./vault/agent.hcl:/vault/config/agent.hcl:ro
      - vault_token:/vault/token
    environment:
      - VAULT_ADDR=https://vault.company.com
      - VAULT_ROLE_ID_FILE=/vault/role-id
      - VAULT_SECRET_ID_FILE=/vault/secret-id
    secrets:
      - source: vault_role_id
        target: /vault/role-id
      - source: vault_secret_id
        target: /vault/secret-id

volumes:
  vault_token:

secrets:
  vault_role_id:
    external: true
  vault_secret_id:
    external: true

6. Hands-On Practice (30-40 minutes)

Exercise 1: Security Audit and Hardening (20 minutes)

Objective: Identify and fix security vulnerabilities in a container setup

Provided Insecure Application:

version: '3.8'

services:
  web:
    build: .
    ports:
      - "80:80"
    volumes:
      - /:/host:rw  # Dangerous host mount
    privileged: true  # Unnecessary privileges
    environment:
      - DB_PASSWORD=plaintext_password  # Exposed secret
      - DEBUG=true
    user: root  # Running as root

  database:
    image: postgres:9.6  # Outdated version
    ports:
      - "5432:5432"  # Exposed to host
    environment:
      - POSTGRES_PASSWORD=admin123
      - POSTGRES_HOST_AUTH_METHOD=trust  # No authentication

Insecure Dockerfile:

FROM ubuntu:18.04

# Running as root
RUN apt-get update && apt-get install -y \
    curl \
    wget \
    sudo

# Copying sensitive files
COPY secrets.txt /app/
COPY private_key.pem /app/

# Exposing unnecessary ports
EXPOSE 22 3000 8080 9090

# Running with full privileges
CMD ["./app"]

Instructions:

Identify all security issues in the provided configuration
Create a secure version addressing each vulnerability
Implement proper secrets management
Add security scanning to the build process
Document the security improvements made

Exercise 2: Implementing Zero-Trust Architecture (15 minutes)

Objective: Design a container architecture with zero-trust principles

Requirements:

No service trusts any other service by default
All communication must be authenticated and encrypted
Implement proper network segmentation
Add monitoring and auditing for all access

Hints:

Use mutual TLS for service-to-service communication
Implement service mesh for traffic control
Add authentication at every layer
Monitor and log all network traffic

Exercise 3: Compliance and Governance (5 minutes)

Objective: Create policies for container security compliance

Challenge: Define policies for:

Image approval process
Runtime security requirements
Access control and permissions
Monitoring and incident response

Extension Ideas:

Implement automated policy enforcement
Create security dashboards and reporting
Set up vulnerability management workflows

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Security Mistakes

Problem: Running containers as root user Cause: Default Docker behavior and lack of security awareness Solution: Always create and use non-root users in containers
Problem: Exposing unnecessary ports and services Cause: Over-permissive network configuration Solution: Follow principle of least privilege for network access
Problem: Hardcoded secrets in images or environment variables Cause: Lack of proper secrets management strategy Solution: Use Docker secrets, external secret stores, or runtime secret injection

Security Scanning and Monitoring

# Regular security maintenance commands

# Update all images
docker images --format "{{.Repository}}:{{.Tag}}" | \
  xargs -I {} docker pull {}

# Clean up unused images and containers
docker system prune -af

# Check for containers with security issues
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" | \
  while read name image status; do
    if [[ "$status" == *"Exited"* ]]; then
        echo "Investigate: $name ($image) - $status"
    fi
  done

# Monitor for privilege escalation attempts
docker events --filter event=exec | \
  while read event; do
    echo "Exec event detected: $event"
  done

Incident Response Procedures

Detection: Automated monitoring and alerting
Containment: Isolate affected containers and networks
Investigation: Analyze logs and system state
Recovery: Restore from known good state
Lessons Learned: Update security practices based on findings

8. Best Practices and Professional Tips (5 minutes)

Security Development Lifecycle

Secure by Design: Build security into the development process
Shift Left: Implement security checks early in the development cycle
Continuous Monitoring: Monitor security posture continuously
Regular Updates: Keep images, dependencies, and tools updated
Training: Ensure team members understand security best practices

Production Security Checklist

All images scanned for vulnerabilities
No containers running as root
Secrets managed securely
Network segmentation implemented
Resource limits configured
Monitoring and alerting in place
Backup and recovery procedures tested
Incident response plan documented

Compliance Considerations

Regulatory Requirements: Understand applicable regulations (SOC 2, PCI DSS, HIPAA)
Audit Trails: Maintain comprehensive logs for compliance auditing
Data Protection: Implement proper data encryption and access controls
Regular Assessments: Conduct periodic security assessments and penetration testing

9. Summary and Review (5 minutes)

Key Takeaways

Defense in Depth: Security requires multiple layers of protection
Least Privilege: Grant minimal necessary permissions and access
Continuous Monitoring: Security is an ongoing process, not a one-time setup
Supply Chain Security: Secure your entire development and deployment pipeline

Quick Review Questions

What are the key differences between container security and traditional VM security?
How would you securely manage database passwords in a multi-container application?
What steps would you take to investigate a suspected container security breach?

For Instructors

Teaching Tips

Use real-world security scenarios and case studies
Demonstrate both secure and insecure configurations for comparison
Emphasize the business impact of security vulnerabilities
Show how security practices integrate with development workflows

Assessment Ideas

Security audit of provided insecure applications
Design secure architectures for given requirements
Create incident response procedures for container security breaches

Adaptation Notes

Beginners: Focus on fundamental security concepts and basic hardening
Advanced: Cover advanced topics like service mesh security and compliance frameworks
Time Management: Balance hands-on security auditing with theoretical concepts
Industry Focus: Adapt examples to specific industry requirements (healthcare, finance, etc.)