Redis Clustering and Scaling

Learning Objectives: By the end of this lesson, you will be able to:

  • Design and implement Redis Cluster for horizontal scaling
  • Configure Redis Sentinel for high availability and automatic failover
  • Build applications that work seamlessly with distributed Redis
  • Handle data partitioning, replication, and consistency in Redis clusters
  • Scale Redis to handle millions of operations per second

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Ever wondered how companies like Twitter handle billions of Redis operations per day, or how Netflix maintains 99.99% uptime even when servers fail? The answer lies in Redis clustering and scaling strategies that distribute data and workload across multiple servers while maintaining lightning-fast performance!

Lesson Overview

  • Duration: 100 minutes
  • Prerequisites: Lessons 1-8 completed, understanding of distributed systems concepts
  • Materials Needed: Go environment, multiple Redis instances, basic networking knowledge

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Scaling Strategies - Vertical vs horizontal scaling, partitioning approaches
  2. Redis Cluster - Sharding, hash slots, and distributed operations
  3. Redis Sentinel - High availability, failover, and service discovery
  4. Client Configuration - Smart clients, connection pooling, and failover handling
  5. Data Migration - Adding/removing nodes, resharding, and zero-downtime scaling
  6. Monitoring & Operations - Cluster health, performance metrics, and troubleshooting

Success Criteria:

By the end of this session, you should be able to design and operate Redis clusters that scale to enterprise requirements while maintaining high availability.

3. Foundation Knowledge (15-20 minutes)

Scaling Approaches

Vertical Scaling (Scale Up)

  • Add more CPU, RAM, or storage to existing server
  • Simpler to implement but has hardware limits
  • Single point of failure
  • Limited by largest available hardware

Horizontal Scaling (Scale Out)

  • Add more servers to distribute load
  • Near-unlimited scaling potential
  • Built-in redundancy and fault tolerance
  • Requires distributed system design

Redis Scaling Solutions

1. Redis Cluster (Built-in Sharding)

[App] → [Redis Cluster Client] → [Master 1] → [Replica 1]
                               → [Master 2] → [Replica 2]
                               → [Master 3] → [Replica 3]

2. Redis Sentinel (High Availability)

[App] → [Sentinel Client] → [Sentinel 1] → [Master] → [Replica 1]
                          → [Sentinel 2]             → [Replica 2]
                          → [Sentinel 3]

3. Client-side Sharding

[App] → [Sharding Logic] → [Redis 1]
                        → [Redis 2]
                        → [Redis 3]

Key Terminology

  • Hash Slot: Unit of data distribution in Redis Cluster (16384 total)
  • Sharding: Distributing data across multiple Redis instances
  • Replication: Copying data to multiple servers for redundancy
  • Failover: Automatic promotion of replica to master during failures
  • Split-brain: Network partition causing multiple masters

4. Detailed Explanation and Examples (25-30 minutes)

Redis Cluster Architecture

Hash Slot Distribution

Redis Cluster uses 16384 hash slots to distribute data:

Key "user:123" → CRC16(user:123) % 16384 → Slot 12539 → Node 2
Key "session:abc" → CRC16(session:abc) % 16384 → Slot 3847 → Node 1

Cluster Topology

Node 1: Slots 0-5460      (Master + Replica)
Node 2: Slots 5461-10922  (Master + Replica)  
Node 3: Slots 10923-16383 (Master + Replica)

Redis Sentinel Architecture

Service Discovery and Failover

1. Sentinels monitor master and replicas
2. Majority consensus required for failover decisions
3. Automatic master promotion during failures
4. Client redirection to new master

5. Live Coding Demonstration (20-25 minutes)

Example 1: Redis Cluster Client Implementation

package main

import (
    "context"
    "fmt"
    "log"
    "sync"
    "time"

    "github.com/redis/go-redis/v9"
)

// ClusterManager handles Redis Cluster operations
type ClusterManager struct {
    client *redis.ClusterClient
    ctx    context.Context
}

// ClusterConfig holds cluster configuration
type ClusterConfig struct {
    Addrs              []string
    Password           string
    ReadTimeout        time.Duration
    WriteTimeout       time.Duration
    PoolSize           int
    MinIdleConns       int
    MaxRetries         int
    RouteByLatency     bool
    RouteRandomly      bool
}

func NewClusterManager(config ClusterConfig) *ClusterManager {
    clusterClient := redis.NewClusterClient(&redis.ClusterOptions{
        Addrs:        config.Addrs,
        Password:     config.Password,
        ReadTimeout:  config.ReadTimeout,
        WriteTimeout: config.WriteTimeout,
        PoolSize:     config.PoolSize,
        MinIdleConns: config.MinIdleConns,
        MaxRetries:   config.MaxRetries,
        
        // Cluster-specific optimizations
        RouteByLatency: config.RouteByLatency,
        RouteRandomly:  config.RouteRandomly,
        MaxRedirects:   8,
        ReadOnly:       false,
    })

    return &ClusterManager{
        client: clusterClient,
        ctx:    context.Background(),
    }
}

// GetClusterInfo retrieves cluster topology information
func (cm *ClusterManager) GetClusterInfo() (*ClusterInfo, error) {
    // Get cluster nodes information
    nodesCmd := cm.client.ClusterNodes(cm.ctx)
    if nodesCmd.Err() != nil {
        return nil, fmt.Errorf("failed to get cluster nodes: %w", nodesCmd.Err())
    }

    // Get cluster slots information
    slotsCmd := cm.client.ClusterSlots(cm.ctx)
    if slotsCmd.Err() != nil {
        return nil, fmt.Errorf("failed to get cluster slots: %w", slotsCmd.Err())
    }

    info := &ClusterInfo{
        Nodes:     parseNodes(nodesCmd.Val()),
        Slots:     parseSlots(slotsCmd.Val()),
        Timestamp: time.Now(),
    }

    return info, nil
}

type ClusterInfo struct {
    Nodes     []NodeInfo    `json:"nodes"`
    Slots     []SlotRange   `json:"slots"`
    Timestamp time.Time     `json:"timestamp"`
}

type NodeInfo struct {
    ID       string   `json:"id"`
    Address  string   `json:"address"`
    Role     string   `json:"role"` // master/replica
    Status   string   `json:"status"` // connected/disconnected
    Slots    []int    `json:"slots,omitempty"`
    MasterID string   `json:"master_id,omitempty"`
}

type SlotRange struct {
    Start  int      `json:"start"`
    End    int      `json:"end"`
    Master NodeInfo `json:"master"`
    Replicas []NodeInfo `json:"replicas"`
}

// DistributedCache implements cache operations across cluster
type DistributedCache struct {
    cluster *ClusterManager
}

func NewDistributedCache(cluster *ClusterManager) *DistributedCache {
    return &DistributedCache{cluster: cluster}
}

// Set stores a key-value pair in the cluster
func (dc *DistributedCache) Set(key string, value interface{}, ttl time.Duration) error {
    return dc.cluster.client.Set(dc.cluster.ctx, key, value, ttl).Err()
}

// Get retrieves a value from the cluster
func (dc *DistributedCache) Get(key string) (string, error) {
    return dc.cluster.client.Get(dc.cluster.ctx, key).Result()
}

// MGet retrieves multiple values efficiently across cluster
func (dc *DistributedCache) MGet(keys []string) (map[string]string, error) {
    // Group keys by node to minimize cross-slot operations
    nodeKeys := dc.groupKeysByNode(keys)
    
    results := make(map[string]string)
    var mu sync.Mutex
    var wg sync.WaitGroup
    errChan := make(chan error, len(nodeKeys))

    // Execute gets in parallel per node
    for _, keyGroup := range nodeKeys {
        wg.Add(1)
        go func(keys []string) {
            defer wg.Done()
            
            values, err := dc.cluster.client.MGet(dc.cluster.ctx, keys...).Result()
            if err != nil {
                errChan <- err
                return
            }

            mu.Lock()
            for i, key := range keys {
                if values[i] != nil {
                    results[key] = values[i].(string)
                }
            }
            mu.Unlock()
        }(keyGroup)
    }

    wg.Wait()
    close(errChan)

    // Check for errors
    if len(errChan) > 0 {
        return nil, <-errChan
    }

    return results, nil
}

// groupKeysByNode groups keys by their cluster node (simplified implementation)
func (dc *DistributedCache) groupKeysByNode(keys []string) [][]string {
    // In a real implementation, this would use the cluster's hash slot mapping
    // For demo purposes, we'll just group sequentially
    groupSize := 10
    var groups [][]string
    
    for i := 0; i < len(keys); i += groupSize {
        end := i + groupSize
        if end > len(keys) {
            end = len(keys)
        }
        groups = append(groups, keys[i:end])
    }
    
    return groups
}

// Pipeline operations across cluster nodes
func (dc *DistributedCache) BatchSet(items map[string]interface{}, ttl time.Duration) error {
    // Group operations by hash slot to minimize cross-slot operations
    pipe := dc.cluster.client.Pipeline()
    
    for key, value := range items {
        pipe.Set(dc.cluster.ctx, key, value, ttl)
    }
    
    _, err := pipe.Exec(dc.cluster.ctx)
    return err
}

// Cross-slot operations (requires special handling)
func (dc *DistributedCache) AtomicIncrement(keys []string) error {
    // For cross-slot atomic operations, we need to use hash tags
    // or handle them specially
    
    if len(keys) == 0 {
        return nil
    }

    // Check if all keys are in the same slot
    if dc.sameSlot(keys) {
        // Can use transaction
        return dc.cluster.client.Watch(dc.cluster.ctx, func(tx *redis.Tx) error {
            _, err := tx.TxPipelined(dc.cluster.ctx, func(pipe redis.Pipeliner) error {
                for _, key := range keys {
                    pipe.Incr(dc.cluster.ctx, key)
                }
                return nil
            })
            return err
        }, keys...)
    } else {
        // Must handle individually (not atomic across cluster)
        pipe := dc.cluster.client.Pipeline()
        for _, key := range keys {
            pipe.Incr(dc.cluster.ctx, key)
        }
        _, err := pipe.Exec(dc.cluster.ctx)
        return err
    }
}

// sameSlot checks if all keys map to the same hash slot
func (dc *DistributedCache) sameSlot(keys []string) bool {
    if len(keys) <= 1 {
        return true
    }
    
    // Use Redis's hash tag extraction logic
    firstSlot := dc.keySlot(keys[0])
    for _, key := range keys[1:] {
        if dc.keySlot(key) != firstSlot {
            return false
        }
    }
    return true
}

// keySlot calculates the hash slot for a key (simplified)
func (dc *DistributedCache) keySlot(key string) int {
    // Simplified - real implementation would use CRC16
    hash := 0
    for _, c := range key {
        hash = hash*31 + int(c)
    }
    return hash % 16384
}

// Health check for cluster nodes
func (cm *ClusterManager) HealthCheck() (*HealthStatus, error) {
    status := &HealthStatus{
        Healthy:   true,
        Nodes:     make(map[string]NodeHealth),
        Timestamp: time.Now(),
    }

    // Check each node's health
    info, err := cm.GetClusterInfo()
    if err != nil {
        status.Healthy = false
        status.Error = err.Error()
        return status, err
    }

    for _, node := range info.Nodes {
        nodeHealth := NodeHealth{
            Address: node.Address,
            Role:    node.Role,
            Status:  node.Status,
        }

        // Ping individual node
        nodeClient := redis.NewClient(&redis.Options{
            Addr: node.Address,
        })
        
        pingStart := time.Now()
        _, err := nodeClient.Ping(cm.ctx).Result()
        pingDuration := time.Since(pingStart)
        
        nodeClient.Close()

        if err != nil {
            nodeHealth.Healthy = false
            nodeHealth.Error = err.Error()
            status.Healthy = false
        } else {
            nodeHealth.Healthy = true
            nodeHealth.Latency = pingDuration
        }

        status.Nodes[node.ID] = nodeHealth
    }

    return status, nil
}

type HealthStatus struct {
    Healthy   bool                   `json:"healthy"`
    Nodes     map[string]NodeHealth  `json:"nodes"`
    Error     string                 `json:"error,omitempty"`
    Timestamp time.Time              `json:"timestamp"`
}

type NodeHealth struct {
    Address string        `json:"address"`
    Role    string        `json:"role"`
    Status  string        `json:"status"`
    Healthy bool          `json:"healthy"`
    Latency time.Duration `json:"latency"`
    Error   string        `json:"error,omitempty"`
}

// Parse helper functions (simplified implementations)
func parseNodes(nodesStr string) []NodeInfo {
    // In real implementation, parse the CLUSTER NODES output
    return []NodeInfo{
        {ID: "node1", Address: "127.0.0.1:7001", Role: "master", Status: "connected"},
        {ID: "node2", Address: "127.0.0.1:7002", Role: "master", Status: "connected"},
        {ID: "node3", Address: "127.0.0.1:7003", Role: "master", Status: "connected"},
    }
}

func parseSlots(slots []redis.ClusterSlot) []SlotRange {
    var ranges []SlotRange
    for _, slot := range slots {
        ranges = append(ranges, SlotRange{
            Start: slot.Start,
            End:   slot.End,
            Master: NodeInfo{
                Address: fmt.Sprintf("%s:%d", slot.Nodes[0].Addr, slot.Nodes[0].Port),
                Role:    "master",
            },
        })
    }
    return ranges
}

// Demo application
func main() {
    // Configure cluster connection
    config := ClusterConfig{
        Addrs: []string{
            "127.0.0.1:7001",
            "127.0.0.1:7002", 
            "127.0.0.1:7003",
        },
        ReadTimeout:    3 * time.Second,
        WriteTimeout:   3 * time.Second,
        PoolSize:       10,
        MinIdleConns:   5,
        MaxRetries:     3,
        RouteByLatency: true,
        RouteRandomly:  false,
    }

    cluster := NewClusterManager(config)
    defer cluster.client.Close()

    cache := NewDistributedCache(cluster)

    // Test cluster operations
    log.Println("=== Cluster Operations Demo ===")

    // Set some values
    items := map[string]interface{}{
        "user:1":    `{"name":"Alice","age":30}`,
        "user:2":    `{"name":"Bob","age":25}`,
        "user:3":    `{"name":"Charlie","age":35}`,
        "session:1": "session_data_1",
        "session:2": "session_data_2",
    }

    err := cache.BatchSet(items, time.Hour)
    if err != nil {
        log.Printf("Batch set failed: %v", err)
    } else {
        log.Printf("Batch set completed successfully")
    }

    // Get multiple values
    keys := []string{"user:1", "user:2", "user:3", "session:1", "session:2"}
    results, err := cache.MGet(keys)
    if err != nil {
        log.Printf("MGet failed: %v", err)
    } else {
        log.Printf("Retrieved %d values", len(results))
        for key, value := range results {
            log.Printf("%s: %s", key, value)
        }
    }

    // Health check
    log.Println("\n=== Cluster Health Check ===")
    health, err := cluster.HealthCheck()
    if err != nil {
        log.Printf("Health check failed: %v", err)
    } else {
        log.Printf("Cluster healthy: %t", health.Healthy)
        for nodeID, nodeHealth := range health.Nodes {
            log.Printf("Node %s (%s): %s, Latency: %v", 
                      nodeID, nodeHealth.Address, 
                      map[bool]string{true: "Healthy", false: "Unhealthy"}[nodeHealth.Healthy],
                      nodeHealth.Latency)
        }
    }

    // Cluster info
    log.Println("\n=== Cluster Information ===")
    info, err := cluster.GetClusterInfo()
    if err != nil {
        log.Printf("Failed to get cluster info: %v", err)
    } else {
        log.Printf("Cluster has %d nodes and %d slot ranges", 
                  len(info.Nodes), len(info.Slots))
        
        for _, node := range info.Nodes {
            log.Printf("Node: %s (%s) - Role: %s, Status: %s", 
                      node.ID, node.Address, node.Role, node.Status)
        }
    }
}

Example 2: Redis Sentinel Implementation

package main

import (
    "context"
    "fmt"
    "log"
    "strings"
    "sync"
    "time"

    "github.com/redis/go-redis/v9"
)

// SentinelManager handles Redis Sentinel operations
type SentinelManager struct {
    sentinel *redis.SentinelClient
    client   *redis.Client
    ctx      context.Context
    config   SentinelConfig
    mu       sync.RWMutex
}

type SentinelConfig struct {
    SentinelAddrs []string
    MasterName    string
    Password      string
    DB            int
    PoolSize      int
    MinIdleConns  int
}

func NewSentinelManager(config SentinelConfig) *SentinelManager {
    // Create sentinel client for service discovery
    sentinelClient := redis.NewSentinelClient(&redis.Options{
        Addr: config.SentinelAddrs[0], // Primary sentinel
    })

    // Create failover client that automatically handles master changes
    failoverClient := redis.NewFailoverClient(&redis.FailoverOptions{
        MasterName:    config.MasterName,
        SentinelAddrs: config.SentinelAddrs,
        Password:      config.Password,
        DB:            config.DB,
        PoolSize:      config.PoolSize,
        MinIdleConns:  config.MinIdleConns,
        
        // Failover-specific settings
        MaxRetries:      3,
        MaxRetryBackoff: 512 * time.Millisecond,
        
        // Connection timeouts
        DialTimeout:  5 * time.Second,
        ReadTimeout:  3 * time.Second,
        WriteTimeout: 3 * time.Second,
    })

    sm := &SentinelManager{
        sentinel: sentinelClient,
        client:   failoverClient,
        ctx:      context.Background(),
        config:   config,
    }

    return sm
}

// GetMasterInfo retrieves current master information
func (sm *SentinelManager) GetMasterInfo() (*MasterInfo, error) {
    result, err := sm.sentinel.Master(sm.ctx, sm.config.MasterName).Result()
    if err != nil {
        return nil, fmt.Errorf("failed to get master info: %w", err)
    }

    return parseMasterInfo(result), nil
}

// GetReplicaInfo retrieves replica information
func (sm *SentinelManager) GetReplicaInfo() ([]ReplicaInfo, error) {
    result, err := sm.sentinel.Slaves(sm.ctx, sm.config.MasterName).Result()
    if err != nil {
        return nil, fmt.Errorf("failed to get replica info: %w", err)
    }

    var replicas []ReplicaInfo
    for _, replica := range result {
        replicas = append(replicas, parseReplicaInfo(replica))
    }

    return replicas, nil
}

// GetSentinelInfo retrieves sentinel information
func (sm *SentinelManager) GetSentinelInfo() ([]SentinelInfo, error) {
    result, err := sm.sentinel.Sentinels(sm.ctx, sm.config.MasterName).Result()
    if err != nil {
        return nil, fmt.Errorf("failed to get sentinel info: %w", err)
    }

    var sentinels []SentinelInfo
    for _, sentinel := range result {
        sentinels = append(sentinels, parseSentinelInfo(sentinel))
    }

    return sentinels, nil
}

type MasterInfo struct {
    Name               string        `json:"name"`
    IP                 string        `json:"ip"`
    Port               string        `json:"port"`
    RunID              string        `json:"runid"`
    Flags              []string      `json:"flags"`
    PendingCommands    int           `json:"pending_commands"`
    LastPingSent       time.Duration `json:"last_ping_sent"`
    LastOkPingReply    time.Duration `json:"last_ok_ping_reply"`
    LastPingReply      time.Duration `json:"last_ping_reply"`
    DownAfterPeriod    time.Duration `json:"down_after_period"`
    InfoRefresh        time.Duration `json:"info_refresh"`
    RoleReported       string        `json:"role_reported"`
    RoleReportedTime   time.Duration `json:"role_reported_time"`
    ConfigEpoch        int           `json:"config_epoch"`
    NumSlaves          int           `json:"num_slaves"`
    NumOtherSentinels  int           `json:"num_other_sentinels"`
    Quorum             int           `json:"quorum"`
    FailoverTimeout    time.Duration `json:"failover_timeout"`
    ParallelSyncs      int           `json:"parallel_syncs"`
}

type ReplicaInfo struct {
    Name               string        `json:"name"`
    IP                 string        `json:"ip"`
    Port               string        `json:"port"`
    RunID              string        `json:"runid"`
    Flags              []string      `json:"flags"`
    PendingCommands    int           `json:"pending_commands"`
    LastPingSent       time.Duration `json:"last_ping_sent"`
    LastOkPingReply    time.Duration `json:"last_ok_ping_reply"`
    LastPingReply      time.Duration `json:"last_ping_reply"`
    DownAfterPeriod    time.Duration `json:"down_after_period"`
    InfoRefresh        time.Duration `json:"info_refresh"`
    RoleReported       string        `json:"role_reported"`
    RoleReportedTime   time.Duration `json:"role_reported_time"`
    MasterLinkDownTime time.Duration `json:"master_link_down_time"`
    MasterLinkStatus   string        `json:"master_link_status"`
    MasterHost         string        `json:"master_host"`
    MasterPort         string        `json:"master_port"`
    SlavePriority      int           `json:"slave_priority"`
    SlaveReplOffset    int           `json:"slave_repl_offset"`
}

type SentinelInfo struct {
    Name               string        `json:"name"`
    IP                 string        `json:"ip"`
    Port               string        `json:"port"`
    RunID              string        `json:"runid"`
    Flags              []string      `json:"flags"`
    PendingCommands    int           `json:"pending_commands"`
    LastPingSent       time.Duration `json:"last_ping_sent"`
    LastOkPingReply    time.Duration `json:"last_ok_ping_reply"`
    LastPingReply      time.Duration `json:"last_ping_reply"`
    DownAfterPeriod    time.Duration `json:"down_after_period"`
    LastHelloMessage   time.Duration `json:"last_hello_message"`
}

// HighAvailabilityCache implements cache with automatic failover
type HighAvailabilityCache struct {
    sentinel *SentinelManager
}

func NewHighAvailabilityCache(sentinel *SentinelManager) *HighAvailabilityCache {
    return &HighAvailabilityCache{sentinel: sentinel}
}

// Set with automatic failover handling
func (hac *HighAvailabilityCache) Set(key string, value interface{}, ttl time.Duration) error {
    return hac.sentinel.client.Set(hac.sentinel.ctx, key, value, ttl).Err()
}

// Get with automatic failover handling
func (hac *HighAvailabilityCache) Get(key string) (string, error) {
    return hac.sentinel.client.Get(hac.sentinel.ctx, key).Result()
}

// SetWithRetry implements retry logic for failover scenarios
func (hac *HighAvailabilityCache) SetWithRetry(key string, value interface{}, ttl time.Duration, maxRetries int) error {
    var lastErr error
    
    for attempt := 0; attempt <= maxRetries; attempt++ {
        err := hac.Set(key, value, ttl)
        if err == nil {
            return nil
        }
        
        lastErr = err
        
        // Check if this is a failover-related error
        if isFailoverError(err) {
            log.Printf("Failover detected on attempt %d, retrying...", attempt+1)
            time.Sleep(time.Duration(attempt*100) * time.Millisecond)
            continue
        }
        
        // Non-failover error, don't retry
        return err
    }
    
    return fmt.Errorf("failed after %d retries: %w", maxRetries, lastErr)
}

// isFailoverError checks if an error is related to failover
func isFailoverError(err error) bool {
    if err == nil {
        return false
    }
    
    errStr := strings.ToLower(err.Error())
    failoverKeywords := []string{
        "connection refused",
        "connection reset",
        "no route to host",
        "timeout",
        "broken pipe",
        "readonly",
    }
    
    for _, keyword := range failoverKeywords {
        if strings.Contains(errStr, keyword) {
            return true
        }
    }
    
    return false
}

// MonitorFailover monitors for failover events
func (sm *SentinelManager) MonitorFailover() {
    // Subscribe to sentinel events
    pubsub := sm.sentinel.Subscribe(sm.ctx, "__sentinel__:hello")
    defer pubsub.Close()

    log.Println("Starting failover monitoring...")

    for {
        select {
        case msg := <-pubsub.Channel():
            log.Printf("Sentinel event: %s", msg.Payload)
            
            // Parse and handle different event types
            if strings.Contains(msg.Payload, "+switch-master") {
                log.Println("Master switch detected!")
                sm.handleMasterSwitch()
            }
            
        case <-sm.ctx.Done():
            log.Println("Failover monitoring stopped")
            return
        }
    }
}

func (sm *SentinelManager) handleMasterSwitch() {
    // Get new master info
    masterInfo, err := sm.GetMasterInfo()
    if err != nil {
        log.Printf("Failed to get new master info: %v", err)
        return
    }
    
    log.Printf("New master: %s:%s", masterInfo.IP, masterInfo.Port)
    
    // Trigger application-specific failover handling
    sm.notifyApplications(masterInfo)
}

func (sm *SentinelManager) notifyApplications(masterInfo *MasterInfo) {
    // In a real application, this would notify other components
    // about the master change
    log.Printf("Notifying applications about master change to %s:%s", 
              masterInfo.IP, masterInfo.Port)
}

// Parse helper functions
func parseMasterInfo(data []interface{}) *MasterInfo {
    info := &MasterInfo{}
    
    for i := 0; i < len(data); i += 2 {
        if i+1 >= len(data) {
            break
        }
        
        key := data[i].(string)
        value := data[i+1].(string)
        
        switch key {
        case "name":
            info.Name = value
        case "ip":
            info.IP = value
        case "port":
            info.Port = value
        case "runid":
            info.RunID = value
        case "flags":
            info.Flags = strings.Split(value, ",")
        case "role-reported":
            info.RoleReported = value
        case "num-slaves":
            fmt.Sscanf(value, "%d", &info.NumSlaves)
        case "num-other-sentinels":
            fmt.Sscanf(value, "%d", &info.NumOtherSentinels)
        case "quorum":
            fmt.Sscanf(value, "%d", &info.Quorum)
        case "parallel-syncs":
            fmt.Sscanf(value, "%d", &info.ParallelSyncs)
        }
    }
    
    return info
}

func parseReplicaInfo(data []interface{}) ReplicaInfo {
    // Similar parsing logic for replica info
    return ReplicaInfo{}
}

func parseSentinelInfo(data []interface{}) SentinelInfo {
    // Similar parsing logic for sentinel info
    return SentinelInfo{}
}

// Demo application
func main() {
    config := SentinelConfig{
        SentinelAddrs: []string{
            "127.0.0.1:26379",
            "127.0.0.1:26380",
            "127.0.0.1:26381",
        },
        MasterName:   "mymaster",
        Password:     "",
        DB:           0,
        PoolSize:     10,
        MinIdleConns: 5,
    }

    sentinel := NewSentinelManager(config)
    defer sentinel.client.Close()
    defer sentinel.sentinel.Close()

    cache := NewHighAvailabilityCache(sentinel)

    // Start failover monitoring in background
    go sentinel.MonitorFailover()

    // Test high availability operations
    log.Println("=== High Availability Demo ===")

    // Set some values with retry logic
    for i := 0; i < 10; i++ {
        key := fmt.Sprintf("ha:test:%d", i)
        value := fmt.Sprintf("value_%d", i)
        
        err := cache.SetWithRetry(key, value, time.Hour, 3)
        if err != nil {
            log.Printf("Failed to set %s: %v", key, err)
        } else {
            log.Printf("Set %s successfully", key)
        }
        
        time.Sleep(500 * time.Millisecond)
    }

    // Get master info
    log.Println("\n=== Sentinel Information ===")
    masterInfo, err := sentinel.GetMasterInfo()
    if err != nil {
        log.Printf("Failed to get master info: %v", err)
    } else {
        log.Printf("Master: %s:%s (RunID: %s)", 
                  masterInfo.IP, masterInfo.Port, masterInfo.RunID)
        log.Printf("Slaves: %d, Sentinels: %d, Quorum: %d", 
                  masterInfo.NumSlaves, masterInfo.NumOtherSentinels, masterInfo.Quorum)
    }

    // Get replica info
    replicas, err := sentinel.GetReplicaInfo()
    if err != nil {
        log.Printf("Failed to get replica info: %v", err)
    } else {
        log.Printf("Found %d replicas", len(replicas))
        for i, replica := range replicas {
            log.Printf("Replica %d: %s:%s", i+1, replica.IP, replica.Port)
        }
    }

    // Keep running to monitor failover events
    log.Println("\nMonitoring for failover events... (Press Ctrl+C to stop)")
    select {}
}

6. Hands-On Practice (30-40 minutes)

Exercise 1: Cluster Setup and Data Migration (15 minutes)

Objective: Set up a Redis cluster and perform data migration

Requirements:

  • Create 6-node cluster (3 masters, 3 replicas)
  • Distribute hash slots evenly
  • Migrate data between nodes
  • Handle resharding operations

Implementation Starter:

# Create cluster nodes
redis-server --port 7001 --cluster-enabled yes --cluster-config-file nodes-7001.conf
redis-server --port 7002 --cluster-enabled yes --cluster-config-file nodes-7002.conf
# ... repeat for ports 7003-7006

# Initialize cluster
redis-cli --cluster create 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 \
  127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 --cluster-replicas 1

# Add new node
redis-cli --cluster add-node 127.0.0.1:7007 127.0.0.1:7001

# Reshard data
redis-cli --cluster reshard 127.0.0.1:7001 --cluster-slots 1000 \
  --cluster-from <source-node-id> --cluster-to <target-node-id>

Exercise 2: Sentinel Configuration and Failover Testing (15 minutes)

Objective: Configure Redis Sentinel for high availability

Requirements:

  • Set up 3 Sentinel instances
  • Configure automatic failover
  • Test master failure scenarios
  • Implement client failover handling

Exercise 3: Performance Benchmarking (10 minutes)

Objective: Benchmark cluster vs single instance performance

Requirements:

  • Compare latency and throughput
  • Test different workload patterns
  • Measure failover recovery time
  • Analyze network overhead

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Issues

  1. Cross-slot Operations in Cluster

    • Problem: Operations span multiple hash slots
    • Solution: Use hash tags {user:123}:profile and {user:123}:settings
  2. Split-brain During Network Partitions

    • Problem: Multiple masters elected
    • Solution: Proper quorum configuration and network monitoring
  3. Sentinel Misconfiguration

    • Problem: Incorrect quorum or failover settings
    • Solution: Test failover scenarios and monitor sentinel logs

Debugging Commands

# Cluster diagnosis
redis-cli --cluster check 127.0.0.1:7001
redis-cli --cluster info 127.0.0.1:7001
redis-cli --cluster nodes 127.0.0.1:7001

# Sentinel monitoring  
redis-cli -p 26379 sentinel masters
redis-cli -p 26379 sentinel slaves mymaster
redis-cli -p 26379 sentinel sentinels mymaster

# Performance monitoring
redis-cli --cluster call 127.0.0.1:7001 info stats
redis-cli --latency -i 1 -h 127.0.0.1 -p 7001

8. Best Practices and Professional Tips (5 minutes)

Production Deployment Guidelines

# Docker Compose for Redis Cluster
version: '3.8'
services:
  redis-node1:
    image: redis:7-alpine
    command: redis-server --cluster-enabled yes --cluster-config-file nodes.conf
    volumes:
      - redis1-data:/data
    ports:
      - "7001:6379"
      - "17001:16379"
    
  redis-node2:
    image: redis:7-alpine  
    command: redis-server --cluster-enabled yes --cluster-config-file nodes.conf
    volumes:
      - redis2-data:/data
    ports:
      - "7002:6379"
      - "17002:16379"

  # ... additional nodes

Performance Optimization

  • Use consistent hashing for client-side sharding
  • Optimize pipeline sizes for cluster operations
  • Monitor cross-slot operation frequency
  • Implement connection pooling per node
  • Use read replicas for read-heavy workloads

Monitoring Checklist

  • ✅ Cluster Health: Monitor node status and slot distribution
  • ✅ Failover Events: Track master switches and recovery times
  • ✅ Network Latency: Monitor inter-node communication
  • ✅ Memory Usage: Track memory distribution across nodes
  • ✅ Client Connections: Monitor connection pooling efficiency

9. Summary and Review (5 minutes)

Key Takeaways

  1. Redis Cluster provides horizontal scaling through automatic sharding
  2. Redis Sentinel enables high availability with automatic failover
  3. Hash slots determine data distribution - design keys accordingly
  4. Cross-slot operations have limitations - use hash tags when needed
  5. Proper monitoring is essential for cluster operations

Scaling Strategy Decision Matrix

Requirement Single Instance Sentinel + Replicas Redis Cluster
High Availability ❌ ✅ ✅
Horizontal Scaling ❌ ❌ ✅
Operational Complexity Low Medium High
Cross-key Operations ✅ ✅ Limited
Maximum Throughput Limited Medium Very High

Quick Review Questions

  1. How many hash slots does Redis Cluster use?
  2. What’s the minimum number of nodes for Redis Cluster?
  3. How does Sentinel determine when to trigger failover?
  4. What are hash tags and when should you use them?
  5. How do you handle cross-slot operations in a cluster?

For Instructors

Teaching Tips

  • Use visual diagrams to show data distribution across nodes
  • Demonstrate actual failover scenarios with node shutdowns
  • Show the performance difference between single instance and cluster
  • Use monitoring tools to visualize cluster health
  • Demonstrate hash slot calculations and key distribution

Common Student Questions

  • Q: “Can I use transactions across cluster nodes?” A: Only if all keys are in the same hash slot, otherwise use individual operations

  • Q: “How do I choose between Sentinel and Cluster?” A: Sentinel for HA with vertical scaling, Cluster for horizontal scaling needs

  • Q: “What happens during network partitions?” A: Depends on quorum - majority partition continues, minority becomes read-only

Assessment Ideas

  • Design cluster architecture for specific requirements
  • Implement client-side failover handling
  • Perform cluster resharding operations
  • Build monitoring dashboard for cluster health
  • Simulate and recover from various failure scenarios