Database chevron_right Search and Analytics Databases - Full-Text Search and Log Analytics

Search and Analytics Databases - Full-Text Search and Log Analytics

calendar_today schedule 17 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Understand how search databases differ from traditional databases
  • Design schemas and mappings for full-text search applications
  • Write complex queries using Elasticsearch Query DSL
  • Implement search features like autocomplete, fuzzy matching, and faceted search
  • Build log analytics pipelines for observability and monitoring

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Type a search query into Amazon or Google. In under 100 milliseconds, you get results from billions of documents, ranked by relevance, with typo corrections, highlighted snippets, and filter options. Now try the same search with a SQL LIKE '%keyword%' query on a large table. The difference is night and day.

Search databases are specifically designed for this use case: finding needles in haystacks, instantly and intelligently.

Lesson Overview

  • Duration: 90 minutes
  • Prerequisites: Understanding of databases and JSON data formats
  • Materials Needed: Elasticsearch or OpenSearch (Docker), command line access

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Search Fundamentals - Inverted indexes, text analysis, and relevance scoring
  2. Elasticsearch Architecture - Indices, shards, and distributed search
  3. Mapping and Analysis - Defining schemas for search
  4. Query DSL - Full-text queries, filters, and aggregations
  5. Search Features - Autocomplete, highlighting, and faceted navigation
  6. Log Analytics - The ELK/EFK stack for observability

Success Criteria:

By the end of this session, you should be able to design a search index, write complex search queries, and understand how to build search-powered applications.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

How Search Works: The Inverted Index

Traditional databases use B-tree indexes that map from keys to values. Search databases use inverted indexes that map from terms to documents.

Forward Index (Traditional):

Document 1 → "The quick brown fox"
Document 2 → "The lazy brown dog"

Inverted Index (Search):

"the"   → [Doc 1, Doc 2]
"quick" → [Doc 1]
"brown" → [Doc 1, Doc 2]
"fox"   → [Doc 1]
"lazy"  → [Doc 2]
"dog"   → [Doc 2]

Real-World Analogy: Think of a book’s index at the back. Instead of scanning every page for “elasticsearch,” you look up “elasticsearch” in the index and go directly to pages 15, 47, and 102.

Why This Matters: Inverted indexes enable sub-millisecond searches across billions of documents. They also enable relevance scoring, fuzzy matching, and linguistic analysis that SQL databases can’t efficiently provide.

Text Analysis Pipeline

When text is indexed, it goes through an analysis pipeline:

"The Quick Brown FOX jumped!" 
    ↓ Character Filter (remove HTML, etc.)
"The Quick Brown FOX jumped"
    ↓ Tokenizer (split into tokens)
["The", "Quick", "Brown", "FOX", "jumped"]
    ↓ Token Filters (lowercase, stemming, etc.)
["the", "quick", "brown", "fox", "jump"]

Common Analyzers:

  • Standard: Lowercase and basic tokenization
  • English: Adds stemming (jumping → jump)
  • Keyword: No analysis, exact matching
  • Custom: Build your own pipeline

Relevance Scoring (TF-IDF and BM25)

Search databases rank results by relevance using algorithms like BM25:

  • Term Frequency (TF): How often does the term appear in this document?
  • Inverse Document Frequency (IDF): How rare is this term across all documents?
  • Field Length: Shorter documents with matches rank higher

Key Terminology

  • Index: A collection of documents (like a database table)
  • Document: A JSON object stored in an index
  • Field: A key-value pair in a document
  • Mapping: Schema definition for an index
  • Shard: A partition of an index for distribution
  • Replica: Copy of a shard for redundancy and read scaling
  • Analyzer: Pipeline for processing text during indexing
  • Query DSL: Domain-specific language for search queries
  • Aggregation: Analytics operations (like GROUP BY)
  • Relevance Score: How well a document matches a query

4. Detailed Explanation and Examples (25-30 minutes)

Elasticsearch Architecture

┌────────────────────────────────────────────────────────┐
│                    Cluster                              │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐                │
│  │  Node 1 │  │  Node 2 │  │  Node 3 │                │
│  │ Primary │  │ Replica │  │ Primary │                │
│  │ Shard 0 │  │ Shard 0 │  │ Shard 1 │                │
│  │ Replica │  │ Primary │  │ Replica │                │
│  │ Shard 1 │  │ Shard 2 │  │ Shard 2 │                │
│  └─────────┘  └─────────┘  └─────────┘                │
└────────────────────────────────────────────────────────┘

Key Concepts:

  • Cluster: One or more nodes working together
  • Node: A single Elasticsearch instance
  • Primary Shard: Original data partition
  • Replica Shard: Copy for redundancy and read performance

Elasticsearch

  • Use Case: Full-text search, log analytics, APM
  • Query Language: Query DSL (JSON)
  • Ecosystem: Kibana visualization, Beats collectors, Logstash processing
  • Best For: General-purpose search and analytics

OpenSearch

  • Use Case: Same as Elasticsearch (AWS fork)
  • Query Language: Compatible with Elasticsearch Query DSL
  • Best For: AWS users, open-source requirements

Apache Solr

  • Use Case: Enterprise search
  • Query Language: Solr Query Parser, JSON API
  • Best For: Mature deployments, extreme configurability

Meilisearch

  • Use Case: Instant search, typo tolerance
  • Query Language: Simple REST API
  • Best For: Developer experience, instant results

Typesense

  • Use Case: Search-as-a-service alternative
  • Query Language: REST API
  • Best For: Easy setup, typo tolerance

Common Misconceptions

  1. Misconception: “Elasticsearch is a primary database” Reality: Search databases are optimized for search, not as systems of record. Use alongside a primary database.

  2. Misconception: “Elasticsearch is just for logs” Reality: It’s excellent for product search, site search, security analytics, and more.

  3. Misconception: “Full-text search is just LIKE queries” Reality: Search databases provide relevance scoring, linguistic analysis, and features SQL can’t match.

5. Live Coding Demonstration (20-25 minutes)

Example 1: Creating an Index and Mapping

// Create an index with explicit mapping
PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stemmer", "english_stop"]
        }
      },
      "filter": {
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "product_analyzer",
        "fields": {
          "keyword": { "type": "keyword" },
          "autocomplete": {
            "type": "text",
            "analyzer": "autocomplete_analyzer"
          }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "product_analyzer"
      },
      "category": {
        "type": "keyword"
      },
      "brand": {
        "type": "keyword"
      },
      "price": {
        "type": "float"
      },
      "rating": {
        "type": "float"
      },
      "in_stock": {
        "type": "boolean"
      },
      "tags": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date"
      },
      "location": {
        "type": "geo_point"
      }
    }
  }
}

// Index documents
POST /products/_doc/1
{
  "name": "Wireless Bluetooth Headphones",
  "description": "Premium noise-canceling headphones with 30-hour battery life",
  "category": "Electronics",
  "brand": "AudioPro",
  "price": 199.99,
  "rating": 4.5,
  "in_stock": true,
  "tags": ["wireless", "bluetooth", "noise-canceling"],
  "created_at": "2024-01-15T10:00:00Z"
}

POST /products/_doc/2
{
  "name": "Mechanical Gaming Keyboard",
  "description": "RGB backlit keyboard with Cherry MX switches",
  "category": "Electronics",
  "brand": "GameGear",
  "price": 149.99,
  "rating": 4.8,
  "in_stock": true,
  "tags": ["gaming", "mechanical", "rgb"],
  "created_at": "2024-01-10T10:00:00Z"
}

POST /products/_doc/3
{
  "name": "Running Shoes Pro",
  "description": "Lightweight running shoes with cushioned soles",
  "category": "Sports",
  "brand": "SportFit",
  "price": 129.99,
  "rating": 4.3,
  "in_stock": false,
  "tags": ["running", "athletic", "lightweight"],
  "created_at": "2024-01-05T10:00:00Z"
}

// Bulk indexing for efficiency
POST /products/_bulk
{"index": {"_id": "4"}}
{"name": "Yoga Mat Premium", "description": "Non-slip yoga mat with alignment lines", "category": "Sports", "brand": "ZenFit", "price": 49.99, "rating": 4.6, "in_stock": true, "tags": ["yoga", "fitness", "non-slip"]}
{"index": {"_id": "5"}}
{"name": "Smart Watch Series X", "description": "Fitness tracker with heart rate monitoring and GPS", "category": "Electronics", "brand": "TechWear", "price": 299.99, "rating": 4.4, "in_stock": true, "tags": ["smartwatch", "fitness", "gps"]}

Example 2: Search Queries

// Simple match query
GET /products/_search
{
  "query": {
    "match": {
      "name": "wireless headphones"
    }
  }
}

// Multi-match across fields
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "wireless bluetooth",
      "fields": ["name^3", "description", "tags^2"],
      "type": "best_fields"
    }
  }
}

// Boolean query (must, should, must_not, filter)
GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "description": "premium" } }
      ],
      "should": [
        { "match": { "tags": "wireless" } },
        { "range": { "rating": { "gte": 4.5 } } }
      ],
      "filter": [
        { "term": { "category": "Electronics" } },
        { "term": { "in_stock": true } },
        { "range": { "price": { "lte": 200 } } }
      ],
      "must_not": [
        { "term": { "brand": "CheapBrand" } }
      ]
    }
  }
}

// Fuzzy search (typo tolerance)
GET /products/_search
{
  "query": {
    "match": {
      "name": {
        "query": "headhpones",
        "fuzziness": "AUTO"
      }
    }
  }
}

// Phrase matching
GET /products/_search
{
  "query": {
    "match_phrase": {
      "description": "noise-canceling headphones"
    }
  }
}

// Wildcard and regex
GET /products/_search
{
  "query": {
    "wildcard": {
      "name": {
        "value": "wire*"
      }
    }
  }
}

// Boosting (prefer in-stock items)
GET /products/_search
{
  "query": {
    "boosting": {
      "positive": {
        "match": { "name": "headphones" }
      },
      "negative": {
        "term": { "in_stock": false }
      },
      "negative_boost": 0.5
    }
  }
}

// Function score (custom relevance)
GET /products/_search
{
  "query": {
    "function_score": {
      "query": { "match": { "name": "headphones" } },
      "functions": [
        {
          "filter": { "term": { "in_stock": true } },
          "weight": 2
        },
        {
          "field_value_factor": {
            "field": "rating",
            "modifier": "log1p",
            "factor": 2
          }
        },
        {
          "gauss": {
            "price": {
              "origin": 100,
              "scale": 50
            }
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

// Basic aggregations
GET /products/_search
{
  "size": 0,
  "aggs": {
    "categories": {
      "terms": {
        "field": "category",
        "size": 10
      }
    },
    "brands": {
      "terms": {
        "field": "brand",
        "size": 10
      }
    },
    "price_stats": {
      "stats": {
        "field": "price"
      }
    },
    "avg_rating": {
      "avg": {
        "field": "rating"
      }
    }
  }
}

// Faceted search (combine search with aggregations)
GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "description": "premium" } }
      ],
      "filter": [
        { "term": { "category": "Electronics" } }
      ]
    }
  },
  "aggs": {
    "brands": {
      "terms": { "field": "brand" }
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "key": "budget", "to": 100 },
          { "key": "mid-range", "from": 100, "to": 200 },
          { "key": "premium", "from": 200 }
        ]
      }
    },
    "rating_histogram": {
      "histogram": {
        "field": "rating",
        "interval": 0.5,
        "min_doc_count": 0
      }
    }
  }
}

// Nested aggregations
GET /products/_search
{
  "size": 0,
  "aggs": {
    "categories": {
      "terms": {
        "field": "category"
      },
      "aggs": {
        "avg_price": {
          "avg": { "field": "price" }
        },
        "avg_rating": {
          "avg": { "field": "rating" }
        },
        "top_products": {
          "top_hits": {
            "size": 3,
            "_source": ["name", "price", "rating"],
            "sort": [{ "rating": "desc" }]
          }
        }
      }
    }
  }
}

// Date histogram (time-based analytics)
GET /products/_search
{
  "size": 0,
  "aggs": {
    "products_over_time": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month",
        "format": "yyyy-MM"
      },
      "aggs": {
        "category_breakdown": {
          "terms": { "field": "category" }
        }
      }
    }
  }
}

Example 4: Search Features (Autocomplete, Highlighting)

// Autocomplete with completion suggester
PUT /products_autocomplete
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "suggest": {
        "type": "completion"
      }
    }
  }
}

// Index with suggestions
POST /products_autocomplete/_doc/1
{
  "name": "Wireless Bluetooth Headphones",
  "suggest": {
    "input": ["Wireless Bluetooth Headphones", "headphones", "wireless"],
    "weight": 10
  }
}

// Autocomplete query
POST /products_autocomplete/_search
{
  "suggest": {
    "product-suggest": {
      "prefix": "wire",
      "completion": {
        "field": "suggest",
        "fuzzy": {
          "fuzziness": 1
        }
      }
    }
  }
}

// Search-as-you-type field
PUT /products_search_as_you_type
{
  "mappings": {
    "properties": {
      "name": {
        "type": "search_as_you_type"
      }
    }
  }
}

POST /products_search_as_you_type/_search
{
  "query": {
    "multi_match": {
      "query": "wire blue",
      "type": "bool_prefix",
      "fields": [
        "name",
        "name._2gram",
        "name._3gram"
      ]
    }
  }
}

// Highlighting search results
GET /products/_search
{
  "query": {
    "match": {
      "description": "premium headphones"
    }
  },
  "highlight": {
    "fields": {
      "description": {
        "pre_tags": ["<strong>"],
        "post_tags": ["</strong>"],
        "fragment_size": 150,
        "number_of_fragments": 3
      },
      "name": {
        "pre_tags": ["<em>"],
        "post_tags": ["</em>"]
      }
    }
  }
}

// Did you mean (spell suggestions)
POST /products/_search
{
  "suggest": {
    "text": "headhpones wireles",
    "simple-phrase": {
      "phrase": {
        "field": "name",
        "size": 3,
        "gram_size": 3,
        "direct_generator": [{
          "field": "name",
          "suggest_mode": "always"
        }]
      }
    }
  }
}

Example 5: Log Analytics (ELK Stack)

// Create index template for logs
PUT /_index_template/logs
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs_policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "message": { "type": "text" },
        "request_id": { "type": "keyword" },
        "user_id": { "type": "keyword" },
        "duration_ms": { "type": "integer" },
        "status_code": { "type": "integer" },
        "path": { "type": "keyword" },
        "method": { "type": "keyword" },
        "error": {
          "properties": {
            "type": { "type": "keyword" },
            "message": { "type": "text" },
            "stack_trace": { "type": "text" }
          }
        }
      }
    }
  }
}

// Index log entries
POST /logs-2024.01/_doc
{
  "@timestamp": "2024-01-15T10:30:00Z",
  "level": "ERROR",
  "service": "api-gateway",
  "host": "server-01",
  "message": "Connection refused to database",
  "request_id": "req-12345",
  "user_id": "user-789",
  "duration_ms": 5032,
  "status_code": 503,
  "path": "/api/users",
  "method": "GET",
  "error": {
    "type": "ConnectionRefusedException",
    "message": "Unable to connect to PostgreSQL",
    "stack_trace": "at com.example.db.Connection.connect()..."
  }
}

// Search for errors
GET /logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "sort": [{ "@timestamp": "desc" }]
}

// Error rate by service
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-24h" } }
  },
  "aggs": {
    "by_service": {
      "terms": { "field": "service" },
      "aggs": {
        "total": { "value_count": { "field": "level" } },
        "errors": {
          "filter": { "term": { "level": "ERROR" } }
        },
        "error_rate": {
          "bucket_script": {
            "buckets_path": {
              "errors": "errors._count",
              "total": "total"
            },
            "script": "params.errors / params.total * 100"
          }
        }
      }
    }
  }
}

// Latency percentiles by endpoint
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "term": { "level": "INFO" } }
      ]
    }
  },
  "aggs": {
    "by_path": {
      "terms": { "field": "path", "size": 20 },
      "aggs": {
        "latency_percentiles": {
          "percentiles": {
            "field": "duration_ms",
            "percents": [50, 90, 95, 99]
          }
        },
        "avg_latency": { "avg": { "field": "duration_ms" } }
      }
    }
  }
}

// Unique users with errors
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        { "term": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-24h" } } }
      ]
    }
  },
  "aggs": {
    "unique_affected_users": {
      "cardinality": { "field": "user_id" }
    },
    "top_affected_users": {
      "terms": {
        "field": "user_id",
        "size": 10,
        "order": { "_count": "desc" }
      }
    }
  }
}

6. Hands-On Practice (30-40 minutes)

Exercise 1: Build a Product Search (15 minutes)

Objective: Create a search implementation for an e-commerce site

Requirements:

  • Search across product name and description
  • Filter by category, price range, and availability
  • Sort by relevance, price, or rating
  • Return facets for filtering

Solution:

// Create index with proper mapping
PUT /ecommerce_products
{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stemmer"]
        }
      },
      "filter": {
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "product_analyzer",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "description": { "type": "text", "analyzer": "product_analyzer" },
      "category": { "type": "keyword" },
      "brand": { "type": "keyword" },
      "price": { "type": "float" },
      "rating": { "type": "float" },
      "in_stock": { "type": "boolean" },
      "tags": { "type": "keyword" }
    }
  }
}

// Search endpoint implementation
POST /ecommerce_products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "wireless headphones",
            "fields": ["name^3", "description", "tags^2"],
            "type": "best_fields",
            "fuzziness": "AUTO"
          }
        }
      ],
      "filter": [
        { "term": { "in_stock": true } },
        { "terms": { "category": ["Electronics", "Audio"] } },
        { "range": { "price": { "gte": 50, "lte": 300 } } }
      ]
    }
  },
  "sort": [
    "_score",
    { "rating": "desc" }
  ],
  "aggs": {
    "categories": {
      "terms": { "field": "category" }
    },
    "brands": {
      "terms": { "field": "brand" }
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "key": "Under $50", "to": 50 },
          { "key": "$50-$100", "from": 50, "to": 100 },
          { "key": "$100-$200", "from": 100, "to": 200 },
          { "key": "Over $200", "from": 200 }
        ]
      }
    },
    "rating_filter": {
      "range": {
        "field": "rating",
        "ranges": [
          { "key": "4+ stars", "from": 4 },
          { "key": "3+ stars", "from": 3 },
          { "key": "2+ stars", "from": 2 }
        ]
      }
    }
  },
  "highlight": {
    "fields": {
      "name": {},
      "description": { "fragment_size": 150 }
    }
  },
  "from": 0,
  "size": 20
}

Exercise 2: Build an Autocomplete System (10 minutes)

Objective: Implement search-as-you-type functionality

Solution:

// Create index with autocomplete capabilities
PUT /autocomplete_demo
{
  "settings": {
    "analysis": {
      "filter": {
        "autocomplete_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20
        }
      },
      "analyzer": {
        "autocomplete": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "autocomplete_filter"]
        },
        "autocomplete_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "autocomplete_search"
      },
      "popularity": {
        "type": "integer"
      }
    }
  }
}

// Index products
POST /autocomplete_demo/_bulk
{"index": {}}
{"name": "Wireless Headphones", "popularity": 100}
{"index": {}}
{"name": "Wireless Keyboard", "popularity": 80}
{"index": {}}
{"name": "Wireless Mouse", "popularity": 90}
{"index": {}}
{"name": "Wired Earbuds", "popularity": 70}

// Autocomplete query
GET /autocomplete_demo/_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "name": {
            "query": "wire",
            "operator": "and"
          }
        }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "popularity",
            "factor": 1.2,
            "modifier": "sqrt"
          }
        }
      ]
    }
  },
  "size": 5
}

Exercise 3: Log Analysis Dashboard Queries (15 minutes)

Objective: Write queries for a log analytics dashboard

Challenge: Create queries for:

  1. Request rate over time
  2. Error rate by service
  3. Slowest endpoints
  4. Geographic distribution of requests

Solution:

// 1. Request rate over time (time series)
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-24h" } }
  },
  "aggs": {
    "requests_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "5m"
      },
      "aggs": {
        "by_status": {
          "terms": {
            "field": "status_code",
            "size": 5
          }
        }
      }
    }
  }
}

// 2. Error rate by service
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-1h" } }
  },
  "aggs": {
    "services": {
      "terms": { "field": "service" },
      "aggs": {
        "total_requests": { "value_count": { "field": "@timestamp" } },
        "errors": {
          "filter": { "range": { "status_code": { "gte": 500 } } }
        },
        "error_rate_percent": {
          "bucket_script": {
            "buckets_path": {
              "errors": "errors._count",
              "total": "total_requests"
            },
            "script": "params.total > 0 ? (params.errors / params.total) * 100 : 0"
          }
        }
      }
    }
  }
}

// 3. Slowest endpoints (P95 latency)
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "exists": { "field": "duration_ms" } }
      ]
    }
  },
  "aggs": {
    "endpoints": {
      "terms": {
        "field": "path",
        "size": 20,
        "order": { "p95_latency": "desc" }
      },
      "aggs": {
        "p95_latency": {
          "percentiles": {
            "field": "duration_ms",
            "percents": [95]
          }
        },
        "avg_latency": { "avg": { "field": "duration_ms" } },
        "request_count": { "value_count": { "field": "path" } }
      }
    }
  }
}

// 4. Geographic distribution (if you have geo data)
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-24h" } }
  },
  "aggs": {
    "by_country": {
      "terms": { "field": "geo.country_code", "size": 50 },
      "aggs": {
        "by_city": {
          "terms": { "field": "geo.city", "size": 10 }
        },
        "unique_users": {
          "cardinality": { "field": "user_id" }
        }
      }
    }
  }
}

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Search returning unexpected results Cause: Not understanding text analysis Solution: Use the _analyze API to see how text is tokenized

  2. Problem: Slow search queries Cause: Not using filters for non-scoring criteria Solution: Use filter context for exact matches; saves scoring computation

  3. Problem: High memory usage Cause: High cardinality aggregations Solution: Limit aggregation bucket sizes; use sampler aggregation

  4. Problem: Relevance scores not as expected Cause: Default scoring doesn’t match business logic Solution: Use function_score to customize; boost important fields

Debugging Tips

  • Use _analyze API: Test how text is processed
  • Use explain=true: See how scores are calculated
  • Use profile=true: Identify slow query components
  • Check mapping: Ensure fields have expected types
  • Monitor cluster health: Watch for yellow/red status

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Index Strategy: Use time-based indices for logs (logs-2024.01.15)
  • Mapping: Always define explicit mappings; don’t rely on dynamic
  • Analyzers: Choose appropriate analyzers for your language/domain
  • Aliases: Use aliases for zero-downtime index changes
  • ILM: Implement Index Lifecycle Management for log retention

Performance Tips

  • Use filters: Non-scoring queries are cacheable
  • Shard size: Target 10-50GB per shard
  • Bulk operations: Batch writes for efficiency
  • Refresh interval: Increase for write-heavy workloads
  • Source filtering: Request only needed fields

Next Steps in Your Learning Journey

  • ELK Stack: Kibana visualization, Logstash pipelines
  • Observability: APM, distributed tracing
  • Search Relevance: Learning to Rank, synonyms, boosting
  • Scaling: Cluster management, cross-cluster search

9. Summary and Review (5 minutes)

Key Takeaways

  1. Search databases use inverted indexes for fast full-text search
  2. Text analysis (tokenization, stemming, filters) is crucial for search quality
  3. Query DSL provides rich query types: match, bool, function_score
  4. Aggregations enable analytics and faceted navigation

Quick Review Questions

  1. What’s the difference between match and term queries? Answer: Match applies analysis to query text (full-text search); term looks for exact keyword values (no analysis)

  2. When should you use filter context vs query context? Answer: Filter for yes/no criteria (category, price range) - cacheable, no scoring; Query for relevance-based matching

  3. What is an inverted index? Answer: Data structure mapping terms to documents containing them, enabling fast lookup by term

For Instructors

Teaching Tips

  • Demo the _analyze API to show how text processing works
  • Compare SQL LIKE queries with Elasticsearch to show speed difference
  • Build a simple search UI to make concepts tangible
  • Show real Kibana dashboards for log analysis

Assessment Ideas

  • Design search schema for specific domain
  • Write queries for business requirements
  • Optimize slow queries
  • Build end-to-end search feature

Adaptation Notes

  • For beginners: Focus on basic queries before aggregations
  • For advanced students: Cover advanced scoring, percolators, machine learning
  • Online delivery: Use Elastic Cloud free trial for shared environment
  • Time constraints: Basic search concepts can be covered in 45 minutes