Go JSON, XML, and Data Processing
Learning Objectives: By the end of this lesson, you will be able to:
- Parse and generate JSON data with Go’s encoding/json package
- Work with XML data for legacy system integration
- Implement data transformation and validation pipelines
- Handle various data formats (CSV, YAML, etc.)
- Build robust data processing applications
- Integrate with external APIs and services
1. Welcome and Introduction (5 minutes)
Icebreaker Activity
Think about all the data flowing through modern applications - API responses, configuration files, user uploads, database exports. How do we make sense of this data soup? Today we’ll learn how Go handles the most common data formats and helps us build powerful data processing systems.
Lesson Overview
- Duration: 2-3 days (6-8 hours)
- Prerequisites: Basic Go syntax, structs, interfaces, HTTP basics
- Materials Needed: Go development environment, API access for testing
2. Learning Objectives and Session Outline (5 minutes)
What We’ll Cover Today:
- JSON Processing - Marshaling, unmarshaling, and advanced techniques
- XML Handling - Parsing and generating XML data
- Data Validation - Ensuring data integrity and correctness
- Data Transformation - Converting between formats and structures
- CSV Processing - Working with tabular data
- API Integration - Consuming and processing external data
- Performance Optimization - Efficient data processing techniques
Success Criteria:
By the end of this session, you should be able to build complete data processing pipelines that handle multiple formats and integrate with external services.
3. Foundation Knowledge (15-20 minutes)
Core Concepts
Data Serialization in Go
Go provides excellent built-in support for data serialization through the encoding package family. The most commonly used are:
encoding/json- JSON processingencoding/xml- XML processingencoding/csv- CSV processing
Real-World Analogy: Think of data serialization like translating between languages. Your Go structs speak “Go,” but APIs speak “JSON,” config files speak “YAML,” and databases speak “SQL.” Serialization is the translation layer.
Why This Matters: Modern applications constantly exchange data with external services, databases, and users. Understanding data processing is essential for building robust, interoperable systems.
Struct Tags and Reflection
Go uses struct tags to control how data is serialized and deserialized. These tags tell the encoder/decoder how to map between Go fields and external formats.
Key Terminology
- Marshal: Convert Go data structures to external format (JSON, XML, etc.)
- Unmarshal: Convert external format back to Go data structures
- Struct Tags: Metadata attached to struct fields for serialization control
- Schema: Definition of data structure and validation rules
- Pipeline: Series of data processing steps
4. Detailed Explanation and Examples (25-30 minutes)
Understanding JSON Processing
JSON is the most common data interchange format in modern web applications. Go’s encoding/json package provides powerful tools for working with JSON data.
Basic JSON Operations
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type User struct {
ID int `json:"id"`
Name string `json:"name"`
Email string `json:"email"`
CreatedAt time.Time `json:"created_at"`
IsActive bool `json:"is_active"`
Profile Profile `json:"profile,omitempty"`
}
type Profile struct {
Bio string `json:"bio,omitempty"`
Website string `json:"website,omitempty"`
Age int `json:"age,omitempty"`
}
func basicJSONExample() {
// Create a user
user := User{
ID: 1,
Name: "Alice Johnson",
Email: "alice@example.com",
CreatedAt: time.Now(),
IsActive: true,
Profile: Profile{
Bio: "Software Engineer",
Website: "https://alice.dev",
Age: 28,
},
}
// Marshal to JSON
jsonData, err := json.Marshal(user)
if err != nil {
log.Fatal(err)
}
fmt.Printf("JSON: %s\n", jsonData)
// Pretty print JSON
prettyJSON, err := json.MarshalIndent(user, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Pretty JSON:\n%s\n", prettyJSON)
// Unmarshal from JSON
var parsedUser User
err = json.Unmarshal(jsonData, &parsedUser)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Parsed User: %+v\n", parsedUser)
}
Advanced JSON Techniques
package main
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"time"
)
// Custom JSON marshaling for special formatting
type CustomTime struct {
time.Time
}
func (ct CustomTime) MarshalJSON() ([]byte, error) {
// Custom time format
return json.Marshal(ct.Time.Format("2006-01-02"))
}
func (ct *CustomTime) UnmarshalJSON(data []byte) error {
// Remove quotes from JSON string
str := strings.Trim(string(data), "\"")
t, err := time.Parse("2006-01-02", str)
if err != nil {
return err
}
ct.Time = t
return nil
}
// Flexible numeric type that can handle strings or numbers
type FlexibleInt int
func (fi *FlexibleInt) UnmarshalJSON(data []byte) error {
// Try as number first
var num int
if err := json.Unmarshal(data, &num); err == nil {
*fi = FlexibleInt(num)
return nil
}
// Try as string
var str string
if err := json.Unmarshal(data, &str); err != nil {
return err
}
num, err := strconv.Atoi(str)
if err != nil {
return err
}
*fi = FlexibleInt(num)
return nil
}
type Event struct {
ID FlexibleInt `json:"id"`
Title string `json:"title"`
Date CustomTime `json:"date"`
Metadata json.RawMessage `json:"metadata,omitempty"`
}
func advancedJSONExample() {
// JSON with mixed types
jsonStr := `{
"id": "123",
"title": "Go Workshop",
"date": "2024-12-25",
"metadata": {"venue": "Conference Hall", "capacity": 100}
}`
var event Event
err := json.Unmarshal([]byte(jsonStr), &event)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Event: %+v\n", event)
fmt.Printf("Date: %s\n", event.Date.Format("January 2, 2006"))
// Process raw metadata later
if event.Metadata != nil {
var metadata map[string]interface{}
json.Unmarshal(event.Metadata, &metadata)
fmt.Printf("Metadata: %+v\n", metadata)
}
}
Working with Dynamic JSON
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func dynamicJSONExample() {
// Unknown JSON structure
jsonStr := `{
"users": [
{"name": "Alice", "age": 30, "skills": ["Go", "Python"]},
{"name": "Bob", "age": 25, "department": "Engineering"}
],
"total": 2,
"active": true
}`
var data map[string]interface{}
err := json.Unmarshal([]byte(jsonStr), &data)
if err != nil {
log.Fatal(err)
}
// Navigate the dynamic structure
if users, ok := data["users"].([]interface{}); ok {
for i, user := range users {
if userMap, ok := user.(map[string]interface{}); ok {
fmt.Printf("User %d:\n", i+1)
for key, value := range userMap {
fmt.Printf(" %s: %v (%s)\n", key, value, reflect.TypeOf(value))
}
}
}
}
// Type assertions for specific values
if total, ok := data["total"].(float64); ok {
fmt.Printf("Total users: %.0f\n", total)
}
if active, ok := data["active"].(bool); ok {
fmt.Printf("Active: %t\n", active)
}
}
XML Processing
package main
import (
"encoding/xml"
"fmt"
"log"
)
type Book struct {
XMLName xml.Name `xml:"book"`
ID string `xml:"id,attr"`
Title string `xml:"title"`
Author Author `xml:"author"`
Price float64 `xml:"price"`
Tags []string `xml:"tags>tag"`
}
type Author struct {
Name string `xml:"name"`
Email string `xml:"email,omitempty"`
}
type Library struct {
XMLName xml.Name `xml:"library"`
Name string `xml:"name,attr"`
Books []Book `xml:"book"`
}
func xmlExample() {
// Create library data
library := Library{
Name: "City Library",
Books: []Book{
{
ID: "1",
Title: "Learning Go",
Author: Author{
Name: "Go Gopher",
Email: "gopher@golang.org",
},
Price: 29.99,
Tags: []string{"programming", "golang", "tutorial"},
},
{
ID: "2",
Title: "Advanced Patterns",
Author: Author{
Name: "Expert Coder",
},
Price: 39.99,
Tags: []string{"programming", "patterns", "advanced"},
},
},
}
// Marshal to XML
xmlData, err := xml.MarshalIndent(library, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Printf("XML:\n%s\n", xml.Header+string(xmlData))
// Unmarshal from XML
var parsedLibrary Library
err = xml.Unmarshal(xmlData, &parsedLibrary)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Parsed Library: %+v\n", parsedLibrary)
// Access individual books
for i, book := range parsedLibrary.Books {
fmt.Printf("Book %d: %s by %s ($%.2f)\n",
i+1, book.Title, book.Author.Name, book.Price)
}
}
5. Live Coding Demonstration (20-25 minutes)
Example 1: API Data Processor
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"sync"
"time"
)
// API Response structures
type GitHubUser struct {
Login string `json:"login"`
ID int `json:"id"`
Name string `json:"name"`
Company string `json:"company"`
Blog string `json:"blog"`
Location string `json:"location"`
Email string `json:"email"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
type GitHubRepo struct {
ID int `json:"id"`
Name string `json:"name"`
FullName string `json:"full_name"`
Description string `json:"description"`
Language string `json:"language"`
StarCount int `json:"stargazers_count"`
ForkCount int `json:"forks_count"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// Processed data structures
type UserProfile struct {
Username string `json:"username"`
DisplayName string `json:"display_name"`
Company string `json:"company,omitempty"`
Location string `json:"location,omitempty"`
JoinDate string `json:"join_date"`
RepoCount int `json:"repo_count"`
TotalStars int `json:"total_stars"`
Languages []string `json:"languages"`
TopRepos []RepoSummary `json:"top_repos"`
}
type RepoSummary struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Language string `json:"language,omitempty"`
Stars int `json:"stars"`
}
type APIProcessor struct {
client *http.Client
baseURL string
cache map[string]interface{}
cacheMu sync.RWMutex
}
func NewAPIProcessor() *APIProcessor {
return &APIProcessor{
client: &http.Client{
Timeout: 10 * time.Second,
},
baseURL: "https://api.github.com",
cache: make(map[string]interface{}),
}
}
func (ap *APIProcessor) fetchUser(username string) (*GitHubUser, error) {
// Check cache first
ap.cacheMu.RLock()
if cached, exists := ap.cache["user_"+username]; exists {
ap.cacheMu.RUnlock()
if user, ok := cached.(*GitHubUser); ok {
return user, nil
}
}
ap.cacheMu.RUnlock()
url := fmt.Sprintf("%s/users/%s", ap.baseURL, username)
resp, err := ap.client.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("API error: %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var user GitHubUser
if err := json.Unmarshal(body, &user); err != nil {
return nil, err
}
// Cache the result
ap.cacheMu.Lock()
ap.cache["user_"+username] = &user
ap.cacheMu.Unlock()
return &user, nil
}
func (ap *APIProcessor) fetchUserRepos(username string) ([]GitHubRepo, error) {
url := fmt.Sprintf("%s/users/%s/repos?sort=stars&direction=desc&per_page=100",
ap.baseURL, username)
resp, err := ap.client.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("API error: %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var repos []GitHubRepo
if err := json.Unmarshal(body, &repos); err != nil {
return nil, err
}
return repos, nil
}
func (ap *APIProcessor) ProcessUserProfile(username string) (*UserProfile, error) {
// Fetch user data
user, err := ap.fetchUser(username)
if err != nil {
return nil, fmt.Errorf("failed to fetch user: %w", err)
}
// Fetch repositories
repos, err := ap.fetchUserRepos(username)
if err != nil {
return nil, fmt.Errorf("failed to fetch repos: %w", err)
}
// Process and aggregate data
profile := &UserProfile{
Username: user.Login,
DisplayName: user.Name,
Company: user.Company,
Location: user.Location,
JoinDate: user.CreatedAt.Format("2006-01-02"),
RepoCount: len(repos),
}
// Calculate statistics
totalStars := 0
languageMap := make(map[string]bool)
var topRepos []RepoSummary
for i, repo := range repos {
totalStars += repo.StarCount
if repo.Language != "" {
languageMap[repo.Language] = true
}
// Get top 5 repositories
if i < 5 {
topRepos = append(topRepos, RepoSummary{
Name: repo.Name,
Description: repo.Description,
Language: repo.Language,
Stars: repo.StarCount,
})
}
}
profile.TotalStars = totalStars
profile.TopRepos = topRepos
// Convert language map to slice
for lang := range languageMap {
profile.Languages = append(profile.Languages, lang)
}
return profile, nil
}
func (ap *APIProcessor) SaveProfile(profile *UserProfile, filename string) error {
data, err := json.MarshalIndent(profile, "", " ")
if err != nil {
return err
}
return os.WriteFile(filename, data, 0644)
}
func apiProcessorExample() {
processor := NewAPIProcessor()
usernames := []string{"torvalds", "geerlingguy", "kelseyhightower"}
for _, username := range usernames {
fmt.Printf("Processing user: %s\n", username)
profile, err := processor.ProcessUserProfile(username)
if err != nil {
fmt.Printf("Error processing %s: %v\n", username, err)
continue
}
// Save to file
filename := fmt.Sprintf("%s_profile.json", username)
err = processor.SaveProfile(profile, filename)
if err != nil {
fmt.Printf("Error saving profile: %v\n", err)
continue
}
// Display summary
fmt.Printf("User: %s (%s)\n", profile.DisplayName, profile.Username)
fmt.Printf("Repositories: %d\n", profile.RepoCount)
fmt.Printf("Total Stars: %d\n", profile.TotalStars)
fmt.Printf("Languages: %v\n", profile.Languages)
fmt.Printf("Saved to: %s\n\n", filename)
}
}
Example 2: Data Validation and Transformation Pipeline
package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"os"
"strconv"
"strings"
"time"
)
// Input data structures
type RawEmployee struct {
ID string `csv:"id"`
FirstName string `csv:"first_name"`
LastName string `csv:"last_name"`
Email string `csv:"email"`
Department string `csv:"department"`
Salary string `csv:"salary"`
HireDate string `csv:"hire_date"`
IsActive string `csv:"is_active"`
}
// Output data structures
type Employee struct {
ID int `json:"id"`
FullName string `json:"full_name"`
Email string `json:"email"`
Department string `json:"department"`
Salary float64 `json:"salary"`
HireDate time.Time `json:"hire_date"`
IsActive bool `json:"is_active"`
YearsOfService int `json:"years_of_service"`
}
type ValidationError struct {
Row int `json:"row"`
Field string `json:"field"`
Value string `json:"value"`
Reason string `json:"reason"`
}
type ProcessingReport struct {
TotalRows int `json:"total_rows"`
ValidRows int `json:"valid_rows"`
InvalidRows int `json:"invalid_rows"`
Errors []ValidationError `json:"errors,omitempty"`
ProcessedAt time.Time `json:"processed_at"`
}
type DataProcessor struct {
validationRules map[string]func(string) error
}
func NewDataProcessor() *DataProcessor {
dp := &DataProcessor{
validationRules: make(map[string]func(string) error),
}
// Define validation rules
dp.validationRules["email"] = dp.validateEmail
dp.validationRules["salary"] = dp.validateSalary
dp.validationRules["department"] = dp.validateDepartment
return dp
}
func (dp *DataProcessor) validateEmail(email string) error {
if !strings.Contains(email, "@") || !strings.Contains(email, ".") {
return fmt.Errorf("invalid email format")
}
return nil
}
func (dp *DataProcessor) validateSalary(salary string) error {
amount, err := strconv.ParseFloat(salary, 64)
if err != nil {
return fmt.Errorf("invalid salary format")
}
if amount < 0 || amount > 1000000 {
return fmt.Errorf("salary out of valid range")
}
return nil
}
func (dp *DataProcessor) validateDepartment(dept string) error {
validDepts := []string{"Engineering", "Marketing", "Sales", "HR", "Finance"}
for _, valid := range validDepts {
if strings.EqualFold(dept, valid) {
return nil
}
}
return fmt.Errorf("invalid department")
}
func (dp *DataProcessor) ProcessCSVFile(filename string) (*ProcessingReport, []Employee, error) {
file, err := os.Open(filename)
if err != nil {
return nil, nil, err
}
defer file.Close()
reader := csv.NewReader(file)
// Read header
header, err := reader.Read()
if err != nil {
return nil, nil, err
}
var employees []Employee
var errors []ValidationError
rowNum := 1 // Start from 1 (header is row 0)
for {
rowNum++
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "csv",
Reason: err.Error(),
})
continue
}
// Create raw employee from CSV row
raw := dp.parseCSVRow(header, record)
// Validate and transform
employee, validationErrors := dp.validateAndTransform(raw, rowNum)
if len(validationErrors) > 0 {
errors = append(errors, validationErrors...)
continue
}
employees = append(employees, employee)
}
report := &ProcessingReport{
TotalRows: rowNum - 1, // Exclude header
ValidRows: len(employees),
InvalidRows: len(errors),
Errors: errors,
ProcessedAt: time.Now(),
}
return report, employees, nil
}
func (dp *DataProcessor) parseCSVRow(header, record []string) RawEmployee {
raw := RawEmployee{}
for i, value := range record {
if i >= len(header) {
break
}
switch header[i] {
case "id":
raw.ID = value
case "first_name":
raw.FirstName = value
case "last_name":
raw.LastName = value
case "email":
raw.Email = value
case "department":
raw.Department = value
case "salary":
raw.Salary = value
case "hire_date":
raw.HireDate = value
case "is_active":
raw.IsActive = value
}
}
return raw
}
func (dp *DataProcessor) validateAndTransform(raw RawEmployee, rowNum int) (Employee, []ValidationError) {
var errors []ValidationError
employee := Employee{}
// Validate and convert ID
id, err := strconv.Atoi(raw.ID)
if err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "id",
Value: raw.ID,
Reason: "invalid integer format",
})
} else {
employee.ID = id
}
// Combine names
employee.FullName = strings.TrimSpace(raw.FirstName + " " + raw.LastName)
if employee.FullName == "" {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "name",
Reason: "name cannot be empty",
})
}
// Validate email
if err := dp.validationRules["email"](raw.Email); err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "email",
Value: raw.Email,
Reason: err.Error(),
})
} else {
employee.Email = strings.ToLower(raw.Email)
}
// Validate department
if err := dp.validationRules["department"](raw.Department); err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "department",
Value: raw.Department,
Reason: err.Error(),
})
} else {
employee.Department = raw.Department
}
// Validate and convert salary
if err := dp.validationRules["salary"](raw.Salary); err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "salary",
Value: raw.Salary,
Reason: err.Error(),
})
} else {
if salary, err := strconv.ParseFloat(raw.Salary, 64); err == nil {
employee.Salary = salary
}
}
// Parse hire date
hireDate, err := time.Parse("2006-01-02", raw.HireDate)
if err != nil {
errors = append(errors, ValidationError{
Row: rowNum,
Field: "hire_date",
Value: raw.HireDate,
Reason: "invalid date format (expected YYYY-MM-DD)",
})
} else {
employee.HireDate = hireDate
employee.YearsOfService = int(time.Since(hireDate).Hours() / 24 / 365)
}
// Parse active status
employee.IsActive = strings.EqualFold(raw.IsActive, "true") || raw.IsActive == "1"
return employee, errors
}
func (dp *DataProcessor) SaveResults(employees []Employee, report *ProcessingReport, outputDir string) error {
// Create output directory
os.MkdirAll(outputDir, 0755)
// Save employees as JSON
employeesData, err := json.MarshalIndent(employees, "", " ")
if err != nil {
return err
}
err = os.WriteFile(fmt.Sprintf("%s/employees.json", outputDir), employeesData, 0644)
if err != nil {
return err
}
// Save processing report
reportData, err := json.MarshalIndent(report, "", " ")
if err != nil {
return err
}
return os.WriteFile(fmt.Sprintf("%s/processing_report.json", outputDir), reportData, 0644)
}
func createSampleCSV() error {
file, err := os.Create("employees.csv")
if err != nil {
return err
}
defer file.Close()
writer := csv.NewWriter(file)
defer writer.Flush()
// Header
writer.Write([]string{"id", "first_name", "last_name", "email", "department", "salary", "hire_date", "is_active"})
// Sample data with some invalid entries
records := [][]string{
{"1", "John", "Doe", "john.doe@company.com", "Engineering", "75000", "2020-01-15", "true"},
{"2", "Jane", "Smith", "jane.smith@company.com", "Marketing", "65000", "2019-06-01", "true"},
{"invalid", "Bob", "Johnson", "bob.invalid.email", "Engineering", "80000", "2021-03-10", "false"}, // Invalid ID and email
{"4", "", "", "alice@company.com", "InvalidDept", "120000", "2018-12-01", "true"}, // Missing name, invalid dept
{"5", "Charlie", "Brown", "charlie@company.com", "Sales", "invalid_salary", "invalid_date", "1"}, // Invalid salary and date
{"6", "Diana", "Wilson", "diana@company.com", "HR", "55000", "2022-01-01", "true"},
}
for _, record := range records {
writer.Write(record)
}
return nil
}
func dataProcessingExample() {
// Create sample CSV file
fmt.Println("Creating sample CSV file...")
err := createSampleCSV()
if err != nil {
log.Fatal(err)
}
// Process the data
processor := NewDataProcessor()
fmt.Println("Processing CSV file...")
report, employees, err := processor.ProcessCSVFile("employees.csv")
if err != nil {
log.Fatal(err)
}
// Display results
fmt.Printf("Processing complete!\n")
fmt.Printf("Total rows: %d\n", report.TotalRows)
fmt.Printf("Valid rows: %d\n", report.ValidRows)
fmt.Printf("Invalid rows: %d\n", report.InvalidRows)
if len(report.Errors) > 0 {
fmt.Println("\nValidation errors:")
for _, err := range report.Errors {
fmt.Printf(" Row %d, Field %s: %s (value: %s)\n",
err.Row, err.Field, err.Reason, err.Value)
}
}
// Save results
err = processor.SaveResults(employees, report, "output")
if err != nil {
log.Fatal(err)
}
fmt.Println("\nResults saved to output/ directory")
fmt.Println("- employees.json: Valid employee records")
fmt.Println("- processing_report.json: Processing summary and errors")
}
Example 3: Multi-Format Configuration Manager
package main
import (
"encoding/json"
"encoding/xml"
"fmt"
"gopkg.in/yaml.v2"
"os"
"path/filepath"
"strings"
)
type DatabaseConfig struct {
Host string `json:"host" yaml:"host" xml:"host"`
Port int `json:"port" yaml:"port" xml:"port"`
Database string `json:"database" yaml:"database" xml:"database"`
Username string `json:"username" yaml:"username" xml:"username"`
Password string `json:"password" yaml:"password" xml:"password"`
SSL bool `json:"ssl" yaml:"ssl" xml:"ssl"`
}
type ServerConfig struct {
Port int `json:"port" yaml:"port" xml:"port"`
Debug bool `json:"debug" yaml:"debug" xml:"debug"`
Workers int `json:"workers" yaml:"workers" xml:"workers"`
}
type AppConfig struct {
XMLName xml.Name `xml:"config,omitempty"`
AppName string `json:"app_name" yaml:"app_name" xml:"app_name"`
Version string `json:"version" yaml:"version" xml:"version"`
Database DatabaseConfig `json:"database" yaml:"database" xml:"database"`
Server ServerConfig `json:"server" yaml:"server" xml:"server"`
Features []string `json:"features" yaml:"features" xml:"features>feature"`
}
type ConfigManager struct {
config AppConfig
}
func NewConfigManager() *ConfigManager {
return &ConfigManager{
config: AppConfig{
AppName: "MyApp",
Version: "1.0.0",
Database: DatabaseConfig{
Host: "localhost",
Port: 5432,
Database: "myapp",
Username: "user",
Password: "password",
SSL: true,
},
Server: ServerConfig{
Port: 8080,
Debug: false,
Workers: 4,
},
Features: []string{"auth", "logging", "metrics"},
},
}
}
func (cm *ConfigManager) LoadFromFile(filename string) error {
data, err := os.ReadFile(filename)
if err != nil {
return fmt.Errorf("failed to read config file: %w", err)
}
ext := strings.ToLower(filepath.Ext(filename))
switch ext {
case ".json":
return json.Unmarshal(data, &cm.config)
case ".yaml", ".yml":
return yaml.Unmarshal(data, &cm.config)
case ".xml":
return xml.Unmarshal(data, &cm.config)
default:
return fmt.Errorf("unsupported config format: %s", ext)
}
}
func (cm *ConfigManager) SaveToFile(filename string) error {
ext := strings.ToLower(filepath.Ext(filename))
var data []byte
var err error
switch ext {
case ".json":
data, err = json.MarshalIndent(cm.config, "", " ")
case ".yaml", ".yml":
data, err = yaml.Marshal(cm.config)
case ".xml":
data, err = xml.MarshalIndent(cm.config, "", " ")
if err == nil {
data = append([]byte(xml.Header), data...)
}
default:
return fmt.Errorf("unsupported config format: %s", ext)
}
if err != nil {
return fmt.Errorf("failed to marshal config: %w", err)
}
return os.WriteFile(filename, data, 0644)
}
func (cm *ConfigManager) ConvertFormat(inputFile, outputFile string) error {
// Load from input file
err := cm.LoadFromFile(inputFile)
if err != nil {
return fmt.Errorf("failed to load input file: %w", err)
}
// Save to output file
err = cm.SaveToFile(outputFile)
if err != nil {
return fmt.Errorf("failed to save output file: %w", err)
}
return nil
}
func (cm *ConfigManager) GetConfig() AppConfig {
return cm.config
}
func (cm *ConfigManager) UpdateDatabase(dbConfig DatabaseConfig) {
cm.config.Database = dbConfig
}
func (cm *ConfigManager) UpdateServer(serverConfig ServerConfig) {
cm.config.Server = serverConfig
}
func (cm *ConfigManager) AddFeature(feature string) {
for _, existing := range cm.config.Features {
if existing == feature {
return // Feature already exists
}
}
cm.config.Features = append(cm.config.Features, feature)
}
func configManagerExample() {
cm := NewConfigManager()
// Save in different formats
formats := []string{".json", ".yaml", ".xml"}
for _, format := range formats {
filename := "config" + format
err := cm.SaveToFile(filename)
if err != nil {
fmt.Printf("Error saving %s: %v\n", filename, err)
continue
}
fmt.Printf("Saved configuration to %s\n", filename)
}
// Demonstrate format conversion
fmt.Println("\nConverting formats...")
conversions := [][2]string{
{"config.json", "config_from_json.yaml"},
{"config.yaml", "config_from_yaml.xml"},
{"config.xml", "config_from_xml.json"},
}
for _, conversion := range conversions {
err := cm.ConvertFormat(conversion[0], conversion[1])
if err != nil {
fmt.Printf("Error converting %s to %s: %v\n",
conversion[0], conversion[1], err)
continue
}
fmt.Printf("Converted %s to %s\n", conversion[0], conversion[1])
}
// Modify configuration
fmt.Println("\nModifying configuration...")
cm.AddFeature("caching")
cm.UpdateServer(ServerConfig{
Port: 9090,
Debug: true,
Workers: 8,
})
// Save modified configuration
err := cm.SaveToFile("config_modified.json")
if err != nil {
fmt.Printf("Error saving modified config: %v\n", err)
} else {
fmt.Println("Saved modified configuration to config_modified.json")
}
// Display final configuration
config := cm.GetConfig()
fmt.Printf("\nFinal configuration:\n")
fmt.Printf("App: %s v%s\n", config.AppName, config.Version)
fmt.Printf("Database: %s@%s:%d\n", config.Database.Username, config.Database.Host, config.Database.Port)
fmt.Printf("Server: port %d, debug %t, workers %d\n", config.Server.Port, config.Server.Debug, config.Server.Workers)
fmt.Printf("Features: %v\n", config.Features)
}
6. Hands-On Practice (30-40 minutes)
Exercise 1: Weather Data Processor (10 minutes)
Objective: Build a tool that processes weather API data
Instructions:
- Create structs for weather data (temperature, humidity, conditions)
- Fetch data from a weather API (or use sample data)
- Process and aggregate daily/weekly statistics
- Save results in multiple formats (JSON, CSV)
Starter Code:
type WeatherData struct {
Date time.Time `json:"date"`
Temperature float64 `json:"temperature"`
Humidity int `json:"humidity"`
Condition string `json:"condition"`
WindSpeed float64 `json:"wind_speed"`
}
func main() {
// TODO: Implement weather data processor
}
Exercise 2: Log Parser and Analyzer (15 minutes)
Objective: Parse log files and extract meaningful insights
Requirements:
- Parse common log formats (Apache, Nginx, custom)
- Extract IP addresses, status codes, response times
- Generate statistics and reports
- Handle large files efficiently
Hints:
- Use regular expressions for parsing
- Implement streaming processing for large files
- Consider using channels for concurrent processing
Exercise 3: Data Migration Tool (15 minutes)
Objective: Convert data between different formats and schemas
Challenge: Create a tool that:
- Reads user data from multiple sources (CSV, JSON, XML)
- Validates and normalizes the data
- Converts between different schemas
- Handles data conflicts and errors gracefully
Extension Ideas:
- Add support for database import/export
- Implement data deduplication
- Add schema evolution support
7. Common Pitfalls and Troubleshooting (10 minutes)
Frequent Mistakes
- Problem: Ignoring unknown fields when unmarshaling JSON
Cause: Default behavior silently ignores unknown fields
Solution: Use
json.DecoderwithDisallowUnknownFields()
// Strict JSON unmarshaling
func strictUnmarshal(data []byte, v interface{}) error {
decoder := json.NewDecoder(bytes.NewReader(data))
decoder.DisallowUnknownFields()
return decoder.Decode(v)
}
- Problem: Memory issues when processing large files Cause: Loading entire file into memory at once Solution: Use streaming parsers and process data in chunks
// Stream processing for large JSON arrays
func processLargeJSONArray(filename string) error {
file, err := os.Open(filename)
if err != nil {
return err
}
defer file.Close()
decoder := json.NewDecoder(file)
// Read opening bracket
token, err := decoder.Token()
if err != nil {
return err
}
// Process each object in the array
for decoder.More() {
var item YourStruct
if err := decoder.Decode(&item); err != nil {
return err
}
// Process item...
}
return nil
}
Debugging Tips
- Use
json.RawMessagefor debugging unknown JSON structures - Validate data schemas before processing
- Implement comprehensive error reporting
- Use tools like
jqfor JSON debugging
8. Best Practices and Professional Tips (5 minutes)
Industry Standards
- Schema Validation: Always validate input data
- Error Handling: Provide detailed error messages with context
- Performance: Use streaming for large datasets
- Security: Sanitize and validate all input data
Performance Optimization
// Efficient JSON processing
type OptimizedProcessor struct {
decoder *json.Decoder
encoder *json.Encoder
buffer *bytes.Buffer
}
func (op *OptimizedProcessor) ProcessStream(input io.Reader, output io.Writer) error {
op.decoder = json.NewDecoder(input)
op.encoder = json.NewEncoder(output)
for op.decoder.More() {
var item interface{}
if err := op.decoder.Decode(&item); err != nil {
return err
}
// Process and transform item
processed := op.transform(item)
if err := op.encoder.Encode(processed); err != nil {
return err
}
}
return nil
}
Next Steps in Your Learning Journey
- Learn about Protocol Buffers and gRPC
- Study streaming data processing patterns
- Explore data validation libraries
- Practice with real-world APIs and datasets
9. Summary and Review (5 minutes)
Key Takeaways
- Go’s standard library provides excellent data processing capabilities
- Struct tags enable flexible serialization control
- Streaming processing is essential for large datasets
- Validation should be implemented at data boundaries
Quick Review Questions
- What’s the difference between
json.Marshalandjson.Encoder? - How do you handle unknown fields in JSON unmarshaling?
- When should you use streaming vs. batch processing?
For Instructors
Teaching Tips
- Use real API data for demonstrations
- Show both batch and streaming processing examples
- Emphasize the importance of data validation
- Demonstrate debugging techniques with malformed data
Assessment Ideas
- Build a complete ETL (Extract, Transform, Load) pipeline
- Create a data migration tool for different formats
- Implement a real-time data processing system
Adaptation Notes
- For beginners: Focus on basic JSON/XML operations
- For advanced students: Include performance optimization and streaming
- Online delivery: Provide sample datasets and API keys for practice