Go File I/O and System Programming
Learning Objectives: By the end of this lesson, you will be able to:
- Perform comprehensive file operations in Go
- Execute system calls and interact with the operating system
- Build command-line tools with proper argument handling
- Create file utilities and system monitoring applications
- Handle file permissions, paths, and cross-platform considerations
1. Welcome and Introduction (5 minutes)
Icebreaker Activity
Have you ever wondered how tools like ls, grep, or top work under the hood? Or how your favorite code editor can watch for file changes and automatically reload? Today we’ll learn how to build these kinds of system-level tools using Go’s powerful file I/O and system programming capabilities.
Lesson Overview
- Duration: 2-3 days (6-8 hours)
- Prerequisites: Basic Go syntax, error handling, packages
- Materials Needed: Go development environment, terminal access
2. Learning Objectives and Session Outline (5 minutes)
What We’ll Cover Today:
- File Operations - Reading, writing, and manipulating files
- Directory Operations - Working with folders and file trees
- Path Manipulation - Cross-platform path handling
- System Calls - Interacting with the operating system
- Command-Line Tools - Building robust CLI applications
- File Monitoring - Watching for file system changes
Success Criteria:
By the end of this session, you should be able to build complete file utilities and system monitoring tools.
3. Foundation Knowledge (15-20 minutes)
Core Concepts
File I/O in Go
Go provides several packages for file operations:
os- Operating system interfaceio- Basic I/O primitivesio/ioutil- I/O utilities (deprecated in Go 1.16+)bufio- Buffered I/Opath/filepath- File path manipulation
Real-World Analogy: Think of file I/O like working with physical filing cabinets. You need to open drawers (files), read documents, write new ones, and organize them properly.
Why This Matters: Most real-world applications need to persist data, read configuration files, or interact with the file system. Understanding these operations is crucial for building practical software.
System Programming Concepts
System programming involves interacting directly with the operating system to access resources like files, processes, and network interfaces.
Key Terminology
- File Descriptor: A handle to an open file
- Buffer: Temporary storage for data during I/O operations
- Path: Location of a file or directory in the file system
- Permissions: Access rights for files and directories
- System Call: A request to the operating system kernel
4. Detailed Explanation and Examples (25-30 minutes)
Understanding File Operations
File operations in Go follow a pattern of opening resources, using them, and properly closing them. Go’s defer statement makes resource cleanup elegant and reliable.
Basic File Reading and Writing
package main
import (
"fmt"
"io"
"os"
)
// Reading entire file into memory
func readEntireFile(filename string) (string, error) {
data, err := os.ReadFile(filename)
if err != nil {
return "", fmt.Errorf("failed to read file %s: %w", filename, err)
}
return string(data), nil
}
// Writing data to file
func writeToFile(filename, content string) error {
err := os.WriteFile(filename, []byte(content), 0644)
if err != nil {
return fmt.Errorf("failed to write to file %s: %w", filename, err)
}
return nil
}
// Reading file with proper resource management
func readFileStream(filename string) error {
file, err := os.Open(filename)
if err != nil {
return fmt.Errorf("failed to open file: %w", err)
}
defer file.Close() // Ensure file is closed
// Read in chunks
buffer := make([]byte, 1024)
for {
n, err := file.Read(buffer)
if err != nil {
if err == io.EOF {
break // End of file reached
}
return fmt.Errorf("read error: %w", err)
}
// Process the data
fmt.Printf("Read %d bytes: %s\n", n, string(buffer[:n]))
}
return nil
}
Buffered I/O for Performance
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// Efficient line-by-line reading
func processFileLines(filename string) error {
file, err := os.Open(filename)
if err != nil {
return err
}
defer file.Close()
scanner := bufio.NewScanner(file)
lineNumber := 1
for scanner.Scan() {
line := scanner.Text()
// Process each line
if strings.Contains(line, "ERROR") {
fmt.Printf("Line %d: %s\n", lineNumber, line)
}
lineNumber++
}
return scanner.Err()
}
// Buffered writing for performance
func writeDataEfficiently(filename string, data []string) error {
file, err := os.Create(filename)
if err != nil {
return err
}
defer file.Close()
writer := bufio.NewWriter(file)
defer writer.Flush() // Ensure all buffered data is written
for _, line := range data {
_, err := writer.WriteString(line + "\n")
if err != nil {
return err
}
}
return nil
}
Directory Operations and File Tree Walking
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// Create directory structure
func createDirectoryStructure(basePath string) error {
dirs := []string{
"project/src",
"project/docs",
"project/tests",
"project/config",
}
for _, dir := range dirs {
fullPath := filepath.Join(basePath, dir)
err := os.MkdirAll(fullPath, 0755)
if err != nil {
return fmt.Errorf("failed to create directory %s: %w", fullPath, err)
}
fmt.Printf("Created directory: %s\n", fullPath)
}
return nil
}
// Walk directory tree and collect file information
func analyzeDirectory(rootPath string) error {
var totalSize int64
fileCount := 0
dirCount := 0
err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
fmt.Printf("Error accessing %s: %v\n", path, err)
return nil // Continue walking
}
if info.IsDir() {
dirCount++
fmt.Printf("DIR: %s\n", path)
} else {
fileCount++
totalSize += info.Size()
fmt.Printf("FILE: %s (%d bytes)\n", path, info.Size())
}
return nil
})
if err != nil {
return err
}
fmt.Printf("\nSummary:\n")
fmt.Printf("Directories: %d\n", dirCount)
fmt.Printf("Files: %d\n", fileCount)
fmt.Printf("Total size: %d bytes\n", totalSize)
return nil
}
// Find files matching a pattern
func findFiles(rootPath, pattern string) ([]string, error) {
var matches []string
err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil
}
if !info.IsDir() {
matched, err := filepath.Match(pattern, info.Name())
if err != nil {
return err
}
if matched {
matches = append(matches, path)
}
}
return nil
})
return matches, err
}
5. Live Coding Demonstration (20-25 minutes)
Example 1: Log File Analyzer
package main
import (
"bufio"
"fmt"
"os"
"regexp"
"sort"
"strings"
"time"
)
type LogEntry struct {
Timestamp time.Time
Level string
Message string
Line string
}
type LogAnalyzer struct {
errorCount int
warningCount int
infoCount int
entries []LogEntry
}
func NewLogAnalyzer() *LogAnalyzer {
return &LogAnalyzer{
entries: make([]LogEntry, 0),
}
}
func (la *LogAnalyzer) ParseLogFile(filename string) error {
file, err := os.Open(filename)
if err != nil {
return fmt.Errorf("failed to open log file: %w", err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
lineNumber := 0
// Regex pattern for common log format: timestamp [LEVEL] message
logPattern := regexp.MustCompile(`^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(\w+)\] (.+)$`)
for scanner.Scan() {
lineNumber++
line := scanner.Text()
matches := logPattern.FindStringSubmatch(line)
if len(matches) == 4 {
timestamp, err := time.Parse("2006-01-02 15:04:05", matches[1])
if err != nil {
fmt.Printf("Warning: Invalid timestamp on line %d\n", lineNumber)
continue
}
entry := LogEntry{
Timestamp: timestamp,
Level: matches[2],
Message: matches[3],
Line: line,
}
la.entries = append(la.entries, entry)
// Count by level
switch strings.ToUpper(entry.Level) {
case "ERROR":
la.errorCount++
case "WARN", "WARNING":
la.warningCount++
case "INFO":
la.infoCount++
}
}
}
return scanner.Err()
}
func (la *LogAnalyzer) GenerateReport(outputFile string) error {
file, err := os.Create(outputFile)
if err != nil {
return err
}
defer file.Close()
writer := bufio.NewWriter(file)
defer writer.Flush()
// Write summary
fmt.Fprintf(writer, "Log Analysis Report\n")
fmt.Fprintf(writer, "==================\n\n")
fmt.Fprintf(writer, "Total entries: %d\n", len(la.entries))
fmt.Fprintf(writer, "Errors: %d\n", la.errorCount)
fmt.Fprintf(writer, "Warnings: %d\n", la.warningCount)
fmt.Fprintf(writer, "Info: %d\n", la.infoCount)
fmt.Fprintf(writer, "\nError Details:\n")
fmt.Fprintf(writer, "--------------\n")
// Sort entries by timestamp
sort.Slice(la.entries, func(i, j int) bool {
return la.entries[i].Timestamp.Before(la.entries[j].Timestamp)
})
// Write error entries
for _, entry := range la.entries {
if strings.ToUpper(entry.Level) == "ERROR" {
fmt.Fprintf(writer, "%s: %s\n",
entry.Timestamp.Format("2006-01-02 15:04:05"),
entry.Message)
}
}
return nil
}
func logAnalyzerExample() {
// Create sample log file
sampleLog := `2024-01-01 10:00:00 [INFO] Application started
2024-01-01 10:00:01 [INFO] Loading configuration
2024-01-01 10:00:02 [ERROR] Failed to connect to database
2024-01-01 10:00:03 [WARN] Retrying database connection
2024-01-01 10:00:04 [INFO] Database connection established
2024-01-01 10:00:05 [ERROR] Invalid user credentials
2024-01-01 10:00:06 [INFO] User logged in successfully`
// Write sample log
err := os.WriteFile("sample.log", []byte(sampleLog), 0644)
if err != nil {
fmt.Printf("Error creating sample log: %v\n", err)
return
}
// Analyze log
analyzer := NewLogAnalyzer()
err = analyzer.ParseLogFile("sample.log")
if err != nil {
fmt.Printf("Error parsing log: %v\n", err)
return
}
// Generate report
err = analyzer.GenerateReport("log_report.txt")
if err != nil {
fmt.Printf("Error generating report: %v\n", err)
return
}
fmt.Println("Log analysis complete. Check log_report.txt for results.")
}
Example 2: File Backup Utility
package main
import (
"crypto/md5"
"fmt"
"io"
"os"
"path/filepath"
"time"
)
type BackupManager struct {
sourceDir string
backupDir string
}
func NewBackupManager(source, backup string) *BackupManager {
return &BackupManager{
sourceDir: source,
backupDir: backup,
}
}
func (bm *BackupManager) CreateBackup() error {
timestamp := time.Now().Format("2006-01-02_15-04-05")
backupPath := filepath.Join(bm.backupDir, fmt.Sprintf("backup_%s", timestamp))
fmt.Printf("Creating backup: %s -> %s\n", bm.sourceDir, backupPath)
err := os.MkdirAll(backupPath, 0755)
if err != nil {
return fmt.Errorf("failed to create backup directory: %w", err)
}
return filepath.Walk(bm.sourceDir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// Calculate relative path
relPath, err := filepath.Rel(bm.sourceDir, path)
if err != nil {
return err
}
destPath := filepath.Join(backupPath, relPath)
if info.IsDir() {
return os.MkdirAll(destPath, info.Mode())
}
return bm.copyFile(path, destPath)
})
}
func (bm *BackupManager) copyFile(src, dst string) error {
sourceFile, err := os.Open(src)
if err != nil {
return err
}
defer sourceFile.Close()
destFile, err := os.Create(dst)
if err != nil {
return err
}
defer destFile.Close()
_, err = io.Copy(destFile, sourceFile)
if err != nil {
return err
}
// Copy file permissions
sourceInfo, err := sourceFile.Stat()
if err != nil {
return err
}
return destFile.Chmod(sourceInfo.Mode())
}
func (bm *BackupManager) VerifyBackup(backupPath string) error {
fmt.Println("Verifying backup integrity...")
return filepath.Walk(bm.sourceDir, func(path string, info os.FileInfo, err error) error {
if err != nil || info.IsDir() {
return err
}
relPath, err := filepath.Rel(bm.sourceDir, path)
if err != nil {
return err
}
backupFilePath := filepath.Join(backupPath, relPath)
// Compare file checksums
sourceHash, err := bm.calculateMD5(path)
if err != nil {
return err
}
backupHash, err := bm.calculateMD5(backupFilePath)
if err != nil {
return err
}
if sourceHash != backupHash {
return fmt.Errorf("checksum mismatch for file %s", relPath)
}
fmt.Printf("✓ %s\n", relPath)
return nil
})
}
func (bm *BackupManager) calculateMD5(filename string) (string, error) {
file, err := os.Open(filename)
if err != nil {
return "", err
}
defer file.Close()
hash := md5.New()
_, err = io.Copy(hash, file)
if err != nil {
return "", err
}
return fmt.Sprintf("%x", hash.Sum(nil)), nil
}
Example 3: System Information Tool
package main
import (
"fmt"
"os"
"os/user"
"runtime"
"time"
)
type SystemInfo struct {
OS string
Architecture string
CPUs int
GoVersion string
User string
Hostname string
WorkingDir string
Timestamp time.Time
}
func GatherSystemInfo() (*SystemInfo, error) {
info := &SystemInfo{
OS: runtime.GOOS,
Architecture: runtime.GOARCH,
CPUs: runtime.NumCPU(),
GoVersion: runtime.Version(),
Timestamp: time.Now(),
}
// Get current user
currentUser, err := user.Current()
if err != nil {
return nil, fmt.Errorf("failed to get current user: %w", err)
}
info.User = currentUser.Username
// Get hostname
hostname, err := os.Hostname()
if err != nil {
return nil, fmt.Errorf("failed to get hostname: %w", err)
}
info.Hostname = hostname
// Get working directory
workingDir, err := os.Getwd()
if err != nil {
return nil, fmt.Errorf("failed to get working directory: %w", err)
}
info.WorkingDir = workingDir
return info, nil
}
func (si *SystemInfo) SaveToFile(filename string) error {
file, err := os.Create(filename)
if err != nil {
return err
}
defer file.Close()
fmt.Fprintf(file, "System Information Report\n")
fmt.Fprintf(file, "========================\n\n")
fmt.Fprintf(file, "Timestamp: %s\n", si.Timestamp.Format("2006-01-02 15:04:05"))
fmt.Fprintf(file, "Operating System: %s\n", si.OS)
fmt.Fprintf(file, "Architecture: %s\n", si.Architecture)
fmt.Fprintf(file, "CPU Cores: %d\n", si.CPUs)
fmt.Fprintf(file, "Go Version: %s\n", si.GoVersion)
fmt.Fprintf(file, "User: %s\n", si.User)
fmt.Fprintf(file, "Hostname: %s\n", si.Hostname)
fmt.Fprintf(file, "Working Directory: %s\n", si.WorkingDir)
return nil
}
func (si *SystemInfo) Display() {
fmt.Println("System Information:")
fmt.Println("==================")
fmt.Printf("OS: %s %s\n", si.OS, si.Architecture)
fmt.Printf("CPUs: %d\n", si.CPUs)
fmt.Printf("Go: %s\n", si.GoVersion)
fmt.Printf("User: %s@%s\n", si.User, si.Hostname)
fmt.Printf("Directory: %s\n", si.WorkingDir)
}
6. Hands-On Practice (30-40 minutes)
Exercise 1: Word Count Utility (10 minutes)
Objective: Build a command-line tool that counts lines, words, and characters in files
Instructions:
- Accept multiple file paths as command-line arguments
- For each file, count lines, words, and characters
- Display results in a formatted table
- Handle errors gracefully
Starter Code:
package main
import (
"fmt"
"os"
)
type FileStats struct {
Name string
Lines int
Words int
Characters int
Error error
}
func analyzeFile(filename string) FileStats {
// TODO: Implement file analysis
return FileStats{Name: filename}
}
func main() {
if len(os.Args) < 2 {
fmt.Println("Usage: wordcount <file1> [file2] ...")
os.Exit(1)
}
// TODO: Process each file and display results
}
Exercise 2: Directory Synchronizer (15 minutes)
Objective: Create a tool that synchronizes two directories
Requirements:
- Compare two directory structures
- Identify files that exist in source but not in destination
- Copy missing files to destination
- Report what was synchronized
Hints:
- Use filepath.Walk to traverse directories
- Compare file modification times
- Handle subdirectories recursively
Exercise 3: Log Rotation Tool (15 minutes)
Objective: Implement a log rotation mechanism
Challenge: Create a system that:
- Monitors log file size
- Rotates logs when they exceed a size limit
- Keeps a specified number of old log files
- Compresses old logs (optional)
Extension Ideas:
- Add timestamp-based rotation
- Implement different rotation strategies
- Add configuration file support
7. Common Pitfalls and Troubleshooting (10 minutes)
Frequent Mistakes
- Problem: Not handling file permissions properly Cause: Insufficient permissions or incorrect permission bits Solution: Check permissions and use appropriate modes
// Check if file is readable
func isReadable(filename string) bool {
file, err := os.Open(filename)
if err != nil {
return false
}
file.Close()
return true
}
// Create file with specific permissions
func createWithPermissions(filename string, perm os.FileMode) error {
file, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, perm)
if err != nil {
return err
}
return file.Close()
}
- Problem: Path separator issues across platforms Cause: Using hardcoded path separators Solution: Use filepath package for cross-platform compatibility
// Wrong - hardcoded separators
badPath := "config/app.conf"
// Right - use filepath.Join
goodPath := filepath.Join("config", "app.conf")
Debugging Tips
- Use
filepath.Abs()to get absolute paths for debugging - Check file permissions with
os.Stat() - Use
os.IsNotExist()to check for file existence errors - Test with different file sizes and types
8. Best Practices and Professional Tips (5 minutes)
Industry Standards
- Always close resources: Use defer to ensure files are closed
- Handle errors appropriately: Don’t ignore I/O errors
- Use buffered I/O: For better performance with large files
- Validate paths: Sanitize user input for file paths
Security Considerations
// Validate file paths to prevent directory traversal
func isValidPath(path string) bool {
cleanPath := filepath.Clean(path)
return !strings.Contains(cleanPath, "..")
}
// Safe file creation
func createFileSecurely(basePath, filename string) (*os.File, error) {
if !isValidPath(filename) {
return nil, fmt.Errorf("invalid filename: %s", filename)
}
fullPath := filepath.Join(basePath, filename)
return os.Create(fullPath)
}
Next Steps in Your Learning Journey
- Learn about file system monitoring with fsnotify
- Study advanced file operations like memory mapping
- Explore cross-platform system programming
- Practice with larger file processing applications
9. Summary and Review (5 minutes)
Key Takeaways
- File operations in Go are straightforward but require proper resource management
- Buffered I/O significantly improves performance for large files
- Cross-platform compatibility requires using the filepath package
- Error handling is crucial for robust file operations
Quick Review Questions
- What’s the difference between os.ReadFile() and reading with os.Open()?
- When should you use buffered I/O?
- How do you safely handle file paths across different operating systems?
For Instructors
Teaching Tips
- Demonstrate file operations with real files students can examine
- Show both small file and large file handling techniques
- Emphasize the importance of resource cleanup
- Use practical examples that students might encounter
Assessment Ideas
- Build a complete file management utility
- Create a system monitoring tool
- Implement a custom backup solution
Adaptation Notes
- For beginners: Focus on basic file operations first
- For advanced students: Include performance optimization and security considerations
- Online delivery: Provide sample files for exercises