What is an Operating System?

An operating system is a layer of systems software between applications and hardware that:

  • Abstracts hardware — simplifies what the hardware looks like (e.g., files instead of disk blocks, sockets instead of raw packets)

  • Arbitrates hardware — manages, controls, and allocates hardware resources across applications

The OS has privileged access to hardware; applications do not.

Key responsibilities:

  • Hide hardware complexity — expose higher-level abstractions (files, sockets, memory pages)

  • Manage resources — allocate CPU, memory, and devices to applications; schedule execution

  • Provide isolation and protection — prevent processes from accessing each other’s memory unless explicitly intended

OS Elements

Operating systems are built from three types of elements:

  • Abstractions — process, thread, file, socket, memory page

  • Mechanisms — create, schedule, open, write, allocate (operations on abstractions)

  • Policies — determine how mechanisms are applied (e.g., LRU page replacement, max open files per process)

Design principle: separate mechanism from policy — build flexible mechanisms that support a range of policies, then optimize for the common case (understand the workload, platform, and usage patterns to pick the right policy).

Memory Management Example

  • Abstraction: memory page (fixed-size addressable region, e.g., 4 KB)

  • Mechanisms: allocate page in DRAM, map page into process address space

  • Policy: Least Recently Used (LRU) — pages not accessed recently are swapped to disk; frequently accessed pages stay in physical memory

OS Protection Boundary

Hardware supports at least two execution modes:

  • Kernel mode (privileged) — a CPU bit is set; instructions that directly manipulate hardware are permitted

  • User mode (unprivileged) — privileged instructions are forbidden; attempting them causes a trap

On a trap, hardware transfers control to the OS at a predefined location. The OS inspects the cause, then grants or denies the request (or terminates the offending process).

System Calls

Applications request OS services through the system call interface: open, send, malloc, read, write, etc.

System call flow:

  1. Application writes arguments to a well-defined location and invokes a system call number

  2. Execution context switches from user mode to kernel mode

  3. OS performs the privileged operation

  4. OS returns results and switches back to user mode, resuming the process at the instruction after the call

Arguments can be passed directly or by reference (address). In synchronous mode, the process blocks until the call completes.

Cost of User/Kernel Transitions

  • Direct cost: ~50–100 ns on a 2 GHz Linux machine (saving/restoring registers, mode switch)

  • Indirect cost: kernel execution pollutes the hardware cache — application data evicted from cache leads to cache misses (cold cache) when the process resumes

OS Services

Core services map to hardware components and higher-level abstractions:

  • Scheduler — controls CPU access

  • Memory manager — allocates physical memory, enforces isolation

  • Block device driver — manages disk access

  • File system — provides the file abstraction on top of block storage

Services are exposed via system calls. Windows and UNIX provide similar categories: process control (create, exit, wait), file management (create, open, read, write), device management, etc.

Key Linux system calls:

  • kill — send a signal to a process

  • setgid — set group identity of a process

  • mount — mount a file system

  • sysctl — read/write system parameters

OS Components

Likely OS components: file system, device driver, scheduler.

Not OS components: file editor (user app), web browser (user app), cache memory (managed by hardware, not OS).

Abstraction vs. arbitration:

  • Arbitration: distributing memory between processes

  • Abstraction: supporting interchangeable speakers or storage devices (hard disk vs. SSD) via a uniform interface (+ device drivers)

OS Architectures

Monolithic OS

Every possible service is compiled into a single OS binary (e.g., multiple file systems for sequential vs. random I/O).

  • Pro: compile-time optimizations; everything packaged together

  • Con: large codebase — hard to maintain, debug, upgrade; high memory footprint

Modular OS

Basic services + APIs in the core; additional functionality loaded as modules (e.g., Linux).

  • Modules implement well-defined interfaces and can be installed/replaced dynamically

  • Pro: easier to maintain/upgrade, smaller footprint, better resource efficiency

  • Con: indirection through interfaces may reduce some optimization opportunities; modules from disparate codebases can introduce bugs

Microkernel

Only the most basic primitives live in the kernel: address spaces, threads, IPC.

Everything else (file systems, device drivers, databases) runs at user level.

  • Pro: very small — easier to verify/test; valuable for safety-critical embedded/control systems

  • Con: limited portability (often specialized per platform); frequent user/kernel crossings for IPC add overhead

Linux and Mac OS X

Linux: modular architecture. Hardware → kernel (I/O, memory, process management as replaceable modules) → standard libraries (libc, system call wrappers) → utilities → user applications.

Mac OS X: hybrid. A Mach microkernel at the core (memory management, thread scheduling, IPC/RPC) with a BSD layer providing UNIX interoperability (POSIX APIs, network I/O). Supports dynamically loadable kernel modules and device driver development (I/O Kit).