03. Operating System Concepts

At the heart of every computer system lies a collection of powerful architectural abstractions designed to turn raw hardware into a programmable, secure, and intuitive environment. this module examines the core conceptual pillars that define modern operating systems.

1. The Process Abstraction

At the foundation of operating system design is the process: an abstraction representing an active program in execution. While a program is a lifeless, passive collection of machine instructions and static data stored on non-volatile media (such as an ELF binary or Windows PE file on an NVMe SSD), a process is a dynamic, living computational entity with an active execution context managed by the kernel.

Process vs. Program: The Core Distinction

A single program can correspond to multiple distinct processes running concurrently. For example, if two users launch separate instances of the text editor nano, or if a user opens three independent terminal shells, each instance constitutes a separate process. Although both instances share the exact same underlying machine code in memory, each maintains completely isolated memory registers, dynamic heap buffers, execution state flags, and private call stacks.

Anatomy of a Process in Memory

When an operating system loads a binary executable into physical RAM to execute, it constructs a private virtual address space organized into standardized logical segments:

• Text Segment (Code): The read-only region containing raw machine instructions executed by the CPU. Marking this region read-only prevents self-modifying code vulnerabilities and allows multiple concurrent instances of the same binary to share physical code pages in memory.
• Data Segment (Initialized): Stores global and static variables explicitly initialized by the programmer before execution (e.g., int max_retries = 5;).
• BSS Segment (Uninitialized): Block Started by Symbol; holds uninitialized global and static variables. Rather than bloating binary disk images with zeroes, the kernel simply records the size of the BSS section and zeroes out physical pages on demand upon allocation.
• Heap: A dynamic memory pool managed at runtime by allocators (such as malloc() or new). The heap grows upward from lower virtual memory addresses toward higher addresses as dynamic allocations expand.
• Stack: A Last-In, First-Out (LIFO) scratchpad region storing active execution frames. Each stack frame houses function arguments, return addresses, saved CPU registers, and local automatic variables. In typical x86 and ARM architectures, the stack originates near the top of the virtual address space and grows downward toward the heap.

The Process Control Block (PCB): Cross-Platform Realities

Because CPU cores can execute only one sequence of instructions at any given instant, the operating system kernel maintains a centralized control structure—the Process Control Block (PCB)—to manage identity, ownership, security bounds, and hardware state for every active execution entity.

However, the internal implementation of a PCB varies substantially between operating system families:

Key Elements Universal to All PCBs

Despite naming differences across kernel architectures, any production operating system maintains the following core invariants inside or associated with its process control tracking:

• Architectural Root Anchor: A hardware-mandated pointer to the top-level page table (e.g., CR3 on x86-64, TTBR0 on ARM64) enabling the MMU to swap address spaces during context switches.
• Execution State Tracking: Flags denoting whether the entity is active, ready to run, sleeping on a synchronization primitive, or in a zombie/terminated state awaiting cleanup.
• Security Token & Credentials: User/group identifiers (POSIX UID/GID) or Windows Access Tokens specifying privileges, security identifiers (SIDs), and audit policies.
• Resource Descriptors & Limits: Quotas and references to system resources, including CPU affinity masks, memory working-set limits, and tables mapping private application references to underlying kernel objects.

Process State Transitions: The Three-State Model

Throughout its lifecycle, a process transitions between three core execution states:

1. Running: The process currently occupies a physical CPU core and its instructions are executing in hardware.
2. Ready: The process possesses all prerequisites to execute, but is temporarily stopped waiting for the CPU scheduler to allocate a core.
3. Blocked (Waiting): The process cannot proceed—even if a CPU core is idle—because it is waiting for an external event to finish, such as reading disk sectors, receiving network packets, or waiting on a timer.

Lifecycle Boundaries: The Classical Five-State Model

While the three-state model captures the dynamic scheduling loop on the CPU, production operating systems must also manage process inception, admission control, and resource reclamation upon exit. The Five-State Process Model extends the execution loop by adding formal entry and exit boundaries:

• New (Created): The process is being instantiated (e.g., via fork() in POSIX, CreateProcess() in Windows, or an OS batch loader). The kernel has allocated the Process Control Block (PCB) and assigned a unique PID, but memory translation structures and page mappings are not yet admitted to the active scheduling queue.
• Terminated (Exit / Zombie): The process has completed execution (via exit(), ExitProcess(), or a terminating signal like SIGKILL). The operating system reclaims its virtual address space, file descriptor tables, and allocated heap memory. However, the PCB remains in the system process table as a zombie until the parent process issues a synchronization call (such as wait() or waitpid()) to collect its exit status code.

Virtual Memory Realities: The Seven-State Process Model

While the 5-state model accounts for lifecycle boundaries, it implicitly assumes that all active processes reside permanently in physical RAM. In reality, modern systems frequently face memory overcommitment—where the aggregate virtual address spaces of all runnable and blocked processes exceed physical DRAM capacity.

To prevent the system from crashing under memory exhaustion, the operating system introduces a Medium-Term Scheduler (The Swapper). The swapper moves dormant or blocked processes from physical RAM to secondary backing stores (a swap partition, swapfile, or pagefile), introducing two distinct suspended states and giving rise to the classical Seven-State Model:

• Ready (In-Memory): The process resides in physical DRAM and is ready for the dispatcher to schedule onto an available CPU core.
• Blocked (In-Memory): The process resides in physical DRAM, but is waiting for an external event (I/O completion, timer, mutex).
• Blocked / Suspended (On Disk): The process was waiting for an event and, to relieve severe memory pressure, the OS swapped its address space out to disk. It cannot run even if a CPU is idle.
• Ready / Suspended (On Disk): The event the process was waiting for has completed, or a runnable process was swapped to disk. The process is ready to execute instructions immediately once the OS swaps its working set back into physical DRAM.

Key Swapping Dynamics in the Seven-State Model

• Autonomous Event Completion in Storage: Notice the horizontal transition from Blocked/Suspended to Ready/Suspended. When a hardware I/O request finishes for a swapped-out process, the interrupt service routine updates the PCB in kernel memory immediately—moving it to the Ready/Suspended queue without requiring an expensive page-in until memory pressure subsides.
• Swapping Policy Trade-offs: The OS prioritizes swapping out Blocked processes over Ready ones, as blocked jobs cannot utilize CPU cycles anyway. However, if severe memory starvation persists, even Ready processes are suspended to allow remaining tasks to finish without destructive page thrashing.

2. Address Spaces & Virtual Memory: Diverse Architectures

To prevent concurrent programs from corrupting one another's data and to decouple software from the limits of physical RAM, operating systems provide the address space abstraction. While contemporary systems predominantly utilize flat, paged virtual memory architectures, operating system design encompasses three distinct structural paradigms:

Memory Architectures Across Operating Systems

• Flat Paged Virtual Memory (Linux, Windows NT, macOS): Memory is partitioned into fixed-size chunks called pages (typically 4 KiB, with 2 MiB / 1 GiB huge pages). The hardware Memory Management Unit (MMU) consults multi-level page tables (e.g., x86-64 4-level or 5-level paging, ARM64 translation tables) to translate virtual addresses into physical frames dynamically. Paging enables fine-grained protection flags (read, write, execute), copy-on-write sharing, and demand paging to swap space or page files.
• Segmented Memory (x86 Protected Mode, Multics, OS/2): Instead of a uniform flat array, memory is divided into variable-length, semantically meaningful logical blocks called segments (e.g., Code Segment, Data Segment, Stack Segment). Each address consists of a segment selector and an offset. While pure segmentation fell out of favor on 64-bit consumer systems due to fragmentation, it provided hardware-enforced bounds checking and capabilities natively.
• Single-Level Store & Single Address Space (IBM OS/400 / IBM i): Rather than maintaining private, isolated address spaces per program, systems like IBM i treat all storage—volatile DRAM and non-volatile disk arrays—as one enormous, globally addressed 64-bit or 128-bit virtual address space. Programs, files, and database records reside permanently within this unified store, accessed via hardware-enforced 16-byte capability pointers rather than traditional file path opens.

3. Files & Hierarchical Directories: Diverse Paradigms

The file abstraction provides a persistent, named container for information that survives process termination and power outages. While contemporary developers frequently assume the Unix model—a linear, untyped byte stream anchored in a single unified directory tree—operating system history reflects fundamentally different approaches to structuring persistent storage.

File Structuring Models: Byte Streams vs. Record Datasets

Operating systems categorize the internal structure of files into three primary models:

• Unstructured Byte Sequences (POSIX / Windows): The OS treats the file as an arbitrary sequence of 8-bit bytes. The kernel imposes no internal boundaries, record structures, or record keys; interpreting the contents is left entirely to user applications.
• Multi-Stream & Forked Files (macOS HFS / Windows NTFS): Files can house multiple discrete data streams attached to a single directory entry. Apple's classic HFS utilized a Data Fork (raw content) and a Resource Fork (compiled icons, menus, and localization strings). Similarly, NTFS supports Alternate Data Streams (ADS), allowing metadata (such as web origin tags like Zone.Identifier) to adhere silently to a file.
• Record-Oriented Datasets (IBM z/OS / MVS / VMS): The operating system defines and enforces internal record structures. Files are sequences of fixed-length (Fixed Blocked) or variable-length records. In Indexed Sequential Access Method (ISAM) or VSAM datasets, the OS kernel itself manages indexed key searches to fetch specific customer or account records without requiring database middleware.

Namespace Topologies: Unified Trees vs. Per-Volume Roots

How an operating system organizes and accesses multiple physical disk volumes varies dramatically:

• Unified Virtual File System (POSIX / Unix): Every physical disk, network share, and pseudo-filesystem is spliced into a single root tree starting at / using the mount operation. Physical boundaries are completely transparent to userspace paths (e.g., /home/user/docs might reside across three distinct NVMe and NFS storage devices).
• Drive Letters & The NT Object Manager (Windows): Exposed to users as segmented per-drive root directories (C:\, D:\). Internally, the Windows NT Object Manager maintains a unified root namespace (\GLOBAL??\C: symlinked to \Device\HarddiskVolume1), with Win32 path prefixes providing compatibility.
• Flat Cataloged Namespaces (IBM Mainframes): Traditional z/OS does not use hierarchical directory trees for datasets. Instead, datasets use dotted qualifiers (e.g., USER1.COBOL.SOURCE(MAIN)) tracked via a centralized system catalog (Master and User Catalogs) mapping names to volume serial identifiers (VOLSERs).

Architectural Comparison of File System Implementations

4. Input/Output (I/O) Subsystems

The primary role of the I/O subsystem is to bridge the massive speed and interface discrepancies between the high-speed CPU memory bus and heterogeneous external peripherals. Operating systems differ substantially in how they model, schedule, and notify applications of completed I/O transactions:

I/O Notification & Completion Models

• Synchronous Blocking & Readiness Polling (Classic POSIX): The calling thread initiates an I/O call (e.g., read()) and transitions into a Blocked state until physical hardware completes the transfer. Multiplexing primitives like select(), poll(), and Linux epoll() or BSD kqueue signal when a file descriptor is ready to perform a non-blocking read or write.
• Asynchronous Completion Ports (Windows IOCP): Instead of waiting for readiness, Windows applications initiate genuine asynchronous operations (via ReadFileEx or overlapped I/O) and specify an I/O Completion Port (IOCP). The NT kernel queues completion packets onto the port as hardware finishes data transfers into application-owned buffers, allowing a small pool of worker threads to service thousands of concurrent sockets without polling.
• Shared Ring-Buffer Submission (Linux io_uring): Modern Linux bridges the gap with twin circular ring buffers (Submission Queue and Completion Queue) shared between user space and kernel space. Applications submit batches of I/O requests without issuing individual system calls, minimizing context switches for high-throughput NVMe and networking workloads.
• Mainframe Channel Architecture (IBM z/Architecture): Mainframes completely decouple I/O execution from the main CPUs using dedicated auxiliary coprocessors called Channel Subsystems. The CPU simply executes a START SUBCHANNEL instruction pointing to a sequence of Channel Command Words (CCWs) in memory; the channel hardware manages bus negotiation, device handshakes, and memory transfers autonomously before raising an I/O interrupt.

5. Protection & Security: Authorization Paradigms

Operating system protection mechanisms ensure that executing software accesses only authorized resources (memory segments, files, devices, network endpoints). Broadly, three distinct authorization and boundary paradigms are employed across systems:

Three Core Security Architectures

• Discretionary Access Control (DAC): The resource owner dictates access permissions. Traditional Unix uses numerical User IDs (UID), Group IDs (GID), and permission bits (rwxrwxrwx). Windows NT expands this via Discretionary Access Control Lists (DACLs) composed of individual Access Control Entries (ACEs) granting or denying specific Security Identifiers (SIDs) fine-grained rights like FILE_APPEND_DATA or WRITE_OWNER.
• Mandatory Access Control (MAC): Centralized security policy enforced by the kernel regardless of user preferences. Frameworks like SELinux, AppArmor, and Windows Mandatory Integrity Control (MIC) assign security labels to subjects and objects (e.g., Low, Medium, High, System integrity levels in Windows, or Type Enforcement domains in SELinux), isolating processes even if compromised by zero-day exploits.
• Capability-Based Security (seL4, Fuchsia, KeyKOS): In pure capability systems, access is granted not by matching identity strings against access lists, but through unforgeable cryptographic or kernel-held tokens called capabilities. Possessing a valid capability grants the holder the authority to perform operations on an object directly, eliminating confused deputy vulnerabilities.

6. The Command Interpreter: Text, Objects, and Batch Job Control

The command interpreter provides the direct interface through which users and automation scripts interact with operating system facilities. Far from being a uniform shell environment, modern and historical command systems reflect contrasting philosophies of data exchange and execution control:

Command Line Paradigms Compared

• Text Stream & Pipeline Shells (sh, bash, zsh): Built around the Unix philosophy of small tools connected via linear ASCII/UTF-8 byte streams. Commands read from standard input and emit unstructured text to standard output, requiring external utilities (such as grep, awk, sed) to parse textual fields. Process creation relies on the classical fork() and exec() primitives.
• Object-Oriented Shells (PowerShell): Instead of untyped byte streams, PowerShell pipelines transmit strongly-typed .NET CLI objects. Commands (called cmdlets) receive and emit structured objects possessing properties and methods, eliminating brittle string scraping and regex parsing. On Windows, execution launches through the Win32 CreateProcess() system call, which creates the address space, initial thread, and handles in a single atomic invocation rather than a split fork/exec.
• Job Control Language (IBM JCL): In mainframe batch computing, user interaction occurs through structured declarative control cards specifying computational jobs, required program binaries, datasets, and volume dependencies (e.g., //JOB, //EXEC, //DD cards). The operating system's Job Entry Subsystem (JES2/JES3) evaluates resource allocations, queues the job, schedules CPU allocation, and handles spooling without requiring interactive shell sessions.