Hardware, Coherence & OS Models

8.1.1 Multiprocessor Hardware & Cache Coherence

Modern multiprocessing systems organize multiple execution cores around shared physical memory. Depending on how memory is interconnected and how bus bandwidth scales, architectures fall into two primary hardware paradigms:

1. Uniform Memory Access (UMA)

In a Uniform Memory Access (UMA) architecture, all processors share a common system bus or crossbar switch connected to a centralized memory bank. Every CPU core experiences identical memory latency regardless of which core initiates the read or write request. While simple to program, UMA suffers from severe scalability limits: as core counts scale beyond 8–16 processors, the shared interconnect bus saturates entirely under memory bandwidth contention.

2. Non-Uniform Memory Access (NUMA)

To bypass UMA bus saturation, modern multi-socket servers and high-performance clusters utilize Non-Uniform Memory Access (NUMA). Physical memory is partitioned and distributed locally across processor nodes connected via a high-speed interconnect fabric (such as AMD Infinity Fabric or Intel Ultra Path Interconnect - UPI).

• Local Memory Access: Accessing memory physically attached to the local socket node is fast (~20 ns).
• Remote Memory Access: Accessing memory residing on a remote socket node requires traversing the interconnect fabric, incurring significantly higher latency (~80 ns).

3. Cache-Coherent NUMA (ccNUMA) & Directory-Based Protocols

While snooping protocols (like MESI) broadcast every cache event across a shared bus, snooping does not scale beyond a small number of cores because the broadcast bus itself becomes a bottleneck. To solve this for large-scale NUMA systems, computer architects developed directory-based cache coherence (ccNUMA).

Pioneered academically by Anant Agarwal and his research team with the Stanford DASH (Directory Architecture for Shared Memory) project in the early 1990s, directory-based protocols eliminate broadcast snooping. Instead, a centralized or distributed directory maintains the sharing status of every memory block. When a core requests a cache line, it queries the directory managing that specific physical address block, which then directs point-to-point invalidation messages solely to the nodes actively caching that line.

The Cache Coherence Problem: Because CPU execution cores operate orders of magnitude faster than main memory, each core relies on high-speed private caches (L1 and L2). If CPU 0 modifies a variable in its private cache, CPU 1's private cache immediately becomes stale. To preserve memory consistency without routing every access back to slow main RAM, hardware controllers monitor memory lines using coherence protocols.

8.1.2 Multiprocessor Operating System Models

Structuring an operating system to manage multiple concurrent execution cores requires fundamental architectural decisions regarding kernel data structure placement, reentrancy, and synchronization boundaries. Computer architects have historically organized multiprocessor operating systems into three distinct models:

1. Each CPU Has Its Own Operating System (Static Partitioning)

Partitions physical memory into N disjoint regions for N processors. Each core executes an independent, private copy of the operating system kernel with its own run queue, process table, and buffer cache, eliminating kernel locking contention but preventing load balancing across cores.

2. Master-Slave Multiprocessors (Asymmetric Multiprocessing - ASMP)

Dedicates a single processor (the master, CPU 0) to executing the kernel, device drivers, and I/O interrupts. All other processors (slaves) execute user-space applications exclusively, creating a severe bottleneck when slave cores scale and flood the master with system call traps.

3. Symmetric Multiprocessing (SMP)

A single unified operating system image resides in shared physical memory. Every core can execute kernel code, service hardware interrupts, and schedule threads concurrently. Modern SMP kernels use fine-grained subsystem locking (per-core run queues, lockless RCU lists) to achieve true parallel performance.

8.1.3 Multiprocessor Synchronization & Cache Contention

On a uniprocessor system, mutual exclusion can be achieved simply by disabling hardware interrupts to prevent timer preemption. On a multiprocessor, disabling interrupts on Core 0 has zero effect on Core 1, which continues executing instructions concurrently.

1. Hardware Atomic Primitives

If an operating system attempts to implement a mutual exclusion lock using standard, non-atomic software read and write instructions, a race condition occurs whenever two cores attempt to claim the lock simultaneously:

To eliminate this vulnerability, processor architectures provide Atomic Read-Modify-Write (RMW) Instructions. These instructions guarantee that reading the old value and writing the new value occur as an indivisible, single-cycle operation across the entire memory hierarchy:

2. The Cache-Line Bouncing Hazard

While spinlocks avoid the expensive latency of a context switch for short critical sections, naive spinlock implementations cause severe hardware degradation through cache-line bouncing.

3. Optimization: Test-and-Test-and-Set (TTAS)

To prevent memory bus saturation, lock designers utilize Test-and-Test-and-Set (TTAS). Instead of blindly issuing atomic write instructions on every iteration, a spinning CPU first inspects the lock using ordinary, non-atomic memory read instructions: