02. Interrupts & Direct Memory Access (DMA)

In Module 01, we established the fundamental communication modes between the CPU and hardware controllers. However, high-throughput operating systems cannot rely on naive polling without stalling computation, nor can they handle gigabit streams by processing complex network stacks inside hardware interrupt routines.

This module explores the silicon-level mechanics of Hardware Interrupts, the dual-phase deferral models used by modern kernels to preserve system responsiveness, and the microarchitectural intricacies of Direct Memory Access (DMA) across caching and virtual memory boundaries.

1. Hardware Interrupt Mechanics & The APIC

A hardware interrupt is an asynchronous electrical event initiated by a peripheral device controller that forces the microprocessor to temporarily suspend its current instruction pipeline, execute an involuntary privilege boundary transition, and jump to a pre-registered kernel routine.

Interrupts form the bedrock of interactive and real-time operating systems. Without hardware interrupts, a central processor would have no mechanism to detect external physical events—such as an inbound network packet, a keystroke, a disk block DMA completion, or a timer tick—without dedicating 100% of its computational cycles to wasteful polling loops.

1. Physical Electrical Signaling: Edge vs. Level Triggering

At the hardware layer, an interrupt begins as a change in electrical voltage across a physical trace connecting the peripheral controller to the interrupt controller. Two distinct electrical conventions govern this signaling:

2. Evolution of the Interrupt Controller: 8259A PIC to Modern APIC

To understand why modern operating systems demand specialized hardware routing, we must contrast the historical PC interrupt controller with the modern symmetric multiprocessing (SMP) APIC architecture:

The Legacy 8259A PIC (Programmable Interrupt Controller)

The original 1981 IBM PC incorporated an Intel 8259A PIC supporting 8 hardware interrupt lines (IRQ0–IRQ7). The IBM PC/AT cascaded a second slave 8259A controller through Master IRQ2, yielding 15 usable lines:

• Master PIC: IRQ0 (Timer), IRQ1 (Keyboard), IRQ2 (Cascade to Slave), IRQ3 (COM2), IRQ4 (COM1), IRQ5 (LPT2), IRQ6 (Floppy Disk), IRQ7 (Spurious / LPT1).
• Slave PIC: IRQ8 (Real-Time Clock), IRQ9 (Redirected IRQ2), IRQ10–IRQ11 (Unassigned / PCI), IRQ12 (PS/2 Mouse), IRQ13 (FPU Coprocessor), IRQ14 (Primary IDE Hard Disk), IRQ15 (Secondary IDE).

Why the 8259A Failed Modern Computing: The 8259A is electrically unscalable. It is hardwired to uniprocessor execution—it can route interrupts to only a single CPU core. Furthermore, 15 interrupt lines proved completely inadequate as modern PCs added multiple sound cards, SCSI host adapters, USB controllers, and network interfaces, leading to severe resource conflicts and shared-line interrupt latency.

The Advanced Programmable Interrupt Controller (APIC Architecture)

To support multi-core symmetric multiprocessing (SMP), Intel and industry partners introduced the APIC Architecture, partitioning interrupt management into two discrete silicon entities:

1. The I/O APIC (Chipset / Platform Controller Hub): Located on the motherboard chipset or root complex. The standard I/O APIC provides 24 (or more) programmable Redirection Table Entries (RTEs).
   Each 64-bit RTE allows the operating system kernel to configure:
   • Interrupt Vector Number: The exact byte index (from 0x20 to 0xFE) to trigger in the processor's IDT.
   • Delivery Mode: Fixed, Lowest Priority (directing the interrupt to the least busy core), System Management Interrupt (SMI), or Non-Maskable Interrupt (NMI).
   • Destination Mode & Target Core: Specifies physical or logical APIC IDs, enabling the OS kernel to bind specific device interrupts to dedicated CPU cores (IRQ Affinity).
2. The Local APIC (LAPIC — On-Die Per Core): Every physical CPU core integrates its own dedicated Local APIC. The Local APIC contains specialized hardware registers:
   • Task Priority Register (TPR): Allows the operating system kernel to dynamically set a priority threshold. The LAPIC will automatically suppress all incoming device interrupts whose priority is lower than or equal to the value in the TPR, without requiring software masking of individual devices.
   • In-Service Register (ISR): A 256-bit register tracking which interrupt vectors are currently being serviced by the CPU core.
   • End of Interrupt Register (EOI): When an Interrupt Service Routine completes, the kernel must write a value of 0 to the Local APIC EOI register (located at physical MMIO address 0xFEE000B0 or via MSR 0x80B in x2APIC mode). Writing to the EOI register clears the highest-priority bit in the In-Service Register, notifying the APIC that the core is ready to accept subsequent interrupts of equal or lower priority.
   • Interrupt Command Register (ICR): Enables the CPU core to issue Inter-Processor Interrupts (IPIs) across the system bus to other cores, driving cross-core scheduling wakeups, thread preemption, and TLB invalidation shootdowns.

3. Message Signaled Interrupts (MSI & MSI-X)

In high-throughput PCIe systems, physical interrupt pins are obsolete. Routing physical copper traces across motherboards introduces signal integrity degradation, latency, and pin-count bottlenecks.

To solve this, PCI 2.2 introduced Message Signaled Interrupts (MSI), and PCI Express expanded it into MSI-X:

The Power of MSI-X Multi-Queue Affinity

MSI-X enables true linear scaling on multi-core servers:

• An enterprise 100 GbE NIC allocates 64 separate hardware receive (RX) descriptor queues.
• Through MSI-X, the operating system kernel assigns Queue 0 to fire Vector 0x50 targeting Core 0, Queue 1 to fire Vector 0x51 targeting Core 1, and so forth.
• Each CPU core services its own independent network stream from its own hardware queue, executing locklessly with 100% CPU cache locality and zero inter-core synchronization contention!

4. Silicon Execution Sequence: IDT & Hardware Stack Switching

When an interrupt vector arrives at a CPU core (either from an MSI-X memory write or the I/O APIC), the processor hardware executes an atomic sequence before software receives control:

/* The x86-64 Interrupt Descriptor Table (IDT) Gate Descriptor (16 Bytes) */
typedef struct {
    uint16_t offset_low;      /* Target ISR Handler Address [Bits 0..15] */
    uint16_t segment_selector;/* Kernel Code Segment Selector (e.g. 0x08) */
    uint8_t  ist;             /* Interrupt Stack Table (Bits 0..2) */
    uint8_t  type_attributes; /* DPL (Privilege), Present Flag, Gate Type (0xE) */
    uint16_t offset_mid;      /* Target ISR Handler Address [Bits 16..31] */
    uint32_t offset_high;     /* Target ISR Handler Address [Bits 32..63] */
    uint32_t reserved;
} __attribute__((packed)) idt_entry_t;

2. The Dual-Phase Interrupt Model: Top-Half vs. Bottom-Half

In Section 1, we analyzed how the APIC and CPU hardware vector an interrupt into an Interrupt Service Routine (ISR). However, modern kernel developers face an acute microarchitectural conflict when designing driver handlers: the Interrupt Latency Paradox.

The Core Separation of Concerns

To resolve the latency paradox, all production operating systems partition interrupt handling into two distinct, cooperative execution stages:

Deep Dive: The Linux Deferred Execution Subsystems

The Linux kernel provides three distinct architectural mechanisms for bottom-half deferral, each tailored to specific latency and concurrency profiles:

Modern Threaded IRQs in Linux

In modern Linux, writing manual softirqs or tasklets is increasingly superseded by Threaded Interrupt Handlers (request_threaded_irq()):

/* Registering a Modern Linux Threaded Interrupt Handler */
int request_threaded_irq(
    unsigned int irq,
    irq_handler_t quick_check_handler, /* Top-Half (Hard-IRQ) */
    irq_handler_t thread_worker,       /* Bottom-Half (Preemptible Kernel Thread) */
    unsigned long irqflags,
    const char *devname,
    void *dev_id
);

/* Top-Half: Runs with hardware interrupts disabled */
static irqreturn_t quick_check_handler(int irq, void *dev_id) {
    struct my_device *dev = (struct my_device *)dev_id;
    if (!(read_reg(dev, REG_STATUS) & STATUS_INTERRUPT_ACTIVE)) {
        return IRQ_NONE; /* Not our interrupt (shared IRQ line) */
    }
    write_reg(dev, REG_ACK, ACK_BIT); /* Acknowledge controller */
    return IRQ_WAKE_THREAD;           /* Kernel automatically wakes thread_worker! */
}

/* Bottom-Half: Runs as dedicated kernel thread `irq/42-mydev` (Can sleep!) */
static irqreturn_t thread_worker(int irq, void *dev_id) {
    mutex_lock(&dev_lock);            /* Legal: Threaded IRQs can acquire mutexes */
    parse_inbound_data(dev_id);
    mutex_unlock(&dev_lock);
    return IRQ_HANDLED;
}

Deep Dive: Windows NT Interrupt Architecture — DIRQL to DPC

The Windows NT kernel implements a rigorous, hardware-enforced priority model governed by Interrupt Request Levels (IRQLs):

The Windows NT Execution Sequence

1. ISR Execution (at DIRQL): When a peripheral asserts an interrupt, the hardware raises the CPU core's IRQL to the device's assigned DIRQL (Device IRQL, e.g. IRQL 12).
   Code at DIRQL is strictly prohibited from touching pageable virtual memory, acquiring user mutexes, or waiting on kernel events. The ISR reads the device registers, copies pointer references, and queues a DPC:

   KeInsertQueueDpc(&DeviceExtension->DpcObject, SystemArgument1, SystemArgument2);

2. Dropping to DISPATCH_LEVEL: The ISR exits, sending EOI. The Windows kernel checks if any other hardware interrupts are pending. If none exist, the kernel drops the processor's IRQL to DISPATCH_LEVEL (IRQL 2).
3. DPC Execution: The Windows kernel maintains a per-processor DPC queue. At DISPATCH_LEVEL, the kernel drains queued DPC routines with hardware interrupts fully enabled (a DIRQL interrupt can preempt a DPC at any time).
4. The Strict Rules of DISPATCH_LEVEL: While at DISPATCH_LEVEL, code is executing outside of any particular thread context. Therefore:
   • Cannot Wait with Non-Zero Timeout: Calling KeWaitForSingleObject with a non-zero timeout is illegal (there is no thread to put to sleep!).
   • No Page Faults Permitted: All code and data buffers accessed inside a DPC must reside in Non-Paged Kernel Pool (RAM-resident). Triggering a page fault at DISPATCH_LEVEL causes Windows to immediately crash with BUGCHECK 0x0A: IRQL_NOT_LESS_OR_EQUAL!

3. Direct Memory Access (DMA) & Cache Coherency

Direct Memory Access (DMA) is the cornerstone of high-throughput operating system I/O. By delegating data movement to dedicated bus-mastering engines on peripheral controllers, the CPU is completely liberated from the cycle-wasting overhead of byte-by-byte Programmed I/O.

However, routing high-speed peripheral data directly into physical DRAM introduces a profound architectural conflict with modern CPU memory hierarchies: The Cache Coherency Dilemma.

The Fundamental Problem: The CPU-DRAM Semantic Gap

Central processing units read and write memory through high-speed, on-die write-back L1, L2, and L3 caches. A write instruction executed by the CPU does not immediately touch physical DRAM; instead, it updates the CPU's local cache line and marks the line as Dirty (Modified).

In contrast, a Direct Memory Access engine is an independent bus master that reads and writes directly to physical DRAM across the system interconnect, bypassing CPU caches entirely. This creates two distinct data corruption hazards:

Architectural Solutions: Hardware Snooping vs. Software Maintenance

Operating systems interface with two fundamentally different hardware caching topologies:

1. Hardware-Coherent Architectures (x86-64 & Enterprise ARM)

On standard x86 and enterprise server architectures, cache coherency is enforced completely in silicon via Bus Snooping and directory protocols (such as MESI/MOESI):

• When a PCIe device writes to physical DRAM, the PCIe Root Complex and Memory Controller broadcast a snoop transaction to all CPU cores across the internal coherent interconnect.
• If a CPU core detects that the DMA write address is present in its L1/L2/L3 cache, the hardware cache controller automatically invalidates the local cache line.
• If the device reads from DRAM while a CPU core holds dirty uncommitted data in its cache, the snoop controller forces the CPU core to flush its dirty line to the bus first, satisfying the device read with valid data.
• Operating System Impact: Device driver developers on x86 do not need to execute manual cache flushes. Hardware guarantees transparent memory coherency.

2. Non-Coherent Architectures (Embedded ARM, Mobile SoCs, DSPs)

To conserve silicon die area, reduce design complexity, and minimize battery power consumption, mobile and embedded System-on-Chip (SoC) architectures frequently omit bus snooping logic.

On non-coherent systems, the operating system kernel is legally responsible for executing manual cache maintenance operations before and after every DMA transfer:

The Linux DMA Mapping API

To write portable device drivers that run identically on hardware-coherent x86 systems and non-coherent ARM platforms, Linux provides the unified DMA Mapping Framework:

/* 1. Consistent (Coherent) DMA Allocation:
      Allocates physically contiguous memory that is mapped as Uncacheable (UC).
      Ideal for circular ring descriptors and command mailboxes accessed continuously. */
dma_addr_t dma_handle;
void *ring_buffer = dma_alloc_coherent(
    dev,
    4096,
    &dma_handle,      /* Physical address given to device */
    GFP_KERNEL
);

/* 2. Streaming DMA Mapping:
      Used for high-throughput packet and disk buffers allocated via standard kmalloc/page-alloc.
      Automatically executes cache clean/invalidate instructions on non-coherent hardware! */
dma_addr_t phys_addr = dma_map_single(
    dev,
    packet_data,
    packet_len,
    DMA_FROM_DEVICE    /* Direction: Inbound receive */
);

/* Give phys_addr to hardware controller... wait for completion interrupt */

/* After interrupt arrives: unmap to invalidate CPU caches before CPU reads packet */
dma_unmap_single(dev, phys_addr, packet_len, DMA_FROM_DEVICE);

The IOMMU: Translation, Isolation, and Virtualization

In early computer architectures, peripherals placed raw host physical addresses directly onto the system memory bus. In modern enterprise systems, all DMA transactions pass through a dedicated hardware memory management unit: the I/O Memory Management Unit (IOMMU) (known as Intel VT-d, AMD-Vi, or ARM SMMU).

The IOMMU solves three critical operating system challenges:

1. IOVA to Physical Address Translation (De-fragmenting Paged Memory)

User applications allocate large memory buffers that appear contiguous in virtual memory, but are fragmented across hundreds of arbitrary 4 KB page frames in physical DRAM.

• Without an IOMMU, the driver must build complex Scatter-Gather Lists, and devices without scatter-gather hardware must copy data through intermediate Bounce Buffers (costing massive CPU memory-copy overhead).
• With an IOMMU, the kernel programs I/O Page Tables. The IOMMU maps a single, contiguous range of I/O Virtual Addresses (IOVA) to the scattered physical page frames in DRAM. The peripheral executes a single, continuous DMA burst without knowing that physical memory is fragmented!
• Furthermore, the IOMMU allows legacy 32-bit DMA devices (limited to 4 GB addressing) to access physical memory located above the 4 GB boundary in 64-bit systems without bounce buffers.

2. Memory Protection & DMA Fault Isolation

Direct Memory Access is inherently hazardous: a compromised PCIe network card or a device with buggy firmware could issue rogue DMA memory writes over the top of the operating system kernel code, page tables, or security tokens.

• The IOMMU implements strict Device Context Tables indexed by the peripheral's PCIe Bus/Device/Function (BDF) identifier.
• Each device is restricted to its own private address translation space. If a peripheral attempts a DMA read or write to an address that has not been explicitly mapped by the OS kernel, the IOMMU blocks the transaction at the hardware layer, halts the transfer, and triggers an uncorrectable PCIe DMA Remapping Fault interrupt to the kernel.

3. Direct Hardware Passthrough in Virtualization (SR-IOV)

In cloud and hypervisor environments, operating systems virtualize I/O:

• Normally, hypervisors emulate hardware devices in software, introducing significant latency.
• With IOMMU support, the hypervisor can assign a physical PCIe network interface or GPU directly to a guest Virtual Machine (VM).
• The IOMMU translates Guest Physical Addresses (GPA) directly into Host Physical Addresses (HPA) at wire speed. The guest VM controls the hardware directly with zero hypervisor intervention and bare-metal performance, while the IOMMU guarantees that the guest VM cannot access host memory outside its assigned slice.