Understanding the Machine (Part 1): Processors, CPUID, and Memory

Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1

Chapter 4 answers a question Chapter 3 raised: your algorithm might be correct on paper and still slow because the CPU spent half its time waiting on RAM. Morley gives a compressed tour of modern hardware so C++ programmers know what they are actually driving.

Previous: Algorithmic Thinking and Complexity (Part 2) | Next: Understanding the Machine (Part 2)

Why this chapter exists

Modern chips are absurdly complicated: multiple cores, hyper-threading, SIMD, out-of-order execution, several cache levels. A 3 GHz clock means one cycle is under half a nanosecond. You cannot use that power if you do not know it exists.

Morley is clear: you do not need CPU design expertise to write good code. You do need it when you are chasing the last slice of performance. And optimization is expensive. Only do it where it matters.

Agner Fog’s optimization guides (agner.org/optimize) are the deep-dive reference Morley recommends.

Processor building blocks

The simplified picture:

• ALU does integer logic and arithmetic on registers.
• FPU handles floats, often with vector (SIMD) registers.
• MMU fetches data from memory into registers.
• Instruction decoder (frontend) feeds the backend that does real work.
• Pipelining runs independent operations in parallel and reorders work to avoid stalls.
• I/O controller talks to disks, network, GPUs over PCIe.

Between cores and main memory sit cache layers: L1 (often split data/instruction), L2, sometimes L3. Lower levels are smaller and faster. Higher levels are larger and slower but cut how often you hit RAM.

Instruction architectures

The ISA defines what instructions exist. Desktop x86-64 is complex (CISC heritage, decades of tuning). ARM and RISC-V lean simpler per instruction. SIMD shows up as optional extensions: SSE, AVX, AVX-512 on x86; NEON on ARM.

You can target extensions at compile time (-march=, -mavx2) or detect them at runtime. On x86, cpuid is how.

CPUID in practice

cpuid takes a leaf in eax and optional subleaf in ecx, then fills eax, ebx, ecx, edx with feature bits. Decoding means reading Intel/AMD docs or the Wikipedia CPUID page.

Morley wraps compiler differences in a my_cpuid macro (MSVC __cpuidex vs GCC/Clang __cpuid_count). He warns: use higher-level tools when you can (hwloc, platform APIs on Android/Windows/Linux). Raw cpuid is easy to get wrong (e.g., ecx must be cleared on some second calls).

The same machinery later queries cache size per level. Intel uses leaf 0x04, AMD uses 0x8000001D. You read manufacturer ID from leaf 0, loop subleaves, decode type/level/line size/associativity/sets, and compute size in KiB.

Threads and OpenMP

Most CPUs have multiple cores; many cores run two hardware threads (Intel Hyper-Threading). For big parallel jobs, split work across cores.

OpenMP adds #pragma omp directives. A nearest-neighbor distance example parallelizes the outer loop with reduction(min:...). CMake finds OpenMP::OpenMP_CXX and links the runtime. Morley’s benchmark: 65,536 points, 8 threads, ~4x speedup (4797 ms → 1161 ms).

Caveats: thread startup has cost. Small workloads can get slower. Apple Clang disables OpenMP by default. C++17’s std::execution::par helps for standard algorithms but is less flexible. Intel TBB is the alternative when you need more control.

The storage spectrum

From fastest/smallest to slowest/largest for active program data:

Registers

Single values (or SIMD packs) the ALU operates on. x86-64 has 16 general 64-bit registers per logical view; register renaming maps logical names to larger physical banks so speculative and parallel execution can proceed.

SIMD adds xmm (128-bit SSE), ymm (256-bit AVX), zmm (512-bit AVX-512), with lower halves aliasing wider registers for compatibility.

Cache

On-die SRAM, layered L1, then L2, then L3. Typical desktop: ~32 KiB L1d/L1i per core, ~1 MiB L2, 8 to 200+ MiB L3. Latency ranges from ~1 to 5 cycles (L1) to tens of cycles (L3). Programmer control is limited: access patterns and prefetch hints, mostly.

Cache lines are usually 64 bytes. Touching one byte can load the whole line. Sequential access wins. That is why struct of arrays beats array of structs for hot loops (games use this heavily).

Main memory (RAM)

Active code and data live here before cache. Latency is ~60 to 100 cycles (~20 to 30 ns) vs cache. Capacity is much larger. Desktop RAM comes as DIMMs (DDR5 today) with timings like CAS latency (CL 40 is normal for DDR5). SoC and laptop boards may integrate RAM differently.

Cache-friendly matrix multiply

Morley implements naive dgemm then a blocked version with a tile that fits in L1/L2 (e.g., 32×32 tile ≈ 8 KiB of doubles). Benchmark on 1024×1024 matrices: basic ~547 ms, blocked ~289 ms (~50% faster). OpenBLAS hits ~24 ms. Same math, different memory story.

The lesson for Part 1: know your storage hierarchy before you worry about SIMD intrinsics. Fix memory first.

Previous: Algorithmic Thinking and Complexity (Part 2) | Next: Understanding the Machine (Part 2)