Understanding the Machine (Part 2): SIMD, Branches, and the OS

Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1

Part 2 of Chapter 4 is where Morley gets hands-on with SIMD, shows how branch style changes generated code, and reminds you that the operating system sits between your pointers and physical RAM.

Previous: Understanding the Machine (Part 1) | Next: Data Structures (Part 1)

SIMD and the saxpy example

Single Instruction Multiple Data (SIMD) runs one operation on 2, 4, 8, or more lanes at once. SSE2 (128-bit), AVX2 (256-bit), and AVX-512 (512-bit) are the x86 ladder most desktops support at least partially.

Morley uses BLAS saxpy: y[i] += a * x[i].

A range-zip loop compiles to scalar SSE moves (movss, one float at a time). Switching to a plain index loop with std::span produces real vector loads (movups, mulps, addps, 4 floats per step). Lesson: write loops the compiler can understand.

Function multiversioning

Compile whole translation units with -mavx2, or use [[gnu::target_clones("sse4.2,avx2,default")]] so GCC/Clang emit multiple bodies plus a dispatcher that picks the best ISA at runtime. default is required as fallback.

You can roll your own dispatcher: declare saxpy_sse42, saxpy_avx2, etc., query CPUID once (cache the result!), and switch. CMake can generate clones from saxpy.cpp.in with per-file -mavx2 flags.

Morley warns: prefer compiler auto-vectorization over hand intrinsics. Manual AVX2 saxpy_hand using immintrin.h did not beat the compiler’s version in his benchmarks.

Alignment and aliasing

Aligned loads are faster when addresses are multiples of vector width. C++17 operator new with std::align_val_t and alignas on stack or class types help. For saxpy the copy cost may not justify alignment; for GEMM it can.

Aliasing means two pointers might touch the same memory. The compiler must preserve order. C99 restrict and compiler __restrict extensions tell the optimizer pointers do not overlap. std::span cannot carry that promise, which is one reason hot inner loops sometimes use raw pointers in dispatchers.

Branch prediction and speculative execution

CPUs guess which way branches go and prefetch instructions and data for the predicted path. Speculative execution may run both sides briefly, discarding the wrong one.

Morley compares two clamp loops on uint16_t values capped at 255:

• clamp_min uses std::min → conditional move (cmova) that waits on the compare.
• clamp_conditional uses an explicit if (v > max) → branch with a store only when needed.

On uniform random inputs up to 65535, the branch version averaged ~500 ns vs ~605 ns for the min version. Only ~0.4% of values actually needed clamping, but pipelining behavior differed. Morley’s point: measure before you “optimize” branches.

Tools like Intel VTune explain micro-architectural stalls. Chapter 15 covers profiling more fully.

Security

Speculation enabled Spectre-class side-channel attacks (2017 and 2018). Security-sensitive code may need speculation barriers. Microsoft publishes guidance; Morley does not dig in, but the warning is real.

The operating system layer

The OS runs in kernel mode with privileges user code lacks. It schedules processes, maps virtual addresses to physical RAM, and mediates hardware.

Virtual memory and pages

Processes see virtual addresses, not physical ones. Reasons: security isolation, and the illusion of more address space than RAM installed.

Memory is split into pages (~4 KiB typical). A page table maps virtual pages to physical frames. Every access would be slow without the TLB (translation look-aside buffer), a small cache of recent mappings. TLB thrashing happens when data spans many pages. Transparent huge pages (Linux) or larger page sizes can help; Morley says let the OS handle it unless you have a strong reason.

Query page size with sysconf(_SC_PAGESIZE) on POSIX or GetSystemInfo on Windows, then cache the result.

CPU affinity

The scheduler picks which thread runs on which core. Migration evicts warm cache. CPU affinity pins a process to specific cores (sched_setaffinity on Linux). Use sparingly; fighting the scheduler can hurt the rest of the system. Prefer higher-level abstractions when they exist.

Cross-platform SIMD note

If you ship on both x86 and ARM, Morley points to portability layers like SIMD-everywhere (simde) and Google’s Highway. They wrap intrinsics so one source path can target multiple ISAs. You still need runtime or compile-time dispatch, but you avoid maintaining wholly separate intrinsic blocks per platform.

Chapter summary

Cache and SIMD are the two big levers for throughput. Not every chip has every extension, and high-level C++ exists partly to hide ISA details. Still, when performance matters, knowing what happens under your loop body makes you a better engineer.

Morley’s saxpy numbers on a 5120-element vector: scalar and SSE4.2 clones around 330 ns, hand-written AVX2 intrinsics ~298 ns, compiler AVX2 clone ~259 ns, AVX-512 ~243 ns. OpenBLAS was ~278 ns on that tiny problem (overhead dominated). The lesson is not “always write intrinsics.” It is “help the compiler, pick the right ISA at runtime, and measure.”

Chapter 5 connects this to data structures: how you lay out memory in containers matters as much as which algorithm you pick.

Previous: Understanding the Machine (Part 1) | Next: Data Structures (Part 1)