Many Cores, SIMD and Accelerators

For thirty years each new processor ran old code faster, mostly by raising the clock. Around 2005 that stopped. Faster clocks made chips too hot to cool, and the growing supply of transistors went into more cores, wider arithmetic units and specialized accelerators instead.

The speed is still there, but it is no longer free. A program gets it only if its work can be split into independent pieces, and the kind of split decides which hardware can use it: many cores for many independent tasks, vector units for the same arithmetic over many values, a GPU for enormous amounts of identical work. This topic matches each kind of work to its hardware and prices the mismatch.

The End of Free Clock Speed

For decades, shrinking transistors let each generation run faster at the same power per area, a relationship known as Dennard scaling. In the mid-2000s it broke down: as transistors kept shrinking, current leaking through them when they were supposed to be off, and the heat of switching them faster, stopped falling in step. Clock speeds levelled out at roughly 3 to 5 gigahertz, where they still are.

Transistor counts did not level out. They kept growing, into the tens of billions on current large chips, so the question for chip designers changed from how fast to run one instruction stream to what to spend all those transistors on. Their answers are the rest of this topic, and each one only helps a program shaped to use it.

More Cores

The first answer was more complete processors on one chip. A laptop now has roughly 8 to 16 cores, and a single server processor can have more than a hundred. Each core is a full copy of everything the earlier topics in this chapter described, with its own pipeline, predictor and registers.

A single-threaded program uses exactly one of them. The rest help only if the work is split into threads or processes that can run at the same time, which is Chapter 11. That chapter also gives Amdahl's law: the part of a job that cannot be split sets a hard ceiling on the speed-up, whatever the core count.

Hardware Threads and the vCPU

A core spends much of its time stalled, waiting for memory, as the previous topics showed. Simultaneous multithreading, which Intel markets as Hyper-Threading, lets one core run two instruction streams at once and fill one stream's stalls with the other's work. The operating system sees two logical processors, but they share one core's arithmetic units, so together they deliver well under twice one core's throughput.

This matters when reading a cloud bill. On most x86 instance types, a cloud vCPU is one of those hardware threads, so two vCPUs are usually one physical core, not two. Some ARM-based instance types are the exception, where a vCPU is a whole core. Check which one you are buying before sizing CPU-bound work.

SIMD: One Instruction, Many Values

SIMD, single instruction multiple data, applies one operation to a whole register of values at once. A 256-bit vector register holds eight 32-bit floats, and one instruction adds them to eight others. Image codecs, compression, JSON parsers, memory copies and numerical libraries such as NumPy all use these instructions, because their inner loops do the same arithmetic over long runs of values.

Measured on the laptop this book was written on, summing a million 64-bit integers took about 24 milliseconds as a Python loop, about 3 milliseconds with the built-in sum function, and about a tenth of a millisecond with NumPy. That is roughly two hundred times faster than the loop. Most of the gain comes from skipping the interpreter on every element, and SIMD adds its share on top.

GPUs and Other Accelerators

A GPU takes the idea to its limit. It runs thousands of simple threads that perform the same operation on different data. That makes it excellent at graphics, at the matrix arithmetic behind machine learning, and at other large, uniform numerical work, and poor at branchy, sequential logic. GPUs also popularized reduced-precision number formats, 16-bit and smaller floats, that trade the exactness of Chapter 2 for more arithmetic per second. The machine learning courses in this catalogue take it from there.

Data must first travel from main memory to the GPU's own memory and back, so a small job can lose more in transfer than it gains in arithmetic. Chips also carry fixed-function units that do a single job, such as video encoding, encryption or neural-network inference, at a fraction of the power a general-purpose core would need for the same work.

Matching the Work to the Hardware

A sequential, branchy job, such as a parser or a compiler, wants one fast core. Many independent requests, such as a web server's traffic, want many cores. The same arithmetic over large arrays wants SIMD or a GPU. Large, uniform numerical work that repays its transfer wants a GPU.

The cost in the systems you run is paying for the wrong kind. A 32-vCPU instance running a single-threaded Python process uses one hardware thread and bills for thirty-two. A GPU instance running a job too small to repay its data transfer is slower than the CPU it replaced, and considerably more expensive.