Why GPUs power AI
The short answer is parallel processing.
The longer answer? A CPU has a handful of powerful cores. Each one handles complex logic well and is optimized to finish a single task as fast as possible. That's what most programs need.
Graphics workloads are a bit different though. Every pixel needs its color, lighting and texture computed, and the whole screen has to be redrawn many times a second to feel smooth. That's millions of tiny, identical tasks, all due within a few milliseconds. A 4K screen has about 8 million pixels, so redrawing it 60 times a second means 500 million pixels a second, each needing dozens to hundreds of operations. A CPU can only do a few at a time, so it falls behind quickly and has little left over for the rest of the program.
Pixels don't depend on each other, though, so they can all be computed in parallel. GPUs are designed around that: thousands of smaller, simpler cores, each slower than a CPU core, all running the same instructions on different data. They're built to get through as much work as possible, not to finish any one task fast (throughput over latency).
Turns out neural networks have the same problem shape. Training and running a model are mostly matrix multiplications, which means huge numbers of multiply-adds, most of them independent of each other. For instance, a language model with 7 billion weights (the numbers it learned during training) does around 7 billion multiply-adds for every token (roughly a word) it generates. A GPU can run thousands of them at the same time.
Still, how did anyone think to use a graphics chip for this? Once you could run your own small programs on GPUs, people started using them for other math by pretending the numbers were pixel colors. NVIDIA later made that a lot easier with CUDA, which lets you write regular code for the GPU. Then in 2012, a neural network called AlexNet was trained on two gaming GPUs and did way better than anything else at recognizing images. After that, pretty much everyone training neural networks moved to GPUs.
These days GPUs are designed with AI in mind too. They have circuits that do a small matrix multiplication in one step, called tensor cores.
Memory matters just as much. A large model's weights can take up tens or hundreds of gigabytes, and generating each token means reading through most of them. So AI GPUs come with lots of fast memory on the card (VRAM) to keep thousands of cores from sitting idle waiting for data.
A chip built to draw explosions in video games ended up being the right chip for AI. Nobody planned that, at least not at first.