Kayvon mentioned that the memory controller was able to take different requests and bundle them in clever ways. Is that usually done in hardware by the memory controller? Software by the CPU? Or is that a comment about how writing data-local code on the programmer side is important?

There are specialized hardware managing and optimizing outstanding memory requests and merge requests using write-combining buffers etc. However, I agree that programmers should also be aware of the memory footprint because irregular memory access patterns often leave little space for hardware optimization.

Here's a paper about memory scheduling. https://www.cs.utexas.edu/~lin/papers/tocs07.pdf. And we can see it's done with a hardware unit with a write queue and read queue. This is just an example.