How does the CPU communicate with the GPU (from a hardware perspective)?

I would imagine that the CPU communicated with the GPU similarly to how it communicated with any other peripheral. GPUs are inserted into motherboards via a PCIe expansion card, so it stands to reason that communication must occur through this interface.

As prof Kayvon mentioned in class, the separation of "host code" and "device code" is very similar to the ISPC setup where we have usual C code and separate special ISPC code. But the most important difference between these two cases is that in ISPC setup both sections of codes are running on CPU, while in this case the device code is running on GPU.

According to our lecture, the "device" function is to be compiled on GPU. It can be callable from any code that's running on GPU.

What other languages can have CUDA interfaces? Is there some default CUDA api that the CUDA compiler exposes, or do languages need to be directly supported?