This offers only a max 10,000x speedup! Honestly this feels really similar to an issue in the 802.11 spec for WiFi where initial headers in packets have to be communicated at 1Mbps regardless of the bandwidth capabilities of the machine/receiver in order to ensure backwards compatibility with older machines (who need to get info from all packets sent for purposes of synchronization). As wifi speeds get faster and faster, those headers become a bigger and bigger percentage of packets, placing a cap on the speed with which devices can communicate over wifi, very similarly to how sequential computation places a cap on the speedup.

Why is it 10,000x vs 1,000x speedup? If 0.1%, or 0.001 is sequential, and Amdahl's Law caps speedup at 1/S, then isn't 1/0.001 = 1000?

I wonder what percentage of serial bottlenecks are due to how programs are written versus the overhead in orchestrating parallelism or data bandwidth issues. It seems to me at this scale, those factors would become far more dominating. I wonder what the practical concern is for serial operations say in ML (maybe you build some custom fancy layer for just 1 part) and how much impact it actually has given existing limitations on parallelism by the hardware and machines you're using and corresponding bandwidths.

@arkhan I'm also wondering why it is 10,000x instead of 1,000x. During the class, I took note of the 10,000x speedup. But from Amdalh's law it should be 1,000x instead of 10,000x.

@timothyyeo @arkhan I think it is a typo and it should be 1000x speed up.