In other words, design the hardware such that it is focusing on optimizing operations that are common in deep learnings, rather than a general processor that is optimizing/balancing for everything.

If I understand correctly, adding these more complex instructions to GPUs makes the GPUs better at performing machine learning tasks, but still not quite as good as a dedicated machine learning ASIC. The reason we would want a GPU that occupies this middle ground is that we may want better machine learning capabilities without the added cost of an ASIC that cannot also fulfill our graphical needs.