In lecture, Kayvon pointed out that, in the previous asynchronous example, we assumed that we could simply make a copy of all of our parameters and send copies to every worker node. This assumption rests on the fact that our system has enough memory to fit all of our parameters on one node. Recall on this slide that the bees knees of AI models use over a hundred billion parameters!
In lecture, Kayvon pointed out that, in the previous asynchronous example, we assumed that we could simply make a copy of all of our parameters and send copies to every worker node. This assumption rests on the fact that our system has enough memory to fit all of our parameters on one node. Recall on this slide that the bees knees of AI models use over a hundred billion parameters!