Another benefit of making DNN training more efficient is reducing environmental impact! When we program on our personal computers, we rarely worry about consuming enough energy to make a dent in the world, but when we are talking about large-scale training of massive DNNs by companies/orgs around the world, the carbon footprint is substantial. Here's an article if you want to know more: https://www.forbes.com/sites/robtoews/2020/06/17/deep-learnings-climate-change-problem/?sh=10068a2e6b43.

A poignant excerpt: AI has a meaningful carbon footprint today, and if industry trends continue it will soon become much worse. Unless we are willing to reassess and reform today’s AI research agenda, the field of artificial intelligence could become an antagonist in the fight against climate change in the years ahead.

I know when I first learned about this, I was definitely shocked at first. But knowing the impact of our work -- not only the social dimension but also environmental -- puts us engineers on a path to do better work in the world.

"In a widely discussed 2019 study, a group of researchers led by Emma Strubell estimated that training a single deep learning model can generate up to 626,155 pounds of CO2 emissions—roughly equal to the total lifetime carbon footprint of five cars. "

I've been doing an ML and a DL class alongside this one, and I spent a good amount of time looking at MobileNets. But I'm wondering, after MobileNet was made to downsize networks for computer vision, were there any other big models that tried to do this? I've seen SqueezeNet, for example, but I'm not sure where to look for more!