Deep Learning Specialization on Coursera
vr 09 april 2021
Having done several courses and projects with neural networks in university I was curious to learn a bit more about recent advancements. Overall I enjoyed doing this series of 5 courses, with a total of 20 weeks of material.
For those new to coursera: How much actual work is in 1 "week" of training depends. Usually for me one "week" maps to one workday of time.
The Deep Learning Specialisaion consists of 5 courses:
Neural Networks and Deep Learning. Consists of the basics. What is an NN? When is it deep learning? If you have done a university level course on neural networks you know the deal here. Well explained, but if you know the basics you can finish this part in a day.
Hyperparameters, regularization and optimization. This is where things got more interesting. Training a NN hasn't funamentaly changed since 1986, when backpropagation was introduced. But a lot of tricks have been found to speed up the training. Also to get the best out of a NN one needs to find suitable hyperparameters. One of the downsides to NNs compared to linear or tree family models.
Structure ML project. I thought this would be the most boring part of the specialisation, but actually this was very helpful. Andrew Ng explains how he evaluates his own projects, what things to pay attention to. For instance its easy to say you need more training data (that never hurts), but maybe that's expensive and you can find errors in your training set by sampling from misclassified observations.
Convolutional networks. A lot of the deep learning hype has come from computer vision. Where once humans had to work hard to create features to extract information from the high dimentional images, now the model can learn low-level feature representations. And creative ways of playing with these models gave rise to deep fake photos, reconstruting photos and paintings in famous styles. In my opinion this part of the specialisation is the best.
Sequence modeling. The part I thought I would find most interesting, but for some reason it felt rushed. More than one video had editing ques that were left in (Andrew restarting his sentence) or outside noise was left in (a phone buzzing). These models where the input lenght and output lenght are not fixed in advance, like in a normal ML model, but are flexible were new to me. And I think there is untapped potential. In the real word many things are sequential in nature and not neccisariy of a fixed length. I expected a nod to survival models, but the attention to language models was interesting as well.
My biggest criticism of the course is that the coding assignments are structured in sucha way that sometimes it feels like you're just filling the formulas from the assignment text. A bit more freedom would make it more helpful. But then again would make it too hard for people new to the subject.
Now, I expected resevoir computing to come up. My bachelors project used an Echo-State NN, and I always expected these type of networks to supersede the "simple" feed forward NN. It seems there is still al lot to research.
For me, the next step would be deep reinforcement learning. The models that learned to play atari games from deepmind that made a lot of news some time ago use Deep Q Networks. This course didn't touch that topic, but I am going to persue to learn more about that in the future.
For now my conclusion is that this course is rightfully popular. It tought me about the recent advancements around neural networks, is well put together, and Andrews has a fun way of explaining the material.