In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks. Activation functions are a crucial part of deep learning models, as they add the non-linearity to neural networks. There is a great variety of activation functions in the literature, and some are more beneficial than others. The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't. This notebook is part of a lecture series on Deep Learning at the University of Amsterdam. The full list of tutorials can be found at [ Ссылка ].
Link to the notebook: [ Ссылка ]
00:00 Setup
07:06 Gradient distributions
12:35 Model training
20:07 Dead neurons
24:20 Conclusion
