Why activation function is needed in Neural Networks???

-

Today allow me to share with you, the relationship between a spider and an activation function  

Activation function is used in Neural network to introduce the non-linearity.

Without the proper activation function, neural network will simply be large Linear Model. 

 Consider the case where you have a data which is linearly separable 



Constructing a model to separate yellow and orange data points is very simple. A simple logistic regression will do this job very perfectly

 Now consider the case where your data set looks like something shown below



What do you think now?? will a simple linear model be able to carve out the hyperplane in such a way to classify both yellow and orange points.

I don’t think so. And it is the fact that a simple linear model may not be able to find complex patterns in a data.

 So now we go to big brother of linear model -Neural network.

Neural Network has this super ability to both carve out linear dependency as well as nonlinear dependency as per dataset.

 

How I perceive activation function is some special kind of function which gives super power to the simple linear model which helps model to capture complex, nonlinear patterns.




As discussed earlier neural network with no activation function or simply identity activation function is just a large Linear Model.

 

Identity Activation Function

                                        


Some famous activation functions used in NEural NEtorks are 

1. Sigmoid Function  



 2. Rectified Linear Unit (ReLU)


3. Leaky ReLU 

Comments

Post a Comment

Popular posts from this blog

Covariance and Correlation

-
Image
While dealing with any ML project one spends most of the time in data preprocessing .Data pre-processing is nothing but playing with the features present in the dataset ,data cleaning,data transformation ,removing or imputing missing feature values etc. Feature engineering corresponds to developing  new features based on current information . It may happen that few features are related to each other be it directly or inversely . Consider three cases:  Let's say radius and circumference are features present in the dataset ,so we know that they are directly dependent .Here if radius increases ,perimeter also increases.       2. Let's say speed and time are features present in the dataset ,so we know that they are inversely               proportional  .Here if speed increases ,time required to cover a distance decreases.         3. Let's say height and hair color are features present in the datas...
Read more

Split it up - Part 1

-
Image
M achine  L earning ???? How can a machine even learn??  For a long period of time I always wondered what this fancy term meant and how would it be actually possible ..!! Let's figure it out today. To get an essence of what happens when we say Machine is learning ,why not understand in the first place how we learn. Well..!!! let us take a glance at those days when we learnt mathematics as a kid. I still remember how my maths teacher used to teach us basic addition and subtraction in class .Initially she helped us to count using fingers and then add or subtract those numbers. With a lot of practice and sometimes getting beaten up by her we finally learnt addition /subtraction. Mathematically speaking after lot of practice /training , our brain learnt a function of addition where if we get numbers to add ,it processed in a way to give the sum. Similarly Machine Learning is all about finding that magic function "f" wherein ,we give some input and desire some output . (EXC fo...
Read more