Blog

Some other blog post that you may want to read is

• Time Series analysis in python part-1
• Time Series analysis in python part-2
• How to perform data analysis in python
• Linear regression in python
• Single Shot Object detection

Classification

Classification comes under the category of supervised learning i.e it learns from a given set of inputs and makes predictions on unseen data. Supervised learning has two main parts which are Regression and classification.

In the previous blog post we learn about regression in this blog post we learn about classification mainly logistic regression and it’s an implementation in python.

Lets us understand classification by taking a real-life example, suppose we have given a medical record of the patient and we have to predict that the given patient has blood sugar or not. 1 represents the patient has blood sugar while 0 represents the patient does not have blood sugar.

There are mainly two types of classification which are

1- Binary or Binomial classification:- In the previous examples that were the problem of binary classification in which target labels are in the form of 0 and 1.

2- Multi-class ,or multinomial classification:- Three or more classes of output come under multi-class classification.

Application of classification are

1- image classification and video classification is perhaps the best application of classification. Almost every industries i.e medical and agriculture using this. Some of them are lung cancer classification, real-time activity classification, medical imaging classification, and plant weeds classification.

2- In the financial sector we are using classification applications to find loan defaulters, credit scoring, loan distribution i.e whether a customer has to give loan or not. Many companies like RedCarpet and KreditBee uses classification to give loans to the customer.

3- In the Sports industry whether a particular team is going to win a match or not based on the previous track records.

4- Text classification and sound classification is used to classify text and sound. Sound classification is used by marine engineers.

Logistic Regression Overview

The name logistic regression sounds inappropriate. This statistical method is not used to model regression problems but we use to solve a classification problem.

Logistic regression is known as the function used at the core of the tactic, the logistic function. In linear regression, the result (dependent variable) is continuous. It can have any one of an infinite number of possible values.

In logistic regression, the result (dependent variable) has only a limited number of possible values. Logistic Regression is employed when the response variable is categorical in nature.

The logistic function also called the sigmoid function is an S-shaped curve that will take any real-valued number and map it into a worth between 0 and 1, but never exactly at those limits.

Logistic regression uses an equation as the representation, similar to linear regression. The central assumption of Logistic Regression is that your input space is often separated into two nice ‘regions’, one for every class, by a linear(read: straight) boundary. Your data must be linearly separable in n dimensions.

The term logistic refers to the log odds probability that is modelled the term odds.

And the term odd is defined as the ratio of the the probability that an event occurs to the probability that it doesn’t occur.

odds= P(event)/1- P(event)

Let us consider the conditional probability

Pr(y=1|X) = p(X)

In the above equation note that 1 is not the number it is the class or category.

We need to model a probability using a curve where X can be anything i.e X ∈ R

where x can be anything in the range of 0 and 1.

There are many ways to accomplish this thing, in the logistic regression we use sigmoid function to accomplish this.

The goal of the learning is to estimate parameter vector B^ in order to make predictions.

As in the linear regression we use least square method to estimate parameters in the same way, logistic regression uses Maximum Likelihood method for parameter estimation.

So the question arises is how does this maximum likelihood works?

Step 1– Consider n samples with labels either 0 or 1.

Step 2– For the sample labelled “1”: Estimate Beta hat (B^) such that p(X) close to 1.

Step 3- For sample labelled “0”: Estimate B^ such that 1- p(X) is as close to possible as 1.

Mathematically it is represented as

where i is with ith sample.

On combining these equations we want to find Beta parameters such that product of both these equation is maximum over all elements of the dataset.

This function that we need to optimize is called as Likelihood function.

After that we can take the log likelihood and convert them into a summation.

Now. we substitute p(x) values then we now group the coefficients of yi.

Now after simplifying the equation we get

Let us recall that our goal is to find the Beta which maximizes the function.

The above equations we get are also called transcendental Equation which cannot be computed exactly that’s why we use numerical methods for approximation

which is Newton raphson method of approximation to find Beta coffecients.

Now let us move on to code.

Logistic Regression in Python from scratch

There are basically 5 steps to solve logistic regression in python.

Step 1- Import all the required libraries

import numpy as np
import matplotlib.pyplot as plt
import numpy.matlib # return matrices
from sklearn.metrics import accuracy_score
numpy.random.seed(42)

Step 2- Create custom dataset

c1 = [2,3]# Cluster center for class 1 data
c2 = [10,11]# Cluster center for class 2 data

no = 50 # No of samples in a class
class1 = np.matlib.repmat(c1,no,1) + 

np.random.randn(no,len(c1))
class2 = np.matlib.repmat(c2, no,1)+ 

np.random.randn(no,len(c2))
D = np.append(class1,class2,axis =0)
Data = np.concatenate((D, np.ones((2*no,1))),axis = 1)

c1_label = np.ones((no,1))
c2_label = -1*np.ones((no,1))

label = np.concatenate((c1_label,c2_label),axis = 0)
Data = Data.T
y = label.T # True label

Step 3- Create validation data

v1 = [5,4]# Cluster center for class1 data of validation data set
v2 = [7,12]# Cluster center for class 2 data of validation data set

v_no = 30# No of samples in a class
v_class1 = np.matlib.repmat(v1,v_no,1) + np.random.randn(v_no,len(v1))

v_class2 = np.matlib.repmat(v2, v_no,1)+ np.random.randn(v_no,len(v2))

v_D = np.append(v_class1,v_class2,axis =0)
v_Data = np.concatenate((v_D, np.ones((2*v_no,1))),axis = 1)

v_c1_label = np.ones((v_no,1))
v_c2_label = -1*np.ones((v_no,1))
v_label = np.concatenate((v_c1_label,v_c2_label),axis = 0)
v_Data = v_Data.T
v_y = v_label.T# Test label

Step 4- plotting custom dataset and validation

import matplotlib.pyplot as plt
plt.plot(class1[:,0],class1[:,1],'ro',class2[:,0],class2[:,1],'bo')
plt.show()

plt.plot(v_class1[:,0],v_class1[:,1],'ro',class2[:,0],class2[:,1],'bo')
plt.show()

Step 5- The fifth step is to define sigmoid function which helps us to output the probabilities between 0 and 1 and also define prediction function.

def sigmoid(x):
     return 1/(1+np.exp(-x))

def prediction(w, Data):
    pred = []
    z = np.dot(w,Data)
    a = sigmoid(z)
    for i in range(0,len(a[0])):
        if (a[0][i] > 0.5): 
            pred.append(1)
        elif (a[0][i] <= 0.5):
            pred.append(-1)
    return pred

Step 6- Training part- In this step, the loss is calculated and minimized with respect to weights.

learning_rate = 0.01 #define hyperparameters
w = np.random.randn(1,3)
for i in range(1,1500):
    
    z = np.dot(w,Data)
    y_pred = prediction(w, Data)
    val = -np.multiply(y,z)
    J = np.sum(np.log(1+np.exp(val)))
    num = -np.multiply(y,np.exp(val))
    den = 1+np.exp(val)
    f = num/den
    gradJ = np.dot(Data,f.T)
    w = w - learning_rate*gradJ.T

    print("Epoch",i,"Loss",J,"Training Accuracy",accuracy_score(y[0], y_pred)*100)
    
    #Result --- 
    
    #('Epoch', 1499, 'Loss', 0.2563074206291997, 'Training Accuracy', 100.0)

we are getting 100% accuracy on training data

let us make prediction on test data. And then plot the result

Test_predict = prediction(w, v_Data)
print("Test Accuracy",accuracy_score(v_y[0], Test_predict)*100)

domain =  np.linspace(0,13,100)
h_x = -(w[0,0]/w[0,1])*domain - (w[0,2]/w[0,1])

plt.plot(v_class1[:,0],v_class1[:,1],'ro',v_class2[:,0],v_class2[:,1],'bo')
plt.plot(domain,h_x)
plt.show()

Wrap up the Session

You now know what logistic regression is and the way you’ll implement it for classification with Python. You’ve used many open-source packages, including NumPy, to work with arrays and Matplotlib to visualize the results. You also used both scikit-learn and StatsModels to create, fit, evaluate, and apply models.
Generally, logistic regression in Python features a straightforward and user-friendly implementation. It usually consists of these steps:

1- Import packages, functions, and classes

2- Get data to figure with and, if appropriate, transform it

3- Create a classification model and train (or fit) it with existing data

4- Evaluate your model to ascertain if its performance is satisfactory

5- Apply your model to make predictions

You’ve come a long way in understanding one of the most important areas of machine learning! If you have questions or comments, then please put them in the comments section below.

So if you like this blog post, please like it and subscribe to our data spoof community to get real-time updates. You can follow our Facebook page to get notifications whenever we upload any post so you can never miss any update from us.