This article introduces stepwise regression in Python which is a machine learning regression model and is an important model that can be used for solving a lot of complex problems. This blog provides a theoretical and in-depth guide to stepwise regression discussing the components of the models, their advantages and limitations, and the criteria for selecting the model. Thus, this article will help data scientists gather all the important information through this blog which will be helpful for them in understanding the predictive power of the model.
I. Introduction to Stepwise Regression in Python
The stepwise regression model is a statistical technique that builds a regression model through the selection of the most relevant variables from a set of predictors. A stepwise regression model is a form of variable selection for automatically determining the predictors which need to be included or excluded from the model based on their contribution towards the predictive power of the model. The stepwise regression model involves particularly two main steps for regression, forward selection and backward elimination. Here we provide a five-step explanation of how the model works:
- Initial model: The first step involves starting with an initial model that doesn’t contain any predictors.
- Forward selection: This step is an important step in which the variables are added one by one to the model on the basis of their individual statistical significance. This selection criterion is done through a significance level such as p-value or through predefined criteria such as AIC or BIC. The variable with the least p-value or highest criteria value is added.
- Backward elimination: Another important step in stepwise regression is backward elimination which is done after the variables are added in the forward selection step. The variables are removed one by one from the model on the basis of their statistical insignificance. The criteria for removing the variables are usually done through significance level or criterion value where the variable with the highest p-value or lowest criteria value is removed.
- Iteration: The 2nd and 3rd steps need to be repeated iteratively until no variables can be added or removed through the model and the iteration would stop in case of the variables not meeting the criteria.
- Final selection: After the iteration stops, the final model is to be selected on the basis of the criteria chosen.
There are other methods with which stepwise regression can be performed like forward selection, backward elimination, and bidirectional elimination. These three methods are discussed further in this blog so that enough information about the model is available to our readers. The type of method chosen depends on the research question, sample size, and other factors. It is also important to note that the stepwise regression model has increased risks of overfitting and sensitivity to the order of variable entry or removal and it is mostly recommended to interpret the results of stepwise regression cautiously. Overall, stepwise regression is quite efficient and can be used by data scientists for different purposes and applications as they see fit.
A. Importance of variable selection in machine learning
Variable selection is an important process in machine learning that must be performed carefully and in line with specific requirements of the model and dataset or the machine learning task. Here we discuss some reasons why variable selection is important in machine learning.
- Dimensionality reduction: Machine learning models suffer from the curse of dimensionality where the performance of the model deteriorates with an increase of variables. Thus, variable selection helps reduce dimensionality by having relevant variables.
- Interpretability: Variable selection helps in choosing important variables which makes a more interpretable model providing insights into the relationship between predictors and target variables.
- Improved performance of model: Having redundant and irrelevant variables in machine learning leads to overfitting which leads to a complex model and selecting only informative variables reduces model complexity and improves generalization performance.
- Data collection and consideration of cost: Models having fewer variables are easier to maintain and help when the model needs to be retained and simplifies the process compared to dealing with a large number of potentially irrelevant variables.
- Computational efficiency: Involving unnecessary variables in the modeling process increases the computational burden and the model needs to process and analyze more data. Selecting relevant variables helps us reduce computational requirements and speeds up training and inference processes.
Therefore, it is important to note that the variable selection process is important and in line with the specific requirements and characteristics of the dataset. Different variable selection techniques such as statistical significance, domain expertise, or feature importance measures can be employed depending upon the requirement.
II. Understanding Stepwise Regression in Python
Stepwise regression is a very important and useful model for machine learning which uses forward selection and backward elimination steps as the important steps for performing regression. This section discusses the need for variable selection in stepwise regression, types of stepwise regression, and the model selection criteria in stepwise regression. Finally, this section also discusses the advantages and limitations of using stepwise regression as a model.
A. The need for variable selection
Variable selection is an important aspect of the stepwise regression model and here are some reasons for validating how is variable selection important:
- Enhanced performance of model: Stepwise regression helps find a suitable subset of predictors for optimizing the model’s performance through a selection of relevant variables. The model’s predictive accuracy can be improved a bit and having irrelevant or redundant variables introduces noise while decreasing the model’s performance. Variable selection helps mitigate that issue by focusing on informative predictors.
- Improved model interpretability: The inclusion of all variables in the model makes it complex while variable selection helps identify a subset of variables that have the most significant impact on dependent variables. This facilitates the interpretation of the relationship between predictors and outcomes.
- Overfitting: Overfitting might occur when a model captures noise in the training data and leads to a poor generalization of new data. Stepwise regression helps mitigate overfitting through the variable selection that has a significant impact on outcomes and excludes variables that may introduce noise, thus, improving the ability of the model to generalize unseen data.
- Multicollinearity: Multicollinearity is the presence of strong correlations among predictor variables. High multicollinearity causes issues with the model such as inflated standard errors or unstable coefficient estimates. Variable selection helps identify and excluded highly correlated variables which improves the stability of the model.
- Efficient use of Resources: Data collection is considered resource intensive in some cases and collecting information on unnecessary variables is time-consuming and costly. Variable selection helps prioritize our preferred variables and optimizes the use of available resources.
Stepwise regression is a useful tool in variable selection but also has a lot of limitations and drawbacks as variable selection is done on statistical criteria which may not be able to capture the full complexity of underlying relationships. Therefore, interpreting the results of stepwise regression is to be done with caution.
B. Types of Stepwise Regression in Python
There are three different types of stepwise regression models which are discussed here:
1. Forward selection
In the forward selection, the stepwise regression model starts with an empty model. It will add variables sequentially that provide the most improvement in model fit. The algorithm will add one variable at a time and iteratively selects the predictor with the highest statistical significance or large improvement in the specified criterion. This process will continue until no additional variable meets the criteria of inclusion.
2. Backward elimination
The backward elimination will start with a model that has all the predictor variables and iteratively removes one variable at a time on the basis of a specified criterion. The algorithm will remove the predictor variable having the least statistical significance or small improvement in chosen criterion until no variables are up for meeting the predefined criterion of removal. The backward elimination model starts with the full model and reduces gradually until the optimal variable subset is reached.
3. Bidirectional elimination (hybrid)
This method combines forward selection and backward elimination to form stepwise regression and starts with an empty model which iteratively adds and removes variables on the basis of criteria. After this backward elimination comes up which removes the variables meeting exclusion criteria and this process continues until no other variables can be added or removed.
C. Model selection criteria
The stepwise regression model uses different selection criteria for the evaluation of performance and relevance of the variables at each step of the selection process. These criteria help guide the stepwise regression model by evaluating statistical significance, model complexity, and goodness of fit. Therefore, it is important to note that choosing selection criteria is subjective and depends on specific goals and requirements of analysis. Some of the common model selection criteria in stepwise regression are as follows:
1. Adjusted R-squared: The R-squared measures the proportion of variance in the outcome variable being explained through the predictors in the model. The adjusted R-squared adjusts for the number of variables in the model providing a measure of goodness of fit that penalizes overfitting. It is typically used for evaluating the model fitness at every step where the model with the highest R-squared value is selected.
2. Akaike Information Criterion (AIC): AIC quantifies the trade-off between the goodness of fit and complexity of the model. AIC gets calculated after each step in stepwise regression where the model with the lowest AIC values is selected. It will penalize the model for complexity and encourages the selection of simple models that provides a good fit to the data. It also balances the quality of model fit with a number of variables included that aim for the informative and parsimonious model.
3. Bayesian Information Criterion (BIC): It is similar to AIC, but puts a stronger penalty for model complexity. It balances model fitness and complexity and tends to favor simple models. It incorporates a larger penalty term for a number of parameters in the model that leads to a selection of a more parsimonious model than AIC.
4. P-value: The p-value criteria measure the statistical significance of the relationship between the predictor and the outcome variable. In forward selection. The variables with the lowest p-value get selected for model inclusion. In the backward selection, variables with the highest p-values get removed. P-value criteria are based on hypothesis testing in which a low p-value suggests a significant association between prediction and outcome.
D. Advantages and limitations of Stepwise Regression in Python
The stepwise regression model provides a lot of advantages and limitations which must be considered while selecting variable selection techniques.
The advantages of Stepwise Regression in Python are:
- Automatic variable selection: The stepwise regression model automates the process of selecting variables by iteratively adding and removing the predictors on the basis of predefined criteria which eliminates the need for manual selection and saves time and effort.
- Interpretability: Stepwise regression results in a more interpretable model by selecting a subset of predictors. The reduced number of variables makes it easy to understand and communicate the relationship between the predictors and outcomes.
- Efficient use of variables: The stepwise regression model helps us identify a subset of variables having the most relevant outcome variables. Including only significant predictors helps the model achieve good predictive performance and reduces the potential noise and overfitting along with irrelevant variables.
- Computational efficiency: Stepwise Regression in Python can be very computationally efficient while dealing with large datasets and a high number of predictors as it narrows down the search space by focusing on the most promising variables and reducing the computational burden.
The limitations of using the Stepwise regression model are:
- Influence of initial variables: The order of adding and removing the variables has a significant effect on the selection process which leads to different models and results.
- Overfitting and instability: Stepwise regression might overfit the model, especially in cases when it is applied to a small dataset or to a dataset having a large number of predictors. Including the variables is solely based on statistical criteria that lead to models capturing noise or chance associations in data.
- Ignorance of important variables: This model relies on statistical criteria for the selection of variables that might overlook important variables which are not statistically significant in the provided dataset. Domain knowledge and expert judgment might be required for considering other relevant predictors not captured in statistical criteria.
- Model dependency: Stepwise regression is dependent on a dataset that limits the generalizability and reproducibility of the selected model.
- False discoveries: Stepwise regression involves multiple hypothesis tests by adding or removing variables which increases the likelihood of false discoveries and variables appearing to be significant by chance.
Thus, stepwise regression does provide automated variable selection and helps identify the relevant predictors, but it has some limitations and drawbacks which is why it is important to interpret the results of the model with caution and consider alternative variable selection methods to validate the selected model on independent dataset so that robustness of the model is ensured.
Applications of Stepwise Regression in Python
Implementation of Stepwise Regression in Python
Dataset info- In order to estimate a 10-year risk of developing coronary heart disease (CHD), the Framingham Heart Study dataset, which is accessible on Kaggle, comprises medical information of over 4,240 people across 15 variables. Data characteristics cover a range of health factors important for predicting heart disease.
Dataset link: Framingham Heart Study Dataset.
Conclusion
With the help of this article, Stepwise Regression in Python has been introduced in detail and we have tried our best to include all the necessary details for working on stepwise regression. Therefore, first, we introduce the stepwise regression model and since it works on the basis of variable selection methods, we also discuss the importance of variable selection in machine learning. Next, we included the details of why we need variable selection in stepwise regression. We also discussed three different types of stepwise regression and model selection criteria of stepwise regression. Finally, we ended this discussion with the advantages and limitations of using a stepwise regression model.
If you like the article and would like to support me, make sure to:
- 👏 Like for this article and subscribe to our newsletter
- 📰 View more content on my DataSpoof website
- 🔔 Follow Me: LinkedIn| Youtube | Instagram | Twitter
