# Ensembles

This category groups posts on ensemble models. Each post focuses on a specific algorithm. The emphasis on understanding how these models actually work at a technical level. Here you will learn to build ensemble models in Python from scratch.

## Implement Gradient Boosting Regression in Python from Scratch

Implement Gradient Boosting Regression in Python from Scratch Motivation for Gradient Boosting Regression in Python In the previous post, we covered how Gradient Boosting works, and outlined the general algorithm for this ensemble technique. Gradient Boosting was initially developed by Friedman 2001, and the general algorithm is referred to as Algorithm 1: Gradient_Boost, in that paper. Furthermore, …

## Understanding the Gradient Boosting Regressor Algorithm

Understanding the Gradient Boosting Regressor Algorithm Motivation: Why Gradient Boosting Regressors? The Gradient Boosting Regressor is another variant of the boosting ensemble technique that was introduced in a previous article. Development of gradient boosting followed that of Adaboost. In an effort to explain how Adaboost works, it was noted that the boosting procedure can be …

## A Complete Introduction to Cross Validation in Machine Learning

A Complete Introduction to Cross Validation in Machine Learning What is Cross Validation? A natural question to ask, when building any predictive model, is how good are the predictions? Having a clear, quantitative measure for the expected model performance, is a key element to any machine learning project.   Cross validation is a family of …

## Understanding the Adaboost Regression Algorithm

Understanding the Adaboost Regression Algorithm Motivation: What is a Adaboost Regressor? In this post, we will describe the Adaboost regression algorithm. We will start with the basic assumptions and mathematical foundations of this algorithm, and work straight through to an implementation in Python from scratch. Adaboost stands for “Adaptive Boosting”, and this was the first boosting technique …

## Understanding the Adaboost Classification Algorithm

Understanding the Adaboost Classification Algorithm Motivation: What is a Adaboost Classifier? In this post, we will describe the Adaboost classification algorithm. We will start with the basic assumptions and mathematical foundations of this algorithm, and work straight through to an implementation in Python from scratch. Adaboost stands for “Adaptive Boosting”, and this was the first boosting technique …

## Introduction to Simple Boosting Classification in Python

Introduction to Simple Boosting Classification in Python Motivation for Boosting Classification This post will consist of a simple introduction to boosting classification in Python. In my previous article, I introduced boosting with a basic regression algorithm. Here, we will adapt that procedure to handle classification problems. Boosting is a popular ensemble technique, and forms the …

## Introduction to Simple Boosting Regression in Python

Introduction to Simple Boosting Regression in Python Motivation: Why Boosting? This post will consist of an introduction to simple boosting regression in Python. Boosting is a popular ensemble technique, and forms the basis to many of the most effective machine learning algorithms used in industry. For example, the XGBoost package routinely produces superior results in competitions …

## Build a Random Forest in Python from Scratch

Build a Random Forest in Python from Scratch Motivation to Build a Random Forest Model For this post, we will Build a Random Forest in Python from scratch. I will include examples in classification and regression. Bagging ensembles are an approach to reduce variance, and thereby increase model performance. In this algorithm, multiple weak learner models produce predictions …

## Build a Bagging Classifier in Python from Scratch

Build a Bagging Classifier in Python from Scratch Motivation to Build a Bagging Classifier In this article, we will build a bagging classifier in Python from the ground-up. Through this exercise it is hoped that you will gain a deep intuition for how bagging works.  We saw in a previous post that the bootstrap method was …