Yesterday we talked about a Decision Tree Classifier making a prediction on the Iris dataset, now let’s see how an ensemble of decision trees can help each other making a more accurate prediction.
The main fact about ensemble methods is that each model is trained on a different subset of the training set. If the sampling is performed with replacement then the method is called bagging, otherwise it’s called pasting.
The Random Forest Classifier is a bagging classifier. Once all are trained, the predictions are aggregated to make a final prediction which is generally more precise.
Now let’s see the code:
from sklearn import datasets
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
X = pd.DataFrame(iris['data'], columns=['sepal length', 'sepal width', 'petal length', 'petal width'])
y = np.array(iris['target'])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=123)
f_clf = RandomForestClassifier()
param_grid = {'max_depth': [2, 3, 4], 'min_weight_fraction_leaf': np.linspace(0.1, 0.5, 5), 'n_estimators': [100]}
grid = GridSearchCV(estimator=f_clf, param_grid=param_grid, cv=3)
grid.fit(X_train, y_train)
best = grid.best_estimator_
y_pred = best.predict(X_test)
print(accuracy_score(y_pred, y_test))
print(grid.best_params_)
The code is similar to the one used in my previous article: I load the set, split it in train and test sets, create the Random Forest Classifier.
Now, because I want it to make a very accurate prediction, I use the GridSearch model to test a lot of hyperparameter combinations and find the best one. Notice the parameter cv=3, which means cross validation: the set is divided into 3 parts and, for each of these, the model is trained on the other two and tested on it. This method is widely used to test models.
Now let’s take a look at the Random Forest hyperparameters:
max_depth is the maximum number of ramifications
min_weight_fraction_leaf is the minimal ratio of samples in a leaf and total samples that needs to occur to create a leaf
n_estimators is the number of Decision Tree Classifiers in the ensemble
Turns out the best combination is:
{'max_depth': 2, 'min_weight_fraction_leaf': 0.2, 'n_estimators': 100}
Which, once trained, gives an accuracy score of:
0.9777777777777778
So there was an improvement to the previous result of the Decision Tree.