sklearn模型调优中的learning curve 和 validation curve

1. learning_curve()：主要是用来判断模型是否过拟合
2. validation_curve()：这个函数主要是用来查看不同参数的取值下模型的准确性

以下是Python机器学习书里面的例子, 我改了部分参数

▌learning_curve

import matplotlib.pyplot as plt

from sklearn.model_selection import learning_curve
from sklearn.decomposition import PCA
from sklearn.svm import SVC


pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca',PCA()),
                    ('svc',SVC(kernel='rbf')),
#                     ('clf', LogisticRegression(penalty='l2', random_state=0,solver='lbfgs')),
                    ])

train_sizes, train_scores, test_scores =\
                learning_curve(estimator=pipe_lr,
                               X=X_train,
                               y=y_train,
                               train_sizes=np.linspace(0.1, 1.0, 10),
                               cv=10,
                               n_jobs=1)

train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(train_sizes, train_mean,
         color='blue', marker='o',
         markersize=5, label='training accuracy')

plt.fill_between(train_sizes,
                 train_mean + train_std,
                 train_mean - train_std,
                 alpha=0.15, color='blue')

plt.plot(train_sizes, test_mean,
         color='green', linestyle='--',
         marker='s', markersize=5,
         label='validation accuracy')

plt.fill_between(train_sizes,
                 test_mean + test_std,
                 test_mean - test_std,
                 alpha=0.15, color='green')

plt.grid()
plt.xlabel('Number of training samples')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')
plt.ylim([0.8, 1.0])
plt.tight_layout()
plt.savefig('learning_curve.png', dpi=300)
plt.show()

▌从下图可以看出，蓝色的training曲线部分的准确率明显是要高于绿色的testing曲线，这说明有过度拟合的情况，其中一个办法是通过增加数据集来解决。

▌validation_curve

from sklearn.model_selection import validation_curve



param_range = ['linear','sigmoid','poly','rbf']
train_scores, test_scores = validation_curve(
                estimator=pipe_lr,
                X=X_train,
                y=y_train,
                param_name='svc__kernel',
                param_range=param_range)

train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(param_range, train_mean,
         color='blue', marker='o',
         markersize=5, label='training accuracy')

plt.fill_between(param_range, train_mean + train_std,
                 train_mean - train_std, alpha=0.15,
                 color='blue')

plt.plot(param_range, test_mean,
         color='green', linestyle='--',
         marker='s', markersize=5,
         label='validation accuracy')

plt.fill_between(param_range,
                 test_mean + test_std,
                 test_mean - test_std,
                 alpha=0.15, color='green')

plt.grid()
plt.xscale('log')
plt.legend(loc='lower right')
plt.xlabel('Parameter C')
plt.ylabel('Accuracy')
plt.ylim([0.8, 1.0])
plt.tight_layout()
plt.savefig('validation_curve.png', dpi=300)
plt.show()

▌在上面的代码中，我将param_range 设为 ['linear','sigmoid','poly','rbf']，这主要是测试在不同的kernel中，模型的准确性有什么的不同。 有一点需要注意的是：因为前面我们使用pineline， 所以后面赋予参数的时候param_name='svc__kernel' 的param_name后面需要紧跟对应的svc，并且指明是svc下的kernel参数, 两者间用两条下划线__。