机器学习之线性回归

以kaggle上的House Prices: Advanced Regression Techniques为例讲述线性回归

from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV, ElasticNetCV

1、回归模型

（1）一般线性回归：Linear Regression without regularization

from sklearn.linear_model import LinearRegression

lr = LinearRegression()

lr.fit(X_train, y_train)

model = lr

rmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10))   #根均方误差

y_train_pred = lr.predict(X_train)

y_test_pred = lr.predict(X_test)

RMSE on Training set : 0.3887932081355326
RMSE on Test set : 0.42716116210823174

（2）岭回归：Linear Regression with Ridge regularization——L2正则化

采用正则化：处理共线性，滤除噪声，防止过拟合；

原理：引入额外的信息偏差，来惩罚极端参数权重；

from sklearn.linear_model import RidgeCV

ridge = RidgeCV(alphas = [0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6, 10, 30, 60])

ridge.fit(X_train, y_train)

alpha = ridge.alpha_

ridge = RidgeCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, 

                          alpha * .9, alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15,

                          alpha * 1.25, alpha * 1.3, alpha * 1.35, alpha * 1.4], 

                cv = 10)

ridge.fit(X_train, y_train)

alpha = ridge.alpha_

model = ridge

rmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10))   #根均方误差

y_train_rdg = ridge.predict(X_train)

y_test_rdg = ridge.predict(X_test)

Ridge RMSE on Training set : 0.11540572328450796
Ridge RMSE on Test set : 0.11642721377799556

（3）使用Lasson正则化的线性回归：Linear Regression with Lasso regularization——L1正则化

LASSO：以最小绝对收缩和选择算子为代表，简单地用权重的绝对值之和替换权重的平方。

与L2正则化相反，L1正则化产生稀疏特征向量，使得大多数特征权重为0。

如果高维数据集具有许多不相关的特征，则稀疏性会很有帮助。

from sklearn.linear_model import LassoCV

lasso = LassoCV(alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 

                          0.3, 0.6, 1], 

                max_iter = 50000, cv = 10)

lasso.fit(X_train, y_train)

alpha = lasso.alpha_

lasso = LassoCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, 

                          alpha * .85, alpha * .9, alpha * .95, alpha, alpha * 1.05, 

                          alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, alpha * 1.35, 

                          alpha * 1.4], 

                max_iter = 50000, cv = 10)

lasso.fit(X_train, y_train)

alpha = lasso.alpha_

model = lasso

rmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10))   #根均方误差

y_train_las = lasso.predict(X_train)
y_test_las = lasso.predict(X_test)

Lasso RMSE on Training set : 0.11411150837458062
Lasso RMSE on Test set : 0.11583213221750702

（4）使用ElsticNet正则化的线性回归：Linear Regression with ElasticNet regularization——L1和L2正则化

ElasticNet是Ridge和Lasso回归之间的折中，用L1产生稀疏性，用L2克服Lasso的一些限制（例如LASSO不能选择比观察更多的特征）

import sklearn.linear_model import ElasticNetCV

elasticNet = ElasticNetCV(l1_ratio = [0.1, 0.3, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1],

                          alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 

                                    0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], 

                          max_iter = 50000, cv = 10)

elasticNet.fit(X_train, y_train)

alpha = elasticNet.alpha_

ratio = elasticNet.l1_ratio_

elasticNet = ElasticNetCV(l1_ratio = [ratio * .85, ratio * .9, ratio * .95, ratio, ratio * 1.05, ratio * 1.1, ratio * 1.15],

                          alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], 

                          max_iter = 50000, cv = 10)

elasticNet.fit(X_train, y_train)

if (elasticNet.l1_ratio_ > 1):

    elasticNet.l1_ratio_ = 1    

alpha = elasticNet.alpha_

ratio = elasticNet.l1_ratio_



elasticNet = ElasticNetCV(l1_ratio = ratio,

                          alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, alpha * .9, 

                                    alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, 

                                    alpha * 1.35, alpha * 1.4], 

                          max_iter = 50000, cv = 10)

elasticNet.fit(X_train, y_train)

if (elasticNet.l1_ratio_ > 1):

    elasticNet.l1_ratio_ = 1    

alpha = elasticNet.alpha_

ratio = elasticNet.l1_ratio_


model = elasticNet

rmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10))   #根均方误差

y_train_ela = elasticNet.predict(X_train)

y_test_ela = elasticNet.predict(X_test)

ElasticNet RMSE on Training set : 0.11411150837458062
ElasticNet RMSE on Test set : 0.11583213221750702

此处ratio=1，即完全使用L1正则化，LASSO回归