样本
[x_i=(x_{i1};x_{i2}; ...; x_{im}) \, {函数值 y_i}
]
每个样本有m个变量
回归面
[f(x_i) = x_i^T omega +b
]
(omega = (omega_1; omega_2; ...; omega_m))
[hat x_i=(x_{i1};x_{i2}; ...; x_{im}; 1)
]
[hat omega = (omega_1; omega_2; ...; omega_m; b)
]
则
[f(x_i) = hat x_i^T hat omega
]
假设有n个样本令
[X = [hat x_1^T; hat x_2^T; ..., hat x_n^T]
]
则均方误差为
[frac 1n (X hat omega - Y)^T (X hat omega - Y)
]
其中(Y = (y_1; y_2; ...; y_n))
损失函数
[J(hat omega^*) = (X hat omega^* - Y)^T (Xhat omega^* - Y)
]
令
[left. frac{{
m d} J(hat omega^*)}{{
m d} hat omega^*} = 0
ight.
]
[hat omega^* = (X^TX)^{-1}X^TY
]
当样本变量较多，样本数量不足时(m>n), (hat omega^*)解不唯一
L2正则化
引入对于(hat omega^*)的L2正则化项
[hat J(hat omega^*) = (X hat omega^* - Y)^T (Xhat omega^* - Y) + frac{lambda}{2} ||hat omega^*||_2^2
]
可以发现，正则化项为(hat omega^*)二范数平方，乘以(frac{lambda}{2})
(lambda)用于控制正则化项和均方误差的权重
[left. frac{{
m d} hat J(hat omega^*)}{{
m d} hat omega^*} = 0
ight.
]
[hat omega^* = (X^TX + frac{lambda}{2}I)^{-1}X^TY
]
python程序
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
#随机生成两个变量的N个样本
N = 50
feature1 = np.random.rand(N)*10
feature2 = np.random.rand(N)*10
splt = np.ones((1, N))
#
temp_X1 = np.row_stack((feature1, feature2))
temp_X = np.vstack((temp_X1, splt))
X_t = np.mat(temp_X)
X = X_t.T
temp_Y = np.random.rand(N)*10
Y_t = np.mat(temp_Y)
Y = Y_t.T
#画样本散点图
fig = plt.figure()
ax1 = Axes3D(fig)
ax1.scatter(feature1, feature2, temp_Y)
#求Omega
Lbd = 0.01 #确定lambda
I_11 = np.eye(2)
I_12 = np.zeros((2, 1))
I_2 = np.zeros((1, 3))
I_t1 = np.hstack((I_11, I_12))
I_t = np.vstack((I_t1, I_2))
I = np.mat(I_t)
Omega = (X.T*X + Lbd/2*I).I*X.T*Y
#画分回归面
xx = np.linspace(0,10, num=50)
yy = np.linspace(0,10, num=50)
xx_1, yy_1 = np.meshgrid(xx, yy)
Omega_h = np.array(Omega.T)
zz_1 = Omega_h[0, 0]*xx_1 + Omega_h[0, 1]*yy_1 + Omega_h[0, 2]
ax1.plot_surface(xx_1, yy_1, zz_1, alpha= 0.6, color= "r")
plt.show()