Normal equation: Method to solve for θ analytically
正规方程：分析求解θ的方法
对于损失函数
[Jleft( {{ heta _0},{ heta _1},...,{ heta _n}} ight) = frac{1}{{2m}}sumlimits_{i = 1}^m {{{left( {{h_ heta }left( {{x^{left( i ight)}}} ight) - {y^{left( i ight)}}} ight)}^2}} ]
只要满足
[frac{partial }{{partial { heta _1}}}Jleft( heta ight) = frac{partial }{{partial { heta _2}}}Jleft( heta ight) = cdot cdot cdot = frac{partial }{{partial { heta _n}}}Jleft( heta ight) = 0]
就可以直接得到所有的参数
[{{ heta _0},{ heta _1},...,{ heta _n}}]
而满足上面的连续等式的解是
[ heta = {left( {{X^T}X} ight)^{ - 1}}{X^T}y]
其中
[X = left[ {egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{1,x_1^{left( 1
ight)},x_2^{left( 1
ight)},...,x_n^{left( 1
ight)}}\
{1,x_1^{left( 2
ight)},x_2^{left( 2
ight)},...,x_n^{left( 2
ight)}}\
egin{array}{l}
cdot \
cdot
end{array}
end{array}}\
{1,x_1^{left( m
ight)},x_2^{left( m
ight)},...,x_n^{left( m
ight)}}
end{array}}
ight]]
是变量的矩阵；
[y = left[ {egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{{y^{left( 1
ight)}}}\
{{y^{left( 2
ight)}}}\
egin{array}{l}
cdot \
cdot
end{array}
end{array}}\
{{y^{left( m
ight)}}}
end{array}}
ight]]
是对应的输出值
| Gradient Descent | Normal Equation |
| Need to choose α | No need to choose α |
| Needs many iterations | Don't need to iterate |
| Works well even when n is large | O(n3)Need to compute () |
| O(kn2) | Slow if n is very large |
如果矩阵不可逆，可以计算伪逆矩阵。