Neural Network and DeepLearning (1.2)使用神经网络识别手写数字

1.Learning with gradient descent（使用梯度下降法进行学习）

cost function（代价函数）

训练神经网络的目的是找到能最小化二次代价函数C(w,b)的权重和偏置。梯度下降：

when we move the ball a small amount Δv₁ in the v₁ direction, and a small amount Δv₂ in the v₂ direction. Calculus tells us that C changes as follows:

defined the gradient of C to be the vector of partial derivatives:

the expression for ΔC can be rewritten as:

choose Δv as:

then:

this guarantees that ΔC≤0, C will always decrease, never increase.

hat is, we'll use Equation (10) to compute a value for Δv, then move the ball's position v by that amount:

Then we'll use this update rule again, to make another move. If we keep doing this, over and over, we'll keep decreasing C until - we hope - we reach a global minimum.

stochastic gradient descent（随机梯度下降）:

works by picking out a randomly chosen mini-batch of training inputs, and training with those:

随机记梯度下降通过随机的选取并训练输入的小批量数据来工作，其中两个求和符号是在当前小批量数据中的所有训练样本X_j,上进行的。

然后我们再挑选另一随机选定的小批量数据去训练。直到我们用完了所有的训练输入，这被称为完成了一个训练迭代期（epoch）。然后我们就会开始一个新的训练迭代期。

备：我们不能提前知道训练数据量的情况下，舍弃1/n和1/m是有效的。

　　小批量数据的大小设置为1时，成为online学习，或递增学习