根据前篇博文《神经网络之后向传播算法》,现在用java实现一个bp神经网络。矩阵运算采用jblas库,然后逐渐增加功能,支持并行计算,然后支持输入向量调整,最后支持L-BFGS学习算法。
上帝说,要有神经网络,于是,便有了一个神经网络。上帝还说,神经网络要有节点,权重,激活函数,输出函数,目标函数,然后也许还要有一个准确率函数,于是,神经网络完成了:
public class Net { List<DoubleMatrix> weights = new ArrayList<DoubleMatrix>(); List<DoubleMatrix> bs = new ArrayList<>(); List<ScalarDifferentiableFunction> activations = new ArrayList<>(); CostFunctionFactory costFunc; CostFunctionFactory accuracyFunc; int[] nodesNum; int layersNum; public Net(int[] nodesNum, ScalarDifferentiableFunction[] activations,CostFunctionFactory costFunc) { super(); this.initNet(nodesNum, activations); this.costFunc=costFunc; this.layersNum=nodesNum.length-1; } public Net(int[] nodesNum, ScalarDifferentiableFunction[] activations,CostFunctionFactory costFunc,CostFunctionFactory accuracyFunc) { this(nodesNum,activations,costFunc); this.accuracyFunc=accuracyFunc; } public void resetNet() { this.initNet(nodesNum, (ScalarDifferentiableFunction[]) activations.toArray()); } private void initNet(int[] nodesNum, ScalarDifferentiableFunction[] activations) { assert (nodesNum != null && activations != null && nodesNum.length == activations.length + 1 && nodesNum.length > 1); this.nodesNum = nodesNum; this.weights.clear(); this.bs.clear(); this.activations.clear(); for (int i = 0; i < nodesNum.length - 1; i++) { // 列数==输入;行数==输出。 int columns = nodesNum[i]; int rows = nodesNum[i + 1]; double r1 = Math.sqrt(6) / Math.sqrt(rows + columns + 1); //r1=0.001; // W DoubleMatrix weight = DoubleMatrix.rand(rows, columns).muli(2*r1).subi(r1); //weight=DoubleMatrix.ones(rows, columns); weights.add(weight); // b DoubleMatrix b = DoubleMatrix.zeros(rows, 1); bs.add(b); // activations this.activations.add(activations[i]); } } }
上帝造完了神经网络,去休息了。人说,我要使用神经网络,我要利用正向传播计算各层的结果,然后利用反向传播调整网络的状态,最后,我要让它能告诉我猎物在什么方向,花儿为什么这样香。
public class Propagation { Net net; public Propagation(Net net) { super(); this.net = net; } // 多个样本。 public ForwardResult forward(DoubleMatrix input) { ForwardResult result = new ForwardResult(); result.input = input; DoubleMatrix currentResult = input; int index = -1; for (DoubleMatrix weight : net.weights) { index++; DoubleMatrix b = net.bs.get(index); final ScalarDifferentiableFunction activation = net.activations .get(index); currentResult = weight.mmul(currentResult).addColumnVector(b); result.netResult.add(currentResult); // 乘以导数 DoubleMatrix derivative = MatrixUtil.applyNewElements( new ScalarFunction() { @Override public double valueAt(double x) { return activation.derivativeAt(x); } }, currentResult); currentResult = MatrixUtil.applyNewElements(activation, currentResult); result.finalResult.add(currentResult); result.derivativeResult.add(derivative); } result.netResult=null;// 不再需要。 return result; } // 多个样本梯度平均值。 public BackwardResult backward(DoubleMatrix target, ForwardResult forwardResult) { BackwardResult result = new BackwardResult(); DoubleMatrix cost = DoubleMatrix.zeros(1,target.columns); DoubleMatrix output = forwardResult.finalResult .get(forwardResult.finalResult.size() - 1); DoubleMatrix outputDelta = DoubleMatrix.zeros(output.rows, output.columns); DoubleMatrix outputDerivative = forwardResult.derivativeResult .get(forwardResult.derivativeResult.size() - 1); DoubleMatrix accuracy = null; if (net.accuracyFunc != null) { accuracy = DoubleMatrix.zeros(1,target.columns); } for (int i = 0; i < target.columns; i++) { CostFunction costFunc = net.costFunc.create(target.getColumn(i) .toArray()); cost.put(i, costFunc.valueAt(output.getColumn(i).toArray())); // System.out.println(i); DoubleMatrix column1 = new DoubleMatrix( costFunc.derivativeAt(output.getColumn(i).toArray())); DoubleMatrix column2 = outputDerivative.getColumn(i); outputDelta.putColumn(i, column1.muli(column2)); if (net.accuracyFunc != null) { CostFunction accuracyFunc = net.accuracyFunc.create(target .getColumn(i).toArray()); accuracy.put(i, accuracyFunc.valueAt(output.getColumn(i).toArray())); } } result.deltas.add(outputDelta); result.cost = cost; result.accuracy = accuracy; for (int i = net.layersNum - 1; i >= 0; i--) { DoubleMatrix pdelta = result.deltas.get(result.deltas.size() - 1); // 梯度计算,取所有样本平均 DoubleMatrix layerInput = i == 0 ? forwardResult.input : forwardResult.finalResult.get(i - 1); DoubleMatrix gradient = pdelta.mmul(layerInput.transpose()).div( target.columns); result.gradients.add(gradient); // 偏置梯度 result.biasGradients.add(pdelta.rowMeans()); // 计算前一层delta,若i=0,delta为输入层误差,即input调整梯度,不作平均处理。 DoubleMatrix delta = net.weights.get(i).transpose().mmul(pdelta); if (i > 0) delta = delta.muli(forwardResult.derivativeResult.get(i - 1)); result.deltas.add(delta); } Collections.reverse(result.gradients); Collections.reverse(result.biasGradients); //其它的delta都不需要。 DoubleMatrix inputDeltas=result.deltas.get(result.deltas.size()-1); result.deltas.clear(); result.deltas.add(inputDeltas); return result; } public Net getNet() { return net; } }
上面是一次正向/反向传播的具体代码。训练方式为批量训练,即所有样本一起训练。然而我们可以传入只有一列的input/target样本实现adapt方式的串行训练,也可以把样本分成很多批传入实现mini-batch方式的训练,这,不是Propagation要考虑的事情,它只是忠实的把传入的数据正向过一遍,反向过一遍,然后把过后的数据原封不动的返回给你。至于传入什么,以及结果怎么运用,是Trainer和Learner要做的事情。下回分解。