反向传播的代码实现
已经理解了反向传播的理论，现在可以理解上一章用来实现反向传播的代码。回顾上一章中Network类里update_mini_batch和backprop方法里的代码。这些方法的代码是直接对上面所描述算法的直译。实际上update_mini_batch方法通过计算当前mini_batch的训练样本，更新了Network的权重和偏移量：
class Network(object):
def update_mini_batch(self, mini_batch, eta):
"""Update the network's weights and biases by applying
gradient descent using backpropagation to a single mini batch.
The "mini_batch" is a list of tuples "(x, y)", and "eta"
is the learning rate."""
nabla_b = [np.zeros(b.shape) for b in self.biases]
nabla_w = [np.zeros(w.shape) for w in self.weights]
for x, y in mini_batch:
delta_nabla_b, delta_nabla_w = self.backprop(x, y)
nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
self.weights = [w-(eta/len(mini_batch))*nw
for w, nw in zip(self.weights, nabla_w)]
self.biases = [b-(eta/len(mini_batch))*nb
for b, nb in zip(self.biases, nabla_b)]
大部分工作是通过这行代码 delta_nabla_b, delta_nabla_w = self.backprop(x, y) 通过调用 backprop 方法算出偏导数和。这里的backprop 方法与上一节的算法很接近。只有一点小小的改变——用了些许不同的方式来索引网络层。这个改变是用了Python的特性，就是使用链表的负索引来从链表的尾部反向计数，如l[-3]，表示链表l的倒数第3个元素。下面是backprop的代码和几个辅助函数，用来计算函数，导数和成本函数的导数。通过这些说明你应该能够自己理解这些代码。如果遇到一些障碍的话，你可以从代码的原生注释（和完整清单）获得帮助。
class Network(object):
...
def backprop(self, x, y):
"""Return a tuple "(nabla_b, nabla_w)" representing the
gradient for the cost function C_x. "nabla_b" and
"nabla_w" are layer-by-layer lists of numpy arrays, similar
to "self.biases" and "self.weights"."""
nabla_b = [np.zeros(b.shape) for b in self.biases]
nabla_w = [np.zeros(w.shape) for w in self.weights]
# feedforward
activation = x
activations = [x] # list to store all the activations, layer by layer
zs = [] # list to store all the z vectors, layer by layer
for b, w in zip(self.biases, self.weights):
z = np.dot(w, activation)+b
zs.append(z)
activation = sigmoid(z)
activations.append(activation)
# backward pass
delta = self.cost_derivative(activations[-1], y) * \
sigmoid_prime(zs[-1])
nabla_b[-1] = delta
nabla_w[-1] = np.dot(delta, activations[-2].transpose())
# Note that the variable l in the loop below is used a little
# differently to the notation in Chapter 2 of the book. Here,
# l = 1 means the last layer of neurons, l = 2 is the
# second-last layer, and so on. It's a renumbering of the
# scheme in the book, used here to take advantage of the fact
# that Python can use negative indices in lists.
for l in xrange(2, self.num_layers):
z = zs[-l]
sp = sigmoid_prime(z)
delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
nabla_b[-l] = delta
nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
return (nabla_b, nabla_w)
...
def cost_derivative(self, output_activations, y):
"""Return the vector of partial derivatives \partial C_x /
\partial a for the output activations."""
return (output_activations-y)
def sigmoid(z):
"""The sigmoid function."""
return 1.0/(1.0+np.exp(-z))
def sigmoid_prime(z):
"""Derivative of the sigmoid function."""
return sigmoid(z)*(1-sigmoid(z))
问题
- 在小批次里用完全基于矩阵的方式进行反向传播
我们实现的随机梯度下降算法在一个小批次里遍历训练样本。可以改进反向传播算法让其同时计算小批次里所有的训练样本的梯度。方案是这样的，开始的时候不再用一个向量的输入，而是替换成一个矩阵，它的列是小批次里的所有向量。前向转播时，乘以一个权重矩阵，加上一个合适的偏移量矩阵，然后再每处运用S型函数。用类似的方式进行反向传播。写出用这种方式实现的反向传播的伪代码。修改network.py让基使用这种完全基于矩阵的方式。这种方式的优势在于可以完全利用现代线性代数库的优势。结果会比遍历小批次快很多。（举例来说，在我的笔记本里，当跑像上一章我们讨论的MNIST分类问题时，要快两个数量级）。实际上，所有正式的反向传播库都使用基于矩阵的方式，或一些变种。