Journal Archive

Howard et al. [8] developed a CNN model capable of deploying in low-memory devices called Mobilenet. This model has two core layers: Depthwise and pointwise convolution.

In standard convolution, current feature maps are formed by a multiplication and summation between previous feature maps and current filters. However, many filters lead to a huge number of learnable weights. To reduce the number of weights, Mobilenet split this convolution into two types of convolutions: First, a depthwise convolution applies a single convolutional filter for each channel in the feature maps. Mathematically, we can formulate depthwise convolution as

where K^ is the depthwise convolutional filter of size D_k × D_k × M. The m^th filter in K^ is applied to the m^th channel in F.

Then, a pointwise convolution (or 1 × 1 convolution) is applied to aggregate information in the feature maps channel-wise. A 3 × 3 depthwise convolutional filter in Mobilenet reduces 8 to 9 times computation than standard convolution.

In experiment, Howard et al. [8] showed that Mobilenet got a high accuracy on fine-grained object classification. So Mobilenet is sufficient for the weeds classification problem.

Typically, CNN architecture models have “stacked” layers: One layer is placed on top of each layer. This type of deep architecture results in vanishing or exploding gradient flows from the bottom to the top layers, thus preventing the model from converging. He et al. [9] presented a residual learning framework to conveniently train an extremely deep model. Mathematically, a building block of the Resnet model is formulated as

where x and y are the input and output feature maps of residual block, F is the residual mapping. In practice, this function is a stack of convolutional layers, each layers contain filters, and each i-th filter has a weights matrix W_i. The summation operation is performed by an element-wise addition of the input feature maps x to the stacked layers and the output feature maps of these stacked layers.

Resnet contains many building blocks. As explained in [10], implementing Eq. (2) recursively, we have

for any shallow layer l. Assign Eq. (5) as x_L and denote the loss function as ϵ, the gradient of ϵ with respect to the feature map x_L is

δϵ/δx_L is unlikely to be zero, so the gradient does not get vanish. With this building blocks, Resnet contains many deep layers and avoid making the gradient vanish or explode.

Glorot and Bengio [6] analyzed layers in an ANN model by estimating the distribution of activation values (output of symmetry, nonlinear activation function) and gradients. They initialized weights followed by a random distribution and used the standard stochastic gradient descent (SGD) to optimize weighs in this model. They found that weights distribution in deeper layers saturate toward 0. With the sigmoid activation function, this behavior prevents the gradients from flowing backward so that deeper layers may not learn valuable features. This phenomenon also occurs for softsign and hyperbolic tangent but is less extreme for sigmoid. In common, SGD from random initialization is inefficient for a deep neural network.

To keep the information flow from saturating, they proposed a new initialization method to preserve the variance of output activation values between layers.

where zⁱ,z^i＇ is the activation outputs of layer i and i^＇ respectively. Finally Eq. (8) led to

where x was the network input.

Assume we use a symmetry, zero-centered activation function with unit derivative at 0. Using 1^st order Taylor function, we can approximate as a linear function as Eq. (10)

In common, at layer i, ANN models calculate a linear combination sⁱ between weights Wⁱ and output of the activation function zⁱ, and the bias term bⁱ is added. That is, sⁱ = zⁱWⁱ+bⁱ, and zⁱ⁺¹f(sⁱ). Assume weights are initialized independently. To obtain similar variance, we calculate

Initializing the bias term as zero lead to Var[b^i-1] = 0, we have

where is the number of nodes in layer. Assume we initialize weights by using a zero-mean distribution function, then

Eq. (19) is a recursive formular. Expanding to all layers, we have

Setting n_i^＇ Var[W^i＇] = 1 satisfy Eq. (8).

We also want similar gradient variance across layers, means that

where is a cost function of the ANN model. Using chain rule, we have

Eq. (24) is also having a recursive form. Expanding to all layers, we have

Setting n_i^＇+1 Var[W^i＇] = 1 satisfy Eq. (21). In total, for all, we have the following system of equation.

Eq. (27) show that the variance of weights in layer i depend on the number of nodes in that layer and the next layer. Assume we initialize weights followed by a uniform distribution, Wⁱ ~ U(-a,a), then

Glorot's method is based on the assumption of using a zero-centered symmetric function as the activation function. However, this function is unsuitable for deep neural networks because of the gradient vanish problem during backpropagation [5]. Many popular CNN models such as VGG [11], Resnet [9], Mobilenet [8] used rectifier nonlinear activation functions because this function accelerates the training convergence [12] and more generalizes the model than symmetry function [13]. He et al. [7] investigated the variance condition of the output of rectifier nonlinear activation function at layer in a stacked CNN model, based on the procedure of Glorot’s method.

where x is k²c × 1 vector, c is the number of channels and k is the filter size. W is a d × n weight matrix containing d filters, n is k²c is the number of connections. Because we stack layers, the number of channels in current layer are equal to the number of filters used in the previous layer, c_l = d_l-1.

At the beginning, we set the bias term b_l = 0 and x_l = f(y_l-1). Assume we initialize weights W_l followed by a zero-mean distribution (E(W_l) = 0), then

Using rectified linear unit function, we have x = f(y) = max(0,y), so

Since y² is a symmetric function around 0 and the input y is zero-mean, we have

So we have

Expand recursive Eq. (43) to all L layers, we have

Similar to Glorot’s method, He’s method maintains variance in the activation output across all layers by setting a condition for all layers,

Denote Δx = δϵ/δx, Δy = δϵ/δy, where Δx is a c × 1vector, represents gradient at a pixel of this layer, Δy represents k × k pixels in d channelks, and ϵ is an objective function. We have

where Wl^ is a c×n^ matrix, and n^=k2d. Assume that W_l is initialized by a symmetric distribution around zero, then E[Δx_l] for all l. By using ReLU as the activation function, f^'(y_l) = 1 or 0 with equal probability, so

So, calculate variance in Eq. (47), we have

Put L layers together, we have

To maintain gradient variance in backpropagation, we set 0.5nl^Varwl=1 for all layers l. He et al. [7] stated that using either n_l or nl^ were sufficient for training the model. This condition aims to keep the gradient from being exponentially large or small.

He’s method show that the weights in layer are initialized with a variance that depend only on the filter size in that layer. If we initialize W_l~U(-a,a), then

Mathematically, a filter of size n_l applied to layer l has a larger variance using He`s method than that of Glorot. By considering Eq. (28) and Eq. (54), Glorot requires an additional filter of size n_l+1 in the next layer l+1, which increases the denominator value, hence smaller variance than He.

We initialized weights using a zero-mean Uniform distribution with variance followed by Glorot and He's methods. Figures (a) and (b) in Figure 3 show the box plot of values in filters, output of activation layers, and gradients between layers. It turns out that these values, using He's method, had a higher variance than Glorot‘s method because mathematically, He's method only needs the filter size in the current layer, while Glorot's method requires the current and next layer. Both Mobilenet and Resnet have large filter sizes at deeper layers, thus reducing its variance then. In particular, variances of the output of activation layers in Mobilenet decreased faster than those in Resnet, starting from the depthwise and pointwise convolution layers. The large filter size configuration caused this issue in the earlier layers in Mobilenet.

Fig. 3. 
Box plot of values in filters, output of activation layers, and gradients across layers, using Glorot and He’s methods. Left column: Mobilenet. Right column: Resnet

Fig. 4. 
Training and validation curve on Mobilenet. (a) Accuracy, Glorot. (b) Accuracy, He. (c) Loss, Glorot. (d) Loss, He.

Fig. 5. 
Training and validation curve on Resnet. (a) Accuracy, Glorot. (b) Accuracy, He. (c) Loss, Glorot. (d) Loss, He.

The first two rows of Figure 6 show the Mobilenet output activation values distribution, using Glorot and He's methods. The first layer in this model is a standard convolution. The left-most figures in 2 rows show that the output of this layer had a higher variance using He’s method than Glorot’s method. The last two figures of Figure 6 and figure (c) in Figure 3 show that, on average, using both initialization methods brought weights approaching zero. Both methods had a small variance when we went to deeper layers, mainly affected by two reasons: small input values after normalization ([-1,1]) and small variance of the zero-mean depthwise and pointwise convolution layers.

Fig. 6. 
Distribution of output of activation function. From top to down, 1^st and 2^nd row use Glorot and He on Mobilenet, 3^rd and 4^th row use Glorot and He on Resnet. From left to right is the result from earliest to lowest layers.

The last two rows of Figure 6 show the distribution of Resnet output activations. Like Mobilenet using Glorot's method, the variance decreased as we went to deeper layers, as shown in figure (d) of Figure 3. In contrast, using He's method increased the variance rapidly. In the last layer, many weights are larger than 10⁵, resulting in the maximum value of softmax activation function saturated to 1. Thus, the gradient of all weights (after applying chain rule) was approximate zero. Those large weight values in deeper layers came from 2 reasons: For one layer, the distribution variance using He's method is larger than using Glorot‘s method, and the skip connection by a summation in Resnet architecture increases the output values in deeper layers.

The distribution of gradients in Mobilenet is shown in the first two rows in Figure 7, using Glorot and He’s methods. Figure (e) in Figure 3 in both methods shows a lower variance in gradient in deeper layers for Glorot‘s method. In contrast, gradients have large values in deep layers for He’s method, except for the values in pointwise convolution layers.

Fig. 7. 
Distribution of output of gradient. From top to down, 1st and 2nd row use Glorot and He on Mobilenet, 3rd and 4th row use Glorot and He on Resnet. From left to right is the result from earliest to lowest layers.

The last two rows in Figure 7 show the distributions of gradient values in the Resnet model, using Glorot and He's methods. It shows that using Glorot‘s method, many weights had non-zero gradients. Except for the first layer, all layers had similar output variance, as shown in figure (f) of Figure 3. Using He's method to initialize weights in the Resnet model, all the gradients across layers were very close to zero. As explained above, large variance in He's method and shortcut connection led to enormous value in deeper layers. Therefore, max(s) in Eq. (55) (softmax function, K is the number of classes) was approximately 1, and other elements in the softmax output vector were close to 0. In this case, the partial derivative in Eq. (56), where δ_ij is the Kronecker delta function, was approximately zero with respect to any z_j.

Figure 4 shows performances of training the Mobilenet, where the weights were initialized using the Glorot and He's methods. It indicates that Glorot‘s method helped the model to converge faster than using He. By using Glorot's method, the model increased in accuracy quickly, faster than He‘s method. In both methods, the model converged at the 30th epoch.

Figure 5 shows performances of training the Resnet. Like what was in Mobilenet, Glorot‘s method helped the model converge a bit faster, starting in the 10th epoch. The model converged at 30th epoch in both methods.

Table 1 shows performances of models using Glorot and He’s methods as initialization methods. In particular, Glorot‘s method showed the ability to generalize the model better than He‘s method when all metrics using Glorot‘s method were 0.01 points higher than He‘s method, except for the loss value showing that Glorot‘s method was lower than He‘s method by nearly 0.03 points. Besides, Resnet had higher performance than Mobilenet, regardless of the initialization method used.

Experiences have shown that, for a model, using Glorot's method to initialize weights makes the model converge faster and more general than He's method. The first few epochs in Figure 4 and Figure 5 indicate that, for the CNU weeds dataset, Glorot's method to initialize the weights placed these weights in appropriate positions on the loss surface in both models. This result is in contrast to [9], where Glorot's method (based on symmetric activation functions) is more suitable for training a weeds classification model than using He's method (based on rectifier nonlinear activation functions). Remind that both models use ReLU as the activation function. This problem occurs when the architectures of both models are not deep enough to show the advantage of He‘s method over Glorot‘s method.

Glorot and He's methods is founded based on the assumption of stacking standard convolution layers such as VGG or conventional ANN model. However, both model architectures have its characteristic that may not fit that assumption. As shown in Figure 6, pointwise and depthwise convolution in Mobilenet reduces the variance of the output of activation function, resulting in the output in deeper layers close to zero. Shortcut connection in Resnet enlarged variance when using He's method, but not in Glorot’s method because the variance in Glorot’s method is smaller than He’s method. This means that we might deal with large weights initialization, resulting in saturation in Glorot’s method is smaller than He’s method.