ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance

 ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance

Here is presented the following paper, published on  arXiv:


Despite the unresolved ‘dying ReLU problem’, the classical ReLU activation function (AF) has been extensively applied in Deep Neural Networks (DNN), in particular Convolutional Neural Networks (CNN), for image classification. The common gradient issues of ReLU pose challenges in applications on academy and industry sectors. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the Absolute Leaky ReLU (ALReLU) AF, a variation of LReLU, is proposed, as an alternative method to resolve the common ’dying ReLU’ on NN-based algorithms for supervised learning. The experimental results demonstrate that by using the absolute values of LReLU’s small negative gradient, has a significant improvement in comparison with LReLU and ReLU, on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets.


Although recent developments of AFs for Shallow and Deep Learning Neural Networks (NN), such as the QReLU/m-QReLU (Parisi et al., 2020a) and m-arcsinh (Parisi et al., 2020b), the repeatable and reproducible functions have remained very limited and confined to three activation functions regarded as ’gold standard’ (Parisi et al., 2020b). The sigmoid and tanh are well-known for their common vanishing gradient issues and only ReLU function seems to be more accurate and scalable for deep neural networks, despite its ’dying ReLU’ problem, which has recently appears to been solved by (Parisi et al., 2020a). Earlier, many other methods are proposed in order to fix these known problems. Some of them are multiple variations of the ReLU AF, such as the Leaky ReLU (LReLU), the Parametric ReLU (PReLU), the Randomised ReLU (RReLU) and the Concatenated ReLU (CReLU). Furthermore, Leaky ReLU (LReLU) introduced (Maas et al. 2013) by providing a small negative gradient for negative inputs into a ReLU function, instead of being 0. A constant variable 𝛼, with a default value of 0.01, was used to compute the output for negative inputs. Using this modification, LReLU can lead to small improvements in classification performance when compared to the ReLU AF. But the most of the above mentioned AFs are tend to have lack of robustness with classification tasks of varying degrees of complexity, e.g., slow or lack of convergence (Vert and Vert, 2006) (Jacot et al., 2018), caused by trapping at local minima (Parisi et al., 2020b). Amongst the mentioned AFs, only the ReLU is really applicable on NNs, with its novel quantum variations (QReLU and m-QReLU) found more scalable than its traditional version only recently (Parisi et al., 2020a).

In this work, a new approach of LReLU is proposed in order to solve the common gradient vanishing and ’dying ReLU’ problems, by using the absolute values of the small negative gradient used on LReLU. This method leads in a significance improvement on training and classification procedures as it is concluded from the results of the numerical evaluation performed. Quantitative evaluation metrics such as accuracy, AUC, recall, precision and F1-score have been computed to reveal the performance of the proposed technique and to provide the necessary criteria for a reliable objective evaluation of the method.

The outline of this paper is as follows: Section 2 contains one of the main contributions of this work, which includes the implementation of ALReLU in Keras. Section 3 presents experimental results of the proposed AF, including an evaluation of the accuracy of the training. Also it is compared to other well defined AFs in the field. Finally, discussion and the main conclusions of the work are devoted on Section 4.

2. Methods and algorithms

2.1 Datasets used from Kaggle and NN models hyperparameters

The following data sets for image, text and tabular data classification were used
in the experiments described and discussed in this study:

  • COVID-19 X-Rays and Pneumonia Datasets
    (Kaggle) having 372 and 5,863
    images respectively.(,
  • Parkinson Detection Spiral Drawing
    & Waves (Kaggle) having 204 total images of Waves and Spirals draws
    images (
  • Histopathologic Cancer Detection
    Dataset having 57,458 32×32 colour images and it was used in Kaggle competition
    on 2019 (
  • Quora Insincere Questions Detection Dataset
    consist by 1,681,928 unique questions in total labeled with 0 if
    considered not toxic and 1 if considered toxic. It was also used in Kaggle
    competition on 2019 (
  • Microsoft Malware Prediction was used
    in Kaggle competition on 2019 too, and has 16,774,736 records in tabular
    format (

The CNN-related hyperparameters to train the datasets of COVID-19, Spiral Drawing
& Waves and Histopathologic Cancer Detection data set is a deep CNN and has
the following layers:

  • five convolutional layers, the first of
    which has kernel size of 5×5 and
    the four last 3×3
  • the following convolutional filters
    for each of the two convolutional layers respectively

(in order from the first layer to the last one): 32, 64,
128, 256, 512

  • Max Pooling and Batch Normalization is
    applied after all convolutional layers;
  • Dropout layer also after all
    convolutional layers leveraged with dropout rates (in order from the first
    layer to the last one): 0.1, 0.2, 0.3, 0.4, 0.5
  • The AFs were also applied after all convolution
  • A Global Average Pooling Layer followed
    by AF, Batch Normalization and dropout with rate 0.3
  • A Dense layer of size 256 followed by AF, Batch Normalization and
    dropout with rate 0.4
  • a final Dense layer with softmax
    activation, having as size of neurons the number of output classes of each
    the dataset


On Quora Insincere Questions Dataset, it was used a Bidirectional LSTM CNN, set
are as follows:

  • As embedding layer, it was used the
    mean of both Glove and Paragram embendings (these embeddings are included
    in the Quora Kaggle Dataset that is referred before.)
  • A Spatial Dropout at 0.2 rate
  • A Bidirectional LSTM layer with 128 RNN units,
  • four convolutional layers, each of
    which has a kernel size 1, 2, 3, 5 and filters of 100, 90, 30, 12
  • Each convolution layers is followed
    by the AF
  • four Global Max Pooling layer
  • A Dense layer of 200, followed by AF
    dropout (0.2) and batch normalization
  • a final Dense layer with softmax
    activation, having 2 neurons as for each class (toxic, not toxic)


A shallow neural network model is used to train Microsoft Malware Prediction dataset
with the following properties:

  • A Dense layer of size 100, followed
    by dropout rate of 0.4, Batch Normalization and AF
  • Another Dense layer of size 100 ,
    followed by dropout rate of 0.4, Batch Normalization and AF
  • a final Dense layer with softmax
    activation, having 2 neurons as for each class (has malware detection, not
    has malware detection)

2.2 The ALReLU AF as an LReLU alternative

Rectified Linear Unit, or ReLU, is one of the most common AFs used in NNs today. It is commonly
used on between layers to add nonlinearity in order to handle more complex and
nonlinear datasets. Fig. 1, demonstrates the ReLU that can be expressed as
follows (Eq. (2) is ReLU derivative):

Despite its success, especially on DNN, ReLU has a few issues. Firstly, ReLU is not continuously differentiable. At x=0, the gradient cannot be computed. Although it is not a serious problem, it slightly impact training performance.

Moreover, ReLU sets all values < 0 to zero. This can benefit on sparse data, but since the gradient of 0 is 0 and hence neurons arriving at large negative values cannot recover from being stuck at 0. The neuron effectively dies and hence the problem is known as the ’dying ReLU’ problem. This can leads the network essentially stops learning and underperforms.

Despite appropriate initialization of the weights to small random values, with large weight updates, the summed input to the traditional ReLU AF is always negative, although the input values fed to the NN. Current improvements to the ReLU, such as the LReLU, allow for a more non-linear output to either account for small negative values or facilitate the transition from positive to small negative values, without eliminating the problem though.

The LReLU is trying to solve these problems by providing a small negative gradient for negative inputs into a ReLU function. Fig. 2 and Eqs. (3) and (4) demonstrate the LReLU and its derivative.

Although theoretically LReLU is solving the ‘dying ReLU’, it is not actually proven to improve the classification performance. Indeed, in several studies the LReLU performance is the same or lower with ReLU. Consequently, this study investigates the development of a modified version of ReLU AF to obviate the problem of the ‘dying ReLU’ by using absolute values of LReLU, i.e., by achieving a positive solution where the solution was negative.

Actually, the idea to use absolute value rectification was first introduced by (Jarrett et al., 2009), and it used for object recognition from images where it makes sense to seek features that are invariant under a polarity reversal of the input illumination (GoodFellow et al., 2016). This work, goes a step further, with the proposal to use the absolute values of LReLU.  

Fig.3 and Eqs. (5) and (6) show the modified version that uses LReLU absolute values (ALReLU) and its derivative:

 The derivative of ALReLU on x<0 is the negative derivative of LReLU. On x>0 it is has the same derivative.

On the other hand, in Fig. 4 it is demonstrated the QReLU (L. Parisi, at al., 2020a). It can be considered that this AF and its derivatives are a little similar with the proposed one

The main difference is that ALReLU has smaller value and derivative. In a theoretically perspective, the ALReLU has also the properties of QReLU, such as the advantage of superposition and entanglement principles. However, this claim is only in theory and it is not proven in this paper. In fact, this work demonstrates the advantages of the little modification of LReLU that leads to the proposed ALReLU AF. The experiments and results on Section 3, indicate the significance impact of this modification.

The following code snippets, show the Keras implementation of the proposed AF, as well as its usage after the Convolution Layers. The derivatives and gradients of AFs are automatically calculated in TensorFlow 2, so they have not implemented manually.


Since the NN models are trained for given datasets, for estimating the performance and the classification accuracy a 9-Fold validation used in COVID-19 dataset and a 5-Fold cross-validation for other datasets. Cross-validation is a statistical method of evaluating and comparing learning algorithms by avoiding overfitting. The K-Fold validation procedure has been executed 4 times for every NN model and dataset. The average of these measures is computed and demonstrated in this section. The results support the theoretical superiority of the proposed ALReLU AF for image and text classification tasks, as well as tabular data classification. The classification performance results are demonstrated in Table 1 and are described above:


  • For CNN train and classification on COVID-19 X-Rays classification: All the evaluation metrics (Weighted Precision, Accuracy, Weighted Recall, AUC, F1) are better than original ReLU and LReLU activation functions. This demonstrated in the Table 1 first row dataset “COVID-19 X-Rays (Kaggle)”. 
  • For CNN train and classification on Spiral Drawings and Waves for Parkinson disease prediction: The results show a better performance of the proposed AF on Weighted Precision, Accuracy, Weighted Recall, F1. Only the AUC was a little lower (66.94% VS 71.9% and 74.38%) comparing with original ReLU and LReLU. (Table 1, second row “Spiral Drawings+Waves (Kaggle)”.
  • For CNN train and classification of Histopathologic Cancer Detection Dataset: The 3 AF have almost similar performance in this case. The proposed AF has a little lower results on Weighted Precision (88% VS 89% and 89%), Accuracy (86.69% VS 87.34% and 87.48%), and AUC (95.3% VS 95.45% and 95.21%). The other metrics are the same. (Table 1, third row “Histopathologic Cancer Detection (Kaggle)”).
  • For LSTM CNN train and classification of Quora Insincere Questions: In this text classification task, the 3 AF have also similar performance. The proposed AF has a little better results on Accuracy and AUC. The other metrics are the same. (Table 1, third row “Quora Insincere Questions (Kaggle)”).
  • For NN train and classification of Microsoft Malware Prediction Dataset: In this tabular data classification task the proposed AF has a little better results on Accuracy and AUC. The other metrics are the same. (Table 1, third row “Microsoft Malware Prediction (Kaggle)”).


In this paper, a different approach on LReLU function was demonstrated, by employing the absolute values of its negative gradient. It was proven a more accurate and robust AF for Neural Networks that used in image, text and tabular data classification. The proposed AF is merely centering on the improvement of the classification accuracy in sections wherein high accuracy and reliability is very important to be achieved, such as in the medical sector. Numerous tests were performed in order to validate the theoretical framework developed in this paper. In fact, the proposed method is used in several experiments of training and classification, on different datasets, and it shows superiority to the well-established ReLU and LReLU in terms of ROC/AUC metrics, recall, precision, F1 scores and accuracy. The important conclusion of these results, is the proposal of an alternative method for solving the common dying and vanishing gradient problems, a serious flag on NN training.


5. References


Luca Parisi, Daniel Neagu, Renfei Ma, and Felician Campean. Qrelu and m-qrelu: Two novel quantum activation functions
to aid medical diagnostics. arXiv preprint arXiv:2010.08031, 2020a

Luca Parisi, Renfei Ma, Narrendar
RaviChandran, Matteo Lanzillotta: hyper-sinh: An Accurate and Reliable Function
from Shallow to Deep Learning in TensorFlow and Keras arXiv preprint
arXiv:2011.07661, 2020b

R´egis Vert and Jean-Philippe Vert. Consistency and convergence rates of one-class svms and related algorithms. Journal of Machine Learning Research, 7(May):817–854, 2006.

Arthur Jacot, Franck Gabriel, and Cl´ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571–8580, 2018.

Maas, A. L., Hannun, A. Y., & Ng, A. Y. (2013). Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning. Atlanta, Georgia, USA: JMLR: W&CP.

Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In ICCV’09 . 16, 24, 27, 174, 193, 226, 363, 364, 523

Deep Learning (Ian
J. Goodfellow, Yoshua Bengio and Aaron Courville), MIT Press, 2016.


Related post