![]() Haffner, “Gradient-based learning applied to document recognition”, Proceedings of the IEEE, vol. Krizhevsky, Learning multiple layers of features from tiny images, 2009. Vollgraf, “Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms”, arXiv, 2017.Ī. Garnett, Reds Curran Associates, Inc., 2019, pp. Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library”, in Advances in Neural Information Processing Systems 32, H. Rozza, “Learning Combinations of Activation Functions”, CoRR, vol. ![]() Kolodyazhniy, “A new learning algorithm for a forecasting neuro-fuzzy network”, Integrated Computer-Aided Engineering, vol. Ba, “Adam: A Method for Stochastic Optimization”, arXiv, 2017. Workshop on Digital Content & Smart Multimedia “DCSMART 2019”, vol. Slepanska, “Formal Neuron Based on Adaptive Parametric Rectified Linear Activation Function and its Learning”, in Proc. Pipa, “Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks”, arXiv, 2018. Baldi, “Learning Activation Functions to Improve Deep Neural Networks”, arXiv, 2015. Trentin, “Networks with Trainable Amplitude of Activation Functions”, Neural Netw., vol. Chang, “A Feedforward Neural Network with Function Shape Autotuning”, Neural Netw., vol. Shao, “The study of neural network adaptive control systems”, Control and Decision, no. Movellan, “Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks”, IEEE Transactions on Systems, Man, and Cybernetics, vol. Misra, “Mish: A Self Regularized Non-Monotonic Activation Function”, arXiv, 2020. Lu, “Universal Activation Function For Machine Learning”, arXiv, 2020.ĭ. Tanaka, “Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network”, arXiv, 2018.ī. Yan, “Deep Learning with S-shaped Rectified Linear Activation Units”, arXiv, 2015. Le, “Searching for Activation Functions”, arXiv, 2017. Doya, “Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning”, arXiv, 2017. Schmidhuber, “Long Short-Term Memory”, Neural Comput., vol. Sun, “Deep Residual Learning for Image Recognition”, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. Hochreiter, “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”, arXiv, 2016. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. Unbehauen, Neural Networks for Optimization and Signal Processing, 1st ed. Hornik, “Approximation capabilities of multilayer feedforward networks”, Neural Networks, vol. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals and Systems, vol. Springer Publishing Company, Incorporated, 2018. ![]() Aggarwal, Neural Networks and Deep Learning: A Textbook, 1st ed. Springer Publishing Company, Incorporated, 2018.Ĭ.C. Chang, Deep Neural Networks in a Mathematical Framework, 1st ed. USA: World Scientific Publishing Co., Inc., 2016.Ī.L. Graupe, Deep Learning Neural Networks: Design and Case Studies. Schmidhuber, “Deep learning in neural networks: An overview”, Neural Networks, vol. Yevgeniy Bodyanskiy, Kharkiv National University of Radio Electronics, Kharkiv The proposed approach is sufficiently simple from the implementation standpoint and provides high performance for the neural network training process. A double-stage parameter tuning process for training the neural networks with AHAF is proposed. The evaluation shows that the neural networks with AHAF activations achieve better classification accuracy comparing to their base implementations that use ReLU and SiL. The effectiveness of the function was evaluated on the image classification task using the Fashion-MNIST and CIFAR-10 datasets. The proposed function can be used as a drop-in replacement for ReLU, SiL and Swish activations for deep neural networks and can evolve to one of such functions during the training. The adaptive hybrid activation function (AHAF) is proposed that combines the properties of the rectifier units and the squashing functions. Kharkiv National University of Radio Electronics, Kharkiv, UkraineĪdaptive hybrid activation function, double-stage parameter turning process, deep neural networks Abstract
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |