ISSN 0869-6632 (Print)
ISSN 2542-1905 (Online)

For citation:

Moskvitin V. М., Semenova N. И. Noise influence on recurrent neural network with nonlinear neurons. Izvestiya VUZ. Applied Nonlinear Dynamics, 2023, vol. 31, iss. 4, pp. 484-500. DOI: 10.18500/0869-6632-003052, EDN: XGRKMR

This is an open access article distributed under the terms of Creative Commons Attribution 4.0 International License (CC-BY 4.0).
Full text PDF(Ru):
PDF icon and_2023-4_7moskvitin-semenova_484-500.pdf
(downloads: 0)
Full text PDF(En):
PDF icon and_2023_4_moskvitin_semenova_eng.pdf
(downloads: 6)
Language: 
Russian
Heading: 
Nonlinear Dynamics and Neuroscience
Article type: 
Article
UDC: 
004.032.26, 530.152.2
DOI: 
10.18500/0869-6632-003052
EDN: 
XGRKMR

Noise influence on recurrent neural network with nonlinear neurons

Autors: 
Moskvitin Viktor Максимович, Saratov State University
Semenova Nadezhda Игоревна, Saratov State University
Abstract: 

The purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them.

Methods. We have considered white Gaussian noise sources. We used additive, multiplicative and mixed noise depending on how the noise is introduced into artificial neurons. The noise impact was estimated using the dispersion (variance) of the output signal.

Results. It is shown that the activation function plays a significant role in noise accumulation. Two nonlinear activation functions have been considered: the hyperbolic tangent and the sigmoid function with range form 0 to 1. It is shown that some types of noise are suppressed in the case of the second function. As a result of considering the influence of coupling matrices, it was found that diagonal coupling matrices with a large blurring coefficient lead to less noise accumulation in the echo network reservoir with an increase in the reservoir memory influence.

Conclusion. It is shown that activation functions of the form of sigmoid with range from 0 to 1 are suitable for suppressing multiplicative and mixed noise. The accumulation of noise in the reservoir was considered for three types of coupling matrices inside the reservoir: a uniform matrix, a band matrix with a small blurring coefficient, and a band matrix with a large blurring coefficient. It has been found that the band matrix echo networks with a high blurring coefficient accumulates the least noise. This holds for both additive and multiplicative noise.

Key words: 
neural networks
recurrent neural networks
echo-state networks
noise influence
white noise
nonlinear activation function
Acknowledgments: 
This work was supported by Russian Science Foundation (Project no. 21-72-00002)
Reference: 
  1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–444. DOI: 10.1038/ nature14539.
  2. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun. ACM. 2017;60(6):84–90. DOI: 10.1145/3065386.
  3. Maturana D, Scherer S. VoxNet: A 3D Convolutional Neural Network for real-time object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 28 September 2015 – 02 October 2015, Hamburg, Germany. New York: IEEE; 2015. P. 922–928. DOI: 10.1109/IROS.2015.7353481.
  4. Graves A, Mohamed AR, Hinton G. Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 26–31 May 2013, Vancouver, BC, Canada. New York: IEEE; 2013. P. 6645–6649. DOI: 10.1109/ICASSP.2013. 6638947.
  5. Kar S, Moura JMF. Distributed consensus algorithms in sensor networks with imperfect communication: Link failures and channel noise. IEEE Transactions on Signal Processing. 2009;57(1):355–369. DOI: 10.1109/TSP.2008.2007111.
  6. Mandic DP, Chambers JA. Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. New York: Wiley; 2001. 304 p. DOI: 10.1002/047084535X.
  7. Bailador G, Roggen D, Troster G, Trivino G. Real time gesture recognition using continuous time recurrent neural networks. In: 2nd International ICST Conference on Body Area Networks. 11th–13th Jun 2007, Florence, Italy. ICST; 2007. 8 p. DOI: 10.4108/bodynets.2007.149.
  8. Hasler J, Marr H. Finding a roadmap to achieve large neuromorphic hardware systems. Frontiers in Neuroscience. 2013;7:118. DOI: 10.3389/fnins.2013.00118.
  9. Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P. Deep learning with limited numerical precision. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. 6-11 July 2015, Lille, France. JMLR; 2015. P. 1737–1746.
  10. Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nature Reviews Physics. 2021;3(6):422–440. DOI: 10.1038/s42254-021-00314-5.
  11. Brunner D, Soriano MC, Mirasso CR, Fischer I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nature Communications. 2013;4(1):1364. DOI: 10.1038/ncomms2368.
  12. Tuma T, Pantazi A, Le Gallo M, Sebastian A, Eleftheriou E. Stochastic phase-change neurons. Nature Nanotechnology. 2016;11(8):693–699. DOI: 10.1038/nnano.2016.70.
  13. Torrejon J, Riou M, Araujo FA, Tsunegi S, Khalsa G, Querlioz D, Bortolotti P, Cros V, Yakushiji K, Fukushima A, Kubota H, Yuasa S, Stiles MD, Grollier J. Neuromorphic computing with nanoscale spintronic oscillators. Nature. 2017;547(7664):428–431. DOI: 10.1038/nature23011.
  14. Psaltis D, Brady D, Gu XG, Lin S. Holography in artificial neural networks. Nature. 1990;343 (6256):325–330. DOI: 10.1038/343325a0.
  15. Bueno J, Maktoobi S, Froehly L, Fischer I, Jacquot M, Larger L, Brunner D. Reinforcement learning in a large-scale photonic recurrent neural network. Optica. 2018;5(6):756–760. DOI: 10.1364/ OPTICA.5.000756.
  16. Lin X, Rivenson Y, Yardimci NT, Veli M, Luo Y, Jarrahi M, Ozcan A. All-optical machine learning using diffractive deep neural networks. Science. 2018;361(6406):1004–1008. DOI: 10.1126/science. aat8084.
  17. Shen Y, Harris NC, Skirlo S, Prabhu M, Baehr-Jones T, Hochberg M, Sun X, Zhao S, Larochelle H, Englund D, Soljacic M. Deep learning with coherent nanophotonic circuits. Nature Photonics. 2017;11(93):441–446. DOI: 10.1038/nphoton.2017.93.
  18. Tait AN, de Lima TF, Zhou E, Wu AX, Nahmias MA, Shastri BJ, Prucnal PR. Neuromorphic photonic networks using silicon photonic weight banks. Scientific Reports. 2017;7(1):7430. DOI: 10.1038/s41598-017-07754-z.
  19. Moughames J, Porte X, Thiel M, Ulliac G, Larger L, Jacquot M, Kadic M, Brunner D. Three dimensional waveguide interconnects for scalable integration of photonic neural networks. Optica. 2020;7(6):640–646. DOI: 10.1364/OPTICA.388205.
  20. Dinc NU, Psaltis D, Brunner D. Optical neural networks: The 3D connection. Photoniques. 2020;(104):34–38. DOI: 10.1051/photon/202010434. 
  21. Moughames J, Porte X, Larger L, Jacquot M, Kadic M, Brunner D. 3D printed multimode splitters for photonic interconnects. Opt. Mater. Express. 2020;10(11):2952–2961. DOI: 10.1364/ OME.402974.
  22. Semenova N, Porte X, Andreoli L, Jacquot M, Larger L, Brunner D. Fundamental aspects of noise in analog-hardware neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science. 2019;29(10):103128. DOI: 10.1063/1.5120824.
  23. Semenova N, Larger L, Brunner D. Understanding and mitigating noise in trained deep neural networks. Neural Networks. 2022;146:151–160. DOI: 10.1016/j.neunet.2021.11.008.
  24. Semenova N, Brunner D. Noise-mitigation strategies in physical feedforward neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science. 2022;32(6):061106. DOI: 10.1063/ 5.0096637.
  25. Jaeger H. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach. GMD-Report 159. Bonn: German National Research Center for Information Technology; 2002. 48 p.
  26. Prokhorov D. Echo state networks: appeal and challenges. In: Proceedings. 2005 IEEE International Joint Conference on Neural Networks. Vol. 3. 31 July 2005 – 04 August 2005, Montreal, QC, Canada. New York: IEEE; 2005. P. 1463–1466. DOI: 10.1109/IJCNN.2005.1556091.
  27. Cerina L, Santambrogio MD, Franco G, Gallicchio C, Micheli A. EchoBay: Design and optimization of echo state networks under memory and time constraints. ACM Transactions on Architecture and Code Optimization. 2020;17(3):1–24. DOI: 10.1145/3404993.
  28. Lukosevicius M, Jaeger H. Reservoir computing approaches to recurrent neural network training. Computer Science Review. 2009;3(3):127–149. DOI: 10.1016/j.cosrev.2009.03.005.
Received: 
06.03.2023
Accepted: 
02.05.2023
Available online: 
12.07.2023
Published: 
31.07.2023
  • 899 reads