SwitchPath enhancing exploration in neural networks learning dynamics

Di Cecco A., Papini A., Metta C., Fantozzi M., Galfré S. G., Morandin F., Parton M.

Deep Learning Theory, Deep Neural Network Algorithms 

We introduce SwitchPath, a novel stochastic activation function that enhances neural network exploration, performance, and generalization, by probabilistically toggling between the activation of a neuron and its negation. SwitchPath draws inspiration from the analogies between neural networks and decision trees, and from the exploratory and regularizing properties of DropOut as well. Unlike Dropout, which intermittently reduces network capacity by deactivating neurons, Switch- Path maintains continuous activation, allowing networks to dynamically explore alternative information pathways while fully utilizing their capacity. Building on the concept of ε-greedy algorithms to balance exploration and exploitation, SwitchPath enhances generalization capabilities over traditional activation functions. The exploration of alternative paths happens during training without sacrificing computational efficiency. This paper presents the theoretical motivations, practical implementations, and empirical results, showcasing all the described advantages of SwitchPath over established stochastic activation mechanisms.

Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 15243 - Proceedings, Part I, pp. 275-291. Pisa, Italy, 14-16/10/2024

Publisher: Springer

Metrics