See at: CNR ExploRA

Bio-inspired approaches for Deep Learning: from spiking neural networks to Hebbian plasticity
Lagani G.
In the past few years, Deep Neural Network (DNN) architectures have achieved outstanding results in several Artificial Intelligence (AI) domains. Even though DNNs draw inspiration from biology, the training methods based on the backpropagation algorithm (\textit{backprop}) lack neuroscientific plausibility. The goal of this dissertation is to explore biologically-inspired solutions for the learning task. These are interesting because they can help to reproduce features of the human brain, for example, the ability to learn from a little experience. The investigation is divided into three phases: first, I explore a novel AI solution based on simulating neuronal biological cultures with a high level of detail, using biologically faithful Spiking Neural Network (SNN) models; second, I investigate neuroscientifically grounded \textit{Hebbian} learning rules, applied to traditional DNNs in combination with backprop, using computer vision as a case study; third, I consider a more applicative perspective, using neural features derived from Hebbian learning for multimedia content retrieval tasks. I validate the proposed methods on different benchmarks, including MNIST, CIFAR, and ImageNet, obtaining promising results, especially in learning scenarios with scarce data. Moreover, to the best of my knowledge, for the first time, I am able to bring bio-inspired Hebbian methods to ImageNet scale, consisting of over 1 million images.Project(s): AI4Media

See at: etd.adm.unipi.it | CNR ExploRA

See at: CNR ExploRA

2023 Conference article Open Access

See at: ceur-ws.org | ISTI Repository | CNR ExploRA

2023 Other Open Access

See at: ISTI Repository | CNR ExploRA

2022 Journal article Open Access

Comparing the performance of Hebbian against backpropagation learning using convolutional neural networks
Lagani G., Falchi F., Gennaro C., Amato G.
In this paper, we investigate Hebbian learning strategies applied to Convolutional Neural Network (CNN) training. We consider two unsupervised learning approaches, Hebbian Winner-Takes-All (HWTA), and Hebbian Principal Component Analysis (HPCA). The Hebbian learning rules are used to train the layers of a CNN in order to extract features that are then used for classification, without requiring backpropagation (backprop). Experimental comparisons are made with state-of-the-art unsupervised (but backprop-based) Variational Auto-Encoder (VAE) training. For completeness,we consider two supervised Hebbian learning variants (Supervised Hebbian Classifiers--SHC, and Contrastive Hebbian Learning--CHL), for training the final classification layer, which are compared to Stochastic Gradient Descent training. We also investigate hybrid learning methodologies, where some network layers are trained following the Hebbian approach, and others are trained by backprop. We tested our approaches on MNIST, CIFAR10, and CIFAR100 datasets. Our results suggest that Hebbian learning is generally suitable for training early feature extraction layers, or to retrain higher network layers in fewer training epochs than backprop. Moreover, our experiments show that Hebbian learning outperforms VAE training, with HPCA performing generally better than HWTA.Source: Neural computing & applications (Print) (2022). doi:10.1007/s00521-021-06701-4
DOI: 10.1007/s00521-021-06701-4
Project(s): AI4EU , AI4Media
Metrics:

See at: ISTI Repository | ISTI Repository | link.springer.com | CNR ExploRA

2022 Other Open Access

See at: ISTI Repository | CNR ExploRA

FastHebb: scaling hebbian training of deep neural networks to ImageNet level
Lagani G., Gennaro C., Fassold H., Amato G.
Learning algorithms for Deep Neural Networks are typically based on supervised end-to-end Stochastic Gradient Descent (SGD) training with error backpropagation (backprop). Backprop algorithms require a large number of labelled training samples to achieve high performance. However, in many realistic applications, even if there is plenty of image samples, very few of them are labelled, and semi-supervised sample-efficient training strategies have to be used. Hebbian learning represents a possible approach towards sample efficient training; however, in current solutions, it does not scale well to large datasets. In this paper, we present FastHebb, an efficient and scalable solution for Hebbian learning which achieves higher efficiency by 1) merging together update computation and aggregation over a batch of inputs, and 2) leveraging efficient matrix multiplication algorithms on GPU. We validate our approach on different computer vision benchmarks, in a semi-supervised learning scenario. FastHebb outperforms previous solutions by up to 50 times in terms of training speed, and notably, for the first time, we are able to bring Hebbian algorithms to ImageNet scale.Source: SISAP 2022 - 15th International Conference on Similarity Search and Applications, pp. 251–264, Bologna, Italy, 5-7/10/2022
DOI: 10.1007/978-3-031-17849-8_20
Project(s): AI4Media
Metrics:

See at: doi.org | link.springer.com | CNR ExploRA

2022 Conference article Open Access

Recent advancements on bio-inspired Hebbian learning for deep neural networks
Lagani G.
Deep learning is becoming more and more popular to extract information from multimedia data for indexing and query processing. In recent contributions, we have explored a biologically inspired strategy for Deep Neural Network (DNN) training, based on the Hebbian principle in neuroscience. We studied hybrid approaches in which unsupervised Hebbian learning was used for a pre-training stage, followed by supervised fine-tuning based on Stochastic Gradient Descent (SGD). The resulting semi-supervised strategy exhibited encouraging results on computer vision datasets, motivating further interest towards applications in the domain of large scale multimedia content based retrieval.Source: SEBD 2022 - 30th Italian Symposium on Advanced Database Systems, pp. 610–615, Pisa, Italy, 2022
Project(s): AI4EU , AI4Media

See at: ceur-ws.org | ISTI Repository | CNR ExploRA

2021 Journal article Open Access

Hebbian semi-supervised learning in a sample efficiency setting
Lagani G., Falchi F., Gennaro C., Amato G.
We propose to address the issue of sample efficiency, in Deep Convolutional Neural Networks (DCNN), with a semi-supervised training strategy that combines Hebbian learning with gradient descent: all internal layers (both convolutional and fully connected) are pre-trained using an unsupervised approach based on Hebbian learning, and the last fully connected layer (the classification layer) is trained using Stochastic Gradient Descent (SGD). In fact, as Hebbian learning is an unsupervised learning method, its potential lies in the possibility of training the internal layers of a DCNN without labels. Only the final fully connected layer has to be trained with labeled examples. We performed experiments on various object recognition datasets, in different regimes of sample efficiency, comparing our semi-supervised (Hebbian for internal layers + SGD for the final fully connected layer) approach with end-to-end supervised backprop training, and with semi-supervised learning based on Variational Auto-Encoder (VAE). The results show that, in regimes where the number of available labeled samples is low, our semi-supervised approach outperforms the other approaches in almost all the cases.Source: Neural networks 143 (2021): 719–731. doi:10.1016/j.neunet.2021.08.003
DOI: 10.1016/j.neunet.2021.08.003
DOI: 10.48550/arxiv.2103.09002
Project(s): AI4EU , AI4Media
Metrics:

See at: arXiv.org e-Print Archive | Neural Networks | ISTI Repository | ZENODO | Neural Networks | doi.org | www.sciencedirect.com | CNR ExploRA

2021 Other Open Access

See at: ISTI Repository | CNR ExploRA

Hebbian Learning GitHub repository
Lagani G.
Pytorch implementation of Hebbian learning algorithms to train deep convolutional neural networks.Project(s): AI4Media

See at: github.com | CNR ExploRA

2020 Other Open Access

See at: ISTI Repository | CNR ExploRA

2019 Conference article Open Access

Hebbian learning meets deep convolutional neural networks
Amato G., Carrara F., Falchi F., Gennaro C., Lagani G.
Neural networks are said to be biologically inspired since they mimic the behavior of real neurons. However, several processes in state-of-the-art neural networks, including Deep Convolutional Neural Networks (DCNN), are far from the ones found in animal brains. One relevant difference is the training process. In state-of-the-art artificial neural networks, the training process is based on backpropagation and Stochastic Gradient Descent (SGD) optimization. However, studies in neuroscience strongly suggest that this kind of processes does not occur in the biological brain. Rather, learning methods based on Spike-Timing-Dependent Plasticity (STDP) or the Hebbian learning rule seem to be more plausible, according to neuroscientists. In this paper, we investigate the use of the Hebbian learning rule when training Deep Neural Networks for image classification by proposing a novel weight update rule for shared kernels in DCNNs. We perform experiments using the CIFAR-10 dataset in which we employ Hebbian learning, along with SGD, to train parts of the model or whole networks for the task of image classification, and we discuss their performance thoroughly considering both effectiveness and efficiency aspects.Source: Image Analysis and Processing - ICIAP 2019, pp. 324–334, Trento, Italia, 9/9/2019, 13/9/2019
DOI: 10.1007/978-3-030-30642-7_29
Project(s): AI4EU
Metrics:

See at: ISTI Repository | Lecture Notes in Computer Science | link.springer.com | CNR ExploRA

2019 Other Open Access

See at: ISTI Repository | CNR ExploRA