Scientists develop new mathematical model for transfer learning in neural networks

Alessandro Ingrosso, researcher at the Donders Institute for Neuroscience, has developed a new mathematical method in collaboration with colleagues in two Italian research institutions, which enables the prediction of the effectiveness of transfer learning in neural networks.

The paper is published in the journal Physical Review Letters.

The problem of limited data

AI systems are typically trained with large amounts of data. Consider an image recognition system trained to identify a dog in a picture. By training this network with large quantities of dog photos, the network learns to identify a dog with high accuracy in new images.

However, for some applications, sufficient training data is not available. “In the medical sector, for example, when diagnosing cancer via ultrasound, there aren’t enough examples available to properly train a neural network,” explains Ingrosso.

“This leads to ‘overfitting,’ where the network fails at generalizing on new, unseen cases. The number of false positives and false negatives can then become very large.”

To solve this problem, the researchers focused on “transfer learning,” a technique where knowledge from a network trained on a large dataset (the “source”) is transferred to a new network (the “target”) that is trained on limited data.

“We studied the simplest model for this transfer learning approach and developed a mathematical theory for networks with one hidden layer,” says Ingrosso.

Innovative combination of analytical methods

In this theory, the researchers combined two different analytical methods: the recently developed “Kernel Renormalization” approach and the classical “Franz-Parisi” formalism from the theory of Spin Glasses. Merging such tools makes it possible to work with specific, real datasets instead of statistical models or approximations.

Ingrosso adds, “Our new method can directly and accurately predict how effective the target network will be in generalizing data when it adopts knowledge from the source network.”

This research provides important new insights for the effective training of AI systems in domains where data is scarce, such as medical diagnostics and other specialized applications.