CVC Seminars: “Continual Learning from Pretrained Models”, Dr. Antonio Carta

Antonio Carta is an Assistant Professor at the Department of Computer Science, University of Pisa. He is a member of the Pervasive AI Lab (University of Pisa, CNR) and the Computational Intelligence and Machine Learning group (University of Pisa). He has a Ph.D. in Computer Science from the University of Pisa where he studied recurrent neural networks and the problem of short-term memory optimization.
His recent research is focused on continual learning, including multi-agent continual learning and learning with privacy constraints. He is a member of ContinualAI and the lead maintainer of Avalanche, one of the most extensive continual learning libraries.

Continual Learning (CL) is a paradigm where an agent learns over time from a stream of data. In this talk, we will discuss how to exploit pretrained models in CL. First, we will talk about “continual pretraining”, a scenario where a large pretrained model is updated over time. The results show that continual pretraining seems much more robust to catastrophic forgetting than supervised CL. Then, we will talk about “Ex-Model CL”, a scenario where a CL agent learns from a stream of pretrained models. Learning from pretrained models allows independent CL agents to freely share their knowledge. We will show some preliminary results in a simplified setting and discuss possible future directions.