How Low Can You Go? Rank, Efficiency and Learning in Image and Speech Neural Networks

Speaker: Prof. Mark Sandler
Queen Mary University of London (QMUL)

December 1^st, 2026 | 3:00 pm
DEIB, "Schiavoni" Room (Bld. 20A)

Contact:  Riccardo Giampiccolo

Abstract

Deep learning models are known to be over-parameterised, and much of their arithmetic goes into dense, fully connected weight matrices. This talk presents Sum of Rank One (SoRO) layers, a structured alternative inspired by the Singular Value Decomposition. Each weight matrix is replaced by a trainable sum of rank-one terms. The rank is fixed before training, so savings in memory and computation come from the layer’s structure rather than from pruning afterwards.
I will describe two studies. In the first, the fully connected layers of LeNet are replaced with SoRO layers for image classification on MNIST and related datasets. These layers can be rank-reduced by more than 90% with negligible loss in accuracy. The second moves to audio: a simple MLP for spoken-digit keyword spotting (“Speech-MNIST”). It shows that representing MFCCs by their correlation structure gives the most effective training. SoRO networks cut computation by two-thirds or more for only a small drop in F1. I will also look at the trade-off between layer width and rank, which suggests that a small subspace of a high-dimensional space works better than a large subspace of a low-dimensional one.
Throughout, I use the evolution of singular values and the epoch-by-epoch rotation of singular vectors to see how networks learn. SoRO layers settle within a few epochs, while conventional layers keep rotating for much longer. This points to cheaper training through early freezing, and gives a new window on what low rank does inside a network.

Short Bio

Mark Sandler is Professor of Signal Processing at Queen Mary University of London (QMUL), where he founded the Centre for Digital Music (C4DM) and now heads the Centre for Fundamentals of AI and Computational Theory. He received his BSc and PhD from the University of Essex and spent 19 years at King’s College London, latterly as Head of Department, before moving to QMUL in 2001. He is a pioneer of digital audio and music research and has published over 500 papers. He is a Fellow of the Royal Academy of Engineering, IEEE, the IET and the AES. He has been a Turing Fellow and an IAS Distinguished Scholar at Tel Aviv University. His current EPSRC-funded research looks at the foundations of deep learning: low-rank network architectures, efficient “green” AI, and the training dynamics of neural networks in audio and vision.