Looking inside visual attention
This project investigated internal attention dynamics in Vision Transformer architectures using pretrained DeiT models. I built a pipeline to extract and analyse attention distributions across transformer layers, making it possible to compare how attention changes with depth and input structure.
Experimental intervention
Beyond visualisation, I conducted experiments that modified attention formulations to study the resulting model behaviour. The goal was to move from observing attention maps to testing how particular attention mechanisms affect a model’s internal representations.
The work was completed in collaboration with Dr Kasra Khosoussi and strengthened my interest in understanding how mathematical structure translates into model behaviour.