By exploring this transformer's architecture, they theoretically proved that it can write a linear model within its hidden states.