Why Do Self-Supervised Models Transfer? On the Impact of Invariance on Downstream Tasks