It's not racism nor colorism, it's a social class thing. Traditionally, the people with a tanned skin were those working in the fields ie. the peasants. Nobles and lords didn't have to do such labours so they had a paler skin therefore it became the norm of beauty. Simple as that.
Colorism is a symptom of classism.