To address OP's question in the post title though - yes of course - it probably leads to low self esteem and imposter syndrome. Key had the benefit of being a man (so doesn't get torn apart as much) and also being very skilled although not to the power vocalist level of some members of his group like onew - so I'm sure people weren't looking down on him.
In contrast someone like Giselle gets dragged so much and people even question her right to be in aespa
Chaeyeong from itzy pre recent surgery was constantly criticised for her looks
When it happens to female idols - it feels more personal and netizens make it like an inherent character flaw is why they are not popular so yeah it must do a real number on them.