It's pretty clear to me that for a lot of people that claim it doesn't matter to them, it actually matters a lot because they are super vocal about it whenever a character does not appeal to them. They can't just ignore it and play the games that actually appeal to them. They gotta complain about it and let everyone know "hey I don't like this game because this character doesn't appeal to me at all!"
There's definitely a nugget of ironic truth to what you're saying. For me, as someone who says that I don't need a character that looks like me or has anything in common with me, I still want characters who are fun, sexy, or cool. I like things being stylized and compelling, and I admit I pass on a lot of games because I don't like the way the characters look.
I guess that's where the difference comes in. I view games as escapism, so I'm not looking to have myself represented. Whereas it seems like that's the primary goal of "the other side" of this debate. Extremely bland-looking, even intentionally ugly characters that are designed to "represent", check boxes, and avoid offense. The recent implosion of Concord was the greatest example of this type of design thinking. That side would see Dragon's Crown and lose their minds, yelling that "it's harmful", "designed for the male gaze", and "completely unrealistic". It's the idea that players need someone they can identify with or champion their values.