I think there's a couple of factors to consider. One being that it's Martin's fictional world. He creates certain structural facts about that world that serve his story. They don't have to make sense with our world because it's not the world we live in. They only have to make sense within the confines of the story. Summers and Winters will be as long as they need to in order to serve the story of A Song of Ice and Fire. Accepted on face value that way, the long seasons are no different than having dragons and wargs, wights and red magic. It's all part of the canon of Martin's world.
The second thing to take into consideration is that I don't think the term "season" means the same thing in Martin's world that it does in ours. Recall that they make references in the books to "light Summer snows" and that sort of thing. In most of the Global North we think of seasons as the four distinct times of year with generalized weather patterns. Winter is cold, Spring is warm, Summer is hot and Fall is cool. Those nations even define the start and end times for the rest of the world (e.g. March 21st is the "official" start of Spring...what does that even mean?)--even if the weather doesn't cooperate. How long into Spring was it still snowing in the South? And Wyoming got a late snowfall just last week.
To put that in perspective, where I live in the Philippines, they don't define their seasons that way at all. April and May is considered "Summer" here, as that is the hottest time of year. The next season is not Fall, but Rainy Season, starting in June and lasting until sometime between September and November. Rainy Season is also referred to as Winter here. And that time between November and March is referred to as "High Season". That's when the weather is less hot, but without as much rain. It's called High Season because that's when all the tourists come, but also because that's when a lot of construction and other business gets accomplished...without blisteringly hot temperatures or monsoons to interfere. A season of high economic activity either way, thus "High Season". People in Southeast Asia prepare for Rainy Season much the way Starks prepare for Winter. You always know it's coming, and you try to get things in order before the weather makes it impossible for you to accomplish things.
So rather than think of seasons as fixed, defined and regimented weather patterns, think of it as a way to explain a generalized set of circumstances.
I think Summer in Westeros is simply the times when weather is most pleasant and economic activity is high. Think of those times you've had a "mild Winter" where you live...and I think that's what Martin would call a "Summer Snow" in Westeros.
I do think The Others have a role in the seasons of Martin's world however, I'm not sure to what extent. Most people in Westeros believe The Others died out centuries ago. But they've had bad Winters in living memory of the characters we know. So either The Others have been stronger all this time and the people of Westeros simply unawares (totally a possibility) or they have less an impact on actual weather than we think.
My two cents.
