Nowadays things seem bleak because instead of games targeting the small portion of the overall players, they try to target everyone who may be at all interested. It's how I feel with most games now.
There's no special award for making a game that's mass-market unfriendly and only going to sell to the developer's friends. There was nothing exclusive about the Wing Commander games. The market was still on the high of the space programs that also fed Star Trek movies and TV series (which have funnily enough suffered the same limbo as WC). People wanted to buy a joystick and an expensive rig so they could ride that high. We had X-Wings and Tie Fighters up to our eyeballs.
Today the high has faded into a sense of "oh, yeah that." Fantasy and modern military replaces science fiction and demon/pig-cop shooters. In another 10-20 years we may see the tastes shift again and be up to our eyeballs in games set in the Romantic and Victorian eras.
Anyway...I look at games now and see people trying to build these giant sprawling epics. These games that are not measured by the weight of their storylines, but of the content of your hourly commitment to them. Most of us who work full-time can probably say that most of our free time isn't spent trying to get that next level in Fallout 3, or traveling across Ferelden to enlist some Elves for some asskicking in Dragon age.
Fallout 3 reminds me a lot of the Privateer games, namely having this story thing over here but having all this other stuff you can explore and do over there. And under it all there are means to get better guns and ships... err... power armor, although you can do the whole game in your Tarsus... err... vault jumpsuit.
The shortness of games in the early 90's was entirely technical. They didn't have terrabyte harddrives, gigabytes of ram, or blue-ray discs. Remember how it took five minutes to load a mission in WC3? Or that your harddrive held less data than a single CD? As technical limitations are withdrawn developers can realize new objectives. Just imagine what kind of sprawling epics the Privateer games would have been if made with today's technology.
But to be optimistic. Maybe if current game devs put down the pen and put it into the hands of real writers. Maybe we could see some decent stories come out of the industry that don't have the words "FOXDIE" "Force" "Mana" or "Mako" or even "Zanarkand" as key McGuffins.
Development teams have plenty of dedicated, professional writers. But just like movies, writers don't top the list of credits. If not for the CIC staffers pointing out who really wrote some of the Wing Commander games we'd just be mindlessly swooning over Mr. Roberts.
And I'm not sure how to respond to a complaint that games try to have an objective in their story. Even Luke Skywalker and Frodo Baggins had their McGuffins to see to.