I think that the reason D&D works the way it works is that as you go up in levels, the game itself is supposed to change. Ordinary men going through a military career will end up as veterans on 2nd, 3rd, or 4th level. Some rare commanders will reach 5th level. But adventurers will want to fight trolls, and giants, and dragons.
More hitpoints is what allows adventurers to proceed to the next type of D&D game, going toe-to-toe with tough brutes.
As they gain even more levels, hit-points start to loose in importance as magic items and spells, and the appropriate defenses gain in importance. Now you want to go up against nagas, and mindflayers, and beholders. No problem!
To me, that's the D&D secret nobody told me about: Every level range has a different "feel" to it, appropriate encounters, strategies, tactics, items, concerns, and so on.
The mental disconnect only happens, I think, if you take your level 8 fighter and fight bandits, or fight alongside henchmen. If you don't want the level 8 fighter to shine like a madman, like a prince of Amber, like a hairy foot god of war, then it's easy to point your finger at the hit-points. An alternative point of view might suggest that perhaps the adventurers should not gain more than a level or two in the first place. That keeps them within the level range where fighting bandits and goblins and remains a cause for caution.
I don't think that one way or the other is intrinsically better than the other way of playing the game. I do think, however, that the default rules imply a certain progression through the monster manual, if you like. Conversely, if you limit hit-points, I think you will have to massively change monsters. I'm not sure whether just reducing hit-points is appropriate enough. My guess is that higher level monsters also need to have their melee attacks reduced, otherwise an unarmed mind-flayer isn't just dangerous because of his mind-blasting and his brain-eating, in addition to that he'll also be a fearsome melee fighter (which he is not, with the default rules).
The entire thing is an interesting thought experiment. I'm not sure I'd want to go there, however, because personally, I like the changing nature of D&D as you go through in levels (and I therefore accept side-effects like knights being infinitely superior fighters to squires and henchmen, etc.).
I'd like to start by saying that I agree with the spirit of this entire remark. There's really nothing wrong with the game as written, and it's a lot of fun to play as a level 8 Fighter and be a powerhouse of death and be tough as nails, able to slay a band of lesser men with only a couple of scratches to show for it. Similarly, I agree that the feel of the game changes dramatically, and not necessarily for the worse. Low-level campaigns are ones where the cauliflower-eared fighting man reigns supreme, and the most deadly threat is the sword in your back, but high level ones are where wizards are the deadly ones, and even the most staid of fighters are brandishing +4 Vorpal Greatswords and have quivers full of Arrows of Dragon Slaying and stuff.
I suppose the most fundamental disconnect comes, as you say, when you have to look at the monsters and realize that what is portrayed as a weak, but mentally powerful mind flayer can wrestle men to the ground and beat them to death with their bare hands, or when you realize that the sort of threats that require a 6th level party can nearly decimate a kingdom. Or when you realize that you're going to have to hit a fiery dude fifteen times with a sword to get it to quit trying to eat princesses. Or when you realize that your fighter can get straight chomped on by a troll a good four times and still get up and smack him with yet another axe chop. It's not particularly interesting, especially when you consider that, despite claims to the contrary, hit points as written are entirely toughness and not some sort of "strength of will and luck" and stuff. When's the last time you had to rest for a day to recover one point of your luck?
Regardless, I don't think there's really a reason that we have to throw out higher levels just because it's tied to health. I can't think of a less impressive fighter than one who never gets any better at dodging as he levels, and only learns to take a punch better. A man like that is a guile-less butcher, not a warrior. With just an easy tweak (perhaps applying the THAC0 as a bonus to Armor Class, and then halving hit point gains after level one?) you can make a fighter that's dodging and weaving as well as getting more grizzled and hard to kill.
And see, it's cool that D&D isn't really meant to emulate sword and sorcery fiction- it's sort of a science-fantasy game, with fire and forget spells and everybody trying to wear as much armor as possible and a high death rate and exploring dungeons created three world-spanning armageddons ago. It's a very cool feel for a game, but it's not the only one possible by any means. With just a couple of tweaks, you can make it into a more historical game, where killing trolls and dragons means that you're planning out a major invasion that would be the focus of a campaign in and of itself, instead of a session of hack and slashery made possible because you've got 80 hit points and seven +12 axes or whatever.
But I digress. Again.