...and the funny thing is that the majority of Manhattan's residential real estate is, in fact, a mess of ugly, uncomfortable five-story buildings (at ridiculous prices).
I'd estimate that less than 5% of Manhattan's residential space exists in buildings taller than 20 stories, and maybe 15% of said space exists in buildings taller than 5 stories.
You are 100% correct, but interestingly more and more high rise stock is going residential. These are just a few examples, but check out 520 West 41st street (106 floors of residential) or 45 East 22nd Street (60 floors of residential). That's hundreds and hundreds of apartments going on tiny footprints. The one on 22nd street is replacing a 4 story nondescript building, in almost laughable fashion. The major buildings going up on Park Ave, in Long Island City, and all along the Brooklyn water front are all residential.
There seems to be a glut of office space, so for now this is where the market is going. Much of it is luxury unfortunately, but not all of it.
And all of the new construction tall buildings are luxury condos, which are barely "affordable" in Queens and start in the millions on Manhattan. I think something like 40% of them are actually even occupied year-round, instead they are used as investment/offshore holding/pied-a-terre for the absurdly wealthy.
I'd estimate that less than 5% of Manhattan's residential space exists in buildings taller than 20 stories, and maybe 15% of said space exists in buildings taller than 5 stories.