I have a 4S still around as my "iPod touch" and it does fine compared to my 5 in terms of usability for apps and music. Remembering back, I didn't somehow feel when I got the iPhone5 and later my Nexus5 that my 4S was "inadequate". I did feel that it weighed a lot compared the iPhone5 however, and was very glad for that.
When thinking about a smartphone, I care more about it's usability as a phone, then as a music/podcast player, and only finally as a media device. Am I out in left field here?
A bit, yes. My phone (n5) is used more as a web browsing device and video consuming device than it is as a music player. It is definitely the screen size. My previous phone (HTC Desire, about the same screen size as an iPhone <6) was pretty much a phone plus ipod.
I think the demand for screen size is also due to the fact that we probably have some ways to go before getting good at designing things for small screened devices.
I am still in 2 minds about my iPhone 6. It is much nicer in most ways than the 5. The extra screen real-estate really helps but at the cost of single-handed operation and comfort when bending at the hips.
I'm sure Apple does market research, but they've proven to be very susceptible to their own dogma.