I have the same problem as OP, which makes me think that this is not common, and yet it affects some people. If it were common, the app wouldn't exist because it is just entirely unusable.
I used to have this problem and I remember it being DNS related. It was a while ago and I don't quite remember what I did to resolve it, but it had something to do with disabling IPv6 dns resolution while not using IPv6 in about:config and possibly something else as well.
Yes I think you're on the right track. It must be some lurking network misconfiguration. Firefox itself is not broken in this way in my experience (which is not short).