Turning off the stylesheet on every page as I browse the web would be a hassle; much better to change a setting once and never have to worry about it again. About the only pages I've found that aren't improved by enforcing a sensible font-choice are pages designed to showcase CSS3 font embedding - and those are pretty rare.

I do wish browsers would have a per-host "allow custom fonts" setting, alongside the tradition per-host "allow images" and "allow cookies" settings. With the potential security implications of downloadable fonts, it seems only sensible - but nobody seems interested in implementing such a thing. :(