Same here. oh-my-zsh was too slow for me and took away one of the advantages of zsh (that of being faster to load than bash). Also, a great number of oh-my-zsh config options are, to me, incredibly annoying (the long completion menu is the worst shell interface ever invented).

I'm in the process of trimming down prezto to exactly what I need, since it also has too much fluff. It's a great resource, I'd recommend it to anyone wanting a slimmer and easier to grok zsh configuration.