We use PagerDuty (http://pagerduty.com) at HipChat and while I absolutely loathe being woken up by it, it's helped us identify issues during off-peak hours much more quickly.

But no matter what systems you have in place or how many hundreds of deploys you've done, there's always a new way for things to break.