We've all been there - your Drupal site is being overwhelmed by people and robots trying to post to your site, or trying to create accounts. These attempts can really hammer your web server, and it's the sort of traffic that isn't going to be helped any by the thirty-five different kinds of caching you have set up on your site.
We've seen thsi recently get much worse with some of our clients, so we set out to do something about it that's simple and reliable. We currently use fail2ban on all of our servers to try to cut back on the attempts at breaking in to the servers through ssh, ftp, etc. Fail2ban reads your server logs, and looks for matching patterns. If a pattern matches, the server can carry out any number of activities, including temporarily banning an IP address through iptables. We thought "why not use fail2ban to block bots that are hammering the server?" So - we came up with this simple plan.
On our servers that have been having particular difficulty, we have implemented a couple of fail2ban rules designed to protect the web server from incoming spam traffic and email address crawlers. The email address crawler rule is part of the fail2ban package, and we just enabled it.