Out of curiosity, How are the bots getting through, anyways? Are they attempting to just access the servers directly, or are they just making throw away accounts? Or something else entirely?
Both... although, primarily just by registering accounts.
The degree of attack from bots is kind of insane. A good thing is that Security is my day job, and also something I enjoy doing as a hobby. ... You should see my plans for my home network when I get some time for that.
Anyway, largely we see attempts to register accounts from all over the place; Russia, China, Belarus, Brazil, etc. In some cases, these seem more scattered in IP ranges, such as might be the case with individual compromised computers. In several cases, it's clear that a full subnet is being used... either compromised or owned/managed by the attackers (or a platform leased to attackers).
Aside from attempts to get forum spam on here by registering accounts, they are also attempting to compromise the server. By getting an account, they're able to see more "input" fields on the site, including posting forms. So it increases the possible attack surface they can try to hack at. I see SQL injection attempts constantly. These are attempts to execute database commands to either modify or steal data from the database. By stealing password fields, they can try to find ways to escalate their privileges, especially if they can compromise a moderator or admin account. And then further try to compromise the server.
There are also other direct attacks on the server, trying to exploit vulnerabilities in Apache, the mail server, PHP... whatever level they can. Much of this is automated, so it's really just suites of tools iterating through their various attacks, keeping track of results, and escalating further with any knowledge they gain from earlier attacks. It's frightening how advanced and automated "hacking" has gotten.
So... what am I doing? Here are some things...
I've been spending the last week or so tuning a suite of detections for these sort of attacks. During that time, they've been running mostly in "Log only" mode. Now that I think I've reasonably ruled out the false positives (rules that would have broken people's normal use of the site), I've decided to activate them so they now BLOCK access when an attack is detected.
This mostly helps protect the site against the more direct attacks, trying to compromise the database or the system itself.
This is essentially a database of IPs that have been seen doing bad things. I have some lists I use for my email server, to help reduce spam. So those lists would keep track of IPs known for sending spam. I also use some lists that try to detect and track IPs seen creating forum spam, or otherwise as malicious in relation to only websites (stealing email addresses, etc.). Most of these lists rate the "severity" or confidence that an IP is bad. When an IP exceeds a certain threshold, I block it. When really high, I don't let the IP even read the site. When in a more gray area, I let them read, but not "POST" (which includes any form submission, including registering).
I also am maintaining my own blacklist. This is something I am still working to refine. But largely, I will be looking for regular "bad" behavior being blocked, and will escalate blocking bad IPs myself for any access to the site (not just when the request triggers an automated detections). I'm also hoping to set it up so that when we (the moderators and I) see and remove clear automated forum spam (not something where a real user violates policy or is being "spammy"), the system will identify all IPs that that account has used (whether registering, posting, or logged in from) and add them to the blacklist.
I will also keep a whitelist. These are to help when some other block (whether a blacklist or another more general restriction) is affecting a real user who would like to use the site. I can whitelist them so they can still enjoy the site.
I use my own black and white lists for my email server as well, so if the 3rd party services don't know an IP is bad (or good), I build my own to reduce spam.
I generally don't like the idea of this, but it's one of the only ways to really keep up. There are numerous countries that I really have very little expectation of valid users from. However, these countries are a very high source of attacks and spam. I'm sure you can imagine a few that are well known for such things, sadly. The server is a low enough load (relative to its power) that I can actually perform a IP-to-country lookup for every web request (each image, each page, each ad, etc.).
I block a handful of countries outright. This is something I've done for years now, but with a more manually maintained list. My new method is more automated to keep up-to-date. Over that time, I've had real users contact me to let me know they got blocked. I add those network ranges to the whitelist mentioned above.
General Firewall Controls
So, aside from the application firewall above (which inspects and detects based on the web traffic), I also monitor for general malicious activity against the server... this includes port scanning (a method where an attacker scans for network service ports to see what's open and to try and identify what sort of system and versions you are running on), brute force password cracking attempts (on any exposed authentication service), and others.
There is more that I do, but that's the basics of it. I don't mind posting publicly what sort of basic controls I have on the site... To me, obscurity of one's controls isn't an ideal form of security. I am not going to disclose every fine detail, but at a high-level... I'm trying to keep things safe.
Obviously, stuff like "patching" and applying updates to the operating system and all other software installed is important... especially any software that is exposed to the Internet (web server, email server, etc.).
Most of these blocks take someone to the standard "403 Forbidden" page
for the site. This page explains that they've been denied access. It briefly explains why and that it is POSSIBLE it is a mistake. And if this mistake occurs to contact me to resolve it. I provide them with their IP address (since most people don't know what their public IP address is) and as that if they contact me to provide me that information. This let's me try to resolve false positive blocking quickly.
So... that's about it. It is now active. And in the time that I've typed this, I've seen about 80 attempts to compromise the web server by sending it invalid requests (detected and blocked), a few attempts to register from blocked countries, and probably about half a dozen attempts to register from IPs known for forum spam.
I am REALLY hoping that these controls, and the few more I have in mind to help the moderators and I feed into the system, reduce spam on the site, and generally provide people with a more pleasant and secure experience on here.