Working on improving security...

Status
Not open for further replies.

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Beardie name(s)
Cailyth, Pinky, & Brain
[Edit: If you get an "Forbidden 403" error message unexpectedly, please reply here or email [email protected]]

Hello all,

I am working on finding new ways to improve the security of the site (and server itself). This includes ways to reduce spam on the forums. Ever since we migrated to a new server, different methods of protections went into place. With the old server, we had some protections well-tuned. However, with some of the newer software, that hasn't happened quite yet. So we're seeing more spam on the site, for one. But also, I'm concerned about possible compromises to the server (after all I also use the system for my own email and such, as well as other sites).

The main thing I wanted to mention is that IF you get some sort of error, especially something like "Access Denied" or "Forbidden", please let me know. Reply on this thread with the details (what you were trying to do; link if possible; copy paste error would be nice). If you can't reply here (i.e. being blocked), email me at [email protected].

Forbidden pages should provide you with your IP address. Having that will help me as well.

I will try to run in "log only" mode for a while, so hopefully that won't impact anyone. But eventually I'll start tightening things up. Some of you may have been impacted when I first turned it on (wasn't in log only mode). Once I saw what looked like false positive blocking, I disabled the active blocking.

Thank you for understanding any brief impacts. I will do my best to keep it to a minimum.

Thanks,
-Alex
 

CooperDragon

BD.org Sicko
Staff member
Moderator
Thanks Alex. I was wondering about the uptick in spam bot postings recently. Please let me know if you need help testing anything.
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
Thanks. I'll certainly let ya know. For now, I'm just gathering data to figure out which attack signatures to use, or which I may need to either tune back or disable. One of the default rules thinks there's an attempt to do a cross site scripting attack just for using a BBCode in the forums... heh.

I also think that when I get the black list stuff back in place, that will also help reduce stuff quite a bit.
 

Rampagingraptor

Sub-Adult Member
Out of curiosity, How are the bots getting through, anyways? Are they attempting to just access the servers directly, or are they just making throw away accounts? Or something else entirely?
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
Rampagingraptor":y500vhtb said:
Out of curiosity, How are the bots getting through, anyways? Are they attempting to just access the servers directly, or are they just making throw away accounts? Or something else entirely?

Both... although, primarily just by registering accounts.

The degree of attack from bots is kind of insane. A good thing is that Security is my day job, and also something I enjoy doing as a hobby. ... You should see my plans for my home network when I get some time for that. :)

Anyway, largely we see attempts to register accounts from all over the place; Russia, China, Belarus, Brazil, etc. In some cases, these seem more scattered in IP ranges, such as might be the case with individual compromised computers. In several cases, it's clear that a full subnet is being used... either compromised or owned/managed by the attackers (or a platform leased to attackers).

Aside from attempts to get forum spam on here by registering accounts, they are also attempting to compromise the server. By getting an account, they're able to see more "input" fields on the site, including posting forms. So it increases the possible attack surface they can try to hack at. I see SQL injection attempts constantly. These are attempts to execute database commands to either modify or steal data from the database. By stealing password fields, they can try to find ways to escalate their privileges, especially if they can compromise a moderator or admin account. And then further try to compromise the server.

There are also other direct attacks on the server, trying to exploit vulnerabilities in Apache, the mail server, PHP... whatever level they can. Much of this is automated, so it's really just suites of tools iterating through their various attacks, keeping track of results, and escalating further with any knowledge they gain from earlier attacks. It's frightening how advanced and automated "hacking" has gotten.

So... what am I doing? Here are some things...

Application Firewall
I've been spending the last week or so tuning a suite of detections for these sort of attacks. During that time, they've been running mostly in "Log only" mode. Now that I think I've reasonably ruled out the false positives (rules that would have broken people's normal use of the site), I've decided to activate them so they now BLOCK access when an attack is detected.

This mostly helps protect the site against the more direct attacks, trying to compromise the database or the system itself.

Real-time Blacklists
This is essentially a database of IPs that have been seen doing bad things. I have some lists I use for my email server, to help reduce spam. So those lists would keep track of IPs known for sending spam. I also use some lists that try to detect and track IPs seen creating forum spam, or otherwise as malicious in relation to only websites (stealing email addresses, etc.). Most of these lists rate the "severity" or confidence that an IP is bad. When an IP exceeds a certain threshold, I block it. When really high, I don't let the IP even read the site. When in a more gray area, I let them read, but not "POST" (which includes any form submission, including registering).

I also am maintaining my own blacklist. This is something I am still working to refine. But largely, I will be looking for regular "bad" behavior being blocked, and will escalate blocking bad IPs myself for any access to the site (not just when the request triggers an automated detections). I'm also hoping to set it up so that when we (the moderators and I) see and remove clear automated forum spam (not something where a real user violates policy or is being "spammy"), the system will identify all IPs that that account has used (whether registering, posting, or logged in from) and add them to the blacklist.

I will also keep a whitelist. These are to help when some other block (whether a blacklist or another more general restriction) is affecting a real user who would like to use the site. I can whitelist them so they can still enjoy the site.

I use my own black and white lists for my email server as well, so if the 3rd party services don't know an IP is bad (or good), I build my own to reduce spam.

Country Blacklisting
I generally don't like the idea of this, but it's one of the only ways to really keep up. There are numerous countries that I really have very little expectation of valid users from. However, these countries are a very high source of attacks and spam. I'm sure you can imagine a few that are well known for such things, sadly. The server is a low enough load (relative to its power) that I can actually perform a IP-to-country lookup for every web request (each image, each page, each ad, etc.).

I block a handful of countries outright. This is something I've done for years now, but with a more manually maintained list. My new method is more automated to keep up-to-date. Over that time, I've had real users contact me to let me know they got blocked. I add those network ranges to the whitelist mentioned above.

General Firewall Controls
So, aside from the application firewall above (which inspects and detects based on the web traffic), I also monitor for general malicious activity against the server... this includes port scanning (a method where an attacker scans for network service ports to see what's open and to try and identify what sort of system and versions you are running on), brute force password cracking attempts (on any exposed authentication service), and others.

More...
There is more that I do, but that's the basics of it. I don't mind posting publicly what sort of basic controls I have on the site... To me, obscurity of one's controls isn't an ideal form of security. I am not going to disclose every fine detail, but at a high-level... I'm trying to keep things safe.

Obviously, stuff like "patching" and applying updates to the operating system and all other software installed is important... especially any software that is exposed to the Internet (web server, email server, etc.).

Most of these blocks take someone to the standard "403 Forbidden" page for the site. This page explains that they've been denied access. It briefly explains why and that it is POSSIBLE it is a mistake. And if this mistake occurs to contact me to resolve it. I provide them with their IP address (since most people don't know what their public IP address is) and as that if they contact me to provide me that information. This let's me try to resolve false positive blocking quickly.


So... that's about it. It is now active. And in the time that I've typed this, I've seen about 80 attempts to compromise the web server by sending it invalid requests (detected and blocked), a few attempts to register from blocked countries, and probably about half a dozen attempts to register from IPs known for forum spam.

I am REALLY hoping that these controls, and the few more I have in mind to help the moderators and I feed into the system, reduce spam on the site, and generally provide people with a more pleasant and secure experience on here. :)

Cheers,
-Alex
 

CooperDragon

BD.org Sicko
Staff member
Moderator
Thanks Alex!

We see attacks just like you're describing on our VoIP servers all the time and have to take similar countermeasures. The country blacklisting works much better for phone servers, especially if no international calling is expected.
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
CooperDragon":2wc4fze9 said:
We see attacks just like you're describing on our VoIP servers all the time and have to take similar countermeasures. The country blacklisting works much better for phone servers, especially if no international calling is expected.
Yeah. VoIP is a major target. VoIP servers are so often overlooked for being updated, and people often expose them to the Internet for "easy management". It's crazy how much can be done with a VoIP server. With companies that have intellectual property they really care about, VoIP is often used as a means to exfiltrate data. VoIP is often ignored by most security controls and left alone for fear of interrupting phone service. What's worse, people commonly put VoIP on the same networks as their computers rather than separate them out to an isolated network like they should be.

We had to spend a chunk of time revamping VoIP here when we started the security organization. We still often find VoIP systems plugged into the wrong ports, and networking configured to let that work (because they don't want to be the reason something isn't working). My thinking is that it isn't working because people plugged it into the wrong place.

It's complicated. But there are definitely concerns with how much of this stuff gets implemented and the lack of controls on them.
 

anastasia6988

Hatchling Member
I was in the introduction section and attempted to post a reply, I received the below error.

Forbidden [Error 403] The page or function you attempted to access has been denied. If you are receiving this error, it is most likely due to a ban that has been put in place on your ISP's address space. The access attempt has been recorded. If you feel you have reached this page in error, please try again. If you feel that the block on your access has been made by mistake, please contact the website administrator, providing the IP address listed below and your username (desired username if new to the site), and what you were trying to do when you received this error message (login, register, etc.). Your IP Address is: 68.84.19.151
Source: posting.php?mode=reply&f=44&t=222839
Copyright © BeardedDragon.org

And sorry I PM'd you first, I just read through this more carefully.

Thank you
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
@anastasia6988 - Resolved. I disabled that specific rule on the forums. It appears that normal use is triggering it.
 

Rampagingraptor

Sub-Adult Member
Thanks for all that info! I actually went to college for Information Assurance (Eastern decided to rename a major, but basically just like any network security degree), so I knew alot of that, but it is really nice to have a bit of a refresher, learn some more info I hadn't quite learned yet, and see how you run you're system! Still, I feel so inexperienced and lacking compared to you in the field. XP
Honestly, I haven't had many opportunities to really get more into the experience side of things yet because I haven't gotten a chance to try a job like it yet. My current job is literally just monitoring some robots to make sure they don't fail on things for some hospitals. A bit important, but not really what I want to do my whole life.
Anyways, I know its not really the major problem, but have you thought of adding a captcha to add a little bit more security from robots? I figure it probably cost a monthly sort of fee, but I would be interested to hear your thoughts on it.
 

PogoLover

Juvie Member
I got blocked from responding to my own thread in Beardie Tales (RIP Sweet Pogo; Welcome Home Happy Go LUCKY). I posted just fine on that thread last night, but this morning I could not. I tried multiple times to make sure that it wasn't just because I was coming into the site through my e-mail notifications. Sorry - I didn't copy the message that told me that it wouldn't accept my post. If I try again and it happens again, I will be sure to copy the message.
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
@Pogo: The rule that was causing that has been disabled for the forums. Hopefully that should take care of the issue you were seeing. Someone else reported the same problem via email. I really wonder what characters or combination of characters are triggering this rule. Many possible causes are things that might look like database commands or database comments, such as a double dash (--), or even apostrophes ('). Most of the ones that were a little too broad like that I disabled, but it seems there are still SOME combinations that may trigger it. But hopefully soon, we will have things tuned nicely to catch as much bad stuff as possible without affecting you all. :)

Thanks for reporting it!

-Alex
 

PogoLover

Juvie Member
beardie":wm8x43p5 said:
@Pogo: The rule that was causing that has been disabled for the forums. Hopefully that should take care of the issue you were seeing. Someone else reported the same problem via email. I really wonder what characters or combination of characters are triggering this rule. Many possible causes are things that might look like database commands or database comments, such as a double dash (--), or even apostrophes ('). Most of the ones that were a little too broad like that I disabled, but it seems there are still SOME combinations that may trigger it. But hopefully soon, we will have things tuned nicely to catch as much bad stuff as possible without affecting you all. :)

Thanks for reporting it!

-Alex

Thank you!!!
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
Rampagingraptor":1683qlg3 said:
Thanks for all that info! I actually went to college for Information Assurance (Eastern decided to rename a major, but basically just like any network security degree), so I knew alot of that, but it is really nice to have a bit of a refresher, learn some more info I hadn't quite learned yet, and see how you run you're system! Still, I feel so inexperienced and lacking compared to you in the field. XP
I've been doing just Information Security for about 11 years now. Have had my CISSP for almost 5 years. I've had the benefit of being part of a small group that was responsible for the policy-level governance and program management for a large company. We leveraged the IT resources that managed the systems themselves. So, I got less hands-on experience with security tech at work. I have the training, and experience to plan. But much of my hands on experience with tools like these is with this site or my own home network. I have to understand the professional systems involved in order to do incident response and forensic analysis, so that has also been helpful professionally.

It's always been a tough balance between being more on the policy side and tech side. Currently, I work for a smaller company than I did before, so I have to have my hands a little more hands on too.

Anyways, I know its not really the major problem, but have you thought of adding a captcha to add a little bit more security from robots? I figure it probably cost a monthly sort of fee, but I would be interested to hear your thoughts on it.
Captchas are pretty much useless these days. The bot networks are far too good at solving them. I've been using question/answer combinations. I just looked into improving the Q&A, and found I was making classic mistakes of using questions that are easily "Google-able", which is what many bots do now... google the question and generate answers based on Google. Many also recommend one question, and if you see new bots getting in, alter the question to be more difficult. I am currently using one that cannot be googled, but anyone registering should know the answer to. "What sort of reptile is this site about?" There are multiple ways to answer it that are supported. Hopefully that's easy enough for humans, but hard enough for bots. :)
 
Status
Not open for further replies.

Members online

Latest resources

Latest posts

Latest profile posts

I miss you so much, Amaris 💔
What is a quick way to warm up a cold beardie? His heating element went out overnight and now he's very cold.
Pearl Girl wrote on moorelori1966's profile.
i feel so sad reading your about me 😢
Clapton is acclimating okay I think. He's quick as lightning so I'm not sure how much I should bring him out of his house yet. He's not at all interested in his salad though. I wonder if I should change what I'm giving him. Least he's eating his crickets.

Things to do:
Buy calcium powder
Material to raise surface for basking spot
Scenery decals for back of tank

Forum statistics

Threads
155,899
Messages
1,255,686
Members
75,965
Latest member
williamyoung
Top Bottom