Page 1 of 1

Site Slowness

PostPosted: Tue Apr 13, 2021 3:32 pm
by beardie
Hey all,

I noticed today that the site became pretty slow to respond. Something seems to have happened a few days ago that slowly depleted our processing reserve (CPU/compute allowance on AWS). It needs to recoup a little spare processing power before I can further diagnose. I'll take a closer look tonight to see what has caused this increased CPU usage and address it.

For now, your patience is appreciated.

Thanks,
-Alex

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 4:32 pm
by beardie
I'm still working on this, but will enable the site again for now. It will be slow. It seems that there was a massive increase in traffic, but it looks suspiciously like something crawling the site and copying content in mass... which is loading down the database. And unfortunately, it looks like it is coming from many IP addresses at the same time. I wouldn't call it a distributed denial of service, as an attempt at that would certainly take us down (we're not on a super powerful server). But it is distributed. That makes it harder to block.

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 6:08 pm
by Valkyrie47
That's scary o.O why would someone attack us 😭
We just need help with our beardies!

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 6:32 pm
by Claudiusx
Thanks for the update. Luckily you are just the man for the job lol.

-Brandon

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 6:35 pm
by CooperDragon
Valkyrie47 wrote:That's scary o.O why would someone attack us 😭
We just need help with our beardies!


Attacks like that are often automated. Not fun at all to deal with, but not a personal or targeted attack necessarily.



Alex, please let me know if you need any help and I'll do what I can.

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 8:04 pm
by beardie
Valkyrie47 wrote:That's scary o.O why would someone attack us 😭
We just need help with our beardies!

I don't think it's truly an "attack", as much as it is someone trying to mirror content or index it (like Google), but that is doing so in an underhanded manner... not sure why, except to avoid being blocked easily.

I have noticed that the user agent (the data that identifies the version of the browser, plugin info, etc.) is VERY similar for all their requests and doesn't seem to overlap with regular users. I've just implemented a block on that and am waiting to see if that works. They can easily get around it, but I can adjust. Hopefully they are just using tools and ignorant as to how to be more sneaky. I think they're likely just using tools with little technical understanding of them.

Re: Site Slowness

PostPosted: Tue Apr 13, 2021 8:30 pm
by beardie
It looks like the blocks are effective.. we shall see. This graph shows that the forbidden response code started being sent for their requests (orange on the right).

[Click image to enlarge]


And this shows just how noisy they were. You can also see they probed a day or two before they went nuts.

[Click image to enlarge]


And this AWS graph shows that our CPU cycles are recuperating. So, overall, looking better. I'll feel better, though, when that CPU credit is up into the 500's (our normal levels).

[Click image to enlarge]


Hopefully, they'll just go away.

Re: Site Slowness

PostPosted: Wed Apr 14, 2021 4:58 pm
by CooperDragon
I'm not super familiar with hosting on AWS. Do they have anything similar to pfBlocker that can maintain blacklists via anti-spam databases and/or geoblock databases? I figure geo blocking would be sticky with a site like this, but perhaps the IPs that were accessing the site are in one of the anti spam databases and could be filtered out.

Re: Site Slowness

PostPosted: Fri Apr 16, 2021 12:03 am
by beardie
CooperDragon wrote:I'm not super familiar with hosting on AWS. Do they have anything similar to pfBlocker that can maintain blacklists via anti-spam databases and/or geoblock databases? I figure geo blocking would be sticky with a site like this, but perhaps the IPs that were accessing the site are in one of the anti spam databases and could be filtered out.

They don't offer much for such things with how I have the server setup. We have an EC2 instance, which is like a virtual server that you do things for yourself with. I prefer that in most ways.

Even if they had black lists, the traffic we were getting was too broad. I was rather surprised at how spread out it was, but they were clearly coordinated. I do have geo-location blocking in place for the site, and other measure to protect the site from spam on the forums. But general traffic... that's a harder thing to guard. I think there were some 60 class C subnets (up to 255 IPs per subnet) involved, with anywhere from 10 to 150 IPs actually being used per subnet. So, it was probably nearly 1000 IP addresses seen in the 2-3 days.

I am glad that I noticed the consistency of their user-agent. That made it easy to block. It may have to be adjusted again (before we get to a depletion of CPU credits; which did take a few days, but I wasn't looking that closely). I doubt they'll put in the effort to diversify the user-agents for all their nodes. It did some variation, but minimal, and easy enough to write a rule to block that.

Re: Site Slowness

PostPosted: Sat Apr 17, 2021 12:45 am
by Drache613
Hello Alex,

I am not real familiar with a lot of that computer language, but understand enough on how
to block certain areas, addresses, etc.
I think that the entire internet & banking systems are being affected due to political moves
going on right now.
The site is doing well though! I can't imagine the behind the scenes work that it takes to keep
it all smooth.

Tracie

Re: Site Slowness

PostPosted: Sat Apr 17, 2021 12:09 pm
by beardie
Thanks Tracie.

The "attacks" stopped about 8 hours after I blocked it. I guess they realized they were caught.

I did get an email from a regular user that was blocked by this method. Upon review, I think in the last few days there were maybe 4 people affected that weren't part of the attack. That's not too bad, given how many visitors we have per day.

I've removed the block now though, as the attack stopped, and there's no need to block regular people who might be inadvertently impacted.