Site Slowness

If you have any comments, questions, suggestions, or problems with the BeardedDragon.org Website, please post them here. Forum Community related topics and the like are also welcome here!

Site Slowness

Postby beardie » Tue Apr 13, 2021 3:32 pm

Hey all,

I noticed today that the site became pretty slow to respond. Something seems to have happened a few days ago that slowly depleted our processing reserve (CPU/compute allowance on AWS). It needs to recoup a little spare processing power before I can further diagnose. I'll take a closer look tonight to see what has caused this increased CPU usage and address it.

For now, your patience is appreciated.

Thanks,
-Alex
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA
- Advertisement -
- Advertisement -

Re: Site Slowness

Postby beardie » Tue Apr 13, 2021 4:32 pm

I'm still working on this, but will enable the site again for now. It will be slow. It seems that there was a massive increase in traffic, but it looks suspiciously like something crawling the site and copying content in mass... which is loading down the database. And unfortunately, it looks like it is coming from many IP addresses at the same time. I wouldn't call it a distributed denial of service, as an attempt at that would certainly take us down (we're not on a super powerful server). But it is distributed. That makes it harder to block.
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA

Re: Site Slowness

Postby Valkyrie47 » Tue Apr 13, 2021 6:08 pm

That's scary o.O why would someone attack us 😭
We just need help with our beardies!
Valkyrie47
Hatchling Poster
 
Posts: 67
Joined: Sun Dec 13, 2020 1:18 pm

Re: Site Slowness

Postby Claudiusx » Tue Apr 13, 2021 6:32 pm

Thanks for the update. Luckily you are just the man for the job lol.

-Brandon
Follow along with all my beardies. Check out my thread here!: Claud's Crew
P.S. We have lots of pictures ;)
Click here if you use a Sunblaster fixture - IMPORTANT
User avatar
Claudiusx
Moderator
Moderator
 
Posts: 19697
Joined: Tue Jun 02, 2009 3:44 pm
Location: California

Re: Site Slowness

Postby CooperDragon » Tue Apr 13, 2021 6:35 pm

Valkyrie47 wrote:That's scary o.O why would someone attack us 😭
We just need help with our beardies!


Attacks like that are often automated. Not fun at all to deal with, but not a personal or targeted attack necessarily.



Alex, please let me know if you need any help and I'll do what I can.
Image
User avatar
CooperDragon
Moderator
Moderator
 
Posts: 27306
Joined: Thu Mar 27, 2014 8:41 am
Location: Madison, WI

Re: Site Slowness

Postby beardie » Tue Apr 13, 2021 8:04 pm

Valkyrie47 wrote:That's scary o.O why would someone attack us 😭
We just need help with our beardies!

I don't think it's truly an "attack", as much as it is someone trying to mirror content or index it (like Google), but that is doing so in an underhanded manner... not sure why, except to avoid being blocked easily.

I have noticed that the user agent (the data that identifies the version of the browser, plugin info, etc.) is VERY similar for all their requests and doesn't seem to overlap with regular users. I've just implemented a block on that and am waiting to see if that works. They can easily get around it, but I can adjust. Hopefully they are just using tools and ignorant as to how to be more sneaky. I think they're likely just using tools with little technical understanding of them.
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA
- Advertisement -
- Advertisement -

Re: Site Slowness

Postby beardie » Tue Apr 13, 2021 8:30 pm

It looks like the blocks are effective.. we shall see. This graph shows that the forbidden response code started being sent for their requests (orange on the right).

[Click image to enlarge]


And this shows just how noisy they were. You can also see they probed a day or two before they went nuts.

[Click image to enlarge]


And this AWS graph shows that our CPU cycles are recuperating. So, overall, looking better. I'll feel better, though, when that CPU credit is up into the 500's (our normal levels).

[Click image to enlarge]


Hopefully, they'll just go away.
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA

Re: Site Slowness

Postby CooperDragon » Wed Apr 14, 2021 4:58 pm

I'm not super familiar with hosting on AWS. Do they have anything similar to pfBlocker that can maintain blacklists via anti-spam databases and/or geoblock databases? I figure geo blocking would be sticky with a site like this, but perhaps the IPs that were accessing the site are in one of the anti spam databases and could be filtered out.
Image
User avatar
CooperDragon
Moderator
Moderator
 
Posts: 27306
Joined: Thu Mar 27, 2014 8:41 am
Location: Madison, WI

Re: Site Slowness

Postby beardie » Fri Apr 16, 2021 12:03 am

CooperDragon wrote:I'm not super familiar with hosting on AWS. Do they have anything similar to pfBlocker that can maintain blacklists via anti-spam databases and/or geoblock databases? I figure geo blocking would be sticky with a site like this, but perhaps the IPs that were accessing the site are in one of the anti spam databases and could be filtered out.

They don't offer much for such things with how I have the server setup. We have an EC2 instance, which is like a virtual server that you do things for yourself with. I prefer that in most ways.

Even if they had black lists, the traffic we were getting was too broad. I was rather surprised at how spread out it was, but they were clearly coordinated. I do have geo-location blocking in place for the site, and other measure to protect the site from spam on the forums. But general traffic... that's a harder thing to guard. I think there were some 60 class C subnets (up to 255 IPs per subnet) involved, with anywhere from 10 to 150 IPs actually being used per subnet. So, it was probably nearly 1000 IP addresses seen in the 2-3 days.

I am glad that I noticed the consistency of their user-agent. That made it easy to block. It may have to be adjusted again (before we get to a depletion of CPU credits; which did take a few days, but I wasn't looking that closely). I doubt they'll put in the effort to diversify the user-agents for all their nodes. It did some variation, but minimal, and easy enough to write a rule to block that.
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA

Re: Site Slowness

Postby Drache613 » Sat Apr 17, 2021 12:45 am

Hello Alex,

I am not real familiar with a lot of that computer language, but understand enough on how
to block certain areas, addresses, etc.
I think that the entire internet & banking systems are being affected due to political moves
going on right now.
The site is doing well though! I can't imagine the behind the scenes work that it takes to keep
it all smooth.

Tracie
User avatar
Drache613
Moderator
Moderator
 
Posts: 41680
Joined: Fri Feb 10, 2006 9:33 pm
Location: May, Texas

Re: Site Slowness

Postby beardie » Sat Apr 17, 2021 12:09 pm

Thanks Tracie.

The "attacks" stopped about 8 hours after I blocked it. I guess they realized they were caught.

I did get an email from a regular user that was blocked by this method. Upon review, I think in the last few days there were maybe 4 people affected that weren't part of the attack. That's not too bad, given how many visitors we have per day.

I've removed the block now though, as the attack stopped, and there's no need to block regular people who might be inadvertently impacted.
Not a vet, just a concerned former keeper
Image
User avatar
beardie
Site Administrator
Site Administrator
 
Posts: 10887
Joined: Tue Nov 27, 2001 1:47 pm
Location: Orange County, CA, USA
- Advertisement -
- Advertisement -


Return to Website Comments

Who is online

Users browsing this forum: No registered users