NOTICE: Website Maintenance Tonight [COMPLETE]

Status
Not open for further replies.

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Beardie name(s)
Cailyth, Pinky, & Brain
Hello all,

Some of you know that once in a while, I perform some major maintenance that requires the website to be down for a little while. Well, tonight it is that time again.

Downtime Start: Appx. 8 PM PST
Downtime End: Appx 11 PM PST (hopefully much sooner)

Results:
- BeardedDragon.org will be running on a more stable server environment
- A fix to the database has been applied to stop the Unicode errors!!!
- You should otherwise not notice a difference other than it continues to work!

Issues:
- Please reply here or email me at beardie at bearded dragon dot org if you experience any issues on the site. I'm trying to catch everything, but over the years, little details don't get tracked as well as I would have liked and may get missed.


For those wanting to know the details...

I've been working on building out a new server for the site. While our old one still works just fine and is powerful enough for the load we have, it is aging a little. Being a physical server with only a single hard drive, there is the risk that the hard drive may fail. In addition, we're pretty overdue on updating the operating system and major server software components (mail server, etc.). I've applied all the minor updates, but not the major upgrades to the software or OS. And for the operating system, it's very difficult (if not impossible) to actually perform a major version upgrade on a remote server.

Sooo... the solution... we're going CLOUD! For many, this is a buzz-word that has been done to death. I've actually never liked the term. But ... well... it's the truth. I've opted to migrate to a virtual server in the Amazon EC2 cloud.

This gives me a few key benefits (among others):
- Ability to upgrade or downgrade (cost savings) the basic resources as needed (CPU, memory, storage)
- Self-service changes (nice to be able to easily set things up without a huge order processing process)
- Better resource monitoring

At this point, I've already prepared the server and migrated the email server, and all of my other websites (personal sites mainly). BeardedDragon.org is the last because it is the biggest and the more complicated setup.


For those wanting the REAL details...

I'm running the new server with:
- CentOS 7 (latest updates)
- Apache 2.4
- MariaDB (replacing MySQL)

In addition, I'm part of the beta group for the Let's Encrypt effort. Some of you know that I switched the website over to use SSL encryption a little while back (two years, I think now?). SSL Certificates can cost money (not a lot, but some) and have typically involved an overly complicated process. Let's Encrypt is a significant effort heavily supported by the EFF (Electronic Frontier Foundation, whom I support) as an initiative to get ALL websites using encryption. Doing so helps reduce having compromised passwords, and other disclosure people don't realize happens all the time when using a non-SSL website. Let's Encrypt provides SSL certificates free of charge! And they make it such that the certificates can be obtained immediately, and renewed in an automatic fashion while maintaining a secure practice.

So, with Let's Encrypt, I'm able to use proper TLS certs for BeardedDragon.org and every one of my websites (before only had it for BD.org). I also can now have properly encrypted connections for all email sent/received.

Along those lines, and with the much newer Apache version, I'm supporting HSTS. This tells your web browser that if you just type "beardeddragon.org" into your web browser, without specifying "https://", that your browser should assume https by default for it. That way, it never tries to use
"http" (unless you explicitly tell it to). There is a minor security risk in http while redirecting to https, and HSTS helps remove that risk and is generally a safer practice is your site is all encrypted.

I'm also taking measures to ensure the forum data is safe from a catastrophic failure. Data is backed up nightly to another cloud service, using CrashPlan (by Code42).

Along the security line of thinking, I'm installing Snort. This is an intrusion detection system. It will monitor all network activity on the server and look for various patterns of potential compromise or communications with known bad systems. I can then take action to block access or otherwise resolve the concern.

Similarly, I'm installing ModSecurity, which is like Snort, but more focused on web server traffic and identifying attacks against the website itself. I've used ModSecurity on the current site for some time, and it does help block attempts to compromise the server.

There are numerous other tech that goes into the behind the scenes... I'll just list of some names for those who care... firewalld (with rules to block all outbound traffic except as specifically allowed), Splunk (for log monitoring and centralization), Greylisting (reduces spam I get so I see your support emails).

As you may guess (if you didn't already know), I'm a security guy. Really... it's my day job. So, when implementing the server for BeardedDragon.org I want to try and be as secure as I can with it.

The BD.org forum software still will need to be updated, but I will handle that later.

Cheers,
-Alex
 

CooperDragon

BD.org Sicko
Staff member
Moderator
Wow, that's quite a lot of moving parts behind the curtain! I'm glad to hear you're moving away from a server with a single drive. If you weren't already moving to Amazon & Crashplan (good move IMO) I would have offered to build a NAS. Please keep us posted on how it goes (especially the migration to MariaDB) and let me know if there's anything I can do to help.
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
I could build a NAS, but it was hosted. So, I'd then have to do a colo for the NAS.

I had Crashplan running already for years to ensure data is backed up well. And before that, I was rsync'ing all important data to an old (and cheap) shared hosting account. BCDR (Business Continuity Disaster Recovery) isn't lost on me. In fact, I'll be helping (not my primary program) support building our BCDR program here at my semi-new job (last 6 months). My primary areas of ownership in security at my current job has been application security, data protection, and access control. Secondary (to varying degrees) on everything else for InfoSec. And we all are in the boat for policy development (the joys of that).

Anyway...rabbit-hole...

There's actually plenty more complexity to it all. I'm actually working on simplifying the implementation to something more uniform. I was recently borrowed as a resource by our infrastructure group to architect a LAMP architecture. I spent a good month researching some of the current tech and rethinking best practices. So, much of what I'm implementing here are based on the things I did for that (organizationally). I figured if I'm going to subject the developers here to that structure, I might as well eat my own dog food with my own stuff. :)

MariaDB migration should be seamless. Maria is binary compatible with MySQL. It all pretty much works the same. It was created by the same guy who created MySQL. Actually, Maria is his other daughter (My was the first one). But MariaDB builds in support for advanced features MySQL has struggled with being able to have implemented. I doubt I'll be using some of that, but nice to know. Red Hat and CentOS officially switched to MariaDB for default database support instead of MySQL.

I expect it will all go smoothly. And that when done, I'll find a few little issues with back-end scripts missing dependencies or paths having changed. And I expect a few permissions issues of files, since I will be changing how permissions work between the Apache process and the website directories (security measures). Eventually, I may play around with a module that Red Hat considers "experimental" (thus can't implement in corp prod environment) that lets the full Apache request process run as specified users for each site. That way there isn't one Apache user that has access to all sites. Instead, I would have a web user for each site, different than the user account that owns the files. This would make things quite secure from a web process perspective.

:)
 

CooperDragon

BD.org Sicko
Staff member
Moderator
Very cool. I'd heard of MariaDB after it emerged from MySQL after that was bought up by Oracle but I haven't messed with it yet. I would imagine it's pretty much the same. Good luck with the migration!
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
Okay! I THINK I'm all done... I hope! :)

It took a lot longer to transfer and apply the database from the older server to the new, but it did finally finish.

PLEASE, if you experience any issues with the site, or oddities, please let me know by replying here. I often find it can be useful for such things to be for all to see, as others can often confirm whether it happens for them too.

Thank you for your patience. I know that some of you get pretty addicted to your evening BD.org fix. ;)

Oh! And check this out!! ☔️ <-- That right there is an emoji character... like smileys. You should now be able to post any emoji character without errors. Most people don't realize that what they type gets converted into an emoji character sometimes and they got unexpected and confusing errors. Hopefully, those are a thing of the past now... AND, if you find out how to paste emoji with your keyboard (often easier with mobile devices), you now have a wider range of simple graphic images you can post with. :)

Cheers!
-Alex
 

Drache613

BD.org Sicko
Staff member
Moderator
Hello Alex,

It looks like it all went smoothly! :D
I wish I more fully understood all of the data & security stuff, along with programming, etc. I have just never done any of that before. It sounds interesting though!
I have noticed that the site seems to be running faster though, have you?


Thanks,
Tracie
 

beardie

BD.org Sicko
Staff member
Administrator
Moderator
Founder
Original Poster
Beardie name(s)
Cailyth, Pinky, & Brain
Yeah. I think that it actually may be a bit faster. I noticed it with the backups that ran last night. I think the hard drives with this virtual server are much faster than the older server. A backup that used to take 15-20 minutes (including compressing it) only took 1 minute last night. I was shocked. Of course, that meant I could reduce the nightly "outage" window from 30 minutes to 5 minutes, which is also nice.

If the disk is faster, that means that all things would seem faster (loading the website code, reading the data out of the database, etc.), resulting in a snappier website. :)
 

Drache613

BD.org Sicko
Staff member
Moderator
Hello Alex,

Wow, that is a lot faster then for the backup maintenance. That is actually kind of surprising, too.
You work really hard keeping it all under control here. We really appreciate it. :D
I am happy to hear the disc drive is much improved from the older one.

Tracie
 
Status
Not open for further replies.

Members online

Latest resources

Latest profile posts

I miss you so much, Amaris 💔
What is a quick way to warm up a cold beardie? His heating element went out overnight and now he's very cold.
Pearl Girl wrote on moorelori1966's profile.
i feel so sad reading your about me 😢
Clapton is acclimating okay I think. He's quick as lightning so I'm not sure how much I should bring him out of his house yet. He's not at all interested in his salad though. I wonder if I should change what I'm giving him. Least he's eating his crickets.

Things to do:
Buy calcium powder
Material to raise surface for basking spot
Scenery decals for back of tank

Forum statistics

Threads
155,897
Messages
1,255,669
Members
75,965
Latest member
williamyoung
Top Bottom