[Complete] Website and App maintenance!
-
@trutherford2388 that is not generally how servers work. It is rare for infrastructure to be entirely mutually interchangeable servers. Even more so if they are deploying a software or configuration refactor that is not backwards compatible, downtime is way better than data corruption caused by running incompatible software.
-
@Flan-Latte we need a support group...
-
@trutherford2388 ideally that would be the best. but you know some stuff just cannot be partially updated like transactional thingy unless it can be disable or something first. I don't know what they are updating tho, however it could be really big backend stuff that it need to be down to simplify what already complicated work.
-
@SomeOldGuy Hey, aren't all we old codgers supposed to be retired. I think somebody forgot to tell us to stop working...
-
Ok... Was purposefully trying npt to step into this; but, yes, there are multiple ways thus could probably been avoided if they had just known 'X'... But they didn't; and honestly I'm still pretty Ok with my experience overall... though my withdrawal symptoms have just not set deep enough... yet :-P
-
Most productive day in years... It was awful.
-
@Thomas-Smith-yjw56xl Codeine and whiskey will help your withdrawal symptoms. Trust me...
-
@shyevsa They're switching to an entirely new (and hopefully better) database backend to support all the things they want to be able to do with tags and scheduling, so it's not something they would have been able to leave the site up for.
-
It would be pretty simple to keep things up if they were only doing something like rebooting systems. System resilience is a thing, after all.
But that's not what they are doing.
It's possible they could have kept some kind of "legacy" version of existing systems up, but that's a whole headache in itself and it makes any kind of fail back super messy. Much better to bite the bullet and shut everything down.
-
@afdsfafa I've only ever worked at 1 company that did not run backup servers and they paid a heavy financial cost and a hit to their reputation when they tried upgrading their servers and caused a crash. Our entire VoIP system and CMS (Customer Management Software) was down. There was no data updates to the DBs (DataBases) and it took 1 week to get all the systems fully operational and about another month for the data from the week long outage to finally be updated.
Every other company I worked at has had backup servers. The company uses server B to upgrade or do maintenance, test all the changes in house before flipping the servers from customers using Server A to Server B and just disconnect Server A from customer usage for about 1-2 days before doing the same to Server A as what was done to Server B.
Also, industry standard for any web based company is to have redundancy servers not only setup, but also physically located several miles apart. (When the east coast lost internet for about a week due to power outages from a hurricane, big tech companies switched to having servers setup something like Server A in Texas and Server B in the east coast somewhere to account for natural disasters.)
Long story short, only one of those starter companies should have a Server setup without any form of redundancy in place to prevent outages.
-
@karasutengu ... might I suggest a combination of Starlink & EcoFlow power solutions.
I'm not sure if I could make it otherwise. -
I don't mind the outage honestly, even the extended no-ETA one. However, some feedback:
- Given it's a long (and apparently, risky) maintenance, more than the one day advanced notice would have been good
- Posting the notice of upcoming maintenance on the main site and in the app would have been ideal (only found out what was going on after things didn't work)
- A better maintenance page on the main site, with status updates (it wasn't immediately obvious to go here for information)
- Show a maintenance page in the app (right now it just returns errors)
- Perhaps roll back if things are not going as expected
- It's usually industry standard that risky things are done off-hours, even though it means nights and weekends
Again, not bothered, just observations from someone that has been doing this for a while.
-
Too late to read anything tonight, hopefully the site will be back up tomorrow. Some of my favorite series comes out on Friday so fingers crossed. Good night. 💤
-
@Rahul-Balaggan how's it going?
-
@Geezer-Weasalopes ok, someone needs to start writing that 😀
-
@jpwong Such things are possible but they always cost money.
For example, spinning up a database server using the most recent backup data and putting that in read-only mode. This would allow folks at least read old content until the new database is ready. (Even if nothing new was published and folks couldn't buy anything, folks like me who have gotten distracted reading the Catchup Series wouldn't have been impacted.)
With the database in read-only mode, there would be no risks of things being out-of-sync when the fancy new updates had finished deploying.
Honestly, if it truly is an entirely new database backend, it might have been even easier to set it to read-only mode while the new database got set up. I mean, if that system was already going to be frozen for the data migration it wouldn't have even cost money...
Hopefully things will be resolved before morning... This has got to be super stressful for the folks working there.
-
@karasutengu the whiskey helped all right, yummy macallan 12!
-
@Anthony-Berardinelli-kbgtvo8-0 Welft, I'm on myyy twenth glasst righth now, and I'm fiielling no pain, aha ha.
-
@karasutengu said in [Extended] Website and App maintenance!:
@Thomas-Smith-yjw56xl Codeine and whiskey will help your withdrawal symptoms. Trust me...
…Was that a codeine/coding pun? Because coding and whiskey might indeed help. Mostly the coding, in my opinion, but your mileage might vary.
-
@Phredis The pun was unintended, but does work pretty well, eh? Congrats on catching it.
PS I'm having trouble typing right now due to my booze buzz, so I'm glad I'm not a coder.
